RESEARCH Adapted Physical Activity Quarterly, 2014, 31, 193-218 http://dx.doi.org/10.1123/apaq.2013-0084 © 2014 Human Kinetics, Inc.
Gauging the Quality of Qualitative Research in Adapted Physical Activity Michelle R. Zitomer and Donna Goodwin University of Alberta Qualitative inquiry is increasingly being used in adapted physical activity research, which raises questions about how to best evaluate its quality. This article aims to clarify the distinction between quality criteria (the wltat) and strategies (the how) in qualitative inquiry. An electronic keyword search was used to identify articles pertaining to quality evaluation published between 1995 and 2012 (n = 204). A five phase systematic review resulted in the identification of 56 articles for detailed review. Data extraction tables were generated and analyzed for commonalities in terminology and meanings. Six flexible criteria for gauging quality were for mulated: reflexivity, credibility, resonance, significant contribution, ethics, and coherence. Strategies for achieving the established criteria were also identified. It is suggested that researchers indicate the paradigm under which they are working and guidelines by which they would like readers to evaluate their work as well as what criteria can be absent without affecting the research value. Keywords: criteria, strategies, trustworthiness, rigor, evaluation Q ualitative research m ethods are becom ing more com m only used internation ally across varied disciplines (Easterby-Sm ith, G olden-Biddle, & Locke, 2008; Elliott, Fischer, & Rennie, 1999; Pitney, 2004; Tem ple, 1998; W hittem ore, Chase, & M andle, 2001). The history o f qualitative research in adapted physical activity arguably dates back to Connolly (1994). H er article “Practicum experiences and journal w riting in adapted physical activity: Im plications for teacher education” was the first qualitative study published in the A dapted Physical Activity Q uarterly (APAQ ). Since this publication, there has been an increasing trend in qualitative papers published in APAQ. The average num ber o f qualitative studies published each year in APAQ between the years 1996 and 2013 was 3.4, with a range o f 1-7 articles. Furtherm ore, in the past 3 years, a consistent increase can be seen from four qualitative studies published in 201 1, to five in 2012, and six in 2013. In sum mary, o f the 119 articles published in APAQ between January 2007 and January 2014, 19% were qualitative, and an additional 5% used mixed m ethods (Causgrove Dunn & Cairney, 2014).
Michelle R. Zitomer and Donna Goodwin are with the University of Alberta in Edmonton. Alberta. Canada. Address author correspondence to Michelle Zitomer at zitomer@ ualberta.ca.
193
194
Zitomer and Goodwin
Contrary to quantitative research that aims to verify theory or earlier conclu sions, qualitative research aims to enrich understanding of human experience and the meaning of actions taken within social and cultural contexts (Elliott et al., 1999; Fossey, Harvey, McDermott, & Davidson, 2002; Long & Godfrey, 2004; Pitney, 2004). Qualitative research methodology is particularly valuable to the field of adapted physical activity, because it enables individuals with disabilities opportunities to voice their experiences and opinions related to their participation in various physical activity contexts. It thus has emancipatory potential by enabling these individuals to exercise their human right to be heard (Fisher, 2012). Qualitative research tends to be subjective, contextual, and ethically more complex than quantitative research because of the integral role the researcher and participants play through their involvement in the research process and the investi gated social context (de Witt & Ploeg, 2006; Lincoln, 1995; Pitney, 2004; Temple, 1998). In the course of their work, qualitative researchers have been tasked with justifying their research to counter allegations that it is undisciplined, atheoretical, anecdotal, methodologically weak, and subjectively indulgent (Denzin & Lincoln, 2008; Long & Godfrey, 2004). To address contentions on the quality of their work, qualitative research ers developed criteria (the “what” of evaluation) against which quality of their research would be judged. In 1985, Lincoln and Guba introduced four criteria for trustworthiness: credibility, transferability, dependability, and confirmability. In 1989, they suggested authenticity criteria of fairness: ontological authenticity, educative authenticity, catalytic authenticity, and tactile authenticity. In 1995, Lincoln published additional criteria that “recognize and validate relationships between the inquirer and those who participate in the inquiry” (p. 278), thereby highlighting the coconstruction of knowledge and the emergence of increasingly diverse qualitative approaches. These criteria included positionality, community, voice, critical subjectivity, reciprocity, sacredness, and sharing the perquisites of privilege. Lincoln’s (1995) article is considered pivotal in the development of qualitative research criteria because it was the first to establish the importance of ethical relationships between the researcher and the participants from the moment the researcher enters the field and throughout a qualitative study process (Lincoln, Lynham, & Guba, 2011). Since the introduction of qualitative criteria, a plethora of terms have emerged. In some cases, multiple terms were used for the same criteria, and in other cases, strategies (i.e., approaches used for achieving the criteria) were incorrectly identi fied as criteria, resulting in confusion among students, readers, and reviewers of qualitative research about what constitutes quality qualitative research (Creswell & Miller, 2000; Morse, Barrett, Mayan, Olson, & Spiers 2002). Given the confusion, it is somewhat understandable that at the time of the writing of this article, the official journal of the International Federation of Adapted Physical Activity, APAQ, did not provide guidelines for journal contributors or reviewers of qualitative submissions. Quality criteria discussions have enriched the complexity of qualitative research methodology and research practice while increasing its responsiveness, richness, and sensitivity both politically and ethically (Lincoln, 1995; Tracy, 2010). Criteria become an expression of quality and serve as a means by which to demonstrate integrity, competence, and the legitimacy of the research process and findings. Criteria also provide basis for understanding events, taking action, and designing
Gauging the Q uality of Qualitative Research
195
future research, as well as evaluating strengths and weaknesses of a research design (Finlay, 2006; Fossey et al„ 2002; Hammersley, 2007; Lazaraton, 2003; Tobin & Begley, 2004). Demonstration of quality reflects researchers’ meticulous investment of time, effort, and care in a thorough process encompassing engage ment of abundant theoretical constructs, data sources, and samples to investigate the phenomena of interest (Savall, Zardet, Bonnet, & Peron, 2008; Tracy, 2010). Criteria represent the core values of inquiry that help us learn, practice, and perfect our craft as students, early-career researchers, readers, and reviewers of qualitative inquiry (Tracy, 2010). Caution is nevertheless needed in thinking about the selection and application of criteria given the paradigm one is working under. Some criteria may apply to specific research approaches but not others, during some periods of the research process and not others, and at all times are relational (Lincoln, 1995; Sparkes & Smith, 2009). Criteriology refers to reliance on regulative norms (criteria) for removing doubt about what is correct or incorrect (good or bad) or the quest for identification of and reliance on definitive criteria for appraising all forms of qualitative research (Emden, Hancock. Schubert, & Darbyshire, 2001; Schwandt, 1996; Sparkes & Smith, 2009). Some have refuted the use of any criteria (checklists) or rule-based judgments for ascertaining goodness because of the multiplicity of methodologies, approaches, and theories underpinning qualitative inquiry in favor of reflexive dia logue (Bochner, 2000; Easterby-Smith et ah, 2008; Stige, Malterud, & Midtgarden, 2009). Often checklists contain strategies, also referred to as mechanisms (Morse et ah, 2002; Temple, 1998) or techniques (Caelli, Ray, & Mill, 2003; Cohen & Crabtree, 2008; Lewis, 2009; Mays & Pope, 1995; Whittemore et ah, 2001), for achieving criteria. For example member checks, triangulation, and methodologi cally relevant sampling strategies have been used to bring legitimacy to qualitative inquiry (Creswell & Miller 2000; Lewis, 2009; Lietz, Langer, & Furman, 2006; Long & Godfrey, 2004; Tracy, 2010). It has been argued that methodological decisions cannot be appraised by uni versal rules because of the "blurring of genres” that has occurred over time as social scientists learned more about qualitative approaches (Denzin & Lincoln, 2005). Schwandt (1996) further stated that we should not look to “following procedures or defining or specifying the right criteria” (p. 70). Instead, he suggested we examine our research practices, as well as the consequences, and outcomes that decisions we make in the research process can have for participant reflections. Moreover, a focus on checklists may inhibit researchers from applying “critical intelligence” to their research endeavors (Schwandt, 1996, p. 69), which may result in a more methoddriven, predictable, and less dynamic research process (Sparkes & Smith. 2009). Ravenek & Rudman (2013) affirm however, that “flexible criteria are needed that are unique to, and unify, qualitative work and, at the same time, are sensitive to diversity within and between paradigms and methodological approaches” (p. 447). We are not proponents of reliance on checklists, but we agree with Ravenek & Rudman that flexible guidelines that are paradigmatically sensitive are needed to connect qualitative researchers in their pursuit of understanding the meaning of human action while also providing space for new methods to emerge (Pilnick & Swift, 2011). Qualitative research in adapted physical activity is characterized by a wide diversity of research approaches including grounded theory, phenomenology, narrative, case study, and ethnography. Therefore, it is important for researchers
196
Zitomer and Goodwin
to become familiar with the variety of research paradigms that underlie different research approaches.
Qualitative Research Paradigms “Paradigms matter when considering how to assess quality because these basic assumptions guide what a researcher seeks to know, how they approach knowing, and what they believe are the best ways to know” (Ravenek & Rudman, 2013, p. 440). Therefore, we offer a brief introduction of research paradigms. A research paradigm encompasses the researcher’s world view based on ontological, episte mological, and methodological assumptions (Guba & Lincoln, 1994). Lincoln et al. (2011) identified five research paradigms: positivism, postpositivism, critical theory, constructivism, and participatory. Given that positivism is primarily associ ated with quantitative research approaches (Morrow, 2005), this overview will focus on postpositivism, critical theory, constructivism, and participatory paradigms.
Postpositivism Postpositivism adopts (a) a critical realism ontology that is based on the idea that reality needs to be subject to critical examination to the extent possible, (b) a modi fied objectivist epistemology, and (c) a modified experimental methodology that focuses on falsifying hypothesis. In this methodology, research moves to natural settings and collection of situated data (Guba & Lincoln, 1994). Quality criteria established for postpositivist qualitative research include credibility (parallel to internal validity), transferability (parallel to generalizability), dependability (parallel to reliability), and confirmability (parallel to objectivity). These criteria have also been associated with early phases in constructivism (Morrow, 2005). Critical Theory Critical theory adopts a historical realism ontology that assumes reality, once per ceived as abstract, can be shaped by social, political, cultural, economic, ethnic, and gender values to be perceived as real. The epistemology is transactional and subjectivist, which means that the known is inseparable from the interaction between the researcher and the participants. The methodology thus is dialogic in a research process that seeks to uncover and critique historical understandings of experiences of suffering, conflict, or collective struggles to transform reality. Quality criteria for this paradigm are the extent to which inquiry is historically situated, acts to diminish ignorance, and inspires action (Guba & Lincoln, 1994). Constructivism Constructivism adopts relativist ontology that holds that reality depends on the context in which it is created (Ravenek & Rudman, 2013), a transactional and subjectivist epistemology, and a hermeneutic, dialectical methodology. Inquiry under the constructivist paradigm aims to produce reconstructed understanding that is considered by some to be more informed and sophisticated than previous constructed understandings (Guba & Lincoln, 1994). The criteria and strategies
Gauging the Quality of Qualitative Research
197
of this paradigm replaced the positivistic notions of internal and external validity, reliability, and objectivity (Pitney, 2004). However, authenticity criteria suggested by Guba and Lincoln ( 1989), along with researcher reflexivity, are considered most appropriate for research under the constructivism paradigm (Morrow, 2005).
Participatory The most recent paradigm is participatory (Lincoln et al., 2011). Like constructiv ism, the participatory paradigm adopts a relativist ontology but emphasizes the co created nature of reality (Ravenek & Rudman, 2013). The epistemology is critical subjectivity in which “the knower participates in the known, articulates a work, in at least four interdependent ways: experiential, presentational, propositional, and practical’ (Heron & Reason, 1997, p. 280). Experiential knowing is knowing through empathic resonance that is participative. Presentational knowing builds on experiential knowing and is expressed through aesthetic creation and musical, vocal, verbal, or movement art forms. Propositional knowing entails knowing things are what they are in conceptual terms that are grounded in experiential articulation of the world. Finally, practical knowing is knowing how to do something. Participa tory methodology involves collaborative forms of inquiry engaging all members in democratic dialogue in which participants are coresearchers and the researcher is a coparticipant (Heron & Reason, 1997). It is important to note that paradigms matter not only when choosing a research approach but also when choosing a specific strand to follow within a given approach. For example, grounded theory has strands that are associated with postpositivism if one chooses an approach based on Glaser or Strauss. However, Charmaz offers a grounded theory approach that is situated in constructivism, but has also been adopted by some researchers working under critical theory paradigm pursuing social justice inquiry (Charmaz, 2011). Our aim here is not to provide a comprehensive historical review of qualitative criteria, but to acquaint readers with criteria and strategies that may be applicable to qualitative research in adapted physical activity. In so doing, we hope to inspire continued dialogue in our field as it advances in its pursuit for research excellence (Lundengren, 1984). The purpose of this article is to review and synthesize criteria for evaluating the quality of qualitative research and to identify strategies toward their achievement. We put forward a flexible and parsimonious framework that may facilitate researchers, readers, and reviewers of qualitative inquiry to constructively evaluate research quality.
Method We conducted a systematic review containing five phases: (a) locating relevant papers (search profile electronically completed), (b) deciding what to include (estab lishing inclusion and exclusion criteria and practical screening of the information tor relevance), (c) article appraisal (article review and summary), (d) data extraction and analysis (compare and contrast for connected understandings), and finally (e) synthesis of the findings and presentation of refined meanings (communicating with audience; Okoli & Schabram, 2010; Walsh & Downe, 2005).
198
Zitomer and Goodwin
Locating Relevant Papers The first author conducted the literature search, which involved an unrestricted keyword search of EBSCO, a multidisciplinary meta-database encompassing 63 databases, including those relevant to education, health sciences, and the humanities and social sciences. Because of the pivotal value of Lincoln’s (1995) criteria, which emphasizes ethical relations between the researcher and participants in qualitative research methodology (Lincoln et al., 2011), we delimited our search for articles to those published between 1995 and 2012. The keywords used were qualitative research or appraisal, or quality, or checklist, or critical, or criteria, or evaluate, or rigor, or assess, or syntheses, or standard, or trustworthiness. Parameters for limiting the search were key words in titles (IT), peer reviewed, scholarly journals published in English between January 1995 and June 2012. The subject limits were qualitative research and research methodology. The keyword search revealed 163 articles. To assure retrieval of articles that may have been absent from the EBSCO search, we searched specific journals for qualitative research individually, including Qualitative Inquiry, Qualitative Health Research, Qualitative Research in Sport, Exercise & Health, Qualitative Research, Qualitative Research in Psychology, Qualitative Research Journal, and Interna tional Journal o f Qualitative Methods. This search revealed 51 additional articles, 10 of which overlapped with the results of the EBSCO search. Thus a total of 204 articles were identified in this phase.
Deciding What to Include Practical screening was completed in two parts. In Part 1, the first author reviewed article titles for clear relevance. Articles were excluded if the titles did not pertain to qualitative research quality (e.g., article titles that discussed quality of life). Ninetyfour articles remained in the EBSCO search folder after the process of reviewing the titles. In Part 2, the first author reviewed abstracts from the EBSCO folder along with the 41 articles from the individual journal search for relevance to the research question. We determined inclusion criteria together, and articles were included if the abstracts contained quality criteria indicators, strategies for achieving quality, or both. Excluded were articles that (a) focused on only one aspect of quality (e.g., reflexivity, ethics) or one strategy (e.g., transcription, audit trail), (b) reflectively evaluated a project on the basis of existing criteria, (c) consisted of book reviews or commentaries, (d) discussed usefulness of tools designed to appraise qualitative research, and (e) did not explicitly discuss criteria or strategies. At the conclusion of the practical screening of abstracts, 39 articles remained in the EBSCO search folder, and 10 articles remained from the individual journal search (see Figure 1).
Article Appraisal In the article-appraisal phase, we reviewed the 49 selected articles, and the first author summarized them in a Microsoft Word document, describing their purpose, process, and findings. The change-tracking feature of Word was used to comment on the relevance of each article for identifying quality criteria for qualitative research. Through a forward and backward search of articles that offered clear criteria sets,
Figure 1 Deciding what to include.
199
200
Zitomer and Goodwin
we identified seven additional articles for appraisal (Levy & Ellis, 2006). A total of 56 articles were examined. In this phase, 31articles were found to offer specific quality criteria or strategies. Articles that offered no criteria/strategies or offered criteria focused on a specific aspect of the research process (e.g., data analysis) were excluded in this phase.
Data Extraction and Analysis Data extraction involved the use of the change-tracking feature to identify informa tion pertaining to criteria or strategies from each article. The first author copied the information into Word documents. Criteria items (i.e., “what” the researchers were attempting to achieve) were separated from strategy items (i.e., the “hows” or techniques for achieving the criteria) and placed into separate documents. The information was then read and reread, searching for commonalities. Items with similar intents were temporarily labeled. When the items were compared, contrasted, and clustered, six criteria emerged. The first author entered the criteria as temporary headings into summary tables and populated them with information from the articles. The second author then read all contributing papers, and the summary tables were edited by discussing discrepancies in use of labels, identification of an item as strategy or criteria, or location of information on the tables until agreement was reached. For example, we found reflexivity confusing because some papers addressed it as a strategy and others as a criterion. In attempting to identify whether it was a criterion or a strategy, we discussed ways in which reflexivity could be viewed as what was trying to be achieved versus how it could be achieved. It became clear that reflexivity can be viewed as both a criterion and a strategy because it can be seen as both a distinct quality of an article and as a connecting thread among all the criteria. Reflexiv ity describes the researcher throughout the process, including bias, assumptions, theoretical positioning, and interactions with participants. Through reflexivity we see how the research process took place (credibility), what the research has to contribute, what ethical issues arose and how they were addressed, and how the process taken is interconnected from beginning through to conclusion. Reflexivity can also have an impact on the article’s resonance with readers.
Synthesis Synthesis involved summarizing the findings that emerged from the data extrac tion and analysis process. We then finalized the six criteria labels on the basis of evaluation of terms most commonly used by different authors to denote criteria. The implications of these findings for students, readers, and reviewers of qualitative research were outlined in the form of this article.
Results The review process revealed a plethora of overlapping terms used to denote qual ity criteria and strategies, as well as the extent of the confusion existing between criteria (the what) and strategies (the how). Articles offering criteria did one of three
Gauging the Quality of Qualitative Research
201
things: (a) offered a clear criteria set, (b) offered criteria mixed with strategy, and (c) stated that they proposed criteria but instead offered strategies. Six criteria emerged from the review: reflexivity, credibility, significant contri bution, ethics, resonance, and coherence. The criteria are intertwined and together represent constructs for evaluating the quality of qualitative research. The findings are presented in three tables. The first table summarizes criteria by author (Table 1), the second outlines strategies for achieving the criteria by author (Table 2), and the third summarizes the six criteria by strategies (Table 3).
Reflexivity Reflexivity is a distinct quality of qualitative research. It involves researcher selfawareness, honesty, and genuineness with her/himself, the research process, and the readers (Morrow, 2005; Tracy, 2010; Walsh & Downe, 2006). It is the means by which the researcher realizes her/his integral part in the investigated phenomena (Horsburgh, 2003). Through reflexivity, the impact of a researcher’s professional and personal background, as well as theoretical assumptions brought to the study topic, design, analysis, and relationship with participants become evident (Cohen & Crabtree, 2008; Fossey et al. 2002; Gomez, 2009; Horsburgh, 2003; Pilnick & Swift, 2011). Thus, reflexivity renders a study transparent by requiring researchers to be methodologically conscious about the research process and self-critical about the personal and psychological states of participants and their actions (Lincoln, 1995; Pilnick & Swift, 2011; Seale. 1999; Sparkes, 1998). Other terms include “opening up the study to scrutiny” (de Witt & Ploeg, 2006, p. 225), continuous interaction and engagement with the phenomena (Stige et al., 2009), positionality (Lincoln, 1995), researcher emotional vulnerability and honesty (Bochner, 2000), audibil ity (Cooney, 2011), transparency (Fossey et al., 2002), criticality, and explicitness (Whittemore et al., 2001). It is important to note that reflexivity can also be seen as a strategy in which the researcher keeps a reflective journal of her/his experiences, assumptions, reac tions, or emerging awareness occurring throughout the entire research process. The journal can either be incorporated into the study or used for the researcher to examine her/himself and be set aside depending on the paradigmatic approach guiding the study (Morrow, 2005). Strategies to achieve reflexivity include owning one’s perspective by reflecting on personal biases and articulating theoretical posi tions and assumptions held (Elliott et al., 1999; Lewis, 2009; Long & Godfrey, 2004; Walsh & Downe, 2006). Reflexivity through activities such as journaling engages researcher reflection on motives and how personal history (research biases) shape and guide the research process, including the data analysis and interpretation (Caelli et al., 2003; Elliott et al., 1999; Lietz et al., 2006; Mays & Pope, 1995). An audit trail of memos, logs, field notes, computer files, and other relevant information creates opportunity to analyze and critique decisions made throughout the research process (Creswell & Miller, 2000; Horsburgh, 2003; Lewis, 2009; Lietz et al., 2006). Some terms that were identified as criteria but can more appropriately be considered as strategies to achieve reflexivity included, tor example, a narrative of two selves (Bochner, 2000) and attending to researcher bias (Cohen & Crabtree, 2008).
c 0 ^
0> C
o
• A story that moves • Ethical selfconsciousness
CD C
W) Qfi
u < Q g
C
3
-1)
o
> x" T3 03 c 3
i ig .3 £C/5 «O
.3
T3
—
xCD «C
CL
03
•—
C
>
5J
3"
CL
CD
O
> "S o 3 o
.2 x W 1)
= o « i! C £ '
*5 « sc Q
l
cl
£
c« 5
o (N 8 = •o U
c
_c
o iS
c •=
Q.
o c
W)
i_
•
£
o
c .2 3 X5 •c ^
W
c .2 3 ca C3 C c -*S 5 § .2 > .3 c ■n .«> « t3 CD c/D o 03 a:
O
£
3
^
5 = £2 p o o -3 g o -fa •pS T3 o .2 a. c c o o ■£ o £ U P CD C3 £ E &.
c* D
c 0>
co
^
c S ’§ 2 2 H > U
c o S'i V2 5 23 1) ■S/3 *5-» C C C£ C (D D 2
o
— QC 3 • o
3
•a 3
3
cc o CL E
T3
? fO
L*C Op 00
03 1
c
E
£
o
>v >»
u 0) 3 c
"S _a> *►a) DC
u. «
.2 C i
c § o _, -
-=8 s
03 C/3 . 2 v° “ - •— « 22 a3 T3
I• aN .
™ .33 ~ ■C - CT * i3 C3
.3 .£ oj
SC/5
’5
o O u >
0-
> 0XJ 30-3 Wj _X r. ^
~o w
> O X > 0) *C —
"O
jC
« c *5
1) d > =3J 3 3 £Sj *p I os IS Q a. Sg
•
TD >,
03 ^ c3 OJ d) C 1/5 •= CL “O
• • • a. D. —
o ■O £ £S .
u, x: w) y §C3 1 §— {— 2 =5 D Di/ . •o •-
«> •a
w
-i-
3 > -C E = c X M
d>
"C a) 3
.C c
d> -a c x c c 03 d> P 3 .& E 3 S O d) % d> «» 'E ^ *
o
o O
CM o A
.
3
CL E 1
x
J c « e- & « s i
“
* = S a £ < . H Q Q
A nalytic technique
S “ z s g i f ? s a
•
>> — !E 1
£
03
C
x u h v o us d>
o o
C
ao
3 O 2 & > X E- Q
d> « uu _ O ^ d>X c •S o ^ .3 ^ a
£ 5 d) g
.3 (N X . ^
15 207
"O C
03 •a ■5
5o ■O Cu o O -S a) ^: < .1 *>
.2
U « 6 . _ P 2 2 a. a5 . °O c E 1 o 03 «{ c/3 T3 go
’a
© 3
Gauging the Quality of Qualitative Research in Adapted Physical Activity Michelle R. Zitomer and Donna Goodwin University of Alberta Qualitative inquiry is increasingly being used in adapted physical activity research, which raises questions about how to best evaluate its quality. This article aims to clarify the distinction between quality criteria (the wltat) and strategies (the how) in qualitative inquiry. An electronic keyword search was used to identify articles pertaining to quality evaluation published between 1995 and 2012 (n = 204). A five phase systematic review resulted in the identification of 56 articles for detailed review. Data extraction tables were generated and analyzed for commonalities in terminology and meanings. Six flexible criteria for gauging quality were for mulated: reflexivity, credibility, resonance, significant contribution, ethics, and coherence. Strategies for achieving the established criteria were also identified. It is suggested that researchers indicate the paradigm under which they are working and guidelines by which they would like readers to evaluate their work as well as what criteria can be absent without affecting the research value. Keywords: criteria, strategies, trustworthiness, rigor, evaluation Q ualitative research m ethods are becom ing more com m only used internation ally across varied disciplines (Easterby-Sm ith, G olden-Biddle, & Locke, 2008; Elliott, Fischer, & Rennie, 1999; Pitney, 2004; Tem ple, 1998; W hittem ore, Chase, & M andle, 2001). The history o f qualitative research in adapted physical activity arguably dates back to Connolly (1994). H er article “Practicum experiences and journal w riting in adapted physical activity: Im plications for teacher education” was the first qualitative study published in the A dapted Physical Activity Q uarterly (APAQ ). Since this publication, there has been an increasing trend in qualitative papers published in APAQ. The average num ber o f qualitative studies published each year in APAQ between the years 1996 and 2013 was 3.4, with a range o f 1-7 articles. Furtherm ore, in the past 3 years, a consistent increase can be seen from four qualitative studies published in 201 1, to five in 2012, and six in 2013. In sum mary, o f the 119 articles published in APAQ between January 2007 and January 2014, 19% were qualitative, and an additional 5% used mixed m ethods (Causgrove Dunn & Cairney, 2014).
Michelle R. Zitomer and Donna Goodwin are with the University of Alberta in Edmonton. Alberta. Canada. Address author correspondence to Michelle Zitomer at zitomer@ ualberta.ca.
193
194
Zitomer and Goodwin
Contrary to quantitative research that aims to verify theory or earlier conclu sions, qualitative research aims to enrich understanding of human experience and the meaning of actions taken within social and cultural contexts (Elliott et al., 1999; Fossey, Harvey, McDermott, & Davidson, 2002; Long & Godfrey, 2004; Pitney, 2004). Qualitative research methodology is particularly valuable to the field of adapted physical activity, because it enables individuals with disabilities opportunities to voice their experiences and opinions related to their participation in various physical activity contexts. It thus has emancipatory potential by enabling these individuals to exercise their human right to be heard (Fisher, 2012). Qualitative research tends to be subjective, contextual, and ethically more complex than quantitative research because of the integral role the researcher and participants play through their involvement in the research process and the investi gated social context (de Witt & Ploeg, 2006; Lincoln, 1995; Pitney, 2004; Temple, 1998). In the course of their work, qualitative researchers have been tasked with justifying their research to counter allegations that it is undisciplined, atheoretical, anecdotal, methodologically weak, and subjectively indulgent (Denzin & Lincoln, 2008; Long & Godfrey, 2004). To address contentions on the quality of their work, qualitative research ers developed criteria (the “what” of evaluation) against which quality of their research would be judged. In 1985, Lincoln and Guba introduced four criteria for trustworthiness: credibility, transferability, dependability, and confirmability. In 1989, they suggested authenticity criteria of fairness: ontological authenticity, educative authenticity, catalytic authenticity, and tactile authenticity. In 1995, Lincoln published additional criteria that “recognize and validate relationships between the inquirer and those who participate in the inquiry” (p. 278), thereby highlighting the coconstruction of knowledge and the emergence of increasingly diverse qualitative approaches. These criteria included positionality, community, voice, critical subjectivity, reciprocity, sacredness, and sharing the perquisites of privilege. Lincoln’s (1995) article is considered pivotal in the development of qualitative research criteria because it was the first to establish the importance of ethical relationships between the researcher and the participants from the moment the researcher enters the field and throughout a qualitative study process (Lincoln, Lynham, & Guba, 2011). Since the introduction of qualitative criteria, a plethora of terms have emerged. In some cases, multiple terms were used for the same criteria, and in other cases, strategies (i.e., approaches used for achieving the criteria) were incorrectly identi fied as criteria, resulting in confusion among students, readers, and reviewers of qualitative research about what constitutes quality qualitative research (Creswell & Miller, 2000; Morse, Barrett, Mayan, Olson, & Spiers 2002). Given the confusion, it is somewhat understandable that at the time of the writing of this article, the official journal of the International Federation of Adapted Physical Activity, APAQ, did not provide guidelines for journal contributors or reviewers of qualitative submissions. Quality criteria discussions have enriched the complexity of qualitative research methodology and research practice while increasing its responsiveness, richness, and sensitivity both politically and ethically (Lincoln, 1995; Tracy, 2010). Criteria become an expression of quality and serve as a means by which to demonstrate integrity, competence, and the legitimacy of the research process and findings. Criteria also provide basis for understanding events, taking action, and designing
Gauging the Q uality of Qualitative Research
195
future research, as well as evaluating strengths and weaknesses of a research design (Finlay, 2006; Fossey et al„ 2002; Hammersley, 2007; Lazaraton, 2003; Tobin & Begley, 2004). Demonstration of quality reflects researchers’ meticulous investment of time, effort, and care in a thorough process encompassing engage ment of abundant theoretical constructs, data sources, and samples to investigate the phenomena of interest (Savall, Zardet, Bonnet, & Peron, 2008; Tracy, 2010). Criteria represent the core values of inquiry that help us learn, practice, and perfect our craft as students, early-career researchers, readers, and reviewers of qualitative inquiry (Tracy, 2010). Caution is nevertheless needed in thinking about the selection and application of criteria given the paradigm one is working under. Some criteria may apply to specific research approaches but not others, during some periods of the research process and not others, and at all times are relational (Lincoln, 1995; Sparkes & Smith, 2009). Criteriology refers to reliance on regulative norms (criteria) for removing doubt about what is correct or incorrect (good or bad) or the quest for identification of and reliance on definitive criteria for appraising all forms of qualitative research (Emden, Hancock. Schubert, & Darbyshire, 2001; Schwandt, 1996; Sparkes & Smith, 2009). Some have refuted the use of any criteria (checklists) or rule-based judgments for ascertaining goodness because of the multiplicity of methodologies, approaches, and theories underpinning qualitative inquiry in favor of reflexive dia logue (Bochner, 2000; Easterby-Smith et ah, 2008; Stige, Malterud, & Midtgarden, 2009). Often checklists contain strategies, also referred to as mechanisms (Morse et ah, 2002; Temple, 1998) or techniques (Caelli, Ray, & Mill, 2003; Cohen & Crabtree, 2008; Lewis, 2009; Mays & Pope, 1995; Whittemore et ah, 2001), for achieving criteria. For example member checks, triangulation, and methodologi cally relevant sampling strategies have been used to bring legitimacy to qualitative inquiry (Creswell & Miller 2000; Lewis, 2009; Lietz, Langer, & Furman, 2006; Long & Godfrey, 2004; Tracy, 2010). It has been argued that methodological decisions cannot be appraised by uni versal rules because of the "blurring of genres” that has occurred over time as social scientists learned more about qualitative approaches (Denzin & Lincoln, 2005). Schwandt (1996) further stated that we should not look to “following procedures or defining or specifying the right criteria” (p. 70). Instead, he suggested we examine our research practices, as well as the consequences, and outcomes that decisions we make in the research process can have for participant reflections. Moreover, a focus on checklists may inhibit researchers from applying “critical intelligence” to their research endeavors (Schwandt, 1996, p. 69), which may result in a more methoddriven, predictable, and less dynamic research process (Sparkes & Smith. 2009). Ravenek & Rudman (2013) affirm however, that “flexible criteria are needed that are unique to, and unify, qualitative work and, at the same time, are sensitive to diversity within and between paradigms and methodological approaches” (p. 447). We are not proponents of reliance on checklists, but we agree with Ravenek & Rudman that flexible guidelines that are paradigmatically sensitive are needed to connect qualitative researchers in their pursuit of understanding the meaning of human action while also providing space for new methods to emerge (Pilnick & Swift, 2011). Qualitative research in adapted physical activity is characterized by a wide diversity of research approaches including grounded theory, phenomenology, narrative, case study, and ethnography. Therefore, it is important for researchers
196
Zitomer and Goodwin
to become familiar with the variety of research paradigms that underlie different research approaches.
Qualitative Research Paradigms “Paradigms matter when considering how to assess quality because these basic assumptions guide what a researcher seeks to know, how they approach knowing, and what they believe are the best ways to know” (Ravenek & Rudman, 2013, p. 440). Therefore, we offer a brief introduction of research paradigms. A research paradigm encompasses the researcher’s world view based on ontological, episte mological, and methodological assumptions (Guba & Lincoln, 1994). Lincoln et al. (2011) identified five research paradigms: positivism, postpositivism, critical theory, constructivism, and participatory. Given that positivism is primarily associ ated with quantitative research approaches (Morrow, 2005), this overview will focus on postpositivism, critical theory, constructivism, and participatory paradigms.
Postpositivism Postpositivism adopts (a) a critical realism ontology that is based on the idea that reality needs to be subject to critical examination to the extent possible, (b) a modi fied objectivist epistemology, and (c) a modified experimental methodology that focuses on falsifying hypothesis. In this methodology, research moves to natural settings and collection of situated data (Guba & Lincoln, 1994). Quality criteria established for postpositivist qualitative research include credibility (parallel to internal validity), transferability (parallel to generalizability), dependability (parallel to reliability), and confirmability (parallel to objectivity). These criteria have also been associated with early phases in constructivism (Morrow, 2005). Critical Theory Critical theory adopts a historical realism ontology that assumes reality, once per ceived as abstract, can be shaped by social, political, cultural, economic, ethnic, and gender values to be perceived as real. The epistemology is transactional and subjectivist, which means that the known is inseparable from the interaction between the researcher and the participants. The methodology thus is dialogic in a research process that seeks to uncover and critique historical understandings of experiences of suffering, conflict, or collective struggles to transform reality. Quality criteria for this paradigm are the extent to which inquiry is historically situated, acts to diminish ignorance, and inspires action (Guba & Lincoln, 1994). Constructivism Constructivism adopts relativist ontology that holds that reality depends on the context in which it is created (Ravenek & Rudman, 2013), a transactional and subjectivist epistemology, and a hermeneutic, dialectical methodology. Inquiry under the constructivist paradigm aims to produce reconstructed understanding that is considered by some to be more informed and sophisticated than previous constructed understandings (Guba & Lincoln, 1994). The criteria and strategies
Gauging the Quality of Qualitative Research
197
of this paradigm replaced the positivistic notions of internal and external validity, reliability, and objectivity (Pitney, 2004). However, authenticity criteria suggested by Guba and Lincoln ( 1989), along with researcher reflexivity, are considered most appropriate for research under the constructivism paradigm (Morrow, 2005).
Participatory The most recent paradigm is participatory (Lincoln et al., 2011). Like constructiv ism, the participatory paradigm adopts a relativist ontology but emphasizes the co created nature of reality (Ravenek & Rudman, 2013). The epistemology is critical subjectivity in which “the knower participates in the known, articulates a work, in at least four interdependent ways: experiential, presentational, propositional, and practical’ (Heron & Reason, 1997, p. 280). Experiential knowing is knowing through empathic resonance that is participative. Presentational knowing builds on experiential knowing and is expressed through aesthetic creation and musical, vocal, verbal, or movement art forms. Propositional knowing entails knowing things are what they are in conceptual terms that are grounded in experiential articulation of the world. Finally, practical knowing is knowing how to do something. Participa tory methodology involves collaborative forms of inquiry engaging all members in democratic dialogue in which participants are coresearchers and the researcher is a coparticipant (Heron & Reason, 1997). It is important to note that paradigms matter not only when choosing a research approach but also when choosing a specific strand to follow within a given approach. For example, grounded theory has strands that are associated with postpositivism if one chooses an approach based on Glaser or Strauss. However, Charmaz offers a grounded theory approach that is situated in constructivism, but has also been adopted by some researchers working under critical theory paradigm pursuing social justice inquiry (Charmaz, 2011). Our aim here is not to provide a comprehensive historical review of qualitative criteria, but to acquaint readers with criteria and strategies that may be applicable to qualitative research in adapted physical activity. In so doing, we hope to inspire continued dialogue in our field as it advances in its pursuit for research excellence (Lundengren, 1984). The purpose of this article is to review and synthesize criteria for evaluating the quality of qualitative research and to identify strategies toward their achievement. We put forward a flexible and parsimonious framework that may facilitate researchers, readers, and reviewers of qualitative inquiry to constructively evaluate research quality.
Method We conducted a systematic review containing five phases: (a) locating relevant papers (search profile electronically completed), (b) deciding what to include (estab lishing inclusion and exclusion criteria and practical screening of the information tor relevance), (c) article appraisal (article review and summary), (d) data extraction and analysis (compare and contrast for connected understandings), and finally (e) synthesis of the findings and presentation of refined meanings (communicating with audience; Okoli & Schabram, 2010; Walsh & Downe, 2005).
198
Zitomer and Goodwin
Locating Relevant Papers The first author conducted the literature search, which involved an unrestricted keyword search of EBSCO, a multidisciplinary meta-database encompassing 63 databases, including those relevant to education, health sciences, and the humanities and social sciences. Because of the pivotal value of Lincoln’s (1995) criteria, which emphasizes ethical relations between the researcher and participants in qualitative research methodology (Lincoln et al., 2011), we delimited our search for articles to those published between 1995 and 2012. The keywords used were qualitative research or appraisal, or quality, or checklist, or critical, or criteria, or evaluate, or rigor, or assess, or syntheses, or standard, or trustworthiness. Parameters for limiting the search were key words in titles (IT), peer reviewed, scholarly journals published in English between January 1995 and June 2012. The subject limits were qualitative research and research methodology. The keyword search revealed 163 articles. To assure retrieval of articles that may have been absent from the EBSCO search, we searched specific journals for qualitative research individually, including Qualitative Inquiry, Qualitative Health Research, Qualitative Research in Sport, Exercise & Health, Qualitative Research, Qualitative Research in Psychology, Qualitative Research Journal, and Interna tional Journal o f Qualitative Methods. This search revealed 51 additional articles, 10 of which overlapped with the results of the EBSCO search. Thus a total of 204 articles were identified in this phase.
Deciding What to Include Practical screening was completed in two parts. In Part 1, the first author reviewed article titles for clear relevance. Articles were excluded if the titles did not pertain to qualitative research quality (e.g., article titles that discussed quality of life). Ninetyfour articles remained in the EBSCO search folder after the process of reviewing the titles. In Part 2, the first author reviewed abstracts from the EBSCO folder along with the 41 articles from the individual journal search for relevance to the research question. We determined inclusion criteria together, and articles were included if the abstracts contained quality criteria indicators, strategies for achieving quality, or both. Excluded were articles that (a) focused on only one aspect of quality (e.g., reflexivity, ethics) or one strategy (e.g., transcription, audit trail), (b) reflectively evaluated a project on the basis of existing criteria, (c) consisted of book reviews or commentaries, (d) discussed usefulness of tools designed to appraise qualitative research, and (e) did not explicitly discuss criteria or strategies. At the conclusion of the practical screening of abstracts, 39 articles remained in the EBSCO search folder, and 10 articles remained from the individual journal search (see Figure 1).
Article Appraisal In the article-appraisal phase, we reviewed the 49 selected articles, and the first author summarized them in a Microsoft Word document, describing their purpose, process, and findings. The change-tracking feature of Word was used to comment on the relevance of each article for identifying quality criteria for qualitative research. Through a forward and backward search of articles that offered clear criteria sets,
Figure 1 Deciding what to include.
199
200
Zitomer and Goodwin
we identified seven additional articles for appraisal (Levy & Ellis, 2006). A total of 56 articles were examined. In this phase, 31articles were found to offer specific quality criteria or strategies. Articles that offered no criteria/strategies or offered criteria focused on a specific aspect of the research process (e.g., data analysis) were excluded in this phase.
Data Extraction and Analysis Data extraction involved the use of the change-tracking feature to identify informa tion pertaining to criteria or strategies from each article. The first author copied the information into Word documents. Criteria items (i.e., “what” the researchers were attempting to achieve) were separated from strategy items (i.e., the “hows” or techniques for achieving the criteria) and placed into separate documents. The information was then read and reread, searching for commonalities. Items with similar intents were temporarily labeled. When the items were compared, contrasted, and clustered, six criteria emerged. The first author entered the criteria as temporary headings into summary tables and populated them with information from the articles. The second author then read all contributing papers, and the summary tables were edited by discussing discrepancies in use of labels, identification of an item as strategy or criteria, or location of information on the tables until agreement was reached. For example, we found reflexivity confusing because some papers addressed it as a strategy and others as a criterion. In attempting to identify whether it was a criterion or a strategy, we discussed ways in which reflexivity could be viewed as what was trying to be achieved versus how it could be achieved. It became clear that reflexivity can be viewed as both a criterion and a strategy because it can be seen as both a distinct quality of an article and as a connecting thread among all the criteria. Reflexiv ity describes the researcher throughout the process, including bias, assumptions, theoretical positioning, and interactions with participants. Through reflexivity we see how the research process took place (credibility), what the research has to contribute, what ethical issues arose and how they were addressed, and how the process taken is interconnected from beginning through to conclusion. Reflexivity can also have an impact on the article’s resonance with readers.
Synthesis Synthesis involved summarizing the findings that emerged from the data extrac tion and analysis process. We then finalized the six criteria labels on the basis of evaluation of terms most commonly used by different authors to denote criteria. The implications of these findings for students, readers, and reviewers of qualitative research were outlined in the form of this article.
Results The review process revealed a plethora of overlapping terms used to denote qual ity criteria and strategies, as well as the extent of the confusion existing between criteria (the what) and strategies (the how). Articles offering criteria did one of three
Gauging the Quality of Qualitative Research
201
things: (a) offered a clear criteria set, (b) offered criteria mixed with strategy, and (c) stated that they proposed criteria but instead offered strategies. Six criteria emerged from the review: reflexivity, credibility, significant contri bution, ethics, resonance, and coherence. The criteria are intertwined and together represent constructs for evaluating the quality of qualitative research. The findings are presented in three tables. The first table summarizes criteria by author (Table 1), the second outlines strategies for achieving the criteria by author (Table 2), and the third summarizes the six criteria by strategies (Table 3).
Reflexivity Reflexivity is a distinct quality of qualitative research. It involves researcher selfawareness, honesty, and genuineness with her/himself, the research process, and the readers (Morrow, 2005; Tracy, 2010; Walsh & Downe, 2006). It is the means by which the researcher realizes her/his integral part in the investigated phenomena (Horsburgh, 2003). Through reflexivity, the impact of a researcher’s professional and personal background, as well as theoretical assumptions brought to the study topic, design, analysis, and relationship with participants become evident (Cohen & Crabtree, 2008; Fossey et al. 2002; Gomez, 2009; Horsburgh, 2003; Pilnick & Swift, 2011). Thus, reflexivity renders a study transparent by requiring researchers to be methodologically conscious about the research process and self-critical about the personal and psychological states of participants and their actions (Lincoln, 1995; Pilnick & Swift, 2011; Seale. 1999; Sparkes, 1998). Other terms include “opening up the study to scrutiny” (de Witt & Ploeg, 2006, p. 225), continuous interaction and engagement with the phenomena (Stige et al., 2009), positionality (Lincoln, 1995), researcher emotional vulnerability and honesty (Bochner, 2000), audibil ity (Cooney, 2011), transparency (Fossey et al., 2002), criticality, and explicitness (Whittemore et al., 2001). It is important to note that reflexivity can also be seen as a strategy in which the researcher keeps a reflective journal of her/his experiences, assumptions, reac tions, or emerging awareness occurring throughout the entire research process. The journal can either be incorporated into the study or used for the researcher to examine her/himself and be set aside depending on the paradigmatic approach guiding the study (Morrow, 2005). Strategies to achieve reflexivity include owning one’s perspective by reflecting on personal biases and articulating theoretical posi tions and assumptions held (Elliott et al., 1999; Lewis, 2009; Long & Godfrey, 2004; Walsh & Downe, 2006). Reflexivity through activities such as journaling engages researcher reflection on motives and how personal history (research biases) shape and guide the research process, including the data analysis and interpretation (Caelli et al., 2003; Elliott et al., 1999; Lietz et al., 2006; Mays & Pope, 1995). An audit trail of memos, logs, field notes, computer files, and other relevant information creates opportunity to analyze and critique decisions made throughout the research process (Creswell & Miller, 2000; Horsburgh, 2003; Lewis, 2009; Lietz et al., 2006). Some terms that were identified as criteria but can more appropriately be considered as strategies to achieve reflexivity included, tor example, a narrative of two selves (Bochner, 2000) and attending to researcher bias (Cohen & Crabtree, 2008).
c 0 ^
0> C
o
• A story that moves • Ethical selfconsciousness
CD C
W) Qfi
u < Q g
C
3
-1)
o
> x" T3 03 c 3
i ig .3 £C/5 «O
.3
T3
—
xCD «C
CL
03
•—
C
>
5J
3"
CL
CD
O
> "S o 3 o
.2 x W 1)
= o « i! C £ '
*5 « sc Q
l
cl
£
c« 5
o (N 8 = •o U
c
_c
o iS
c •=
Q.
o c
W)
i_
•
£
o
c .2 3 X5 •c ^
W
c .2 3 ca C3 C c -*S 5 § .2 > .3 c ■n .«> « t3 CD c/D o 03 a:
O
£
3
^
5 = £2 p o o -3 g o -fa •pS T3 o .2 a. c c o o ■£ o £ U P CD C3 £ E &.
c* D
c 0>
co
^
c S ’§ 2 2 H > U
c o S'i V2 5 23 1) ■S/3 *5-» C C C£ C (D D 2
o
— QC 3 • o
3
•a 3
3
cc o CL E
T3
? fO
L*C Op 00
03 1
c
E
£
o
>v >»
u 0) 3 c
"S _a> *►a) DC
u. «
.2 C i
c § o _, -
-=8 s
03 C/3 . 2 v° “ - •— « 22 a3 T3
I• aN .
™ .33 ~ ■C - CT * i3 C3
.3 .£ oj
SC/5
’5
o O u >
0-
> 0XJ 30-3 Wj _X r. ^
~o w
> O X > 0) *C —
"O
jC
« c *5
1) d > =3J 3 3 £Sj *p I os IS Q a. Sg
•
TD >,
03 ^ c3 OJ d) C 1/5 •= CL “O
• • • a. D. —
o ■O £ £S .
u, x: w) y §C3 1 §— {— 2 =5 D Di/ . •o •-
«> •a
w
-i-
3 > -C E = c X M
d>
"C a) 3
.C c
d> -a c x c c 03 d> P 3 .& E 3 S O d) % d> «» 'E ^ *
o
o O
CM o A
.
3
CL E 1
x
J c « e- & « s i
“
* = S a £ < . H Q Q
A nalytic technique
S “ z s g i f ? s a
•
>> — !E 1
£
03
C
x u h v o us d>
o o
C
ao
3 O 2 & > X E- Q
d> « uu _ O ^ d>X c •S o ^ .3 ^ a
£ 5 d) g
.3 (N X . ^
15 207
"O C
03 •a ■5
5o ■O Cu o O -S a) ^: < .1 *>
.2
U « 6 . _ P 2 2 a. a5 . °O c E 1 o 03 «{ c/3 T3 go
’a
© 3
