323

Work 50 (2015) 323–334 DOI 10.3233/WOR-141982 IOS Press

Speaking of Research

Enhancing rigor and practice of scoping reviews in social policy research: Considerations from a worked example on the Americans with Disabilities Act Sarah Parker Harris, Robert Gould∗ and Glenn Fujiura Department of Disability and Human Development, University of Illinois-Chicago, Chicago, IL, USA

Abstract. BACKGROUND: There is increasing theoretical consideration about the use of systematic and scoping reviews of evidence in informing disability and rehabilitation research and practice. Indicative of this trend, this journal published a piece by Rumrill, Fitzgerald and Merchant in 2010 explaining the utility and process for conducting reviews of intervention-based research. There is still need to consider how to apply such rigor when conducting more exploratory reviews of heterogeneous research. OBJECTIVES: This article explores the challenges, benefits, and procedures for conducting rigorous exploratory scoping reviews of diverse evidence. METHODS: The article expands upon Rumrill, Fitzgerald and Merchant’s framework and considers its application to more heterogeneous evidence on the impact of social policy. RESULTS: A worked example of a scoping review of the Americans with Disabilities Act is provided with a procedural framework for conducting scoping reviews on the effects of a social policy. The need for more nuanced techniques for enhancing rigor became apparent during the review process. CONCLUSIONS: There are multiple methodological steps that can enhance the utility of exploratory scoping reviews. The potential of systematic consideration during the exploratory review process is shown as a viable method to enhance the rigor in reviewing diverse bodies of evidence. Keywords: Policy analysis, exploratory review, ADA, systematic review

1. Introduction Various systematic review techniques are used to consolidate evidence for a stakeholder group’s needs or to inform decision-making processes. While systematic reviews of evidence may be useful to share findings, reviews of social policy evidence often reflect opinions, beliefs, and goals of the group presenting the ∗ Corresponding author: Robert Gould, Department of Disability and Human Development (MC 626), 1640 W. Roosevelt Rd., Chicago, IL 60608-6904, USA. Tel.: +1 312 413 2299; E-mail: [email protected].

findings. Evidence can be susceptible to a range of criticism regarding how the researcher selects and generates conclusions about the existing evidence base [33]. In response to such criticism, an emerging body of literature in the area of rehabilitation and social science research has called for a more standardized approach to policy review [36]. Indicative of this trend, this journal recently published an article by Rumrill, Fitzgerald and Merchant [26] that describes the utility of scoping reviews for documenting trends in research on various rehabilitation interventions. They explain how rigorous scoping reviews are useful used to bring theory to praxis, and to reduce bias when informing stakehold-

c 2015 – IOS Press and the authors. All rights reserved 1051-9815/15/$35.00 

324

S.P. Harris et al. / Enhancing rigor and practice of scoping reviews in social policy research

ers about problems in existing policies or systems. Additionally, Rumrill, Fitzgerald and Merchant include a theoretical example of how to conduct a scoping review with a primary purpose of tracking shared research findings across studies. Their work provides a significant contribution to scholars interested in developing or utilizing scoping reviews by creating a methodological framework that can be built upon for reviewing and disseminating research. Indeed, their approach has been widely applied to scoping reviews covering a range of intervention-based research topics reported in the journal including: work capacity and return-to-work assessments [6]; knowledge transfer for youth using technology [35]; and identity formation in relation to occupation [4]. The framework developed by Rumrill, Fitzgerald and Merchant [26] adds a critical addition to the literature on the systematic review of intervention-based research. However, there is still a considerable knowledge gap on how to apply such rigor to more diverse types of evidence, such as the implementation of social policy. We argue in this article that exploratory reviews can similarly benefit from the systematic rigor identified in Rumrill, Fitzgerald and Merchant’s work, and when approached in this way, such a review can be used to identify gaps in existing research. We extend their framework to more exploratory reviews and assess its application for systematically reviewing heterogeneous bodies of research. To do this, we first consider the need for rigor in exploratory reviews, especially when they are conducted in advance of full systematic reviews. We then address the challenges and benefits of using non-interventionist reviews as a way of understanding the impact of social policy, and the extent to which the review process can inform future reviews. Finally, we offer a worked example of an exploratory scoping review of the Americans with Disabilities Act (ADA) research, and compare this review with the more results-based scoping reviews conducted for the purpose of disseminated findings (such as suggested by Rumrill, Fitzgerald and Merchant). We will turn now to a brief discussion of why conducting a scoping review in advance of a systematic review can impact various methodological considerations during the scoping stage.

2. Considering a scoping review within the context of a full systematic review As Rumrill, Fitzgerald and Merchant [26] note, exploratory reviews are most often considered precur-

sors to a larger systematic review project. So how does the methodological process of such a review change when it is planned in conjunction with a larger review project? While a review question may not immediately suggest the need for a full systematic review, scoping reviews are typically conducted because of the anecdotal state of evidence and a response to the demand for more systematic evidence [12]. When fragmented or disparate findings about the state of evidence exist, initial scoping reviews can set the stage to understand what evidence is available to inform future systematic inquiries. Scoping reviews can also provide useful summative accounts to identify research gaps by confirming or negating previous anecdotal claims on the state of evidence in an area of study. The potential benefit of generating findings useful for future systematic review from the scoping process is highly dependent on the degree of rigor and synthesis that goes into the initial review. Without heightened attention to rigor in this exploratory stage of the review process, the review can simply result in recreating weak claims and suggestions of existing evidence – a type of error that is sometimes referred to in systematic review research as the “garbage in, garbage out” phenomena [9]. While not a systematic review per se, the preplanned processes for review that Rumrill, Fitzgerald and Merchant [26] mention greatly enhances the transparency, rigor, and thus the utility of scoping findings. The value of evidence generated through the scoping review process to be used in future systematic review is largely dependent on the degree that the review team adheres to rigorous analysis techniques and scrupulous recording processes that result in more credible research findings. This realization applies to all stages of the research, including the initial scoping review. Personal beliefs and opinions tend to impact the selection of materials and the interpretations of findings in non-rigorous review. Opinions can present bias that is reflective of authors’ personal stances of what constitutes valid evidence to inform decisions. Even in very preliminary or investigatory reviews that seek to understand the broad expanse of literature on topic, the selection of literature and the choices of where to locate resource are potential sources of bias. Methodological conclusions deriving from reviews can easily become skewed in favor of the author’s personal worldview if there is not careful consideration to meeting the research problem with appropriate methods and research questions [14]. Some scholars critique the mis-assessment of evidence as reflective of poor aca-

S.P. Harris et al. / Enhancing rigor and practice of scoping reviews in social policy research

demic rigor that can lead to faulty conclusions [7]. Such deficiencies can be characteristic of inadequate research, but can also be the result of summarizing conclusions that have not been adequately researched or critiqued. Without sufficient critical analysis of research evidence, inconclusive synthesis and findings are left unchallenged and can easily be replicated during the review process. Finding appropriate methods for all types of review entails minimizing biases while adhering to clear and rigorous academic standards. In selecting an appropriate technique, it becomes most apparent how the larger systematic review goals shape the methodology and practical considerations of how to conduct an adequately rigorous scoping review.

3. Exploratory reviews and policy evidence Rigorous review techniques have traditionally been limited to assessing measurable impacts of an intervention, and favor the inclusion of research using experimental designs [17,20]. However, emerging techniques for conducting scoping reviews have included a range of evidence generated from both qualitative and quantitative methods. While the preliminary investigatory nature of some exploratory reviews may suggest an iterative process without need for set systematic methods [2], other exploratory reviews benefit from enhanced consideration of rigor and process. Researchers interested in conducting exploratory scoping reviews of complex social phenomena can experience similar benefits of drawing upon established systematic methods which includes: a heightened accountability for academic rigor [19]; increased standardization and acceptance in scholarly circles [11]; and a greater and more transparent communication of the procedure on how reviews are conducted [1]. In practice, policy researchers have demonstrated the substantial benefits for comprehensive and rigorous reviews of research evidence in informing policy. In medical/health sciences, for example, systematic reviews have become the ‘gold standard’ for research on the effectiveness of medications, psychosocial interventions and organizational management practices [34]. Systematic reviews have had groundbreaking impact on national health and wellbeing by strengthening legislative decision-making processes. Such reviews are often based on rigorous and more exploratory types of review. For example, a prominent study that entailed an exploratory review of existing research on causes of automobile accidents led

325

to conclusions that specific laws against drunk driving could be tested to reduce traffic fatalities [30]. The evidence from the review was used to spearhead sweeping reforms after policy makers had previously advised against tightening blood alcohol concentration laws [34]. Reviews of this sort reflect an increased demand for scientific evidence-based research about the effects of policy in practice. The conceptual roots of applying systematic research to policy decisions are closely linked to a national trend where policy makers and other stakeholder groups increasingly seek “truthful” research evidence to inform decision-making processes. The trend primarily began in medical research but has since spread to other areas of social sciences [27]. Since the inception of evidence-based medical practices and policies, the pursuit of best practices is now one of the primary scholarly pursuits for all of the “help-based” fields in social and health sciences [20]. The benefits of enhancing the rigor of reviews of policy research transcend their utility in transforming research into sound policy. Moreover, such reviews are reflective of a larger philosophical pursuit to make unbiased and informed policy makers and practitioners, facilitate accurate and transparent decision making, and utilize scarce resources to best help marginalize groups [25]. The longstanding effect of existing policy review projects exemplifies the potential this area of research can have on disseminating and applying evidence on the impacts of social policy. This paper now turns to further consideration of how a review project can inform policy research in this manner through a worked example of a scoping review on the Americans with Disabilities Act.

4. Worked example of the Americans with Disabilities Act (ADA) In Rumrill, Fitzgerald and Merchant’s [26] framework, it is suggested that the first step of conducting a scoping review is creating research questions for the study. The step is similar in exploratory reviews, although it is worth considering the significant deliberation that takes place before creating review questions or deciding upon a more exploratory review process. The exploratory nature does not simply mean that one dives in to the review process without baseline knowledge of the area of inquiry. Careful consideration takes place to match the conceptual goals of the review to project goals, while also identifying assumptions about the body of research. Cohesions between

326

S.P. Harris et al. / Enhancing rigor and practice of scoping reviews in social policy research

different stages of the systematic review process ensures that the type of research collected, the creation of research questions, and the purpose of the stages of review are all in alignment [11]. The purpose and aims of the review are often driven by existing evidence. The review questions may build upon a pre-assessment of the literature and the project may involve identifying widespread assumptions from the research. The scoping review often builds from existing knowledge to help focus the potential areas for systematic inquiry. In this worked example, the project was driven from the state of evidence as purported by nationally recognized stakeholder groups. The state of research is such that the broad scope of empirical research on the ADA covers a myriad of topics, methodologies, research questions, and stakeholder groups, and is often considered to present offsetting or inconclusive findings [23]. Various stakeholders identified the need for systematic inquiry about the state of ADA evidence to counter a lack of ongoing empirical study and disagreements in the existing research [5,23,29]. Before such conclusions could be explored, it was first necessary to identify how and from where these conclusions were derived. 4.1. Step 1: Research development The methodological approach for the exploratory review was planned within the context of a larger systematic review project so that the initially collected data could be built upon for future inquiry. To capture the proliferation of methods and content of available ADA-related research, the project team is conducting a three-stage review process that includes scoping, rapid evidence assessments, and systematic reviews. The framework for conducting the three stages of reviews is outlined in Table 1. These stages closely follow the suggestions of Rumrill, Fitzgerald and Merchant [26], but offer an additional detail of how processes more indicative of exploratory reviews (i.e. outsider review and iterative coding schemes) can still be planned as substantially rigorous steps in the research process. The methodological processes outlined in the table reflect commonly used processes for systematic reviews, supplemented with more innovative approaches to research synthesis necessary to review a heterogeneous body of research. Bringing together the findings of diverse studies in a systematic way can be achieved through the application of conventional scoping review principles and methods (i.e. reducing bias, extensive search-

ing, basic aggregation), supplemented with more novel approaches (i.e. interpretative analysis, comparison to anecdotal evidence, and stakeholder feedback) [24,28, 37]. While these processes are intended to enhance the rigor of exploratory reviews, the process of reviewing policy is often more exploratory due to the breadth and depth of evidence. There is also difficulty in assessing direct outcomes of the policies due to the multitude of social and contextual factors that impact policy implementation and outcomes [32]. The research approach displayed in the table may read as a linear process, but components of each stage (i.e. development, data collection, insider feedback, and analysis) may be occurring simultaneously or adapting to reflect ongoing feedback and stakeholder needs. 4.2. Step 2: Formation of expert panel With knowledge of the specific challenges to scoping both policy related-research and research specifically on the ADA, the research team formed an expert panel of ADA stakeholders. Rumrill, Fitzgerald and Merchant [26] briefly mention this step as an optional component of review used in the latter stage of the review process to disseminate findings. We found the step to be imperative when attempting to assess how a social policy has been studied. Review projects often rely on consultations from content experts to exhaustively locate materials that are seldom disseminated, confirm findings and assumptions about the overarching themes from categories of research, and validate key research findings [25]. Recently, there has been further consideration on how the participation of experts impacts the rigor of the study by influencing the range and quality of evidence that is included in the review project [28]. The benefit of added rigor to confirm findings and locate hard-to-reach materials, as well as working with context experts has been invaluable to ensure that findings are useful and applicable to the constituent groups that the research most impacts. We formed an ADA expert panel consisting of key disability-related stakeholders, including representatives from the National Council on Disability, the Disability Rights Education and Defense Fund (DREDF), the National ADA Network, Mathematica Policy Research Group, the US Business Leadership Network, various universities, and other pertinent organizations. This broad representation was determined based on the larger review goals to inform the wide range of ADA stakeholders in charge of the law’s research and im-

S.P. Harris et al. / Enhancing rigor and practice of scoping reviews in social policy research

327

Table 1 Framework for conducting systematic reviews of ADA-related research Scoping review Research development 1 Define Review Questions 2 Form Expert Panel 3 Develop Inclusion Criteria 4 Write Protocol 5 Consult with Expert Panel (refine questions, criteria & protocol; identify key research topics/priorities) Data collection 6 Locate and Screen Studies 7 Select Studies 8 Data Extraction (limit to abstract/title) 9 Consult with Expert Panel (identify emerging issues and data gaps; refine research topics/priorities) Data analysis 10 Data Analysis (descriptive)

11

Data Synthesis (limit to abstract; descriptive)

12

Consult with Expert Panel (identify and refine research topics/priorities) Interpretation (descriptive)

13

Dissemination 14 Consult with Expert Panel (identify and refine research topics/priorities, develop strategies for dissemination) 15 Knowledge Translation (dissemination and knowledge translation of findings)

Rapid assessment

Systematic review

As with steps 1–5

As with steps 1–5

As with steps 6–7

As with steps 6–7

Data Extraction (sample of full papers) As with step 9

Data Extraction (inclusive of full papers) As with step 9

Data Analysis (critical appraisal; overview of key research findings & topics) Data Synthesis (sample of full papers; descriptive/thematic) As with step 12

Data Analysis (critical appraisal; comparative identification & analysis) Data Synthesis (inclusive of full papers; e.g. meta-synthesis) As with step 12

Interpretation (availability of evidence base; potential outcomes of interest)

Interpretation (empirical state of knowledge; quality of evidence; breadth, depth, gaps in research)

As with step 14

As with step 14

As with step 15

As with step 15

plementation. The research team worked in collaboration with the ADA expert panel throughout the scoping review to decide what is evidence, what should be included as evidence, and what is important in the evidence. The formation of the expert panel was vital to ensure the applicability of findings, and to address the various stakeholder knowledge gaps that initiated the call for the systematic review project. 4.3. Step 3: Research questions and search parameters After the larger research goals were developed, the scoping research questions were finalized with the assistance of the expert panel. In reviews of interventionist studies and/or reviews that intend to disseminate shared findings across studies, the process of creating review questions is relatively straightforward. Reviews of these type typically addresses question regarding ‘what works’ or ‘what are the effects of a given intervention and can lead towards a statistical metaanalysis [10]. When conducting a more exploratory

scoping review, the process tends to be more iterative to reflect the larger project goals. The project team often has a level of freedom to develop goals, but also they must respond to stakeholder feedback and/or funding requests. The larger project goal of this review was developed in response to a request by the National Institute on Disability and Rehabilitation Research for a systematic review to “increase the use of available ADA-related research findings to inform behavior, practices, or policies that improve equal access in society for individuals with disabilities”. This goal dictated the stakeholder involvement in developing questions to increase the utility of findings. To address these goals we initially proposed two levels of questions: 1) key questions to describe the scope of current, published research; and 2) draft sample-type questions for the rapid evidence assessment review and systematic reviews that describe the level of detail we hope to be able to address in future stages of review. Sample questions are illustrated in Table 2. We proposed using the scoping studies to determine the research landscape of how the ADA has been em-

328

S.P. Harris et al. / Enhancing rigor and practice of scoping reviews in social policy research Table 2 Sample review questions

Key Question for Scoping Review What English-language studies have been conducted and/or published from 1990 onwards that empirically study the Americans with Disabilities Act? – What disability and stakeholder subgroups are represented in the literature on the ADA? – What individual-level and system-level factors are examined in ADA research? – What topics and titles are represented in the literature on the ADA? – What methodologies and research designs represented in the literature on the ADA? Sample Key Questions for Rapid Assessment and Systematic Reviews Disability subgroup research – What evidence exists that accessible public transportation has improved societal attitudes towards people with sensory disabilities in major urban cities? Stakeholder subgroup research – What evidence exists for the perspectives of union employees being prioritized over the perspectives of self-advocates in qualitative studies of deinstitutionalization and nursing homes? Individual level-factors – What evidence exists that unmet technology training needs of people with disabilities impede their participation in social networking? System level-factors – What evidence exists for emergency response teams effectiveness in delivering equitable services to people with disabilities in disaster situations? Specific title research – What evidence exists for Title II increasing accessibility at public recreation facilities for people with disabilities as a direct result of legal action? Research design – What evidence exists that methodological challenges in the collection of disability specific statistics are related to incongruent definitions of disability in state and federal disability policies?

pirically studied. Findings from the scoping component will inform the number and focus of key questions developed for the rapid evidence and systematic reviews. In this way, the selection of topics will be useful to policymakers, practitioners and researchers. Without knowing specific stakeholder needs at this point of the review, the initial inclusion/exclusion criteria were necessarily broad. The inclusion criteria for the scoping reviews consisted of citations to all studies identified as examining the ADA by a literature search using the following parameters: (a) published or dated from 1990; (b) be written in English; (c) have been carried out in the United States; (d) relate to the ADA; and (e) be based on published studies reporting the gathering of primary or secondary data or the collating and synthesis of existing information to answer ADA-related research questions. 4.4. Step 4: Data collection Data collection for the project occurs at three places during the review process: (1) identification and inclusion of studies at the abstract and full article level; (2) coding of abstracts; and (3) extraction of data from articles at the abstract and full article level (note: this last stage will take place in future systematic reviews).

Although Rumrill, Fitzgerald and Merchant [26] suggest that scoping reviews conducted for the purpose of dissemination should immediately grapple with findings in the full text, this can be far more challenging for scoping reviews related to the impact of social policy. Our initial search strategy revealed over 30,000 research records that required analysis of abstracts and introductions to extract data of potential use for future research and systematic review; far too many records to assess in full text. In systematic reviews, the population of included articles is dependent on the search strategies and the inclusion/exclusion criteria. Previous studies have noted the challenges in conducting non-standard approaches to reviews because of the need to locate and review extensive and heterogeneous types of evidence [2,25,28]. Concurrently one of the strengths of scoping studies is the breadth and depth of evidence. Stakeholder groups can help reduce the amount of articles for scoping reviews and subsequent systematic reviews by identifying small set of studies for in-depth review [24]. Moreover, it is important to have a scoping team whose members provide the methodological and context expertise needed for decisions regarding breadth and depth [19]. Arriving at a sampling frame for a scoping or systematic review is often an iterative process. Re-

S.P. Harris et al. / Enhancing rigor and practice of scoping reviews in social policy research

searchers commonly work with a trained librarian and consult with the expert panel throughout this process to revise the search terms until confidence is reached that the strategy is capturing applicable material and minimizing the amount of extraneous material. 4.4.1. Identification and inclusion of articles Scoping the full base of research evidence in regards to the implementation of a policy requires gathering a large and diverse body of evidence. While the primary purpose of a scoping review is to map the state of evidence in a given area, there is much debate about what constitutes evidence when it comes to research on social policy such as the ADA. Again, our expert panel was consulted to assist us in the process of narrowing search terms to reveal applicable evidence. In addition to debates on the type of evidence to include, the fragmentation and volume of research on the ADA makes it hard to know what is currently useful. Rumrill, Fitzgerald and Merchant [26] explain that useful evidence for scoping reviews is often retrieved through various electronic search strategies, and directly contacting context experts. We found that in reviewing evidence on a social policy, sourcing records is more complex than in other types of scoping or systematic reviews. For example, a substantial source of evidence for key ADA decisions to date has been organizational reports and technical research. Given the non-availability of many strictly qualitative and quantitative accounts, search strategies for exploratory reviews must augment electronic searches with more traditional methods of reviewing including back-tracking of references and engaging with key experts in the field [25]. ADA-related evidence, as with much research on the impact of social policy, can come from published and/or unpublished research from a variety of sources: – Published studies are found in academic databases, journal articles, books and book chapters, dissertations, background material used in court case decisions; as well as research conducted or commissioned by government departments, disability, technical, and other stakeholder agencies, ADA Regional Centers, research institutes, think tanks, and employer and business-related organizations. – Unpublished studies (‘grey literature’) are found in many of the above same sources, but especially in government agencies, non-government and technical organizations, dissertations, and research institutes. Because grey literature is unpublished, obtaining the research and reports requires the expertise and knowledge of key contacts and stakeholders working in the field.

329

The review of records is a complex and arduous process to address the broad scoping question of what evidence exists on the ADA.1 4.4.2. Data extraction The data extraction process of a review conducted in advance of a systematic review requires an added degree of rigor. This step closely resembles what Rumrill, Fitzgerald and Merchant [26] refer to as the “categorical charting” step of the scoping review. However, we found that the exploratory review process benefited from a degree of iterative coding as well as closely examining narrative descriptors that the expert panel identified as essential factors of ADA implementation (i.e. reasonable accommodation, antidiscrimination charges). Once the full range of records has been collected, data extraction occurs. This step involves the identification and collection of information relevant to the review question to lay the ground for the next step of synthesizing the collective evidence [11]. A data extraction form, code-book, and training procedures were developed specifically for this review. The content variables and coding options for the extraction form were developed based on expert panel feedback, pilot tested, and revised for reliability. The form contained information on: research purposes and questions, theoretical framework, method and design, sampling strategy, sample demographics, study setting, data collection and analysis techniques, techniques to optimize validity or minimize bias, techniques to protect human subjects, strength of study evidence, findings and discussion, and other relevant data to address the key questions. Data extraction needs to be an iterative process in which researchers continually extract and update the data extraction form. Having a data extraction form and system in place at the scoping stage, greatly eases the categorization, searching, and coding of literature later on in the systematic review process. Additionally, the use of software to manage the data at all stages of the review is essential. The project uses EPPI Reviewer 4.0 for managing and analyzing data.2 1 Full

detail of the selection process and the records included and excluded appears in a separate article submitted for publication that documents the results of the scoping review. 2 A software program developed and maintained by the EPPICentre at the Social Science Research Unit at the Institute of Education, University of London, for use in research synthesis. We uploaded studies for screening, coding, data extraction and synthesis that can be additionally analyzed during the future stages of review.

330

S.P. Harris et al. / Enhancing rigor and practice of scoping reviews in social policy research

4.5. Step 5: Data analysis (synthesis, interpretation, and consultation with expert panel) The way that data is explained and synthesized is often dictated by the funding requirements or publication goals. The method of dissemination for this information is also derived from a dissemination plan that reflects stakeholder needs. The analysis stage of the scoping review process included descriptive analysis and synthesis of the data which was completed using a numerical summary (e.g. overall number of studies included, types of study design, topics and/or titles studied, characteristics of disability sub-groups and/or stakeholders, years of publication). Due to the breadth and depth of research on the ADA, data was synthesized at the abstract level for the scoping studies and will be at the full article level for the rapid evidence and systematic reviews. When insufficient detail was provided in article abstracts, the introductory and research design descriptions were also consulted. The final result of this coding and analytical process was descriptive data about the number of studies that featured various elements that could be compared for trends in approaches (i.e. sub-factors such as costs compliance), topic areas (i.e. employment, healthcare), stakeholders typically analyzed in these studies, and common research designs. This information is useful to compare to existing anecdotal evidence and to develop suggestions for future research through a baseline assessment of the type of evidence available to inform specific systematic review questions.

5. Limitations and lessons learned A key challenge to conducting scoping reviews of broad and diverse topic areas such as social policy is the retrieval of a necessarily broad and diverse expanse of literature including grey and other hard-tolocate research materials. The availability of resources devoted to the search process can inform the extent to which exhaustive searching can be conducted, and theoretical saturation of relevant research findings can be achieved [11]. The primary challenge for this research on the ADA has been the retrieval of unpublished and print materials. While difficult to locate, this type of information has shown to be a rich source of underutilized and unexplored policy evidence. Reviewing unpublished and print materials posits several challenges regarding the location and analysis of data. The majority of the difficulties arise from obtaining mate-

rial from before the internet became readily available. These materials are expensive to ship, and libraries are often reluctant to send them. For similar projects, we would recommend the inclusion of comprehensive appraisal criteria (or when appropriate, a smaller time frame of included studies) to limit the amount of research collected, even at the scoping review level. In more exploratory reviews, a degree of appraisal may be necessary to limit evidence to pertinent and relevant research findings. We discovered that there were vital resources that did not fit with our framework, and additionally, that a large portion of research met inclusion criteria, but likely would not be informative for future review. One way to enhance the relevance of included research for scoping reviews of policy studies and other non-interventionist research may be the use of more sophisticated inclusion criteria that also contains appraisal criteria. While other systematic review methodologists have argued that appraisal may be an inappropriate step for scoping reviews [38], the breadth and depth of research on the impact of the social policy requires additional deliberation and methodological consideration. It is necessary to limit included research so that it is analyzable and findings are useful for involved stakeholders. A degree of appraisal may be appropriate for exploratory reviews when inclusion criteria are so broad that it results in the collection of informal research without explanation of methodological process or rigor. A primary purpose of enhancing academic rigor for exploratory scoping reviews is to increase the utility of such reviews for a broad array of practitioners, researchers, and policy makers. Additional suggestions were made by the expert panel to increase the usefulness of findings. It was also suggested that future reviews of policy evidence might benefit from scoping analysis of what research is not available. This sentiment echoes emerging theoretical considerations of interpreting research suggesting that missing data or the exclusion of population groups from analysis also reveals an array of sociological evidence [11]. Although we identified seldom studied or under-studied areas of the ADA, our approach did not include any steps for comparative analysis of topic areas that are imagined to be under the scope of the ADA but do not have any research. There is not a standardized approach to scoping reviews for this process, as reviews typically cover well-researched areas to generate summative evidence. However, developing a process for comparative analysis of this type would be extremely relevant for future exploratory scoping reviews that have a goal to di-

S.P. Harris et al. / Enhancing rigor and practice of scoping reviews in social policy research

rect future research. This approach would not necessarily be appropriate for all types of exploratory review, such as those created with the intention of conducting a systematic review approach that requires a substantial base of evidence to create summative and/or generalizable findings.

6. Discussion: Informing scoping reviews of policy research and non-interventionist evidence The process of conducting an exploratory review of a non-intervention based research question has revealed useful considerations for future practitioners of scoping reviews. Rumrill, Fitzgerald and Merchant [26] offer insight on the importance of standalone scoping reviews and provide a methodological framework for reviewing evidence. More exploratory type of reviews can also generate useful evidence when adhering to similarly rigorous methodological processes. There are several key differences in the exploratory type of scoping review conducted in preparation of future research compared to systematic reviews/stand-alone reviews that disseminate findings. While the goals of these reviews suggest different conceptual approaches, both types of scoping reviews benefit from considering how rigorous attention to systematic methodology can enhance potential evidence for informing future research and practice. The benefits of adding rigor to reviews of policy evidence are that they enhance the utility of findings generated from the review process. As explored in our example review on the ADA, a key difference between the types of findings generated using the scoping techniques suggested by Rumrill, Fitzgerald and Merchant [26], and the provided example, is the charting of process data versus outcome data. For example, the aforementioned authors provide an example of findings that might be charted in a scoping review of workplace and discrimination such as the trend of a higher degree of workplace discrimination in different industries. In our study, there is less attention to conclusive analysis at this stage of the review, as this type of evidence will be generated further on in the study with more rigorous evaluation of existing research. A more exploratory type of review first identifies the broader body of research that discusses this type of evidence (i.e. accounts of workplace discrimination), and then discusses contributing factors, sources of evidence, and consideration of the methodological process to generate such evidence. The goal of the study provided in

331

this article is to interrogate how the area of interest was studied and pertains less to describing outcome based evidence at this point of the project. The goal of generating such evidence would be inappropriate based on the review goals and the current state of ADA research evidence that is often contested as fragmented or anecdotal [18,23]. Other reviewers of heterogeneous bodies of social policy evidence may find that the focus on process will similarly be useful to inform future systematic research. A key challenge of transforming evidence on research process into eventual outcome-related data is the synthesis of a range of evidence from differing research paradigms, questions, and research designs. It has been acknowledged that much of the broad research on ADA research is extremely heterogeneous, covering a variety of different populations with often offsetting findings, or disagreements in conclusions [16,31]. Much of these disagreements are due to the lack of clear longitudinal data kept as indicators of the policy’s progress, and also the difficulty in assessing cause/effect relationships of social policy [32]. The realization that there is a great difficulty in assessing the direct impact of a social policy is pertinent for future reviewers to understand the need to consult a broad range of evidence to assess a policy’s impact. The state of evidence on a social policy makes any type of findings-based review without rigorous appraisal more susceptible to unit of analysis error where different studies may feature a wide variety of variables analyzed ranging from court outcomes, to individual experiences, to group experiences. In exploratory scoping reviews (and systematic reviews in general), the unit of analysis is most typically the study itself [11]. The end result of such errors in systematic review is that conclusions that are repeated across study may receive a higher-than-appropriate weight or predictive value. The exploratory nature of a review of social policy evidence thus merits considerable attention to appraisal and rigor in addition to the framework that was initially suggested by Rumrill, Fitzgerald and Merchant [26]. Without a more sophisticated analysis and vivid description of study settings (such as proposed in the latter stages of our review) it would be misleading to present conclusive findings at the scoping stage of the review. This level of in-depth analysis would be inappropriate for scoping review that often features exhaustive searching across a broad range of evidence. Even after narrowing down our scoping review to a smaller group of relevant records, it would be erroneous to track purported findings without thick con-

332

S.P. Harris et al. / Enhancing rigor and practice of scoping reviews in social policy research

textual description due to the volume of research and the varying predictive/descriptive capabilities of different methodological approaches used in this research. In preparation for future systematic review, it is suggested that exploratory scoping reviews are particularly relevant approaches for tracking process elements and descriptors that can further inform future analysis of outcomes. 6.1. Challenges to rigorous reviews of diverse research Researchers must consider and address the various ways that the review can become biased during each stage of the review process. This is particularly true for exploratory reviews that require researchers to make a variety of choices on the type and range of research reviewed. The degree that researchers’ opinions and beliefs play in the selection of evidence incidentally is one of the primary factors influencing bias in conducting reviews [33]. Rumrill, Fitzgerald and Merchant [26] exemplify how such systematic steps greatly influence the utility of scoping reviews of interventionist research. The reduction of bias is often reflective of using a clear search strategy, and inclusion/exclusion criteria that is often expanded so that only relevant research is included. The process of reducing bias is less straightforward, however, when the state of evidence is more diverse. In this case, a narrow search strategy may not be appropriate for the research problem. Typically, scoping reviews are generated in well-established fields (i.e. health or medicine) and involve aggregating data from similarly crafted studies to determine correlated relationships or what works to achieve desired outcomes. The most prevalent techniques for systematic scoping consist of aggregating the results of multiple studies that are very similar in research method and design, population group, and purpose [11]. This review process is often limited to studies testing intervention methods or explaining shared findings of similar studies [17,20]. Policy implementation research, such as that on the ADA, differs greatly from much of the medical and intervention research that is typically included in reviews and draws on a greater diversity of testing methods. The synthesis process is also more complicated. ADA-related research consists of qualitative, quantitative and mixed methods studies that address a myriad of sub-topics areas, which range widely from: quantitative analyses of hiring rates and costs [29], to qualitative discourse analyses of cultural representations of disability [13] to

mixed methods analyses of the policy’s quality of life implications [15]. The wide range of methods used to study the state of research on a policy suggests a review approach that can adequately synthesize the diverse body of research. The inclusion of diverse research in systematic reviews has only recently been considered as a viable endeavor. Rumrill, Fitzgerald and Merchant [26] note that one of the advantages of scoping reviews is that they also provide a substantial resource for more diverse types of evidence than would be included in a typical systematic review. This is a challenge when considering the appropriate evidence to include in scoping reviews conducted in advance of further systematic reviews. The contribution that nonexperimental and qualitative research can make to inform policy and practice has been limited in the past because of uncertainty about how to include such research in systematic reviews [24]. In turn, researchers have often excluded qualitative or mixed method studies from evidence-based reviews, focusing largely on meta-analysis, which has become synonymous with systematic reviews. However, meta-analysis is not always the most suited or appropriate technique to synthesize data [10,25], and in recent years there has been a growing interest in synthesizing the findings of methodologically diverse studies including qualitative and mixed-methods analysis [12]. Such methods are being used in social science, education, medical and health care areas, and are now widely considered to be an alternative approach to meta-analysis [3,8,21,28]. The ultimate decision of methodological approach for the full systematic review is reciprocally related to the scoping process. It is beneficial for the techniques and types of evidence collecting for the scoping review to fit within the large conceptual goal of the project, and often the results of the scoping review will reveal the type of evidence that is available to address systematic review research questions.

7. Conclusion Exploratory reviews of diverse research evidence are useful to establish baseline knowledge about a broad field of inquiry and to consolidate the diverse types of evidence generated across various research approaches and designs. While there has been growing attention to rigorous methods for scoping reviews of interventionist research that seek to disseminate findings, this article describes alternative approaches to re-

S.P. Harris et al. / Enhancing rigor and practice of scoping reviews in social policy research

view for more exploratory and heterogeneous bodies of research. Such exploratory reviews can similarly benefit from enhanced rigor and epistemological consideration on how to improve practice and the utility of findings. Comparable to other approaches for conducting scoping reviews, exploratory reviews also have great potential to efficiently inform readers about a large body of research. Exploratory reviewing conducted in advance of a systematic review also have the benefit to be quickly operationalized into additional research that can guide evidence-based policy decisions and inform best practice. Exploratory reviews of non-interventionist studies present a variety of theoretical and epistemological challenges that were discussed in this article. When reviewing non-interventionist research, such as with the analysis of the impact of social policy like the ADA, there is still considerable room for establishing rigorous processes and techniques for scoping reviews. This paper presents only one such approach that will be further refined throughout the full systematic review process. It is the research team’s hope that the full review project will generate useful evidence on the impact of disability policy and can also be useful to guide other exploratory scoping reviews in the future.

Acknowledgement This research is funded by NIDRR Grant # H133A 11014.

References [1]

[2]

[3]

[4]

[5]

[6]

Armstrong R, Hall BJ, Doyle J, Waters E. ‘Scoping the scope’ of a cochrane review. Journal of Public Health 2011 March 01; 33(1): 147-150. Barbour RS, Barbour M. Evaluating and synthesizing qualitative research: the need to develop a distinctive approach. J Eval Clin Pract 2003; 9(2): 179-186. Britten N, Campbell R, Pope C, Donovan J, Morgan M, Pill R. Using meta ethnography to synthesise qualitative research: A worked example. Journal of Health Services Research & Policy 2002 October 01; 7(4): 209-215. Bryson-Campbell M, Shaw L, O’Brien J, Holmes J, Magalhaes L. A scoping review on occupational and self identity after a brain injury. Work 2013 01; 44(1): 57-67. Collignon FC. Is the ADA Successful? Indicators for Tracking Gains. The ANNALS of the American Academy of Political and Social Science 1997 January 01; 549(1): 129-147. Cronin S, Curran J, Iantorno J, Murphy K, Shaw L, Boutcher N, et al. Work capacity assessment and return to work: A scoping review. Work 2013 01; 44(1): 37-55.

[7]

333

Davies P. What is needed from research synthesis from a policy-making perspective? In: Popay, J, editor. Moving beyond effectiveness in evidence synthesis. London: National Institute for Health and Clinical Excellence; 2006. 97-103. [8] Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A. Synthesising qualitative and quantitative evidence: A review of possible methods. Journal of Health Services Research & Policy 2005 January 01; 10(1): 45-53B. [9] Egger M, Dickersin K, Smith GD. Problems and Limitations in Conducting Systematic Reviews. Systematic Reviews in Health Care: BMJ Publishing Group; 2001; 2008. p. 43-68. [10] Fitzgerald SM, Rumrill P, J. Speaking of research. Metaanalysis as a tool for understanding existing research literature. Work 2003 07; 21(1): 97-103. [11] Gough D, Oliver S, Thomas J. An introduction to systematic reviews, Sage London, 2012. [12] Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal 2009 06; 26(2): 91-108. [13] Haller B. How the news frames disability: Print media coverage of the Americans with disabilities act. In: Altman B, Barnartt S, editors. Expanding the scope of social science research on disability. Research in social science and disability 2001; 1: 55-83. [14] Hannes K, Macaitis K. A move to more systematic and transparent approaches in qualitative evidence synthesis: update on a review of published papers. Qualitative Research 2012 August 01; 12(4): 402-442. [15] Harrison TC. Has the Americans with Disabilities Act Made a Difference? A Policy Analysis of Quality of Life in the PostAmericans with Disabilities Act Era. Policy, Politics, & Nursing Practice 2002 November 01; 3(4): 333-347. [16] Hoffman S. Settling the Matter: Does Title I of the ADA Work? Alabama Law Review 2008; 59(2): 305. [17] Jesson J, Matheson L, Lacey FM. Doing your literature review: Traditional and systematic techniques. London: Sage Publications, 2011. [18] Kovacs Burns K, Gordon GL. Analyzing the Impact of Disability Legislation in Canada and the United States. Journal of Disability Policy Studies 2010 March 01; 20(4): 205-218. [19] Levac D, Colquhoun H, O’Brien KK. Scoping studies: advancing the methodology. Implement Sci 2010; 5(1): 1-9. [20] Littell JH, Corcoran J, Pillai V. Systematic reviews and metaanalysis. New York: Oxford University Press, 2008. [21] Lucas PJ, Baird J, Arai L, Law C, Roberts HM. Worked examples of alternative methods for the synthesis of qualitative and quantitative research in systematic reviews. BMC medical research methodology 2007; 7(1): 4. [22] Mertens DM. Transformative Mixed Methods Research. Qualitative Inquiry 2010 July 01; 16(6): 469-474. [23] National Council on Disability. Implementation of the Americans with Disabilities Act: Challenges, best practices, and new opportunities for success. Washington DC: US NCD; 2007. Available from: http://www.ncd.gov/publications/2007/ 07262007. [24] Oliver S, Harden A, Rees R, Shepherd J, Brunton G, Garcia J, et al. An emerging framework for including different types of evidence in systematic reviews for public policy. Evaluation 2005; 11(4): 428-446. [25] Petticrew M, Roberts H. Systematic reviews in the social sciences: A practical guide. Malden, MA: Blackwell Publishing, 2008. [26] Rumrill PD, Fitzgerald SM, Merchant WR. Using scoping lit-

334

[27]

[28]

[29]

[30]

[31]

[32]

S.P. Harris et al. / Enhancing rigor and practice of scoping reviews in social policy research erature reviews as a means of understanding and interpreting existing literature. Work 2010 03; 35(3): 399-404. Sackett DL, Rosenberg WM, Gray J, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. BMJ: British Medical Journal 1996; 312(7023): 71. Sandelowski M, Barroso J, Voils CI. Using qualitative metasummary to synthesize qualitative and quantitative descriptive findings. Res Nurs Health 2007; 30(1): 99-111. Schwochau S, Blanck PD. Economics of the Americans with Disabilities Act, Part III: Does the ADA Disable the Disabled. Berkeley J. Emp. & Lab. L. 2000; 21: 271. Shults RA, Elder RW, Sleet DA, Nichols JL, Alao MO, Carande-Kulis VG, et al. Reviews of evidence regarding interventions to reduce alcohol-impaired driving. Am J Prev Med 2001 11; 21(4, Supplement 1): 66-88. Silverstein R. Anatomy of Change: The Need for Effective Disability Policy Change Agents. Arch Phys Med Rehabil 2010 2; 91(2): 173-177. Silverstein R, Julnes G, Nolan R. What policymakers need and must demand from research regarding the employment rate of persons with disabilities. Behav Sci Law 2005; 23(3): 399-448.

[33]

[34]

[35]

[36]

[37]

[38]

Suri H. Epistemological pluralism in research synthesis methods. International Journal of Qualitative Studies in Education 2013 08/01; 2013/10; 26(7): 889-911. Sweet M, Moynihan R, Fund MM. Improving population health: the uses of systematic reviews. Produced in Collaboration with the Centers for Disease Control and Prevention (CDC): Milbank Memorial Fund; 2007. Available from http://www.milbank.org/reports/0712populationhealth/0712 populationhealth.html. To-Miles F, Shaw L. Knowledge transfer with children and adolescents in promoting comfort, health, and safety in technology use: Strategies and opportunities. Work 2012 11; 43(3): 387-397. Tranfield D, Denyer D, Smart P. Towards a Methodology for Developing Evidence-Informed Management Knowledge by Means of Systematic Review. Br J Manage 2003 09; 14(3): 207-222. Voils CI, Sandelowski M, Barroso J, Hasselblad V. Making Sense of Qualitative and Quantitative Findings in Mixed Research Synthesis Studies. Field Methods 2008 February 01; 20(1): 3-25. Walsh D, Downe S. Meta-synthesis method for qualitative research: A literature review. J Adv Nurs 2005; 50(2): 204-211.