Evaluation and Program Planning 48 (2015) 1–9

Contents lists available at ScienceDirect

Evaluation and Program Planning journal homepage: www.elsevier.com/locate/evalprogplan

Protocol: Realist synthesis of the impact of unemployment insurance policies on poverty and health Agnes Molnar a,*, Patricia O’Campo a,b, Edwin Ng a, Christiane Mitchell a, Carles Muntaner a,b,c, Emilie Renahy a, Alexander St. John a, Ketan Shankardass a,b,d a

Centre for Research on Inner City Health, Li Ka Shing Knowledge Institute, 209 Victoria Street, 3rd Floor, Toronto, ON, Canada M5B 1T8 Dalla Lana School of Public Health, University of Toronto, Health Sciences Building, 400-155 College Street, Toronto, ON, Canada M5T 3M7 Bloomberg Faculty of Nursing, University of Toronto, Health Sciences Building, 130-155 College Street, Toronto, ON, Canada M5T 3M7 d Department of Psychology, Wilfrid Laurier University, Sciences Building, 75 University Avenue West, Waterloo, ON, Canada N2L 3C5 b c

A R T I C L E I N F O

A B S T R A C T

Article history: Received 4 December 2013 Received in revised form 7 August 2014 Accepted 9 September 2014 Available online 19 September 2014

Unemployment insurance is an important social protection policy that buffers unemployed workers against poverty and poor health. Most unemployment insurance studies focus on whether increases in unemployment insurance generosity are predictive of poverty and health outcomes. Less work has used theory-driven approaches to understand and explain how and why unemployment insurance works, for whom, and under what circumstances. Given this, we present a realist synthesis protocol that seeks to unpack how contextual influences trigger relevant mechanisms to generate poverty and health outcomes. In this protocol, we conceptualize unemployment insurance as a key social protection policy; provide a supporting rationale on the need for a realist synthesis; and describe our process on identifying context-mechanism-outcome pattern configurations. Six methodological steps are described: initial theory development, search strategy; selection and appraisal of documents; data extraction; analysis and synthesis process; and presentation and dissemination of revised theory. Our forthcoming realist synthesis will be the first to build and test theory on the intended and unintended outcomes of unemployment insurance policies. Anticipated findings will allow policymakers to move beyond ‘black box’ approaches to consider ‘mechanism-based’ explanations that explicate the logic on how and why unemployment insurance matters. ß 2014 Elsevier Ltd. All rights reserved.

Keywords: Health Poverty Protocol Unemployment insurance Realist synthesis Social protection Context Mechanism Outcome

1. Introduction Recent research has examined the effects of social protection on poverty and health within and between wealthy nations (CSDH (2008); Marmot, Friel, Bell, Houweling, & Taylor, 2008; Muntaner et al., 2011; Nelson, 2013). Social protection refers to policies and programs designed to prevent, manage, and overcome situations that adversely affect the well-being of individuals and populations (Norton, Conway, & Foster, 2002). Given that social protection mitigates the risks associated with common life events such as unemployment (e.g., when a worker who is actively searching for employment is unable to find work), the effects of social protection are likely to include poverty and health. In particular, unemployment insurance is a type of social protection that works in several

* Corresponding author. E-mail addresses: [email protected] (A. Molnar), [email protected] (C. Mitchell). http://dx.doi.org/10.1016/j.evalprogplan.2014.09.002 0149-7189/ß 2014 Elsevier Ltd. All rights reserved.

ways to protect and support unemployed workers from adverse outcomes. Existing studies find that increases in unemployment insurance generosity are predictive of both lower poverty rates and improved health outcomes (Ferrarini, Nelson, & Sjo¨berg, 2014; Kessler, Turner, & House, 1988; Rodriguez, 2001; Rodriguez, Frongillo, & Chandra, 2001; Scruggs & Allan, 2006). It appears that collective resources (e.g., government-sponsored cash benefits) provided during unemployment, a life event that requires additional resources, protects unemployed workers against falling into poverty or experiencing poor health (Lundberg et al., 2008). However, less work has applied theory-driven approaches to explain how and why unemployment insurance works, for whom, and under what circumstances. In this paper, we present a protocol for a realist synthesis which aims to unpack how contextual influences trigger relevant mechanisms to generate poverty and health outcomes. This protocol paper is organized in three sections. In Section 1, we define unemployment insurance, provide a rationale for using realist synthesis, and state study objectives and questions. In

2

A. Molnar et al. / Evaluation and Program Planning 48 (2015) 1–9

Section 2, we detail our methodological steps for our realist synthesis: initial theory development, search strategy, selection and appraisal of documents, data extraction, analysis and synthesis process, and presentation and dissemination of revised theory. We conclude in Section 3 by discussing how our realist synthesis protocol makes a novel contribution to our knowledge base. 1.1. Unemployment insurance Unemployment insurance systems comprise social welfare payments made by states or other authorized bodies to unemployed individuals, and are designed to alleviate the income insecurity of those out of work for conditional periods of time. Unemployment insurance systems vary considerably across wealthy countries, and sometimes even vary within sub-national jurisdictions (e.g., provinces or states) (Bambra & Eikemo, 2009; Katz & Meyer, 1990). Cross-national differences in unemployment insurance include variations in funding sources (e.g., worker, state, employer), eligibility criteria (e.g., be unemployed through no fault of the worker), replacement rates (e.g., the percentage of income replaced by unemployment insurance), replacement durations (e.g., the time unemployment insurance is available during unemployment), waiting periods (e.g., the time for which no benefit is paid), and active labour market policies (e.g., programs that help and re-train the unemployed find work) (Eurofound, 2007; Kenworthy, 2008; Scruggs & Allan, 2006). In countries with historically weak unemployment insurance schemes (e.g., Greece, Italy, and United States), replacement rates fall below subsistence levels and coverage rates are meagre (Gallie & Paugam, 2000). In contrast, countries with generous systems (e.g., Denmark and Sweden), replacement rates are high, benefit durations are long, and eligibility criteria are based on social citizenship rights rather than means-testing (Gallie & Paugam, 2000). Although the determinants of unemployment insurance generosity extend beyond the scope of the current protocol, it is worth noting the importance of political actors and institutions in shaping and influencing generosity. Historically, political parties and labour unions have used their power in government and the labour market, respectively, to increase the willingness of welfare states to institute generous unemployment insurance schemes. The key argument is that political parties and labour unions committed to achieving egalitarian outcomes (e.g., lower levels of poverty, narrower income inequalities) are more likely to advocate for generous unemployment insurance schemes given their design to compensate for market failures, ensure socially acceptable standards of living, and facilitate those out of work to re-enter the labour force. While we acknowledge that political actors and institutions are key considerations in determining the generosity of unemployment insurance, our protocol’s analytic goal is to build and test theory on how unemployment insurance is supposed to work and its expected impacts on poverty and health. Existing studies find that countries with generous unemployment insurance systems are effective in alleviating the financial burden of unemployed workers as well as protecting them against severe income losses and preventing the onset of absolute and relative forms of poverty (Allan & Scruggs, 2004). Generous unemployment insurance schemes are particularly important during economic downturns because large segments of the working population face greater risks for unemployment and longer spells of joblessness. On one hand, unemployment insurance has the intended consequence of protecting workers against the adverse economic consequences of unemployment. On the other, it has been argued that unemployment insurance might have the unintended consequence of reducing the incentive to work among the unemployed, or what economists consider a ‘‘moral hazard’’ (Chetty, 2008; Howell & Azizoglu, 2011). In any

case, the effects of the current global recession among most European countries are certain—unemployment rates are persistently high, unemployment spells are determinedly long, poverty rates have steadily increased, and several public health indicators such as suicide rates have worsened (Karanikolos et al., 2013; Stuckler, Basu, Suhrcke, Coutts, & McKee, 2009). Given these current trends, we argue that more theory-driven research is needed to illuminate how social protection policies such as unemployment insurance can strike an optimal balance between ensuring adequate income supports, creating attractive work incentives, ameliorating material hardships, improving population health, and reducing health inequalities. A major step toward these ends involves developing a deeper understanding of the mechanisms that connect the context of unemployment insurance policies with various outcomes, including for example income security, poverty, and physical and mental health (Shiller, 2008; Stuckler et al., 2009). 1.2. Rationale for using realist synthesis As reviewed here, the associations between unemployment insurance, poverty, and health have been established. However, minimal research examined the theory behind how unemployment insurance policies protect against financial distress and poor health. Realist synthesis is a theory-driven approach to evidence synthesis that can reveal how unemployment insurance interacts with contexts, which then trigger mechanisms, which produce poverty and health outcomes. A key implication is unemployment insurance may work well in one context to reduce poverty and improve health but poorly or not all in another context. In this paper, we present a realist synthesis protocol on unemployment insurance to identify, characterize, and synthesize the underlying mechanisms through which unemployment insurance affect poverty and health. By doing so, our work leads to the development, refinement, and exposition of an original theoretical framework that unpacks context, mechanism, and outcome (CMO) pattern configurations (Pawson, Greenhalgh, Harvey, & Walshe, 2004). In our protocol, contexts refer to the conditions in which unemployment insurance policies are introduced and that affect the activation of mechanisms (Pawson & Tilley, 1997). The concept of mechanisms refers to the ‘‘. . . underlying entities, processes, or structures which operate in particular contexts to generate outcomes of interest’’ (Astbury & Leeuw, 2010, p. 368). Central to understanding mechanisms is that they are often hidden, may work in one context but another, and are responsible for generating outcomes. Outcomes consist of both the intended and unintended consequences of unemployment insurance policies, and result from the activation of different mechanisms in varied contexts (Pawson & Tilley, 1997). The clear advantage of the CMO approach involves its set of analytical strategies that assist in uncovering the mechanisms responsible for bringing about specific outcomes (Blamey & Mackenzie, 2007; Coryn, Noakes, Westine, & Schro¨ter, 2011; Marchal, van Belle, van Olmen, Hoere´e, & Kegels, 2012; Sridharan & Nakaima, 2012). Calls for evidence-based policy options are increasingly relying on systematic reviews that focus on the impacts of specific policies, programs, and interventions (Bambra, 2011). Systematic reviews compile and organize fragmented bodies of empirical research by pooling, assessing, and synthesizing evidence on selective interventions. Recent examples include reviews on the effectiveness of interventions on various social determinants of health (e.g., housing, work environment) (Bambra et al., 2010) and types of interventions that exacerbate health inequalities (Lorenc, Petticrew, Welch, & Tugwell, 2013). The greatest advantage of systematic reviews involves its ability to uncover what policy,

A. Molnar et al. / Evaluation and Program Planning 48 (2015) 1–9

program, or intervention works or does not work within a given sector at a given time (Pawson, 2006). In recent years, public health researchers have augmented systematic reviews with realist synthesis methodologies (Greenhalgh, Kristjansson, & Robinson, 2007; O’Campo et al., 2011). Whereas systematic reviews adhere to strict parameters that tend to produce narrow findings, realist reviews uncover the underlying mechanisms that explain the logic of policies, programs, and interventions (Pawson & Manzano-Santaella, 2012). By concentrating on mechanisms, realist syntheses first make explicit the underlying theories on how and why an intervention might work within a specific context, and second gather different types of valid evidence to confirm or refute these theories (O’Campo et al., 2011; Pawson et al., 2004). Other benefits include the inclusion of both quantitative and qualitative evidence (e.g., reviewing academic literature and key documents, interviewing key informants; synthesizing statistical results) as well as the use of stakeholder engagement during the data collection and review processes (Kastner et al., 2011). Despite the growing number of realist syntheses appearing in the public health literature (Gagliardi et al., 2011; Jagosh et al., 2011; Kastner et al., 2011; Robert, Ridde, Marchal, & Fournier, 2012; Wong, Greenhalgh, Westhorp, Buckingham, & Pawson, 2013), there remains a dearth of realist syntheses that unpack unemployment insurance, poverty, and health into context – mechanism – outcome configurations. In this protocol, we apply a theory-driven approach to undertake a realist synthesis on how and why unemployment insurance is supposed to work in generating our outcomes of interest. 1.3. Objectives and questions Our realist synthesis aims to identify, characterize and systematically synthesize the literature on unemployment benefit policies to reveal the mechanisms by which these policies in various contexts can reduce or eliminate social and health inequalities arising from social and labour market conditions. More specifically, our aim is to develop, refine, and propose a theoretical framework that depicts CMO pattern configurations. In essence, we will augment the literature on unemployment insurance policies by articulating the mechanisms through which these policies affect proximal poverty rates and/or health outcomes. Specifically, we ask the following realist synthesis questions: How, why, and under what circumstances does unemployment insurance impact short- and long-term poverty? How, why, and under what circumstances does unemployment insurance impact well-being, physical, and mental health? Our realist synthesis is conducted within a larger European Union funded research project called the SOPHIE project (Evaluating the Impact of Structural Policies on Health Inequalities and their Social Determinants, and Fostering Change) (SOPHIE, 2014). The SOPHIE project aims to develop innovative methodologies to generate new evidence on the impact of structural policies on social and health inequalities while paying particular attention to the involvement of affected stakeholders (http://www.sophie-project.eu/index.htm). 2. Methods In this section, we describe the six methodological steps that will guide our realist synthesis of unemployment insurance policies. 2.1. Initial theory development Consistent with theory-driven approaches, the primary aim of research synthesis is explanation building, which involves the

3

articulation of underlying theories, and the interrogation of existing evidence to determine whether identified theories are pertinent and productive (Pawson, 2006; Pawson, Greenhalgh, Harvey, & Walshe, 2005). In combing for working theories that underline unemployment insurance, our steps are two-fold: (i) searching for initial theories in the literature, and (ii) consulting with experts. These initial efforts will assist with building theory and informing our initial theoretical framework—a cornerstone of high-quality of realist syntheses. In developing and articulating our initial theoretical framework, we will concentrate on identifying theories that explain how and why unemployment insurance policies may be successful or not with reducing poverty and improving health. Insights gleaned from browsing the literature and consulting with experts will be combined to form the building blocks of our initial theoretical framework. 2.1.1. Searching for initial theories in the literature To build an initial understanding unemployment insurance, poverty, and health, we will search for initial theories in the literature. This process will assist in the identification of provisional theories and key experts to contact for consultations. We will informally and rapidly browse academic (e.g., peerreviewed articles, dissertations, theses) and grey (e.g., government reports, white papers) literatures in search of potentially relevant articles in pertinent databases (e.g., ProQuest, Web of Knowledge, and JSTOR) and general search engines (e.g., Google Scholar). Search results will be reviewed and screened independently by each research team member. Articles that provide insights on how and why unemployment insurance policies might affect poverty and health will be highlighted and retrieved for further review. Among retrieved studies, we will search for published theories that have conceptualized unemployment insurance, poverty or health within the same framework, and for ‘rich’ descriptions of potential mechanisms that speak to the factors that determine the success and failure of unemployment insurance policies. 2.1.2. Consultations with experts During the initial theory development stage, we will also consult with unemployment insurance experts. These experts will be identified during our informal browsing of academic and grey literatures. In order to obtain varied perspectives about unemployment insurance, we will consult different experts with different knowledge bases, including for example, policymakers (i.e., expertise on how policies are formulated in governments), policy analysts (i.e., expertise on how the costs and benefits of different policies are evaluated), researchers (i.e., expertise on the current state of knowledge on the effect of policies), advocates (i.e., expertise on the political process on how to influence public policy and resource allocation), and front-line personnel (i.e., expertise with the recipients of unemployment insurance benefits). Unemployment insurance experts will be contacted through e-mail and invited to participate in a phone interview to facilitate the building of theory and the identification of initial propositions. We will conduct 5 to 10 interviews with experts (e.g., 1 or 2 interviewees from each expert category noted above), and will ask them about how and why unemployment insurance policies work to alleviate poverty and to improve health. If expert interviews in a given area of knowledge (e.g., with researchers) offer minimal insights on mechanisms, we will conduct additional interviews with experts from that area (e.g., more researchers will be targeted for interviews). Taken together, initial theory development efforts will generate a working list of propositions to be tested and refined as we proceed with our realist synthesis. Our initial mechanisms of interest are two-fold: (1) ‘materialist’ mechanisms that stress the importance of material resources and opportunities such as income replace and re-entry into the labour force; and (2)

A. Molnar et al. / Evaluation and Program Planning 48 (2015) 1–9

4

‘psycho-social’ mechanisms that emphasize psychological processes and interactions with the social environment such as stress, stigma, social support, and employment strain and security. Regarding outcomes, we are interested in the consequences of unemployment insurance policies. Consistent with the realist approach, we will not rely on a single measure of poverty (e.g., percent of population below 50% of median income) or health (e.g., all-cause mortality rates). Instead, we will consider a wide range of outcomes by relying on the same poverty and health measures and operational definitions used by researchers of selected and appraised studies. This strategy will allow us to consider the widest range of possible outcomes, to identify additional CMO pattern configurations, and to investigate more deeply the workings of unemployment insurance. Given this, our poverty outcomes of interest may include but are not be limited to shortand long-term forms of absolute poverty (e.g., defined as insufficient income to meet basic living needs), relative poverty (e.g., defined in relation to the overall distribution of income), relative deprivation (e.g., constructed indexes based on access to resources other than income such as heating and clothing), and standard budgets (e.g., based on the cost of a basket of goods and services that represent a basic living standard). Health outcomes include the impact of unemployment insurance policies on population health, health inequalities, physical and mental health, and well-being outcomes, including for example, health behaviours and quality-of-life indicators. During initial theory development, our ultimate goal is to construct provisional propositions (e.g., hypotheses) that suggest how unemployment insurance policies might alter and shape contexts, trigger relevant mechanisms, which in turn, generate intended and unintended poverty and health outcomes. To make this initial theory process more concrete, we brainstormed a draft set of propositions (e.g., hypotheses) and present them in Figs. 1 and 2. For example, one proposition suggests that restrictive eligibility criteria for unemployment insurance such as meanstesting increases the risk of unemployed workers falling into poverty because it denies cash benefits those in financial need (Fig. 1). Another proposition suggests that expansive unemployment insurance promotes and protects the well-being of unemployed workers by replacing high rates of loss income for long periods of time, which in turn, buffer against the stressors of being out of work (Fig. 2). 2.2. Search strategy Our search strategy, for the collection of evidence to test our initial propositions, will follow a purposive, iterative approach.

Unemployment insurance characteristics Eligibility criteria Wage replacement rates/benefit levels Duration of benefits Ability to earn some income while receiving benefits Waiting period

After our initial development efforts, we will perform an electronic database search to bring together different sources of evidence that supports, refines, or refutes our initial theoretical framework. A medical librarian, trained in systematic reviews in the social and health sciences, will perform individual electronic searches in the following four databases: Ovid Medline, Social Sciences Citation Index by Web of Science, Ovid EMBASE, and ProQuest. The following four databases will be searched simultaneously: International Bibliography of the Social Sciences, Worldwide Political Science Abstracts, Political and International Studies (PAIS) International, and EBSCO; as well as the following six databases: FRANCIS, Sociological Abstracts, Applied Social Science Index and Abstracts, PsycINFO, EconLit, and International Political Science Abstract. In order to ensure that the number of retrieved records remains manageable, we will search for studies from 2000 to the present. No restrictions will be applied to language and type of publication. Search terms will be expansive, including unemployment insurance terms such as: ‘‘unemployment insurance’’, ‘‘employment insurance’’, ‘‘unemployment assistance’’, ‘‘employment assistance’’, ‘‘unemployment protection’’, ‘‘employment protection’’, ‘‘unemployment benefit*’’, ‘‘jobseeker’s allowance’’, ‘‘jobseeker’s benefit*’’, ‘‘workseeker’s allowance’’, ‘‘workseeker’s benefit*’’, and ‘‘unemployment compensation’’. Mechanism search terms for poverty and health will include:‘‘risk’’, ‘‘at-risk’’, ‘‘materialist’’, ‘‘neo-materialist’’, ‘‘psycho-social’’, ‘‘stress*’’, and ‘‘stigma*’’. Boolean operators will be used to combine search terms to narrow and broaden the retrieval results of our searches. To increase the comprehensiveness of our search strategy, a medical librarian will be consulted to improve our keyword search strategy and assist with searching of electronic databases. 2.3. Selection and appraisal of documents During the selection and appraisal of documents, we will screen potentially relevant abstracts and articles for: (1) relevance (does the article contribute to building or testing our theory?); and (2) rigour (does the study use methods that are credible and trustworthy?) (Pawson et al., 2004). Whereas critical appraisal tools for empirical studies are well-established (Booth, Papaioannou, & Anthea, 2012; Hannes, 2011), the most appropriate tools to appraise qualitative investigations are often debated among experts (Booth et al., 2012). In our case, we will use critical appraisal instruments and frameworks that increase transparency, accuracy, utility, and propriety (Petticrew, 2006). In the case of realist reviews, the practice of assessing study quality with ‘hierarchies of evidence’ is often rejected by realist authors (Pawson & Manzano-Santaella, 2012), who contend that a

Initial propositions Restrictive eligilibity criteria such as means-testing increases the risk of unemployed workers falling into poverty because it denies cash benefits those in financial need. Generous wage replacement rates impact poverty among unemployed workers through the replacement of loss income, which ensures that material needs such as food and housing remain met. Waiting days affect short -term poverty because the number of days unemployed workers must wait to start receiving unemployment insurance benefits determines the number of days with no cash benefits. Fig. 1. Initial propositions between unemployment insurance and poverty.

Poverty outcomes Short-term and longterm poverty Absolute and relative poverty

A. Molnar et al. / Evaluation and Program Planning 48 (2015) 1–9

Unemployment insurance characteristics Eligibility criteria Wage replacement rates/benefit levels Duration of benefits Ability to earn some income while receiving benefits Waiting period

Initial propositions

5

Health outcomes

Eligibility criteria impacts the psycho-social health of unemployed workers when means-testing is used to determine eligbility, which generates higher levels of stress compared to universalistic approaches in which all unemployed workers are entitled to receive benefits.

Well-being

Expansive unemployment insurance promotes and protects the well-being of unemployed workers by replacing high rates of loss income for long periods of time, which in turn, buffer against the stressors of being out out of work.

Population health

Physical health Mental health

Universal coverage impacts the mental health of unemployed workers since universal coverage is less stigmatizing than means-testing for eligibility.

Fig. 2. Initial propositions between unemployment insurance and health.

diversity of study methods is needed to produce the most complete and richest understanding of a given topic. The essential realist task to is describe what has been observed, including all potentially valuable insights, rather than simply excluding studies with ‘less rigorous’ methods. The underlying rationale is that the value of individual studies might only be realized at the point of synthesis, and not necessarily during the appraisal process. Hence, the responsibility for assessing study quality is ‘devolved to the reader from the researcher’ (Hannes, 2011). First, our research team will screen all potentially relevant abstracts. Abstracts will be independently reviewed by two research team members, and evaluated against the following three criteria: (1) Does the abstract refer to unemployment insurance policies?; (2) Does the abstract describe poverty or health as dependent outcomes?; and (3) Does the abstract report empirical methods to test hypotheses related to our proposed mechanisms? Abstracts will be coded as ‘Yes’ if all three inclusion criteria are satisfied, as ‘Maybe’ if the abstract satisfies at least one criteria, and ‘No’ if none of the criteria are met. In cases where no abstract is available, the title of articles will be screened for eligibility, and potentially relevant studies will be coded as ‘Promising’. Before screening the abstracts, all research team members will participate in several training exercises to increase the accuracy and reliability in selecting appropriate articles (e.g., a random sample of abstracts will be distributed among the research team, members will independently review and assess the abstracts, and selection results will be compared and discussed). After the initial screening of abstracts, the full-text of articles coded as ‘Yes’, ‘Maybe’, and ‘Promising’ will be retrieved, and evaluated by two independent reviewers for a second time to ensure that one or more of our inclusion are met. Disagreements about articles to be included and excluded will be resolved through group consensus. In our protocol, quality appraisal refers to ‘‘what has been observed without the aim of excluding further studies’’ (Hannes, 2011). Selected articles will be critically appraised with respect to methodological rigor and soundness, including research questions, sample descriptions, data collection procedures, and data analytical techniques. Primary findings will be evaluated on whether ‘how and why CMOs work’ are transferable to other contexts. All relevant information on study objectives, methods, and findings will be extracted with the use of a validated appraisal tool (Pluye, Gagnon, Griffiths, & Johnson-Lafleur, 2009), thus increasing the inter-rater reliability among independent reviewers. Regarding the appraisal of theory, our goal is to identify and build theory by proposing how and why unemployment insurance

policies may work as a prelude to testing and refining our initial theory. Specifically, we will be searching for rich theoretical descriptions on relevant mechanisms between unemployment insurance policies and poverty and health. Table 1 displays our full set of inclusion and exclusion criteria when selecting and appraising articles. 2.4. Data extraction The data extraction stage will be carried by research team members working in pairs. Extraction results will be regularly shared and discussed with the larger research team to increase transparency, ensure consistency, and enable reflexive feedback. Review teams will present potential mechanisms, and provide justification for their inclusion or exclusion from our initial theoretical framework. Box 1 presents the standardized data collection tool that will guide our extraction duties. During data extraction, our objectives are to collect information that specifically address our research questions. To these ends, our extraction efforts are designed to understand: ‘‘how and why does unemployment insurance impact short- or long-term poverty and physical and mental health?’’; ‘‘who experiences these outcomes?’’ and ‘‘what are the mechanisms and contexts that generate poverty and health outcomes?’’ In effect, the supporting rationale to extract data is to facilitate a process of frequent, repetitive framework refinement (Noyes, 2011). In doing so, individual reviewers will contribute to the collective goal of developing and refining our theoretical framework by reviewing primary articles, extracting and synthesizing data, and interpreting findings through several re-iterative cycles. Review teams will be responsible for ensuring accuracy and achieving agreement on methodological details and primary study findings. We will read, several times, the papers that have met inclusion criteria to identify (i) evidence that might support our initial theory, and (ii) emergent mechanisms not articulated by our initial theory concerning how UI impacts poverty and health. New mechanisms that emerge during our research team discussions will be added to our initial theoretical framework. The ‘richness’ of CMO configurations will reflect whether descriptions of mechanisms are ‘thick’ or ‘thin’, and whether analytic rigour is derived from empirical testing or authors’ interpretation or both. ‘Thick’ descriptions of mechanisms will be based on empirical results and will offer detailed evidence on what it is about unemployment insurance that actually brings about changes in poverty and health. Given the hidden nature of mechanisms, ‘thick’ descriptions will

A. Molnar et al. / Evaluation and Program Planning 48 (2015) 1–9

6 Table 1 Inclusion and exclusion criteria.

Study focus Study design

Study population Policy outcome

Inclusion criteria

Exclusion criteria

Unemployment benefit policies (e.g., insurance, assistance, benefit, allowance, compensation) Includes but not limited to Theory or conceptual framework Case study Qualitative description Randomized controlled trial (RCT) Non-RCT experimental or quasi-experimental study (cohort/case control/cross sectional study) Quantitative descriptive studies (incidence or prevalence study without comparison group) Mixed-methodology case study (i.e. sequential explanatory/exploratory/triangulation/embedded study design) Multiple case study (i.e. comparative/cross case/ study, typology, most different design) Systematic review Meta-analysis Adults, unemployed Includes but not limited to Short and long-term poverty Health inequalities Physical and mental health Well-being

Inquire in the impacts of (un)employment or re-employment, without discussing policy interventions Opinion-driven studies Editorial review Commentary (except for expert-informed commentary) Opinion piece Letters

Box 1. Summary of data extraction tool

 Page: Page number from which the mechanism is drawn  Paragraph: Paragraph on the page from which the mechanism is drawn  Starts with ‘‘___’’: First few words of sentence from which the mechanism is drawn  Jurisdiction: Which jurisdiction(s) is the data drawn?  Context: For whom? In what circumstances? Where? Context may not be immediately apparent in the section of text from which you draw the mechanism/outcome.  Outcome: What is the outcome (i.e., decreased poverty, improved health behaviour)?  Mechanism: How? Why?  Mechanism Number: Does this mechanism in the article reflected in the mechanisms from the initial framework? If yes, assign mechanism with the associated number from the initial framework (e.g., P1, H1)  Mechanism is based on: Is the evidence of the mechanism based on: empirical evidence, reference to literature, or author opinion/speculation?  Richness: Is there thick or thin description of the mechanism?  Additional UB articles: References found in the article that may lead to new mechanisms  Comments: General comments/concerns about the article

reveal how and why unemployment insurance either enables or constrains resources that generate poverty and health outcomes. In contrast, ‘thin’ descriptions will be derived from the interpretation or opinion of authors and will offer minimal mechanismbased insights. The reference lists of included studies will also be hand searched to identify other potentially relevant articles and CMOs. 2.5. Analysis and synthesis process Analysis and synthesis will be undertaken by each member of the research team, geared toward the identification of generative

Non-welfare state population Outcome of the policy is related to e.g. unemployment or re-employment without discussing health or poverty outcomes

explanations of causation, and treated as an iterative process. The aim here is to test and refine our CMO pattern configurations using primary findings in data sources. Emerging results will be regularly shared and discussed during weekly meetings to ensure that inferences are assessed to be consistent and accurate by all research team members. First, we will analyze data for recurrent patterns of contexts and outcomes. Recurrent patterns of outcomes are also known as demi-regularities, or semi-predictable patterns or pathways in the data (Jagosh et al., 2012). For example, higher wage replacement rates may buffer against poverty among unemployed workers. Second, we will attempt to explain context-outcome patterns by identifying mechanisms. Our analytic technique will focus on the description of mechanisms. Descriptions will be rated as ‘thick’ or ‘thin’ depending on whether the description of mechanisms provide a nuanced and insightful understanding on how and why unemployment insurance brings about changes (O’Campo et al., 2011; O’Campo et al., 2009). We may find, for example, that higher wage replacement rates reduce the risk of falling into poverty through a process of income smoothing, or the reduction of fluctuations from income security during employment to income insecurity during unemployment. Third, we will synthesize identified mechanisms to understand its interaction with contexts to influence poverty and health outcomes. Synthesis will start with studies that offer ‘thicker’ descriptions of underlying mechanisms as well as those confirmed through empirical testing. Studies that provide only ‘thin’ descriptions, or mechanisms that rely on the interpretation and opinion of authors, will be treated as theoretical insights (not evidence) and given smaller weight in the analysis and synthesis. Forth, we will use an iterative purposive searching process given that our realist synthesis will likely require additional information to understand and explain emerging findings. Snowball techniques will be used to identify additional studies from primary studies to further explanation building, including for example, to continuously search, locate, and track references in the footnotes, endnotes, and references of potentially relevant articles. Additional studies will likely that suggest other contextual influences and descriptors of mechanisms, which we will then use iteratively to repeat our search strategy.

A. Molnar et al. / Evaluation and Program Planning 48 (2015) 1–9

2.6. Presentation and dissemination of revised theory Consistent with the dissemination goals the SOPHIE project, our realist review will be followed by a multi-sector engagement process. This engagement process will be designed to gather evidence on the priorities of multiple stakeholders regarding public policies and social interventions that can potentially reduce poverty and health inequalities based on labour market conditions. Experts and stakeholders will be identified and recruited in collaboration with other SOPHIE research teams to obtain a representative group of agencies, organizations, and institutions working on unemployment insurance policies within the European Union. Fig. 3 presents a flow chart that visually depicts our successive methodological steps and major review duties. Note that the curved dotted line that runs backwards from step 5 (analysis and synthesis process) to step 2 (search strategy) allows for iterative purposive searching. 3. Discussion In this protocol paper, we have provided background information and described methodological steps about our forthcoming realist synthesis on ‘how, why, and under what circumstances does unemployment insurance impact poverty and health’. In this last section, we discuss how our realist synthesis protocol advances our understanding of unemployment insurance policies. First, our

Step 1. Initial theory development

work is the first to undertake a realist synthesis on unemployment insurance, poverty, and health and one of the few that focuses on macro-level policies. Moreover, we have presented an initial set of propositions or hypotheses that can be tested in the realist synthesis of the impact unemployment insurance on poverty and health. The focus on theory will enable us to search for causal mechanisms that operate in different contexts to generate desired and undesirable outcomes. Second, our work will synthesize different theories from disciplines as diverse as political sociology, public health, labour economics, and social psychology in order to explain poverty and health outcomes. In this respect, our theoretical explanations will constitute ‘‘middle-range theories’’, or explanations with sufficient specificity to generate testable propositions about unemployment insurance on one hand, and on the other, sufficient generalizability about types of social protection policies (e.g., disability, pensions) (Merton, 1967). And third, our realist synthesis protocol represents a ‘policy-friendly’ approach to synthesizing the evidence on unemployment insurance. By comparing ‘how unemployment insurance is supposed to work’ to ‘how such policies actually work’, we will gain a realist understanding about the foreseen and unforeseen events that influence unemployment insurance policies in generating their desired outcomes. Such knowledge has the potential to guide policymakers on which contexts to modify (e.g., active labour market policies) or what kinds of resources to enable (e.g., specialized retraining to gain new skills), which in turn, activate

Search for initial theories

Consult experts

Step 2. Search strategy

Search electronic databases using keywords and terms

Step 3. Selection and appraisal of documents

Screen potentially relevant abstracts and articles for relevance and rigour and against inclusion criteria Retrieve full-text of articles

Step 4. Data extraction

Extract relevant data using standardized collection tool

Hand search reference lists for additional potentially relevant articles

Step 5. Analysis and synthesis process

Analyze data for context and outcome patterns and synthesize mechanisms Iterative search process

Step 6. Presentation and dissemination of revised theory

7

Present revised theoretical findings with multiple stakeholders

Fig. 3. Flow chart of methodological steps and major review duties.

8

A. Molnar et al. / Evaluation and Program Planning 48 (2015) 1–9

the mechanisms that generate desired outcomes (e.g., lower rates of poverty and improved levels of health). Role of the funding source This research was supported by the European Community’s Seventh Framework Program (FP7/2007–2013) to the SOPHIE Project (Evaluating the Impact of Structural Policies on Health Inequalities and their Social Determinants and Fostering Change), Grant no. 278173. The authors also gratefully acknowledge the support of The Canadian Institute for Health Research, Grant no. 96566, and the Ontario Ministry of Health and Long-Term Care. Acknowledgement Special thanks to Carolyn Ziegler, medical librarian at St. Michael’s Hospital, Toronto, Canada for her assistance with selecting keyword search terms and electronic databases. References Allan, J., & Scruggs, P. L. (2004). Political partisanship and welfare state reform in advanced industrial societies. American Journal of Political Science, 48(3), 496– 512. Astbury, B., & Leeuw, F. L. (2010). Unpacking black boxes: Mechanisms and theory building in evaluation. American Journal of Evaluation, 31(3), 363–381. Bambra, C. (2011). Real world reviews: A beginner’s guide to undertaking systematic reviews of public health policy interventions. Journal of Epidemiology and Community Health, 65(1), 14–19. Bambra, C., & Eikemo, T. A. (2009). Welfare state regimes, unemployment and health: A comparative study of the relationship between unemployment and self-reported health in 23 European countries. Journal of Epidemiology and Community Health, 63, 92–98. Bambra, C., Gibson, M., Sowden, A., Wright, K., Whitehead, M., & Petticrew, M. (2010). Tackling the wider social determinants of health and health inequalities: Evidence from systematic reviews. Journal of Epidemiology and Community Health, 64(4), 284–291. Blamey, A., & Mackenzie, M. (2007). Theories of change and realistic evaluation peas in a pod or apples and oranges? Evaluation, 13(4), 439–455. Booth, A., Papaioannou, D., & Anthea, S. (2012). Systematic approaches to a successful review. London: Sage Publications. Chetty, R. (2008). Moral hazard versus liquidity and optimal unemployment insurance. Journal of Political Economy, 116(2), 173–234. Coryn, C. L., Noakes, L. A., Westine, C. D., & Schro¨ter, D. C. (2011). A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation, 32(2), 199–226. CSDH (2008). Closing the gap in a generation: Health equity through action on the social determinants of health: Final report of the commission on social determinants of health. Geneva: World Health Organization. Eurofound (2007). Social partners and social security systems: European foundation for the improvement of living and working conditions. Eurofound. Ferrarini, T., Nelson, K., & Sjo¨berg, O. (2014). Unemployment insurance and deteriorating self-rated health in 23 European countries. Journal of Epidemiology and Community Health. http://dx.doi.org/10.1136/jech-2013-203721 Gagliardi, A. R., Legare, F., Brouwers, M. C., Webster, F., Wiljer, D., Badley, E., et al. (2011). Protocol: Developing a conceptual framework of patient mediated knowledge translation, systematic review using a realist approach. Implementation Science, 6 http://dx.doi.org/10.1186/1748-5908-6-25 Gallie, D., & Paugam, S. (2000). Welfare regimes and the experience of unemployment in Europe. Oxford: Oxford University Press. Greenhalgh, T., Kristjansson, E., & Robinson, V. (2007). Realist review to understand the efficacy of school feeding programmes. British Medical Journal, 335(7625), 858– 861. Hannes, K. (2011). Chapter 4: Critical appraisal of qualitative research. In B. A. Noyes, J. Hannes, K. Harden, A. Harris, J. Lewin, & S. C. Lockwood (Eds.), Supplementary guidance for inclusion of qualitative research in cochrane systematic reviews of interventions (vol. 1). York: Cochrane Collaboration. Howell, D. R., & Azizoglu, B. M. (2011). Unemployment benefits and work incentives: The US labour market in the Great Recession. Oxford Review of Economic Policy, 27(2), 221–240. Jagosh, J., Pluye, P., Macaulay, A. C., Salsberg, J., Henderson, J., Sirett, E., et al. (2011). Assessing the outcomes of participatory research: Protocol for identifying, selecting, appraising and synthesizing the literature for realist review. Implementation Science, 6. http://dx.doi.org/10.1186/1748-5908-6-24 Jagosh, J., Macaulay, A. C., Pluye, P., Salsberg, J., Bush, P. L., Henderson, J., et al. (2012). Uncovering the benefits of participatory research: Implications of a realist review for health research and practice. Milbank Quarterly, 90(2), 311–346. Karanikolos, M., Mladovsky, P., Cylus, J., Thomson, S., Basu, S., Stuckler, D., et al. (2013). Financial crisis, austerity, and health in Europe. The Lancet, 381(9874), 1323–1331.

Kastner, M., Estey, E., Perrier, L., Graham, I., Grimshaw, J., Straus, S. E., et al. (2011). Understanding the relationship between the perceived characteristics of clinical practice guidelines and their uptake: Protocol for a realist review. Implementation Science, 6(69), 1–9. Katz, L. F., & Meyer, B. D. (1990). The impact of the potential duration of unemployment benefits on the duration of unemployment. Journal of Public Economics, 41(1), 45– 72. Kenworthy, L. (2008). Jobs with equality. Oxford: Oxford University Press. Kessler, R., Turner, B., & House, J. (1988). Effects of unemployment on health in a community survey: Main, modifying, and mediating effects. Journal of Social Issues, 44(4), 69–85. Lorenc, T., Petticrew, M., Welch, V., & Tugwell, P. (2013). What types of interventions generate inequalities? Evidence from systematic reviews. Journal of Epidemiology and Community Health, 67(2), 190–193. Lundberg, O., Yngwe, M. A˚., Stja¨rne, M. K., Elstad, J. I., Ferrarini, T., Kangas, O., et al. (2008). The role of welfare state principles and generosity in social policy programmes for public health: An international comparative study. The Lancet, 372(9650), 1633–1640. Marchal, B., van Belle, S., van Olmen, J., Hoere´e, T., & Kegels, G. (2012). Is realist evaluation keeping its promise? A review of published empirical studies in the field of health systems research. Evaluation, 18(2), 192–212. Marmot, M., Friel, S., Bell, R., Houweling, T. A., & Taylor, S. (2008). Closing the gap in a generation: Health equity through action on the social determinants of health. The Lancet, 372(9650), 1661–1669. Merton, R. (1967). On theoretical sociology. Five essays old and new. New York, NY: The Free Press. Muntaner, C., Borrell, C., Ng, E., Chung, H., Espelt, A., Rodriguez-Sanz, M., et al. (2011). Politics, welfare regimes, and population health: controversies and evidence. Sociology of Health & Illness, 33(6), 946–964. Nelson, K. (2013). Social assistance and EU poverty thresholds 1990–2008. Are European welfare systems providing just and fair protection against low income? European Sociological Review, 29(2), 386–401. Norton, A., Conway, T., & Foster, M. (2002). Social protection: Defining the field of action and policy. Development Policy Review, 20(5), 541–567. Noyes, J. L. S. (2011). Chapter 5: Extracting qualitative evidence. In J. Noyes, A. Booth, K. Hannes, A. Harden, J. Harris, S. Lewin, & C. Lockwood (Eds.), Supplementary guidance for inclusion of qualitative research in cochrane systematic reviews of interventions. Oxford: Cochrane Collaboration Qualitative Methods Group. O’Campo, P., Kirst, M., Schaefer-McDaniel, N., Firestone, M., Scott, A., & McShane, K. (2009). Community-based services for homeless adults experiencing concurrent mentalhealth and substance use disorders: A realist approach to synthesizing evidence. Journal of Urban Health, 86(6), 965–989. O’Campo, P., Kirst, M., Tsamis, C., Chambers, C., & Ahmad, F. (2011). Implementing successful intimate partner violence screening programs in health care settings: Evidence generated from a realist-informed systematic review. Social Science & Medicine, 72(6), 855–866. Pawson, R. (2006). Evidence-based policy: A realist perspective. London: Sage Publications. Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2004). Realist synthesis: An introduction ESRC research methods programme. RMP methods paper. University of Manchester. Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005). Realist review—A new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy, 10(suppl 1), 21–34. Pawson, R., & Manzano-Santaella, A. (2012). A realist diagnostic workshop. Evaluation, 18(2), 176–191. Pawson, R., & Tilley, N. (1997). Realistic evaluation. London: Sage Publications Ltd. Petticrew, M. R. H. (2006). Systematic reviews in the social sciences. Oxford: Blackwell Publishing. Pluye, P., Gagnon, M.-P., Griffiths, F., & Johnson-Lafleur, J. (2009). A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. International Journal of Nursing Studies, 46(4), 529–546. Robert, E., Ridde, V., Marchal, B., & Fournier, P. (2012). Protocol: A realist review of user fee exemption policies for health services in Africa. BMJ Open, 2(1.) http:// dx.doi.org/10.1136/bmjopen-2011-000706 Rodriguez, E. (2001). Keeping the unemployed healthy: The effect of means-tested and entitlement benefits in Britain, German and the United States. American Journal of Public Health, 91(9), 1403–1411. Rodriguez, E., Frongillo, E. A., & Chandra, P. (2001). Do social programmes contribute to mental well-being? The long-term impact of unemployment on depression in the United States. International Epidemiological Association, 30, 163–170. Scruggs, L., & Allan, J. (2006). Welfare-state decommodification in 18 OECD countries: A replication and revision. Journal of European Social Policy, 16(1), 55–72. Shiller, R. J. (2008). The subprime solution: How today’s global financial crisis happened and what to do about it. Princeton: Princeton University Press. SOPHIE (2014). Evaluating the impact of structural policies on health inequalities and their social determinants, and fostering change. hhttp://www.sophie-project.eu/ index.htmi. Sridharan, S., & Nakaima, A. (2012). Towards an evidence base of theory-driven evaluations: Some questions for proponents of theory-driven evaluation. Evaluation, 18(3), 378–395. Stuckler, D., Basu, S., Suhrcke, M., Coutts, A., & McKee, M. (2009). The public health effect of economic crises and alternative policy responses in Europe: An empirical analysis. The Lancet, 374(9686), 315–323.

A. Molnar et al. / Evaluation and Program Planning 48 (2015) 1–9 Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., & Pawson, R. (2013). RAMESES publication standards: Realist syntheses. BMC Medicine, 11(21), 1–14. Agnes Molnar holds an MD and PhD in Health Sciences. She contributed to the research as a post-doctoral fellow in the ACHIEVE Research Partnership: Action for Health Equity Interventions at the Centre for Research on Inner City Health at St. Michael’s Hospital. E-mail: [email protected]. Patricia O’Campo is a social epidemiologist, director of the Centre for Research on Inner City Health at St. Michael’s Hospital, professor at the Dalla Lana School of Public Health, University of Toronto and adjunct professor at Johns Hopkins Bloomberg School of Public Health. E-mail: O’[email protected]. Edwin Ng completed his PhD in Social Science and Health in the Dalla Lana School of Public Health, University of Toronto. Currently, he is an ACHIEVE post-doctoral fellow at the Centre for Research on Inner City Health at St. Michael’s Hospital. E-mail: [email protected]. Christiane Mitchell holds a Master in Public Health, and is currently a research coordinator at the Centre for Research on Inner City Health, at St. Michael’s Hospital.

9

Carles Muntaner is professor at the Bloomberg Faculty of Nursing and Dalla Lana School of Public Health, University of Toronto. His research interests include social inequities in health, social epidemiology, work organization, employment conditions, race/ethnicity, and the philosophy of population health. E-mail: [email protected]. Emilie Renahy completed her PhD in public health at the Research Team on Social Determinants of Health and Healthcare (DS3) at the French National Institute for Health and Medical Research (Inserm) and Pierre et Marie Currie University (Paris 6). She is currently an ACHIEVE post-doctoral fellow at the Centre for Research on Inner City Health at St. Michael’s Hospital. E-mail: [email protected] Alexander St. John is currently a law student at Western University, London, Ontario, and was a research coordinator at the Centre for Research on Inner City Health, St. Michael’s Hospital during the preparation of this study protocol. E-mail: [email protected]. Ketan Shankardass is a researcher with training in epidemiology, public health, biology, and geography. He is currently an assistant professor in the Department of Psychology and the Health Sciences Program at Wilfrid Laurier University. E-mail: [email protected].