EDUCATION

Research methods for graduate students: A practical framework to guide teachers and learners Patricia F. Pearce, MPH, PhD, FNP, FAANP (Associate Professor)1 , Becky J. Christian, PhD, RN (Professor and Interim Chair, Family, Child Health, & Caregiving Department School of Nursing)2 , Sandra L. Smith, PhD, APRN, NNP-BC (Associate Professor)3 , & David E. Vance, PhD, MGG (Associate Professor, Associate Director, Center for Nursing Research)2 1

Loyola University College of Social Sciences, School of Nursing, New Orleans, Louisiana University of Alabama at Birmingham School of Nursing, Birmingham, Alabama 3 University of Louisville School of Nursing, Louisville, Kentucky 2

Keywords Research; education; critical thinking; pedagogy; evidence-based practice; curriculum. Correspondence Patricia F. Pearce, MPH, PhD, FNP, FAANP, Loyola University, College of Social Sciences School of Nursing, 6363 St. Charles Avenue, Stallings Hall #212C, New Orleans, LA 70118. Tel: 205-541-8988; E-mail: [email protected], [email protected] Received: June 2013; accepted: August 2013 doi: 10.1002/2327-6924.12080

Abstract Purpose: The purpose of this article is to present the Arrow Framework for Research Design, an organizing framework that facilitates teaching and learning of research methods, providing logical organization of interrelationships between concepts, content, and context of research methods, and practice application. The Arrow Framework was designed for teaching and learning research methods to facilitate progression of knowledge acquisition through synthesis. Data sources: The framework was developed over several years and used successfully to teach masters, DNP, and PhD nursing students across five universities. The framework is presented with incremental graphics and narrative for teaching. Conclusion: The Arrow Framework provides user-friendly information, in an organized and systematic approach demonstrated as successful for teaching and learning the foundational language of research, facilitating synthesis and application in scholarly endeavors. Implications for practice: The Arrow Framework will be useful for educators and students in teaching and learning research language, relationships, and application of methods. The materials are easily adaptable to slide or paper presentation, and meet learner needs for narrative and visual presentation. Teaching research design to graduate students is critical to meet the expectation that students are to understand the scientific underpinnings of nursing science and appropriate use of evidence that are essential for well-educated practitioners.

Introduction Understanding the scientific underpinnings of nursing is a critical skill for all nurses, and is the hallmark of a nursing professional, as well as a major component of the Essentials for the master’s degree in nursing (MSN), the clinical doctor of nursing practice (DNP) programs, and recommendations for doctor of philosophy (PhD) programs (American Association of Colleges of Nursing [AACN], 2001, 2006, 2011). Expectations for nurses, such as those identified by the Institute of Medicine (2010), to lead change in the quest for advancing health care require an understanding of evidence, as well as

the generation and practical use of evidence upon which to base practice. These themes are echoed in a recent call to action for nursing (Benner, 2010), and in nursing research and evidence-based practice textbooks (Grove, Burns, & Gray, 2013; Hall & Roussel, 2014; Melnyk & Fineout-Overholt, 2011; Polit & Beck, 2012). Regardless of program emphasis, a critical factor to understanding science lies in understanding an encompassing perspective of evidence, including the language of research, research methods, and the use of data. Completion of any graduate program and related scholarly projects encompasses use of all levels of Bloom’s taxonomy that hierarchically builds upon the acquisition of

C 2013 The Author(s) Journal of the American Association of Nurse Practitioners 26 (2014) 19–31 

 C 2013 American Association of Nurse Practitioners

19

Research methods for graduate students

knowledge, then comprehension, application, analysis, synthesis, and evaluation (Anderson & Krathwohl, 2001; Bloom, 1974; Forehand, 2005). The emphasis in the clinically oriented MSN and DNP programs is to prepare advanced practice nurses (APRN) and highly skilled clinicians, who must be able to discuss evidence adeptly to function fully in the rapidly evolving healthcare arena. MSN- and DNP-prepared nurses, including APRNs, must be able to critically appraise and critique published reports that include evidence, and then translate that evidence into practice and practice-related scholarly projects (AACN, 2006, 2011). Further, for APRNs, understanding of evidence is required in order to evaluate their practice outcomes and productivity. Thus, to understand the process, application, and translation of science, graduate nursing education necessarily must include coursework for MSN and DNP students that is rich in research methods, basic statistics, and the overall use of evidence, but not as in-depth as required in a PhD program (AACN, 2006, 2011).

Purpose The purpose of this article is to present the Arrow Framework for Research Design, an organizing framework that facilitates teaching and learning of research methods, with logical and practical organization of the interrelationships between concepts, content, and context of research methods, as well as application of evidence to practice. The Arrow Framework was developed from several years of experience in teaching research methods and statistics, and then used effectively by four faculty at five universities, in teaching hundreds of MSN, DNP, and PhD students. The practical framework provides both educators and students with an overall schematic and visual displays for organizing research methods content, understanding the hierarchy of designs and level of evidence, facilitating the educator’s ability to present the related information on the research designs within the paradigms in an understandable manner, and providing ample opportunities for faculty to address the paradigmatic philosophical and historical underpinnings. Further, use of the Arrow Framework enhances the student’s ability to grasp sufficient research knowledge to critically evaluate research reports, develop their scholarly projects, and use the information adeptly in practice. The framework is introduced in a layered, hierarchical, systematic format, and presented sequentially in a visual format to students to facilitate critical thinking. Application of the framework to the major research paradigms, quantitative, qualitative, and mixed methods, and their related components is addressed, and examples are included to illustrate the concepts. The quanti20

P. F. Pearce et al.

tative paradigm (positivist), the qualitative paradigm (also called naturalistic, interpretive, or constructivist), and the mixed methods paradigm (pragmatic) are well-established (Creswell, 2012, 2013; Denzin & Lincoln, 2011; Polit & Beck, 2012). The philosophical foundations are emphasized in the framework using only categorical descriptors. Augmenting class discussions about the foundations of research and evidence-based practice is critical for student understanding and helpful for students to understand the overall paradigms and designs.

Background and significance The use of evidence, therefore necessarily research, is the foundation of nursing practice, established by Florence Nightingale in the late 1800s (Dossey, 2010). Educating nurses in the language and practice of research, however, is challenging and many barriers exist to integrating research into practice (DiCenso, Guyatt, & Ciliska, 2005; Fineout-Overholt, Melnyk, & Schultz, 2005; Funk, Tornquist, & Champagne, 1995; Hall & Roussel, 2014; Jones, Crookes, & Johnson, 2011; Melnyk & FineoutOverholt, 2011; Melnyk et al., 2004; Polit & Beck, 2012; Schmidt & Brown, 2011). Understanding research language is one barrier that has been well documented in the literature (Kelly, Turner, Gabel Speroni, McLaughlin, & Guzzetta, 2013; McLaughlin, Gabel Speroni, Kelly, Guzzetta, & Desale, 2013). In a systematic review of 45 published reports on research utilization, Squires, Estabrooks, Gustavsson, and Wallin (2011) demonstrated that a positive attitude of nurses to research was the primary factor related to research utilization. APRN identified similar barriers to using evidence in practice; yet, APRNs are licensed to provide advanced care and, therefore, they must be highly skilled at evaluating evidence at a more in-depth level, applying evidence in practice, and generating practice-based initiatives that rely on evidence and outcomes. In a descriptive qualitative study of transition from RN to APRN role, understanding evidence and its use in practice was identified as by participants as a high priority (Spoelstra & Robbins, 2010). Data are considered to be evidence and nurses use data and evidence in every patient encounter. For example, measurement of laboratory values provides data that are essential for everyday practice. Each value is scrutinized for the expected range in relationship with patient age and gender. Critical decision making is necessary regarding the validity of the measurement, as well as application in the contextual issues related to the patient situation. However, laboratory values are not generally considered by nurses and students to be a form of data or evidence. Further, nurses and students have difficulty translating

P. F. Pearce et al.

the use of these data and evidence from clinical practice to the language and context of research. Similarly, differential diagnosis requires an extensive understanding of data and evidence from history and physical exam to determine a culminating diagnosis, but these related data are not labeled necessarily as evidence by APRNs. Although data in clinical practice are considered clinical data, or clinical evidence, and are not collected generally for research purposes, clinical examples help to translate research concepts and principles for clinicians. Introducing research language, such as dependent and independent variables, validity and reliability, level of evidence, and causality, typically produces glazed looks or silence from students. By providing the linkages to clinical practice, the concepts become more clear, and the application of information is more easily understood, and students are better able to generalize the clinical concepts to research. Understanding the basic concepts and definitions related to research is essential for developing foundational knowledge about evidence, and moving toward critical evaluation and application of the more in-depth details of research. Every graduate program includes one or more research courses with labels ranging from scholarly inquiry, evidencebased practice, research, or research methods, and with some that include statistical content and others with statistics as a separate course. Information in these courses is scaled to the level of the educational program, with more in-depth information and expectations for learning, progressing from BSN, through MSN, DNP, and PhD programs. Often research is taught in graduate nursing programs as a survey course, yet the content material is dense and difficult to cover thoroughly without creating confusion. Students are expected to move from knowledge acquisition through application and synthesis in one semester or through several sequential courses. However, the baseline for all levels requires students to understand research language, such as research design, analysis, critique, and level of evidence (AACN, 2006, 2011; Benner, 2010; Polit & Beck, 2012).

The Arrow Framework for research design In order to illustrate the range of research designs, as well as distinct differences in research paradigms and the relationships of the designs to each other, a graphic in the form of an arrow was created as the visual framework. The arrow serves as the foundation for representing research paradigms, and then superimposing additional information, building the detail and complexity of research methods. A visual graphical display is a useful technique for presenting information in a manner that supports both teaching and learning a new language, cre-

Research methods for graduate students

ating a framework to make complex information more organized, providing logical linkages between concepts, and addressing information from knowledge acquisition through synthesis (Forehand, 2005; Harris, 1996; Tufte, 1990, 2001). The foundational arrow depicting the relationship between qualitative, quantitative, and mixed methods research paradigms (see Figure 1) then is augmented with extensive, incremental layering of related research methods, scientific rigor, and analysis details. The arrow was selected as the visual form for presenting research methods framework because the arrow shows the continuum or range, without prioritizing one facet over another, as recommended for high-quality symbol use in graphic design (Harris, 1996; Tufte, 1990, 2001). Thus, the arrow visually demonstrates the continuum of research paradigms. As additional concepts and more complexity are introduced, the arrow is expanded to include multiple layers with additional concepts superimposed to depict these interrelationships. With the introduction of each new facet of research, the original information moves to the background (gray), and new information highlighted in bold. Thus, the basic arrow transforms into a series of arrows used throughout the course to illustrate and translate the increasing complexity of research designs.

Major research paradigms With a need for educators to explain the research continuum, and students to understand the predominant traditions in research, commentary accompanies the Arrow Framework (see Figure 1) to explain the range of research, including the philosophical foundations of the research paradigms. The left side of the arrow represents the qualitative design paradigm, the right side represents the quantitative design paradigm, and the center depicts mixed methods. To support students’ understanding of the continuum and range, an analogy to the differences between political parties ranging from conservative and liberal political viewpoints found in many countries has demonstrated usefulness, and generally is easily understood. Thus, using the horizontal axis, the right side of the arrow is perceived as the quantitative research paradigm that is more conservative, traditional, and corresponds to deductive scientific method that students learned in gradeschool science; while the left side is depicted as the qualitative research paradigm that is less traditional, less conservative, more liberal viewpoints, predominantly using more inductive processes than deductive; and the middle section, mixed methods, represents a more moderate stance that includes components of designs from each of the anchoring ends, or paradigms. On the vertical axis, extremes fall at the outer ends of the arrow, and the baseline, with 21

Research methods for graduate students

P. F. Pearce et al.

Major Research Paradigms Research Designs

QUALITATIVE

MIXED METHODS

QUANTITATIVE

Figure 1 Three major research paradigms.

the least complex at the inner lower aspects of the arrow. The philosophical and historical nuances of the paradigms can be introduced with this initial arrow in order that students understand the basis for each paradigm. It is important to note that the characteristics of the various paradigms are not necessarily right or wrong, or better or worse from a research methods perspective, but simply represent differing approaches along a continuum of research.

Research designs Once the idea of a continuum with the major research paradigms is introduced, the next step involves superimposing the associated research design labels with the respective paradigms on the arrow (see Figure 2). Critical to this presentation is explaining the stair-stepped layering pattern comprising each paradigm that is represented in upward (vertical) and outward (horizontal) placement of the terms. The easiest manner of explaining the Arrow Framework is with the stair-step design pattern; the increasing design complexity is depicted with movement of the designs outward along the continuum . The most basic design in each paradigm begins with descriptive design, moving vertically upward and horizontally outward (to the right for quantitative, and left for qualitative) on the arrow to the more complex design, representing interpretive orientation. A caveat is that the hierarchical nature of quantitative design is clear and generally accepted; however, for the qualitative paradigm, the hierarchy of designs is dependent on the complexity of the analysis and overall procedures (Creswell & Plano Clark, 2011; Denzin & Lincoln, 2011; Morse & Field, 1995; Morse & Niehaus, 2009; Tashakkori & Teddlie, 2003a). Thus, a bracket to the left in the qualitative design area of the Arrow Framework indicates the level of complexity in these designs, especially as related to interpretive analysis, can vary; thus, the designs do not fall into a clear hierarchical fashion. For mixed methods, the complexity is determined by the types of design combined from the quantitative and qualitative paradigms, the emphasis on the type of de22

sign, and the sequencing of the related activities (Creswell & Plano-Clark, 2006; Denzin & Lincoln, 2011; Morse & Field, 1995; Morse & Niehaus, 2009; Tashakkori & Teddlie, 2003a). The listing of designs provides the student with information regarding generally accepted level of evidence, ranging from the lowest to the highest levels within the qualitative and quantitative designs, as well as across the research paradigms. Because mixed methods design is understood most easily in context of the designs selected from the quantitative and qualitative paradigms, generally presentation of the details of mixed methods design follows discussion of quantitative and qualitative designs. However, students must be cautioned that the research question posed by the investigator drives study design choice, and design choice subsequently influences level of data and overall evidence. A qualitative design is considered as high level of evidence for research questions aimed at understanding the meaning of a phenomenon; whereas, a quantitative design is more frequently used for comparative questions (Polit & Beck, 2012). The major research designs are presented in the framework (Figure 2), and although this list is extensive, it is not exhaustive and other designs may be included at the discretion of the faculty. The designs and related level of evidence produced are extensively detailed in research methods textbooks (Grove et al., 2013; Polit & Beck, 2012).

Research question and hypothesis With readings and discussion, students generally grasp a basic understanding of the research designs and paradigms, but with introduction of research questions and hypotheses, students can become confused. Explanation of the differences and uses of research questions and hypotheses is important, because research questions and hypotheses should link directly to the research design choice, and this relationship to design is shown in the Arrow Framework graphic (Figure 3). Superimposing the idea of research question and hypothesis, this arrow covers the entire paradigmatic span, in which a research question

Research methods for graduate students

P. F. Pearce et al.

Major Research Paradigms

Qualitative and Quantitative Designs Meta-Synthesis Meta-Summary Discourse Analysis Phenomenology Grounded Theory Ethnography Narrative Interpretive Descriptive

Meta-Analysis Multi-Site RCT Interventional/RCT Experimental Quasi-Experimental Correlational Cross-Sectional Case Control Descriptive

QUALITATIVE

MM

QUANTITATIVE

Figure 2 Major research designs by paradigm, emphasizing qualitative and quantitative. Note. MM, mixed methods; RCT, randomized controlled trial. For qualitative research designs (bracket on left) between descriptive (baseline, foundational design) and metasummary and metasynthesis that require inclusion of multiple already existent research, the designs can be layered in a variety of different orders, representing complexity in interpretation. There is no one hierarchical schema for qualitative (constructivist) design. The Arrow Framework reflects overall designs and a general framework orientation. Major Research Paradigms Research Designs Research Question and Hypothesis Hypothesis Research Question Meta-Synthesis Meta-Summary Discourse Analysis Phenomenology Grounded Theory Ethnography Narrative Interpretive Descriptive QUALITATIVE

Meta-Analysis Multi-Site RCT RCT Experimental Quasi-Experimental Correlational Cross-Sectional Case Control Descriptive MM

QUANTITATIVE

Figure 3 Research questions and hypotheses across paradigms. Note. MM, mixed method; RCT, randomized control trial.

can be used appropriately, and designates a formal hypothesis as used only in quantitative design when sufficient evidence exists to formulate a hypothesis (Figure 3).

Sampling design The next layer of design characteristics focuses on sampling design, a critical factor for any research or project design (Grove et al., 2013; Polit & Beck, 2012). Definitions and application of sampling design must be discussed, but positioning within the arrows helps students to solidify their thoughts about the relationship of sampling design to the respective research paradigms, and the research

designs within the paradigms. Figure 4 includes designation of the predominant form of sampling with purposive sampling, unique to qualitative research, and convenience sampling in quantitative design. Probability sampling aligns with the more complex quantitative research designs and rarely, if ever, occurs in qualitative designs. Students are often confused by sampling designs reported in published literature, thus, a critical piece of information in explaining sampling design is that researchers often use a creative mix of various forms of sampling, including strata, staged, quota, etc. (Polit & Beck, 2012). In mixed methods design, multiple forms are also used. The tendency in sampling design decisions is for heightened 23

Research methods for graduate students

P. F. Pearce et al.

Major Research Paradigms Research Designs Sampling Design Probability Purposive

Convenience

Meta-Synthesis Meta-Summary Discourse Analysis Phenomenology Grounded Theory Ethnography Narrative Interpretive Descriptive QUALITATIVE

Meta-Analysis Multi-Site RCT RCT Experimental Quasi-Experimental Correlational Cross-Sectional Case Control Descriptive MM

QUANTITATIVE

Figure 4 Sampling design across research designs and paradigms. Note. MM, mixed method; RCT, randomized control trial.

representativeness in the quantitative designs and for more emergent sampling in qualitative designs. However, it is essential for students to understand how researchers mix sampling designs, and how to identify integrated sampling designs. The arrow schema provides an overall depiction, with the details instructed in discussions of sampling designs overall.

Data types and level of measurement Data types and level of measurement are introduced after sampling design in the Arrow Framework (Figure 5). Each research paradigm is hallmarked by the specific type of data that comprise the predominant focus. In qualitative research, investigators emphasize narrative text, while numeric orientation is emphasized in quantitative research, and both forms are used in mixed methods design. The distinction between qualitative and quantitative designs is not to imply that qualitative researchers ignore numbers, counting, or other numeric facets of their research, but to emphasize the preponderant orientation to data in each paradigm (Chang, Voils, Sandelowski, Hasselblad, & Crandell, 2009; Morse & Field, 1995; Polit & Beck, 2012; Sandelowski, 2001, 2010). It is important to note that the complexity of interpretation of both narrative and numeric data increases as the designs become more complex, that is, with movement upward and outward in the designs listed on the arrow, the more complex the level of data and interpretation, and the more removed from the raw data, regardless of qualitative or quantitative research designs. For example, in qualitative design, metasynthesis requires inclusion of all published qualitative research reports on the given topic, 24

thus, is more complex and highly interpretive than descriptive design studies, and is placed farther out and upward on the arrow. In contrast, with descriptive qualitative research, analysis stays in close relationship to the raw data (Sandelowski, 2000b, 2010). Designs between descriptive and the metasummary and metasynthesis designs in the qualitative paradigm are reordered to represent increasing complexity in analysis and interpretation. In the quantitative paradigm, meta-analysis is considered the highest form of evidence (Polit & Beck, 2012); however, meta-analysis is a form of secondary analysis that is removed from the raw data analysis in the primary research studies, because the data are often drawn from published studies, or raw data are frequently obtained from researchers for the secondary analysis.

Scientific rigor Overall scientific rigor, required in the paradigms, while conceptually has similar application across the research paradigms, varies in philosophical basis, function, and application, the components of scientific rigor are particularly difficult for students to understand (Figure 6). The difficulty is especially true regarding systematic data collection and analysis of quantitative data, and the psychometric properties of reliability and validity of measurement tools. The use of the terms reliability and validity in relationship to properties of a data collection instrument, or the manner in which measurements are performed, are frequently confused by students with the terms internal and external validity in relationship to the reported study. It is critical for students to understand contributions to internal validity from conceptualization, identification of

Research methods for graduate students

P. F. Pearce et al.

Major Research Paradigms Research Designs Data Types Narrative

Numeric

Meta-Synthesis Meta-Summary Discourse Analysis Phenomenology Grounded Theory Ethnography Narrative Interpretive Descriptive QUALITATIVE

Meta-Analysis Multi-Site RCT RCT Experimental Quasi-Experimental Correlational Cross-Sectional Case Control Descriptive MM

QUANTITATIVE

Figure 5 Data types for each paradigm: narrative and numeric. Note. MM, mixed method; RCT, randomized control trial. Major Research Paradigms Research Designs Rigor Narrative

Numeric

Meta-Synthesis Meta-Summary Discourse Analysis Phenomenology Grounded Theory Ethnography Narrative Interpretive Descriptive QUALITATIVE

Confirmability Creativity Congruence Criticality Explicitness Thoroughness Sensitivity Vividness

Meta-Analysis Multi-Site RCT RCT Experimental Quasi-Experimental Correlational Cross-Sectional Case Control Descriptive MM

Trustworthiness Credibility Consistency Transferability

QUANTITATIVE

Measures --Reliability and Validity Internal Validity External Validity

Figure 6 Scientific rigor. Note. MM, mixed method; RCT, randomized control trial.

study variables, sampling design, and data collection procedures and analysis. External validity, or generalizability, entails the larger picture of understanding of whether or not the research results can be placed into context of the larger target population or similar populations, geographic location and settings, and time. The underlying constructs, definitions, and operationalization of internal and external validity in relation to scientific rigor are essential for the design of a quantitative research study, and for understanding research and for evaluating a published report. For example, it is critical for students

to understand that investigators who conduct quantitative research emphasize the importance of external validity; while the potential for applying findings to other populations is emphasized less in qualitative design, but is considered as one aspect of transferability in the qualitative paradigm. In the quantitative research paradigm, reliability and validity of data collection measures or tools are paramount, including: (a) varying forms of reliability (e.g., internal consistency, test–retest, parallel form), (b) validity (e.g., face, content, construct, convergent, discriminant), (c) application 25

Research methods for graduate students

for data collection processes (inter-rater and intrarater reliability), as well as the overall notions of internal validity and external validity (Grove et al., 2013; Polit & Beck, 2012). Each of these procedures requires detailed information for students; however, the Arrow Framework provides a context for the forms of rigor used. A vital component for student understanding is the linkage between the level of data, measurement parameters and processes, and level of evidence. Rigor in qualitative design includes emphasis on credibility, trustworthiness, overall consistency, and transferability and numerous techniques are used to assure that rigor is met (Denzin & Lincoln, 2011; Morse, 1999; Morse, Barrett, Mayan, Olson, & Spiers, 2002; Sandelowski, 1986, 1993). The basic components expand further to include components of criticality, congruence, explicitness, integrity, sensitivity, and vividness (Whittemore, Chase, & Mandle, 2001). These components of scientific rigor are superimposed on the arrow graphic in relationship to the qualitative and quantitative research paradigms (Figure 6), and are used selectively in mixed methods design. Thus, rigor is desired and useful for each of the major paradigms, but viewed in a different manner, and established with differing techniques and procedures.

Data analysis Figure 7 represents information needed for the two major steps of data analysis, crossing the spectrum of research paradigms and research designs in the Arrow Framework. Although in most studies there are multiple steps in data analysis, those more comprehensive steps are simplified for presentation purpose, and discussed with more detail in course instruction. In Step 1, descriptive statistics, such as frequencies for categorical variables (nominal and ordinal), and measures of central tendency and variability for continuous variables (interval and ratio data) are used by both qualitative and quantitative researchers. At minimum, investigators use these statistical calculations for describing sample characteristics and overall demographics. Although investigators working in qualitative research emphasize primarily textual, narrative data, counting or tallying in terms of codes or related study artifacts is important. In quantitative research, in addition to the sample demographics, results for all variables are included in the descriptive statistics step analysis (Step 1). It is critically important to emphasize to students that basic descriptive statistics (e.g., frequency, measures of central tendency, and variability) are used in describing the characteristics of the sample in both qualitative and quantitative 26

P. F. Pearce et al.

design paradigms in order to understand study context, as well as applicability (Figure 7). Data analysis Step 2 (see Figure 7) varies substantially across the paradigms and research designs. In qualitative research, data analysis emphasizes words or narrative text, thus, words are the data that are managed and analyzed, and are the primary unit of analysis. Qualitative analysis procedures generally align with the methodologic tradition, but remain flexible. In general, content analysis begins with the identification of categories, themes, and patterns for basic descriptive analysis of data. The Arrow Framework includes the major traditions, with baseline descriptive orientation at the lowermost portion of the arrow with metasynthesis at the uppermost portion vertically. For quantitative research design (Figure 7), numericbased analysis ranges from simple descriptive statistics, through inferential (parametric and nonparametric; e.g., chisquare, t-test, ANOVA, correlation), and multivariate statistics (e.g., multiple regression, ANCOVA, MANOVA, MANCOVA, GLM, survival, path analysis, SEM). Understanding the hierarchy of statistical tests and the related assumptions is essential. Placement on the Arrow Framework is intended to represent these designs and statistical analyses in their hierarchical context. More details related to each analytic method can be found in standard research, statistics, and qualitative research methods textbooks (Denzin & Lincoln, 2011; Grove et al., 2013; Polit & Beck, 2012; Tabachnick & Fidell, 1996).

Mixed methods design Mixed methods design is represented at the center of the Arrow Framework, overlapping qualitative and quantitative designs (see Figure 8), with curved symbols representing the flow of designs from each paradigm. The depiction oversimplifies mixed methods design, but provides a basic visual display to indicate that mixed methods design involves components from each paradigm. Creswell and Plano-Clark (2011) identifies six mixed method designs: three basic (i.e., Convergent, Explanatory Sequential, Exploratory Sequential ) and three advanced (i.e., Embedded, Transformative, and Multiphase). Mixed methods design includes different combinations of patterns with different emphasis on research paradigms. For example, sequential mixed methods design emphasizes a quantitative design followed by a qualitative component (QUANT -> qual), a primary qualitative design followed by quantitative component (QUAL -> quant), or a balanced, simultaneous design with equal emphasis (QUANTQUAL; Creswell, 2012, 2013; Tashakkori & Teddlie, 2003a, 2003b). For discussion on mixed methods design, the visual display can be expanded to

Research methods for graduate students

P. F. Pearce et al.

Major Research Paradigms Research Designs Data Analysis Narrative

Numeric

Meta-Synthesis Meta-Summary Discourse Analysis Phenomenology Grounded Theory Ethnography Narrative Interpretive Descriptive QUALITATIVE

Step One Descriptives

Meta-Analysis Multi-Site RCT RCT Experimental Quasi-Experimental Correlational Cross-Sectional Case Control Descriptive MM

QUANTITATIVE

Frequencies (Nominal, Ordinal) Central Tendency (Interval, Ratio)

Step Two Qualitative Analysis Meta-Synthesis Critique&Analysis Discourse Analysis Phenomenological Analysis Grounded Theory Analysis Ethnographic Analysis Narrative Analysis Interpretive Analysis Descriptive Content Analysis Categories, Themes, & Patterns

Multivariate & Inferential Statistics SEM Causal Modeling/Path Analysis Survival Analysis Logistic Regression Multiple Regression General Linear Model ANCOVA, MANCOVA ANOVA, MANOVA T-test Correlation Chi-Square

Figure 7 Data analysis across research paradigms and research designs.

Major Research Paradigms Research Designs

QUALITATIVE

MIXED METHODS QUANTITATIVE

Figure 8 Display of mixed methods with other major paradigms.

include several arrows depicting the varying forms of mixed methods sequencing options. Selection of the type and sequence of mixed methods designs is dependent on the research questions posed by the investigator, but mixed method design for research is used increasingly, and can maximize the strengths in each paradigm and design, while minimizing the weaknesses (Creswell & Plano-Clark, 2006; Morse & Niehaus, 2009; Sandelowski, 2000a; Tashakkori & Teddlie, 2003a, 2003b).

Until students have a basic understanding of the quantitative and qualitative research paradigms, and overall research designs, the introduction of mixed method design research adds a layer of complexity that is potentially confusing for students. Thus, mixed method design is best discussed in detail following discussion of quantitative and qualitative designs, because use of mixed method design integrates design based on contribution from qualitative and quantitative designs and techniques.

27

Research methods for graduate students

Application of framework The Arrow Framework has been used effectively in face-to-face classroom teaching, as well as in both synchronous and asynchronous online, distance-accessible research methods courses. The framework is easily replicated in basic nonanimated or animated PowerPoint slides and documents for teaching similar to the figures in this article, adding the layers of complexity to illustrate the interrelationships among concepts as needed for instruction. For use in application-type exercises that are especially critical for adult learners (Knowles, Holton, & Swanson, 2011; Merriam, 2007), the Arrow Framework can be used following discussion regarding a particular facet of research methods. For example, walking students step-by-step through a critique of a published research article is useful, with integrated use of the Arrow Framework for each section of the article, and again as the culminating point of the critique. Queries can be posed to facilitate student understanding, such as What research design (or sampling design, etc.) did the investigators use in this research, or where exactly on the arrow would the design reported in this article fall? Or, the investigators did not identify the design used in their reported study—what are the design characteristics found in the report, and where would those characteristics lead you to place the design on the arrow? Followup with questions regarding the rationale for a student’s decision to select placement at a particular point on the framework helps the student to articulate the specific criteria used in their decision making, moves the student to application within the Arrow Framework, and identifies the relevance to their own scholarly or practice endeavors. Student evaluation comments regarding use of the Arrow Framework have been very positive. For example, the arrow used as an instructional method was really easy to follow (Fall Semester 2010); Most importantly, the arrow helped me to read and understand published research (Fall Semester 2011); and Visual aids like the arrow really helped to understand the material (Fall Semester 2011). Students have corresponded with faculty following graduation regarding the usefulness of the Arrow Framework as well, for example, “I thought of you today when I was reading a research article: THE ARROW. I can’t thank you enough for helping to make sense of it all” (Pearce, 2012, personal communication ).

Examples For those students who are experienced clinicians, making the linkage between clinical evidence and research, as well as evidence-based practice, is a critical component for their understanding that evidence is im28

P. F. Pearce et al.

portant and useful for their practices and their scholarly endeavors. If teaching and learning are to be effective and students are to move to the next level of understanding and application of what they have learned, it is the responsibility of faculty teaching research methods to provide appropriate and logical scaffolding examples that link the learner experience with the new material being introduced (Goldstein, 1999; Vygotsky, 1978; Vygotsky & Kozulin, 1986). But, that next level of understanding can only occur if the students understand basic tenets of data or evidence. Students are cautioned to use clinical analogies carefully, because clinically relevant examples and clinical data are not gathered necessarily for research purposes. The following examples are designed to help students generalize their clinical knowledge to the larger domain of research and evidence. Examples such as the procedures for collecting and evaluating a laboratory result (e.g., CBC, lipids, hemoglobin A1C), or evaluative radiology tests (e.g., chest x-ray, ultrasound), and safety in medication administration are easily adaptable examples relevant to all nurses regardless of educational program or specialty. These procedures involve a required level of overall rigor for related activities, such as patient-preparation procedures, fasting parameters, and timing of blood draws, that if not monitored will culminate in spurious results. Age of the patient can determine how a particular procedure is completed, and how the result is interpreted, given varying context and patient characteristics (e.g., age, gender). Inter-rater and intrarater reliability can be exemplified with explanations and probing questions, such as “if there is no specific, detailed protocol for completing a blood draw, then how can there be any assurance that the blood draw result is valid or reliable?” When describing instrumentrelated reliability and validity, a corollary can be made to laboratory results and how norms are developed, how laboratory equipment is calibrated, and how results are reported and interpreted. An additional example that is universally understood by all nurses, and quite useful in discussing reliability and validity of instruments, is the procedure of blood pressure (BP) measurement. Discussion of the equipment addresses aneroid, mercury-based, and digital sphygmomanometer measurement techniques, as well as arterial lines and more invasive procedures. For standard settings, it is helpful to include the notion of a gold standard that by which all other measurements or measurement tools are compared. Mathematical models (e.g., regression models) serve as a basis for programming digital techniques. However, those techniques are heavily reliant on years of data and clinical experience generated with arterial lines, mercury-based, and aneroid-based BP measurement. Validity of measurement (does the [type] sphygmomanometer

Research methods for graduate students

P. F. Pearce et al.

Table 1 Helpful hints for using the Arrow Framework for educators and students Educators

Students

• Build the Arrow Framework in sequential order for PowerPoint presentations, handouts, and discussion • Emphasize the points on the Arrow Framework you are highlighting by using a darker or colored font to mark specific point • Emphasize vocabulary and terms in relationship to the language of research, noting the position of the terms in application to designs on the Arrow Framework • Explain the horizontal and vertical placement of the designs on the Arrow Framework in relationship to level of data and level of evidence. Expand to add further designs as needed • Highlight labels for the designs, so that students become familiar with traditional labeling • Discuss design (e.g., experimental), providing the characteristics or assumptions that match the design • Highlight information in context to the design lower or higher in complexity to demonstrate the “stepping” of requirements to meet design assumptions • Use a published article for presentation and discussion of the specific design, identifying the research question and/or hypothesis, and then the specific design • Discuss the specific characteristics of the design and ask students to identify where on the arrow the design would be located • Encourage students to apply the Arrow Framework in their readings of research • Provide examples that they can check and exercises that specifically use published literature

• Use the Arrow Framework while you are reading about designs—think about where the design fits best on the Arrow • Focus on the design that is being presented on the Arrow Framework. Note the level of data and level of evidence for the specific design in relation to other designs • Focus on understanding the vocabulary of research, noting the position of terms in relation to the designs on the Arrow Framework. This will help to learn the language in application to the designs • Consider the horizontal and vertical axis of the Arrow Framework that move from least complex to more complex designs, which parallel level of data and level of evidence • Learn the design labels—these are labels that are reflected in published literature, and each includes particular activities, level of data, and procedures overall • Each design has specific characteristics that are hallmarks of the specific design. These characteristics will come from your readings and can be viewed against the Arrow Framework • Think of each design in relationship to a design lower and higher to it in complexity on the Arrow Framework, comparing the differences in characteristics. You will find the characteristics build as you move through the designs • Think about the Arrow Framework while you are reading a published research report • Identify the research question and/or hypothesis, and identify the location on the Arrow Framework where the design would fall. This will help you to understand level of design complexity, level of data, and level of evidence • When reading a published research report, identify where on the continuum the research should be on the Arrow Framework. You may want to draw the Arrow in the white space of the paper and mark with a star or other symbol. This will help you gather you about where in the overall hierarchy the design is positioned, and will help you know what to expect when you make critical judgments

actually measure what is purported to measure?), as well as reliability (e.g., can we depend on this measurement tool to always, measure the same way and produce the same result, and what about the individual doing the BP measurement—is each nurse doing it the same way?) are linked as clinical examples, but used to exemplify the importance of reliable and valid measurements in research . Examples from professional experience are particularly useful in helping students reframe the notion of evidence, research, and related parameters, and are especially helpful to make these concepts more relevant. Medication safety in an acute care setting is an example used to explain the introduction of bias, or error, with universal understanding and appeal for nurses. Using detail of all steps in medication administration, from the initiation of an order through pharmacy handling, delivery, and medication administration, including the people and departments involved is a directly applicable example of the many points at which er-

ror can occur (Bell, Cretin, Marken, & Landman, 2004; Miller & Bovbjerg, 2002). Queries, such as at what point might the possibility of error be introduced, and what is random error and what is systematic error?, are more easily linked with examples drawn from their professional work experiences.

Conclusion It is critical for students to understand the scientific underpinnings of nursing. Science is based necessarily on the use of evidence. Use of the Arrow Framework for teaching and learning research design has been demonstrated to be effective and practical as a model for teaching research methods to MSN, DNP, and PhD students by four faculty across five universities (see Table 1). The Arrow Framework provides a conceptual and practical pedagogical approach to providing students a model for understanding the varying aspects of research methods and

29

Research methods for graduate students

helps students understand the language of research, hierarchy of evidence, and the application of the complex information more easily than rote memorization. Use of the Arrow Framework improves the student’s ability to learn how to critique and synthesize research evidence and translate the research concepts to their courses, scholarly projects, and practice. To that end, the use of the Arrow Framework provides a powerful scaffolding experience, and empowers students in learning difficult and complex research content, making sense of the evidence, and seeing relevant application to practice and their scholarly endeavors. Further, the Arrow Framework provides educators a visual display that assists in the organization of research content in a way that makes sense to students and enhances learning and application of research concepts. Moreover, the Arrow Framework provides a practical solution for teaching complex research content that helps students make the critical linkages between concepts and understanding of evidence in a straightforward manner. In this way, student learning about the scientific underpinnings of nursing is enhanced and a better understanding of the complexity of research and the use of evidence is achieved.

References American Association of Colleges of Nursing (2001). Indicators of quality in research-focused doctoral programs in nursing (position statement). Retrieved from http://www.aacn.nche.edu/publications/positons/qualityindicators. htm American Association of Colleges of Nursing (AACN) (2006). Essentials of master’s education in nursing. Retrieved from http://www.aacn.nche.edu /education-resources/MastersEssentials11.pdf American Association of Colleges of Nursing (AACN) (2011). Essentials of doctoral education for advanced nursing practice. Retrieved from http://www. aacn.nche.edu/publications/position/DNPEssentials.pdf Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives (complete ed.). New York, NY: Longman. Bell, D. S., Cretin, S., Marken, R. S., & Landman, A. B. (2004). A conceptual framework for evaluating outpatient electronic prescribing systems based on their functional capabilities. Journal of the American Medical Informatics Association, 11(1), 60–70. doi:10.1197/jamia.M1374M1374 [pii] Benner, P. E. (2010). Educating nurses: A call for radical transformation (1st ed.). San Francisco, CA: Jossey-Bass. Bloom, B. S. (1974). Taxonomy of educational objectives: The classification of educational goals. New York, NY: McKay. Chang, Y., Voils, C. I., Sandelowski, M., Hasselblad, V., & Crandell, J. L. (2009). Transforming verbal counts in reports of qualitative descriptive studies into numbers. Western Journal of Nursing Research, 31(7), 837–852. doi:0193945909334434 [pii]10.1177/0193945909334434 Creswell, J. W. (2012). Six mixed methods designs for health and social science research. Paper presented at the UAB Lister Hill Center for Health Policy: Research Methods Workshop, Birmingham, AL. Creswell, J. W. (2013). Research design: Qualitative, quantitative, and mixed methods approaches (4th ed.). Thousand Oaks, CA: Sage Publications. Creswell, J. W., & Plano-Clark, V. L. (2006). Designing and conducting mixed methods research. Thousand Oaks, CA: Sage Publications.

30

P. F. Pearce et al.

Creswell, J. W., & Plano Clark, V. L. (2011). Designing and conducting mixed methods research (2nd ed.). Los Angeles, CA: Sage Publications. Denzin, N. K., & Lincoln, Y. S. (2011). The Sage handbook of qualitative research (4th ed.). Thousand Oaks, CA: Sage Publications. DiCenso, A., Guyatt, G., & Ciliska, D. (2005). Evidence-based nursing: A guide to clinical practice. Toronto, Ontario, Canada: Elsevier. Dossey, B. M. (2010). Florence Nightingale: Mystic, visionary, healer (commemorative ed.). Philadelphia, PA: F. A. Davis. Fineout-Overholt, E., Melnyk, B. M., & Schultz, A. (2005). Transforming health care from the inside out: Advancing evidence-based practice in the 21st century. Journal of Professional Nursing, 21(6), 335–344. doi:S8755–7223(05)00145–6 [pii]10.1016/j.profnurs.2005.10.005 Forehand, M. (2005, 2010). Bloom’s taxonomy: Original and revised. In M. Orey (Ed.), Emerging perspectives on learning, teaching, and technology. Retrieved from http://projects.coe.uga.edu/epltt/ Funk, S. G., Tornquist, E. M., & Champagne, M. T. (1995). Barriers and facilitators of research utilization: An integrative review. Nursing Clinics of North America, 30(3), 395–407. Goldstein, L. S. (1999). The relational zone: The role of caring relationships in the co-construction of mind. American Educational Research Journal, 36(3), 647–673. Grove, S. K., Burns, N., & Gray, J. (2013). The practice of nursing research: Appraisal, synthesis, and generation of evidence (7th ed.). St. Louis, MO: Elsevier/Saunders. Hall, H. R., & Roussel, L. A. (2014). Evidence-based practice: An integrative approach to research, administration, and practice. Burlington, MA: Jones & Bartlett Learning. Harris, R. L. (1996). Information graphics: A comprehensive illustrated reference—Visual tools for analyzing, managing, and communicating. Atlanta, GA: Management Graphics. Institute of Medicine (U.S.). (2010). The future of nursing: Leading change, advancing health. Retrieved from http://www.thefutureofnursing.org Jones, S. C., Crookes, P. A., & Johnson, K. M. (2011). Teaching critical appraisal skills for nursing research. Nurse Education in Practice, 11, 327–332. doi:10.1016/j.nepr.2011.03.002 Kelly, K. P., Turner, A., Gabel Speroni, K., McLaughlin, M. K., & Guzzetta, C. E. (2013). National survey of hospital nursing research, part 2: Facilitators and hindrances. Journal of Nursing Administration, 43(1), 18–23. doi:10.1097/NNA.0b013e3182786029 Knowles, M. S., Holton, E. F., & Swanson, R. A. (2011). The adult learner: The definitive classic in adult education and human resource development (7th ed.). Boston, MA: Elsevier. McLaughlin, M. K., Gabel Speroni, K., Kelly, K. P., Guzzetta, C. E., & Desale, S. (2013). National survey of hospital nursing research, part 1: Research requirements and outcomes. Journal of Nursing Administration, 43(1), 10–17. doi:10.1097/NNA.0b013e3182786045 Melnyk, B. M., & Fineout-Overholt, E. (2011). Evidence-based practice in nursing & healthcare: A guide to best practice (2nd ed.). Philadelphia, PA: Wolters Kluwer/Lippincott Williams & Wilkins. Melnyk, B. M., Fineout-Overholt, E., Fischbeck Feinstein, N., Li, H., Small, L., Wilcox, L., & Kraus, R. (2004). Nurses’ perceived knowledge, beliefs, skills, and needs regarding evidence-based practice: Implications for accelerating the paradigm shift. Worldviews: Evidence Based Nursing, 1(3), 185–193. doi:WVN04024 [pii]10.1111/j.1524–475X.2004.04024.x Merriam, S. (2007). Learning in adulthood: A comprehensive guide (3rd ed.). San Francisco, CA: Jossey-Bass. Miller, R. H., & Bovbjerg, R. R. (2002). Efforts to improve patient safety in large, capitated medical groups: Description and conceptual model. Journal of Health Politics, Policy and Law, 27(3), 401–440. Morse, J. M. (1999). Myth #93: Reliability and validity are not relevant to qualitative inquiry. Qualitative Health Research 9, 717–718. doi: 10.1177/104973299129122171 Morse, J. M., Barrett, M., Mayan, M., Olson, K., & Spiers, J. (2002). Verification strategies for establishing reliability and validity in qualitative research. International Journal of Qualitative Methods, 1(2), 1–19.

P. F. Pearce et al.

Morse, J. M., & Field, P.-A. (1995). Qualitative research methods for health professionals (2nd ed.). Thousand Oaks, CA: Sage Publications. Morse, J. M., & Niehaus, L. (2009). Mixed method design: Principles and procedures. Walnut Creek, CA: Left Coast Press. Polit, D. F., & Beck, C. T. (2012). Essentials of nursing research: Generating and appraising evidence for nursing practice (9th ed.). Philadelphia, PA: Wolters Kluwer/Lippincott/Williams & Wilkins Health. Sandelowski, M. (1986). The problem of rigor in qualitative research. ANS. Advances in Nursing Science, 8(3), 27–37. Sandelowski, M. (1993). Rigor or rigor mortis: The problem of rigor in qualitative research revisited. ANS. Advances in Nursing Science, 16(2), 1–8. Sandelowski, M. (2000a). Combining qualitative and quantitative sampling, data collection, and analysis techniques in mixed-method studies. Research in Nursing and Health, 23(3), 246–255. Sandelowski, M. (2000b). Whatever happened to qualitative description? Research in Nursing and Health, 23(4), 334–340. Sandelowski, M. (2001). Real qualitative researchers do not count: The use of numbers in qualitative research. Research in Nursing and Health, 24(3), 230–240. Sandelowski, M. (2010). What’s in a name? Qualitative description revisited. Research in Nursing and Health, 33(1), 77–84. doi:10.1002/nur.20362 Schmidt, N. A., & Brown, J. M. (2011). EBP for nurses: Appraisal and application research (2nd ed.). Sudbury, MA: Jones & Bartlett Publishing.

Research methods for graduate students

Spoelstra, S. L., & Robbins, L. B. (2010). A qualitative study of role transition from RN to APN. International Journal of Nursing Education Scholarship, 7(1), 1–15. doi:http://dx.doi.org/10.2202/1548–923X.2020 Squires, J. E., Estabrooks, C. A., Gustavsson, P., & Wallin, L. (2011). Individual determinants of research utilization by nurses: A systematic review update. Inplementation Science, 6, 1–14. doi:10.1186/1748–5908–6–1 Tabachnick, B. G., & Fidell, L. S. (1996). Using multivariate statistics (3rd ed.). New York, NY: Harper Collins. Tashakkori, A., & Teddlie, C. (2003a). Handbook of mixed methods in social and behavioral research. Thousand Oaks, CA: Sage Publications. Tashakkori, A., & Teddlie, C. (2003b). Issues and dilemmas in teaching research methods courses in social and behavioural sciences: US perspective. International Journal of Social Research Methodology: Theory & Practice, 6(1), 61–77. Tufte, E. R. (1990). Envisioning information. Cheshire, CT: Graphics Press. Tufte, E. R. (2001). The visual display of quantitative information (2nd ed.). Cheshire, CT: Graphics Press. Vygotsky, L. S. (1978). Mind in Society: The development of higher psychological processes. Cambridge, MA: Harvard University Press. Vygotsky, L. S., & Kozulin, A. (1986). Thought and language. Cambridge, MA: MIT Press. Whittemore, R., Chase, S. K., & Mandle, C. L. (2001). Validity in qualitative research. Qualitative Health Research, 11(4), 522–537.

31