JNPD
Journal for Nurses in Professional Development & Volume 30, Number 1, 11Y15 & Copyright B 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins
Revisiting the Delphi Technique Implications for Nursing Professional Development Marilyn Asselin, PhD, RN, BC
ƒ
Mary Harper, PhD, RN-BC
The Delphi technique is a multistage survey involving rounds of structured questions with a panel of identified experts. The approach is well suited for a range of applications in nursing professional development practice that involve priority setting and consensus building. This article provides an overview of the technique, steps in using the approach, and implications and suggestions for its use in nursing professional development practice.
content formation (Goligher, Ferguson, & Kenny, 2012; Ito, Ota, & Matsuda, 2011). These examples indicate that the Delphi may be useful in diverse circumstances. One arena in which the Delphi technique may be useful is the nursing professional development (NPD) practice. This article provides an overview and steps in using the approach, methodological considerations, and implications and suggestions for its use in NPD practice.
THE DELPHI TECHNIQUE INTRODUCTION The Delphi technique, widely used and accepted across a range of disciplines, has gained increased recognition in health, nursing, and medical research (Keeney, Hasson, & McKenna, 2011). The technique is a form of survey design research consisting of a multistage survey involving two or more rounds of structured questions with a panel of identified experts. The goal is to reach consensus on an issue, question, problem, or idea. Traditionally, this technique has been used to identify priorities for conducting or funding research (Back-Pettersson, Hermansson, Sernert, & Bjorkelund, 2008; Grundy & Ghazi, 2009). Although the Delphi technique has been used primarily in research (Blackwood, Albarran, & Latour, 2010; Harper, Asselin, Kurtz, MacArthur, & Perron, 2012; Keeney et al., 2011), examples of alternate uses of the Delphi are found in the literature. For example, Calsyn et al. (2012) used the Delphi process to revise an HIV prevention program to make it more relevant to ethnically diverse groups. In nursing, the Delphi has been used for both instrument and conceptual model validation (Chang, Gardner, Duffield, & Ramis, 2010; Culley, 2011), development of a wound care documentation framework (Kinunen, Saranto, Ensio, Iivanainen, & Dykes, 2012), and educational objective and Marilyn Asselin, PhD, RN, BC, is Associate Professor, College of Nursing, University of Massachusetts Dartmouth. Mary Harper, PhD, RN-BC, is Director of Nursing Professional Development, Association for Nursing Professional Development, Chicago, Illinois. The authors have disclosed that they have no significant relationship with, or financial interest in, any commercial companies pertaining to this article. ADDRESS FOR CORRESPONDENCE: Marilyn Asselin, PhD, RN, BC, College of Nursing, University of Massachusetts, 285 Old Westport Road, North Dartmouth, MA 02747 (email: [email protected]). DOI: 10.1097/01.NND.0000434028.30432.34 Journal for Nurses in Professional Development
The Delphi technique was originally developed in 1950s as a way to predict or forecast the impact of various military technologies, such as the atomic bomb, in the event of war (Kenney et al., 2011). The technique continued to be used as a forecasting method, first in science and technology and later in public policy and economics (Vernon, 2009). The technique was first used in nursing in the 1970s (Kenney et al., 2011; Vernon, 2009) and then later in radiography, dentistry, dietetics, and physiotherapy (Vernon, 2009). Current use of the Delphi technique is focused primarily on gaining consensus or setting priorities (Kenney et al., 2011) and may be used to address research questions that are often not associated with established scientific procedures (Vernon, 2009). The Delphi technique consists of a multistage survey, including structured questions, with a group of identified ‘‘experts’’. The premise underlying the technique is that group opinion based on a structured format yields more accurate and reliable data than individual opinion or opinion from unstructured methods (Keeney et al., 2011). Thus, it is a group decision-making process. There are several key characteristics of the Delphi technique. First, the Delphi consists of multiple rounds, usually two to three. Each stage or round of the survey builds on the results from the previous rounds until consensus is reached. Second, the participants include a group of identified experts or specialists who have met certain criteria related to knowledge, expertise, and skill. They are invited to participate and share their ‘‘expert opinion’’ on the topic by answering the questions in the survey. Third, the communication with the experts is structured in some way, consisting, for example, of a set of predetermined questions, potential answer options, or open-ended questions. Data collected may be qualitative or quantitative in nature or a combination of both. www.jnpdonline.com
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
11
Fourth, participants receive regular feedback at the outset of each round. This feedback consists of individual as well as summarized group feedback. Keeney et al. (2011) notes that this later process promotes motivation, encourages participants to become more involved, and encourages consensus. As in a quantitative research study, the researcher has an objective role in structuring and administering the study. However, with a Delphi, the researcher is also a facilitator, ensuring that responses are returned to experts and facilitating the process of expert communication. The researcher also leads the data analysis, if working with a Delphi team, and details an objective process for analyzing the data and presenting it in a meaningful way. Vernon (2009) notes that there are multiple advantages in using the Delphi technique. The technique is a relatively simple design and can be cost effective to administer. Because of its relatively simple design, it is not overly time consuming for the participant. The technique also offers flexibility of use for a variety of purposes. Another advantage of the Delphi is that participants remain anonymous to one another. Anonymity among panelists allows for changing positions based on group feedback without the need to defend such change, avoiding undue influence of reputable experts and inability of one member to dominate the expert panel (de Meyrick, 2003). Finally, important points or issues can be quickly identified, and conclusions can be rapidly reached. Although no acceptable response rate for Delphi studies has been identified, Vernon (2009) notes that response rates may vary widely across studies with rates ranging as low as 8%Y100%. Strategies such as informed consent, limited time between rounds, limited number of rounds, continued communication with panel members, and short surveys may improve response rates.
CONDUCTING A DELPHI STUDY The decision to conduct a Delphi may be made whenever consensus is needed or as an aid to priority setting (Keeney et al., 2011; Vernon, 2009), particularly when strong, diverse opinions exist among stakeholders. It may also be constructive to use the Delphi to ensure that all stakeholders have equal input into the decision-making process without undue influence from others. The Delphi may be relatively straightforward and inexpensive to conduct; however, before embarking on the process, consideration must be given to the resources needed, especially time for data analysis and feedback to participants. Although the Delphi process is straightforward, it consists of several sequential steps. Once a decision has been made to conduct a Delphi with a clearly stated question developed, the participants must be identified (Vernon, 2009). Typically, these participants are considered ‘‘experts,’’ but depending on the purpose of the Delphi, an 12
www.jnpdonline.com
expert may be anyone with intimate experience or knowledge of the topic (Keeney et al., 2011). For example, a Delphi to determine the educational needs of hospital nurses may include staff nurses who indicate what educational offerings they would prefer and would be likely to participate in. On the other hand, nursing leadership may be selected to identify topics related to the strategic plan, risk management issues, and quality initiatives. Clear delineation of the characteristics of the participants is essential to promote the rigor and accuracy of the results. After the identification of participants, the size of the participant panel must be determined. Whereas some Delphi experts maintain that no ideal sample size has been identified (Keeney et al., 2011; Vernon, 2009), others posit that more than 30 participants are unnecessary (De Villiers, De Villiers, & Kent, 2005). Recent research using the Delphi technique indicates expert panel sizes ranging from 12 to 110 (Blackwood et al., 2010; Chang et al., 2010; Harper et al., 2012; Ito et al., 2011; Lock, 2011). Practical considerations for the size of the participant panel for a Delphi in NPD practice include the number of available participants along with resources for data analysis and feedback to the participant panel. For example, it may be realistic to include all nurses on a particular nursing unit in a Delphi used to determine priority unit-specific educational needs, whereas including all employed nurses for a facility-wide educational needs assessment may be too time and labor intensive. Once a participant panel has been identified, the next step in conducting a Delphi study is to determine the type of Delphi to be used. The Delphi consists of rounds of surveys in which participants rate items based on given criteria until consensus is achieved (Keeney et al., 2011). Two common types of Delphi are the classical and the modified. When either of these methods is conducted electronically, it is referred to as an e-Delphi. In a classic Delphi, the first step of the process is to obtain feedback from the expert panel in a qualitative format. An open-ended question is posed concerning the topic of the Delphi, and the expert panel generates the ideas that inform subsequent rounds. For example, a nursing leadership participant group may be asked, ‘‘What educational topics are needed for staff nurses to promote achievement of the strategic plan, avoid risk management issues, and to reach quality benchmarks?’’ The advantage of the classical method is obtaining expert opinion and input (Keeney et al., 2011). The disadvantage is that the number of ideas received may be enormous and difficult to manage. An example of a classic Delphi may be found in Goligher et al.’s (2012) use to identify key objectives for education for healthcare providers on care of mechanically ventilated patients. In this study, the investigators allowed 14 participants to generate the initial list of educational objectives in response to an open-ended question. Two hundred objectives were generated, 109 January/February 2014
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
of which were duplicates. The remaining 91 items were used as items for the ensuing Delphi rounds. In a modified Delphi, the initial ideas are generated by the investigators through literature or document review, interviews, or focus groups (Keeney et al., 2011). Although this approach may effectively control the number of ideas for subsequent rounds, it may limit options and create bias. For an educational needs assessment, investigators may review the organization’s strategic plan, risk management, and quality data and perhaps interview key nursing leaders to generate the initial list of topics for the survey. In an example of a modified Delphi, Harper et al. (2012) conducted a Delphi on NPD research priorities. Potential research priorities were generated by the research team through a review of pertinent research and government reports. The resulting topics were presented to the expert panel for rating in the initial round of the Delphi. Whether conducting a classical or modified Delphi survey, use of the e-Delphi method has many advantages. Electronic survey tools have simplified data collection and timeliness of Delphi surveys. In addition, Web-based surveys are easily accessible to the participants through URL links. Furthermore, results are immediately accessible to the investigator with basic data analysis such as frequency and mean already calculated. Finally, individual results are available, thereby facilitating feedback for subsequent rounds. As stated, the initial rounds of the classical and modified Delphi differ (Keeney et al., 2011). In the classical Delphi, an open-ended question format is used to generate ideas. These data are evaluated with like items being combined and categorized to form the list of items to be rated in subsequent rounds. With the exception of this initial round, the process is the same in both the classical and modified Delphi. Before initiating the quantitative survey process in which participants rate each item on a well-defined Likert-type scale, the definition of consensus must be established. The level of consensus depends on the purpose of the Delphi and typically ranges from 51% to 100%, with 70% agreement being common (Keeney et al., 2011; Vernon, 2009). Investigators may look at measures of central tendency (mean, median, and mode) as well as at the percentage of responses that fall within a predetermined range of responses. For example, Harper et al. (2012) defined consensus using a 5-point Likert-type scale as ‘‘65% of participants indicating scores of 3, 4, or 5 for an item or items that at least 85% of the participants score as ‘4’ or ‘5’ and no participants score as ‘1’ or ‘2’’’ (p. 139). Once participants complete the initial quantitative survey round in which they rate items on a Likert-type scale, data are analyzed for central tendency and dispersion of scores. Items that have reached consensus may be reJournal for Nurses in Professional Development
moved from subsequent rounds of the survey to shorten the survey and time required for completion (Keeney et al., 2011). In addition, investigators may opt to remove items that have been rejected by consensus of all participants (Vernon, 2009). Before subsequent rounds, participants receive feedback from the previous round indicating their personal responses along with measures of central tendency of the overall group. Standard deviation may be included to reflect the dispersion of scores. This feedback allows participants to reflect on their responses and change them if desired. Delphi rounds may be discontinued after a predetermined number of rounds, typically three, or when consensus has been reached to either accept or reject ideas.
ETHICAL CONSIDERATIONS, METHODOLOGICAL RIGOR, AND LIMITATIONS As in any research study, attention to ethical considerations is critical when designing and conducting the Delphi study. As stated, approval to conduct the study should be sought from the agency’s institutional research review committee when the Delphi is research related. The procedure for identifying the expert panel should be clearly documented, and expert members should receive a written explanation of the purpose, procedures, benefits, time line, and time involvement for participation in the study as well as assurance that the responses of each individual will be kept confidential. Delphi studies offer quasi-anonymity rather than true anonymity. Because the expert group is a small, defined group, it is possible that the participants may have knowledge of the other members. However, the researcher can ensure that responses of any one panel member are not disclosed to other panel members, thereby preventing undue influence of more highly recognized experts. Codes are assigned to each expert, and the data for each round are thus linked to that code. There is ongoing debate about establishing methodological rigor for the Delphi. Kenney et al. (2011) note that the basis of this debate centers on whether the Delphi aligns with a positivist quantitative paradigm in which validity and reliability are used or a qualitative paradigm where criteria associated with credibility or trustworthiness of data, such as audit trails, are used to establish rigor. Irrespective of this debate, methodological strategies should be chosen to limit threats to the integrity of the technique. Careful thought should be given to the composition of the panel. Criteria for the expert panel members should be clearly delineated noting knowledge, skill, and experience levels as well as practice setting. Allowing for some heterogeneity of panel members, about practice settings for example, may yield a broad range of opinions and be more representative of the opinions of the population of www.jnpdonline.com
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
13
interest. The level of consensus should be established before the first round as well as procedures for communicating with the experts. It is critical that the aim of the Delphi be clearly communicated to everyone involved with the Delphi and that it is included as a part of each round. In addition, the researcher should document the process for analyzing the data from each round. This includes a step-by-step description of how responses from experts will be collapsed into categories or themes. Typically, a form of content analysis is used to analyze the qualitative responses. Actions and decisions that occur during the study should be documented and serve as an audit trail of thoughts and decisions. Ideally, involving another person or persons in the review of data and corroboration of findings enhances the credibility of the study. Credibility is also enhanced through the ongoing rounds and feedback from the panel of experts. Although there are many advantages in using the Delphi technique, it is important to be aware of potential limitations in using this approach. Reaching consensus does not mean that the correct answer or priority has been reached; only that consensus has been reached among the panel of chosen experts (Keeney et al., 2011). Moreover, the findings may not reflect the opinions and views of the larger population of experts or representative population. In the quest to reach consensus, variations in points of view, which may be important to consider, may be overlooked. Depth of inquiry in a topic may not be reached, but the Delphi technique may serve to identify a topic to be more fully explored at a later time. Furthermore, there is a potential for researcher bias in the interpretation of the data. This can be avoided by taking steps, as described above. Kenney et al. also note that the effect of group dynamics may influence expert responses in subsequent Delphi rounds. This influence may take the form of pressure to conform similar to the notion of ‘‘group think,’’ which involves group members forming quick responses or opinions that match those of others in the group. This may be more likely to occur when the members of the expert panel have knowledge of each other. In addition, because this technique requires participation in several rounds, attrition of experts may occur because of round fatigue (Vernon, 2009) or time delays between rounds (Kenney et al., 2011). Finally, as in any study, events occurring outside of the study may influence the experts’ responses and, if known to the researcher, noted in the study limitations.
USE OF THE DELPHI IN NPD PRACTICE As stated, the Delphi technique is useful for priority setting and developing consensus (Kenney et al., 2011; Vernon, 2009). This technique allows for input from a wide variety of stakeholders without extra time-consuming meetings and may promote timeliness of decision making in a variety 14
www.jnpdonline.com
of areas. Within NPD practice, the Delphi may be employed for activities such as developing research priorities, conducting educational needs assessments, identifying competencies for evaluation, developing educational objectives and content for courses, and delineating role responsibilities. Although these examples are not exhaustive, they may stimulate thinking about innovative ways to use the Delphi in other areas of NPD practice. One use of the Delphi in NPD practice is setting research priorities. Although often used on a national or international level to determine research priorities for a specialty (Blackwood et al., 2010; Harper et al., 2012), the Delphi could prove useful within a single organization. For example, research priority setting may occur on a unit level and engage all staff from the unit. Conversely, the priority setting could occur at an organizational level, engaging nursing leadership to ensure that research aligns with the organization’s mission, vision, and strategic goals. Perhaps, nursing leadership could be used to generate potential research ideas and then staff nurses selected to rate the importance of each, or vice versa. Engagement of staff in the process may promote buy-in and participation when the research is conducted. Another use of the Delphi in NPD practice is for educational needs assessment, as suggested in the previous section. Nursing leadership and nursing staff involvement are important to ensure identification of realistic needs that align with organizational initiatives. Using the Delphi for educational needs assessment not only identifies educational topics that are needed but assists with prioritizing these needs. Selection of competencies for assessment/evaluation is another use of the Delphi in NPD practice. For example, Lock (2011) used a Delphi method to identify core competencies required for licensure in Indonesia. Similarly, NPD departments could use a Delphi method to determine core competencies for assessment during orientation. In addition, a Delphi method may be helpful in narrowing a vast array of competencies for ongoing assessment as required by accrediting bodies. The Delphi method is also useful to determine objectives and content for educational activities (Goligher et al., 2012; Ito et al., 2011). As NPD specialists are challenged to manage competing priorities, identification of ‘‘need to know’’ content may ensure optimizing time spent in program development as well as staff time in participation in programs. Content of orientation, inservices, and continuing education programs could all be guided by Delphi surveys. Finally, the Delphi technique may be helpful in determining essential components of nursing roles both within NPD and in other practice areas. For example, Chang et al. (2010) used a Delphi survey to validate an advanced practice nursing tool. This tool identified the elements of practice January/February 2014
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
for the advanced practice role. Likewise, as NPDVand otherVdepartments are confronted with increasing demands, sometimes with shrinking resources, identification of key role responsibilities may be warranted.
SUMMARY The Delphi technique, commonly used in a broad range of disciplines, has potential for use in NPD as specialists face challenges in complex healthcare systems, seek to define and clarify practice roles, and expand the evidence base for practice. As a multistaged survey, the technique may be used to set priorities or gain consensus in a variety of situations such as establishing research priorities, conducting needs assessments, selecting competencies and educational content, or identifying practice role responsibilities. It may be of particular use for initially engaging staff in the research process. As in any research, particular attention to ethical considerations and methodological rigor is essential.
References Back-Pettersson, S., Hermansson, E., Sernert, N., & Bjorkelund, C. (2008). Research priorities in nursingVA Delphi study among Swedish nurses. Journal of Clinical Nursing, 17(16), 2221Y2231. Blackwood, R., Albarran, J., & Latour, J. (2010). Research priorities of adult intensive care nurses in 20 European countries: A Delphi study. Journal of Advanced Nursing, 67(3), 550Y562. Calsyn, D., Burlew, A., Hatch-Maillette, M., Wilson, J., Beadnell, B., & Wright, L. (2012). Real Men Are SafeVCulturally adapted: Utilizing the Delphi process to revise Real Men Are Safe for an ethnically diverse group of men in substance abuse treatment. AIDS Education and Prevention, 24(2), 117Y131.
Journal for Nurses in Professional Development
Chang, A., Gardner, G., Duffield, C., & Ramis, M. (2010). A Delphi study to validate an advanced practice nursing tool. Journal of Advanced Nursing, 66(10), 2320Y2330. Culley, J. (2011). Use of a computer mediated Delphi process to validate a mass casualty conceptual model. Computers, Informatics, Nursing, 29(5), 272Y279. de Meyrick, J. (2003). The Delphi method and health research, Health Education, 103(1), 7Y16. De Villiers, M. R., De Villiers, P. J., & Kent, A. P. (2005). The Delphi technique in health sciences education research. Medical Teacher, 27(7), 639Y643. Goligher, E., Ferguson, N., & Kenny, L. (2012). Core competency in mechanical ventilation: Development of educational objectives using the Delphi technique. Critical Care Medicine, 40(10), 2828Y2832. Grundy, M., & Ghazi, R. (2009). Research priorities in haematooncology nursing: Results of a literature review and a Delphi study. European Journal of Oncology Nursing, 13(4), 235Y249. Harper, M., Asselin, M., Kurtz, A., MacArthur, S., & Perron, S. (2012). Research priorities for nursing professional development: A modified e-Delphi study. Journal for Nurses in Staff Development, 28(3), 137Y142. Ito, C., Ota, K., & Matsuda, M. (2011). Educational content in nurse education in Japan: A Delphi study. Nursing Ethics, 18(3), 441Y454. Keeney, S., Hasson, F., & McKenna, H. (2011). The Delphi technique in nursing and health research. West Sussex, UK: Wiley-Blackwell. Kinunen, U., Saranto, K., Ensio, A., Iivanainen, A., & Dykes, P. (2012). Developing the standardized wound care documentation model: A Delphi study to improve the quality of patient care documentation. Journal of Wound, Ostomy & Continence Nursing, 39(4), 397Y407. Lock, L. (2011). Selecting examinable nursing core competencies: A Delphi project. International Nursing Review, 58, 347Y353. Vernon, W. (2009). The Delphi technique: A review. International Journal of Therapy and Rehabilitation, 16(2), 69Y76.
www.jnpdonline.com
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
15
Journal for Nurses in Professional Development & Volume 30, Number 1, 11Y15 & Copyright B 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins
Revisiting the Delphi Technique Implications for Nursing Professional Development Marilyn Asselin, PhD, RN, BC
ƒ
Mary Harper, PhD, RN-BC
The Delphi technique is a multistage survey involving rounds of structured questions with a panel of identified experts. The approach is well suited for a range of applications in nursing professional development practice that involve priority setting and consensus building. This article provides an overview of the technique, steps in using the approach, and implications and suggestions for its use in nursing professional development practice.
content formation (Goligher, Ferguson, & Kenny, 2012; Ito, Ota, & Matsuda, 2011). These examples indicate that the Delphi may be useful in diverse circumstances. One arena in which the Delphi technique may be useful is the nursing professional development (NPD) practice. This article provides an overview and steps in using the approach, methodological considerations, and implications and suggestions for its use in NPD practice.
THE DELPHI TECHNIQUE INTRODUCTION The Delphi technique, widely used and accepted across a range of disciplines, has gained increased recognition in health, nursing, and medical research (Keeney, Hasson, & McKenna, 2011). The technique is a form of survey design research consisting of a multistage survey involving two or more rounds of structured questions with a panel of identified experts. The goal is to reach consensus on an issue, question, problem, or idea. Traditionally, this technique has been used to identify priorities for conducting or funding research (Back-Pettersson, Hermansson, Sernert, & Bjorkelund, 2008; Grundy & Ghazi, 2009). Although the Delphi technique has been used primarily in research (Blackwood, Albarran, & Latour, 2010; Harper, Asselin, Kurtz, MacArthur, & Perron, 2012; Keeney et al., 2011), examples of alternate uses of the Delphi are found in the literature. For example, Calsyn et al. (2012) used the Delphi process to revise an HIV prevention program to make it more relevant to ethnically diverse groups. In nursing, the Delphi has been used for both instrument and conceptual model validation (Chang, Gardner, Duffield, & Ramis, 2010; Culley, 2011), development of a wound care documentation framework (Kinunen, Saranto, Ensio, Iivanainen, & Dykes, 2012), and educational objective and Marilyn Asselin, PhD, RN, BC, is Associate Professor, College of Nursing, University of Massachusetts Dartmouth. Mary Harper, PhD, RN-BC, is Director of Nursing Professional Development, Association for Nursing Professional Development, Chicago, Illinois. The authors have disclosed that they have no significant relationship with, or financial interest in, any commercial companies pertaining to this article. ADDRESS FOR CORRESPONDENCE: Marilyn Asselin, PhD, RN, BC, College of Nursing, University of Massachusetts, 285 Old Westport Road, North Dartmouth, MA 02747 (email: [email protected]). DOI: 10.1097/01.NND.0000434028.30432.34 Journal for Nurses in Professional Development
The Delphi technique was originally developed in 1950s as a way to predict or forecast the impact of various military technologies, such as the atomic bomb, in the event of war (Kenney et al., 2011). The technique continued to be used as a forecasting method, first in science and technology and later in public policy and economics (Vernon, 2009). The technique was first used in nursing in the 1970s (Kenney et al., 2011; Vernon, 2009) and then later in radiography, dentistry, dietetics, and physiotherapy (Vernon, 2009). Current use of the Delphi technique is focused primarily on gaining consensus or setting priorities (Kenney et al., 2011) and may be used to address research questions that are often not associated with established scientific procedures (Vernon, 2009). The Delphi technique consists of a multistage survey, including structured questions, with a group of identified ‘‘experts’’. The premise underlying the technique is that group opinion based on a structured format yields more accurate and reliable data than individual opinion or opinion from unstructured methods (Keeney et al., 2011). Thus, it is a group decision-making process. There are several key characteristics of the Delphi technique. First, the Delphi consists of multiple rounds, usually two to three. Each stage or round of the survey builds on the results from the previous rounds until consensus is reached. Second, the participants include a group of identified experts or specialists who have met certain criteria related to knowledge, expertise, and skill. They are invited to participate and share their ‘‘expert opinion’’ on the topic by answering the questions in the survey. Third, the communication with the experts is structured in some way, consisting, for example, of a set of predetermined questions, potential answer options, or open-ended questions. Data collected may be qualitative or quantitative in nature or a combination of both. www.jnpdonline.com
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
11
Fourth, participants receive regular feedback at the outset of each round. This feedback consists of individual as well as summarized group feedback. Keeney et al. (2011) notes that this later process promotes motivation, encourages participants to become more involved, and encourages consensus. As in a quantitative research study, the researcher has an objective role in structuring and administering the study. However, with a Delphi, the researcher is also a facilitator, ensuring that responses are returned to experts and facilitating the process of expert communication. The researcher also leads the data analysis, if working with a Delphi team, and details an objective process for analyzing the data and presenting it in a meaningful way. Vernon (2009) notes that there are multiple advantages in using the Delphi technique. The technique is a relatively simple design and can be cost effective to administer. Because of its relatively simple design, it is not overly time consuming for the participant. The technique also offers flexibility of use for a variety of purposes. Another advantage of the Delphi is that participants remain anonymous to one another. Anonymity among panelists allows for changing positions based on group feedback without the need to defend such change, avoiding undue influence of reputable experts and inability of one member to dominate the expert panel (de Meyrick, 2003). Finally, important points or issues can be quickly identified, and conclusions can be rapidly reached. Although no acceptable response rate for Delphi studies has been identified, Vernon (2009) notes that response rates may vary widely across studies with rates ranging as low as 8%Y100%. Strategies such as informed consent, limited time between rounds, limited number of rounds, continued communication with panel members, and short surveys may improve response rates.
CONDUCTING A DELPHI STUDY The decision to conduct a Delphi may be made whenever consensus is needed or as an aid to priority setting (Keeney et al., 2011; Vernon, 2009), particularly when strong, diverse opinions exist among stakeholders. It may also be constructive to use the Delphi to ensure that all stakeholders have equal input into the decision-making process without undue influence from others. The Delphi may be relatively straightforward and inexpensive to conduct; however, before embarking on the process, consideration must be given to the resources needed, especially time for data analysis and feedback to participants. Although the Delphi process is straightforward, it consists of several sequential steps. Once a decision has been made to conduct a Delphi with a clearly stated question developed, the participants must be identified (Vernon, 2009). Typically, these participants are considered ‘‘experts,’’ but depending on the purpose of the Delphi, an 12
www.jnpdonline.com
expert may be anyone with intimate experience or knowledge of the topic (Keeney et al., 2011). For example, a Delphi to determine the educational needs of hospital nurses may include staff nurses who indicate what educational offerings they would prefer and would be likely to participate in. On the other hand, nursing leadership may be selected to identify topics related to the strategic plan, risk management issues, and quality initiatives. Clear delineation of the characteristics of the participants is essential to promote the rigor and accuracy of the results. After the identification of participants, the size of the participant panel must be determined. Whereas some Delphi experts maintain that no ideal sample size has been identified (Keeney et al., 2011; Vernon, 2009), others posit that more than 30 participants are unnecessary (De Villiers, De Villiers, & Kent, 2005). Recent research using the Delphi technique indicates expert panel sizes ranging from 12 to 110 (Blackwood et al., 2010; Chang et al., 2010; Harper et al., 2012; Ito et al., 2011; Lock, 2011). Practical considerations for the size of the participant panel for a Delphi in NPD practice include the number of available participants along with resources for data analysis and feedback to the participant panel. For example, it may be realistic to include all nurses on a particular nursing unit in a Delphi used to determine priority unit-specific educational needs, whereas including all employed nurses for a facility-wide educational needs assessment may be too time and labor intensive. Once a participant panel has been identified, the next step in conducting a Delphi study is to determine the type of Delphi to be used. The Delphi consists of rounds of surveys in which participants rate items based on given criteria until consensus is achieved (Keeney et al., 2011). Two common types of Delphi are the classical and the modified. When either of these methods is conducted electronically, it is referred to as an e-Delphi. In a classic Delphi, the first step of the process is to obtain feedback from the expert panel in a qualitative format. An open-ended question is posed concerning the topic of the Delphi, and the expert panel generates the ideas that inform subsequent rounds. For example, a nursing leadership participant group may be asked, ‘‘What educational topics are needed for staff nurses to promote achievement of the strategic plan, avoid risk management issues, and to reach quality benchmarks?’’ The advantage of the classical method is obtaining expert opinion and input (Keeney et al., 2011). The disadvantage is that the number of ideas received may be enormous and difficult to manage. An example of a classic Delphi may be found in Goligher et al.’s (2012) use to identify key objectives for education for healthcare providers on care of mechanically ventilated patients. In this study, the investigators allowed 14 participants to generate the initial list of educational objectives in response to an open-ended question. Two hundred objectives were generated, 109 January/February 2014
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
of which were duplicates. The remaining 91 items were used as items for the ensuing Delphi rounds. In a modified Delphi, the initial ideas are generated by the investigators through literature or document review, interviews, or focus groups (Keeney et al., 2011). Although this approach may effectively control the number of ideas for subsequent rounds, it may limit options and create bias. For an educational needs assessment, investigators may review the organization’s strategic plan, risk management, and quality data and perhaps interview key nursing leaders to generate the initial list of topics for the survey. In an example of a modified Delphi, Harper et al. (2012) conducted a Delphi on NPD research priorities. Potential research priorities were generated by the research team through a review of pertinent research and government reports. The resulting topics were presented to the expert panel for rating in the initial round of the Delphi. Whether conducting a classical or modified Delphi survey, use of the e-Delphi method has many advantages. Electronic survey tools have simplified data collection and timeliness of Delphi surveys. In addition, Web-based surveys are easily accessible to the participants through URL links. Furthermore, results are immediately accessible to the investigator with basic data analysis such as frequency and mean already calculated. Finally, individual results are available, thereby facilitating feedback for subsequent rounds. As stated, the initial rounds of the classical and modified Delphi differ (Keeney et al., 2011). In the classical Delphi, an open-ended question format is used to generate ideas. These data are evaluated with like items being combined and categorized to form the list of items to be rated in subsequent rounds. With the exception of this initial round, the process is the same in both the classical and modified Delphi. Before initiating the quantitative survey process in which participants rate each item on a well-defined Likert-type scale, the definition of consensus must be established. The level of consensus depends on the purpose of the Delphi and typically ranges from 51% to 100%, with 70% agreement being common (Keeney et al., 2011; Vernon, 2009). Investigators may look at measures of central tendency (mean, median, and mode) as well as at the percentage of responses that fall within a predetermined range of responses. For example, Harper et al. (2012) defined consensus using a 5-point Likert-type scale as ‘‘65% of participants indicating scores of 3, 4, or 5 for an item or items that at least 85% of the participants score as ‘4’ or ‘5’ and no participants score as ‘1’ or ‘2’’’ (p. 139). Once participants complete the initial quantitative survey round in which they rate items on a Likert-type scale, data are analyzed for central tendency and dispersion of scores. Items that have reached consensus may be reJournal for Nurses in Professional Development
moved from subsequent rounds of the survey to shorten the survey and time required for completion (Keeney et al., 2011). In addition, investigators may opt to remove items that have been rejected by consensus of all participants (Vernon, 2009). Before subsequent rounds, participants receive feedback from the previous round indicating their personal responses along with measures of central tendency of the overall group. Standard deviation may be included to reflect the dispersion of scores. This feedback allows participants to reflect on their responses and change them if desired. Delphi rounds may be discontinued after a predetermined number of rounds, typically three, or when consensus has been reached to either accept or reject ideas.
ETHICAL CONSIDERATIONS, METHODOLOGICAL RIGOR, AND LIMITATIONS As in any research study, attention to ethical considerations is critical when designing and conducting the Delphi study. As stated, approval to conduct the study should be sought from the agency’s institutional research review committee when the Delphi is research related. The procedure for identifying the expert panel should be clearly documented, and expert members should receive a written explanation of the purpose, procedures, benefits, time line, and time involvement for participation in the study as well as assurance that the responses of each individual will be kept confidential. Delphi studies offer quasi-anonymity rather than true anonymity. Because the expert group is a small, defined group, it is possible that the participants may have knowledge of the other members. However, the researcher can ensure that responses of any one panel member are not disclosed to other panel members, thereby preventing undue influence of more highly recognized experts. Codes are assigned to each expert, and the data for each round are thus linked to that code. There is ongoing debate about establishing methodological rigor for the Delphi. Kenney et al. (2011) note that the basis of this debate centers on whether the Delphi aligns with a positivist quantitative paradigm in which validity and reliability are used or a qualitative paradigm where criteria associated with credibility or trustworthiness of data, such as audit trails, are used to establish rigor. Irrespective of this debate, methodological strategies should be chosen to limit threats to the integrity of the technique. Careful thought should be given to the composition of the panel. Criteria for the expert panel members should be clearly delineated noting knowledge, skill, and experience levels as well as practice setting. Allowing for some heterogeneity of panel members, about practice settings for example, may yield a broad range of opinions and be more representative of the opinions of the population of www.jnpdonline.com
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
13
interest. The level of consensus should be established before the first round as well as procedures for communicating with the experts. It is critical that the aim of the Delphi be clearly communicated to everyone involved with the Delphi and that it is included as a part of each round. In addition, the researcher should document the process for analyzing the data from each round. This includes a step-by-step description of how responses from experts will be collapsed into categories or themes. Typically, a form of content analysis is used to analyze the qualitative responses. Actions and decisions that occur during the study should be documented and serve as an audit trail of thoughts and decisions. Ideally, involving another person or persons in the review of data and corroboration of findings enhances the credibility of the study. Credibility is also enhanced through the ongoing rounds and feedback from the panel of experts. Although there are many advantages in using the Delphi technique, it is important to be aware of potential limitations in using this approach. Reaching consensus does not mean that the correct answer or priority has been reached; only that consensus has been reached among the panel of chosen experts (Keeney et al., 2011). Moreover, the findings may not reflect the opinions and views of the larger population of experts or representative population. In the quest to reach consensus, variations in points of view, which may be important to consider, may be overlooked. Depth of inquiry in a topic may not be reached, but the Delphi technique may serve to identify a topic to be more fully explored at a later time. Furthermore, there is a potential for researcher bias in the interpretation of the data. This can be avoided by taking steps, as described above. Kenney et al. also note that the effect of group dynamics may influence expert responses in subsequent Delphi rounds. This influence may take the form of pressure to conform similar to the notion of ‘‘group think,’’ which involves group members forming quick responses or opinions that match those of others in the group. This may be more likely to occur when the members of the expert panel have knowledge of each other. In addition, because this technique requires participation in several rounds, attrition of experts may occur because of round fatigue (Vernon, 2009) or time delays between rounds (Kenney et al., 2011). Finally, as in any study, events occurring outside of the study may influence the experts’ responses and, if known to the researcher, noted in the study limitations.
USE OF THE DELPHI IN NPD PRACTICE As stated, the Delphi technique is useful for priority setting and developing consensus (Kenney et al., 2011; Vernon, 2009). This technique allows for input from a wide variety of stakeholders without extra time-consuming meetings and may promote timeliness of decision making in a variety 14
www.jnpdonline.com
of areas. Within NPD practice, the Delphi may be employed for activities such as developing research priorities, conducting educational needs assessments, identifying competencies for evaluation, developing educational objectives and content for courses, and delineating role responsibilities. Although these examples are not exhaustive, they may stimulate thinking about innovative ways to use the Delphi in other areas of NPD practice. One use of the Delphi in NPD practice is setting research priorities. Although often used on a national or international level to determine research priorities for a specialty (Blackwood et al., 2010; Harper et al., 2012), the Delphi could prove useful within a single organization. For example, research priority setting may occur on a unit level and engage all staff from the unit. Conversely, the priority setting could occur at an organizational level, engaging nursing leadership to ensure that research aligns with the organization’s mission, vision, and strategic goals. Perhaps, nursing leadership could be used to generate potential research ideas and then staff nurses selected to rate the importance of each, or vice versa. Engagement of staff in the process may promote buy-in and participation when the research is conducted. Another use of the Delphi in NPD practice is for educational needs assessment, as suggested in the previous section. Nursing leadership and nursing staff involvement are important to ensure identification of realistic needs that align with organizational initiatives. Using the Delphi for educational needs assessment not only identifies educational topics that are needed but assists with prioritizing these needs. Selection of competencies for assessment/evaluation is another use of the Delphi in NPD practice. For example, Lock (2011) used a Delphi method to identify core competencies required for licensure in Indonesia. Similarly, NPD departments could use a Delphi method to determine core competencies for assessment during orientation. In addition, a Delphi method may be helpful in narrowing a vast array of competencies for ongoing assessment as required by accrediting bodies. The Delphi method is also useful to determine objectives and content for educational activities (Goligher et al., 2012; Ito et al., 2011). As NPD specialists are challenged to manage competing priorities, identification of ‘‘need to know’’ content may ensure optimizing time spent in program development as well as staff time in participation in programs. Content of orientation, inservices, and continuing education programs could all be guided by Delphi surveys. Finally, the Delphi technique may be helpful in determining essential components of nursing roles both within NPD and in other practice areas. For example, Chang et al. (2010) used a Delphi survey to validate an advanced practice nursing tool. This tool identified the elements of practice January/February 2014
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
for the advanced practice role. Likewise, as NPDVand otherVdepartments are confronted with increasing demands, sometimes with shrinking resources, identification of key role responsibilities may be warranted.
SUMMARY The Delphi technique, commonly used in a broad range of disciplines, has potential for use in NPD as specialists face challenges in complex healthcare systems, seek to define and clarify practice roles, and expand the evidence base for practice. As a multistaged survey, the technique may be used to set priorities or gain consensus in a variety of situations such as establishing research priorities, conducting needs assessments, selecting competencies and educational content, or identifying practice role responsibilities. It may be of particular use for initially engaging staff in the research process. As in any research, particular attention to ethical considerations and methodological rigor is essential.
References Back-Pettersson, S., Hermansson, E., Sernert, N., & Bjorkelund, C. (2008). Research priorities in nursingVA Delphi study among Swedish nurses. Journal of Clinical Nursing, 17(16), 2221Y2231. Blackwood, R., Albarran, J., & Latour, J. (2010). Research priorities of adult intensive care nurses in 20 European countries: A Delphi study. Journal of Advanced Nursing, 67(3), 550Y562. Calsyn, D., Burlew, A., Hatch-Maillette, M., Wilson, J., Beadnell, B., & Wright, L. (2012). Real Men Are SafeVCulturally adapted: Utilizing the Delphi process to revise Real Men Are Safe for an ethnically diverse group of men in substance abuse treatment. AIDS Education and Prevention, 24(2), 117Y131.
Journal for Nurses in Professional Development
Chang, A., Gardner, G., Duffield, C., & Ramis, M. (2010). A Delphi study to validate an advanced practice nursing tool. Journal of Advanced Nursing, 66(10), 2320Y2330. Culley, J. (2011). Use of a computer mediated Delphi process to validate a mass casualty conceptual model. Computers, Informatics, Nursing, 29(5), 272Y279. de Meyrick, J. (2003). The Delphi method and health research, Health Education, 103(1), 7Y16. De Villiers, M. R., De Villiers, P. J., & Kent, A. P. (2005). The Delphi technique in health sciences education research. Medical Teacher, 27(7), 639Y643. Goligher, E., Ferguson, N., & Kenny, L. (2012). Core competency in mechanical ventilation: Development of educational objectives using the Delphi technique. Critical Care Medicine, 40(10), 2828Y2832. Grundy, M., & Ghazi, R. (2009). Research priorities in haematooncology nursing: Results of a literature review and a Delphi study. European Journal of Oncology Nursing, 13(4), 235Y249. Harper, M., Asselin, M., Kurtz, A., MacArthur, S., & Perron, S. (2012). Research priorities for nursing professional development: A modified e-Delphi study. Journal for Nurses in Staff Development, 28(3), 137Y142. Ito, C., Ota, K., & Matsuda, M. (2011). Educational content in nurse education in Japan: A Delphi study. Nursing Ethics, 18(3), 441Y454. Keeney, S., Hasson, F., & McKenna, H. (2011). The Delphi technique in nursing and health research. West Sussex, UK: Wiley-Blackwell. Kinunen, U., Saranto, K., Ensio, A., Iivanainen, A., & Dykes, P. (2012). Developing the standardized wound care documentation model: A Delphi study to improve the quality of patient care documentation. Journal of Wound, Ostomy & Continence Nursing, 39(4), 397Y407. Lock, L. (2011). Selecting examinable nursing core competencies: A Delphi project. International Nursing Review, 58, 347Y353. Vernon, W. (2009). The Delphi technique: A review. International Journal of Therapy and Rehabilitation, 16(2), 69Y76.
www.jnpdonline.com
Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.
15
