ORIGINAL CONTRIBUTION

Content Validation of a Novel Screening Tool to Identify Emergency Department Patients With Significant Palliative Care Needs Naomi George, MD, Nina Barrett, NP, Laura McPeake, MD, Rebecca Goett, MD, Kelsey Anderson, and Janette Baird, PhD

Abstract Background: The emergency department (ED) is increasingly used by patients with life-limiting illness. These patients are frequently admitted to the hospital, where they suffer from poorly controlled symptoms and are often subjected to marginally effective therapies. Palliative care (PC) has emerged as the specialty that cares for patients with advanced illness. PC has been shown to reduce symptoms, improve quality of life, and decrease resource utilization. Unfortunately, most patients who could benefit from PC are never identified. At present, there exists no validated screening tool to identify significant unmet PC needs among ED patients with life-limiting illness. Objectives: The objective was to develop a simple, content-valid screening tool for use by ED providers to identify ED patients with significant PC needs. A positive screen would result in an inpatient PC consultation. Methods: An initial screening tool was developed based on a critical review of the literature. Content validity was determined by a two-round modified Delphi technique using a panel of PC experts. The expert panel reviewed the items of the tool for accuracy and necessity using a Likert scale and provided narrative feedback. Expert’s responses were aggregated and analyzed to revise the tool until consensus was achieved. Greater than 80% agreement, as well as meeting Lawshe’s critical values, was required to achieve consensus. Results: Fifteen experts completed two rounds of surveys to reach consensus on the content validity of the tool. Three screening items were accepted with minimal revisions. The remaining items were revised, condensed, or eliminated. The final tool contains 13 items divided into three steps: 1) presence of a lifelimiting illness, 2) unmet PC needs, and 3) hospital admission. The majority of panelists (86%) endorsed adoption of the final screening tool. Conclusions: Use of a modified Delphi technique resulted in the creation of a content-validated screening tool for identification of ED patients with significant unmet PC needs. Further validation testing of the instrument is warranted. ACADEMIC EMERGENCY MEDICINE 2015;22:823–837 © 2015 by the Society for Academic Emergency Medicine

T

he aging of the population coupled with advancements in medical care has resulted in a growing number of people living with chronic life-limiting illnesses. This has led to increased use of the emergency department (ED) by patients suffering from advanced illnesses and frequent hospitalizations toward the end of life.1 For these patients, hospitaliza-

tion is often characterized by unnecessary suffering from burdensome physical symptoms and psychological distress, as well as invasive interventions that are frequently incongruent with patient and caregiver wishes.2,3 Palliative care (PC) has emerged as the specialty that seeks to support and care for patients with advanced illness.

From the Department of Emergency Medicine, Alpert Medical School, Brown University (NG, LM, RG, JB), Providence, RI; the New York University School of Medicine (NB), New York, NY; and Brown University (KA), Providence, RI. Funded by the University Emergency Medicine Foundation, Seed Grant. Received November 21, 2014; revision received January 20, 2015; accepted January 25, 2015. The authors have no potential conflicts to disclose. Supervising Editor: Brian Hiestand, MD, MPH. Address for correspondence and reprints: Naomi George, MD; e-mail: [email protected].

© 2015 by the Society for Academic Emergency Medicine doi: 10.1111/acem.12710

ISSN 1069-6563 PII ISSN 1069-6563583

823 823

824

Palliative care focuses on improving quality of life by preventing and relieving suffering through treatment of physical, psychosocial, and spiritual problems.4 PC is not synonymous with hospice care. PC encompasses hospice care, but is a much broader entity. Ideally, PC is initiated early in a patient’s disease course and can be delivered concurrently with curative therapies.5 In contrast, hospice care is only available in the terminal stage of disease and is exclusive of providing concurrent curative care. Many high-quality studies, including several randomized trials,6–8 have examined the benefits of PC and shown it leads to significant improvements in patient quality of life and mood,9 increased patient and caregiver satisfaction,3,10,11 reduction in health care costs,12– 14 decreased intensive care unit (ICU) days,15–18 hospital length of stay, physical symptomatology, and improved resource usage.19 Despite the clear benefits of PC, the vast majority of patients who could benefit from PC are never identified or referred at any point during the course of their illness.20,21 This situation is particularly dire among the medically underserved, whose access to PC is often severely limited and whose goals of care are frequently unaddressed. Similarly, the physical and psychological distress of patients living with advanced nonmalignant diseases, such as heart failure or advanced dementia, is often underrecognized. These patients are less likely to receive PC compared to patients with cancer.22–25 The current system of referral relies on primary care doctors, hospitalists, and specialists to provide referral; however, all have been slow to integrate PC into their practices and consistently fail to make appropriate and/or timely referral to PC for their patients.23,26–28 There are several reasons why PC screening should fall under the purview of ED providers. First, demographic shifts have led to increased use of the ED by patients with advanced chronic illness.29–31 Over 75% of older adults visit the ED in their final 6 months of life, and 51% have at least one visit in their last month of life.32 There is also a high prevalence of unmet PC needs among ED patients.33,34 During times of high symptom burden and psychological distress, patients may interact more with ED providers than with their own primary care providers.35 The ED is increasingly the locus of care for the medically underserved and/or high-risk patients, including those whose access to PC has been severely limited by insurance status and available community health resources.36,37 Finally, critical decisions regarding a patient’s trajectory and disposition are made during the ED encounter, which could provide an opportunity for appropriate and timely referral to PC.38 For the minority of patients who do ultimately receive PC referrals, there is a long latency between hospital admission and PC referral.39 This time is often characterized by increased health care resource utilization that is incongruent with patient and caregiver wishes.40–42 While many patients with advanced illness see their medical specialists frequently in the last year of life, the emphasis on active treatment often overshadows concomitant symptom management.43

George et al. • EM PALLIATIVE CARE SCREENING TOOL

Finally, there are significant racial and ethnic disparities in PC and hospice referral. In 1996, 93% of enrollees in the Medicare hospice benefit were white.44 The Institute of Medicine has called for increased access to end-of-life care for poor and inner-city patients, many of whom are from minority populations.20,21,45,46 EDbased PC consultation could increase access to such services for racial and ethnic minorities who tend to seek care in the ED in disproportionately large numbers.47–49 Identified barriers to PC consultation in the ED include the acuity of medical problems, lack of privacy,50 rapid decision-making by new physicians, absence of a complete medical history, and an ED culture that prioritizes aggressive disease-modifying therapy51 and problem-focused assessment.5,52,53 Moreover, many ED providers erroneously believe that their patients’ PC needs are already being met in other clinical settings.54,55 ED practitioners have also cited lack of 24-hour coverage by consultants as a major barrier to PC consultation. National organizations, including the National Institutes of Health56,57 and the Agency for Healthcare Research and Quality,58 have called for the integration of ED and PC care to help better meet the need of patients with advanced illness. In 2013, as part of the “Choosing Wisely” campaign, American College of Emergency Physicians (ACEP) endorsed “timely engagement of PC services” as a key performance measure.59 However, at present, there are no validated screening tools to identify unmet PC needs among ED patients with all life-limiting illnesses. In this study we address this problem by developing and validating an ED screening tool to identify patients with any advanced chronic illness who will likely benefit from inpatient PC consultation. Our objective was to develop an evidenced-based ED PC screening and referral tool based on critical review of the literature and to provide content validation of this screening tool among PC experts. In addition to achieving content validity, we endeavored to develop a screening tool that would increase the timeliness, consistency, and frequency of appropriate referrals to PC consultation for ED patients admitted to the hospital. We attempted to minimize potential barriers to widespread adoption in the ED by striving to ensure that the resulting tool would be brief and user-friendly and would not require additional resources such as PC specialists (case managers, social workers, or others) to complete. Our goal was to design an instrument that could be completed by a single ED provider based on information gathered during a routine ED patient encounter. Content validity, the degree to which elements of an assessment instrument are relevant to and representative of the targeted construct for a particular assessment purpose,60,61 is a critical element of testing validation. Given that the use of a content-invalid assessment instrument degrades the clinical inferences derived from the obtained data,62 we felt that content validation was the necessary first step to rational instrument development. We anticipate further testing of sensitivity, specificity, and feasibility of this instrument.

ACADEMIC EMERGENCY MEDICINE • July 2015, Vol. 22, No. 7 • www.aemj.org

METHODS We undertook a three-phrase process for developing the screening tool: 1) a critical review of the literature and an initial screening tool development, 2) pilot-phase testing, and 3) content validation with modified Delphi technique. The study was granted exempt status by the hospital institutional review board. Initial Construct and Item Development To generate initial structure and content for the screening tool, two investigators (NG, NB) performed a critical review of the literature including previously published articles on PC interventions or demonstration projects in the ED. We also reviewed articles on barriers to ED PC integration. English-language articles were searched in the PubMed, Cochrane, and CINAHL databases from January 1990 through October 15, 2013. Citations found in relevant articles were reviewed, as well as consensus opinion and policy statements of national PC organizations. Complete search terms can be found in Figure 1. Articles were included if they discussed provider-based PC screening, PC interventions in the ED, and attitudes toward and barriers to providing PC in the ED. PC screening tools for ICU and hospital patients were also reviewed. Pilot Phase The preliminary tool was piloted with four volunteer clinicians, two clinicians with advanced trained in PC and two clinicians with emergency medicine training. These volunteers had no other connection to the study besides this role. Each volunteer reviewed the structure and items of the screening tool, and open discussion was used to modify and adjust the language and content based on feedback. Content Validation Phase Content validity of an instrument is established by content analysis performed by qualified experts in the domain of the content to be assessed.63 We selected a modified Delphi technique to evaluate the content validity of our screening tool. Delphi technique is a structured process commonly used to generate consensus among experts and is frequently employed when seeking to address content validity in the absence of a criterion standard. The technique involves a collaborative process that enables a group of panelists to anonymously share their knowledge and opinions and build consensus over multiple rounds of revisions.64 This method guides panelists toward consensus and reveals

Figure 1. Literature review search terms.

825

which items are valid, which require modification, and which are invalid representations of content domains. While no consensus recommendations exist for the structure of a modified Delphi panel, we modeled the structure of our Delphi technique on the recommendations reported by Boulkedid et al.65 in their 2011 systematic review of the topic, including: 1) the use of a preset level of percent agreement and stop criteria, 2) use of a 1 to 9 Likert rating scale, 3) two to three rounds, 4) heterogeneous panel, 5) and mixed qualitative and quantitative feedback. Our goal was to establish a panel of 10 to 20 experts; therefore, we initially invited 40 potential panelists. A PC expert was defined as having a minimum of 3 years of clinical experience as a PC clinician and/or as holding a position as either a faculty educator in PC or a clinician-researcher in PC (as determined by publication of one or more PC-themed peer-reviewed articles). We invited authors of articles found in our literature review (n = 25), as well as clinician-educators at geographically diverse, hospital-based PC training programs (n = 15). Experts were asked to commit to respond to two questionnaires, and each received a gift card for participation. A panel of PC experts was used to assess content validity of the screening tool as a whole, as well as of the individual items contained within. The study questionnaire was based on Dillman’s criteria for survey design, including respondent-friendly design, static question formatting, and mixed-mode design.66 An online survey tool (Survey Monkey) was used as the platform. Each round consisted of Likert scale and open comments. The first round was completed between April and May 2014. The survey consisted of 56 questions regarding the screening tool as approved in pilot testing. Respondents completed a 9-point Likert scale specifying how necessary and accurate they felt each item was in representing key PC domains. They were also asked the likelihood of the tool successfully identifying patients with significant PC needs. A rating of “1” was defined as “strongly disagree” while a “9” was defined as “strongly agree.” Panelists were also asked whether any important domains of PC had been left out. Each question in the survey enabled panelists to optionally provide written explanations for their responses and/or recommendations for revision. Invitations to the second round online survey were only sent to those panelists who completed the first round. The second round took place in June 2014. The general format of round 2 was identical to that of round 1, with the addition of a summary of responses from round 1. It also included an enhanced explanation and graphics explaining the two-step screening model, as this had been the cause of some confusion in round 1. Investigators tabulated quantitative and qualitative results from both rounds. Descriptive statistics were used to describe the necessity and accuracy of the tool, as a whole as well as of each individual item. Comments were reviewed by investigators (NG, NB, RG, LM) and categorized using a thematic analysis to identify common themes and key patterns. Items were revised based on this analysis.

826

Consensus Two predetermined metrics were used to determine when consensus was reached; percent agreement > 80% and Lawshe’s content validity ratio (CVR).67 An item was considered to have reached consensus when there was an 80% or greater agreement between panelists that the item was both necessary and accurate and the item achieved critical value based on the number of respondents (CVR of > 0.411 in round 1 and > 0.425 in round 2, range = –1 to 1). Critical values for CVR were set using a one-tailed test (p < 0.05) based on the corrected tables of critical values published by Wilson et al.68 Lawshe’s CVR was chosen over other quantifiable methods that determine content validity because the Lawshe method focuses on deeming an item “essential” or “not essential,” while other methods focus on inter-rater agreement. An expert ranking of 1 through 3 on the Likert scale indicated that the item was unnecessary; 4 through 6 indicated that it was important but not necessary; 7 through 9 indicated that the item was necessary. A similar scale was used for accuracy. If an item received greater than 80% consensus that it was both necessary and accurate (i.e., greater than 80% ranked is 7 through 9 on the scale), and met the critical valued for the CVR, then it was accepted with no modifications. If greater than 80% of panelists believed that an item was necessary, but only 50% to 80% of panelists deemed it accurate, then it was modified based on aggregated feedback in an effort to achieve a necessary and accurate item. If 50% to 80% of panelists agreed that an item was necessary, then the item was deemed “important” but not necessary. Meaningful concepts from the item were reworked based on narrative feedback and re-pre-

Figure 2. Critical literature review.

George et al. • EM PALLIATIVE CARE SCREENING TOOL

sented in the second round. If less than 50% of panelists thought an item was necessary, the item was discarded regardless of the accuracy level. Because the literature on the Delphi technique does not provide a validated scoring system, we chose a high level of agreement on the Likert scale (>7) to count an item as necessary/accurate. While other studies include all responses greater than five or six, we felt the strength of agreement would be more meaningful at higher levels. RESULTS Critical Review of the Literature and Initial Screening Tool Development Literature review revealed an overall dearth of evidence regarding ED PC projects and/or screening tools. Relevant articles were reviewed for strengths and weaknesses. We identified five articles that met our predetermined criteria for relevance. A flow chart of the literature review is available in Figure 2. All ED PC referral or integration projects that we examined had some measure of success in terms of increasing PC evaluation and referral from the ED.69–73 Common weaknesses of prior efforts included absence of evidenced-based development of an ED-specific screening tool, narrow inclusion criteria (i.e., cancer patients only), use of lengthy PC patient assessment tools, and reliance on additional staff to perform screening. Expert and consensus opinions on PC screening were also reviewed, including screening recommendations for ED, hospital, and ICU settings. The preliminary framework for the screening tool was based on a review of current concepts in PC as described by expert opinion including the National Quality Forum,74 screen-

ACADEMIC EMERGENCY MEDICINE • July 2015, Vol. 22, No. 7 • www.aemj.org

ing models published by the Improving Palliative Care in Emergency Medicine and ICU groups (IPAL-EM and IPAL-ICU, respectively), and screening methods by the Center to Advance Palliative Care.75–77 Using these resources, a preliminary three-step screening tool was developed. In an effort to preemptively address ED provider concerns, we reviewed articles regarding attitudes and perceived barriers of ED providers toward PC integration.78 Two key issues cited by ED providers as barriers to PC consultation were absence of 24-hour availability and having incomplete patient information available during a fast-paced ED encounter. To help address these concerns, we made two key provisions in the construction of the screening tool. First, we designed the tool to screen only for PC needs among those being admitted to the hospital. Currently, hospital-based PC consultation is far more robust than community-based services. The prevalence of a PC service in U.S. hospitals with 50 or more beds is estimated to be 75%.79 Outpatient PC availability is markedly lower; for instance, data from California revealed that only 8% of hospitals had outpatient PC referral available.80 Thus, in most practice environments we predict it will be far more feasible for the ED provider to convey the results of the screening test to an inpatient PC consultation service rather than to a (perhaps nonexistent) outpatient PC consultation service. Given that hospital-based PC is more widely available, it is more likely that PC consultation can be arranged for patients admitted to the hospital from the ED, rather than discharged from the ED to the community. Research has shown that ED providers perceive that the incomplete clinical picture available in the ED is a significant barrier to screening for PC needs among their patients.53,78 While it is true that an ED clinician cannot perform a comprehensive PC assessment for a patient based solely on a single clinical encounter in the ED, screening based on easily identifiable attributes of advanced disease states is likely feasible with prompting and cognitive cues. Therefore, we reviewed articles regarding disease-specific thresholds for PC to include appropriate cues for ED providers to sufficiently identify advanced disease states.2,81–92 Based on this review, we developed a preliminary ED-specific tool that has only three questions: Step 1— Does the patient have a life-limiting illness? Step 2— Does the patient have unmet PC needs? Step 3—Is the patient being admitted to the hospital? To assist the ED provider in completing the preliminary screening tool, 13 items were included for guidance. In the preliminary tool, Step 1 included seven items. These items were based on current disease-specific PC needs2,82–93 and current hospice eligibility guidelines. We felt that these items represented a comprehensive but brief list of lifelimiting illnesses (dementia, cancer, neurodegenerative disease, chronic renal failure, end-stage liver disease, heart failure, and chronic pulmonary disease). Step 2 initially included five clarifying items designed to represent the key PC construct domains.93 We identified five major PC constructs applicable to the emergency setting: 1) physical aspects of care including pain and symptom management, 2) social aspects of care

827

and functional status, 3) psychosocial care, 4) caregiver needs, and 5) quality of life. Although it is a key construct of PC, we chose to exclude spiritual aspects of care from our screening tool as we felt this domain may be difficult for ED providers to engage with given the limitations in continuity of care in the ED and may be more appropriately addressed by PC consultants. Our Preliminary Screening Tool and supporting evidence are available in Figure 3. Pilot Phase The preliminary screening tool was given to the pilot group for evaluation. As a result of the pilot review, two additional items were recommended for inclusion under Step 1, and multiple items were revised. Specifically, respondents highlighted the need to address two additional life-threatening illnesses that often present to the ED: those with severe sepsis and those sustaining serious trauma. These items were incorporated given the high prevalence of these problems in ED populations and significant associated mortality. Based on the recommendation of the group, changes were made in the survey’s length, layout, and wording. The Pilot Phase Screening Tool contains three steps and 15 items and can reviewed in Figure 4. Delphi Panel Invitations to participate in the study were sent out to 40 expert PC physicians and nurse practitioners (NPs). Seventeen invitees (42.5%) consented to participate in the survey. Of those who consented to participate, 94% (n = 16) completed round 1, and 93% (n = 15) completed the second and final round. The panel represented a geographically diverse group that was sex-balanced (among respondents, NPs predominantly identified as women, and MDs as men) and included providers from multiple training backgrounds, including advanced practice nurses in PC and fellowship-trained PC physicians. Experts practiced in various types of settings (outpatient PC, academic hospitals, county hospitals, etc.) with various levels of PC services available (PC consultation service, inpatient PC unit, etc.). Characteristics of panel members are included in Table 1. Modified Delphi Panel: Round 1 Results After pilot testing and modification, the pilot screening tool (Figure 4) was given to the Delphi panel for review. Sixteen experts completed the online survey. Step 1 contained a single question designed to identify life-limiting illness. Nine clarifying items were included in Step 1 to help ED providers identify life-limiting illness. Step 2 also contained a single question (Does this patient have significant PC needs?) and was composed of six items representing the main PC content domains. Panelists deemed Step 1 to be necessary (100% agreement, p < 0.001) and accurate (93.7% agreement, p < 0.001). Of the nine items in Step 1, all received critical approval (>80% agreement) for “necessary.” In total, three of the nine items received critical approval for “accurate,” and six items received between 50 and 80% approval. As a whole, Step 2 was also deemed necessary (87.5% agreement, p < 0.001) and accurate (81.3% agreement, p < 0.05). Overall, three of the six items received critical

828

George et al. • EM PALLIATIVE CARE SCREENING TOOL

Supporting Literature

Construct Item Step 1: Select any Life-limiting or Life-threatening condition that applies:

Quest, 2011 Weissman, 2011 Ross, 2008

Dementia (requiring significant support with ADLs)

Ahronheim, 2000

Cancer (Metastatic or locally incurable)

Smith, 2012

Progressive CNS Disease (requiring significant support with ADLS)

Hussain, 2013 Foley, 2001

Chronic Renal Failure (On dialysis)

Hussain, JA 2012 Cohen, 2006

End Stage Liver Disease

Poonja, 2013 Boyd, 2011

Congestive Heart Failure: (marked limitation of ADLS)

LeMond, 2011 Howlett, 2011

Advanced COPD (requiring home O2)

Cao, 2006

Step 2: Select all Secondary Criteria that apply: You would not be surprised if this patient died within 12 months

Moss, 2008 Moss, 2010

> 1 Admission or ED visits for issues related to the same chronic illness in the past 6 months

Earle, 2003

Difficult to control physical or psychological symptoms: pain, dyspnea, depression, etc.

Nelson, 2001 Solano, 2006

Functional decline: Feeding intolerance, unintended weight loss, etc.

Harris, 2013

New or worsened complex care requirements

National Quality Forum, 2006

Significant Caregiver Distress or Limited Social Support

National Quality Forum, 2006

Step 3: Is the patient being admitted to the Hospital?

Yes

Consult Palliative Care

No

Screening complete

Figure 3. Preliminary screening tool and supporting literature.

approval for “necessary” and “accurate” and were finalized; three of the six items received 50% to 80% approval and were modified. Respondents provided 208 comments in round 1; the majority of comments consisted of suggestions for improvements that could be made to increase simplicity and accuracy of the items (77%). Disagreement or criticism of an item constituted 23% of comments. Results from round 1 are presented in Table 2. Modified Delphi Panel: Round 2 Results Round 2 had a high participation rate (93.7%). Results (both quantitative and qualitative) from round 1 were used to modify those items that did not meet critical approval for accuracy. Detailed results from round 1 were provided to participants to consider when formulating their responses to the modified screening tool presented in round 2. Seven items from round 1 were modified and resubmitted to panelists for further validation. The items for identifying “Advanced CNS Disease” and “Advanced Dementia” in Step 1 were collapsed into a single item to improve the brevity of the tool and also increase the accuracy of the item. Likewise, in Step 2 an item for identifying complex care needs was incorpo-

rated into an item regarding caregiver distress. Other items were heavily edited based on comments from round 1. Respondents in round 2 felt that the tool as a whole was accurate (86.6% agreement) and would adopt the tool in their settings (82% agreement). In total, six of seven revised items received critical approval for being both “necessary” and “accurate.” One item, “Frequent ED Visits,” did not receive critical approval for “necessary” or “accurate.” A total of 46 comments were provided in round 2. Of these comments, 42 (91.3%) expressed approval of the modified items, while four comments (8.6%) expressed doubt regarding validity of specific items. Results from round 2 are presented in Table 3. Figure 5 charts the evolution of the modifications made by the panelists. To guide future sensitivity and specificity testing, we obtained the opinion of our expert group regarding the appropriate “threshold” of PC needs in Step 2 necessary for accurate identification of patients with a high likelihood of benefiting from PC consultation. Panelists were asked to recommend a minimum number of items in Step 2 that would be necessary to generate a positive screening that would result in referral: 58% of panelists felt that only one item was necessary, while 42% sug-

ACADEMIC EMERGENCY MEDICINE • July 2015, Vol. 22, No. 7 • www.aemj.org

829

Step 1: Select ALL Life-limiting conditions that apply:

Step 2: Select ALL Secondary Criteria that apply:

Figure 4. Pilot-phase screening tool.

gested that the threshold for initiating PC consultation should be two or more items. The final tool can be viewed in Figure 6. DISCUSSION The proposed ED-PC screening tool is novel in several ways. First, the tool is designed to be brief and contain as few questions as necessary while still representing the key domains of PC. Time constraints are a significant barrier to the adoption of any initiative in the ED setting.94 Thus, we sought to make the screening tool as brief as possible while retaining content validity. A total of 13 items specifically designed to provide guidance to ED providers were validated by PC experts. This screening tool is considerably shorter than most other PC tools described in literature, which often contain dozens of items.95,96 In contrast to the existing tools, this one is anticipated to be better suited to the rapid pace of care delivery in the ED. Second, this is the only content-valid ED PC screening tool for patients with either malignant or nonmalignant illness. By asking our expert panelists to comment

on the necessity and accuracy of each screening item, as well as recommend additions or omissions, we ensured that the content of the screening tool sufficiently reflected the key PC content domains, including a broad definition of life-limiting illness. Referral rates for PC remain low among providers in all settings (including among ED providers) in part due to provider bias and provider knowledge gaps. Thus, our goal was to develop a tool that could increase appropriate referrals by addressing sources of bias where possible. While physicians are inclined to refer patients with metastatic cancer for PC consults, they are much less likely to consider PC for patients with organ failure, dementia, COPD, or heart disease. These patients collectively represent the majority of patients with significant unmet PC needs.10,97 We were careful to structure the tool in such a way that it will be evident to the provider that many patients with advanced illnesses should also be evaluated for unmet PC needs. Additionally, due the simplicity of its design and language, this tool has the potential to be accessible to ED providers with multiple training backgrounds, including physicians, NPs, physician assistants, and registered nurses.

830

George et al. • EM PALLIATIVE CARE SCREENING TOOL

Table 1 Demographic Characteristics of Delphi Panelists Characteristic Sex Male Female Years of practice 2 to 4 5 to 10* More than 10 Practice site† Hospital inpatient PC consult service Hospital inpatient PC service Hospice Outpatient PC ED Other Professional role Physician, attending Nurse practitioner

Round 1

Round 2

9 (50.0) 7 (50.0)

9 (60.0) 6 (40.0)

2 (13.3) 8 (53.3) 5 (33.3)

2 (13.3) 9 (60.0) 4 (26.7)

11 2 6 5 5 4

(68.8) (12.5) (37.5) (31.3) (31.3) (25.0)

11 (73.3) 4 (26.7)

10 4 4 7 5 0

(66.7) (26.7) (26.7) (46.7) (33.3) (0.0)

10 (66.7) 5 (33.3)

Data are reported as n (%). PC = palliative care. *Only respondents who completed round 1 were invited to complete round 2. However, in round 1 not all participants provided complete demographic information. In round 2, participants were encouraged to provide complete demographic information. †Respondents were allowed to select multiple choices for “Practice Site” to accurately represent providers who work in multiple settings.

Finally, this screening tool is novel in that it was designed to be completable by ED providers based solely on information likely to be ascertained during routine patient encounters. Unlike previously published PC screening tools, this one should not require a provider to perform an additional patient interview, nor should it require a PC specialist or case manager for completion.15,69,70,72 Ultimately, for this screening tool to increase rates of PC consultation as well as decrease the long latency between hospital admission and PC consultation, the results of the screening tool must be put to use. Multiple interventions based on this screening tool are conceivable. We propose a collaboration between ED and PC services, such that the PC team promptly reviews all positively screened patients and provides PC consultation on cases deemed appropriate within several days of admission. This proposal is based on an ideal system in which the PC team staffing and coverage is sufficient for the possible increase in caseload. It is easy to imagine multiple adaptations to this scheme based on local practice. There are several strengths to the methodology of this study. First, the diversity of the expert panel contributes to the generalizability of the consensus results and the screening tool. Panelists worked in multiple hospital systems with different norms and practice patterns. The panelists represent a geographically diverse group with wide variation in training backgrounds

Table 2 Round 1 Results: Mean Score, Percent Agreement, and Content Validity Ratio of Screening Items Necessary Item Step 1 Dementia CNS disease Cancer ESRD COPD CHF ESLD Septic shock/ multiorgan failure Accelerated death Section 1 summary‡ Step 2 12 months >1 ED admission Difficult symptoms Complex care Functional decline Caregiver distress Section 2 summary§

Mean* (SD)

Accurate

% Agreement

Lawshe’s Critical Value (–1 to 1)

pvalue

(0.5) (0.5) (0.0) (1.2) (0.9) (0.6) (0.3) (1.1)

100 100 100 93.8 93.8 100 100 93.8

1.000 1.000 1.000 0.875 0.875 1.000 1.000 0.875

0.001 0.001 0.001 0.001 0.001 0.001 0.001 0.001

6.5 6.7 8.0 6.4 7.0 7.3 8.3 6.8

7.7 (2.1) 8.6 (0.7)

86.7 100

0.733 1.000

8.7 7.7 8.1 7.5 8.4 8.2 7.9

100 86.7 86.7 73.3 93.3 93.3 86.7

1.000 0.733 0.733 0.467 0.867 0.867 0.733

8.8 8.8 9.0 8.2 8.6 8.7 8.9 8.5

(0.7) (2.1) (1.6) (1.7) (1.0) (1.2) (1.3)

Mean (SD)

% Agreement

Lawshe’s Critical Value (–1 to 1)

pvalue

(2.0) (2.2) (1.4) (2.2) (2.0) (1.7) (1.1) (2.2)

52.9 68.8 87.5 62.5 68.8 73.3 93.8 56.3

0.059 0.375 0.750 0.250 0.375 0.467 0.875 0.125

0.100 0.100 0.005 0.100 0.100 0.050 0.001 0.100

0.05 0.001

6.0 (2.7) 7.7 (1.0)

46.7 93.3

–0.067 0.867

† 0.001

0.001 0.005 0.005 0.050 0.001 0.001 0.005

8.5 6.5 7.5 7.4 7.7 7.5 7.7

100 60.0 73.3 66.7 80.0 73.3 80.0

1.000 0.200 0.467 0.333 0.600 0.467 0.600

0.001 0.100 0.050 0.100 0.025 0.050 0.025

(0.8) (2.6) (2.1) (1.7) (1.7) (1.9) (1.2)

CHF = congestive heart failure; CNS = central nervous system; COPD = chronic obstructive pulmonary disease; ESLD = end-stage liver disease; ESRD = end-stage renal disease. *Scale was from 1 to 9. †p-value for Lawshe CVR less than zero is not defined. ‡In Section 1 summary, panelists were asked to rate the accuracy and essentialness of Section 1 in terms of identifying patients with life-limiting illness. §In Section 2 summary, panelists were asked to rate the accuracy and essentialness of Section 2 in terms of identifying patients with significant unmet palliative care needs.

ACADEMIC EMERGENCY MEDICINE • July 2015, Vol. 22, No. 7 • www.aemj.org

831

Table 3 Round 2 Results: Mean Score, Percent Agreement, and Content Validity Ratio of Screening Items Necessary Item Dementia/CNS CNS disease† ESRD COPD CHF‡ ESLD‡ Septic shock Accelerated death 12 months‡ ≥2 ED admissions Difficult symptoms‡ Complex care† Functional decline‡ Caregiver distress Overall assessment of instrument Adoption Would you endorse adoption of instrument in the ED?

Mean* (SD) 8.0 (1.0) – 7.7 (1.5) 8.3 (0.9) – – 8.4 (0.8) 7.7 (1.9) – 7.0 (2.0) – – – 7.7 (1.2) 7.3 (1.6)

7.4 (1.49)

% Agreement 93.3 – 86.7 100 – – 100 86.7 – 66.7 – – – 85.7 73.3

86.7%

Accurate

Content Validity Ratio (–1 to 1)

pvalue

Mean* (SD)

% Agreement

Content Validity Ratio (–1 to 1)

pvalue

0.867 – 0.733 1.000 – – 1.000 0.733 – 0.333 – – – 0.714 0.467

0.001 – 0.005 0.001 – – 0.001 0.005 – 0.100 – – – 0.005 0.05

8.1 (0.8) – 7.5 (1.3) 8.0 (0.9) – – 8.3 (0.8) 7.5 (1.7) – 6.9 (2.1) – – – 7.0 (1.4) 7.4 (1.5)

100 – 86.7 100 – – 100 86.7 – 66.7 – – – 85.7 86.7

1.000 – 0.733 1.000 – – 1.000 0.733 – 0.333 – – – 0.714 0.733

0.001 – 0.005 0.001 – – 0.001 0.005 – 0.100 – – – 0.005 0.005

0.7333

0.001

– = not included in round 2. CHF = congestive heart failure; CNS = central nervous system; COPD = chronic obstructive pulmonary disease; CNS = central nervous system; ESLD = end-stage liver disease; ESRD = end-stage renal disease. *Scale was from 1 to 9. †Item eliminated in round 1, not included for review in round 2. ‡Item fully approved in round 1, not included for review in round 2.

within PC, reflective of the diversity of the field. The high response rate (93.7%) of panelists from round 1 to round 2 is a strength and decreases the likelihood that selection bias significantly contributed to the consensus achieved by the panel. Additionally, we required an approval of >80% for each item in order to be accepted by the panel. This high level of consensus was required to ensure content validity, so only the most essential items would be included and maintain the brevity essential for ED feasibility. Additional evidence of high-level of consensus is found in the Lawshe’s CVR, which revealed very high levels of agreement for all included items. Item “stability” (the degree to which items are revised, omitted, or added across rounds) has been proposed as an additional measure of agreement for Delphi panels. Item stability was evidenced in this study by the absolute reduction in narrative comments expressing disagreement, which dropped from 23% of comments in the first round to only 8.6% in the second round. To balance the twin aims of developing a tool with a high level of content validity that was also clear and brief, we used a combined approach of critical review of the literature and the modified Delphi technique. Whereas the Delphi technique typically begins by asking experts open-ended questions to generate initial items, and then narrows the items through iterative discussion, we opted instead to generate the items and structure first by literature review and then solicited further suggestions for items from expert panelist during the first round of review. While this hybrid methodol-

ogy introduces the threat of bias, we felt that it was an important strategy to ensure that the final screening tool could be feasibly implemented in an ED environment. Furthermore, we felt that by offering panelists an opportunity to suggest additional items, potential bias was mitigated. LIMITATIONS This study represents one of the first attempts to systematically develop a PC screening tool for the ED, and thus there are several important limitations that can be attributed to the relative infancy of the field and dearth of supporting literature specific to the ED PC practices. While a comprehensive PC assessment that addresses each of the eight domains of PC (as highlighted by the National Quality Institute and others) may be optimal in theory, in practice a comprehensive PC assessment is often incongruent with the pace and constraints of the ED environment. Indeed, prior ED PC demonstration projects have been met with limited success partly because their aim was to provide comprehensive PC assessment, which is both time- and resource-intensive. In this study, we attempted to distill the eight content domains of PC into only those domains that could be feasibly and meaningfully assessed in the ED, all while preserving the content validity of the tool. Therefore, we did not include spiritual care or ethical and legal aspects of care. While this poses a potential limitation to the content validity of our tool, we attempted to mitigate this by incorporating caregiver distress, patient

832

George et al. • EM PALLIATIVE CARE SCREENING TOOL

Item not reaching critical approval Section 1

Item as it originally appeared

Aggregated feedback of round 1 panelists •

Advanced Dementia: Advanced Dementia

CNS

The patient requires assistance with more self-care

Advanced CNS Disease (e.g. stroke, ALS, Parkinson’s):

•

Increase specificity (cite FAST score) Reduce length (combine Dementia and CNS into one item)

•

See above

• •

eGFR < 30mL/min too high Use creatinine levels, which are more available to ED providers

Modified item

Advanced Dementia or CNS Disease (e.g. history of stroke, ALS, Parkinson’s): Assistance needed for most self-care (e.g. ambulation, toileting) and/or minimally verbal. See above

The patient requires assistance with most self-care End Stage Renal Disease: End Stage Renal Disease

eGFR 6

High Chance of Accelerated Death: Hip fracture or major trauma in patients over 65, advanced AIDS, other

Provider Discretion – High Chance of Accelerated Death: Examples: Hip fracture > age 80; Major trauma in the elderly (rip fractures, intracranial bleed); etc

Section 2

> 1 ED Visit

Complex Care

Caregiver Distress

Greater than 1 hospital admission or ED visit in that past 6 months for the same condition

• •

1 visit not enough Should not specify “for the same condition.” Patients present with different complaints

2 or more ED visits or hospital admissions in the past 6 months

New or worsened complex care requirements

• • •

Difficult to interpret Not strong enough by itself Reduce Length

Caregiver and Patient Distress:

Caregiver is concerned about ability to meet care requirements

•

See above

e.g. Caregiver cannot meet long-term care needs; uncertainty/distress about prognosis or goals-of-care See above

Figure 5. Chart of Delphi modifications to screening tool.

distress, and goals-of-care points into the tool. Notably, none our expert panelists recommended the addition of these domains. Use of the modified Delphi technique itself can lead to a possible reduction in content validity, as anonymity can lead to lack of accountability for responses.98 The size of the expert panel was based on recommended Delphi methodology and is similar in size to other

Delphi panels described in the literature.65 However, while a smaller panel is more advantageous in terms of discussion and consensus, the results may be limited by skewed sampling of experts.99 An attempt was made to diminish this effect by inviting a diverse group of expert panelists to participate, in terms of years in practice, education level, sex, practice type and setting, and geographic location.

ACADEMIC EMERGENCY MEDICINE • July 2015, Vol. 22, No. 7 • www.aemj.org

833

Figure 6. Palliative care screening tool.

This study only included two rounds. While this may have limited true consensus, the approval level was high after the second round, and the commenters provided few opportunities for substantial change to the items or format. Therefore, it was determined that participants should not be invited to review the items in a third round. Furthermore, conducting more rounds puts the process at risk of attrition of respondents, which is a competing threat to validity. Our goal for this study was to develop a content-valid screening tool based on current evidence and validated by experts. Content validity is merely one aspect of validity, and future investigation into the accuracy, reliability,

and acceptability of the screening tool must be conducted to further validate this tool. Based on initial feedback from the Delphi process, we recommend requiring one or more “life-limiting illness” and two or more “PC needs” as the minimum threshold to qualify as a positive screening. Finally, while there is no criterion standard for which patients require PC consults in any setting, we plan to use “agreement by PC consultant” (in combination with outcome measures) as a reference standard for future studies of sensitivity, specificity, and predictive validity of this screening tool. A pilot of the tool, within a clinical setting, is planned to study the feasibility of implementation and efficacy of the tool in practice.

834

CONCLUSIONS A modified Delphi technique resulted in a content-validated, three-step screening tool for identification of ED patients with significant unmet palliative care needs. The screening tool is unique in that is designed for the ED environment and is the only screening tool content validated by palliative care experts. Subsequent investigation into ED palliative care screening with this tool is well supported by the findings of this study. The authors acknowledge Chad Van Ginkel and Mary Martha Gnatt.

References 1. Xu KT, Nelson BK, Berk S. The changing profile of patients who used emergency department services in the United States: 1996 to 2005. Ann Emerg Med 2009;54:805–10. 2. SUPPORT Principal Investigators. A controlled trial to improve care for seriously ill hospitalized patients. The study to understand prognoses and preferences for outcomes and risks of treatments (SUPPORT). JAMA 1995;274:1591–8. 3. Nelson JE, Meier DE, Oei EJ, et al. Self-reported symptom experience of critically ill cancer patients receiving intensive care. Crit Care Med 2001;29:277–82. 4. World Health Organization. WHO Definition of Palliative Care. Available at: www.who.int/cancer/ palliative/definition/en/. Accessed April 24, 2015 5. Lamba S. Early goal-directed palliative therapy (EGPT): a step to move palliative care upstream. J Palliat Med 2009;12:767. 6. Temel JS, Greer JA, Muzikansky A, et al. Early palliative care for patients with metastatic non-smallcell lung cancer. N Engl J Med 2010;363:733–42. 7. Bakitas M, Lyons KD, Hegel MT, et al. Effects of a palliative care intervention on clinical outcomes in patients with advanced cancer: the Project ENABLE II randomized controlled trial. JAMA 2009;302:741–9. 8. Pantilat SZ, O’Riordan DL. Hospital-based palliative medicine consultation: a randomized controlled trial. Arch Intern Med 2010;170:2038–40. 9. Singer PA, Martin DK, Kelner M. Quality end-of-life care: patients’ perspectives. JAMA 1999;281:163–8. 10. Lynn J. Learning to care for people with chronic illness facing the end of life. JAMA 2000;284:2508–11. 11. Lynn J, Wilkinson AM. Quality end of life care: the case for a MediCaring demonstration. Hosp J 1998;13:151–63. 12. Cowan JD. Hospital charges for a community inpatient palliative care program. Am J Hosp Palliat Care 2004;21:177–90. 13. Ciemins EL, Blum L, Nunley M, Lasher A, Newman JM. The economic and clinical impact of an inpatient palliative care service: a multifaceted approach. J Palliat Med 2007;10:1347–55. 14. Axelsson B, Christensen SB. Evaluation of a hospital-based palliative support service with particular regard to financial outcome measures. Palliat Med 1998;12:41–9.

George et al. • EM PALLIATIVE CARE SCREENING TOOL

15. O’Mahony S, McHenry J, Blank AE, et al. Preliminary report of the integration of a palliative care team into an intensive care unit. J Palliat Med 2010;24:154–65. 16. Norton SA, Hogan LA, Holloway RG, TemkinGreener H, Buckley MJ, Quill TE. Proactive palliative care in the medical intensive care unit: effects on length of stay for selected high-risk patients. Crit Care Med 2007;35:1530–5. 17. Penrod JD, Deb P, Luhrs C, et al. Cost and utilization outcomes of patients receiving hospital-based palliative care consultation. J Palliat Med 2006;9:855–60. 18. O’Mahony S, Blank AE, Zallman L, Selwyn PA. The benefits of a hospital-based inpatient palliative care consultation service: preliminary outcome data. J Palliat Med 2005;8:1033–9. 19. Brody EC, Newman J, Harrington C. The effects of an inpatient palliative care team on discharge disposition. J Palliat Med 2010;13:541–8. 20. Meier DE, Morrison RS, Cassel CK. Improving palliative care. Ann Intern Med 1997;127:225–30. 21. Coventry PA, Grande GE, Richards DA, Todd CJ. Prediction of appropriate timing of palliative care for older adults with non-malignant life-threatening disease: a systematic review. Age Ageing 2005;34:218–27. 22. Murray SA, Boyd K, Sheikh A. Palliative care in chronic illness. Br Med J 2005;330:611–2. 23. Murray SA, Sheikh A. Palliative care beyond cancer: care for all at the end of life. Br Med J 2008;336:958–9. 24. Formiga F, Chivite D, Ortega C, Casas S, Ramon JM, Pujol R. End-of-life preferences in elderly patients admitted for heart failure. Quart J Med 2004;97:803–8. 25. Goodlin SJ, Hauptman PJ, Arnold R, et al. Consensus statement: palliative and supportive care in advanced heart failure. J Card Fail 2004;10:200–9. 26. Snow CE, Varela BR, Pardi DA, Adelman RD, Said S, Reid MC. Identifying factors affecting utilization of an inpatient palliative care service: a physician survey. J Palliat Med 2009;12:231–7. 27. Antonio CP, Barretto ACP, Carlos H, et al. Re-hospitalizations and mortality still at alarming rates in advanced heart failure. J Cardiac Fail 2007;13(6 Suppl 2):S153. 28. Walshe C. Implementation and impact of the gold standards framework in community palliative care: a qualitative study of three primary care trusts. Palliat Med 2008;22:736–43. 29. Wilper AP, Woolhandler S, Lasser KE, et al. Waits to see an emergency department physician: U.S. trends and predictors, 1997-2004. Health Aff (Millwood) 2008;27:84–95. 30. Bailey C, Murphy R, Porock D. Trajectories of end-of-life care in the emergency department. Ann Emerg Med 2011;57:362–9. 31. Lynn J. Sick to Death. Berkeley, CA: University California Press, 2004. 32. Smith AK, McCarthy E, Weber E, et al. Half of older Americans seen in emergency department in last month of life; most admitted to hospital, and

ACADEMIC EMERGENCY MEDICINE • July 2015, Vol. 22, No. 7 • www.aemj.org

33.

34.

35.

36.

37.

38.

39.

40.

41.

42.

43.

44.

45.

46. 47.

48.

many die there. Health Aff (Millwood) 2012;31:1277–85. Gardiner C, Gott M, Ingleton C, et al. Extent of palliative care needs in the acute hospital setting: a survey of two acute hospitals in the UK. Palliat Med 2013;27:76–83. Beynon T. How common are palliative care needs among older people who die in the emergency department? Emerg Med J 2011;28:491–5. Rosenberg M, Lamba S, Misra S. Palliative medicine and geriatric emergency care: challenges, opportunities, and basic principles Clin Geriatr Med 2013;29:1–29. U.S. Department of Health and Human Services, Health Resources and Services Administration. Shortage Designation: Health Professional Shortage Areas & Medically Underserved Areas/Populations. Available at: http://www.hrsa.gov/ shortage/index.html. Accessed April 24, 2015 Bernstein SL, Haukoos JS. Public health, prevention, and emergency medicine: a critical juxtaposition. Acad Emerg Med 2008;15:190–3. Lukin W, Doulgas C, O’Conor A. Palliative care in the emergency department: an oxymoron or just good medicine? Emerg Med Australas 2012;24:102–4. Osta BE, Palmer JL, Paraskevopoulos T, et al. Interval between first palliative care consult and death in patients diagnosed with advanced cancer at a comprehensive cancer center. J Palliat Med 2008;11:51–7. Brumley R, Enguidanos S, Jamison P, et al. Increased satisfaction with care and lower costs: results of a randomized trial of in-home palliative care. J Am Geriatr Soc 2007;55:993–1000. Riley GF, Lubitz JD. Long-term trends in Medicare payments in the last year of life. Health Serv Res. 2010;45:565–76. Teno JM, Fisher E, Hamel MB, Coppola K, Dawson N. Medical care inconsistent with patients’ treatment goals: association with 1-year Medicare resource use and survival. J Am Geriatr Soc 2002;50:496–500. Weissman DE, Block SD. ACGME requirements for end-of-life training in selected residency and fellowship programs: a status report. Acad Med 2002;77:299–304. Crawley L, Payne R, Bolden J, Payne T, Washington P, Williams S. Palliative and end-of-life care in the African American community. JAMA 2000;284:2518–21. Foley KM, Gelband H. Improving Palliative Care for Cancer. Washington DC: National Academies Press, 2001. Freeman HP, Payne R. Racial injustice in health care. N Engl J Med 2000;342:1045–47. Richardson LD, Ragin DF, Hwang U, et al. Emergency medicine patients’ access to healthcare (EMPATH) study: racial/ethnic, gender and age related differences in emergency department use [abstract]. Acad Emerg Med 2003;10:524. Grumbach K. Primary care and public emergency department overcrowding. Am J Public Health 1993;83:372–8.

835

49. Guttman N, Zimmerman DR, Nelson MS. The many faces of access: reasons for medically nonurgent emergency department visits. J Health Polit Policy Law 2003;28:1089–120. 50. Rogers D. The increasing geriatric population and overcrowding in the emergency department: one hospital’s approach. J Emerg Nurs 2009;35:447–50. 51. Smith AK, Fisher J, Schonberg MA, et al. Am I doing the right thing? Provider perspectives on improving palliative care in the emergency department. Ann Emerg Med 2009;54:86–93. 52. Lamba S, Nagurka R, Zielinski A, Scott SR. Palliative care provision in the emergency department: barriers reported by emergency physicians. J Palliat Med 2013;16:143–7. 53. Meier DE, Beresford L. Fast response is key to partnering with the emergency department. J Palliat Med 2007;10:641–5. 54. Stone SC, Mohanty S, Grudzen CR, et al. Emergency medicine physicians’ perspectives of providing palliative care in an emergency department. J Palliat Med 2011;14:1333–8. 55. Chan GK. End-of-life models and emergency department care. Acad Emerg Med 2004;11:79–86. 56. National Institutes of Health. State-of-the-Science Conference: Statement on Improving End-of-Life Care. Available at: http://consensus.nih.gov/2004/ 2004EndOfLifeCareSOS024html.htm. Accessed April 24, 2015. 57. Meier DE. Increased access to palliative care and hospice services: opportunities to improve value in health care. Milbank Q 2011;89:343–80. 58. Quest TE, Asplin BR, Cairns CB, Hwang U, Pines JM. Research priorities for palliative and end-oflife care in the emergency setting. Acad Emerg Med 2011;18:70–6. 59. American College of Emergency Physicians. ACEP Announces List of Tests as Part of Choosing Wisely Campaign. Available at: http://www.acep.org/Clinical—Practice-Management/ACEP-Announces-Listof-Tests-As-Part-of-Choosing-Wisely-Campaign/. Accessed April 24, 2015. 60. Anastasi A, Urbina S. Psychological Testing, 7th ed. New York, NY: Prentice Hall, 1997. 61. Rungtusanatham M. Let’s not overlook content validity. Decision Line 1998;29:10–13. 62. Haynes SN, Richard DC, Kubany ES. Content validity in psychological assessment: a functional approach to concepts and method. Psychol Assess 1995;7:238–47. 63. Allen MJ, Yen WM. Introduction to Measurement Theory, 2nd ed. Prospect Heights, IL: Waveland Press, 2002. 64. Powell C. The Delphi technique: myths and realities. J Adv Nurs 2003;41:376–82. 65. Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One 2011;6:e20476. 66. Dillman S. Mail and Internet Surveys: The Tailored Design Method. New York, NY: Wiley, 2000. 67. Lawshe CH. A quantitative approach to content validity. Pers Psychol 1975;28:563–75.

836

68. Wilson RF, Pan W, Schumsky DA. Recalculation of the critical values for Lawshe’s content validity ratio. Meas Eval Couns Dev 2012;45:197–210. 69. Glajchen M, Lawson R, Homel P, DeSandre P, Todd K. A rapid two-stage screening protocol for palliative care in the emergency department: a quality improvement initiative. J Pain Symptom Manage 2011;42:657–62. 70. Mion LC, Palmer RM, Anetzberger G, Meldon S. Establishing a case finding and referral system for at-risk older individuals in the emergency department setting: the SIGNET model. J Am Geriatr Soc 2001;49:1379–86. 71. Mahony SO, Blank A, Simpson J, et al. Preliminary report of a palliative care and case management project in an emergency department for chronically ill elderly patients. J Urban Health 2008;85:443–51. 72. Richards CT, Gisondi MA, Chang CH, et al. Palliative care symptom assessment for patients with cancer in the emergency department: validation of the screen for palliative and end-of-life care needs in the Emergency Department instrument. Palliat Med 2011;14:757–64. 73. Wu FM, Newman JM, Lasher A, Brody AA. Effects of initiating palliative care consultation in the emergency department on inpatient length of stay. J Palliat Med 2013;16:1362–7. 74. National Quality Forum. A National Framework and Preferred Practices for Palliative and Hospice Care Quality: A Consensus Report. Available at: http://www.qualityforum.org/Publications/2006/12/ A_National_Framework_and_Preferred_Practices_ for_Palliative_and_Hospice_Care_Quality.aspx. Accessed April 24, 2015. 75. Center to Advance Palliative Care. Palliative Care ED Screening Tool: A Technical Assistance Resource from the IPAL-EM Project2011. Available at: http://www.google.com/url?sa=t&rct=j&q= &esrc=s&source=web&cd=2&ved=0CCoQFjAB& url=http%3A%2F%2Fsummaem.org%2Fpublic% 2Fgeriatricem%2Fresources%2FShared%2520Documents%2FPalliative%2520Care%2520ED%2520 Screening%2520Tool.docx&ei=Y-Y6VbPvIe-1sASEtIGgBQ&usg=AFQjCNGNKxl9WlUKuby16AmPx 5LCdtUFMg&bvm=bv.91665533,d.cWc&cad=rja. Accessed April 24, 2015. 76. Weissman DE, Meier DE. Identifying patients in need of a palliative care assessment in the hospital setting: a consensus report from the Center to Advance Palliative Care. J Palliat Med 2011;14:17–23. 77. Mosenthal AC, Weismann DE, Curtis JR, et al. Integrating palliative care in the surgical and trauma intensive care unit: a report from the Improving Palliative Care in the Intensive Care Unit (IPAL-ICU) Project Advisory Board and the Center to Advance Palliative Care. Crit Care Med 2012;40:1199–206. 78. Grudzen CR, Richardson LD, Hopper SS, Ortiz JM, Whang C, Morrison RS. Does palliative care have a future in the emergency department? Discussions with attending emergency physicians. J Pain Sympt Manage 2012;43:1–9.

George et al. • EM PALLIATIVE CARE SCREENING TOOL

79. Center to Advance Palliative Care. Growth of Palliative Care in U.S. Hospitals, 2013 Snapshot. Available at: http://www.hpcanys.org/palliativecarecenter/files/ Growth%20of%20Palliative%20Care%20in%20US %20Hospitals%20-%20A%202013%20Snapshot. pdf. Accessed April 24, 2015 80. Berger G, O’Riordan D, Kerr K, Pantilat S. Prevalence and characteristics of outpatient palliative care services in California. Arch Intern Med 201;171:2057–9. 81. Ahronheim JC, Morrison RS, Morris J, Baskin S, Meier DE. Palliative care in advanced dementia: a randomized controlled trial and descriptive analysis. J Palliat Med 2000;3:265–73. 82. Smith TJ, Temin S, Alesi ER, Abernethy AP, Balboni TA, Basch EM. American Society of Clinical Oncology provisional clinical opinion: the integration of palliative care into standard oncology care. J Clin Oncol 2012;30:880–7. 83. Hussain J, Adams D, Campbell C. End-of-life care in neurodegenerative conditions: outcomes of a specialist palliative neurology service. Int J Palliat Nurs 2013;19:162–9. 84. Foley KM, Carver AC. Palliative care in neurology. Neurol Clin 2001;19:789–99. 85. Hussain JA, Russon L. Supportive and palliative care for people with end-stage renal disease. Br J Hosp Med 2012;73:640–4. 86. Cohen LM, Moss AH, Weisbord SD, Germain MJ. Renal palliative care. J Palliat Med 2006;9:977– 92. 87. Poonja Z, Brisebois A, van Zanten S, Tandon P, Meeberg G, Karvellas CJ. Patients with cirrhosis and denied liver transplants rarely receive adequate palliative care or appropriate management. Clin Gastroenterol Hepatol 2014;4:692–8. 88. Boyd K, Kimbell B, Murray S, Iredale J. Living and dying well with end-stage liver disease: time for palliative care? Hepatology 2012;55:1650–1. 89. Howlett JG. Palliative care in heart failure: addressing the largest care gap. Curr Opin Cardiol 2011;26:144–8. 90. LeMond L, Allen LA. Palliative care and hospice in advanced heart failure. Prog Cardiovasc Dis 2008;54:168–78. 91. Cao Z, Ong KC, Eng P, Tan WC, Ng TP. Frequent hospital readmissions for acute exacerbation of COPD and their associated factors. Respirology 2006;11:188–95. 92. Stuart B, The NHO Medical Guidelines for NonCancer Disease and Local Medical Review Policy. Hospice access for patients with diseases other than cancer. Hosp J 1999;14:139–54. 93. Gade G, Venohr I, Conner D, et al. Impact of an inpatient palliative care team: a randomized control trial. J Palliat Med 2008;11:180–90. 94. McCusker M, Ceronsky L, Crone C, et al. Institute for Clinical Systems Improvement. Palliative Care for Adults. Available at: https://www.icsi.org/_asset/ k056ab/PalliativeCare.pdf. Accessed April 24, 2015 95. Chen JC, Goetz MB, Feld JE, et al. A provider participatory implementation model for HIV testing in an ED. Am J Emerg Med 2011;29:418–26.

ACADEMIC EMERGENCY MEDICINE • July 2015, Vol. 22, No. 7 • www.aemj.org

96. Portenoy RK, Thaler HT, Kornblith AB, Lepore JM. The Memorial Symptom Assessment Scale: an instrument for the evaluation of symptom prevalence, characteristics and distress. Eur J Cancer 1994;30:1326–36. 97. Emaniel L, Alpert HR, Emanuel E. Concise screening questions for clinical assessment of terminal care: the needs near end of life screening tool. J Palliat Med 2001;4:465–74. 98. Udris EM, Fihn SD, McDonell MB, Curtis JR. Differences in health care utilization at the end of life

837

among patients with chronic obstructive pulmonary disease and patients with lung cancer. Arch Intern Med 2006;166:326–31. 99. Penciner R, Langhan T, Lee R, Mcewen J, Woods RA, Bandiera G. Using a Delphi process to establish consensus on emergency medicine clerkship competencies. Med Teach 2011;33:333–9. 100. Balasubramanian R, Agarwal D. Delphi technique – a review. Int J Public Health Dent 2013; 3:16–25.