Qual Life Res DOI 10.1007/s11136-014-0780-y

PATIENT ENGAGEMENT SPECIAL SECTION

Data collection challenges in community settings: insights from two field studies of patients with chronic disease Richard J. Holden • Amanda M. McDougald Scott Peter L. T. Hoonakker • Ann S. Hundt • Pascale Carayon

•

Accepted: 8 August 2014  Springer International Publishing Switzerland 2014

Abstract Purpose Collecting information about health and disease directly from patients can be fruitfully accomplished using contextual approaches, ones that combine more and less structured methods in home and community settings. This paper’s purpose was to describe and illustrate a framework of the challenges of contextual data collection. Methods A framework is presented based on prior work in community-based participatory research and organizational science, comprised of ten types of challenges across four broader categories. Illustrations of challenges and suggestions for addressing them are drawn from two mixed method, contextual studies of patients with chronic disease in two regions of the USA. Results The first major category of challenges was concerned with the researcher–participant partnership, for example, the initial lack of mutual trust and understanding between researchers, patients, and family members. The second category concerned patient characteristics such as R. J. Holden (&) Department of BioHealth Informatics, Indiana University School of Informatics and Computing, Indiana University-Purdue University Indianapolis, Walker Plaza (WK), 719 Indiana Avenue, WK 319, Indianapolis, IN 46202, USA e-mail: [email protected] A. M. McDougald Scott Institute on Family and Neighborhood Life, Clemson University, Clemson, SC, USA P. L. T. Hoonakker
A. S. Hundt
P. Carayon Center for Quality and Productivity Improvement, University of Wisconsin-Madison, Madison, WI, USA P. Carayon Department of Industrial and Systems Engineering, University of Wisconsin-Madison, Madison, WI, USA

cognitive limitations and a busy personal schedule that created barriers to successful data collection. The third concerned research logistics and procedures such as recruitment, travel distances, and compensation. The fourth concerned scientific quality and interpretation, including issues of validity, reliability, and combining data from multiple sources. The two illustrative studies faced both common and diverse research challenges and used many different strategies to address them. Conclusion Collecting less structured data from patients and others in the community is potentially very productive but requires the anticipation, avoidance, or negotiation of various challenges. Future work is necessary to better understand these challenges across different methods and settings, as well as to test and identify strategies to address them. Keywords Community-based participatory research
Field study
Methodology
Mixed methods
Chronic illness
Patient engagement

Introduction Collecting information about health and disease directly from patients is believed to uncover unique and otherwise unmeasured insights into patients’ experiences [1, 2]. Patient-reported data collection spans continua of structure and setting (Fig. 1). For example, near one extreme lies the standardized, numeric pain scale intended for administration in clinic to a single patient to produce a single value [3]. Near the other extreme is a longitudinal series of openended in-home interviews with a whole family, producing an intricate, joint (sometimes disjoint!) narrative about ‘‘how we managed dad’s pain [4, 5].’’ We use the term

123

Qual Life Res

Fig. 1 Data collection with patients spans continua of structure and setting. Data collection approaches tending toward the upper-right are called ‘‘contextual’’ and involve at least some less structured and qualitative methods carried out in home and community settings. Each dot represents one example among many possible designs and methods

contextual to refer to data collection approaches bearing a closer resemblance to the latter example than the former. In other words, contextual data collection tends to occur in patients’ homes and communities and to include less structured methods. The purpose of this paper was to describe the practical challenges of a contextual approach to patient-reported data collection. For each challenge, we present illustrations from contextual research that we have conducted with vulnerable and typically older adults with chronic diseases and their caregivers. The benefits and challenges of contextual data collection Common phenomena assessed with patient-reported data include quality of life, service satisfaction, impairment, and mood [6, 7]. Patients can also self-report a host of other health-related perceptions, abilities, limitations, routines, and events such as hospitalizations or experienced breakdowns in healthcare delivery [8, 9]. All of these phenomena occur in context, and many cannot be separated from that context. More concretely, consider the construct of barriers to self-care in patients with chronic disease [10, 11]. Such barriers are dynamic: occurring and changing over time. They span several levels and domains, from personal and biological (e.g., age and disease progression), to household and social (e.g., caregiver situation and social isolation), to healthcare system and economic (e.g., lack of insurance or access to care) [12, 13]. Some barriers are meaningful only in light of personal values and beliefs,

123

such as patients who relinquish control of their health to family members, clinicians, or deities [14]. Other barriers are found in the local environment and not always known or easily articulated by patients, including air pollutants, social influence, or living in a car-dependent locale or food desert [15]. Thus, while self-care barriers can be measured with a standardized self-report instrument (e.g., [16]), many times they are also assessed with semi-structured interviews and observations, many of which take place in patients’ homes and communities, involve multiple informants per household, and are spread over several encounters [17]. Generally speaking, if a phenomenon is context-dependent, then contextual research is useful and necessary. Accordingly, several communities of practice in health and healthcare are turning to contextual approaches. Scholars in public health and health services research are increasingly engaging in community-based participatory research (CBPR) [18–22], an approach that ‘‘equitably involves all partners with a research topic of importance to the community with the aim of combining knowledge and action for social change to improve community health and eliminate health disparities [23–25].’’ Health psychologists and sociologists rely on focus groups, personal interviews, and observations in the community to inform and test theories of health behavior [26, 27]. Engineers and social scientists alike who adopt a systems approach to understand health find they must enter the system in order to learn about it [28, 29], just as those in pursuit of patient engagement find they must engage with patients. Even the psychometrician whose ultimate goal is instrument development benefits from a contextual understanding of the domain to achieve content validity [30]. Furthermore, when patient-reported data are collected and implemented in clinical practice, it is important to understand the contextual factors that will shape their appropriate capture and use [31]. Many of these efforts employ mixed methods research, which is well situated to more fully capture complex phenomena in context [32–34]. However, the benefits of a contextual approach come at the cost of actually doing it! Contextual approaches are attended by several organizational and methodological challenges. For example, Green and Thorogood [26] describe some of the ‘‘practical issues’’ in contextual health-related research, including getting access to eligible participants, managing privacy and intrusiveness issues in patients’ homes, providing incentives and childcare to participants, and obtaining accurate responses to sensitive topics. In order to support researchers and others in collecting patient-reported data in context, the remainder of the paper presents and illustrates a framework of the types of challenges that can be encountered in a contextual approach. Illustrations of the framework draw on the authors’

Qual Life Res

experiences conducting two research studies with similar goals and patient populations: the Caring Hearts study and the Keystone Beacon Community (KBC) Project.

Methods Here, we describe, first, how we derived and refined a framework of challenges associated with contextual data collection and, second, the authors’ two studies that provided illustrations of the framework. Framework development The framework was iteratively developed, beginning with a preliminary framework combining: •

•

A review of the challenges encountered over several decades of community-based participatory research (CBPR) [18–20], and Models from the field of systems engineering describing the practicalities of conducting organizational field research [35, 36].

The preliminary framework was then reduced to eliminate challenges not pertinent to contextual data collection

(e.g., those strictly related to interventions). Experiences from our two studies, described below, were then listed and categorized according to the framework. Experiences were identified as follows: (a) during data collection, using researcher field memos, written notes in recruitment documents, and e-mails between researchers and (b) after data collection, using document and transcript review, group discussion sessions to stimulate recall, and interviews with study personnel. As all authors were directly involved in data collection, they were able to identify a very large number of data collection challenges based on their memories and field notes. Lastly, the framework was refined so that categories better matched the nature of our experiences and to aggregate smaller categories. All of these activities were performed by each author separately, discussed face to face among members of each study, and then reviewed and revised by all authors together via conference calls and e-mail. The final framework is presented in the Results. Study descriptions Illustration and refinement of the framework were based on two studies, Caring Hearts and KBC. Both were mixed methods field studies of community-dwelling patients with

Table 1 Descriptions of two community-based studies of patients with chronic disease

Study objective

Caring Hearts Study

Keystone Beacon Community (KBC) Project

Understand the person, task, technology, and context factors in chronic disease and chronic heart failure (CHF) self-care.

Understand the patient’s perspectives on case management.

Ultimate goal to develop technology to support CHF self-care. Design

Participant demographics

Mixed methods, longitudinal field study of elderly patients with CHF and their informal caregivers.

Ultimate goal to reduce (re)admissions and emergency department visits by patients with CHF and COPD. Initially case study, but progressively action research, of hospital-discharged patients with CHF or COPD enrolled in case management

Mean age = 73, SD = 6.67

Survey respondents aged 45–85? (*M = 75 years old).

29 % non-White, 55 % male

Due to HIPAA regulations, no demographics collected on focus group and interview participants.

16 % \12 years education; 50 % household income \$35,000 *50 % B90 days from hospital discharge

Setting

Urban and rural Mid-south US counties within 200 mile radius of a large academic medical center.

Five counties in central Pennsylvania (US) within 60 mile radius of one of four hospitals.

Methods

Standardized self-administered paper survey (n = 59, 95 % response rate)

Standardized self-administered paper survey (n = 160, 40 % response rate)

Short interviews (30 min, n = 46)

4 focus groups (60 min, n = 9)

Extended interviews (90 min, n = 46)

Interviews (30–60 min, n = 10)

Outpatient clinic visit observations, audio recorded (n = 41) Research team Funding and IRB approval

Medical record review (n = 63) Principal investigator: Human factors engineer/psychologist

Principal investigator: Human factors engineer

Team with sociology, psychology, and nursing backgrounds

Team with engineering, sociology and psychology backgrounds

National Institutes of Health

The study was funded by the Office of The National Coordinator (ONC) for Health Information Technology

Approved by University IRB

Approved by University IRB and IRB of the hospital system

CHF chronic (congestive) heart failure, COPD chronic obstructive pulmonary disease; IRB institutional review board

123

Qual Life Res

chronic disease. Despite being carried out by separate research teams, both were based on applying a systems approach [29, 37, 38] to understand the structures, processes, and outcomes related to management of chronic disease. Table 1 reports the studies’ characteristics. Notable differences were as follows: geographic location, included disease groups, primary study focus, and study design. The studies were similar in their use of mixed methods and focus on individuals vulnerable due to age, minority status, multiple diseases and disability, low income, lacking health insurance, low literacy, lack of resources, and often some combination of these. Both studies included home visits in settings that varied from government housing to high-end homes in urban settings.

The two studies used somewhat different recruitment and data collection procedures (Table 2). In particular, Caring Hearts recruited patients directly based on medical record screening, used audio and video recording, and collected data in both homes and clinics. KBC relied on a clerical staff outside the study team for recruitment, did not use video, and provided a smaller total amount of compensation.

Results Both studies provided rich qualitative and quantitative data from patients and their caregivers. However, during data collection, we encountered several challenges that may

Table 2 Recruitment and data collection procedures in the two studies

Participant pool

Caring Hearts Study

Keystone Beacon Community (KBC) Project

Eligible patients scheduled for outpatient cardiology clinic visit (N = 31)

Eligible patients who had received case management. The survey had an intervention and control group. The control group consisted of eligible patients who had been discharged from a hospital not participating in the project (and therefore did not receive KBC case management)

Eligible patients enrolled in another study who agreed to contact from other studies (N = 32) Recruitment approach

Research team screens medical record for eligibility. Potential participants contacted by phone to explain study; recruitment proceeds if interested; appointment scheduled. Recruited individuals briefed and consented at first appointment.

Interviews and focus groups: based on inclusion criteria, clerical staff called the patients at home and asked whether the patient was willing to participate in the study Survey: all eligible patients who had been discharged from the hospital in the past week received a (paper & pencil) survey, with stamped return envelope. Patients were sent three reminders

Multiple attempts made to reach individuals for recruitment and in case of no-show. Reminder phone calls prior to appointment date; mailed reminders by request. Privacy and confidentiality issues

Written consent forms. Separate video consent and release form with option to have face blurred. Medical record data kept separate from personal identifiers.

Interviews and focus groups: clerical staff at the health insurance organization recruited the patients. Researchers only received an alias, and (if necessary for home interview) an address. Surveys were completed anonymously

Data collection settings

Outpatient clinic or office

Focus groups: in hospital and clinic meeting rooms

Patient’s or family member’s home.

Interviews: at home and via phone

Targets of data collected

Health: general and cardiovascular

Health

Health behavior (self-care, adherence)

Health behavior (self-care)

Personal routines, habits

Perceptions of case management

Living situation, home and community environment

Perceptions of hospital stay

Barriers to self-care

Demographics (survey only)

Survey: in home

Self-care resources, strategies Participant incentives Data capture

Demographics Up to $65US mailed check. Gift card in lieu of check, upon request. Audio recording, video recording if permitted, and written notes

Focus group and interview participants received a $25 gift card; survey respondents did not receive an incentive. Observation (field) and debriefing notes Audio recordings Paper-and-pencil surveys

123

Qual Life Res

have had an effect on the quality of the data and therefore on the ultimate usefulness of our research. Using a combination of our shared experiences and prior models of field research implementation [19, 35, 36], Fig. 2 presents a framework of ten categories of challenges across four general types: those concerning (1) the researcher–participant partnership, (2) participant characteristics, (3) research logistics and procedures, and (4) scientific quality and interpretation. These challenges are described with respect to the two studies in Tables 3, 4, 5, and 6 and the text below. Challenges concerning researcher–participant partnership (Table 3) Researcher–participant differences in priorities In our studies, researchers prioritized ethical conduct and collection of accurate, reliable, and on-topic data in a concise and objective way. Participants’ priorities for the research encounter included these and other goals such as friendship and a chance to socialize, payment, or an opportunity to speak on a topic of importance to them. A few enrolled in our studies were seeking extra healthcare services or another way to lodge complaints about the health system. Many were not used to the more contextual health-related research and several did not understand why anyone would want to ask them questions, come to their home, or video-record the encounter. Some responded with

confusion or mistrust, others with amusement. The frequent participant questions, ‘‘How did I do?’’ and ‘‘Was this at all useful?’’ were signs of participants’ potential uncertainty about researchers’ priorities. Mistrust and misunderstanding of research and researchers Some participants questioned the benefits and purpose of participation. At home visits, participants sought researcher identification and university affiliation. Some were wary of videotaping in their homes, either from discomfort or not understanding its purpose, including worry that robbers could use the video. Several felt researchers were fishing for a right or wrong answer, especially around socially desirable topics (e.g., adherence). In some cases, there was a perception that researchers were part of the clinical care team. For some, this perception appeared to elicit ‘‘sugarcoated’’ descriptions of health behavior. For others, it prompted requests for medical advice and other information. Among other requests for assistance, one woman requested that researchers arrange a lecture circuit for her to speak about the ills of smoking. Differences in language, perspective, and personal norms Several participants had difficulty reading or understanding standardized and ad hoc probing questions. Sometimes, it

Fig. 2 Framework of challenges associated with contextual data collection (adapted from [19, 35, 36])

123

Qual Life Res Table 3 Examples of data collection challenges concerning researcher–participant partnership Researcher–participant differences in priorities Participant voices ignorance of research study or research in general Participant unsure whether participation was helpful Participant discontinues participation after first encounter, stating they have nothing more to contribute Participant refuses aspects of the study (e.g., videotaping and home visit) when not expecting them Patient desires someone else (family member and caregiver) to participate on their behalf Researcher’s value for expedient, on-point data results in participants’ interests being dismissed Participant’s value for conversation and own goals results in time inefficiencies for researcher Mistrust and misunderstanding of research and researchers Mistrust for researchers’ intention, perceptions of hidden motives Distrust of health system carries over into distrust of researchers, research study Participant wary that researcher will take advantage of them in return for inadequate benefits Questioning of need for certain data (address, income, social activities) Hesitancy to consent to videotaping of home out of concern for invasion of privacy and questions about how recordings will be used Researcher viewed as clinical expert, asked for medical advice Researcher perceived as talking with doctors, having control over care plan Participant asks for connections to university/medical center for personal advancement Unwillingness to reveal sensitive information (at least initially) or ‘‘sugarcoating’’ of situation Participant provides different answers to male vs. female researchers Differences in language, perspective, and personal norms Researcher asked to help to read research materials (e.g., survey and knowledge test), interpret medical jargon Participant does not understand question with medical content or language Words interpreted differently depending on participant background or education Participant home environment unwelcoming or insalubrious for researcher Participant worldview or beliefs in conflict with research questions or goals

was a language issue, as when participants interpreted selfcare ‘‘barriers’’ and ‘‘obstacles’’ to be physical barriers/ obstacles. As another example, ‘‘tricks and strategies’’ were problematic terms because participants did not label their actions as such. Several participants lacked basic terminology or knowledge for effectively communicating about health issues, including not knowing the name and nature of medical conditions and being unable to separate medical conditions and events (e.g., heart failure exacerbation from heart attack). Sometimes, researchers and participants operated on different assumptions. For example, some interview queries, perhaps incorrectly, attempted to separate specific diseases from the patient’s overall life experience. Personal choices were sometimes challenging for researchers to understand or accommodate. Some participants’ homes were unwelcoming or insalubrious: ‘‘keep out’’ signs, dogs, potentially psychotic or antisocial behavior, and odor. In two cases, the research team declined a home visit because such conditions risked safety and health. Another source of difficulty was worldview differences between participants and researchers. For example, participants’ narratives about mistrust in

123

traditional medicine or divine and mystical interventions made it challenging to ask about medical treatments and health behavior. Challenges concerning participant characteristics (Table 4) Participants’ competing life and health demands Patients in our studies sometimes had demands, needs, or schedules that interfered with participation. Difficulties scheduling in-person data collection arose from deaths of family and friends, having multiple clinical or research appointments, and work or volunteer obligations. One participant double-booked a home research interview with a physical therapy visit. Several had health-related disruptions or relocations that affected data collection. Transportation was another major issue. Some refused to participate because they would not impose on whoever provided their transportation or declined extra travel. Inclinic data collection was sometimes curtailed because participants were delayed in traffic or needed to leave earlier to get home earlier. For those who lived further

Qual Life Res Table 4 Examples of data collection challenges concerning participant characteristics Participants’ competing life and health demands Participant has multiple appointments or obligations, resulting in scheduling conflicts or difficulties Participant in multiple studies within institutions, resulting in confusion about which study is which, and feeling overburdened by research Participant cannot drive, lives far away, or has other transportation constraints Extenuating circumstances such as hospitalizations, holidays/travel, deaths in the family, or relocation (e.g., to rehabilitation facilities and assisted living) prevent scheduling data collection Medical interruptions during interviews (e.g., hypoglycemic event and frequent visits to restroom) Participant does not feel well enough to participate or is unable due to illness, hospitalization Participant does not own home where data collected, thus not comfortable with extensive interview or use of video recording Participants’ psychosocial, cognitive, and perceptual limitations Participant not talkative, inarticulate Disinterest in reflecting on illness or thinking deeply about unpleasant topics Forgetting of and confusion about scheduled data collection appointments Memory deficits about disease, therapy, or clinicians (even despite being shown photographs) Participant and researcher have difficulty understanding one another due to accents, rate of speech, hearing impairment Home interview complicated by lighting and noise conditions, causing difficulty communicating Fatigue and disengagement resulting in inattentiveness or misinformation No guided home tours because of fatigue or physical impairment

away, visits to the medical campus were, as one patient said, ‘‘all-day affairs.’’ This imposed challenges on researchers to find time for data collection without interfering with other appointments or meals, and at times resulted in rushed data collection. Other examples of health-related issues during data collection were patients’ physical impairments precluding guided observations of their house, shortness of breath or fatigue hindering conversation, and interview interruptions due to acute events. One participant was interviewed at a time of medical transition in which his routines were changing considerably, which resulted in many responses about the past and the future, as opposed to about the present. Another participant declined an in-home visit because of embarrassment over his frequent urination.

Participants’ psychosocial, cognitive, and perceptual limitations Individuals with mood or cognitive issues were sometimes difficult to engage or understand. Mood problems caused distress, inattentiveness, and perseveration on a narrow range of topics during data collection. Some with mood and motivation issues indefinitely put off or did not agree to follow-up visits. There were memory and confusion issues among participants in our studies, which was not surprising given their average age. Several could not recall diagnostic or therapeutic information. One woman with chronic heart failure (CHF), when asked about her low-sodium diet,

produced and read from an outdated sheet about soft foods to eat post-stroke. Such memory and comprehension deficits were exacerbated by these individuals having multiple medical conditions, clinicians, medications, and treatments. Some but not all patients used documents to help remember. Others relied on caregivers. For example, during the short interview, the aforementioned woman with CHF had her husband present to compensate for her cognitive deficits, but the husband did not attend the in-home follow-up. At the extreme end, one participant with sleep and concentration problems fell asleep at a home interview while her daughter was answering a question. Hearing problems and noisy environments (e.g., waiting rooms, noise from medical devices, and vaulted ceilings) made communication difficult. Some participants had a stroke or other impairment that made their speech incomprehensible, making it hard to understand and respond to their answers in real time. In one case, once the interview began, it quickly became clear that the wife was hard of hearing. This meant that we had to write our questions on paper, hand the paper to her, and wait for her to respond on paper with comments that included input by her husband. Challenges concerning research logistics and procedures (Table 5) Patient identification and recruitment One important challenge was identifying patients clinically and cognitively well enough to participate in interviews or

123

Qual Life Res Table 5 Examples of data collection challenges concerning research logistics and procedures Patient identification and recruitment Unclear how to define eligibility when patient and medical record provide conflicting or inadequate information (e.g., functional status and when first diagnosed with disease) Participants and researchers define inclusion criteria differently, e.g., what defines ‘‘heart failure’’ or cognitively intact? Questions about whether caregiver should be included, especially when patient and caregiver do not get along or have drastically different views of situation Gatekeeping by clinicians or recruitment personnel, resulting in potential selection bias Clinicians insist on being informed of research conducted with patients in their clinic and must buy-in and not be disrupted by the research Participant refusal to participate due to mistrust, lack of perceived benefit, or perceived burden Recruiting done directly by researchers requires special skills, major time demands Logistical issues Participant agrees to participate, but proves difficult to reach or schedule, sometimes rendering participant ineligible by the time appointment is scheduled Participant (or researcher) forgets or needs to reschedule appointment Participant too sick to participate or was admitted to the hospital, requiring rescheduling or rendering participant ineligible Participant lives or moves far away from research center, rendering them ineligible or increasing burden of data collection Difficulties with transportation to or navigation within research center Travel time for researchers to rural homes in various communities Participant transferred to another facility during study without informing researcher Participant died after initial interview or recruitment Researcher arrives at home but participant absent, forgot about appointment, changes their mind, or too busy to participate Data collection very time-consuming Maintaining participant privacy and confidentiality Challenges of complying with federal US (e.g., HIPAA) requirements Challenges of navigating local (e.g., institutional review board and clinic) requirements Aliases used but not always remembered by participant Lack of private area in clinic to interview participant Family members, friends, bystanders, or visitors present during interviews Conflicts with compensation Displeasure with delay between participation and compensation or form of payment Unwillingness to provide social security number for compensation record keeping Mailed check cannot be retrieved or cashed (e.g., if homebound or not using banking institution) Compensation too low for some, very high and potentially coercive for others Participant expects treatment, information, or help in return for participation

focus groups. In the KBC Project, initial eligibility criteria narrowed a large patient population to 15–41 eligible individuals, \20 % of the target sample. In both studies, relaxing inclusion constraints helped to enlarge and diversify the sample but created other challenges, described later. Different recruiting strategies in the two studies (see Table 2) produced unique challenges. Caring Hearts researchers used electronic health record data to screen and directly contact eligible individuals. This direct recruitment strategy replaced an earlier approach of contacting busy clinicians to suggest potential participants, a process that imposed considerable delays and inadvertent ‘‘gatekeeping’’ by clinicians. The disadvantages of direct recruiting by research personnel were the associated time costs and risk of breeched confidentiality. In the KBC Project,

123

recruiting was done by an intermediary: because of IRB and HIPAA requirements, patients were recruited by clerical personnel and researchers were only provided with aliases. In both studies, the participation of patients’ caregivers (e.g., family members) was optional but desirable when caregivers played central roles in patients’ health. This raised questions about defining eligibility, for example, if a patient has literacy or cognitive deficits, but his wife is always involved in his care, should they be considered one unit and screened and recruited as such? In some cases, the caregiver was unavailable at the same time as the patient, chose to not participate at all, or attended some but not all research encounters. In one case, an elderly woman with trouble comprehending had one daughter present at the first encounter and another at the second. It was generally

Qual Life Res

difficult to recruit both the patient and caregiver over the phone. In the KBC Project, recruitment was done in a deidentified fashion, and when both patient and caregiver had medical issues, it was difficult to tell during home research visits who was who.

unreasonable delays, researchers had to use a quiet corner of the waiting room or a hallway table outside the waiting room. This meant dealing with concerns about eavesdropping and privacy. Conflicts with compensation

Logistical issues There were times when the researcher arrived at a participant’s home and the participant was not there, had forgotten, or changed their mind about participation, despite reminder phone calls. Some died, were admitted to the hospital, or moved without the researcher being informed. The studies, as is the case with much contextual research, were relatively time-consuming for both researchers and participants. Participants and researchers often spent over 3 h together, including as much as two continuous hours, plus multiple phone calls for recruitment, scheduling, and reminders. Research travel time ranged from 10 min to 3 h one way. Given the potential power differential between them and researchers, a key ethical challenge was to obtain enough information without being too demanding or intrusive. Maintaining participant privacy and confidentiality Both studies had to address ethical human subjects considerations and specific policies and regulations. Special procedures for medical record review and obtaining consent and data from vulnerable (e.g., cognitively impaired) groups [39] had to be reviewed by the Institutional Review Board (IRB) and closely followed. In the KBC Project, medical assistants at the healthcare organization had participating patients choose an alias. Typically, patients chose names such as John or Jane Doe, John Wayne, Marilyn Monroe, but other aliases included Merry Christmas and Jesus Christ. As a result, one focus group had two John Does. The alias was used to greet participants in person and by phone, which resulted in confusion when participants forgot their aliases. One call to a participant went as follows: Researcher: Participant: Researcher: Participant:

Hello, Merry Christmas? (referring to patient alias) Hi! Merry Christmas to you too! (explaining) We are the researchers in Wisconsin No, we are in Pennsylvania. You dialed the wrong number

In the Caring Hearts Study, there were privacy issues related to securing an interview room in the clinic. When a room was not available or going there would have imposed

There were some issues with getting research payments to patients within a time frame deemed appropriate by researchers, participants, or both. In one study, the researchers’ institution did not permit cash payments, which created delays due to time to process payments by check, mail delivery problems, and administrative errors. Some participants desired to use payments immediately for specific purchases. One woman needed the money to purchase a cane. Another couple planned to use their payment to purchase a meal after their interview and felt that the researchers’ promise of a check 3–4 weeks after the interview was unfair and dishonest. They forcefully and publicly demanded that their data be destroyed. Other individuals refused payment or insisted things be done with the payment that the researchers could not accommodate (e.g., donate to clinic). Challenges concerning scientific quality and interpretation (Table 6) Questions of scientific quality, interpretation, and integration of data In contextual data collection, it is possible to gauge and correct for data quality and the validity of interpretations during data collection activity. For example, in an interview or focus group, a researcher can verify data or test their interpretations using follow-up probes. Researcher also use ‘‘member checking’’ to verify interpretations with former participants. However, in our studies, gauging data quality and our interpretations was complicated by multiple, sometimes conflicting sources of evidence. For example, in Caring Hearts, an elderly woman spoke eloquently about the importance of adherence and taking control of her health, but medical record data and home observations revealed a lack of both. Survey-reported data about dietary adherence often contradicted interview data and examination of patients’ pantries and refrigerators. Identifying and pursuing the more accurate information source(s) was difficult because of the time requirement to go through and compare the tremendous amount of data in real time. Researchers struggled to identify during actual data collection whether information was inaccurate or contradictory due to the researcher asking the wrong questions or using the wrong words. Asking participants directly but non-

123

Qual Life Res Table 6 Examples of data collection challenges concerning scientific quality and interpretation Questions of scientific quality, interpretation, and integration of data Missing self-report data due to inability or unwillingness to provide information Missing data in medical records or patients’ personal records Conflicting data provided by different sources (patient vs. caregiver) or at different times Different information given by participants to nurses versus physicians versus researchers Different, sometimes conflicting data provided to different research personnel, depending on participant–researcher relationship, race or sex differences Difficulty determining the reason for contradictory information in real time Difficulty capturing certain contextual data in real time, such as gestures, tone, and smell Potential for biased sampling when clinicians recommend or select patients for participations Participants have hard time remembering events in question or speaking about the research topic, but researchers are not aware of this Analyzing large volumes of qualitative and mixed data to assess data quality and validity in real time is time-consuming Difficulty interpreting whether participants’ experience and perceptions align with the group to which they were assigned (e.g., diagnosis category, intervention vs. control condition)

Table 7 Suggested strategies to overcome challenges in contextual data collection Buy-in: Emphasize to participants that they are the expert on themselves and explain the importance of working together. When participants and communities feel respected and valued, they are more likely to help. Furthermore, if they are fed back results and signs of success from their participation, short and long term, they are likely to continue to help [18, 21, 22, 36, 45] Trust: Communicate openly and honestly, erring on the side of over-informing. Alleviating immediate mistrust and misunderstanding could be accomplished by being transparent (e.g., showing one’s notes and explaining each researcher action) and demonstrating that no harm is occurring during the research. If researchers are aware of possible longstanding issues of mistrust, they can dedicate more time to addressing community members’ concerns and desires before launching into the research [18, 21, 22, 36] Transparency: Expectations should be clearly stated up front. While it is appropriate to redirect data collection toward research goals, it is important to not outright dismiss what participants want, say, or do not say. During recruiting and initial encounters, researchers can provide information on the study using teach-back guidelines (http://www.teachbacktraining.org), being as complete as possible about all that could happen [18, 21, 22, 43, 45] Accommodation: Because participants make time for researchers in their complicated schedules and life, researchers should make the most of their time. They must not over-impose or pursue goals unrelated to their research. It is helpful to call ahead of appointments or provide written or e-mail reminders as needed. Accommodations of location, travel, and timing may be necessary. Mutual flexibility is expected: in some cases, researchers concerned for safety or comfort can insist on a neutral location or other accommodations [18, 21, 22, 36, 45] Openness: Contextual data collection requires representative sampling, which can require additional work and an open mind when interacting with participants. Screening and recruitment may need to be controlled by the research team to avoid individuals or groups being excluded by gatekeepers, inadvertently or not. Researchers who differ from participants should be aware of the language they use, the assumptions they hold, and how their role or actions may be interpreted. Over time, researchers can develop vocabulary and style that participants understand without feeling patronized. Researchers should anticipate requests for help or advice and if appropriate should prepare responses, including connecting participants to resources. Openness requires understanding and empathy without overstepping one’s role [18, 21, 22] Anticipation: Ask participant ahead of time whether others will be present and have criteria for whether others (and who) will be included. After inclusion criteria are set, be open to hearing all that the multiple participants have to offer on the subject at hand—it can create valuable context for the collected data. Anticipate special privacy and confidentiality issues that will arise with multiple participants, vulnerable groups, and research carried out in people’s homes; local human subjects bodies, other researchers, or community groups can help to identify potential issues [22] Compensation: Explain as early as possible the purpose of compensation and any regulations surrounding them, including delays in payment and the need to collect personal information. Participants may need to have an explanation of partial payments and different pay scales. Be prepared to provide participants with contact information of the researchers and human subjects regulatory body in case further questions about compensation arise. Some flexibility may be required, for example, obtaining cash or gift card alternatives for those unable to cash checks. Using food or food-related compensation (e.g., gift card for ice cream shop) should be carefully evaluated when working with individuals with disease-related dietary restrictions [18, 21]

confrontationally about contradictory data was also difficult; in Caring Hearts, such questions were typically reserved toward the end of the interview.

123

We also found it difficult to assign participants to groups based on external data such as knowledge of who received an intervention or had a certain diagnosis. This is because

Qual Life Res

participants’ experiences and perceptions sometimes contradicted these assignments. For instance, in the KBC Project, some patients assigned to the inpatient case manager intervention group did not remember having seen such a person, whereas a quarter of those in the control group (i.e., no case management) did.

Discussion In this paper, we presented a framework of the challenges of contextual data collection, based on several related literatures and refined and exemplified using the authors’ experiences conducting two similar studies in community settings with patients with chronic disease. The framework included ten types of challenges related to research–participant partnership, participant characteristics, research logistics and procedures, and scientific data quality and interpretation. We experienced various challenges in these categories, some of which posed threats to the scientific quality of the data, the ethical treatment of in some cases vulnerable individuals, and the ultimate success of the projects. For example, issues of mutual understandability and trust may have directly impacted data accuracy and completeness, while issues of recruitment and compensation may have biased the sample. The presence of these potentially impactful challenges is notable for researchers and practitioners in the area of patient-reported outcomes measurement. There has been strong work in that field addressing technical instrumentation and measurement issues such as questionnaire psychometrics or interpreting changes in values over time [40, 41]. Beyond these concerns, the collection and use of patientreported outcomes data has a number of implementation challenges [42], some of which were illuminated in this paper. More broadly, implementation challenges are especially common and important to address in any contextual data collection, where data collection is situated in the field, involves various less structured methods, measures a broader scope of the ‘‘system,’’ and requires patients to be engaged. In conducting contextual data collection, we and others have experienced challenges related to establishing meaningful partnerships and relationships with patients and other community stakeholders; accommodating patients’ characteristics, schedules, and daily life; recruitment, confidentiality, and compensation; and making sense of collected data. Further, different contextual data collection projects face different challenges. Even in our two similar research studies, differences in recruiting methods, sample characteristics, and data capture methods resulted in dissimilar data collection challenges. This means that lessons from our experiences

may be transferrable but should be complemented by lessons learned from other studies (for a few additional lessons learned, see [10, 11, 13, 16–18, 20, 43–45]). Fortunately, we found ways to manage and sometimes avoid these challenges. In Table 7, we list a number of strategies that were helpful for us, as well as for others who have conducted contextual or community-based research [18, 21, 22, 43, 45]. An understandable reaction to our description of the challenges to contextual approaches to data collection might be to avoid such approaches altogether, in favor of ones where stricter control is possible. However, even deploying structured methods such as standardized surveys has many challenges, from issues of non-response [42] to ones of displaying and translating patient-reported data [7, 31]. The challenges are also not isolated to community settings. For example, in a study of a hospital catheterization laboratory, we still had difficulty recruiting patients and family members who had driven from further away, had scheduling conflicts, or were preoccupied or anxious. Furthermore, contextual data collection is of utmost importance for accurately describing phenomena such as lived experience, pain, (dis)ability, or satisfaction. To appropriately understand a patient’s health or behavior, it is important to assess these in their natural ecology, whether at school, in the clinic, at home, or in the community [20, 46, 47]. Doing so produces unique insights: for example, in a study of individuals with chronic pain, Walker et al. [48] found that the crucial issue was not the experience of the pain itself but rather the frustration of dealing with medical, policy, and legal systems. Ultimately, collected data need to be used. Knowing a patient’s symptom severity, pain level, or composite quality of life score is useful for some purposes, for example, as a repeated outcome measure in a clinical trial, but insufficient for designing a complex intervention, especially when the intervention must be integrated into a patient’s ecology. One of the best examples of a mismatch between the design of an intervention and patients’ actual experiences of disease is Diana Forsythe’s [49] seminal work on a digital information system for people with migraines. The system was designed from the perspective of neurologists and ignored the point of view of the patients themselves, and its design and use suffered accordingly. Thus, for scientific and practical reasons, we strongly recommend addressing, not ignoring, the contextual data collection-related challenges we identified. At the same time, we note that the challenges to contextual or community-based data collection can produce limitations that need to be considered in interpreting findings. In most cases, findings from contextual approaches should be considered in conjunction with less contextual studies, in order to build a sturdy evidence base.

123

Qual Life Res

Conclusion Conducting community-based or action research can be challenging, but also necessary and rewarding. We provided illustration and suggestions from our own studies in an effort to facilitate future efforts to plan and conduct contextual data collection, particularly among elderly or otherwise vulnerable individuals in community settings. We also urge the continued discovery and reporting of challenges unique to specific types of research (e.g., focus groups on sensitive topics), populations (e.g., non-English speaking), and settings (e.g., assisted living communities). Overall, exemplary ongoing efforts to improve instrumentation and the technical quality of patient-reported data collection must now be complemented with systematic efforts to understand and address the challenges of collecting data in and about context. Acknowledgments We thank the participants in our studies—the many patients and their family members who graciously welcomed us into their lives and homes and whose interactions with us were far more rewarding than ‘‘challenging.’’ Funding for the Caring Hearts Study was provided by grants from the National Institute on Aging (NIA) of the US National Institutes of Health (NIH) (K01AG044439) and grants UL1 TR000445 and KL2 TR000446 from the National Center for Advancing Translational Sciences (NCATS/NIH) through the Vanderbilt CTSA. Funding for the Keystone Beacon Community Project was provided by the US Office of the National Coordinator for Health Information Technology (ONC) through the Beacon award program [award No. 90BC001301]. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding bodies.

References 1. Greenhalgh, J. (2009). The applications of PROs in clinical practice: What are they, do they work, and why? Quality of Life Research, 18(1), 115–123. 2. Higginson, I. J., & Carr, A. J. (2001). Measuring quality of life: Using quality of life measures in the clinical setting. British Medical Journal, 322(7297), 1297–1300. 3. von Korff, M., Jensen, M. P., & Karoly, P. (2000). Assessing global pain severity by self-report in clinical and health services research. Spine, 25, 3140–3151. 4. Ra´heim, M. (2006). Lived experience of chronic pain and fibromyalgia: Women’s stories from daily life. Qualitative Health Research, 16, 741–761. 5. Delvecchio Good, M. J., Browdin, P. E., Good, B. J., & Kleinman, A. (Eds.). (1992). Pain as human experience: An anthropological perspective. Berkley, CA: University of California Press. 6. Doward, L. C., & McKenna, S. P. (2004). Defining patientreported outcomes. Value in health, 7, S4–S8. 7. Wu, A. W., Jensen, R. E., Salzberg, C., Snyder, C. (2013). Advances in the use of patient reported outcome measures in electronic health records. http://www.pcori.org/assets/2013/11/ PCORI-PRO-Workshop-EHR-Landscape-Review-111913.pdf. 8. Coulter, A., & Ellins, J. (2006). Patient-focused Interventions: A review of the evidence. London: The Health Foundation.

123

9. Eton, D. E. A. (2013). A systematic review of patient-reported measures of burden of treatment in three chronic diseases. Patient Related Outcome Measures, 4, 7–20. 10. Bayliss, E. A., Steiner, J. F., Fernald, D. H., Crane, L. A., & Main, D. S. (2003). Descriptions of barriers to self-care by persons with comorbid chronic diseases. Annals of Family Medicine, 1, 15–21. 11. Bayliss, E. A., Ellis, J. L., & Stiener, J. F. (2007). Barriers to selfmanagement and quality-of-life outcomes in seniors with multimorbidities. Annals of Family Medicine, 5, 392–402. 12. Sabate, E. (2003). Adherence to long-term therapies: Evidence for action (pp. 1–194). Geneva (Switzerland): World Health Organization. 13. McEntee, et al. (2009). Patient-, provider-, and system-level barriers to heart failure care. Journal of Cardiovascular Nursing, 24, 290–298. 14. Schoenberg, N. E., Amey, C. H., & Coward, R. T. (1998). Stories of meaning: Lay perspectives on the origin and management of noninsulin dependent diabetes mellitus among older women in the United States. Social Science and Medicine, 47, 2113–2125. 15. Braveman, P., Egerter, S., & Williams, D. R. (2011). The social determinants of health: Coming of age. Annual Review of Public Health, 32, 381–398. 16. Stetson, B., Schlundt, D., Rothschild, C., Floyd, J. E., Rogers, W., & Mokshagundam, S. P. (2011). Development and validation of The Personal Diabetes Questionnaire (PDQ): A measure of diabetes self-care behaviors, perceptions, and barriers. Diabetes Research and Clinical Practice, 91, 321–332. 17. Harkness, K., Spaling, M. A., Currie, K., Strachan, P. H., & Clark, A. M. (2014). A systematic review of patient heart failure self-care strategies. Journal of Cardiovascular Nursing. [Epub ahead of print]. 18. Israel, B. A., Krieger, J., Vlahov, D., Ciske, S., Foley, M., Fortin, P., et al. (2006). Challenges and facilitating factors in sustaining community-based participatory research partnerships: Lessons learned from the Detroit, New York City, and Seattle urban research centers. Journal of Urban Health, 83(6), 1022–1040. 19. Israel, B. A., Schulz, A. J., Parker, E. A., & Becker, A. B. (1998). Review of community-based research: Assessing partnership approaches to improve public health. Annual Review of Public Health, 19, 173–202. 20. Barkin, S., Schlundt, D., & Smith, P. (2013). Communityengaged research perspectives: Then and now. Academic Pediatric, 13(2), 93–97. 21. Reason, P., & Bradbury, H. (Eds.). (2008). The SAGE handbook of action research: Participative inquiry and practice. London: SAGE Publications Ltd. 22. McIntyre, A. (2008). In V. Knight (Ed.), Participatory action research. Qualitative research methods series 52. Thousand Oaks, CA: Sage Publications, Inc. 23. Scholars, K. H. (2001). Program overview: Community track. http://www.kellogghealthscholars.org/about/community.cfm. 24. Wallerstein, N., & Duran, B. (2010). Community-based participatory research contributions to intervention research: The intersection of science and practice to improve health equity. American Journal of Public Health, 100(S1), S40–S46. 25. Viswanathan, M., Ammerman, A., Eng, E., Gartlehner, G., Lohr, K. N., Griffith, D., et al. (2004). Community-based participatory research: Assessing the evidence. Rockville, MD: AHRQ. 26. Green, J., & Thorogood, N. (Eds.). (2009). Qualitative methods for health research (2nd ed.). Thousand Oaks, CA: Sage. 27. Pope, C., & Mays, N. (2006). Qualitative research in health care (3rd ed.). Malden, MA: Blackwell Publishing Ltd. 28. Council, National. Research. (2011). Health care comes home: The human factors. Washington, DC: The National Academies Press.

Qual Life Res 29. Holden, R. J., Carayon, P., Gurses, A. P., Hoonakker, P., Hundt, A. S., Ozok, A. A., et al. (2013). SEIPS 2.0: A human factors framework for studying and improving the work of healthcare professionals and patients. Ergonomics, 56(11), 1669–1686. 30. Tashiro, J. (2002). Exploring health promoting lifestyle behaviors of Japanese college women: Perceptions, practices, and issues. Health Care for Women International, 23, 59–70. 31. Greenhalgh, J., Long, A. F., & Flynn, R. (2005). The use of patient reported outcome measures in routine clinical practice: Lack of impact or lack of theory? Social Science and Medicine, 60, 833–843. 32. Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013). Achieving integration in mixed methods designs—principles and practices. Health Services Research, 48(6pt2), 2134–2156. 33. Scammon, D. L., et al. (2013). Connecting the dots and merging meaning: Using mixed methods to study primary care delivery transformation. Health Services Research, 48(6pt2), 2181–2207. 34. Miller, W. L., et al. (2013). Integrating mixed methods in health services and delivery system research. Health Services Research, 48(6pt2), 2125–2267. 35. Carayon, P. (2010). Human factors in patient safety as an innovation. Applied Ergonomics, 41(5), 657–665. 36. Holden, R. J., Or, C. K. L., Alper, S. J., Rivera, A. J., & Karsh, B. T. (2008). A change management framework for macroergonomic field research. Applied Ergonomics, 39, 459–474. 37. Carayon, P., et al. (2006). Work system design for patient safety: The SEIPS model. Quality & Safety in Health Care, 15, i50–i58. 38. Carayon, P., et al. (2014). Human factors systems approach to healthcare quality and patient safety. Applied Ergonomics, 45(1), 14–25. 39. N.B.A.C. (2001). Ethical and policy issues in research involving human participants: Report and recommendations of the National Bioethics Advisory Commission. Rockville, MD: NBAC. 40. Revicki, D., Hays, R. D., Cella, D., & Sloan, J. (2008). Recommended methods for determining responsiveness and minimally

41.

42.

43.

44.

45.

46.

47.

48.

49.

important differences for patient-reported outcomes. Journal of Clinical Epidemiology, 61, 102–109. Fayers, P., & Machin, D. (Eds.). (2007). Quality of life: The assessment, analysis, and interpretation of patient-reported outcomes (2nd ed.). Hoboken, NJ: Wilen. Hjollund, N. H. I., Larsen, L. P., Biering, K., Johnsen, S. P., Riiskajaer, E., Schougaard, L. M. (2014). Use of patient-reported outcome (PRO) measures at group and patient levels: Experiences from the generic integrated PRO system, WestChronic. Interactive Journal of Medical Research, 3(1). doi:10.2196/ijmr. 2885 Molinari, A. L., et al. (2011). Barriers to conducting research with Community-dwelling adults who have intellectual disabilities. Intellectual and Developmental Disabilities, 49(5), 392–396. Wright, P. B., Curran, G. M., Stewart, K. E., & Booth, B. M. (2013). A qualitative analysis of provider barriers and solutions to HIV testing for substance users in a small, largely rural Southern state. The Journal of Rural Health, 29, 420–431. Lasker, R. D., & Weiss, E. S. (2003). Broadening participation in community problem solving: A multidisciplinary model to support collaborative practice and research. Journal of Urban Health, 80(1), 14–34. Fisher, E. B., Brownson, C. A., O’Toole, M. L., Shetty, G., Anwuri, V. V., & Glasgow, R. E. (2005). Ecological approaches to self-management: The case of diabetes. American Journal of Public Health, 95, 1523–1535. Richard, L., Gauvin, L., & Raine, K. (2010). Ecological models revisited: Their uses and evolution in health promotion over two decades. Annual Review of Public Health, 32, 307–326. Walker, J., Holloway, I., & Sofaer, B. (1999). In the system: The lived experience of chronic back pain from the perspectives of those seeking help from pain clinics. Pain, 80, 621–628. Forsythe, D. E. (1996). New bottles, old wine: Hidden cultural assumptions in a computerized explanation system for migraine sufferers. Medical Anthropology Quarterly, 10, 551–574.

123