At the Intersection of Health, Health Care and Policy Cite this article as: David Grande, Sarah E. Gollust, Maximilian Pany, Jane Seymour, Adeline Goss, Austin Kilaru and Zachary Meisel Translating Research For Health Policy: Researchers' Perceptions And Use Of Social Media Health Affairs, 33, no.7 (2014):1278-1285 (published online June 6, 2014; 10.1377/hlthaff.2014.0300)
The online version of this article, along with updated information and services, is available at: http://content.healthaffairs.org/content/33/7/1278.full.html
For Reprints, Links & Permissions: http://healthaffairs.org/1340_reprints.php E-mail Alerts : http://content.healthaffairs.org/subscriptions/etoc.dtl To Subscribe: http://content.healthaffairs.org/subscriptions/online.shtml
Health Affairs is published monthly by Project HOPE at 7500 Old Georgetown Road, Suite 600, Bethesda, MD 20814-6133. Copyright © 2014 by Project HOPE - The People-to-People Health Foundation. As provided by United States copyright law (Title 17, U.S. Code), no part of Health Affairs may be reproduced, displayed, or transmitted in any form or by any means, electronic or mechanical, including photocopying or by information storage or retrieval systems, without prior written permission from the Publisher. All rights reserved.
Not for commercial use or unauthorized distribution Downloaded from content.healthaffairs.org by Health Affairs on August 22, 2015 at University of Sussex
Web First
10.1377/hlthaff.2014.0300 HEALTH AFFAIRS 33, NO. 7 (2014): 1278–1285 ©2014 Project HOPE—The Peopleto-People Health Foundation, Inc.
doi:
David Grande (dgrande@whar ton.upenn.edu) is an assistant professor of medicine at the University of Pennsylvania, in Philadelphia. Sarah E. Gollust is an assistant professor in the Division of Health Policy and Management, School of Public Health, University of Minnesota, in Minneapolis. Maximilian Pany is an undergraduate student at Swarthmore College, in Pennsylvania. Jane Seymour is a graduate student in public health at the University of Pennsylvania. Adeline Goss is a student at the Perelman School of Medicine, University of Pennsylvania. Austin Kilaru is a student at the Perelman School of Medicine, University of Pennsylvania. Zachary Meisel is an assistant professor of emergency medicine at the University of Pennsylvania.
By David Grande, Sarah E. Gollust, Maximilian Pany, Jane Seymour, Adeline Goss, Austin Kilaru, and Zachary Meisel
Translating Research For Health Policy: Researchers’ Perceptions And Use Of Social Media As the United States moves forward with health reform, the communication gap between researchers and policy makers will need to be narrowed to promote policies informed by evidence. Social media represent an expanding channel for communication. Academic journals, public health agencies, and health care organizations are increasingly using social media to communicate health information. For example, the Centers for Disease Control and Prevention now regularly tweets to 290,000 followers. We conducted a survey of health policy researchers about using social media and two traditional channels (traditional media and direct outreach) to disseminate research findings to policy makers. Researchers rated the efficacy of the three dissemination methods similarly but rated social media lower than the other two in three domains: researchers’ confidence in their ability to use the method, peers’ respect for its use, and how it is perceived in academic promotion. Just 14 percent of our participants reported tweeting, and 21 percent reported blogging about their research or related health policy in the past year. Researchers described social media as being incompatible with research, of high risk professionally, of uncertain efficacy, and an unfamiliar technology that they did not know how to use. Researchers will need evidence-based strategies, training, and institutional resources to use social media to communicate evidence. ABSTRACT
T
he translation of research evidence into clinical practice is often slow. Considerable attention has been paid to delays in this translation process, such as the delay in adopting the use of beta-blockers for acute myocardial infarction after evidence supporting their use became available.1 The National Research Council recently expressed concern about a similar issue: whether scientific evidence is being adequately communicated to policy makers and adopted in public policy.2 The scientific evidence produced by health policy and health services researchers provides a foundation of knowledge for public policy relat1278
Health A ffairs
J u ly 2 0 1 4
33:7
ed to health care and population health. To maximize the return on public investments in research, the findings from this research need to be communicated effectively to policy makers and other health care stakeholders. Moreover, the demand for evidence is likely to grow as the nation embarks on implementation of the Affordable Care Act and addresses the challenges in health care related to cost, quality, and access that lie ahead. Researchers and policy makers confront substantial barriers to effective communication. For example, the two groups have different incentives, social networks, time frames, vocabularies, and values.3–5 Academic researchers work in
Downloaded from content.healthaffairs.org by Health Affairs on August 22, 2015 at University of Sussex
an environment that values traditional scholarly productivity, has a long time horizon for knowledge generation, and is largely based within the social networks of academe.5 In contrast, policy makers require relevant, timely research that is accessible and delivered by trusted individuals with whom they have established relationships.6– 11 Policy makers value scientific evidence, but it is just one item to be considered in a setting full of competing influences.12 Researchers vary in the degree to which they actively disseminate their work, such as writing policy briefs or working with public relations specialists to generate press coverage.13 Social media represent an expanding tool that could facilitate communication of scientific evidence to policy makers. Technologies such as Twitter have created new communication networks and channels that can deliver information in targeted and efficient ways.14 For example, a health legislative aide can follow on Twitter trusted individuals who have strong reputations for culling and translating policy-relevant findings from research journals. Similarly, bloggers can translate new research into formats that policy makers might find more accessible than traditional journal articles.15 These tools complement traditional dissemination methods such as issuing press releases, because they may better target specialized audiences, quickly link to related content (for example, policy briefs), create opportunities to sustain communication beyond a single day of press coverage, and facilitate direct relationships with key information intermediaries or policy makers. Academic journals are increasingly using Twitter, as are major health agencies, health officials, and health care organizations. As an example, Twitter’s website indicated 73,000 followers for Health Affairs and 182,000 for the New England Journal of Medicine as of April 22, 2014; the Centers for Disease Control and Prevention regularly disseminates public health information to 290,000 followers on Twitter. Taking advantage of the promise of social media for translation and dissemination will require engagement by researchers and their institutions. However, little is known about researchers’ attitudes toward social media and their capacity to use these media.16 In this article we present findings about the attitudes of health services and health policy researchers toward social media compared to more traditional communication channels. We also measured researchers’ use of social media.
Study Data And Methods Study Population And Recruitment We conducted a mixed-methods survey that included an experimental component. The survey was administered in person in June 2013 as a structured and semistructured interview, containing both closed-ended and open-ended questions. Our respondents were 215 health services and health policy researchers recruited from a random sample of 325 people who had preregistered for the 2013 AcademyHealth Annual Research Meeting, the largest gathering of US health policy researchers. We limited our sample to researchers whose primary affiliation was with an academic institution. To invite each of the 325 people to be interviewed at the meeting, we sent them a recruitment letter introducing the study, along with $2. The letter stated that “the study will examine strategies to improve knowledge dissemination from academic researchers to policy makers.” We sent follow-up e-mail to schedule interviews with responders, and we contacted nonresponders up to two additional times by e-mail. Participants were entered in a raffle for a $200 retail gift card. Survey Instrument We designed the instrument to compare researchers’ attitudes toward different methods of disseminating research and to measure the frequency of the researchers’ dissemination behaviors. We measured attitudes in the following three ways. ▸ VIGNETTE - BASED RANDOMIZED EXPERIMENT : We presented each respondent with a vignette in which we described a researcher who published a study with implications for health policies on childhood obesity in a peerreviewed journal. We randomly assigned each participant one of three versions of the vignette, in which the researcher disseminated her findings through traditional media, social media, or direct outreach to policy makers. The experimental design and vignettes are described in online Appendix A1.17 We asked participants to rate the activity presented in the vignette across five domains: the efficacy of the researcher’s mode of dissemination, their confidence about disseminating their own work in that mode, their peers’ respect for that mode of dissemination, how much the academic promotion process values the mode, and their overall impression of the researcher described in the vignette. We asked participants to rate each domain on a scale from 1 to 10, except for overall impression. In that case, we asked participants to rate the researcher using a feeling thermometer (a rating between 50 and 100 degrees meant that the respondent felt warm toward the researcher, while a rating between 0 and 50 degrees meant that the respondent felt July 2014
33:7
Downloaded from content.healthaffairs.org by Health Affairs on August 22, 2015 at University of Sussex
Health Affa irs
1279
Web First cool toward the researcher).18 ▸ RATING OF EFFICACY OF DISSEMINATION METHODS : On a scale of 1 to 10, we asked each participant to rate the efficacy of eight channels to “communicate specifically with policy makers and other decision makers.” Two of the eight channels were scholarly (editorials or commentaries in peer-reviewed journals and original research articles), one was traditional media (articles in newspapers about a research study), two were direct outreach to policy makers (directly contacting policy makers and issuing policy briefs from an academic center or institute), and three were social media (Twitter, Facebook, and blogs on journal websites). In the vignette experiment we provided a contextual narrative that included characteristics of the researcher, her research publication, and its successful dissemination. However, in this section of the survey, participants rated the dissemination method without additional context. ▸ OPEN - ENDED QUALITATIVE QUESTIONS : We asked participants how researchers should be using social media, if at all, to disseminate their work. ▸ FREQUENCY OF RESPONDENTS ’ DISSEMINATION ACTIVITIES : In addition, we measured respondents’ reported frequency of dissemination behaviors. We did this by asking if at any time in the previous year they had engaged in dissemination activities using scholarly channels (“published an editorial in a journal where you made policy recommendations”), traditional media channels (“news media covered a study you published,” were “interviewed by a news reporter about an issue related to your area of expertise”), direct outreach to policy makers (“published a policy brief,” “had direct contact with a policymaker,” “directly contacted an interest group or community group”), and social media channels (“blogged, used Twitter, or Facebook to communicate about research or health policy expertise”). We conducted cognitive pretesting of the survey instrument on a convenience sample of five health services researchers and adapted the instrument accordingly. Survey Administration The in-person structured interviews were conducted in a designated area at the AcademyHealth Annual Research Meeting. Trained research assistants conducted the interviews and recorded participants’ responses in writing for the close-ended questions and with a digital audio recorder for the openended questions. We obtained consent prior to all interviews. Of the 325 researchers in the sample, 215 participated, for a response rate of 66 percent. Twenty-eight participants were unable to com1280
H e a lt h A f fai r s
J u ly 201 4
3 3: 7
Little is known about researchers’ attitudes toward social media and their capacity to use these media.
plete the interview during the meeting; instead, they were interviewed by telephone. The study was approved by the University of Pennsylvania’s Institutional Review Board. Analysis We used analysis of variance to compare both mean scores across the three experimental groups—participants exposed to vignettes that described dissemination through social media, traditional media, or direct outreach to policy makers—for each of the domains that we measured and mean academic rank. We used multivariate linear regression models to test for interactions between experimentalgroup assignment and academic rank on the key outcomes. We used chi-square tests of association to test for significant (p < 0:05) differences in reported dissemination behaviors by academic rank. For open-ended questions, we entered deidentified transcripts of responses into the structured qualitative software QSR NVivo, version 10.0. We used an inductive process to identify the main themes that emerged in the qualitative text, first comparing emerging themes across a small sample of text using a consensus approach and then applying the resulting themes to the full sample.19 Limitations Our research had several limitations. First, we recruited participants from a sample of university health services and health policy researchers attending the AcademyHealth Annual Research Meeting. The researchers attending the meeting may not be representative of university researchers overall. In fact, these researchers’ applied focus and policy interests should bias them toward more, not less, interest in translation to policy settings. However, our participants were drawn from a random sample (not a convenience sample) of people who had preregistered for the meeting. Therefore, we are relatively confident that the sample is representative of this population. Second, we relied on self-reports to measure behavior. This method could be vulnerable to some recall bias as well as to social desirability
Downloaded from content.healthaffairs.org by Health Affairs on August 22, 2015 at University of Sussex
bias, depending on the behavior. Third, as is the case with all survey research, our results could be vulnerable to nonresponse bias. This is true in spite of our high response rate of 66 percent.
Study Results Exhibit 1 describes the characteristics of the 215 participants. There were no significant differences in participants’ demographic characteristics across the three experimental groups. Vignette Ratings Participants rated two of the dissemination methods—traditional media and direct outreach to policy makers—favorably across the five domains we measured (Exhibit 2). Dissemination through social media received neutral scores in the domains of confidence, peer respect, and academic promotion. It also ranked below the other two methods in the domain of overall impression but was evaluated almost as favorably as the other two in efficacy. As a secondary outcome, we tested for differences in the ratings of the three dissemination methods by academic rank. We found a significant difference by academic rank in how participants rated the efficacy of the methods, with
senior faculty considering social media less favorably than junior faculty (Exhibit 3). However, we found no significant differences in the other domains (Appendices A2, A3, and A4).17 Ratings Of Efficacy Of Dissemination Methods We asked all participants, regardless of their experimental-group assignment, to rate the efficacy of various dissemination methods. Respondents rated more traditional dissemination methods (direct outreach to policy makers, traditional media, and policy briefs) favorably and social media methods (Twitter, journal blogs, and Facebook) unfavorably (Exhibit 4). Higher academic rank was associated with less favorable attitudes toward Twitter (p ¼ 0:02), blogs on journal websites (p ¼ 0:03), and newspaper articles (p ¼ 0:03). Because the participants had been differentially exposed to a randomly assigned vignette, we tested for priming effects—that is, whether exposure to different vignettes earlier in the survey (based on random assignment to an experimental group) influenced the reported attitudes in questions that appeared later in the survey. We found none. Reported Dissemination Activities In The Past Year Sixty-five percent or more of the par-
Exhibit 1 Survey Participants’ Characteristics, By Experimental Group, 2013 Experimental group All participants (N=215)
Traditional media (n=78)
Social media (n=67)
Direct outreach to policy makers (n=70)
Number
Percent
Number
Percent
Number
Percent
Number
Percent
4 35 1 9 3 163
1.9 16.3 0.5 4.2 1.4 75.8
2 12 1 5 1 57
2.6 15.4 1.3 6.4 1.3 73.1
2 12 0 1 1 51
3.0 17.9 0.0 1.5 1.5 76.1
0 11 0 3 1 55
0.0 15.7 0.0 4.3 1.4 78.6
0–10 11–20 21–30 31 or more
95 62 38 20
44.2 28.8 17.7 9.3
36 20 13 9
46.2 25.6 16.7 11.5
34 21 8 4
50.8 31.3 11.9 6.0
25 21 17 7
35.7 30.0 24.3 10.0
Academic rankc Assistant professor Associate professor Full professor Other Sexd
87 48 62 18
40.5 22.3 28.8 8.4
35 15 21 7
44.9 19.2 26.9 9.0
30 16 14 7
44.8 23.9 20.9 10.5
22 17 27 4
31.4 24.3 38.6 5.7
125 90
58.1 41.9
29 49
37.2 62.8
26 41
38.8 61.2
35 35
50.0 50.0
Characteristic Degreea JD MD MD and JD MD and doctorate Master’s Doctorate Years since degreeb
Female Male
SOURCE Authors’ analysis of survey data. NOTES The three experimental groups were assigned one of three versions of a vignette that described different methods of disseminating peer-reviewed research, as explained in the text. Significance refers to associations between participants’ characteristics and experimental-group assignment. ap ¼ 0:80. bp ¼ 0:36. cp ¼ 0:28. dp ¼ 0:24.
July 2014
33:7
Downloaded from content.healthaffairs.org by Health Affairs on August 22, 2015 at University of Sussex
Health A ffairs
1281
Web First Exhibit 2 Survey Participants’ Attitudes Toward Dissemination Methods, 2013 Experimental group Attitude Perceived efficacy of dissemination methoda
Traditional media 6.7
Social media 6.6
Direct outreach to policy makers 7.3
p value 0.06
Confidence in dissemination methoda
7.0
5.4
6.0
The online version of this article, along with updated information and services, is available at: http://content.healthaffairs.org/content/33/7/1278.full.html
For Reprints, Links & Permissions: http://healthaffairs.org/1340_reprints.php E-mail Alerts : http://content.healthaffairs.org/subscriptions/etoc.dtl To Subscribe: http://content.healthaffairs.org/subscriptions/online.shtml
Health Affairs is published monthly by Project HOPE at 7500 Old Georgetown Road, Suite 600, Bethesda, MD 20814-6133. Copyright © 2014 by Project HOPE - The People-to-People Health Foundation. As provided by United States copyright law (Title 17, U.S. Code), no part of Health Affairs may be reproduced, displayed, or transmitted in any form or by any means, electronic or mechanical, including photocopying or by information storage or retrieval systems, without prior written permission from the Publisher. All rights reserved.
Not for commercial use or unauthorized distribution Downloaded from content.healthaffairs.org by Health Affairs on August 22, 2015 at University of Sussex
Web First
10.1377/hlthaff.2014.0300 HEALTH AFFAIRS 33, NO. 7 (2014): 1278–1285 ©2014 Project HOPE—The Peopleto-People Health Foundation, Inc.
doi:
David Grande (dgrande@whar ton.upenn.edu) is an assistant professor of medicine at the University of Pennsylvania, in Philadelphia. Sarah E. Gollust is an assistant professor in the Division of Health Policy and Management, School of Public Health, University of Minnesota, in Minneapolis. Maximilian Pany is an undergraduate student at Swarthmore College, in Pennsylvania. Jane Seymour is a graduate student in public health at the University of Pennsylvania. Adeline Goss is a student at the Perelman School of Medicine, University of Pennsylvania. Austin Kilaru is a student at the Perelman School of Medicine, University of Pennsylvania. Zachary Meisel is an assistant professor of emergency medicine at the University of Pennsylvania.
By David Grande, Sarah E. Gollust, Maximilian Pany, Jane Seymour, Adeline Goss, Austin Kilaru, and Zachary Meisel
Translating Research For Health Policy: Researchers’ Perceptions And Use Of Social Media As the United States moves forward with health reform, the communication gap between researchers and policy makers will need to be narrowed to promote policies informed by evidence. Social media represent an expanding channel for communication. Academic journals, public health agencies, and health care organizations are increasingly using social media to communicate health information. For example, the Centers for Disease Control and Prevention now regularly tweets to 290,000 followers. We conducted a survey of health policy researchers about using social media and two traditional channels (traditional media and direct outreach) to disseminate research findings to policy makers. Researchers rated the efficacy of the three dissemination methods similarly but rated social media lower than the other two in three domains: researchers’ confidence in their ability to use the method, peers’ respect for its use, and how it is perceived in academic promotion. Just 14 percent of our participants reported tweeting, and 21 percent reported blogging about their research or related health policy in the past year. Researchers described social media as being incompatible with research, of high risk professionally, of uncertain efficacy, and an unfamiliar technology that they did not know how to use. Researchers will need evidence-based strategies, training, and institutional resources to use social media to communicate evidence. ABSTRACT
T
he translation of research evidence into clinical practice is often slow. Considerable attention has been paid to delays in this translation process, such as the delay in adopting the use of beta-blockers for acute myocardial infarction after evidence supporting their use became available.1 The National Research Council recently expressed concern about a similar issue: whether scientific evidence is being adequately communicated to policy makers and adopted in public policy.2 The scientific evidence produced by health policy and health services researchers provides a foundation of knowledge for public policy relat1278
Health A ffairs
J u ly 2 0 1 4
33:7
ed to health care and population health. To maximize the return on public investments in research, the findings from this research need to be communicated effectively to policy makers and other health care stakeholders. Moreover, the demand for evidence is likely to grow as the nation embarks on implementation of the Affordable Care Act and addresses the challenges in health care related to cost, quality, and access that lie ahead. Researchers and policy makers confront substantial barriers to effective communication. For example, the two groups have different incentives, social networks, time frames, vocabularies, and values.3–5 Academic researchers work in
Downloaded from content.healthaffairs.org by Health Affairs on August 22, 2015 at University of Sussex
an environment that values traditional scholarly productivity, has a long time horizon for knowledge generation, and is largely based within the social networks of academe.5 In contrast, policy makers require relevant, timely research that is accessible and delivered by trusted individuals with whom they have established relationships.6– 11 Policy makers value scientific evidence, but it is just one item to be considered in a setting full of competing influences.12 Researchers vary in the degree to which they actively disseminate their work, such as writing policy briefs or working with public relations specialists to generate press coverage.13 Social media represent an expanding tool that could facilitate communication of scientific evidence to policy makers. Technologies such as Twitter have created new communication networks and channels that can deliver information in targeted and efficient ways.14 For example, a health legislative aide can follow on Twitter trusted individuals who have strong reputations for culling and translating policy-relevant findings from research journals. Similarly, bloggers can translate new research into formats that policy makers might find more accessible than traditional journal articles.15 These tools complement traditional dissemination methods such as issuing press releases, because they may better target specialized audiences, quickly link to related content (for example, policy briefs), create opportunities to sustain communication beyond a single day of press coverage, and facilitate direct relationships with key information intermediaries or policy makers. Academic journals are increasingly using Twitter, as are major health agencies, health officials, and health care organizations. As an example, Twitter’s website indicated 73,000 followers for Health Affairs and 182,000 for the New England Journal of Medicine as of April 22, 2014; the Centers for Disease Control and Prevention regularly disseminates public health information to 290,000 followers on Twitter. Taking advantage of the promise of social media for translation and dissemination will require engagement by researchers and their institutions. However, little is known about researchers’ attitudes toward social media and their capacity to use these media.16 In this article we present findings about the attitudes of health services and health policy researchers toward social media compared to more traditional communication channels. We also measured researchers’ use of social media.
Study Data And Methods Study Population And Recruitment We conducted a mixed-methods survey that included an experimental component. The survey was administered in person in June 2013 as a structured and semistructured interview, containing both closed-ended and open-ended questions. Our respondents were 215 health services and health policy researchers recruited from a random sample of 325 people who had preregistered for the 2013 AcademyHealth Annual Research Meeting, the largest gathering of US health policy researchers. We limited our sample to researchers whose primary affiliation was with an academic institution. To invite each of the 325 people to be interviewed at the meeting, we sent them a recruitment letter introducing the study, along with $2. The letter stated that “the study will examine strategies to improve knowledge dissemination from academic researchers to policy makers.” We sent follow-up e-mail to schedule interviews with responders, and we contacted nonresponders up to two additional times by e-mail. Participants were entered in a raffle for a $200 retail gift card. Survey Instrument We designed the instrument to compare researchers’ attitudes toward different methods of disseminating research and to measure the frequency of the researchers’ dissemination behaviors. We measured attitudes in the following three ways. ▸ VIGNETTE - BASED RANDOMIZED EXPERIMENT : We presented each respondent with a vignette in which we described a researcher who published a study with implications for health policies on childhood obesity in a peerreviewed journal. We randomly assigned each participant one of three versions of the vignette, in which the researcher disseminated her findings through traditional media, social media, or direct outreach to policy makers. The experimental design and vignettes are described in online Appendix A1.17 We asked participants to rate the activity presented in the vignette across five domains: the efficacy of the researcher’s mode of dissemination, their confidence about disseminating their own work in that mode, their peers’ respect for that mode of dissemination, how much the academic promotion process values the mode, and their overall impression of the researcher described in the vignette. We asked participants to rate each domain on a scale from 1 to 10, except for overall impression. In that case, we asked participants to rate the researcher using a feeling thermometer (a rating between 50 and 100 degrees meant that the respondent felt warm toward the researcher, while a rating between 0 and 50 degrees meant that the respondent felt July 2014
33:7
Downloaded from content.healthaffairs.org by Health Affairs on August 22, 2015 at University of Sussex
Health Affa irs
1279
Web First cool toward the researcher).18 ▸ RATING OF EFFICACY OF DISSEMINATION METHODS : On a scale of 1 to 10, we asked each participant to rate the efficacy of eight channels to “communicate specifically with policy makers and other decision makers.” Two of the eight channels were scholarly (editorials or commentaries in peer-reviewed journals and original research articles), one was traditional media (articles in newspapers about a research study), two were direct outreach to policy makers (directly contacting policy makers and issuing policy briefs from an academic center or institute), and three were social media (Twitter, Facebook, and blogs on journal websites). In the vignette experiment we provided a contextual narrative that included characteristics of the researcher, her research publication, and its successful dissemination. However, in this section of the survey, participants rated the dissemination method without additional context. ▸ OPEN - ENDED QUALITATIVE QUESTIONS : We asked participants how researchers should be using social media, if at all, to disseminate their work. ▸ FREQUENCY OF RESPONDENTS ’ DISSEMINATION ACTIVITIES : In addition, we measured respondents’ reported frequency of dissemination behaviors. We did this by asking if at any time in the previous year they had engaged in dissemination activities using scholarly channels (“published an editorial in a journal where you made policy recommendations”), traditional media channels (“news media covered a study you published,” were “interviewed by a news reporter about an issue related to your area of expertise”), direct outreach to policy makers (“published a policy brief,” “had direct contact with a policymaker,” “directly contacted an interest group or community group”), and social media channels (“blogged, used Twitter, or Facebook to communicate about research or health policy expertise”). We conducted cognitive pretesting of the survey instrument on a convenience sample of five health services researchers and adapted the instrument accordingly. Survey Administration The in-person structured interviews were conducted in a designated area at the AcademyHealth Annual Research Meeting. Trained research assistants conducted the interviews and recorded participants’ responses in writing for the close-ended questions and with a digital audio recorder for the openended questions. We obtained consent prior to all interviews. Of the 325 researchers in the sample, 215 participated, for a response rate of 66 percent. Twenty-eight participants were unable to com1280
H e a lt h A f fai r s
J u ly 201 4
3 3: 7
Little is known about researchers’ attitudes toward social media and their capacity to use these media.
plete the interview during the meeting; instead, they were interviewed by telephone. The study was approved by the University of Pennsylvania’s Institutional Review Board. Analysis We used analysis of variance to compare both mean scores across the three experimental groups—participants exposed to vignettes that described dissemination through social media, traditional media, or direct outreach to policy makers—for each of the domains that we measured and mean academic rank. We used multivariate linear regression models to test for interactions between experimentalgroup assignment and academic rank on the key outcomes. We used chi-square tests of association to test for significant (p < 0:05) differences in reported dissemination behaviors by academic rank. For open-ended questions, we entered deidentified transcripts of responses into the structured qualitative software QSR NVivo, version 10.0. We used an inductive process to identify the main themes that emerged in the qualitative text, first comparing emerging themes across a small sample of text using a consensus approach and then applying the resulting themes to the full sample.19 Limitations Our research had several limitations. First, we recruited participants from a sample of university health services and health policy researchers attending the AcademyHealth Annual Research Meeting. The researchers attending the meeting may not be representative of university researchers overall. In fact, these researchers’ applied focus and policy interests should bias them toward more, not less, interest in translation to policy settings. However, our participants were drawn from a random sample (not a convenience sample) of people who had preregistered for the meeting. Therefore, we are relatively confident that the sample is representative of this population. Second, we relied on self-reports to measure behavior. This method could be vulnerable to some recall bias as well as to social desirability
Downloaded from content.healthaffairs.org by Health Affairs on August 22, 2015 at University of Sussex
bias, depending on the behavior. Third, as is the case with all survey research, our results could be vulnerable to nonresponse bias. This is true in spite of our high response rate of 66 percent.
Study Results Exhibit 1 describes the characteristics of the 215 participants. There were no significant differences in participants’ demographic characteristics across the three experimental groups. Vignette Ratings Participants rated two of the dissemination methods—traditional media and direct outreach to policy makers—favorably across the five domains we measured (Exhibit 2). Dissemination through social media received neutral scores in the domains of confidence, peer respect, and academic promotion. It also ranked below the other two methods in the domain of overall impression but was evaluated almost as favorably as the other two in efficacy. As a secondary outcome, we tested for differences in the ratings of the three dissemination methods by academic rank. We found a significant difference by academic rank in how participants rated the efficacy of the methods, with
senior faculty considering social media less favorably than junior faculty (Exhibit 3). However, we found no significant differences in the other domains (Appendices A2, A3, and A4).17 Ratings Of Efficacy Of Dissemination Methods We asked all participants, regardless of their experimental-group assignment, to rate the efficacy of various dissemination methods. Respondents rated more traditional dissemination methods (direct outreach to policy makers, traditional media, and policy briefs) favorably and social media methods (Twitter, journal blogs, and Facebook) unfavorably (Exhibit 4). Higher academic rank was associated with less favorable attitudes toward Twitter (p ¼ 0:02), blogs on journal websites (p ¼ 0:03), and newspaper articles (p ¼ 0:03). Because the participants had been differentially exposed to a randomly assigned vignette, we tested for priming effects—that is, whether exposure to different vignettes earlier in the survey (based on random assignment to an experimental group) influenced the reported attitudes in questions that appeared later in the survey. We found none. Reported Dissemination Activities In The Past Year Sixty-five percent or more of the par-
Exhibit 1 Survey Participants’ Characteristics, By Experimental Group, 2013 Experimental group All participants (N=215)
Traditional media (n=78)
Social media (n=67)
Direct outreach to policy makers (n=70)
Number
Percent
Number
Percent
Number
Percent
Number
Percent
4 35 1 9 3 163
1.9 16.3 0.5 4.2 1.4 75.8
2 12 1 5 1 57
2.6 15.4 1.3 6.4 1.3 73.1
2 12 0 1 1 51
3.0 17.9 0.0 1.5 1.5 76.1
0 11 0 3 1 55
0.0 15.7 0.0 4.3 1.4 78.6
0–10 11–20 21–30 31 or more
95 62 38 20
44.2 28.8 17.7 9.3
36 20 13 9
46.2 25.6 16.7 11.5
34 21 8 4
50.8 31.3 11.9 6.0
25 21 17 7
35.7 30.0 24.3 10.0
Academic rankc Assistant professor Associate professor Full professor Other Sexd
87 48 62 18
40.5 22.3 28.8 8.4
35 15 21 7
44.9 19.2 26.9 9.0
30 16 14 7
44.8 23.9 20.9 10.5
22 17 27 4
31.4 24.3 38.6 5.7
125 90
58.1 41.9
29 49
37.2 62.8
26 41
38.8 61.2
35 35
50.0 50.0
Characteristic Degreea JD MD MD and JD MD and doctorate Master’s Doctorate Years since degreeb
Female Male
SOURCE Authors’ analysis of survey data. NOTES The three experimental groups were assigned one of three versions of a vignette that described different methods of disseminating peer-reviewed research, as explained in the text. Significance refers to associations between participants’ characteristics and experimental-group assignment. ap ¼ 0:80. bp ¼ 0:36. cp ¼ 0:28. dp ¼ 0:24.
July 2014
33:7
Downloaded from content.healthaffairs.org by Health Affairs on August 22, 2015 at University of Sussex
Health A ffairs
1281
Web First Exhibit 2 Survey Participants’ Attitudes Toward Dissemination Methods, 2013 Experimental group Attitude Perceived efficacy of dissemination methoda
Traditional media 6.7
Social media 6.6
Direct outreach to policy makers 7.3
p value 0.06
Confidence in dissemination methoda
7.0
5.4
6.0
