News & Analysis Medical News & Perspectives .........p1781
Lab Reports ............................................p1785
News From the CDC ............................p1786
Striving for a More Perfect Peer Review
Mucus’ Effects on Inflammation May Be Used to Treat Gut Diseases
e-Cigarette Use Grows Among Youths
FDA Lays Out Rules for Regulating Mobile Medical Apps Dietary Supplement Linked to Cases of Acute Hepatitis news@JAMA: From JAMA's Daily News Site
Cellular Immune Response May Be Key to Universal Flu Vaccine Evolution of a Human Pathogen in a New Host Population Scientists Publish Comprehensive Human Urine Metabolome
Legionella Poses Major Challenge in Waterborne Disease Outbreaks Thousands of US Deaths From Stroke and Heart Disease Are Preventable Neural Tube Defect Cluster Stresses Importance of Folic Acid Supplements
Medical News & Perspectives
Striving for a More Perfect Peer Review Editors Confront Strengths, Flaws of Biomedical Literature Bridget M. Kuehn, MSJ
Ted Grudzinski/American Medical Association
C
hicago—“Most results in clinical research are false positives or substantially exaggerated,” said John Ioannidis, MD, to an audience of journal editors from around the world, gathered in September for the Seventh International Congress on Peer Review and Biomedical Publication, a quadrennial research meeting organized by JAMA and the BMJ (http://bit.ly /19gG1C8). His stinging assessment of the peerreviewed literature was one of many reminders that the field of biomedical publishing remains a work in progress nearly 2 and a half decades after meeting organizers launched the Peer Review Congress as a means to promote research into the field and hold one another accountable. Such robust criticism continued throughout the meeting, with researchers and journal editors presenting evidence of bias, failure to publish results or report author conflicts of interest, and sloppy or inaccurate reporting of results. But meeting director and JAMA Contributing Deputy Editor Drummond Rennie, MD, wouldn’t have it any other way. The meeting, he explained, started as a way to subject journals and editors to the same rigorous peer review they are expected to dish out to authors. “The peer review system of organized criticism is beautiful and powerful as an idea,” he said.
advance of the inaugural Peer Review Congress in 1989, he wrote about the importance of researching the peer review process and harshly criticized the status quo (Rennie D. JAMA. 1986;256[17]:23912392): “There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too con-
tradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print.” The meetings, which are held every 4 years, invited academic study of the processes and behavior of journals. For example, presentations at the 1993 meeting examined the quality and training of peer reviewers, ways to handle author
“Guarding the Guardians” When Rennie issued the first call for research on peer review in 1986, in
At a recent meeting in Chicago, John Ioannidis, MD, said most clinical study findings are wrong or much weaker than they appear.
jama.com
Downloaded From: http://jama.jamanetwork.com/ by a Purdue University User on 06/09/2015
JAMA November 6, 2013 Volume 310, Number 17
1781
News & Analysis
Revealing Bias Rennie said the “biggest triumph” of the Peer Review Congress has been to bring together researchers interested in measuring bias, particularly that related to financial conflicts of interest and the powerful effects they can have on the literature. “The really striking and important papers [at the Peer Review Congress] have been on science’s biggest problem, which is bias, intentional and subconscious,” he said. Bringing attention to bias and how it can skew study results allows organizations to put safeguards in place to correct for it, Rennie noted. By identifying and defining such biases, one can warn researchers to allow for them when they design a trial and analyze the results, he said. Greater knowledge of bias and its effects has led to reform. A desire to help ensure consistent and transparent application of safeguards led to the creation of CONSORT (http://bit.ly/14qUNEF), which has developed standards for reporting of trials based on the evidence base that has grown from the Peer Review Congress, and STARD (Standards for the Reporting of Diagnostic Accuracy Studies). “The CONSORT and STARD initiatives have revolutionized the transparent reporting of trials and are being shown to improve the conduct of trials of therapies and tests,” Rennie said. But as Rennie predicted in the 1986 call for research (“No matter how much we improve the system we are unlikely to perfect it,” he wrote), there was no shortage of issues to discuss this year. 1782
Suspect Results Ioannidis, a vocal critic of statistically suspect studies, kicked off the meeting by reminding attendees that the scientific literature is littered with results that are too good to be true. “Most papers look beautiful when you see them,” he said. But he noted that reviewers may not be seeing the real data when they examine studies. Poor practices or perverse incentives in academia may drive publication of questionable studies. An analysis by Ioannidis and Daniele Fanelli found that behavioral studies by US researchers are more likely to report outsized effects compared with similar studies from other countries (Fanelli D and Ioannidis JP. Proc Natl Acad Sci U S A. 2013; 110[37]:15031-15036). Ioannidis suggested at the meeting that perhaps US researchers face more intense pressure to publish. Another questionable practice Ioannidis highlighted is the validation of studies by individuals involved in the original research. “Validation is done by the same people for their own work. We need to think about whether that is the way to go,” he said. Ioannidis called for greater transparency, better reporting of research results, and greater emphasis on reproducibility. He highlighted the work of EQUATOR (http://bit.ly/kYEsm), which is a clearinghouse of information on how to improve
From the news@JAMA blog • Clinicaltrials.gov Database Reveals “Sausage Making” in Clinical Research (http: //jama.md/17Wa4ds) • Scientific Journals Feel the Heat of Peer Review (http://jama.md/1aAjVh5)
reporting of trial results and to audit reporting quality in the literature. Rennie noted that all the methods included in EQUATOR build off the the original CONSORT. Ioannidis also said that registration of clinical trials is an advance, but one that needs continued refinement. For example, he suggested that more types of research, including observational studies, be registered. “I’m a strong supporter of registration for all types of research,” he said at the meeting. In addition, he recommended giving more effort to reproducing study results. He pointed to efforts to promote such validation by the Center for Open Science (http: //bit.ly/158d73R) and the Reproducibility Initiative (http://bit.ly/Pabixg) and noted that he is also creating his own program to improve study validity.
Misconduct Other speakers at the event highlighted evidence of ethical failings by authors or journal editors.
Drummond Rennie, MD, meeting organizer and Contributing Deputy Editor at JAMA, said the Peer Review Congress has helped draw attention to the corrosive effects of bias on the medical literature.
JAMA November 6, 2013 Volume 310, Number 17
Downloaded From: http://jama.jamanetwork.com/ by a Purdue University User on 06/09/2015
jama.com
Ted Grudzinski/American Medical Association
misconduct, and methods for improving statistical reporting and readability (Cotton P. JAMA. 1993;270[23]:2775-2778). The first meeting in 1989 brought together 270 attendees, including editors of such leading journals as the British Medical Journal, The Lancet, JAMA, the New England Journal of Medicine, and Science. This year’s meeting drew more than 500 participants from more than 30 countries, with many of the original organizing journals represented along with a new guard of open-access journals such as those published by the Public Library of Science (PLOS). Social media extended the meeting’s reach even further this year, with more than 70 000 Twitter accounts receiving live updates.
News & Analysis
Elizabeth Wager, PhD, a freelance publications consultant and former chair of the Committee on Publication Ethics (COPE), drew attention to authors who are suspiciously productive. She and her colleagues used a computer program to analyze trends in authorship in the MEDLINE database from 2008 to 2012. They found the maximum number of publications for an author per year to be 43 for all types of studies and 15 for clinicaltrials.However,shenotedthat24ofthe40 mostprolificauthorswerelistedonmorethan 1 article every 10 working days in a year. She said that academic institutions should be as skeptical about overproduction as they are about underproduction, because it raises questionsabouttheauthor’sroleineachofthe publications and casts doubt on the results. Evenwhenproblematicauthorsareidentified,journalsmaybeslowtoreact,suggested
research presented by Martin R. Tramèr, MD, PhD, of the University of Geneva, and coauthoredbyWager.Tramèrandhiscolleaguesexaminedthefateof88articlesbydiscreditedresearcherJoachimBoldt,MD,PhD,whichwere supposed to be retracted. A letter in 2011 publishedin14journalsreportedthatinvestigators failed to find evidence of proper institutional board approval for the studies (http://bit.ly /14qJj41).YetTramèrandcolleaguesfoundthat only 5 of the articles had been fully retracted basedonthestandardsoftheCOPE(http://bit .ly/14qKgcw). Failure to disclose conflicts of interest remains a problem. Kristine Rasmussen, of the Nordic Cochrane Center, analyzed publications by Danish authors who were not employees of drug companies to determine the prevalence of unreported conflicts of interest. She found that 45% of
conflicts of interest related to an author’s relationship with the trial sponsor were not disclosed in the article. Roberta Scherer, PhD, of Johns Hopkins Bloomberg School of Public Health, noted that fewer than half of studies presented at meetings are ever published and that there may be some bias in which studies eventually make it to publication. She noted that authors cited a range of reasons for not publishing, including an expectation that the study would be rejected, lack of time, lack of resources, and difficulties with coauthors. She concluded that the responsibility for nonpublication lies with authors, not journals or editors, and echoed calls from previous Peer Review Congresses that it is unethical not to publish results when individuals participating in studies expect the results to be used to improve medicine.
FDA Lays Out Rules for Regulating Mobile Medical Apps Mike Mitka, MSJ
T
he US Food and Drug Administration (FDA) has issued its final guidance for developers of mobile medical apps being used on smartphones, tablets, and other devices for a variety of healthrelated tasks, from tracking an individual’s fitness to wireless remote control of computed tomography machines. The FDA, citing industry estimates, said that500millionsmartphoneusersworldwide will use a health care app by 2015. Some predict apps will allow physicians to diagnose patients with potentially life-threatening conditionsoutsideoftraditionalhealthcaresettings and help individuals manage their health and wellnessandgainaccesstousefulinformation whenever and wherever they need it. The FDA’s final guidance (http://1.usa .gov/17pXNlI), issued September 23, was eagerly awaited by software developers and investors who were hesitant to jump into the market before understanding the playing field for medical apps. For the majority of medical apps, which pose minimal risk to consumers and patients, the FDA will take a hands-off approach. The agency will regulate apps that are intended to be used as an accessory to a regulated medical device or that transform a mobile platform into a regulated medical device.
“This approach to overseeing mobile medical apps is consistent with our existing approach to overseeing medical device func-
The US Food and Drug Administration will regulate mobile medical apps that turn smartphones and tablets into regulated medical devices or accessories to such medical devices.
tionality of a product and the risks it poses to patients regardless of the shape, size, or the platform,” the FDA noted, saying it will
jama.com
Downloaded From: http://jama.jamanetwork.com/ by a Purdue University User on 06/09/2015
use the same regulatory standards and riskbased approach that the agency applies to other medical devices. Examples of apps that will require FDA regulatory oversight are those that use a sensor or lead connected to a mobile platform to act like an electrocardiograph system or those that use an attachment to the mobile platform to act like an oximeter to measure blood oxygen saturation. Examples of mobile medical apps already cleared or approved by the FDA can be found at http: //tinyurl.com/lo4rax2. “Today’s actions are an important step forward for [mobile] health,” said Jeffrey Shuren, MD, JD, director of the FDA’s Center for Devices and Radiological Health, during a press briefing. “For patients and practitioners, the mobile medical apps guidance should further energize innovative approaches to helping patients and consumers to live healthier lives.” Agreeing with Shuren’s statement was David Collins, MHA, of the Healthcare Information and Management Systems Society (HIMSS), a not-for-profit organization comprising more than 52 000 individuals, a majority of whom work in health care provider, governmental, and not-for-profit organizations around the world, plus more JAMA November 6, 2013 Volume 310, Number 17
1783
Lab Reports ............................................p1785
News From the CDC ............................p1786
Striving for a More Perfect Peer Review
Mucus’ Effects on Inflammation May Be Used to Treat Gut Diseases
e-Cigarette Use Grows Among Youths
FDA Lays Out Rules for Regulating Mobile Medical Apps Dietary Supplement Linked to Cases of Acute Hepatitis news@JAMA: From JAMA's Daily News Site
Cellular Immune Response May Be Key to Universal Flu Vaccine Evolution of a Human Pathogen in a New Host Population Scientists Publish Comprehensive Human Urine Metabolome
Legionella Poses Major Challenge in Waterborne Disease Outbreaks Thousands of US Deaths From Stroke and Heart Disease Are Preventable Neural Tube Defect Cluster Stresses Importance of Folic Acid Supplements
Medical News & Perspectives
Striving for a More Perfect Peer Review Editors Confront Strengths, Flaws of Biomedical Literature Bridget M. Kuehn, MSJ
Ted Grudzinski/American Medical Association
C
hicago—“Most results in clinical research are false positives or substantially exaggerated,” said John Ioannidis, MD, to an audience of journal editors from around the world, gathered in September for the Seventh International Congress on Peer Review and Biomedical Publication, a quadrennial research meeting organized by JAMA and the BMJ (http://bit.ly /19gG1C8). His stinging assessment of the peerreviewed literature was one of many reminders that the field of biomedical publishing remains a work in progress nearly 2 and a half decades after meeting organizers launched the Peer Review Congress as a means to promote research into the field and hold one another accountable. Such robust criticism continued throughout the meeting, with researchers and journal editors presenting evidence of bias, failure to publish results or report author conflicts of interest, and sloppy or inaccurate reporting of results. But meeting director and JAMA Contributing Deputy Editor Drummond Rennie, MD, wouldn’t have it any other way. The meeting, he explained, started as a way to subject journals and editors to the same rigorous peer review they are expected to dish out to authors. “The peer review system of organized criticism is beautiful and powerful as an idea,” he said.
advance of the inaugural Peer Review Congress in 1989, he wrote about the importance of researching the peer review process and harshly criticized the status quo (Rennie D. JAMA. 1986;256[17]:23912392): “There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too con-
tradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print.” The meetings, which are held every 4 years, invited academic study of the processes and behavior of journals. For example, presentations at the 1993 meeting examined the quality and training of peer reviewers, ways to handle author
“Guarding the Guardians” When Rennie issued the first call for research on peer review in 1986, in
At a recent meeting in Chicago, John Ioannidis, MD, said most clinical study findings are wrong or much weaker than they appear.
jama.com
Downloaded From: http://jama.jamanetwork.com/ by a Purdue University User on 06/09/2015
JAMA November 6, 2013 Volume 310, Number 17
1781
News & Analysis
Revealing Bias Rennie said the “biggest triumph” of the Peer Review Congress has been to bring together researchers interested in measuring bias, particularly that related to financial conflicts of interest and the powerful effects they can have on the literature. “The really striking and important papers [at the Peer Review Congress] have been on science’s biggest problem, which is bias, intentional and subconscious,” he said. Bringing attention to bias and how it can skew study results allows organizations to put safeguards in place to correct for it, Rennie noted. By identifying and defining such biases, one can warn researchers to allow for them when they design a trial and analyze the results, he said. Greater knowledge of bias and its effects has led to reform. A desire to help ensure consistent and transparent application of safeguards led to the creation of CONSORT (http://bit.ly/14qUNEF), which has developed standards for reporting of trials based on the evidence base that has grown from the Peer Review Congress, and STARD (Standards for the Reporting of Diagnostic Accuracy Studies). “The CONSORT and STARD initiatives have revolutionized the transparent reporting of trials and are being shown to improve the conduct of trials of therapies and tests,” Rennie said. But as Rennie predicted in the 1986 call for research (“No matter how much we improve the system we are unlikely to perfect it,” he wrote), there was no shortage of issues to discuss this year. 1782
Suspect Results Ioannidis, a vocal critic of statistically suspect studies, kicked off the meeting by reminding attendees that the scientific literature is littered with results that are too good to be true. “Most papers look beautiful when you see them,” he said. But he noted that reviewers may not be seeing the real data when they examine studies. Poor practices or perverse incentives in academia may drive publication of questionable studies. An analysis by Ioannidis and Daniele Fanelli found that behavioral studies by US researchers are more likely to report outsized effects compared with similar studies from other countries (Fanelli D and Ioannidis JP. Proc Natl Acad Sci U S A. 2013; 110[37]:15031-15036). Ioannidis suggested at the meeting that perhaps US researchers face more intense pressure to publish. Another questionable practice Ioannidis highlighted is the validation of studies by individuals involved in the original research. “Validation is done by the same people for their own work. We need to think about whether that is the way to go,” he said. Ioannidis called for greater transparency, better reporting of research results, and greater emphasis on reproducibility. He highlighted the work of EQUATOR (http://bit.ly/kYEsm), which is a clearinghouse of information on how to improve
From the news@JAMA blog • Clinicaltrials.gov Database Reveals “Sausage Making” in Clinical Research (http: //jama.md/17Wa4ds) • Scientific Journals Feel the Heat of Peer Review (http://jama.md/1aAjVh5)
reporting of trial results and to audit reporting quality in the literature. Rennie noted that all the methods included in EQUATOR build off the the original CONSORT. Ioannidis also said that registration of clinical trials is an advance, but one that needs continued refinement. For example, he suggested that more types of research, including observational studies, be registered. “I’m a strong supporter of registration for all types of research,” he said at the meeting. In addition, he recommended giving more effort to reproducing study results. He pointed to efforts to promote such validation by the Center for Open Science (http: //bit.ly/158d73R) and the Reproducibility Initiative (http://bit.ly/Pabixg) and noted that he is also creating his own program to improve study validity.
Misconduct Other speakers at the event highlighted evidence of ethical failings by authors or journal editors.
Drummond Rennie, MD, meeting organizer and Contributing Deputy Editor at JAMA, said the Peer Review Congress has helped draw attention to the corrosive effects of bias on the medical literature.
JAMA November 6, 2013 Volume 310, Number 17
Downloaded From: http://jama.jamanetwork.com/ by a Purdue University User on 06/09/2015
jama.com
Ted Grudzinski/American Medical Association
misconduct, and methods for improving statistical reporting and readability (Cotton P. JAMA. 1993;270[23]:2775-2778). The first meeting in 1989 brought together 270 attendees, including editors of such leading journals as the British Medical Journal, The Lancet, JAMA, the New England Journal of Medicine, and Science. This year’s meeting drew more than 500 participants from more than 30 countries, with many of the original organizing journals represented along with a new guard of open-access journals such as those published by the Public Library of Science (PLOS). Social media extended the meeting’s reach even further this year, with more than 70 000 Twitter accounts receiving live updates.
News & Analysis
Elizabeth Wager, PhD, a freelance publications consultant and former chair of the Committee on Publication Ethics (COPE), drew attention to authors who are suspiciously productive. She and her colleagues used a computer program to analyze trends in authorship in the MEDLINE database from 2008 to 2012. They found the maximum number of publications for an author per year to be 43 for all types of studies and 15 for clinicaltrials.However,shenotedthat24ofthe40 mostprolificauthorswerelistedonmorethan 1 article every 10 working days in a year. She said that academic institutions should be as skeptical about overproduction as they are about underproduction, because it raises questionsabouttheauthor’sroleineachofthe publications and casts doubt on the results. Evenwhenproblematicauthorsareidentified,journalsmaybeslowtoreact,suggested
research presented by Martin R. Tramèr, MD, PhD, of the University of Geneva, and coauthoredbyWager.Tramèrandhiscolleaguesexaminedthefateof88articlesbydiscreditedresearcherJoachimBoldt,MD,PhD,whichwere supposed to be retracted. A letter in 2011 publishedin14journalsreportedthatinvestigators failed to find evidence of proper institutional board approval for the studies (http://bit.ly /14qJj41).YetTramèrandcolleaguesfoundthat only 5 of the articles had been fully retracted basedonthestandardsoftheCOPE(http://bit .ly/14qKgcw). Failure to disclose conflicts of interest remains a problem. Kristine Rasmussen, of the Nordic Cochrane Center, analyzed publications by Danish authors who were not employees of drug companies to determine the prevalence of unreported conflicts of interest. She found that 45% of
conflicts of interest related to an author’s relationship with the trial sponsor were not disclosed in the article. Roberta Scherer, PhD, of Johns Hopkins Bloomberg School of Public Health, noted that fewer than half of studies presented at meetings are ever published and that there may be some bias in which studies eventually make it to publication. She noted that authors cited a range of reasons for not publishing, including an expectation that the study would be rejected, lack of time, lack of resources, and difficulties with coauthors. She concluded that the responsibility for nonpublication lies with authors, not journals or editors, and echoed calls from previous Peer Review Congresses that it is unethical not to publish results when individuals participating in studies expect the results to be used to improve medicine.
FDA Lays Out Rules for Regulating Mobile Medical Apps Mike Mitka, MSJ
T
he US Food and Drug Administration (FDA) has issued its final guidance for developers of mobile medical apps being used on smartphones, tablets, and other devices for a variety of healthrelated tasks, from tracking an individual’s fitness to wireless remote control of computed tomography machines. The FDA, citing industry estimates, said that500millionsmartphoneusersworldwide will use a health care app by 2015. Some predict apps will allow physicians to diagnose patients with potentially life-threatening conditionsoutsideoftraditionalhealthcaresettings and help individuals manage their health and wellnessandgainaccesstousefulinformation whenever and wherever they need it. The FDA’s final guidance (http://1.usa .gov/17pXNlI), issued September 23, was eagerly awaited by software developers and investors who were hesitant to jump into the market before understanding the playing field for medical apps. For the majority of medical apps, which pose minimal risk to consumers and patients, the FDA will take a hands-off approach. The agency will regulate apps that are intended to be used as an accessory to a regulated medical device or that transform a mobile platform into a regulated medical device.
“This approach to overseeing mobile medical apps is consistent with our existing approach to overseeing medical device func-
The US Food and Drug Administration will regulate mobile medical apps that turn smartphones and tablets into regulated medical devices or accessories to such medical devices.
tionality of a product and the risks it poses to patients regardless of the shape, size, or the platform,” the FDA noted, saying it will
jama.com
Downloaded From: http://jama.jamanetwork.com/ by a Purdue University User on 06/09/2015
use the same regulatory standards and riskbased approach that the agency applies to other medical devices. Examples of apps that will require FDA regulatory oversight are those that use a sensor or lead connected to a mobile platform to act like an electrocardiograph system or those that use an attachment to the mobile platform to act like an oximeter to measure blood oxygen saturation. Examples of mobile medical apps already cleared or approved by the FDA can be found at http: //tinyurl.com/lo4rax2. “Today’s actions are an important step forward for [mobile] health,” said Jeffrey Shuren, MD, JD, director of the FDA’s Center for Devices and Radiological Health, during a press briefing. “For patients and practitioners, the mobile medical apps guidance should further energize innovative approaches to helping patients and consumers to live healthier lives.” Agreeing with Shuren’s statement was David Collins, MHA, of the Healthcare Information and Management Systems Society (HIMSS), a not-for-profit organization comprising more than 52 000 individuals, a majority of whom work in health care provider, governmental, and not-for-profit organizations around the world, plus more JAMA November 6, 2013 Volume 310, Number 17
1783
