VOLUME
32
䡠
NUMBER
6
䡠
FEBRUARY
20
2014
JOURNAL OF CLINICAL ONCOLOGY
C O R R E S P O N D E N C E
Public Database of Federally Funded Clinical Trials Data TO THE EDITOR: I read with interest the article by Nguyen et al1 and the accompanying editorial by Prasad and Bennett2 documenting the underreporting of cancer clinical trials in the United States. Nguyen et al lay out reasons why “the entire system” works to delay reporting of these results by academic authors. I propose that in the current Era of Big Data, the United States can do better than simply requiring that authors declare results. Immense amounts of high-quality data are collected in the course of these studies, and the reporting of end points reveals only one facet of the whole. To comply with the US Food and Drug Administration, study authors should turn over the mature data set in its entirety. Formal analysis of this data can come before, at the same time, or later. A data repository from the multitudes of intergroup cancer studies would be a phenomenal resource. The analogy here is the SEER database maintained by the National Cancer Institute (NCI). SEER offers easy tools for interested lay people to learn facts about cancer epidemiology. The SEERSTAT program is available for dedicated researchers. One can download the entire database. SEER data has added value to many other research endeavors and given rise to many other projects. The NCI could set up a standard format for the raw data from intergroup and other federally funded studies, to allow research endeavors that span multiple datasets. There could be research tools
analogous to SEER. Once the format and tools are established, other owners of high-quality clinical trial data might also deposit their datasets with the NCI to be added to those of the intergroups—these might include European studies, pharmaceutical studies, and so on. Federal funding for a study means that the clinical data produced by the study should not be secret. The promise of big data and free information in the age of connectivity is the promise of the mass mind. There are many, many smart people in the world. There are important truths which become evident when data can be studied by anyone interested. A public database of clinical trial data would maximize the contributions of clinical trial participants, and validate their altruistic hopes that their participation could improve the world for those who come after.
William Read Winship Cancer Institute, Atlanta, GA
AUTHOR’S DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST
The author(s) indicated no potential conflicts of interest. REFERENCES 1. Nguyen T-A-H, Dechartres A, Belgherbi S, et al: Public availability of results of trials assessing cancer drugs in the United States. J Clin Oncol 31:2998-3003, 2013. 2. Prasad SM, Bennett CL: Finishing the picture: Problems with public reporting of clinical trials. J Clin Oncol 31:2981-2982, 2013
DOI: 10.1200/JCO.2013.53.2358; published online ahead of print at www.jco.org on January 6, 2014
■ ■ ■
Reply to W. Read We read with interest the comment by Read1 and fully agree with his proposition to create a national clinical trials database including raw data in a standard format from all academic clinical trials performed in the United States. Such a project will serve the whole scientific community as well as patients. The availability of raw data ensures complete transparency and allows for performing additional research projects on existing data without data collection cost. Data sharing is crucial in an era of personalized medicine and individual patient data meta-analyses. The idea of publishing raw data is not new. In the first issue of Biometrika, in 1901, Galton proposed to make available a wellarranged and well-bound copy of data in some place where the data could be accessed under reasonable conditions.2,3 However, many researchers are reluctant to release their raw data,4-6 despite incentives from researchers5-7 and medical journal editors.8,9 Survey results published in 2009 showed that only one in 10 contacted investigators who had published in a PLOS journal agreed to share their raw data, despite the clear data-sharing policy of the PLOS journals.10 Things are moving ahead, with a growing tendency for data sharing among academic sponJournal of Clinical Oncology, Vol 32, No 6 (February 20), 2014: pp 603-609
sors, health regulatory agencies, and some pharmaceutical companies.11-14 Nevertheless, practical and ethical aspects of data sharing are challenging. A first point is related to data archiving, because data are frequently not archived in a sustainable format and are poorly described using nonstandard terminology. The development of a repository would allow for centrally archiving raw data and harmonizing data in terms of structure, formatting and annotation but this has a cost so who is going to fund it? The major challenge of data sharing is protection of patient confidentiality and anonymity. A minimum standard for ensuring adequate anonymity in datasets was recently proposed,15 but for public availability of anonymized data, patient consent should be obtained in trials,9,15 which can be difficult in practice. Then, concerning data sharing itself, important questions debated are who should have access? and how? A recent article proposed and discussed the pros and cons of four possible models for expanded access to participantlevel data: (1) complete open access for everyone, (2) submission of queries to data holders who would run the analyses, (3) review of the request by the trial sponsor, and (4) review of the request by an independent review board.16 The availability of raw data is the ultimate step in transparency.17 Nevertheless, one should remember that not all trials are © 2014 by American Society of Clinical Oncology
Information downloaded from jco.ascopubs.org and provided by at UNIVERSITY OTAGO on October 4, 2014 from Copyright © 2014 American Society of Clinical Oncology. All rights reserved. 139.80.149.115
603
Correspondence
registered, have published results or have their results posted. Compliance with mandatory posting of results, as required by the Food and Drug Administration Amendments Act, is low.18,19 Moreover, this law concerns trials performed in the United States, with no similar law in Europe or elsewhere. So, before the ultimate step of data sharing, considerable improvement in transparency could be achieved by all trials being registered and their results available in journals or posted in clinical trial registries, as required by the All Trials campaign.20 In the United States where posting trial results is mandatory, efforts should be made to more rigorously apply the law. Furthermore, we advocate extending the mandatory posting of results for all clinical trials worldwide.
Agnes Dechartres INSERM U738; Universite´ Paris Descartes—Sorbonne Paris Cite´; Assistance Publique-Hoˆpitaux de Paris, Hoˆpital Hoˆtel-Dieu, Centre d’Epide´miologie Clinique, Paris, France
Philippe Ravaud INSERM U738; Universite´ Paris Descartes—Sorbonne Paris Cite´; Assistance Publique-Hoˆpitaux de Paris, Hoˆpital Hoˆtel-Dieu, Centre d’Epide´miologie Clinique; French Cochrane Centre, Paris, France; Columbia University, Mailman School of Public Health, New York, NY
ACKNOWLEDGMENT
Supported by the academic Grant No. DEQ20101221475, program “Equipe espoir de la Recherche,” Fondation pour la Recherche Me´dicale, Paris, France. AUTHORS’ DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST
The author(s) indicated no potential conflicts of interest.
2. Perneger TV: Sharing raw data: Another of Francis Galton’s ideas. BMJ 342:d3035, 2011 3. Galton F: Biometry. Biometrika 1:7-10, 1901 4. Kirwan JR: Making original data from clinical studies available for alternative analysis. J Rheumatol 24:822-825, 1997 5. Vickers AJ: Whose data set is it anyway? Sharing raw data from randomized trials. Trials 7:15, 2006 6. Vickers AJ: Making raw data more widely available. BMJ 342:d2323, 2011 7. Gotzsche PC: Strengthening and opening up health research by sharing our raw data. Circ Cardiovasc Qual Outcomes 5:236-237, 2012 8. Godlee F, Groves T: The new BMJ policy on sharing data from drug and device trials. BMJ 345:e7888, 2012 9. Groves T: Sharing the raw data from medical research [corrected]. BMJ 338:b1252, 2009 10. Savage CJ, Vickers AJ: Empirical study of data sharing by authors publishing in PLoS journals. PLoS One 4:e7078, 2009 11. Ross JS, Krumholz HM: Ushering in a new era of open science through data sharing: The wall must come down. JAMA 309:1355-1356, 2013 12. Zarin DA: Participant-level data and the new frontier in trial transparency. N Engl J Med 369:468-469, 2013 13. European Medicines Agency. Release of data from clinical trials. http:// www.ema.europa.eu/ema/index.jsp?curl⫽pages/special_topics/general/ general_content_000556.jsp 14. Nisen P, Rockhold F: Access to patient-level data from GlaxoSmithKline clinical trials. N Engl J Med 369:475-478, 2013 15. Hrynaszkiewicz I, Norton ML, Vickers AJ, et al: Preparing raw clinical data for publication: Guidance for journal editors, authors, and peer reviewers. BMJ 340:c181, 2010 16. Mello MM, Francer JK, Wilenzick M, et al: Preparing for responsible sharing of clinical trial data. N Engl J Med 369:1651-1658, 2013 17. Zarin DA, Tse T: Medicine: Moving toward transparency of clinical trials. Science 319:1340-1342, 2008 18. Nguyen TA, Dechartres A, Belgherbi S, et al: Public availability of results of trials assessing cancer drugs in the United States. J Clin Oncol 31:2998-3003, 2013 19. Prayle AP, Hurley MN, Smyth AR: Compliance with mandatory reporting of clinical trial results on ClinicalTrials.gov: Cross sectional study. BMJ 344:d7373, 2012 20. Sense About Science: All Trials Registered/All Results Reported. 2013 www.alltrials.net
REFERENCES 1. Read W: Public database of federally funded clinical trials data. J Clin Oncol 32:603, 2014
DOI: 10.1200/JCO.2013.53.7217; published online ahead of print at www.jco.org on January 6, 2014
■ ■ ■
Reply to W. Read We read with interest the proposal from Read1 regarding his proposal for a National Clinical Trials Database containing mature datasets from clinical trials with federal funding after his reading of the article from Nguyen et al2 and our accompanying editorial.3 Currently, these datasets are maintained by a dizzying number of organizations and individual researchers.4 While we agree that data funded by public dollars should be made publically available, the ideal shape, structure, and format of such a database are not entirely intuitive and require foresight, planning, and thought.5 The ability to query, search and download the multiple different data sets is not an inconsequential task and a standard format would need to be considered to facilitate use of such a tool. In addition, protection of patient confidentiality remains essential to study participants. Read1 notes the example of the SEER database as an analogous endeavor, but like other government-sponsored data sets including the National Health Interview Survey, Medical Expenditure Panel Survey, and the Health Care Cost and Utilization Project, SEER benefits from central budgets, organization, and effort. We would offer the example of the development of the Human Genome Project as a model of publically shared and easily navigable Big Data generated 604
© 2014 by American Society of Clinical Oncology
from public and private endeavors. Currently, DNA sequencing data is available to anyone on the Internet through the unrestricted public GenBank database,6 and many for-profit and nonprofit organizations have developed tools to search and interpret these data to optimize utility. While public and private databases represent separate challenges, this distinction is likely to be smaller in the future. For example, publication of individual patient data from clinical trials where the relevant information is held by the European Medicines Agency (EMA) is moving forward, with advocates requesting open access to these data based on the patients’ altruism and skepticism over the completeness of published trial information.7 In June 2013, the EMA proposed a draft policy making patient-level data in the possession of the EMA publicly accessible.8 The importance of full reporting has been emphasized in a recent study noting that unpublished clinical study reports provide significantly more information regarding patient-related benefits and harms compared with journal publications or registry reports.9 The EMA reports that access to full data sets of completed trials will lead to improved design of subsequent trials, facilitate reporting of comprehensive meta-analyses of patient-level databases, improve a drug’s value in the marketplace, facilitate comparative effectiveness research efforts, and minimize the likelihood of JOURNAL OF CLINICAL ONCOLOGY
Information downloaded from jco.ascopubs.org and provided by at UNIVERSITY OTAGO on October 4, 2014 from Copyright © 2014 American Society of Clinical Oncology. All rights reserved. 139.80.149.115
32
䡠
NUMBER
6
䡠
FEBRUARY
20
2014
JOURNAL OF CLINICAL ONCOLOGY
C O R R E S P O N D E N C E
Public Database of Federally Funded Clinical Trials Data TO THE EDITOR: I read with interest the article by Nguyen et al1 and the accompanying editorial by Prasad and Bennett2 documenting the underreporting of cancer clinical trials in the United States. Nguyen et al lay out reasons why “the entire system” works to delay reporting of these results by academic authors. I propose that in the current Era of Big Data, the United States can do better than simply requiring that authors declare results. Immense amounts of high-quality data are collected in the course of these studies, and the reporting of end points reveals only one facet of the whole. To comply with the US Food and Drug Administration, study authors should turn over the mature data set in its entirety. Formal analysis of this data can come before, at the same time, or later. A data repository from the multitudes of intergroup cancer studies would be a phenomenal resource. The analogy here is the SEER database maintained by the National Cancer Institute (NCI). SEER offers easy tools for interested lay people to learn facts about cancer epidemiology. The SEERSTAT program is available for dedicated researchers. One can download the entire database. SEER data has added value to many other research endeavors and given rise to many other projects. The NCI could set up a standard format for the raw data from intergroup and other federally funded studies, to allow research endeavors that span multiple datasets. There could be research tools
analogous to SEER. Once the format and tools are established, other owners of high-quality clinical trial data might also deposit their datasets with the NCI to be added to those of the intergroups—these might include European studies, pharmaceutical studies, and so on. Federal funding for a study means that the clinical data produced by the study should not be secret. The promise of big data and free information in the age of connectivity is the promise of the mass mind. There are many, many smart people in the world. There are important truths which become evident when data can be studied by anyone interested. A public database of clinical trial data would maximize the contributions of clinical trial participants, and validate their altruistic hopes that their participation could improve the world for those who come after.
William Read Winship Cancer Institute, Atlanta, GA
AUTHOR’S DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST
The author(s) indicated no potential conflicts of interest. REFERENCES 1. Nguyen T-A-H, Dechartres A, Belgherbi S, et al: Public availability of results of trials assessing cancer drugs in the United States. J Clin Oncol 31:2998-3003, 2013. 2. Prasad SM, Bennett CL: Finishing the picture: Problems with public reporting of clinical trials. J Clin Oncol 31:2981-2982, 2013
DOI: 10.1200/JCO.2013.53.2358; published online ahead of print at www.jco.org on January 6, 2014
■ ■ ■
Reply to W. Read We read with interest the comment by Read1 and fully agree with his proposition to create a national clinical trials database including raw data in a standard format from all academic clinical trials performed in the United States. Such a project will serve the whole scientific community as well as patients. The availability of raw data ensures complete transparency and allows for performing additional research projects on existing data without data collection cost. Data sharing is crucial in an era of personalized medicine and individual patient data meta-analyses. The idea of publishing raw data is not new. In the first issue of Biometrika, in 1901, Galton proposed to make available a wellarranged and well-bound copy of data in some place where the data could be accessed under reasonable conditions.2,3 However, many researchers are reluctant to release their raw data,4-6 despite incentives from researchers5-7 and medical journal editors.8,9 Survey results published in 2009 showed that only one in 10 contacted investigators who had published in a PLOS journal agreed to share their raw data, despite the clear data-sharing policy of the PLOS journals.10 Things are moving ahead, with a growing tendency for data sharing among academic sponJournal of Clinical Oncology, Vol 32, No 6 (February 20), 2014: pp 603-609
sors, health regulatory agencies, and some pharmaceutical companies.11-14 Nevertheless, practical and ethical aspects of data sharing are challenging. A first point is related to data archiving, because data are frequently not archived in a sustainable format and are poorly described using nonstandard terminology. The development of a repository would allow for centrally archiving raw data and harmonizing data in terms of structure, formatting and annotation but this has a cost so who is going to fund it? The major challenge of data sharing is protection of patient confidentiality and anonymity. A minimum standard for ensuring adequate anonymity in datasets was recently proposed,15 but for public availability of anonymized data, patient consent should be obtained in trials,9,15 which can be difficult in practice. Then, concerning data sharing itself, important questions debated are who should have access? and how? A recent article proposed and discussed the pros and cons of four possible models for expanded access to participantlevel data: (1) complete open access for everyone, (2) submission of queries to data holders who would run the analyses, (3) review of the request by the trial sponsor, and (4) review of the request by an independent review board.16 The availability of raw data is the ultimate step in transparency.17 Nevertheless, one should remember that not all trials are © 2014 by American Society of Clinical Oncology
Information downloaded from jco.ascopubs.org and provided by at UNIVERSITY OTAGO on October 4, 2014 from Copyright © 2014 American Society of Clinical Oncology. All rights reserved. 139.80.149.115
603
Correspondence
registered, have published results or have their results posted. Compliance with mandatory posting of results, as required by the Food and Drug Administration Amendments Act, is low.18,19 Moreover, this law concerns trials performed in the United States, with no similar law in Europe or elsewhere. So, before the ultimate step of data sharing, considerable improvement in transparency could be achieved by all trials being registered and their results available in journals or posted in clinical trial registries, as required by the All Trials campaign.20 In the United States where posting trial results is mandatory, efforts should be made to more rigorously apply the law. Furthermore, we advocate extending the mandatory posting of results for all clinical trials worldwide.
Agnes Dechartres INSERM U738; Universite´ Paris Descartes—Sorbonne Paris Cite´; Assistance Publique-Hoˆpitaux de Paris, Hoˆpital Hoˆtel-Dieu, Centre d’Epide´miologie Clinique, Paris, France
Philippe Ravaud INSERM U738; Universite´ Paris Descartes—Sorbonne Paris Cite´; Assistance Publique-Hoˆpitaux de Paris, Hoˆpital Hoˆtel-Dieu, Centre d’Epide´miologie Clinique; French Cochrane Centre, Paris, France; Columbia University, Mailman School of Public Health, New York, NY
ACKNOWLEDGMENT
Supported by the academic Grant No. DEQ20101221475, program “Equipe espoir de la Recherche,” Fondation pour la Recherche Me´dicale, Paris, France. AUTHORS’ DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST
The author(s) indicated no potential conflicts of interest.
2. Perneger TV: Sharing raw data: Another of Francis Galton’s ideas. BMJ 342:d3035, 2011 3. Galton F: Biometry. Biometrika 1:7-10, 1901 4. Kirwan JR: Making original data from clinical studies available for alternative analysis. J Rheumatol 24:822-825, 1997 5. Vickers AJ: Whose data set is it anyway? Sharing raw data from randomized trials. Trials 7:15, 2006 6. Vickers AJ: Making raw data more widely available. BMJ 342:d2323, 2011 7. Gotzsche PC: Strengthening and opening up health research by sharing our raw data. Circ Cardiovasc Qual Outcomes 5:236-237, 2012 8. Godlee F, Groves T: The new BMJ policy on sharing data from drug and device trials. BMJ 345:e7888, 2012 9. Groves T: Sharing the raw data from medical research [corrected]. BMJ 338:b1252, 2009 10. Savage CJ, Vickers AJ: Empirical study of data sharing by authors publishing in PLoS journals. PLoS One 4:e7078, 2009 11. Ross JS, Krumholz HM: Ushering in a new era of open science through data sharing: The wall must come down. JAMA 309:1355-1356, 2013 12. Zarin DA: Participant-level data and the new frontier in trial transparency. N Engl J Med 369:468-469, 2013 13. European Medicines Agency. Release of data from clinical trials. http:// www.ema.europa.eu/ema/index.jsp?curl⫽pages/special_topics/general/ general_content_000556.jsp 14. Nisen P, Rockhold F: Access to patient-level data from GlaxoSmithKline clinical trials. N Engl J Med 369:475-478, 2013 15. Hrynaszkiewicz I, Norton ML, Vickers AJ, et al: Preparing raw clinical data for publication: Guidance for journal editors, authors, and peer reviewers. BMJ 340:c181, 2010 16. Mello MM, Francer JK, Wilenzick M, et al: Preparing for responsible sharing of clinical trial data. N Engl J Med 369:1651-1658, 2013 17. Zarin DA, Tse T: Medicine: Moving toward transparency of clinical trials. Science 319:1340-1342, 2008 18. Nguyen TA, Dechartres A, Belgherbi S, et al: Public availability of results of trials assessing cancer drugs in the United States. J Clin Oncol 31:2998-3003, 2013 19. Prayle AP, Hurley MN, Smyth AR: Compliance with mandatory reporting of clinical trial results on ClinicalTrials.gov: Cross sectional study. BMJ 344:d7373, 2012 20. Sense About Science: All Trials Registered/All Results Reported. 2013 www.alltrials.net
REFERENCES 1. Read W: Public database of federally funded clinical trials data. J Clin Oncol 32:603, 2014
DOI: 10.1200/JCO.2013.53.7217; published online ahead of print at www.jco.org on January 6, 2014
■ ■ ■
Reply to W. Read We read with interest the proposal from Read1 regarding his proposal for a National Clinical Trials Database containing mature datasets from clinical trials with federal funding after his reading of the article from Nguyen et al2 and our accompanying editorial.3 Currently, these datasets are maintained by a dizzying number of organizations and individual researchers.4 While we agree that data funded by public dollars should be made publically available, the ideal shape, structure, and format of such a database are not entirely intuitive and require foresight, planning, and thought.5 The ability to query, search and download the multiple different data sets is not an inconsequential task and a standard format would need to be considered to facilitate use of such a tool. In addition, protection of patient confidentiality remains essential to study participants. Read1 notes the example of the SEER database as an analogous endeavor, but like other government-sponsored data sets including the National Health Interview Survey, Medical Expenditure Panel Survey, and the Health Care Cost and Utilization Project, SEER benefits from central budgets, organization, and effort. We would offer the example of the development of the Human Genome Project as a model of publically shared and easily navigable Big Data generated 604
© 2014 by American Society of Clinical Oncology
from public and private endeavors. Currently, DNA sequencing data is available to anyone on the Internet through the unrestricted public GenBank database,6 and many for-profit and nonprofit organizations have developed tools to search and interpret these data to optimize utility. While public and private databases represent separate challenges, this distinction is likely to be smaller in the future. For example, publication of individual patient data from clinical trials where the relevant information is held by the European Medicines Agency (EMA) is moving forward, with advocates requesting open access to these data based on the patients’ altruism and skepticism over the completeness of published trial information.7 In June 2013, the EMA proposed a draft policy making patient-level data in the possession of the EMA publicly accessible.8 The importance of full reporting has been emphasized in a recent study noting that unpublished clinical study reports provide significantly more information regarding patient-related benefits and harms compared with journal publications or registry reports.9 The EMA reports that access to full data sets of completed trials will lead to improved design of subsequent trials, facilitate reporting of comprehensive meta-analyses of patient-level databases, improve a drug’s value in the marketplace, facilitate comparative effectiveness research efforts, and minimize the likelihood of JOURNAL OF CLINICAL ONCOLOGY
Information downloaded from jco.ascopubs.org and provided by at UNIVERSITY OTAGO on October 4, 2014 from Copyright © 2014 American Society of Clinical Oncology. All rights reserved. 139.80.149.115
