HHS Public Access Author manuscript Author Manuscript
J Bone Miner Res. Author manuscript; available in PMC 2017 July 01. Published in final edited form as: J Bone Miner Res. 2016 July ; 31(7): 1317–1319. doi:10.1002/jbmr.2881.
The Road to Reproducibility in Animal Research Robert L. Jilka Center for Osteoporosis and Metabolic Bone Diseases, Division of Endocrinology and Metabolism, University of Arkansas for Medical Sciences, Little Rock, AR 72205, USA Robert L. Jilka: [email protected]
Author Manuscript
Abstract Reproducibility of research findings is the hallmark of scientific advance. However, the recently noted lack of reproducibility and transparency of published research using animal models of human biology and disease has alarmed funders, scientists and the public. Improved reporting of methodology and better use of statistical tools are needed to enhance the quality and utility of published research. Reporting guidelines like ARRIVE have been devised to achieve these goals, but most biomedical research journals, including JBMR, have not been able to obtain high compliance. Co-operative efforts among authors, reviewers and editors - empowered by increased awareness of their responsibilities, and enabled by user-friendly guidelines – are needed to solve this problem.
Author Manuscript
Introduction
Author Manuscript
Highly publicized reports of poor reproducibility of published research using animal models of human biology and disease have alarmed funders, the pharmaceutical industry, the public, and the scientific community. Manolagas and Kronenberg have called attention to this issue, and described its negative impact on the field of bone and mineral research (1). Indeed, reproducibility is the hallmark of scientific advance, but this process is hindered by inadequate or opaque reporting of methodology (2), and misapplication of statistical tools like P values (3). These problems must be addressed because advances in the understanding of human biology and disease will increasingly depend on well-defined animal models. For example, assembly of large data sets containing detailed information on the genome, and transcriptome and phenome of specific mouse strains has increased understanding of mouse models of human disease, and will likely provide clues to the design of better models (4,5). Moreover, the ease and wide availability of CRISPR/Cas9 gene editing methodology (6) has accelerated the development of genetically modified mice, and greatly facilitates genetic modification of other species. Part of the reproducibility problem may be ascribed to that fact that modern biomedical research increasingly requires interdisciplinary teams of scientists. Such teams are particularly important in bone and mineral research because of the complex array of specialized cells in bone, the diverse microenvironments in which they and their progenitors Disclosures The author states that he has no conflicts of interest.
Jilka
Page 2
Author Manuscript
reside, and the involvement of endocrine as well as local factors in the regulation of these cells. Nevertheless, team members inevitably bring diverse clinical, animal, and basic research training and experience to their task, and therefore varying standards for analyzing and communicating their work. Regardless of the size of the research team, advances in biomedical knowledge best occurs when all laboratories investigating a particular problem follow the same publication standards.
Animal Research: Reporting In Vivo Experiments (ARRIVE)
Author Manuscript
The ARRIVE guidelines were developed in 2011 to aid investigators in the design, analysis and publication of animal experiments in manner that ensures, as far as practicable, transparency and reproducibility (7). ARRIVE focuses on details such as the characteristics of the animal model, its relevance to human biology and disease, as well as aspects of animal health and husbandry, experimental design and methodology, and statistical methods. Based on conversations with my scientific colleagues, JBMR Editor-in-Chief Juliet Compston, ASBMR Publications staff, and the ASBMR Publications Committee (of which I was Chair), a working group was formed in early 2014 to identify guidelines for the publication of animal-based bone and mineral research. This group recommended that JBMR adopt the ARRIVE guidelines to aid authors in reporting their findings, and to provide a tool for reviewers and editors to assess whether the standards espoused by the guidelines are being met in submitted manuscripts.
Author Manuscript
In October of 2014, JBMR announced that a completed ARRIVE checklist must accompany all submitted manuscripts reporting studies using animals. The announcement was accompanied by the Manolagas and Kronenberg Perspective, which also explained and endorsed ARRIVE (1). While the Journal of Orthopedic Research also requires ARRIVE, other journals such as PlosOne state that completion of the ARRIVE checklist may be required for publication. Most biomedical research journals, however, endorse ARRIVE but do not require completion of a checklist. Despite endorsement, however, subsequent surveys revealed that many published articles still lack key information required by ARRIVE (8–11).
Author Manuscript
To assess the efficacy of the JBMR approach, ASBMR Publications staff and members of the ASBMR Publications Committee analyzed manuscripts that had been submitted to JBMR between October of 2014 and December of 2015, and subsequently accepted for publication. The audit showed that only about half of the published manuscripts with a completed ARRIVE checklist contained the appropriate information – similar to findings in other journals. It seems clear that many authors, and importantly many JBMR reviewers, are not yet convinced that it is necessary to devote the effort needed to ensure maximum transparency and reproducibility of animal-based research as outlined by ARRIVE. Because of our similar interests and goals, this result should trouble basic, translational and clinical investigators in the field of bone and mineral research.
Next steps The demand for greater transparency and reproducibility in animal studies, and indeed all research efforts, is rising. Adherence to standards for transparency and reproducibility is
J Bone Miner Res. Author manuscript; available in PMC 2017 July 01.
Jilka
Page 3
Author Manuscript
now a requirement for funding from the NIH (12), and other funding agencies will likely follow. Moreover, the Federation of American Societies for Experimental Biology, of which ASBMR is a member, recently called on the Publications Committees and Editorial Boards of member Society journals to consider how transparency and reproducibility of research can be improved (13).
Author Manuscript
Few would argue that authors ultimately bear responsibility for their published work, but thorough peer review and strong editorial leadership are also needed to ensure that reporting standards are met. As cogently summarized by Csiszar (14), the current system of anonymous peer review, established in the 1950’s, is only the latest of a variety of approaches developed over the past 200 years to protect against “shoddy science” and to maintain public accountability. It is almost certain that peer review will continue to evolve with the accelerating pace and sophistication of biomedical research, and the emergence of new electronic approaches for the dissemination and preservation of research findings. Postpublication comment and review may become more common in this environment.
Author Manuscript
For the ARRIVE guideline checklist to be an effective tool, it must be easy to use. However, several interrelated factors often make this tool more irksome than helpful. ARRIVE is based on the CONSORT (Consolidated Standards of Reporting Trials) guidelines developed for publishing the results of clinical trials. Because of this heritage, some of the information required by ARRIVE seems superfluous or at least beyond that historically included in most life science publications. For instance, some might argue that it is not necessary to report the source of a particular inbred mouse strain used in the study. They would be wrong. Witness the recent discovery of a mutation in the Dock2 gene that exists only in the C57BL/6NHsd substrain of C57BL/6 mice (15) – a situation that probably compromises interpretation of many gene-targeting studies in which founders were backcrossed into this substrain. Another difficulty is that the sheer breadth of animal-based research makes it difficult to devise a single all-encompassing checklist: pre-clinical studies have different requirements than cell culture studies. Finally, the process of entering manuscript locations into the ARRIVE checklist can only be described as tedious.
Author Manuscript
To address these problems, ASBMR Publications staff and members of the ASBMR Publications Committee have developed a more concise version of the ARRIVE checklist for use by JBMR authors and reviewers. Eventually, it is anticipated that authors will be able to electronically link each component of the checklist to the relevant part of the manuscript. The ASBMR is also developing approaches to increase awareness of the importance of animal research reporting guidelines, which will include education for authors and reviewers. These changes should improve compliance with ARRIVE in articles published in JBMR and perforce the transparency and reproducibility of animal-based bone and mineral research.
Acknowledgments This Perspective is made possible by conversations with colleagues at the UAMS Center for Osteoporosis and Metabolic Bone Diseases, members of the ASBMR Publications Committee, and ASBMR Publications staff. The author’s research is supported by the NIH (P01 AG13918).
J Bone Miner Res. Author manuscript; available in PMC 2017 July 01.
Jilka
Page 4
Author Manuscript
References
Author Manuscript Author Manuscript
1. Manolagas SC, Kronenberg HM. Reproducibility of results in preclinical studies: a perspective from the bone field. J Bone Miner Res. 2014; 29:2131–2140. [PubMed: 24916175] 2. Landis SC, Amara SG, Asadullah K, et al. A call for transparent reporting to optimize the predictive value of preclinical research. Nature. 2012; 490:187–191. [PubMed: 23060188] 3. Baker M. Statisticians issue warning over misuse of P values. Nature. 2016; :531.doi: 10.1038/ nature.2016.19503 [PubMed: 27127824] 4. Mouse phenome database at the Jackson Laboratory. http://phenome.jax.org/ 5. Mouse genome informatics. http://www.informatics.jax.org/ 6. Hsu PD, Lander ES, Zhang F. Development and applications of CRISPR-Cas9 for genome engineering. Cell. 2014; 157:1262–1278. [PubMed: 24906146] 7. Kilkenny C, Browne WJ, Cuthill IC, Emerson M, Altman DG. Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research. PLoS Biol. 2010; 8:e1000412. [PubMed: 20613859] 8. Baker D, Lidster K, Sottomayor A, Amor S. Two Years Later: Journals Are Not Yet Enforcing the ARRIVE Guidelines on Reporting Standards for Pre-Clinical Animal Studies. PLoS Biol. 2014; 12:e1001756. [PubMed: 24409096] 9. Gulin JE, Rocco DM, Garcia-Bournissen F. Quality of Reporting and Adherence to ARRIVE Guidelines in Animal Studies for Chagas Disease Preclinical Drug Research: A Systematic Review. PLoS Negl Trop Dis. 2015; 9:e0004194. [PubMed: 26587586] 10. Bramhall M, Florez-Vargas O, Stevens R, Brass A, Cruickshank S. Quality of methods reporting in animal models of colitis. Inflamm Bowel Dis. 2015; 21:1248–1259. [PubMed: 25989337] 11. Florez-Vargas O, Bramhall M, Noyes H, Cruickshank S, Stevens R, Brass A. The quality of methods reporting in parasitology experiments. PLoS One. 2014; 9:e101131. [PubMed: 25076044] 12. 2016 Rigor and Reproducibility. http://grants.nih.gov/reproducibility/index.htm 13. 2016 Enhancing Research Reproducibility: Recommendations from the Federation of American Societies for Experimental Biology. https://www.faseb.org/Portals/2/PDFs/opa/2016/ FASEB_Enhancing%20Research%20Reproducibility.pdf 14. Csiszar A. Peer review: troubled from the start. Nature. 2016; 532:306–308. [PubMed: 27111616] 15. Mahajan, Demissie E, Mattoo H, et al. Striking Immune Phenotypes in Gene-Targeted Mice Are Driven by a Copy-Number Variant Originating from a Commercially Available C57BL/6 Strain. Cell Rep. 2016; 15:1–9. [PubMed: 27052168]
Author Manuscript J Bone Miner Res. Author manuscript; available in PMC 2017 July 01.
J Bone Miner Res. Author manuscript; available in PMC 2017 July 01. Published in final edited form as: J Bone Miner Res. 2016 July ; 31(7): 1317–1319. doi:10.1002/jbmr.2881.
The Road to Reproducibility in Animal Research Robert L. Jilka Center for Osteoporosis and Metabolic Bone Diseases, Division of Endocrinology and Metabolism, University of Arkansas for Medical Sciences, Little Rock, AR 72205, USA Robert L. Jilka: [email protected]
Author Manuscript
Abstract Reproducibility of research findings is the hallmark of scientific advance. However, the recently noted lack of reproducibility and transparency of published research using animal models of human biology and disease has alarmed funders, scientists and the public. Improved reporting of methodology and better use of statistical tools are needed to enhance the quality and utility of published research. Reporting guidelines like ARRIVE have been devised to achieve these goals, but most biomedical research journals, including JBMR, have not been able to obtain high compliance. Co-operative efforts among authors, reviewers and editors - empowered by increased awareness of their responsibilities, and enabled by user-friendly guidelines – are needed to solve this problem.
Author Manuscript
Introduction
Author Manuscript
Highly publicized reports of poor reproducibility of published research using animal models of human biology and disease have alarmed funders, the pharmaceutical industry, the public, and the scientific community. Manolagas and Kronenberg have called attention to this issue, and described its negative impact on the field of bone and mineral research (1). Indeed, reproducibility is the hallmark of scientific advance, but this process is hindered by inadequate or opaque reporting of methodology (2), and misapplication of statistical tools like P values (3). These problems must be addressed because advances in the understanding of human biology and disease will increasingly depend on well-defined animal models. For example, assembly of large data sets containing detailed information on the genome, and transcriptome and phenome of specific mouse strains has increased understanding of mouse models of human disease, and will likely provide clues to the design of better models (4,5). Moreover, the ease and wide availability of CRISPR/Cas9 gene editing methodology (6) has accelerated the development of genetically modified mice, and greatly facilitates genetic modification of other species. Part of the reproducibility problem may be ascribed to that fact that modern biomedical research increasingly requires interdisciplinary teams of scientists. Such teams are particularly important in bone and mineral research because of the complex array of specialized cells in bone, the diverse microenvironments in which they and their progenitors Disclosures The author states that he has no conflicts of interest.
Jilka
Page 2
Author Manuscript
reside, and the involvement of endocrine as well as local factors in the regulation of these cells. Nevertheless, team members inevitably bring diverse clinical, animal, and basic research training and experience to their task, and therefore varying standards for analyzing and communicating their work. Regardless of the size of the research team, advances in biomedical knowledge best occurs when all laboratories investigating a particular problem follow the same publication standards.
Animal Research: Reporting In Vivo Experiments (ARRIVE)
Author Manuscript
The ARRIVE guidelines were developed in 2011 to aid investigators in the design, analysis and publication of animal experiments in manner that ensures, as far as practicable, transparency and reproducibility (7). ARRIVE focuses on details such as the characteristics of the animal model, its relevance to human biology and disease, as well as aspects of animal health and husbandry, experimental design and methodology, and statistical methods. Based on conversations with my scientific colleagues, JBMR Editor-in-Chief Juliet Compston, ASBMR Publications staff, and the ASBMR Publications Committee (of which I was Chair), a working group was formed in early 2014 to identify guidelines for the publication of animal-based bone and mineral research. This group recommended that JBMR adopt the ARRIVE guidelines to aid authors in reporting their findings, and to provide a tool for reviewers and editors to assess whether the standards espoused by the guidelines are being met in submitted manuscripts.
Author Manuscript
In October of 2014, JBMR announced that a completed ARRIVE checklist must accompany all submitted manuscripts reporting studies using animals. The announcement was accompanied by the Manolagas and Kronenberg Perspective, which also explained and endorsed ARRIVE (1). While the Journal of Orthopedic Research also requires ARRIVE, other journals such as PlosOne state that completion of the ARRIVE checklist may be required for publication. Most biomedical research journals, however, endorse ARRIVE but do not require completion of a checklist. Despite endorsement, however, subsequent surveys revealed that many published articles still lack key information required by ARRIVE (8–11).
Author Manuscript
To assess the efficacy of the JBMR approach, ASBMR Publications staff and members of the ASBMR Publications Committee analyzed manuscripts that had been submitted to JBMR between October of 2014 and December of 2015, and subsequently accepted for publication. The audit showed that only about half of the published manuscripts with a completed ARRIVE checklist contained the appropriate information – similar to findings in other journals. It seems clear that many authors, and importantly many JBMR reviewers, are not yet convinced that it is necessary to devote the effort needed to ensure maximum transparency and reproducibility of animal-based research as outlined by ARRIVE. Because of our similar interests and goals, this result should trouble basic, translational and clinical investigators in the field of bone and mineral research.
Next steps The demand for greater transparency and reproducibility in animal studies, and indeed all research efforts, is rising. Adherence to standards for transparency and reproducibility is
J Bone Miner Res. Author manuscript; available in PMC 2017 July 01.
Jilka
Page 3
Author Manuscript
now a requirement for funding from the NIH (12), and other funding agencies will likely follow. Moreover, the Federation of American Societies for Experimental Biology, of which ASBMR is a member, recently called on the Publications Committees and Editorial Boards of member Society journals to consider how transparency and reproducibility of research can be improved (13).
Author Manuscript
Few would argue that authors ultimately bear responsibility for their published work, but thorough peer review and strong editorial leadership are also needed to ensure that reporting standards are met. As cogently summarized by Csiszar (14), the current system of anonymous peer review, established in the 1950’s, is only the latest of a variety of approaches developed over the past 200 years to protect against “shoddy science” and to maintain public accountability. It is almost certain that peer review will continue to evolve with the accelerating pace and sophistication of biomedical research, and the emergence of new electronic approaches for the dissemination and preservation of research findings. Postpublication comment and review may become more common in this environment.
Author Manuscript
For the ARRIVE guideline checklist to be an effective tool, it must be easy to use. However, several interrelated factors often make this tool more irksome than helpful. ARRIVE is based on the CONSORT (Consolidated Standards of Reporting Trials) guidelines developed for publishing the results of clinical trials. Because of this heritage, some of the information required by ARRIVE seems superfluous or at least beyond that historically included in most life science publications. For instance, some might argue that it is not necessary to report the source of a particular inbred mouse strain used in the study. They would be wrong. Witness the recent discovery of a mutation in the Dock2 gene that exists only in the C57BL/6NHsd substrain of C57BL/6 mice (15) – a situation that probably compromises interpretation of many gene-targeting studies in which founders were backcrossed into this substrain. Another difficulty is that the sheer breadth of animal-based research makes it difficult to devise a single all-encompassing checklist: pre-clinical studies have different requirements than cell culture studies. Finally, the process of entering manuscript locations into the ARRIVE checklist can only be described as tedious.
Author Manuscript
To address these problems, ASBMR Publications staff and members of the ASBMR Publications Committee have developed a more concise version of the ARRIVE checklist for use by JBMR authors and reviewers. Eventually, it is anticipated that authors will be able to electronically link each component of the checklist to the relevant part of the manuscript. The ASBMR is also developing approaches to increase awareness of the importance of animal research reporting guidelines, which will include education for authors and reviewers. These changes should improve compliance with ARRIVE in articles published in JBMR and perforce the transparency and reproducibility of animal-based bone and mineral research.
Acknowledgments This Perspective is made possible by conversations with colleagues at the UAMS Center for Osteoporosis and Metabolic Bone Diseases, members of the ASBMR Publications Committee, and ASBMR Publications staff. The author’s research is supported by the NIH (P01 AG13918).
J Bone Miner Res. Author manuscript; available in PMC 2017 July 01.
Jilka
Page 4
Author Manuscript
References
Author Manuscript Author Manuscript
1. Manolagas SC, Kronenberg HM. Reproducibility of results in preclinical studies: a perspective from the bone field. J Bone Miner Res. 2014; 29:2131–2140. [PubMed: 24916175] 2. Landis SC, Amara SG, Asadullah K, et al. A call for transparent reporting to optimize the predictive value of preclinical research. Nature. 2012; 490:187–191. [PubMed: 23060188] 3. Baker M. Statisticians issue warning over misuse of P values. Nature. 2016; :531.doi: 10.1038/ nature.2016.19503 [PubMed: 27127824] 4. Mouse phenome database at the Jackson Laboratory. http://phenome.jax.org/ 5. Mouse genome informatics. http://www.informatics.jax.org/ 6. Hsu PD, Lander ES, Zhang F. Development and applications of CRISPR-Cas9 for genome engineering. Cell. 2014; 157:1262–1278. [PubMed: 24906146] 7. Kilkenny C, Browne WJ, Cuthill IC, Emerson M, Altman DG. Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research. PLoS Biol. 2010; 8:e1000412. [PubMed: 20613859] 8. Baker D, Lidster K, Sottomayor A, Amor S. Two Years Later: Journals Are Not Yet Enforcing the ARRIVE Guidelines on Reporting Standards for Pre-Clinical Animal Studies. PLoS Biol. 2014; 12:e1001756. [PubMed: 24409096] 9. Gulin JE, Rocco DM, Garcia-Bournissen F. Quality of Reporting and Adherence to ARRIVE Guidelines in Animal Studies for Chagas Disease Preclinical Drug Research: A Systematic Review. PLoS Negl Trop Dis. 2015; 9:e0004194. [PubMed: 26587586] 10. Bramhall M, Florez-Vargas O, Stevens R, Brass A, Cruickshank S. Quality of methods reporting in animal models of colitis. Inflamm Bowel Dis. 2015; 21:1248–1259. [PubMed: 25989337] 11. Florez-Vargas O, Bramhall M, Noyes H, Cruickshank S, Stevens R, Brass A. The quality of methods reporting in parasitology experiments. PLoS One. 2014; 9:e101131. [PubMed: 25076044] 12. 2016 Rigor and Reproducibility. http://grants.nih.gov/reproducibility/index.htm 13. 2016 Enhancing Research Reproducibility: Recommendations from the Federation of American Societies for Experimental Biology. https://www.faseb.org/Portals/2/PDFs/opa/2016/ FASEB_Enhancing%20Research%20Reproducibility.pdf 14. Csiszar A. Peer review: troubled from the start. Nature. 2016; 532:306–308. [PubMed: 27111616] 15. Mahajan, Demissie E, Mattoo H, et al. Striking Immune Phenotypes in Gene-Targeted Mice Are Driven by a Copy-Number Variant Originating from a Commercially Available C57BL/6 Strain. Cell Rep. 2016; 15:1–9. [PubMed: 27052168]
Author Manuscript J Bone Miner Res. Author manuscript; available in PMC 2017 July 01.
