International Journal of Rheumatic Diseases 2015; 18: 488–494
SYSTEMATIC REVIEW
Quality of reporting of interventional animal studies in rheumatology: a systematic review using the ARRIVE guidelines Kimberley H.J. TING,1 Catherine L. HILL1,2 and Samuel L. WHITTLE1 1
Rheumatology Unit, The Queen Elizabeth Hospital, Woodville South, South Australia, Australia, and 2The Health Observatory, Discipline of Medicine, The University of Adelaide, Adelaide, South Australia, Australia
Abstract Aim: To systematically investigate the quality of reporting of published interventional animal studies in experimental rheumatology. Methods: Original scientific publications in Annals of the Rheumatic Diseases (ARD) and Arthritis and Rheumatism (A&R) from January to December 2012 were identified. Studies were included if they used animal experimental model(s) and involved a treatment intervention. Data were extracted regarding disease type, animal model, intervention type and funding. Each study was assessed for quality of reporting, using the ARRIVE guidelines as a checklist. Results: Forty-one studies (15 ARD, 26 A&R) were analyzed. Ethics approval was not reported or unclear in 22%. Randomization was not reported or unclear in 82.9% of the papers. Only 19.5% and 9.8% of papers reported attrition rate and important adverse events, respectively. Sample size calculation or allocation method was not reported in any paper. Only one study published negative results. Conclusion: A number of key study design principles are poorly reported in experimental animal research investigating potential treatments in rheumatology. We support the widespread implementation of the ARRIVE guidelines in the rheumatology literature to promote the publication of manuscripts that allow rigorous appraisal of scientific quality. Key words: ARRIVE, experimental rheumatology, systematic review.
INTRODUCTION Animal research remains the forerunner in generating new knowledge, and advancing innovative drug development. However, it has been increasingly recognized that successful treatments in animal studies have not been replicated in the clinical trials that have followed. In a 2006 review of interventional in vivo animal studies published in seven leading, high-impact scientific journals, only about one-third of these translated to inter-
Correspondence: Dr. Kimberley H.J. Ting, 11 First Avenue, St Peters, Adelaide, SA 5069, Australia. Email: [email protected]
ventional randomized trials in humans, and only 10% were subsequently approved for use in patients.1 Studies of intervention require rigorous design and reporting to ensure the generation of useful data. While the Consolidated Standards of Reporting Trials (CONSORT) guidelines have improved the quality of human randomized clinical trials (RCTs), less attention has been paid to animal studies.2–4 Poor-quality animal studies may prevent the successful translation of research from bench to bedside and risk the unethical use of animals in research that does not advance scientific knowledge. A comprehensive review of 271 publications in experimental animal research found frequent omissions in fundamental study design concepts, such
© 2015 Asia Pacific League of Associations for Rheumatology and Wiley Publishing Asia Pty Ltd
Reporting of experimental rheumatology
as randomization and sample size.5 Akin to the CONSORT guidelines for human RCTs, the Animal Research: Reporting of In vivo Experiments (ARRIVE) guidelines (2010) have been developed to improve the standards of reporting in animal experiments (Fig. 1).6 To date, quality of reporting of animal research has best been shown in the stroke literature;7,8 it has yet to be described in the field of rheumatology. The aim of this study was to systematically investigate the quality of published animal studies in the field of experimental rheumatology research, focusing on treatment interventions.
important and supplemented ARRIVE item number 5, that is, ‘indicate the nature of the ethical review permissions, relevant licenses, and national or institutional guidelines for the care and use of animals, that cover the research.’ The two investigators were blinded to the assessment of the other, and a third adjudicator made the final assessment where discrepancies in answers were identified.
RESULTS
We confined our search to the two highest impact rheumatology journals in 2012: Annals of the Rheumatic Diseases (ARD) and Arthritis and Rheumatism (A&R). A reviewer then systematically identified original scientific publications from January to December 2012.
The number of studies screened in total was 76 (25 ARD, 51 A&R). Following inclusion and exclusion criteria, a total of 41 studies (15 ARD, 26 A&R) were reviewed. Three studies were concise reports. Studies investigated a range of rheumatic diseases, most commonly inflammatory arthritis (Table 1). Industry contributed to funding in 26.8%, with 2.4% receiving funding from industry alone (Table 1). Only one study published negative results.
Inclusion and exclusion criteria for quality assessment
Reporting of published interventional animal studies
All articles had to be original published articles – review articles, commentaries or communications were excluded. As the ARRIVE guidelines only apply to in vivo animal experiments, studies of interventional animal studies were included. Studies which were exclusively mechanistic or only included in vitro experiments were excluded.
Table 2 summarises our evaluation of the quality of reporting in experimental rheumatology.
MATERIAL AND METHODS Search strategy
Evaluation of publication quality Data were extracted regarding disease type, animal model, intervention type and funding. Using the ARRIVE guidelines, two investigators independently extracted data by answering either ‘yes’ (met by the publication), ‘no’ (not met by the publication), or ‘unclear’ to score 29 items for each original article. This included the 20 core items, and selected sub-items from the ARRIVE guidelines that we felt deserved specific attention with respect to study design and method. For an example, ‘number of animals in each experiment’ and ‘sample size calculation’ (10a, 10b respectively), which are a single ARRIVE item, were separately addressed as sub-items. We also assessed randomization and blinding as separate sub-items, rather than in the combined item found in the ARRIVE guidelines. In addition, we added to our checklist ‘was ethical approval attained?’ Although this is not an item or subitem within the ARRIVE guidelines, we felt that this was
International Journal of Rheumatic Diseases 2015; 18: 488–494
Ethics A statement regarding ethics approval was not reported or unclear in 22%. Further details regarding the nature of the ethical review permission attained was reported in only 39%.
Items pertaining to the quality of reporting Explanation of study background, including context and experimental rationale, was well reported. However, only 65.9% clearly described an objective or stated a hypothesis. Qualities important to the reproducibility of a study were poorly reported. For example, details of animal strain and species were reported in only 53.7%, and housing and husbandry reported in only 4.9%. Study limitations, including potential sources of bias, were not reported in 87.8%.
Study design There were several key items in study design that were poorly reported. Reporting of randomization and assessor blinding occurred in 17.1% and 29.3%, respectively. None of the 41 interventional studies reported sample size calculation or details of allocation method. Attrition rate was unclear or not reported in 80.5%.
489
K. H. J. Ting et al.
Figure 1 ARRIVE guidelines
490
International Journal of Rheumatic Diseases 2015; 18: 488–494
Reporting of experimental rheumatology
Figure 1 Continued.
International Journal of Rheumatic Diseases 2015; 18: 488–494
491
K. H. J. Ting et al.
Table 1 Disease investigated and funding received % Disease type Inflammatory arthritis Systemic Sclerosis Systemic lupus erythematosus Osteoarthritis Polymyositis Septic arthritis Periprosthetic osteolysis Funding Government and/or institution Government/institution/industry Industry only No fund/unclear
41.5 19.5 17.1 14.6 2.4 2.4 2.4 68.3 24.4 2.4 4.9
Statistics Although 75.6% of publications included details of the statistical methods used for each analysis, only 39% reported the results of each analysis carried out with a measure of precision such as standard error or confidence interval.
Harm Only 9.8% of the studies gave details of all important adverse events.
DISCUSSION We found that key elements in study design and scientific procedure were poorly reported in the experimental rheumatology literature. Description of a method for treatment allocation, which addresses bias, and sample size calculation, which ensures adequate power, were most notable with none of the 41 studies reporting these measures. Despite its fundamental importance to scientific method, a hypothesis or objectives statement was not clearly reported in more than one-third of the studies. Our findings are consistent with Vesterinen et al.’s analysis of cerebrovascular animal research that found 27% stated a primary hypothesis, 8% allocation concealment and 1% stated a sample size.7 Our analysis revealed poor reporting of randomization and assessor blinding, with the subsequent potential for inflated effect sizes. Further, only 12.2% and 9.8% reported any methodological limitations (including bias), or significant adverse events, respectively. This risks successful translation to interventional clinical trials. The reproducibility of many studies was limited by poor explanation of the animal species and models
492
used, and poor reporting of animal strain. These variables are important to prevent genetic confounding, and provide the variability that is required to reflect disease in humans.9 We also found that experimental procedures and environmental conditions such as housing were not well defined, making results vulnerable to phenotypic heterogeneities and complicating healthy comparisons between studies. The value of such scientific procedure and quality design has recently been reiterated by the European League Against Rheumatism Study Group on Animal Models recommendations.9 We were surprised that a statement of granted ethical approval was not reported or unclear in 22%. Furthermore, only 39% of papers reported the nature of the ethical review permissions. Ongoing animal research requires sound ethical accountability.10 We identified only one negative study. It has been estimated that publication bias in animal studies inflates the effect size of interventions by one-third.11 In our study we found that 61% failed to clearly define experimental outcomes a priori. This raises suspicion for the presence of selective outcomes reporting bias. Other studies of the experimental animal literature have also reported an over-representation of studies with statistically significant results.12 Our study has several limitations. The first is that it analyzes the quality of the reporting of animal studies. We cannot comment on the quality of each study overall, and indeed it would be difficult to assess whether such studies are truly flawed, when reporting of methodological principles is deficient. The small number of papers included prevented a more detailed analysis of factors affecting reporting. For example, it would be interesting to examine the effect, if any, on reporting of industry sponsorship. Restricting the current study to two journals may preclude generalization of these results. However, given the high impact of these two journals we expected that these were ‘model examples’ of reporting of interventional animal experiments. The extent to which flawed methodology or reporting in animal studies leads to translational block is unclear.13 Certainly, there are inherent limitations to animal studies, such as replicating the nature of a chronic inflammatory disease in experiments with limited time frames. However, this should not detract from the utility of animal models. Rather, we emphasize that incomplete reporting may limit the value of publications as instruments to inform future experimentation. We acknowledge that it will likely take more than just endorsement of the ARRIVE guidelines to positively impact on reporting standards. In other fields, such as
International Journal of Rheumatic Diseases 2015; 18: 488–494
Reporting of experimental rheumatology
Table 2 Evaluation of the quality of reporting in experimental rheumatology, based on the ARRIVE guidelines6 ARRIVE item Title Abstract Background Study context and experimental rationale How and why animal species and models being used address objectives Objectives/hypothesis Ethics Indicate nature of the ethical review permissions, licences or guidelines Ethical statement of approval Study design No. of experimental and control groups Randomization and details of the experimental unit Blinding performed Experimental procedures (how, when, where, why) Experimental animals (Details of species, strain, sex and other relevant information, e.g., source) Housing, husbandry and welfare-related assessments Sample size Total no. of animals used in each experiment and in each experimental group Sample size calculation and replications if relevant Details of allocation method Experimental outcomes Statistical methods Details of the statistical methods used Specify unit of analysis for each dataset Methods to assess whether data meet assumptions of statistical approach Baseline data of animals Numbers analyzed (including no. of animals not included and why) Outcomes and estimation Adverse events (Details of all important adverse events and modifications) Interpretation/scientific implications Interpretation of results Study limitations Implications for replacement, refinement or reduction of use of animals Generalisability/translation Funding
neuroimmunology, it is evident that post-endorsement of the ARRIVE guidelines has yet to correlate with improved reporting of animal studies.14 Additional measures in the peer-review and publication process, and guidance for authors on how the ARRIVE guidelines should be applied may be more effective strategies for improving reporting quality. Nonetheless we believe that adopting the ARRIVE guidelines, as a quality standard of reporting, is the first step in addressing some of the aforementioned deficiencies. It is hoped that this will provide the impetus to improve trial reporting and
International Journal of Rheumatic Diseases 2015; 18: 488–494
1 2 3 3a 3b 4 5 n/a 6 6a 6b, 6c 6b 7a, b, c, d 8a, b 9a, b, c 10 10a 10b,c 11a, b 12 13 13a 13b 13c 14 15a, b 16 17a, b 18 18a 18b 18c 19 20
Reported (%)
Unclear (%)
Not reported (%)
73.2 87.8
14.6 4.9
12.2 7.3
100 75.6 65.9
0 12.2 19.5
0 12.2 14.6
39 78
9.8 9.8
51.2 12.2
58.5 17.1 29.3 82.9 53.7
31.7 0 12.2 7.3 29.3
9.8 82.9 58.5 9.8 17
4.9
2.4
92.7
31.7
14.6
53.7
0 0 39
0 9.8 31.7
100 90.2 29.3
75.6 78 4.9 0 19.5 39 9.8
17.1 12.2 0 0 26.8 14.6 2.4
7.3 9.8 95.1 100 53.7 46.3 87.8
97.6 12.2 0 87.8 95.1
2.4 0 0 2.4 2.4
0 87.8 100 9.8 2.4
quality, similar to the effect of the CONSORT guidelines on RCTs.4,15 No registry yet exists for pre-clinical trials. A preclinical registry would allow investigators to receive credit for their work, particularly for unfavorable findings, which are otherwise at a publishing disadvantage. It would also reduce wastage of animals and allow subsequent investigators access to all relevant data. Several studies suggest that at best 10–11% of therapeutic interventions translate from the level of preclinical research to clinical approval.1,16 Improved preclinical
493
K. H. J. Ting et al.
disclosure may help in identifying roadblocks in translation, and prevent progression to premature clinical trials. In conclusion, our study identified that published animal studies investigating potential treatments in the two highest impact rheumatology journals exhibit poor reporting of key design principles. We support the implementation of the ARRIVE guidelines in the rheumatology literature in order to promote the publication of manuscripts that allow rigorous appraisal of scientific quality. Not only is this an ethical imperative, but it may also improve study quality and optimize translation to human clinical trials.
ACKNOWLEDGEMENTS No financial grants, funding or industrial links or affiliations to declare. All authors contributed to the study conception and design, acquisition of data, analysis and interpretation of data, drafting and revising of the manuscript.
REFERENCES 1 Hackam DG, Redelmeier DA (2006) Translation of research evidence from animals to humans. JAMA 296, 1731–2. 2 Begg C, Cho M, Eastwood S et al. (1996) Improving the quality of reporting of randomised controlled trials: the CONSORT statement. JAMA 276, 637–9. 3 Schulz KF, Altman DG, Moher D (2010) CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Obstet Gynecol 115, 1063–70. 4 Plint AC, Moher D, Morrison A et al. (2006) Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A Systematic Review. Med J Aust 185, 263–7. 5 Kilkenny C, Parsons N, Kadyszewski E et al. (2009) Survey of the quality of experimental design, statistical analysis
494
6
7
8
9
10
11
12
13
14
15
16
and reporting of research using animals. PLoS ONE 4, e7824. Kilkenny C, Browne WJ, Cuthill IC, Emerson M, Altman DG (2010) Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research. PLoS Biol 8, e1000412. Vesterinen HV, Egan K, Deister A et al. (2011) Systematic survey of the design, statistical analysis, and reporting of studies published in the 2008 volume of the journal of cerebral blood flow and metabolism. J Cereb Blood Flow Metab 31, 1064–72. Van der Worp HB, de Haan P, Morrema E, Kalkman CJ (2005) Methodological quality of animal studies on neuroprotection in focal cerebral ischaemia (original communication). J Neurol 252, 1108–14. Kollias G, Papadaki G, Apparailly F et al. (2011) Animal models for arthritis: innovative tools for prevention and treatment. Ann Rheum Dis 70, 1357–62. Varga O, Hansen AP, Sandoe P, Olsson IAS (2010) Improving transparency and ethical accountability in animal studies. EMBO Rep 11, 500–3. Sena ES, van der Worp HB, Bath PMW, Howells DW, Macleod MR (2010) Publication bias in reports of animal stroke studies leads to major overstatement of efficacy. PLoS Biol 8, e1000344. Tsilidis KK, Panagiotou OA, Sena ES et al. (2013) Evaluation of excess significance bias in animal studies of neurological diseases. PLoS Biol 11, e1001609. Van der Worp HB, Macleod MR (2011) Preclinical studies of human disease: time to take methodological quality seriously (editorial). J Mol Cell Cardiol 51, 449–50. Baker D, Lidster K, Sottomayor A, Amor S (2014) Two years later: journals are not yet enforcing the ARRIVE guidelines on reporting standards for pre-clinical animal studies. PLoS Biol 12 (1), e1001756. Hill CL, LaValley MP, Felson DT (2002) Secular Changes in the quality of published randomized clinical trials in rheumatology. Arthritis Rheum 46, 779–85. Kimmelman J, Anderson JA (2012) Should preclinical studies be registered? (editorial letter). Nat Biotechnol 30, 488–9.
International Journal of Rheumatic Diseases 2015; 18: 488–494
SYSTEMATIC REVIEW
Quality of reporting of interventional animal studies in rheumatology: a systematic review using the ARRIVE guidelines Kimberley H.J. TING,1 Catherine L. HILL1,2 and Samuel L. WHITTLE1 1
Rheumatology Unit, The Queen Elizabeth Hospital, Woodville South, South Australia, Australia, and 2The Health Observatory, Discipline of Medicine, The University of Adelaide, Adelaide, South Australia, Australia
Abstract Aim: To systematically investigate the quality of reporting of published interventional animal studies in experimental rheumatology. Methods: Original scientific publications in Annals of the Rheumatic Diseases (ARD) and Arthritis and Rheumatism (A&R) from January to December 2012 were identified. Studies were included if they used animal experimental model(s) and involved a treatment intervention. Data were extracted regarding disease type, animal model, intervention type and funding. Each study was assessed for quality of reporting, using the ARRIVE guidelines as a checklist. Results: Forty-one studies (15 ARD, 26 A&R) were analyzed. Ethics approval was not reported or unclear in 22%. Randomization was not reported or unclear in 82.9% of the papers. Only 19.5% and 9.8% of papers reported attrition rate and important adverse events, respectively. Sample size calculation or allocation method was not reported in any paper. Only one study published negative results. Conclusion: A number of key study design principles are poorly reported in experimental animal research investigating potential treatments in rheumatology. We support the widespread implementation of the ARRIVE guidelines in the rheumatology literature to promote the publication of manuscripts that allow rigorous appraisal of scientific quality. Key words: ARRIVE, experimental rheumatology, systematic review.
INTRODUCTION Animal research remains the forerunner in generating new knowledge, and advancing innovative drug development. However, it has been increasingly recognized that successful treatments in animal studies have not been replicated in the clinical trials that have followed. In a 2006 review of interventional in vivo animal studies published in seven leading, high-impact scientific journals, only about one-third of these translated to inter-
Correspondence: Dr. Kimberley H.J. Ting, 11 First Avenue, St Peters, Adelaide, SA 5069, Australia. Email: [email protected]
ventional randomized trials in humans, and only 10% were subsequently approved for use in patients.1 Studies of intervention require rigorous design and reporting to ensure the generation of useful data. While the Consolidated Standards of Reporting Trials (CONSORT) guidelines have improved the quality of human randomized clinical trials (RCTs), less attention has been paid to animal studies.2–4 Poor-quality animal studies may prevent the successful translation of research from bench to bedside and risk the unethical use of animals in research that does not advance scientific knowledge. A comprehensive review of 271 publications in experimental animal research found frequent omissions in fundamental study design concepts, such
© 2015 Asia Pacific League of Associations for Rheumatology and Wiley Publishing Asia Pty Ltd
Reporting of experimental rheumatology
as randomization and sample size.5 Akin to the CONSORT guidelines for human RCTs, the Animal Research: Reporting of In vivo Experiments (ARRIVE) guidelines (2010) have been developed to improve the standards of reporting in animal experiments (Fig. 1).6 To date, quality of reporting of animal research has best been shown in the stroke literature;7,8 it has yet to be described in the field of rheumatology. The aim of this study was to systematically investigate the quality of published animal studies in the field of experimental rheumatology research, focusing on treatment interventions.
important and supplemented ARRIVE item number 5, that is, ‘indicate the nature of the ethical review permissions, relevant licenses, and national or institutional guidelines for the care and use of animals, that cover the research.’ The two investigators were blinded to the assessment of the other, and a third adjudicator made the final assessment where discrepancies in answers were identified.
RESULTS
We confined our search to the two highest impact rheumatology journals in 2012: Annals of the Rheumatic Diseases (ARD) and Arthritis and Rheumatism (A&R). A reviewer then systematically identified original scientific publications from January to December 2012.
The number of studies screened in total was 76 (25 ARD, 51 A&R). Following inclusion and exclusion criteria, a total of 41 studies (15 ARD, 26 A&R) were reviewed. Three studies were concise reports. Studies investigated a range of rheumatic diseases, most commonly inflammatory arthritis (Table 1). Industry contributed to funding in 26.8%, with 2.4% receiving funding from industry alone (Table 1). Only one study published negative results.
Inclusion and exclusion criteria for quality assessment
Reporting of published interventional animal studies
All articles had to be original published articles – review articles, commentaries or communications were excluded. As the ARRIVE guidelines only apply to in vivo animal experiments, studies of interventional animal studies were included. Studies which were exclusively mechanistic or only included in vitro experiments were excluded.
Table 2 summarises our evaluation of the quality of reporting in experimental rheumatology.
MATERIAL AND METHODS Search strategy
Evaluation of publication quality Data were extracted regarding disease type, animal model, intervention type and funding. Using the ARRIVE guidelines, two investigators independently extracted data by answering either ‘yes’ (met by the publication), ‘no’ (not met by the publication), or ‘unclear’ to score 29 items for each original article. This included the 20 core items, and selected sub-items from the ARRIVE guidelines that we felt deserved specific attention with respect to study design and method. For an example, ‘number of animals in each experiment’ and ‘sample size calculation’ (10a, 10b respectively), which are a single ARRIVE item, were separately addressed as sub-items. We also assessed randomization and blinding as separate sub-items, rather than in the combined item found in the ARRIVE guidelines. In addition, we added to our checklist ‘was ethical approval attained?’ Although this is not an item or subitem within the ARRIVE guidelines, we felt that this was
International Journal of Rheumatic Diseases 2015; 18: 488–494
Ethics A statement regarding ethics approval was not reported or unclear in 22%. Further details regarding the nature of the ethical review permission attained was reported in only 39%.
Items pertaining to the quality of reporting Explanation of study background, including context and experimental rationale, was well reported. However, only 65.9% clearly described an objective or stated a hypothesis. Qualities important to the reproducibility of a study were poorly reported. For example, details of animal strain and species were reported in only 53.7%, and housing and husbandry reported in only 4.9%. Study limitations, including potential sources of bias, were not reported in 87.8%.
Study design There were several key items in study design that were poorly reported. Reporting of randomization and assessor blinding occurred in 17.1% and 29.3%, respectively. None of the 41 interventional studies reported sample size calculation or details of allocation method. Attrition rate was unclear or not reported in 80.5%.
489
K. H. J. Ting et al.
Figure 1 ARRIVE guidelines
490
International Journal of Rheumatic Diseases 2015; 18: 488–494
Reporting of experimental rheumatology
Figure 1 Continued.
International Journal of Rheumatic Diseases 2015; 18: 488–494
491
K. H. J. Ting et al.
Table 1 Disease investigated and funding received % Disease type Inflammatory arthritis Systemic Sclerosis Systemic lupus erythematosus Osteoarthritis Polymyositis Septic arthritis Periprosthetic osteolysis Funding Government and/or institution Government/institution/industry Industry only No fund/unclear
41.5 19.5 17.1 14.6 2.4 2.4 2.4 68.3 24.4 2.4 4.9
Statistics Although 75.6% of publications included details of the statistical methods used for each analysis, only 39% reported the results of each analysis carried out with a measure of precision such as standard error or confidence interval.
Harm Only 9.8% of the studies gave details of all important adverse events.
DISCUSSION We found that key elements in study design and scientific procedure were poorly reported in the experimental rheumatology literature. Description of a method for treatment allocation, which addresses bias, and sample size calculation, which ensures adequate power, were most notable with none of the 41 studies reporting these measures. Despite its fundamental importance to scientific method, a hypothesis or objectives statement was not clearly reported in more than one-third of the studies. Our findings are consistent with Vesterinen et al.’s analysis of cerebrovascular animal research that found 27% stated a primary hypothesis, 8% allocation concealment and 1% stated a sample size.7 Our analysis revealed poor reporting of randomization and assessor blinding, with the subsequent potential for inflated effect sizes. Further, only 12.2% and 9.8% reported any methodological limitations (including bias), or significant adverse events, respectively. This risks successful translation to interventional clinical trials. The reproducibility of many studies was limited by poor explanation of the animal species and models
492
used, and poor reporting of animal strain. These variables are important to prevent genetic confounding, and provide the variability that is required to reflect disease in humans.9 We also found that experimental procedures and environmental conditions such as housing were not well defined, making results vulnerable to phenotypic heterogeneities and complicating healthy comparisons between studies. The value of such scientific procedure and quality design has recently been reiterated by the European League Against Rheumatism Study Group on Animal Models recommendations.9 We were surprised that a statement of granted ethical approval was not reported or unclear in 22%. Furthermore, only 39% of papers reported the nature of the ethical review permissions. Ongoing animal research requires sound ethical accountability.10 We identified only one negative study. It has been estimated that publication bias in animal studies inflates the effect size of interventions by one-third.11 In our study we found that 61% failed to clearly define experimental outcomes a priori. This raises suspicion for the presence of selective outcomes reporting bias. Other studies of the experimental animal literature have also reported an over-representation of studies with statistically significant results.12 Our study has several limitations. The first is that it analyzes the quality of the reporting of animal studies. We cannot comment on the quality of each study overall, and indeed it would be difficult to assess whether such studies are truly flawed, when reporting of methodological principles is deficient. The small number of papers included prevented a more detailed analysis of factors affecting reporting. For example, it would be interesting to examine the effect, if any, on reporting of industry sponsorship. Restricting the current study to two journals may preclude generalization of these results. However, given the high impact of these two journals we expected that these were ‘model examples’ of reporting of interventional animal experiments. The extent to which flawed methodology or reporting in animal studies leads to translational block is unclear.13 Certainly, there are inherent limitations to animal studies, such as replicating the nature of a chronic inflammatory disease in experiments with limited time frames. However, this should not detract from the utility of animal models. Rather, we emphasize that incomplete reporting may limit the value of publications as instruments to inform future experimentation. We acknowledge that it will likely take more than just endorsement of the ARRIVE guidelines to positively impact on reporting standards. In other fields, such as
International Journal of Rheumatic Diseases 2015; 18: 488–494
Reporting of experimental rheumatology
Table 2 Evaluation of the quality of reporting in experimental rheumatology, based on the ARRIVE guidelines6 ARRIVE item Title Abstract Background Study context and experimental rationale How and why animal species and models being used address objectives Objectives/hypothesis Ethics Indicate nature of the ethical review permissions, licences or guidelines Ethical statement of approval Study design No. of experimental and control groups Randomization and details of the experimental unit Blinding performed Experimental procedures (how, when, where, why) Experimental animals (Details of species, strain, sex and other relevant information, e.g., source) Housing, husbandry and welfare-related assessments Sample size Total no. of animals used in each experiment and in each experimental group Sample size calculation and replications if relevant Details of allocation method Experimental outcomes Statistical methods Details of the statistical methods used Specify unit of analysis for each dataset Methods to assess whether data meet assumptions of statistical approach Baseline data of animals Numbers analyzed (including no. of animals not included and why) Outcomes and estimation Adverse events (Details of all important adverse events and modifications) Interpretation/scientific implications Interpretation of results Study limitations Implications for replacement, refinement or reduction of use of animals Generalisability/translation Funding
neuroimmunology, it is evident that post-endorsement of the ARRIVE guidelines has yet to correlate with improved reporting of animal studies.14 Additional measures in the peer-review and publication process, and guidance for authors on how the ARRIVE guidelines should be applied may be more effective strategies for improving reporting quality. Nonetheless we believe that adopting the ARRIVE guidelines, as a quality standard of reporting, is the first step in addressing some of the aforementioned deficiencies. It is hoped that this will provide the impetus to improve trial reporting and
International Journal of Rheumatic Diseases 2015; 18: 488–494
1 2 3 3a 3b 4 5 n/a 6 6a 6b, 6c 6b 7a, b, c, d 8a, b 9a, b, c 10 10a 10b,c 11a, b 12 13 13a 13b 13c 14 15a, b 16 17a, b 18 18a 18b 18c 19 20
Reported (%)
Unclear (%)
Not reported (%)
73.2 87.8
14.6 4.9
12.2 7.3
100 75.6 65.9
0 12.2 19.5
0 12.2 14.6
39 78
9.8 9.8
51.2 12.2
58.5 17.1 29.3 82.9 53.7
31.7 0 12.2 7.3 29.3
9.8 82.9 58.5 9.8 17
4.9
2.4
92.7
31.7
14.6
53.7
0 0 39
0 9.8 31.7
100 90.2 29.3
75.6 78 4.9 0 19.5 39 9.8
17.1 12.2 0 0 26.8 14.6 2.4
7.3 9.8 95.1 100 53.7 46.3 87.8
97.6 12.2 0 87.8 95.1
2.4 0 0 2.4 2.4
0 87.8 100 9.8 2.4
quality, similar to the effect of the CONSORT guidelines on RCTs.4,15 No registry yet exists for pre-clinical trials. A preclinical registry would allow investigators to receive credit for their work, particularly for unfavorable findings, which are otherwise at a publishing disadvantage. It would also reduce wastage of animals and allow subsequent investigators access to all relevant data. Several studies suggest that at best 10–11% of therapeutic interventions translate from the level of preclinical research to clinical approval.1,16 Improved preclinical
493
K. H. J. Ting et al.
disclosure may help in identifying roadblocks in translation, and prevent progression to premature clinical trials. In conclusion, our study identified that published animal studies investigating potential treatments in the two highest impact rheumatology journals exhibit poor reporting of key design principles. We support the implementation of the ARRIVE guidelines in the rheumatology literature in order to promote the publication of manuscripts that allow rigorous appraisal of scientific quality. Not only is this an ethical imperative, but it may also improve study quality and optimize translation to human clinical trials.
ACKNOWLEDGEMENTS No financial grants, funding or industrial links or affiliations to declare. All authors contributed to the study conception and design, acquisition of data, analysis and interpretation of data, drafting and revising of the manuscript.
REFERENCES 1 Hackam DG, Redelmeier DA (2006) Translation of research evidence from animals to humans. JAMA 296, 1731–2. 2 Begg C, Cho M, Eastwood S et al. (1996) Improving the quality of reporting of randomised controlled trials: the CONSORT statement. JAMA 276, 637–9. 3 Schulz KF, Altman DG, Moher D (2010) CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Obstet Gynecol 115, 1063–70. 4 Plint AC, Moher D, Morrison A et al. (2006) Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A Systematic Review. Med J Aust 185, 263–7. 5 Kilkenny C, Parsons N, Kadyszewski E et al. (2009) Survey of the quality of experimental design, statistical analysis
494
6
7
8
9
10
11
12
13
14
15
16
and reporting of research using animals. PLoS ONE 4, e7824. Kilkenny C, Browne WJ, Cuthill IC, Emerson M, Altman DG (2010) Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research. PLoS Biol 8, e1000412. Vesterinen HV, Egan K, Deister A et al. (2011) Systematic survey of the design, statistical analysis, and reporting of studies published in the 2008 volume of the journal of cerebral blood flow and metabolism. J Cereb Blood Flow Metab 31, 1064–72. Van der Worp HB, de Haan P, Morrema E, Kalkman CJ (2005) Methodological quality of animal studies on neuroprotection in focal cerebral ischaemia (original communication). J Neurol 252, 1108–14. Kollias G, Papadaki G, Apparailly F et al. (2011) Animal models for arthritis: innovative tools for prevention and treatment. Ann Rheum Dis 70, 1357–62. Varga O, Hansen AP, Sandoe P, Olsson IAS (2010) Improving transparency and ethical accountability in animal studies. EMBO Rep 11, 500–3. Sena ES, van der Worp HB, Bath PMW, Howells DW, Macleod MR (2010) Publication bias in reports of animal stroke studies leads to major overstatement of efficacy. PLoS Biol 8, e1000344. Tsilidis KK, Panagiotou OA, Sena ES et al. (2013) Evaluation of excess significance bias in animal studies of neurological diseases. PLoS Biol 11, e1001609. Van der Worp HB, Macleod MR (2011) Preclinical studies of human disease: time to take methodological quality seriously (editorial). J Mol Cell Cardiol 51, 449–50. Baker D, Lidster K, Sottomayor A, Amor S (2014) Two years later: journals are not yet enforcing the ARRIVE guidelines on reporting standards for pre-clinical animal studies. PLoS Biol 12 (1), e1001756. Hill CL, LaValley MP, Felson DT (2002) Secular Changes in the quality of published randomized clinical trials in rheumatology. Arthritis Rheum 46, 779–85. Kimmelman J, Anderson JA (2012) Should preclinical studies be registered? (editorial letter). Nat Biotechnol 30, 488–9.
International Journal of Rheumatic Diseases 2015; 18: 488–494
