GENERAL ORTHOPAEDICS
The impact factor of a journal is a poor measure of the clinical relevance of its papers
P. Kodumuri, B. Ollivere, J. Holley, C. G. Moran From Queens Medical Centre, Nottingham, United Kingdom
We evaluated the top 13 journals in trauma and orthopaedics by impact factor and looked at the longer-term effect regarding citations of their papers. All 4951 papers published in these journals during 2007 and 2008 were reviewed and categorised by their type, subspecialty and super-specialty. All citations indexed through Google Scholar were reviewed to establish the rate of citation per paper at two, four and five years post-publication. The top five journals published a total of 1986 papers. Only three (0.15%) were on operative orthopaedic surgery and none were on trauma. Most (n = 1084, 54.5%) were about experimental basic science. Surgical papers had a lower rate of citation (2.18) at two years than basic science or clinical medical papers (4.68). However, by four years the rates were similar (26.57 for surgery, 30.35 for basic science/medical), which suggests that there is a considerable time lag before clinical surgical research has an impact. We conclude that high impact journals do not address clinical research in surgery and when they do, there is a delay before such papers are cited. We suggest that a rate of citation at five years post-publication might be a more appropriate indicator of importance for papers in our specialty. Cite this article: Bone Joint J 2014;96-B:414–19.
P. Kodumuri, MRCS, Specialty Registrar Queens Medical Centre, 16 Topliff Road, Chilwell, Nottingham, NG9 5AS, UK. B. Ollivere, MD, FRCS, Consultant Trauma and Orthopaedic Surgeon J. Holley, MRCS, Core Surgical Trainee C. G. Moran, MD, FRCS, Professor of Trauma and Orthopaedics Queens Medical Centre, Derby Road, Nottingham, NG7 2UH, UK. Correspondence should be sent to Mr P. Kodumuri; e-mail: [email protected] ©2014 The British Editorial Society of Bone & Joint Surgery doi:10.1302/0301-620X.96B3. 32279 $2.00 Bone Joint J 2014;96-B:414–19. Received 1 May 2013; Accepted after revision 13 November 2013
414
Since its introduction in the 1960s, the impact factor has steadily gained importance in the field of research. Garfield introduced it in 19551, Other indicators of the importance of a journal within scientific literature followed, including the Science Citation Index in 19612,3 and the Journal Citations Reports (JCR) in 1975.4,5 The Impact factor is calculated by dividing the number of citations in the current year by the number of citable or source items published in that journal in the previous two years. A list of impact factors of most journals is released by Thomson Reuters, a commercial company, as a part of JCR every year.5 Unfortunately the denominator, the ‘source items published’ is not available in the public domain as it is deemed a commercial secret.6 The strengths and weaknesses of the impact factor have been widely discussed.7-11 Several studies have assessed its effect in various scientific fields.12-14 It has become a benchmark by which to assess the quality of research15 in the wider scientific community. It is used as a surrogate marker to assess the excellence of a journal, and by many universities and grant funding bodies to assess the performance of academic departments and the quality of research.16 It is used as a quantitative tool to
rank, evaluate, categorise, and compare journals.2 Organisations, research departments and University bodies encourage authors to publish their research in journals with the highest impact factor in their field.17 Journals in each medical or surgical specialty are ranked according to impact factor, by Thomson Reuters: this ranking is considered by many academic bodies to reflect the quality of the published research. Some research assessment exercises give little or no credit for work that has not been published in a journal ranked in the top five within their specialty. The aim of this study was to assess whether the citation index of the highest ranked journals of trauma and orthopaedics reflects the content of the journals, the rate of citation at two years (when the impact factor of a journal is calculated) and the subsequent rate of citation at four or five years.
Methods We collected data from the 13 journals of trauma and orthopaedics with the highest impact factors in 2007 and 2008 according to Thomson Reuters (Table I). We categorised those that published mostly experimental or basic research as scientific journals and those that published mostly clinical research as THE BONE & JOINT JOURNAL
THE IMPACT FACTOR OF A JOURNAL IS A POOR MEASURE OF THE CLINICAL RELEVANCE OF ITS PAPERS
415
Table I. List of the top 13 journals in trauma by impact factor for 2007 and 2008 according to Thomsons Reuters Impact factor Journal
2007
2008
Arthritis and Rheumatism Journal of Bone and Mineral Research Arthritis Research and Therapy Osteoporosis International Bone Osteoarthritis and Cartilage American Journal of Sports Medicine Medicine and Science in Sports and Exercise JBJS (Am) Sports Medicine JBJS (Br) Injury Journal of Orthopaedic Trauma
7.677 6.004 4.035 3.893 3.966 3.793 3.397 2.864 2.487 3.619 1.868 1.509 1.429
6.787 6.443 4.485 4.29 4.145 4.082 3.646 3.399 3.313 3.018 2.196 1.946 1.877
Table II. Classification of papers Type
Subspecialty
Superspecialty
Experimental Diagnostic Review Clinical operative Clinical non-operative
Basic research Elective clinical Trauma clinical
Rheumatology Genetics Hip Knee Shoulder Spine Paediatrics Hand surgery Cancer/Tumour Infection Metabolic bone disease Bone density/Osteoporosis Biomechanics Tissue engineering Molecular Miscellaneous
clinical journals. All publications were cross-referenced with Google Scholar for citations at two, four and five years (for 2007) and two and four years (for 2008). We did this to simulate the impact factor published by Thomson Reuters, as the denominator is not available. Papers were then divided by type (experimental; diagnostic; review; clinical operative and non-operative). We further classified these by subspecialty (basic research, clinical (trauma or elective)) and superspecialty (molecular; osteoporosis or bone density; rheumatology; hip; knee; foot and ankle; hand surgery; paediatric orthopaedics; spine; biomechanics; malignancy; sports medicine and miscellaneous) (Table II).
Results We evaluated 4951 papers from 13 journals over the specified two-year period. Of these, 1914 were published in clinical journals and 3037 in scientific journals. Analysis of the papers by type showed that the ‘experimental’ category dominated the total number of publications (n = 2031, 41%). The vast majority of publications (n = 1765, 92.2%) VOL. 96-B, No. 3, MARCH 2014
in the scientific journals belonged to the ‘experimental’ category (Fig. 1). Basic research studies (n = 2197, 44.4%) formed the majority of subspecialty articles, of which 62.7% (n = 1904) were in scientific journals. The top five journals with the highest impact factor published three papers about operative orthopaedics (0.15%); none were on trauma. ‘Trauma clinical’ (n = 856) and ‘clinical operative’ (n = 830) papers were mostly published in clinical journals with a relatively low impact factor (Fig. 2). Analysis of super-specialties showed that ‘rheumatology’ was the most commonly published super-specialty (n = 402, 8.1%;) followed closely by ‘knee’ (n = 392, 7.9%;) (Fig. 3). Table III summarises the type of research published in each journal. We also compared the distribution of papers between Arthritis and Rheumatism (highest impact factor in our group) and the JBJS (AM) (Figs 4 and 5). We evaluated the research work in each journal individually and compared their subsequent citations for all subspecialties and super-specialties. Research in the basic sciences had a mean rate of citation of 4.2 at two years, 20.5 at four years and 30.35
416
P. KODUMURI, B. OLLIVERE, J. HOLLEY, C. G. MORAN
2031
Experimental
1782
Elective clinical Trauma clinical
856
Review
Basic research Clinical non-operative Clinical operative
461
Diagnostic
315 0
750
1500
2250
2197 922 830 0
3000
750
1500
2250
Fig. 1
Fig. 2
Bar chart showing type of papers.
Bar chart showing subspecialty of papers.
Rheumatology Knee Molecular Hip Bone density Shoulder Spine Paediatrics Pelvis Foot and ankle Infection Hand Cancer
3000
402 392 340 283 279 177 129 95 79 62 54 24
0
125
250
375
500
Fig. 3 Bar chart showing superspecialty of papers.
Table III. Journals with their most commonly published type of paper Journal
Mostly published type
Proportion (%)
Arthritis and Rheumatism Arthritis and Research Therapy Journal of Bone and Mineral Research Osteoporosis International Bone Osteoarthritis and Cartilage American Journal of Sports Medicine Medicine and Science in Sports and Exercise JBJS (Am) Sports Medicine JBJS (Br) Injury Journal of Orthopaedic Trauma
Experimental Experimental Experimental Clinical non-operative Experimental Clinical non-operative Clinical operative Experimental Clinical operative Experimental Clinical operative Review Clinical operative
50 58.25 50.7 62.6 62 63 31.8 92.8 48.7 75.3 59.2 33.8 60.2
at five years. Review articles received the most citations at each evaluation. By contrast research into operative trauma was cited less frequently at two years (2.18). However, this increased at the four- (12.56) and five-year (25.20) evaluations. ‘Elective operative’ also had a low citation rate at two years (2.25) but increased at four (14.63) and five years (25.60) (Fig. 6). Finally, we questioned whether the impact factor at two years was a true reflection of clinical research in terms of citations. Clinical journals had higher rankings based on these citations at two years than the impact factor rankings (Tables IV and V).
Discussion Our aim was to evaluate if the impact factor is a true reflection of clinical research in trauma and orthopaedics. We found that journals publishing basic research have a high rate of citation at two years compared with journals that concentrate on clinical research. Clinical research is cited less frequently in the first two years but thereafter the rate of citation increases to approach that of basic science by five years. This almost certainly reflects the nature of clinical trials, which take much longer to set up and undertake than laboratory studies. THE BONE & JOINT JOURNAL
THE IMPACT FACTOR OF A JOURNAL IS A POOR MEASURE OF THE CLINICAL RELEVANCE OF ITS PAPERS
53
Experimental
80
Trauma clinical
109
Clinical non-operative Hip
95
Knee
66 368
Elective clinical
0
125
250
379 375
Basic research Experimental Clinical operative Trauma clinical Clinical non-operative Hip Knee Elective clinical
283
Clinical operative
375
0 1 159 9 8 269
0
500
417
100
200
300
400
Fig. 5
Fig. 4
Bar chart showing distribution of papers in Arthritis and Rheumatism (n = 651).
Bar chart showing distribution of papers in JBJS (AM) (n = 561).
35 30 25
Basic sciences Clinical nonoperative Trauma operative Operative elective
30.35 29.50 26.50 25.60 20.50
20
20.15 14.63
15
12.56
10
7.92 4.20
5
2.25 2.18
0 0 year
2 yr
3 yr
4 yr
Fig. 6 Graphic representation of citations of various subspecialty papers in all journals.
Table IV. Impact factor and citation rankings of journals in the study in 2008. Journal
Ranking as per impact factors (2008)
Citations at two years
Ranking at two year citations
Citations at four years
Ranking at four year citations
Arthritis and Rheumatism Arthritis and Research Therapy Journal of Bone and Mineral Research Osteoporosis International Bone Osteoarthritis and Cartilage American Journal of Sports Medicine Medicine and Science in Sports and Exercise JBJS (Am) Sports Medicine JBJS (Br) Injury Journal of Orthopaedic Trauma
1 2 3 4 5 6 7 8 9 10 11 12 13
11.63 6.32 10.33 14.16 7.91 11.24 14.72 14.63 7.57 13.73 4.71 3.97 7.59
5 11 7 3 8 6 1 2 10 4 12 13 9
48.84 20.66 33 36.68 24.84 24.32 24.47 23.46 30.23 22.86 18.96 12.53 26.41
1 11 3 2 6 8 7 9 4 10 12 13 5
Universities and research funding bodies use the ranking list published by Thomson Reuters, even though it is obvious from the list that few of the journals in the top ten are directly relevant to trauma and orthopaedics. It is of concern that the five journals with highest impact factors published only three clinical papers (0.15%) on operative surgery in the two-year period of study. None were on VOL. 96-B, No. 3, MARCH 2014
trauma, which is the leading cause of death worldwide in patients under the age of 40 years.18 Although this is entirely appropriate for the journals, it puts surgical specialties at a considerable disadvantage if research assurance exercises use impact factor as a surrogate marker for excellence. Our study suggests that clinical research in surgery receives a high rate of citation, comparable to that of basic
418
P. KODUMURI, B. OLLIVERE, J. HOLLEY, C. G. MORAN
Table V. Impact factor and citation rankings of journals in the study in 2007. Journal Arthritis and Rheumatism Arthritis and Research Therapy Journal of Bone and Mineral Research Osteoporosis International Bone Osteoarthritis and Cartilage American Journal of Sports Medicine Medicine and Science in Sports and Exercise JBJS (Am) Sports Medicine JBJS (Br) Injury Journal of Orthopaedic Trauma
Ranking as per Citation at impact factors (2007) two years
Ranking at two-year citations
Citation at four years
Ranking at Citation at four-year citations five years
Ranking at five-year citations
1
32.6
1
44
1
53.64
1
2
21.8
2
32.1
2
42.66
2
3
17
3
23
7
32.08
3
4
10.8
8
17.6
9
29.62
7
5 6
13.12 16.08
7 4
23.16 24.37
6 4
31.69 30.87
4 6
7
14.31
5
24.35
5
25.77
8
8
4.1
12
7.3
12
10
12
9 10 11 12 13
14.1 2.5 5.74 8.84 10.07
6 13 11 10 9
24.5 5.9 10 18.35 16.12
3 13 11 8 10
31.5 8.1 13.77 25.1 22.5
5 13 11 9 10
research, but only after four or five years. This is particularly important in the context of research assessment and grant-giving bodies, which use the impact factor as a surrogate for the quality and value of research. If clinical or operative studies are published in journals with a lower impact factor, they may be deemed inferior, or of less clinical relevance. The new Research Excellence Framework (REF) in the UK is moving away from using impact factor to assess the quality of research, but the rate of citation is likely to remain a factor in any assessment. This will replace the Research Assessment Exercise (RAE) in 2014 as the system for assessing the quality of research in Higher Education Institutions (HEI) in the UK. The REF will produce assessment outcomes for each HEI submission, thus appraising research funding, providing accountability and establishing yardsticks for institutional reputation. The pilot exercise19 performed by Higher Education Funding Council for England (HEFCE), one of the four bodies involved in REF, evaluated the inequalities in citation scores. Their advisory group has warned against the use of citation data in business processes such as staff recruitment and funding research. The REF aims to evaluate all research output across all disciplines on a fair and equal basis.20,21 When assessing the performance of an academic department of trauma and orthopaedic surgery, it may be more appropriate to use a rate of citation at five years rather than two. Clearly, this is an area that requires further study. Several attempts to overcome the limitations of impact factors by new bibiliometric indicators have been performed without complete success.22-24 The Impact Factor is based on the rate of citation at two years and is not a true reflection of clinical research in
trauma and orthopaedics. This means that academic departments in our specialty are disadvantaged in the REF exercise. This study confirms that it is very difficult to publish clinical work about trauma and orthopaedics in journals with a high impact factor. This does not reflect the quality of the work, but simply the content and readership of the journals. We suggest that the impact factor calculated at five years or more might be a better quantitative tool by which to assess clinical research. The authors would like to thank Prof. B. E. Scammell for input with the manuscript. No benefits in any form have been received or will be received from a commercial party related directly or indirectly to the subject of this article. This article was primary edited by A. Ross and first proof edited by J. Scott.
References 1. Garfield E. Citation indexes to science: a new dimension in documentation through association of ideas. Science 1955;122:108–111. 2. No authors listed. Thomson Reuters. The Thomson Reuters Impact Factor. http:// thomsonreuters.com/products_services/science/free/essays/impact_factor/ (date last accessed 15 November 2013). 3. Garfield E, Sher I. Genetics Citation Index. In: Garfield E. Essays of an information scientist. Volume 7. Philadephia: ISI Press, 1985:517–520. 4. No authors listed. Thomson Reuters. History of Citation Indexing. http://thomsonreuters.com/products_services/science/free/essays/history_of_citation_indexing/ (date last accessed 15 November 2013). 5. Garfield E. SCI journal citation reports: a bibliometric analysis of science journals in the ISI database. Philadelphia: Institute for Scientific Information, 1993. 6. Golubic R, Rudes M, Kovacic N, Marusic M, Marusic A. Calculating impact factor: how bibliographical classification of journal items affects the impact factor of large and small journals. Sci Eng Ethics 2008;14:41–49. 7. Dimitrov JD, Kaveri SV, Bayry J. Metrics: journal's impact factor skewed by a single paper. Nature 2010;466:179. 8. Rossner M, Van Epps H, Hill E. Irreproducible results: a response to Thomson Scientific. J Exp Med 2008;205:260–261. 9. Rossner M, Van Epps H, Hill E. Show me the data. J Cell Biol 2007;179:1091–1092. 10. Pendlebury DA. The use and misuse of journal metrics and other citation indicators. Arch Immunol Ther Exp (Warsz) 2009;57:1–11. THE BONE & JOINT JOURNAL
THE IMPACT FACTOR OF A JOURNAL IS A POOR MEASURE OF THE CLINICAL RELEVANCE OF ITS PAPERS
11. Glänzel W, Moed HF. Journal impact measures in bibliometric research. Scientometrics 2002;53:171–193. 12. Pierce GN. Too much impact for the impact factor: are a new generation of scientists in peril? Can J Physiol Pharmacol 2012;90:. 13. Satyanarayana K. Impact factor and other indices to assess science, scientists and scientific journals. Indian J Physiol Pharmacol 2010;54:197–212. 14. AbdullGaffar B. Impact factor in cytopathology journals: what does it reflect and how much does it matter? Cytopathology 2012;23:5–320. 15. Parsons NR, Hiskens R, Price CL, Achten J, Costa ML. A systematic survey of the quality of research reporting in general orthopaedic journals. J Bone Joint Surg [Br] 2011;93-B:1154–1159. 16. Fuyuno I, Cyranoski D. Cash for papers: putting a premium on publication. Nature 2006;441:792. 17. McVeigh ME, Mann SJ. The journal impact factor denominator: defining citable (counted) items. J Am Med Assoc 2009;302:1107–1109. 18. Murray CJ, Lopez AD. Alternative projections of mortality and disability by cause 1990-2020: Global Burden of Disease Study. Lancet 1997;349:1498–1504.
VOL. 96-B, No. 3, MARCH 2014
419
19. No authors listed. Higher Education Funding Council for England: analysis of data from the pilot exercise to develop bibliometric indicators for the REF: the effect of using normalised citation scores for particular staff characteristics, 2011. http:// www.hefce.ac.uk/pubs/year/2011/201103/name,63893,en.html (date last accessed 15 November 2013). 20. No authors listed. Research Excellence Framework: assessment framework and guidance on submissions, 2011. http://www.ref.ac.uk/media/ref/content/pub/assessmentframeworkandguidanceonsubmissions/GOS%20including%20addendum.pdf (date last accessed 15 November 2013). 21. No authors listed. Research Excellence Framework: Research outputs (REF2). http://www.ref.ac.uk/faq/researchoutputsref2/ (date last accessed 15 November 2013). 22. Bergstrom CT, West JD, Wiseman MA. The Eigenfactor metrics. J Neurosci 2008;28:11433–11434. 23. Mathur VP, Sharma A. Impact factor and other standardized measures of journal citation: a perspective. Indian J Dent Res 2009;20:81–85. 24. Bollen J, Van de Sompel H, Hagberg A, Chute R. A principal component analysis of 39 scientific impact measures. PloS One 2009;4:6022.
The impact factor of a journal is a poor measure of the clinical relevance of its papers
P. Kodumuri, B. Ollivere, J. Holley, C. G. Moran From Queens Medical Centre, Nottingham, United Kingdom
We evaluated the top 13 journals in trauma and orthopaedics by impact factor and looked at the longer-term effect regarding citations of their papers. All 4951 papers published in these journals during 2007 and 2008 were reviewed and categorised by their type, subspecialty and super-specialty. All citations indexed through Google Scholar were reviewed to establish the rate of citation per paper at two, four and five years post-publication. The top five journals published a total of 1986 papers. Only three (0.15%) were on operative orthopaedic surgery and none were on trauma. Most (n = 1084, 54.5%) were about experimental basic science. Surgical papers had a lower rate of citation (2.18) at two years than basic science or clinical medical papers (4.68). However, by four years the rates were similar (26.57 for surgery, 30.35 for basic science/medical), which suggests that there is a considerable time lag before clinical surgical research has an impact. We conclude that high impact journals do not address clinical research in surgery and when they do, there is a delay before such papers are cited. We suggest that a rate of citation at five years post-publication might be a more appropriate indicator of importance for papers in our specialty. Cite this article: Bone Joint J 2014;96-B:414–19.
P. Kodumuri, MRCS, Specialty Registrar Queens Medical Centre, 16 Topliff Road, Chilwell, Nottingham, NG9 5AS, UK. B. Ollivere, MD, FRCS, Consultant Trauma and Orthopaedic Surgeon J. Holley, MRCS, Core Surgical Trainee C. G. Moran, MD, FRCS, Professor of Trauma and Orthopaedics Queens Medical Centre, Derby Road, Nottingham, NG7 2UH, UK. Correspondence should be sent to Mr P. Kodumuri; e-mail: [email protected] ©2014 The British Editorial Society of Bone & Joint Surgery doi:10.1302/0301-620X.96B3. 32279 $2.00 Bone Joint J 2014;96-B:414–19. Received 1 May 2013; Accepted after revision 13 November 2013
414
Since its introduction in the 1960s, the impact factor has steadily gained importance in the field of research. Garfield introduced it in 19551, Other indicators of the importance of a journal within scientific literature followed, including the Science Citation Index in 19612,3 and the Journal Citations Reports (JCR) in 1975.4,5 The Impact factor is calculated by dividing the number of citations in the current year by the number of citable or source items published in that journal in the previous two years. A list of impact factors of most journals is released by Thomson Reuters, a commercial company, as a part of JCR every year.5 Unfortunately the denominator, the ‘source items published’ is not available in the public domain as it is deemed a commercial secret.6 The strengths and weaknesses of the impact factor have been widely discussed.7-11 Several studies have assessed its effect in various scientific fields.12-14 It has become a benchmark by which to assess the quality of research15 in the wider scientific community. It is used as a surrogate marker to assess the excellence of a journal, and by many universities and grant funding bodies to assess the performance of academic departments and the quality of research.16 It is used as a quantitative tool to
rank, evaluate, categorise, and compare journals.2 Organisations, research departments and University bodies encourage authors to publish their research in journals with the highest impact factor in their field.17 Journals in each medical or surgical specialty are ranked according to impact factor, by Thomson Reuters: this ranking is considered by many academic bodies to reflect the quality of the published research. Some research assessment exercises give little or no credit for work that has not been published in a journal ranked in the top five within their specialty. The aim of this study was to assess whether the citation index of the highest ranked journals of trauma and orthopaedics reflects the content of the journals, the rate of citation at two years (when the impact factor of a journal is calculated) and the subsequent rate of citation at four or five years.
Methods We collected data from the 13 journals of trauma and orthopaedics with the highest impact factors in 2007 and 2008 according to Thomson Reuters (Table I). We categorised those that published mostly experimental or basic research as scientific journals and those that published mostly clinical research as THE BONE & JOINT JOURNAL
THE IMPACT FACTOR OF A JOURNAL IS A POOR MEASURE OF THE CLINICAL RELEVANCE OF ITS PAPERS
415
Table I. List of the top 13 journals in trauma by impact factor for 2007 and 2008 according to Thomsons Reuters Impact factor Journal
2007
2008
Arthritis and Rheumatism Journal of Bone and Mineral Research Arthritis Research and Therapy Osteoporosis International Bone Osteoarthritis and Cartilage American Journal of Sports Medicine Medicine and Science in Sports and Exercise JBJS (Am) Sports Medicine JBJS (Br) Injury Journal of Orthopaedic Trauma
7.677 6.004 4.035 3.893 3.966 3.793 3.397 2.864 2.487 3.619 1.868 1.509 1.429
6.787 6.443 4.485 4.29 4.145 4.082 3.646 3.399 3.313 3.018 2.196 1.946 1.877
Table II. Classification of papers Type
Subspecialty
Superspecialty
Experimental Diagnostic Review Clinical operative Clinical non-operative
Basic research Elective clinical Trauma clinical
Rheumatology Genetics Hip Knee Shoulder Spine Paediatrics Hand surgery Cancer/Tumour Infection Metabolic bone disease Bone density/Osteoporosis Biomechanics Tissue engineering Molecular Miscellaneous
clinical journals. All publications were cross-referenced with Google Scholar for citations at two, four and five years (for 2007) and two and four years (for 2008). We did this to simulate the impact factor published by Thomson Reuters, as the denominator is not available. Papers were then divided by type (experimental; diagnostic; review; clinical operative and non-operative). We further classified these by subspecialty (basic research, clinical (trauma or elective)) and superspecialty (molecular; osteoporosis or bone density; rheumatology; hip; knee; foot and ankle; hand surgery; paediatric orthopaedics; spine; biomechanics; malignancy; sports medicine and miscellaneous) (Table II).
Results We evaluated 4951 papers from 13 journals over the specified two-year period. Of these, 1914 were published in clinical journals and 3037 in scientific journals. Analysis of the papers by type showed that the ‘experimental’ category dominated the total number of publications (n = 2031, 41%). The vast majority of publications (n = 1765, 92.2%) VOL. 96-B, No. 3, MARCH 2014
in the scientific journals belonged to the ‘experimental’ category (Fig. 1). Basic research studies (n = 2197, 44.4%) formed the majority of subspecialty articles, of which 62.7% (n = 1904) were in scientific journals. The top five journals with the highest impact factor published three papers about operative orthopaedics (0.15%); none were on trauma. ‘Trauma clinical’ (n = 856) and ‘clinical operative’ (n = 830) papers were mostly published in clinical journals with a relatively low impact factor (Fig. 2). Analysis of super-specialties showed that ‘rheumatology’ was the most commonly published super-specialty (n = 402, 8.1%;) followed closely by ‘knee’ (n = 392, 7.9%;) (Fig. 3). Table III summarises the type of research published in each journal. We also compared the distribution of papers between Arthritis and Rheumatism (highest impact factor in our group) and the JBJS (AM) (Figs 4 and 5). We evaluated the research work in each journal individually and compared their subsequent citations for all subspecialties and super-specialties. Research in the basic sciences had a mean rate of citation of 4.2 at two years, 20.5 at four years and 30.35
416
P. KODUMURI, B. OLLIVERE, J. HOLLEY, C. G. MORAN
2031
Experimental
1782
Elective clinical Trauma clinical
856
Review
Basic research Clinical non-operative Clinical operative
461
Diagnostic
315 0
750
1500
2250
2197 922 830 0
3000
750
1500
2250
Fig. 1
Fig. 2
Bar chart showing type of papers.
Bar chart showing subspecialty of papers.
Rheumatology Knee Molecular Hip Bone density Shoulder Spine Paediatrics Pelvis Foot and ankle Infection Hand Cancer
3000
402 392 340 283 279 177 129 95 79 62 54 24
0
125
250
375
500
Fig. 3 Bar chart showing superspecialty of papers.
Table III. Journals with their most commonly published type of paper Journal
Mostly published type
Proportion (%)
Arthritis and Rheumatism Arthritis and Research Therapy Journal of Bone and Mineral Research Osteoporosis International Bone Osteoarthritis and Cartilage American Journal of Sports Medicine Medicine and Science in Sports and Exercise JBJS (Am) Sports Medicine JBJS (Br) Injury Journal of Orthopaedic Trauma
Experimental Experimental Experimental Clinical non-operative Experimental Clinical non-operative Clinical operative Experimental Clinical operative Experimental Clinical operative Review Clinical operative
50 58.25 50.7 62.6 62 63 31.8 92.8 48.7 75.3 59.2 33.8 60.2
at five years. Review articles received the most citations at each evaluation. By contrast research into operative trauma was cited less frequently at two years (2.18). However, this increased at the four- (12.56) and five-year (25.20) evaluations. ‘Elective operative’ also had a low citation rate at two years (2.25) but increased at four (14.63) and five years (25.60) (Fig. 6). Finally, we questioned whether the impact factor at two years was a true reflection of clinical research in terms of citations. Clinical journals had higher rankings based on these citations at two years than the impact factor rankings (Tables IV and V).
Discussion Our aim was to evaluate if the impact factor is a true reflection of clinical research in trauma and orthopaedics. We found that journals publishing basic research have a high rate of citation at two years compared with journals that concentrate on clinical research. Clinical research is cited less frequently in the first two years but thereafter the rate of citation increases to approach that of basic science by five years. This almost certainly reflects the nature of clinical trials, which take much longer to set up and undertake than laboratory studies. THE BONE & JOINT JOURNAL
THE IMPACT FACTOR OF A JOURNAL IS A POOR MEASURE OF THE CLINICAL RELEVANCE OF ITS PAPERS
53
Experimental
80
Trauma clinical
109
Clinical non-operative Hip
95
Knee
66 368
Elective clinical
0
125
250
379 375
Basic research Experimental Clinical operative Trauma clinical Clinical non-operative Hip Knee Elective clinical
283
Clinical operative
375
0 1 159 9 8 269
0
500
417
100
200
300
400
Fig. 5
Fig. 4
Bar chart showing distribution of papers in Arthritis and Rheumatism (n = 651).
Bar chart showing distribution of papers in JBJS (AM) (n = 561).
35 30 25
Basic sciences Clinical nonoperative Trauma operative Operative elective
30.35 29.50 26.50 25.60 20.50
20
20.15 14.63
15
12.56
10
7.92 4.20
5
2.25 2.18
0 0 year
2 yr
3 yr
4 yr
Fig. 6 Graphic representation of citations of various subspecialty papers in all journals.
Table IV. Impact factor and citation rankings of journals in the study in 2008. Journal
Ranking as per impact factors (2008)
Citations at two years
Ranking at two year citations
Citations at four years
Ranking at four year citations
Arthritis and Rheumatism Arthritis and Research Therapy Journal of Bone and Mineral Research Osteoporosis International Bone Osteoarthritis and Cartilage American Journal of Sports Medicine Medicine and Science in Sports and Exercise JBJS (Am) Sports Medicine JBJS (Br) Injury Journal of Orthopaedic Trauma
1 2 3 4 5 6 7 8 9 10 11 12 13
11.63 6.32 10.33 14.16 7.91 11.24 14.72 14.63 7.57 13.73 4.71 3.97 7.59
5 11 7 3 8 6 1 2 10 4 12 13 9
48.84 20.66 33 36.68 24.84 24.32 24.47 23.46 30.23 22.86 18.96 12.53 26.41
1 11 3 2 6 8 7 9 4 10 12 13 5
Universities and research funding bodies use the ranking list published by Thomson Reuters, even though it is obvious from the list that few of the journals in the top ten are directly relevant to trauma and orthopaedics. It is of concern that the five journals with highest impact factors published only three clinical papers (0.15%) on operative surgery in the two-year period of study. None were on VOL. 96-B, No. 3, MARCH 2014
trauma, which is the leading cause of death worldwide in patients under the age of 40 years.18 Although this is entirely appropriate for the journals, it puts surgical specialties at a considerable disadvantage if research assurance exercises use impact factor as a surrogate marker for excellence. Our study suggests that clinical research in surgery receives a high rate of citation, comparable to that of basic
418
P. KODUMURI, B. OLLIVERE, J. HOLLEY, C. G. MORAN
Table V. Impact factor and citation rankings of journals in the study in 2007. Journal Arthritis and Rheumatism Arthritis and Research Therapy Journal of Bone and Mineral Research Osteoporosis International Bone Osteoarthritis and Cartilage American Journal of Sports Medicine Medicine and Science in Sports and Exercise JBJS (Am) Sports Medicine JBJS (Br) Injury Journal of Orthopaedic Trauma
Ranking as per Citation at impact factors (2007) two years
Ranking at two-year citations
Citation at four years
Ranking at Citation at four-year citations five years
Ranking at five-year citations
1
32.6
1
44
1
53.64
1
2
21.8
2
32.1
2
42.66
2
3
17
3
23
7
32.08
3
4
10.8
8
17.6
9
29.62
7
5 6
13.12 16.08
7 4
23.16 24.37
6 4
31.69 30.87
4 6
7
14.31
5
24.35
5
25.77
8
8
4.1
12
7.3
12
10
12
9 10 11 12 13
14.1 2.5 5.74 8.84 10.07
6 13 11 10 9
24.5 5.9 10 18.35 16.12
3 13 11 8 10
31.5 8.1 13.77 25.1 22.5
5 13 11 9 10
research, but only after four or five years. This is particularly important in the context of research assessment and grant-giving bodies, which use the impact factor as a surrogate for the quality and value of research. If clinical or operative studies are published in journals with a lower impact factor, they may be deemed inferior, or of less clinical relevance. The new Research Excellence Framework (REF) in the UK is moving away from using impact factor to assess the quality of research, but the rate of citation is likely to remain a factor in any assessment. This will replace the Research Assessment Exercise (RAE) in 2014 as the system for assessing the quality of research in Higher Education Institutions (HEI) in the UK. The REF will produce assessment outcomes for each HEI submission, thus appraising research funding, providing accountability and establishing yardsticks for institutional reputation. The pilot exercise19 performed by Higher Education Funding Council for England (HEFCE), one of the four bodies involved in REF, evaluated the inequalities in citation scores. Their advisory group has warned against the use of citation data in business processes such as staff recruitment and funding research. The REF aims to evaluate all research output across all disciplines on a fair and equal basis.20,21 When assessing the performance of an academic department of trauma and orthopaedic surgery, it may be more appropriate to use a rate of citation at five years rather than two. Clearly, this is an area that requires further study. Several attempts to overcome the limitations of impact factors by new bibiliometric indicators have been performed without complete success.22-24 The Impact Factor is based on the rate of citation at two years and is not a true reflection of clinical research in
trauma and orthopaedics. This means that academic departments in our specialty are disadvantaged in the REF exercise. This study confirms that it is very difficult to publish clinical work about trauma and orthopaedics in journals with a high impact factor. This does not reflect the quality of the work, but simply the content and readership of the journals. We suggest that the impact factor calculated at five years or more might be a better quantitative tool by which to assess clinical research. The authors would like to thank Prof. B. E. Scammell for input with the manuscript. No benefits in any form have been received or will be received from a commercial party related directly or indirectly to the subject of this article. This article was primary edited by A. Ross and first proof edited by J. Scott.
References 1. Garfield E. Citation indexes to science: a new dimension in documentation through association of ideas. Science 1955;122:108–111. 2. No authors listed. Thomson Reuters. The Thomson Reuters Impact Factor. http:// thomsonreuters.com/products_services/science/free/essays/impact_factor/ (date last accessed 15 November 2013). 3. Garfield E, Sher I. Genetics Citation Index. In: Garfield E. Essays of an information scientist. Volume 7. Philadephia: ISI Press, 1985:517–520. 4. No authors listed. Thomson Reuters. History of Citation Indexing. http://thomsonreuters.com/products_services/science/free/essays/history_of_citation_indexing/ (date last accessed 15 November 2013). 5. Garfield E. SCI journal citation reports: a bibliometric analysis of science journals in the ISI database. Philadelphia: Institute for Scientific Information, 1993. 6. Golubic R, Rudes M, Kovacic N, Marusic M, Marusic A. Calculating impact factor: how bibliographical classification of journal items affects the impact factor of large and small journals. Sci Eng Ethics 2008;14:41–49. 7. Dimitrov JD, Kaveri SV, Bayry J. Metrics: journal's impact factor skewed by a single paper. Nature 2010;466:179. 8. Rossner M, Van Epps H, Hill E. Irreproducible results: a response to Thomson Scientific. J Exp Med 2008;205:260–261. 9. Rossner M, Van Epps H, Hill E. Show me the data. J Cell Biol 2007;179:1091–1092. 10. Pendlebury DA. The use and misuse of journal metrics and other citation indicators. Arch Immunol Ther Exp (Warsz) 2009;57:1–11. THE BONE & JOINT JOURNAL
THE IMPACT FACTOR OF A JOURNAL IS A POOR MEASURE OF THE CLINICAL RELEVANCE OF ITS PAPERS
11. Glänzel W, Moed HF. Journal impact measures in bibliometric research. Scientometrics 2002;53:171–193. 12. Pierce GN. Too much impact for the impact factor: are a new generation of scientists in peril? Can J Physiol Pharmacol 2012;90:. 13. Satyanarayana K. Impact factor and other indices to assess science, scientists and scientific journals. Indian J Physiol Pharmacol 2010;54:197–212. 14. AbdullGaffar B. Impact factor in cytopathology journals: what does it reflect and how much does it matter? Cytopathology 2012;23:5–320. 15. Parsons NR, Hiskens R, Price CL, Achten J, Costa ML. A systematic survey of the quality of research reporting in general orthopaedic journals. J Bone Joint Surg [Br] 2011;93-B:1154–1159. 16. Fuyuno I, Cyranoski D. Cash for papers: putting a premium on publication. Nature 2006;441:792. 17. McVeigh ME, Mann SJ. The journal impact factor denominator: defining citable (counted) items. J Am Med Assoc 2009;302:1107–1109. 18. Murray CJ, Lopez AD. Alternative projections of mortality and disability by cause 1990-2020: Global Burden of Disease Study. Lancet 1997;349:1498–1504.
VOL. 96-B, No. 3, MARCH 2014
419
19. No authors listed. Higher Education Funding Council for England: analysis of data from the pilot exercise to develop bibliometric indicators for the REF: the effect of using normalised citation scores for particular staff characteristics, 2011. http:// www.hefce.ac.uk/pubs/year/2011/201103/name,63893,en.html (date last accessed 15 November 2013). 20. No authors listed. Research Excellence Framework: assessment framework and guidance on submissions, 2011. http://www.ref.ac.uk/media/ref/content/pub/assessmentframeworkandguidanceonsubmissions/GOS%20including%20addendum.pdf (date last accessed 15 November 2013). 21. No authors listed. Research Excellence Framework: Research outputs (REF2). http://www.ref.ac.uk/faq/researchoutputsref2/ (date last accessed 15 November 2013). 22. Bergstrom CT, West JD, Wiseman MA. The Eigenfactor metrics. J Neurosci 2008;28:11433–11434. 23. Mathur VP, Sharma A. Impact factor and other standardized measures of journal citation: a perspective. Indian J Dent Res 2009;20:81–85. 24. Bollen J, Van de Sompel H, Hagberg A, Chute R. A principal component analysis of 39 scientific impact measures. PloS One 2009;4:6022.
