SPECIAL TOPIC Journal Impact Factor versus the Evidence Level of Articles Published in Plastic Surgery Journals Maria A. Rodrigues, P.T. Ana C. B. Tedesco, P.T. Fabio X. Nahas, M.D., Ph.D., M.B.A. Lydia M. Ferreira, M.D., Ph.D., M.B.A. São Paulo, Brazil
Background: The aim of this study was to assess the correlation between impact factor and the level of evidence of articles in plastic surgery journals. Methods: The four plastic surgery journals with the top impact factors in 2011 were selected. Articles were selected using the PubMed database between January 1 and December 31, 2011. The journal evidence index was calculated by dividing the number of randomized clinical trials by the total number of articles published in the specific journal, multiplied by 100. This index was correlated to the impact factor of the journal and compared with the average of the other journals. Two investigators independently evaluated each journal, followed by a consensus and assessment of the interexaminer concordance. The kappa test was used to evaluate the concordance between the two investigators and Fisher’s exact test was used to evaluate which journal presented the highest number of randomized clinical trials. Results: The journal evidence index values were as follows: Plastic and Reconstructive Surgery, 1.70; Journal of Plastic, Reconstructive and Aesthetic Surgery, 0.40; Aesthetic Plastic Surgery, 0.56; and Annals of Plastic Surgery, 0.35. The impact factors of these journals in 2011 were as follows: Plastic and Reconstructive Surgery, 3.382; Journal of Plastic, Reconstructive and Aesthetic Surgery, 1.494; Aesthetic Plastic Surgery, 1.407; and Annals of Plastic Surgery, 1.318. After consensus, the quantity of adequate studies was low and similar between these journals; only the journal Plastic and Reconstructive Surgery showed a higher journal evidence index. Conclusions: The journal Plastic and Reconstructive Surgery exhibited the highest journal evidence index and had the highest impact factor. The number of adequate articles was low in all of the assessed journals. (Plast. Reconstr. Surg. 133: 1502, 2014.)
T
he methodologic quality and relevance of published studies are the main factors that promote their citation and affect the impact factor of a journal. Nevertheless, poor-quality clinical studies are still published in various medical fields.1,2 Some scientists reject evidence-based medicine studies because of the presence of sources of bias that threaten the reliability of the data, such as the choice of a single language for data selection, prejudice, citation bias, database bias, and multiple publication bias.3 Randomized clinical trials provide the best available source of scientific information and the only scientifically reliable method of assessing the efficacy of interventions. Therefore, the design, development, and the journal From the Division of Plastic Surgery, Federal University of São Paulo. Received for publication November 5, 2013; accepted December 12, 2013. Copyright © 2014 by the American Society of Plastic Surgeons DOI: 10.1097/PRS.0000000000000214
1502
of publication of randomized clinical trials must be of high quality.4 Poorly described randomized clinical trials are implicitly considered to have also been poorly performed and to have low quality, and the intervention effect is usually considered to be overestimated.5,6 Evidence-based medicine provides an approach for selecting the best treatment for a given patient and may therefore contribute to reductions in mortality and the costs of health care and improvement in patients’ satisfaction with treatment.7 A consensus meeting was organized to state the purpose of evidence-based medicine, and the adaptation of research within this process was also defined.8 Plastic surgery researchers are resistant to evidence-based medicine; therefore, there is still a long way to go in this field. Disclosure: The authors have no financial interest to declare in relation to the content of this article.
www.PRSJournal.com
Volume 133, Number 6 • Impact Factor vs Journal Evidence Index Through careful and efficient analysis of the available literature, evidence-based medicine must complement the surgeon’s judgment in decisionmaking. However, the knowledge of plastic surgeons is based mostly on the clinical experience of experts; in other words, technical experience and common sense are considered the crucial factors for success in plastic surgery.9 The type and quality of the selected articles, the level of the editorial board members and reviewers, the language used, and an indexed database are important factors that contribute to increase the impact factor of a journal.10 Furthermore, publication in a journal with a high impact factor contributes to establish the reputation of authors, institutions, and, eventually, countries.11 Novel methods of evaluating studies and journals facilitate the development of scientific research. Therefore, the present study was designed to assess the relationship between the impact factor and the evidence level of articles of plastic surgery journals. Also, this study aims to investigate interexaminer concordance in the assessment of the methodologic quality of studies with the highest level of evidence published in plastic surgery journals.
MATERIALS AND METHODS The inclusion criteria were as follows: the four plastic surgery journals with the highest impact factors listed in the Journal Citation Report of Web of Knowledge; journals that publish articles on general reconstructive or cosmetic surgery; and journals listed for at least 5 years in the Journal Citation Report. Nonsystematic review articles were excluded. The following plastic surgery journals were selected among those with the highest impact factors (Table 1): Plastic and Reconstructive Table 1. Top Impact Factor Journals of Plastic Surgery in 2011* Journal Plastic and Reconstructive Surgery Archives of Facial Plastic Surgery Journal of Plastic, Reconstructive, and Aesthetic Surgery Aesthetic Surgery Journal Clinics in Plastic Surgery Aesthetic Plastic Surgery Annals of Plastic Surgery Facial Plastic Surgery Scandinavian Journal of Plastic and Reconstructive Surgery Journal of Plastic Surgery and Hand Surgery *From Journal Citation Report of Web of Knowledge.
Impact Factor 3.382 1.646 1.494 1.469 1.422 1.407 1.318 0.963 0.935 0.017
Surgery; Journal of Plastic, Reconstructive, and Aesthetic Surgery; Aesthetic Plastic Surgery; and Annals of Plastic Surgery. The journals ranked as second and fifth based on impact factor (i.e., Archives of Facial Plastic Surgery and Clinics in Plastic Surgery, respectively, were excluded from the analysis because the former does not cover all areas of plastic surgery and the latter exclusively publishes nonsystematic literature reviews rather than randomized clinical trials). The journal ranked as fourth, the Aesthetic Surgery Journal, was also excluded because it was not listed on the Journal Citation Report for at least 5 years. Following the classification of the journals, the studies were selected by searching the PubMed database, as all of those journals are found in this database. Studies published from January 1 to December 31, 2011, were selected. Filters for systematic reviews, meta-analyses, clinical trials, and randomized clinical trials were applied separately to each journal that had been previously selected using the “advanced search” available at the PubMed Web site. Letters, editorials, and communications were excluded. The journal evidence index was calculated as the number of appropriate randomized clinical trials published by a journal divided by its total number of full-length articles published in a single year, multiplied by 100. This index was correlated with the journal impact factor and the weighted mean of the journal evidence index of the other journals. The Cook et al.12 classification, based on the Cochrane criteria, was used. Articles are classified into seven different levels of evidence, as follows: I, systematic review; II, mega trial (randomized clinical trial with >1000 patients); III, randomized clinical trial with fewer than 1000 patients; IV, cohort study design; V, case-control study design; VI, case series; and VII, case reports. Two examiners with a similar level of expertise and who had independently attended the Cochrane online course on systematic review performed the same stages of study selection and independently classified the articles based on their study design and patient selection. The following criteria were established to assess the randomized clinical trials: (1) description of the randomization techniques, including allocation concealment and stages of group distribution; and (2) performance of treatment comparisons. For systematic reviews, the following criteria were established: (1) to assess whether the clinical question was formulated; (2) to assess whether the description of the strategy used for searching
1503
Plastic and Reconstructive Surgery • June 2014 for clinical studies was present; and (3) whenever possible, to assess whether statistical analysis of the results was performed. A consensus between examiners was performed to assess which studies were considered adequate among those that did not show concordance. Statistical Analysis Statistical analysis was performed using the kappa test to assess interexaminer concordance, and the Fisher’s exact test was used to identify journals with the largest number of adequate studies. The Altman criteria13 were used to classify the reliability coefficient obtained in the kappa test.
Table 2. Number of Adequate Randomized Clinical Trials after Interexaminer Concordance for Each Journal and the Total Number of Published Studies* Journals
Level I, II, and III
Total No. of Studies
JEI
IF (2011)
PRS JPRAS APS AnPS Total Average
11 03 02 01 17 4.25
646 745 354 285 2.030 507.5
1.70 0.40 0.56 0.35 3.01 0.84
3.382 1.494 1.407 1.318
JEI, journal evidence index; IF, impact factor; PRS, Plastic and Reconstructive Surgery; JPRAS, Journal of Plastic, Reconstructive, and Aesthetic Surgery; APS, Aesthetic Plastic Surgery; AnPS, Annals of Plastic Surgery. *The journal evidence index and the average journal evidence index of plastic surgery journals in the studied period and the journal impact factor during the year of 2011 are also displayed.
RESULTS
DISCUSSION
A total of 116 articles, levels I, II and III— systematic reviews, mega trials, and randomized clinical trials—were identified, of which 71 were systematic reviews and 45 were randomized clinical trials, although only 100 were included in the present study. It was verified by the examiners that 16 articles did not fit into the characteristics of randomized clinical trials or systematic reviews. Subsequently, following the examiners’ consensus, an additional four articles were not considered as high–evidence level studies and were excluded. Plastic and Reconstructive Surgery published 64 (55.18 percent) of the selected articles, the Journal of Plastic, Reconstructive, and Aesthetic Surgery published 18 (15.52 percent), and Aesthetic Plastic Surgery and Annals of Plastic Surgery published nine (7.75 percent) each. To analyze the interexaminer concordance, the kappa reliability coefficient was calculated, which indicated moderate to excellent reliability according to the following Altman criteria: poor (κ < 0.20); reasonable (0.21 ≥ κ ≤ 0.40); moderate (0.41 ≥ κ ≤ 0.60); good (0.61 ≥ κ ≤ 0.80); and excellent (0.81 ≥ κ ≤ 1.00). Table 2 shows the absolute and mean journal evidence index and impact factor values of the investigated journals. Table 3 shows that the interexaminer concordance was moderate relative to Plastic and Reconstructive Surgery (p < 0.001) and the Journal of Plastic, Reconstructive, and Aesthetic Surgery (p = 0.058), good relative to Aesthetic Plastic Surgery (p = 0.018), and excellent relative to Annals of Plastic Surgery (p = 0.003). Table 4 shows that following consensus among the examiners, the incidence of adequate studies was similar in most journals, with Aesthetic Plastic Surgery being the journal with the highest percentage of studies considered adequate.
One major concern among editors of scientific journals is the identification and publication of studies with a high level of evidence, as these decisions enable doctors to make decisions based on up-to-date and consistent information and result in the best possible treatment for patients.9 The impact factor guides researchers in the selection of a journal in which to publish their studies, as researchers generally agree that the best articles are published in journals with high impact factors.
1504
Table 3. Interexaminer Concordance Regarding the Number of Level I Studies per Journal Published in 2011* Examiner 1 Examiner 2 Plastic and Reconstructive Surgery† Adequate Inadequate Total Journal of Plastic, Reconstructive, and Aesthetic Surgery‡ Adequate Inadequate Total Aesthetic Plastic Surgery§ Adequate Inadequate Total Annals of Plastic Surgery‖ Adequate Inadequate Total
Adequate
Inadequate
Total
17 7 24
7 33 40
24 40 64
6 3 9
2 7 9
8 10 10
5 0 5
1 3 4
6 3 9
3 0 3
0 6 6
3 6 9
*Statistically significant difference for p < 0.05. †κ = 0.533, p < 0.001 (moderate). ‡κ = 0.444, p = 0.058 (moderate). §κ = 0.769, p = 0.018 (good). ‖κ = 1.000, p = 0.003 (excellent).
Volume 133, Number 6 • Impact Factor vs Journal Evidence Index Table 4. Number of Articles Listed According to PubMed as Being Systematic Reviews and Randomized Clinical Trials That Were Considered Adequate and Inadequate, According to the Examiners, per Year, in Each Journal* Journal PRS JPRAS APS AnPS Total
Adequate
% Adequate
Inadequate
% Inadequate
Total
22 7 5 3 37
36.0 41.2 55.5 33.3 38.5
39 10 4 6 59
63.9 58.8 44.4 66.7 61.5
61 17 9 9 96
PRS, Plastic and Reconstructive Surgery; JPRAS, Journal of Plastic, Reconstructive, and Aesthetic Surgery; APS, Aesthetic Plastic Surgery; AnPS, Annals of Plastic Surgery. *p = 0.6998, Fisher’s exact test; statistically significant difference for p < 0.05.
Since July of 2011, plastic surgery journals began to use a pyramid scale to classify articles according to their level of evidence; it has improved the use of the acquired knowledge by the readers in their practices.14 In turn, the journal evidence index assesses the proportion of adequate randomized clinical trials published by a journal in a single year. This type of assessment is important to illustrate the influence of evidencebased medicine on plastic surgery. Some journals also publish a large number of letters, editorials, or communications, which could influence the results. For this reason, these types of publications were excluded from the calculation of the total number of published articles in the current study. Plastic and Reconstructive Surgery exhibited the highest impact factor and highest journal evidence index and was the only journal with a journal evidence index above the average. These results likely explain the preference of authors for sending their studies to this journal. In addition, the journal evidence index of Aesthetic Plastic Surgery was higher compared with that of the Journal of Plastic, Reconstructive, and Aesthetic Surgery and Annals of Plastic Surgery, although the impact factor of the Journal of Plastic, Reconstructive, and Aesthetic Surgery was higher. There is always a tendency for researchers to submit articles with the highest evidence levels to journals with higher impact factors. However, it is not true that the others journals do not receive these types of articles, as we could demonstrate in this study. The idea of the present study was to evaluate whether there would be a clear relation between the impact factor and the number of high–evidence level studies, which could not be established. Between 2007 and 2011, Aesthetic Plastic Surgery was the journal whose impact factor exhibited the greatest percentage increase (151.7 percent), followed by the Journal of Plastic, Reconstructive, and Aesthetic Surgery (139.8 percent), Plastic and Reconstructive Surgery (63.1 percent), and Annals
of Plastic Surgery (28.1 percent), according to the Journal Citation Reports. Therefore, Aesthetic Plastic Surgery was the journal with the greatest impact factor increase and was ranked second in journal evidence index. A high journal evidence index will be reflected in the impact factor in the ensuing few years. Although Plastic and Reconstructive Surgery is the journal with the largest number of level I evidence articles, it did not exhibit a similar proportionality in impact factor progression. The present study found a low frequency of randomized clinical trials published in the field of plastic surgery (0.84 percent), which agrees with the results of Moraes et al.,14 who found a mere 2.4 percent of randomized clinical trials in a journal of orthopedics and 1.7 percent in another journal from the same field. Sinno et al.15 showed that only 22 percent of the articles studied in the four main plastic surgery journals presented level I or II evidence. Another study16 that has evaluated the aesthetic surgery literature has shown that most studies were level IV (50 percent) and V (23 percent) evidence. Veiga et al.17 investigated the distribution of randomized clinical trials in journals of plastic surgery from 1996 to 2003 and found that 38.2 percent (n = 13) were published in Plastic and Reconstructive Surgery, 29.4 percent (n = 10) were published in the Journal of Plastic, Reconstructive, and Aesthetic Surgery, 5.9 percent (n = 2) were published in Annals of Plastic Surgery, and 26.5 percent (n = 9) were published in other journals. These authors performed the same analysis for the period from 2004 to 2008 and found that 46.4 percent (n = 13) of the randomized clinical trials were published in Plastic and Reconstructive Surgery, 7.1 percent (n = 2) were published in the Journal of Plastic, Reconstructive, and Aesthetic Surgery, 3.6 percent (n = 1) were published in Annals of Plastic Surgery, and 42.9 percent (n = 12) were published in other journals. An increase, albeit small, in the total number of published randomized clinical trials when these periods were compared was noted, according to Veiga et al.17
1505
Plastic and Reconstructive Surgery • June 2014 Standardization of the rules for publication and the journal’s identification of the evidence level of scientific articles increase the demand for higher quality articles with reliable results, thus improving the contributions made by science to society at large. To assess the limitations and power of randomized clinical trials, readers must be provided data regarding the methodologic quality of the study. This requirement, however, remains poorly met in spite of the overall advancement of science. Flaws in reporting make proper identification of randomized clinical trials difficult or even impossible. In addition, these flaws potentially contribute to incorrect decision-making in clinical practice when the results of biased studies are attributed undeserved reliability. For that reason, Consolidated Standards of Reporting Trials, recommendations established in the 1990s, have increasingly been used as standards to improve the quality of randomized clinical trials.18 We agree with Moraes et al.19 that a lack of tradition in scientific research by researchers leads to low methodologic quality, especially when the clinical question is not clear. The use of scales to assess methodologic quality by authors submitting randomized clinical trials should be suggested by all journals to increase the reliability of the results, and this system could also be applied to systematic reviews. The interexaminer concordance for the articles published in the assessed journals varied from moderate to excellent, as measured by the kappa reliability coefficient,13 thus ensuring the reliability of the results. To increase reliability, factors such as allocation concealment, randomization, masking, blinding, relevant clinical question, statistical analysis, and sample size should be improved in plastic surgery articles. If an objective description of these factors is not properly displayed in the article, the study may not be considered as a level I evidence article. It is clearly very difficult to conduct studies of surgery with high levels of evidence, and they are seldom performed because they are time consuming and very expensive.14,17 These factors account for the small number of adequate articles found. In addition, when such studies are performed, doubts are often cast on patient selection and results because of the low quality of the description of these factors in the articles. One additional factor that makes research in plastic surgery difficult is the scarcity of validated instruments for assessment to compare the results of surgical procedures, as the assessment of the results is generally based on the judgment of the
1506
patient and doctor. Thus, the use of standardized instruments should be promoted to advance the field of plastic surgery. The difficulty of reaching an interexaminer consensus in the present study highlights the difficulties met by investigators in the selection of articles for systematic reviews on plastic surgery. It is important for investigators to concern themselves with establishing what journals publish the studies with the best evidence and appropriate descriptions, rather than focusing on the journals that carry the largest numbers of cited articles. The impact factor considers only citations of recent publications, and it decreases with the increase of the total number of articles published in the journal. Bernstein and Gray20 suggests the use of the content factor, which considers the total number of citations of articles of a journal in a given year, regardless of the year in which the cited article was published, to all the articles the journal had published up to and including the year in question. The main advantage of the content factor is that it considers all previous citations. Evidence-based medicine does not deny the value of personal experience but recommends forming this knowledge based on evidence provided by clinical studies, which thereby reduces uncertainty in the field of health.21 In spite of the large number of published studies; few clinical trials exhibit level I, II, and III evidence. In addition, these clinical trials often do not provide the data needed to ensure their reliability or enable their inclusion in systematic reviews.
CONCLUSIONS The journal Plastic and Reconstructive Surgery exhibited the highest journal evidence index and had the highest impact factor. Interexaminer concordance in the analysis of the methodologic quality of articles published in the main journals of plastic surgery varied from moderate to excellent. The number of adequate articles was low in all of the assessed journals. Fabio Xerfan Nahas, M.D., Ph.D., M.B.A. Division of Plastic Surgery Federal University of São Paulo Rua Napoleão de Barros 715, 4° andar CEP 04024-002 São Paulo, Brazil [email protected]
REFERENCES 1. Riera R. Designs of studies published in two Brazilian journals of orthopedics and sports medicine, recently indexed in the ISI Web of Science. Sao Paulo Med J. 2009;127:355–358.
Volume 133, Number 6 • Impact Factor vs Journal Evidence Index 2. Hirji KF. No short-cut in assessing trial quality: A case study. Trials 2009;10:1–10. 3. Aalaei-Andabili SH, Alavian SM. Principles of meta-analysis should be well understood. Obes Surg. 2012;22:1926–1927. 4. Veiga Filho J, Castro AA, Veiga DF, et al. Quality of reports of randomized clinical trials in plastic surgery. Plast Reconstr Surg. 2005;115:320–323. 5. Karri V. Randomised clinical trials in plastic surgery: Survey of output and quality of reporting. J Plast Reconstr Aesthet Surg. 2006;59:787–796. 6. Hochman B, Nahas FX, Oliveira Filho RS, Ferreira LM. Research designs (in Portuguese). Acta Cir Bras. 2005;(Suppl 2):2–9. 7. McCarthy JE, Chatterjee A, McKelvey TG, Jantzen EM, Kerrigan CL. A detailed analysis of level I evidence (randomized controlled trials and meta-analyses) in five plastic surgery journals to date: 1978 to 2009. Plast Reconstr Surg. 2010;126:1774–1778. 8. Eaves FF III, Rohrich RJ, Sykes JM. Taking evidence-based plastic surgery to the next level: Report of the second Summit on Evidence-based Plastic Surgery. Aesthet Surg J. 2013;33:735–743. 9. Eaves F III, Pusic AL. Why evidence-based medicine matters to aesthetic surgery. Aesthet Surg J. 2012;32:117–119. 10. Tedesco AB, Nahas FX, Ferreira LM. The importance of the use of descriptors of the Medical Subject Heading (MeSH) in plastic surgery journals. Plast Reconstr Surg. 2010;126:222e. 11. Petroianu A. Quantitative criteria to assess the value of the publication of scientific articles (in Portuguese). Rev Assoc Med Bras. 2003;49:173–176.
12. Cook DJ, Guyatt GH, Laupacis A, Sackett DL, Goldberg RJ. Clinical recommendations using levels of evidence for antithrombotic agents. Chest 1995;108(Suppl):227S–230S. 13. Altman DG. Practical Statistics for Medical Research. 1st ed. London: Chapman & Hall; 1991:69–81. 14. Moraes VY, Moreira CD, Tamaoki MJC, Faloppa F, Belloti JC. Randomized clinical trials in orthopedics and traumatology: Systematic assessment of the national evidence (in Portuguese). Rev Bras Ortop. 2010;45:601–605. 15. Sinno H, Neel OF, Lutfy J, Bartlett G, Gilardino M. Level of evidence in plastic surgery research. Plast Reconstr Surg. 2011;127:974–980. 16. Chuback JE, Yarascavitch BA, Eaves F III, Thoma A, Bhandari M. Evidence in the aesthetic surgical literature over the past decade: How far have we come? Plast Reconstr Surg. 2012;129:126e–134e. 17. Veiga DF, Veiga-Filho J, Pellizzon RF, Juliano Y, Ferreira LM. Evolution of reports of randomised clinical trials in plastic surgery. J Plast Reconstr Aesthet Surg. 2011;64:703–709. 18. Dainesi SM, Aligieri P. How might the CONSORT statements ensure quality in clinical study reporting? (in Portuguese). Rev Assoc Med Bras. 2005;51:66. 19. Moraes VY, Belloti JC, Moraes FY, et al. Hierarchy of evidence relating to hand surgery in Brazilian orthopedic journals. Sao Paulo Med J. 2011;129:94–98. 20. Bernstein J, Gray CF. Content factor: A measure of a journal’s contribution to knowledge. PLoS One 2012;7: e41554. 21. El Dib RP. How to practice evidence-based medicine. J Vasc Bras. 2007;6:1–4.
1507
Background: The aim of this study was to assess the correlation between impact factor and the level of evidence of articles in plastic surgery journals. Methods: The four plastic surgery journals with the top impact factors in 2011 were selected. Articles were selected using the PubMed database between January 1 and December 31, 2011. The journal evidence index was calculated by dividing the number of randomized clinical trials by the total number of articles published in the specific journal, multiplied by 100. This index was correlated to the impact factor of the journal and compared with the average of the other journals. Two investigators independently evaluated each journal, followed by a consensus and assessment of the interexaminer concordance. The kappa test was used to evaluate the concordance between the two investigators and Fisher’s exact test was used to evaluate which journal presented the highest number of randomized clinical trials. Results: The journal evidence index values were as follows: Plastic and Reconstructive Surgery, 1.70; Journal of Plastic, Reconstructive and Aesthetic Surgery, 0.40; Aesthetic Plastic Surgery, 0.56; and Annals of Plastic Surgery, 0.35. The impact factors of these journals in 2011 were as follows: Plastic and Reconstructive Surgery, 3.382; Journal of Plastic, Reconstructive and Aesthetic Surgery, 1.494; Aesthetic Plastic Surgery, 1.407; and Annals of Plastic Surgery, 1.318. After consensus, the quantity of adequate studies was low and similar between these journals; only the journal Plastic and Reconstructive Surgery showed a higher journal evidence index. Conclusions: The journal Plastic and Reconstructive Surgery exhibited the highest journal evidence index and had the highest impact factor. The number of adequate articles was low in all of the assessed journals. (Plast. Reconstr. Surg. 133: 1502, 2014.)
T
he methodologic quality and relevance of published studies are the main factors that promote their citation and affect the impact factor of a journal. Nevertheless, poor-quality clinical studies are still published in various medical fields.1,2 Some scientists reject evidence-based medicine studies because of the presence of sources of bias that threaten the reliability of the data, such as the choice of a single language for data selection, prejudice, citation bias, database bias, and multiple publication bias.3 Randomized clinical trials provide the best available source of scientific information and the only scientifically reliable method of assessing the efficacy of interventions. Therefore, the design, development, and the journal From the Division of Plastic Surgery, Federal University of São Paulo. Received for publication November 5, 2013; accepted December 12, 2013. Copyright © 2014 by the American Society of Plastic Surgeons DOI: 10.1097/PRS.0000000000000214
1502
of publication of randomized clinical trials must be of high quality.4 Poorly described randomized clinical trials are implicitly considered to have also been poorly performed and to have low quality, and the intervention effect is usually considered to be overestimated.5,6 Evidence-based medicine provides an approach for selecting the best treatment for a given patient and may therefore contribute to reductions in mortality and the costs of health care and improvement in patients’ satisfaction with treatment.7 A consensus meeting was organized to state the purpose of evidence-based medicine, and the adaptation of research within this process was also defined.8 Plastic surgery researchers are resistant to evidence-based medicine; therefore, there is still a long way to go in this field. Disclosure: The authors have no financial interest to declare in relation to the content of this article.
www.PRSJournal.com
Volume 133, Number 6 • Impact Factor vs Journal Evidence Index Through careful and efficient analysis of the available literature, evidence-based medicine must complement the surgeon’s judgment in decisionmaking. However, the knowledge of plastic surgeons is based mostly on the clinical experience of experts; in other words, technical experience and common sense are considered the crucial factors for success in plastic surgery.9 The type and quality of the selected articles, the level of the editorial board members and reviewers, the language used, and an indexed database are important factors that contribute to increase the impact factor of a journal.10 Furthermore, publication in a journal with a high impact factor contributes to establish the reputation of authors, institutions, and, eventually, countries.11 Novel methods of evaluating studies and journals facilitate the development of scientific research. Therefore, the present study was designed to assess the relationship between the impact factor and the evidence level of articles of plastic surgery journals. Also, this study aims to investigate interexaminer concordance in the assessment of the methodologic quality of studies with the highest level of evidence published in plastic surgery journals.
MATERIALS AND METHODS The inclusion criteria were as follows: the four plastic surgery journals with the highest impact factors listed in the Journal Citation Report of Web of Knowledge; journals that publish articles on general reconstructive or cosmetic surgery; and journals listed for at least 5 years in the Journal Citation Report. Nonsystematic review articles were excluded. The following plastic surgery journals were selected among those with the highest impact factors (Table 1): Plastic and Reconstructive Table 1. Top Impact Factor Journals of Plastic Surgery in 2011* Journal Plastic and Reconstructive Surgery Archives of Facial Plastic Surgery Journal of Plastic, Reconstructive, and Aesthetic Surgery Aesthetic Surgery Journal Clinics in Plastic Surgery Aesthetic Plastic Surgery Annals of Plastic Surgery Facial Plastic Surgery Scandinavian Journal of Plastic and Reconstructive Surgery Journal of Plastic Surgery and Hand Surgery *From Journal Citation Report of Web of Knowledge.
Impact Factor 3.382 1.646 1.494 1.469 1.422 1.407 1.318 0.963 0.935 0.017
Surgery; Journal of Plastic, Reconstructive, and Aesthetic Surgery; Aesthetic Plastic Surgery; and Annals of Plastic Surgery. The journals ranked as second and fifth based on impact factor (i.e., Archives of Facial Plastic Surgery and Clinics in Plastic Surgery, respectively, were excluded from the analysis because the former does not cover all areas of plastic surgery and the latter exclusively publishes nonsystematic literature reviews rather than randomized clinical trials). The journal ranked as fourth, the Aesthetic Surgery Journal, was also excluded because it was not listed on the Journal Citation Report for at least 5 years. Following the classification of the journals, the studies were selected by searching the PubMed database, as all of those journals are found in this database. Studies published from January 1 to December 31, 2011, were selected. Filters for systematic reviews, meta-analyses, clinical trials, and randomized clinical trials were applied separately to each journal that had been previously selected using the “advanced search” available at the PubMed Web site. Letters, editorials, and communications were excluded. The journal evidence index was calculated as the number of appropriate randomized clinical trials published by a journal divided by its total number of full-length articles published in a single year, multiplied by 100. This index was correlated with the journal impact factor and the weighted mean of the journal evidence index of the other journals. The Cook et al.12 classification, based on the Cochrane criteria, was used. Articles are classified into seven different levels of evidence, as follows: I, systematic review; II, mega trial (randomized clinical trial with >1000 patients); III, randomized clinical trial with fewer than 1000 patients; IV, cohort study design; V, case-control study design; VI, case series; and VII, case reports. Two examiners with a similar level of expertise and who had independently attended the Cochrane online course on systematic review performed the same stages of study selection and independently classified the articles based on their study design and patient selection. The following criteria were established to assess the randomized clinical trials: (1) description of the randomization techniques, including allocation concealment and stages of group distribution; and (2) performance of treatment comparisons. For systematic reviews, the following criteria were established: (1) to assess whether the clinical question was formulated; (2) to assess whether the description of the strategy used for searching
1503
Plastic and Reconstructive Surgery • June 2014 for clinical studies was present; and (3) whenever possible, to assess whether statistical analysis of the results was performed. A consensus between examiners was performed to assess which studies were considered adequate among those that did not show concordance. Statistical Analysis Statistical analysis was performed using the kappa test to assess interexaminer concordance, and the Fisher’s exact test was used to identify journals with the largest number of adequate studies. The Altman criteria13 were used to classify the reliability coefficient obtained in the kappa test.
Table 2. Number of Adequate Randomized Clinical Trials after Interexaminer Concordance for Each Journal and the Total Number of Published Studies* Journals
Level I, II, and III
Total No. of Studies
JEI
IF (2011)
PRS JPRAS APS AnPS Total Average
11 03 02 01 17 4.25
646 745 354 285 2.030 507.5
1.70 0.40 0.56 0.35 3.01 0.84
3.382 1.494 1.407 1.318
JEI, journal evidence index; IF, impact factor; PRS, Plastic and Reconstructive Surgery; JPRAS, Journal of Plastic, Reconstructive, and Aesthetic Surgery; APS, Aesthetic Plastic Surgery; AnPS, Annals of Plastic Surgery. *The journal evidence index and the average journal evidence index of plastic surgery journals in the studied period and the journal impact factor during the year of 2011 are also displayed.
RESULTS
DISCUSSION
A total of 116 articles, levels I, II and III— systematic reviews, mega trials, and randomized clinical trials—were identified, of which 71 were systematic reviews and 45 were randomized clinical trials, although only 100 were included in the present study. It was verified by the examiners that 16 articles did not fit into the characteristics of randomized clinical trials or systematic reviews. Subsequently, following the examiners’ consensus, an additional four articles were not considered as high–evidence level studies and were excluded. Plastic and Reconstructive Surgery published 64 (55.18 percent) of the selected articles, the Journal of Plastic, Reconstructive, and Aesthetic Surgery published 18 (15.52 percent), and Aesthetic Plastic Surgery and Annals of Plastic Surgery published nine (7.75 percent) each. To analyze the interexaminer concordance, the kappa reliability coefficient was calculated, which indicated moderate to excellent reliability according to the following Altman criteria: poor (κ < 0.20); reasonable (0.21 ≥ κ ≤ 0.40); moderate (0.41 ≥ κ ≤ 0.60); good (0.61 ≥ κ ≤ 0.80); and excellent (0.81 ≥ κ ≤ 1.00). Table 2 shows the absolute and mean journal evidence index and impact factor values of the investigated journals. Table 3 shows that the interexaminer concordance was moderate relative to Plastic and Reconstructive Surgery (p < 0.001) and the Journal of Plastic, Reconstructive, and Aesthetic Surgery (p = 0.058), good relative to Aesthetic Plastic Surgery (p = 0.018), and excellent relative to Annals of Plastic Surgery (p = 0.003). Table 4 shows that following consensus among the examiners, the incidence of adequate studies was similar in most journals, with Aesthetic Plastic Surgery being the journal with the highest percentage of studies considered adequate.
One major concern among editors of scientific journals is the identification and publication of studies with a high level of evidence, as these decisions enable doctors to make decisions based on up-to-date and consistent information and result in the best possible treatment for patients.9 The impact factor guides researchers in the selection of a journal in which to publish their studies, as researchers generally agree that the best articles are published in journals with high impact factors.
1504
Table 3. Interexaminer Concordance Regarding the Number of Level I Studies per Journal Published in 2011* Examiner 1 Examiner 2 Plastic and Reconstructive Surgery† Adequate Inadequate Total Journal of Plastic, Reconstructive, and Aesthetic Surgery‡ Adequate Inadequate Total Aesthetic Plastic Surgery§ Adequate Inadequate Total Annals of Plastic Surgery‖ Adequate Inadequate Total
Adequate
Inadequate
Total
17 7 24
7 33 40
24 40 64
6 3 9
2 7 9
8 10 10
5 0 5
1 3 4
6 3 9
3 0 3
0 6 6
3 6 9
*Statistically significant difference for p < 0.05. †κ = 0.533, p < 0.001 (moderate). ‡κ = 0.444, p = 0.058 (moderate). §κ = 0.769, p = 0.018 (good). ‖κ = 1.000, p = 0.003 (excellent).
Volume 133, Number 6 • Impact Factor vs Journal Evidence Index Table 4. Number of Articles Listed According to PubMed as Being Systematic Reviews and Randomized Clinical Trials That Were Considered Adequate and Inadequate, According to the Examiners, per Year, in Each Journal* Journal PRS JPRAS APS AnPS Total
Adequate
% Adequate
Inadequate
% Inadequate
Total
22 7 5 3 37
36.0 41.2 55.5 33.3 38.5
39 10 4 6 59
63.9 58.8 44.4 66.7 61.5
61 17 9 9 96
PRS, Plastic and Reconstructive Surgery; JPRAS, Journal of Plastic, Reconstructive, and Aesthetic Surgery; APS, Aesthetic Plastic Surgery; AnPS, Annals of Plastic Surgery. *p = 0.6998, Fisher’s exact test; statistically significant difference for p < 0.05.
Since July of 2011, plastic surgery journals began to use a pyramid scale to classify articles according to their level of evidence; it has improved the use of the acquired knowledge by the readers in their practices.14 In turn, the journal evidence index assesses the proportion of adequate randomized clinical trials published by a journal in a single year. This type of assessment is important to illustrate the influence of evidencebased medicine on plastic surgery. Some journals also publish a large number of letters, editorials, or communications, which could influence the results. For this reason, these types of publications were excluded from the calculation of the total number of published articles in the current study. Plastic and Reconstructive Surgery exhibited the highest impact factor and highest journal evidence index and was the only journal with a journal evidence index above the average. These results likely explain the preference of authors for sending their studies to this journal. In addition, the journal evidence index of Aesthetic Plastic Surgery was higher compared with that of the Journal of Plastic, Reconstructive, and Aesthetic Surgery and Annals of Plastic Surgery, although the impact factor of the Journal of Plastic, Reconstructive, and Aesthetic Surgery was higher. There is always a tendency for researchers to submit articles with the highest evidence levels to journals with higher impact factors. However, it is not true that the others journals do not receive these types of articles, as we could demonstrate in this study. The idea of the present study was to evaluate whether there would be a clear relation between the impact factor and the number of high–evidence level studies, which could not be established. Between 2007 and 2011, Aesthetic Plastic Surgery was the journal whose impact factor exhibited the greatest percentage increase (151.7 percent), followed by the Journal of Plastic, Reconstructive, and Aesthetic Surgery (139.8 percent), Plastic and Reconstructive Surgery (63.1 percent), and Annals
of Plastic Surgery (28.1 percent), according to the Journal Citation Reports. Therefore, Aesthetic Plastic Surgery was the journal with the greatest impact factor increase and was ranked second in journal evidence index. A high journal evidence index will be reflected in the impact factor in the ensuing few years. Although Plastic and Reconstructive Surgery is the journal with the largest number of level I evidence articles, it did not exhibit a similar proportionality in impact factor progression. The present study found a low frequency of randomized clinical trials published in the field of plastic surgery (0.84 percent), which agrees with the results of Moraes et al.,14 who found a mere 2.4 percent of randomized clinical trials in a journal of orthopedics and 1.7 percent in another journal from the same field. Sinno et al.15 showed that only 22 percent of the articles studied in the four main plastic surgery journals presented level I or II evidence. Another study16 that has evaluated the aesthetic surgery literature has shown that most studies were level IV (50 percent) and V (23 percent) evidence. Veiga et al.17 investigated the distribution of randomized clinical trials in journals of plastic surgery from 1996 to 2003 and found that 38.2 percent (n = 13) were published in Plastic and Reconstructive Surgery, 29.4 percent (n = 10) were published in the Journal of Plastic, Reconstructive, and Aesthetic Surgery, 5.9 percent (n = 2) were published in Annals of Plastic Surgery, and 26.5 percent (n = 9) were published in other journals. These authors performed the same analysis for the period from 2004 to 2008 and found that 46.4 percent (n = 13) of the randomized clinical trials were published in Plastic and Reconstructive Surgery, 7.1 percent (n = 2) were published in the Journal of Plastic, Reconstructive, and Aesthetic Surgery, 3.6 percent (n = 1) were published in Annals of Plastic Surgery, and 42.9 percent (n = 12) were published in other journals. An increase, albeit small, in the total number of published randomized clinical trials when these periods were compared was noted, according to Veiga et al.17
1505
Plastic and Reconstructive Surgery • June 2014 Standardization of the rules for publication and the journal’s identification of the evidence level of scientific articles increase the demand for higher quality articles with reliable results, thus improving the contributions made by science to society at large. To assess the limitations and power of randomized clinical trials, readers must be provided data regarding the methodologic quality of the study. This requirement, however, remains poorly met in spite of the overall advancement of science. Flaws in reporting make proper identification of randomized clinical trials difficult or even impossible. In addition, these flaws potentially contribute to incorrect decision-making in clinical practice when the results of biased studies are attributed undeserved reliability. For that reason, Consolidated Standards of Reporting Trials, recommendations established in the 1990s, have increasingly been used as standards to improve the quality of randomized clinical trials.18 We agree with Moraes et al.19 that a lack of tradition in scientific research by researchers leads to low methodologic quality, especially when the clinical question is not clear. The use of scales to assess methodologic quality by authors submitting randomized clinical trials should be suggested by all journals to increase the reliability of the results, and this system could also be applied to systematic reviews. The interexaminer concordance for the articles published in the assessed journals varied from moderate to excellent, as measured by the kappa reliability coefficient,13 thus ensuring the reliability of the results. To increase reliability, factors such as allocation concealment, randomization, masking, blinding, relevant clinical question, statistical analysis, and sample size should be improved in plastic surgery articles. If an objective description of these factors is not properly displayed in the article, the study may not be considered as a level I evidence article. It is clearly very difficult to conduct studies of surgery with high levels of evidence, and they are seldom performed because they are time consuming and very expensive.14,17 These factors account for the small number of adequate articles found. In addition, when such studies are performed, doubts are often cast on patient selection and results because of the low quality of the description of these factors in the articles. One additional factor that makes research in plastic surgery difficult is the scarcity of validated instruments for assessment to compare the results of surgical procedures, as the assessment of the results is generally based on the judgment of the
1506
patient and doctor. Thus, the use of standardized instruments should be promoted to advance the field of plastic surgery. The difficulty of reaching an interexaminer consensus in the present study highlights the difficulties met by investigators in the selection of articles for systematic reviews on plastic surgery. It is important for investigators to concern themselves with establishing what journals publish the studies with the best evidence and appropriate descriptions, rather than focusing on the journals that carry the largest numbers of cited articles. The impact factor considers only citations of recent publications, and it decreases with the increase of the total number of articles published in the journal. Bernstein and Gray20 suggests the use of the content factor, which considers the total number of citations of articles of a journal in a given year, regardless of the year in which the cited article was published, to all the articles the journal had published up to and including the year in question. The main advantage of the content factor is that it considers all previous citations. Evidence-based medicine does not deny the value of personal experience but recommends forming this knowledge based on evidence provided by clinical studies, which thereby reduces uncertainty in the field of health.21 In spite of the large number of published studies; few clinical trials exhibit level I, II, and III evidence. In addition, these clinical trials often do not provide the data needed to ensure their reliability or enable their inclusion in systematic reviews.
CONCLUSIONS The journal Plastic and Reconstructive Surgery exhibited the highest journal evidence index and had the highest impact factor. Interexaminer concordance in the analysis of the methodologic quality of articles published in the main journals of plastic surgery varied from moderate to excellent. The number of adequate articles was low in all of the assessed journals. Fabio Xerfan Nahas, M.D., Ph.D., M.B.A. Division of Plastic Surgery Federal University of São Paulo Rua Napoleão de Barros 715, 4° andar CEP 04024-002 São Paulo, Brazil [email protected]
REFERENCES 1. Riera R. Designs of studies published in two Brazilian journals of orthopedics and sports medicine, recently indexed in the ISI Web of Science. Sao Paulo Med J. 2009;127:355–358.
Volume 133, Number 6 • Impact Factor vs Journal Evidence Index 2. Hirji KF. No short-cut in assessing trial quality: A case study. Trials 2009;10:1–10. 3. Aalaei-Andabili SH, Alavian SM. Principles of meta-analysis should be well understood. Obes Surg. 2012;22:1926–1927. 4. Veiga Filho J, Castro AA, Veiga DF, et al. Quality of reports of randomized clinical trials in plastic surgery. Plast Reconstr Surg. 2005;115:320–323. 5. Karri V. Randomised clinical trials in plastic surgery: Survey of output and quality of reporting. J Plast Reconstr Aesthet Surg. 2006;59:787–796. 6. Hochman B, Nahas FX, Oliveira Filho RS, Ferreira LM. Research designs (in Portuguese). Acta Cir Bras. 2005;(Suppl 2):2–9. 7. McCarthy JE, Chatterjee A, McKelvey TG, Jantzen EM, Kerrigan CL. A detailed analysis of level I evidence (randomized controlled trials and meta-analyses) in five plastic surgery journals to date: 1978 to 2009. Plast Reconstr Surg. 2010;126:1774–1778. 8. Eaves FF III, Rohrich RJ, Sykes JM. Taking evidence-based plastic surgery to the next level: Report of the second Summit on Evidence-based Plastic Surgery. Aesthet Surg J. 2013;33:735–743. 9. Eaves F III, Pusic AL. Why evidence-based medicine matters to aesthetic surgery. Aesthet Surg J. 2012;32:117–119. 10. Tedesco AB, Nahas FX, Ferreira LM. The importance of the use of descriptors of the Medical Subject Heading (MeSH) in plastic surgery journals. Plast Reconstr Surg. 2010;126:222e. 11. Petroianu A. Quantitative criteria to assess the value of the publication of scientific articles (in Portuguese). Rev Assoc Med Bras. 2003;49:173–176.
12. Cook DJ, Guyatt GH, Laupacis A, Sackett DL, Goldberg RJ. Clinical recommendations using levels of evidence for antithrombotic agents. Chest 1995;108(Suppl):227S–230S. 13. Altman DG. Practical Statistics for Medical Research. 1st ed. London: Chapman & Hall; 1991:69–81. 14. Moraes VY, Moreira CD, Tamaoki MJC, Faloppa F, Belloti JC. Randomized clinical trials in orthopedics and traumatology: Systematic assessment of the national evidence (in Portuguese). Rev Bras Ortop. 2010;45:601–605. 15. Sinno H, Neel OF, Lutfy J, Bartlett G, Gilardino M. Level of evidence in plastic surgery research. Plast Reconstr Surg. 2011;127:974–980. 16. Chuback JE, Yarascavitch BA, Eaves F III, Thoma A, Bhandari M. Evidence in the aesthetic surgical literature over the past decade: How far have we come? Plast Reconstr Surg. 2012;129:126e–134e. 17. Veiga DF, Veiga-Filho J, Pellizzon RF, Juliano Y, Ferreira LM. Evolution of reports of randomised clinical trials in plastic surgery. J Plast Reconstr Aesthet Surg. 2011;64:703–709. 18. Dainesi SM, Aligieri P. How might the CONSORT statements ensure quality in clinical study reporting? (in Portuguese). Rev Assoc Med Bras. 2005;51:66. 19. Moraes VY, Belloti JC, Moraes FY, et al. Hierarchy of evidence relating to hand surgery in Brazilian orthopedic journals. Sao Paulo Med J. 2011;129:94–98. 20. Bernstein J, Gray CF. Content factor: A measure of a journal’s contribution to knowledge. PLoS One 2012;7: e41554. 21. El Dib RP. How to practice evidence-based medicine. J Vasc Bras. 2007;6:1–4.
1507
