531700 research-article2014
WJNXXX10.1177/0193945914531700Western Journal of Nursing ResearchEditorial
Editorial
Moving Beyond Counting Publications to Assess Impact
Western Journal of Nursing Research 2015, Vol. 37(3) 283–287 © The Author(s) 2014 Reprints and permissions: sagepub.com/journalsPermissions.nav DOI: 10.1177/0193945914531700 wjn.sagepub.com
Researchers oftentimes wonder about the impact of their work on their field. Administrators seek some grade of how their scientists measure up relative to their peers. Hiring and tenure committees also seek a metric by which to evaluate a candidate. The field of bibliometrics aims to address these needs by providing indices that attempt to quantify and standardize the publication output of researchers, sometimes called their impact. There are a multitude of indices that have been created in just the last few years. In this column, we briefly present the main metrics in the field, describe their strengths and weaknesses, and show where they can be calculated online. The quantification of researcher impact was jumpstarted by the introduction of the h-index (Hirsch, 2005). The h-index for an individual is the highest number of his or her articles that have each been cited the same number of times. For example, if someone’s body of work has been cited 1, 3, 4, 12, and 28 times, her h-index is three because she has three articles that have been cited at least 3 times each. One issue that is immediately apparent is that the h-index does not account for the length of the researcher’s career. Surely, someone who produced the above five articles over a period of a year is a more productive researcher than someone who took 10 years to do the same. Hirsch (2005) proposed another measure, called the m-quotient, which is the h-index divided by the number of years since the researcher’s first publication. Continuing the above example, the first researcher’s m-quotient would still be 3 while the second researcher’s m-quotient drops to 0.3, showing a clear difference in productivity. After Hirsch’s article was published, issues with the h-index and m-quotient were noted by other researchers (Alonso, Cabrerizo, Herrera-Viedma, & Herrera, 2009; Bornmann & Hans-Dieter, 2007; Costas & Bordons, 2007; Jin, 2006; Rousseau, 2006; Saleem, 2011; Wendl, 2007). For example, the actual number of citations for the mostly highly cited articles is irrelevant to the index: Someone whose work has been cited 1, 2, 3, 3, and 3 times will have an h-index of three, just like in the example above, even though in this case there are fewer total citations. Also, while the m-quotient accounts for
Downloaded from wjn.sagepub.com at PURDUE UNIV LIBRARY TSS on May 22, 2015
284
Western Journal of Nursing Research 37(3)
Table 1. Common Scientific Impact Indices. Index
Formulation
Access
Citation
h-index
The largest number in which h articles are cited h times each
Hirsch, 2005
g-index
The highest number of papers that, when combined, have g2 citations The square root of h×g
Google Scholar WACSPhi (web application to calculate the single publication h-index [and further metrics] based on Google Scholar; Thor & Bornmann, n.d.) Publish or Perish Scholar H-Index Calculator Scopus Web of Science Publish or Perish
Egghe, 2006
Publish or Perish
Alonso, Cabrerizo, HerreraViedma, and Herrera, 2010 Zhang, 2009
hg-index
e-index i10-index A-index AR-index
m-quotient
R-index
Square root of the excess citations in the top h citations The number of articles with at least 10 citations The average number of citations in the top h studies The square root of the sum of the average number of articles per year in the top h articles h divided by number of years since a researcher’s first publication The square root of the number of citations in the top h studies
Google Scholar
Google Scholar, n.d. Jin, 2006
Publish or Perish (variation of the index)
Jin, 2007a
WACSPhi
Hirsch, 2005
Jin, 2007b
Downloaded from wjn.sagepub.com at PURDUE UNIV LIBRARY TSS on May 22, 2015
285
Editorial
the confounding effect of a researcher’s career span, it does not correct for how long individual papers have been able to accrue citations. Many researchers took on the task of creating an improved index of researcher impact. While there are too many indices to review here, three variations have become more prominent than others (Table 1 shows several more variations). Egghe (2006) created the g-index, calculated as the highest number of papers that together has at least g2 citations, to account for the number of times the top publications have been cited. In the first example, g = 5 because all five articles in total have at least 52 citations (48 > 25). In the second case, g = 3 because the top three articles have at least 32 citations (g ≠ 4 or greater because the top four articles do not have over 42 citations total). Because the h and g indices measure different aspects of researcher impact, Alonso, Cabrerizo, Herrera-Viedma, and Herrera (2010) created the hg-index, which takes the square root of the product of the two indices (in our examples, the hg indices are 3.87 and 3). Further focusing on the quantity of citations in the top articles, the AR-index takes the square root of the sum of the citations of a paper divided by the number of years since its publication, for the top h citations (Jin, 2007b). This index uses the h-index but also accounts for how long a publication has been citable to account for the time factor. The power of an index is only one aspect of its usefulness; an index has to be easily accessible to quickly compare different researchers. There are several different free solutions for calculating researcher impact indices. The most prominent are online citation databases, such as Google Scholar, Scopus, and the Web of Science. Publish or Perish is a computer application that computes a variety of indices based on Google Scholar results and calculates the most indices among the different tools (Harzing, 2007). Thor and Bornmann (n.d.) created a web application accessible in Internet browsers that calculates variations of the h-index and the m-quotient. Overall, the original h-index is by far the most common metric as it can be calculated by all of these tools. While perhaps the most comprehensive measure, the hg-index has to be manually calculated from its constituent parts either by hand or with Publish or Perish. Besides calculating an overall career index, bibliometric tools allow investigators to find component pieces of the impact indices, which are useful for analyzing the strength of individual publications. For example, investigators should evaluate which of their papers, considering the age of the articles, have had more impact than other papers. Many investigators are surprised to see papers that they did not consider especially important in their overall body of work to actually be their most cited articles. If the purpose of research is to generate new knowledge, not to obtain grants or
Downloaded from wjn.sagepub.com at PURDUE UNIV LIBRARY TSS on May 22, 2015
286
Western Journal of Nursing Research 37(3)
accumulate publications, future research ideas could be partially shaped by seeing the impact of past work. While certainly useful, impact indices cannot measure all criteria that define a good researcher. Attempts to compare researchers should consider lengths of research careers because older papers have more opportunities to be cited than newer papers. Evaluative uses of impact assessment should also consider whether the researcher remains an active investigator because inert researchers may have significant impact scores based on studies published decades earlier. Impact indices may be less useful for evaluating junior faculty than to evaluate more established investigators. Also, comparisons across different areas of science may have limited utility. In some situations, it might be useful to separate impact as a research team member from impact of publications derived from investigator-led work. Despite the limitations of these investigator impact metrics, they are clearly far more valuable than simply counting the number of publications. For evaluating scientists, impact metrics based on specific articles are much more useful than journal impact factor because journal impact factor does not address the impact of individual authors and their work. Recently, consideration of the future of bibliometry stimulated thinking about alternative metrics for social media, called altmetrics (Howard, 2013). Altmetrics attempt to capture the online impact of research through measurements such as the number of tweets, blog posts, and downloads of articles. Online impact could be more suitable for junior faculty scholars for whom bibliometrics are less appropriate, because index values take years to accumulate. The rapidly developing research in altmetrics may provide valuable information about impact as novel measures become accepted and standardized. While the h-index is the most common indicator in computer applications and is also easy to calculate manually, there are several alternatives that attempt to provide more thorough metrics. Researchers and evaluation committees should be aware of the scope of each index and be cognizant of their flaws when using them to make important career decisions. Vicki S. Conn, PhD, RN, FAAN Editor Keith C. Chan, PhD Research Specialist University of Missouri
Downloaded from wjn.sagepub.com at PURDUE UNIV LIBRARY TSS on May 22, 2015
287
Editorial References
Alonso, S., Cabrerizo, F., Herrera-Viedma, E., & Herrera, F. (2010). hg-index: A new index to characterize the scientific output of researchers based on the h- and g-indices. Scientometrics, 82, 391-400. Alonso, S., Cabrerizo, F. J., Herrera-Viedma, E., & Herrera, F. (2009). h-index: A review focused in its variants, computation and standardization for different scientific fields. Journal of Informetrics, 3, 273-289. Bornmann, L., & Hans-Dieter, D. (2007). What do we know about the h index. Journal of the American Society for Information Science and Technology, 58, 1381-1385. Costas, R., & Bordons, M. (2007). The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level. Journal of Informetrics, 1, 193-203. Egghe, L. (2006). Theory and practise of the g-index. Scientometrics, 69, 131-152. Google Scholar. (n.d.). Google Scholar Metrics help. Retrieved from http://scholar. google.com/intl/en/scholar/metrics.html#metrics Harzing, A. W. (2007). Publish or Perish. Retrieved from http://www.harzing.com/ pop.htm Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102, 16569-16572. doi:10.1073/pnas.0507655102 Howard, J. (2013, June 3). Rise of “altmetrics” revives questions about how to measure impact of research. The Chronicle of Higher Education. Retrieved from http://chronicle.com/article/Rise-of-Altmetrics-Revives/139557/ Jin, B. (2006). h-index: An evaluation indicator proposed by scientist. Science Focus, 1(1), 8-9. Jin, B. (2007a). The AR-index: Complementing the h-index. ISSI Newsletter, 3(1), 6. Jin, B. (2007b). The R- and AR-indices: Complementing the h-index. Chinese Science Bulletin, 52, 855-863. Rousseau, R. (2006). New developments related to the Hirsch index. Science Focus, 1(4), 23-25. Saleem, T. (2011). The Hirsch index—A play on numbers or a true appraisal of academic output? International Archives of Medicine, 4, 25. doi:10.1186/17557682-4-25 Thor, A., & Bornmann, L. (n. d.) Web application to calculate the single publication h index (and further metrics) based on Google Scholar. Retrieved from http://labs. dbs.uni-leipzig.de/gsh/ Wendl, M. C. (2007). H-index: However ranked, citations need context. Nature, 449(7161), 403. Zhang, C. T. (2009). The e-index, complementing the h-index for excess citations. PloS One, 4(5), e5429. doi:10.1371/journal.pone.0005429
Downloaded from wjn.sagepub.com at PURDUE UNIV LIBRARY TSS on May 22, 2015
WJNXXX10.1177/0193945914531700Western Journal of Nursing ResearchEditorial
Editorial
Moving Beyond Counting Publications to Assess Impact
Western Journal of Nursing Research 2015, Vol. 37(3) 283–287 © The Author(s) 2014 Reprints and permissions: sagepub.com/journalsPermissions.nav DOI: 10.1177/0193945914531700 wjn.sagepub.com
Researchers oftentimes wonder about the impact of their work on their field. Administrators seek some grade of how their scientists measure up relative to their peers. Hiring and tenure committees also seek a metric by which to evaluate a candidate. The field of bibliometrics aims to address these needs by providing indices that attempt to quantify and standardize the publication output of researchers, sometimes called their impact. There are a multitude of indices that have been created in just the last few years. In this column, we briefly present the main metrics in the field, describe their strengths and weaknesses, and show where they can be calculated online. The quantification of researcher impact was jumpstarted by the introduction of the h-index (Hirsch, 2005). The h-index for an individual is the highest number of his or her articles that have each been cited the same number of times. For example, if someone’s body of work has been cited 1, 3, 4, 12, and 28 times, her h-index is three because she has three articles that have been cited at least 3 times each. One issue that is immediately apparent is that the h-index does not account for the length of the researcher’s career. Surely, someone who produced the above five articles over a period of a year is a more productive researcher than someone who took 10 years to do the same. Hirsch (2005) proposed another measure, called the m-quotient, which is the h-index divided by the number of years since the researcher’s first publication. Continuing the above example, the first researcher’s m-quotient would still be 3 while the second researcher’s m-quotient drops to 0.3, showing a clear difference in productivity. After Hirsch’s article was published, issues with the h-index and m-quotient were noted by other researchers (Alonso, Cabrerizo, Herrera-Viedma, & Herrera, 2009; Bornmann & Hans-Dieter, 2007; Costas & Bordons, 2007; Jin, 2006; Rousseau, 2006; Saleem, 2011; Wendl, 2007). For example, the actual number of citations for the mostly highly cited articles is irrelevant to the index: Someone whose work has been cited 1, 2, 3, 3, and 3 times will have an h-index of three, just like in the example above, even though in this case there are fewer total citations. Also, while the m-quotient accounts for
Downloaded from wjn.sagepub.com at PURDUE UNIV LIBRARY TSS on May 22, 2015
284
Western Journal of Nursing Research 37(3)
Table 1. Common Scientific Impact Indices. Index
Formulation
Access
Citation
h-index
The largest number in which h articles are cited h times each
Hirsch, 2005
g-index
The highest number of papers that, when combined, have g2 citations The square root of h×g
Google Scholar WACSPhi (web application to calculate the single publication h-index [and further metrics] based on Google Scholar; Thor & Bornmann, n.d.) Publish or Perish Scholar H-Index Calculator Scopus Web of Science Publish or Perish
Egghe, 2006
Publish or Perish
Alonso, Cabrerizo, HerreraViedma, and Herrera, 2010 Zhang, 2009
hg-index
e-index i10-index A-index AR-index
m-quotient
R-index
Square root of the excess citations in the top h citations The number of articles with at least 10 citations The average number of citations in the top h studies The square root of the sum of the average number of articles per year in the top h articles h divided by number of years since a researcher’s first publication The square root of the number of citations in the top h studies
Google Scholar
Google Scholar, n.d. Jin, 2006
Publish or Perish (variation of the index)
Jin, 2007a
WACSPhi
Hirsch, 2005
Jin, 2007b
Downloaded from wjn.sagepub.com at PURDUE UNIV LIBRARY TSS on May 22, 2015
285
Editorial
the confounding effect of a researcher’s career span, it does not correct for how long individual papers have been able to accrue citations. Many researchers took on the task of creating an improved index of researcher impact. While there are too many indices to review here, three variations have become more prominent than others (Table 1 shows several more variations). Egghe (2006) created the g-index, calculated as the highest number of papers that together has at least g2 citations, to account for the number of times the top publications have been cited. In the first example, g = 5 because all five articles in total have at least 52 citations (48 > 25). In the second case, g = 3 because the top three articles have at least 32 citations (g ≠ 4 or greater because the top four articles do not have over 42 citations total). Because the h and g indices measure different aspects of researcher impact, Alonso, Cabrerizo, Herrera-Viedma, and Herrera (2010) created the hg-index, which takes the square root of the product of the two indices (in our examples, the hg indices are 3.87 and 3). Further focusing on the quantity of citations in the top articles, the AR-index takes the square root of the sum of the citations of a paper divided by the number of years since its publication, for the top h citations (Jin, 2007b). This index uses the h-index but also accounts for how long a publication has been citable to account for the time factor. The power of an index is only one aspect of its usefulness; an index has to be easily accessible to quickly compare different researchers. There are several different free solutions for calculating researcher impact indices. The most prominent are online citation databases, such as Google Scholar, Scopus, and the Web of Science. Publish or Perish is a computer application that computes a variety of indices based on Google Scholar results and calculates the most indices among the different tools (Harzing, 2007). Thor and Bornmann (n.d.) created a web application accessible in Internet browsers that calculates variations of the h-index and the m-quotient. Overall, the original h-index is by far the most common metric as it can be calculated by all of these tools. While perhaps the most comprehensive measure, the hg-index has to be manually calculated from its constituent parts either by hand or with Publish or Perish. Besides calculating an overall career index, bibliometric tools allow investigators to find component pieces of the impact indices, which are useful for analyzing the strength of individual publications. For example, investigators should evaluate which of their papers, considering the age of the articles, have had more impact than other papers. Many investigators are surprised to see papers that they did not consider especially important in their overall body of work to actually be their most cited articles. If the purpose of research is to generate new knowledge, not to obtain grants or
Downloaded from wjn.sagepub.com at PURDUE UNIV LIBRARY TSS on May 22, 2015
286
Western Journal of Nursing Research 37(3)
accumulate publications, future research ideas could be partially shaped by seeing the impact of past work. While certainly useful, impact indices cannot measure all criteria that define a good researcher. Attempts to compare researchers should consider lengths of research careers because older papers have more opportunities to be cited than newer papers. Evaluative uses of impact assessment should also consider whether the researcher remains an active investigator because inert researchers may have significant impact scores based on studies published decades earlier. Impact indices may be less useful for evaluating junior faculty than to evaluate more established investigators. Also, comparisons across different areas of science may have limited utility. In some situations, it might be useful to separate impact as a research team member from impact of publications derived from investigator-led work. Despite the limitations of these investigator impact metrics, they are clearly far more valuable than simply counting the number of publications. For evaluating scientists, impact metrics based on specific articles are much more useful than journal impact factor because journal impact factor does not address the impact of individual authors and their work. Recently, consideration of the future of bibliometry stimulated thinking about alternative metrics for social media, called altmetrics (Howard, 2013). Altmetrics attempt to capture the online impact of research through measurements such as the number of tweets, blog posts, and downloads of articles. Online impact could be more suitable for junior faculty scholars for whom bibliometrics are less appropriate, because index values take years to accumulate. The rapidly developing research in altmetrics may provide valuable information about impact as novel measures become accepted and standardized. While the h-index is the most common indicator in computer applications and is also easy to calculate manually, there are several alternatives that attempt to provide more thorough metrics. Researchers and evaluation committees should be aware of the scope of each index and be cognizant of their flaws when using them to make important career decisions. Vicki S. Conn, PhD, RN, FAAN Editor Keith C. Chan, PhD Research Specialist University of Missouri
Downloaded from wjn.sagepub.com at PURDUE UNIV LIBRARY TSS on May 22, 2015
287
Editorial References
Alonso, S., Cabrerizo, F., Herrera-Viedma, E., & Herrera, F. (2010). hg-index: A new index to characterize the scientific output of researchers based on the h- and g-indices. Scientometrics, 82, 391-400. Alonso, S., Cabrerizo, F. J., Herrera-Viedma, E., & Herrera, F. (2009). h-index: A review focused in its variants, computation and standardization for different scientific fields. Journal of Informetrics, 3, 273-289. Bornmann, L., & Hans-Dieter, D. (2007). What do we know about the h index. Journal of the American Society for Information Science and Technology, 58, 1381-1385. Costas, R., & Bordons, M. (2007). The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level. Journal of Informetrics, 1, 193-203. Egghe, L. (2006). Theory and practise of the g-index. Scientometrics, 69, 131-152. Google Scholar. (n.d.). Google Scholar Metrics help. Retrieved from http://scholar. google.com/intl/en/scholar/metrics.html#metrics Harzing, A. W. (2007). Publish or Perish. Retrieved from http://www.harzing.com/ pop.htm Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102, 16569-16572. doi:10.1073/pnas.0507655102 Howard, J. (2013, June 3). Rise of “altmetrics” revives questions about how to measure impact of research. The Chronicle of Higher Education. Retrieved from http://chronicle.com/article/Rise-of-Altmetrics-Revives/139557/ Jin, B. (2006). h-index: An evaluation indicator proposed by scientist. Science Focus, 1(1), 8-9. Jin, B. (2007a). The AR-index: Complementing the h-index. ISSI Newsletter, 3(1), 6. Jin, B. (2007b). The R- and AR-indices: Complementing the h-index. Chinese Science Bulletin, 52, 855-863. Rousseau, R. (2006). New developments related to the Hirsch index. Science Focus, 1(4), 23-25. Saleem, T. (2011). The Hirsch index—A play on numbers or a true appraisal of academic output? International Archives of Medicine, 4, 25. doi:10.1186/17557682-4-25 Thor, A., & Bornmann, L. (n. d.) Web application to calculate the single publication h index (and further metrics) based on Google Scholar. Retrieved from http://labs. dbs.uni-leipzig.de/gsh/ Wendl, M. C. (2007). H-index: However ranked, citations need context. Nature, 449(7161), 403. Zhang, C. T. (2009). The e-index, complementing the h-index for excess citations. PloS One, 4(5), e5429. doi:10.1371/journal.pone.0005429
Downloaded from wjn.sagepub.com at PURDUE UNIV LIBRARY TSS on May 22, 2015
