HHS Public Access Author manuscript Author Manuscript
Acad Med. Author manuscript; available in PMC 2017 April 01. Published in final edited form as: Acad Med. 2016 April ; 91(4): 556–562. doi:10.1097/ACM.0000000000001025.
Towards Independence: Resubmission Rate of Unfunded National Heart, Lung, and Blood Institute R01 Research Grant Applications among Early Stage Investigators
Author Manuscript
Josephine E.A. Boyington, PhD, MPH, CNS, Health scientist administrator/program director, Clinical Applications and Prevention Branch, Division of Cardiovascular Sciences, National Heart, Lung, and Blood Institute, National Institutes of Health, Bethesda, Maryland Melissa D. Antman, PhD, Health scientist administrator, Office of Extramural Policy and Training, Division of Extramural Research Activities, National Heart, Lung, and Blood Institute, National Institutes of Health, Bethesda, Maryland Katherine C. Patel, MSPH, and Mathematical statistician, Division of Statistical Analysis and Reporting, Office of Planning, Analysis, and Communication, Office of Extramural Research, Office of the Director, National Institutes of Health, Bethesda, Maryland
Author Manuscript
Michael S. Lauer, MD Director, Division of Cardiovascular Sciences, National Heart, Lung, and Blood Institute, National Institutes of Health, Bethesda, Maryland
Abstract Purpose—The current, budget-driven low rate of the National Institutes of Health (NIH) funding for biomedical research has raised concerns about the ability of new investigators to become independent scientists and their willingness to persist in efforts to secure funding. The authors sought to determine resubmission rates for unfunded National Heart, Lung, and Blood Institute (NHLBI) early stage investigator (ESI) independent research grant (RO1) applications and to identify predictors of resubmission.
Author Manuscript
Method—The authors used a retrospective cohort study design and retrieved applications submitted in fiscal years 2010–2012 from NIH electronic research administrative sources. They defined ESI applicants as those who have received no prior R01 (or equivalent) funding and were
Correspondence should be addressed to Josephine E. A. Boyington, 6701 Rockledge Drive, Suite 10224, Rockledge II, Bethesda, MD 20892; telephone: (301) 435-0446; [email protected]. Other disclosures: None reported. Ethical approval: This work was conducted as part of the authors’ National Institutes of Health employment activities, and all records were de-identified. Disclaimers: The information in this report represents the work of the authors alone, and does not necessarily indicate the views of the National Institutes of Health or the federal government of the United States. Previous presentations: This work was presented internally at the National Institutes of Health Portfolio Analysis Research Symposium and at National Heart, Lung, and Blood Institute’s Staff Meetings.
Boyington et al.
Page 2
Author Manuscript
within 10 years of completion of their terminal research degree or medical residency training. ESI applications at the NHLBI were eligible for special funding consideration if they scored above, but within ten points of, the payline. The primary outcome was application resubmission after failing to secure funding with the first R01 submission. Results—Over half of the unfunded applications were resubmitted. Some of these were discussed and “percentiled,” and among these applications, the only significant predictor of resubmission was the percentile score. Over half (59%) of the ESI R01grants funded by NHLBI in FY-2010–2012 had percentile scores above, but within ten points of the NHLBI payline, and benefitted from the special funding considerations. Conclusions—The only independent predictor of resubmission of NHLBI ESI R01 grant applications was percentile score; applicant demographics and institutional factors were not predictive of resubmission.
Author Manuscript Author Manuscript
Sally Rockey, the immediate past Deputy Director for Extramural Research at the National Institutes of Health (NIH), began the April 29, 2014, post of her blog “Rock Talk” writing, “The strength of the biomedical research enterprise depends on new researchers becoming independent NIH-funded researchers”1; however, for many new investigators, receipt of an NIH independent research (R01) award presents a significant challenge.2,3 Richard Harris, in his recently posted National Public Radio story “When Scientists Give Up,”4 described the pursuits of two early career scientists—both of whom unsuccessfully sought independent research grant awards and subsequently abandoned their research careers. National Heart, Lung, and Blood Institute (NHLBI) staff members have heard similar stories from other early career investigators. One possible manifestation of these junior researchers’ frustration might be their decision not to resubmit an application after failing to secure funding with the first peer-reviewed R01 grant application. We decided, therefore, to measure and ascertain possible correlates for the resubmission of unfunded R01 grants submitted to the NHLBI by early stage investigators (ESIs) during fiscal year (FY) 2010, 2011, and 2012.
Terms, Considerations, and R01 Review Processes
Author Manuscript
We focused on the R01 grant mechanism because it is the hallmark of research independence and because NIH affords funding benefits to ESIs through this mechanism.2,5–7 NIH defines an ESI as any applicant who is within ten years of completion of his/her terminal research degree or medical residency and has never received an R01 grant (or equivalent).5,8 Policies related to early career scientists include the clustering and review of ESIs’ applications apart from those from established investigators,5,8 the provision of opportunities to resubmit unfunded grants earlier and thereby reduce the time to resubmission by four months,9 and fiscal policies that ensure that new investigators are funded at the same rate as established investigators.10 Within the confines of these NIH policies, individual NIH institutes freely develop and implement strategies that ensure a scientifically-diverse and workforce-balanced portfolio. During FY 2010–2012, all unfunded NHLBI ESI R01 applications were eligible for special funding considerations if their percentile scores (see below) were within 10 points of the
Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Page 3
Author Manuscript
payline (i.e., the cutoff point set by NHLBI for funding grant applications, after balancing projected grant numbers, total grant budgets, and funds available).11,12 All applications submitted to NIH are reviewed, but not all applications are discussed at peer-review meetings. Discussed applications receive criterion scores for five core variables, an overall impact score, and a percentile score. Applications that are not discussed receive criterion scores but no overall impact score and no percentile score. It is generally assumed that these applications are above the 50th percentile of the applications submitted for review.
Author Manuscript
The percentile score13 is a measure of an application’s relative rank compared to other applications reviewed by a particular study section (i.e., a panel of scientific experts [i.e., “reviewers”], convened by the NIH Center for Scientific Review or an NIH institute, to provide initial scientific review).14 It reflects reviewers’ global assessment of the application’s scientific and technical merit. The score ranges from 1st through 99th percentile, and favorable scores are below the 50th percentile.13
Author Manuscript
The overall impact score reflects reviewers’ judgment of a project’s ability to influence its field, and it is determined by reviewers’ ratings of the overall merit of an application, particularly after taking into consideration the five core peer review criteria—(1) significance, (2) approach, (3) investigator, (4) innovation, and (5) environment—in addition to other application relevant factors such as human subjects protections. In their consideration of significance, reviewers reflect on the extent to which a research “project address[es] an important problem or a critical barrier to progress in the field,” 15,16 and in their consideration of approach, they assess how well a research project’s “strategy, methodology, and analyses [are] well-reasoned and appropriate to accomplish the stated objectives of the project.”15,16 Reviewers considering the investigator criterion ask the following key questions: “Are the PD/PIs [program director/principal investigators], collaborators, and other researchers well suited to the project?” and “If Early Stage Investigators or New Investigators, or in the early stages of independent careers, do they have appropriate experience and training?”15 The consideration of innovation focuses on “how much a project can shift the current research or clinical paradigms by utilizing novel theoretical concepts, approaches, or methodologies, instrumentation, or interventions” and, finally, the review of environment centers on whether “the scientific environment in which the work will be done contribute[s] to the probability of success”; whether “the institutional support, equipment and other physical resources available to the investigators [are] adequate for the project proposed”; and whether the project would “benefit from unique features of the scientific environment, subject populations, or collaborative arrangements?”15,16
Author Manuscript
The overall impact score is not an average of the five core criteria scores; rather it is the average of the individual ratings of overall merit given by all reviewers after assessment of the five scored criteria listed above, plus, as mentioned, additional concerns such as the protection of human subjects, the welfare and care of vertebrate animals, and the consideration of biohazards.17,18 The relationship between overall impact score and percentile score is that the percentile score normalizes the application’s overall impact score across study sections and thereby attenuates unintended variations in scoring.19
Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Page 4
Author Manuscript
Method In our effort to examine resubmissions (our primary end point), we extracted and deidentified all unfunded, competing, new (A0) R01 applications that had undergone peer review during FY 2010 – 2012. We extracted the applications from the NIH electronic research administration “Information for Management, Planning, and Coordination II (IMPAC II)” database. As potential predictors of resubmission, we considered grant-based factors (e.g., year of original submission and peer-review scores), applicant-based factors (e.g., demographics, training, prior funding), and institution-based factors (e.g., ranking in receipt of NIH funding, and type of institution [research, medical, other]).
Author Manuscript
We determined variable frequencies with descriptive analyses, and we used Random Forest methodology,20 a robust machine-learning-based predictors’ ensemble approach, to identify the most important predictor variables for inclusion in subsequent logistic regression models. We first analyzed the overall pool of applications and subsequently the subset of applications that were discussed and percentiled. We performed all analyses at the NIH, using R3.1.0 (packages Hmisc, rms, ggplot2, and randomForestSRC).
Results We extracted, for applications reviewed in FY 2010–2012, a total of 34,240 unfunded A0 NIH grant applications. Of the 4,587 NHLBI applications, 833 (18.2%) were ESI applications. (Comparatively, of the 29,653 remaining NIH grants, 5,268 (17.8%) were ESI applications.) We focused our analyses on the 821 NHLBI ESIs applications that were not formally withdrawn. Table 1 summarizes grant-, applicant-, and institution-based characteristics.
Author Manuscript
Resubmitted applications The overall resubmission rate was 51.4% (422 of 821). Of the pool of 821 applications, 382 (46.5%) were discussed and given a percentile score and 294 (35.8%) scored less than 50 (Table 1). Of the applications with percentile score less than 50, 82.3% (242 of 294) were resubmitted, whereas among the 527 applications with a percentile score equal to or greater than 50, only 180 (34.2%) were resubmitted (P < .001). Using the random forest machinelearning approach on the entire pool of applications, we observed that the most important correlate, by far, of resubmission was a percentile score less than 50. The investigator, approach, significance, innovation, and environment criterion scores were less correlated with resubmission (Figure 1). Neither applicant- nor institutional-based variables emerged as important correlates.
Author Manuscript
In the corresponding logistic regression model, the only independent predictors were percentile score of less than 50 (P < .001) and the investigator criterion score (P = .01). We noted that 560 of 821 (68.2%) of the ESI applications were from male PIs and that the sample of applications from groups underrepresented in the biomedical sciences was small, thereby limiting our ability to conduct subgroup analyses.
Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Page 5
Discussed and percentiled applications
Author Manuscript
For the 382 applications that were both discussed and received a percentile ranking, we examined the association between, overall impact score, raw percentile score, percentile distance from payline, and resubmission status. Overall impact score did not emerge as a key predictor variable; however, we found that the raw percentile score and the percentile distance from the payline were essentially equivalent predictors of resubmission. We calculated “percentile distance” as a derived variable, by measuring the distance of an application’s percentile score from the operative NHLBI payline at the time of funding consideration. The payline at NHLBI generally changes over the course of the fiscal year, being more conservative (lower) at the beginning of the year, and more generous (higher) towards the end. With the calculation of percentile distance, we were able to create a variable with seemingly greater face value than raw percentile score and to also account for the impact of payline changes on resubmission.
Author Manuscript
As such, we tested percentile distance along with other candidate predictors in a random forest model and found that it was, by far, the strongest correlate of resubmission, followed by the significance and approach criterion scores (Figure 2). In the logistic regression model including percentile distance, the five criterion scores, requested budget, and gender, the only independent predictor was percentile distance (P < .001). We further examined the relationship between percentile distance and the probability of resubmission and found that as percentile distance increased, the probability of resubmission decreased (Figure 3). Unfunded applications eligible for NHLBI special funding consideration FY 2010–2012
Author Manuscript
During FY 2010–2012, NHLBI considered 342 total (new and resubmitted) ESI R01 applications for funding. Two hundred and two (59%) of these had percentile scores above, but within ten points of the NHLBI R01 payline, and were, therefore, awarded funding based on NHLBI’s special funding consideration for ESIs.
Discussion In the current budget-driven fiscal climate, early career investigators may be frustrated by failure to secure an R01 grant and may express this frustration by not resubmitting unfunded applications. We found that slightly more than half of NHLBI ESIs (51.4%) resubmitted their unfunded applications, and that in the regression model of the overall pool of applications, a percentile score of less than 50 and the investigator criterion score were the two significant predictors of resubmission.
Author Manuscript
As noted above, percentile, reflects the relative rank of an application compared to others reviewed by a particular study section, and the investigator criterion score reflects reviewers’ judgment of a PI and his/her team’s competence to execute a proposed project. A higher investigator criterion score, therefore, indicates the need for the PI to gain more practical and relevant research skills, the need for a stronger and more complementary research team, or both. Compared to other factors that could affect the investigator criterion score, skill set and an appropriate research team are more easily changeable than, for example, demographics (race/ethnicity) or degree type (MD vs. PhD). A recent internal NIH
Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Page 6
Author Manuscript
analysis using overall NIH data determined that degree type has no measurable impact on R01 success rates.21 In contrast, a recent NIH-commissioned external study that sought to understand the relationship between race/ethnicity and grant receipt, documented differential probabilities of R01 grant receipts by race/ethnicity.22 Due to insufficient data, the study did not report institute-specific, race-based results, but did show that overall, Hispanic and Black applicants were less likely than others to resubmit a revised application.22
Author Manuscript
Our study differs from previous work in that we focused specifically on early stage investigators and on a critical antecedent behavior for grant receipt—that is resubmission of unfunded R01 grants. Although, we focused on the vulnerable group of early career investigators, we were underpowered to conduct subgroup analyses (e.g., race/ethnicity) because of our small sample size. We found correlations between application-based characteristics and resubmission of an unfunded R01 application but no independent association of applicant- or institution-based characteristics with resubmission. We did observe twice as many male as female applicants in our cohort of 821. Ley and colleagues23 and Dunbar and colleagues24 have reported similar observations, and have commented that observed differences are evidence of the greater exodus of females at the independent career stage and not a result of gender-based differences in success rates. Apart from the proportional differences in the number of applicants by gender, we found that gender was not a significant predictor of resubmission.
Author Manuscript
The finding that discussed (thus percentiled) applications were resubmitted at a higher rate is reassuring since it indicates that an objectively-derived application attribute (i.e., percentile score) seemingly drives resubmission behavior. As noted earlier, we determined percentile distance to be an essentially equivalent surrogate for raw percentile score, and to be the only significant predictor of resubmission for discussed applications. Consequently, suggesting that the primary driver in applicant’s decision to resubmit should be the application’s overall percentile score and less so the individual criterion or overall impact scores.
Author Manuscript
NHLBI funded 342 ESI R01 applications in FY 2010–2012. Of these, 202 (59%) were within 10 points of the NHLBI payline that was operative when these applications were being considered for funding, and were therefore awarded based on NHLBI’s special funding consideration for ESIs. We were unable to assess whether knowledge of the funding advantage offered by NHLBI and of other related NIH policies (e.g., favorable resubmission time allowances, clustered review, and rapid processing of new investigator applications) affected resubmission behavior. Given these policies—along with the most recent NIH application submission policy25 that allows resubmission of a previously unsuccessful application—we predict that resubmission of new applications will continue at the same or higher rate. We note three important limitations of this study. First, we could not use data prior to FY 2010 because at the NIH, the American Recovery and Reinvestment Act, the designation of the ESI category, and the new scoring and summary statement percentiling approaches were all implemented in 2009.5,26,27 Second, we did not analyze outcomes for other NIH institutes and centers that may have differing ESI policies, so we cannot speak to ESI resubmission behavior outside of the NHLBI. Lastly, we had too few foreign/international
Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Page 7
Author Manuscript
applicants (n = 2) and too few applicants from groups underrepresented in the biomedical sciences, to allow for robust subgroup analyses. In the future, we plan to assess the long-term research career trajectory and performance of NHLBI newly independent investigators, the relative effect of NHLBI special funding advantage on research career performance among beneficiaries and non-beneficiaries, and the combined effect of NIH ESI-focused policies on research career independence.
Conclusions
Author Manuscript
The NIH and the biomedical research communities must contend with a stringent budgetdriven climate, resulting in shared concerns about the future of new investigators.28,29 For the early career biomedical research scientist, receipt of an NIH R01 (or equivalent) award constitutes a major step towards career independence2,6,7; however, reaching this goal is a challenge for many.2,28,29 Collectively, the length of time between research idea and receipt of an NIH R01 research project award, the multiple years of flat NIH budgets, the ever increasing pool of applications, and the current grant success and funding rates1,30 all portend continued, substantial challenges for early career scientists seeking research independence. Consequently, for many whose cache of research ideas, funding options, and opportunities for establishing career independence are few, R01 grant resubmission could constitute a critical step towards independent research careers.
Author Manuscript
As the third largest NIH institute (with a FY 2014 budget of approximately 2.9 billion dollars31), the NHLBI recognizes the immense challenges facing young investigators pursuing heart-, lung-, blood- and sleep- related research. Within the confines of NIH policies, NHLBI has engaged strategies, including favorable funding advantages for ESIs, to facilitate their timely entry into the biomedical research pipeline.
Acknowledgments The authors thankfully acknowledge Dr. Lawrence Fine, Dr. Paul Sorlie and Mr. Matthew Eblen all of the National Institutes of Health, for their support and invaluable assistance with various aspects of this project. Funding: The authors are employees of the National Institutes of Health and this work was conducted as part of their work activities. No extra funding was received.
References
Author Manuscript
1. Rockey, S. [Accessed September 30, 2015] A Look at Programs Targeting New Scientists. Rock Talk. Apr 29. 2014 http://nexus.od.nih.gov/all/2014/04/29/a-look-at-programs-targeting-newscientists/ 2. Alberts B, Kirschner MW, Tilghman S, Varmus H. Rescuing US biomedical research from its systemic flaws. Proc Natl Acad Sci. 2014; 111:5773–5777. [PubMed: 24733905] 3. Harris, A. [Accessed September 30, 2015] Young, Brilliant and Underfunded. The New York Times. Oct 2. 2014 http://www.nytimes.com/2014/10/03/opinion/young-brilliant-andunderfunded.html?_r=3 4. Harris, R. [Accessed September 30, 2015] When Scientists Give Up. National Public Radio. Sep 9. 2014 http://www.npr.org/blogs/health/2014/09/09/345289127/when-scientists-give-up?refresh=true 5. National Institutes of Health, Office of Extramural Research. [Accessed September 30, 2015] Encouraging Early Transition to Research Independence: Modifying the NIH New Investigator
Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Page 8
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Policy to Identify Early Stage Investigators. NOT-OD-08-121. Updated August 12, 2009. http:// grants.nih.gov/grants/guide/notice-files/NOT-OD-08-121.html 6. National Instutes of Health, The Advisory Committee to the Director. [Accessed September 30, 2015] Draft Report of the Advisory Committee to the Director Working Group on Diversity in the Biomedical Research Workforce. Jun 13. 2012 http://acd.od.nih.gov/Diversity%20in%20the %20Biomedical%20Research%20Workforce%20Report.pdf 7. Garrison HH, Deschams AM. NIH research funding and early career physician scientists: Continuing challenges in the 21st century. FASEB J. 2014; 3:1049–1058. [PubMed: 24297696] 8. National Institutes of Health, Office of Extramural Research. [Accessed September 30, 2015] New and Early Stage Investigator Policies: Definition of New Investigator. Updated February 10, 2014. http://grants.nih.gov/grants/new_investigators/#definition 9. National Institutes of Health, Office of Extramural Research. [Accessed September 30, 2015] Full Implementation to Shorten the Review Cycle for New Investigator R01 Applications Reviewed in Center for Scientific Review (CSR) Recurring Study Sections. NOT-OD-07-083. Updated October 5, 2011. http://grants.nih.gov/grants/guide/notice-files/NOT-OD-07-083.html 10. National Institutes of Health, Office of Extramural Research. [Accessed September 30, 2015] Revised New and Early Stage Investigator Policies. Notice Number: NOT-OD-09-013. Updated May 8, 2013. http://grants.nih.gov/grants/guide/notice-files/NOT-OD-09-013.html 11. National Institutes of Health, National Institute of Allergy and Infectious Diseases (NIAID). [Accessed October 6, 2015] NIAID Glossary of Funding and Policy Terms and Acronyms-P. http://www.niaid.nih.gov/researchfunding/grant/strategy/pages/7payline.aspx 12. National Institutes of Health, National Institute of Allergy and Infectious Diseases (NIAID). [Accessed October 6, 2015] Research Funding: Understand Paylines and Percentiles. http:// www.niaid.nih.gov/researchfunding/glossary/pages/p.aspx 13. National Institutes of Health Office of Extramural Research. [Accessed September 30, 2015] Grants and Funding Glossary and Acronym List: Percentile. Updated March 20, 2013. http:// grants.nih.gov/grants/glossary.htm#Percentile 14. National Institutes of Health, National Institute of Allergy and Infectious Diseases (NIAID). NIAID Glossary of Funding and Policy Terms and Acronyms-S. Study Section. http:// www.niaid.nih.gov/researchfunding/glossary/pages/s.aspx#ss 15. National Institutes of Health; Office of Extramural Research. [Accessed September 30, 2015] Grants and Funding: Definitions of Criteria and Considerations for Research Project Grant (RPG/R01/R03/R15/R21/R34) Critiques. Updated April 15, 2015. https://grants.nih.gov/grants/ peer/critiques/rpg.htm 16. National Institutes of Health. [Accessed October 7, 2015] Enhancing Peer Review: The NIH Announces Enhanced Review Criteria for Evaluation of Research Applications Received for Potential FY2010 Funding. Notice Number: NOT-OD-09-025. http://grants.nih.gov/grants/guide/ notice-files/NOT-OD-09-025.html 17. National Institutes of Health. [Accessed October 7, 2015] Scoring System and Procedure. https:// grants.nih.gov/grants/peer/guidelines_general/scoring_system_and_procedure.pdf 18. National Institutes of Health, National Institute of Allergy and Infectious Diseases (NIAID). [Accessed October 7, 2015] Research Funding: Overall score. http://www.niaid.nih.gov/ researchfunding/glossary/pages/o.aspx#overallscore 19. National Institutes of Health Office of Extramural Research. [Accessed September 30, 2015] Grants and Funding Glossary and Acronym List: Impact Score. Updated March 20, 2013. http:// grants.nih.gov/grants/glossary.htm#ImpactScore 20. Biau G. Analysis of a Random Forests Model. J Mach Learn Res. 2012; 13:1063–1095. 21. Rockey, S. [Accessed September 29, 2015] Does Your Institution Matter?. Rock Talk. May 14. 2012 http://nexus.od.nih.gov/all/2012/05/14/does-your-institution-matter/ 22. Ginther DK, Schaffer WT, Schnell J, et al. Race, Ethnicity, and NIH Research Awards. Science. 2011; 333:1015–1019. [PubMed: 21852498] 23. Ley TJ, Hamilton BH. The Gender Gap in NIH Grant Applications. Science. 2008; 322:1472– 1474. [PubMed: 19056961]
Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Page 9
Author Manuscript Author Manuscript
24. Dunbar, CE.; Abkowitz, JL.; Berliner, N.; Shurin, SB. [Accessed September 30, 2015] “Ask and Ye Shall Receive” – Women, Stand Up and Be Counted. The Hematologist. May 30. 2014 http:// www.hematology.org/Thehematologist/Op-Ed/2883.aspx 25. National Institutes of Health, Office of Extramural Research. [Accessed September 30, 2015] NIH and AHRQ Announce Updated Policy for Application Submission. Notice Number: NOTOD-14-074. Updated April 22, 2014. http://grants.nih.gov/grants/guide/notice-files/NOTOD-14-074.html 26. National Institutes of Health. [Accessed October 7, 2015] The NIH Director: NIH’s Role in the American Recovery and Reinvestment Act (ARRA). http://www.nih.gov/about/director/ 02252009statement_arra.htm 27. National Institutes of Health. [Accessed October 7, 2015] NIH and the American Recovery and Reinvestment Act : NIH Grants Funded by the American Recovery and Reinvestment Act of 2009. http://report.nih.gov/recovery/ARRAFunding.aspx 28. Langer JS. Enabling scientific innovation. Science. 2012; 338:171. [PubMed: 23066044] 29. Ioannidis JP. More time for research: Fund people not projects. Nature. 2011; 477:529–531. [PubMed: 21956312] 30. U.S. Department of Health and Human Services. [Accessed September 30, 2015] NIH Research Portfolio online Reporting Tools (RePORT). NIH Data Book. Success Rates and Funding Rates, R01-Equivalent investigators: Competing applicants, awardees, and funding rates. Updated June 15, 2015. http://report.nih.gov/nihdatabook/ 31. Consolidated and Further Continuing Appropriations Act of 2015; 113th Congress of the United States; Public Law No: 113-235. December 16, 2014. https://www.congress.gov/113/plaws/ publ235/PLAW-113publ235.pdf
Author Manuscript Author Manuscript Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Page 10
Author Manuscript Author Manuscript Figure 1.
Author Manuscript Author Manuscript
Variable importance and probability of resubmission for all National Heart, Lung, and Blood Institute (NHLBI) FY 2010–2012 early stage investigator unfunded A0 R01 application, by applicant-, application- and institution-based factors. Source: National Institutes of Health (NIH) Information for Management, Planning, and Coordination II (IMPAC II) database. Approach, Investigator, Significance, Innovation, and Environment are the five peer reviewed criterion on which NIH grant applications are scored. Requested budget is the total budget requested in an application. FY is the fiscal year in which the application was reviewed. F-grant indicates fellowship training grant award; K-award indicates career development grant awards; and T-grant means training grant award. ICR indicates “Initial Council Review” (the NIH institutes generally make grant-funding decisions after grants have been presented at one of their advisory council meetings). SEP indicates a “Special Emphasis Panel,” which is a review group set up to conduct a one-time review of grants; RPG indicates “Research project grant,” which is an award that supports scientific research projects. NIH institutional rank is the rank quintile in which an institutions falls, based on the total amount of funds received from NIH in the last 5 years.
Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Page 11
Author Manuscript Author Manuscript
Figure 2.
Author Manuscript
Variable importance and probability of resubmission for discussed and percentiled National Heart, Lung, and Blood Institute (NHLBI) FY 2010–2012 early stage investigator unfunded A0 R01 application, by grant-, application- and institution-based factors. Source: National Institutes of Health (NIH) Information for Management, Planning, and Coordination II (IMPAC II) database. Approach, Investigator, Significance, Innovation, and Environment are the five peer reviewed criterion on which NIH grant applications are scored. Requested budget is the total budget requested in an application. FY is the fiscal year in which the application was reviewed. F-grant indicates fellowship training grant; K-award indicates career development grant award; and T-grant means training grant. ICR indicates “Initial Council Review” (the NIH institutes generally make grant funding decisions after grants have been presented at one of their advisory council meetings). SEP indicates a “Special Emphasis Panel,” which is a review group set up to conduct a one-time review of grants. RPG indicates “research project grant,” which is an award that supports scientific research projects. Payline is the cutoff point set by NHLBI for funding grant applications, after balancing projected grant numbers, total grant budgets, and funds available. Distance from payline, a derived variable, is the calculated distance of the difference between the overall impact score and the payline operative at the time funding decisions are made. NIH Institutional Rank is the rank quintile in which an institutions falls based on the total amount of funds received from NIH in the last 5 years.
Author Manuscript Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Page 12
Author Manuscript Author Manuscript
Figure 3.
RandomForest Adjusted probability of resubmission for percentiled grants according to distance from payline. The shaded space connotes the confidence interval of the points in the plotted line.
Author Manuscript Author Manuscript Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Page 13
Table 1
Author Manuscript
Frequencies of Baseline Grant-, Applicant-, and Institution-Based Variables for NHLBI Unfunded Early-Stage Investigator Applications, FY 2010–2012a Variables
No. (% of 821)
Grant-based Submitted FY10
243 (29.6)
Submitted FY11
281 (34.2)
Submitted FY12
297 (36.2)
Special emphasis panel reviewb
62 (7.6)
All percentiled grantsc
382 (46.5)
Grants with a percentile score < 50c
294 (35.8)
Grants with previous non-R01 awardsd
135 (16.4)
Author Manuscript
Applicant-based Demographicse Male-PI applications
560 (68.2)
Female-PI applications
260 (31.7)
Whitef
514 (62.6)
Asianf
239 (29.1)
Blackf
20 (2.4)
Hispanicf
36 (4.4)
Training
Author Manuscript
MD
207 (25.2)
PhD
476 (57.9)
Mixed degrees
6 (0.7)
MPH
132 (16.1)
Other degrees
1 (0.1)
Previous training-related grant awards Previous K-grant awardeed
246 (30.0)
Previous T-32 recipientd
262 (31.9)
Previous F-grant awardeed
87 (10.6)
Institution-based Foreign or international
2 (0.2)
Author Manuscript
Higher education
155 (18.9)
Medical school
489 (59.6)
Research institution Hospital Other institution
52 (6.3) 105 (12.8) 18 (2.2)
Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Variables
Page 14
No. (% of 821)
Author Manuscript
Rank 1 institutions g
326 (39.7)
Rank 2 institutions g
279 (34.0)
Rank 3 institutions g
119 (14.5)
Rank 4 institutions g
91 (11.1)
Rank 5 institutions g
6 (0.7)
Abbreviations: NHLBI indicates National Heart, Lung, and Blood Institute; FY, fiscal year; PI Principal investigator. a
Data source: National Institutes of Health Information for Management, Planning, and Coordination II (IMPAC II) database.
b
Special emphasis panels are review panels of scientific experts that are convened for special, one-time grant review events.
c
Percentiled applications are those that undergo discussion and receive overall impact scores. Lower percentiles represent better scores or a more favorable review.
Author Manuscript
d
The National Institutes of Health funds multiple types of grants including but not limited to research project grants known as the R-series grants (e.g., R01), career award grants known as the K-Series (e.g., K01), and training grants known as the T-or F-series (e.g., T-32). e
The average age of the applicants was 40.9 years (standard deviation = 0.17). One candidate did not provide gender.
f Twelve applications did not provide the race or ethnicity of the PI. g Rank indicates the relative position of institutions with respect to receipt of NIH funds. This was an NIH-internally generated variable. Rank 1 indicates institutions that receive enough NIH funds to fall into the top quintile; Rank 2 indicates those in the second quintile; Rank 3 indicates those that fall into the third quintile; Rank 4 indicates institutions that fall into the fourth quintile; and Rank 5 indicates institutions that fall into the bottom quintile in terms of NIH funds received.
Author Manuscript Author Manuscript Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Page 15
Table 2
Author Manuscript
Mean review criterion scores for NHLBI Unfunded Early-Stage Investigator Applications, FY 2010–2012 Criterion
Mean scorea (SD)
Approach score
3.7 (0.04)
Significance score
3.6 (0.04)
Innovation score
3.6 (0.04)
Investigator score
4.9 (0.03)
Environment score
2.3 (0.03)
Abbreviations: NHLBI indicates National Heart, Lung, and Blood Institute; FY, fiscal year; SD, standard deviation. a
Scores can range from 1 = exceptional to 9 = Poor.
Author Manuscript Author Manuscript Author Manuscript Acad Med. Author manuscript; available in PMC 2017 April 01.
Acad Med. Author manuscript; available in PMC 2017 April 01. Published in final edited form as: Acad Med. 2016 April ; 91(4): 556–562. doi:10.1097/ACM.0000000000001025.
Towards Independence: Resubmission Rate of Unfunded National Heart, Lung, and Blood Institute R01 Research Grant Applications among Early Stage Investigators
Author Manuscript
Josephine E.A. Boyington, PhD, MPH, CNS, Health scientist administrator/program director, Clinical Applications and Prevention Branch, Division of Cardiovascular Sciences, National Heart, Lung, and Blood Institute, National Institutes of Health, Bethesda, Maryland Melissa D. Antman, PhD, Health scientist administrator, Office of Extramural Policy and Training, Division of Extramural Research Activities, National Heart, Lung, and Blood Institute, National Institutes of Health, Bethesda, Maryland Katherine C. Patel, MSPH, and Mathematical statistician, Division of Statistical Analysis and Reporting, Office of Planning, Analysis, and Communication, Office of Extramural Research, Office of the Director, National Institutes of Health, Bethesda, Maryland
Author Manuscript
Michael S. Lauer, MD Director, Division of Cardiovascular Sciences, National Heart, Lung, and Blood Institute, National Institutes of Health, Bethesda, Maryland
Abstract Purpose—The current, budget-driven low rate of the National Institutes of Health (NIH) funding for biomedical research has raised concerns about the ability of new investigators to become independent scientists and their willingness to persist in efforts to secure funding. The authors sought to determine resubmission rates for unfunded National Heart, Lung, and Blood Institute (NHLBI) early stage investigator (ESI) independent research grant (RO1) applications and to identify predictors of resubmission.
Author Manuscript
Method—The authors used a retrospective cohort study design and retrieved applications submitted in fiscal years 2010–2012 from NIH electronic research administrative sources. They defined ESI applicants as those who have received no prior R01 (or equivalent) funding and were
Correspondence should be addressed to Josephine E. A. Boyington, 6701 Rockledge Drive, Suite 10224, Rockledge II, Bethesda, MD 20892; telephone: (301) 435-0446; [email protected]. Other disclosures: None reported. Ethical approval: This work was conducted as part of the authors’ National Institutes of Health employment activities, and all records were de-identified. Disclaimers: The information in this report represents the work of the authors alone, and does not necessarily indicate the views of the National Institutes of Health or the federal government of the United States. Previous presentations: This work was presented internally at the National Institutes of Health Portfolio Analysis Research Symposium and at National Heart, Lung, and Blood Institute’s Staff Meetings.
Boyington et al.
Page 2
Author Manuscript
within 10 years of completion of their terminal research degree or medical residency training. ESI applications at the NHLBI were eligible for special funding consideration if they scored above, but within ten points of, the payline. The primary outcome was application resubmission after failing to secure funding with the first R01 submission. Results—Over half of the unfunded applications were resubmitted. Some of these were discussed and “percentiled,” and among these applications, the only significant predictor of resubmission was the percentile score. Over half (59%) of the ESI R01grants funded by NHLBI in FY-2010–2012 had percentile scores above, but within ten points of the NHLBI payline, and benefitted from the special funding considerations. Conclusions—The only independent predictor of resubmission of NHLBI ESI R01 grant applications was percentile score; applicant demographics and institutional factors were not predictive of resubmission.
Author Manuscript Author Manuscript
Sally Rockey, the immediate past Deputy Director for Extramural Research at the National Institutes of Health (NIH), began the April 29, 2014, post of her blog “Rock Talk” writing, “The strength of the biomedical research enterprise depends on new researchers becoming independent NIH-funded researchers”1; however, for many new investigators, receipt of an NIH independent research (R01) award presents a significant challenge.2,3 Richard Harris, in his recently posted National Public Radio story “When Scientists Give Up,”4 described the pursuits of two early career scientists—both of whom unsuccessfully sought independent research grant awards and subsequently abandoned their research careers. National Heart, Lung, and Blood Institute (NHLBI) staff members have heard similar stories from other early career investigators. One possible manifestation of these junior researchers’ frustration might be their decision not to resubmit an application after failing to secure funding with the first peer-reviewed R01 grant application. We decided, therefore, to measure and ascertain possible correlates for the resubmission of unfunded R01 grants submitted to the NHLBI by early stage investigators (ESIs) during fiscal year (FY) 2010, 2011, and 2012.
Terms, Considerations, and R01 Review Processes
Author Manuscript
We focused on the R01 grant mechanism because it is the hallmark of research independence and because NIH affords funding benefits to ESIs through this mechanism.2,5–7 NIH defines an ESI as any applicant who is within ten years of completion of his/her terminal research degree or medical residency and has never received an R01 grant (or equivalent).5,8 Policies related to early career scientists include the clustering and review of ESIs’ applications apart from those from established investigators,5,8 the provision of opportunities to resubmit unfunded grants earlier and thereby reduce the time to resubmission by four months,9 and fiscal policies that ensure that new investigators are funded at the same rate as established investigators.10 Within the confines of these NIH policies, individual NIH institutes freely develop and implement strategies that ensure a scientifically-diverse and workforce-balanced portfolio. During FY 2010–2012, all unfunded NHLBI ESI R01 applications were eligible for special funding considerations if their percentile scores (see below) were within 10 points of the
Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Page 3
Author Manuscript
payline (i.e., the cutoff point set by NHLBI for funding grant applications, after balancing projected grant numbers, total grant budgets, and funds available).11,12 All applications submitted to NIH are reviewed, but not all applications are discussed at peer-review meetings. Discussed applications receive criterion scores for five core variables, an overall impact score, and a percentile score. Applications that are not discussed receive criterion scores but no overall impact score and no percentile score. It is generally assumed that these applications are above the 50th percentile of the applications submitted for review.
Author Manuscript
The percentile score13 is a measure of an application’s relative rank compared to other applications reviewed by a particular study section (i.e., a panel of scientific experts [i.e., “reviewers”], convened by the NIH Center for Scientific Review or an NIH institute, to provide initial scientific review).14 It reflects reviewers’ global assessment of the application’s scientific and technical merit. The score ranges from 1st through 99th percentile, and favorable scores are below the 50th percentile.13
Author Manuscript
The overall impact score reflects reviewers’ judgment of a project’s ability to influence its field, and it is determined by reviewers’ ratings of the overall merit of an application, particularly after taking into consideration the five core peer review criteria—(1) significance, (2) approach, (3) investigator, (4) innovation, and (5) environment—in addition to other application relevant factors such as human subjects protections. In their consideration of significance, reviewers reflect on the extent to which a research “project address[es] an important problem or a critical barrier to progress in the field,” 15,16 and in their consideration of approach, they assess how well a research project’s “strategy, methodology, and analyses [are] well-reasoned and appropriate to accomplish the stated objectives of the project.”15,16 Reviewers considering the investigator criterion ask the following key questions: “Are the PD/PIs [program director/principal investigators], collaborators, and other researchers well suited to the project?” and “If Early Stage Investigators or New Investigators, or in the early stages of independent careers, do they have appropriate experience and training?”15 The consideration of innovation focuses on “how much a project can shift the current research or clinical paradigms by utilizing novel theoretical concepts, approaches, or methodologies, instrumentation, or interventions” and, finally, the review of environment centers on whether “the scientific environment in which the work will be done contribute[s] to the probability of success”; whether “the institutional support, equipment and other physical resources available to the investigators [are] adequate for the project proposed”; and whether the project would “benefit from unique features of the scientific environment, subject populations, or collaborative arrangements?”15,16
Author Manuscript
The overall impact score is not an average of the five core criteria scores; rather it is the average of the individual ratings of overall merit given by all reviewers after assessment of the five scored criteria listed above, plus, as mentioned, additional concerns such as the protection of human subjects, the welfare and care of vertebrate animals, and the consideration of biohazards.17,18 The relationship between overall impact score and percentile score is that the percentile score normalizes the application’s overall impact score across study sections and thereby attenuates unintended variations in scoring.19
Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Page 4
Author Manuscript
Method In our effort to examine resubmissions (our primary end point), we extracted and deidentified all unfunded, competing, new (A0) R01 applications that had undergone peer review during FY 2010 – 2012. We extracted the applications from the NIH electronic research administration “Information for Management, Planning, and Coordination II (IMPAC II)” database. As potential predictors of resubmission, we considered grant-based factors (e.g., year of original submission and peer-review scores), applicant-based factors (e.g., demographics, training, prior funding), and institution-based factors (e.g., ranking in receipt of NIH funding, and type of institution [research, medical, other]).
Author Manuscript
We determined variable frequencies with descriptive analyses, and we used Random Forest methodology,20 a robust machine-learning-based predictors’ ensemble approach, to identify the most important predictor variables for inclusion in subsequent logistic regression models. We first analyzed the overall pool of applications and subsequently the subset of applications that were discussed and percentiled. We performed all analyses at the NIH, using R3.1.0 (packages Hmisc, rms, ggplot2, and randomForestSRC).
Results We extracted, for applications reviewed in FY 2010–2012, a total of 34,240 unfunded A0 NIH grant applications. Of the 4,587 NHLBI applications, 833 (18.2%) were ESI applications. (Comparatively, of the 29,653 remaining NIH grants, 5,268 (17.8%) were ESI applications.) We focused our analyses on the 821 NHLBI ESIs applications that were not formally withdrawn. Table 1 summarizes grant-, applicant-, and institution-based characteristics.
Author Manuscript
Resubmitted applications The overall resubmission rate was 51.4% (422 of 821). Of the pool of 821 applications, 382 (46.5%) were discussed and given a percentile score and 294 (35.8%) scored less than 50 (Table 1). Of the applications with percentile score less than 50, 82.3% (242 of 294) were resubmitted, whereas among the 527 applications with a percentile score equal to or greater than 50, only 180 (34.2%) were resubmitted (P < .001). Using the random forest machinelearning approach on the entire pool of applications, we observed that the most important correlate, by far, of resubmission was a percentile score less than 50. The investigator, approach, significance, innovation, and environment criterion scores were less correlated with resubmission (Figure 1). Neither applicant- nor institutional-based variables emerged as important correlates.
Author Manuscript
In the corresponding logistic regression model, the only independent predictors were percentile score of less than 50 (P < .001) and the investigator criterion score (P = .01). We noted that 560 of 821 (68.2%) of the ESI applications were from male PIs and that the sample of applications from groups underrepresented in the biomedical sciences was small, thereby limiting our ability to conduct subgroup analyses.
Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Page 5
Discussed and percentiled applications
Author Manuscript
For the 382 applications that were both discussed and received a percentile ranking, we examined the association between, overall impact score, raw percentile score, percentile distance from payline, and resubmission status. Overall impact score did not emerge as a key predictor variable; however, we found that the raw percentile score and the percentile distance from the payline were essentially equivalent predictors of resubmission. We calculated “percentile distance” as a derived variable, by measuring the distance of an application’s percentile score from the operative NHLBI payline at the time of funding consideration. The payline at NHLBI generally changes over the course of the fiscal year, being more conservative (lower) at the beginning of the year, and more generous (higher) towards the end. With the calculation of percentile distance, we were able to create a variable with seemingly greater face value than raw percentile score and to also account for the impact of payline changes on resubmission.
Author Manuscript
As such, we tested percentile distance along with other candidate predictors in a random forest model and found that it was, by far, the strongest correlate of resubmission, followed by the significance and approach criterion scores (Figure 2). In the logistic regression model including percentile distance, the five criterion scores, requested budget, and gender, the only independent predictor was percentile distance (P < .001). We further examined the relationship between percentile distance and the probability of resubmission and found that as percentile distance increased, the probability of resubmission decreased (Figure 3). Unfunded applications eligible for NHLBI special funding consideration FY 2010–2012
Author Manuscript
During FY 2010–2012, NHLBI considered 342 total (new and resubmitted) ESI R01 applications for funding. Two hundred and two (59%) of these had percentile scores above, but within ten points of the NHLBI R01 payline, and were, therefore, awarded funding based on NHLBI’s special funding consideration for ESIs.
Discussion In the current budget-driven fiscal climate, early career investigators may be frustrated by failure to secure an R01 grant and may express this frustration by not resubmitting unfunded applications. We found that slightly more than half of NHLBI ESIs (51.4%) resubmitted their unfunded applications, and that in the regression model of the overall pool of applications, a percentile score of less than 50 and the investigator criterion score were the two significant predictors of resubmission.
Author Manuscript
As noted above, percentile, reflects the relative rank of an application compared to others reviewed by a particular study section, and the investigator criterion score reflects reviewers’ judgment of a PI and his/her team’s competence to execute a proposed project. A higher investigator criterion score, therefore, indicates the need for the PI to gain more practical and relevant research skills, the need for a stronger and more complementary research team, or both. Compared to other factors that could affect the investigator criterion score, skill set and an appropriate research team are more easily changeable than, for example, demographics (race/ethnicity) or degree type (MD vs. PhD). A recent internal NIH
Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Page 6
Author Manuscript
analysis using overall NIH data determined that degree type has no measurable impact on R01 success rates.21 In contrast, a recent NIH-commissioned external study that sought to understand the relationship between race/ethnicity and grant receipt, documented differential probabilities of R01 grant receipts by race/ethnicity.22 Due to insufficient data, the study did not report institute-specific, race-based results, but did show that overall, Hispanic and Black applicants were less likely than others to resubmit a revised application.22
Author Manuscript
Our study differs from previous work in that we focused specifically on early stage investigators and on a critical antecedent behavior for grant receipt—that is resubmission of unfunded R01 grants. Although, we focused on the vulnerable group of early career investigators, we were underpowered to conduct subgroup analyses (e.g., race/ethnicity) because of our small sample size. We found correlations between application-based characteristics and resubmission of an unfunded R01 application but no independent association of applicant- or institution-based characteristics with resubmission. We did observe twice as many male as female applicants in our cohort of 821. Ley and colleagues23 and Dunbar and colleagues24 have reported similar observations, and have commented that observed differences are evidence of the greater exodus of females at the independent career stage and not a result of gender-based differences in success rates. Apart from the proportional differences in the number of applicants by gender, we found that gender was not a significant predictor of resubmission.
Author Manuscript
The finding that discussed (thus percentiled) applications were resubmitted at a higher rate is reassuring since it indicates that an objectively-derived application attribute (i.e., percentile score) seemingly drives resubmission behavior. As noted earlier, we determined percentile distance to be an essentially equivalent surrogate for raw percentile score, and to be the only significant predictor of resubmission for discussed applications. Consequently, suggesting that the primary driver in applicant’s decision to resubmit should be the application’s overall percentile score and less so the individual criterion or overall impact scores.
Author Manuscript
NHLBI funded 342 ESI R01 applications in FY 2010–2012. Of these, 202 (59%) were within 10 points of the NHLBI payline that was operative when these applications were being considered for funding, and were therefore awarded based on NHLBI’s special funding consideration for ESIs. We were unable to assess whether knowledge of the funding advantage offered by NHLBI and of other related NIH policies (e.g., favorable resubmission time allowances, clustered review, and rapid processing of new investigator applications) affected resubmission behavior. Given these policies—along with the most recent NIH application submission policy25 that allows resubmission of a previously unsuccessful application—we predict that resubmission of new applications will continue at the same or higher rate. We note three important limitations of this study. First, we could not use data prior to FY 2010 because at the NIH, the American Recovery and Reinvestment Act, the designation of the ESI category, and the new scoring and summary statement percentiling approaches were all implemented in 2009.5,26,27 Second, we did not analyze outcomes for other NIH institutes and centers that may have differing ESI policies, so we cannot speak to ESI resubmission behavior outside of the NHLBI. Lastly, we had too few foreign/international
Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Page 7
Author Manuscript
applicants (n = 2) and too few applicants from groups underrepresented in the biomedical sciences, to allow for robust subgroup analyses. In the future, we plan to assess the long-term research career trajectory and performance of NHLBI newly independent investigators, the relative effect of NHLBI special funding advantage on research career performance among beneficiaries and non-beneficiaries, and the combined effect of NIH ESI-focused policies on research career independence.
Conclusions
Author Manuscript
The NIH and the biomedical research communities must contend with a stringent budgetdriven climate, resulting in shared concerns about the future of new investigators.28,29 For the early career biomedical research scientist, receipt of an NIH R01 (or equivalent) award constitutes a major step towards career independence2,6,7; however, reaching this goal is a challenge for many.2,28,29 Collectively, the length of time between research idea and receipt of an NIH R01 research project award, the multiple years of flat NIH budgets, the ever increasing pool of applications, and the current grant success and funding rates1,30 all portend continued, substantial challenges for early career scientists seeking research independence. Consequently, for many whose cache of research ideas, funding options, and opportunities for establishing career independence are few, R01 grant resubmission could constitute a critical step towards independent research careers.
Author Manuscript
As the third largest NIH institute (with a FY 2014 budget of approximately 2.9 billion dollars31), the NHLBI recognizes the immense challenges facing young investigators pursuing heart-, lung-, blood- and sleep- related research. Within the confines of NIH policies, NHLBI has engaged strategies, including favorable funding advantages for ESIs, to facilitate their timely entry into the biomedical research pipeline.
Acknowledgments The authors thankfully acknowledge Dr. Lawrence Fine, Dr. Paul Sorlie and Mr. Matthew Eblen all of the National Institutes of Health, for their support and invaluable assistance with various aspects of this project. Funding: The authors are employees of the National Institutes of Health and this work was conducted as part of their work activities. No extra funding was received.
References
Author Manuscript
1. Rockey, S. [Accessed September 30, 2015] A Look at Programs Targeting New Scientists. Rock Talk. Apr 29. 2014 http://nexus.od.nih.gov/all/2014/04/29/a-look-at-programs-targeting-newscientists/ 2. Alberts B, Kirschner MW, Tilghman S, Varmus H. Rescuing US biomedical research from its systemic flaws. Proc Natl Acad Sci. 2014; 111:5773–5777. [PubMed: 24733905] 3. Harris, A. [Accessed September 30, 2015] Young, Brilliant and Underfunded. The New York Times. Oct 2. 2014 http://www.nytimes.com/2014/10/03/opinion/young-brilliant-andunderfunded.html?_r=3 4. Harris, R. [Accessed September 30, 2015] When Scientists Give Up. National Public Radio. Sep 9. 2014 http://www.npr.org/blogs/health/2014/09/09/345289127/when-scientists-give-up?refresh=true 5. National Institutes of Health, Office of Extramural Research. [Accessed September 30, 2015] Encouraging Early Transition to Research Independence: Modifying the NIH New Investigator
Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Page 8
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Policy to Identify Early Stage Investigators. NOT-OD-08-121. Updated August 12, 2009. http:// grants.nih.gov/grants/guide/notice-files/NOT-OD-08-121.html 6. National Instutes of Health, The Advisory Committee to the Director. [Accessed September 30, 2015] Draft Report of the Advisory Committee to the Director Working Group on Diversity in the Biomedical Research Workforce. Jun 13. 2012 http://acd.od.nih.gov/Diversity%20in%20the %20Biomedical%20Research%20Workforce%20Report.pdf 7. Garrison HH, Deschams AM. NIH research funding and early career physician scientists: Continuing challenges in the 21st century. FASEB J. 2014; 3:1049–1058. [PubMed: 24297696] 8. National Institutes of Health, Office of Extramural Research. [Accessed September 30, 2015] New and Early Stage Investigator Policies: Definition of New Investigator. Updated February 10, 2014. http://grants.nih.gov/grants/new_investigators/#definition 9. National Institutes of Health, Office of Extramural Research. [Accessed September 30, 2015] Full Implementation to Shorten the Review Cycle for New Investigator R01 Applications Reviewed in Center for Scientific Review (CSR) Recurring Study Sections. NOT-OD-07-083. Updated October 5, 2011. http://grants.nih.gov/grants/guide/notice-files/NOT-OD-07-083.html 10. National Institutes of Health, Office of Extramural Research. [Accessed September 30, 2015] Revised New and Early Stage Investigator Policies. Notice Number: NOT-OD-09-013. Updated May 8, 2013. http://grants.nih.gov/grants/guide/notice-files/NOT-OD-09-013.html 11. National Institutes of Health, National Institute of Allergy and Infectious Diseases (NIAID). [Accessed October 6, 2015] NIAID Glossary of Funding and Policy Terms and Acronyms-P. http://www.niaid.nih.gov/researchfunding/grant/strategy/pages/7payline.aspx 12. National Institutes of Health, National Institute of Allergy and Infectious Diseases (NIAID). [Accessed October 6, 2015] Research Funding: Understand Paylines and Percentiles. http:// www.niaid.nih.gov/researchfunding/glossary/pages/p.aspx 13. National Institutes of Health Office of Extramural Research. [Accessed September 30, 2015] Grants and Funding Glossary and Acronym List: Percentile. Updated March 20, 2013. http:// grants.nih.gov/grants/glossary.htm#Percentile 14. National Institutes of Health, National Institute of Allergy and Infectious Diseases (NIAID). NIAID Glossary of Funding and Policy Terms and Acronyms-S. Study Section. http:// www.niaid.nih.gov/researchfunding/glossary/pages/s.aspx#ss 15. National Institutes of Health; Office of Extramural Research. [Accessed September 30, 2015] Grants and Funding: Definitions of Criteria and Considerations for Research Project Grant (RPG/R01/R03/R15/R21/R34) Critiques. Updated April 15, 2015. https://grants.nih.gov/grants/ peer/critiques/rpg.htm 16. National Institutes of Health. [Accessed October 7, 2015] Enhancing Peer Review: The NIH Announces Enhanced Review Criteria for Evaluation of Research Applications Received for Potential FY2010 Funding. Notice Number: NOT-OD-09-025. http://grants.nih.gov/grants/guide/ notice-files/NOT-OD-09-025.html 17. National Institutes of Health. [Accessed October 7, 2015] Scoring System and Procedure. https:// grants.nih.gov/grants/peer/guidelines_general/scoring_system_and_procedure.pdf 18. National Institutes of Health, National Institute of Allergy and Infectious Diseases (NIAID). [Accessed October 7, 2015] Research Funding: Overall score. http://www.niaid.nih.gov/ researchfunding/glossary/pages/o.aspx#overallscore 19. National Institutes of Health Office of Extramural Research. [Accessed September 30, 2015] Grants and Funding Glossary and Acronym List: Impact Score. Updated March 20, 2013. http:// grants.nih.gov/grants/glossary.htm#ImpactScore 20. Biau G. Analysis of a Random Forests Model. J Mach Learn Res. 2012; 13:1063–1095. 21. Rockey, S. [Accessed September 29, 2015] Does Your Institution Matter?. Rock Talk. May 14. 2012 http://nexus.od.nih.gov/all/2012/05/14/does-your-institution-matter/ 22. Ginther DK, Schaffer WT, Schnell J, et al. Race, Ethnicity, and NIH Research Awards. Science. 2011; 333:1015–1019. [PubMed: 21852498] 23. Ley TJ, Hamilton BH. The Gender Gap in NIH Grant Applications. Science. 2008; 322:1472– 1474. [PubMed: 19056961]
Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Page 9
Author Manuscript Author Manuscript
24. Dunbar, CE.; Abkowitz, JL.; Berliner, N.; Shurin, SB. [Accessed September 30, 2015] “Ask and Ye Shall Receive” – Women, Stand Up and Be Counted. The Hematologist. May 30. 2014 http:// www.hematology.org/Thehematologist/Op-Ed/2883.aspx 25. National Institutes of Health, Office of Extramural Research. [Accessed September 30, 2015] NIH and AHRQ Announce Updated Policy for Application Submission. Notice Number: NOTOD-14-074. Updated April 22, 2014. http://grants.nih.gov/grants/guide/notice-files/NOTOD-14-074.html 26. National Institutes of Health. [Accessed October 7, 2015] The NIH Director: NIH’s Role in the American Recovery and Reinvestment Act (ARRA). http://www.nih.gov/about/director/ 02252009statement_arra.htm 27. National Institutes of Health. [Accessed October 7, 2015] NIH and the American Recovery and Reinvestment Act : NIH Grants Funded by the American Recovery and Reinvestment Act of 2009. http://report.nih.gov/recovery/ARRAFunding.aspx 28. Langer JS. Enabling scientific innovation. Science. 2012; 338:171. [PubMed: 23066044] 29. Ioannidis JP. More time for research: Fund people not projects. Nature. 2011; 477:529–531. [PubMed: 21956312] 30. U.S. Department of Health and Human Services. [Accessed September 30, 2015] NIH Research Portfolio online Reporting Tools (RePORT). NIH Data Book. Success Rates and Funding Rates, R01-Equivalent investigators: Competing applicants, awardees, and funding rates. Updated June 15, 2015. http://report.nih.gov/nihdatabook/ 31. Consolidated and Further Continuing Appropriations Act of 2015; 113th Congress of the United States; Public Law No: 113-235. December 16, 2014. https://www.congress.gov/113/plaws/ publ235/PLAW-113publ235.pdf
Author Manuscript Author Manuscript Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Page 10
Author Manuscript Author Manuscript Figure 1.
Author Manuscript Author Manuscript
Variable importance and probability of resubmission for all National Heart, Lung, and Blood Institute (NHLBI) FY 2010–2012 early stage investigator unfunded A0 R01 application, by applicant-, application- and institution-based factors. Source: National Institutes of Health (NIH) Information for Management, Planning, and Coordination II (IMPAC II) database. Approach, Investigator, Significance, Innovation, and Environment are the five peer reviewed criterion on which NIH grant applications are scored. Requested budget is the total budget requested in an application. FY is the fiscal year in which the application was reviewed. F-grant indicates fellowship training grant award; K-award indicates career development grant awards; and T-grant means training grant award. ICR indicates “Initial Council Review” (the NIH institutes generally make grant-funding decisions after grants have been presented at one of their advisory council meetings). SEP indicates a “Special Emphasis Panel,” which is a review group set up to conduct a one-time review of grants; RPG indicates “Research project grant,” which is an award that supports scientific research projects. NIH institutional rank is the rank quintile in which an institutions falls, based on the total amount of funds received from NIH in the last 5 years.
Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Page 11
Author Manuscript Author Manuscript
Figure 2.
Author Manuscript
Variable importance and probability of resubmission for discussed and percentiled National Heart, Lung, and Blood Institute (NHLBI) FY 2010–2012 early stage investigator unfunded A0 R01 application, by grant-, application- and institution-based factors. Source: National Institutes of Health (NIH) Information for Management, Planning, and Coordination II (IMPAC II) database. Approach, Investigator, Significance, Innovation, and Environment are the five peer reviewed criterion on which NIH grant applications are scored. Requested budget is the total budget requested in an application. FY is the fiscal year in which the application was reviewed. F-grant indicates fellowship training grant; K-award indicates career development grant award; and T-grant means training grant. ICR indicates “Initial Council Review” (the NIH institutes generally make grant funding decisions after grants have been presented at one of their advisory council meetings). SEP indicates a “Special Emphasis Panel,” which is a review group set up to conduct a one-time review of grants. RPG indicates “research project grant,” which is an award that supports scientific research projects. Payline is the cutoff point set by NHLBI for funding grant applications, after balancing projected grant numbers, total grant budgets, and funds available. Distance from payline, a derived variable, is the calculated distance of the difference between the overall impact score and the payline operative at the time funding decisions are made. NIH Institutional Rank is the rank quintile in which an institutions falls based on the total amount of funds received from NIH in the last 5 years.
Author Manuscript Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Page 12
Author Manuscript Author Manuscript
Figure 3.
RandomForest Adjusted probability of resubmission for percentiled grants according to distance from payline. The shaded space connotes the confidence interval of the points in the plotted line.
Author Manuscript Author Manuscript Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Page 13
Table 1
Author Manuscript
Frequencies of Baseline Grant-, Applicant-, and Institution-Based Variables for NHLBI Unfunded Early-Stage Investigator Applications, FY 2010–2012a Variables
No. (% of 821)
Grant-based Submitted FY10
243 (29.6)
Submitted FY11
281 (34.2)
Submitted FY12
297 (36.2)
Special emphasis panel reviewb
62 (7.6)
All percentiled grantsc
382 (46.5)
Grants with a percentile score < 50c
294 (35.8)
Grants with previous non-R01 awardsd
135 (16.4)
Author Manuscript
Applicant-based Demographicse Male-PI applications
560 (68.2)
Female-PI applications
260 (31.7)
Whitef
514 (62.6)
Asianf
239 (29.1)
Blackf
20 (2.4)
Hispanicf
36 (4.4)
Training
Author Manuscript
MD
207 (25.2)
PhD
476 (57.9)
Mixed degrees
6 (0.7)
MPH
132 (16.1)
Other degrees
1 (0.1)
Previous training-related grant awards Previous K-grant awardeed
246 (30.0)
Previous T-32 recipientd
262 (31.9)
Previous F-grant awardeed
87 (10.6)
Institution-based Foreign or international
2 (0.2)
Author Manuscript
Higher education
155 (18.9)
Medical school
489 (59.6)
Research institution Hospital Other institution
52 (6.3) 105 (12.8) 18 (2.2)
Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Variables
Page 14
No. (% of 821)
Author Manuscript
Rank 1 institutions g
326 (39.7)
Rank 2 institutions g
279 (34.0)
Rank 3 institutions g
119 (14.5)
Rank 4 institutions g
91 (11.1)
Rank 5 institutions g
6 (0.7)
Abbreviations: NHLBI indicates National Heart, Lung, and Blood Institute; FY, fiscal year; PI Principal investigator. a
Data source: National Institutes of Health Information for Management, Planning, and Coordination II (IMPAC II) database.
b
Special emphasis panels are review panels of scientific experts that are convened for special, one-time grant review events.
c
Percentiled applications are those that undergo discussion and receive overall impact scores. Lower percentiles represent better scores or a more favorable review.
Author Manuscript
d
The National Institutes of Health funds multiple types of grants including but not limited to research project grants known as the R-series grants (e.g., R01), career award grants known as the K-Series (e.g., K01), and training grants known as the T-or F-series (e.g., T-32). e
The average age of the applicants was 40.9 years (standard deviation = 0.17). One candidate did not provide gender.
f Twelve applications did not provide the race or ethnicity of the PI. g Rank indicates the relative position of institutions with respect to receipt of NIH funds. This was an NIH-internally generated variable. Rank 1 indicates institutions that receive enough NIH funds to fall into the top quintile; Rank 2 indicates those in the second quintile; Rank 3 indicates those that fall into the third quintile; Rank 4 indicates institutions that fall into the fourth quintile; and Rank 5 indicates institutions that fall into the bottom quintile in terms of NIH funds received.
Author Manuscript Author Manuscript Acad Med. Author manuscript; available in PMC 2017 April 01.
Boyington et al.
Page 15
Table 2
Author Manuscript
Mean review criterion scores for NHLBI Unfunded Early-Stage Investigator Applications, FY 2010–2012 Criterion
Mean scorea (SD)
Approach score
3.7 (0.04)
Significance score
3.6 (0.04)
Innovation score
3.6 (0.04)
Investigator score
4.9 (0.03)
Environment score
2.3 (0.03)
Abbreviations: NHLBI indicates National Heart, Lung, and Blood Institute; FY, fiscal year; SD, standard deviation. a
Scores can range from 1 = exceptional to 9 = Poor.
Author Manuscript Author Manuscript Author Manuscript Acad Med. Author manuscript; available in PMC 2017 April 01.
