�������

Science moves forward when scientists take risks in their work and explore new, untested ideas. For example, one of the hottest new ideas in cancer treatment involves using patients’ own immune cells to treat their cancer. New therapies based on this idea, such as CAR T-cell therapy, have rendered many types of previously untreatable leukemia treatable. The scientific work that led to this breakthrough happened in the face of decades of widespread skepticism that targeted immune therapy for cancer would be possible. The researchers who worked on these ideas when they were novel risked failure, but still pursued them.

Research into novel concepts like targeted cancer therapy might be called edge science, in contrast with studies focused on refining well-established ideas. Without broad and vigorous pursuit of edge science ideas, scientific progress stagnates. But there is an inherent problem in encouraging such pioneering work: Research based on unproven ideas is risky and prone to failure. That makes such research difficult to fund and to attract the critical mass of scientists needed to develop an idea.

The U.S. National Institutes of Health plays an important role in addressing this failure in the market for scientific research. With a $37 billion annual budget, NIH is the world’s largest funder of biomedical research. One of its explicit missions is to “foster fundamental creative discoveries (and) innovative research strategies” (National Institutes of Health, 2017). As a public institution, it can be thought of as “patient capital,” a funding source with a longtime horizon and an understanding that good ideas frequently lead down blind alleys. By that standard, NIH ought to be putting money into novel ideas that cannot get funding from private sources.

That at least is the theory. But a growing number of researchers have questioned whether NIH is doing all it should to support groundbreaking science (Alberts, Kirschner, Tilghman, and Varmus, 2011; Cook- Deegan, 1996; Kolata, 2009). For reasons of science and politics, NIH may be subject to a conservative bias. Because it spends public money, it is under pressure to produce visible results—and these can more readily be achieved with incremental advances on proven ideas than with scientific breakthroughs. Moreover, the senior scientists who serve on NIH grant application review panels may favor tried and true approaches they are familiar with and know are publishable instead of more speculative avenues of investigation (Joyner, Paneth, and Ioannidis, 2016). There is also evidence that NIH increasingly is funding scientists at later stages in their careers, when they are more likely to investigate well-established ideas.

Measuring NIH Funding of Edge Science

To address this question, we performed a quantitative analysis to measure the extent to which NIH funds novel ideas. Our study made use of an innovative method to determine the novelty of each article: a textual analysis of 24 million biomedical research articles in the MEDLINE database published between 1950 and 2017 with an American first author. An article was considered novel if the newest idea on which it built upon was relatively recent in the sense that the idea had first appeared in any biomedical research paper at most a few years prior. In determining the novelty of the ideas that each contribution built upon, our analysis also controlled for the idea type of each idea (e.g. gene, protein, drug, or diagnostic tool) and the research area of the journal in which the article was published (e.g. cardiology, neoplasms, or molecular biology). We categorized contributions that built on at least one relatively recent idea as novel, while we categorized contributions based on ideas with a longer history in the literature as more traditional science. Using this new measure of novelty, we then compared papers that acknowledged NIH funding against those funded by other sources.

Our key finding was that NIH funded edge science at a higher rate than it funded less-innovative science during the 67-year period we analyzed. But this positive result is subject to several important qualifications that raise questions about the extent of NIH’s commitment to edge science.

First, from 2010 to 2016, NIH disproportionately funded biomedical research based neither on the most recent ideas nor on the most longstanding ideas, but rather on those of intermediate vintage introduced into the literature between 1990 and 2005. Specifically, NIH funded research based on 10- to 25-year-old ideas at a 55 percent rate compared with a 45 percent funding rate for more recent or older ideas.

By contrast, from 1990 to 1999, NIH funded research based on new ideas at a higher rate than it funded research drawing on well-established ideas. This indicates that NIH has become less likely to support edge science over the past two decades.

Second, NIH’s propensity to support edge science varies across idea types. It is concentrated in a limited number of idea types in basic science where many ideas are relatively novel, such as genomics, proteomics, and the general field of subcellular biology. When it comes to research on clinical ideas as opposed to basic science, NIH has no preference for novelty. Moreover, once we account for NIH’s disproportionate funding of a few idea types that are hotbeds of innovation, NIH’s funding rates for edge science and traditional science are about the same. In other words, NIH disproportionately directs funds to innovative areas of basic science, and that explains its overall preference for innovation. If NIH funded basic science research and clinical research at the same rate, its preference for novelty would disappear.

These results are disheartening and consistent with a growing body of scholarship that finds NIH review panels becoming more conservative and risk-averse (Nicholson and Ioannidis, 2012; Li and Agha, 2015; Li, 2017). This is consistent with evidence that suggests biomedical researchers as a group have become more cautious in their research choices in recent years (Joyner, Paneth, and Ioannidis, 2016). One sign that NIH may be backing away from edge science is that its grant recipients are older than they used to be. In the 1980s, the median age of NIH grantees was in the thirties. Today, the median age is in the mid-forties. That is a problem because younger scientists are more likely to try out new ideas (Packalen and Bhattacharya, 2017).

Policy Reforms to Promote Edge Science

NIH recognizes the danger of underfunding high-risk ideas and has taken a number of steps to counter a creeping conservative bias and boost support for novel biomedical research. These policies include increasing the number of training awards, paying bonuses to young researchers, and developing methods for identifying high-risk ideas. More needs to be done. NIH should embed promotion of groundbreaking research more deeply in the grant-making process and in monitoring and evaluating the work it supports. Here are some policy reforms that could help NIH carry out its mission of supporting groundbreaking research:

• NIH should reform the review process and rethink how review panel members are selected. In particular, policies should be adopted that increase the number of accomplished younger scientists on these committees.
• NIH should change the way it measures success to increase tolerance of failure. NIH should adopt the approach that is standard in the world of venture capital that many failures are needed in order to innovate (Peifer, 2017; Zaringhalam, 2016).
• NIH should develop ways to directly measure the novelty of ideas. Currently, NIH grant proposals that reviewers deem innovative are given preference, but assessment of innovation is subjective. Review panels should use specific and concrete measures of innovation to adjudge proposal. Methods like the textual analysis described in the policy brief may be helpful.
• NIH should find additional ways to reward scientists working on novel ideas by taking steps to advance their careers. NIH can lend its prestige to these innovators when they apply for jobs and seek promotion. And it should find ways of evaluating grantee performance based on the novelty of the work. That implies downgrading the importance of the number of scholarly citations a research project generates, since it may take time for the importance of breakthroughs to be recognized.

Conclusion

If the world’s foremost supporter of biomedical research has indeed become less open to edge science, as our analysis indicates, it bodes poorly for science. While our finding is somewhat discouraging, it is not necessarily surprising. Some existing studies have found evidence that the scientific community as a whole has been less disposed to accept the newest ideas in the 2000s than it was during 1990s. If this is true, it is an ominous signal that progress in science—and in medicine in particular—is in danger of stagnating. Also from this perspective, we consider it essential that NIH and other funding agencies recommit to supporting work that tries out new ideas. While such work is risky, it is also essential for scientific progress as it helps the most fruitful new ideas develop from a germ of an idea to transformative discoveries. 

References

Alberts, B., Kirschner, M.W., Tilghman, S., and H. Varmus. (2011) Rescuing U.S. Biomedical Research from Its Systemic Flaws, Proceedings of the National Academy of Sciences, 111(16), 5773-5777.

Cook-Deegan, R.M. (1996) Does NIH need a DARPA? Issues in Science and Technology, 13(2), 25-28.

Joyner, M.J., Paneth, N., and J.P.A. Ioannidis. (2016) What Happens When Underperforming Big Ideas Become Entrenched? JAMA.

Kolata, G. (2009) Grant System Leads Cancer Researchers to Play it Safe, New York Times, June 28.

Li, D., and L. Agha. (2015) Big Names or Big Ideas? Do Peer-Review Panels Select the Best Science Proposals? Science, 348(6233), 343-348.

Li, D. (2017) Expertise versus Bias in Evaluation: Evidence from the NIH, American Economic Journal: Applied Economics, 9(2), 60-92.

National Institutes of Health (2017) .

Nicholson, J.M., and J.P.A. Ioannidis. (2012) Conform and Be Funded, Science 492, 34-36.

Packalen, M., and J. Bhattacharya. (2017) Age and the Trying Out of New Ideas, Journal of Human Capital (forthcoming).

Packalen, M., and J. Bhattacharya. (2018) Does the NIH Fund Edge Science? (NBER Working Paper Series No. 24860) Cambridge, Mass. .

Peifer, M. (2017) The Argument for Diversifying NIH Grant Portfolio, Molecular Biology of the Cell, 28, 2935-2940.

Zaringhalam, M. (2016) Failure in Science Is Frequent and Inevitable— and We Should Talk More about It, Scientific American Blog ( guest-blog/failure-in-science-is-frequent-and-inevitable-and-we-should-talk-more-about-it/, accessed October 2016).