Our Bibliography of Peer Review Literature

The peer review process is used by organizations around the world to identify good, promising science. However, peer review can be implemented in many ways, and surprisingly, dialogue among organizations managing peer review is limited. Although literature about peer review (and particularly peer review of grant applications) is scant, there have been some studies addressing review practices, challenges and guidelines.

Compiled below is a reference list of scientific articles concerning the processes, outcomes and validations of the scientific peer review process, particularly focusing on peer review of applications for funding. This list is sorted by topic area and represents a centralized body of knowledge intended as a reference for those conducting/participating in peer review and those who are interested in the outcomes of the process.

Assignment Load

Cook,WD, Golany B, Kress M, Penn M. Optimal Allocation of Proposals to Reviewers to Facilitate Effective Ranking. Manage Sci. 2005;51(4):655-61.

Li L, Wang Y, Liu G, Wang M, Wu X. Context-Aware Reviewer Assignment for Trust Enhanced Peer Review. PLoS One. 2015 Jun 19;10(6):e0130493.

Conflict of Interest & Bias

Abdoul H, Perrey C, Tubach F, Amiel P, Durand-Zaleski I, Alberti C. Non-financial Conflicts of Interest in Academic Grant Evaluation: A Qualitative Study of Multiple Stakeholders in France. PLoS One. 2012;7(4):e35247.

Bekelman JE, Li Y, Gross CP. Scope and Impact of Financial Conflicts of Interest in Biomedical Research: A Systematic Review. JAMA. 2003 Jan 22-29;289(4):454-65.

Benos D. Ethics: Detecting Misconduct. Nature 2006.

Bhattacharjee Y. Science Funding. NSF’s ‘Big Pitch’ Tests Anonymized Grant Reviews. Science. 2012 May 25;336(6084):969-70.

Bornmann L. Research Misconduct—Definitions, Manifestations and Extent. Publications. 2013;1(3):87-98.

De Martino B, Kumaran D, Seymour B, Dolan RJ. Frames, Biases, and Rational Decision-Making in the Human Brain. Science. 2006 Aug 4;313(5787):684-7.

Drazen JM, De Leeuw PW, Laine C, Mulrow C, Deangelis CD, Frizelle FA, Godlee F, Haug C, Hébert PC, James A, Kotzin S, Marusic A, Reyes H, Rosenberg J, Sahni P, Van Der Weyden MB, Zhaori G. Toward More Uniform Conflict Disclosures: The Updated ICMJE Conflict of Interest Reporting Form. Natl Med J India. 2010 Jul-Aug;23(4):196-7.

Kotchen TA, Lindquist T, Miller Sostek A, Hoffmann R, Malik K, Stanfield B. Outcomes of National Institutes of Health Peer Review of Clinical Grant Applications. J Investig Med. 2006 Jan;54(1):13-9.

Langfeldt L. Decision-making in Expert Panels Evaluating Research: Constraints, Processes and Bias. 2001;dissertation: The University of Oslo.

Lee C. Sugimoto CR, Zhang G, Cronin B. Bias in Peer Review. J Am Soc Inf Sci Technol. 2013;64(1):2-17.

Li D. Expertise vs. Bias in Evaluation: Evidence from the NIH. 2013;Northwestern University.

Li L, Wang Y, Liu G, Wang M, Wu X. Context-Aware Reviewer Assignment for Trust Enhanced Peer Review. PLoS One. 2015 Jun 19;10(6):e0130493.

Lo B, Field MJ, Editors. Conflict of Interest in Medical Research, Education, and Practice. Committee on Conflict of Interest in Medical Research, Education, and Practice. 2009;Institute of Medicine

Oleinik A. Conflict(s) of Interest in Peer Review: Its Origins and Possible Solutions. Sci Eng Ethics. 2014 Mar;20(1):55-75.

Pham-Kanter G. Revisiting Financial Conflicts of Interest in FDA Advisory Committees. Milbank Q. 2014 Sep;92(3):446-70.

Rockey SJ, Collins FS. Managing Financial Conflict of Interest in Biomedical Research. JAMA. 2010 Jun 16;303(23):2400-2.

Wager E. Ethics: What is it for? Nature 2006.

Wood SF, Mador JK. Science and Regulation. Uncapping Conflict of Interest? Science. 2013 Jun 7;340(6137):1172-3.

Critique & Text Analysis

Bornmann L, Wolf M, Daniel HD. Closed Versus Open Reviewing of Journal Manuscripts: How Far do Comments Differ in Language Use? Scientometrics. 2012;91(3):843-56.

Fuller EO, Hasselmeyer EG, Hunter JC, Abdellah FG, Hinshaw AS. Summary Statements of the NIH Nursing Research Grant Applications. Nurs Res. 1991 Nov-Dec;40(6):346-51.

Tausczik YR, Pennebaker JW. The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods. J Lang Soc Psychol. 2010;29(1):24-54.

Decision Making, Emotion, Risk

De Martino B, Kumaran D, Seymour B, Dolan RJ. Frames, Biases, and Rational Decision-Making in the Human Brain. Science. 2006 Aug 4;313(5787):684-7.

Denes-Raj V, Epstein S. Conflict Between Intuitive and Rational Processing: When People Behave Against Their Better Judgment. J Pers Soc Psychol. 1994 May;66(5):819-29.

Finucane ML, Alhakami A, Slovic P, Johnson SM. The Affect Heuristic in Judgments of Risks and Benefits. J Behav Decis Mak. 2000;13(1):1-17.

Stephen Gallo, Lisa Thompson, Karen Schmaling, Scott Glisson. Risk evaluation in peer review of grant applications. Environment Systems and Decisions. 24 February, 2018.

Gregory R, Mendelsohn R. Perceived Risk, Dread, and Benefits. Risk Anal. 1993;13(3):259-64.

Lipshitz R, Strauss O. Coping with Uncertainty: A Naturalistic Decision-Making Analysis. Organ Behav Hum Decis Process. 1997;69(2):149-63.

Pachur T, Hertwig R, Steinmann F. How do People Judge Risks: Availability Heuristic, Affect Heuristic, or Both? J Exp Psychol Appl. 2012 Sep;18(3):314-30.

Rand DG, Epstein ZG. Risking Your Life Without a Second Thought: Intuitive Decision-Making and Extreme Altruism. PLoS One. 2014 Oct 15;9(10):e109687.

Sjöberg L. Factors in Risk Perception. Risk Anal. 2000;20(1):1-12.

Slovic P, Finucane ML, Peters E, MacGregor DG. Risk as Analysis and Risk as Feelings: Some Thoughts About Affect, Reason, Risk, and Rationality. Risk Anal. 2004 Apr;24(2):311-22.

Slovic P, Peters E. Risk Perception and Affect. Curr Dir Psychol Sci. 2006;15(6):322-5.

Gender

Bornmann L, Mutz R, Daniel HD. Gender Differences in Grant Peer Review: A Meta-Analysis. J Informetr. 2007 Jul;1(3):226-38.

Homma MK, Motohashi R, Ohtsubo H. Japan’s Lagging Gender Equality. Science. 2013 Apr 26;340(6131):428-30.

van Arensbergen P, van der Weijden I, van den Besselaar P. Academic Talent Selection in Grant Review Panels]. In book: (Re)Searching Academic Careers, Chapter: 2. 2014. Publisher: Russian Academy of Sciences.

General Peer Review

Abdoul H, Perrey C, Amiel P, Tubach F, Gottot S, Durand-Zaleski I, Alberti C. Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer Practices. PLoS One. 2012;7(9):e46054.

Alberts B, Kirschner MW, Tilghman S, Varmus H. Rescuing US Biomedical Research from its Systemic Flaws. Proc Natl Acad Sci U S A. 2014 Apr 22;111(16):5773-7.

Bandeh-Ahmadi A. Editorial Decisions and Measuring Information Contents of Reviews. Available at SSRN. 2013.

Benos DJ, Bashari E, Chaves JM, Gaggar A, Kapoor N, LaFrance M, Mans R, Mayhew D, McGowan S, Polter A, Qadri Y, Sarfare S, Schultz K, Splittgerber R, Stephenson J, Tower C, Walton RG, Zotov A. The Ups and Downs of Peer Review. Adv Physiol Educ. 2007 Jun;31(2):145-52.

Bhattacharya A. Science Funding: Duel to the Death. Nature. 2012 Aug 2;488(7409):20-2.

Bornmann L. Scientific Peer Review: An Analysis of the Peer Review Process from the Perspective of Sociology of Science Theories. Human Architecture: Journal of the Sociology of Self-Knowledge. 2008;VI(2):23-38.

Cantor M, Gero S. The Missing Metric: Quantifying Contributions of Reviewers. R Soc Open Sci. 2015 Feb 11;2(2):140540.

Chang WR, McLean IP. CUSUM: A Tool for Early Feedback About Performance? BMC Med Res Methodol. 2006 Mar 2;6:8.

Cole S, Rubin L, Cole J. Peer Review in the National Science Foundation: Phase One of a Study. Washington, DC: National Academy of Sciences, 1978.

Council of Canadian Academies. Informing Research Choices: Indicators and Judgment. 2012.

Demicheli V, Di Pietrantonj C. Peer Review for Improving the Quality of Grant Applications. Cochrane Database Syst Rev. 2007 Apr 18;(2):MR000003.

DiSilvestro RL. The First Half Century: A History of AIBS. BioScience. 1997 Nov;47(10):643-9.

Dumanis SB, Ullrich L, Washington PM, Forcelli PA. It’s Money! Real-World Grant Experience Through a Student-Run, Peer-Reviewed Program. CBE Life Sci Educ. 2013 Fall;12(3):419-28.

Fleurence RL, Forsythe LP, Lauer M, Rotter J, Ioannidis JP, Beal A, Frank L, Selby JV. Engaging Patients and Stakeholders in Research Proposal Review: The Patient-Centered Outcomes Research Institute. Ann Intern Med. 2014 Jul 15;161(2):122-30.

Fliesler SJ. Rethinking Grant Peer Review. Science. 1997 Mar 7;275(5305):1399.

Gallo SA et al. The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies. 2014 PLoS ONE 9(9): e106474. doi:10.1371/journal.pone.0106474

Gillespie GW, Jr., Chubin DE, Kurzon GM. Experience with NIH Peer Review: Researchers’ Cynicism and Desire for Change. Sci Technol Human Values. 1985;10(3):45-54.

Glass B, Krauss RW. AIBS Presidents Revisit the Past. BioScience. 1997 Nov;47(10):650-6.

Glenn JF. Uses and Needs for Peer Review in Army Medical Research. Technol Innov. 2010;12:241-7.

Greaves S, Scott J, Clarke M, Miller M, Hannay T, Thomas A, Campbell P. Overview: Nature’s Peer Review Trial. Nature. 2006.

Guthrie S, Guerin B, Wu H, Ismail S, Wooding S. Alternatives to Peer Review in Research Project Funding. Santa Monica, CA: RAND Corporation, 2013.

Hand E. Extra Scrutiny for ‘Grandee Grantees’. Nature. 2012 Feb 20;482(7386):450-1.

Hartmann I, Neidhardt F. Peer Review at the Deutsche Deutsche Forschungsgemeinschaft. Scientometrics. 1990 Nov 01;19(5-6):419-25.

Hartonen T, Alava MJ. How Important Tasks Are Performed: Peer Review. Sci Rep. 2013;3:1679.

Herbert DL, Barnett AG, Clarke P, Graves N. On the Time Spent Preparing Grant Proposals: An Observational Study of Australian Researchers. BMJ Open. 2013 May 28;3(5). pii: e002800.

Herbert DL, Barnett AG, Graves N. Funding: Australia’s Grant System Wastes time. Nature. 2013 Mar 21;495(7441):314.

Herbert DL, Coveney J, Clarke P, Graves N, Barnett AG. The Impact of Funding Deadlines on Personal Workloads, Stress and Family Relationships: A Qualitative Study of Australian Researchers. BMJ Open. 2014 Mar 28;4(3):e004462.

Herrmann-Lingen C, Brunner E, Hildenbrand S, Loew TH, Raupach T, Spies C, Treede RD, Vahl CF, Wenz HJ. Evaluation of Medical Research Performance—Position Paper of the Association of the Scientific Medical Societies in Germany (AWMF). Ger Med Sci. 2014 Jun 26;12:Doc11.

Hickam D, Totten A, Berg A, Rader K, Goodman S, Newhouse R, Editors. The PCORI Methodology Report. 2013.

Ho RC, Mak KK, Tao R, Lu Y, Day JR, Pan F. Views on the Peer Review System of Biomedical Journals: An Online Survey of Academics from High-Ranking Universities. BMC Med Res Methodol. 2013 Jun 7;13:74.

IOM. Strategies for managing the Breast Cancer Research Program: A Report to the U.S. Army Medical Research and Development Command. Washington, DC: National Academy Press, 1993.

Irwin D, Gallo SA, Glisson SR. Opinion: Learning from Peer Review. The Scientist. 2013 May 24.

Ismail S, Farrands A, Wooding S. Evaluating Grant Peer Review in the Health Sciences: A review of the Literature. Santa Monica, CA: RAND Corporation, 2009.

Jefferson T, Wager E, Davidoff F. Measuring the Quality of Editorial Peer Review. JAMA. 2002 Jun 5;287(21):2786-90.

Johnson VE. Statistical Analysis of the National Institutes of Health Peer Review System. Proc Natl Acad Sci U S A. 2008 Aug 12;105(32):11076-80.

King JC, Lawrence TS, Murphy SB, Davidson NE, Mayer RJ. The American Society of Clinical Oncology Cancer Foundation Grants Program: A 25-Year Report and a Look Toward the Future. J Clin Oncol. 2010 Mar 20;28(9):1616-21.

Koonin E, Landweber L, Lipman D, Dignon R. Systems: Reviving a Culture of Scientific Debate. Nature. 2006.

Kupfer DJ, Murphree AN, Pilkonis PA, Cameron JL, Giang RT, Dodds NE, Godard KA, Lewis DA. Using Peer Review to Improve Research and Promote Collaboration. Acad Psychiatry. 2014 Feb;38(1):5-10.

Lahiri D. Perspective: The Case for Group Review. Nature 2006.

Lee K, Bero L. Ethics: Increasing Accountability. Nature 2006.

Lee K, Brownstein JS, Mills RG, Kohane IS. Does Collocation Inform the Impact of Collaboration? PLoS One. 2010 Dec 15;5(12):e14279.

Lyman RL. A Three-Decade History of the Duration of Peer Review. Journal of Scholarly Publishing. 2013 Apr;44(3):211-20.

McGarity TO. Peer Review in Awarding Federal Grants in the Arts and Sciences. Berkeley Tech LJ. 1994;9(1).

McGeary M, Hanna KE. Strategies to Leverage Research Funding: Guiding DOD’s Peer Reviewed Medical Research Programs. Washington, DC: National Academies Press, 2004.

Mervis J. Peer Review. Beyond the Data. Science. 2011 Oct 14;334(6053):169-71.

Mietchen D. The Transformative Nature of Transparency in Research Funding. PLoS Biol. 2014 Dec 30;12(12):e1002027.

Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. Int J Surg. 2010;8(5):336-41.

Nicholson J, Ioannidis J. Research Grants: Conform and be Funded. Nature. 2012 Dec 6;492:34-36.

Pawelczyk JA, Strawbridge LM, Schultz AM, Liverman CT. A Review of NASA Human Research Program’s Scientific Merit Processes: Letter Report. Washington, DC: National Academies Press, 2012.

Pederson T. The “Study” Role of Past National Institutes of Health Study Sections. Mol Biol Cell. 2012 Sep;23(17):3281-4.

Parliamentary Office of Science and Technology. Peer Review. 2002;182.

Ranalli B. A Prehistory of Peer Review: Religious Blueprints from the Hartlib Circle. Spontaneous Generations. 2011;5(1):12-18.

Riley B. Systems: Trusting Data’s Quality. Nature. 2006.

Roebber PJ, Schultz DM. Peer Review, Program Officers and Science Funding. PLoS One. 2011 Apr 12;6(4):e18680.

Russell AS, Thorn BD, Grace M. Peer Review: A Simplified Approach. J Rheumatol. 1983 Jun;10(3):479-81.

Sandewall E. Systems: Opening up the Process. Nature. 2006.

Sandstrom U, Hallsten M. Persistent Nepotism in Peer-Review. Scientometrics. 2008 Feb;74(2):175-89.

Smith MA, Kaufman NJ, Dearlove AJ. External Community Review Committee: A New Strategy for Engaging Community Stakeholders in Research Funding Decisions. Prog Community Health Partnersh. 2013 Fall;7(3):301-12.

Smith R. Classical Peer Review: An Empty Gun. Breast Cancer Res. 2010 Dec 20;12 Suppl 4:S13.

Smith R. Peer Review: A Flawed Process at the Heart of Science and Journals. J R Soc Med. 2006 Apr;99(4):178-82.

Spier R. The History of the Peer-Review Process. Trends Biotechnol. 2002 Aug;20(8):357-8.

Squazzoni F, Gandelli, C. Peer Review Under the Microscope. An Agent-Based Model of Scientific Collaboration. Simulation Conference (WSC), Proceedings of the 2012 Winter. Dec 2012:1-12.

Squazzoni F, Takacs, K. Social Simulation that “Peers into Peer Review.” J Artif Soc Soc Simulat. 2011 Oct 31;14(4):3.

Stahel PF, Moore EE. Peer Review for Biomedical Publications: We Can Improve the System. BMC Med. 2014 Sep 26;12(1):179.

Turner RS. Best Practices in Peer Review Assure Quality, Value, Objectivity. Journal of the National Grants Management Association. 2009;17(1):43-8.

United States General Accounting Office. Federal Research: Peer Review Practices at Federal Science Agencies Vary. Report to Congressional Requesters. 2009 March.

Walker R, Rocha da Silva P. Emerging Trends in Peer Review-a Survey. Front Neurosci. 2015 May 27;9:169.

Wessely S. Peer Review of Grant Applications: What do we Know? Lancet. 1998 Jul 25;352(9124):301-5.

Zare RN, Winnacker EL. China’s Science Funding. Science. 2011 Oct 28;334(6055):433.

van Arensbergen P, van der Weijden I, van den Besselaar P. Academic Talent Selection in Grant Review Panels]. In book: (Re)Searching Academic Careers, Chapter: 2. 2014. Publisher: Russian Academy of Sciences.

General Science

American Society for Biochemistry and Molecular Biology. Towards a Sustainable Biomedical Research Enterprise. 2014.

Boyack KW, Klavans R, Borner K. Mapping the Backbone of Science. Scientometrics. 2005;64(3):351-74.

Cohen LR, Noll RG. The Future of the National Laboratories. PNAS. 1996;93(23):12678-85.

Couzin-Frankel J. Chasing the Money. Science. 2014 Apr 4;344(6179):24-5.

Davis, G. Doctors Without Orders. American Scientist. 2005;93(3, supplement).

Furmana JL, Porter ME, Stern S. The Determinants of National Innovative Capacity. Res Policy. 2002;31(6):899-933.

Garrison HH, Deschamps AM. Physician Scientists: Assessing the Workforce. FASEB. 2013.

Ioannidis JP. How to Make More Published Research True. PLoS Med. 2014 Oct 21;11(10):e1001747.

Lopes GR, da Silva R, Moro MM, Palazzo Moreira de Oliveira J. Scientific Collaboration in Research Networks: A Quantification Method by Using Gini Coefficient. IJCSA. 2012;9(2):15-31.

National Institute of Allergy and Infectious Diseases. FY 2014 Budget.

National Institute of Allergy and Infectious Diseases. The Edge of Discovery: A Portrait of the National Institute of Allergy and Infectious Diseases. June 2009;NIH Publication No. 09-6389.

Readers’ Poll Results: Funding Environment. Science. 2013 Apr 26;340(6131):429.

Rice JA, Wu CO. Nonparametric Mixed Effects Models for Unequally Sampled Noisy Curves. Biometrics. 2001 Mar;57(1):253-9.

Sullivan Faith K. Patterns of Creation and Discovery: An Analysis of Defense Laboratory Patenting and Innovation. 2013;dissertation:Pardee Rand Graduate School.

Taleb NN, Douady R. Mathematical Definition, Mapping, and Detection of (Anti)Fragility. Quant Finance. 2013;13(11):1677-89.

Van den Besselaar P, Leydesdorf L. Past Performance as Predictor of Successful Grant Applications: A Case Study. Den Haag, Rathenau Instituut SciSA rapport 0704.

H-Index, Citations, & Bibliometrics

Auranen O, Nieminen M. University Research Funding and Publication Performance—An International Comparison. Res Policy. 2010;39(6):822-34.

Belter CW. Can Bibliometric Indicators Predict Institutional Citation Patterns? Presented at ACRL 2013, 10-13 April, Indianapolis IN.

Bornmann L, Leydesdorff L. On the Meaningful and Non-Meaningful Use of Reference Sets in Bibliometrics. J Informetr. 2014;8(1):273-5.

Boyack KW, Jordan P. Metrics Associated with NIH Funding: A High-Level View. J Am Med Inform Assoc. 2011 Jul-Aug;18(4):423-31.

Cantor M, Gero S. The Missing Metric: Quantifying Contributions of Reviewers. R Soc Open Sci. 2015 Feb 11;2(2):140540.

Doyle JM, Quinn K, Bodenstein YA, Wu CO, Danthi N, Lauer MS. Association of Percentile Ranking with Citation Impact and Productivity in a Large Cohort of de novo NIMH-funded R01 Grants. Mol Psychiatry. 2015 Jun 2.

Farhadi H, Salehi H, Md Yunus M, Chadegani AA, Farhadi M, Fooladi M, Ebrahim NA. Does it Matter Which Citation Tool is Used to Compare the H-Index of a Group of Highly Cited Researchers? Aust J Basic Appl Sci. 2013;7(4):198-202.

Garfield E. Current Comments: Lifetime Citations Rates. Essays of an Information Scientist. 1980 Jan;4(2):355-8.

Gauffriau M, Larsen PO, Maye I, Roulin-Perriard A, von Ins M. Publication, Cooperation and Productivity Measures in Scientific Research. Scientometrics. 2007;73(2).175-214.

Hirsch JE. An Index to Quantify an Individual’s Scientific Research Output. Proc Natl Acad Sci U S A. 2005 Nov 15;102(46):16569-72.

Hirsch JE. Does the H Index Have Predictive Power? Proc Natl Acad Sci U S A. 2007 Dec 4;104(49):19193-8.

Kaltman JR, Evans FJ, Danthi NS, Wu CO, DiMichele DM, Lauer MS. Prior Publication Productivity, Grant Percentile Ranking, and Topic-Normalized Citation Impact of NHLBI Cardiovascular R01 Grants. Circ Res. 2014 Sep 12;115(7):617-24.

Khan NR, Thompson CJ, Taylor DR, Gabrick KS, Choudhri AF, Boop FR, Klimo P Jr. Part II: Should the H-Index be Modified? An Analysis of the M-Quotient, Contemporary H-Index, Authorship Value, and Impact Factor. World Neurosurg. 2013 Dec;80(6):766-74.

Kinney AL. National Scientific Facilities and Their Science Impact on Nonbiomedical Research. Proc Natl Acad Sci U S A. 2007 Nov 13;104(46):17943-7.

Larivière V, Macaluso B, Archambault E, Gingras Y. Which Scientific Elite? On the Concentration of Funding, Productivity and Citations. In Proceedings of the 12th International Conference of the International Society for Scientometrics and Informetrics. 2009.

Lazaridis T. Ranking University Departments Using the Mean H-Index. Scientometrics. 2010 Feb;82(2):211-16.

Marx W, Schier H, Andersen OK. Using Time Dependent Citation Rates (Sales Curves) for Comparing Scientific Impacts. 2006. arXiv:physics/0611284 [physics.soc-ph].

Molinari JF, Molinari A. A New Methodology for Ranking Scientific Institutions. Scientometrics. 2008 Apr;75(1):163-74.

Nieminen P, Carpenter J, Rucker G, Schumacher M. The Relationship Between Quality of Research and Citation Frequency. BMC Med Res Methodol. 2006 Sep 1;6:42.

Sidiropoulos A, Katsaros D, Manolopoulos Y. Generalized Hirsch H-Index for Disclosing Latent Facts in Citation Networks. Scientometrics. 2007 Aug 1;72(2):253-280.

Adrian G Barnett, Scott R. Glisson, Stephen Gallo. Do funding applications where peer reviewers disagree have higher citations? A cross-sectional study. f1000research.com. Retrieved from URL.

Sypsa V, Hatzakis A. Assessing the Impact of Biomedical Research in Academic Institutions of Disparate Sizes. BMC Med Res Methodol. 2009 May 29;9:33.

Waltman L, van Eck NJ, van Leeuwen TN, Visser MS, van Raan AF. On the Correlation Between Bibliometric Indicators and Peer Review: Reply to Opthof and Leydesdorff. Scientometrics. 2011 Sep;88(3):1017-1022.

Innovation

Horrobin DF. The Philosophical Basis of Peer Review and the Suppression of Innovation. JAMA. 1990 Mar 9;263(10):1438-41.

Kaime EM, Moore KH, Goldberg SF. CDMRP: Fostering Innovation Through Peer Review. Technol Innov. 2010;12:233-40.

Luukkonen T. Conservatism and Risk-Taking in Peer Review: Emerging ERC Practices. Research Evaluation. 2012;21:48-60.

Nicholson J, Ioannidis J. Research Grants: Conform and be Funded. Nature. 2012 Dec 6;492:34-36.

Inter-rater Reliability

Colliver J. Commentary on Cichetti’s “Reliability of Peer Review.” Teach Learn Med. 2002;14(3):142-3.

Fleiss JL, Shrout PE. The Effects of Measurement Errors on Some Multivariate Procedures. Am J Public Health. 1977 December; 67(12): 1188-91.

Fogelholm M, Leppinen S, Auvinen A, Raitanen J, Nuutinen A, Väänänen K. Panel Discussion Does Not Improve Reliability of Peer Review for Medical Research Grant Proposals. J Clin Epidemiol. 2012 Jan;65(1):47-52.

Giraudeau B, Leyrat C, Le Gouge A, Léger J, Caille A. Peer Review of Grant Applications: A Simple Method to Identify Proposals with Discordant Reviews. PLoS One. 2011;6(11):e27557.

Hallgren KA. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutor Quant Methods Psychol. 2012;8(1):23-34.

Hargens LL, Herting JR. Neglected Considerations in the Analysis of Agreement Among Journal Referees. Scientometrics. 1990 Jul 01;19(1-2):91-106.

Johnson VE. Statistical Analysis of the National Institutes of Health Peer Review System. Proc Natl Acad Sci U S A. 2008 Aug 12;105(32):11076-80.

Kravitz RL, Franks P, Feldman MD, Gerrity M, Byrne C, Tierney WM. Editorial Peer Reviewers’ Recommendations at a General Medical Journal: Are They Reliable and do Editors Care? PLoS One. 2010 Apr 8;5(4):e10072.

Sattler DN, McKnight PE, Naney L, Mathis R. Grant Peer Review: Improving Inter-Rater Reliability with Training. PLoS One. 2015 Jun 15;10(6):e0130450.

NIH & NSF

Berg J. NIH-Wide Correlations Between Overall Impact Scores and Criterion Scores. NIGMS Feedback Loop Blog. 2010 Sep 30.

Berg J. Scoring Analysis with Funding and Investigator Status. NIGMS Feedback Loop Blog. 2010 Sep 14.

Collins FS, Tabak LA. Policy: NIH Plans to Enhance Reproducibility. Nature. 2014 Jan 30;505(7485):612-3.

Gillespie GW, Jr., Chubin DE, Kurzon GM. Experience with NIH Peer Review: Researchers’ Cynicism and Desire for Change. Sci Technol Human Values. 1985;10(3):45-54.

Kaiser J. Biomedical Research. A Call for NIH Youth Movement. Science. 2014 Oct 10;346(6206):150-1.

Moore RF, Catevenis K, Pearson K, Wagner RM. AIRI Statistics: Trends in NIH Extramural Funding. Presented at the Association of Independent Research Institutes (AIRI) 51st Annual Meeting. 2012, 3 Oct, Philadelphia, PA.

NIH. NIH Success Rate Definition. 2009 Feb.

NIH. NIH Success Rate Definition. 2012 Feb.

NIH. Research Project Grant (RPG) Critique Template. R01/R03/R15/R21/R34 Review. 2012 Mar 5.

NIH. NIH Reviewer Orientation. 2012 Dec 14.

NIH. Scoring and Review Changes. 2009 Oct 22.

NIH. Scoring System and Procedure. 2009 Mar 20.

NIH. Chair Orientation. 2012 Mar 5.

NIH. Enhancing Peer Review: Implementation of Recommended Actions. Transcript accompanying video of Dr. Alan Willard. 2009 Feb.

Open Peer Review

Bornmann L, Wolf M, Daniel HD. Closed Versus Open Reviewing of Journal Manuscripts: How Far do Comments Differ in Language Use? Scientometrics. 2012;91(3):843-56.

Hettyey A, Griggio M, Mann M, Raveh S, Schaedelin FC, Thonhauser KE, Thoss M, van Dongen WF, White J, Zala SM, Penn DJ. Peerage of Science: Will it Work? Trends Ecol Evol. 2012 Apr;27(4):189-90.

Koop T, Pöschl U. Systems: An Open, Two-Stage Peer-Review Journal. Nature. 2006.

Mietchen D. The Transformative Nature of Transparency in Research Funding. PLoS Biol. 2014 Dec 30;12(12):e1002027.

Moylan EC, Harold S, O’Neill C, Kowalczuk MK. Open, Single-Blind, Double-Blind: Which Peer Review Process do You Prefer? BMC Pharmacol Toxicol. 2014 Sep 30;15:55.

Walker R, Rocha da Silva P. Emerging Trends in Peer Review-a Survey. Front Neurosci. 2015 May 27;9:169.

Panel Discussion

Cole S, Cole JR, Simon GA. Chance and Consensus in Peer Review. Science. 1981 Nov 20;214(4523):881-6.

Fogelholm M, Leppinen S, Auvinen A, Raitanen J, Nuutinen A, Väänänen K. Panel Discussion Does Not Improve Reliability of Peer Review for Medical Research Grant Proposals. J Clin Epidemiol. 2012 Jan;65(1):47-52.

Johnson VE. Statistical Analysis of the National Institutes of Health Peer Review System. Proc Natl Acad Sci U S A. 2008 Aug 12;105(32):11076-80.

Obrecht M, Tibelius K, D’Aloisio G. Examining the Value Added by Committee Discussion in the Review of Applications for Research Awards. Research Evaluation. 2007 Jun;16(2):79-91.

Peer Review Technology

Anderson C. Technical Solutions: Wisdom of the Crowds. Nature 2006.

Doran MR, Lott WB, Doran SE. Communication: Use Multimedia in Grant Applications. Nature. 2014 Jan 16;505(7483):291.

Driskell JE, Radtke PH, Salas E. Virtual Teams: Effects of Technological Mediation on Team Performance. Group Dyn. 2003 Dec;7(4):297-323.

Jenkins SL, Iyengar R, Diverse-Pierluissi MA, Chan AM, Devi LA, Sobie EA, Ting AT, Weinstein DC. Teaching Resources. Using Web-Based Discussion Forums as a Model of the Peer-Review Process and a Tool for Assessment. Sci Signal. 2008;1(9):tr2.

Vo, NM, Quiggle, GM, Wadhwani, K. Comparative outcomes of face-to-face and virtual review meetings. International Journal of Surgery Open. 2016. Volume 4 , 38 - 41

Vo, NM, Trocki, R. Virtual and Peer Reviews of Grant Applications at the Agency for Healthcare Research and Quality. Southern Medical Journal. 2015. 108. 622-626.

Resubmission

Bornmann L, Mutz R, Daniel H-R. Latent Markov Modeling Applied to Grant Peer Review. J Informetr. 2008;2(4):217-28.

Reviewer/PI Demographics

Jones BF, Weinberg BA. Age Dynamics in Scientific Creativity. Proc Natl Acad Sci U S A. 2011 Nov 22;108(47):18910-4.

Kaplan D, Lacetera N, Kaplan C. Sample Size and Precision in NIH Peer Review. PLoS One. 2008 Jul 23;3(7):e2761.

Scoring Scale

Cicchetti DV, Shoinralter D, Tyrer PJ. The Effect of Number of Rating Scale Categories on Levels of Interrater Reliability: A Monte Carlo Investigation. Appl Psychol Meas. 1985 Mar;9(1):31-6.

Green JG, Calhoun F, Nierzwicki L, Brackett J, Meier P. Rating Intervals: An Experiment in Peer Review. FASEB J. 1989 Jun;3(8):1987-92.

Team Performance

Barabási AL. Sociology. Network Theory—the Emergence of the Creative Enterprise. Science. 2005 Apr 29;308(5722):639-41.

Borgatti SP, Mehra A, Brass DJ, Labianca G. Network Analysis in the Social Sciences. Science. 2009 Feb 13;323(5916):892-95.

Cur?eu PL, Jansen RJ, Chappin MM. Decision Rules and Group Rationality: Cognitive Gain or Standstill? PLoS One. 2013;8(2):e56454.

Driskell JE, Radtke PH, Salas E. Virtual Teams: Effects of Technological Mediation on Team Performance. Group Dyn. 2003 Dec;7(4):297-323.

Duch J, Waitzman JS, Amaral LA. Quantifying the Performance of Individual Players in a Team Activity. PLoS One. 2010 Jun 16;5(6):e10937.

Evans JA, Foster JG. Metaknowledge. Science. 2011 Feb 11;331(6018):721-5.

Guimerà R, Uzzi B, Spiro J, Amaral LA. Team Assembly Mechanisms Determine Collaboration Network Structure and Team Performance. Science. 2005 Apr 29;308(5722):697-702.

Jeffcott SA, Mackenzie CF. Measuring Team Performance in Healthcare: Review of Research and Implications for Patient Safety. J Crit Care. 2008 Jun;23(2):188-96.

Langfeldt L. Decision-Making in Expert Panels Evaluating Research: Constraints, Processes and Bias. 2001;dissertation: The University of Oslo.

Manthous CA, Hollingshead AB. Team Science and Critical Care. Am J Respir Crit Care Med. 2011 Jul 1;184(1):17-25.

Roebber PJ, Schultz DM. Peer Review, Program Officers and Science Funding. PLoS One. 2011 Apr 12;6(4):e18680.

Rogelberg SG, O’Connor MS, Sederburg M. Using the Stepladder Technique to Facilitate the Performance of Audioconferencing Groups. J Appl Psychol. 2002 Oct;87(5):994-1000.

Rosen MA, Weaver SJ, Lazzara EH, Salas E, Wu T, Silvestri S, Schiebel N, Almeida S, King HB. Tools for Evaluating Team Performance in Simulation-Based Training. J Emerg Trauma Shock. 2010 Oct;3(4):353-9.

Stokols D, Misra S, Moser RP, Hall KL, Taylor BK. The Ecology of Team Science: Understanding Contextual Influences on Transdisciplinary Collaboration. Am J Prev Med. 2008 Aug;35(2 Suppl):S96-115.

Wickens CD, Holland JG, Parasuraman R, Banbury S. Engineering Psychology and Human Performance (4th ed). Upper Saddle River, NJ: Prentice Hall, 2012.

Training

Garfield JM, Kaye AD, Kolinsky DC, Urman RD. A Systematic Guide for Peer Reviewers for a Medical Journal. J Med Pract Manage. 2015 Mar-Apr;30(6 Spec No):13-7.

Rosen MA, Weaver SJ, Lazzara EH, Salas E, Wu T, Silvestri S, Schiebel N, Almeida S, King HB. Tools for Evaluating Team Performance in Simulation-Based Training. J Emerg Trauma Shock. 2010 Oct;3(4):353-9.

Sattler DN, McKnight PE, Naney L, Mathis R. Grant Peer Review: Improving Inter-Rater Reliability with Training. PLoS One. 2015 Jun 15;10(6):e0130450.

Validation & Impact

Armstrong PW, Caverson MM, Adams L, Taylor M, Olley PM. Evaluation of the Heart and Stroke Foundation of Canada Research Scholarship Program: Research Productivity and Impact. Can J Cardiol. 1997 May;13(5):507-16.

Beacham L, Li E, Wasserman A. Measuring the Performance of Extramural Funding at the National Library of Medicine. The College of William and Mary. 2008.

Berg J. Productivity Metrics and Peer Review Scores. NIGMS Feedback Loop Blog. 2011 Jun 2.

Burroughs Wellcome Fund. Biomedical Sciences: Program Outcomes. 2010 Report.

Cabezas-Clavijo A, Robinson-García N, Escabias M, Jiménez-Contreras E. Reviewers’ Ratings and Bibliometric Indicators: Hand in Hand When Assessing Over Research Proposals? PLoS One. 2013 Jun 28;8(6):e68258.

Claveria LE, Guallar E, Cami J, Conde J, Pastor R, Ricoy JR, Rodriguez-Farre E, Ruiz-Palomo F, Munoz E. Does Peer Review Predict the Performance of Research Projects in Health Sciences? Scientometrics. 2000 Jan 1;47(1):11-23.

Cole S, Cole JR, Simon GA. Chance and Consensus in Peer Review. Science. 1981 Nov 20;214(4523):881-6.

Crespia GA, Geunab A. An Empirical Study of Scientific Production: A Cross Country Analysis, 1981-2002. Res Policy. 2008;37(4):565-79.

Doyle JM, Quinn K, Bodenstein YA, Wu CO, Danthi N, Lauer MS. Association of Percentile Ranking with Citation Impact and Productivity in a Large Cohort of de novo NIMH-funded R01 Grants. Mol Psychiatry. 2015 Jun 2.

Eloy JA, Svider PF, Kanumuri VV, Folbe AJ, Setzen M, Baredes S. Do AAO-HNSF CORE Grants Predict Future NIH Funding Success? Otolaryngol Head Neck Surg. 2014 May 20. [Epub ahead of print]

Gallo SA et al. The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies. 2014 PLoS ONE 9(9): e106474. doi:10.1371/journal.pone.0106474

Goldbeck-Wood S. Evidence on Peer Review—Scientific Quality Control or Smokescreen? BMJ. 1999 January 2;318(7175):44-5.

Holbrook JB, Hrotic S. Blue Skies, Impacts, and Peer Review. RT. 2013 Jul;1(1):2013.

Jackson JL, Srinivasan M, Rea J, Fletcher KE, Kravitz RL. The Validity of Peer Review in a General Medicine Journal. PLoS One. 2011;6(7):e22475.

Jacob B, Lefgren L. The Impact of Research Grant Funding on Scientific Productivity. NBER Working Paper No. 13519. 2007.

Kaltman JR, Evans FJ, Danthi NS, Wu CO, DiMichele DM, Lauer MS. Prior Publication Productivity, Grant Percentile Ranking, and Topic-Normalized Citation Impact of NHLBI Cardiovascular R01 Grants. Circ Res. 2014 Sep 12;115(7):617-24.

Kotchen TA, Lindquist T, Miller Sostek A, Hoffmann R, Malik K, Stanfield B. Outcomes of National Institutes of Health Peer Review of Clinical Grant Applications. J Investig Med. 2006 Jan;54(1):13-9.

Laurence WF, Useche C, Laurance SG, Bradshaw CJA. Predicting Publication Success for Biologists. BioScience. 2013;63(10):817-23.

Lindner MD, Nakamura RK. Examining the Predictive Validity of NIH Peer Review Scores. PLoS One. 2015 Jun 3;10(6):e0126938.

Martin MR, Lindquist T, Kotchen TA. Why are Peer Review Outcomes Less Favorable for Clinical Science than for Basic Science Grant Applications? Am J Med. 2008 Jul;121(7):637-41.

Mason JL, Lei M, Faupel-Badger JM, Ginsburg EP, Seger YR, Dijoseph L, Schnell JD, Wiest JS. Outcome Evaluation of the National Cancer Institute Career Development Awards Program. J Cancer Educ. 2013 Mar;28(1):9-17.

Mervis J. Peering Into Peer Review. Science. 2014 Feb 7;343(6171):596-8.

Rinia EJ, Leeuwen Th N, Vuren HG, Raan AFJ. Influence of Interdisciplinarity on Peer-Review and Bibliometric Evaluations in Physics Research. Research Policy. 2001 Mar 1;30(3):357-61.

Stephen A. Gallo, Scott R. Glisson. External Tests of Peer Review Validity Via Impact Measures. Front. Res. Metr. Anal. 23 August 2018

Vale RD. Evaluating How We Evaluate. Mol Biol Cell. 2012 Sep;23(17):3285-9.

Waltman L, van Eck NJ, van Leeuwen TN, Visser MS, van Raan AF. On the Correlation Between Bibliometric Indicators and Peer Review: Reply to Opthof and Leydesdorff. Scientometrics. 2011 Sep;88(3):1017-1022.

Yglesias E. Improving Peer Review in the Federal Government. Technol Innov. 2010 Mar;12(3):225-32.