Our Bibliography of Peer Review Literature

The peer review process is used by organizations around the world to identify good, promising science. However, peer review can be implemented in many ways, and surprisingly, dialogue among organizations managing peer review is limited. Although literature about peer review (and particularly peer review of grant applications) is scant, there have been some studies addressing review practices, challenges and guidelines.

Compiled below is a reference list of scientific articles concerning the processes, outcomes and validations of the scientific peer review process, particularly focusing on peer review of applications for funding. This list is sorted by topic area and represents a centralized body of knowledge intended as a reference for those conducting/participating in peer review and those who are interested in the outcomes of the process.

Assignment Load

Cook,WD, Golany B, Kress M, Penn M. Optimal Allocation of Proposals to Reviewers to Facilitate Effective Ranking. Manage Sci. 2005;51(4):655-61.

Li L, Wang Y, Liu G, Wang M, Wu X. Context-Aware Reviewer Assignment for Trust Enhanced Peer Review. PLoS One. 2015 Jun 19;10(6):e0130493.

Conflict of Interest & Bias

Abdoul H, Perrey C, Tubach F, Amiel P, Durand-Zaleski I, Alberti C. Non-financial Conflicts of Interest in Academic Grant Evaluation: A Qualitative Study of Multiple Stakeholders in France. PLoS One. 2012;7(4):e35247.

Arms W. Ethics: Trust and Reputation on the Web. Nature 2006.

Bekelman JE, Li Y, Gross CP. Scope and Impact of Financial Conflicts of Interest in Biomedical Research: A Systematic Review. JAMA. 2003 Jan 22-29;289(4):454-65.

Benos D. Ethics: Detecting Misconduct. Nature 2006.

Bhattacharjee Y. Science Funding. NSF’s ‘Big Pitch’ Tests Anonymized Grant Reviews. Science. 2012 May 25;336(6084):969-70.

Bornmann L. Research Misconduct—Definitions, Manifestations and Extent. Publications. 2013;1(3):87-98.

Bornmann L, Daniel HD. Selection of Research Fellowship Recipients by Committee Peer Review. Reliability, Fairness and Predictive Validity of Board of Trustees’ Decisions. Scientometrics. 2005;63(2):297-320.

Coats AJ, Nijjer SS, Francis DP. Protecting the Pipeline of Science: Openness, Scientific Methods and the Lessons from Ticagrelor and the PLATO Trial. Int J Cardiol. 2014 Oct 20;176(3):600-4.

De Martino B, Kumaran D, Seymour B, Dolan RJ. Frames, Biases, and Rational Decision-Making in the Human Brain. Science. 2006 Aug 4;313(5787):684-7.

Drazen JM, De Leeuw PW, Laine C, Mulrow C, Deangelis CD, Frizelle FA, Godlee F, Haug C, Hébert PC, James A, Kotzin S, Marusic A, Reyes H, Rosenberg J, Sahni P, Van Der Weyden MB, Zhaori G. Toward More Uniform Conflict Disclosures: The Updated ICMJE Conflict of Interest Reporting Form. Natl Med J India. 2010 Jul-Aug;23(4):196-7.

Gallo SA, LeMaster M, Glisson SR. Frequency and Type of Conflicts of Interest in the Peer Review of Basic Biomedical Research Funding Applications: Self-Reporting Versus Manual Detection. Sci Eng Ethics. 2015 Feb 4.

Hodgson C. Evaluation of Cardiovascular Grant-in-aid Applications by Peer Review: Influence of Internal and External Reviewers and Committees. Can J Cardiol. 1995 Nov;11(10):864-8.

Kotchen TA, Lindquist T, Miller Sostek A, Hoffmann R, Malik K, Stanfield B. Outcomes of National Institutes of Health Peer Review of Clinical Grant Applications. J Investig Med. 2006 Jan;54(1):13-9.

Langfeldt L. Decision-making in Expert Panels Evaluating Research: Constraints, Processes and Bias. 2001;dissertation: The University of Oslo.

Langfeldt L. The Policy Challenges of Peer Review: Managing Bias, Conflict of Interests and Interdisciplinary Assessments. Res Eval. 2006;15(1):31-41.

Lee C. Sugimoto CR, Zhang G, Cronin B. Bias in Peer Review. J Am Soc Inf Sci Technol. 2013;64(1):2-17.

Li D. Expertise vs. Bias in Evaluation: Evidence from the NIH. 2013;Northwestern University.

Li L, Wang Y, Liu G, Wang M, Wu X. Context-Aware Reviewer Assignment for Trust Enhanced Peer Review. PLoS One. 2015 Jun 19;10(6):e0130493.

Lo B, Field MJ, Editors. Conflict of Interest in Medical Research, Education, and Practice. Committee on Conflict of Interest in Medical Research, Education, and Practice. 2009;Institute of Medicine

Marsh HW, Jayasinghe UW, Bond NW. Improving the Peer-Review Process for Grant Applications: Reliability, Validity, Bias, and Generalizability. Am Psychol. 2008 Apr;63(3):160-8.

NIH. Managing Conflict of Interest in NIH Peer Review of Grants and Contracts.

NIH. Revised Policy: Managing Conflict of Interest in the Initial Peer Review of NIH Grant and Cooperative Agreement Applications. NOT-OD-11-120. 2011 Sep 26.

Oleinik A. Conflict(s) of Interest in Peer Review: Its Origins and Possible Solutions. Sci Eng Ethics. 2014 Mar;20(1):55-75.

Pham-Kanter G. Revisiting Financial Conflicts of Interest in FDA Advisory Committees. Milbank Q. 2014 Sep;92(3):446-70.

Rockey SJ, Collins FS. Managing Financial Conflict of Interest in Biomedical Research. JAMA. 2010 Jun 16;303(23):2400-2.

Stehbens WE. Basic Philosophy and Concepts Underlying Scientific Peer Review. Med Hypotheses. 1999 Jan;52(1):31-6.

Wager E. Ethics: What is it for? Nature 2006.

Wood SF, Mador JK. Science and Regulation. Uncapping Conflict of Interest? Science. 2013 Jun 7;340(6137):1172-3.

Critique & Text Analysis

Bantum EO, Owen JE. Evaluating the Validity of Computerized Content Analysis Programs for Identification of Emotional Expression in Cancer Narratives. Psychol Assess. 2009 Mar;21(1):79-88.

Bornmann L, Wolf M, Daniel HD. Closed Versus Open Reviewing of Journal Manuscripts: How Far do Comments Differ in Language Use? Scientometrics. 2012;91(3):843-56.

Fuller EO, Hasselmeyer EG, Hunter JC, Abdellah FG, Hinshaw AS. Summary Statements of the NIH Nursing Research Grant Applications. Nurs Res. 1991 Nov-Dec;40(6):346-51.

Hume KM, Giladi AM, Chung KC. Factors Impacting Successfully Competing for Research Funding: An Analysis of Applications Submitted to the Plastic Surgery Foundation. Plast Reconstr Surg. 2015 Feb;135(2):429e-35e.

Kaatz A, Magua W, Zimmerman DR, Carnes M. A Quantitative Linguistic Analysis of National Institutes of Health R01 Application Critiques from Investigators at One Institution. Acad Med. 2015 Jan;90(1):69-75

Tausczik YR, Pennebaker JW. The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods. J Lang Soc Psychol. 2010;29(1):24-54.

Vener KJ, Calkins BM. Analysis of SBIR Phase I and Phase II Review Results at the National Institutes of Health. FASEB J. 1991 Sep;5(12):2640-4.

Decision Making, Emotion, Risk

De Martino B, Kumaran D, Seymour B, Dolan RJ. Frames, Biases, and Rational Decision-Making in the Human Brain. Science. 2006 Aug 4;313(5787):684-7.

Denes-Raj V, Epstein S. Conflict Between Intuitive and Rational Processing: When People Behave Against Their Better Judgment. J Pers Soc Psychol. 1994 May;66(5):819-29.

Finucane ML, Alhakami A, Slovic P, Johnson SM. The Affect Heuristic in Judgments of Risks and Benefits. J Behav Decis Mak. 2000;13(1):1-17.

Gregory R, Mendelsohn R. Perceived Risk, Dread, and Benefits. Risk Anal. 1993;13(3):259-64.

Hagman W. Affective Biases and Heuristics in Decision Making: Emotion Regulation as a Factor for Decision Making Competence. 2013. Linköping University. Master Thesis.

Langfeldt L. The Decision-Making Constraints and Processes of Grant Peer Review, and Their Effects on the Review Outcome. Social Studies of Science. 2001 Dec;31(6):820-41.

Lipshitz R, Strauss O. Coping with Uncertainty: A Naturalistic Decision-Making Analysis. Organ Behav Hum Decis Process. 1997;69(2):149-63.

Pachur T, Hertwig R, Steinmann F. How do People Judge Risks: Availability Heuristic, Affect Heuristic, or Both? J Exp Psychol Appl. 2012 Sep;18(3):314-30.

Rand DG, Epstein ZG. Risking Your Life Without a Second Thought: Intuitive Decision-Making and Extreme Altruism. PLoS One. 2014 Oct 15;9(10):e109687.

Sheeran P, Harris PR, Epton T. Does Heightening Risk Appraisals Change People’s Intentions and Behavior? A Meta-Analysis of Experimental Studies. Psychol Bull. 2014 Mar;140(2):511-43.

Sjöberg L. Factors in Risk Perception. Risk Anal. 2000;20(1):1-12.

Slovic P, Finucane ML, Peters E, MacGregor DG. Risk as Analysis and Risk as Feelings: Some Thoughts About Affect, Reason, Risk, and Rationality. Risk Anal. 2004 Apr;24(2):311-22.

Slovic P, Peters E. Risk Perception and Affect. Curr Dir Psychol Sci. 2006;15(6):322-5.

Taleb NN. Silent Risk: Technical Incerto, Lectures on Risk and Probability. Available at SSRN. 2014;1.

Gender

Bornmann L, Mutz R, Daniel HD. Gender Differences in Grant Peer Review: A Meta-Analysis. J Informetr. 2007 Jul;1(3):226-38.

Homma MK, Motohashi R, Ohtsubo H. Japan’s Lagging Gender Equality. Science. 2013 Apr 26;340(6131):428-30.

Kaufman RR, Chevan J. The Gender Gap in Peer-Reviewed Publications by Physical Therapy Faculty Members: A Productivity Puzzle. Phys Ther. 2011 Jan;91(1):122-31.

Mutz R, Bornmann L, Daniel HD. Does Gender Matter in Grant Peer Review?: An Empirical Investigation Using the Example of the Austrian Science Fund. Z Psychol. 2012;220(2):121-129.

van Arensbergen P, van der Weijden I, van den Besselaar P. Academic Talent Selection in Grant Review Panels]. In book: (Re)Searching Academic Careers, Chapter: 2. 2014. Publisher: Russian Academy of Sciences.

General Peer Review

AICPA. Summary of the Nature, Objectives, Scope, Limitations of, and Procedures Performed in System and Engagement Reviews and Quality Control Reviews (as referred to in a peer review report). 2012.

Abdoul H, Perrey C, Amiel P, Tubach F, Gottot S, Durand-Zaleski I, Alberti C. Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer Practices. PLoS One. 2012;7(9):e46054.

Alberts B, Kirschner MW, Tilghman S, Varmus H. Rescuing US Biomedical Research from its Systemic Flaws. Proc Natl Acad Sci U S A. 2014 Apr 22;111(16):5773-7.

Bandeh-Ahmadi A. Editorial Decisions and Measuring Information Contents of Reviews. Available at SSRN. 2013.

Benos DJ, Bashari E, Chaves JM, Gaggar A, Kapoor N, LaFrance M, Mans R, Mayhew D, McGowan S, Polter A, Qadri Y, Sarfare S, Schultz K, Splittgerber R, Stephenson J, Tower C, Walton RG, Zotov A. The Ups and Downs of Peer Review. Adv Physiol Educ. 2007 Jun;31(2):145-52.

Berg JM. Science Policy: Well-Funded Investigators Should Receive Extra Scrutiny. Nature. 2012 Sep 13;489(7415):203.

Bhattacharya A. Science Funding: Duel to the Death. Nature. 2012 Aug 2;488(7409):20-2.

Bielski A, Harris R, Gillis N. Summary Report of Comments Received on NIH System to Support Biomedical and Behavioral Research and Peer Review. 2007 November.

Bornmann L. Scientific Peer Review: An Analysis of the Peer Review Process from the Perspective of Sociology of Science Theories. Human Architecture: Journal of the Sociology of Self-Knowledge. 2008;VI(2):23-38.

Brown T. Perspective: ‘I Don’t Know What to Believe’. Nature 2006.

Cantor M, Gero S. The Missing Metric: Quantifying Contributions of Reviewers. R Soc Open Sci. 2015 Feb 11;2(2):140540.

Carter G. Peer Review, Citations, and Biomedical Research Policy: NIH Grants to Medical School Faculty. RAND Corporation, 1974.

Chang WR, McLean IP. CUSUM: A Tool for Early Feedback About Performance? BMC Med Res Methodol. 2006 Mar 2;6:8.

Chetty R. How Small Changes Can Influence Reviewer Behavior. Elsevier. 2014.

Cole S, Rubin L, Cole J. Peer Review in the National Science Foundation: Phase One of a Study. Washington, DC: National Academy of Sciences, 1978.

Council of Canadian Academies. Informing Research Choices: Indicators and Judgment. 2012.

Das NK, Froehlich LA. Quantitative Evaluation of Peer Review of Program Project and Center Applications in Allergy and Immunology. J Clin Immunol. 1985 Jul;5(4):220-7.

Demicheli V, Di Pietrantonj C. Peer Review for Improving the Quality of Grant Applications. Cochrane Database Syst Rev. 2007 Apr 18;(2):MR000003.

Derraik JG. The Principles of Fair Allocation of Peer-Review: How Much Should a Researcher be Expected to Contribute? Sci Eng Ethics. 2014 Sep 7.

DiSilvestro RL. The First Half Century: A History of AIBS. BioScience. 1997 Nov;47(10):643-9.

Dickersin K, Ssemanda E, Mansell C, Rennie D. What do the JAMA Editors Say When They Discuss Manuscripts That They Are Considering for Publication? Developing a Schema for Classifying the Content of Editorial Discussion. BMC Med Res Methodol. 2007 Sep 25;7:44.

Dumanis SB, Ullrich L, Washington PM, Forcelli PA. It’s Money! Real-World Grant Experience Through a Student-Run, Peer-Reviewed Program. CBE Life Sci Educ. 2013 Fall;12(3):419-28.

Fleurence RL, Forsythe LP, Lauer M, Rotter J, Ioannidis JP, Beal A, Frank L, Selby JV. Engaging Patients and Stakeholders in Research Proposal Review: The Patient-Centered Outcomes Research Institute. Ann Intern Med. 2014 Jul 15;161(2):122-30.

Fliesler SJ. Rethinking Grant Peer Review. Science. 1997 Mar 7;275(5305):1399.

Gallo SA et al. The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies. 2014 PLoS ONE 9(9): e106474. doi:10.1371/journal.pone.0106474

Garrison HH, Deschamps AM. NIH Research Funding and Early Career Physician Scientists: Continuing Challenges in the 21st Century. FASEB J. 2014 Mar;28(3):1049-58.

Gillespie GW, Jr., Chubin DE, Kurzon GM. Experience with NIH Peer Review: Researchers’ Cynicism and Desire for Change. Sci Technol Human Values. 1985;10(3):45-54.

Gillett R. Research Performance Indicators Based on Peer Review: A Critical Analysis. High Educ Q. 2007;43(1):20-38.

Glass B, Krauss RW. AIBS Presidents Revisit the Past. BioScience. 1997 Nov;47(10):650-6.

Glenn JF. Uses and Needs for Peer Review in Army Medical Research. Technol Innov. 2010;12:241-7.

Graves N, Barnett AG, Clarke P. Funding Grant Proposals for Scientific Research: Retrospective Analysis of Scores by Members of Grant Review Panel. BMJ. 2011 Sep 27;343:d4797.

Greaves S, Scott J, Clarke M, Miller M, Hannay T, Thomas A, Campbell P. Overview: Nature’s Peer Review Trial. Nature. 2006.

Groves T. Quality and Value: How Can We Get the Best Out of Peer Review? Nature. 2006.

Guthrie S, Guerin B, Wu H, Ismail S, Wooding S. Alternatives to Peer Review in Research Project Funding. Santa Monica, CA: RAND Corporation, 2013.

Hand E. Extra Scrutiny for ‘Grandee Grantees’. Nature. 2012 Feb 20;482(7386):450-1.

Hartmann I, Neidhardt F. Peer Review at the Deutsche Deutsche Forschungsgemeinschaft. Scientometrics. 1990 Nov 01;19(5-6):419-25.

Hartonen T, Alava MJ. How Important Tasks Are Performed: Peer Review. Sci Rep. 2013;3:1679.

Herbert DL, Barnett AG, Clarke P, Graves N. On the Time Spent Preparing Grant Proposals: An Observational Study of Australian Researchers. BMJ Open. 2013 May 28;3(5). pii: e002800.

Herbert DL, Barnett AG, Graves N. Funding: Australia’s Grant System Wastes time. Nature. 2013 Mar 21;495(7441):314.

Herbert DL, Coveney J, Clarke P, Graves N, Barnett AG. The Impact of Funding Deadlines on Personal Workloads, Stress and Family Relationships: A Qualitative Study of Australian Researchers. BMJ Open. 2014 Mar 28;4(3):e004462.

Herrmann-Lingen C, Brunner E, Hildenbrand S, Loew TH, Raupach T, Spies C, Treede RD, Vahl CF, Wenz HJ. Evaluation of Medical Research Performance—Position Paper of the Association of the Scientific Medical Societies in Germany (AWMF). Ger Med Sci. 2014 Jun 26;12:Doc11.

Hickam D, Totten A, Berg A, Rader K, Goodman S, Newhouse R, Editors. The PCORI Methodology Report. 2013.

Ho RC, Mak KK, Tao R, Lu Y, Day JR, Pan F. Views on the Peer Review System of Biomedical Journals: An Online Survey of Academics from High-Ranking Universities. BMC Med Res Methodol. 2013 Jun 7;13:74.

Houser SR. How to Obtain a National Heart, Lung, and Blood Institute-Sponsored K08 and K99/R00 Grant in the Current Funding Climate. Circ Res. 2012 Mar 30;110(7):907-9.

IOM. Strategies for managing the Breast Cancer Research Program: A Report to the U.S. Army Medical Research and Development Command. Washington, DC: National Academy Press, 1993.

Irwin D, Gallo SA, Glisson SR. Opinion: Learning from Peer Review. The Scientist. 2013 May 24.

Ismail S, Farrands A, Wooding S. Evaluating Grant Peer Review in the Health Sciences: A review of the Literature. Santa Monica, CA: RAND Corporation, 2009.

Jefferson T. Quality and Value: Models of Quality Control for Scientific Research. Nature. 2006.

Jefferson T, Wager E, Davidoff F. Measuring the Quality of Editorial Peer Review. JAMA. 2002 Jun 5;287(21):2786-90.

Jennings C. Quality and Value: The True Purpose of Peer Review. Nature. 2006.

Johnson VE. Statistical Analysis of the National Institutes of Health Peer Review System. Proc Natl Acad Sci U S A. 2008 Aug 12;105(32):11076-80.

Kemp E. Open Letter to Senior Editors of Peer-Review Journals Publishing in the Field of Stem Cell Biology. EuroStemCell. 2009 Jul.

King JC, Lawrence TS, Murphy SB, Davidson NE, Mayer RJ. The American Society of Clinical Oncology Cancer Foundation Grants Program: A 25-Year Report and a Look Toward the Future. J Clin Oncol. 2010 Mar 20;28(9):1616-21.

Koonin E, Landweber L, Lipman D, Dignon R. Systems: Reviving a Culture of Scientific Debate. Nature. 2006.

Kostoff RN. Research Program Peer Review: Purposes, Principles, Practices, Protocols. 2004. Office of Naval Research.

Kupfer DJ, Murphree AN, Pilkonis PA, Cameron JL, Giang RT, Dodds NE, Godard KA, Lewis DA. Using Peer Review to Improve Research and Promote Collaboration. Acad Psychiatry. 2014 Feb;38(1):5-10.

Lahiri D. Perspective: The Case for Group Review. Nature 2006.

Lee C. Perspective: Peer Review of Interdisciplinary Scientific Papers. Nature 2006.

Lee K, Bero L. Ethics: Increasing Accountability. Nature 2006.

Lee K, Brownstein JS, Mills RG, Kohane IS. Does Collocation Inform the Impact of Collaboration? PLoS One. 2010 Dec 15;5(12):e14279.

Lyman RL. A Three-Decade History of the Duration of Peer Review. Journal of Scholarly Publishing. 2013 Apr;44(3):211-20.

McGarity TO. Peer Review in Awarding Federal Grants in the Arts and Sciences. Berkeley Tech LJ. 1994;9(1).

McGeary M, Hanna KE. Strategies to Leverage Research Funding: Guiding DOD’s Peer Reviewed Medical Research Programs. Washington, DC: National Academies Press, 2004.

Mervis J. Peer Review. Beyond the Data. Science. 2011 Oct 14;334(6053):169-71.

Mietchen D. The Transformative Nature of Transparency in Research Funding. PLoS Biol. 2014 Dec 30;12(12):e1002027.

Moed HF, Glanzel W, Schmoch U. Handbook of Quantitative Science and Technology Research: The Use of Publications and Patent Statistics in Studies of S&T Systems. New York, NY: Kluwer Academic Publishers, 2004.

Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. Int J Surg. 2010;8(5):336-41.

Moore J. Perspective: Does Peer Review Mean the Same to the Public as it Does to Scientists? Nature 2006.

Myers ER, Alciati MH, Ahlport KN, Sung NS. Similarities and Differences in Philanthropic and Federal Support for Medical Research in the United States: An Analysis of Funding by Nonprofits in 2006-2008. Acad Med. 2012 Nov;87(11):1574-81.

Nicholson J, Ioannidis J. Research Grants: Conform and be Funded. Nature. 2012 Dec 6;492:34-36.

Ozonoff D. Quality and Value: Statistics in Peer Review. Nature. 2006.

Pawelczyk JA, Strawbridge LM, Schultz AM, Liverman CT. A Review of NASA Human Research Program’s Scientific Merit Processes: Letter Report. Washington, DC: National Academies Press, 2012.

Pederson T. The “Study” Role of Past National Institutes of Health Study Sections. Mol Biol Cell. 2012 Sep;23(17):3281-4.

Parliamentary Office of Science and Technology. Peer Review. 2002;182.

Ranalli B. A Prehistory of Peer Review: Religious Blueprints from the Hartlib Circle. Spontaneous Generations. 2011;5(1):12-18.

Riley B. Systems: Trusting Data’s Quality. Nature. 2006.

Roebber PJ, Schultz DM. Peer Review, Program Officers and Science Funding. PLoS One. 2011 Apr 12;6(4):e18680.

Russell AS, Thorn BD, Grace M. Peer Review: A Simplified Approach. J Rheumatol. 1983 Jun;10(3):479-81.

Sandewall E. Systems: Opening up the Process. Nature. 2006.

Sandstrom U, Hallsten M. Persistent Nepotism in Peer-Review. Scientometrics. 2008 Feb;74(2):175-89.

Schroter S, Groves T, Højgaard L. Surveys of Current Status in Biomedical Science Grant Review: Funding Organisations’ and Grant Reviewers’ Perspectives. BMC Med. 2010 Oct 20;8:62.

Servick K. Bioinformatics. Top Contenders Blast Pentagon’s New Bioterror Detection Prize. Science. 2013 Aug 2;341(6145):449.

Shashok K. Content and Communication: How Can Peer Review Provide Helpful Feedback About the Writing? BMC Med Res Methodol. 2008 Jan 31;8:3.

Shideed O, Al-Gasseer N. Appraisal of the Research Grant Schemes of the World Health Organization Regional Office for the Eastern Mediterranean: The Way Forward. East Mediterr Health J. 2012 May;18(5):515-21.

Sieber J. Quality and Value: How Can We Research Peer Review? Nature 2006.

Smith MA, Kaufman NJ, Dearlove AJ. External Community Review Committee: A New Strategy for Engaging Community Stakeholders in Research Funding Decisions. Prog Community Health Partnersh. 2013 Fall;7(3):301-12.

Smith R. Classical Peer Review: An Empty Gun. Breast Cancer Res. 2010 Dec 20;12 Suppl 4:S13.

Smith R. Peer Review: A Flawed Process at the Heart of Science and Journals. J R Soc Med. 2006 Apr;99(4):178-82.

Spier R. The History of the Peer-Review Process. Trends Biotechnol. 2002 Aug;20(8):357-8.

Squazzoni F, Gandelli, C. Peer Review Under the Microscope. An Agent-Based Model of Scientific Collaboration. Simulation Conference (WSC), Proceedings of the 2012 Winter. Dec 2012:1-12.

Squazzoni F, Takacs, K. Social Simulation that “Peers into Peer Review.” J Artif Soc Soc Simulat. 2011 Oct 31;14(4):3.

Stahel PF, Moore EE. Peer Review for Biomedical Publications: We Can Improve the System. BMC Med. 2014 Sep 26;12(1):179.

Stehbens WE. Basic Philosophy and Concepts Underlying Scientific Peer Review. Med Hypotheses. 1999 Jan;52(1):31-6.

Turner RS. Best Practices in Peer Review Assure Quality, Value, Objectivity. Journal of the National Grants Management Association. 2009;17(1):43-8.

United States General Accounting Office. Federal Research: Peer Review Practices at Federal Science Agencies Vary. Report to Congressional Requesters. 2009 March.

Van de Sompel H. Technical Solutions: Certification in a Digital Era. Nature 2006.

Walker R, Rocha da Silva P. Emerging Trends in Peer Review-a Survey. Front Neurosci. 2015 May 27;9:169.

Wessely S. Peer Review of Grant Applications: What do we Know? Lancet. 1998 Jul 25;352(9124):301-5.

Zare RN, Winnacker EL. China’s Science Funding. Science. 2011 Oct 28;334(6055):433.

van Arensbergen P, van der Weijden I, van den Besselaar P. Academic Talent Selection in Grant Review Panels]. In book: (Re)Searching Academic Careers, Chapter: 2. 2014. Publisher: Russian Academy of Sciences.

General Science

American Society for Biochemistry and Molecular Biology. Towards a Sustainable Biomedical Research Enterprise. 2014.

Bond Q, Gallin EK. Commentary: Recognizing and Tracking Philanthropy’s Critical Support of Health Research. Acad Med. 2012 Nov;87(11):1466-7.

Boudreau KJ, Eva G, Lakhani KR, Riedl C. Looking Across and Looking Beyond the Knowledge Frontier: Intellectual Distance and Resource Allocation in Science. Available at SSRN. 2014.

Boyack KW, Klavans R, Borner K. Mapping the Backbone of Science. Scientometrics. 2005;64(3):351-74.

Buttliere BT. Using Science and Psychology to Improve the Dissemination and Evaluation of Scientific Work. Front Comput Neurosci. 2014 Aug 19;8:82.

Cohen LR, Noll RG. The Future of the National Laboratories. PNAS. 1996;93(23):12678-85.

Couzin-Frankel J. Chasing the Money. Science. 2014 Apr 4;344(6179):24-5.

Davis, G. Doctors Without Orders. American Scientist. 2005;93(3, supplement).

Furmana JL, Porter ME, Stern S. The Determinants of National Innovative Capacity. Res Policy. 2002;31(6):899-933.

Garrison HH. Ngo K. Education and Employment of Biological and Medical Scientists 2011: Data from National Surveys. FASEB. 2012 Feb.

Garrison HH, Deschamps AM. Physician Scientists: Assessing the Workforce. FASEB. 2013.

Halffman W, Leydesdorff L. Is Inequality Among Universities Increasing? Gini Coefficients and the Elusive Rise of Elite Universities. Minerva. 2010 Mar;48(1):55-72.

Ioannidis JP. How to Make More Published Research True. PLoS Med. 2014 Oct 21;11(10):e1001747.

Lopes GR, da Silva R, Moro MM, Palazzo Moreira de Oliveira J. Scientific Collaboration in Research Networks: A Quantification Method by Using Gini Coefficient. IJCSA. 2012;9(2):15-31.

Muscio A, Quaglione D, Vallanti G. Does Government Research Funding to Universities Substitute, Complement or Leverage Industry Funding? Working Papers CELEG. 2010.

Enhancing the Postdoctoral Experience for Scientists and Engineers: A Guide for Postdoctoral Scholars, Advisers, Institutions, Funding Organizations, and Disciplinary Societies. National Academy of Sciences, National Academy of Engineering, Institute of Medicine. 2000.

National Institute of Allergy and Infectious Diseases. FY 2014 Budget.

National Institute of Allergy and Infectious Diseases. The Edge of Discovery: A Portrait of the National Institute of Allergy and Infectious Diseases. June 2009;NIH Publication No. 09-6389.

Press WH. Presidential Address. What’s so Special About Science (and How Much Should We Spend on it?). Science. 2013 Nov 15;342(6160):817-22.

Readers’ Poll Results: Funding Environment. Science. 2013 Apr 26;340(6131):429.

Rice JA, Wu CO. Nonparametric Mixed Effects Models for Unequally Sampled Noisy Curves. Biometrics. 2001 Mar;57(1):253-9.

Sullivan Faith K. Patterns of Creation and Discovery: An Analysis of Defense Laboratory Patenting and Innovation. 2013;dissertation:Pardee Rand Graduate School.

Taleb NN, Douady R. Mathematical Definition, Mapping, and Detection of (Anti)Fragility. Quant Finance. 2013;13(11):1677-89.

Van den Besselaar P, Leydesdorf L. Past Performance as Predictor of Successful Grant Applications: A Case Study. Den Haag, Rathenau Instituut SciSA rapport 0704.

H-Index, Citations, & Bibliometrics

Auranen O, Nieminen M. University Research Funding and Publication Performance—An International Comparison. Res Policy. 2010;39(6):822-34.

Belter CW. Can Bibliometric Indicators Predict Institutional Citation Patterns? Presented at ACRL 2013, 10-13 April, Indianapolis IN.

Bornmann L, Daniel HD. The State of H Index Research. Is the H Index the Ideal Way to Measure Research Performance? EMBO Rep. 2009;10(1):2-6.

Bornmann, L, de Moya Anegon F, Leydesdorff L. The New Excellence Indicator in the World Report of the SCImago Institutions Rankings 2011. 2011.

Bornmann L, Leydesdorff L. On the Meaningful and Non-Meaningful Use of Reference Sets in Bibliometrics. J Informetr. 2014;8(1):273-5.

Bornmann L, Marx W. How good is research really? Measuring the Citation Impact of Publications with Percentiles Increases Correct Assessments and Fair Comparisons. EMBO Rep. 2013 Mar 1;14(3):226-30.

Bornmann L. Marx W. Methods for the Generation of Normalized Citation Impact Scores in Bibliometrics: Which Method Best Reflects the Judgements of Experts? 2014.

Bornmann L, Mutz R, Neuhaus C, Daniel HD. Citation Counts for Research Evaluation: Standards of Good Practice for Analyzing Bibliometric Data and Presenting and Interpreting Results. ESEP. 2008; 8(1):93-102.

Boyack KW, Chen MC, Chacko G. Characterization of the Peer Review Network at the Center for Scientific Review, National Institutes of Health. PLoS One. 2014 Aug 13;9(8):e104244.

Boyack KW, Jordan P. Metrics Associated with NIH Funding: A High-Level View. J Am Med Inform Assoc. 2011 Jul-Aug;18(4):423-31.

Cantor M, Gero S. The Missing Metric: Quantifying Contributions of Reviewers. R Soc Open Sci. 2015 Feb 11;2(2):140540.

Carter G. Peer Review, Citations, and Biomedical Research Policy: NIH Grants to Medical School Faculty. RAND Corporation, 1974.

Danthi NS, Wu CO, DiMichele DM, Hoots WK, Lauer MS. Citation Impact of NHLBI R01 Grants Funded Through the American Recovery and Reinvestment Act as Compared to R01 Grants Funded Through a Standard Payline. Circ Res. 2015 Feb 27;116(5):784-8.

Didegah F. Determinants of Research Citation Impact: A Combined Statistical Modelling.

Doyle JM, Quinn K, Bodenstein YA, Wu CO, Danthi N, Lauer MS. Association of Percentile Ranking with Citation Impact and Productivity in a Large Cohort of de novo NIMH-funded R01 Grants. Mol Psychiatry. 2015 Jun 2.

Farhadi H, Salehi H, Md Yunus M, Chadegani AA, Farhadi M, Fooladi M, Ebrahim NA. Does it Matter Which Citation Tool is Used to Compare the H-Index of a Group of Highly Cited Researchers? Aust J Basic Appl Sci. 2013;7(4):198-202.

Garfield E. Current Comments: Lifetime Citations Rates. Essays of an Information Scientist. 1980 Jan;4(2):355-8.

Gauffriau M, Larsen PO, Maye I, Roulin-Perriard A, von Ins M. Publication, Cooperation and Productivity Measures in Scientific Research. Scientometrics. 2007;73(2).175-214.

Harzing AW. A Preliminary Test of Google Scholar as a Source for Citation Data: A Longitudinal Study of Nobel Prize Winners. Scientometrics. 2013;4(3):1057-75.

Hirsch JE. An Index to Quantify an Individual’s Scientific Research Output. Proc Natl Acad Sci U S A. 2005 Nov 15;102(46):16569-72.

Hirsch JE. Does the H Index Have Predictive Power? Proc Natl Acad Sci U S A. 2007 Dec 4;104(49):19193-8.

Ioannidis JP. Concentration of the Most-Cited Papers in the Scientific Literature: Analysis of Journal Ecosystems. PLoS One. 2006 Dec 20;1:e5.

Kaltman JR, Evans FJ, Danthi NS, Wu CO, DiMichele DM, Lauer MS. Prior Publication Productivity, Grant Percentile Ranking, and Topic-Normalized Citation Impact of NHLBI Cardiovascular R01 Grants. Circ Res. 2014 Sep 12;115(7):617-24.

Khan NR, Thompson CJ, Taylor DR, Gabrick KS, Choudhri AF, Boop FR, Klimo P Jr. Part II: Should the H-Index be Modified? An Analysis of the M-Quotient, Contemporary H-Index, Authorship Value, and Impact Factor. World Neurosurg. 2013 Dec;80(6):766-74.

Kinney AL. National Scientific Facilities and Their Science Impact on Nonbiomedical Research. Proc Natl Acad Sci U S A. 2007 Nov 13;104(46):17943-7.

Knoth P, Herrmannova D. Towards Semantometrics: A New Semantic Similarity Based Measure for Assessing a Research Publication’s Contribution. D-Lib Magazine. 2014;20(11/12).

Larivière V, Macaluso B, Archambault E, Gingras Y. Which Scientific Elite? On the Concentration of Funding, Productivity and Citations. In Proceedings of the 12th International Conference of the International Society for Scientometrics and Informetrics. 2009.

Lazaridis T. Ranking University Departments Using the Mean H-Index. Scientometrics. 2010 Feb;82(2):211-16.

Marx W, Schier H, Andersen OK. Using Time Dependent Citation Rates (Sales Curves) for Comparing Scientific Impacts. 2006. arXiv:physics/0611284 [physics.soc-ph].

Meho LI. Rogers I. Citation Counting, Citation Ranking, and H-Index of Human-Computer Interaction Researchers: A Comparison Between Scopus and Web of Science. J Am Soc Inf Sci Technol. 2008 Sep;59(11):1711-26.

Molinari JF, Molinari A. A New Methodology for Ranking Scientific Institutions. Scientometrics. 2008 Apr;75(1):163-74.

Nieminen P, Carpenter J, Rucker G, Schumacher M. The Relationship Between Quality of Research and Citation Frequency. BMC Med Res Methodol. 2006 Sep 1;6:42.

Opthof T, Leydesdorff L. Caveats for the Journal and Field Normalizations in the CWTS (“Leiden”) Evaluations of Research Performance. J Informetr. 2010;4(3):423-30.

Sidiropoulos A, Katsaros D, Manolopoulos Y. Generalized Hirsch H-Index for Disclosing Latent Facts in Citation Networks. Scientometrics. 2007 Aug 1;72(2):253-280.

Sypsa V, Hatzakis A. Assessing the Impact of Biomedical Research in Academic Institutions of Disparate Sizes. BMC Med Res Methodol. 2009 May 29;9:33.

Thomas Reuters. Using Bibliometrics: A Guide to Evaluating Research Performance with Citation Data. 2008.

Waltman L, van Eck NJ, van Leeuwen TN, Visser MS, van Raan AF. On the Correlation Between Bibliometric Indicators and Peer Review: Reply to Opthof and Leydesdorff. Scientometrics. 2011 Sep;88(3):1017-1022.

van Raan AFJ. Comparison of the Hirsch-Index with Standard Bibliometric Indicators and With Peer Iudgment for 147 Chemistry Research Groups. Scientometrics. 2006;67(3):491-502.

Innovation

Boudreau KJ, Eva G, Lakhani KR, Riedl C. Looking Across and Looking Beyond the Knowledge Frontier: Intellectual Distance and Resource Allocation in Science. Available at SSRN. 2014.

Horrobin DF. The Philosophical Basis of Peer Review and the Suppression of Innovation. JAMA. 1990 Mar 9;263(10):1438-41.

Kaime EM, Moore KH, Goldberg SF. CDMRP: Fostering Innovation Through Peer Review. Technol Innov. 2010;12:233-40.

Kaplan D. Point: Statistical Analysis in NIH Peer Review—Identifying Innovation. FASEB J. 2007 Feb;21(2):305-8.

Luukkonen T. Conservatism and Risk-Taking in Peer Review: Emerging ERC Practices. Research Evaluation. 2012;21:48-60.

Nicholson J, Ioannidis J. Research Grants: Conform and be Funded. Nature. 2012 Dec 6;492:34-36.

Inter-rater Reliability

Bornmann L, Mutz R, Daniel HD. A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and its Determinants. PLoS One. 2010 Dec 14;5(12):e14331.

Cicchetti DV. The Reliability of Peer Review for Manuscript and Grant Submissions: A Cross-Disciplinary Investigation. Behav Brain Sci. 1991;14:119-86.

Colliver J. Commentary on Cichetti’s “Reliability of Peer Review.” Teach Learn Med. 2002;14(3):142-3.

Fleiss JL, Shrout PE. The Effects of Measurement Errors on Some Multivariate Procedures. Am J Public Health. 1977 December; 67(12): 1188-91.

Fogelholm M, Leppinen S, Auvinen A, Raitanen J, Nuutinen A, Väänänen K. Panel Discussion Does Not Improve Reliability of Peer Review for Medical Research Grant Proposals. J Clin Epidemiol. 2012 Jan;65(1):47-52.

Giraudeau B, Leyrat C, Le Gouge A, Léger J, Caille A. Peer Review of Grant Applications: A Simple Method to Identify Proposals with Discordant Reviews. PLoS One. 2011;6(11):e27557.

Hallgren KA. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutor Quant Methods Psychol. 2012;8(1):23-34.

Hargens LL, Herting JR. Neglected Considerations in the Analysis of Agreement Among Journal Referees. Scientometrics. 1990 Jul 01;19(1-2):91-106.

Hodgson C. How Reliable is Peer Review? An Examination of Operating Grant Proposals Simultaneously Submitted to Two Similar Peer Review Systems. J Clin Epidemiol. 1997 Nov;50(11):1189-95.

Jayasinghe UW, Marsh HW, Bond, N. A Multilevel Cross-Classified Modelling Approach to Peer Review of Grant Proposals: The Effects of Assessor and Researcher Attributes on Assessor Ratings. J R Statist Soc A. 2003;166(3):279-300.

Johnson VE. Statistical Analysis of the National Institutes of Health Peer Review System. Proc Natl Acad Sci U S A. 2008 Aug 12;105(32):11076-80.

Kravitz RL, Franks P, Feldman MD, Gerrity M, Byrne C, Tierney WM. Editorial Peer Reviewers’ Recommendations at a General Medical Journal: Are They Reliable and do Editors Care? PLoS One. 2010 Apr 8;5(4):e10072.

Marsh HW, Jayasinghe UW, Bond NW. Improving the Peer-Review Process for Grant Applications: Reliability, Validity, Bias, and Generalizability. Am Psychol. 2008 Apr;63(3):160-8.

Mutz R, Bornmann L, Daniel HD. Heterogeneity of Inter-Rater Reliabilities of Grant Peer Reviews and its Determinants: A General Estimating Equations Approach. PLoS One. 2012;7(10):e48509.

Rothwell PM, Martyn CN. Reproducibility of Peer Review in Clinical Neuroscience. Is Agreement Between Reviewers Any Greater Than Would be Expected by Chance Alone? Brain. 2000 Sep;123 (Pt 9):1964-9.

Sattler DN, McKnight PE, Naney L, Mathis R. Grant Peer Review: Improving Inter-Rater Reliability with Training. PLoS One. 2015 Jun 15;10(6):e0130450.

NIH & NSF

Berg J. NIH-Wide Correlations Between Overall Impact Scores and Criterion Scores. NIGMS Feedback Loop Blog. 2010 Sep 30.

Berg J. Scoring Analysis with Funding and Investigator Status. NIGMS Feedback Loop Blog. 2010 Sep 14.

Bielski A, Harris R, Gillis N. Summary Report of Comments Received on NIH System to Support Biomedical and Behavioral Research and Peer Review. 2007 November.

Collins FS, Tabak LA. Policy: NIH Plans to Enhance Reproducibility. Nature. 2014 Jan 30;505(7485):612-3.

Gillespie GW, Jr., Chubin DE, Kurzon GM. Experience with NIH Peer Review: Researchers’ Cynicism and Desire for Change. Sci Technol Human Values. 1985;10(3):45-54.

Kaiser J. Biomedical Research. A Call for NIH Youth Movement. Science. 2014 Oct 10;346(6206):150-1.

Moore RF, Catevenis K, Pearson K, Wagner RM. AIRI Statistics: Trends in NIH Extramural Funding. Presented at the Association of Independent Research Institutes (AIRI) 51st Annual Meeting. 2012, 3 Oct, Philadelphia, PA.

NCI. The NCI Consumers’ Guide to Peer Review. 2009 Nov.

NIH. 2007-2008 Peer Review Self-Study. 2008.

NIH. Managing Conflict of Interest in NIH Peer Review of Grants and Contracts.

NIH. Instructions for Use of the Review Critique Template. 2009 Apr 21.

NIH. NIH Peer Review: Grants and Cooperative Agreements. 2013.

NIH. NIH Success Rate Definition. 2009 Feb.

NIH. NIH Success Rate Definition. 2012 Feb.

NIH. Overall Impact Versus Significance. 2012 Mar 5.

NIH. R01 Guide for Reviewers. Investigator Initiated Research Project Grant Applications. 2010 Feb 2.

NIH. Research Project Grant (RPG) Critique Template. R01/R03/R15/R21/R34 Review. 2012 Mar 5.

NIH. NIH Reviewer Orientation. 2012 Dec 14.

NIH. Revised Policy: Managing Conflict of Interest in the Initial Peer Review of NIH Grant and Cooperative Agreement Applications. NOT-OD-11-120. 2011 Sep 26.

NIH. A Self Study by the NIH in Partnership with the Scientific Community to Strengthen Peer Review in Changing Times. 2008.

NIH. Scoring and Review Changes. 2009 Oct 22.

NIH. Scoring System and Procedure. 2009 Mar 20.

NIH. Side-by-side comparison of enhanced and former review criteria. 2009 Dec 1.

NIH. Center for Scientific Review Advisory Council Meeting. October 25, 2011.

NIH. Moving Forward After an Unsuccessful A1 Application. 2012 May.

NIH. Chair Orientation. 2012 Mar 5.

NIH. Department of Health and Humans National Institutes of Health FY 2013 Budget. 2012.

NIH. Encouraging Early Transition to Research Independence: Modifying the NIH New Investigator Policy to Identify Early Stage Investigators. NOT-OD-08-121. 2008 Sep 26.

NIH. Enhancing Peer Review: Implementation of Recommended Actions. Transcript accompanying video of Dr. Alan Willard. 2009 Feb.

NIH. Enhancing Peer Review: The NIH Announces Enhanced Review Criteria for Evaluation of Research Applications Received for Potential FY2010 Funding. NOT-OD-09-025. 2008 Dec 2.

NIH. Enhancing Peer Review: The NIH Announces New Scoring Procedures for Evaluation of Research Applications Received for Potential FY2010 Funding. NOT-OD-09-024. 2008 Dec 2.

NIH. Evaluation Summary of New 1-9 Scoring System. 2009 Oct 13.

NSF. National Science Foundation’s Merit Review Criteria: Review and Revisions. 2011 Dec 14.

Tabak LW. Report on Enhancing Peer Review at NIH Implementation Plan. NIH. 2008 Jun 6.

Open Peer Review

Bornmann L, Wolf M, Daniel HD. Closed Versus Open Reviewing of Journal Manuscripts: How Far do Comments Differ in Language Use? Scientometrics. 2012;91(3):843-56.

DeCoursey T. Perspective: The Pros and Cons of Open Peer Review. Nature 2006.

Hettyey A, Griggio M, Mann M, Raveh S, Schaedelin FC, Thonhauser KE, Thoss M, van Dongen WF, White J, Zala SM, Penn DJ. Peerage of Science: Will it Work? Trends Ecol Evol. 2012 Apr;27(4):189-90.

Koop T, Pöschl U. Systems: An Open, Two-Stage Peer-Review Journal. Nature. 2006.

Mietchen D. The Transformative Nature of Transparency in Research Funding. PLoS Biol. 2014 Dec 30;12(12):e1002027.

Moylan EC, Harold S, O’Neill C, Kowalczuk MK. Open, Single-Blind, Double-Blind: Which Peer Review Process do You Prefer? BMC Pharmacol Toxicol. 2014 Sep 30;15:55.

Stehbens WE. Basic Philosophy and Concepts Underlying Scientific Peer Review. Med Hypotheses. 1999 Jan;52(1):31-6.

Walker R, Rocha da Silva P. Emerging Trends in Peer Review-a Survey. Front Neurosci. 2015 May 27;9:169.

Panel Discussion

Cole S, Cole JR, Simon GA. Chance and Consensus in Peer Review. Science. 1981 Nov 20;214(4523):881-6.

Fogelholm M, Leppinen S, Auvinen A, Raitanen J, Nuutinen A, Väänänen K. Panel Discussion Does Not Improve Reliability of Peer Review for Medical Research Grant Proposals. J Clin Epidemiol. 2012 Jan;65(1):47-52.

Johnson VE. Statistical Analysis of the National Institutes of Health Peer Review System. Proc Natl Acad Sci U S A. 2008 Aug 12;105(32):11076-80.

Martin MR, Kopstein A, Janice JM. An Analysis of Preliminary and Post-Discussion Priority Scores for Grant Applications Peer Reviewed by the Center for Scientific Review at the NIH. PLoS One. 2010 Nov 17;5(11):e13526.

Obrecht M, Tibelius K, D’Aloisio G. Examining the Value Added by Committee Discussion in the Review of Applications for Research Awards. Research Evaluation. 2007 Jun;16(2):79-91.

Peer Review Technology

Akerman R. Technical Solutions: Evolving Peer Review for the Internet. Nature 2006.

Anderson C. Technical Solutions: Wisdom of the Crowds. Nature 2006.

Bloom T. Systems: Online Frontiers of the Peer-Reviewed Literature. Nature. 2006.

Bohannon J. National Science Foundation. Meeting for Peer Review at a Resort That’s Virtually Free. Science. 2011 Jan 7;331(6013):27.

Doran MR, Lott WB, Doran SE. Communication: Use Multimedia in Grant Applications. Nature. 2014 Jan 16;505(7483):291.

Driskell JE, Radtke PH, Salas E. Virtual Teams: Effects of Technological Mediation on Team Performance. Group Dyn. 2003 Dec;7(4):297-323.

Gallo SA, Carpenter AS, Glisson SR (2013) Teleconference Versus Face-to-Face Scientific Peer Review of Grant Application: Effects on Review Outcomes. PLoS ONE 8(8).

Jenkins SL, Iyengar R, Diverse-Pierluissi MA, Chan AM, Devi LA, Sobie EA, Ting AT, Weinstein DC. Teaching Resources. Using Web-Based Discussion Forums as a Model of the Peer-Review Process and a Tool for Assessment. Sci Signal. 2008;1(9):tr2.

Resubmission

Bornmann L, Mutz R, Daniel H-R. Latent Markov Modeling Applied to Grant Peer Review. J Informetr. 2008;2(4):217-28.

Reviewer/PI Demographics

Jones BF, Weinberg BA. Age Dynamics in Scientific Creativity. Proc Natl Acad Sci U S A. 2011 Nov 22;108(47):18910-4.

Kaplan D, Lacetera N, Kaplan C. Sample Size and Precision in NIH Peer Review. PLoS One. 2008 Jul 23;3(7):e2761.

Zerhouni E. Some Observations on Demographics of NIH-Funded Scientists: Policy Implications for New Investigators. 2007 Dec 7.

Scoring Scale

Cicchetti DV, Shoinralter D, Tyrer PJ. The Effect of Number of Rating Scale Categories on Levels of Interrater Reliability: A Monte Carlo Investigation. Appl Psychol Meas. 1985 Mar;9(1):31-6.

Green JG, Calhoun F, Nierzwicki L, Brackett J, Meier P. Rating Intervals: An Experiment in Peer Review. FASEB J. 1989 Jun;3(8):1987-92.

NIH. Evaluation Summary of New 1-9 Scoring System. 2009 Oct 13.

Team Performance

Barabási AL. Sociology. Network Theory—the Emergence of the Creative Enterprise. Science. 2005 Apr 29;308(5722):639-41.

Borgatti SP, Mehra A, Brass DJ, Labianca G. Network Analysis in the Social Sciences. Science. 2009 Feb 13;323(5916):892-95.

Cur?eu PL, Jansen RJ, Chappin MM. Decision Rules and Group Rationality: Cognitive Gain or Standstill? PLoS One. 2013;8(2):e56454.

Driskell JE, Radtke PH, Salas E. Virtual Teams: Effects of Technological Mediation on Team Performance. Group Dyn. 2003 Dec;7(4):297-323.

Duch J, Waitzman JS, Amaral LA. Quantifying the Performance of Individual Players in a Team Activity. PLoS One. 2010 Jun 16;5(6):e10937.

Evans JA, Foster JG. Metaknowledge. Science. 2011 Feb 11;331(6018):721-5.

Guimerà R, Uzzi B, Spiro J, Amaral LA. Team Assembly Mechanisms Determine Collaboration Network Structure and Team Performance. Science. 2005 Apr 29;308(5722):697-702.

Jeffcott SA, Mackenzie CF. Measuring Team Performance in Healthcare: Review of Research and Implications for Patient Safety. J Crit Care. 2008 Jun;23(2):188-96.

Langfeldt L. Decision-Making in Expert Panels Evaluating Research: Constraints, Processes and Bias. 2001;dissertation: The University of Oslo.

Manthous CA, Hollingshead AB. Team Science and Critical Care. Am J Respir Crit Care Med. 2011 Jul 1;184(1):17-25.

Olbrecht M, Bornmann L. Panel Peer Review of Grant Applications: What Do We Know From Research in Social Psychology On Judgment and Decision-Making in Groups? Res Eval. 2010;19 (4):293-304.

Roebber PJ, Schultz DM. Peer Review, Program Officers and Science Funding. PLoS One. 2011 Apr 12;6(4):e18680.

Rogelberg SG, O’Connor MS, Sederburg M. Using the Stepladder Technique to Facilitate the Performance of Audioconferencing Groups. J Appl Psychol. 2002 Oct;87(5):994-1000.

Rosen MA, Weaver SJ, Lazzara EH, Salas E, Wu T, Silvestri S, Schiebel N, Almeida S, King HB. Tools for Evaluating Team Performance in Simulation-Based Training. J Emerg Trauma Shock. 2010 Oct;3(4):353-9.

Stokols D, Misra S, Moser RP, Hall KL, Taylor BK. The Ecology of Team Science: Understanding Contextual Influences on Transdisciplinary Collaboration. Am J Prev Med. 2008 Aug;35(2 Suppl):S96-115.

Wickens CD, Holland JG, Parasuraman R, Banbury S. Engineering Psychology and Human Performance (4th ed). Upper Saddle River, NJ: Prentice Hall, 2012.

Training

Garfield JM, Kaye AD, Kolinsky DC, Urman RD. A Systematic Guide for Peer Reviewers for a Medical Journal. J Med Pract Manage. 2015 Mar-Apr;30(6 Spec No):13-7.

Rosen MA, Weaver SJ, Lazzara EH, Salas E, Wu T, Silvestri S, Schiebel N, Almeida S, King HB. Tools for Evaluating Team Performance in Simulation-Based Training. J Emerg Trauma Shock. 2010 Oct;3(4):353-9.

Sattler DN, McKnight PE, Naney L, Mathis R. Grant Peer Review: Improving Inter-Rater Reliability with Training. PLoS One. 2015 Jun 15;10(6):e0130450.

Validation & Impact

Armstrong PW, Caverson MM, Adams L, Taylor M, Olley PM. Evaluation of the Heart and Stroke Foundation of Canada Research Scholarship Program: Research Productivity and Impact. Can J Cardiol. 1997 May;13(5):507-16.

Beacham L, Li E, Wasserman A. Measuring the Performance of Extramural Funding at the National Library of Medicine. The College of William and Mary. 2008.

Berg J. Productivity Metrics and Peer Review Scores. NIGMS Feedback Loop Blog. 2011 Jun 2.

Bornmann, L, Daniel H-D. Reliability, Fairness and Predictive Validity of Committee Peer Review: Evaluation of the Selection of Post-Graduate Fellowship Holders by the Boehringer Ingelheim Fonds. B.I.F. FUTURA. 2004;19.

Bornmann L, Daniel HD. Selection of Research Fellowship Recipients by Committee Peer Review. Reliability, Fairness and Predictive Calidity of Board of Trustees’ Decisions. Scientometrics. 2005;63(2):297-320.

Bornmann L, Wallon G, Ledin A. Does the Committee Peer Review Select the Best Applicants for Funding? An Investigation of the Selection Process for Two European Molecular Biology Organization Programmes. PLoS One. 2008;3(10):e3480.

Bornmann L, Daniel HD. The Usefulness of Peer Review for Selecting Manuscripts for Publication: A Utility Analysis Taking as an Example a High-Impact Journal. PLoS One. 2010 Jun 28;5(6):e11344.

Burroughs Wellcome Fund. Biomedical Sciences: Program Outcomes. 2010 Report.

Cabezas-Clavijo A, Robinson-García N, Escabias M, Jiménez-Contreras E. Reviewers’ Ratings and Bibliometric Indicators: Hand in Hand When Assessing Over Research Proposals? PLoS One. 2013 Jun 28;8(6):e68258.

Claveria LE, Guallar E, Cami J, Conde J, Pastor R, Ricoy JR, Rodriguez-Farre E, Ruiz-Palomo F, Munoz E. Does Peer Review Predict the Performance of Research Projects in Health Sciences? Scientometrics. 2000 Jan 1;47(1):11-23.

Cole S, Cole JR, Simon GA. Chance and Consensus in Peer Review. Science. 1981 Nov 20;214(4523):881-6.

Crespia GA, Geunab A. An Empirical Study of Scientific Production: A Cross Country Analysis, 1981-2002. Res Policy. 2008;37(4):565-79.

Danthi NS, Wu CO, DiMichele DM, Hoots WK, Lauer MS. Citation Impact of NHLBI R01 Grants Funded Through the American Recovery and Reinvestment Act as Compared to R01 Grants Funded Through a Standard Payline. Circ Res. 2015 Feb 27;116(5):784-8.

Danthi N, Wu CO, Shi P, Lauer M. Percentile Ranking and Citation Impact of a Large Cohort of National Heart, Lung, and Blood Institute-Funded Cardiovascular R01 Grants. Circ Res. 2014 Feb 14;114(4):600-6.

Doyle JM, Quinn K, Bodenstein YA, Wu CO, Danthi N, Lauer MS. Association of Percentile Ranking with Citation Impact and Productivity in a Large Cohort of de novo NIMH-funded R01 Grants. Mol Psychiatry. 2015 Jun 2.

Eloy JA, Svider PF, Kanumuri VV, Folbe AJ, Setzen M, Baredes S. Do AAO-HNSF CORE Grants Predict Future NIH Funding Success? Otolaryngol Head Neck Surg. 2014 May 20. [Epub ahead of print]

Escobar-Alvarez SN, Myers ER. The Doris Duke Clinical Scientist Development Award: Implications for Early-Career Physician Scientists. Acad Med. 2013 Nov;88(11):1740-6.

Fang D, Meyer RE. Effect of Two Howard Hughes Medical Institute Research Training Programs for Medical Students on the Likelihood of Pursuing Research Careers. Acad Med. 2003 Dec;78(12):1271-80.

Gallo SA et al. The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies. 2014 PLoS ONE 9(9): e106474. doi:10.1371/journal.pone.0106474

Goldbeck-Wood S. Evidence on Peer Review—Scientific Quality Control or Smokescreen? BMJ. 1999 January 2;318(7175):44-5.

Hodgson C. Evaluation of Cardiovascular Grant-in-Aid Applications by Peer Review: Influence of Internal and External Reviewers and Committees. Can J Cardiol. 1995 Nov;11(10):864-8.

Holbrook JB, Hrotic S. Blue Skies, Impacts, and Peer Review. RT. 2013 Jul;1(1):2013.

Ip EH, Wasserman R, Barkin S. Comparison of Intraclass Correlation Coefficient Estimates and Standard Errors Between Using Cross-Sectional and Repeated Measurement Data: The Safety Check Custer Randomized Trial. Contemp Clin Trials. 2011 Mar;32(2):225-32.

Jackson JL, Srinivasan M, Rea J, Fletcher KE, Kravitz RL. The Validity of Peer Review in a General Medicine Journal. PLoS One. 2011;6(7):e22475.

Jacob B, Lefgren L. The Impact of Research Grant Funding on Scientific Productivity. NBER Working Paper No. 13519. 2007.

Kaltman JR, Evans FJ, Danthi NS, Wu CO, DiMichele DM, Lauer MS. Prior Publication Productivity, Grant Percentile Ranking, and Topic-Normalized Citation Impact of NHLBI Cardiovascular R01 Grants. Circ Res. 2014 Sep 12;115(7):617-24.

Kotchen TA, Lindquist T, Miller Sostek A, Hoffmann R, Malik K, Stanfield B. Outcomes of National Institutes of Health Peer Review of Clinical Grant Applications. J Investig Med. 2006 Jan;54(1):13-9.

Lauer MS. Thought Exercises on Accountability and Performance Measures at the National Heart, Lung, and Blood Institute (NHLBI): An Invited Commentary for Circulation Research. Circ Res. 2011 Feb 18;108(4):405-9.

Laurence WF, Useche C, Laurance SG, Bradshaw CJA. Predicting Publication Success for Biologists. BioScience. 2013;63(10):817-23.

Li D, Agha L. Research Funding. Big Names or Big Ideas: Do Peer-Review Panels Select the Best Science Proposals? Science. 2015 Apr 24;348(6233):434-8.

Lindner MD, Nakamura RK. Examining the Predictive Validity of NIH Peer Review Scores. PLoS One. 2015 Jun 3;10(6):e0126938.

Marsh HW, Jayasinghe UW, Bond NW. Improving the Peer-Review Process for Grant Applications: Reliability, Validity, Bias, and Generalizability. Am Psychol. 2008 Apr;63(3):160-8.

Martin MR, Lindquist T, Kotchen TA. Why are Peer Review Outcomes Less Favorable for Clinical Science than for Basic Science Grant Applications? Am J Med. 2008 Jul;121(7):637-41.

Mason JL, Lei M, Faupel-Badger JM, Ginsburg EP, Seger YR, Dijoseph L, Schnell JD, Wiest JS. Outcome Evaluation of the National Cancer Institute Career Development Awards Program. J Cancer Educ. 2013 Mar;28(1):9-17.

Mervis J. Peering Into Peer Review. Science. 2014 Feb 7;343(6171):596-8.

Moed HF, Glanzel W, Schmoch U. Handbook of Quantitative Science and Technology Research: The Use of Publications and Patent Statistics in Studies of S&T Systems. New York, NY: Kluwer Academic Publishers, 2004.

Pion G, Ionescu-Pioggia M. Bridging Postdoctoral Training and a Faculty Position: Initial Outcomes of the Burroughs Wellcome Fund Career Awards in the Biomedical Siences. Acad Med. 2003 Feb;78(2):177-86.

Rinia EJ, Leeuwen Th N, Vuren HG, Raan AFJ. Comparative Analysis of a Set of Bibliometric Indicators and Central Peer Review Criteria: Evaluation of Condensed Matter Physics in the Netherlands. Research Policy. 1998 May;27(1):95-107.

Rinia EJ, Leeuwen Th N, Vuren HG, Raan AFJ. Influence of Interdisciplinarity on Peer-Review and Bibliometric Evaluations in Physics Research. Research Policy. 2001 Mar 1;30(3):357-61.

Schroter S, Tite L, Hutchings A, Black N. Differences in Review Quality and Recommendations for Publication Between Peer Reviewers Suggested by Authors or by Editors. JAMA. 2006 Jan 18;295(3):314-7.

Shashok K. Standardization vs Diversity: How Can We Push Peer Review Research Forward? MedGenMed. 2005 Feb 17;7(1):11.

Star Metrics. Science and Technology for America’s Reinvestment: Measuring the Effect of Research on Innovation, Competitiveness and Science.

Turner J. Best Practices in Merit Review: A Report to the US Department of Energy. 2010 Dec.

Vale RD. Evaluating How We Evaluate. Mol Biol Cell. 2012 Sep;23(17):3285-9.

Waltman L, van Eck NJ, van Leeuwen TN, Visser MS, van Raan AF. On the Correlation Between Bibliometric Indicators and Peer Review: Reply to Opthof and Leydesdorff. Scientometrics. 2011 Sep;88(3):1017-1022.

Yglesias E. Improving Peer Review in the Federal Government. Technol Innov. 2010 Mar;12(3):225-32.