AIBS Examines Method to Identify Scientific Communities

12/15/2020 4:39 PM UTC

AIBS, in collaboration with the NET ESolutions Corporation, published findings from an analysis of citations of scientific articles which introduced a meta-method for large-scale discovery of scientific communities of practice.

The idea of ‘Invisible Colleges,’ self-assembled groups of scientists with common scientific interests, has been around for several hundred years, yet the study of the behavior of such groups has been limited to small samples and case studies. In this work, a new bibliometric method is presented for identifying such groups. This method was validated with expert review of the thematic relatedness of clusters of articles.

This study is a first step in designing and testing a meta-method that could enable large-scale identification of communities of different sizes and types, based on different search criteria, data collection technique, choice of clustering algorithm, adjustable parameters, and cluster selection criteria. This method could be used in identifying expertise and/or conflicts of interest for peer review.


The manuscript, entitled “Finding Scientific Communities in Citation Graphs: Articles and Authors” was just published online in the Quantitative Science Studies journal and is open-access and available at

AIBS Investigates Effectiveness of Panel Discussion

05/19/2020 2:54 PM UTC

Often in the peer review of research grant proposals, panel discussion is used as a way to take advantage of a broader set of expertise and perspectives for making funding decisions. The American Institute of Biological Sciences (AIBS) has published findings exploring reviewer experiences with panel discussion and examining their perceived quality and effectiveness as well as their influence on scoring.

The results from this mixed-methods analysis, generated from a survey sent to over 13,000 scientists, suggest the majority of reviewers viewed panel discussion favorably in terms of participation, clarifying differing opinions, informing unassigned reviewers, chair facilitation, and even in selecting the best science. However, a substantial proportion of the respondents mentioned sources of bias in the review meeting and areas of poor facilitation during panel discussion. While there was a strong acknowledgement of the importance of the chair in addressing these areas, 32% of respondents did not find the chair of their most recent panel to have performed these roles effectively. It is likely that improving chair training in leadership and the management of discussion would improve the peer review process.

The manuscript, entitled “Grant reviewer perceptions of the quality, effectiveness, and influence of panel discussion” was just published online in the Journal of Research Integrity and Peer Review and is available via open access.

New AIBS Publication on Peer Reviewer Levels of Participation and Motivations

08/01/2019 4:50 PM UTC

AIBS published findings from an analysis of scientist participation levels in the grant peer review process, as well as their motivations to do so. These results, generated from a survey sent to over 13,000 scientists, show that while 76% of respondents participated in the peer review of research applications, an uneven distribution of participation was found across this sample, with a sub-set of reviewers shouldering higher review loads (the top 10% reviewing 3 times the amount of the bottom 40% of respondents). This sub-set was estimated to be close to maximum capacity in terms of review availability, highlighting concerns about the sustainability of the peer review system. However, most reviewer respondents indicated that participating in peer review has positively affected their careers, and that giving back to the scientific community is the most important motivation for reviewing.

The manuscript, entitled “The Participation and Motivations of Grant Peer Reviewers: A Comprehensive Survey” was just published online in the Journal for Science and Engineering Ethics and a pre-print version is available at bioRxiv.

An enhanced PDF link is here.

AIBS Client, Doris Duke Charitable Foundation, Publishes on Gender Bias in Research Funding

02/27/2019 3:37 PM UTC

The Doris Duke Charitable Foundation (DDCF), a client of the AIBS Scientific Peer Advisory and Review Services, has recently published a correspondence article examining potential sources of gender bias in grant selection processes and what can be done to improve gender equity.

Please see the article: Promoting gender equity in grant making: what can a funder do? The Lancet. Vol 393, Issue 10171, PE9-E11, February 09, 2019

New AIBS Research on the Participation and Motivations of Grant Peer Reviewers

11/29/2018 7:00 PM UTC

The American Institute of Biological Sciences (AIBS) has posted findings from an analysis of data from a survey of scientists on their self-reported levels of participation and motivations to review research funding applications.

The results, generated from a survey sent to over 13,000 scientists, suggest participation was unevenly distributed across this sample of respondents, with some reviewers working at nearly the maximum of their capacity. The most prevalent motivation was to give back to the scientific community and the most common reason to decline was lack of time. Interestingly, few suggested that expectations from the funding agency were a motivation to review.

Finally, most felt that review participation positively influenced their careers through improving grantsmanship and exposure to new scientific ideas. The manuscript, entitled “Participation and Motivations of Grant Peer Reviewers: A Comprehensive Survey of the Biomedical Research Community” was posted on the bioRxiv online archive in late November, 2018 and is now in the process of journal submission.

AIBS Presents at the Health Research Alliance Members Meeting

11/20/2018 4:21 PM UTC

AIBS presented results from recent retrospective studies at the Health Research Alliance (HRA) Members Meeting.

HRA is a collaborative member organization of nonprofit research funders committed to maximizing the impact of biomedical research to improve human health and represents over 75 nonprofit funders having invested nearly $14 billion in biomedical research and training.

AIBS presented data related to the efficiency and effectiveness of the peer review process, along with representatives from NIH and Academia.

A copy of the presentation can be viewed here.

Congratulations to Dr. Arnold, 2018 Nobel Prize in Chemistry Recipient

10/05/2018 4:14 PM UTC

The American Institute of Biological Sciences (AIBS) extends its congratulations to Dr. Frances H. Arnold, a recipient of the 2018 Nobel Prize in Chemistry. Dr. Arnold has previously participated on a peer review panel that AIBS organized on behalf of the Department of Defense to assess a funded research program.

Dr. Arnold is sharing the chemistry prize with with Dr. George P. Smith and Dr. Gregory P. Winter for the first directed evolution of enzymes. The research has led to “more environmentally friendly manufacturing of chemical substances such as pharmaceuticals and the production of renewable fuels.”

Dr. Arnold is the first female American to win this prize in Chemistry in its 117 year history.

AIBS Presents Research at NIH Meeting

09/25/2018 2:25 PM UTC

Dr. Stephen Gallo, Chief Scientist, AIBS presented “Expectations and Characteristics of the Peer Review of Grant Applications” at the NIH Center for Scientific Review’s Advisory Council meeting on September 24, 2018.

Validity of Peer Review Examined by AIBS

08/24/2018 2:50 PM UTC

The American Institute of Biological Sciences (AIBS) has just published a literature review summarizing results of empirical tests on the validity of peer review decisions using impact measures of investigator output.

These results were published as part of a research topic for the journal Frontiers in Research Metrics and Analytics. Of all of the results, less than half were US based and the majority focused on bibliometric measurements of applicant/project success. Only 25% used more than one type of metric.

Nevertheless, the vast majority of studies provided evidence for at least some level of predictive validity of review decisions, although many detected sizable type I and II errors. Moreover, many of the observed effects were small and several studies suggest a coarse power to discriminate poor proposals from better ones, but not amongst the top tier proposals or applicants.

The article can be accessed here.

Does Application Peer Review Disagreement Lead to Higher Citation Levels? AIBS Publication Investigates

08/24/2018 2:47 PM UTC

The American Institute of Biological Sciences (AIBS) has recently published findings in the journal F100 on research it conducted in collaboration with the Queensland University of Technology to assess whether contentious funding applications (possibly higher risk) resulted in a higher return on investment (higher reward). We examined the scoring from 227 funded applications and their eventual relative citation impact.

Statistical analysis revealed no association between relative citations and either of the two measures of disagreement, although as this is the first study to empirically examine this association, it would be useful to test this in other funding schemes and in larger sample sizes. Publication link:

New AIBS Publication on Perceptions of Risk Evaluation in Grant Peer Review by Applicants and Reviewers

02/26/2018 6:39 PM UTC

The American Institute of Biological Sciences (AIBS) has published findings from research it conducted on reviewer and applicant perceptions of criteria usage and risk evaluation in grant peer review.

These results, generated from a survey sent to over 13,000 scientists, suggest discrepancies between reviewers and applicants in terms of whether the reviews addressed innovation and risk; while most reviewers indicate these elements were essential and were factored into their ranking decisions, most applicants suggest these factors were not addressed in the feedback they received. Thus, these results suggest a potential source of bias in the review process.

As others have suggested, there may be different interpretations of excellence used by reviewers, as well as different risk tolerances; this is an area of future research. The manuscript, entitled “Risk evaluation in peer review of grant applications” was published online in late February, 2018 in the journal Environment Systems and Decisions (

Army Materiel Command Hall of Fame Inductee Key Work Peer Reviewed by AIBS SPARS

02/13/2018 8:41 PM UTC

In 2001, AIBS SPARS provided a key peer review for the work of Army Materiel Command Hall of Fame Inductee Dr. G. Richard Price. Price is a retired Army scientist whose research has been focused in the areas of noise and hearing. AIBS SPARS provided peer review on a model developed by Price and his team, the Auditory Hazard Assessment Algorithm for Humans. This is still the only method of assessing noise hazard for the entire range of impulses that are relevant to all of DOD. For more on Price and his remarkable career, click here.

Further Additions to AIBS SPARS Client Roster in 2017

12/14/2017 11:10 PM UTC

Throughout 2017, AIBS SPARS continued to expand the number of organizations utilizing our peer review support services. As announced previously, we’ve welcomed BioTechnical Communications, Inc., Alaska’s IDeA Network of Biomedical Research Excellence, and the BrightFocus Foundation as new clients over the past 12 months. Two other organizations that we’ve initiated partnerships with in 2017 are the Paralyzed Veterans of America and the Emergency Medicine Foundation.

Since 1946, Paralyzed Veterans of America (PVA) has been a leading advocate for quality health care, spinal cord research and education, veterans’ benefits, and civil rights for veterans and all people with disabilities. For over 40 years, the PVA Research and Education Foundations have funded a wide variety of research projects, programs and fellowships, as well as education initiatives and conferences.

The mission of the Emergency Medicine Foundation (EMF) is to promote education and research that develops the careers of emergency medicine researchers, improves patient care, and provides the basis for effective health policy. It was founded in 1972 by the American College of Emergency Physicians (ACEP). To date, EMF has awarded more than $16 million in research grants to advance emergency medicine science and health policy.

We’re very excited to be supporting the research efforts of these world-class organizations. AIBS provides a range of services to both organizations including creating customized web portals for submission materials (applications/progress reports, etc.) and facilitating/executing expert reviews for applications and reports through the use of our web-based system, SCORES.

By being a valued partner in science and providing critical infrastructure and information to decision makers, we are working to further the important goals and impact of these organizations.

AIBS SPARS New Client Update

08/08/2017 7:06 PM UTC

AIBS SPARS continues to focus on our organzational vision to advance the biological sciences and their applications to human welfare, and to foster and encourage research and education in the biological sciences, including the medical, environmental, and agricultural sciences. To this end we are working to partner with like-minded groups; providing independent peer review services and agile technology and program support.

Recently, we have executed a few new service contracts:

  • We are supporting BioTechnical Communications, Inc. (BTC) providing pre-submission proposal review services (preliminary peer review for applicants prior to formally submitting the proposal to funding agencies). The initial focus of the effort will be to provide these services for Small Business Innovation Research (SBIR) projects developed by researchers at historically black colleges and universities (HBCUs).

  • We are continuing our long standing support of the Kansas University Medical Center (KUMC) and their IDeA Network of Biomedical Research Excellence (K-INBRE) by providing peer review services for bridging grant applications.

AIBS: Informing Decisions that Improve Lives Around the World

08/07/2017 1:48 PM UTC

A recent New York Times article highlights the important role that science can play in international diplomacy and development. This article reports on the positive outcomes of programs to combat HIV/AIDS in Africa. These efforts contributed to better public health in Africa and, according to survey data, have enhanced the perception and reputation of the United States among individuals in the countries where these programs were conducted. This good will is nice, but it is also beneficial to efforts to combat terrorism, and tackle environmental and public health threats that do not respect geographic boundaries.

In 2011 and 2012, the US Centers for Disease Control and Prevention (CDC) tasked the American Institute of Biological Sciences (AIBS) with assessing the merit of research applications submitted in response to the President’s Emergency Plan for AIDS Relief (PEPFAR). In partnership with the CDC, AIBS convened peer review panels charged with identifying meritorious applications that would help to achieve the goals of primary prevention of HIV infection, improving the care and treatment of HIV/AIDS, strengthening the capacity of countries to collect and use surveillance data, and developing, validating and/or evaluating public health programs.

This review process brought together some of the world’s foremost experts on HIV/AIDS prevention and monitoring in Africa to select meritorious research applications. These reviews were accomplished using AIBS’ proprietary processes for conducting merit review of proposals. The objective of the reviews was to identify programs with demonstrated success in HIV/AIDS prevention and treatment that could be expanded to address this public health problem in Africa. The federal government used the recommendations from these reviews to inform its funding decisions.

AIBS has for decades been committed to promoting the use of the best available information to inform decisions. Our Scientific Peer Advisory and Review Services (SPARS) department works with clients to conduct merit (peer) review of research funding proposals, oversight of ongoing research programs.

Spotlight on New AIBS SPARS Clients

04/05/2017 7:38 PM UTC

Over the last several months, AIBS SPARS has begun partnerships with several new clients including the Avon Breast Cancer Crusade, the University of Alaska, and the BrightFocus Foundation.

As a valued partner in science, AIBS SPARS will support the research programs of these foundations by facilitating various aspects of their administration and evaluation processes. The ultimate goal of AIBS SPARS is provide optimal services that inform critical decisions for our clients.

For the Avon Breast Cancer Crusade (ABCC), AIBS SPARS is providing peer review services for breast cancer research applications submitted from seven geographic regions across the US (including Houston, Washington, DC, Boston, Chicago, San Francisco, Los Angeles, and New York). ABCC is focused on the search for new preventive strategies to address the growing number of breast cancer cases around the globe and increase our understanding of not only the causes of breast cancer, but also the development and treatment of metastatic disease. This is AIBS SPARS’ second year supporting ABCC in the review of breast cancer research applications.

For the University of Alaska, AIBS SPARS is providing independent peer review services for Alaska’s IDeA Network of Biomedical Research Excellence (AK INBRE) (specifically for the Faculty Pilot Research Grant program). The INBRE promotes the development, coordination and sharing of research resources and expertise that will expand the research opportunities and increase the number of competitive investigators in the IDeA-eligible states. Alaska’s INBRE theme focuses on interfaces among the environment, behavior, health, and disease in people and animals in Alaska. This work complements AIBS SPARS’ long standing support of the University of Kansas’ INBRE program.

For the BrightFocus Foundation, AIBS SPARS is providing peer review services for the foundation’s Special Opportunity Award applications, these are invite only submissions in areas of interest to the foundation. The BrightFocus Foundation awards grants for research on the causes, prevention, or treatment of Alzheimer’s disease, Glaucoma, and Macular Degeneration. The BrightFocus Foundation offers annual RFAs in each of these three research programs, as well as soliciting application submissions in these topic areas throughout the year from researchers for Special Opportunity Awards.

AIBS SPARS is excited to be partnering with and supporting these organizations.

AIBS Publications Inform Health Research Policy in Canada

02/27/2017 6:44 PM UTC

AIBS research and publications on the science of peer review have recently had some significant impact on research policy making in Canada.

The Governing Council of the Canadian Institutes of Health Research (CIHR) has issued a report from their international peer review expert panel which convened in mid-January 2017 to discuss reforms to the Canadian research funding process (initiated in 2009), which have lead to an erosion of trust between CIHR and its stakeholders (the report can be found here).

Many of these reforms were related to the peer review process and in this report, with the noted overall lack of evidence in the literature on these processes, AIBS publications were referenced heavily. From past AIBS research, evidence for the relative effectiveness of teleconference panels was cited. Our most recent publication was also referred to in the discussion about relative scientific expertise and bias, which has implications on how CIHR currently recruits reviewers. Our work about the detection of conflicts of interest was also mentioned.

Based on this literature analysis and the lessons learned from previous implementations, the panel suggests teleconference panels are still an appropriate and cost-efficient option, and suggests current expertise recruitment methods need to be altered to include specialists and generalists as well as more reliance on the international community of scientists.

New AIBS Publication on the Relationship Between Scientific Merit Score and Reviewer Expertise Score

11/04/2016 8:18 PM UTC

The American Institute of Biological Sciences (AIBS) has published findings from research it conducted on the relationship between the scientific merit scores of grant applications and the self-assessed reviewer expertise scores of the evaluators in an unblinded system. These results, which replicate those of a previous blinded study, indicate a linear, negative relationship, whereby reviewers with expertise “close” to the application tend to score more harshly.

This supports the idea that reviewers tend to focus on the negative, and with higher expertise comes the ability to glean more information which is, on average, focused on weaknesses. These results, which included the examination of reviewer and applicant demographics, also suggest that social networks and status could have subtle effects on the relationship between reviewer scoring and expertise. The manuscript, entitled “The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications” was published in late October in the journal Plos One (

This publication was subsequently highlighted in a news roundup article by The Scientist (

AIBS' Science of Peer Review Program Presentation

03/07/2016 4:35 PM UTC

AIBS’ Dr. Stephen Gallo will offer an Ignite presentation at the International Symposium on Science of Science, which will be held at the U.S. Library of Congress in Washington, DC, on 22-23 March 2016. This workshop brings together social, computational and natural scientists to discuss the science of science and innovation policy, and the potential for this area of scholarship to transform the practice of science, the redesign of institutions, and the practice of science policy.

The talk, entitled “The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications” will explore how scientists leverage their expertise in the review process and how evaluator and applicant demographics can modify a review process. Gallo is the Technical Operations Manager in support of the AIBS Scientific Peer Advisory and Review Services (SPARS) division.

Stay engaged, follow us on Twitter @AIBS_SPARS for news and information about peer review research and programs, @AIBS_Policy for science policy news and updates, @BioScienceAIBS for the latest updates from about upcoming articles and podcasts.

SPARS Welcomes New Clients in 2015

01/26/2016 4:26 PM UTC

In 2015, AIBS SPARS began peer review and support service contracts with two new clients: the Foundation for Physical Therapy and the Matthew Larson Foundation.

As a valued partner in science, AIBS SPARS will support the research programs of these foundations by facilitating aspects of their administration and evaluation processes. The ultimate goal of AIBS SPARS is provide optimal services that inform critical decisions for our clients.

For the Foundation for Physical Therapy (FPT), AIBS SPARS is supporting the application process for a number of award mechanisms for FPT research grants, fellowships, and scholarships (application submission websites, technical support, compliance checking). In addition, AIBS SPARS is supporting the FPT’s Scientific Review Committee’s evaluation of the submitted applications (confidentiality/conflict of interest agreements, expertise assessments, online review forms, discussion facilitation/management, and report preparation). Since 1979, FPT has awarded more than $14 million to launch and fund the research careers of more than 500 physical therapists around the country in scientific, clinical, and health services research.

For the Matthew Larson Foundation (MLF), AIBS SPARS is supporting the LOI/application submission and review process for MLF’s Medical Research Grant, which supports studies on the treatment and cure of pediatric brain and spinal tumors. AIBS SPARS is providing services for a LOI/application submission website, technical support, as well as supporting the Medical Advisory Committee and independent expertise reviews, including confidentiality/conflict of interest agreements, expertise assessment and assignments, online review forms, conferral facilitation, and report preparation. Since 2007, nearly $2 million have been provided by MLF to support many important research projects focused on the treatment and cure of pediatric brain and spinal tumors.

AIBS SPARS is excited to be partnering with and supporting the administration and oversight of important research programs from these two world-class foundations.

SPARS Webinars on Communication Setting/Group Dynamics and Conflict of Interest/Bias in Peer Review

10/06/2015 7:41 PM UTC

SPARS is proud to be participating in several webinars for AIBS’ Topics on Leadership in Biological Program. These events are free and open to the public, but pre-registartion is required to attend. To learn more about this initiative, please visit

The first webinar occurred on Thursday, August 27, 2015. SPARS Technical Operations Manager, Dr. Stephen Gallo, presented findings from recent research evaluating the group dynamics of face-to-face versus teleconference peer review panels to evaluate research proposals.

The second webinar will take place on Thursday, October 15, 2015 at 2pm Eastern. Dr. Stephen Gallo of AIBS and Dr. Carole Lee of the University of Washington will present findings from recent research and analyses evaluating the presence and impact of biases and conflicts of interest in peer review.

Recordings of these webinars and others can be viewed or downloaded: here.

New Study Improves Understanding of Best Practices in Peer-review of Research Proposals

09/29/2015 4:29 PM UTC

The American Institute of Biological Sciences (AIBS) has published findings from research it conducted on the relationship between panel discussion and scoring in teleconference and face-to-face scientific peer-review panels. The study, part of AIBS’ Science of Peer Review initiative, appears in the journal BMJ Open.

Peer-review is a process used by most governmental and many non-governmental scientific research programs in the United States to review, analyze, and identify the most promising research. Most experts agree that the peer-review has played a critical role in driving the nation’s R&D system, which has been the foundation for economic growth and advancements in security, health, and environmental stewardship throughout the 20th and 21st centuries. In 2014 alone, the U.S. government spent approximately $136 billion on R&D.

The study was conducted by AIBS to provide a better understanding of how the use of teleconferencing for grant peer-review panels may alter outcomes as compared to traditional face-to-face peer-review panel meetings.

“A better understanding of how setting or the use of technology influences the dynamics of a peer-review panel will enable the research evaluation community to make more informed funding decisions,” said Scott Glisson, AIBS Co-Interim Executive Director. “To fund the highest impact research we need to understand how factors that can influence the functioning of a peer-review panel may impact deliberations and ultimately funding decisions.”

This study considered the influences of the use of teleconferencing because funding agencies are often interested in using this technology to reduce costs and the time burden placed on panel members who travel to on-site panel meetings.

Previous research by AIBS found that discussion times were significantly shorter for teleconference settings as compared to face-to-face, but the influence of discussion on application scoring was unclear. This analysis compared proposal scores by reviewers before and after the peer-review meeting. The authors measured the magnitude and direction of score changes. Comparisons of these score shifts were made for face-to-face and teleconference settings, which provided insights into the effect of communication medium on the subsequent scoring patterns.

“Scoring shifts post-discussion were, on average, small in both settings. Discussion was important for at least 10% of applications, regardless of setting, with these applications moving over the threshold to receive funding or not,” said Dr. Stephen Gallo, an author of the study. “Small, but statistically significant differences in post-discussion scoring patterns were uncovered between settings, including a decrease in the magnitude of score shifts in the teleconference panels as compared to face-to-face. However, discussion time had little influence on the magnitude of these score shifts.”

Interestingly, panel discussion was found to often result in poorer scores when compared to the initial premeeting scores. In other words, review scores worsened after panels came together and discussed a proposal. This was true regardless of setting.

The subtle differences observed between settings were potentially due to reduced reviewer engagement in teleconferences. More research is needed to understand the extent of this phenomenon, as well as the psychology of decision-making, team performance, and persuasion to better elucidate the effects of peer-review panel setting.

AIBS Study Investigates Conflict of Interest in Scientific Peer Review: Frequency, Type, and Detection Methods

03/06/2015 8:27 PM UTC

New research from the American Institute of Biological Sciences (AIBS) found that peer review managers play an important role in identifying potential conflicts of interest (COI) in biomedical research grant peer reviews. The study, Frequency and Type of Conflicts of Interest in the Peer Review of Basic Biomedical Research Funding Applications: Self-Reporting Versus Manual Detection, was published in the journal Science and Engineering Ethics.

Peer review is the widely used process by which panels of experts evaluate research proposals to help funders identify the best research to fund. A cornerstone of the process is the integrity of the review panel, which includes a fair and non-conflicted evaluation of the proposed research. Despite this and the widespread belief that peer review is a foundation upon which U.S. science is built, few studies of the peer review process have actually been conducted.

“Peer review is so central to the way we do science, it is important that we study the process. With good data, we can ensure the vibrancy of peer review and develop models and best practices that promote the integrity of peer review,” said Dr. Stephen Gallo, AIBS Technical Operations Manager and the lead author of the study.

This research is the most recent study from AIBS, which has conducted peer review of research proposals on behalf of government and non-government research funders for over 50 years.

To inform the future of peer review, Gallo and colleagues conducted a retrospective analysis of COI data from peer review panels that evaluated 282 biomedical research applications. The overall ‘conflicted-ness’ of these panels was significantly lower than that reported for regulatory review panels, which have been studied by others.

The AIBS study found that 35 percent of conflicts were self-reported by review panel members. Importantly, peer review panel managers identified 65 percent of conflicts.

“The people who organize panels play an important role in identifying conflicts of interest,” said Gallo.

Overall, this study suggests that the scientific community should dedicate some energy to improving COI reporting and detection methods. In light of increasing demands on reviewers’ time, administrators will also need to make this process as efficient as possible while maintaining the highest ethical and review standards.

AIBS Analysis of Scientific Peer Review Offers Insights into Research Productivity

09/02/2014 6:30 PM UTC

In a paper published today in the journal PLOS One, investigators with the American Institute of Biological Sciences report findings from an analysis of the research output from a series of biomedical research grants funded after undergoing a scientific peer review process. The results, reported in The Validation of Peer Review Through Research Impact Measures and the Implications for Funding Strategies, offer insights for future research on peer review and potential models for increasing research productivity.

“Some form of peer review is used at the majority of research granting organizations to determine the best research to fund,” said Dr. Joseph Travis, President of AIBS and a biologist at Florida State University. “Peer review makes a significant contribution to how billions of dollars in research funding from government and private sources are awarded,” said Travis, a coauthor of the study.

In recent years, this process has been questioned, particularly with regard to how well peer review predicts the ultimate impact of the funded research.

“We conducted a retrospective analysis of peer review and project output data for 2,063 projects from an eight year period. Of these, 227 were funded and we examined whether correlations exist among the assessment of scientific merit using a peer review system and the scientific output from this program,” said Dr. Steve Gallo, Technical Operations Manager for AIBS and the lead investigator on the study.

Citation impact, or the number of times a research paper is referenced by others, is a common way to assess research impact. Analysis revealed that peer review scores associated with individual applications were correlated with the total time-adjusted citation output of these funded projects.

Gallo states, “citation impact did not correlate with the amount of funds awarded per application or with the total annual programmatic budget.” The number of funded applications per year did correlate well with total annual citation impact, suggesting that improving funding success rates by reducing the size of awards may be one strategy to optimize the scientific impact of research program portfolios.

“This strategy must be weighed against the need for a balanced research portfolio and the inherently high costs of some kinds of research,” said Travis.

The relationship observed between peer review scores and publication output lays the groundwork for establishing a model system for future prospective testing of the validity of peer review formats and procedures.

“This is something AIBS is looking at now,” said Gallo.

SPARS Wins Contract to Support Effort for Commercialization of Medical Technology

08/13/2014 4:26 PM UTC

The American Institute of Biological Sciences (AIBS) Scientific Peer Advisory and Review Services (SPARS) division has won a contract with the University City Science Center (UCSC) to provide independent scientific merit review services in support of the UCSC QED Program, a funding and business development program for researchers among 21 academic institutions located in the mid-Atlantic region and clustered within and around Greater Philadelphia.

The QED Program model catalyzes cooperation among institutional stakeholders who seek to accelerate the development of technologies in research organization laboratories and pull these technologies into the marketplace.

AIBS Expert to Participate in International Panel on Future of Peer Review

06/09/2014 4:26 PM UTC

On June 10, 2014, Dr. Stephen Gallo of the American Institute of Biological Sciences (AIBS) will participate in a panel discussion as part of the 2014 Tri-Agency Peer Review Workshop to be held in Ottawa, Canada. The meeting, “Inside - Out: Perspectives on Maintaining High Quality Peer Review in Changing Times”, is sponsored by the Canadian Institutes of Health Research, Natural Sciences and Engineering Council, Social Sciences and Humanities Research Council, and the Canada Foundation for Innovation.

“We are pleased that Dr. Gallo is participating in this important discussion,” said Dr. Richard O’Grady, Executive Director of AIBS. “Peer review is central to the way we do science and how we maintain high quality standards. This discussion is an excellent forum to explore how to sustain this quality at a time when society is continuously considering how to be more efficient and to identify more cost effective ways to work. AIBS, which has a long and distinguished history of providing high quality peer review services, has been looking at these questions for a number of years.”

Indeed, Gallo, who is Technical Operations Manager for AIBS’ Scientific Peer Advisory and Review Services, has published findings from research he and AIBS colleagues have done to identify issues associated with remotely conducted panels.

Gallo will participate on a panel with Dr. Richard Nakamura, Director of the Center for Scientific Review at the U.S. National Institutes of Health, and Dr. Sophie Stevance, Assistant Professor at the University of Laval in Quebec.

“I am looking forward to this important and timely discussion,” said Gallo. “There is a growing recognition of the need to look at how we do peer review and how we can use teleconference or other technologies to reduce peer review panel costs. It is critical, however, that we understand the strengths and weaknesses of different peer review models. These are some of the reasons the AIBS peer review program has been evaluating different models.”

Study Explores Effects of Review Setting on Scientific Peer Review

08/07/2013 10:26 PM UTC

Research findings published today in PLOS ONE report that the setting in which a scientific peer review panel evaluates grant applications does not necessarily impact the outcome of the review process. However, the research found that the average amount of discussion panelists engage in during the review is reduced. The investigation examined more than 1,600 grant application reviews coordinated by the American Institute of Biological Sciences Scientific Peer Advisory and Review Services (AIBS SPARS) on behalf of a federal agency over a four-year period.

The researchers compared two years when grant applications were reviewed using an in-person peer review panel to two years when panels were conducted via teleconference or videoconference.

Funding organizations routinely bring experts together to review research grant applications. A process known as scientific peer review, the goal of these panels is to identify the best research applications.

“There are no studies exploring whether the review setting significantly alters the quality of the peer review process,” stated Dr. Stephen Gallo, the lead author of this study and Technical Operations Manager for AIBS SPARS.

“Our goal is always a reliable and high-quality peer-review process. It is important that we understand the strengths and weaknesses of different peer review methods,” said Scott Glisson, Director of AIBS SPARS and an author of this study.

The findings appear in “Teleconference Versus Face-To-Face Scientific Peer Review of Grant Applications: Effects on Review Outcomes published in PLOS ONE.

“Little difference was found in most of the review metrics between face-to-face and teleconference settings,” said Gallo. Application scoring was only modestly affected and reviewers used the full scoring range regardless of review setting. The reviewer ratings were highly reliable in both settings.

Often, the greatest anticipated difference between in-person and teleconference panels is the amount of time allocated to discussing applications. This study found teleconference or videoconference panels allocated less time to application discussions than in-person panels.

“This is a first of its kind study that provides valuable data to help research program managers select appropriate models for conducting peer review,” said Glisson.

More research is needed. “We should know whether the reduced amount of discussion and peripheral interactions that occur in a teleconference setting influence the final panel outcomes, and, ultimately the productivity of the research that is funded,” said Gallo.

2011: The Year in (peer) Review: The World and Home

01/12/2012 5:21 AM UTC

Looking back at 2011, AIBS supported the peer review of science that brought a spectrum of social and life sciences to bear on human burdens as ancient and seemingly intractable as malaria, cancer, and wound infection, and as contemporary as blast-induced trauma, the reintegration of returning warriors home, and designing wildlife-sensitive green energy infrastructure.

SPARS reviewed the military “neglected tropical diseases” research that, this year, culminated in the phase III clinical trials of the RST,S vaccine. The trials over 15,000 children under the age of five across seven African countries. All over the world, 800,000 people a year die from this ancient disease, and half the world’s population is at risk. This year, RTS.S reduced the risk of clinical malaria in treated children by 56% and severe malaria by 47%, SPARS helped.

The science we reviewed this year took a world view, and SPARS supported the Philadelphia’s QED program that works to move science out of the lab and into the marketplace. Because of our peer review support, a portable, radiation-free, breast-scanning device, about the size of a calculator, secured licensing to bring breast cancer screening to women in remote, poor, or underserved regions of the globe.

This year too, we hosted the review of the CDC’s President’s Emergency Plan for AIDS Relief, PEPFAR, the largest effort by any nation to combat a single disease internationally. SPARS peer review screened good science that aimed to develop effective combination anti-retroviral therapy, train 140,000 new health care workers, care for twelve million HIV-infected people and orphans, and prevent twelve million new infections worldwide.

The life and social science we reviewed this year was innovative and wildly diverse: harnessing novel strategies that worked within - rather than against - biological systems: some developed therapeutic predatory bacteria to combat the wound infection of trauma victims, some designed rugged prosthetic ankles that let people run if they wanted to with less pain, and science that helped count Golden Eagles in order to site wind energy stations that respects the hunting, breeding, and migration life history of raptors, their landscape, and their prey.

While the science we reviewed this year reflected the needs of humanity and the planet, it frankly reflected the needs of our nation. We reviewed biomedical and bioengineering science that addressed the physical and psychological injuries of wounded warriors, and social science that addressed the needs of returning warriors, their families, and society as a whole, as everyone reintegrates to home and work.

SPARS Helps Marine Biologists and Alaskan Natives Count Seals

04/29/2010 7:37 PM UTC

SPARS Helps Marine Biologists and Native Alaskan Hunters Sample Alaska Harbor Seals

For Alaska’s Ecosystem and Its Native Culture, Harbor Seals are Critical

One hundred and two people live in Tatitlek, Alaska, a traditional Alutiiq village in a forested coastal glen - the impassible Chugash Mountains, like snowy ramparts, at their backs and the vast Prince William Sound spreading out before them. Intimately linked to the mountain and sea that surround them, Tatitlek is much like the hundreds of native Alaskan coastal communities for whom a subsistence lifestyle is tradition, and for whom harbor seals (Phoca vitulina) play a central role in a community’s diet and, because of that, its culture.

In 2008, of the 1,462 harbor seals harvested in the state of Alaska, Tatitlek harvested 141 of them. With one harbor seal supplying about 56 pounds of useable meat, fur, and oil, every person in Tatitlek, had 77.4 pounds - about a quarter pound of seal a day - 39% of their meat that year.

Not only are harbor seals important to native Alaskan people, they are both a key marine predator and an important prey animal, and they live in a nearly unbroken sweep of coast from Dixon Entrance south of Juneau, west more than a thousand miles curving down the Aleutian Archipelago, then north to Kuskokwim Bay in the Bering Sea.

But, since the 1970s, throughout that wide distribution, the Alaskan harbor seal population - which is divided into three “stocks” for wildlife management purposes - has declined precipitously - west of the Gulf of Alaska, populations have dropped by 50 to 90%, but the decline is a strange, erratic one - populations are relatively stable in the southeast. The cause or causes are unknown – ideas include predation and/or competition from stellar sea lions or sharks, resource competition from recently returned hump-backed whales, pollution, and temperature increase.

SPARS Review: Genetic Research Finds Alaska Harbor Seal Stocks Too Few For Healthy Management

In 2005, in response to a request by the National Marine Fisheries Service (NMFS)/Alaska Native Harbor Seal Commission, the Southwest Fisheries Science Center (SWFSC) submitted the report “The analysis of population genetic structure in Alaskan harbor seals Phoca vitulina) as a framework for the identification of management stocks” by O’Corry-Crowe et al. SPARS directed its peer review, assembling a panel of five reviewers - two anonymous,  all experts in conservation ecology, biostatistics, and marine mammal biology.

The research they reviewed sampled 881 harbor seals in 180 sites, evaluating population subdivisions and seal dispersal patterns using mitochondrial DNA. The data indicated that the three stock units used by the state to manage marine mammal populations inappropriately lump together up to 12 smaller, genetically distinct populations. They concluded that continuing to manage the population using stocks too large, risks the extirpation of the smaller, genetically distinct sub-groups, reducing genetic diversity -  that variation within species that allows populations to adapt to environmental changes, resist certain diseases, and avoid inbreeding.

…managing the population using stocks too large, risks the extirpation of the smaller, genetically distinct sub-groups.

In addition to identifying genetically distinct populations, the O’Corry-Crowe report also suggested that the spatially erratic nature of the drops in harbor seal population were actually reflecting the unrecognized sub-population diversity as those sub-populations responded (well, or less well) to predation, prey abundance, resource competition, and pollution. The report stated that the population declines were occurring “on similar spatial scales to the genetic findings presented,” and that declines did not correspond “to currently recognized stock structure”.

In Rocky Inlets and Drifting Pack-Ice’s Hard to Count Harbor Seals

After the two-day review in Juneau, the three on-site SPARS peer reviewers agreed with the investigators’ conclusions generally, but recognized that sample coverage in data collection had been limited. It’s tremendously difficult and expensive to sample or count harbor seals. They live most of their lives in the water, hauling out in rocky inlets and pack-ice only to pup and molt, they are similar in appearance to other seals making them hard to discern from a distance, and they are scattered across enormous fields of floating ice that shift with ocean and wind currents.

Because it’s so hard to sample and monitor harbor seals, it was recognized that identification of meaningful management stocks should incorporate traditional Alaskan ecological knowledge. Native knowledge is frequently employed to interpret trends in harbor seal distribution, abundance, and foraging ecology relevant to stock structure. In fact, researchers stated that “active collaboration with Alaska native subsistence hunters and directed sampling is necessary” to cover difficult-to-sample areas.

Alaskan Traditional Ecological Knowledge Used to Monitor and Count Harbor Seals

Scientists monitoring Alaska wildlife routinely collaborate with native hunters. Since 1997, indigenous hunters, the only group legally allowed to hunt harbor seals, have provided the Alaska Native Harbor Seal Commission with biosamples of their kills. Whiskers identify changes in diet. Blubber, kidney, and liver samples are tested for heavy metal contaminants, protecting both seal populations and public health. Hunters collect skin samples to monitor, as in the O’Corry-Crowe study, stock identity to understand how closely related harbor seals are in different parts of the state.

Not only do native hunters work with scientists to collect biological samples, but because a subsistence lifestyle requires exceptional skill and traditional knowledge of resources and the environment, native hunters are a valuable resource for qualitative information, guiding scientific enquiry.

Beth Mathews, Assistant Professor of Biology at the University of Alaska Southeast, who attended the O’Corry-Crowe review, gives an example in which “a hunter in Yakutat had observed harbor seal pups in the stomachs of sleeper sharks. That raised new ideas about population pressures on harbor seals, and inspired research into whether those pups were being eaten live or not, looking at sleeper sharks as either a predator or a scavenger.’

That Yakutat hunter had spent a lot of time out in the field, said Mathews, and had the expertise to notice it was something unusual. “Traditional knowledge was extremely valuable in bringing this important ecological interaction to the attention of scientists. The decline in harbor seals - 70% Glacier Bay since 1990 - cautiously suggested that sleeper sharks were responsible.”

Anne Hoover-Miller, scientist at the Alaska Sealife Center who also attended the review, agrees that, because “…native seal hunters are out in the world out there, when they’re catching seals,” they can provide valuable qualitative information to researchers. At community meetings, she said, “…the hunters were really concerned about the loss of animals, then we’d hear there were improvements in the north. Those are my first clues,” she says, “that a region is showing improvement or that they’re concerned about this.”

“NMFS has a hard time surveying because of weather, but the local people can take advantage of the weather more easily and fill in the gaps.” Sentinel programs, notably the Aleutian archipelago’s Aleut Mammal Commission, and harvest monitoring program through the Alaska Native Harbor Seal Commission are extremely useful. “If native hunters think something is amiss, it probably is.”

“Yakutat hunters have been concerned that the number of seals were going down, hunter insight can trigger hypothesis generation. If native hunters think something is amiss, it probably is.”

FAMRI-Funded Research Helps Harness Cancer-Killing Viruses

04/08/2010 1:46 AM UTC

SPARS Manages FAMRI Peer Review

The not-for-profit foundation, Flight Attendant Medical Research Institute (FAMRI;, sponsors scientific and medical research to combat the diseases caused by exposure to tobacco smoke.

Since 2001, the foundation has relied on SPARS to manage large- and small-conference peer review meetings that have marshaled the wide-ranging expertise of hundreds of reviewers and funded nearly 500 projects addressing the prevention, early detection, treatment, and cure of diseases and medical conditions caused by tobacco smoke.

Cancer Cells: Difficult to Find, Difficult to Fight.

Cancer cells, cells growing out of control, are diffused within healthy cells, especially at early stages. Because they are strewn and woven throughout healthy cells, they are difficult to target precisely. Because they are difficult to target, traditional anticancer therapies are blunt instruments: chemotherapy targets not just cancer, but fast-growing healthy cells like bone marrow, hair, and skin; radiation cannot always focus precisely, and lumpectomies may miss malignant or premalignant tissue.

But, in a series of elegant and innovative FAMRI-funded studies, Dr. Yuman Fong of Memorial Sloan-Kettering Cancer Center is harnessing the propensity of viruses to cause cell death. His lab is harnessing that propensity, and aiming it at cancer cells. Fong’s lab is designing viruses that selectively infect tumor cells and can kill them within hours.

FAMRI Research Harnesses Viruses to Kill Cancer

A virus’s strategy is to capture a cell’s replicating machinery, producing thousands of viruses until the infected cell bursts, or lyses, releasing the newly made viruses, and spreading viral infection that affects the cancer, but not the patient. A cancer cell is a virus’s perfect target - its replicating machinery is out of control - it has no protective ‘off’ switch. Dr. Fong’s laboratory focuses on engineering viruses (herpes simplex 2, adeno, vaccinia, myxoma, and vesicular stomatitis) that specifically infect and lyse cancer cells (oncolytic viruses), while sparing normal cells.

FAMRI Research Harnesses Viruses to Find Cancer

In a FAMRI-supported 2004 study, the investigators used a herpes simplex virus called NV1066 that they engineered to contain a gene for a protein called “green fluorescent protein” (GFP). They wanted to use the modified virus to infect early cancer cells and cancer cells in hard-to-reach tissues like nerves. It did just that, and only the cancer cells with the GFP glowed green.

These experiments showed that viruses can be used to improve early detection and identification of cancer, and to help guide surgical resection. “In animal and in human studies, they [the viruses] have been able to detect one tumor cell in a background of one million normal cells, a 50 times improvement over traditional cytology),” writes the PI.

FAMRI Research Designed for Global Application

This work has global ramifications. The investigators in Dr. Fong’s lab point out that using this technique, the cancer cells that become virus-tagged and glow green under the appropriate light source can be seen by personnel with minimal training using available, inexpensive machines. Dr. Fong suggests that these qualities may make the techniques appropriate “even in rural areas of developing nations where cigarette smoking is still popular…” allowing detection of many types of cancers including lung, oral, pancreatic, and stomach, and could be immediately applicable worldwide.

Steadily building on the success of previous work, another FAMRI-supported study by Fong et al., in 2007, continued this work, with an added twist. They asked, “What cellular change in the process of malignant transformation allows the virus to selectively infect the cancer cell?”

Dr. Fong and colleagues have carried out a great deal of FAMRI-funded research; they have been working steadily since 2004 to develop oncolytic viral therapies, some of which are now entering Phase 1 clinical trials.

Blast Physics Peer Review Means Better Helmet Design

03/17/2010 2:00 AM UTC

Traumatic Brain Injury (TBI) is perhaps the defining injury of the US wars in Iraq and Afghanistan. The Military Health System diagnosed TBI in 43,779 patients between 2003 and 2007.

TBI victims suffer from a stunning range of functional brain changes both immediately after the trauma, and decades later. Memory, language, learning, emotion, and behavior are skewed. TBI can cause phantom sensation, and epilepsy; a brain injury event seems to increase the risk for Alzheimer’s disease, Parkinson’s, and other age-associated brain disorders.

As devastating to individuals, families, and society as TBI is, the biomechanical mechanisms describing what happens when a blast-wave meets a human head are not clearly understood. But, in 2007, SPARS recruited neurophysiologists and blast experts to review research that sought to describe how blast wave energy is transferred to a human head at different blast strengths and different head angles, addressing this knowledge gap.

Successfully funded, that research went on to explain mathematically that shock waves can be focused in the orbital cavities - the eye sockets - and the strength of the blast significantly increased by 10 times or more, “…focused like a parabolic mirror, like the headlights of a car,” said the study PI.

The model showed that the blast hits the inner part of a soldier’s helmet, travels through the space between the head and helmet, then washes up and over to hit the back of the soldier’s head. With information directly from SPARS peer-reviewed research, the shape of helmets may be changed to mitigate this wave.

Further, associated research was able to use the original data to continue to describe the physiology of blast-induced brain injury - “…not just what happens in the instant of the blast impact,” says the PI, “but what goes on in the brain after the blast for minutes, even hours.” guiding immediate treatment and subsequent therapy.