AIBS Analysis of Scientific Peer Review Offers Insights into Research Productivity

Tuesday, Sep 2, 2014,

In a paper published today in the journal PLOS One, investigators with the American Institute of Biological Sciences report findings from an analysis of the research output from a series of biomedical research grants funded after undergoing a scientific peer review process. The results, reported in The Validation of Peer Review Through Research Impact Measures and the Implications for Funding Strategies, offer insights for future research on peer review and potential models for increasing research productivity.

“Some form of peer review is used at the majority of research granting organizations to determine the best research to fund,” said Dr. Joseph Travis, President of AIBS and a biologist at Florida State University. “Peer review makes a significant contribution to how billions of dollars in research funding from government and private sources are awarded,” said Travis, a coauthor of the study.

In recent years, this process has been questioned, particularly with regard to how well peer review predicts the ultimate impact of the funded research.

“We conducted a retrospective analysis of peer review and project output data for 2,063 projects from an eight year period. Of these, 227 were funded and we examined whether correlations exist among the assessment of scientific merit using a peer review system and the scientific output from this program,” said Dr. Steve Gallo, Technical Operations Manager for AIBS and the lead investigator on the study.

Citation impact, or the number of times a research paper is referenced by others, is a common way to assess research impact. Analysis revealed that peer review scores associated with individual applications were correlated with the total time-adjusted citation output of these funded projects.

Gallo states, “citation impact did not correlate with the amount of funds awarded per application or with the total annual programmatic budget.” The number of funded applications per year did correlate well with total annual citation impact, suggesting that improving funding success rates by reducing the size of awards may be one strategy to optimize the scientific impact of research program portfolios.

“This strategy must be weighed against the need for a balanced research portfolio and the inherently high costs of some kinds of research,” said Travis.

The relationship observed between peer review scores and publication output lays the groundwork for establishing a model system for future prospective testing of the validity of peer review formats and procedures.

“This is something AIBS is looking at now,” said Gallo.

AIBS SPARS Wins Contract to Support Effort for Commercialization of Medical Technology

Wednesday, Aug 13, 2014,

The American Institute of Biological Sciences (AIBS) Scientific Peer Advisory and Review Services (SPARS) division has won a contract with the University City Science Center (UCSC) to provide independent scientific merit review services in support of the UCSC QED Program, a funding and business development program for researchers among 21 academic institutions located in the mid-Atlantic region and clustered within and around Greater Philadelphia. The QED Program model catalyzes cooperation among institutional stakeholders who seek to accelerate the development of technologies in research organization laboratories and pull these technologies into the marketplace.

AIBS Expert to Participate in International Panel on Future of Peer Review

Monday, Jun 9, 2014,

Dr. Stephen Gallo will discuss factors influencing scientific peer review of grants

For release: June 9, 2014

Contact: Robert Gropp, AIBS, 202-628-1500 x 250

RESTON, VA - On June 10, 2014, Dr. Stephen Gallo of the American Institute of Biological Sciences (AIBS) will participate in a panel discussion as part of the 2014 Tri-Agency Peer Review Workshop to be held in Ottawa, Canada. The meeting, “Inside - Out: Perspectives on Maintaining High Quality Peer Review in Changing Times”, is sponsored by the Canadian Institutes of Health Research, Natural Sciences and Engineering Council, Social Sciences and Humanities Research Council, and the Canada Foundation for Innovation.

“We are pleased that Dr. Gallo is participating in this important discussion,” said Dr. Richard O’Grady, Executive Director of AIBS. “Peer review is central to the way we do science and how we maintain high quality standards. This discussion is an excellent forum to explore how to sustain this quality at a time when society is continuously considering how to be more efficient and to identify more cost effective ways to work. AIBS, which has a long and distinguished history of providing high quality peer review services, has been looking at these questions for a number of years.”

Indeed, Gallo, who is Technical Operations Manager for AIBS’ Scientific Peer Advisory and Review Services, has published findings from research he and AIBS colleagues have done to identify issues associated with remotely conducted panels.

Gallo will participate on a panel with Dr. Richard Nakamura, Director of the Center for Scientific Review at the U.S. National Institutes of Health, and Dr. Sophie Stevance, Assistant Professor at the University of Laval in Quebec.

“I am looking forward to this important and timely discussion,” said Gallo. “There is a growing recognition of the need to look at how we do peer review and how we can use teleconference or other technologies to reduce peer review panel costs. It is critical, however, that we understand the strengths and weaknesses of different peer review models. These are some of the reasons the AIBS peer review program has been evaluating different models.”

Study Explores Effects of Review Setting on Scientific Peer Review

Wednesday, Aug 7, 2013,

New research finds that using teleconference technology may not alter the outcomes of research grant review panels

For Release: August 7, 2013, 05:00 PM Eastern

Contact: Robert Gropp, 202-628-1500 x 250, rgropp@aibs.org

RESTON, VIRGINIA - Research findings published today in PLOS ONE report that the setting in which a scientific peer review panel evaluates grant applications does not necessarily impact the outcome of the review process. However, the research found that the average amount of discussion panelists engage in during the review is reduced. The investigation examined more than 1,600 grant application reviews coordinated by the American Institute of Biological Sciences Scientific Peer Advisory and Review Services (AIBS SPARS) on behalf of a federal agency over a four-year period.

The researchers compared two years when grant applications were reviewed using an in-person peer review panel to two years when panels were conducted via teleconference or videoconference.

Funding organizations routinely bring experts together to review research grant applications. A process known as scientific peer review, the goal of these panels is to identify the best research applications.

"There are no studies exploring whether the review setting significantly alters the quality of the peer review process," stated Dr. Stephen Gallo, the lead author of this study and Technical Operations Manager for AIBS SPARS.

"Our goal is always a reliable and high-quality peer-review process. It is important that we understand the strengths and weaknesses of different peer review methods," said Scott Glisson, Director of AIBS SPARS and an author of this study.

The findings appear in "Teleconference Versus Face-To-Face Scientific Peer Review of Grant Applications: Effects on Review Outcomes" published in PLOS ONE.

"Little difference was found in most of the review metrics between face-to-face and teleconference settings," said Gallo. Application scoring was only modestly affected and reviewers used the full scoring range regardless of review setting. The reviewer ratings were highly reliable in both settings.

Often, the greatest anticipated difference between in-person and teleconference panels is the amount of time allocated to discussing applications. This study found teleconference or videoconference panels allocated less time to application discussions than in-person panels.

"This is a first of its kind study that provides valuable data to help research program managers select appropriate models for conducting peer review," said Glisson.

More research is needed. "We should know whether the reduced amount of discussion and peripheral interactions that occur in a teleconference setting influence the final panel outcomes, and, ultimately the productivity of the research that is funded," said Gallo.

# # #

Follow AIBS SPARS on Twitter: @AIBS_SPARS.

- END -

2011: The Year in (peer) Review: The World and Home

Wednesday, Jan 11, 2012,

Looking back at 2011, AIBS supported the peer review of science that brought a spectrum of social and life sciences to bear on human burdens as ancient and seemingly intractable as malaria, cancer, and wound infection, and as contemporary as blast-induced trauma, the reintegration of returning warriors home, and designing wildlife-sensitive green energy infrastructure.

SPARS reviewed the military "neglected tropical diseases" research that, this year, culminated in the phase III clinical trials of the RST,S vaccine. The trials over 15,000 children under the age of five across seven African countries. All over the world, 800,000 people a year die from this ancient disease, and half the world's population is at risk. This year, RTS.S reduced the risk of clinical malaria in treated children by 56% and severe malaria by 47%, SPARS helped.

The science we reviewed this year took a world view, and SPARS supported the Philadelphia's QED program that works to move science out of the lab and into the marketplace. Because of our peer review support, a portable, radiation-free, breast-scanning device, about the size of a calculator, secured licensing to bring breast cancer screening to women in remote, poor, or underserved regions of the globe.

This year too, we hosted the review of the CDC's President's Emergency Plan for AIDS Relief, PEPFAR, the largest effort by any nation to combat a single disease internationally. SPARS peer review screened good science that aimed to develop effective combination anti-retroviral therapy, train 140,000 new health care workers, care for twelve million HIV-infected people and orphans, and prevent twelve million new infections worldwide.

The life and social science we reviewed this year was innovative and wildly diverse: harnessing novel strategies that worked within - rather than against - biological systems: some developed therapeutic predatory bacteria to combat the wound infection of trauma victims, some designed rugged prosthetic ankles that let people run if they wanted to with less pain, and science that helped count Golden Eagles in order to site wind energy stations that respects the hunting, breeding, and migration life history of raptors, their landscape, and their prey.

While the science we reviewed this year reflected the needs of humanity and the planet, it frankly reflected the needs of our nation. We reviewed biomedical and bioengineering science that addressed the physical and psychological injuries of wounded warriors, and social science that addressed the needs of returning warriors, their families, and society as a whole, as everyone reintegrates to home and work.

SPARS Helps Marine Biologists and Alaskan Natives Count Seals

Thursday, Apr 29, 2010,

SPARS Helps Marine Biologists and Native Alaskan Hunters Sample Alaska Harbor Seals

For Alaska's Ecosystem and Its Native Culture, Harbor Seals are Critical
One hundred and two people live in Tatitlek, Alaska, a traditional Alutiiq village in a forested coastal glen - the impassible Chugash Mountains, like snowy ramparts, at their backs and the vast Prince William Sound spreading out before them. Intimately linked to the mountain and sea that surround them, Tatitlek is much like the hundreds of native Alaskan coastal communities for whom a subsistence lifestyle is tradition, and for whom harbor seals (Phoca vitulina) play a central role in a community’s diet and, because of that, its culture.  

In 2008, of the 1,462 harbor seals harvested in the state of Alaska, Tatitlek harvested 141 of them. With one harbor seal supplying about 56 pounds of useable meat, fur, and oil, every person in Tatitlek, had 77.4 pounds - about a quarter pound of seal a day - 39% of their meat that year.

Not only are harbor seals important to native Alaskan people, they are both a key marine predator and an important prey animal, and they live in a nearly unbroken sweep of coast from Dixon Entrance south of Juneau, west more than a thousand miles curving down
the Aleutian Archipelago, then north to Kuskokwim Bay in the Bering Sea.

But, since the 1970s, throughout that wide distribution, the Alaskan harbor
seal population - which is divided into three "stocks" for wildlife management
purposes - has declined precipitously - west of the Gulf of Alaska,
populations have dropped by 50 to 90%, but the decline is a strange, erratic one
- populations are relatively stable in the southeast. The cause or causes are unknown – ideas include predation and/or competition from stellar sea lions or sharks, resource competition from recently returned hump-backed whales, pollution, and temperature increase.

SPARS Review: Genetic Research Finds Alaska Harbor Seal Stocks Too Few For
Healthy Management

In 2005, in response to a request by the National Marine Fisheries
Service (NMFS)/Alaska Native Harbor Seal Commission, the Southwest
Fisheries Science Center (SWFSC) submitted the report "The analysis of
population genetic structure in Alaskan harbor seals Phoca
vitulina
) as a framework for the identification of management
stocks" by O'Corry-Crowe et al. SPARS directed its peer review, assembling a panel of five reviewers - two anonymous,  all experts in conservation ecology, biostatistics, and marine mammal biology.

The research they reviewed sampled 881 harbor seals in 180 sites, evaluating population subdivisions and seal dispersal patterns using mitochondrial DNA. The data indicated that the three stock units used by the state to manage marine mammal populations inappropriately lump together up to 12 smaller, genetically distinct populations. They concluded that continuing to manage the population using stocks too large, risks the extirpation of the smaller, genetically distinct sub-groups, reducing genetic diversity -  that variation within species that allows populations to adapt to environmental changes, resist certain diseases, and avoid inbreeding.

...managing the population using stocks too large, risks the
extirpation of the smaller, genetically distinct sub-groups.


In addition to identifying genetically distinct populations, the O'Corry-Crowe report also suggested that the spatially erratic nature of the drops in harbor seal population were actually reflecting the unrecognized sub-population diversity as those sub-populations responded (well, or less well) to predation, prey abundance, resource competition, and pollution. The report stated that the population declines were occurring "on similar spatial scales to the genetic findings presented," and that declines did not correspond "to currently recognized stock structure".

In Rocky Inlets and Drifting Pack-Ice's Hard to Count Harbor Seals
After the two-day review in Juneau, the three on-site SPARS peer reviewers agreed with the investigators' conclusions generally, but recognized that sample coverage in data collection had been limited. It's tremendously difficult and expensive to sample or count harbor seals. They live most of their lives in the water, hauling out in rocky inlets and
pack-ice only to pup and molt, they are similar in appearance to other seals making them hard to discern from a distance, and they are scattered across enormous fields of floating ice that shift with ocean and wind currents.

Because it’s so hard to sample and monitor harbor seals, it was recognized that identification of meaningful management stocks should incorporate traditional Alaskan ecological knowledge. Native knowledge is frequently employed to interpret trends in harbor seal distribution, abundance, and foraging ecology relevant to stock structure. In fact, researchers stated that "active collaboration with Alaska native subsistence hunters and directed sampling is necessary" to cover difficult-to-sample areas.

Alaskan Traditional Ecological Knowledge Used to Monitor and Count
Harbor Seals
;

Scientists monitoring Alaska wildlife routinely collaborate with native hunters. Since 1997, indigenous hunters, the only group legally allowed to hunt harbor seals, have provided the Alaska Native Harbor Seal Commission with biosamples of their kills. Whiskers identify changes in diet. Blubber, kidney, and liver samples are tested for heavy metal contaminants, protecting both seal populations and public health. Hunters collect skin samples to monitor, as in the O’Corry-Crowe study, stock identity to understand how closely related harbor seals are in different parts of the state.

Not only do native hunters work with scientists to collect biological samples, but because a subsistence lifestyle requires exceptional skill and traditional knowledge of resources and the environment, native hunters are a valuable resource for qualitative information, guiding scientific enquiry.

Beth Mathews, Assistant Professor of Biology at the University of Alaska Southeast, who attended the O'Corry-Crowe review, gives an example in which "a hunter in Yakutat had observed harbor seal pups in the stomachs of sleeper sharks. That raised new ideas about population pressures on harbor seals, and inspired research into whether those pups were being eaten live or not, looking at sleeper sharks as either a predator or a scavenger.'

That Yakutat hunter had spent a lot of time out in the field, said Mathews, and had the expertise to notice it was something unusual. “Traditional knowledge was extremely valuable in bringing this important ecological interaction to the attention of scientists. The decline in harbor seals - 70% Glacier Bay since 1990 - cautiously suggested that sleeper sharks were responsible.”

Anne Hoover-Miller, scientist at the Alaska Sealife Center who also attended the review, agrees that, because "…native seal hunters are out in the world out there, when they're catching seals," they can provide valuable qualitative information to researchers. At community meetings, she said, "…the hunters were really concerned about the loss of animals, then we'd hear there were improvements in the north. Those are my first clues,” she says, “that a region is showing improvement or that they’re concerned about this."

"NMFS has a hard time surveying because of weather, but the local people can take advantage of the weather more easily and fill in the gaps." Sentinel programs, notably the Aleutian archipelago's Aleut Mammal Commission, and harvest monitoring program through the Alaska Native Harbor Seal Commission are extremely useful. “If native hunters think something is amiss, it probably is."

"Yakutat hunters have been concerned that the number of seals were going down, hunter insight can trigger hypothesis generation. If native hunters think something is amiss, it probably is.""

FAMRI-Funded Research Helps Harness Cancer-Killing Viruses

Wednesday, Apr 7, 2010,

SPARS Manages FAMRI Peer Review

The not-for-profit foundation, Flight Attendant Medical Research Institute (FAMRI; http://www.famri.org), sponsors scientific and medical research to combat the diseases caused by exposure to tobacco smoke.

Since 2001, the foundation has relied on SPARS to manage large- and small-conference peer review meetings that have marshaled the wide-ranging expertise of hundreds of reviewers and funded nearly 500 projects addressing the prevention, early detection, treatment, and cure of diseases and medical conditions caused by tobacco smoke.

Cancer Cells: Difficult to Find, Difficult to Fight.

Cancer cells, cells growing out of control, are diffused within healthy cells, especially at early stages. Because they are strewn and woven throughout healthy cells, they are difficult to target precisely. Because they are difficult to target, traditional anticancer therapies are blunt instruments: chemotherapy targets not just cancer, but fast-growing healthy cells like bone marrow, hair, and skin; radiation cannot always focus precisely, and lumpectomies may miss malignant or premalignant tissue.

But, in a series of elegant and innovative FAMRI-funded studies, Dr. Yuman Fong of Memorial Sloan-Kettering Cancer Center is harnessing the propensity of viruses to cause cell death. His lab is harnessing that propensity, and aiming it at cancer cells. Fong's lab is designing viruses that selectively infect tumor cells and can kill them within hours.

FAMRI Research Harnesses Viruses to Kill Cancer A virus's strategy is to capture a cell's replicating machinery, producing thousands of viruses until the infected cell bursts, or lyses, releasing the newly made viruses, and spreading viral infection that affects the cancer, but not the patient. A cancer cell is a virus's perfect target - its replicating machinery is out of control - it has no protective 'off' switch. Dr. Fong's laboratory focuses on engineering viruses (herpes simplex 2, adeno, vaccinia, myxoma, and vesicular stomatitis) that specifically infect and lyse cancer cells (oncolytic viruses), while sparing normal cells.

FAMRI Research Harnesses Viruses to Find Cancer

In a FAMRI-supported 2004 study, the investigators used a herpes simplex virus called NV1066 that they engineered to contain a gene for a protein called "green fluorescent protein" (GFP). They wanted to use the modified virus to infect early cancer cells and cancer cells in hard-to-reach tissues like nerves. It did just that, and only the cancer cells with the GFP glowed green.

These experiments showed that viruses can be used to improve early detection and identification of cancer, and to help guide surgical resection. "In animal and in human studies, they [the viruses] have been able to detect one tumor cell in a background of one million normal cells, a 50 times improvement over traditional cytology)," writes the PI.

FAMRI Research Designed for Global Application

This work has global ramifications. The investigators in Dr. Fong's lab point out that using this technique, the cancer cells that become virus-tagged and glow green under the appropriate light source can be seen by personnel with minimal training using available, inexpensive machines. Dr. Fong suggests that these qualities may make the techniques appropriate "even in rural areas of developing nations where cigarette smoking is still popular..." allowing detection of many types of cancers including lung, oral, pancreatic, and stomach, and could be immediately applicable worldwide.

Steadily building on the success of previous work, another FAMRI-supported study by Fong et al., in 2007, continued this work, with an added twist. They asked, "What cellular change in the process of malignant transformation allows the virus to selectively infect the cancer cell?"

Dr. Fong and colleagues have carried out a great deal of FAMRI-funded research; they have been working steadily since 2004 to develop oncolytic viral therapies, some of which are now entering Phase 1 clinical trials.

Blast Physics Peer Review Means Better Helmet Design

Tuesday, Mar 16, 2010,

Traumatic Brain Injury (TBI) is perhaps the defining injury of the US wars in Iraq and Afghanistan. The Military Health System diagnosed TBI in 43,779 patients between 2003 and 2007.

TBI victims suffer from a stunning range of functional brain changes both immediately after the trauma, and decades later. Memory, language, learning, emotion, and behavior are skewed. TBI can cause phantom sensation, and epilepsy; a brain injury event seems to increase the risk for Alzheimer's disease, Parkinson's, and other age-associated brain disorders.

As devastating to individuals, families, and society as TBI is, the biomechanical mechanisms describing what happens when a blast-wave meets a human head are not clearly understood. But, in 2007, SPARS recruited neurophysiologists and blast experts to review research that sought to describe how blast wave energy is transferred to a human head at different blast strengths and different head angles, addressing this knowledge gap.

Successfully funded, that research went on to explain mathematically that shock waves can be focused in the orbital cavities - the eye sockets - and the strength of the blast significantly increased by 10 times or more, "...focused like a parabolic mirror, like the headlights of a car," said the study PI.

The model showed that the blast hits the inner part of a soldier’s helmet, travels through the space between the head and helmet, then washes up and over to hit the back of the soldier’s head. With information directly from SPARS peer-reviewed research, the shape of helmets may be changed to mitigate this wave.

Further, associated research was able to use the original data to continue to describe the physiology of blast-induced brain injury - "...not just what happens in the instant of the blast impact," says the PI, "but what goes on in the brain after the blast for minutes, even hours." guiding immediate treatment and subsequent therapy.