Friday, Nov 4, 2016,
The American Institute of Biological Sciences (AIBS) has published findings from research it conducted on the relationship between the scientific merit scores of grant applications and the self-assessed reviewer expertise scores of the evaluators in an unblinded system. These results, which replicate those of a previous blinded study, indicate a linear, negative relationship, whereby reviewers with expertise “close” to the application tend to score more harshly.
This supports the idea that reviewers tend to focus on the negative, and with higher expertise comes the ability to glean more information which is, on average, focused on weaknesses. These results, which included the examination of reviewer and applicant demographics, also suggest that social networks and status could have subtle effects on the relationship between reviewer scoring and expertise. The manuscript, entitled “The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications” was published in late October in the journal Plos One (http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0165147).
This publication was subsequently highlighted in a news roundup article by The Scientist (www.the-scientist.com/?articles.view/articleNo/47428/title/Tough-Crowd/).
Monday, Mar 7, 2016,
AIBS’ Dr. Stephen Gallo will offer an Ignite presentation at the International Symposium on Science of Science, which will be held at the U.S. Library of Congress in Washington, DC, on 22-23 March 2016. This workshop brings together social, computational and natural scientists to discuss the science of science and innovation policy, and the potential for this area of scholarship to transform the practice of science, the redesign of institutions, and the practice of science policy.
The talk, entitled “The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications” will explore how scientists leverage their expertise in the review process and how evaluator and applicant demographics can modify a review process. Gallo is the Technical Operations Manager in support of the AIBS Scientific Peer Advisory and Review Services (SPARS) division.
Stay engaged, follow us on Twitter @AIBS_SPARS for news and information about peer review research and programs, @AIBS_Policy for science policy news and updates, @BioScienceAIBS for the latest updates from about upcoming articles and podcasts.
Tuesday, Jan 26, 2016,
As a valued partner in science, AIBS SPARS will support the research programs of these foundations by facilitating aspects of their administration and evaluation processes. The ultimate goal of AIBS SPARS is provide optimal services that inform critical decisions for our clients.
For the Foundation for Physical Therapy (FPT), AIBS SPARS is supporting the application process for a number of award mechanisms for FPT research grants, fellowships, and scholarships (application submission websites, technical support, compliance checking). In addition, AIBS SPARS is supporting the FPT’s Scientific Review Committee’s evaluation of the submitted applications (confidentiality/conflict of interest agreements, expertise assessments, online review forms, discussion facilitation/management, and report preparation). Since 1979, FPT has awarded more than $14 million to launch and fund the research careers of more than 500 physical therapists around the country in scientific, clinical, and health services research.
For the Matthew Larson Foundation (MLF), AIBS SPARS is supporting the LOI/application submission and review process for MLF’s Medical Research Grant, which supports studies on the treatment and cure of pediatric brain and spinal tumors. AIBS SPARS is providing services for a LOI/application submission website, technical support, as well as supporting the Medical Advisory Committee and independent expertise reviews, including confidentiality/conflict of interest agreements, expertise assessment and assignments, online review forms, conferral facilitation, and report preparation. Since 2007, nearly $2 million have been provided by MLF to support many important research projects focused on the treatment and cure of pediatric brain and spinal tumors.
AIBS SPARS is excited to be partnering with and supporting the administration and oversight of important research programs from these two world-class foundations.
Tuesday, Oct 6, 2015,
SPARS is proud to be participating in several webinars for AIBS’ Topics on Leadership in Biological Program. These events are free and open to the public, but pre-registartion is required to attend. To learn more about this initiative, please visit http://www.aibs.org/events/latest_events.html
The first webinar occurred on Thursday, August 27, 2015. SPARS Technical Operations Manager, Dr. Stephen Gallo, presented findings from recent research evaluating the group dynamics of face-to-face versus teleconference peer review panels to evaluate research proposals.
The second webinar will take place on Thursday, October 15, 2015 at 2pm Eastern. Dr. Stephen Gallo of AIBS and Dr. Carole Lee of the University of Washington will present findings from recent research and analyses evaluating the presence and impact of biases and conflicts of interest in peer review.
Recordings of these webinars and others can be viewed or downloaded here: https://spars.aibs.org/webinars.html
Tuesday, Sep 29, 2015,
The American Institute of Biological Sciences (AIBS) has published findings from research it conducted on the relationship between panel discussion and scoring in teleconference and face-to-face scientific peer-review panels. The study, part of AIBS' Science of Peer Review initiative, appears in the journal BMJ Open.
Peer-review is a process used by most governmental and many non-governmental scientific research programs in the United States to review, analyze, and identify the most promising research. Most experts agree that the peer-review has played a critical role in driving the nation's R&D system, which has been the foundation for economic growth and advancements in security, health, and environmental stewardship throughout the 20th and 21st centuries. In 2014 alone, the U.S. government spent approximately $136 billion on R&D.
The study was conducted by AIBS to provide a better understanding of how the use of teleconferencing for grant peer-review panels may alter outcomes as compared to traditional face-to-face peer-review panel meetings.
"A better understanding of how setting or the use of technology influences the dynamics of a peer-review panel will enable the research evaluation community to make more informed funding decisions," said Scott Glisson, AIBS Co-Interim Executive Director. "To fund the highest impact research we need to understand how factors that can influence the functioning of a peer-review panel may impact deliberations and ultimately funding decisions."
This study considered the influences of the use of teleconferencing because funding agencies are often interested in using this technology to reduce costs and the time burden placed on panel members who travel to on-site panel meetings.
Previous research by AIBS found that discussion times were significantly shorter for teleconference settings as compared to face-to-face, but the influence of discussion on application scoring was unclear. This analysis compared proposal scores by reviewers before and after the peer-review meeting. The authors measured the magnitude and direction of score changes. Comparisons of these score shifts were made for face-to-face and teleconference settings, which provided insights into the effect of communication medium on the subsequent scoring patterns.
"Scoring shifts post-discussion were, on average, small in both settings. Discussion was important for at least 10% of applications, regardless of setting, with these applications moving over the threshold to receive funding or not," said Dr. Stephen Gallo, an author of the study. "Small, but statistically significant differences in post-discussion scoring patterns were uncovered between settings, including a decrease in the magnitude of score shifts in the teleconference panels as compared to face-to-face. However, discussion time had little influence on the magnitude of these score shifts."
Interestingly, panel discussion was found to often result in poorer scores when compared to the initial premeeting scores. In other words, review scores worsened after panels came together and discussed a proposal. This was true regardless of setting.
The subtle differences observed between settings were potentially due to reduced reviewer engagement in teleconferences. More research is needed to understand the extent of this phenomenon, as well as the psychology of decision-making, team performance, and persuasion to better elucidate the effects of peer-review panel setting.
Friday, Mar 6, 2015,
New research from the American Institute of Biological Sciences (AIBS) found that peer review managers play an important role in identifying potential conflicts of interest (COI) in biomedical research grant peer reviews. The study, Frequency and Type of Conflicts of Interest in the Peer Review of Basic Biomedical Research Funding Applications: Self-Reporting Versus Manual Detection, was published in the journal Science and Engineering Ethics.
Peer review is the widely used process by which panels of experts evaluate research proposals to help funders identify the best research to fund. A cornerstone of the process is the integrity of the review panel, which includes a fair and non-conflicted evaluation of the proposed research. Despite this and the widespread belief that peer review is a foundation upon which U.S. science is built, few studies of the peer review process have actually been conducted.
“Peer review is so central to the way we do science, it is important that we study the process. With good data, we can ensure the vibrancy of peer review and develop models and best practices that promote the integrity of peer review,” said Dr. Stephen Gallo, AIBS Technical Operations Manager and the lead author of the study.
This research is the most recent study from AIBS, which has conducted peer review of research proposals on behalf of government and non-government research funders for over 50 years.
To inform the future of peer review, Gallo and colleagues conducted a retrospective analysis of COI data from peer review panels that evaluated 282 biomedical research applications. The overall ‘conflicted-ness’ of these panels was significantly lower than that reported for regulatory review panels, which have been studied by others.
The AIBS study found that 35 percent of conflicts were self-reported by review panel members. Importantly, peer review panel managers identified 65 percent of conflicts.
“The people who organize panels play an important role in identifying conflicts of interest,” said Gallo.
Overall, this study suggests that the scientific community should dedicate some energy to improving COI reporting and detection methods. In light of increasing demands on reviewers’ time, administrators will also need to make this process as efficient as possible while maintaining the highest ethical and review standards.
Tuesday, Sep 2, 2014,
In a paper published today in the journal PLOS One, investigators with the American Institute of Biological Sciences report findings from an analysis of the research output from a series of biomedical research grants funded after undergoing a scientific peer review process. The results, reported in The Validation of Peer Review Through Research Impact Measures and the Implications for Funding Strategies, offer insights for future research on peer review and potential models for increasing research productivity.
“Some form of peer review is used at the majority of research granting organizations to determine the best research to fund,” said Dr. Joseph Travis, President of AIBS and a biologist at Florida State University. “Peer review makes a significant contribution to how billions of dollars in research funding from government and private sources are awarded,” said Travis, a coauthor of the study.
In recent years, this process has been questioned, particularly with regard to how well peer review predicts the ultimate impact of the funded research.
“We conducted a retrospective analysis of peer review and project output data for 2,063 projects from an eight year period. Of these, 227 were funded and we examined whether correlations exist among the assessment of scientific merit using a peer review system and the scientific output from this program,” said Dr. Steve Gallo, Technical Operations Manager for AIBS and the lead investigator on the study.
Citation impact, or the number of times a research paper is referenced by others, is a common way to assess research impact. Analysis revealed that peer review scores associated with individual applications were correlated with the total time-adjusted citation output of these funded projects.
Gallo states, “citation impact did not correlate with the amount of funds awarded per application or with the total annual programmatic budget.” The number of funded applications per year did correlate well with total annual citation impact, suggesting that improving funding success rates by reducing the size of awards may be one strategy to optimize the scientific impact of research program portfolios.
“This strategy must be weighed against the need for a balanced research portfolio and the inherently high costs of some kinds of research,” said Travis.
The relationship observed between peer review scores and publication output lays the groundwork for establishing a model system for future prospective testing of the validity of peer review formats and procedures.
“This is something AIBS is looking at now,” said Gallo.
Wednesday, Aug 13, 2014,
The American Institute of Biological Sciences (AIBS) Scientific Peer Advisory and Review Services (SPARS) division has won a contract with the University City Science Center (UCSC) to provide independent scientific merit review services in support of the UCSC QED Program, a funding and business development program for researchers among 21 academic institutions located in the mid-Atlantic region and clustered within and around Greater Philadelphia. The QED Program model catalyzes cooperation among institutional stakeholders who seek to accelerate the development of technologies in research organization laboratories and pull these technologies into the marketplace.
Monday, Jun 9, 2014,
On June 10, 2014, Dr. Stephen Gallo of the American Institute of Biological Sciences (AIBS) will participate in a panel discussion as part of the 2014 Tri-Agency Peer Review Workshop to be held in Ottawa, Canada. The meeting, “Inside - Out: Perspectives on Maintaining High Quality Peer Review in Changing Times”, is sponsored by the Canadian Institutes of Health Research, Natural Sciences and Engineering Council, Social Sciences and Humanities Research Council, and the Canada Foundation for Innovation.
“We are pleased that Dr. Gallo is participating in this important discussion,” said Dr. Richard O’Grady, Executive Director of AIBS. “Peer review is central to the way we do science and how we maintain high quality standards. This discussion is an excellent forum to explore how to sustain this quality at a time when society is continuously considering how to be more efficient and to identify more cost effective ways to work. AIBS, which has a long and distinguished history of providing high quality peer review services, has been looking at these questions for a number of years.”
Indeed, Gallo, who is Technical Operations Manager for AIBS’ Scientific Peer Advisory and Review Services, has published findings from research he and AIBS colleagues have done to identify issues associated with remotely conducted panels.
Gallo will participate on a panel with Dr. Richard Nakamura, Director of the Center for Scientific Review at the U.S. National Institutes of Health, and Dr. Sophie Stevance, Assistant Professor at the University of Laval in Quebec.
“I am looking forward to this important and timely discussion,” said Gallo. “There is a growing recognition of the need to look at how we do peer review and how we can use teleconference or other technologies to reduce peer review panel costs. It is critical, however, that we understand the strengths and weaknesses of different peer review models. These are some of the reasons the AIBS peer review program has been evaluating different models.”
Wednesday, Aug 7, 2013,
Research findings published today in PLOS ONE report that the setting in which a scientific peer review panel evaluates grant applications does not necessarily impact the outcome of the review process. However, the research found that the average amount of discussion panelists engage in during the review is reduced. The investigation examined more than 1,600 grant application reviews coordinated by the American Institute of Biological Sciences Scientific Peer Advisory and Review Services (AIBS SPARS) on behalf of a federal agency over a four-year period.
The researchers compared two years when grant applications were reviewed using an in-person peer review panel to two years when panels were conducted via teleconference or videoconference.
Funding organizations routinely bring experts together to review research grant applications. A process known as scientific peer review, the goal of these panels is to identify the best research applications.
"There are no studies exploring whether the review setting significantly alters the quality of the peer review process," stated Dr. Stephen Gallo, the lead author of this study and Technical Operations Manager for AIBS SPARS.
"Our goal is always a reliable and high-quality peer-review process. It is important that we understand the strengths and weaknesses of different peer review methods," said Scott Glisson, Director of AIBS SPARS and an author of this study.
The findings appear in "Teleconference Versus Face-To-Face Scientific Peer Review of Grant Applications: Effects on Review Outcomes" published in PLOS ONE.
"Little difference was found in most of the review metrics between face-to-face and teleconference settings," said Gallo. Application scoring was only modestly affected and reviewers used the full scoring range regardless of review setting. The reviewer ratings were highly reliable in both settings.
Often, the greatest anticipated difference between in-person and teleconference panels is the amount of time allocated to discussing applications. This study found teleconference or videoconference panels allocated less time to application discussions than in-person panels.
"This is a first of its kind study that provides valuable data to help research program managers select appropriate models for conducting peer review," said Glisson.
More research is needed. "We should know whether the reduced amount of discussion and peripheral interactions that occur in a teleconference setting influence the final panel outcomes, and, ultimately the productivity of the research that is funded," said Gallo.
Wednesday, Jan 11, 2012,
Looking back at 2011, AIBS supported the peer review of science that brought a spectrum of social and life sciences to bear on human burdens as ancient and seemingly intractable as malaria, cancer, and wound infection, and as contemporary as blast-induced trauma, the reintegration of returning warriors home, and designing wildlife-sensitive green energy infrastructure.
SPARS reviewed the military "neglected tropical diseases" research that, this year, culminated in the phase III clinical trials of the RST,S vaccine. The trials over 15,000 children under the age of five across seven African countries. All over the world, 800,000 people a year die from this ancient disease, and half the world's population is at risk. This year, RTS.S reduced the risk of clinical malaria in treated children by 56% and severe malaria by 47%, SPARS helped.
The science we reviewed this year took a world view, and SPARS supported the Philadelphia's QED program that works to move science out of the lab and into the marketplace. Because of our peer review support, a portable, radiation-free, breast-scanning device, about the size of a calculator, secured licensing to bring breast cancer screening to women in remote, poor, or underserved regions of the globe.
This year too, we hosted the review of the CDC's President's Emergency Plan for AIDS Relief, PEPFAR, the largest effort by any nation to combat a single disease internationally. SPARS peer review screened good science that aimed to develop effective combination anti-retroviral therapy, train 140,000 new health care workers, care for twelve million HIV-infected people and orphans, and prevent twelve million new infections worldwide.
The life and social science we reviewed this year was innovative and wildly diverse: harnessing novel strategies that worked within - rather than against - biological systems: some developed therapeutic predatory bacteria to combat the wound infection of trauma victims, some designed rugged prosthetic ankles that let people run if they wanted to with less pain, and science that helped count Golden Eagles in order to site wind energy stations that respects the hunting, breeding, and migration life history of raptors, their landscape, and their prey.
While the science we reviewed this year reflected the needs of humanity and the planet, it frankly reflected the needs of our nation. We reviewed biomedical and bioengineering science that addressed the physical and psychological injuries of wounded warriors, and social science that addressed the needs of returning warriors, their families, and society as a whole, as everyone reintegrates to home and work.
Thursday, Apr 29, 2010,
SPARS Helps Marine Biologists and Native Alaskan Hunters Sample Alaska Harbor Seals
For Alaska's Ecosystem and Its Native Culture, Harbor Seals are Critical
One hundred and two people live in Tatitlek, Alaska, a traditional Alutiiq village in a forested coastal glen - the impassible Chugash Mountains, like snowy ramparts, at their backs and the vast Prince William Sound spreading out before them. Intimately linked to the mountain and sea that surround them, Tatitlek is much like the hundreds of native Alaskan coastal communities for whom a subsistence lifestyle is tradition, and for whom harbor seals (Phoca vitulina) play a central role in a community’s diet and, because of that, its culture.
In 2008, of the 1,462 harbor seals harvested in the state of Alaska, Tatitlek harvested 141 of them. With one harbor seal supplying about 56 pounds of useable meat, fur, and oil, every person in Tatitlek, had 77.4 pounds - about a quarter pound of seal a day - 39% of their meat that year.
Not only are harbor seals important to native Alaskan people, they are both a key marine predator and an important prey animal, and they live in a nearly unbroken sweep of coast from Dixon Entrance south of Juneau, west more than a thousand miles curving down
the Aleutian Archipelago, then north to Kuskokwim Bay in the Bering Sea.
But, since the 1970s, throughout that wide distribution, the Alaskan harbor
seal population - which is divided into three "stocks" for wildlife management
purposes - has declined precipitously - west of the Gulf of Alaska,
populations have dropped by 50 to 90%, but the decline is a strange, erratic one
- populations are relatively stable in the southeast. The cause or causes are unknown – ideas include predation and/or competition from stellar sea lions or sharks, resource competition from recently returned hump-backed whales, pollution, and temperature increase.
SPARS Review: Genetic Research Finds Alaska Harbor Seal Stocks Too Few For
In 2005, in response to a request by the National Marine Fisheries
Service (NMFS)/Alaska Native Harbor Seal Commission, the Southwest
Fisheries Science Center (SWFSC) submitted the report "The analysis of
population genetic structure in Alaskan harbor seals Phoca
vitulina) as a framework for the identification of management
stocks" by O'Corry-Crowe et al. SPARS directed its peer review, assembling a panel of five reviewers - two anonymous, all experts in conservation ecology, biostatistics, and marine mammal biology.
The research they reviewed sampled 881 harbor seals in 180 sites, evaluating population subdivisions and seal dispersal patterns using mitochondrial DNA. The data indicated that the three stock units used by the state to manage marine mammal populations inappropriately lump together up to 12 smaller, genetically distinct populations. They concluded that continuing to manage the population using stocks too large, risks the extirpation of the smaller, genetically distinct sub-groups, reducing genetic diversity - that variation within species that allows populations to adapt to environmental changes, resist certain diseases, and avoid inbreeding.
...managing the population using stocks too large, risks the
extirpation of the smaller, genetically distinct sub-groups.
"Yakutat hunters have been concerned that the number of seals were going down, hunter insight can trigger hypothesis generation. If native hunters think something is amiss, it probably is.""
Wednesday, Apr 7, 2010,
SPARS Manages FAMRI Peer Review
The not-for-profit foundation, Flight Attendant Medical Research Institute (FAMRI; http://www.famri.org), sponsors scientific and medical research to combat the diseases caused by exposure to tobacco smoke.
Since 2001, the foundation has relied on SPARS to manage large- and small-conference peer review meetings that have marshaled the wide-ranging expertise of hundreds of reviewers and funded nearly 500 projects addressing the prevention, early detection, treatment, and cure of diseases and medical conditions caused by tobacco smoke.
Cancer Cells: Difficult to Find, Difficult to Fight.
Cancer cells, cells growing out of control, are diffused within healthy cells, especially at early stages. Because they are strewn and woven throughout healthy cells, they are difficult to target precisely. Because they are difficult to target, traditional anticancer therapies are blunt instruments: chemotherapy targets not just cancer, but fast-growing healthy cells like bone marrow, hair, and skin; radiation cannot always focus precisely, and lumpectomies may miss malignant or premalignant tissue.
But, in a series of elegant and innovative FAMRI-funded studies, Dr. Yuman Fong of Memorial Sloan-Kettering Cancer Center is harnessing the propensity of viruses to cause cell death. His lab is harnessing that propensity, and aiming it at cancer cells. Fong's lab is designing viruses that selectively infect tumor cells and can kill them within hours.
FAMRI Research Harnesses Viruses to Kill Cancer A virus's strategy is to capture a cell's replicating machinery, producing thousands of viruses until the infected cell bursts, or lyses, releasing the newly made viruses, and spreading viral infection that affects the cancer, but not the patient. A cancer cell is a virus's perfect target - its replicating machinery is out of control - it has no protective 'off' switch. Dr. Fong's laboratory focuses on engineering viruses (herpes simplex 2, adeno, vaccinia, myxoma, and vesicular stomatitis) that specifically infect and lyse cancer cells (oncolytic viruses), while sparing normal cells.
FAMRI Research Harnesses Viruses to Find Cancer
In a FAMRI-supported 2004 study, the investigators used a herpes simplex virus called NV1066 that they engineered to contain a gene for a protein called "green fluorescent protein" (GFP). They wanted to use the modified virus to infect early cancer cells and cancer cells in hard-to-reach tissues like nerves. It did just that, and only the cancer cells with the GFP glowed green.
These experiments showed that viruses can be used to improve early detection and identification of cancer, and to help guide surgical resection. "In animal and in human studies, they [the viruses] have been able to detect one tumor cell in a background of one million normal cells, a 50 times improvement over traditional cytology)," writes the PI.
FAMRI Research Designed for Global Application
This work has global ramifications. The investigators in Dr. Fong's lab point out that using this technique, the cancer cells that become virus-tagged and glow green under the appropriate light source can be seen by personnel with minimal training using available, inexpensive machines. Dr. Fong suggests that these qualities may make the techniques appropriate "even in rural areas of developing nations where cigarette smoking is still popular..." allowing detection of many types of cancers including lung, oral, pancreatic, and stomach, and could be immediately applicable worldwide.
Steadily building on the success of previous work, another FAMRI-supported study by Fong et al., in 2007, continued this work, with an added twist. They asked, "What cellular change in the process of malignant transformation allows the virus to selectively infect the cancer cell?"
Dr. Fong and colleagues have carried out a great deal of FAMRI-funded research; they have been working steadily since 2004 to develop oncolytic viral therapies, some of which are now entering Phase 1 clinical trials.
Tuesday, Mar 16, 2010,
Traumatic Brain Injury (TBI) is perhaps the defining injury of the US wars in Iraq and Afghanistan. The Military Health System diagnosed TBI in 43,779 patients between 2003 and 2007.
TBI victims suffer from a stunning range of functional brain changes both immediately after the trauma, and decades later. Memory, language, learning, emotion, and behavior are skewed. TBI can cause phantom sensation, and epilepsy; a brain injury event seems to increase the risk for Alzheimer's disease, Parkinson's, and other age-associated brain disorders.
As devastating to individuals, families, and society as TBI is, the biomechanical mechanisms describing what happens when a blast-wave meets a human head are not clearly understood. But, in 2007, SPARS recruited neurophysiologists and blast experts to review research that sought to describe how blast wave energy is transferred to a human head at different blast strengths and different head angles, addressing this knowledge gap.
Successfully funded, that research went on to explain mathematically that shock waves can be focused in the orbital cavities - the eye sockets - and the strength of the blast significantly increased by 10 times or more, "...focused like a parabolic mirror, like the headlights of a car," said the study PI.
The model showed that the blast hits the inner part of a soldier’s helmet, travels through the space between the head and helmet, then washes up and over to hit the back of the soldier’s head. With information directly from SPARS peer-reviewed research, the shape of helmets may be changed to mitigate this wave.
Further, associated research was able to use the original data to continue to describe the physiology of blast-induced brain injury - "...not just what happens in the instant of the blast impact," says the PI, "but what goes on in the brain after the blast for minutes, even hours." guiding immediate treatment and subsequent therapy.