A STATISTICAL-MODEL VALIDATING TRIAGE FOR THE PEER-REVIEW PROCESS - KEEPING THE COMPETITIVE APPLICATIONS IN THE REVIEW PIPELINE

Triage of grant application at the National Institutes of Health (NIH) is a process whereby an initial screening of applications by a scient ific peer review group eliminates applications that are not competitiv e for awards. The process of application triage has been limited to th ose applications submitted to the NIH in response to an RFA (Request f or Applications). A hypergeometric model was developed to determine th e extent to which five, six, seven, or eight member triage teams or su bsets of 12-to-20 member full committees could provide a statistically defensible triage decision. Although the intent of triage is to remov e from review those applications that are noncompetitive, the model wa s weighted in favor of the applicant to minimize the likelihood that h ighly competitive applications would be eliminated. Within the assumpt ions and rules developed, it was determined that there was little like lihood that the latter would occur. For example, in the worst case sce nario, the greatest probability that a highly competitive application would be knocked out of competition is P less-than-or-equal-to 0.014 i n the case of a five-member triage subset of a 20-member committee. Us ing the latter case, the model was tested on a set of 73 applications that were submitted to the National Cancer Institute for action at the February 1993 National Cancer Advisory Board. The model selected for triage required that each application be assigned to five reviewers, t hat each reviewer be blinded to the review assignments of the other re viewers, and that four noncompetitive votes be registered to triage ou t an application. Each of 19 applications received four to five noncom petitive votes, and were triaged out of the review process. The remain ing 54 applications were then reviewed according to the usual NIH revi ew process. Four of the applications received three noncompetitive tri age votes each and were either rated as not recommended for further co nsideration (NRF, n=2)) or received priority scores greater-than-or-eq ual-to 250 (n=2) (The smaller the priority score the better the techni cal merit). Thirteen of the 53 applications received two noncompetitiv e votes. Of the latter, two were not recommended for further considera tion and the remaining 11 received priority scores between in excess o f 200. The distribution of competitive applications was such that fund ing was limited to those applications with priority scores of less tha n 190. Thus, the data suggest that the conservative model is valid suc h that the likelihood of eliminating a highly competitive application from consideration for funding is remotely small. With this model, the process of triage is fair to applicants on the one hand and is also e ffective in reducing consultant workloads on the other. The model coul d be applied to many different types of review situations (private spo nsors as well as federal), especially when few awards are to be issued relative to the number of competing applications.