Review Strategy Brown Bag on June 19, 2014

Brenda Lange-Gustafson

2008 VATID(Partnerships)

RFA 08-001, “Cooperative Partnerships for Biodefense and Emerging Infectious Diseases”- Vaccines, Adjuvants, Therapeutics, Immunotherapeutics, and Diagnostics (VATID).U01

Receipt 199

6 panels (3 Brenda; 3 Michelle)

Reviewer Load: usually ~5 assignments; mostly 3 per application, sometimes 4

Released scores at the end

#Applications# (%)discu# funded (score range) others <212 skipped

SEP 1Immunotherapeutics2510 (40%)6 (139-193) skipped 204

SEP 2Vaccines/Adjuvants3313 (39%)6 (132-179) skipped 204

SEP 3Diagnostics3514 (40%)10 (147-204)

SEP 4Therapeutics3715 (43%)11 (139-212) skipped 206

SEP 5Vaccines3413 (39%)5 (150-192)skip 201,202,204,205

SEP 6Therapeutics3615 (40%)10 (132-200)

This was one of the early ones to split,before it became usual. HIV/AIDs CTUs and Leadership had been split and TC; Malaria and Neglected Tropical Diseases (FTF;Annie)had gotten permission from the Sallys to split without splitting the pool of money. Thus, we (Ed, Michelle and I) informally gave HH the following justifications:

Challenges facing:

  • Knew we were expecting ~ 200 from LOIs
  • We were recruiting at the same time of RCE recruitment, who had 100 reviewers of type we needed
  • There was reviewer fatigue- many reviewers who repeatedly had been called and served were saying no(not for this review yet)since they were often assigned 10 reviews for VATIDs.
  • We wanted a high quality review as measured by inclusion of significant number of senior scientists; all applications get a fair review, including those streamlined.
  • Previous VATID reviewers (especially diagnostics) asked for discipline-specific panels

Suggestion: Split the panels as above. Streamline up to 70% of the applications. All applications will receive full critiques. Assuming 200 applications, that leaves 60 applications to be discussed. Since only 15-20 applications will be funded, this should provide sufficient numbers of applications for the Program Officer. According to the previous year’sVATID figures, 17 out of ~ 270 applications were funded. They had scores between 129 and 167. Five applications scoring in this range were not funded (140, 149, 149, 150, 157).

Presented to Program and they agreed.

Advantages:

  • Expect that more reviewers, especially more senior ones, will say yes to a teleconference.
  • More reviewers means we can reduce reviewer workload, ensuring thorough review per applic
  • May allow us to recruit more foreign, unconflicted reviewers.
  • Allows for attractive, subject focused panels.
  • Big savings in travel, hotel and reimbursement costs (>$175,000).
  • If 30-35 applications per committee, we could streamline 50-60% and get the remaining applications discussed in a 4-5 hour telephone review.
  • Allows time for full discussion of the most competitive applications so the ones with the highest scientific merit can be determined.
  • Reduces reviewer burden of discussing applications that have no chance of being funded.

Disadvantages

  • Inherent difficulty with standardization of review across panels.
  • Program might need to prepare a funding plan using criteria other than strict score order depending on outcome (e.g. if all immunotherapeutics scored worse than other panels and program wanted to fund some from that area.Also, there is the potential that one research category could have all the applications streamlined, thus there would be no chance to fund, for example, an adjuvant application. However, the 2007 funding data (see above) indicates that program is not “reaching down” to fund equally across these research fields. They funded mostly by score.
  • Foreign reviewers would face less advantageous time schedule (but we could allow partial attendance if necessary).

Strategies to help with a long teleconference review:

  • Impress upon them that this still is an arduous review and that they would have to be considered absent from their university (i.e., no coming and going from the review to deal with issues in the lab or administrative issues). Review as they would like their application reviewed.
  • Emphasize importance of participation in discussion by non-assigned reviewers.
  • Emphasize learning from the people with expertise very different from your own and taking all comments into account prior to scoring.
  • There will be breaks.
  • Critiques due ~ 1 week prior to meeting to allow reviewers to have read them and have focused discussions.

Strategies for standardization:

  • Provided common pre-review TC/documents.
  • Shared reviewers/chairs if possible;at recruitment, ask for participation in multiple panels.
  • Employed common streamlining procedures.

Based on LOIs, Michelle did a pre-recruitment and had ~ 100 reviewers

Final:

# Prof level71 (57%)

# Assoc level41 (33%)

# Assist level12 (10%)

124

2011: Initiative: RFA AI-10-017, “Partnerships for Next Generation Biodefense Diagnostics (R01)”.

  • For early stage product development projects to establish proof-of-concept for potential next generation diagnostics products that do not involve nucleic acid amplification and
  • That demonstrate the technology’s capability to detect NIAID Category A, B, or C priority agents in live infectious disease animal models (for invivo detection technologies) or human clinical samples (for other technologies).
  • Expect 7-10 awards

Receipt: 73 applications; three reviews from 15 Feb-28 Feb, 2011; wait to release scores until end.

Rationale: Split panels and TC due to earlier success (VATID) of recruiting more senior people for teleconference and saving money by not having a face to face (emphasis at the time since VATID).

Equivalent panels- not split by expertise or pathogen to try for standardization

# Applications# (%)discussed# funded (score range)/ next score/reviewer info

SEP 12310 (44%)2 (18-27) next 30 [7/19rev from SEP 2; 5 from SEP 3]

SEP 22511 (44%)2 (23-31) next 37 [7/25 rev fr SEP1; 12 fr SEP 3]

SEP 3 2511 (44%)4 (26-34) next 35 [12/24rev fr SEP 2; 5 fr SEP 1]

Standardization:

  • Overlapping chairs/reviewers
  • 42 reviewers total
  • 25 participated on 2 panels (asked in recruitment letter; 8-10 total reviews if 2 panels)
  • Chairs: Chair #1 was going to chair all; father became ill.I picked Chairs #2 and #3 from members of Panels 1 and 2 respectively; thus, they had experienced a session with same “philosophical”/operational decisions made prior to chairing.
  • Provided common pre-review teleconference.
  • Used common streamlining procedures: similar percentages reviewed as long as preliminary scores were also similar.