Review Strategy: Two-Stage Streamlining Review

Case Study: RFA-AI-14-055: Centers for Medical Countermeasures against Radiation Consortium (CMCRC)

Review 2015, SRO contactsDrs. Paul Amstad and Lou Rosenthal

Review Strategy: Two-Stage Streamlining Review

Case Study: RFA-AI-14-055: Centers for Medical Countermeasures against Radiation Consortium (CMCRC)

Review 2015,SRO contacts Drs. Paul Amstad and Lou Rosenthal

Abstract

A two-stage review strategy was employed to handle a large response of multi-project, U19 applications. Stage 1 of the review was a FACA teleconference meeting of the entire review panel to make streamlining decisions. Stage 2 of the review was a face-to-face Special Emphasis Panel (SEP) to discuss the most meritorious applications. Strengths of strategy included increasing the amount of time at the face-to-face SEP to discuss applications and the potential for reducing the number of SEP attendees due to the streamlining of assigned applications. A disadvantage of the strategy was the prolonged review timeline and the potential for information from the FACA teleconference meeting to be leaked to the community prior to the SEP. Technical challenges involved the transfer of meeting contents from the Stage 1 to Stage 2 meetings.

Driving Factors for Strategy Choice

A large responseof multi-projectapplications that would be challenging to review in single meeting due to volume and complexity.A large number of reviewers was required to cover all applications.

Overview

THE CMCRC RFA resulted in the submission of 19 U19 (317 Reviewable Units (RUs)) applications that had a minimum of 3 Projects and 4 Cores each. Sixty reviewers were required to review the applications. The review was conducted in two stages; Stage 1 involveda FACA teleconference of the entire review panel to make streamlining decisions. The committee decided to streamline approximately 40% of the applications. The streamlining teleconference meeting lasted approximately 1 hour. Stage 2 occurred 2-3 weeks after the initial streamlining teleconference and was conducted as a face-to-face SEP. The second stage meeting lasted two and a half days. Reviewers assigned only to applications that were streamlined were not required to attend the Stage 2 review, although they could attend if they wished. Thosewith only minimal assignments remaining were allowed, at their request, to join the SEP via teleconference. Fifty-one of the original 60 reviewers attended the Stage 2 face-to-face meeting in person and an additional seven via teleconference.

Detailed Approach

Stage 1

The Stage 1 review was conducted as a FACA teleconference 2-3 weeks prior to the Stage 2 face-to-face SEP.
A total of 60 reviewers were recruited to review the 19 U19 applications (317 RUs).
All but one or two reviewers, including two chairpersons, attended the Stage 1 review to conduct streamlining decisions. The few who could not attend gave concurrence for streamlining decisions prior the Stage 1 review
A twelve day READ phase preceded the Stage 1 review. Although there was no formal EDIT phase prior to Stage 1, reviewers had the opportunity to send updated scores to the SRO up to 48 hours prior to the Stage 1 teleconference and these were posted in IAR.
Streamlining was based on the mean preliminary Project scores for each U19 application. Streamlining tables (summary, score details, individual project and core scores) were posted in Meeting Materials in IAR the day before the Stage 1 teleconference. Although Preliminary Core scores were provided in Meeting Materials, core scores were not considered in the initial ranking.
Approximately 40% of the applications were slated to be streamlined at the Stage 1 meeting. The Chairpersons emphasized the confidentiality of the meeting at the teleconference.
The Stage 1 teleconference review lasted about 1 hour.
After the Stage 1 review was concluded, the preliminary summary statement (PSS) of all critiques was generated and then applications, critiques and reviewers were transferred to the second meeting.
Stage 2
Reviewers made travel plans for the Stage 2 meeting after the results from the Stage 1 streamlining meeting. Overseas reviewers were permitted to make travel arrangements prior to the Stage 1 meeting.
The Stage 2 review was conducted as a single Special Emphasis Panel (SEP) convened as a two and one half day, face-to-face meeting.
51 of the 60 original reviewers attended the face-to-face meeting in person. Seven reviewers joined via teleconference. Two did not attend since their assigned applications were streamlined in Stage 1.One chose to attend even though all assigned applications were streamlined.
Of the original 19 U19 applications submitted, 11 were discussed in Stage 2.
Each application was reviewed in its entirety as follows:
Review of projects (at least 3): numeric scoring (1-9 scoring range)
Review of cores (at least 4): administrative core and service cores, numeric scoring (1-9 scoring range); consortium wide cores, scored Acceptable or Unacceptable
Review of overall applications: numeric scoring (1-9 scoring range)

UniqueFeatures of the Strategy

Stage 1 is a FACA meeting so streamlining decisions made at the Stage 1 were final.
Streamlining prior to the face-to-face Stage 2 meeting provided more time to the reviewers to discuss meritorious applications.
Reviewers who are assigned to only streamlined applications did not have to attend the face-to-face meeting.
Since streamlining and critique submission occurred several weeks prior to the face-to-face meeting, reviewers came to the face-to-face meeting relaxed and in a good frame of mind for the review meeting. SRO meeting preparation for the face-to-face meeting was simpler as well.

Lessons Learned

Including an extra FACA meeting for streamlining extended the review timeline by 2-3 weeks.
Even though SROs and Chairpersons emphasized the confidentiality of the streamlining Stage 1 meeting, there was still concern that information may have leaked to the external community. However, there was no evidence that information from the Stage 1 meeting reached the community.
Although it had been hoped that streamlining prior to the face-to-face meeting would reduce the number of reviewers at the Stage 2 face-to-face meeting, the Stage 1 streamlining did not significantly impact attendance at the Stage 2 meeting due to the multi-disciplinary nature of the applications and review assignments.
An alternative strategy could have been used to circumvent the problems associated with the transfer of critiques, etc from Stage 1 to Stage 2. See Technical Challenges below.

Technical Challenges

The biggest technical challenge of this strategyconcerned the transfer of the critiques, applications and reviewers from the Stage 1 meeting to the Stage 2 meeting. First the applications had to be transferred and then this was followed by critiques and reviewers with their review assignments. Additionally, there were text features lost in the critiques when moving from pdf to Word versions during the transfer. This process is time consuming for the SRO. In hindsight one could have set up 2 meetings one without applications (streamlining) and one with applications (face to face meeting). It would not have required transferring applications, assignments, critiques etc. Both meetings would have the same roster.

Questions and Answers

Due to the challenges associated with transferring critiques from the Stage 1 to the Stage 2 meeting, is it worth asking the reviewers to re-upload their critiques?

That is a possible solution but there is a risk of the reviewers editing their critiques between the meetings.

Can reviewers upload critiques to the two meetings at the same time?

At this point, this is not possible. Applications can only exist in one meeting and since Stage 1 and Stage 2 were both FACA meetings the applications had to stay with one meeting at a time. This could have been avoided by making the Stage 1 meeting a Workgroup but then you lose the benefits of a FACA meeting.

Can you simply rename a meeting instead of transferring everything into a new meeting?

No, once a meeting is named it is final.

How did the community feel about the process?

It seems like the reviewers liked the two stages. The two weeks between Stage 1 and Stage 2 gave them the opportunity to relax.

Review Strategy: Two-Stage Streamlining Review