Review Strategy: Two-Stage Review – Internet Assisted Meeting (IAM) Discussion for Multiproject Applications

Case Study: RFA-AI-15-041,Human Immunology Project Consortium (HIPC)

Review 2016, SRO contactsDrs. Paul Amstad and Zhuqing (Charlie) Li

Table of Contents:

Abstract

Driving Factors for Strategy Choice

Overview

Detailed Approach

Unique Features of the Strategy

Lessons Learned

Technical Challenges

Timeline

Questions and Answers

Review Strategy: Two Stage Review- Internet Assisted Meeting (IAM) Discussion for Multiproject Applications

Case Study: RFA-AI-15-041, Human Immunology Project Consortium (HIPC)

Review 2016,SRO contacts Drs. Paul Amstad and Zhuqing (Charlie) Li

Abstract

A two-stage review strategy was employed to handle a large response of multi-project, U19 applications that required streamlining. Stage 1 of the review was conducted via Internet Assisted Meeting (IAM) that permitted an online “chat” discussion and preliminary scoring of the overall merit of the U19 applications. Stage 2 of the review was a face-to-face Special Emphasis Panel (SEP)that included streamlining of U19 applications based on the preliminary overall impact scores determined during the IAM Stage 1 review followed by the full review of non-streamlined applications. Strengths of strategy included streamlining decisions based on the scientific merit of the overall application instead of the average preliminary scores of individual projects and cores. Disadvantages of the strategy were the increased time and effort required by both reviewers and SRP staff to participate and monitor the IAM Stage 1 review and the added cost of honorariumfor the IAM discussion. Technical challenges involved setting up IAM so that reviewers only had access to the discussion board for the overall application and not the projects and cores.

Driving Factors for Strategy Choice

A large responseof multi-projectapplications that would be challenging to review in single meeting due to volume and complexity.Streamlining would be required to permit adequate time to discuss the most meritorious applications. However, there was concern raised by both reviewers and NIAID staff that prior streamlining strategies that included ranking applications based solely on preliminary project scores may notadequately reflect the scientific merit of the overall applications; synergy and cohesiveness of the overall application are not considered when streamlining is based solely on preliminary project scores. Also, the merit of cores is not considered if streamlining is based on project scores.The IAM chat provides reviewers with a forum to discuss the overall merit of applications prior to giving a preliminary overall impact score to establish ranking of applications for streamlining. IAM discussion increased reviewers’ confidence in their streamlining decision.

Overview

  • THE HIPC RFA resulted in the submission of 19 U19 (145Reviewable Units (RUs)) applications that had a minimum of 2 Projects,an administrative core, a clinical core and a data management core. Additional science cores were optional. Sixty-two reviewers were required to review the applications. The review was conducted in two stages.Stage 1 involveda two-day IAM online chat discussion of the overall merit of the applications. At the end of the IAM discussion, reviewers were asked to post a preliminary overall impact score in IAR for their assigned applications. Average preliminary overall impact scores were then used to rank the applications for streamlining. Stage 2 involved a three day, face-to-face review meeting (52 reviewers in person, 10 teleconference) that began with a streamlining decision based on preliminary overall application scores, followed by a discussion of the most meritorious applications. Eight of the 19 U19 applications were streamlined.Both review stages (IAM and face-to-face) were conducted as a single FACA meeting within a seven day period.

Detailed Approach

  • Stage 1
  • Stage 1 included a 2 day IAM discussion of the overall merit of the multi-project U19 applications.
  • Before the start of the discussion, Discussant 1 (Dis 1) was asked to provide introductory comments (Dis 1’s assessment of the overall merit of the application).
  • All assigned reviewers were then asked to participate in the discussion by reading comments, providing new comments and responding to comments by others. At the end of the IAM discussion, assigned reviewers were asked to post a preliminary overall impact score in IAR for theirassigned applications. Average preliminary overall impact score of assigned reviewers was the basis for streamlining.
  • IAM was monitored by multiple SROs when in session. Each SRO monitored approximately three U19 application discussions. Monitoring included making sure that comments were focused on the overall application, appropriate and did not include discussion scores.
  • IAM was put on hold outside business hours. During the hold phase reviewers could still read and post comments, but the posted comments could only be read once the SRO had read them and removed the hold on IAM.
  • Only assigned reviewers were asked to provide a preliminary overall impact score in IAR. However, unassigned reviewers can still participate in the discussion of all non-conflicted applications.
  • Reviewers were asked not to post preliminary overallimpact scores in IAR or discussion scores in IAM.
  • Conflicts were managed by propagation of conflict settings from Peer Review.
  • Program staff were permitted to monitor the discussion but did not have permission to post comments.
  • Reviewers were paid $200 for participation in the two day IAM meeting.
  • Stage 2
  • Stage 2 was a 3 day face to face meeting to discuss non-streamlined applications.
  • Streamlining was done at the beginning of the face-to-face meeting (8 out of 19 applications were streamlined). A streamlining chart was provided to the reviewers at the beginning of the Stage 2 review with the applications ranked in order of preliminary overall impact score provided during the Stage 1 IAM discussion.
  • Both IAM and face-to-face meetings were done as one continuous FACA meeting.
  • Reviewers were also paid for their participation in a three day face-to-face meeting.

UniqueFeatures of the Strategy

  • Combining both stages in single FACA meeting prevented the SRO from having to transfer applications to a new meeting (resulting in alarge reduction in SRO workload). Committee Management permitted the combined, single FACA meeting since the total time frame of the two stages was relatively short (one week).
  • IAM discussion of the overall application during Stage 1 allowed reviewers to assess the overall scientific merit of the application, including the cohesiveness and synergy of the program. These elements are not considered when streamlining is based solely on the preliminary scores for individual projects and cores of multi-project applications. Therefore, streamlining decisions are made with the same review criteria used for making subsequent funding decisions.
  • The IAM discussion worked well. There was large reviewer participation, insightful and well-formulated comments, good adherence to discussion topics, and tendency to reach consensus towards the end of the discussion.
  • The preliminary scores for the overall application had a wider spread than the preliminary scores for the individual projects. The score spread for the overall after the IAM discussion was 2.5-6.5 while the score spread for the projects 2.3-4.9.
  • Streamlining based on overall impact score was successful. Due to the wider spread of scores reviewers were satisfied with their streamlining decision.

Lessons Learned

  • It is important that IAM stays open after the close of discussion (i.e. put IAM on hold) in order for reviewers to go back to read the comments before deciding on the preliminary overall impact score.
  • Reviewers suggested leaving IAM open during the face-to-face meeting so they could refresh their memory of the specific discussion points.
  • Some reviewers complained of the extra work involved in the IAM discussion.

Technical Challenges

  • During Stage 1, it was important that reviewers only have access to the discussion board of the overall application and not the projects or cores. Otherwise reviewers could post their overall comments into the wrong discussion page. To get around this, projects and cores were designated ND so that reviewers could only select the discussion page of the overall program.
  • If a reviewer included a preliminary score for an overall application in IAR, it was transferred to IAM. SROs had to request that the score be manually removed by the IAM team. The IAM team was very responsive to these requests.

Timeline

Questions and Answers

  • Were all reviewers required to participate and post during the Stage 1 IAM meeting?

Yes, all reviewers were required to provide at least one comment to their assigned applications. Some needed reminding. In general, Dis 1 did a great job starting the discussion and the other assigned reviewers followed.

  • What were the nature of the IAM comments? How was the quality of the discussion?

The majority of the comments responded directly to the introductory comments provided by Dis 1. Some reviewers posted their own preliminary critiques for the overall program. The complexity of the comments were quite variable and depended largely on the level of disagreement. Overall, the online discussion was good.

  • Did the Stage 1 IAM discussion have an effect on the quality or length of the discussion of the overall application at the Stage 2 face-to-face meeting?

No, the IAM discussion did not replace or shorten the face-to-face discussion. The quality of the discussion at the Stage 2 meeting was very good.

  • Could unassigned reviewers participate in the discussion of applications during the Stage 1 IAM review?

They were permitted to participate if they were not in conflict. However, most did not with the exception of the Chairs.

  • Were comments from the Stage 1 IAM review captured for the summary statement? What about for ND applications?

No, the IAM discussion was only used for providing preliminary overall impact score for streamlining and the IAM Stage 1 discussion was not captured in the summary statement. However, much of what was discussed at the Stage 1 IAM meeting was also discussed in the face-to-face meeting. For ND applications, reviewers were given standard instructions to update their critiques but no specific instructions to include comments from the Stage 1 IAM discussion.

  • The preliminary scores for the overall applicationafter IAM discussion had a larger spread than the average preliminary scores for the projects. Why do you think this was the case?

Reviewers appeared to have more confidence in their scoring after the discussion in IAM and were comfortable to give scores in the range from 4-7.

  • Did the overall scores of non -streamlined applications change much between the IAM discussion and the face-to-face discussion?

Preliminary scores were from 23-40 after the IAM discussion and ranged from 20-44 after the face-to-face review.

  • Were the applications with more discussion during the Stage 1 IAM meeting more likely to be streamlined?

No, the applications that were in the “grey zone” received the most discussion. Really poor applications were not heavily discussed since there was already a lot of agreement.

  • Did the presence of specific expertise influence the IAM discussion and scoring?

There was a mix of immunology, infectious disease and systems biology experts assigned to each application. There was a nice sharing of information and ideas to assist each other.

  • Do you think the increased effort of reviewers and staff is justified for this approach?

Yes, the reviewers felt more confident in the streamlining decision and the IAM discussion helped clarify applications in the grey zone.

  • Did your applicants know you were conducting the review in two stages?

No, but several asked why the review was taking so long.

  • Did you inform your reviewers during recruitment of the added involvement of an IAM meeting?

No, we informed them during the pre-review orientation.