Conduct of A Meeting:

At-A-Glance Checklist

Several Days BEFORE Meeting

Meet with all staff participants (Program, CMP/GMP) to set who'll say what, and discuss any other last minute or unique issues of which SRO should be aware;

Ascertain number of observers/visitors/program staff, etc expected, and relay this through the RS to hotel (chairs, refreshments, phone system, microphone setup);

Determine exactly who is listed as speaking on the agenda (e.g., Contract Specialist, Contract Officer, Program Officer, Grants Management Specialist);

Familiarize uninitiated reviewers with procedures ahead of time;

Meet with/call Chair to go over procedures, expectations, special issues for review, questions reviewers have raised;

Finalize hotel logistics (physical layout, rooming list, transportation information)

Discuss expectations and division of tasks with RS for the meeting.

On Day of Meeting

Arrive early, to greet reviewers in relaxed fashion, check room setup, solve problems;

Greet Program, OA, GMP staff and observers;

Start on time;

Open with introductions; panel introductions are led by Chair;

Orient committee as to contents of folders, ask them to sign pre-review COl certifications electronically or on paper and, if paper, put into accordion folders to be collected at the end of the meeting

Orient committee as to:

  • purpose of meeting
  • objective of committee
  • COl and confidentiality

Ask for any corrections to roster information

SRO describes how review will proceed, covering:

  • how discussions of each application/proposal will proceed (following agenda, chair will call on reviewers according to assignment list);
  • what is and what is not considered reviewable;
  • how documentation will be completed during review and used later in Summary Statements (grants) or Technical Evaluation Reports (TERs) for debriefing unsuccessful offerors (contracts) -- reminder to keep annotating TESS forms during discussion;
  • what review criteria are to be used and how to apply them;
  • who scores: all members not in conflict will score every application/proposal
  • basis of scoring: based on totality of what reviewers have read and heard discussed, not only on their own specific area of expertise

Describe how to score numerically (grants: whole numbers only: scale of 1-9; contracts: allocation of points within pre-weighted Technical Evaluation Criteria)

(Grants) Completing the paper score sheet or using electronic scoring in IAR -- reviewers need to put something in every blank. Entries may be a numeric score, ND to specify “not discussed”/”streamlined”; NP to specify “not present”; NRFC to specify “not recommended for further consideration”; CF to specify “conflict” or AB to signify “present but abstaining” from voting. Reviewers can no longer enter CF if this field is not pre-populated.

When and how budget and special issues (protection of human subjects from research risks; appropriate representation of women, minorities and children in the proposed research; vertebrate animal welfare issues; protection of investigators and the environment from biohazards) are to be considered (only for scored applications; acceptable proposals).

Grant Review Highlights:

For each application, Chair announces application, reviewers with conflicts exit, and then the assigned reviewers are announced.

Sequence of events: Chair asks for preliminary scores, primary reviewer gives non evaluative summary, then primary, secondary, tertiary reviewers speak, each indicating agreement/disagreement with previous speakers and adding only new information; then open discussion to entire panel; then discuss human subjects/animal welfare/biohazard issues; then assigned reviewers restate final scores, with rest of panel scoring privately. Then, for scored applications, the committee discusses the budget. Reviewers will retain score sheets at their places until the end of the meeting. Reviewers may adjust any score for a previously reviewed application up until final adjournment, so that each reviewer leaves the meeting satisfied that they were personally consistent in their scoring throughout the meeting. SRO may seek clarification of any issues during discussion.

For multiproject applications, panel first discusses and scores Projects, including special issues/budgets, then cores/special issues/budgets, finally concluding with overall Program Project discussion before giving final PPG score reflecting synergy, etc.

If the SRO uses final scoring in IAR, the SRO should provide instructions on the use of paper backup score sheets, if provided.

The RS collects score sheets, if used, and checks that all are handed in. The RS collects any revised paper critiques, extra comment sheets, and CDs.

The RSor SRO verifies all electronic scores are entered and saved.

Reviewers sign and hand in post-review COl certification forms.

The Chair and SRO should sign and date the Minutes in IAR.

Contract Review Highlights:

Sequence of events: For first proposal, Chair reads first Technical Evaluation Criterion (TEC); asks for preliminary scores of that TEC section; discussion of strengths/weaknesses proceeds by each scorable section of the TESS, in turn, with primary, secondary, tertiary reviewers leading off as for grants, followed by other discussants. Assigned reviewers state a range of enthusiasm within the point value assigned for that TEC, and then each reviewer privately records their own point score for that criterion. Proceed in the same manner to review the next TEC, following the reading by the Chair. After all TEC have been discussed, scored and tallied on the TESS, the panel votes with show of hands on whether the proposal is "Technically Acceptable" or "Unacceptable" (see definitions in Introduction to Contract Review module).Then, the special issues and budget are addressed only for Acceptable proposals.

Budget issues -- consider direct costs only, level of effort, appropriateness of effort, supplies, travel (quantitative, qualitative) but not at level of item pricing (not scorable).

Special issues -- SRO provides clear explanation (brief, well-considered) of how to handle protection issues (HS, VA, biohazard), and how to handle representation issues (gender, minority, children). For HS/VA, data sharing, model organisms, GDS. HS/VA/biohazard issues are considered in the scoring of proposals if one or more TEC sections refer to human/animal studies. (Note: Discussions are ongoing as to how/whether an unacceptable rating for HS/AW is to be considered in the final rating of a proposal.)

After a proposal review concludes, reviewers sign the TESS in eRSS

Note: Reviewers may request to revise their score for any previously completed TESS up until the time of final adjournment, so that each reviewer leaves the meeting satisfied that they applied consistent stringency in scoring each proposal.

Reviewers sign and hand in post-review COl certification forms or sign the post COI in eRSS.

The Chair and SRO should sign and date the Minutes.

Comments by other NIH staff:

Program staff talks 10-15 minutes, with Q and A about intent of NIAID, Program, review sequelae, but not about conduct of review or reviewable elements

Contracts: Contracting Officer/Contracts Specialist, to describe what will happen after conclusion of review meeting, i.e., events leading to award.

Grants: Grants Management Specialist may wish to make a few remarks, but usually does not.

Streamlining (grants only):

Streamlining is conducted after introductory remarks and before the discussion of the first application. See “Special Review Situations”.

During the Review Discussions: Reminders that May be Necessary:

Institute staff (review, contracts, grants, program) may not offer any evaluative statements concerning any application/proposal, or performance of any applicant/offeror, but may provide factual background required by the committee.

For contract proposals:Reviewers must confine discussion strictly to points that belong under the particular evaluation criterion under consideration (don't blur the lines between the criteria). Hold other comments for subsequent discussion under the appropriate criterion.

Capture all crucial commentary.

Use of supplementary comment sheets or submitting a critique as an unassigned reviewer in grant reviews is encouraged.

In contract reviews, ALL comments must be on the TESS –no extra comments sheets should be used. Reviewers should update their TESS as proposal review discussions proceed rather than waiting until end of discussion. Appropriate for the same strength/weakness to appear in multiple TESS forms for the proposal under review;

Consensus on scores or issues is not necessary (but is desirable), as long as all opinions have an opportunity to be heard. Each reviewer must vote based on their own professional opinions.

For contract proposals, committee decisions to find a proposal Acceptable/Unacceptable are made by majority vote. Reviewers who may dissent with the majority can complete their TESS in accordance with their own decision.

For grant applications, decisions to streamline must be unanimous among all non-conflicted reviewers. Decisions to assign a score of NRFC are made by majority vote.

Grant applications reviewed by NIAID are not percentiled.

HS/VA/biohazard protection/representation issues. (SRO takes definitive notes during discussion, to determine/confirm codes.)

Throughout the discussion, the SRO should work through the Chair and minimize his or her direct interruptions.

SRO Reminders to Panel Before Adjournment

Thank Chair and all reviewers on behalf of the NIAID and gives final instructions to:

  • Maintain strict confidentiality; refer any inquiries to SRO.
  • Destroy any remaining review materials in a confidential manner.
  • Revise critiques to reflect final opinions after discussions.
  • Send SRO any information for non-flat reimbursement and mileage.
  • Register on SPRS for reimbursement of expenses and honoraria.