Guidelines for Special Emphasis Panel (SEP) and
Technical Evaluation Panel (TEP) Chairpersons
The Chairpersonplays a key role in the function and success of a special emphasis panel (grant applications) or technical evaluation panel (contract proposals). The leadership that you and the Scientific Review Officer (SRO) provide is essential to ensuring that the highest quality science reviewed by your special emphasis panel receives the best scores. The panel should operate in a fashion that yields high quality and fair reviews.
To help you in this endeavor, the guidelines below were adapted from those developed by several former study section chairpersons in collaboration with SROs in the Center for Scientific Review (CSR) and the experiences of NIAID SROs. These are not "rules of conduct”, but they are suggestions and guidelines that you can incorporate to fit your own style and the needs of your special emphasis or technical evaluation panel.
Chairing the meeting: You are responsible for the flow and focus of the scientific discussion at the meeting. How you chair the meeting will strongly influence whether the discussionsare fair and focused, and appropriate scores are assigned. Here are some suggestions about how to facilitate useful discussions. Keep in mind, the SRO is there to assist you as needed.
Preparation before the meeting: Many Chairpersons find that it helps to maintain focused, balanced and fair discussions if they are familiar with most or all of the applications/proposals* under review. To compensate for the time this involves, you may ask the SRO to assign fewer applications/proposals to you for explicit review. However, it is important that you remain an assigned reviewer of some applications/proposals, since you will establish a role model for other panel members.
Soon after the applications/proposals are sent to reviewers, the SRO will check to make sure that reviewers are comfortable with their assignments. Panel members are encouraged to call the SRO prior to the meeting if they have reservations about their expertise for any of their assigned applications/proposals.
If there is a pre-review orientation teleconference, make every effort to participate. The presentations from NIAID Review, Program, and Contracts or Grants Management staff during the teleconference assist the panel in their preparation for the review meeting.
It is also important to maintain contact with the SRO, particularly if you have questions or want to discuss an important issue. You should also plan to meet with the SRO just before the review meeting to go over any last minute issues and for final preparations. This meeting can be the evening before (preferred), or the morning of, the review meeting and provides an opportunity to establish a working relationship. (Note: If the review meeting is via teleconference, you and the SRO should have a planning telephone conversation a day or two before the review.)
Begin with synopses. At the meeting, the review of an application/proposal starts with each assigned reviewer briefly indicating his/her overall level of enthusiasm for the application. This synopsis can include a rough numerical score or range of scores. Generally, the primary reviewer (or a designated discussion leader) should give a brief, non-evaluative summary/overview of the application/proposal.
Spend time wisely. Meetings should move at a pace that ensures reviews are fair and of high quality; review discussions should focus on key issues and on the applications whose fate is less clear.
(a) Discourage panel members from reading reviews in their entirety. Rather, encourage reviewers to state those issues that most influenced their levels of enthusiasm. This approach helps to keep the committee engaged and the discussions focused. However, make sureimportant issues are discussed completely.
(b)Invest time where it is most needed. Spend the most time on applications/proposals where there is greatest disagreement, especially if the application/proposal is likely to be in the best 20 to 30% of those reviewed. Less time should be spent on those applications/proposals where there is uniform high or low enthusiasm. However, for contract proposals, all of the technical evaluation criteria must be discussed for each proposal, regardless of scoring range.
(c)Remember that discussion does not always lead to consensus. It is essential that all major issues are aired, and the reasons for differences of opinion are clear to all.
However, once this has been accomplished, all opinions have been heard, and further progress toward consensus is not being made, you should terminate the discussion. Recapitulate the key arguments, have the reviewers state their "post discussion" enthusiasm, and then ask panel members to vote as they see fit.
Applications/proposals in which discussion does not lead to consensus need special attention. Remember that the only substantial information conveyed to an applicant (other than score) will be the written critiques of the reviewers and a summary of the discussion, prepared by the SRO. (For contract proposals, the Offeror will be given a debriefing based on the comments reviewers write down on their TESSs, which are then used by the SRO to generate a Technical Evaluation Report.) Applicants/Offerors need to receive the clearest possible assessment of their application/ proposal. In that spirit, remind reviewers that they can (and should) modify their reviews in light of the discussion. In addition, you or the SRO may ask discussants who raised particularly important points to write a brief comment. The SRO's written evaluation of the discussion is particularly critical. If the SRO asks you to review a summary, be willing to do so.
Promote balanced discussion.
(a)Help the panel concentrate on the most important issues. Reign in wandering discussions. Interrupt if necessary.
(b) Don't allow one person to monopolize the discussion. Create an open atmosphere and encourage reticent reviewers to speak up. Once the assigned reviewers have finished presenting their critiques, encourage the rest of the panel to express their opinions.
Be a guardian of fairness.
(a)Though this rarely occurs, watch for evidence that a reviewer may be influenced by inappropriate personal interests (competition, scientific bias, personal antagonism, etc.). If you sense that this might be happening during the review, determine a diplomatic way to handle it, such as inviting the opinion of other members of the committee. After the meeting, speak in private to the SRO if you have serious concerns about the fairness of any of the reviews.
(b)Beware of your own biases: Although you shouldn't hesitate to state your scientific opinion when appropriate, be cognizant of your role as Chairperson, and don't champion your favorite areas of science over others.
Promoteconsistent scoring. To maintain fairness and to provide the best advice to the Institute about scientific and technical merit, a broad score range should be used. (For grant applications, even if 50% of the applications are streamlined, reviewers are encouraged to use the full score range (1 - 9) for the remaining applications. For contract proposals, the full range of allowable points for eachof the technical evaluation criteria should be used.) Maintaining a consistent approach to scoring from the beginning to the end of the meeting and from reviewer to reviewer requires constant diligence by the Chairpersonand the SRO. Set an example by your own scoring behavior, and feel free to challenge a reviewer who appears to be scoring improperly (in either direction). In addition, remind committee members who plan to vote outside the range of scores discussed by the assigned reviewers, to state their opinion clearly before the committee.
Ensure that criteria-based scoring is used properly. During the discussion, each criterion should be explicitly assessed and the reviewers should state how it affected the overall evaluation. Because it is up to the reviewers to determine the appropriate weight of each criterion in determining the score, serious consideration must be given to each criterion.(Note: For contract proposals, the technical criteria to be evaluated will have pre-weighted point values.)
Speak up. Evaluate whether the discussion corresponds to the score assigned. If not, say so; other panel members likely will agree with you but have remained silent.
Makesure other issues are discussed. Make sure that other issues important to the NIH are aired adequately. These include comments or concerns about: vertebrate animal welfare; human subjects protection; the inclusion of genders, minorities, and children; biohazards; and if required, the following: data sharing plans; recombinant DNA/select agent use; genome-wide association studies; and, the justification/unique resources of a foreign site for foreign applications.
Budget. Make sure that budget issues are addressedproperly at the end of the discussion, after the finalscore recommendations are made. Work as a team with the SRO to make sure that scoring decisions were not affected by an inappropriate consideration of the budget.
* application = grant application; proposal = contract proposal
1