Division of Independent Review (DIR)
Reviewer Instruction Guide
Web-Teleconference and Face-to-Face
Last Update: February 3, 2016
Teleconference and Face-to-Face Reviews
TABLE OF CONTENTS
1 INTRODUCTION 2
2 OVERVIEW 2
3 HRSA RESPONSIBILITIES 2
3.1 Review Administrator 2
3.2 Program Staff 3
3.3 Grants Management Specialist 3
3.4 DIR Contractor 3
4 REVIEWER RESPONSIBILITIES 4
4.1 Conflict of Interest 4
4.2 Confidentiality 5
5 CHAIRPERSON RESPONSIBILITIES 5
6 REVIEW PROCESS 5
6.1 Pre-Review 5
6.2 Review 7
6.3 Post-Review 9
ATTACHMENT A—GUIDELINES FOR WRITING STRENGTHS AND WEAKNESSES 11
ATTACHMENT B - WHAT DO I DO WITH THE BUDGET: A GUIDE FOR HRSA REVIEWERS…………………………………………………………………………………………..….13
ATTACHMENT C - HRSA SCORING RUBRIC………………………………………………………..16
Teleconference and Face-to-Face Reviews
1 INTRODUCTION
The Health Resources and Services Administration (HRSA) Division of Independent Review (DIR) is responsible for planning, directing, and managing the objective review of new, competing continuation, and program expansion supplement applications for HRSA discretionary grant and cooperative agreement programs. The primary purpose of an objective review is to ensure that each application receives a fair and objective assessment consistent with the published review criteria.
HRSA achieves this goal with the assistance of Reviewers who possess the expertise and/or skill-sets appropriate for the program under the Funding Opportunity Announcement. Reviewers are expected to provide a comprehensive evaluation of the merits of applications. HRSA program officials use those evaluations to make award decisions.
This document provides instructions for Reviewers, including a timeline for deliverables associated with the review process, and describes roles and responsibilities of HRSA and DIR Contractor staff involved in the review process.
2 OVERVIEW
DIR conducts objective reviews using the Application Review Module (ARM), a web-based application review tool. It is a user-friendly system designed specifically to meet the objective review requirements of HRSA Grant programs and provide a cost effective and efficient system to review grant applications. Reviewers will use this tool when evaluating and scoring assigned applications. For ARM specific procedures and instructions, Reviewers should refer to the separate ARM Reviewer Manual.
Reviewers begin the process by independently evaluating and scoring assigned applications. For Teleconferences, Reviewers then convene via telephone and the internet as an objective review committee consisting of one or more review panels to discuss the applications. For Face-to-Face review meetings, Reviewers then convene as an objective review committee consisting of multiple review panels at a designated central location, usually in the Washington, DC metro area, to discuss the applications.
Each application is assigned to three Reviewers. Prior to the meeting, Reviewers evaluate and score their assigned applications based on how thoroughly they determine each application meets the published review criteria in the program Funding Opportunity Announcement (FOA). Reviewers record their initial scores and note strengths and weaknesses by criterion within ARM. During the objective review panel meeting, the assigned Reviewers will be asked to present their evaluations and scores for each application. Following the presentations, the full panel will discuss each application to develop a final summary statement. Each Reviewer on the panel will enter and submit final criterion scores for each application into ARM. ARM calculates an overall score for each application based on the average of the individual Reviewer’s scores. See section 6, “Review Process,” provides detailed instructions for Reviewers in the conduct of the review.
3 HRSA RESPONSIBILITIES
3.1 Review Administrator
A Federal Review Administrator (RA) is assigned to each review. The RA manages the review process and provides assistance to Chairs and Reviewers to ensure compliance with applicable statutes, regulations, and policies and to maintain an objective review process.
Typically, the RA:
· Ensures the review is fair and objective;
· Provides oversight and direction during the objective review process;
· Manages Reviewer potential or actual conflicts of interest and ensures that Reviewers maintain the confidentiality of applications and review information throughout the review process;
· Convenes and facilitates the mandatory pre-review conference call;
· Oversees the objective review panel discussions, including the production of summary statements and scoring of applications; and
· Participates in the Reviewer evaluation and post-review debriefing session.
3.2 Program Staff
HRSA program staff members serve as a resource during the objective review process to ensure that program policies, procedures, and goals are carried out in accordance with prescribed requirements (i.e., legislative requirements, HRSA policy, and program-specific policy). Specifically, program staff:
· Participate in the pre-review conference call;
· Upon request, provide program expertise, particularly regarding application materials, application guidance, legislation, review criteria, and applicable funding factors (preferences, priorities, and/or special considerations); and
· Respond to any questions that require further explanation regarding programmatic issues.
3.3. Grants Management Specialist
Grants Management Specialists (GMS) are responsible for ensuring that applicants and applications comply with HRSA statutes, regulations, and policies that pertain to financial and business management matters. In collaboration with program staff, grants management staff screen applications to determine eligibility. A GMS is assigned to each review as a resource. Specifically they:
· Participate in the pre-review conference call;
· Respond to any questions that require further explanation regarding grants management issues, such as:
o Whether budget items are allowable;
o Applicant eligibility;
o Applicant compliance with required assurances; and
o Federal and HRSA grants policy, regulations, and procedures.
3.4 DIR Contractor
A contractor assists DIR RAs by providing technical, administrative, and logistical support services for the objective review. Specifically, the contractor assists with:
· Distributing review materials;
· Collecting and tracking of documents received from Reviewers;
· Consolidating individual Reviewer evaluations prior to the objective review panel meeting on each application; and
· Coordinating payment of honoraria to Reviewers.
Any Reviewer questions that are not addressed in the pre-review conference call should be addressed to the RA.
4 REVIEWER RESPONSIBILITIES
The primary responsibility of a Reviewer is to independently, fairly, and competently evaluate the merits of applications consistent with the program purpose and published review criteria. Each Reviewer is generally assigned to read and score up to eight grants or cooperative agreement applications. The number of applications assigned may vary depending on the program complexity, the number of eligible applications received, length of the applications and the time available for independent reviewing. All current and potential reviewers must attend a HRSA DIR Quarterly Reviewer Training session.
It is imperative that Reviewers follow the DIR Reviewer Instruction Guide and the FOA and complete assigned tasks in accordance with established deadlines. The potential integrity of the review process may be compromised if a Reviewer does not complete the assigned work or does not deliver the work on schedule. Reviewers must make all necessary arrangements to be available to complete review assignments, attend and participate in the entire objective review committee meeting. If for some reason a Reviewer believes that they are unable to fulfill these duties, or if unforeseen circumstances prevent the Reviewer from completing their duties, they must immediately contact their assigned RA and or the contractor representative. Honorarium will be paid only after all of the assigned responsibilities have been fulfilled.
4.1 Conflict of Interest
During the pre-review conference call, the RA will provide additional guidance regarding conflicts of interest. To achieve the goal of a fair and objective review, Reviewers must not carry any prejudices, biases or conflicts of interest into the review process. Application and panel assignments are made by DIR in consultation with Program staff to avoid biases, e.g., applications are assigned to Reviewers from a different geographic area than the applicant. The Reviewer is obligated to declare any and all potential or actual conflicts of interest, as soon as possible.
Upon receiving the review materials, all Reviewers are required to complete, sign, date and return the Conflict of Interest and Confidentiality Certification Form as directed. The Reviewer can attest to having no conflict of interest with the assigned applications or can include details regarding potential or actual conflicts of interest. In the latter situation, the DIR RA will reassign all applications to comply with HRSA policy prohibiting the participation of Reviewers where he or she has a potential or actual conflict of interest regarding any application to be reviewed by the objective review committee. No person who serves as an officer, director, trustee, partner, or employee of an applicant, its parent or subsidiary organization may serve as a Reviewer on the committee.
If the Reviewer believes that there exists a conflict of interest with any assigned application for review, or if it could be perceived that there is a conflict of interest, they must immediately inform the RA. It is never too late to disclose a conflict of interest; however, Reviewers should attempt to identify any conflict of interest as early as possible to avoid compromising the integrity of the review.
At the beginning of each application’s review, the Chairperson will ask the Reviewers to affirm that no potential or actual conflict exists. In the rare instance in which a Reviewer discovers that he or she has a conflict of interest or a reasonably perceived conflict, he or she will be recused for the duration of that application’s review. For Teleconferences, the Chairperson will be asked to note that the reviewer was recused for conflict of interest, and the reviewer will disconnect from the teleconference until notified via phone call to rejoin the review. For Face-to-Face review meetings, the recused Reviewer will be asked to note on the score sheet that he or she was recused for conflict of interest, sign a score sheet (without providing a score), and will leave the room until notified the discussion of that application has concluded.
4.2 Confidentiality
All review materials and the proceedings of objective review committee meetings are privileged information for use only by HRSA staff, DIR Contractors, and assigned Reviewers. At no time prior to, during, or after the review of applications should Reviewers discuss the applications, comments, recommendations, evaluations, scores, names of applicants, or names of other Reviewers with anyone not involved in the objective review process. Questions regarding applications to be reviewed should be directed to the RA. Direct contact or communication with applicants or associated persons is strictly prohibited.
Please note that all comments made by the Reviewer are treated as confidential information. In situations where an applicant (or other individual) requests review-related information via the Freedom of Information Act (FOIA), the Federal Government is required to release the names of all Reviewers who participated in the review activity; however, the names of the individual Reviewers will NOT be associated with specific application assignments.
5 CHAIRPERSON RESPONSIBILITIES
Each objective review panel has a Chairperson whose primary responsibility is to preside over the panel discussions and assure that each application is accorded a fair, ethical and equitable review. The Chairperson must also manage the schedule to assure timely yet thorough review of all applications. The Chairperson does not participate as a Reviewer and therefore does not contribute subject matter to the discussion, assist in the determination of an application’s merit, or score individual applications. The Chairperson does ensure that the contributions, views, and questions of all Reviewers are heard and given equal consideration, and manages the development of the final summary statement and the editing process. At the end of the review, the Chairperson certifies the results of the review.
6 REVIEW PROCESS
6.1 Pre-Review
6.1.1 Pre-Review Conference Call
Reviewers and Chairpersons are required to participate in a pre-review conference call with the RA, Grants Management staff, and the Program staff. The purpose of the conference call is to provide an overview of the review process and give Reviewers the opportunity to address with HRSA staff any questions or issues regarding review policies and procedures. If a Reviewer is unable to attend, an acceptable explanation must be discussed with the contractor and RA in advance of the conference call. Instructions on how to access the recorded conference call will be provided and the Reviewer must make time to listen to the recorded session prior to the objective review committee meeting and report to the contractor and RA that they have listened to the recorded session. Subjects addressed during the pre-review conference call include:
· Confirmation regarding access to review materials in ARM;
· General information regarding the objective review process;
· Presentation of Reviewer responsibilities including confidentiality and conflicts of interest;
· Guidance on developing and writing clear, concise, and accurate evaluations;
· Questions regarding honoraria ;
· Questions regarding program objectives or background information (Program);
· Questions regarding grants policy or application budget considerations (Grants Management); and,
· Questions regarding review logistics.
Immediately after the conference call, an ARM login and password as well as other information required to begin the evaluation process will be emailed to each Reviewer.
Any Reviewer questions that arise during or after the pre-review conference call should be addressed to the RA.
6.1.2 Review of Applications and Preparation and Submission of Preliminary Documents
Reviewers are required to read each assigned application in its entirety, evaluate and score each one based on the published review criteria, and enter their scores and comments in ARM. Please note that reviewers are required to evaluate and score only those applications assigned to them on the Reviewer Assignment Sheet. Reviewers must also read and familiarize themselves with the other applications assigned to their panel.
6.1.2.1 Completing the Evaluation
To complete the evaluation, Reviewers must enter the scores by criterion as well as the strengths and weaknesses for each assigned application in ARM (ARM will calculate the total score). For assistance in preparing these comments, see Attachment A— Guidelines for Writing Strengths and Weaknesses.
Reviewers must independently consider and separately score each criterion. The total score for an application (calculated by ARM) is the sum of all the criteria scores. All Reviewer criterion scores must be justified with written strengths or weaknesses. It is recommended that Reviewers read the entire application before entering their scores and comments in ARM. In addition to scores, strengths and weaknesses, Reviewers enter budget recommendations and, if applicable, funding factors. When Reviewers complete the evaluation in ARM, they must click on the “Submit to Chair” button. Please note that once this process is completed, Reviewers are no longer able to modify their scores or comments. In preparation for the objective review committee meeting, reviewers should print the merged evaluations for reference and presentation of their assigned applications. Saving the document for reference at the objective review is encouraged.