How to Score the Questionnaires by Peer Reviewers

There are many grant application review systems and methods of scoring being used by grant giving organizations around the world. Experience has shown that the overall system should not be too complicated. The system described here mayseem complicated but once it has been run it should be seen as straightforward to implement.

No review system is perfect. All are subject to the opinions of individual reviewers which may not always be rational or fully informed.

The system proposed in this document is based on scoring systems used by the European Union, US National Science Foundation, and the US SBIR and STTR programs, and my personal experience as a grant application reviewer for several organizations.

  1. General Features
  1. The score assigned to an application should be a quantitative representation of the merit of the application alone and not by influenced by any other knowledge a reviewer may have about the applicant. That is, judge only what is written in an application. This can be difficult if you know the applicant could have written a better application.
  1. If a reviewer has any business relationship with the applicant’s project, or there is any other conflict of interest, he/she should exclude himself/herself from reviewing the application.
  1. For transparency, the review and scoring system should be provided to applicants before they prepare an application for grant support so the applicants are fully aware of the evaluation procedure.
  1. For transparency, unsuccessful grant applicants should be allowed to see their scores (but not the names of the reviewers) to help them understand the weaknesses in their application and prepare better applications in the future.

These are ideal conditions. In practice, compromises may sometimes have to be made.

  1. Review Procedure
  1. Each application should be reviewed by at least 4 reviewers who have knowledge of the field of science, technology, and markets for the innovation being scored.
  1. All reviewers attend a briefing meeting at which the review and the scoring system are explained and any questions answered.
    [After the system has been run several times, reviewers can be given sample applications, with sensitive data removed, as examples of a strong and a weak application]
  1. Each reviewer is provided with the grant application and scoring system.
  1. Each reviewer reviews and scores each application separately from the other reviewers.
  1. Each reviewer submits his/her preliminary score for each application reviewed to the manager of the review procedure.
  1. The manager of the review procedure calls a meeting of all the reviewers. If some of the reviewers are at a distance they can join the discussion by phone or video link. Procedures should minimize the burden for reviewers both before and at the review meeting.Prior to this meeting, reviewers should not discuss the applications or their scores with each other or with the applicants.
  2. During the review meeting the preliminary scores of each reviewer for each application and the arithmetical average for each application are displayed.
    [It is useful to multiply all averaged score by 10 as most people find whole numbers easier to deal with than decimals].
  1. The reviewers explain the reason for giving their scores and each application is discussed. Very strong and very weak applications may not need much discussion.
  2. Special attention should be given to cases where the scores of one or more reviewers differ widely. Each reviewer has an opportunity to change their scores based on the group discussion, and the arithmetical average is adjusted accordingly. For example, it may be decided by the group that an exceptional strength in one area might outweigh a weakness,which can be corrected, in another area. Or, one reviewer may have missed something important in an application.
  3. Final scores for each application are displayed to the review group. Winners are selected from the highest score applications based on the budget available. [There will frequently be natural “gaps” in the scores which can help the choice of a group of winners and perhaps a second group which could be funded if there are sufficient funds].

Where a 1 to 5 score is assigned to Questionnaire section the meaning is:

5 Excellent / Exceptionally strong with essentially no, or only minor, weaknesses
4 Very Good / Very strong with only some minor weaknesses
3 Good / Strong but with numerous minor weaknesses
2 Satisfactory / Some strengths but also some moderate weaknesses
1 Poor / Very few strengths and several major weaknesses

Alistair M. Brett

Science Technology Innovation Global Expert Team (STI GET)

The World Bank

28/1/2010