Instructions to Panelists for FY04 Application Review

FY2016 Panel Manual

Welcome

Welcome to the Missouri Arts Council Panel. Your role as a panelist is vital: we could not grant public funds without your help. As an experienced panel member your assistance is essential. Our goal is to provide funding that is fair, accountable, and accessible.

1 Before the Panel Meeting

Let’s begin by getting ready. The email you received from your program specialist included an agenda, username, password, deadline, meeting date and time. Please contact your program specialist if you have any questions throughout this process. They are your contact person at MAC and your support base here. We want to make sure you have the tools and answers you need to do your job as panelist.

We use an online panel review process. This offers several advantages. You will be able to review applications, view support material, and score and write evaluations, all online. Whether you use a desktop, laptop, or iPad, you will need access to a high speed internet connection and an updated internet browser such as Chrome, Firefox, or Safari. While Internet Explorer is supported, some users have experienced conflicts.

The Panel List includes your fellow panelists and identifies conflicts of interest. This list is for internal use only. Let your program specialist know if you have a conflict of interest that is not listed, or if your contact information is inaccurate or changes. In order to preserve panel privacy and integrity we do not allow applicants to contact you directly. We also ask panelists not to contact applicants. Panelists should rely on the application and support material. Applicants are not allowed to lobby or pressure panelists. If you are feeling pressured by an applicant, please tell your program specialist as soon as possible.
Evaluation criteria and scoring guide. Panelists review applications according to criteria. The criteria and weighting depends on the program the applicant applies to. It is important to note that the listed criteria are the only ones that should be applied to the application. While it may seem reasonable to apply other criteria, it is not fair if it is not applied across the board by all panelists. Please do not use additional criteria or compare applications to each other.

Application Management Supplement includes information for each application to assist in scoring Management Ability. Please note that it addresses only a few items of the Management Ability criteria and should be only part of the determination of the management ability score.
Applicant Support Materials: This year applicants had to upload support material online.

All the applications are eligible for funding. Some may include projects that cannot use MAC funds. This is addressed with stipulations in the applicant management supplement above.

Contact your program specialist if you have difficulty completing an evaluation or have questions. Plan about 45-90 minutes per application. Since the review process is time consuming, it is important to schedule your application review sessions over a four week period. Don’t’ wait until the last week. Some panelists recommend that you review a few applications each night, or several each weekend, whatever works best with your schedule.

The online review system has many features and benefits, but sometimes there are glitches in the system. Please contact your program specialist immediately if you experience any problems or something doesn’t look right.

Start at . Do not add www to the address. If you reset your password because you forgot it, please notify MAC staff. Enter your username and password to log into the grant system. Once you have logged in, you will see a link to your assigned panel. For example, Discipline Visual Arts Panel Review Form. When you open that link, you will see the applicants with their application number, name, criteria scores and total score. Next to that are four icons:

View Support Material: You may access attachments and support materials here.

Download Support Material: Another way to access attachments and support materials. However, this method does require space on your computer for saving.

Generate PDF Formset: Creates a viewable/printable/saveable PDF of the application, which includes links to the attachments and support materials.

Show Scorecard: This is where you will score (using a dropdown box), comment, or declare a conflict of interest. You must submit the scorecard in order to save your work. We recommend that you submit/save often. You may edit your scores and comments by clicking the star icon, make the revision, and selecting submit. The scorecard does not have editing tools.

New this Year: MAC added General Comments to the Scorecard to allow panelists to offer feedback that does not involve a score including suggestions, resources, and observations. The system will require that you select a score but it will not be used nor affect the applicant’s average score. Ignore the total score and tags box at the bottom of the page as they are not used in the process.

For each application you will:

Provide a score between 1 (lowest) to 10 (highest) for each criterion. The score should correspond with your comments. Note that the totaled score on your scorecard does not include the weighted average. That will be reflected on the final average score. Select the score that best reflects the applicant overall in each criteria. Assume that the applicant should get a 10 and then reduce the score to where it predominantly fits within the scoring guide. The applicant does not need to achieve every item in the category, score them where they predominantly fit. If the applicant does not achieve an acceptable level, you may score them less than six. Applicants with a total average score of less than six are not funded and may not apply for other funding for the same program in that fiscal year. If you give a score below six it should be consistent with the proportion of the criteria achieved. If they have offered some part of the criteria, they should get some points. For example, a score of one in management ability suggests that they have none of the required attachments, there is no clarity to their application, they have consistently late reports, and the budget is completely unfeasible, and exhibit considerable financial issues. Do not compare applicants to each other, only against the criteria.

Provide written evaluation for each criterion (1-3 sentences). This should address the proposal’s strengths and weaknesses and/or provide constructive criticism. Avoid repeating the narrative or comparing to other applications. The comment should be comparable with the numeric score, for example, a primarily positive comment should have a good score, while a primarily negative comment should accompany a lower score. Evaluations should express your thoughts about how the application and support materials address each criterion.

The most common error is to reiterate the application narrative (like proposed activity or budget details). While this information might be meaningful to you, the significance isn’t clear to applicants. Connect the description to the proposal’s strength/weakness or provide constructive criticism.

Example: 2 of 15 board members are African American.
This is a non-evaluative statement. It could be positive (strength) or negative (weakness), depending on the applicant’s community and target audience. You can turn this description into an evaluative statement regarding the Community Involvement criterion. See examples below:

▫ Concerned that only 2 of 15 board members are African American. Encourage the applicant to diversify the ethnic/racial composition of its board and staff. A more diverse board and staff should strengthen the applicant’s community outreach. Also, the ethnic diversity of the community should be reflected in the demographics of the staff and board.

▫ Pleased to see the strong effort to include ethnic minorities on the board. 2 of 15 board members are African American. This is an achievement for this rural community with 3% ethnic minority population.

Another common error is a discrepancy between the numeric score and written evaluation. Sometimes written evaluation ends up more critical/negative than the score reflects. While other times, the initial score was too low compared to the comments. To ensure an accurate evaluation, review the proposal, numeric score, and written evaluation and adjust the score or written evaluation accordingly. Below are some discrepancy examples. The ≠ symbol means “doesn’t equal.”

Artistic Quality: 9 ≠ The artistic presentations have variable quality. Need formal evaluation process.
A lower score is more appropriate for these comments.
Community Involvement: 6 ≠ The film series has an impressive amount of collaborative support from various organizations. Selected films appeal to different cultures that are represented in the community.
A higher score is a better match for these comments.
Management Ability: 7 ≠ The grant is an extremely well written. I am especially impressed to see this progression from a monthly strategic project grant. They have excellent support materials.
A higher score is a more compatible with these comments.

Remember to submit/save frequently when using the online panel system.

If you have a conflict of interest with one of the applicant organizations, select the checkbox “I have a conflict of interest with this user” within the appropriate scoring form. Declaring a conflict of interest will not hurt the organization’s average score.

Definition of Quality: While it is our goal to fund programs that are of high quality, MAC believes that applicants cannot be measured by a single standard of quality. A high quality performance in one area of the State may not be considered to be of high quality in another area.

MAC Request: Applicant scores shouldn’t be reduced for requesting 50% of the project budget since this is encouraged under MAC guidelines.

University/College programs are funded as a service to their community. Involvement by students, staff, or faculty is not expected. At least 30% of the audience should come from the community and be marketed to them.

A stipulation shouldn’t impact the management ability score. If it applies to a project that is ineligible for funding, they may be using it as part of their match. A good application will indicated which projects are not using MAC funds.

It is important that you complete your Panel Evaluation Form by the deadline date in order to return the results to panelists in time for the panel meeting. The program specialist needs time to collect the data, review the comments, and return them to you. As soon as you have completed all of your scores and comments, please notify your program specialist by email or phone.

Your Program Specialist will send the following items 2-3 days before the panel meeting:

Written Evaluation: Your Program Specialist will combine the written statements from all of the panelists into one document, editing for spelling and clarity.
Average Scores: The individual scores will be entered into a spreadsheet. The high and low scores for each application are removed before calculating the average score. The resulting spreadsheets will show the average scores by application number and by score (highest to lowest).

When you review the written evaluation and average scores,

Make sure that the meaning/intent hasn’t changed when they were edited for spelling and clarity.
Decide if you want to revise any of your scores and/or comments.
Identify any application that you'd like to discuss at the panel meeting.
Consider suggestions to improve the application or issues that you would like MAC to address.
The panelists' assessments vary. This is normal and MAC doesn't expect a consensus.

2 During the Panel Meeting

MAC staff will coordinate the panel meeting conference call from the MAC office (Old Post Office, 815 Olive Street, Gallery Level, Suite 16, St. Louis). Panel meeting chair and panel members may join the MAC staff in person or by conference call. However, MAC cannot reimburse panelists for travel expenses. The panel meetings are open to the public and applicants may observe the proceedings at the MAC office or call in on the conference line. Applicants are not allowed to address the panel members and the panel may not ask the audience questions. However, if an applicant in the audience observes erroneous information presented by a panelist or MAC staff, that person may immediately contact MAC by phone if not in person. The panel chair in consultation with MAC staff will bring this concern to the panel’s attention only if it is relevant to the discussion.

The Panel Chair, a MAC Board Member, will begin the meeting by asking panel members, staff, and audience to identify themself. The panelists on the phone will identify themselves when they speak to help staff record the panel meeting minutes. During the orientation, panelists will have an opportunity to update the conflicts of interest list. The meeting is conducted using the most recent edition of Robert's Rules of Order.

The Program Specialist will present each application in application number order, announcing the application number, applicant name, initial average score, and range of panelists' scores. If necessary, the Program Specialist may ask for clarification and/or corrections to the presented panelists’ written evaluation. They may also note any stipulations or recommendations.

Panel chair will moderate discussion and ask the following question after each application: Are there any comments or changes that would affect the average score, or any motions for recommendations or stipulations? If a panelist wants to change their applicant score they will announce that they are modifying their score to a specific score. No motion, second or discussion is necessary. The score will be adjusted in the system so that a new total average score can be calculated and announced.

After all the applications have been presented, the panel makes a formal motion to approve all of the final average scores. This requires a second and then a vote.

At this point the Program Specialist will review issues that were raised during the meeting. If any ideas or questions come up during the panel meeting about policies, process, programs, services, or issues critical to the discipline, they will be held for later discussion. MAC welcomes your suggestions and ideas.

The meeting ends with a formal motion to adjourn, marking the end of your duties as volunteer panelist. Your responsibility is completed. If applicants approach you for information about the panel review process, how they performed, etc., please advise them to talk to their MAC Program Specialist. This will protect you and the process as well as insure that a consistent message is delivered to all of the applicants. You will receive a email survey after all the panels are completed asking for your feedback on this process.

3 After the Panel Meeting

The panel recommendation is presented to the full Council. The Council approves the budget and grant funding at the June MAC Board Meeting. The grant amount is determined by the budget, application request, and applicant score as recommended by the panel. The formula directly relates to the application scores, rewarding those applications that most clearly and directly address the criteria. The grant request is multiplied by the score as a percentage. This formula is applied to each application multiple times until the total grant recommendation equals the funds available for the program.

After July 1, MAC sends written notification of the MAC board funding decisions to applicants. This notification will include panel comments (not attributed to the panelist) to help applicants understand how their application was reviewed and how to make a stronger application. Also, if applicable, any panel recommendations and/or grant stipulations. The program specialist will also send the results to the panel.

If you have any questions, contact your program specialist directly. Thank you again for your invaluable assistance as a panelist.

P:\PANELS\Panels FY16\Panel Book Material\FY16 Panel Manual.doc Updated 4/9/15

Page 1