[Facility Name]

[Exercise Name]

[Exercise Date]

Evaluator Guidelines and Exercise Evaluation Guide User Guide

The goal of this exercise evaluation is to validate strengths and identify improvement opportunities for the [Plan]. In the [Exercise Name], evaluation will attempt to validate plan contents, procedures, and protocols of participating agencies and determine their level of capability with regard to the exercise target capabilities. Discussion-based validation attempts to answer the following questions:

  • Are plans, procedures, and protocols established, and were participants knowledgeable of them?
  • Are personnel identified to execute and support the plans and procedures?
  • Has sufficient training been provided to the identified personnel?
  • Is appropriate equipment identified and available for use to execute procedures?

Evaluation Steps

  1. Pre-exercise Preparation

Prior to the exercise, you should be familiar with the exercise objectives, scenario for discussion, key evaluation areas and Exercise Evaluation Guide contents

  • Review appropriate plans, procedures, and protocols to include the [Plan].
  • Attend required evaluator training and other briefings.
  • Review appropriate exercise materials, including the exercise schedule and evaluator instructions.
  • Review the Exercise Evaluation Guides and other supporting materials for your area.
  1. Observing the Exercise

During the exercise, you should not focus on filling out the Exercise Evaluation Guide Analysis forms; listen closely to the participants’ discussion and take detailed notes relative to your assigned area of evaluation. You should be familiar with the activities and tasks for evaluation, but do not jump ahead to analysis until the exercise has ended. As applicable make references in your notes to applicable exercise objectives or Target Capabilities.

  • Do not prompt players with specific responses or interfere with player performance in any way.
  • Your primary duty is to document player discussion. After the exercise, that information will be used to determine whether the exercised capabilities and plans were effectively demonstrated and to identify strengths and improvement items.
  • Reference Exercise Evaluation Guides to confirm objectives are being met; advise facilitator if they are not
  1. Data Analysis

Following the exercise, you should use your Exercise Evaluation Guide Data Analysis Sheet to help organize your notes in categorized data for the After Action Report. The Analysis Sheet lists all of the critical tasks that should have been discussed, based upon exercise objectives. Similar tasks are grouped within Activity tables. As you complete the forms, you will identify strengths and areas for improvement for each Activity. These bullet points will serve as your observation titles for the After Action Report input. Post exercise as you are conducting your analysis, you should:

  • Participate in the Hot Wash, and take notes on findings identified by players. Before the Hot Wash, do not discuss specific issues or problems with participants.
  • After the Hot Wash, summarize your notes and prepare for the Controller and Evaluator Debriefing. Have your summary ready for the Lead Evaluator.
  • Use data to determine whether the exercise capabilities and plans were effectively implemented or demonstrated.
  • Use data to identify Observations as strengths or areas for improvement.
  1. After Action Report Development

In this step, we take our detailed notes from observing the exercise that we analyzed at the activity level, and provide detailed recommendations for continuous improvement. The After Action Report input form has four major fields that should be completed for each strength or area for improvement.

  • Observation – this is a short one sentence statement that identifies the strength or area for improvement as noted in the Activity analysis tables. This field is also followed by check boxes to identify the capability preparedness element that the root cause of this observation falls under
  • Reference – This field provides an area to reference Sections of the NVS plan, other State Plans, or industry standards that could be used to facilitate implementing recommendations for improvement or capturing strengths.
  • Analysis – This field provides an area to document detailed data on what you observed/overheard during exercise discussion specific to this individual Observation. This is where you will provide the Who, Why and What of the strength or area for improvement based on your notes taken.
  • Recommendations – These are suggested steps to allow the NVS Team to further improve the NVS Plan or share strengths with other agencies. Recommendations should be targeted specifically to the individual observation, and the capability element that we determined to be the root cause of the observation.

Do not forget, you should complete one AAR Input form for each bullet from your Activity analysis tables. The exercise planning team will use these input forms to directly build the exercise After Action Report.

1