Section 9: Assessing Your Program

Section 9:

Assessing Your Program

Evaluation enables the camp director, leadership staff, and session facilitators to determine whether the staff Orientation and subsequent training experiences justified the investment of time and staff resources, and whether the Orientation experience helped to achieve the goals and objectives of the camp. Even the most effective Orientation and training programs can be improved. Suggestions for these improvements should be sought from all staff members and from data provided by existing evaluation documents.

Evaluation is usually divided into two types: formative and summative. Formative evaluation occurs during training; this allows training to be adapted or revised to meet the needs of the participants. This not only makes the training more effective, but proves to the staff that you care about their opinions. Summative evaluation occurs after the fact; it determines the effectiveness of training, and what must be done additionally to train staff in the future.

PreSession Response Forms

Facilitators may wish to gauge the attitudes of participants prior to a training experience. PreSession Response Forms are an especially valuable tool if you wish to target the immediate needs of your staff. Such a form is provided in Appendix R.

PostSession Evaluation Forms

Camp directors should provide a way for staff to evaluate each session during Orientation. Designed properly, evaluations will provide information regarding the value of the workshop to each particular participant by asking participants how they will use the information and training which were shared in the workshop. See Appendix S for a sample session evaluation form.

From these evaluations and their own observations, leadership staff should be able to answer a number of questions regarding the Orientation program, including:

  • Were the training sites appropriate?
  • Was the training conducted as planned? What last minute changes were made, and why?
  • Did the trainers have the appropriate knowledge, skills, and style?
  • Did the session content maintain participants' interest?
  • Was sufficient opportunity provided for participants to ask questions and receive feedback?
  • Were transitions smooth during training?
  • Was the time allotted for Orientation sufficient?
  • Were the materials appropriate and adequate?
  • What were strengths and weaknesses of the day-to-day schedule?
  • Which logistics created problems? How can these issues be resolved for future training?
  • What participant comments (from evaluations) can help to improve the training process?
  • How effective are staff in carrying out their assigned tasks and in accomplishing their individual goals?
  • Did participants have the prerequisite knowledge and skills necessary for obtaining maximum benefit from the training?
  • What training is still needed for this season?

Participant Feedback Sessions

As an alternative to session feedback forms, leaders can stage participant feedback sessions. Participants are asked questions similar to those included in session questionnaires. When necessary, facilitators can seek elaboration or clarification of ideas which cannot be done on written forms. Such sessions should be conducted daily so that adjustments can be made to training while Orientation is in progress.

These sessions should not be conducted by staff who led the day’s sessions. It is also important to create a diverse mix of staff respondents in order to hear from all sides.

Session Observations

Enlisting qualified observers to watch training sessions will improve subsequent experiences. A competent observer should be able to comment upon:

  • the relevance of session content,
  • the participants’ acquisition of that content,
  • the quality and flow of the learning activities, and
  • the degree of participant involvement and interest.

Observers can include leadership staff, professional teachers, DHQ staff, guest facilitators, and staffers with many years of experience. The primary goal of this method is to identify positive and negative aspects of the training session which may not be readily apparent to either the participants or the trainer since their attention is focused on participating rather than observing.

The observer should be willing to debrief the session with the presenter, tactfully discussing both the strengths and weaknesses of the delivery. A more important issue, however, is whether the content was received, and ways in which to improve the delivery in the future.

Staff Journals

A more intense and personal way to gather data regarding the Orientation experience is to have staff members keep journals. These journals can be blank pages in the Orientation program or staff manual, or thin spiral bound notebooks which sell for as little as twenty cents each.

In order for the journals to be most effective, it is suggested that guiding questions or prompts be given to staff before, during, or after each training session. Although prompts should be custom written to fit the training experiences offered, the Session Openers/ Closers on 4:11 provide ideas for generic prompts. Additional response ideas may include the following:

Prior to a Session:

  • Preview the title of this workshop. What do you feel you already know about this topic? What else would you like to learn?
  • Reread your most recent response in this journal. What emotions were you feeling at the time? How do those compare to your present emotions?
  • Read the quotation provided (on a tablet, screen, monitor). Do you agree? Or, why do you think this is true? Or, in what way does this relate to the camp experience?

During a Session:

  • Choose one of the ideas shared thus far. Explain how you can use that idea with your campers.
  • Jot down the five most important words you have heard since the start of this session. Then, write a single sentence using these five words (and others) which expresses an important thought you could take from this session.
  • Write down an idea or thought you had which was “triggered” by the facilitator, or by another member of your group.

Staff members can volunteer to share their responses aloud, responses can be shared within small groups, or journals can be randomly collected for selective reading by the leadership staff. Just be certain that staff realize that this is a training journal, not a personal journal, and that the responses (for the most part) are meant to be shared.

If you do wish to allow staff the option of privacy, you might suggest they simply fold the upper right corner of the page over to the center of the journal, with the guarantee that such pages will not be read. Then, respect those wishes!

Another way to encourage staff to respond to prompts is to establish a Message Mural in a central location (see Appendix F, page 53). This format allows not only the individualized feedback that directors seek, but also allows staff members to respond to peer comments, and to “piggyback” upon good ideas. Plus, “burning questions” can receive a number of responses in a very short time.

Final Staff Evaluations

As mentioned earlier in this manual, evaluations should be handed out during the Orientation week so that staff members will know precisely what is required of them (see AppendixL2). Supervisors can ask staff to record anecdotal information under each trait in order to assess their own progress. That simple requirement really “turns the tables” on the whole notion of assessment, since it forces staff members to cast a critical eye upon themselves. Then, at conference time, supervisor and staff member can compare notes. The staff member is able to boast in an honest way, and supervisors are given a more realistic picture of what the staff member has accomplished beyond what they were able to observe firsthand.

Another type of evaluation which many camps use is a reverse evaluation, in which staff evaluates their supervisors. Again, it is only fair that supervisors be provided a copy of this form prior to the process, as the form itself is an incredibly powerful tool for training. This type of evaluation needs to occur early in the season if it is to serve any value.

Staff Feedback

At summer’s end, all staff members should be given the opportunity to provide feedback and suggestions on personnel, facilities, and program. Allow staff to answer anonymously. This feedback should be reviewed as soon as possible, since some of the best suggestions may require implementation before the start of the next summer. Since the natural tendency is to remember only the good times and forget the bad, such feedback acts as a concrete reminder of issues that must still be resolved.

A feedback form should include questions such as:

  • In what ways can the existing program be improved?
  • In what ways can the site be improved?
  • What suggestions would you make for changes in next year’s schedule?
  • How can any existing procedures or routines be improved for next summer?
  • What kind of additional training might have made your job easier this summer?
  • What suggestions do you have for next year’s Orientation and in-service training sessions?

Program Area Briefs

Section 3 described how a Program Area Brief is intended to do two things:

  1. Summarize a program area or event, and
  2. Make recommendations for future use of area or future programming of the event.

A well done program brief can help the camp director to evaluate the effectiveness of a particular program site, while providing clues for how to improve the site in coming seasons.

The Brief will also document the fact that all presented activities were age appropriate and well designed, in the event that a question arises after the summer concerning any given site.

See Appendix I for a sample outline of a Program Brief.

Home Calls

Rarely will parents or other caregivers take the time to write or call the camp to relate a positive story. They are, in fact, more likely to call if there is a problem. It is incumbent upon the camp itself, then, to contact parents of campers to assess their level of satisfaction. This is not a task that should be relegated to the end of the summer. The data that can be collected from just two or three calls is enough to provide your leadership with numerous ways to improve the camp’s services.

You may also wish to design a survey to be sent to parents; keep in mind, however that the return rate will be higher if you

  • keep the question format simple (1-10 rating or true/false);
  • keep the language simple; and
  • provide an addressed, stamped envelope.

Agency Contacts

As stated in Ann R. Bauman's Training of Trainers Manual, “Those who pay for training want to assess their investment. Those who receive training want to know what progress they have made, and those who deliver training want to ascertain whether the objectives were valid, whether they were accomplished, and whether the training provided skills that will help participants on the job."

Since the camp is so reliant upon the social service agencies which send children each season, it is an excellent idea to contact those groups for feedback. You may discover a recurring complaint that can be easily remedied for next year, if not this season. Agencies may also be able to offer advice on streamlining the administrative process, which in the long run can net additional time for what matters the most at camp: time spent with your staff and campers.

Camper Surveys

A final assessment of your program can come from the campers themselves. Many schools have devised surveys which allow teachers to read the criterion aloud, and prereaders circle a smiley face or a frowny face to indicate their opinions. Therefore, don’t be reluctant to survey your campers for fear that they won’t “understand” what is being asked.

Perhaps the best way to collect feedback is to simply talk with campers on a daily basis. Ask them about their favorite activity, their favorite counselor, their new friends. Ask them what they’ll remember most when they get home. Finally, ask them, “If you were in charge here, what would you do differently to make this a better camp?” You might be surprised by the simple, yet awesome, ideas that they suggest.

The Salvation Army Eastern Territory CampStaff Orientation and Training Manual

page 9:1