Evaluation

One of the priorities of the pilot project was to measure the impact of the educational sessions. An assessment of the four educational sessions was conducted for three primary purposes:

Establish outcomes of the educational sessions

Measure knowledge gained

Determine changes for subsequent communities

There are countless examples of evaluation instruments from which Extension educators could choose. Identifying the most effective evaluation method for use with a one-time education program was challenging. Because this was a pilot program, different evaluation instruments were used to determine what instrument might be the most effective for these sessions.

The evaluation instruments used quantitative and qualitative self-reported knowledge gained to identify specific topics and areas of interest in knowledge gained. One session utilized a retrospective pretest posttest. This was administered at the conclusion of the session and asked respondents to refer back to a prior point in time. A retrospective pretest-posttest evaluation can be an effective evaluation strategy for one-time workshops (Nielsen, 2011; Rockwell and Kohn, 1989). Participants were asked to indicate knowledge before and after participating in the session on a scale of 1-4 with one being, “not at all, 2 “very little”, 3 “somewhat” and 4 “very much” (Table 1).

The other sessions used a post test to measure knowledge gained. The highest self-reported knowledge gained was from an Extension representative and oil and gas industry. Qualitative questions were also included for session participants to indicate “what would you like to learn more about” and “recommendations for improving this pilot program in other communities”.

All of the evaluations are attached as a part of this curriculum. The evaluations provided valuable information for the presenters of the program as well as the stakeholders in the community. The evaluation results provided insight and information to aid in the future coordination of community planning.

One of the suggestions for implementations for the next participating community will be the development and widespread use of one effective and simple evaluation. This will allow a more consistent comparison of statements in the evaluation from session to session.

Table 1:

Evaluation Summary of Shale Energy Development Community Planning Sessions

Delivered in Guernsey County, Ohio

Planning Session / Brief Session Description / Session Design/Speakers / Evaluation Results
Shale Development 101 (Project Launch) / Introduce purpose of sustainable planning process, review shale energy development and community impacts / OSU Extension presented the shale energy overview and facilitated the participants in identifying social, environmental and economic issues /
  • 96% had better understanding of planning process
  • 96% gained knowledge of community issues
  • 87% had better understanding of shale impacts on community

Social Sustainability / Identify anticipated social impacts and possible resources, strategies, and best practices / Facilitated panel discussion with impacted landowners and shale industry representatives / Comments to open ended questions:
  • Liked “real world” speakers
  • Emphasized need to have plan in place

Environmental Sustainability / Identify anticipated environmental impacts and possible resources, strategies, and best practices / Panel presentations and discussion from the Ohio Department of Natural Resources, industry representatives and OSU environmental experts / Pre-post knowledge questionnaire:
  • Gain in knowledge about environmental impacts and regulatory oversight
  • Gain knowledge of industry operations

Economic Sustainability / Identify anticipated economic impacts, review research results from economic impact analysis, and help local leaders develop strategies to prepare for a sustainable future / Capstone speaker from Penn State and a presentation and discussion on the economic cluster analysis from OSU Extension / 5 Point Scale Used:
  • 5.0 Learned how shale impacts other states
  • 4.88 Know more about emerging shale trends
  • 4.0 Better understand shale economic impacts

Creation of Sustainable Growth Management Chapter and Final Plan / Share each workgroup’s plan to formulate shared planning issues that tie together and formulate consensus planning priorities and planning issues to build sustainability / Presentations by workgroup chairs and
OSU Extension facilitated the development of shared planning issues and priorities based on multidimensional goals / No formal evaluation

References:

Neilsen, R. B. (2011). A retrospective pretest-postest evaluation of a one-time personal finance training. Journal of Extension [On-line], 49 (1) Article 1FEA4. Available at:

Rockwell, S. K. & Kohn, H (1989). Post-then-pre-evaluation. Journal of Extension [On-line], 27 (2) Article 2FEA5. Available at:

Prepared by: Cynthia Bond OSU Extension 2014