Critical Thinking and Program Evaluation PHPH-614

Course Data

Number: PHPH-614

Title: Critical Thinking and Program Evaluation

Credit-hours: 3

Department: Health Promotion and Behavioral Sciences

School/College: School of Public Health and Information Sciences

Type: Lecture

Course Description

The course establishes basic skills for public health evaluation. Students learn critical thinking regarding evaluation purpose, procedures and findings, terminology, specific techniques, and the application and dissemination of evaluation results.

Course Objectives

·  articulate the context and purpose of evaluation

·  define basic evaluation terminology

·  identify steps to conceptualize the nature of an evaluation problem

·  create a culturally competent evaluation logic model

·  outline levels of evaluation, including objects of interest and suitable measurement techniques at each level

·  describe strategies for collaboration with evaluation stakeholders

·  demonstrate principles for communicating, disseminating, and sustaining evaluation findings

·  apply ethical principles in evaluation

Prerequisites

Admission to graduate study.

Course Instructors

Name / Office / Phone / Email
Muriel Harris, Ph.D.
Course Director / 4060 K Wing / 852-4061 /

The course instructors welcome conversations with students outside of class. Students may correspond with instructors by email or set up appointments by contacting . Her phone number is 852-8040. Students should also contact Ms. Sacksteder with questions they might have regarding the mechanics or operation of the course.

Course Topics and Schedule

IMPORTANT NOTE: The schedule and topics may change as the course unfolds. Changes are posted on Blackboard.

Class / Content / Assignment
1 / Introduction to course
2 / Team assignment and -project selection
3 / I: Context & Purpose of Evaluation
4 / Evaluation in the context of strategic planning
5 / Terminology of evaluation
6 / Engaging stakeholders
7 / Team / Coach meetings
8 / Ethics in evaluation
9 / Cultural competence in evaluation
10 / II: Resources / Team / Coach meetings
11 / Budget needs
12 / Data sources / information technology
13 / Personnel resources
14 / Team / Coach meetings
15 / III: Evaluation methods
16 / Levels of evaluation
17 / Evaluation objectives and measures
18 / Midterm examination
19 / Quantitative and qualitative assessment
20 / Designing evaluation logic models
21 / Team / Coach meetings
22 / Measurement techniques in evaluation
23 / Quality assurance in evaluation
24 / IV: Findings and results
25 / Program impacts of evaluation results
26 / Writing and disseminating evaluation reports
27 / Team presentations
28 / Team presentations
Final exam

Course Materials

Blackboard

The primary mechanism for communication in this course, other than class meetings, is UofL’s Blackboard system at http://ulink.louisville.edu/ or http://blackboard.louisville.edu/. Instructors use Blackboard to make assignments, provide materials, communicate changes or additions to the course materials or course schedule, and to communicate with students other aspects of the course. It is imperative that students familiarize themselves with Blackboard, check Blackboard frequently for possible announcements, and make sure that their e-mail account in Blackboard is correct, active, and checked frequently.

Required Texts

Grembowski, David E. The Practice of Health Program Evaluation. Thousand Oaks, CA: Sage Publications, 2001.

Centers for Disease Control and Prevention. Framework for Program Evaluation in Public Health. MMWR 1999;48(No. RR-11).

http://www.cdc.gov/mmwr/preview/mmwrhtml/rr4811a1.htm

Other Required Reading

Selected reading from Community Toolbox, a public health resource maintained by the University of Kansas. http://ctb.ku.edu/index.jsp

Prepared Materials Used by Instructors

Materials used by instructors in class are available to students via Blackboard no later than 24 hours following the class. These may include outlines, citations, slide presentations, and other materials. There is no assurance that the materials include everything discussed in the class.

Course Policies

Attendance and Class Participation

It is expected that students attend and actively participate in class meetings. They are encouraged to contact instructors in advance when they are to be absent. IMPORTANT NOTE:

The course has a built-in team project, designed to serve as a laboratory for course concepts. While there is some time during regularly scheduled class periods for team project work, students also have to be available from time to time for meeting with their teams and Louisville Metro Department of Public Health and Wellness personnel to conduct project-related work. Some of this time is outside of scheduled class periods.

Student Evaluation

The grading system is designed to encourage and reward mastery of the course learning objectives. The following are be used to determine the student’s grade:

·  Logic models (10% of grade).

Students must submit one-page logic models for three different evaluation problems. They are free to imagine evaluation scenarios to use as the subjects for the logic models, with no imposed limits on resources or evaluation activities. Instructions and guidelines are provided when the assignment is presented in class.

Papers are graded by course instructors on a continuous numerical scale.

·  Midterm Exam (20%).

In Class 18, students complete a written exam based on course concepts presented up to that date. Study materials are provided to help students prepare for the exam.

·  Individual team project reports (20%)

Students submit written reports about their team projects. Reports should include an overview of the project, with emphasis on the student’s specific role or activities. Detailed guidelines are provided in class.

·  Team Project presentation (20%)

In Classes 27 and 28, each team presents its projects to the assembled class. The presentation may provide an overview of various aspects of the project, but a major focus should be on evaluation issues and results. Presentations are graded on the basis of content, presentation style and audiovisual qualities.

·  Faculty and program coordinator evaluation of presentations (5%)

Faculty project coaches and agency preceptors attending the presentations in Classes 27 and 28 assess the quality of those presentations, using a standardized evaluation form.

·  Student self-evaluation of their team project (5%).

Students complete an evaluation of their team project, with opportunities to reflect on the dynamics of the team and their contribution to the team’s work and progress.

·  Final Exam (20%).

In finals week, students complete a written exam based on all the concepts presented in the course. The exam consists of a variety of question formats: some short answer and some essays.

Grading

The course is graded with the letter grades A, B, C, F, with +/- designations. There is no traditional D grade falling between C and F because the course requires students to perform at or above a C level. Performance below that level requires students to repeat the course.

Grading Scale

A+ 98-100% B- 80-82%

A 93-97% C+ 78-79%

A- 90-92% C 73-77%

B+ 88-89% C- 70-72%

B 83-87% F <70%

Student and Team Guidelines for Community Engagement Projects

Students in the course participate with selected team projects at the Louisville Metro Department of Public Health and Wellness (LMDPHW). The teams begin their engagement with the semester projects around the second week of the course and continue working until the end of the semester. Teams are responsible for learning all they can about the selected project and trying to find practical ways they can support it. About the selected project they should learn:

·  the social, political, economic, and organizational background that lead to the creation of the program;

·  assessment data that provided a rationale;

·  any theoretical basis for the planning and implementation;

·  goals and objectives, and how the project is related to the mission of the parent

·  agency;

·  implementation activities and strategies;

·  personnel;

·  budget expenses and resources;

·  the evaluation plan.

These data are obtained from documents provided by program personnel, meetings with program personnel, field observations of program components, and actual participation and support of program elements. In some cases, aspect of the above eight items are undeveloped or missing entirely. Students may have an opportunity to develop those missing elements.

Each team has a member of the faculty of the school assigned to it as a coach. Coaches meet with teams on a weekly basis to provide guidance, mentoring, practical suggestions, and accountability to support continued progress and completion of the projects. Each month a formal appointment during class time is scheduled for teams and coaches to meet. Coaches do not routinely interact with project coordinators or other agency staff but are a point of contact for the agency as needed. The amount of direct interaction of the coaches with the community project staff varies at the discretion of each coach.

To satisfy course requirements, students complete individual written reports and participate in group presentations based on their project work.

It is expected that each student contributes in a substantial way to his or her selected projects. Because the projects are diverse, it is not feasible to outline rigid rules for what student contributions entail. However, in general, students and their teams are expected to do the following:

·  At the beginning of the project work, teams meet with the agency staff member who supervises, coordinates, or manages (hereafter called program coordinator) the selected project. The purpose of this meeting is an introduction of immediate agency personnel and the team members. Further, this is an opportunity for students to get an overview of the program, including: 1) how and why the program was established; 2) how the program is organized and what services it provides; 3) what is the program’s administrative structure; 4) who are the agency and community stakeholders; 5) what is the source of funding; 5) how the program is or will be evaluated; 6) other details thought to be important by the program coordinator.

·  Students then reads all available background material describing the project. Examples of background material include a grant proposal, an annual report, education and training handbooks, brochures, and handouts, data summaries, agency directives, and so forth.

·  The next step is for students and teams to identify ways they can participate in the project. The program coordinators are encouraged to provide specific tasks for students and teams to complete. These activities could include providing staff support for a community event, developing instructional material, organizing and scheduling an activity, offering library and other research support, completing data management and evaluation tasks, and generally assisting staff of the program. It is expected that teams engage in project evaluation to the extent possible. It is explicitly not the intention for students and teams to only shadow or observe. Instead, it is desired and intended that students make a practical contribution to the success of the project. In many cases it is possible to define at the outset a specific deliverable or product for which the team is responsible.

·  When the teams begin working with the projects, they have had completed very little of the MPH degree program. However, it is hoped that with this limited public health background and other educational and work experiences they are able to come to understand the projects and make suggestions for making them better. Some of their suggestions may have merit while other suggestions may be wrong or unworkable. The important point is that as their experience with the project grows over the course of the semester, they have opportunities for open, respectful dialogue with program staff regarding their observations and suggestions. The agency staff members are asked to provide opportunities for student/team dialogue from time to time.

·  Teams should pay particular attention to how the project is going to be evaluated. They should make every effort to participate in evaluation procedures appropriate to the stage of the project during the semester. If there are discrepancies between existing evaluation plans and the principles presented in this class, students should make an effort to expand the evaluation plan appropriately, within the organizational and budgetary limits of the LMDPHW.

·  At the completion of the student/team projects, the teams prepare and deliver a presentation based on their project activities, with special focus on project evaluation. Program coordinators and associated staff from LMDPHW are invited to attend those presentations.

·  Individual project reports and electronic copies of team presentations are due no later than finals week. The presentation files should include all slides used in the team’s presentation; the first slide should specify the name of each team member.

·  The individual report outlines program elements with a focus on the activities and roles carried out by the individual student. The project report is graded on the basis of the organization of the document, the logic and cohesion of the narrative, the quality of spelling, grammar, and sentence structure, the thoroughness of project description, and the proficiency of the comparison of the project’s evaluation plan with evaluation principles taught in class. It is expected that any sources used in developing the report are cited with complete references, using one of the standard reference styles. Students are encouraged to use narrative, enhanced with charts, graphs, tables, photos, and so forth as appropriate. The report should include a table of contents and an executive summary, to appear at the beginning of the report. The executive summary should be no more than one page, with key points from the report. There is no minimum page length. However, students need 3-7 pages to do a thorough job, while more than 10 would probably be unnecessarily lengthy. Students are encouraged to discuss their papers with their team's coach for suggestions regarding how to develop and complete the report.

Project Timeline
Class / Action/Event
2 / Student teams form; identify project preference
Teams have first meeting with coaches; weekly meetings continue through January
Teams begin engagement with LMDPHW projects
*teams schedule initial meeting with project personnel
*teams read and synthesize existing program documents
*teams and project staff identify engagement opportunities and potential deliverables
*teams continue with ongoing assignments, appointments
7 / Formal meeting of teams with coaches; turn in half-page progress report, signed by team coach.
15 / Formal meeting of teams with coaches; turn in half-page progress report, signed by team coach.
21 / Formal meeting of teams with coaches; turn in half-page progress report, signed by team coach.
27 and 28 / Teams present projects; LMDPHW personnel invited
Finals week / Students submit reports and team presentations to instructors
Teams complete deliverables; submit to LMDPH
LMDPHW personnel provide team performance assessment to course instructor

Professional Respect and Ethics