Critical Thinking and Program Evaluation PHPB-614

Course Data

Number:PHPB-614

Title:Critical Thinking and Program Evaluation

Department:Health Promotion and Behavioral Sciences

School/College:School of Public Health and Information Sciences

Status:Approved

Version:2006.08.23

Data updated:03/03/2009

Course Description

The course establishes basic skills for public health evaluation. Students learn critical thinking regarding evaluation purpose, procedures and findings, terminology, specific techniques, and the application and dissemination of evaluation results.

Course Objectives

1.articulate the context and purpose of evaluation

2.define basic evaluation terminology

3.identify steps to conceptualize the nature of an evaluation problem

4.create a culturally competent evaluation logic model

5. outline levels of evaluation, including objects of interest and suitable measurement techniques at each level

6.describe strategies for collaboration with evaluation stakeholders

7. demonstrate principles for communicating, disseminating, and sustaining evaluation findings

8.apply ethical principles in evaluation

Prerequisites

Admission to graduate study.

Course Instructors

Name / Office / Phone / Email
Muriel Harris, Ph.D.
Course Director / 4060 K Wing / 852-4061 /

The course instructors welcome conversations with students outside of class. Students may correspond with instructors by email or set up appointments by contacting . Her phone number is 852-8040. Students should also contact Ms. Sacksteder with questions they might have regarding the mechanics or operation of the course.

Course Topics and Schedule

IMPORTANT NOTE: The schedule and topics may change as the course unfolds. Changes will be posted on Blackboard.

Class / Content / Assignment
1 / Introduction to course
2 / Team assignment and -project selection
3 / I: Context & Purpose of Evaluation
4 / Evaluation in the context of strategic planning
5 / Terminology of evaluation
6 / Engaging stakeholders
7 / Team / Coach meetings
8 / Ethics in evaluation
9 / Cultural competence in evaluation
10 / II: Resources / Team / Coach meetings
11 / Budget needs
12 / Data sources / information technology
13 / Personnel resources
14 / Team / Coach meetings
15 / III: Evaluation methods
16 / Levels of evaluation
17 / Evaluation objectives and measures
18 / Midterm examination
Spring Break: no classes
19 / Quantitative and qualitative assessment
20 / Designing evaluation logic models
21 / Team / Coach meetings
22 / Measurement techniques in evaluation
23 / Quality assurance in evaluation
24 / IV: Findings and results
25 / Program impacts of evaluation results
26 / Writing and disseminating evaluation reports
27 / Team presentations
28 / Team presentations
Final exam

Course Materials

Blackboard

The primary mechanism for communication in this course, other than class meetings, is UofL’s Blackboard system at or Instructors use Blackboard to make assignments, provide materials, communicate changes or additions to the course materials or course schedule, and to communicate with students other aspects of the course. It is imperative that students familiarize themselves with Blackboard, check Blackboard frequently for possible announcements, and make sure that their e-mail account in Blackboard is correct, active, and checked frequently.

Required Texts

Grembowski, David E. The Practice of Health Program Evaluation. Thousand Oaks, CA: Sage Publications, 2001.

Centers for Disease Control and Prevention. Framework for Program Evaluation in Public Health. MMWR 1999;48(No. RR-11).

Other Required Reading

Selected reading from Community Toolbox, a public health resource maintained by the University of Kansas.

Prepared Materials Used by Instructors

Materials used by instructors in class are available to students via Blackboard no later than 24 hours following the class. These may include outlines, citations, slide presentations, and other materials. There is no assurance that the materials include everything discussed in the class.

Course Policies

Attendance and Class Participation

It is expected that students will attend and actively participate in class meetings. They are encouraged to contact instructors in advance when they will be absent. IMPORTANT NOTE:

The course has a built in team project, designed to serve as a laboratory for course concepts. While there will be some time during regularly scheduled class periods for team project work, students will also have to be available from time to time for meeting with their teams and Louisville Metro Health Department personnel to conduct project-related work. Some of this time will be outside of scheduled class periods.

Student Evaluation

The grading system is designed to encourage and reward mastery of the course learning objectives. The following will be used to determine the student’s grade:

  • Logic models (10% of grade).

Students must submit one-page logic models for three different evaluation problems. They are free to imagine evaluation scenarios to use as the subjects for the logic models, with no imposed limits on resources or evaluation activities. Instructions and guidelines will be provided when the assignment is presented in class.

Papers will be graded by course instructors on a continuous numerical scale.

  • Midterm Exam (20%).

On March 1, students will complete a written exam based on course concepts presented up to that date. Study materials will be provided to help students prepare for the exam.

  • Individual team project reports (20%)

Students will submit written reports about their team projects. Reports should include an overview of the project, with emphasis on the student’s specific role or activities. Detailed guidelines will be provided in class.

  • Team Project presentation (20%)

On April 17 and 19, each team will present their projects to the assembled class. The presentation may provide an overview of various aspects of the project, but a major focus should be on evaluation issues and results. Presentations will be graded on the basis of content, presentation style and audiovisual qualities.

  • Faculty and program coordinator evaluation of presentations (5%)

Faculty project coaches and agency preceptors attending the presentations on April 17/19 will assess the quality of those presentations, using a standardized evaluation form.

  • Student self-evaluation of their team project (5%).

Students will complete an evaluation of their team project, with opportunities to reflect on the dynamics of the team and their contribution to the team’s work and progress.

  • Final Exam (20%).

On April 30, students will complete a written exam based on all the concepts presented in the course. The exam will consist of a variety of question formats: some short answer and some essays.

Grading

The course will be graded with the letter grades A, B, C, F, with +/- designations. The is no traditional D grade falling between C and F because the course requires students to perform at or above a C level. Performance below that level will require students to repeat the course.

Grading Scale

A+ 98-100%B-80-82%

A93-97%C+78-79%

A-90-92%C73-77%

B+88-89%C-70-72%

B83-87%F<70%

Other Policies

Syllabus Revision

The course director reserves the right to modify any portion of this syllabus. A best effort will be made to provide an opportunity for students to comment on a proposed change before the change takes place.

Inclement Weather

This course will adhere to the University’s policy and decisions regarding cancellation or delayed class schedules. Adjustments will be made to the class schedule as necessary to take into account any delays or cancellations of this class. Local television and radio stations broadcast University delays or closings. The UofL web site ( and telephone information line (502-852-5555) also broadcast delays or closings.

Grievances

Students who have grievances regarding the course should contact the course director. If students are not satisfied with the response, they should seek guidance from the Dean of the GraduateSchool or the Dean of the School of Public Health and Information Sciences.

Disabilities

In accordance with the Americans with Disabilities Act, students with bona fide disabilities will be afforded reasonable accommodation. The DisabilityResourceCenter will certify a disability and advise faculty members of reasonable accommodations. More information is located at

Academic Honesty

Students are required to comply with the academic honesty policies of the University, GraduateSchool, and School of Public Health and Information Sciences. These policies prohibit plagiarism, cheating, and other violations of academic behavior. More information is located within the SPHIS Student Catalog.

Additional Policy Information

Consult the UofL Graduate Student Handbook for more about UofL policies. (

Community Engagement Projects

Student / Team Guidelines

Students in Critical Thinking and Program Evaluation will participate with selected team projects at the Louisville Metro Health Department. The teams will begin their engagement with the semester projects around the second week of the course, and will continue working until the end of the semester. Teams will be responsible for learning all they can about the selected project, and trying to find practical ways they can support it. About the selected project they should learn:

1) the social, political, economic, and organizational background that lead to the creation of the program;

2)assessment data that provided a rationale;

3) any theoretical basis for the planning and implementation;

4) goals and objectives, and how the project is related to the mission of the parent

agency;

5) implementation activities and strategies;

6) personnel;

7) budget expenses and resources;

8) the evaluation plan.

These data will be obtained from documents provided by program personnel, meetings with program personnel, field observations of program components, and actual participation and support of program elements. In some cases, aspect of the above eight items will be undeveloped or missing entirely. Students may have an opportunity to develop those missing elements.

Each team will have a member of the faculty of the School of Public Health and Information Sciences assigned to them as a coach. Coaches will be meeting with teams on a weekly basis to provide guidance, mentoring, practical suggestions, and accountability to support continued progress and completion of the projects. Each month a formal appointment during class time is scheduled for teams and coaches to meet. Coaches will not routinely interact with project coordinators or other agency staff, but will be a point of contact for the agency as needed. The amount of direct interaction of the coaches with the community project staff will vary with the discretion of each coach.

To satisfy course requirements, students will be completing individual written reports and participating in group presentations based on their project work.

It is expected that each student will contribute in a substantial way to their selected projects. Because the projects will be diverse, it is not feasible to outline rigid rules for what student contributions will entail. However, in general, students and their teams will be expected to do the following:

1.At the beginning of the project work, teams will meet with the agency staff member who supervises, coordinates, or manages (hereafter called program coordinator) the selected project. The purpose of this meeting will be for an introduction of immediate agency personnel and the team members. Further, this is an opportunity for students to get an overview of the program, including: 1) how and why the program was established; 2) how the program is organized and what services it provides; 3) what is the program’s administrative structure; 4) who are the agency and community stakeholders; 5) what is the source of funding; 5) how the program is or will be evaluated; 6) other details thought to be important by the program coordinator.

2.Students will then read all available background material describing the project. Examples of background material include a grant proposal, an annual report, education and training handbooks, brochures, and handouts, data summaries, agency directives, and so forth.

3.The next step will be for the students and teams to identify ways they can participate in the project. The program coordinators are encouraged to provide specific tasks for students and teams to complete. These activities could include providing staff support for a community event, developing instructional material, organizing and scheduling an activity, offering library and other research support, completing data management and evaluation tasks, and generally assisting staff of the program. It is expected that teams engage in project evaluation to the extent possible. It is explicitly not the intention for students and teams to only shadow or observe. Instead, it is desired and intended that students make a practical contribution to the success of the project. In many cases it will be possible to define at the outset a specific deliverable or product for which the team will be responsible.

4.When the teams begin working with the projects they will have completed very little of the MPH degree program. However, it is hoped that with this limited public health background and other educational and work experiences they will be able to come to understand the projects and make suggestions for making them better. Some of their suggestions will have merit while other suggestions might be wrong or unworkable. The important point is that as their experience with the project grows over the course of the semester they will have opportunities to have open, respectful dialogue with program staff regarding their observations and suggestions. The agency staff members are asked to provide opportunities for student/team dialogue from time to time.

5.Teams should pay particular attention to how the project is going to be evaluated. They should make every effort to participate in evaluation procedures appropriate to the stage of the project during the semester. If there are discrepancies between existing evaluation plans and the principles presented in this class, students should make an effort to expand the evaluation plan appropriately, within the organizational and budgetary limits of the LMHD.

6.At the completion of the student/team projects, the teams will be preparing and delivering a presentation based on their project activities, with special focus on project evaluation. Program coordinators and associated staff from LMHD will be invited to attend those presentations.

7.Individual project reports and electronic copies of team presentations are due no later than April 27. The presentation files should include all slides used in the team’s presentation; the first slide should specify the name of each team member.

8.The individual report will outline program elements, but should focus on the activities and roles carried out by the individual student. The project report will be graded on the basis of the organization of the document, the logic and cohesion of the narrative, the quality of spelling, grammar, and sentence structure, the thoroughness of project description, and the proficiency of the comparison of the project’s evaluation plan with evaluation principles taught in class. It is expected that any sources used in developing the report will be cited with complete references, using one of the standard reference styles. Students are encouraged to use narrative, enhanced with charts, graphs, tables, photos, and so forth as appropriate. The report should include a table of contents and an executive summary, to appear at the beginning of the report. The executive summary should be no more than one page, with key points from the report. There is no minimumpage length. However, students will need 3-7 pages to do a thorough job, while more than 10 would probably be unnecessarily lengthy. Students are encouraged to discuss their papers with their team's coach for suggestions regarding how to develop and complete the report.

Professional Respect and Ethics

The MPH Program and the course instructors recognize that students will be guests in the LMHD facilities. For this and other reasons, three ethical obligations are emphasized. First, any and all patient or client specific data is absolutely confidential, and may not be shared, openly discussed, or disseminated in any way. Second, details of day to day agency functions, exchanges between personnel, administrative issues, and so forth are considered sensitive; students will be expected to use discretion and refrain from disclosing information that might put the agency in an unfavorable light in the community. Finally, while the team projects will be an integral part of the LMHD functions, not just a “make-work” activity, it is understood that the LMHD has to respond to organizational, budgetary, and political forces that will at times supersede its commitment to this project. This means that sometimes projects will be impacted in undesirable and unforeseen ways. Teams should view this with respect, and recognize it as an opportunity to learn the reality of everyday of public health.

Project Timeline

Class 2Student teams formed; identify project preference

Teams have first meeting with coaches; weekly meetings continue through January

Teams begin engagement with LMHD projects

*teams schedule initial meeting with project personnel

*teams read and synthesize existing program documents

*teams and project staff identify engagement opportunities

and potential deliverables

*teams continue with ongoing assignments, appointments

Class 7Formal meeting of teams with coaches; turn in half-page progress report,

signed by team coach.

Class 15Formal meeting of teams with coaches; turn in half-page progress report,

signed by team coach.

Class 21Formal meeting of teams with coaches; turn in half-page progress report,

signed by team coach.

Classes 27 and 28Teams present projects; LMHD personnel invited

Finals weekStudents submit reports and team presentations to instructors

(date TBA)Teams complete deliverables; submit to LMHD

Finals weekLMHD personnel provide team performance assessment to SPHIS

(date TBA)

Page 1 of 9