1
SOCW 6324
Research and Evaluation Methods in Social Work II
Summer 2011
Instructor: Sung Seek Moon, Ph.D.
Office Number: Social Work Complex 208B
Office Telephone Number: 817-272-1022
Email Address:
Office Hours: Tuesday 16:50-17:50and by appointment
Time of class meetings: 18:00-21:45
Recycling and sustainability: Please help our fragile environment by recycling this paper when finished, as well as plastic bottles, cans, etc., in the many recycling stations available in the building. Thanks.
Description of Course Content
EPAS Policy Statement (Research): Qualitative and quantitative research content provides understanding of a scientific, analytic, and ethical approach to building knowledge for practice. The content prepares students to develop, use and effectively communicate empirically based knowledge, including evidence-based interventions. Research knowledge is used by students to provide high-quality services; to initiate change; to improve practice, policy, and social service delivery; and to evaluate their own practice.
Catalogue Description of Course Content:Advanced course in the application of research principles and techniques. Topics include regression and statistical control, analysis of variance, questionnaire construction, evaluation research, and computerized tabulation and analysis of data. Mini-projects require the student to apply these techniques in the context of social work practice. Required of all students.
Expanded Description of Course Content: Building on previous courses, particularly Research I, in this course an evidence informed approach to social work practice and the delivery of human service program is emphasized. In this course quantitative and qualitative research methods and commonly used statistical procedures and approaches are applied to the evaluation of social work practice interventions and the evaluation of human service programs. In this course these research skills and knowledge are presented from the perspective of promoting diversity and social and economic justice in the evaluation of social work intervention and the delivery of human service programs.
Student Learning Outcomes:
1.Demonstrate an understanding of the role of research in the evaluation of social work practice and the delivery of human service programs.
2.Display an understanding of ethical, political and managerial implications of conducting evaluation research in human service organizations.
3.Articulate a rational for how evaluation results can be used to improve human services, advance the interest of stakeholders and inform social work practice.
4.Utilize empirical studies as a knowledge base to support professional interventions, programs, and decisions in human service agencies.
5.Develop goals and objectives for practice and program evaluation inclusive of diversity and social and economic justice.
6.Design a valid social work practice and program evaluation with an understanding of issues in evaluation related to cultural diversity, gender, sensitive topics, sexual orientation, and ethical concerns.
7.Determine and use relevant statistical procedures to analyze evaluation data in order to determine effectiveness.
8. Professional present evaluation results to target identified audience.
Note: The course instructor reserves the option to modify the course syllabus throughout the course offering by adding guest speakers, audio visual media, instructional technology, or supplemental materials and/or modifying assignments or making substitutions so long as course objectives are met and the overall grading criteria are maintained. The instructor may assess the class members’ understanding of the course content at any time by administering a pop quiz or other evaluation.
Requirements: Prerequisite: SOCW 5322.
Computer Software:
We will use SINGWIN, Excel, and SPSS for statistical analysis in this class.
Although the use of all three of these will be demonstrated on many occasions in class, this is not a technology course. The focus of this class will be on the implementation, understanding, and interpretation of statistical findings used in practice and program evaluation research, not on instruction on how to work the SPSS and Excel software. Note that one of the mechanics of using SINGWIN is described in detail in the Bloom, Fischer and Orme textbook. Those students who are totally unfamiliar with SPSS and EXCEL software may also want to consider:
1.The Office of Information Technology at UTA offers an Introduction to SPSS course and an Introduction to EXCEL course free to UTA students. For more information about these courses and other technology courses go to:
2.An online SPSS and/or EXCEL tutorial: Google SPSS and EXCEL and choose the ones that fit your needs; some are more advanced than others.
SPSS Access:
The SPSS Software, though not available on every public computer on campus, is available at many different computer sites.
- The School of Social Work Computer Lab (Technology Classroom). For information about the hours this lab is available for student use go to:
- The School of Social Work E-Library (SWEL). For information about the hours this site is open see:
- Additional Computer Labs on Campus for SPSS:
Fine Arts Building rms. 404/411/411A/412/412A
Business Building rms. 338/340
Ransom Hall (All)**open 24 hours
University Hall rm. B004. For more information about these labs go to:
Required Textbooks and Other Course Materials:
Required Texts:
Bloom, M., Fischer, J., & Orme, J.G. (2009). Evaluating practice: Guidelines for the accountable
professional (6th ed.). Englewood Cliffs, NJ: Prentice-Hall.
Supplemental Texts:
York, R. O. (2009). Evaluating human services: A practical approach for the human service professional. Boston,
MA: Pearson.
American Psychological Association. (2001). Publication manual of the American Psychological Association (5th Ed.).Washington DC: Author.
Fischer, J. & Corcoran, K. (2007). Measures for clinical practice: A sourcebook (4th ed).New York: Free Press.
Rubin, A., & Babbie, E. (2008). Research methods for social work (6th Ed.). Belmont, CA: Thomson Brooks/Cole. **This is the textbook used in Research I.
Course Outline/Topics and Readings:
Topics / Readings / Assignments / Related articlesWeek One
June 7 / Introductions;
Course overview and expectations; Description of projects to be completed for the course; The importance of program evaluation
An Overview of Evaluation Research: Scope of evaluation; Types of evaluation; Selected models of evaluation / York, Ch. 3
Week Two
June 14 / Social Service Programs:
Planning, designing, and tracking the program
Needs Assessment
Client Satisfaction / Kettner et al., Chs 5, 6, & 7
York, Ch. 13
Royse et al., Ch. 3
Royse et al., Ch. 8
Week Three
June 21 / Ethics & Diversity in Evaluation Research: Ethical Guidelines; Research with Special Populations
Process evaluation and qualitative research: Overview of process evaluation, overview of qualitative approach
Content Analysis / York, Ch. 5
York, Ch. 6
York, Ch. 7 / Buttell & Carney (2006)
Week Four
June 28 / Phases of outcome evaluation
Conceptualizing and Measuring Targets and Objectives/Goals: Basic principles of conceptualization and measurement; Specifying problems and goals; Behavioral observation; Individualized rating scales; Standardized scales; Logs; Reactivity and nonreactive measures; Selecting a measure / York, Ch. 8
Bloom et al., Chs. 2, 3, 4, 5, 6, 7, 8, 9, & 10 / Mancini et al
(2004)
Clarke et. al (2003)
Week Five
July 5 / Exam 1
Week Six
July 12 / Evaluation Designs : Basic principles of single-system designs; Baselining; Experimental single system design; Multiple designs for single systems; Designs for comparing interventions; Selecting a design. / Bloom et al., Chs 11, 12, 13, 14, 15, 16, 17, & 18 / Part 1 and Part 2. Critical Review of a measurement instrument
Week Seven
July 19 / Group Research Design
Analyzing Group Data (SPSS)
**Meet in computer lab / Royse et al., Ch. 9
Royse et al., Ch. 13
Week Eight
July 26 / Analyzing Single System Data
**Meet in computer lab / Bloom et al., Ch. 19, 20, 21, 22, 23
Week Nine
August 2 / Analyzing Single System Data
**Meet in computer lab / Bloom et al., Ch. 19, 20, 21, 22, 23 / Program Evaluation Paper Due
Week Ten
August 9 / Presentation / Final Exam
** This schedule is subject to change to fit the progress of the class**
Descriptions of major assignments and examinations with due dates:
ASSIGNMENTS: Assess Student Learning Outcomes 5 & 6
I. Intermediate Program Evaluation Paper (1. Introduction, 2. Outcomes – Critical Review) Due: July 12
1. Introduction
a)Describe the agency and the program under investigation.
b)Describe the specific program under investigation.
2. Outcomes
a) Identify the agency's goals as long-term outcomes using the proper format. State the original goals, if any, and the rewritten ones.
b) Identify the agency's program objectives or outcomes (usually the initial and intermediate outcomes) to be evaluated. If the program has no initial or intermediate outcomes you will need to develop them.
c) Each initial or intermediate outcome must have outcome indicators associated with it.
3. Methodology
a) Measurements: For each outcome, describe the measures that enabled you to determine whether the outcomes were successfully accomplished. These should be consistent with the outcome variable stated in outcome indicator(s). They are to be listed separately by construct.
Outcome measurement(s) – Critical Review of a measurement instrument.
For this assignment you will participate in the process of critically reviewing an existing measurement instrument. Students will work in groups of 3 or 4 * to identify an instrument for review and conducting a structured review of the instrument’s construct, operationalization, means of administration, and suggested revisions.
For the Critical Review, the following steps are to be taken and documented:
1. Select a standardized scale for review.
2. Identify the construct to being measured. The construct can be a strength, problem, skill, or attitude. It should have relevance to social work practice.
3. Provide an operational definition of the construct.
4. Review the adequacy of the instrument’s attention to the essential features, rationales for, and indicators of the construct based on a simple review of relevant literature. This review should not exceed 2-3 paragraphs and can be limited to 2 academic sources, referenced in APA style in your paper.
5. Apply the seven guidelines of selecting a standardized scale (Bloom et al., pp. 176-184) to review the selected standardized scale.
6. Pilot the instrument. Administer the instrument to at least 3 people in order to obtain feedback on the format, wording, and ease of use of your instrument. You do NOT need to (and should not) include any analysis of the data obtained from piloting the instrument. Your purpose is to find out if the format and/or wording are clear, if the answers your subjects provide elicit the type of response you expected, the ease of administration, etc.
You are looking for responses like:
“I was a bit confused by question 3 (and why)”
“I thought questions 2 and 3 were asking the same thing”
“I was offended by question 4 because...”
6. Make suggestions for revision of your instrument based on feedback. Be sure to assess face validity, clarity of instructions, and ease of administration.
7. Submit a report documenting all of the steps above. Include the original instrument, the revised instrument, and a conclusion discussing the challenges your group encountered. What are the strengths and limitations of the final instrument? What recommendations for change might you make based on your experience of developing the instrument?
8. Brief presentation in class on July 12.
The purpose of this assignment is to experience the process, so your documentation of the steps taken is more important than the final product. *Size of small group: Students may work alone if it fits your life structure. Groups should be no larger than 4 people.
II. Final Program Evaluation Paper (Due August 2): Student Learning Outcomes 5, 6, & 7
The purpose of this paper is to perform a real or simulated evaluation of a program connected to service provision in a human service agency or other organization that provides social work related services. Requirements include the use of developing a research design that includes both qualitative and quantitative data gathering and program analysis. The final paper for the program evaluation is due on the assigned date, at the beginning of the class period. Failure to turn in any part of the assignment on the due date at the beginning of class will result in an immediate loss of 5 points. Another 5 points will be deducted if the assignment is not received by 6 pm on the due date, and a continued loss of 5 points for each day late will be calculated. The final paper will not be accepted later than a week following the due date, resulting in an automatic zero for the assignment.
– Students will work in groups of 3 or 4*. Students can work with different group members. Students will make arrangements with a human service agency or other organization and develop an evaluation plan and conduct an outcome evaluation of an agency program. Please note, this assignment calls for evaluating an existing program not collecting data on human subjects.
1. Introduction
a. Describe the agency and the program under investigation.
b. Describe the specific program under investigation.
2. Outcomes
a) Identify the agency's goals as long-term outcomes using the proper format. State the original goals, if any, and the rewritten ones.
b) Identify the agency's program objectives or outcomes (usually the initial and intermediate outcomes) to be evaluated. If the program has no initial or intermediate outcomes you will need to develop them.
c) Each initial or intermediate outcome must have outcome indicators associated with it.
3. Methodology
a. Design
Select a single system design with the reasons why your selected design will be the most appropriate in relation to your client’s situations or agency environments.
b. Measurement
Describe the measures that enabled you to determine whether the outcome was successfully accomplished. These should be consistent with the outcome variable stated in the outcome indicators.
c. Data Collection Procedures
Provide a description of the data collection procedures that were used. Did you use telephone interviews, questionnaire completion, observations, mailed questionnaires, etc.? What steps did you use to collect data?
4. Results
a. Descriptive statistics (client characteristics, central tendency, variation, trend, effect size) should be reported.
b. All possible statistical approaches should be exercised and report the results of them, then discuss which statistical approach will be the most appropriate for your data.
c. Graphs or tables can be placed in the text or at the end of the paper, but not in the
appendix.
d.Computer Printout must always be seen as preliminary or intermediate results. That is, they must be converted into a presentable format. Place your computer output or other calculations in an appendix.
5. Discussion
a. In this section summarize the meaning of the findings and draw conclusions about the program. You can make a list of important points that became evident by the research. You may make recommendations for program change if appropriate.
b. If the program being evaluated was not effective, explain the possible reasons why. Also, if you think the program really worked, but your evaluation methodology was poor, describe weaknesses in the evaluation. Decide how feedback will be provided to the agency director and other stakeholders.
6. Class Presentations
As part of this assignment you will prepare a class presentation using Microsoft PowerPoint Presentation Graphics to present your evaluation plan and results. All members of the group are expected to participate equally in the project and presentation and will be asked to submit collectively a report. These results will be based on a hypothetical data analysis, using the computer or a hand analysis if appropriate, and tables or graphs. Include a hardcopy of the PowerPoint presentation attached with your paper.
*Size of small group: Students may work alone if it fits your life structure. Groups should be no larger than 3 people.
III.Tests and Exams: Student Learning Outcomes 1, 2, 3, 4, & 7
Two in-class exams will be given on 7/5 and 8/9, the total will count for 50% of the grade for this course. No make-up exams, either before or after the scheduled date, will be given without a documented (doctor’s note) excuse. Students who do not complete the exam on the specified date and time will receive a 0 for that exam. The exams will cover material from the readings, lectures, class discussion, and homework assignments. Each exam will focus primarily on content presented since the previous exam but may also include material covered earlier. The format of the exams will be mixed, potentially including multiple choices, true/false, matching, fill-in-the-blank, and short-answer essay.
Grading Policy: Students will be evaluated throughout the semester on homework, written assignments, and examinations. Grades will be based on the student’s performance in the following areas:
Percent of Grade1. Attendance (Attendance will be taken each class session. There is a limit of two excused absences for the class. An excused absence is limited to health and family emergencies. 0.5 point is lost for each unexcused absence or for any absence not discussed and approved by the instructor ahead of time.). More than 3 absences including excused absences will be automatically failed. / 5.5%
2. Intermediate Program Evaluation Project and Critical review of a measurement (paper & brief presentation) / 21.5%
3. Mid-term Exam / 25%
4. Final Exam / 25%
5. Final Program Evaluation Project / 23%
Total / 100%
*If you miss one of the above mandatory required works, you cannot receive a passing grade.
Grading Scale: A = 90-100%
B = 80%-89%
C = 70%-79%
D = 60%-69%
Fail = 59 points or lower
To avoid receiving a failing grade due to absences, it is the student’s responsibility to withdraw from or drop the class according to University guidelines and time frames.An incomplete grade for the semester will only be considered in the event of a documented medical emergency.
Students are expected to keep a copy of all assignments submitted.
Additional University and Class Policies
Attendance Policy:
Students are expected to be on time and attend all class sessions, complete reading assignments and be prepared to participate in class discussions and small group activities.
Each class missed without an appropriate excuse (emergence situations), preferably provided before class, will result in a 2 point deduction from the class participation core. Also, more than 10 minutes late will result in a 1 point deduction. NOTE: Excessive absences (2 or more, including appropriately excused absences) will result in the reduction of a student’s final grade by one letter grade. Three or more absences (including appropriately excused absences) will result in “AutomaticFail”. Active participation is expected, integrating personal and professional experience, assigned readings, and critical thought.