AAHLE 2016: Best Practices for Creating and Implementing an EOC Evaluation Survey
BEST PRACTICES FOR CREATING AND IMPLEMENTING AN
END-OF-COURSE EVALUATION SURVEY
Jennifer Ann Morrow, Ph.D.
Sarah A. Nadel, M.A.
Elizabeth Taylor, Ph.D.
University of Tennessee
© 2016 by Jennifer Ann Morrow, Ph.D.
All rights reserved. No part of this document may be
reproduced or transmitted in any form or by any means,
electronic, mechanical, photocopying, recording, or
otherwise, without prior written permission of Jennifer Ann Morrow.
Morrow, Nadel, & Taylor (2016)Page 1 of 14
AAHLE 2016: Best Practices for Creating and Implementing an EOC Evaluation Survey
Outline of Roundtable Presentation
What is the Purpose of End-of-Course (EOC) Evaluations?
Question: What is the purpose of EOCs?
Question: What are the main reasons institutions should utilize and make changes based on their EOCs results?
Question: What problems exist with your institution’s EOC?
Question: What could be done to improve or alleviate these problems?
Best Practices for Creating a New EOC Evaluation Survey
Step I – Identify Specific Purpose at Your Institution
Question: How do you identify your purpose?
Step II - Create an EOC Evaluation Survey Taskforce
Question: What are the possible responsibilities of the Taskforce?
Question: Who should be invited to be a member of the EOC Taskforce?
Step III – Meet with Key Stakeholders
Question: Who are the possible Key Stakeholders within your institution?
Question: What are the possible interview questions/conversation topics you can use?
Step IV – Review EOC Evaluation Survey Literature and Institutions’ Surveys
Question: Why is it important to conduct literature reviews of other EOCs?
Question: Why is it important to look at what EOC surveys are utilized at other institutions?
Step V – Collect and Analyze Faculty and Student Feedback
Question: Why is it important to gain feedback regarding your current EOC survey?
Step VI: Create EOC Evaluation Survey Questions and Solicit Feedback
Question: What are some ways to begin to develop your questions?
Question: Why is it important to solicit feedback on your newly developed EOC survey questions?
Step VII: Pilot Study of New EOC Survey and Analysis of Items/Feedback
Question: Why is it important to ensure the EOC questions are valid/reliable and understandable?
Step VIII: Expert Review of New EOC Survey
Question: What is it important to seek Expert Review of the new EOC survey?
Step IX: Assigning of Responsible Parties to Oversee EOC and Future Revisions
Question: Who should oversee the EOCs at your institution?
Actions within Each Step
Question: What are ways to ensure you are constantly communicating with stakeholders?
Question: What are some possible barriers that you may encounter during the EOC creation process?
Office of Institutional Research and Assessment Form Link
University of Tennessee - Knoxville End of Course Evaluation Instruments
Resources
What is the Purpose of End-of-Course (EOC) Evaluations?
Question: What is the purpose of EOCs?
Question: What are the main reasons institutions should utilize and make changes based on their EOCs results?
Question: What problems exist with your institution’s EOC?
Question: What could be done to improve or alleviate these problems?
Best Practices for Creating a New EOC Evaluation Survey
Morrow, Nadel, & Taylor (2016)Page 1 of 14
AAHLE 2016: Best Practices for Creating and Implementing an EOC Evaluation Survey
Step I – Identify Specific Purpose at Your Institution
Before you begin to develop a new end-of-course evaluation survey, you should identify what the purpose of your EOC evaluation survey will be.
Question: How do you identify your purpose?
Step II - Create an EOC Evaluation SurveyTaskforce
Developing a taskforce of several people from various areas within your institution can begin the new EOC evaluation survey development process.
Question: What are the possible responsibilities of the Taskforce?
Question: Who should be invited to be a member of the EOC Taskforce?
Step III – Meet with Key Stakeholders
It is important to assess what administrators, faculty, and students know and think about EOCs and their use. Meet with your institutions key stakeholders to get their “buy-in” and support.
Question: Who are the possible Key Stakeholders within your institution?
Question: What are the possible interview questions/conversation topics you can use?
Step IV – Review EOC Evaluation Survey Literature and Institutions’ Surveys
It is important to do research on other EOCs being utilized within higher institutions.
Question: Why is it important to conduct literature reviews of other EOCs?
Question: Why is it important to look at what EOC surveys are utilized at other institutions?
Step V – Collect and Analyze Faculty and Student Feedback
It is important to reach out to faculty and students for their feedback regarding your current EOCs.
Question: Why is it important to gain feedback regarding your current EOC survey?
Step VI: Create EOC Evaluation Survey Questions and Solicit Feedback
It is imperative that as you are developing your new questions, you gain feedback from a variety of stakeholders and experts regarding your newly developed questions.
Question: What are some ways to begin to develop your questions?
Question: Why is it important to solicit feedback on your newly developed EOC survey questions?
Step VII: Pilot Study of New EOC Survey and Analysis of Items/Feedback
It is essential that you conduct a pilot study and run analyses to ensure the new questions are valid/reliable and understandable.
Question: Why is it important to ensure the EOCquestions are valid/reliable and understandable?
Step VIII: Expert Review of New EOC Survey
It is important to seek the review of your EOC survey questions from a content and assessment expert.
Question: What is it important to seek Expert Review of the new EOC survey?
Step IX: Assigning of Responsible Parties to Oversee EOC and Future Revisions
The specific roles and responsibilities should be assigned by someone within your institution who has the ability to set policy and procedures for handling and dealing with the EOC survey data and revisions.
Question: Who should oversee the EOCs at your institution?
Actions within Each Step
Throughout your EOC survey development you need to ensure you constantly communicate with your stakeholders. Additionally, you need to be aware of any barriers in the development process and find ways to address them.
Action #1: Constant Communication with Stakeholders
Question: What are ways to ensure you are constantly communicating with stakeholders?
Action #2: Identify and Address Barriers
Question: What are some possible barriers that you may encounter during the EOC creation process?
Office of Institutional Research and Assessment Form Link
The following link includes forms that are utilized in the current End-of-Course (EOC) Evaluations within our institution (called SAIS). Each form can be utilized within a different class type (i.e., large lecture, lab sections, distance learning courses). These forms will be phased out in Fall 2016.
University of Tennessee - Knoxville End of Course Evaluation Instruments
These construct tables contain the new EOC items that were developed and are currently being tested at our institution. These forms will comprise our EOC evaluation in Fall 2016.
Rating Scale (all forms):
1=Strongly Disagree
2=Disagree
3=Neutral
4=Agree
5=Strongly Agree
0=Not Applicable
Construct Table Form V: Core Form
9 QUESTIONS (7 CLOSED-ENDED; 2 OPEN-ENDED)
Construct / Marsh and Roche’s 9 Factors (1993, 1997) / QuestionsEnthusiasm/Rapport
The perceived ability of the instructor to reach out and connect with students.
Availability
The perceived level of availability of the instructor beyond regularly scheduled meeting times. / Enthusiasm/Rapport / The instructor created an atmosphere that invited you to seek additional help.
The instructor created a respectful and positive learning environment.
Course Dimensions
The perceived organization and effectiveness of the course as delivered. / Breadth of Coverage
Course Delivery / The instructor contributed to your understanding of course content.
Organization/Clarity / The class sessions were well organized.
Grading (Feedback) / The instructor provided useful feedback on course assignments.
Course Resources / The course materials (readings, homework, laboratories, etc.) enhanced your learning in this course.
Group Interaction
The perceived opportunity for student-to-student and student-to-instructor interaction. / Group Interaction
Active Learning
The perceived opportunity for student to practice or actively engage course content.
Value
The perceived level of value of the course experience. / Learning/Value / The course challenged you to learn something new.
Workload/Difficulty
The perceived level of academic challenge the course presented to the student. / Workload/Difficulty
Open-Ended Questions / Is there any additional feedback you would like to provide about the instructor (e.g., teaching style, time management, accessibility, etc.)?
Is there any additional feedback you would like to provide about the course (e.g., workload, content, technology, etc.)?
Construct Table Form VA: Courses with an Online Component
7 QUESTIONS (6 CLOSED-ENDED; 1 OPEN-ENDED)
Construct / Marsh and Roche’s 9 Factors (1993, 1997) / QuestionsEnthusiasm/Rapport
The perceived ability of the instructor to reach out and connect with students.
Availability
The perceived level of availability of the instructor beyond regularly scheduled meeting times. / Enthusiasm/Rapport / The instructor responded to your email inquiries within a reasonable timeframe (i.e., 48-72 hours).
There were opportunities for interaction between you and the instructor.
Course Dimensions
The perceived organization and effectiveness of the course as delivered. / Breadth of Coverage
Course Delivery / The instructor was effective at teaching an online course.
Organization/Clarity / The course platform was well organized (e.g., Blackboard, Moodle).
Grading (Feedback)
Course Resources / The technology tools were appropriate for the type of online course.
Group Interaction
The perceived opportunity for student-to-student and student-to-instructor interaction. / Group Interaction
Active Learning
The perceived opportunity for student to practice or actively engage course content.
Value
The perceived level of value of the course experience. / Learning/Value / The online resources/components of this course contributed to your understanding of the course content.
Workload/Difficulty
The perceived level of academic challenge the course presented to the student. / Workload/Difficulty
Open-Ended Questions / Is there any additional feedback you would like to provide regarding the online format of the course (e.g., workload, content, technology, etc.)?
Construct Table Form VB: Courses with a Lab Component
12 QUESTIONS (10 CLOSED-ENDED; 2 OPEN-ENDED)
Construct / Marsh and Roche’s 9 Factors (1993, 1997) / QuestionsEnthusiasm/Rapport
The perceived ability of the instructor to reach out and connect with students.
Availability
The perceived level of availability of the instructor beyond regularly scheduled meeting times. / Enthusiasm/Rapport / The lab instructor responded to your email inquiries within a reasonable timeframe (i.e., 48-72 hours).
The lab instructor created a respectful and positive learning environment.
Course Dimensions
The perceived organization and effectiveness of the course as delivered. / Breadth of Coverage / The lab instructor used good examples and illustrations.
Course Delivery / The lab instructor demonstrated the proper way to use lab supplies/materials.
The lab instructor contributed to your understanding of the course content.
Organization/Clarity
Grading (Feedback) / The lab instructor provided useful feedback on lab assignments.
Course Resources / The lab materials enhanced your learning in this course.
Group Interaction
The perceived opportunity for student-to-student and student-to-instructor interaction. / Group Interaction
Active Learning
The perceived opportunity for student to practice or actively engage course content.
Value
The perceived level of value of the course experience. / Learning/Value / The lab instructor created opportunities for hands-on learning.
The lab provided opportunities to apply knowledge to real world situations or problems.
The lab instructor was knowledgeable about the course material.
Workload/Difficulty
The perceived level of academic challenge the course presented to the student. / Workload/Difficulty
Open-Ended Questions / Is there any additional feedback you would like to provide about the lab instructor (e.g., teaching style, time management, accessibility, etc.)?
Is there any additional feedback you would like to provide about the lab (e.g., workload, content, technology, etc.)?
Construct Table Form VC: Courses with a Teaching Assistant/Peer Mentor
7 QUESTIONS (6 CLOSED-ENDED; 1 OPEN-ENDED)
Construct / Marsh and Roche’s 9 Factors (1993, 1997) / QuestionsEnthusiasm/Rapport
The perceived ability of the instructor to reach out and connect with students.
Availability
The perceived level of availability of the instructor beyond regularly scheduled meeting times. / Enthusiasm/Rapport / The teaching assistant/peer mentor responded to your email inquiries within a reasonable timeframe (i.e., 48-72 hours).
The teaching assistant/peer mentor created a respectful and positive learning environment.
Course Dimensions
The perceived organization and effectiveness of the course as delivered. / Breadth of Coverage / The teaching assistant/peer mentor used good examples and illustrations.
Course Delivery / The teaching assistant/peer mentor contributed to your understanding of the course content.
Organization/Clarity
Grading (Feedback) / The teaching assistant/peer mentor provided useful feedback on assignments.
Course Resources / The teaching assistant/peer mentor enhanced your learning in this course.
Group Interaction
The perceived opportunity for student-to-student and student-to-instructor interaction. / Group Interaction
Active Learning
The perceived opportunity for student to practice or actively engage course content.
Value
The perceived level of value of the course experience. / Learning/Value / The teaching assistant/peer mentor was knowledgeable about the course material.
Workload/Difficulty
The perceived level of academic challenge the course presented to the student. / Workload/Difficulty
Open-Ended Questions / Is there any additional feedback you would like to provide about the teaching assistant/peer mentor (e.g., teaching style, time management, accessibility, etc.)?
Construct Table Form VD: Courses with Discussion/Recitation Component
10 QUESTIONS (8 CLOSED-ENDED; 2 OPEN-ENDED)
Construct / Marsh and Roche’s 9 Factors (1993, 1997) / QuestionsEnthusiasm/Rapport
The perceived ability of the instructor to reach out and connect with students.
Availability
The perceived level of availability of the instructor beyond regularly scheduled meeting times. / Enthusiasm/Rapport / The discussion/recitation instructor responded to your email inquiries within a reasonable timeframe (i.e., 48-72 hours).
The discussion/recitation instructor created a respectful and positive learning environment.
Course Dimensions
The perceived organization and effectiveness of the course as delivered. / Breadth of Coverage
Course Delivery / The discussion/recitation instructor contributed to your understanding of the course content.
The discussion/recitation instructor was skilled at developing classroom discussion.
Organization/Clarity / The class discussions were well organized.
Grading (Feedback)
Course Resources / The discussion/recitation materials enhanced your learning in this course.
Group Interaction
The perceived opportunity for student-to-student and student-to-instructor interaction. / Group Interaction / The class discussions provided an opportunity to learn from other students.
Active Learning
The perceived opportunity for student to practice or actively engage course content.
Value
The perceived level of value of the course experience. / Learning/Value / The discussion/recitation components of this course contributed to your understanding of the course content.
Workload/Difficulty
The perceived level of academic challenge the course presented to the student. / Workload/Difficulty
Open-Ended Questions / Is there any additional feedback you would like to provide about the discussion/recitation instructor (e.g., teaching style, time management, accessibility, etc.)?
Is there any additional feedback you would like to provide about the discussion/recitation (e.g., workload, content, technology, etc.)?
Resources
Amrein-Beardsley, A., & Haladyna, T. (2012). Validating a theory-based survey to evaluate teaching effectiveness in higher education. Journal on Excellence in College Teaching, 23 (1), 17-42.
Angelo, T., & Cross, K. (1993). Class assessment techniques: A handbook for college teachers. San Francisco: Jossey-Bass.
Banta, T. W., Lund, J. P., Black, K. E., Oblander, F. W. (Eds.) (1996). Assessment in practice: Putting principles to work on college campuses. San Francisco: JosseyBass.DOI: 10.1016/S0099-1333(96)90096-8
Bloom, B. S. (1968). Learning for mastery. UCLA: Center for the Study of Evaluation of Instructional Programs, (1)2. Los Angeles, CA.
Bresciani, M. J., Zelna, C. L., & Anderson, J. A. (2004). Assessing student learning and 2 development: A handbook for practitioners. Washington, D.C.: National Association of Student Personnel Administrators.
Cho, J., Otani, K., & Kim, B. J. (2014). Differences in student evaluations of limited-term lecturers and full-time faculty. Journal on Excellence in College Teaching, 25(2), 5-24.
Glasnapp, D.R., Poggio, J.P., & Ory, J.C. (1978). End-of-course and long-term retention outcomes for
mastery and nonmastery learning paradigms. Psychology in Schools,15(4), 595-603, 595-603. DOI: 10.1002/1520-6807(197810)15:4
Keeling, R. (Ed.). (2004). Learning reconsidered. A campus-wide focus on the student experience. Washington, D.C.: National Association of Student Personnel Administrators & American College Personnel Association.
Marlin, J.W., & Niss, J.F. (1980). End-of-course evaluations as indicators of student learning and
instructor effectiveness. The Journal of Economic Education, 11(2), 16-27. DOI: 10.1177/0273475308324086
Suskie, L. A. (2009). Assessing student learning: A common sense guide (2nd Ed). San Francisco: Jossey-Bass
Twan Huybers (2014) Student evaluation of teaching: the use of best–worst scaling, Assessment & Evaluation in Higher Education, 39(4), 496-513, DOI: 10.1080/02602938.2013.851782