Online Course Design 2008: Evaluation and Revising Online Courses
Introduction
Session Goals
• Understand the difference between assessment and evaluation
• Define formative, summative and confirmative evaluation and understand the importance of each
• Explore theories and methods of evaluation
• Create a course evaluation and revision plan
Course Evaluation: Why?
• Fix things that are broken
• Ensure learning outcomes are being achieved
• Discover causes for failures…proactively fix other problems
• Discover potential usability/accessibility issues
• What works in theory doesn’t always work in practice
• Constant maintenance and improvements to content and instructional strategies
• Dynamic nature of online learning
What Do We Evaluate?
• Alignment of course materials with learning objectives
• Effectiveness of testing instruments
• Quality of instructional materials
• Quality of external resources
• Effectiveness of instructional strategies
• Usability of tools and technology
• Effectiveness of teaching skills
Notes:
© Jennifer Freeman
Online Course Design 2008: Evaluation and Revising Online Courses
Why / What / Who / When / HowFormative Evaluation
Summative Evaluation
Confirmative Evaluation
© Jennifer Freeman
Online Course Design 2008: Evaluation and Revising Online Courses
Formative Course Evaluation
• Why?
– Uncover problems early on; fix broken stuff (hopefully before students find it)
– Discover potential usability/accessibility issues
– Improve functionality
– Dynamic nature of online learning
• What? When?
– An ongoing process, usually done both during development and while being taught
– Asks the question, “How are we doing?”
• Who? (Who will use this evaluation information?)
– Course development team
– Instructor
• How? (What should be evaluated?)
– Instructional materials
– Instructional strategies
– Use of tools and technology
Questions to Ask
• Do learning activities and assessments align with the learning objectives?
• Do learning materials meet quality standards?
– Are learning materials error-free?
– Are learning materials accessible?
– Are learning materials usable?
• Are the technology tools appropriate and working properly?
Notes:
Formative Evaluation: Gathering data
• Course development rubrics
• Checklists
• Focus group feedback
• For consistency and repeatability
– To reduce bias, evaluation should not be done by members of the course development team…you need a “fresh pair of eyes”
– Develop a testing process/protocol/list of questions/evaluation forms
– Determine a standardized, set time for testing so it occurs regularly as a normal part of the process
• Help desk error logs
• Student FAQ discussion threads
• “Extra credit for errors found” idea
• Faculty notes jotted down during semester
Notes:
Summative Course Evaluation
• Why?
– Examine effectiveness
– Improve functionality
– Discover causes for failures…fix existing problems
– What works in theory doesn’t always work in practice
– Constant maintenance and improvements to content and instructional strategies…the dynamic nature of online learning
• What? When?
– Usually done after the completion of each semester
– Asks the question, “How did we do?”
• Who? (will use this evaluation information?)
– Instructor
– Course development team
– Administration
• How? (What should be evaluated?)
– effectiveness of instructional materials and strategies
– the learning environment (interactivity and communication)
– the instructor’s teaching skills
– availability and ease of use of tools and technology
– activities and assessments
– instructor satisfaction with the online teaching experience
– student satisfaction with the online learning experience
Notes:
Summative Evaluation: Questions to Ask
• Did the students succeed? (grades)
– Did the learning activities and assessments align with the learning objectives?
– Were assignments and assessment appropriate to the content? (test item analysis)
– Was time adequate to convey material and complete tasks?
• Level of instructor and student satisfaction (participation and opinion)
– Were learning materials easy to use and accessible? What content did students frequently have problems with? What areas of the course are error-prone?
– Was the communication and interactivity between students adequate?
– Was the communication and interactivity between students and the instructor(s) adequate?
– Were there any concerns about motivation?
– What tools did the instructor or students frequently have problems with? Should we continue to use chosen tools?
• Are program/department needs being met?
– accreditations, prerequisites for other courses, competencies
– Is the course scalable?
Summative Evaluation: Gathering data
• Student grades
• Student surveys
• Instructor satisfaction surveys
• Learner self-assessments
• Pretest/posttest comparisons
• Assessment or test item analysis
• Focus group feedback
• Help desk error logs
• Discussion forum and chat archives
Notes:
Confirmative Course Evaluation
• Why?
– Discover long-term level of effectiveness of the course within the program / department
– Discover long-term level of effectiveness of the course for students
– Identify large-scale changes necessary to the curriculum
– Constant maintenance and improvements to technology, content and strategies…the dynamic nature of online learning
• What? When?
– Usually done some time after the completion of at least two semesters
– Asks the question, “How are we doing now?”
• Who? (will use the evaluation information?)
– Instructor
– Administration
• How? (What is being evaluated?)
– Students’ long-term retention of learning, usefulness to their long-term goals
– Long term effectiveness of the course within the program
– LMS and other technology/tools
Questions to Ask
• Are program/department needs being met (accreditations, prerequisites for other courses, competencies)
• Trends in level of student satisfaction
• Course still valuable / meaningful / current / relevant to students’ long-term goals? (program/career)
• Is the course scaleable?
• Is the course sustainable?
• Learning environment; are technology, tools still meeting our needs?
Gathering data
Program student surveys
• Departmental / administrative opinions
• Faculty peer review of learning materials
• Employer surveys (professional development)
• Retention data
• Help desk logs
• Learning management system study / survey
Revision Plans
Sample Formative Evaluation and Revision Plan• Checkpoint #1 – syllabus, outline and first lesson
• Checkpoint #2 – half of the course, viewed on multiple platforms
• Checkpoint #3 – Entire course proofread/edited
• Checkpoint #4 – Entire course Q/A checked
• Checkpoint #5 – Final check (previous errors)
• Student survey after first three lessons
• Instructor survey after first three lessons
• Examination of help desk error logs
• Analysis of problems found
– How urgent is it?
– How long will it take to fix?
• Assign each issue a priority score
• Establish a threshold below which a course will be postponed
• Prioritized list of change requests…when’s the best time to revise?
• Assign corrections and establish a deadline for each
• Make note of unaddressed issues
Sample Summative Evaluation and Revision Plan
• Analyze student and faculty satisfaction surveys, identify common themes or trends
• Analyze assessment
• grade distribution
• item analysis as needed
• pre-test / post-test comparisons
• Analyze help desk logs, identify common issues
• Examine course archives (discussions, chats, etc.)
• Compile list of issues (including issues noted during formative phase that have yet to be addressed)
• Research solutions as needed
• Determine time needed to fix each issue
• Assign priority ratings
• Assign tasks and establish deadlines
Sample Checkpoint #1 Rubric
Improvement / Satisfactory / Excellent
Course Syllabus
/ Several required syllabus elements are missing; errors may be present / Syllabus is missing some required elements; typos and errors may be present / Syllabus is complete and detailed with few or no missing elements or errors; / Syllabus is complete and detailed with no errors; a printable version of the syllabus is included
Instructional Design / Organization or presentation of content is difficult to follow; sequencing is not logical; learning objectives are missing or unclear / Sequencing may be logical, but some course content may be difficult to find; more than 20% of course content is external to Blackboard / Course content is well-sequenced; most information is easy to find; clear learning objectives are stated, most are observable or measurable / Content is well-sequenced; advance organizers and/or summaries are present; clear learning objectives are stated and are observable and measurable
Visual Design / Design is inconsistent; use of colors and/or graphics is missing or distracting; course is text-heavy and difficult to read; horizontal scrolling may be required / Design may be inconsistent; some content is enhanced by colors and/or graphics; horizontal scrolling may be required / Design is consistent; all text is well-formatted; most content is enhanced by pleasing, unobtrusive colors and graphics; no horizontal scrolling is required / Design is consistent; all content is enhanced by pleasing, unobtrusive colors and graphics; no horizontal scrolling is required
Testing / Assessment Plan / No testing or assessment plan is in place; instructions for student learning activities or assignments are missing or unclear / Assessments may not align with stated learning objectives; assignments and activities do not include instructions / Course includes subjective assessments (discussions, homework, papers, projects); most assessments align with stated learning objectives; assignments and activities may need more / more clear instructions / Students will be evaluated through both objective and subjective assessments; all assessments align with stated learning objectives; opportunities for self- and peer-assessment are provided; assignments and activities include detailed, clear instructions
Sample Q/A Review Checklist
Every course should receive a thorough review of its technical aspects. This review might include the following:
____ review all content on every page - check at least 30% of course in each supported browser (IE, Firefox, Netscape, Safari), including at least one example each of media, java, javascript, etc.
____ review text for font size and style consistency, with special attention to headers
and footers
____ check every link – URL correct? Each external link opens a new window?
____ check all dates to be sure they are the same as in the syllabus and that they are
appropriate for the semester in which the course will be taught
____ check all times listed; each should specify time zone
____ take each test
____ check functionality of gradebook after each test
____ check all graphics
____ note any LMS functions not being used
____ check every media clip (usually not necessary to view/listen to the entire clip)
____ check that technical support directions are present
____ check for access to all materials; note any special instructions needed
____ confirm that all navigation items work correctly
____ check javascripts (mouse-overs, pull down menus, etc.)
____ check Flash movies and interactives
____ confirm that at least one meaningful post is in the discussion area
© Jennifer Freeman
Online Course Design 2008: Evaluation and Revising Online Courses
Sample Online Course Q/A Review FormCourse Number:
Use this area to describe problems based on the review principles document. Please list the general course area, page numbers where the problem occurs (if applicable), a brief description of the problem, and the web browser you were using when you encountered the error. Please be as specific as possible so that we can quickly find and fix the problems you encounter. We have listed an example for you below
Course Area / Page # / Description of Problem / Browser Used / Priority / CommentsName:
Freelancer Hours:
Date Completed:
______
Corrections Made by:
Date Corrections Completed: / Reviewer Rating
**** Excellent
*** Adequate
** Needs Improvement
* Unsatisfactory
© Jennifer Freeman
Online Course Design 2008: Evaluation and Revising Online Courses
SAMPLE FACULTY SATISFACTION SURVEY1. How many semesters have you taught online? ______
2. Did you author or co-author this course? ____Yes ____No
3. Please indicate your satisfaction level with each of the following:
(1 = very dissatisfied, 5 = very satisfied)
1 / 2 / 3 / 4 / 5 / n/a
Quality / amount of training you received (instructional design, technology tools, LMS)
Administrative support (policies and resources)
Deadlines and the course development schedule
Support received from instructional design and development staff during development
Support received from instructional design and development staff while teaching
Support received from technical support staff (help desk) during development
Support received from technical support staff (help desk) while teaching
Quality / amount of resources available to your students
Pacing of your course while being taught
Quality / amount of interactive content
Quality / amount of student-student interactivity
Quality / amount of student-instructor interactivity
Overall satisfaction teaching online this semester
4. How satisfied were you with each of the following features of your course in [LMS] :
(1 = very dissatisfied, 5 = very satisfied)
1 / 2 / 3 / 4 / 5 / n/a
Structure of lectures and notes
Appearance of lectures and notes
Structure of activities and assessments
Appearance of activities and assessments
Syllabus design
Syllabus content
Navigation
Quality / number of tools available
Discussion area
Text chat tool
VoIP chat tool
Survey tool
Email tool
ePortfolio tool
Wikis, blogs, student home pages
Testing tools
Gradebook
Adaptive release functionality
Overall Ease of use
5. How would you rate your satisfaction teaching an online course in general?
___Very satisfied ___Satisfied ___Neutral ___Dissatisfied ____Very Dissatisfied
6. In general, how would you compare teaching your online course to an on-campus course?
___Much more difficult ___More difficult ___About the same ___Less difficult ____Much less difficult
7. Would you consider teaching other online courses in the future? ____Yes ____No
8. Do you have any suggestions as to how the Distance Learning Team could better support your teaching and/or your class?
SAMPLE STUDENT SATISFACTION SURVEY
How satisfied were you with each of the following aspects of your online course? (1 = very dissatisfied, 5 = very satisfied)
1 / 2 / 3 / 4 / 5 / n/a
Layout, design and appearance of course
Ease of navigation / finding needed information
Pacing of course
Activities and assessments
Tools available to you through [LMS]
Syllabus
Lectures / notes
Discussion area
Chat tools
Testing tools
ePortfolio, blogs, wikis, student homepages
Support received from technical support staff (help desk)
Support received from your instructor
Quality / amount of interactive content
Quality / amount of student-student interactivity
Quality / amount of student-instructor interactivity
Quality / amount of resources available to you (library, tutoring, etc.)
Overall satisfaction with this course
Overall satisfaction with this instructor
1. Had you ever taken an online course BEFORE this semester? ____Yes ____No
2. Did your advisor speak to you about the advantages and disadvantages of taking an online course? ____Yes ____No
3. Describe your motivation for taking this course online as opposed to on-campus.
4. How would you compare this course to an on-campus course in the difficulty level of the coursework?
___Much more difficult ___More difficult ___About the same ___Less difficult ____Much less difficult
5. How would you compare this course to other online courses you’ve taken in the difficulty level of the coursework?
___Much more difficult ___More difficult ___About the same ___Less difficult ____Much less difficult
6. How would you compare this course to an on-campus course in terms of the time you spent?
___Much more time ___More time ___About the same ___Less time ____Much less time
7. How would you compare this course to an other online courses you’ve taken in terms of the time you spent?
___Much more time ___More time ___About the same ___Less time ____Much less time
8. How easy did you find [LMS] to use?
___Very easy ___Easy ___Neutral ___Difficult ____Very difficult
9. What did you like BEST about taking this course online?
10. What did you like LEAST about taking this course online?
11. Would you take another online course? ____Yes ____No
12. If you answered "Maybe" or "No" above, please explain why.
SAMPLE FACULTY PEER REVIEW FORM
Course Name: / Date:
Course Number: / Instructor:
SYLLABUS / Syllabus includes all required elements ____Yes ____No
Syllabus is easy to read and navigate ____Yes ____No
Overall quality of syllabus ___Excellent ___Good ___Fair ___Poor
GOALS AND OBJECTIVES / Goals and objectives are stated at the course level ____Yes ____No
Goals and objectives are stated at the lesson level ____Yes ____No
Goals and objectives are observable and measurable ____Yes ____No
Overall quality of course objectives ___Excellent ___Good ___Fair ___Poor
CONTENT STRUCTURE / Course content is well sequenced ____Yes ____No
Course includes a site map ____Yes ____No
Information is easy to find ____Yes ____No
Overall quality of course structure ___Excellent ___Good ___Fair ___Poor
COMMENTS
INSTRUCTIONS FOR ASSIGNMENTS AND ACTIVITIES / Clear instructions are provided ____Yes ____No
Student expectations are provided ____Yes ____No
Grading rubrics are provided ____Yes ____No
COMMUNICATION AND INTERACTIVITY / Instructor contact information is provided ____Yes ____No
Instructor communication plan is provided ____Yes ____No
Course announcements are provided ____Yes ____No
Course discussions are provided ____Yes ____No
Course chats are provided ____Yes ____No
Students use the discussions ____Yes ____No
Students use the chats ____Yes ____No
Students work in pairs or groups ____Yes ____No
Students self- or peer-assess ____Yes ____No
COMMENTS:
COURSE CONTENT / Information sources are identified ____Yes ____No
Copyright permissions are in place ____Yes ____No
Content is credible ____Yes ____No
Content breadth is sufficient ____Yes ____No
Content is presented clearly ____Yes ____No
Content is error- and typo-free ____Yes ____No
Student resources are identified ____Yes ____No
Student learning styles’ are considered ____Yes ____No
Content is accessible ____Yes ____No
TECHNOLOGY / Most LMS tools are used ____Yes ____No
Technology requirements are identified ____Yes ____No
Technical support contact is identified ____Yes ____No
Course media and tools function properly ____Yes ____No
COMMENTS:
© Jennifer Freeman