PREFACE 4
PROGRAM QUALITY ASSURANCE – AN OVERVIEW 5
Program Quality Assurance Process Audit (PQAPA) 5
Program Quality Assurance at Algonquin College 6
PROGRAM QUALITY REVIEW 7
PQR Goals 7
PQR Process 7
Schedule 7
Roles 9
Orientation 11
Release Time 11
Timeline 12
Need Assistance? 12
THE FINAL REPORT – AN OVERVIEW 13
The Main Document 13
The Appendices 14
DOCUMENT REVIEW 15
Program Monograph 15
Program Council Meeting Minutes 16
Advisory Committee Meeting Minutes 16
Further Study Pathways 16
Reports from External Accreditation or Licensing Bodies 17
Program Learning Outcomes 17
Program of Study 17
Course Outlines and Course Section Information 17
DATA REVIEW 18
Registration Data 18
Student Success Data 19
Student, Graduate and Employer Satisfaction Data 21
Financial Data 22
CURRICULUM MAPPING 23
Purposes of Curriculum Mapping 23
Degree Level Standard 23
Degree Level Learning Outcomes 25
Performing Curriculum Mapping 25
EVALUATION AGAINST THE QUALITY CRITERIA 28
Evaluation Criteria Listing 28
Responding to the Evaluation Criteria 31
Evaluation Criteria Listing – with help text 31
CONDUCTING FOCUS GROUPS 49
Memos from the Chair and Team Leader 49
Setting the Focus Group Agendas 50
Tips for Planning and Facilitating Focus Groups 50
Faculty Focus Group 51
External Stakeholders Focus Group 52
Student Focus Group 56
COMPLETING THE FINAL REPORT – AN OVERVIEW 62
ANNUAL FOLLOW UP 63
GLOSSARY AND LIST OF ACRONYMS 64
APPENDIX A – THE FINAL REPORT 65
APPENDIX B – THE ONTARIO QUALIFICATIONS FRAMEWORK 72
APPENDIX C – TEMPLATES 73
APPENDIX D – KEY PERFORMANCE INDICATORS 97
APPENDIX E – PROGRAM MIX DATA 98
PREFACE
This Program Quality Review (PQR) Team Leader guide is written to aid Program Quality Reviews for Bachelor’s Degrees offered at Algonquin College. It is an adaptation of the main Team Leader Guide for Ontario College Credentials, located at http://algonquincollege.com/acad_dev/MinistryApprove.htm .
The final PQR report, also referred to as the ‘program self-study’ is included in documentation pertaining to PEQAB degree consent renewal, and provided to members of an external Program Evaluation Committee, who independently carry out a quality assessment of the Bachelor’s Degree program.
Documentation and guidelines pertaining to any degree consent renewal, including the external Quality Assessment are available from the Program Quality Assurance Administrator.
PROGRAM QUALITY ASSURANCE – AN OVERVIEW
Program Quality Assurance Process Audit (PQAPA)
One of the prerequisites for granting greater autonomy to the Colleges of Applied Arts and Technology in Ontario to govern their own programs is the implementation of a process to assure program quality. The Ontario Colleges of Applied Arts and Technology Act, 2002 outlined two elements of this expectation:
· quality assurance at the individual college level, and
· a self-regulating process at the system level.
The Program Quality Review process forms a major part of adherence to providing quality assurance at the College level, and is therefore essential to College operations.
At the system level, that is, at the level which encompasses all Colleges of Applied Arts and Technology in Ontario, a separate quality assurance process takes place, the Program Quality Assurance Process Audit (PQAPA). This is run through the Ontario College Quality Assurance Service (OCQAS[1]). OCQAS was established to provide effective and efficient mechanisms that ensure specific program quality and consistency standards are met by the Colleges of Applied Arts and Technology (CAAT) in Ontario.
The following six[2] criteria define institutional policies and practices that OCQAS expects a college to have developed and implemented to ensure the quality of their programs:
Criterion 1. Program-level learning outcomes for all programs of instruction are set, are consistent with the college mission and the programs’ intended purpose, and are appropriate for the credential offered upon successful completion of the program.
Criterion 2. Admission, credit for prior learning, promotion, graduation, and other related academic policies support program development and student achievement of program learning outcomes.
Criterion 3. Programs conform to the Framework for Programs of Instruction[3] and the Credentials Framework, are consistent with accepted college system nomenclature / program titling principles, and maintain relevance.[4]
Criterion 4. Methods of program delivery and student evaluation are consistent with the program learning outcomes.
Criterion 5. Human, physical, financial, and support resources to support student achievement of program learning outcomes are available and accessible.
Criterion 6. Regular program quality assessment that involves faculty, students, industry representatives, and others as appropriate for the purpose of continual improvement is in place and happens.
Program Quality Review evaluation criteria are based on the quality criteria above.
3
Program Quality Assurance at Algonquin College
Program review is annual event at Algonquin College, and currently consists of four processes:
· Annual Curriculum Review
· Annual Program Mix Review
· Cyclical Program Quality Review
· Cyclical Review of Online Elective General Education courses[5]
Annual Curriculum Review
Each year program faculty members revise and update curriculum to maintain its currency and to ensure that graduates will have the skills necessary to be successful in the workplace.
Feedback from faculty, students, and advisory committee members is taken into consideration before updates are made to keep the program current. Revisions to the curriculum and the program narrative information are made in GeneSIS, the student information system, and are subsequently reviewed in the Office of the Vice President Academic. Once the changes are approved, a new program version is created. The information in GeneSIS becomes the foundation for course loading, fees loading, scheduling, registration and College publications.
Program Mix Review
Staff in the Academic Operations and Planning department compile data that reflect the Quality Measures for a program. Admissions and registration data, employment rates, capstone questions from Key Performance Indicators (KPI) and course assessment surveys, and program costing data are reviewed. Programs not meeting college benchmarks are reviewed and strategies are developed by the Chair and Dean to improve the quality of the program and/or the financial contribution.
Program Quality Review
Program Quality Review (PQR) ensures that each program goes through a comprehensive review every five years. The PQR process builds on annual review activities by including analysis of three to five years of data compiled for the program mix review, and soliciting direct feedback from stakeholders and students into the program review. Each PQR results in an implementation plan containing resourced actions to maintain and improve the program over the next five years.
This manual guides Team Leaders when conducting a Program Quality Review.
Cyclical Review of Online Elective General Education courses
Online General Education courses form an integral part of many programs offered at the College. In 2009, the review process for these courses was formalized. Each of the online General Elective courses in the College pool is reviewed according to a published cycle, first by the General Education committee for overall adherence of the course to General Education objectives, and then in detail by individual faculty and student reviewers.
PROGRAM QUALITY REVIEW
PQR Goals
· To assess the program against established quality criteria in order to determine whether the program meets or exceeds College and Ontario College Quality Assurance Service (OCQAS) standards (for Ontario College Credentials) or Postsecondary Education Quality Assessment Board (PEQAB) standards (for Bachelors’ Degrees).
· To establish and implement a quality improvement plan[6] for the program, in response to the findings of the review.
PQR Process
Program Quality Review follows a systematic process, and is outlined in Figure 1. After the Team Leader has been identified to conduct the review, information pertinent to the program is collected and analyzed, allowing an initial evaluation of the program against established quality criteria. The program information reviewed includes published and internal documentation, program data, current curriculum maps, and input collected from key players in the program – faculty, students, and external stakeholders.
Once the review is complete, the evaluation against the criteria is finalized, and all findings and recommendations for improvement are captured in a report. An implementation plan for the recommendations is drawn up, with an update on the actions provided to the PQAA at least annually until completion.
The PQR Team Leader coordinates the PQR, with assistance from the Chair.
Note that this guide is written to provide assistance with each of the main components of PQR:
· Document Review
· Data Review
· Curriculum Mapping
· Evaluation Against the Quality Criteria
· Conducting Focus Groups
· Completing the Final PQR Report
Schedule
Programs are reviewed according to a defined schedule, normally every five years. The schedule is revised annually, in consultation with the Academic Departments, and aligned with other Quality Assurance requirements, for example, accreditations or consent renewals for bachelor’s degrees.
The PQR schedule is published at http://www.algonquincollege.com/acad_dev/services.htm.
Cyclical Program Quality Review
(Figure 1)
Roles
Program Quality Assurance Administrator
Program Quality Reviews are coordinated by the Program Quality Assurance Administrator (PQAA), who reports to the Dean, Academic Development, who in turn reports to the Vice President Academic.
The responsibilities of the Program Quality Assurance Administrator include
· ensuring the College is compliant with Ministry requirements regarding Program Quality Assurance.
· consulting with major academic constituencies College-wide regarding Program Quality Review.
· preparing documents related to Program Quality Review.
· planning and organizing orientation/training workshops for Team Leaders.
· supporting the Program Quality Review process in programs.
· meeting regularly with PQR Team Leaders during the review process to monitor progress and provide guidance.
· facilitating evaluation of the Program Quality Review process.
· arranging printing of the final report; forward copies to the TL, Chair, Dean and Vice President Academic.
Program Quality Review Team Leader
The PQR Team Leader is normally a faculty member who is assigned to conduct Program Quality Review for the program in which s/he teaches.
The responsibilities of the Team Leader include
· attending all meetings scheduled by the Program Quality Assurance Administrator.
· attending all orientation/training workshops related to the Team Leader role.
· reporting to the Chair weekly to monthly on the progress of the Program Quality Review.
· analyzing data and incorporating the results of the analysis in the final PQR report.
· convening meetings of the program faculty team.
· convening a Student Focus Group.
· convening an External Stakeholders / Advisory Committee Focus Group.
· facilitating Student and External Stakeholder Focus Groups for a program from another School and submitting reports for these meetings.
· evaluating the Program Quality Review process.
· submitting the final report to the Chair and PQAA, who will review and sign-off. (The Chair then forwards the report to the Dean for final review/sign-off, and notifies the PQAA who will advise the Vice President Academic of its completion.)
Dean
The responsibilities of the Dean include
· identifying programs for Program Quality Review.
· supporting the Program Quality Review process.
· reviewing and signing-off on the Program Quality Review report.
· following up regarding results and recommendations included in the report.
· reporting to the Vice President Academic on the progress of Program Quality Review and regarding the follow-up on the recommendations.
Chair
The responsibilities of the Chair include
· identifying Team Leaders (TL) each spring for programs identified to be in the next cycle. It is preferred that a full-time professor or coordinator is assigned.
· acting as a facilitator for Focus Groups in the event of illness of the TL.
· meeting with each TL in May to discuss the upcoming process, and to encourage and facilitate attendance at the Aligning and Building Curriculum Conference in the upcoming Fall and following Spring.
· attending information meetings for Chairs.
· providing release time for the TL as outlined in the PQR process
· meeting with the TL early in the process to review the program data and initial assessment of the quality criteria. Assisting with agenda setting for focus groups and helping to create a list of those to invite.
· providing the relevant budget code for all expenses.
· sending an email to all program faculty inviting them to participate in the PQR process and to fill out the credentials and experience sheet.
· providing a summary report of faculty credentials and experience to the Team Leader, to be included in the final report.
· arranging secretarial support as outlined in the PQR process
· meeting regularly with the TL and monitoring progress.
· attending focus group meetings and introducing the group members to the process.
· ensuring feedback is provided to all those who participated in the focus groups.
· providing feedback regarding the PQR process.
· collaborating in the development of the final recommendations and implementation plan, ensuring adequate budgetary support for those recommendations that are to be implemented immediately (i.e. within one year) and consideration of those recommendations for which funds are not currently available.
· including PQR recommendations in their Performance Contract.
· facilitating the implementation of the final recommendations.
· tracking progress on the implementation plan, and preparing the submission of the PQR Annual Follow Up Report until all recommendations have been implemented.
Orientation
Orientation for Team Leaders is scheduled outside of classroom contact weeks to minimize the negative impact of training within teaching weeks.
The orientation process for Team Leaders involves a series of workshops, each three hours in length. Upon completing the Team Leader orientation process, Team Leaders are able to
· evaluate a program against identified criteria; identify quality in a program when measured against identified standards and determine how to reinforce/maintain this quality.
· lead focus groups for internal and external stakeholders.
· map curriculum to validate that the program of study meets current Ministry requirements.
· analyze program data and make inferences based on trends.
· analyze curriculum maps to identify gaps and redundancies.
· formulate realistic and attainable recommendations to improve quality where needed.
Release Time
Team Leaders are entitled to receive the following release time for the duties and responsibilities associated with Program Quality Review:
Fall 2011 or Winter 2012:
Release from teaching hours for the equivalent of one three-hour course including evaluation and preparation time which, for consistency purposes, has been equated to six hours of assigned workload. Faculty members who are reviewing a three-year program, a four-year program, or the equivalent of three or more years of curriculum, will be assigned a nine hour workload. All authorized travel expenses incurred by Team Leaders to facilitate Focus Groups at another campus will be reimbursed.
In addition to the Team Leaders’ release time, additional financial support is provided to departments as follows:
· Support Staff: up to 48 hours per PQR at $15 per hour
· Part-time Faculty: up to 16 hours per PQR at $30 per hour
· Hospitality[7]: up to $100 per PQR to contribute towards light refreshments at focus groups.