Ontario Association of Career Colleges

Ontario Association of Career Colleges

Program Assessment Process Consultation Response

Date: July 4, 2008

Submitted to:

Shamira Madhany, Director, Postsecondary Accountability Branch, MTCU

Richard Jackson, Director, Student Support Branch, MTCU

Submitted by:

Paul Kitchin, Executive Director

Ontario Association of Career Colleges

Table of Contents

A.Introduction3

B.Executive Summary3

C.Critical Outcomes5

D.OACC Feedback5

Part 1 Qualifications of Assessors5

1.1 Academic Qualifications of Assessors5

1.2 Definition of Field5

1.3 Qualifications Flexibility – Emerging Program Areas6

1.4 Assessment of Regulated and Professionally Programs6

1.5 Conflict of Interest Clarification6

1.6 Tag Team Assessments6

Part 2 Program Review Standards7

2.1 RICC Concerns7

2.2 Assessors and RICC7

2.3 Flexibility for Streamlined Assessments7

2.4 Administrative and Compliance Verification7

Part 3 Program Quality Assessment Report8

3.1 Site Visit Criteria8

3.2 Consistent Terminology9

3.3 Benchmarks for Assessment Duration9

3.4 Separation of Programmatic and Operational Assessment9

3.5 NACC Programs9

3.6 Iterative Assessment Process Between a PCC and the Assessor10

3.7 Conditional Program Approvals10

Part 4 Implementation of Process10

4.1 Program Standards10

4.1.1 Program Categories10

4.1.2 Approval of Regulated and Professionally Recognized

Programs 11

4.1.3 Roll Out for Program Standards11

4.2 Timing of Re-approvals11

4.3 Credential Shift12

4.4 Variable Rigor12

4.5 No Pre-approved Assessor on the Roster for a Field12

4.6 RICC Manual and Training12

4.7 Program Approval Applications Currently in the System13

Part 5 Timing of Implementation13

5.1 Sufficient Transition Period13

5.2 RICC Matters13

5.3 Pilot Projects14

5.3.1 Pilot – Assessment Process14

5.3.2 Pilot – New Program Standards14

E.Program Assessment Industry – Big Picture Questions14

F.Summation15

A.Introduction

The Ontario Association of Career Colleges (OACC) very strongly supports the objectives of the PCC Act, 2005 that are intended to improve student protection, quality assurance, accountability and sector capacity within the private career college sector in Ontario. In particular, OACC supports the adoption of the Ontario Qualifications Framework and the movement toward establishing and meeting program standards. As outlined in our submission to the Ministry as a part of the review of the PCC act in 2004,

OACC believes that program standards and the credentials framework form the foundation for providing quality assurance to potential students and to the employers that hire career college graduates.

Thus, there is full support for the concept of moving to program standards that incorporate the criteria included in the Ontario Qualifications Framework, and an acknowledgement that the use of a meaningful and efficient program assessment process will be a key to ensuring that PCCs deliver high quality programs that adhere to agreed to standards.

Accordingly, we are pleased to have been invited by the Ministry to participate in the consultation process with respect to the development and implementation of a new process for the approval of vocational programs to be offered at private career colleges that will ultimately provide such assurance of quality. It is OACC’s position that programs of study must be well designed, appropriately assessed, effectively delivered and continually maintained in order to assure their quality.

We understand from your letter dated May 12, 2008 and sent to OACC on May 27, 2008, the draft documents that accompanied that letter, and the briefing that OACC committee members received from Charlotte McCloskey and Donna Vogel on June 5, 2008 that it is the Ministry’s intention to develop:

-a comprehensive set of program review standards against which programs would be evaluated, and

-a more comprehensive assessment report, and

-a set of qualifications for program quality assessors, and

-a roster of pre-approved external program quality assessors that PCCs could select from to evaluate their programs

OACC has reviewed the draft materials and considered the information and clarifications that were provided to OACC by the Ministry on June 5, 2008. As a result, our feedback as detailed in section D of this paper has been gathered into the five broad categories of Qualifications of Assessors, program Review standards, Program Quality Assessment Report, Implementation of Process, and Timing of implementation. It should be noted that there is, of necessity, an overlap between the five categories.

B. Executive Summary

OACC fully supports the concept of moving to program standards that incorporate the criteria included in the Ontario Qualifications Framework, and use a meaningful and effective program assessment process however; OACC finds it impossible to separate the concept of program assessment from that of program standards from that of the qualifications framework. The use of a program assessment process must raise the level of assurance that private career college programs of study meet appropriate standards, reduce program approval times, and facilitate the introduction of new programs.

This paper identifies a number of concerns with the proposed assessment process and suggests a number of alternate approaches and actions that bear consideration in addressing such concerns.

The concerns itemized in this summary and many others are discussed in greater detail throughout the paper.

-There are concerns that the proposed qualifications will lead to a lack of qualified assessors, an inability to approve new programs in emerging fields, and an overlap of the roles of assessors and Program consultants (particularly in respect to administrative compliance matters).

There are concerns about the ability of RICC to effectively handle program applications and align with the assessment process.

There is concern that the process does not allow for sufficient flexibility, or variable rigor or appeals of assessment decisions.

-There is concern that there are no guidelines on the expected time to be taken to complete program assessments making costing unpredictable.

There is concern that some program fields may have no pre-qualified assessors.

Overall, there is deep concern that rushing to implement the process before the matters raised in this paper have been properly dealt with would be very problematic for the Ministry and for the career college sector.

In light of the concerns expressed and a number of questions of clarification that have been posed, OACC has some suggested approaches and actions for consideration and discussion over the coming months.

It is suggested that there be joint reviews by the Ministry and the sector on a variety of topics including but not limited to academic criteria for assessors, the definition of program field, accepting attestations from regulatory bodies or professional associations, allowing the use of assessment teams, allowing conditional program approvals, and allowing assessment streamlining under appropriate circumstances.

It is suggested that a focus group of users be established to test and improve RICC to better align it with the assessment process, develop an RICC user manual and establish a training plan for assessors and users.

-It is suggested that a group or groups be formed to work on establishing guidelines or benchmarks for the time required to complete an assessment, an appeals process, a roll out plan for the development of program standards, a roll out plan for program re-approvals an approach to NACC programs, and the schedule for approving assessors.

It is also suggested that pilot projects be initiated over the next year regarding the proposed new assessment process and regarding the establishment of new program standards where none exist.

OACC looks forward to receiving the answers and clarifications that are being sought from the Ministry and having the opportunity to discuss these matters further.

C.Critical Outcomes

It is OACC’s position that the use of external program quality assessors in the program approval process must achieve three critical outcomes. It must raise the level of assurance within the postsecondary education community, the employer community, and the general public that private career college programs of study meet appropriate standards. It must also significantly reduce the time it is currently taking to get programs approved. Finally, it must facilitate the approval of new programs of study in emerging fields. One strength of the PCC sector has been its ability to react quickly to the current and future needs of the labour market through accelerated development of new programs of study. A delay of more than 6 months to approve a program after it has been fully designed is unacceptable. OACC is seeking assurance from the Ministry that all program approval and re-approval applications submitted with a program assessor’s approval should be approved in a timely manner. Given that the proposed process includes updating and improving third party program assessors who will be validating that the program approval applications are complete, and that all program information will have been loaded into RICC in advance, there should be no reason for approvals to take longer than 30 days for clean applications.

D.OACC Feedback

Part 1Qualifications of Assessors

1.1Academic Qualifications of Assessors

It is not clear to OACC that entry-level training programs would require an assessor that has a terminal degree in the field of study. OACC suggests that a more appropriate requirement would be that the assessor be an individual that has a credential that is at least one level above the proposed credential and demonstrated extensive related industry experience. For example, an assessor who has a Bachelor degree should be able to assess a diploma or certificate program.

1.2Definition of Field

Given that there is close to 3,000 programs currently being offered in the PCC sector, OACC has a real concern that it will be very difficult to attract enough assessors with the proposed qualifications. OACC believes that this concern will be magnified unless the definition of “field” in reference to programs and assessor qualifications is allowed to be at the broadest possible level. OACC suggests that the fields could be Business, Healthcare, Technology, Beauty, etc.

OACC sees no reason why an assessor with a degree in computer Science could not assess a variety of technology programs including network specialist, web design, or programmer for example. OACC believes that the ultimate goal of the assessment is that we want the assessor to be able to say the proposed program is reasonable, similar to what a C.A. does in an audit. OACC believes that the use of a narrower definition of “field” will be problematic.

1.3Qualifications Flexibility – Emerging Program Areas

OACC believes that the qualifications for assessors need to include some flexibility so that there is a mechanism for assessing programs in emerging fields. The proposed requirement that assessors have 3 to 5 years experience will preclude anyone from qualifying from being an assessor for such emerging program areas. In such cases, OACC suggests that it may be necessary to approve emerging programs more based on demands from employers and from the labour pool. Perhaps the first program approved becomes the standard until there is enough of a pool of programs that have been approved and program standards can be established through a peer review of the existing programs. Institutions could then have one year to meet the standard.

1.4Assessment of Regulated and Professionally Recognized Programs

For programs that are regulated by regulatory bodies or are professionally recognized by professional associations, OACC suggests that there is no need to assess such programs. A letter from the regulating body or professional association that states that they have evaluated and approved a program should override the need to follow the program report and review process that is being proposed. Examples of professional associations are CIDA for internal designers I North America or AATO for architects in Ontario.

1.5Conflict of Interest Clarification

OACC is concerned that the proposed conflict of interest requirements could hamper the ability of PCCs to engage a qualified assessor in some cases. OACC suggests that there needs to be flexibility that would allow a Ministry pre-approved assessor that had done some prior contract work for a PCC to be eligible to assess a program as long as they are not assessing a program that they have helped to design. Also, OACC suggests that having assessed a program previously for a PCC should not preclude that assessor from assessing subsequent programs for the same PCC. If agreeable, OACC suggests that these clarifications be included in the assessor qualifications document.

1.6Tag Team Assessments

OACC has some concern that the proposed process requires the identification of assessors that have both the content knowledge and the pedagogical and design expertise, which could severely limit the number of individuals that could qualify. OACC suggests that some flexibility be built in to allow for a team of two people to do assessments. The first individual would be an individual with the design knowledge who could engage content experts, many of whom might come from the PCC sector itself, establishing a quasi peer review type of approach. This is a great opportunity for the sector to build capacity, thereby preparing itself to eventually provide peer review as part of an accreditation framework. Building the capacity to assess might also increase the ability of individual career colleges to develop and deliver quality programming.

Part 2 – Program Review Standards

2.1RICC Concerns

OACC has concerns that the current program approval application on RICC does not include many of the questions that are included in the proposed assessment. OACC suggests that before the new assessment process can be launched, it will be necessary to align the program approval application on RICC to the assessment. OACC also suggests that once that alignment is completed, there needs to be testing done of the application on RICC using real data. Feedback from OACC members that have recently tried to use the RICC program approval application have found it very time consuming, and not as user friendly as it could be, with very little room for entering the required data.

2.2Assessors and RICC

It is still unclear to OACC how the assessor will be expected to relate to the program approval application on RICC. Will the assessor be accessing the RICC version of the application or will they be sent hard copies printed from RICC? Will any of the assessors report be entered on RICC? If so, has sufficient time been allotted to test this part of RICC before launching the new assessor process? Will the assessment report be submitted to the Superintendent through RICC or will it be a hard copy?

2.3Flexibility for Streamlined Assessments

OACC is concerned that there may be some instances where the assessment process may require assessor activity that may not be necessary in all cases, thereby causing extra time and costs. OACC suggests that some work still needs to be done to identify any program areas where an assessment activity such as site visits might not be totally necessary. In many cases, site visits are necessary for programs that have extensive equipment requirements. However, does it make sense to perform a site visit in order to assess a new accounting program? Additionally, are there cases where the provision of a purchase order covering the required equipment would suffice instead of conducting a site visit? OACC suggests that for a career college that already has programs that have met the Standard, have passed the assessment and have been approved, consideration might be given to waiving some of the process requirements for approving new programs under the same college Director or owner, such as site visits and/or whatever administrative compliance matters remain part of the process.

2.4Administrative and Compliance Verification

OACC is somewhat confused over where the responsibilities of the assessor leave off and the responsibilities of the Program Consultants take over with respect to administrative compliance issues under the proposed process. There are several references to the assessor checking if a PCC is in compliance with the Act in the draft documentation. Examples include, responsibility for checking the proposed credential against the Ontario Qualifications Framework, admissions, promotion and graduation requirements consistent with the stated learning outcomes of the program and in compliance with the requirements of the PCCA, advanced standing policies, Promotion and graduation requirements, instructor qualifications etc. To what degree will assessors be responsible for verifying administrative compliance with the legislation and regulations, and what level of administrative compliance audit will they be expected to perform? OACC is concerned that requiring assessors to complete compliance checks may lengthen the assessment process, drive up the cost of assessments, and further restrict the number of people who might be qualified to be approved as assessors.

Additionally, in the delivery section of the draft document on review standards, what is meant by “quality assurance policies”? Is this more of a compliance matter than an assessment matter?

Generally, OACC is concerned that there is an expectation that assessors will need to be experts on the legislation and regulations in order to conduct administrative compliance checks. Again, we are not sure if this is an appropriate role for an assessor.

OACC is curious about the reference in the documentation to the requirement for assessors to check for assessments by students, as this cannot be done until after the program has been approved and delivery of the program has commenced.

The documentation makes reference to assessors checking for a demonstration of labour market demand. What criteria would assessors be checking labour market demand against?

As discussed in the section above on Flexibility for Streamlined Assessments, OACC suggests that for a career college that already offers programs that have met the standard, have passed the assessment and have been approved, consideration might be given to waiving some of the process requirements for approving new programs such as the administrative compliance matters that remain part of the process.

In the draft documentation there was reference to subject specific standards. OACC is asking for some clarification on what the Ministry means by subject specific standards.

Part 3 – Program Quality Assessment Report

3.1Site Visit Criteria

The draft documentation states that “In most cases there will be site visits” conducted. Under what circumstances does the Ministry contemplate that site visits may not be required?

3.2Consistent Terminology

For sake of consistency, OACC suggests that the terminology with respect to the “assessors” be the same throughout the documents. Currently the terminology flips from assessor to quality assessor to program quality assessor.