Leading a Course Review

Table of Contents

Introduction

Learning Outcomes

Module Topics

How the quality of a course is monitored at the national level

How course quality is monitored at Curtin

Course quality and review process

Annual Course Review

Comprehensive Course Review

Guidelines for interpreting course performance data

Practical Activity

Required Reading

Additional Resources

References

1 Academic Leadership for Course Coordinators Program

Introduction

Hello and welcome to Leading a Course Review

In this module you will be exploring the need to review the key performance indicators for a course or major on an annual basis. You will identify a range of sources for obtaining key course performance indicator data and analyse the data to identify areas of strength and for improvement. It will give you an opportunity to explore a range of institutional data to use to undertake and annual ‘health check’ on the performance of the course/major that you coordinate. With this information you will be able to complete the Annual Course Review required as part of the University Course Review policy.

Please Note:There are a number of weblinks throughout this module which are current at the time of publication, if any are broken or fail to open,please advise your Program Coordinator or post a message to the Academic Leaders Café in Blackboard in the forum titled, ‘Help us to help you!’ Thank you.

Learning Outcomes

On successful completion participants will:

  • Interpret key performance indicators for annual course review.
  • Work with a course team to identify course strengths and devise strategies to maintain them.
  • Work with a course team to identify course areas for improvement and devise strategies to address them.

Module Topics

This module looks at Leading a Course Review and includes guidelines on the types of data and their interpretation to assist with the evaluation of course quality and development of strategies for course improvement.

The topics to be covered are:

  • How the quality of a course is monitored at the national level.
  • How course quality is monitored at Curtin.
  • Some key principles of course financial viability
  • Course quality and review process.
  • Annual Course Review.
  • Comprehensive Course Review.
  • Guidelines for interpreting course performance data.

How the Quality of a Course is Monitored at the National Level

National Level

In 2003, the Australian Government announced the Learning and Teaching Performance Fund (LTPF) as part of the Our Universities: Backing Australia’s Future initiative. The purpose of the fund is to reward higher education providers that best demonstrate excellence in learning and teaching. Funding of $54.4 million in 2006, which increased to approximately $82 million in 2007 and $109 million in 2008 will be allocated as part of the Government's renewed focus on teaching quality in Australian universities. For latest information on the LTPF, go to the Department of Education, Employment and Workplace Relations ( and search for Learning and Teaching Performance Fund.

Following feedback on the 2006 LTPF performance measures, DEST (Department of Education Science and Training) modified them in 2007 to include seven measures and to calculate results in four discipline clusters. DEST has continued to apply an ‘adjustment process’ (as for 2006) to take into account individual differences across universities. Note that for the 2007 calculations:

  • LTPF measures were drawn from data about on-shore undergraduate courses.
  • The Course Experience Questionnaire (CEQ) scores (used for three of the measures) drew on post-graduation onshore undergraduate course level feedback from 2005 (from students who exited their course in 2004).
  • Progress rates included all undergraduate students (not just first year students).
  • Both further full and part-time study was included as measures of teaching and learning success.
  • All of the indicators were weighted equally.

Performance Indicators

The performance indicators used in the Learning and Teaching Performance Fund (LTPF) are derived from the Australian Graduate Survey (AGS) which incorporates the Graduate Destination Survey (GDS) and the Course Experience Questionnaire (CEQ), and DEST’s annual university statistics collection.

The AGS is a national survey conducted by each university in association with Graduate Careers Australia (GCA). The survey consists of two components, the GDS and the CEQ for coursework graduates, or the Postgraduate Research Experience Questionnaire (PREQ) for higher degree by research students.

All graduates who studied at, or via, an Australian campus are invited to complete the AGS around four months after the completion of their course. The survey can be completed online or in a paper format.

The aggregated data from the AGS has a variety of uses:

  • Students may use the GDS data as part of their job-hunting process, as it gives the job titles, employers and average annual salary of students who completed the same course in previous years.
  • Universities and Government Departments use the data as key performance indicators, against which their performance is measured.
  • Good Universities Guide uses the data as part of their process for determining the ratings of national universities.

The AGS is Australia’s official annual review of the activities of university graduates who have recently completed their qualifications. Information is collected about their employment status, the type of work gained and any further study undertaken. The following GDS data are used in the LTPF:

  • graduate full-time employment, that is the number of Australian graduates in full-time employment expressed as a proportion of all domestic bachelor graduates available for full-time work (taken from responses to Questions 3 & 6 of the GDS)
  • graduate part-time or full-time study, that is the proportion of domestic bachelor graduates proceeding to further full-time or part-time study (taken from responses to Question 3 & 6 of the GDS).

The CEQ covers the attitudes of graduates towards their courses and the skills they acquired while undertaking tertiary education. The following CEQ data are used in the fund:

  • Generic Skills Scale that is the level of satisfaction with generic skills acquired (taken from responses to Questions 6, 14, 23, 32, 42, 43 of the CEQ).
  • Good Teaching Scale that is the level of satisfaction with good teaching (taken from the responses to Questions 1, 3, 10, 15, 16, 27 of the CEQ).
  • Overall Satisfaction Index that is the proportion of graduates who were satisfied with the overall quality of their courses (taken from responses to Question 49 of the CEQ).

Each year, the department collects comprehensive information about students attending Australian higher education institutions. The data are sourced from the institutions themselves. The fund uses the following data from:

  • student progressrates, that is the proportion of subject load passed by students
  • retentionrates, that is the proportion of students who either completed in that year or were retained in the subsequent year.

The four Field of Education groups are:

  1. Science, Computing, Engineering, Architecture and Agriculture (SCEAA).
  2. Business, Law & Economics (BLE).
  3. Humanities, Arts & Education (HAE).
  4. Health (H).

Adjustment process

The data collected from the AGS and the DEST’s university statistics holdings are ‘raw’ or ‘crude’ data. DEST applies an adjustment process to the crude scores for each of the seven performance indicators. The adjustment process is designed to provide a ‘level playing field’ so that all universities participating in the fund may be assessed on their learning and teaching performance alone, independent of external influences such as socio-economic status of students, location etc.

In order for universities to improve their ranking and achieve funding (and improve reputation), they need to perform clearly above average in all performance indicators in any cluster (bearing in mind that all universities are attempting to do this).

The basic building block of a university’s performance in the LTPF is the course, and it is only by having high performing courses that universities can improve their national performance.

Course Coordinators, together with their Heads of School, play a key role in monitoring and improving the course. You can find out how your own course performed in relation to your Field of Education (FOE) group by going to Curtin’s homepage ( and searching for Information Reportal. SelectCourse Review Data, and then enter the Course Study Package Number.

Unpacking the performance indicators

It is fair to say that 1) Employment and 2) Further study are less within the sphere of influence of the course team. Even so, the onus is on the Course Coordinator to ensure that the course is designed to ensure that graduates are fully prepared to achieve employment - connections with industry through Advisory Boards are essential, and fieldwork and work integrated learning to develop graduate employability skills during the course often helps students to forge links with potential employers. Also, students need to be fully aware of the opportunities and benefits of further study.

However, student responses to the AGS are clearly within the course team’s sphere of influence. The three CEQ indicators (Generic Skills Scale, Good Teaching Scale, and Overall Satisfaction Index) are directly related to the quality of the student’s learning experience.

The Generic Skills Scale items are:

  • The course helped me develop my ability as a team member.
  • The course sharpened my analytic skills.
  • The course developed my problem-solving skills.
  • The course improved my skills in written communications.
  • As a result of my course, I feel confident about tackling unfamiliar problems.
  • My course helped me to develop the ability to plan my own work.

The Good Teaching Scale items are:

  • The staff put a lot of time into commenting on my work.
  • The teaching staff normally gave me helpful feedback on how I was going.
  • The teaching staff of this course motivated me to do my best work.
  • My lecturers were extremely good at explaining things.
  • The teaching staff worked hard to make their subjects interesting.
  • The staff made a real effort to understand difficulties I might be having with my work.

The Overall Satisfaction Index is a one item measure which, it is generally agreed, correlates most closely with the Good Teaching and Generic Skills scales and the Clear Goals and Standards CEQ scale.

The remaining LTPF indicators, Progress Rates and Retention Rates are clearly strongly related to these CEQ indicators and are readily amenable to intervention strategies to improve them.

In essence, these indicators focus on what we might call the ‘nuts and bolts’ of good teaching: students indicate highest satisfaction when:

  • beyond acquiring discipline knowledge, they are challenged to achieve higher order thinking, communication and personal management skills
  • teaching staff are passionate about what they teach, and focus on student learning - they are organised, clear, set fair and focussed assessments and give valuable feedback that promoted further learning.

How Course Quality is Monitored at Curtin

Course sustainability

As part of Curtin’s Strategic and Teaching and Learning Enabling Plans, there is a need to ensure sustainability of all courses, both in the planning phase, and through the Course Review Process.Sustainability refers to:

  • Strategic direction - whether the course fits with the strategic direction of the university and community needs.
  • Financial viability - whether the course and the units within it are financially viable, cost-effective and efficiently managed.This depends upon student enrolments, unit and course funding and the costs associated with how the teaching and learning is delivered.
  • Course quality - teaching and learning within courses is informed by current research, and is a high quality offering as determined by student profile, demand, evaluation (CEQ, GDS, eVALUate) and professional recognition and/or accreditation.

Further details on specific criteria to measure course sustainability visit the Curriculum 2010 website via Curtin’s homepage ( and search for C2010.

Course Coordinators play a critical role in monitoring the ongoing sustainability of courses, and implementing strategies to address identified limitations.

Some key principles of course financial viability

All new courses must provide a financial plan with five year projections to demonstrate sustainability at the time of initial application.

Ongoing financial viability is determined by:

 Student demand - all courses have an allocated quota, and there is a need to ensure there is sufficient demand for the course. Insufficient demand may be due to:

  • lack of knowledge about the course
  • insufficient marketing
  • poor quality course
  • strong competition from elsewhere.

 Unit viability - each unit must be costed in terms of funding received (income) offset again the cost of delivering the unit (expenditure). One of the key university strategies to ensure financial viability and to manage staff workloads is to ensure that coursework units operate with a minimum of 10 students (and preferably 20 students). Currently, 70% of the units at Curtin have <20 students enrolled, making unit delivery very costly. If courses run units with fewer than 10 students, they will need to demonstrate that the unit is financially viable.

 Course viability - the overall course viability is determined by a range of factors including Commonwealth Government funding, enrolments, university and divisional levies and method of unit delivery. Unit funding is determined by the assigned Field of Education (FOE) and the corresponding Cluster within which the FOE sits. The funding is determined by DEST and the cost of delivery of a unit, is not always a reflection of the level of funding received.In addition, the type of staff who are involved in the unit (eg. sessional academic vs. permanent, lecturer vs. Associate Professor) will also affect the cost of unit delivery due to their salary level and associated on-costs. Therefore to deliver a total course which is viable, some units may need to be highly profitable to offset the costs of delivering more expensive units.However, all units should strive to be viable in their own right to ensure maximum efficiency.

Course Quality and Review Process

Course reviews at Curtin consist of an Annual Course Review and a Comprehensive Course Review which is completed at least every five years. The key principles in Course Review at Curtin are:

  • Curtin is committed to a planned, systematic review of all courses to ensure that for every award the curriculum is contemporary and challenging, the structure is efficient, and that the student receives a high quality teaching and learning experience.
  • Course quality is the professional responsibility of all staff who contribute to any award course. Each staff member has the right and responsibility to contribute to the review process as appropriate.
  • Course reviews incorporate student, external and other stakeholder feedback, professional and accreditation reviews, acknowledging their importance and relevance, and have three primary foci: course quality (curriculum design with clear, appropriate and developmental learning outcomes; engaging learning experiences; assessment clearly linked to outcomes; continual enhancement and efficient, management); course viability (cost effectiveness and sustainability); and course relevance (the course aligns with Curtin’s strategic priorities and external stakeholder and graduate employment confirms the course is meeting expectations). Course reviews incorporate student, community and other stakeholder feedback, and satisfy professional and accreditation requirements.
  • All courses use the results of monitoring and review to continuously enhance course quality.

Conduct of Course Reviews

Courses may be reviewed individually or in groups. It may be appropriate to review several award courses or majors or cognate disciplines as a group of courses. For example:

  • within the Bachelor of Commerce, finance and banking majors could be reviewed as a group
  • within postgraduate discipline areas a graduate certificate, graduate diploma and masters could be a group
  • double-degrees or significantly overlapping units could also be considered as a group.

Groups of courses are determined by the owning organisation in consultation with the Deputy Vice-Chancellor (Education).

The schedule for Annual and Comprehensive Course Reviews is determined by the Deputy Vice-Chancellor (Education) in consultation with the Pro Vice-Chancellors and owning organisations.

The scope of the course review at each level is:

Annual Course Review - monitors and analyses key indicators and their related measures and targets regarding student profile and demand, and the quality of teaching and learning. The major focus is to analyse course performance and identify initiatives for enhancing the teaching and learning quality, resources and market responsiveness.

Comprehensive Course Review – reviews and analyses the entire academic program for an award; its regulations, structure (its units, major and minor sequences), currency of the curriculum, quality of teaching and learning, management, fieldwork, projects and work experience, and any other aspects which comprise the award course. The major focus is to review the course curriculum map (which shows how the learning outcomes and their associated assessment tasks contribute to the achievement of course learning outcomes); to review the level of engagement in learning experiences in all modes, and to monitor student perceptions of how all aspects of the course support their achievement of the course learning outcomes.

Annual Course Review

Annual Course Review is the annual monitoring of a course or group of courses in regard to the student profile, teaching and learning quality, resources and market responsiveness, to identify and implement initiatives for enhancement. Course review and improvement are integral to Curtin’s quality enhancement processes and forms part of the University’s Planning and Quality Framework. When reviewing courses, all staff involved in the teaching of a course should have the opportunity to be involved and be encouraged to view the process as a self-evaluation of teaching and learning effectiveness. A course review enables staff to reassess the focus of the course, course and unit learning outcomes, the curriculum, teaching methods and pedagogy, and incorporate changes in response to student and external stakeholder feedback. Course viability can also be assessed, and assist in identifying how courses can be taught in order to maximise the resources available.