Cascade Project: Evaluation Report

DRAFT v4

December 2010

Henriette Lundgren
Cascade Project Evaluator

+44 (0) 7511 096672 / Marion Manton
Project Manager

+44 (0) 1865 280986
Bridget Lewis / Nicola Warren

Table of contents

Evaluation summary and recommendations 3

Background, aims and structure of this report 5

1 Focus area 1: Online assignment handling 6

1.1 IT support time 7

1.2 Administration handling time 7

1.3 Customer satisfaction 8

1.4 Ease of use 9

1.5 Conclusion: new system improvements on their way 10

2 Focus areas 2 and 3: VLE support and generic content 10

2.1 Overall adoption rate 12

2.2 User friendliness (administrators) 13

2.3 Usage rate (students) 14

2.4 Customer satisfaction (students) 16

2.5 User acceptance (academics) 17

2.6 IT set-up and support time 18

2.7 Conclusion: high acceptance of VLE support for courses 18

3 Focus area 4: Course design 19

3.1 Technology engagement 20

3.2 Conclusion: technology use in course design increases 21

4 Focus area 5: Online enrolment and payment 21

4.1 Administration handling time 22

4.2 Adoption rate 23

4.3 Customer satisfaction 26

4.4 Conclusion: growing number of online enrolments – less handling time 27

5 Overall project evaluation 28

5.1 Project management 28

5.2 Stakeholder engagement 29

5.3 Learning as a result of the project 29

5.4 Tangible benefits 29

5.5 Conclusion: achievements against aims and objectives 30

Appendices 32

Evaluation summary and recommendations

The Cascade project evaluation was conducted in three steps. First, an Evaluation Plan was agreed (April 2010); next baseline data was collected and presented in the Baseline Report (June 2010). Finally, a second quantitative and qualitative data set was gathered (November 2010) to close the evaluation loop.

For the online assignment handling area, evaluation data was drawn partially from the new as well as from the legacy system. Data from pilot studies were collected as the new and improved online assignment handling system will not be fully rolled out until the academic year 2011-12. Our preliminary time-motion analysis of the new system has shown significant time savings in comparison to paper-based assignment handling, but the handling efficiency still lags behind the legacy online assignment handling system where routine use has decreased administration time. In terms of student adoption and satisfaction, the new system has received positive responses during the pilots. Further efficiency and service improvements are anticipated once the new assignment handling system is fully implemented.

VLE support for courses as well as online delivery of generic content have both been well received and adopted by members of the Department. This is especially true in the area of award-bearing courses where the take up of the VLE resources has been good. A significant challenge will be supporting staff and students as numbers increase. However, tools have been put in place to manage this issue, and there is cause to be optimistic about the volume of IT support that will be required. It is anticipated that VLE supported courses, with the integration of generic content, will be an increasingly important tool for administrative staff, tutors, academics and students. In these focus areas, we see full achievements on both aims: efficiency and service. As Departmental staff become more familiar with the VLE and are able to set up their courses more efficiently they will focus more of their attention on other value-adding course activities.

Technology use in course design has increased over the last two years. This trend can be in large part directly attributed to the Cascade project work as the project has supported a general trend of technology engagement and confidence within the Department. All new programmes coming through the Department’s approval process in the last nine months now plan at least basic use of the VLE, and because of that we see the innovation aim of this focus area as having been achieved.

Online enrolment and payment is now almost as popular as paper-based enrolment. This reduces dramatically the administration handling time and error rate when compared to the manual approach. In the case of a standard online enrolment, the time savings in comparison to paper-based enrolment are immense. Where online enrolment requires an extra feedback loop via email with the course administrator, administration handling time savings still exist but are less significant. In terms of student satisfaction with the online enrolment website, a consistently high level of customer satisfaction can be observed. In this area, improved efficiency and service is clearly identifiable.

As a consequence of the evaluation analysis and conclusions, a number of recommendations were established to guide the Department’s efforts beyond the end of the project.

Recommendation on Online Assignment Handling System: The Department should drive the full implementation of the new system further, including the proposed roll-out to all legacy system users and new courses in 2011-12 and to all award-bearing courses subsequently.

Recommendation on VLE Implementations, including Generic Content: The Department should focus on award-bearing courses where VLEs are used over a longer period of time and therefore maximise the ‘shelf-life’ and return on investment.

Recommendation on VLE Improvements: Through the collection of feedback from all key stakeholders – administrative staff, part-time tutors, students and full-time academic staff – course VLEs can be integrated in cycles of continuous improvement. The provision of additional support sessions to all staff when they meet as course teams is recommended to embed the use of the self-service resources and documentation available from the project.

Recommendation on Course Design: The Department should continue to offer support to academic staff, to assist them to plan how they will use technology effectively to deliver the curriculum when they are designing new courses.

Recommendation on Online Enrolment and Payment: The Department will identify all courses where online enrolment and payment could be successfully implemented i.e. for those courses where the process can be fully automated. By doing so, the Department will reduce the number of errors associated with manual processing, improve transaction times, ensure the enrolment and payment service can be offered 24/7, while at the same time generating significant financial value through the reduction in staffing costs associated with automating the process.

Detailed findings in each focus area (Sections 1-4) and for the overall project (Section 5) are presented in this report. Supporting evidence can be found in the Appendices.

Background, aims and structure of this report

The first step in the evaluation of the Cascade project was to devise an Evaluation Plan – submitted to JISC in March 2010 – and to define the measures of success. In order to construct this Evaluation Plan, all available Cascade documents, such as the original JISC call, the project plan and activity plans for the five focus areas of the project were reviewed and synthesised. Next, several informal meetings were conducted with the project team members. Two days were spent identifying aims and key measures of success for the individual focus areas. Finally, an evaluation plan matrix was agreed, which outlined the various evaluation questions, activities, and suggested data-collection methods.

The second step was to take a snapshot of the initial status of the project to establish a baseline. The goal of the resulting Baseline Report – submitted to JISC in June 2010 – was to provide a sound basis on which the success of the project could later be evaluated. The baseline was established following the same structure as the Evaluation Plan; starting with focus area 1 on online assignment handling and concluding with focus area 5 on online enrolment and payment. Where possible, data collection, analysis, and interpretation was separated into three distinct sub-sections. The Baseline Report focused entirely on the description of baseline information and did not include any summative evaluation on the progress of the project.

The third and final step in the evaluation process was to summarise our findings in this Evaluation Report. This report draws from various data sources and meetings and an evaluation workshop that took place on 10 November 2010 with the project team. In this report, we aim to compare the baseline data to the data that was collected throughout the project. As well as reporting on the various focus and evaluation areas, we also aimed to link each area back to the core project aims, i.e. to increase efficiency, to innovate and to improve levels of service:

Figure 1.1: Pyramid of project aims in focus and evaluation areas

(Source: Cascade Evaluation Plan)

The Evaluation Report will conclude with a section on overall project evaluation and lessons learned during the project.

1  Focus area 1: Online assignment handling

Online assignment handling was recognised early on as a high priority for the project, due to the clear benefits it offered the Department both in terms of time and cost savings and in the potential to offer an improved service to students. This was also recognised at the wider University level where there was a parallel pilot of online assignment handling, with which we collaborated to identify areas of common practice. Within the Department, a legacy online assignment submission system (referred to as ‘CASS’) was being used by a few programmes. However this had considerable functionality issues and needed replacing, and as such was not scalable for wider adoption across the full range of the Department's course offerings.

Initially, the plan was to have the new online assignment-handling system available to all courses that wanted to use it for the start of the 2010-11 academic year. However, the launch of the new system was delayed to allow more time for extensive testing and piloting. All those staff involved in the piloting of the system have opted to adopt the system. We are now offering the opportunity for a number of courses to trial the system for the remainder of the current academic year, and we anticipate a very significant adoption rate for the start of the 2011-12 academic year.

Consequently, an evaluation of this focus area will be done using the following structure:

# / Evaluation area / Main stakeholder(s) / Evaluation feasibility check / Section in Evaluation Report
1.1 / IT support time / TALL IT support team / Only a small amount of pilot data available; reference to legacy system / 1.1
1.2 / Administration- handling time / Registry / Only small amount of pilot data available; comparison paper vs CASS vs new assignment-handling system / 1.2
1.3 / Adoption rate / Department / Summative data not available – indicative data discussed in focus area conclusion / 1.5
1.4 / Customer-service response time / Students / Data not available to evaluate / NA
1.5 / Customer satisfaction / Tutors, Registry and students / Only small amount of pilot data available / 1.3
1.6 / Ease of use / Students / Only small amount of pilot data available / 1.4

Table 1.1: Overview of evaluation data for Focus area 1

As the new online assignment-handling system was not fully implemented at the time of the second data collection, only four of the six evaluation areas have been included in this report.

1.1  IT support time

In our Baseline Report, we gave an overview of the IT support time it took to support users of the Department’s legacy online assignment-submission system during the period August 2007 to July 2009. The main data we referred to came from an Excel spreadsheet held in the Department that detailed the amount of time spent providing IT support to students, administrative staff and academic staff to support their use of the old online submission system.

Ideally, we would have liked to collect the same data for the new online assignment-handling system in order to compare IT support times. However, as the new system was not available as a production service during the evaluation period, no directly comparable data could be collected. We therefore decided to continue reporting IT support data on the legacy system instead:

Baseline / Baseline / New data
2007/08 / 2008/09 / 2009/10
Total IT support time (CASS) / 3,050min / 5,005min / 4,875min
Number of programmes supported / 7 / 9 / 8

Table 1.2: IT support time CASS

For more detail on the reported IT support time, please see Appendix 1.1.

As Table 1.2 shows, the total IT support time for the legacy system rose from 3,050min in 2007/08 to 5,005min in 2008/09. This increase in support time was mainly due to an increased number of courses being set up and supported. In comparison, the total support time in 2009/10 decreased again to 4,875min as a consequence of the reduction in the number of courses supported.

IT support time was also tracked for the new online assignment-handling system during pilot studies. Apart from bug reporting by teams such as the Registry etc. there were no more than two calls to the staff providing IT support for the new system. This meant that the vast majority of academics and students were able to use the system without any issues; a major improvement on the previous system which, despite familiarity with the system still requires significant amounts of IT support, as shown above. While this might be taken to be extremely encouraging, the IT Manager cautions on projecting too optimistically considering the relatively small numbers who have used the system thus far. We do know that we have designed out key problem areas in the previous system. However, the IT Manager still suggests, from experience, that there may be unexpected issues that will only become evident as the service is rolled out to more programmes.