OER International Evaluation

OER International Programme /
Evaluation Report /
Prepared for
The Higher Education Academy
by
Dr Rebekah Southern
Higher Education Independent Evaluator

January 2013

OER International Programme

Evaluation Report

Commissioned by:
The Higher Education Academy (HEA)
Innovation Way,
York Science Park,
Heslington,
York,
YO10 5BR
Email:
Registered No 4930131 /
Delivered by:
Southern Horizons (UK) Ltd
c/o Torrington Chambers
58 North Road East
Plymouth
PL4 6AJ
Email:
Company No: 08098961
VAT No: GB 136 9958 59 /

With thanks to the HEA OER International team and to the project leads who gave their time to take part in the evaluation

Executive Summary

  • Open Educational Resources (OER) have been of increasing interest to policy makers and educators over recent years with thousands of resources and courses now available globally over the internet. Following two, year-long, OER Programmes which involved over 90 institutions across the UK, JISC and the Higher Education Academy (HEA) launched a third Programme in July 2012 which included, for the first time, an explicit international dimension.
  • Because the HEA was in a period of transition the call was disseminated following the appointment of an HEA consultant. The consultant was appointed to commission project teams to participate in the programme, proposing case studies that illustrated how OER could be used to promote their educational work or their institution internationally. The Programme started behind schedule which has had implications for delivery.
  • Also of note is the fact that during the commissioning process interested parties were given some discretion to develop projects with an international dimension that aligned with their institution’s specific priorities, whilst building on their previous OER projects. The resulting Programme therefore comprised a highly diverse range of small scale projects, funded up to a maximum of £7,000 each, to be conducted within a three month window from November 2012 to January 2013.
  • An independent external evaluation was commissioned by the HEA from Southern Horizons (UK) Limited and covered seventeen of the participating projects. The purpose of the evaluation was to assess the success of the OER International Programme from the perspectives of the key stakeholders, namely participating projects and HEA management. The evaluation commenced in November 2012 at a workshop for project leads and methods included the administration of a baseline questionnaire at the start of their delivery period, followed by an interview with the project lead (or nominated representative) towards their project completion. Interviews were also held with key HEA representatives.
  • Project leads were initially invited to indicate from a list, within the baseline questionnaire, their main motivations for engaging with the call. Reasons varied considerably across projects but analysis showed the most popular reason was to ‘build on previous or current OER projects.’ Other motivations featuring strongly in the analysis included, ‘enhancing an understanding of OERs for an international market’ and ‘raising the profile of OERs within their institution.’
  • To provide a high level perspective on the overall emphasis of the different projects, project leads were also invited to select from a list their target beneficiaries. They were asked to indicate the importance of each to their OER International project and to elaborate upon the actual impacts expected for those groups, together with their measures of success. Interestingly, the most frequently mentioned target beneficiary (fourteen respondents) was participants’ ‘own institution,’ followed by ‘International tutors’ and ‘the UK HE sector.’
  • The follow up interviews explored projects’ actual achievements in relation to the above. Analysis showed that where projects had aspirations for either prospective or existing international students, most had not been fully achieved during the lifetime of the project but there was some optimism that their aspirations would be realised over time. Feedback was mixed in terms of international tutors. Where projects included this group in general, unspecific terms, they had sometimes ‘fallen off the radar’ but for those projects that gave international tutors a high priority, a number of results were achieved.Analysis further showed that significant progress had been made with project partners, including both UK based and international partners where a number of new relationships had been forged. It also showeda number of significant outcomes for many of the participating institutions themselves including raised awareness and understanding of OER across different internal communities and high level support for further OER development. In many respects, it is too early to assess whether anticipated impacts have been achieved for the sector as a whole as project outputs had not been released at the time of writing this report.
  • Given the importance attached to the institutional impacts,respondents to the baseline questionnaire were asked to rate on a scale of 1 to 5 (where 1 was ‘poor’ and 5 was ‘excellent’), the extent to which their project fitted with their own institution’s Teaching and Learning Strategy. They were also asked to rate on a scale of 1 to 5 (where 1 was ‘poor’ and 5 was ‘excellent’), their understanding of developing OERs for an international audience. Each of these questions was revisited during the follow up interviews. Analysis shows some positive changes in rating to have been made in relation to each, particularly the latter. Moreover, even where the rating had not changed, participants spoke of a range of learning outcomes.
  • Evidence on the actual delivery of the projects, however, was more mixed. Whilst some respondents felt that their project had gone well overall, others had clearly experienced difficulties, many of which had been anticipated in advance. Whilst all projects were aware of the timeframe before they became involved, individual challenges were in many cases compounded by the short duration of the project which included the Christmas break part way through. The evaluation found that several projects had to adapt their methodology as a result of these challenges or make compromises and some interviewees expressed frustration that they had not been able to deliver as much as they would have liked because of the constraints. Some were clearly behind schedule at the point of the evaluation interview and in other cases disappointment was expressed about the quality and/or reach of resulting outcomes. However, only two of the projects delivered substantially less than they stated in their initial plan and this stemmed from having a lower starting point than was originally envisaged.
  • Despite the frustrations expressed, interviewees were also unanimous in their belief that their project had been useful and for some, aspects of it had surpassed expectations. Several interviewees spontaneously mentioned some unexpected outcomes which included the strength of their existing partnerships, the development of new relationships (both internal and external), receiving excellent feedback on their resources and gaining new insights as well practical experience with various platforms and on-line communities.Additionally, some projects had engaged either student interns or recent graduates which proved successful.
  • Interviewees were also invited to pinpoint learning from their own delivery that could be shared more widely. Some common themes emerged including: the importance of good project management; the value of strategic positioning (for both short and longer term objectives); engagement with the university community; methods for gaining wider stakeholder feedback; and the role of external collaboration.
  • Feedback was also garnered through the evaluation on working with the HEA in relation to this call. This was set within the context of significant personnel changes taking place within the organisation during the early stages, in addition to the slightly unusual methods of recruiting projects, both of which caused delays.Respondents were asked to rate their engagement, at the baseline stage, on a scale of 1 to 5 (where 1 was ‘poor’ and 5 was ‘excellent’). Responses ranged between two and five with over half of respondents giving a rating of ‘three.’For those initially giving a lower rating, common issues were identified around the timeframe of the project and the level of engagement required as well as a number of delays. Additionally, there was a degree of confusion about the nature and requirements of the call itself.
  • Notwithstanding the above, a number of positive comments were also made about the HEA project staff and the support received and this was echoed within the follow up interviews. There was, overall positive movement over the lifetime of the project as new staff came on board, with more respondents providing a ‘four’ or ‘five’ rating in the later stages. However, the timeframe of the project remained an issue for many, as well as confusion around reporting templates.
  • Additionally, the evaluation explored participants’ experiences of engaging with the wider Programme community where comments were frequently expressed around what was seen as a missed opportunity to develop relationships with other projects funded through this call. Few of the projects had exchanged ideas with any of the others during their project delivery although most would like to have done so. Time, in many ways, necessitated against this.
  • Overall, evidence from the evaluation indicates that this has been a very interesting Programmein terms of the opportunity to extend the reach of previous OER projects to an international audience. The method of initial engagement with projects was somewhat unusual and offered a greater degree of discretion in terms of the types of activities developed. Although this flexibility created a degree of confusion for some at the outset, it also enabled them to develop projects that fitted with their institutional priorities. Many appreciated the ability to weave an international dimension into their work.
  • Also unusual was the timeframe and the amount of funding made available, where the evidence was more mixed. All projects had achieved something of value, including some unexpected outcomes, with impacts particularly evident for the participating institutions themselves and for the development of wider relationships. However, many projects have delivered less than they would have hoped; a number of compromises have had to be made and; the quality and/or ‘reach’ of the resulting outputs have been questioned. Against this backdrop, a good degree of learning has nevertheless taken place which will be shared for wider benefit. Hence the Programme of activities has clearly been of value.
  • Within the above there are also some potential learning points for the HEA itself. The first is around clarity of purpose in regard to funding opportunities. Given the tensions experienced around timeframe and resources, set against the quality of outcomes, consideration could be given to the management of, and expectations surrounding, this type of approach in future. The second relates to clarity (and timeliness) of communication: ensuring that language, supporting documentation and templates are clear; and that the messaging is consistent. One suggestion put forward during the evaluation was that a peer review process could be introduced for project calls to ensure that they can be easily understood in future.The final point concerns the community of practice and what might be described as a missed opportunity. Interestingly, many of the projects would still appreciate the opportunity to engage with others funded through this Programme and this may be particularly relevant where there are aspirations to further develop or embed the activities that have been generated.

TABLE OF CONTENTS

Executive Summary

Section One: Introduction

Section Two: Evaluation Methods

Section Three: Engagement with the Call

Section Four: Project Beneficiaries

Section Five: Internal Impacts

Section Six: Overview of Project Delivery

Section Seven: Lessons Learned

Section Eight: Engagement with the HEA

Section Nine: Concluding Comments

OER International Programme: Evaluation Report

Section One: Introduction

Open Educational Resources (OER) have been of increasing interest to policy makers and educators over recent years with thousands of resources and courses now available globally over the internet. Within the UK, JISC and the Higher Education Academy (HEA) initially ran two, year-long OER programmes: the UKOER pilot programme in 2009-10, and Phase 2 in 2010-11. This work involved over 90 UK institutions releasing OERs, alongside associated work to provide support and guidance. When the Department of Business, Innovation and Skills (BIS) Online Learning Task Force recommended in 2012 that HEFCE made additional investments around OER, JISC and the HEA were invited to develop a further year-long funded programme. Given the importance of internationalisation to UK universities and the global reach of OER, this additional programme of work included a ‘Promoting UK OER Internationally’ strand, designed specifically to enable universities to devise and implement policies within their institutions that would promote previously created OER to an international audience. It was anticipated that up to 20 projects would be commissioned, building on work undertaken within the previous funding rounds.

The resulting Programme comprised a highly diverse range of small scale projects, funded up to a maximum of £7,000 each to be conducted within a three month window from November 2012 to January 2013. This document summarises key findings from an evaluation of this HEA commissioned OER International Programme. The evaluation was conducted by an external evaluator from Southern Horizons (UK) Limited and coveredseventeen of the eighteen participating projects:

UniversityProject title

Aston University:Global Dimensions in HE

Anglia Ruskin University:Open Resources: Improving Access and Marketing Internationally (ORIgAMI)

Falmouth University:Mining user data to inform international strategy for OER

Newcastle University:Promoting international use and understanding of OER in health sciences

Open University:Evaluating the SCORE Microsites

Plymouth University:International OERs with iTunes U

Sheffield Hallam University:Open Practice in International Teacher Education (OPITE)

Southampton Solent University:Sharing OER Employability Resources on an International Platform

University of the Arts London:ONCE (Open Networks for Culture and Education)

University of Birmingham:Project CoPILOT: Community of Practice for Information Literacy Online Teaching

University of Leeds:Promoting Leeds’ UK OER Internationally: a case study

University of Leicester:iTunesU Reach

University of Lincoln:Preparation for Academic Practice with OER for International Students

University of Liverpool:An Insider’s Guide to Becoming a Business Academic in the UK: Insights for the International Academic Community

University of Nottingham:Pathway2Nottingham

University of Oxford:Oxford OER International

University of Southampton:iTunes and You

Section Two: Evaluation Methods

The purpose of the evaluation was to assess the success of the OER International Programme from the perspectives of the key stakeholders, namely participating projects and HEA management.The evaluation process commenced in November 2012 at a workshop for project leads that was held as part of the UKOER Final Programme meeting. In total 11 projects were represented. The workshop provided an overview of the Programme and an introduction to the evaluation. Participants also engaged in a small group exercise based around aspects of self-evaluation and were introduced to the main evaluation tools, including a self-completion baseline questionnaire. Following the workshop all project leads, including those not present on the day, were invited to complete the baseline questionnaire which was designed to capture feedback about their motivations for participating in the Programme; their intended beneficiaries and measures of success; anticipated challenges to individual project delivery; and, engagement with the HEA in relation to the call.

Responses to the baseline questionnaire provided a snap-shot of the projects at the start of their work. Each project lead (or nominated representative) then took part in a follow up interview, mostly conducted by telephone, during January 2013 towards the completion of their projects. All interviews were informed by results from the baseline questionnaire and focused primarily on understanding their achievements and capturing learning points for wider dissemination. In addition, interviews were held with key members of staff from the HEA during January 2013 in order to understand the process and projects from a commissioning perspective.

An overview of the projects and their individual outcomes is provided in the Final Synthesis Report and individual case studies will also be made available. The present report focuses on insights from the evaluation around projects’ original motivations for engaging with the call; impacts for their intended beneficiaries; internal impacts for the participating institutions themselves; project delivery, including lessons learned; and feedback on projects’ engagement with the HEA.

The evaluation adhered to strict research ethics throughout and all respondents were assured of anonymity. Hence, the report aggregates the evaluation findings and draws out common themes without naming individuals or institutions, except where the information is already available in the public domain.

Section Three: Engagement with the Call

In July 2012 the HEA/JISC grant funding call- ‘UK Open Educational Resources (OER) Phase Three Programme: Promoting UK OER Internationally’ was launched and, because the HEA was in a period of transition, it was disseminated following the appointment of an HEA consultant. The consultant was appointed to commission project teams to participate in the programme, proposing case studies that illustrated how OER could be used to promote their educational work or their institution internationally. The Programme started behind schedule which has had implications for delivery, as will be highlighted more fully in later sections of the report. Also of note in terms of early engagement, is the fact that interested parties were given some discretion to develop projects with an international OER dimension that also aligned with their institution’s specific strategic priorities, whilst building on their previous OER projects. This resulted in a diversity of individual projects being commissioned.