Report of the Panel appointed to undertake a review of the Agency for Quality Assurance in Higher Education and Research of Andalusia (AGAE) for the purposes of full membership of the European Association for Quality Assurance in Higher Education (ENQA)

Final

Report

16.01.2009

Overview

Introduction

1 The regulations of the European Association for Quality Assurance in Higher Education (ENQA) require all member agencies to undergo an external review at least once every five years. External reviews are expected to include consideration of how far agencies meet the criteria for full membership of ENQA.

These criteria are identical to the European Standards and Guidelines in Quality Assurance (ESG) in the European Higher Education Area, adopted by Ministers responsible for Higher Education in Bergen in 2005. Conformity with the ESG will also in due course constitute the principal criterion for inclusion on a Register of Quality Assurance Agencies, the establishment of which was agreed at the London ministerial summit held in May 2007.

Membership of both the Association and the Register requires a satisfactory external review; one review is expected to be sufficient to cover both purposes. Given that there is currently no national regulation for external review of quality assurance agencies for higher education in Spain, a request was made by AGAE for the European Association for Quality Assurance in Higher Education (ENQA) to be responsible for the external evaluation of the Agency.

ENQA has identified for its own purposes two types of nationally co-ordinated external review. These are:

a) a review the sole purpose of which is to fulfil the periodic external review requirement of ENQA membership; and

b) a review which has a number of purposes, one of which is to fulfil the periodic external review requirement of ENQA membership.

Terms of Reference for the Review

2 AGAE is the main instrument for the oversight of quality assurance in Higher Education institutions (HEIs) in Andalusia, an autonomous region in Spain, which AGAE carries out through assessment activities, process certification and the accreditation of learning and research processes.

In addition to investigating to what extent AGAE meets the criteria for full membership of ENQA, the review also aims to provide feedback on AGAE’s broader role and distinctive programmes in the context of the Andalusian Knowledge System, with particular reference to its teaching staff assessment programmes and its research grants evaluation procedures. This review has therefore been conducted as a Type ‘B’ review with the following purposes:

  • an external assessment to establish the extent to which AGAE complies with the European Standards and Guidelines (ESG) for Quality Assurance in the European Higher Education Area (EHEA), in order for it to fulfil the periodic external review requirement for membership of ENQA, culminating in a report to the ENQA Board
  • an overview of the effectiveness of both AGAE’s teaching staff assessment programmes and research grants evaluation procedures, for the information and guidance of the AGAE and the AGAE Board

Scope of the review

3 In fulfilling these purposes the review team was asked to consider the following core tasks and operations of the AGAE:

i) Evaluation of Institutional Quality

  • Evaluation of quality assurance in higher education institutions. The assessment of departments, centres, degrees and services.
  • Assessment of proposals for the new Postgraduate Programmes (M.Sc. and Ph.D.) of Andalusian public Universities

ii) Assessment of university teaching staff

  • Teaching staff accreditation: assessment of the teaching and research activities of temporary staff for report to the Universities in the granting of confirmation of tenure
  • Evaluations for granting of Emeritus Professor status
  • As a partner in the ‘Docentia’ programme (a collaborative programme between AGAE and the National Agency for Quality Assessment and Accreditation of Spain (ANECA) for the development of a methodology for teaching staff evaluation)
  • Other teaching staff assessment and evaluation exercises, such as the ‘Good teaching performance incentives’ scheme, linked to the granting of additional salary rewards to individual university staff.

iii) Evaluation of Research Programmes, groups and initiatives

  • Research Projects Assessment and Evaluation.
  • Research Activity Evaluation of Groups under the Andalusian Research, Development and Innovation Scheme (PAIDI)
  • Other Research Activities Evaluation, such as that linked to the granting of Research Incentive payments.

Membership of the Panel

4 The members of the Panel were:

Séamus Puirséil, lately Chief Executive of the Higher Education & Training Awards Council for Ireland (Chairman)

Professor Domingo Do Campo, Department of Signal Theory and Communications, University of Vigo, (and formerly Rector of the University of Vigo), Spain

Thierry Malan, lately General Inspector for Administration of National Education and Research, Ministry of Education and Research, France

Rossella Iraci Capuccinello, doctoral postgraduate student, Lancaster University, UK, and lately member of the Executive Committee of the European Students Association.

Paul Mitchell, (Secretary to the Panel), independent HE consultant, UK.

Approach

5 The review was conducted through a transparent process managed independently of the AGAE. The Panel sought to conduct the Review in a professional and courteous way, but in a manner which was both constructively searching and challenging. The review process as a whole proceeded smoothly and responsively. During the site visits, the Panel was met throughout with unfailing courtesy and by a willingness at all levels to engage in discussion and exploration of key issues.

Methodology

6 In fulfilling the purposes of the review the Panel has:

  • considered the broader context within which AGAE operates
  • considered a self-evaluation document prepared by AGAE
  • sought additional documentation relevant to the Panel’s lines of enquiry, both before and during the site visit
  • during a three-day site visit to AGAE’s offices in Cordoba (19-21 November 2008) (Appendix 1), met with a range of stakeholders (determined by the Panel) representative of all AGAE’s operations, including:
  • the Secretary General of the Andalusian Ministry of Innovation, Science and Enterprise
  • AGAE Board members
  • Director of AGAE, senior advisers and all AGAE professional staff
  • Panel members/reviewers with experience of operating AGAE processes at institutional level
  • senior staff and students from HEIs with independent experience of AGAE quality assurance and assessment processes
  • expert adviser on the local labour market
  • meetings with co-ordinators of AGAE’s Scientific Committees and members of AGAE’s Research, Development and Innovation Committee and with staff who have experienced research evaluation
  • meetings with AGAE staff responsible for teaching staff assessment, teaching incentives and other teaching assessment programmes

Self-evaluation document

7 The self-evaluation document submitted by AGAE comprised an account of the following principal areas:

  • Approach to the Review
  • Sources of Information
  • Analysis of Stakeholder Submissions
  • Background and current state of Higher Education and Research and Quality Assurance in Andalusia
  • The Andalusian University System: structure and figures
  • The Andalusian Agency for Quality Assurance in Higher Education and Research (AGAE)
  • AGAE Functions and European Standards and Guidelines
  • ESG Part 2: Assessment against European standards and guidelines for the external quality assurance of higher education
  • ESG Part 3: Assessment against European standards and guidelines for external quality assurance agencies
  • Use of External Quality Assurance Procedures for Higher Education
  • Procedures used by AGAE (amplified in Annexes comprising key procedural documents)

The Panel considered the self-evaluation to be analytical and reflective. In particular it welcomed AGAE’s own analysis, both in the document and in discussion, of its strengths and weaknesses and identification of those areas where enhancements to its operations could be made, with which the Panel concur.

A summary of the supporting documentation made available to the Panel is summarised at Appendix 2.

The Local Context

8 The University system of the Autonomous Region of Andalusia consists of ten public universities. The universities comprise: University of Almeria (UAL), University of Cadiz (UCA), University of Cordoba (UCO), University of Granada (UGR), University of Huelva (UHU), University of Jaen (UJA), University of Malaga (UMA), University of Seville (USE), University of Pablo de Olavide (UPO) and the International University of Andalusia. In the academic year 2007-2008, public universities in Andalusia offered 123 different degree programmes at undergraduate and postgraduate level. The structure of these degree programmes, as in the rest of Spain and in other parts of the EU, is currently undergoing substantial change to conform to the Bologna recommendations for a three-cycle framework in accordance with the European Higher Education Area (EHEA) qualifications framework. These universities had a total enrolment of 223,511 students in the academic year 2007-08, of whom 123,965 were female and 99,546 were male. By nationality, Andalusian universities registered a total of 4,674 international students in 2007-08, mostly located at the Universities of Granada and Malaga. Total full time equivalent teaching staff and researchers numbered 17,087, including 9,126 tenured professors and 7,961 tenured and non-tenured lecturers. In addition, there are 8,480 administrative and support staff.

9 The role played by the Andalusian University system is essential for the economic and social development of the region. Universities in Andalusia are responsible for guarantying quality in education, promoting research and innovation, facilitating technology transfer, and encouraging entrepreneurship. They are also accountable for educating according to democratic values, and promoting international cooperation and cultural development. It must be emphasised that Andalusian universities are ultimately responsible for ensuring the effective convergence of the university system with the European Higher Education Area (EHEA).

10 Formal responsibility for the University System in Andalusia was transferred from the national Government to the autonomous region of Andalusia in 1986. Since then, the Autonomous Government of Andalusia has had responsibility for oversight of the Andalusian University System within the 1983 National Law. The Innovation and Modernisation Scheme for Andalusia (PIMA), put in place in 2005, gave the Regional Ministry of Innovation, Science and Enterprise a duty to achieve a level of excellence in university activities of teaching, research and technology, and knowledge transfer so as to increase competitiveness within Europe. The Andalusian Board of Universities (CAU) was then established to serve as a consulting body to the Regional Ministry of Innovation, Science and Enterprise on University issues. Regional government’s decisions must be approved by the CAU. Equally, society at large is represented at university level through the Social Council (CS) which helps to support the growth of an open and committed university system.

11 A few years after responsibility for the University system was transferred from the National Government to the Autonomous Regional Government of Andalusia, efforts were initiated to develop a quality assurance system for the Region. Such efforts led in 1998 to the establishment of a Consortium for Quality Assurance of the University System in Andalusia (UCUA). UCUA was created as a consortium between the Regional Ministry of Science and Education (later to become the Regional Ministry of Innovation, Science and Enterprise) (CICE) and the Public Universities in Andalusia, with the mission to promote quality improvement in Andalusian Universities through its main instrument, the institutional assessment of quality. The Spanish Universities Act (2001), brought in significant changes to legislation in the regional Autonomous Communities with the introduction of mechanisms to promote academic excellence through a new quality system that emphasized features such as being objective, independent and transparent, and which provided the assurance of merit and capability in teaching staff selection and promotion, and quality improvement in all other areas of university activity. In 2003, a reform of the framework legislation for Spanish Universities led to legislative changes proposed by the Autonomous Regional Government that affected UCUA. The Andalusian University Act ((2003)), which was embedded into the basic framework defined by the national Spanish Universities Act, expanded on the jurisdiction of the government of Andalusia over higher education and research.

12 The new Andalusian Act created AGAE (Agency for Quality Assurance in Higher Education and Research of Andalusia) in 2005 and indicated that UCUA would be integrated in the new AGAE. The 2005 legislation designated AGAE as the main instrument for quality promotion and assurance, to be carried out through assessment, process certification and the accreditation of learning and research processes in the Andalusian Knowledge System and within the framework of the European Higher Education Area (EHEA). The new AGAE incorporated the legacy of the UCUA consortium, but had a different legal status, as an administrative autonomous agency of the Government of Andalusia, with full legal identity and capacity, and its own assets with which it carries out its functions. These were extended to cover institutional quality assessment and teaching staff and research evaluation.

13 The Andalusian Science and Knowledge Law of December 3rd, 2007 will bring further significant changes by incorporating the AGAE into an independent, public corporation under a different name of Andalusian Knowledge Agency (AKA). The new Agency will have two separate parts: one will maintain the same functions as those currently developed by AGAE, while the other will develop activities in the fields of promotion and funding of University activities, and of research projects. The new AKA is to be established sometime in 2009, and its operational principles are also based on independence, objectivity, transparency, efficiency and fairness.

14 The AGAE is headed by a Board of Directors (Consejo Rector) and an Executive Director. Its operations are carried out through three committees with delegated functions in the following areas: Higher Education (HEA); Research, Development and Innovation (RDIA); and Management (MA). There is also an Evaluation, Certification and Accreditation Committee (comprising the Executive Director, individuals from the three specialist committees and a group of external advisors) could be described as a Quality Assurance Committee for AGAE’s own operations. The Board of Directors is the governing body of AGAE. It is made up of the Regional Minister of Innovation, Science and Enterprise (President), the Executive Director of AGAE (Vice-President), five members nominated independently by the Andalusian Board of Universities, three members from AGAE’s organisational structure and the Secretary General. The Board’s main functions are to establish the Agency’s strategic goals and monitor their execution in line with the Andalusian University Act, to approve the programme of annual activities and to exercise financial oversight. The Director is the Agency’s Executive Director and is responsible to the Board for the implementation of its strategies and policies and for the overall performance of the Agency. The Executive Director’s main functions are to supervise the attainment of the Agency’s goals, to represent AGAE externally at the corporate level and to liaise with Government and other stakeholders on the Agency’s activities. The Executive Director is appointed by the Regional Ministry of Innovation, Science and Enterprise from among ‘individuals of recognised prestige’ in University and research circles for a period of four years, on a renewable basis. There are currently fourteen full time professional staff working for AGAE, together with a wide range of contracted part-time expert reviewers and advisers.

15 The legislation envisages assessment, accreditation and certification of quality as the main quality mechanisms for assuring standards (see AGAE Statutes: Decree 1/2005, January 11th art. 6).The Agency has established two committees to exercise delegated powers in making decisions on individual applications for assessment, accreditation or certification. These two committees focus on the following two areas of responsibility:

Higher Education Area (HEA)

Evaluation of institutional quality and assessment of university teaching staff

Research, Development and Innovation Area (RDIA)

Evaluation of research programmes, groups and initiatives.

A common feature of all AGAE’s major processes for quality assessment is the appointment of panels of experts (preferably from outside Andalusia) to advise AGAE within a strategic framework set by the Agency on whether the individual institutions, programmes or proposed research projects meet the relevant criteria for quality assurance. The number of applications has been very large and varied and are summarised in AGAE’s annual reports, found in the evidence pack.

AGAE Compliance with the Standards and Guidelines for Quality Assurance in the European Higher Education Area

(The numbering system applied below to each standard of the ESG is that used in ENQA's 1995 report to ministers in Bergen)

2.4 Part 2: European standards and guidelines for the external quality assurance of higher education

AGAE compliance

16 AGAE’s general approach to the review process is set out in its Strategic Plan 2008-2010 and more explicitly in the criteria set out in its Service Letter issued to institutions, which sets out the procedures and guidelines under which the work is carried out. The reference points used by AGAE are contained in the Service Letter, which itself is compatible with section 1 of the ESG. The procedures and guidelines for the work are thus in accordance with recognized good practice in terms of consistency, independent judgement, openness and clear communication. The institutions' self-evaluations of their own processes are a key element in this review process. Based on individual examples the Panel has explored, AGAE’s processes are effective in testing and challenging institutions' internal quality assurance policies and their procedures for managing quality and standards.

Panel judgement:

Fully compliant