Report to the
Department of Education and Training
APRIL 2015
MID TERM Programme evaluation
Collaborative research networks
Final report
ACIL ALLEN CONSULTING PTY LTD
ABN 68 102 652 148
Level FIFTEEN
127 Creek Street
Brisbane QLD 4000
Australia
T+61 7 3009 8700
F+61 7 3009 8799
Level TWO
33 Ainslie Place
CANBERRA ACT 2600
AUSTRALIA
T+61 2 6103 8200
F+61 2 6103 8233
Level NINE
60 Collins Street
MELBOURNE VIC 3000
AUSTRALIA
T+61 3 8650 6000
F+61 3 9654 6363
Level one
50 Pitt Street
SYDNEY NSW 2000
AUSTRALIA
T+61 2 8272 5100
F+61 2 9247 2455
Suite C2 Centa Building
118 Railway Street
WEST PERTH WA 6005
AUSTRALIA
T+61 8 9449 9600
F+61 8 9322 3955
Acilallen.com.au
suggested citation for this report
ACIL Allen Consulting 2015, Collaborative Research networks MID TERM programme evaluation Report to the Department of Education and Training
© Acil allen consulting 2015
ACIL Allen Consulting
Executive Summary
The Collaborative Research Networks (CRN) programme was announced in the 2009-10 Federal Budget and ends in 2016. To date $81.1 million has been allocated across fifteen CRN projects. The aim of the programme is to increase the capacity of smaller and regional universities to carry out research and provide researcher training. Funding is provided for regional universities to partner with other universities and institutions to build their capabilities in areas of common interest.
In the short term, the programme aims to improve the efficiency of research at smaller and regional universities, as well as lead to more effective collaborations between universities and other research institutions. The longer term benefits include broadening the geographic scope of research activity in Australia to address wider national research and innovation goals, as well as increasing the number of Australian research groups operating at world-class level.
Observations
The CRN programme has helped develop research capacity by enabling asustainable framework for the establishment of collaborative research consortia that should facilitate a more robust research and innovation system. Research capacity has been developed through engaging higher degree by research (HDR) students, often with joint supervision, as well as by encouragingstaff to be more research oriented and building collaborative research programs. This has helped drive an increase in joint grant applications and grants won. Some of the achievements include:
The number of HDR students (Masters and PhD) engaged through the CRN programme is 219 (May 2014), compared to 81 prior to CRN and a target of 155.
The number of journal papers published has increased from a baseline of 20 to 157, exceeding the target of 125 journal publications (May 2014).
The number of joint grant applications submitted by CRN participants is 370 (May 2014), from a baseline of 26. This far exceeds the target of 139.
The number of successful joint grant applications by CRN participants is 39 (May 2014), from a baseline of 1. This exceeds the target of 15 successful grant applications.
The value of grants won is $4.3 million (May 2014), which is below the target of $5.4 million, but is an 85percent increase over the baseline of $2.6 million[1].
All recipients of CRN funding acknowledge that the programme has contributed significantly to the further development of the research capacity in their institution. Evidence collected through this mid-cycle review suggests that the program has been successful in encouraging progress towards achieving this outcome. The CRN programme has created change by driving a more strategic focus on research and creating opportunities to test and develop partnerships. It has helped to generate capacity through a range of institutional reforms (for example, improved research governance structures, skills training, mentoring, focussed research, multidisciplinary approaches to research, strategic planning and clearer career pathways for researchers); and generated secondary benefits such as new linkages, different business models, products (for example in health, agriculture and resource management) and services (for example, research skills training courses).
Our analysis has found thatsignificant pre-planning and the involvement and commitment of senior management throughout the exerciseare key factors fortimely establishment of effective partnerships.Many CRN recipients reported that collaborative consortia take time to establish, although the CRNs which engaged in pre-planning and had buy-in from senior management early on found this to be less of a problem.
The CRN programme reinforces the benefits to be obtained from a flexible approach that allows for a mix of basic research and applied short term research outputs.
The CRN programme has generally been able to demonstrate value for money outcomes. The administrative costs seem to be similar to other government grant programmes. CRN participants did identify other costs associated with the programme – these included typical aspects of ‘on-costs’ such as staffing changes, accounting for in-kind contributions, governance and partnership management costs, travel etc. However, these were not regarded as being outside the norm for programmes such as CRN.
The perspective from the Regional Universities Network (RUN) is that the CRN programme has been transformational for the regional universities involved. It has generated several secondary benefits such as greater internal and external networking and cross-institutional learnings. The CRN programme has also built secondary skills and linkages by encouraging corporate areas to be part of the delivery equation, which has contributed to greater administrative efficiencies in terms of better reporting, coordination and business planning.
The setting and monitoring of Key Performance Indicators (KPI) has produced mixed results. There are some strong commonalities across the KPIs set by different CRN projects. There are also considerable differences in the sets of KPIs. However, ACIL Allen sees this as largely reflective of the flexible nature of the programme.
Some CRN projects have high quality KPIs that are specific, measurable, achievable, relevant and timely, whereas other CRN projects have KPIs that are too general, hard to measure, and unrealistic. Many KPIs lack a time component and some are worded in ways that make them difficult to measure.
It should be possible for a future similar programme to have clear guidelines on how KPIs should be formulated and the types of KPIs that should be used. ACIL Allen envisages that there could be a common (or core) set of KPIs across all projects and also additional KPIs that are tailored to the specific project. This would allow for a greater degree of comparability across all projects, without affecting the flexibility that is at the core of the CRN programme.
Evaluation Methodology
This report evaluates the appropriateness, effectiveness, efficiency, integration, performance assessment and strategic policy alignment of the CRN programme at its midterm point, using the Department of Finance’s Expenditure Review Principles.A series of questions, set out in the body of the report, was developed to assess outcomes against these elements.
A range of sources was utilised to inform this evaluation, including (but not limited to) documents relevant to the CRN programme, including funding agreements and progress reports; case studies; stakeholder consultations (including five lead and five partner institutions and the RUN; asuccessful survey of all CRN lead institutions and their partners;and documents more broadly related to the Australian innovation system.
Stakeholder Outcomes
Overall there is strong support from the stakeholders we consulted for the relevance of the CRN objectives to the Australian innovation system as a whole. The vast majority of survey respondents agreed or strongly agreed that each of the CRN objectives is relevant to the Australian innovation system. A small number were neutral (especially around improving the level of collaboration between different parts of the innovation system, and, in particular, between universities) and only one or two respondents disagreed with the relevance of any objective.
ACIL Allen finds that the CRN programme has been effective at achieving its short term objectives. Lead and partner institutions report an increase in the number of higher degree by research students, the number of research staff as well as the number of (joint) research publications and grant applications. In total at least 100 HDR students, 36 research staff, 460 publications and 365 grant applications can be associated with the CRN programme, though it should be noted that these numbers are not necessarily additional to what would have happened without the CRN programme.
The majority (14 out of 15) of lead institutions agree that the CRN programme has allowed them to do research that they would otherwise not have been able to do. The majority of partner institution respondents (25 out of 42) also agree, though a small number do not. The CRN programme has led to an increase in the number of grant applications submitted by CRN lead institutions. The majority of CRN lead institutions report an increase in the number of competitive grants won. The value of the grants won has also typically increased.
Other benefits identified include improved collaboration with researchers at partner institutions; increased research capability and skills; broadening of research networks beyond the CRN partner institutions;and increased collaboration within a respondent’s own institution.
The CRN programme aimed to deliver benefits for lead universities. These benefits included an increased number of research active and research only staff; an improved ability to attract high quality research students and staff; increased research funding; and improved capacity to manage research. This aligns with the objectives of the CRN programme that were aimed primarily at increasing the research capacity at smaller, less research intensive universities.
The funding model
The expectation was that CRN lead institutions research efforts would, over time, become self-sustaining as they increased their research funding to maintain the work initiated through the CRN projects. It is unclear at this stage whether all the CRN projects have done enough to maintain their current level of research activities.
Participants in the CRN have suggested that the timeframe of the CRN programme is too short to allow the collaborations established through the programme to become self-sustaining. Eight universities have suggested moving onto a 3+2 funding model in which an initial three year funding period is supplemented by a further two years of transitional funding, after which the changes induced under the CRN programme would be expected to become self-sustaining.
Key Findings
Given the timing of this evaluation (with some CRN participants coming to the end of their projects and others with significantly more time to elapse) it is not feasible with the available data to fully evaluate the success of the programme in achieving its outcomes, in particular, sustainability of collaborations and enhanced research capability. However, based on the information we have been able to obtain and the feedback received from CRN participants and others, it is possible to make some suggestions about the programme. In particular, the Department may wish to consider the following if considering the design of any future programme like the CRN:
- Clarify the perception of the need for formal, legally binding partnership agreements and make it clear that less costly (and time consuming to establish) approaches, such as MOUs, letters of exchange and/or project management level agreements, are acceptable (see Section 2.3).
- Review the eligibility criteria for partnerships, e.g. broaden the criteria or clarify the flexible nature of allowable partnerships (see Section 2.3).
- Review the Department’s reporting requirements. In particular, improve guidance and ensure reports include appropriate metrics and consider the frequency of reporting required. For example, it would be possible to reduce compliance load by changing to annual reporting. However the obligation to provide a six month progress report in the establishment year to monitor start-up issues should be retained (see Sections 4.1 and 6.1).
- Clarify the allowable use of funds. In particular, to ensure that institutions are actively encouraged to undertake regular face to face meetings, particularly during the early stages of the project (see Section 6.2).
- Encourage early and effective engagement of senior management in the project establishment phase (Section 6.2).
- Without affecting the flexibility that is at the core of the CRN programme, develop clear guidelines on how KPIs should be formulated. Ideally there should be a set of KPIs that are common across all projects to allow better comparisons between projects and a second set of KPIs that are tailored to the specific project. KPIs should preferably follow the SMART framework (see Section 6.1).
- If a similar programme was to be funded in the future then the design of the funding model should be carefully considered, with the aim of maximising capacity building and making the impact of the programme as sustainable as possible. One option would be to change the funding model to a five year cycle with a minimum three year start-up period – then, subject to a satisfactory report on work done to date, two further years of funding could be provided to assist in the transition to alternative funding streams (see Section 2.3).
- Undertake another review of the programme when all CRN projects have been completed to better assess the sustainability of outcomes. ACIL Allen believes that after July 2017 would be an appropriate time to carry out such a review (see Section 3.1).
Contents
Executive Summary
1Background and context
1.1About the CRN programme
1.2Evaluation methodology
2Appropriateness
2.1CRN programme objectives and key activities
2.2Demonstrated need for the CRN programme
2.3Funding model and governance
2.4Key findings
3Effectiveness
3.1Outcomes of the CRN programme
3.2Alignment of outcomes with CRN objectives
3.3Key findings
4Efficiency
4.1Administrative costs
4.2Benefits of the CRN programme
4.3Key findings
5Integration
5.1Integration
5.2Key findings
6Performance assessment
6.1Performance assessment criteria
6.1.1Types of KPIs
6.1.2Quality of KPIs
6.2Have KPIs been met?
6.3Key findings
7Strategic policy alignment
8Findings
8.1Key Findings
9References
Appendix ASurvey of CRN lead and partner institutions
Appendix BList of consultations
List of boxes
Box 1Expenditure Review Principles
Box 2Advanced Technologies for Leading Teaching and Education Practice
Box 3Growth in research capacity and funding
Box 4Building capacity in quantitative research
Box 5MDBfutures CRN – leveraging CRN funds
Box 6Complementary research funding programmes
List of figures
Figure 1CRN Programme Logic
Figure 2Total research income by university
Figure 3Total research funding for CRN lead institutions (excluding BIITE)
Figure 4Effect of CRN on number and value of competitive grants won
Figure 5Impact of the CRN programme
Figure 6Programme administration costs compared to other government grant programmes; by institution type
Figure 7Integration of the CRN programme with other funding programmes
List of tables
Table 1List of CRN project and funding amounts
Table 2Activities undertaken by CRN projects
Table 3Baseline, target and actual numbers for selected research capacity and capability indicators
Table 4Greatest benefits of CRN - per cent of times ranked in top 3
Table A1Number of responses by CRN and lead/partner status
Table A2The CRN has allowed my institution to engage in research that it otherwise would not have been able to do
Table A3The funding process was appropriate for this type of research funding programme
Table A4My institution's involvement with CRN has led to an increase in the number of competitive grants won
Table A5My institution's involvement with CRN has led to an increase in the value of competitive grants won
Table A6Greatest benefit of CRN - per cent of times ranked in top 3
Table A7What per cent of objectives have you achieved to date?
Table A8What per cent of objectives do you expect to achieve by then end of the CRN programme?
Table A9Greatest barriers to achieving objectives - per cent of times ranked in top 3
Table A10Please rate the impact participation in the CRN programme has had on your institution in the following areas
Table A11Impact of the CRN programme
Table A12Has your institution implemented new frameworks or processes to support collaboration beyond the life of the CRN programme?
Table A13How would you rate the overall impact of the CRN project on the way research is conducted in your institution?
Table A14Compared to other government grant programmes (e.g. ARC and NHMRC grants), how would you rate the administrative costs associated with the CRN in the following areas?
Table A15The CRN programme complements the…
Table A16The CRN programme duplicates the…
Table A17The collaborations formed under the CRN programme are of high quality
Table A18The CRN programme has fostered strong relationships between partnering institutions
Table A19The collaborations formed under the CRN programme are likely to continue after the programme ends
Table A20Please indicate how relevant each of the following CRN programme objectives is to your institution
Table A21Please indicate how relevant each of the following CRN programme objectives is to the Australian innovation system
Table A22The CRN programme has delivered value for money for my institution
Table B1List of consultations
1
1Background and context
1.1About the CRN programme
The Collaborative Research Networks (CRN) programme was announced in the 2009-10 Federal Budget. In May 2011, $61.5 million was allocated to establish twelve CRN projects; in May 2012 a further $19.6 million was allocated to establish three more CRN projects, bringing the total funding to $81.1 million for fifteen projects. The project funding period ends 30 June 2016.
The aim of the programme is to encourage less research intensive smaller and regional higher education institutions to develop their research capacity and adapt to a research system driven more strongly by performance outcomes by teaming up with other institutions in areas of common interest.
The programme was driven by a desire to broaden the geographic scope of research activity in Australia andparticularly to enhance strategic research at smaller and regional universities.
Figure 1 shows the programme logic for the CRN programme, indicating inputs, activities, outputs, outcomes and goals for the programme. The activities funded through the CRN programme include supporting higher degree by research students, early career researchers (ECR) and undertaking collaborative research projects. These activities lead to short term outputs (e.g. increased number of publications research staff, research based graduations, and increased amount of competitive funding won), which in turn feed into the short and long term projected outcomes of the programme.