Response to the CERF Program Evaluation
Overview
CERF
The Commonwealth Environment Research Facilities (CERF) program began in 2005 as a four year $100 million environmental research grants initiative focussed on strong ‘public good’ research. The evaluation focuses on this initial investment, noting that the program has since been made ongoing and renamed the National Environmental Research Program (NERP). CERF was created to provide results accessible to government, industry and the community. It promotes research in areas of special strength or need as identified in the program’s priority research areas.It was established in the format of a grants administration program not as a research program. Towards the end of the program CERF research was refocussed to better meet the policy decision making needs of the Department of Sustainability, Environment, Water, Population and Communities (DSEWPaC), previously the Department of the Environment, Water Heritage and the Arts.
CERF Evaluation
In May 2008, the Department developed an evaluation framework for the CERF program to guide the evaluation of the program. Two reviews were undertaken during the life of the program. A mid term review was undertaken by an independent evaluation company – by Courage Partners. This informed both the final evaluation and design of the National Environmental Research Program. The final and more comprehensive program review commenced in September 2009 and finished in June 2010. This was undertaken by an independent evaluator – Urbis Pty Ltd.
The Evaluation produced four confidential internal reports and one final summary report. To maintain the confidentiality of interview participants, only the final summary report is being released for public review. This document is the department’s response to the issues and recommendations raised in that final summary report.
National Environmental Research Program
The design of CERF’s successor, the National Environmental Research Program (NERP), commenced during the period of this second evaluation. Lessons learnt during the early and mid stages of the CERF Evaluation were incorporated into the initial and the detailed design of NERP.
Australia’s Innovation Agenda
On 12 May 2009, the Australian Government released its innovation policy agenda to 2020 – Powering Ideas: an innovation agenda for the 21st century. Through this strategy the Australian Government has adopted seven National Innovation Priorities. These focus on the production, diffusion and application of new knowledge. They address the country’s long-term weakness in business innovation, and in collaboration between researchers and industry. The National Innovation Priorities of relevance to both CERF and NERP are:
- Priority 1: Public research funding supports high-quality research that addresses national challenges and opens up new opportunities;
- Priority 2: Australia has a strong base of skilled researchers to support the national research effort in both the public and private sectors;
- Priority 5: The innovation system encourages a culture of collaboration within the research sector and between researchers and industry; and
- Priority 7: The public and community sectors work with others in the innovation system to improve policy development and service delivery.
The key objective of NERP is to “Improve our capacity to understand, manage and conserve Australia’s unique biodiversity and ecosystems through the generation of world-class research and its delivery to Australian environmental decision makers and other stakeholders”. It is supporting research that:
- better informs environmental management, policy and decision making, both in the short-term and into the future,
- highly innovative and aims to achieve world-class research and an international standing in the chosen field of research,
- enhances Australia’s environmental research capacity, and
- provides results accessible to government, industry and the community.
The design of NERP will assist the Australian Government meet its innovation agenda Priorities 1, 2, 5 and 7.
Page 1 of 21
CERF Evaluation and DSEWPaC Response
This table is a response to Appendix C of the final report.
Finding / Recommendation / ResponseStrategic alignment, structure and purpose / Refinement of Program goals and desired outcomes
1. /
- Evaluating Program impact (i.e. response to original problem) at the end of the first term was difficult to achieve. Program documentation and reporting demonstrated that:
- The definition of needs to be addressed by the Program was poor.
- There was a lack of clarity about the strategic rationale for commissioned research.
- The Program focused on outputs rather than outcomes.
In developing NERP DSEWPaC undertook an internal assessment of its strategic research priority needs. These priorities were rephrased to research questions and formed a key focus of the NERP application and assessment process.
Knowledge broking and improved communication links between the DSEWPaC and research hubs are a major investment focus of NERP, to ensure the results of research delivered to priority government needs.
The Guidelines and strategies being put in place to guide the program will have a greater focus on its researchers and the department working together to build hub research programs that address priority needs. A monitoring and evaluation framework will be put in place at commencement of the program. This will provide direct feedback on the impact of the program as it progresses.
2. /
- The CERF Program is the Commonwealth’s single most significant opportunity to produce research to serve its Environment Portfolio.
- The Program defined public good environment research as an activity that contributes to the knowledge base for Australian environmental decision-makers and builds the capacity of the research sector. Consequently:
- There was considerable confusion among CERF end users about who the primary end users of the Program were supposed to be. The most commonly mentioned end users included state government, DSEWPaC, land and water managers, natural resource managers, NGOs and policy makers.
- The nominated priority research areas (PRAs) were very broad. This fostered diversity of research and investment. However, failing to be specific about what information would be most useful to Commonwealth decision makers means that some of the research produced will not be particularly useful to the Minister.
- Required contributions to knowledge simply had to relate to the PRAs and grant recipients did not have to articulate what would be produced, or how that information would be applied as part of their proposal.
- Required contributions to capacity were ill defined and consequently open to interpretation by individual researchers and projects. It was often understood as building the capacity of the environmental research sector in the medium-long term through the training of PhD and early career researchers; attracting and supporting world-class researchers to work in Australia; and investing in under-resourced research areas.
- The key end user of CERF research should be understood to be the Commonwealth Government and more specifically the Minister for the Environment and associated Portfolio.
- The priority research areas are more closely defined to enable strategic commission of research to address Government policy and information needs.
- Required contributions to capacity focus on enabling research participants and end users to optimise Program results rather than building the capacity of the environmental research sector and that of environmental decision-makers more broadly.
- Required contributions to knowledge are more focused on providing the Minister and Portfolio with the information needed to make sound decisions about identified priority biodiversity policy, regulation and management needs.
- Information produced is free and freely available.
The strategic priorities were identified through an internal process that addressed the strategic needs of DSEWPaC and the apparent gaps in existing information sourcing. NERP outputs will better meet the Australian Government’s research information needs.
The NERP Guidelines specify the capacity and expertise of researchers that DSEWPaC is seeking, and the requirement for them to contribute to DSEWPaC’s and the portfolio’s decision making capacity.
As with CERF, all NERP research data will be made publically available and free. DSEWPaC is providing additional advice and clear contractual obligations to funding recipients on how to meet this requirement.
3. /
- There was no evidence-based process at Program inception to define the most important knowledge gaps. The consultation process was fairly limited. This was a missed opportunity to engage Portfolio agencies in thinking about its research needs, strategising about the use of evidence to inform policy and building a connection to the Program itself. It also meant that the priority research areas were not particularly specific.
The broad research directions for NERP were identified through department/relevant portfolio workshops with executive staff, researcher consultation, and the Minister’s Office. This lead to the development of the research questions which formed a key focus of the NERP application and assessment process.
A Departmental Advisory Committee (DAC), composed of senior DSEWPaC and portfolio agency staff, was established to advise NERP on portfolio research needs.
4. /
- The criteria for research activities were quite general, and framed as activity or outputs rather than as outcomes.
Research activities / Capacity building
5. /
- In order to produce multi-disciplinary applied research, the Program requires certain kinds of capacity to be strengthened and maintained. These skills, qualifications, organisational cultures and experiences need to be identified and central to Program and project design and activity. E.g. that staff in key Portfolio agencies have capacity to seek out and make use of evidence to inform policy, regulation and decision-making. To date, the specific nature of capacity required to support the success of the Program has not been properly identified or documented.
The NERP has also been designed as a key investment within the new strategic focus of the departmental information reform process. This process builds on the department’s Review of Research and Related Information Management, which has improved use of information, assessment of risk, and adoption of knowledge, as essential strategic improvements.
NERP hubs will employ knowledge brokers who will work with the NERP Team members to build relationships with the department and portfolio agencies. These hub and departmental staff will facilitate information flow to inform decision making.
6. /
- CERF was the first large research and development program that DEWHA (now DSEWPaC) had ever managed. The CERF administrative team were not trained or experienced in the management or delivery of research and development.
- DEWHA (now DSEWPaC) did well to create a Reference Committee of esteemed researchers that could guide the Department in this venture and inform the application assessment process. Consulted stakeholders agree that the selection of projects was well executed and resulted in some high quality work, sometimes in spite of the poor Program guidelines.
- The CERF team did not have sufficient resources to plan for or manage Program legacy; to inform research development to any significant degree; or to facilitate knowledge brokering within the Department and across the Portfolio. Consequently, its major focus has been on contract management. It has however succeeded in conducting an annual CERF Conference.
7. /
- There are no formal systems or processes for promoting the use of research across the Portfolio.
- Nor is there a capacity building strategy for the Program.
DSEWPaC is implementing new knowledge acquisition and management processes. These processes are intended to enhance information sharing across DSEWPaC and provide a strategic and efficient approach to information acquisition and management. The NERP Team has been transferred to a new Information Management Division that will coordinate this process on an ongoing basis.
Capacity in the Portfolio and in the research community will grow as the program is implemented. The NERP Team has expanded its skills base to have greater involvement in liaising with portfolio line areas, and to improve communication between researchers and policy makers. NERP hubs are required to commit greater resources to communications and liaising directly with policy makers.
8. /
- Hubs, fellowships and significant projects usually report their contribution to capacity in terms of activities they have completed rather than outcomes achieved (e.g. ‘conducted training’ rather than ‘improved understanding of…’). Reporting that focuses on activities rather than outcomes does not enable the Program to assess value for money or demonstrate its real contribution to the Australian public.
9. /
- Hubs, fellowships and significant projects are not required to plan for or manage their legacy. They are not instructed to have processes in place to record or share important lessons and findings; to communicate with key stakeholders; or to manage knowledge generated.
10. /
- There are no formal systems in place to facilitate knowledge sharing across the projects and hubs, aside from the annual CERF Conference. There is a significant opportunity to facilitate continuous improvement and in so doing enhance efficiency and effect of the Program through clear and timely information sharing during the research term.