Audit and Evaluation Branch Evaluation of the MSC Transition
Evaluation of the
Meteorological Service of Canada (MSC) Transition Project
July 2008
Environment Canada
Audit and Evaluation Branch Evaluation of the MSC Transition Project
Report Clearance Steps
Planning phase completed / September 2007Report sent for management response / April 2008
Management response received / June 2008
Report completed / June 2008
Report approved by Departmental Evaluation Committee (DEC) / July 2008
Acronyms used in the report
AES / Atmospheric Environment ServicesADM
AMDAR / Assistant Deputy Minister
Aircraft Meteorological Data Relay
CIOB
DMF
DMS / Chief Information Officer Branch
Data Management Framework
Departmental Management Services
DSO / Departmental Service Organization
EC / Environment Canada
EQ / Evaluation Question
IT
MSC / Information Technology
Meteorological Service of Canada
M
NL / Million
National (Research) Laboratory
NSO / National Service Office
NSU / National Service Unit
OPG
PAA
QMS
RWIS / Outcome Project Group
Program Activity Structure
Quality Management System
Road Weather Information System
S&T
SPC / Science and Technology
Storm Prediction Centre
TBS / Treasury Board Secretariat
WEP / Weather and Environmental Predictions
WES / Weather and Environmental Services
Acknowledgments
The Evaluation Project Team including Robert Tkaczyk, Lindsay Fitzpatrick, Karine Kisilenko and Martine Perrault and led by Janet King and Shelley Tice under the direction of the Director, Shelley Borys, would like to thank those individuals who contributed to this project and particularly all interviewees who provided insights, and comments crucial to this evaluation.
Prepared by the Evaluation Division, Audit and Evaluation Branch
Table of Contents
EXECUTIVE SUMMARY i
1.0 INTRODUCTION 1
2.0 BACKGROUND AND CONTEXT 2
2.1 Profile 2
2.2 Program Logic Model 5
2.3 Roles and Responsibilities 7
2.4 Resources 7
3.0 EVALUATION DESIGN 9
3.1 Purpose and Scope 9
3.2 Evaluation Methodology and Approach 9
4.0 FINDINGS 14
4.1 Relevance 14
4.2 Success 16
4.3 Cost-Effectiveness/Alternatives 28
4.4 Design and Delivery 31
5.0 CONCLUSIONS 38
6.0 RECOMMENDATIONS 40
Annex A Planned Resource Allocations by Transition Component 49
Annex B Evaluation Issues and Questions 50
Annex C Relationship between Expenditure Review Questions and the MSC
Transition Project Evaluation Questions 53
Annex D Key Documents 54
Annex E Master Interview Guide 58
Annex F Summary of Findings 63
Annex G Budgeted and Actual Expenditures versus Treasury Board Funding for the
First Four Years of the Transition Project (2003–2004 to 2006–2007) 64
Environment Canada
Audit and Evaluation Branch Evaluation of the MSC Transition Project
EXECUTIVE SUMMARY
In 2003, the Government of Canada announced increased funding to revitalize and transform the operations of the Meteorological Service of Canada (MSC). The objectives of this project, known as the Transition Project, were to help the MSC become a more sustainable organization, to advance science- and weather-related services, and to help Canadians adapt in ways that safeguard security, health and safety, economic prosperity and environmental quality. The government allocated approximately $74million (M) in new funds over a five-year period and $5M per year thereafter. This funding was to be combined with $11M from internal allocations over the same five-year period and approximately $2M annually thereafter.
An evaluation of the progress made to modernize Environment Canada’s weather services over the 2003–2004 to 2007–2008 period was required by Treasury Board. The importance of evaluating this significant and transformative measure was also endorsed by Environment Canada’s Audit and Evaluation Committee[1] and the conduct of an evaluation was included in the department’s 2007–2008 Audit and Evaluation Plan.
The purpose of the evaluation is to assess
· relevance: whether the Transition Project goals were consistent with federal priorities and addressed actual needs;
· success: the extent to which the Transition Project has met its intended outcomes;
· cost-effectiveness / alternatives: whether the most appropriate and cost-effective means were used to achieve those outcomes; and
· design and delivery: whether the Transition Project was delivered in the best possible way.
The evaluation addresses activities funded under the Transition Project over the fiveyear project period from fiscal year 2003–2004 through 2007–2008. Multiple methodologies were used to collect information. These methods included
· document and file reviews;
· secondary data analyses and reports;
· a media scan;
· key informant interviews (58 people in total: 36 internal staff and 22 external stakeholders); and
· three facilitated workshops with Environment Canada employees (47 people in total).
The major conclusions of the evaluation are presented by evaluation issue.
Relevance
· The objectives and intended outcomes of the MSC Transition Project a) were aligned with Environment Canada’s mandate for meteorological services and departmental strategic outcomes as set out in corporate documents; b) were related to priorities identified in the Government of Canada’s speeches from the Throne and budgets; and c) addressed actual needs to modernize Environment Canada’s weather services.
Success
· The Transition Project has made progress on most of the activities undertaken. These activities have resulted in the consolidation of weather offices, a focus on highimpact weather, increased training and recruitment of staff, the introduction of new products and services and the lifecycle management and modernization of some equipment. Activities that have not been completely implemented and require further attention include the decommissioning and cleanup of some climate and hydrometric sites and the development and implementation of quality assurance processes and data management frameworks.
· Despite the achievement of these activities, the intended outcomes at the immediate and ultimate outcome levels will only be partially achieved by project end (2007–2008). For example, while there is evidence of ongoing recruitment in the Meteorology Group and Engineering and Support Staff categories, there is a widespread perception, both internal and external, that current succession planning efforts are inadequate to address the impact of an aging workforce. Notwithstanding the progress made to modernize monitoring networks through network rationalization, considerable work remains to be done. Steps have been taken to enhance the quality assurance of and access to key data, but the data management framework is still under development and quality assurance processes are considered to be under-resourced. Further, even though progress has been made in decommissioning sites and stations and installing new equipment at reference climate stations, the decommissioning of sites has not occurred at the rate expected. These partial achievements weaken the ability of the Transition Project to attain its overarching outcome, which is to make MSC a more sustainable organization.
· The progress achieved under the Transition Project was affected by factors internal to the project itself:
1) The costs of some activities (e.g. consolidation and modernization of forecast offices) were underestimated and resources had to be redirected from other activities (e.g. from restoring and developing key skill sets).
2) Staffing, training and consequently relocation processes took longer to get into place than expected (e.g. National Service Offices).
· The ability to implement activities and achieve the intended outcomes was also affected by factors external to the project:
1) The ability to achieve outcomes was affected by department-wide changes to governance and resource allocation processes. Budget decisions became the responsibility of boards, not line-based organizations, and resources were centralized and allocated on a priority basis.
2) Government decisions were not matched with funds (e.g. the cost of the offices in Gander, Newfoundland and Winnipeg, Manitoba), meaning resources had to be redirected from Environment Canada’s A-Base programs.
3) The implementation of activities was delayed by policy requirements (e.g.land claims negotiations) and new procedures for the management of property.
Cost-effectiveness / alternatives
· Environment Canada interviewees widely agreed that the Transition Project used the most appropriate and effective means of achieving the intended outcomes and that there were no obvious alternatives. Although there is compelling qualitative evidence that the project provided value for the money received, a complete assessment could not be done due to a lack of financial information.
Design and delivery
· While the concept and design of the project was considered to be the best way of achieving the intended outcomes, the project was not delivered entirely as planned. Some activities were dropped and others have not been completed to date, notably in the monitoring component of the Transition Project.
· As previously indicated, MSC underestimated some of the funding requirements and resources were reallocated within the project.
· Pockets of dissatisfaction with the project exist among Environment Canada interviewees and some stakeholders. Environment Canada employees are concerned about the potential to achieve the outcomes sought and some stakeholders express concern over the delivery and quality of services and the adequacy of Environment Canada’s efforts to address the impact of an aging workforce.
· As a result of the inability to access all financial information, the evaluation was not able to confirm whether or not the project had remained within budget.
Recommendations
The following recommendations are based on the findings and conclusions of the evaluation. The first three recommendations are directed to the Weather and Environmental Services (WES) Board and the fourth recommendation is addressed both to the WES Board and the Departmental Management Services (DMS) Board.
1. Take stock and communicate which activities need to be continued in support of the modernization and sustainability of Environment Canada’s weather services.
Given the ongoing importance of modernizing Environment Canada’s weather services, attention is needed to maintain the progress and momentum achieved to date. Priority attention, in particular, is required to ensure the sustainability of Environment Canada’s weather services. In deciding which of the activities initiated under the Transition Project need to continue, the WES Board needs to
a) Identify the status of the activities and how the activities relate to the ongoing modernization and sustainability of MSC
The overarching objective of the Transition Project was to help MSC become a more sustainable organization and infrastructure. The achievement of the other ultimate outcomes — strengthened linkages between production, science and service; improved services for Canadians and key stakeholders; and adaptation of Canadians to the environment in ways that safeguard security, health, the economy and quality of the environment — depend on the existence of a sustainable organization and infrastructure. Thus, in considering which activities should be continued, the WES Board needs to assess how these activities relate to the sustainability sought under the project and, further, how these activities contribute to the overall sustainability of Environment Canada’s weather services funded by existing A-Base resources.
In this context, the WES Board also needs to consider the risk to the mandates and priorities of internal stakeholders (e.g. the Environmental Protection Board) and external stakeholders, including other federal departments and agencies, other levels of government, the private sector and Canadians at large of not continuing the activities launched under the Transition Project” to align with what was sent for response.
b) Determine the resource requirements of ongoing activities
The modernization efforts of the Transition Project were to be complemented by other A-Base programs; hence resource allocation should be considered within the broader context of A-Base funding for weather services. Seven million dollars annually has been allocated for the implementation of activities initiated under the Transition Project. In considering the budget allocations, the WES Board needs to consider how to best use and leverage the existing resources (financial and human) provided under the project with MSC’s other A-Base programs.
c) Determine the adequacy of current succession planning efforts
External stakeholders in particular voiced concern over the impact of an aging workforce on the capacity of Environment Canada to deliver high-quality weather services. The WES Board needs to assess the capacity of Environment Canada and the adequacy of current succession planning efforts to support sustainable and high-quality weather services to stakeholders.
d) Articulate a strategic direction and implement an action plan
Based on the information collected above, the WES Board needs to decide which activities will continue to be funded, how the resources will be allocated and linked to existing A-Base programs, and what needs to be done to ensure the capacity needed to deliver quality weather services. The strategic direction should be accompanied by an action plan which would set out specific measurable deliverables, timelines and accountabilities.
2. Communicate the strategic direction and action plan
The strategic direction and the action plan articulated by the WES Board need to be communicated to
a) MSC staff and other internal stakeholders
The vision for the Transition Project was clear and, overall, MSC staff were motivated to achieve the intended outcomes. Over time, Environment Canada interviewees[2] became discouraged by their perceived inability to achieve the intended outcomes. Modernization of Environment Canada’s weather services is an ongoing requirement of the department, and employees need to be clear about the direction of programming. The strategic direction and the action plan formulated by the WES Board need to be communicated to the MSC, to the Science and Technology (S&T) Branch and to the staff of the Chief Information Officer (CIOB) Branch.
b) External stakeholders outside of Environment Canada
Many external stakeholders are of the view that the 2004–2005 changes in the organizational structure of MSC and the turnover in personnel have moved Environment Canada away from being a sustainable organization. External stakeholders are concerned that Environment Canada does not have the capacity or succession planning efforts in place needed to create a stable organization. The changes in governance at Environment Canada, along with MSC’s efforts to address succession planning issues, need to be addressed with external stakeholders to ensure their continued support and the continued credibility of Environment Canada’s weather services.
3. Strengthen performance measurement, monitoring and reporting at the outcome level
While information on the progress of the project was collected at the activity level, information on how these activities contributed to the achievement of immediate and ultimate outcomes was not consistently collected and aggregated. Despite the existence of a performance measurement strategy, that strategy was not implemented and information at the outcome level was not readily available. The absence of performance information weakened the ability of the department to tell its full performance story as to how weather services have been modernized over the past five years and to identify ongoing requirements for programming and resources. To demonstrate good stewardship and accountability for the use of public funds and informed decision making, the WES Board needs to strengthen performance measurement, monitoring and reporting at the outcome level.