______
Extraordinary Meeting of
Presidents of Regional Associations and
Presidents of Technical Commissions
Mechanism for WMO contributions to the GFCS, with participation by representatives of WMO and co-sponsored programmes
GENEVA, 18 October 2017 / PRA-PTC Ext.-2 2017/INF.4
MEASURING PROGRESS IN WMO MEMBER CLIMATE SERVICES CAPACITY:
FROM 2010-2015
The following summarizes the results of an evaluation of WMO Members’ climate services provision capacity in 2010 in comparison with a subsequent evaluation in 2015. Both classify climate services on a scale from 0-4 (table 1).
Table 1 Climate services categories (summary)
Category / Criteria0 / Does not fully meet the criteria for Category 1
1 / Basic – Design, operation and maintenance of national observing systems; data management including quality assurance; development and maintenance of data archives; climate monitoring; climate diagnostics and climate analysis; climate assessment; dissemination of climate products via a variety of media; and, participation in regional climate outlook forums and some interaction with users.
2 / Essential – Meet the criteria for Category 1; Develop and provide operational monthly and longer climate predictions including seasonal climate outlooks; conduct or participate in regional and national climate outlook forums; interact with users in various sectors to identify their requirements and, provide advice on climate information and products.
3 / Full – Meet the criteria for Category 2; Develop and/or provide tailored and downscaled climate products on timescales ranging from seasonal to climate change in order to meet the needs of major sectors; engage at least with some user communities; provide a strong user interface along with technical expertise for training climate specialists and for developing curricula; and, provide some level of regional cooperation and support.
4 / Advanced – Meet the criteria for Category 3; Provide advanced climate services with research and modelling capabilities for climate and applied climate studies underpinned by a high level of global/regional cooperation and support.
Climate services capacities in 2010
In 2010, following World Climate Conference Three in 2009, and the agreement to establish the Global Framework for Climate Services (GFCS), the WMO Secretariat conducted an initial assessment of numbers of Members in each of the above climate services categories to establish a baseline for GFCS implementation for inclusion in the report of the High Level Taskforce for the GFCS (table 2)[1].
Table 2 Member climate service capacities in 2010
Category / Number of Members0 – Less than Basic Climate Services / 6
1 - Basic Climate Services / 64
2 - Essential Climate Services / 56
3 - Full Climate Services / 39
4 - Advanced Climate Services / 24
Total / 189
According to this analysis, in the absence of a uniformly applicable quantitative indicator directly representing the status of climate services in each of the Member states/territories, the assessment of Member climate services capabilities was largely influenced by the economic and development status, with higher-income and better-developed countries classified as having higher levels of climate services capacities and vice versa (figure 1). Some perceptions of the levels of participation in the relevant WMO programmes and activities also contributed to the assessment, but not in any quantifiable manner.
Figure 1 Member climate services capacities as evaluated by the WMO Secretariat (2010) [Map incorrectly shows six Members with 0 values as missing: Liberia, Somalia, Afghanistan, Bhutan, Lao People’s Democratic Republic, Haiti]
Climate services capacities in 2015
In 2015, WMO conducted its biennial survey of Members on “impacts of achieved results”. This survey includes about 50 questions on various specific capacities specifically related to climate services provision. Based on the responses, the WMO Secretariat updated Members’ climate services capacities classifications (table 3). As only slightly over half of WMO Members responded to the 2015 survey, not all Member classifications could be updated, however.
Table 3 Member climate service capacities in 2015
Category / Number of Members0 - Less than Basic Climate Services / -
1 - Basic Climate Services / 1
2 - Essential Climate Services / 24
3 - Full Climate Services / 61
4 - Advanced Climate Services / 18
Total / 104
According to this analysis, only one Member remains at the basic level. This very low number may be partly due to the fact that many of the Members originally classified as having basic levels of climate services in 2010 did not respond to the survey, with RAs II and V having the highest percentages of Members with missing data for 2015 (62% and 71%, respectively) (figure 2).
Figure 2 Member climate services capacities evaluated based on WMO biennial “impacts of achieved results” survey responses (2015)
Climate services in 2015 as compared to 2010
Both categorizations are available only for 100 Members and territories for which data is available for both years, comprising 52% of WMO Members. Of these:
- For 48%, the climate services categorization increased by one or more categories
- There was an overall upward movement of 52 categories across all Members
- 11% of Members moved down one category
- The RAs with the highest percentages of Members whose classifications increased were RA I (75%), followed by RA IV (62%) and RA III (56%) (Table 4)
Table 4Climate services classificationcomparison 2010-2015
Regional Association / I / II / III / IV / V / VI / AllTotal increase in categories / 27 / 10 / 7 / 7 / 1 / 0 / 52
# Members which moved up / 21 / 6 / 5 / 8 / 2 / 6 / 48
# Members which moved down / 1 / 0 / 0 / 3 / 1 / 6 / 11
# Members with no change / 6 / 7 / 4 / 2 / 3 / 19 / 41
# Members with data for 2010 and 2015 / 28 / 13 / 9 / 13 / 6 / 31 / 100
% of Members which moved up / 0.75 / 0.46 / 0.56 / 0.62 / 0.33 / 0.19 / 0.48
% of Members which moved down / 0.04 / 0.00 / 0.00 / 0.23 / 0.17 / 0.19 / 0.11
% of Members with no change / 0.21 / 0.54 / 0.44 / 0.15 / 0.50 / 0.61 / 0.41
# Members with 2010 data / 53 / 34 / 12 / 22 / 21 / 49 / 191
# Members with missing 2015 data / 25 / 21 / 3 / 9 / 15 / 18 / 91
% of Members with missing 2015 data / 0.47 / 0.62 / 0.25 / 0.41 / 0.71 / 0.37 / 0.48
Average category in 2010 (out of 4) / 1.44 / 1.91 / 2.17 / 2 / 1.94 / 2.8
The majority of advances in category between 2010 and 2015 were from basic to essential (category 1->2) and essential to full (category 2->3) (figure 3). The lowest percentages of Member classification increase were in RA VI (19%), which had the highest average initial 2010 classification (3 out of 4), and RA V (33%), which has the highest percentage of missing data.
Of 70 Members classified as having basic (1) or no (0) services in 2010, only 22 provided data in 2015. Of these, 0 stayed in the same class, 11 advanced one class, 10 advanced two classes, and 1 advanced three classes.
Most of the 11 Members which moved down one class in 2015 were originally categorized as advanced (4) in 2010. Since it is unlikely that the capabilities of advanced Members degraded over the five year period, this phenomenon is more likely an artefact of differences in the data and categorization methodologies,as well as criteria used in the analyses, than an actual trend.
Figure 3 Movement of the category status of Members between 2010 and 2015 as given by the data and methods used for categorization
Discussion
There are two important considerations for interpretation of the above results:
- The criteria used for categorization were similar in each case; the criteria applied in 2015 were adapted from the criteria used in 2010, but
- The data and the methods, used to assess the degree to which the criteria were met, were both different in 2015 than they were in 2010.
We examine these considerations,and their impacts on the above results, in more detail below.
Classification criteria
The criteria summarized in table 1 are elaborated at greater length in a methodological paper underpinning the 2010 categorization that appears in the 2011 GFCS High Level Task Force Report. For the analysis of the 2015 data, these same criteria were slightly consolidated and rearranged so that the outputs of objective 1 of the Results-based framework for WMO contributions to the GFCS could be mapped onto them, but otherwise they are essentially the same in both cases.
Data and methods
The major differences between the two analyses lie in the data and methods:
- 2010 – In the absence of systematically collected data on climate services capacity at the time of the advent of the GFCS, the 2010 categorizationwere necessarily based on composite data, including responses to available surveys related to climate services, United Nations classifications based on economic status, WMO surveys of technological (modelling and forecasting) capabilities, and Member participation in relevant training activities. For the purpose of providing the High Level Taskforce on the GFCS preliminary estimates of the total number of countries in each of the categories, the countries were subjectively assigned representative categories based on these varied sources, applying the professional knowledge of the Secretariat.
- 2015 – Conversely, the 2015 classification is based on survey data reflecting Members’ self-reported capabilities in the broad areas of basic systems and service delivery, as identified in outputs 6-10 under objective 1 in the Results-based framework for WMO contributions to the GFCS.[2] This methodology maps each question from the 2015 WMO biennial survey to these outputs, and weights each response on the basis of its relative contribution to the output. The method then groups the outputs under two headings: basic systems (output 6) and service delivery (outputs 7-10) and weights each output grouping equally, to arrive at an overall index. The final categorization for each Member is assigned based on boundaries defined for each of the four categories.
Although the above scheme entails subjective judgements, the associations among the Results-based framework outputs, the survey question responses, and the criteria for categorization are relatively clear. A critical ambiguity arises, however, from the fact that the 1-5 scale involved in many of the survey questions is arbitrary (“On a scale of 1-5, how would you rate the quality of . . .”). The methodology for the 2015 analysis takes the survey responses approximately at face value, that is, a survey response of “1” is taken to mean basic level and so on. Yet, for categorization purposes, the assessments of whether a given component of an operational system or link in the service delivery value chain is at a basic, essential, full, or advanced level are intended to be based on unambiguous, verifiable criteria such as are summarized in table 1.
Thegeneral increase in Member climate services capabilities categories at the lower end of the scale in 2015 as compared with in 2010 documented in table 4 mayarise from the above discrepancy. Similarly, the reduction of some Members’ categories in table 4 and figure 3, from the advanced to the essential level, is also more likely due to the above rather than to an actual decrease in Member capacity. Indeed, both the low-end increase and the high-end reduction are consistent with a tendency to avoid the highest and lowest values when responding to survey questions on an undefined, 1-5 scale. Although it would be possible to re-scale the responses, to bring more Members back to the basic level and boost previously advanced Members back to the advanced level, any attempt to do so would also be arbitrary in the absence of an unambiguous assessment of the degree to which each Member’s capabilities align with the criteria for the four climate services categories.
Conclusions and recommendations
The 2015 survey responses, summarized in the report of the 18-19 May 2017 meeting of the mechanism for WMO contributions to the GFCS and related materials, provides a great deal of useful information for assessing the state of climate services across WMO Members broadly. It reflects Members’ perceptions of what they are doing well and less well, and provides a basis for broad comparisons of climate services capacities among WMO regions. For the reasons explained above, it does not appear to provide a sufficient basis for revising the original 2010 Member climate services categorization, however.
The requirement for updating Member climate services capacities by category can be addressed by combining the next WMO biennial survey, in 2017, with the exercise currently underway by Members – to complete the climate services checklist that was developed as requested following the May meeting. The checklist, which is also based on the criteria underpinning the 2010 and 2015 analyses, unambiguously stratifies constituent elements of climate services provision into basic, essential, full, and advanced categories. Questions included in the forthcoming 2017 survey, significantly reduced in number from those in the 2015 survey, map onto the checklist and include guidance on the meanings of any 1-5 scale responses. The 2017 survey, which will be administered through the WMO country profile database, will reflect data pre-imported into it from completed checklists, where available. When checklists for all Members have been completed, and are being continuously updated, the survey will provide a biennial basis for surveying the status across the WMO Membership.
More generally there is also clearly a need for data from more Members. This is particularly the case in RAs II and V, and for Members initially classified as having the least advanced capacity for providing climate services. RCOFs, and regional association and their climate working group meetings, NCOFs, and other similar events, will provide opportunities for completion of additional checklists. Importing or otherwise transferring data from the checklists to the surveys will assist with regularly updating the status of climate services across WMOMembers.
The desirability of combining the survey responses and the checklist data underscores the need to identify focal points for climate services in each country, who can track responses to surveys and checklists over time and who can be contacted to verify key information. These focal points would broadly be responsible for tracking outputs under objective 1 of the results based framework for WMO contributions to the GFCS. In that capacity they would also provide national points of contact for regional association working groups and task teams on climate and the GFCS, as well as for technical commissions and WMO and co-sponsored programmes. TORs for these focal points are under development by the Commission for Climatology.
[1]WMO, 2011. Climate Knowledge for Action, Report of the High Level Taskforce for the GFCS, WMO No. 1065, 248pp.
[2]6) Adequate observing networks, data, data management, monitoring, and forecasting systems
7) Decision support tools and systems (identified, designed and improved, including any necessary research)
8) Capacity development services
9) Decision-support products and services (established or strengthened)
10) Monitoring systems