Transforming Assessment and Evaluation Working Group

Transforming Assessment and Evaluation Working Group

Action Plan in Brief

Jan. 31, 2018

Working Group Purpose

Affirming the importance of the transformation agenda for SDG #17, this Working Group supports emergence of a powerful assessment and evaluation system fortransformation globally.We are building a global network with those who are working on assessment evaluation of global systems change to energize, stimulate and open up new pathways for learning, acting and reflecting.

The Challenge

Issues with assessment and evaluation approaches with respect to transformation include:

• The program evaluation sector is a professional practice that is relatively young, rapidly evolving, and politically charged. It is becoming a global endeavor practiced by a wide range of people with over 150 national and regional networks in Africa, Asia, Australasia, North and South America, and Europe including aboriginal and indigenous evaluators who bring a wide and diverse perspective.
• Evaluation “grew up in the projects,” with a methodological focus that has been largely quantitative and focused on relatively short-term, time bounded projects or programs, with frameworks designed to illuminate cause-effect relationships. As a result, program evaluation processes are often incremental, slow, and focused on discrete issues and not well matched to the wicked global systems change challenge.
• While progress is being made with assessment and measurement of the 17 Sustainable Development Goals (SDGs), to date, the work is rooted largely in traditional quantitative methods namely tracking progress toward indicators at the scale of the nation-state for the purpose of accountability. The implications of global systems change are too important to allow A&E to remain as an accounting function left only to statisticians and methodologists.
• Assessment and evaluation needs to be itself transformed to reflect the systems and complexity challenges identified throughout the world including inequality, unsustainable consumption patterns, weak governance and corruption.
• While not perfect, the SDG’s provide a suitable platform to both make the case for, and prototype and crystalize transformation of assessment and evaluation at the scale of, global systems change that transcends the nation-state. Without transforming the nature of how we measure progress toward SDG’s, we believe Campbell’s Law[1] will prevail. Applied to SDGs, the law posits: the more any single SDG indicator is used for high stakes decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

The Plan

Transformation of A&E will require engaging multiple perspectives and exploring different definitions of problems/situations/interventions that are being evaluated. We anticipate that this work will lead to identifying alternative moral points of views and principles that guide collective action. The work will require several entry points:

1. Operationalizing the construct of transformation and global systems change evaluation;
2. Conducting systems analysis by identifying and mapping the global network of people interested and engaged in global systems change evaluation;
3. Connecting with leaders within the global network who seek to deepen the understanding of the challenge, co-create action strategies and co-produce proto-type frameworks for assessing and evaluating response to global systems change; and
4. Radical learning by implementing and testing innovative evaluation practices on global systems change initiatives that are transboundary and transcultural, integrate across SDG’s, and have long time horizons.

1)Operationalize the Concept of Transformation and Global Systems Change Evaluation

Although the literature of transformation is growing, it remains an under-empiricized and misunderstood construct. Hence, as we seek to develop a more global perspective to better understand response to global systems change, we will need to operationalize the concept of evaluation at this scale and appropriate units of analysis. For example, transformation is a sensitizing concept that’s only meaningful when applied to a given context. Transformation should be interpreted contextually and dynamically; it requires examples that add meaning to understanding its potential. Thus, transformation should not be subject to narrow measurement or narrow operationalization because it occurs in non-linear and often unpredictable ways. Ironically, we often “know it when we see it”, and therefore may not require massive investment in evaluation. This is known as inter-ocular significance: reality that hits you between the eyes. The challenge in operationalizing the concept is actually engaging with multiple perspectives, multiple kinds of data, qualitative and quantitative, case studies, indicators, and global to local scales in an integrated systemic, synthesize ways to understand the global patterns of transformation and better operationalize the concept.

How we engage in operationalizing the concept of transformation and global systems change evaluation is part of the change process. There are many positive uses that can be derived from engaging in a process to better understand and articulate a concept. Engaging with a global network on this topic can actually change or transform the system itself and the kinds of concepts that we bring to evaluation. A more inclusive approach to collectively define what we mean by this new frontier will influence the overall culture and ideally increase uptake and broaden understanding of the topic. Thus, we propose to conduct a highly dynamic interactive process of meaning-making to support and reinforce interventions, increases ownership and ongoing innovation of the concept.

2)System Analysis

A second entry point is to conduct more detailed analysis and create a map of the people and institutions working on assessment and evaluation of progress toward SDG’s. Increasingly, SDG attainment is becoming more important because they are linked to national priorities and global agreements. SDG’s are not the whole picture by any means, and are actually disconnected to other global goals such as nationally determined contributions to the Paris Agreement for climate goals. Nevertheless, a growing number of people are being asked to assess and evaluate programs that link to both SDG’s and national priorities.

Much of what we're missing from a system perspective is an understandging of who’s doing what? where? and to what extent/impact? While we know there is a global network, we know of no entity that is looking at the larger patterns of change that are unfolding regarding the investment in time and energy to assess and evaluate progress to SDG’s. We propose to apply the field of data visualization and gegraphic informations systems to create a custom map of the system. With such a map in place, we believe we will increase understanding of our profession, better connect with global networks such as EvalSDGs, Earth-Eval, Climate-Eval, Voluntary Organisations of Professional Evaluators and similar groups who will be essential to invite into a network as they are more likely to attend to the SDGs across goals. We hope to identity the SDG monitoring organisations in each country from government, private sector and civil society. Such a task requires the boundaries and data gathering strategies to be constructed properly from the beginning with clear articulation of utility to insure a high quality data visualization. We believe such a map can support collaboration and be used to track whether the intensity of evaluation vs. monitoring increases over time. Such analysis might inspire more effort, for example, if countries see they have no or few “nodes” in a given geography or given thematic focus. We recognize that getting this systems analysis “right” will be a major endeavor and clarity of purpose and use will be paramount.

3)Connecting

Who are you planning on connecting – what’s different that’s not happening now? What types of activities do you plan to support the connections? EG: The WG will lead a process to share its “Manifesto”, a document it has prepared to describe the current situation and actions to address the A&E challenges.

As we operationalize the ideas (Entry Point #1) we will use the systems map (Entry Point #2) to invite people within this global network to engage in a range of activities that include co-creation of action strategies and co-production of frameworks for assessing and evaluating response to global systems change. Transformation requires developing new sets of metrics that integrates current knowledge of how ecology, economics, psychology and sociology collectively contribute to establishing and measuring sustainable wellbeing. The new metrics must garner broad support from those who practice global systems change evaluation. For example, global systems change evaluation will require some form of face validity (it measures what it intended to measure) only if it is recognized as valid and useful by stakeholders. Facilitating engagement of multiple perspectives will be a major goal to test practical application of less than perfect data when deciding on issues of global systems change. If valid evidence is gathered in a timeframe that affects decision-making, how does this compare to the need for more perfect and validated data that is only available after decisions have been taken.

A global network needed to unlock new funding for evaluation includes transforming how we use evaluative thinking to engage diverse communities. Appreciative inquiry and new forms of interactive facilitation can surface new insights and inspire collective will of whole communities to learn how to transform. If linked to the primary values and interests of stakeholders, the transformation of evaluation will empower communities and inspire innovation for non-traditional sources of funding. Finally, connection with a global network will build a strong community to help pilot and develop the needed methods, tools, and approaches to respond to issues associated with climate change, dying oceans, clean drinking water, refugees, poverty, terrorism, human trafficking, virulent infectious diseases, evolving super-viruses, and feeding a growing global populations.

4)Radical Learning and Acting

Evaluating progress to address wicked challenges of interrelated issues of poverty, hunger, well-being, education, and ecosystem health will require radical learning. The relationships are non-linear, dynamic and complex. Transforming the systems of evaluation for SDG’s is part of transformation process. Integration of indicators across SDGs and moving both up and down scale from the nation state are critical steps. By making sense of the interrelationships and interdependencies across SDG’s, actions will be guided that respect these realities.

Learning will require both not just “inside” and “outside the box” thinking, but transformation thinking that can be characterized as questioning “is it a box”. Focusing “inside and outside the box” is crucial for many challenges that are relatively well defined to guide use of existing methods decision-making, data analysis, certain procedures. Addressing wicked problems require more radical thinking that challenges fundamental assumptions (is it a box?) and focuses on dynamics such as ripple effects, unintended consequences, surprises, and critical incidences. Radical learning includes new forms of open-ended data collection, evolution of indicators, not just to measure what was decided at a point in time, but to keep exploring with open-ended field work to document what else is turning up. Radical learning also embraces failure as a pre-condition to innovation – so celebration of failure becomes central to radical learning.

By applying radical learning logic, we seek to develop deeper understanding of the measurement and assessment systems for response to global systems change. This principle calls on working group to collaborate with the other SDG Forum working groups to facilitate radical learning across the groups, provide timely feedback and illuminate sense-making processes to help conceptualize, design, and test new and innovative approaches.

Radical learning requires the field of evaluation as a whole to become more self-critical and accountable to the effect it has on assessment and measurement of global systems change and to what degree it actually supports or inhibits transformation. Resources are needed to identify effective evaluation frameworks and methods (such as radical learning indicators of transformation) that focus on the process of transformation that can be disseminated an applied in context at a trans-national scale.

1

Summary A&E Action Plan
Operationalize Concept & System Analysis / Connecting Across the System / Acting/Learning/Reflecting / Outputs
Year 1 / Defining a common language around global systems change evaluation, including case studies from multiple perspectives, uses of multiple types of data, qualitative and quantitative strategies, issues relating to indicators, global to local scales of analysis, examples of synthesis as a way to understand what the global patterns of transformation / Webinars, podcasts, blogs will grow a global network to build a strong community to help pilot and develop the needed methods, tools, and approaches to respond to issues associated with climate change, dying oceans, clean drinking water, refugees, poverty, terrorism, human trafficking, virulent infectious diseases, evolving super-viruses, and feeding a growing global populations. / Piloting new forms of Global Systems Change evaluation Global systems evaluation as a model of the transformative potential of evaluation and SDGs – linked with areas around the globe that are facing significant challenges in responding to global systems change. / Custom data visualization of the global system of evaluators transiting performance of SDGs
Podcasts (12-18 per year on the topic)
Blogs(12-18 per year on the topic)
Webinars (6-8 per year)
Network analysis
Linkages with other working groups
Year 2 / Developing and testing training modules for how to build core competencies for the practice of global systems change evaluation. Developing strong on-line and open access web presence that features models for operationalizing the practice of global systems change evaluation / Series of meetings, conferences, trainings, and continued development of podcasts, website and blogs that increases connectivity and critical incidences. First annual meeting to be held in conjunction with existing global evaluation meeting that focuses on global systems change evaluation. / Protoyping with innovative governance structures, transformative scenario analysis and decision-makers such as IIASA Arctic Futures Initiative, the Arctic Conservation of Arctic Flora and Fauna and learning across 6 Arctic municipalities that are linked to the mayors of the following communities: Longyearbyen, Svalbard; Oulu, Finland; Tromsø, Norway, Akureyri, Iceland; Nuuk, Greenland; Iqaluit, Canada. / Detailed series of training modules, web-based courses, in person trainings and case study development that forms the basis of a certification program on global systems change evaluation.
Year 3 / Systems mapping that shows the growth and development of a robust network focused on global systems change evaluation. / Wide range of practices that increases connectivity including publishing of detailed handbooks and textbooks that support training modules on how to practice global systems change evaluation. / Learning journeys to areas where transformative evaluation has occurred to grow and share radical learning. Further develop learning journeys as a business model to visit areas where transformation of evaluation is taking place. / Detailed business plan that combines training, certification program, learning journeys so the work can continue will less reliance on external funding.

1

[1]The law states "The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor."