Forth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
Seville, 12-13May 2011
Exploratory Modelling and Analysis, an approach for model-based foresight under deep uncertainty
1Introduction
Future oriented technology analysis (FTA) is understood as the field interested in analysing future technology and its consequences. FTA relies on a wide array of methods for exploring futures. FTA is an umbrella term for the work done in various future-oriented research fields, including technology forecasting, future studies, foresight, and innovation studies. These various fields have their own methods and techniques. Across these fields, various methods and techniques rely at least in part on mathematical and computer models.
We call decision support through models model-based decision support. The reason for using models might be understood in light of the rise of Newtonian mechanics and its success in predicting a wide array of phenomena. This success gave rise to a mechanistic worldview, according to which the world is like a clock. If the mechanisms of the clock are known, any future state of the clock can be predicted. Similarly, if the mechanisms underlying a phenomenon are known, we can predict how this phenomenon will develop in the future. With the rise of computers, more and more mechanisms can be codified into a model, and more and more phenomena can be predicted.
However, the use of models to make predictions can be seriously misleading if there are profound uncertainties. The system of planets is a relatively small system in terms of components and can be very well observed, and thus its behaviour can be predicted with great accuracy. However, for many other phenomena, such as the world’s climate, or systems in which humans are involved, the situation is different. In these cases, there are many components and mechanisms that interact in a variety of ways, and the system can only partly be observed. The use of predictive models for such systems is problematic. There have been scientists who have realized this. Some claim “the forecast is always wrong” (Ascher, 1978), others say “all models are wrong” (Sterman, 2002), and yet again others speak of “useless arithmetic” (Pilkey and Pilkey-Jarvis, 2007). Such comments raise the question whether models can be used at all in decisionmaking under uncertainty.
In their agenda setting paper on FTA, Porter et al. (2004) note that “there are many irreducible uncertainties inherent in the forces driving toward an unknown future beyond the short term and predictions need not be assumed to constitute necessary precursors to effective action”. There is a need for model-based support for the design of robust strategies across this spectrum of irreducible uncertainties. The RAND Corporation developed a technique called Exploratory Modelling and Analysis (EMA) tailored to this. This paper explores the potential of EMA for FTA. It thus explicitly addresses one of the FTA challenges of Porter et al. (2004). Particular attention is given to the potential of EMA in offering decision support for shaping systemic and structural transformation.
This paper first discusses the wide ranging literature on uncertainty and uncertainty classification, resulting in a typology of levels of uncertainty. This typology is used to specify more clearly the types of uncertainty for which EMA is well suited. It is argued that EMA is particularly well suited for model-based decision support given heterogeneous or even conflicting information. Next, EMA is introduced and the steps of an EMA study are specified. To further elucidate what EMA is and in order to assess the potential of EMA for FTA, three case studies are reported on. Each of these cases is related to a grand societal challenge, is grounded in a systems perspective, and aims to offer decision-support for crafting policies that can shape and drive change. The paper will close with a discussion of the potential of EMA for FTA in light of these cases. Section 2 present an uncertainty typology. Section 3 provides more background on EMA. Section 4 contains three short cases. Section 5 contains the concluding remarks and a discussion on the potential of EMA for FTA.
2Classifying Uncertainty
To assess the extent to which models can be used for model-based decision support, it is necessary to first specify in more detail what is meant with uncertainty. A variety of conceptual schemes, definitions, and typologies of uncertainty have been put forward in different scientific fields. For example, in risk analysis the distinction between aleatory and epistemic uncertainty emerged (Hoffman and Hammonds, 1994, Helton, 1994). Epistemic uncertainty denotes the lack of knowledge or information in any phase or activity of the modelling process. Aleatory uncertainty denotes the inherent variation associated with the physical system or the environment under consideration. Others have tried to clarify where uncertainty manifests itself in the form of a source or location of uncertainty (e.g. Walker et al., 2003, Morgan and Henrion, 1990), and still others have tried to classify the severity of the uncertainty in the form of a level of uncertainty (e.g. Courtney, 2001, van Asselt, 2000, Walker et al., 2003). That is, where does the uncertainty manifest itself along the continuum ranging from deterministic knowledge to total ignorance?
The question of where uncertainty manifests itself along the continuum ranging from deterministic knowledge to total ignorance might very well have the longest history of the different dimensions of uncertainty, dating back to philosophical questions debated among the ancient Greeks about the certainty of knowledge and perhaps even further. Its modern history begins around 1921 when Knight made a distinction between risk and uncertainty (Samson et al., 2009, Rechard, 1999, Knight, 1921). More recent, various alternative conceptualizations have been put forward (e.g. Makridakis et al., 2009, Kwakkel et al., 2010b, van Asselt, 2000, Walker et al., 2003, Funtowicz and Ravetz, 1990, van der Sluijs, 1997, Petersen, 2006, Courtney, 2001). A common theme in these various conceptualizations of the level of uncertainty is the identification of more than two levels. For this paper, we define four levels of uncertainty. The least uncertain is Level 1 uncertainty, or shallow uncertainty. In case of Level 1 uncertainty, probabilities can be used to specify the likelihood or plausibility of the uncertain alternatives. In case of Level 2 uncertainty, or medium uncertainty, alternatives can be enumerated and rank ordered in terms of their likelihood, but how much more likely or less likely cannot be specified. This level of uncertainty is encountered when one is able to enumerate alternatives and is able to say whether they are more likely, equally likely, or less likely, without being able or willing to quantify this further. In case of Level 3 uncertainty, or deep uncertainty, alternatives can be enumerated, but for various reasons, such as that decisionmakers or experts cannot agree or don’t know, even a rank ordering is ruled out. The strongest form of uncertainty is Level 4 uncertainty, or recognized ignorance. Here, alternatives cannot even be enumerated. However, even when alternatives cannot be enumerated, merely keeping open the possibility of being wrong or of being surprised is still possible. Table 1 shows the resulting levels of uncertainty, their description, and some examples.
Table 1: The Four Levels of Uncertainty
Level of Uncertainty / Description / ExamplesLevel 1
(shallow uncertainty) / Being able to enumerate multiple alternatives and being able to provide probabilities (subjective or objective) / Being able to enumerate multiple possible futures or alternative model structures, and specify their probability of occurring
Level 2
(medium uncertainty) / Being able to enumerate multiple alternatives and being able to rank order the alternatives in terms of perceived likelihood. However, how much more likely or unlikely one alternative is compared to another cannot be specified / Being able to enumerate multiple possible futures or alternative model structures, and being able to judge them in terms of perceived likelihood
Level 3
(deep uncertainty) / Being able to enumerate multiple alternatives without being able to rank order the alternatives in terms of how likely or plausible they are judged to be / Being able to enumerate multiple possible futures or specify multiple alternative model structures, without being able to specify their likelihood
Level 4
(recognized ignorance) / Being unable to enumerate multiple alternatives, while admitting the possibility of being surprised / Keeping open the possibility of being wrong or being surprised
Extensive model-based techniques are available for handling Level 1 uncertainty. Exploring the sensitivity of model outcomes to variations in the values of input parameters, is for example well supported by Monte Carlo sampling and related techniques. Handling Level 2 and Level 3 uncertainty in model-based decision support is more limited. Here, analysts need to take recourse to techniques such as scenario’s, Delphi, etc. Level 4 uncertainty is even more problematic. How can we handle unknown unknowns? Adaptivity and flexibility in planning appear to be the main techniques for preparing for this (Collingridge, 1980, Holling, 1978, Albrechts, 2004, Erikson and Weber, 2008, Kwakkel et al., 2010a).
To what extent can models be used across the different levels of uncertainty? For level 1 uncertainty, there is not really a problem. Monte Carlo sampling and related techniques, uncertainty propagation methods etc, offer the analyst ample opportunity to assess the implications of uncertainties about the exact values of model parameters on outcomes. For level 2-4, the situation is different. Uncertainty about the future world can be addressed through for example scenarios. These scenarios can be quantified and serve as input for model-runs. But even these techniques are not free of problems. Goodwin and Wright (2010) (p. 355) argue that “all the extant forecasting methods – including the use of expert judgment, statistical forecasting, Delphi and prediction markets – contain fundamental weaknesses.” And Popper, et al. (2009) state that the traditional methods “all founder on the same shoals: an inability to grapple with the long-term’s multiplicity of plausible futures.” Moreover, if the uncertainty is not only about the future, but is also about the models or aspects of the models, the situation becomes more problematic. In addition, how are models to be used to support the development of flexible adaptive plans that are robust across the uncertainties? This suggests that there is a need for a methodology for using models for deeper levels of uncertainty (Porter et al., 2004).
3Exploratory Modelling and Analysis
Various scientific fields are involved in providing model-based decision support. In these various fields, people are grappling with the treatment of deeper levels of uncertainty while using models. A common theme across these different sciences appears to be a shift away from predictive use of models towards more explorative use of models (Pilkey and Pilkey-Jarvis, 2007, Sarewitz et al., 2000).
Exploratory Modelling and Analysis (EMA) is a research methodology that uses computational experiments to analyse complex and uncertain systems (Bankes, 1993, Agusdinata, 2008). Porter et al. (2004), in their agenda setting paper on FTA, explicitly mention EMA as being of potential interest to FTA. To our knowledge, the potential of EMA for FTA has however not been investigated yet. This paper can be seen as a (belated) attempt to rectify this.
EMA can be contrasted with consolidativemodelling techniques in whichknown facts are consolidated into a single model. Most models are intended to be predictive and use consolidative modelling techniques, in which known facts are consolidated into a single ‘best estimate’ model. The consolidated model is subsequently used to predict system behaviour (Hodges, 1991, Hodges and Dewar, 1992). In such uses, the model is assumed to be an accurate representation of that portion of the real world being analysed. However, the consolidative approach is valid only when there is sufficient knowledge at the appropriate level and of adequate quality available. Under deeper uncertainties, these conditions are not met.
Even if the consolidative modelling approach cannot be used, there is often a wealth of information, knowledge, and data available that can be used to inform decisionmaking. For example, through databases, scientific research, and in the form of mental models. EMA is a research methodology that aims at utilizing the available information, knowledge, and data. EMA specifies multiple models that are consistent with the available information. Instead of building a single model and treating it as a reliable representation of the information, an ensemble of models is created and the implications of these models are explored. A single model run drawn from this set of models is not a prediction. Rather, it provides a computational experiment that reveals how the world would behave if the assumptions any particular model makes about the various uncertainties were correct. By conducting many such computational experiments, one can explore the implications of the various assumptions. EMA aims at offering support for exploring this set of models across the range of plausible parameter values and drawing inferences from this exploration(Agusdinata, 2008, Bankes, 1993) that can be used for decisionmaking, without falling into the pitfall of trying to predict that which is unpredictable.
EMA is not focused narrowly on optimizing a (complex) system to accomplish a particular goal or answer a specific question, but can be used to address ‘beyond what if’ questions, such as “Under what circumstances would this policy do well? Under what circumstances would it likely fail?” It is exceptionally valuable in stimulating ’out of the box’ thinking and supporting the development of adaptive plans.The typical steps in EMA are: (i) conceptualize the problem; (ii) develop an ensemble of fast and simple model of the system of interest; (iii) specify the uncertainties that are to be explored; (iv) specify a variety of policy options; (v) calculate and compare the performance of the various options across the ensemble of future worlds; (vi) iterate through steps (iii) to (v) until a satisfying policy option emerges. EMA is first and foremost an alternative way of using models, knowledge, data, and information. Many well established techniques, such as Monte Carlo sampling, factorial methods, and optimization techniques, can be usefully and successfully employed in the context of EMA.
In this paper, we argue that by using models differently, the challenges associated with decisionmaking under deep uncertainty can be overcome. Instead of trying to predict, the models are used to explore what could happen across various uncertainties. In this way, decisionmaking can proceed despite the presence of deep uncertainty, for decisions can be designed to be robust across the explored range of possible futures. This exploratory way of using models fits with the trend in foresight to move away from prediction and more towards exploring what can happen. Exploratory modelling provides a rigorous methodology for exploring the uncertainty space. Thus addressing one of the often mentioned shortcomings of foresight, namely its impressionistic character (Erikson and Weber, 2008).
4Illustrations of EMA
In this paper, EMA is illustrated using three cases. These cases differ in application domain, the type of models used, and the purpose of the study. In this way, together these cases offer a good overview of what EMA is about, what can be done with it, and what its potential is. Each of the cases is related to important societal challenges. The first case explores uncertainties related to the availability of minerals/metals that are crucial for the sustainable development of all developed and developing societies. The second case shows how EMA can be used to develop adaptive plans for guiding airport development. Airports are a major driver for regional and national economic development. Future uncertainty is increasing because contextual conditions are less stable, new technical solutions are emerging, and evaluation criteria are contested (e.g. noise and emissions versus economic benefits) (Störmer et al., 2009). EMA offers a suitable technique to explore the potential implications of these uncertainties and assists in developing a plan that can adapt over time to how uncertainties unfold. The third case presents an EMA study into transition pathways for the Dutch electricity system. Recent contextual developments constitute a backdrop of change for the Dutch electricity system. Institutional change driven by liberalization, changing economic competitiveness of the dominant fuels, new technologies, and changing end-user preferences regarding electricity supply are some examples of these developments. EMA is used to explore plausible transition trajectories in the face of these developments given technological uncertainty about investment and operating costs, and fuel efficiency of various alternative technologies; political uncertainty about future CO2 abatement policies such as emission trading; and socio-economic uncertainty about fuel prices, investment decisions of suppliers, and load curves.
4.1Mineral scarcity
The first case explores uncertainties related to the availability of minerals/metals that are crucial for the sustainable development of all developed and developing societies. Potential mineral/metal scarcity poses a serious challenge for civil protection in at least three ways: