Journal Name: Climatic Change

Online Resource4

Article Title: The unseen uncertainties in climate change: reviewing comprehension of anIPCC scenario graph

Journal Name: Climatic Change

Authors: Rosemarie McMahon (1), Michael Stauffacher (2), Reto Knutti (3),

(1) Institute for Atmospheric and Climate Science, ETH Zurich, Switzerland. Email:

(2) Institute for Environmental Decisions, ETH Zurich, Switzerland. Email:

(3) Institute for Atmospheric and Climate Science, ETH Zurich, Switzerland. Email:

Qualitative Assessment

In this document supplementary interview statements have been outlined. These interview statements (plus those in the main article)are part of the study results and the various subsections to which they refer have been outlined hereunder. The interview statements in this document were not included in the main article because of limited space. All interview statementsare catalogued along with an associated role and/or educational background for the participant as well as an identification number e.g. [Swiss MP, Engineering Degree, 10]

See subsection 3.1.1. Graph Salience

Novice description of the graph / “There is just one black line and then there are many coloured lines, and something grows and distributes I think”[Doctoral in German Literature, 8]
“That we have the scale of time developing and we have a margin of error obviously and we have a certain point in time where evolvement starts to diversity”[Game Designer, 30]
“Main message of this global warming graph is that a prognosis for the future is difficult to make. Each colour has some bias so it is not quite sure and the significance of those results is not clear, absolutely not clear. I am using this graph in my own presentations to show people that this is the IPCC report with lots of international experts and scientist to see what’s going on in the future and that global warming its not sure and we have a lot of bias here and while this is statistically interesting, for me, the one thing I tell people is they are not quite sure and be aware and pay attention of those political guided answers they want to give in every field of the world”[Swiss MP, Engineering Degree, 10]
“So I would say these grey areas would be like thresholds or such and then the lines are the methods. I would have said that this is something more procedural, that it starts with the same thing but then the black line is what happened or something and then what’s expected to happen. With some tolerance again”[Doctoral in Geometry, 15]
“I have a bit of difficulty understanding what the graph is telling me because global, what is that amount of warming? So basically between 1900 and 2000 cooling down. I mean without knowing what this is, it just kind of tells me that the warming is kind of at least increasing although not entirely, completely”[Doctoral in Computer Science, 6]
“Well if you want me to be more confident, I really have to read it and go into it lets put it that way. So at this stage, I don’t understand what they want to tell me”[Professor in Cognitive Neuroscience,16]
“Okay the title (reading) so I think it’s to do with climate and the different models rating the speed of climate change but this is all I can tell without reading in more detail. Someone is doing research in global climate change and they try to simulate models to try to understand all the different options that affect the temperature of the planet”[Doctoral in Physics, 17]
“Multi model average (reading title). Seems like an inconvenient truth and worry. So the message is bad news”[Masters in Political Science, 18]
“Projection for the future, it seems there are different cases from worst case and then probably best case”[Doctoral in Physics, 19]
“Well things are going up and there’s a range of outcomes depending on the assumptions or the data they use but generally speaking these lines are going up so I think that’s the message”[Professor in Political Science, 22]
“This looks pretty familiar to me. The message is it’s very likely that we will have an increase in temperature and it’s very likely that it would go up to six degrees, I mean I talked to other people and they say that even the most pessimistic model is still too optimistic. And we will have by the end of the century we will have more than six degrees of an increase”[Professor in Communication Studies, 27]
“Well lots of variables are developing over time, most of them increasing, but some are decreasing and I’d have to read to figure out what they mean”[Professor in Psychology, 34]
“Developments in respect to some I don’t know, circumstances that will happen or not happen”[NGO, Degree in Geography, 40]
“The message is clear. I know we shouldn’t shoot over two degree. I know we need some measurements to reach this A2, I know this is a lot of renewable energy. Yes, I know there is a lot of uncertainty. The goal is 2 degrees and you have to know that. Without this knowledge you won’t see why this is a problem as some people will say what’s the problem it’s a little bit warmer here. This is the type of arguments you will hear”[Scientific Journalist, 41]
“I would see the word warming and the red. So that would be a negative message”[Swiss MP, Degree in Literature and History, 42]
“Well I can give you an explanation of this graph but I know the background already. If you had to show me this and I wasn’t working in this very job I most probably would have had to read and then try to understand but it would be difficult. There is certainly a message in this graph that related to what economic activity and population growth translated to how emissions develop and then you see this and how active the policy is, you end up actually same story, you end up in a very different range which is significant in terms of global warming. But without that background, I wouldn’t have been able to understand”[Governmental Administrator, 43]
Experts informative description of the graph / “The quick message would be first of all, for most plausible assumptions of human behaviour climate change and warming is going to continue and it depends on two factors, it depends on the choice of our emission scenario, which is not just a climate question but a human behaviour question and these are the different colours. For each of these human behaviours there is an uncertainty related to how well we understand and simulate the climate system, which is the shadings” [Professor in Atmospheric Science, 12]
“Yes here they kind of messed up the message. Because they have this commitment running there which is totally ridiculous, this should not have been there and then constant concentrations, no one understands constant concentration, it’s a concept that is totally yeah. I mean if you ask a lay person how you get constant concentrations then they say ‘switch off emissions’ but that’s not because the concentrations go down so it’s really something ah, kind of, you’d need to put exactly the number of emissions in the air to get it so it’s really an academic thing that comes next to scenarios that give some kind of an indication of the future”[Doctoral in Atmospheric Science, 11]

See subsection 3.1.2. Scientific Literacy

Knowledgeable in scientific concept such as, probabilities / “Likely? For me it would mean something with probability. It will come to pass, likely is a good chance to happen and I would plan in a way that it could happen”[Professor in Agricultural Science, 14]
“Well ok unlikely would be I think mostly used a bit more positive than likely so it would mean the same that it probably won’t happen unless something unexpected happens but it would be more tendency towards might”[Doctoral in Geometry, 15]
“Possible (sharp immediate answer). Ah! ok , so likely, well it’s a very, very general term. (Slight laugh) And the problem is, I work a lot with probability! So it’s nearly impossible for me to basically (pause) the problem is also that I think I know where you want to go with this. So I think you should probably, if you want to have a clean study, you should exclude me from this particular area. Well I mean if you want to, I can…Yes, ok. Ok, so likely I think is between seventy to one hundred percent and unlikely and again you know, this depends completely on (pause) I mean it depends on the frame, it depends of the person who says it, I think about it, it’s extremely rational, but I think in principle, I’d say just as a ballpark, unlikely would be something from zero to twenty percent”[Professor in Cognitive Neuroscience, 16]
“For me it means very probable it’s likely to happen something, very probable happens and eighty percent. Unlikely for me means, if a diplomat says unlikely means no but if a physicist says unlikely there is still room for manoeuvre so unlikely from me is not impossible. Its around ten percent still likely”[Doctoral in Physics, 17]
“I’m not sure about that term Likely. I don’t know (hesitant). Almost certain. 90%.../...Model? I am now a member of the parliament since 2007 and I rarely work with models before but have now started to use models in my work in the health commission. I would describe them the different possibilities how it would go. The model is a theoretical way of solving the problem.”[Swiss MP, Degree in Literature and History, 42]
“The interpretation of a scientist is any statistical likelihood and its does have an interpretation in the sense that it does not have a frequentist interpretation of a repeat experiment like throwing a dice or something where you can basically count the number of occurrences or the number of six that you throw or something like that. Well I mean it is a likelihood in the Bayesian sense that you can describe it as a degree of belief. So it quantifies your confidence whether something is going to happen but you know it is only going to happen once or not at all so its not like you can count the number of times it happens and divide it by the total number of events so it’s about confidence in whether something will happen or not”[Professor in Atmospheric Science, 12]
Good understanding of model concepts / “A model is some kind of hypothesis with some kind of mathematical background that tries to reflect a certain situation e.g. part of environment or certain process in the environment”[Lobbyist, Doctoral in Environmental Science, 5]
“Oh a model is an extraction of the real world or of something that can also be non real that you want to capture but again it is an abstraction so it doesn’t capture every single detail but the essentials and if the model is really good then the complexity is reduced but is still capable of capturing the essence of things”[Professor Mathematics, 7]
“A model? We have had classes on this! It is a representation either mathematical or a small scale tool which synthesises a hypothesis or some theory so that you build up a model”[Professor in Agricultural Science, 14]
“A model. Yeah, well. So a model is a simplified perspective of a how a particular thing works. And very often and it’s best if you can express it mathematically not verbally, because then you don’t leave any ambiguities and it basically allows you to predict what’s going to happen if certain things change in that specific domain”[Professor in Cognitive Neuroscience, 16]
“There are different types of models and it depends on the situation. There are architectural models, mental models and mathematical models. It depends on what you want to represent and how you want it to be represented”[Doctoral in Neuroscience, 20]
“A model is a simplified representation of reality. Trying to capture particular behaviour or particular aspects of that reality we are interested in, and in a formalized way. So we can do the tests on the model that we cannot do on reality”[Doctoral in Atmospheric Science, 11]
“Well there are lots of types of models but in climate a model is a quantitative description of the relevant processes that govern the workings of the climate system and this is first written down as equations and some assumptions and then its transferred into computer code and its run forward in time to study the climate and make predictions” [Professor in Atmospheric Science, 12]
Responses less precise for complex tasks, such as Task 6 (Would you expect the average of several model results to be always better than one single model result?) / “That depends on how good the model is. I mean if the model is good, then yes, one model is better because I mean you are just reduce variants. But if the model is bad, then I mean that doesn’t help”[Doctoral in Computer Science, 6]
“No it depends on the model. Every model has an emphasis of one characteristic or one property or one emphasis would be the right word and then when you average them all you don’t have emphasis on anything”[Doctoral in Geometry, 15]
“No. Clearly not. No (voice raised slightly) clearly not, I mean models are competing. I actually think the competition between models allows you find out, which one is most accurate. So the average of models is almost worse, than a single model”[Professor in Cognitive Neuroscience, 16]
“Well this is not true because if you have three bad models then the averages isn’t good and if you have one good model then you get better results. You would need to know what type of model you have”[Doctoral in Physics, 17]
“No. I don’t see a reason why you should consider the average. Usually you take averages when you have a reason to think that the average is more meaningful than a single value. At least for a single model you understand what you have, with an average you don’t know any more what was the model.”[Doctoral in Physics, 19]
“I think that’s not true. Well let’s say it depends on what you mean by the average. Because you cannot just average models, you can possibly average predictions of different models. If they make predictions for the same variants. Even then I wouldn’t say averaging models will necessarily give you a more accurate picture, because if you average a good model with a bad model, then your average could be worse than the good model”[Professor in Psychology, 34]
“Yes of course. Of course. Okay some models have different parameters so if you take the average it must be better, it must be better because some are extremely bad and some are better and I think the average is the best”[Scientific Journalist, 41]
“No no way, several models is not better than one model, it’s political between models. Basically that is a stupid way of dealing and it’s a way of making a political consensus among scientists, its not intelligent system basically”[Swiss MP, Masters in Engineering, 4]
“If all the models show that the planet is going to heat up because of the rise of carbon dioxide, I think either all make the same mistake or they prove that they are on the right way. If they all make a mistake then one single model could be better than a thousand others. Just because there is a misunderstand in the basis of the models I would say in general several models are better”[Scientific Journalist, 2]
“If many models have a good average, I would prefer many models yes”[Doctoral in German Literature, 8]
Experts clear understanding of scientific concepts with an emphasis on climate science / “The average of several models is most of the times better than one single model. Or most robust because different models ideally have different concepts behind them or have been made by different people so that means that they are supported by more people that have looked at it so it’s maybe slightly different lines of understanding that support that concept”[Doctoral in Atmospheric Science, 11]
“Well (laughing) that has been the focus of my research over many years, you could probably argue that there is something to be gained from having more than one model in the same way that you would learn something from getting the opinion of two doctors rather than one if you are seriously ill. To the degree that these models are independent and provide additional independent information that you haven’t had before then more models would provide you with a better picture of reality but of course to the degree that these models are based on the same assumptions and the same biases and the same things are wrong in these models then it doesn’t help but actually to quantify how much more models increase your confidence is really hard” [Professor in Atmospheric Science, 12]
A somewhat sceptical view of models among some academics / “I mean I don’t know how accurately we can predict the weather in all these parameters in the next one hundred years”[Doctoral in Neuroscience, 20]
“You see these predictions right and the models that make predictions into the future, it would be great to have such things right but if you change the initial conditions completely or the initial assumptions you get something completely different and somehow I have my doubts that models can work that well”[Professor Mathematics, 7]
“No. I would not expect that because a model has as I said, a hypothesis behind it, a certain way of thinking, this is reflected some kind of mathematical approach and when I have different models that means I have different hypothesis and different ways of thinking and then different approaches to reflect this same situation and since I have been in the environmental modelling I know that you can make any model to say anything about the same situation and the total opposite and it depends on so many factors, the system boundaries, the estimate that you enter, the way of how you put it in a mathematical approach and so on so no I would always expect that one model gives you one answer, different models give you a different answer, neither is better there are just different models that have a different advantages and different limitations and so they give different answers that have to be evaluated differently”[Lobbyist, Doctoral in Environmental Science, 5].
“I personally think that people over estimate the impact on the climate because it changes and we don’t know why and in which direction. There was a story that they faked the results in Britain and it was not a global warming it was a global cool down. We see climate change and its clear its happening but I don’t believe it is due to be human intervention it’s probably just the climate on its own and we don’t know what is the reason. People predict weather for five days and this is a prediction for 100 years based on models. I would not trust this model as I don’t trust forecast for five days”[Doctoral in Physics, 17]
“I don’t know. I mean all this prediction stuff is so intrinsically uncertain”[Professor in Artificial Intelligence, 36]
Some novice readers understood the difference between a model and scenario / “Well I would assume that the model takes into account the details of the scenario so they are not exactly the thing but they are related. So different scenarios would give rise to different models or visa versa”[Doctoral in Electrical Engineering, 33]
“Now I should know it or maybe I knew it twenty years ago but for me its kind of a scenario but maybe yes. A scenario would be ok, we have different developments that we can influence we can maybe stop emissions or we can still allow an increase and then we calculate what happens and this would be a scenario. So yes a scenario and a model would be the same thing”[NGO, Masters in Geography, 40]
Difficult for me to say, I’m not an expert in this field but probably, I mean it depends on the definition, a scenario might be different inputs to the same model”[Doctoral in Computer Graphics, 23]
“It’s kind of a different thing, to me at least. But I cannot describe, I don’t know, to me it seems like in a model you try to mimic something, and in a scenario, it sounds a little bit funny, but I think in terms of scenario it seems like a computer game. So the scenario is what the game designers just picked it to be, of course they try model something, but it’s not like a model of something”[Doctoral in Computer Science, 6]
“I am not sure as it really depends on the models that you use but a model is never perfect so I would assume the model is always more uncertain. Mmmm, I don’t know I think that would be interesting to know what’s the difference between a scenario and a model. I think maybe it’s not the same thing but I couldn’t say what is the difference”[Graphic Designer, 21]
Models were considered more uncertain than a scenario / “I think there is greater uncertainties in the model as it is never a perfect representation of reality”[Doctoral in Neuroscience, 20]
“It’s not the same. A model can give you a result you want depending on how you fit it. A scenario is a choice of a set of hypothesis that seems to be likely. The model has more variability whereas the scenario is the reduction of one result of the model so yes the model is more uncertain than the scenario”[Swiss MP, Masters in Engineering, 4]
“The model is more uncertain. I would say the model as you have to develop a model and a scenario must be based on data you have”[Swiss MP, Degree in Literature and History, 42]
“No as I said, you can construct a model in whatever way that you want and two models will be completely different or very close and then also in the same way one model you can feed it with assumptions and different ways of reflecting the hypothesis of the situation which can lead to just as much uncertainty. So the model can be as uncertain as the scenario”[Lobbyist, Doctoral in Environmental Science, 5]

See subsection 3.1.3. Content Knowledge: