Forecasting using Structured Analogies

A Self-Certification Course

Last modified on 4 April, 2007

Designed by
Kesten C. Green and J. Scott Armstrong / Reviewed by
Paul Goodwin

The structured-analogies method is likely to be useful for forecasting whenever experts know about similar situations from the past or when databases of situations that are more-or-less analogous to the target are available. Structuredanalogies wasdeveloped to forecast the decisions that people will make in conflict situations such as buyer-seller negotiations, employer-union disputes, commercial competition, hostile takeover bids, civil unrest, international trade negotiations, counter-terrorism, and warfare.

Decisions in conflict situations are difficult to forecast. For example, when experts use their unaided judgment to make predictions about such situations, their forecasts are no better than guessing. The structured-analogies method makes better use of experts by eliciting in a formal way i) their knowledge about situations that were similar to the target situation, and ii) their judgments of the similarity of these situations to the target. An administrator then analyzes the information the experts provide to derive forecasts. Research to date suggests that these forecasts are likely to be more accurate than unaided-judgmental forecasts.

The materials in this course mostly relate to the problem of conflict forecasting. For other applications, such as predicting software costs or demand forecasting, the tasks of formally describing the target situation and identifying and rating analogies will be more straightforward because the structures of these situations are likely to be relatively homogenous.

The Course includes the following materials:

• Course Checklist

• Course Guide

• Pre-Course Test

• Method Guide

• Certification

• Examination

• Examination Answers

• Certificate of Completion

Course Checklist

Self-learning courses have been found to be more effective than traditional courses taught in classrooms. For effective learning, you must follow the steps described in the Course. You can do so when and where you like.

This is also a self-certification course. In order to obtain your Certificate, you will be asked the six questions in the Checklist shown below. Your answers to the questions are your testimony as to your diligence, effort, and success in fulfilling the Course’s self-learning requirements. In other words, you will be asked to sign that you have followed the Course as designed or, if not, to indicate how you modified the Course.

Please keep track of your progress using this Course Checklist:

1. / I completed all tasks prescribed in the Course.
2. / I kept a diary of my learning in which I described what I did and what I learned from each task.
3. / I completed the Pre-Course Test and the Certification Examination before looking at the Model Examination Answers.
4. / I fully understand the answers that I gave and I could explain them to another person.
5. / The time I spent on the tasks described in the Course Guide was… / minutes
6. / The number of times I applied the principles covered in this Course to a real problem was…
One of my applications was…
{Approximately 200 words) / times

Note that the Certification Examination is only one aspect of the Course. Completing the Examination will not in itself develop your skills in the use of structured analogies The Certificate of Completion (on the last page of this document) allows you to provide a record of your efforts in completing the Course.

Course Guide

Forecasting Using Structured Analogies

The numbers on the right of this Guide indicate the recommended minimum times to allow for the tasks and for diary entries. As you work through the Course, keep a diary of what you did, what you learned, and how you intend to apply what you learned.

As you can see, we recommend at least seventeen hours to become certified in this area. Record the actual time you spent on each task as you complete it.

Minimum / Actual
(minutes)
1. / Answer the questions in the Pre-Course Test and print a copy. / 20
2. / Familiarize yourself with conflictforecasting.com. / 60
3. / Study the following and, as you do, make plans for small applications.
Green, K. C. & Armstrong, J. S. (2007). Structured analogies for forecasting. International Journal of Forecasting. [Forthcoming with commentary]
For additional information, see:
Green, K. C. & Armstrong, J. S. (2005). The war in Iraq: Should we have expected better forecasts? Foresight, 2, 50-52.
Green, K. C. & Armstrong, J. S. (2007). The value of expertise for forecasting decisions in conflicts. Interfaces. [Forthcoming with commentaries]
Tetlock, P. E. (1992). Good judgment in international politics: three psychological perspectives. Political Psychology, 13, 517-539.
Note: Those who have completed the course may find these books of interest:
Neustadt, R. E. & May, E. R. (1986). Thinking in time: the uses of history for decision makers. New York: Free Press.
Klein, G. A. (1998). Sources of Power: How People Make Decisions. Cambridge, MA: MIT Press. / 150
4. / Study the conflict descriptions provided on conflictforecasting.com under Resources for Researchers so that you can write similar descriptions. / 60
5. / Use the Method Guide to help you prepare materials to use in applying the structured analogies method to at least one real forecasting problem. / 480
6. / Apply structured analogies using experts or a database to derive forecastsfor at least one real situation. / 120
7. / Take the Certification Examination. / 30
8. / Use your Certification Examination answers to grade your Pre-Course Test answers using grades A through F. / 10
9. / Open the Examination Answers file and use the answers in that document to grade your Certification Examination answers using grades A through F. / 10
10. / Develop and record the action steps you will take in order to obtain the knowledge to improve on your weak answers. / 20
11. / Complete your actions steps. / 60
Total minutes 1,020

Pre-Course Test

Forecasting Using Structured Analogies

Answer the following questions as best you can before looking at the course material. You will be able to refer back to your answers to this pre-course test after completing the course to assess how much you have learned.

  1. What is an analogy for the purpose of forecasting with structured analogies?
  1. Why does the structured analogies method tend to provide forecasts that are more accurate than forecasts obtained from experts who are using their unaided judgment?
  1. What is the role of the administrator in structured analogies?
  1. How expert do the experts need to be when using structured analogies? (Discuss)
  1. “For the purpose of structured analogies, an analogy can be a metaphor or a fable that has similarities to the target situation.”

True False

  1. Describe how to use structured analogies to forecast decisions in conflict situations.
  1. What are the important characteristics of the description of a target situation?
  1. “The primary reason structured analogies works is that analogies provide useful data when there is little relevant data about the target situation itself.”

True False

  1. Describe real problems where structured analogies could help to improve forecast accuracy.
  1. Is it a good idea to get more than one forecast for the one target situation? (Discuss)

Method Guide

Forecasting Using Structured Analogies

Forecasting using the structured analogies method involves five main steps: An administrator (1) describes the target situation, and (2) selects experts. The experts then work independently to (3) identify and describe analogous situations, and (4) rate the similarity of the analogies to the target. Finally, the administrator (5) derives forecasts.

(1) Describe the target situation

Before you start writing your description, determine what must be forecast in order to help the client make the best decision.

When the forecast requirement is clear, write a description of the situation. Descriptions of about one page in length are sufficiently long even for complex situations such as conflicts. (For examples see Conflict Descriptions in the Resources for Researchers section at conflictforecasting.com.) A conflict description should include information about the parties to the conflict and their goals, relevant history, current positions and expectations, the nature of the parties’ interaction, and the issue to be forecast. Longer descriptions and unnecessary detail are likely to overburden the experts who will be asked to identify and analyze analogies.

Shorter descriptions are likely to be better when the forecasting task is simpler than those encountered in conflict forecasting, for example predicting ticket sales for a proposed rock concert or enrolments at a new suburban gym. Include information on variables that might have an influence on the thing being forecast. For example, ticket sales for a rock band’s concert might be influenced by the local and regional population, weather, competing events, the venue, and the relative drawing power of the band relative to others.

Provide an objective description of the situation. Avoid implicit forecasts and unwarranted assumptions. When forecasting decisions in conflicts, avoid speculation about the motives and objectives of people in the conflict and, instead, let the facts, including public statements, speak for themselves.

A good understanding of the situation and considerable care is important at this stage. To this end, obtain information from independent experts, have the description reviewed, and test the material (described below). This process can require many rewrites and may consume much calendar time.

In situations where it is possible to define a set of discrete outcomes that is complete and that is relevant to the client’s own decision-making requirements, include a list of possible outcomes with the situation description. Keep the client involved by checking how forecasts will relate to the client’s decisions. Doing so will increase the likelihood that the forecasts will be used. A short list of possible outcomes provides clarity for experts who may be unfamiliar with the target situation, ensures forecasts are useful, and makes coding easier.

Based on research to date, we recommend that the list should include no more than four outcomes. When forecasting decisions in conflicts, the specification of three decision options—in essence Party-A’s favored decision, Party-B’s favored decision, and a compromise decision—often works well. In cases where a more specific forecast is required, break the forecasting task into parts and forecast each part. For example, having forecast that the decision favored by Party-B is made, use structured analogies to forecast what if any concessions are made to Party-A.

Because the decisions might be sensitive to how the situation is described, consider using alternative descriptions. To ensure independence, the alternative description should be given to a different group of experts.

If it is feasible, as where people with relevant expertise are plentiful, test situation descriptions by interviewing experts about their understanding of the target situation after they have read the description. Change the description in order to eliminate any problems that you identify.

(2) Select experts

For the purpose of forecasting with structured analogies, experts are people who are likely to be able to identify situations from the past that are similar to the target situation and who know enough about such situations to rate their similarity to the target.

Experts may be employees of the organization commissioning the forecasts, former employees, consultants, or university professors. Alternatively or additionally, knowledge about analogous situations might be available in the form of databases into which information about situations such as software development projects or civil wars has been coded against important variables.

The limited evidence to date suggests that forecasts derived from the information and analysis provided by people who are more knowledgeable about, and who were more directly involved with, the situations they identify as analogous are more likely to be accurate than forecasts from people with less knowledge. In practice, the relevance of an individual’s expertise may not be clear until after they have completed the expert portion of the structured analogies process. Therefore, the administrator should recruit as many diverse experts as can be justified by the expected net benefit of an accurate forecast.

(3) Experts identify and describe analogies

The administrator asks experts to think of as many situations as they can that are similar to the target situation. The situations should be real situations from the past with which the expert is familiar and for which the outcomes are known. Ask them to identify situations that are well-known or historical—for example, “Jefferson’s war against the Barbary Pirates” or “Apple Computer’s marketing campaign to launch the iPod”—or briefly describe situations that are not well-known—for example, “First Electric’s 1998 launch in New Zealand as a supplier of electricity to households in competition with an incumbent monopoly”.

Use a questionnaire to elicit the desired information from experts in a structured way. Figure 1, below, provides an example.

Figure 1

Example questions for conflict forecasting application of structured analogies method

If using databases, the administrator should decide on a selection rule before selecting analogies or ask experts to use the databases as memory aids or both. Code the target situation using the same variables as are used in the databases. If judgment is involved in the coding, ask diverse experts to do the coding and average the values assigned by the experts for each of the variables. If there is strong empirical evidence or theoretical grounds to believe that some variables are more important than others, ask experts to assign weights to the database variables and use the average weights in the selection rule.

If experts are used to identify analogies, ask them to note the source of their knowledge about each of the analogies they identify. Using the earlier examples, experts might give responses such as “Five years of research on the Barbary Pirates” or “Have published five articles on my research on alternative approaches to marketing PCs” or “I led the project.” The responses to these questions may help the administrator assess the degree of confidence that should be placed in the information provided.

(4) Experts rate similarity

When experts use their judgment to assess similarity, ask them to describe similarities and differences between the analogy and the target. Only when they have done this, should they be asked to provide an overall rating of similarity. This should be done in a structured way as illustrated in Figure 1.

(5) Derive forecasts

Having in the previous step carefully considered how an analogy relates to the target, experts should be well-placed to make judgments about what the outcome of the analogy they are considering implies for the outcome of the target. If it was possible in step 1 to prepare a list of possible target situation outcomes, ask the experts to choose from the list the outcome that is most similar to the outcome of the analogous situation as shown in Figure 1.

If it was not possible to prepare a list, ask the experts toprovide a succinct description of the outcome of the analogous situation. The outcome description should address the client’s need for a forecast and should avoid detail that does not bear on that need. The administrator may need to provide some guidance on this requirement.

Having obtained information and assessment of analogies from experts, it is the administrator’s job to derive forecasts from that data. Green and Armstrong (2007) found that a simple mechanical rule provided the most accurate forecasts. They took the outcome implied by an expert’s highest-rated (most similar) analogy as that expert’s forecast. A decision tree that includes solutions to tied ratings is included in the Green and Armstrong article.

Combine the individual experts’ forecasts. If the thing being forecast is nominal, such as in Figure 1, rather than quantitative, such as the first year sales of a new product, the combined forecast is the modal forecast. For example, if seven out of ten forecasts are that the parties decide to go to war, the administrator would predict war. In the case of quantitative forecasts, the combined forecast is the mean of the individual forecasts.

If some experts were more familiar with their top-rated analogies than others, consider weighting the forecasts of experts who were more personally involved in the situation more highly than the forecasts of experts who were not so intimately familiar with theirs. The evidence on this principle is modest, and so it would be as well to avoid weighting unless there are big differences in familiarity and, if there are, to avoid extreme weights.

Certification Examination

Forecasting Using Structured Analogies

  1. What is an analogy for the purpose of forecasting with structured analogies?
  1. Why does the structured analogies method tend to provide forecasts that are more accurate than forecasts obtained from experts who are using their unaided judgment?
  1. What is the role of the administrator in structured analogies?
  1. How expert do the experts need to be when using structured analogies? (Discuss)
  1. “For the purpose of structured analogies, an analogy can be a metaphor or a fable that has similarities to the target situation.”

True False

  1. Describe how to use structured analogies to forecast decisions in conflict situations.
  1. What are the important characteristics of the description of a target situation?
  1. “The primary reason structured analogies works is that analogies provide useful data when there is little relevant data about the target situation itself.”

True False

  1. Describe real problems where structured analogies could help to improve forecast accuracy.
  1. Is it a good idea to get more than one forecast for the one target situation? (Discuss)