Factors Influencing the Selection of Method(S)

1

22 ISMORSECOND DRAFT
Jan Foghelin
Karin Mossberg Sonnek
FOI

Factors Influencing the Selection of Method(s)

The normative approach of choosing method

In general, a method is a series of steps taken to complete a certain task or to reach a certain objective. An OA-method is used for the purpose of making better decisions. In this paper, the term method is used in a broad meaning, including mathematical models over war gaming to soft OA.

A reasonable assumption could be that there should be a simple relation between a problem and the selection of method(s) to solve it. You could imagine a “best practice” for solving a certain class of problems. To select a method for a given problem, some factors could be considered, such as:

-The nature of the problem, which, according to Ackoff and Pidd, can be considered as a mess, a problem or a puzzle. This can also be expressed as how well defined the problem is, and if there exists an unambiguous solution or not.

-The aim of performing a study (solving a problem).One aim may be to discover or create something (often with emphasis on the process), another to derive a result or recommendation.

-The information sources (which for example can be literature studies, observations, experiments, interviews, -questionnaires ora focus groups) and the quality of the dataacquired. Sometimes it is possible to get more detailed data to an increased cost (time and money).

-The uncertainties involved, if they are known and in such case if they are dissolvable.

-The resources, such as experts, money and available time.

-The requirements for validation of the model/result.

Even if it would be possible to make a taxonomy for methods, where different alternatives on the above factors would result in a given method, we haven’t found that a useful approach.

Instead, we often use the diagram in figure 1, where we have positioned different methods, used at our division, according to where they usually are applied within a study: In the problem formulation, for generating alternatives to be studied, in the analysis, or for assessment of different alternatives. The diagram is used more as a check-list, and requires the user already to have a good knowledge of the different methods mentioned.

Figure 1. Methods grouped with reference to their usage in a studying process.

Another possible clustering of different methods is shown in figure 2, where the methods are classified according if they are quantitative or qualitative, and to the way of working when solving the problem. Some methods can be positioned in more than one box.The qualitative methods are often characterised by a high degree of interactivity between the analysts and the customer.

Class of methods
Quantitative methods / Qualitative methods
Way of working / Problem-solving
Problem given to analyst. Analyst mainly working on her/his own. / e.g.
• Simulation
•Analytical (Mathematical)
models / e.g.
• Cognitive mapping
Process-orientation
Customer(s) and analyst(s) working in a team. / e.g.
• MCDM
• Computer supported war
gaming / e.g.
• War gaming (explorative)
• Methods for creating
scenarios
• Morphological analysis

Figure 2. OA-support. Taxonomy of methods.

To our experience, there is seldom any need to find the best method; it’s enough to choose a method that gives a sufficiently good results. It is often necessary to use several methods in order to enlighten the problem from different views.

Choosing a method in reality

Reality seems however to be more complex than just considering the factors discussed in the previous section. Elaborating on this reality is the purpose of this paper.

A problem does not exist in a vacuum. It is an integral part of some contexts. The most important with examples from the defence sector, are the following:

-The problem-owner,which in this paper also is referred to as the customer (e.g. MOD or a Service).

-The substance of the problem and its environment (e.g. choice of the best mechanised battalion for a certain task. Influencing factors of the choice; scenario, enemy behaviour, equipment alternatives, organisationalternatives …)

-The analyst and the organization of the analyst.

Figure 3. Factors influencing the choice of method in reality.

The customer i.e. the problem-owner

There are customers for whom a black-box solution to a problem can be acceptable or even desirable. In such case the customer delegate to the analyst to choose method, and thereby she/he has a very small impact on the choice.

A black-box solution is characterised by;

-A clear-cut problem, for which the customer can furnish the necessary input data.

-A general trust in the professionalism of the analyst and other involved in the problem-solving.

-An acceptance of the proposed solution, without any deep discussions about assumptions, methods or data.

-Feed-back to the customer and to the analyst will come, from the function of the proposed solution

The black-box model is normally only applicable to lower level tactical problems of the customer organisation. (e.g. some logistic problems). For higher level strategic problem the customer often wants more involvement and understanding.

If the customer isn’t satisfied with the above solution, there are several different patterns of customerinvolvement:

-A genuine interest by the customer to understand the methods used and the impact of assumptions and impact data (including uncertainties). There could also be a dialogue before the start of the study of the problem concerning advantages and disadvantages of different methods.

-A wish by the customer to be actively involved in the study process. For most of the customers this will mean restrictions on the choice of method. Soft OA-methods and gaming are easier methods to use in this situation than more mathematically orientated methods. From a study process point of view it could many times be and advantage to involve the customer in the process early. It could increase the quality and facilitate the acceptance of the result.

-A wish by the customer to use a certain method. This could be a method which the customer has good experiences of or it could be the talk-of-the-town method for the moment. Many times this wish does not create difficulties. There could however be cases where the proposed method is quite simply unsuitable. It is a pedagogic, not always easy, task for the analyst to explain why the proposed method is not a good choice. It’s therefore important for the analysts to learn to argument both for and against different methods.

-Restrictions on resources (money, time…….) by the customer.It is a natural instinct by an analyst to accomplish an ambitions study. This could be the case for the customer as well but in most cases there are some restrictions. Restrictions could exclude certain methods. War gaming cannot be used without the active involvement of the customer.Simulation models need input data which can take time and money to get. A new simulation model requires an investment.
Restrictions on resources should be seriously discussed between the customer and the analyst before the study has started. These dialogues are not always successful e.g.

• The analyst could be to optimistic about resources to construct a simulation model or time needed for a war game.
• The customer wants an impossible “quick fix” i.e. high quality results without necessary knowledge about the logic of the problem and/or input data. There sometimes seems to be customer believing in “magic methods” which can solve any problem without any sort of input.

More resources, on the other hand, allows for investment in tools (e.g. a simulations model) and use of experts (e.g. for technical data). The use of a well-structured model and high-quality data will of course increase the quality of the output. The transparency of the way the results are derived is of great value as part of the quality.
The necessity of good data for good results of a study is often stated – rightly so. We would like however to emphasise the importance of knowing the logic of what you are modelling. The logic of an armoured warfare battle during the cold war may have been known, but not the new types of asymmetric warfare. Balancing (logic, data, and model construction) a model within a study, given resource constraints, is an optimising problem in itself. This balancing problem should be discussed with the customer.

-The customer wants a predetermined solution to the problem. This is the wellknown dilemma when the customer not really wants to have an independent analysis but a proof of a prejudice. This is normally not a type of study the analyst wants to do (and hopefully not the customer either) but if it should be done use soft-OAand show the customer´s assessment of logic and other inputs.

Substance and environment

The notionof open and closed systems could be used in two ways discussion type of problems and different ways of solving them.

The first distinction of open and closed concerns the problem itself (the nature of the problem). It can either be self-contained, which means it can be solved within the customer’s organisation (a closed problem) or it has important interactions with itsenvironment, and requires it to be considered in the analysis (an open problem). A special case, frequent in the defence area, constitutes the hostile environment. You have then to handle an environment which is counteracting and not given.

Figure 4. A closed and an open problem.

Comments on selection of methods in this case:

-While the self-contained problems tend to be more straightforward and easier to handle when it comes to selection of methods the open system can be much more demanding. Reasons for this

• Problems of delimitation, which parts of the environment are the most important
• Problems to handle an aggressor. (Repertoire, surprises …)

-It is sometimes too easy to standardise the environment and concentrate on the internal problems, which can sometimes lead to the wrong alternative. It is important to use a method which allows you to consider the whole range of environments.

The second distinction of open and closed system concerns the problem solver. They (customer(s) + analyst(s)) could be seen as a closed system (e.g. locked into a manor in NE England over a week-end with the given or taken task to solve a problem) or an open system. The environment to the open system could consist of specialists, other analysts, and other people belonging to the customer organisation than the care group

Figure 5. A closed and an open system of problem solvers.

Before comments on the choice of method a short note on the concept of model.

A model is a simplified transparent representation of a chosen part of reality which could be used to facilitate problem solving. It normally requires knowledge of the logic of reality and data to be useful.

About the selection of methods:

-A closed system of customers + analysts is often restricted in time. The aim of the decision-making is more of searching for an acceptable solution which could be reached by consensus rather than the optimal solution. Interactive methods are the best for this way of problem-solving with the analyst contributing as a facilitator.
It is important that the methods used can engage all the participants (often with quite different backgrounds) and promote the handling of conflicts within the group.

Analyst, analyst’s organisation

First a few words about the role of an analyst. The role could be defined and constrained to the use of a method in a professional way. For us however the starting point is much wider (problem formulating, problem structuring, acquiring of inputs in a wide meaning, selection of method(s), presentation…..). That said in a specific case, you always have to discuss limitations with your customer.

Most analysts know many more methods by name (and basic ideas) than they professionally are able to apply. It is one thing to read about a method or hear about one in a conference and quite another to apply it on a messy problem for an important customer.

OA-organisations are less confined in their choice of methods. But even they have their limitations concerning reduction of methods depending on;

-Historical experiences of what has worked or not

-Background of the analysts (which tend to be self-reproducing)

-“Normal” type of problem

-“Culture” e.g. definition of OA (hard vs soft…).

There is a tendency that our OA-organisation will be riskaversiwe in it’s choice of methods. In an ideal world you can test different methods by using them in parallel or on less important problem. It is however not so easy to finance this. Sometimes it may also be the opposite, some analysts tends to choose a new method since they are curious about it, rather than to be assured about a reliable result.

Some conclusions

Other factors influencing on choice of methods from an analyst’s point of view are the following:

-Uncertainties.
Great uncertainties require methods being able to handle many alternatives and present them in well-arranged way. (e.g. scenarios + simple assessment model + “traffic light” presentation)

-Logic and data.
To put it very simply:
Well-known logic and good data hard OA

Shaky logic and low-quality data soft OA
If you have a lot of resources the best is to, as a first step, to transfer the shaky logic and low-quality data to well-known logic and good data and be able to use hard OA as a second step.

There could be different sources of logic and data e.g.:

-The customer organisation (a critical attitude by the analyst is necessary. For different reasons there could be a bias).

-The organisation of the analyst. (A little warning for using (simulation) models which are not appropriate for the actual problem).

-From a third party (quality control!).

-The analyst do some research on his/her own.

Customer-involvement.
There are several reasons for trying to get the customer involved in the study process

• You build in a trust in the results
• You can get data/assessments which are difficult to acquire by direct asking.

A customer involvement limits the choice of methods.

References

Ackoff, R.L. (1974) "Redesigning the future: A social Approach to Societal Planning"

Foghelin, J: Analytiker. ”Några observationer beträffande efterfrågan och tillgång”. FOA Rapport D 10161, juli 1990. (Analysts. Some observations concerning demand and supply). (In Swedish).

Foghelin, J: ”Analysmetoder – en metanot”. FOI Memo 03-925. 2003-04-03.

(Methods of Analysis – a Metanote) (In Swedish).

Foghelin, J: “Lessons learned by Personal Experience from Interacting with Decision-makers”. 21ISMOR, 31 August-3 September, 2004, UK.

Pidd, M. (1996) "Tools for thinking"

1(5)