SUPPLEMENTARY INFORMATION TO ACCOMPANY:

“Client perceptions of reported outcomes of group model building”

To improve replicability of this study, this section includes both of the evaluation tools used. These are a semi-structured interview, and a written questionnaire. Both the interviews and the questionnaire also contained questions not relevant to this study, which have been removed.

The semi-structured interview consists of themes, primary questions and secondary prompts (Kvale and Brinkman, 2008, see Table A1). The first theme enquires about the interviewees experience with group-decision processes. The second theme involves open questions to discover what outcomes the interviewees value. The third theme is more structured, and asks interviewees whether they value specific outcomes from the literature.

Table A1: Themes questions and prompts from semi-structured interviews

Theme / Primary Questions / Secondary questions / Additional prompts
Use of group-decision processes / How you have used group-decision processes in your work, and how you might use them in the future? / Can you tell me about a process that you have run or led?
Can you tell me about a process that you have commissioned?
Can you tell me about a process that you have been a participant in?
How often do you use group-decision processes?
Which groups have been involved in these processes?
Can you tell me about a time that involved (each group)?
Can you think of any other types of group-decision processes that you are involved in that we haven’t discussed already? / Who commissioned the process?
Who led or ran the process?
Who was involved?
How was the process managed or run?
What steps were involved?
Most valued outcomes / When you are running these group-decision processes, what objectives are important to you? / When you think about the group decision processes you have been involved in, how do you decide which ones were successful?
How do you decide which ones were less successful?
What are your goals when you design or commission a group decision process?
Have their been times when the obejctives have been different? / Why were these objectives important?
Why did these situations differ from the others?
Ratings of specific outcomes / Now I am going to describe certain findings for techniques I have been researching, and I would like you to think about whether that outcome is important to you, and how or where you might use that technique because of that finding. Each of these findings relates to decisions made through a group meeting or workshop environment. / Technique A improves the quality of communication between the participants.
Technique B results in increased new insights among participants.
Technique C results in increased consensus between participants.
Technique D results in greater commitment by participants to implementing the conclusions of the workshop.
Technique E makes strategy implementation more effective.
Technique F results in mental model change in participants.
Technique G results in mental model alignment between participants.
Technique H results in power-leveling between participants, where less-powerful members feel less-disadvantaged in their ability to contribute.
Participants using Technique I learn from and are persuaded by each other.
Participants who use Technique J tend not to revert to previous modes of thinking even 12 months afterwards.
Participants who use Technique K continue to use mental models that are more alike 12 months afterwards. / Is this outcome important to you?
When (or in what setting) would this outcome be important to you?
Are there any examples that you have been involved in where this outcome was particularly important? Why?
Are there any examples where this outcome would be less important or unimportant? Why?

The written questionnaire consisted of demographic questions and numerical scale questions that asked research subjects to rate individual outcomes of group decision processes from 1 (is of no importance) to 7 (is very important), as shown in Figure A1.

Written questionnaire

Age ….. yearsGender …..

I have been employed by the New Zealand government for ….. years

My management level is (circle one):

Level 3

Level 4

Level 5

Other level/non-manger

My highest academic qualification is (circle one)

High-School

Under-Graduate

Post-Graduate

Please rate each of these findings by how important this characteristic is in your work, where 7 is very important and 1 is of no importance.

No Very
importance important
Improved quality of communication between the participants / 1 2 3 4 5 6 7
Increased new insights among participants / 1 2 3 4 5 6 7
Increased consensus between participants / 1 2 3 4 5 6 7
Greater commitment by participants to implementing workshop conclusions / 1 2 3 4 5 6 7
More effective strategy implementation / 1 2 3 4 5 6 7
Mental model change in participants / 1 2 3 4 5 6 7
Mental model alignment between participants / 1 2 3 4 5 6 7
Less-powerful able to contribute / 1 2 3 4 5 6 7
Participants learn from and are persuaded by each other / 1 2 3 4 5 6 7
Participants tend not to revert to previous modes of thinking even 12 months afterwards / 1 2 3 4 5 6 7
Participants use mental models that are more alike even 12 months afterwards / 1 2 3 4 5 6 7

Figure A1: Written questionnaire

In interpreting the interview transcripts it was important to use a consistent definition for each outcome (Table A2). These were also used when interviewees asked for an explanation for a particular interview question.

Table A2: Definitions for outcomes explored in this study

Commitment to conclusions / The intensity of dedication by group participants to the content of the decision made, at the conclusion of the group decision process.
Communication quality / The extent to which the information content and action aspect of interpersonal interaction of group members is comprehensible and relevant.
Consensus / The extent to which the conclusions of the group decision process are generally agreed by group participants.
Mental model change / A change in the cognitive basis for the participants’ understanding of reality.
Enduring mental model change / A mental model change that persists after the conclusion of the group decision process.
Mental model alignment / The extent to which participants have a shared or compatible understanding of the problem and its context.
Enduring mental model alignment / The persistence of that shared or compatible understanding after the conclusion of the group decision process.
Effective strategy implementation / The extent to which the group decision process results in the activity and mobilisation of resources to achieve the intent of the decision.
Persuasive content / The degree to which the group decision process facilitates participants adopting the views previously help by other participants.
Power levelling / The degree to which the impact is minimised of relative personal and institutional power of group participants, on the interaction between those participants.
Insight / The impact of the decision process on participant’s understanding of the problem and its context.
Views of non-participants / The extent to which the conclusions of the group decision process are judged favourably by individuals who did not take part in the process.
Process efficiency / The time and resources required to achieve the other intended outcomes of the group decision process.
Further working together / The willingness by group decision participants to engage with other group participants in the future.
Willingness to endorse / The extent to which group decision participants would defend the conclusions of the decision process to others.
Reduced attachment to language / The relative focus of participants on the substantive consequences of the group decision, rather than its words.
Participant disclosure / The extent to which group decision participants volunteer information not known to others.
Tie-breaking / The extent to which a decision can be made between two equally supported options.
Completeness / The degree to which the group decision process has allowed participants to consider all aspects relevant to making an informed decision.