DRAFT – April 10, 2006

Collaborative Time-Sensitive Targeting
Design of Experiment: Study 1

Stacey D. Scott

Humans and Automation Lab

Massachusetts Institute of Technology

Introduction

The goal of this project is to facilitate decision making in collaborative time-sensitive targeting (TST) operations. In particular, our aim is to help team members make appropriate decisions that are in line with the priorities of the overall mission. In the chaos and uncertainties of modern time-sensitive targeting operations, it is often easy to lose sight of the overall mission objectives while trying to manage ongoing, emergent problems of local operations. This particular experiment focuses on evaluating a set of displays which have been designed to help supervisors of a team of UAV operators achieve overall mission success while helping their UAV operators solve local problems.

Drawing from recent work in the Computer-Support Cooperative Work research field (Carroll, Neale et al., 2003; Carroll, Rosson et al., 2006; Millen, Muller et al.; Muller, Geyer et al.), we have designed several computer interface displays to support team supervision in a Collaborative TST scenario. These displays augment traditionally available battlefield and targeting information with activity awareness information to help TST team supervisors maintain awareness of the activities of their team members in the context of the overall mission progress. These displays are intended to serve two purposes that we feel will help TST team supervisors’ decision making:

-to help them understand how the current battlefield situation relates to their operators’ individual activities/performance. This information should help the TST team supervisors understand which operators are effectively handling their current mission tasking and which operators might need assistance or a change in tasking, and

-to help them prioritize the team’s current problems in the context of the overall mission priorities.

Objectives

The overall objective of this experiment is to evaluate how well the proposed team supervision displays facilitate decision making in Collaborative TST operationsin a laboratory user experiment. The goals of this experiment are twofold. The first goal is to evaluate the proposed displays’ effectiveness for supporting the supervisory role of a mission commander in Collaborative TST mission operations. The second goal is more general and involves exploring some of the open questions in the new research approach of providing activity awareness to help further our understanding of these types of displays, which in turn will help us improve our supervisor displays.

Research Questions

To achieve these goals, the follow research questions will be investigated in this user study:

  1. Do the proposed activity awareness displays enable effective decision making in the supervision of collaborative TST operations?
  2. Do these displays facilitate decision making that result in successful overall mission performance?
  3. Do these displays facilitate/encourage good decision making strategies?
  4. Reactive / proactive?
  5. Do these displays enable appropriate prioritization of emergent team problems given the overall mission status?
  6. Local or global problem solving? Impact on overall mission priorities (e.g., convoy safety)
  7. Does operational tempo impact the ability of the proposed displays to facilitate decision making (i.e., does operational tempo impact 1a-1c? If so, how?)
  8. Do the proposed activity awareness displays provide the necessary information to make the appropriate decisions?
  9. If so:
  10. Is this information presented in a form that is easily understandable or accessible when it is needed?
  11. Is this information presented at an appropriate level of detail?
  12. If not:
  13. What information is missing?

Experimental Task: Generalized Team Time-Sensitive Targeting (GTT) Task

In order to investigate these questions a representative collaborative TST task scenario, the Generalized Team TST (GTT) task,was developed upon which to build the experimental simulation environment. The GTT task involves a team of TST operators working together to secure a large geographic area (the area of responsibility (AOR)) to ensure the safe passage for an important political convoy that will be traveling through the area in the near future. During the task, the team will be required to surveil the area using a number of high autonomous Unmanned Aerial Vehicles (UAVs) for potential threats. Once hostile targets have been identified, the TST team must coordinate with an external strike team to engage these hostile contacts before they are in weapons range of the convoy. The team will be required to monitor incoming intelligence reports in order to extract information relating to their AOR and potentially communicate with other teams as necessary to clarify intelligence reports.

In order to secure the AOR, the team will be required to utilize a number of highly-autonomous Unmanned Aerial Vehicles (UAVs) under their supervision. The team will be required to monitor the progress of these UAVs as they provide surveillance of the large AOR and to reroute the UAVs from their original surveillance course, as necessary to secure the area. The team may also be required to coordinate with other teams to utilize assets outside of their immediate control to help secure the AOR.

The TST team consists of four UAV operators responsible for controlling a number of highly autonomous UAVs and one mission commander overseeing the team’s mission progress. In particular, the UAV operators are responsible for supervising the progress of four UAVs surveilling the AOR and coordinating with the strike team to destroy identified targets in the area. The mission commander is responsible for managing the workload of the three main UAV operators on his team by moving a spare UAV operator around the team to helping them balance their workload throughout the mission. The mission command is also responsible for making strategic decisions to ensure the safety of the convoy, such as requesting the convoy hold their current position if their intended path is not deemed safe for passage or requesting help from external assets to help the team secure the area.

While there are many collaborative components to the GTT task, this project will focus on the decision-making and performance of the mission commander (i.e. the team supervisor) managing the overall tasking of the TST team. In this experiment, the tasking actions of the UAV operators will be simulated, along with the rest of the battlefield environment, and indicated by the changing battlefield situation and health and status information shown on the team supervision displays, described below. In the experimental conditions where the UAVs operators are present their tasking actions will still be simulated; their role within the experiment will be monitor their own region and notify the team supervisor when the situation in their region deteriorates.

GTT Team Supervision Task

Within the context of the GTT task scenario, experimental participants will be asked to assume the role of the TST team supervisor (the UAV operators will be experimental confederates). As the team supervisor, their task will be to monitor the ongoing (simulated) mission progress using the team supervision displays (see Figure 1) and to make certain intervention decisions when necessary to successfully complete their mission (i.e., ensure safe, expeditious passage of the convoy their area of responsibility). These decisions include:

Internal team interventions:

  1. Assign the spare operator to a region where the current operator is overloaded,
  2. Remove spare operator from a region can be managed by a single operator again,

External interventions:

  1. Request the convoy hold its current position, and
  2. Request the convoy resume their progress.

These decisions will be commanded through a mobile tabletPC interface that participants will be provided during the experimental sessions.

Participants will also be asked to use the tabletPC interface to record each occurrence of a target being scheduled for a strike within 10 seconds[ss1]prior to the start of its threat envelop (i.e., the period of time it is within weapons range of the convoy, based on its location and its weapons capability). For example, if a target is schedule to be destroyed at 5 seconds before the convoy will be within range of its weapons, the participant should record the ID of this target this as soon as they notice this tight-scheduling situation. This subtask will be referred to as the target buffer breach detection task.

GTT Team Supervision Displays

Figure 1 shows the two team supervision displays, which include a variety of activity awareness information that is dynamically updated as the mission progresses. For example, in the battlefield situation map display in Figure 1a the color of the UAVs (the symbols) represents their current tasking (blue: surveillance, orange: supporting a strike, grey: last know location, possibly destroyed). The color of the targets (thesymbols) represents the team’s current progress on the prosecution of that target (red: identified, orange: waiting for strike, yellow: waiting for battle damage assessment after strike, grey: destroyed). In the Mission Status display, the current strike plan is displayed to illustrate when the team is expected to support (e.g., pinpoint a target) a target strike. This display also provides the team supervisor information about the current workload of the individual UAV operators and the communication status to external resources like AWACS, the strike team, and a satellite link.


(a) /
(b)[SS2]
Figure 1. The Team Supervision displays: (a) the Situation Map display and (b) the Mission Status display.

The displays shown in Figure 1 will be displayed on large, wall screen displays in the Humans and Automation Lab team testing environment. The team supervisor will indicate their decisions throughout the experimental sessions on a button panel provided on a mobile tabletPC. These decisions will be input into the simulation, changing the situation as appropriate.[ss3]

Experiment Design

The experiment will be a 2(Targets) x 2(Operators Present) Mixed Design, with repeated measures on the Targets factorsand between subjects on the Operator Presence factor.

Independent Variables

Targets. The two levels of the Targets factor include: low and high. This factor relates to the number of targets located in the region that the team should discover and destroy. The actual number of targets which constitute ‘low’ and ‘high’ will be determined through pilot testing of the experimental software. Varying the level of this factor enables the investigation of the impact of operational tempo on the decision making support provided by the team supervision displays. A low number of targets emerging at a relatively low frequency (to be determined through pilot testing) will provide a slow operational tempo. In contrast, a high number of targets emerging at a relatively high frequency (to be determined through pilot testing) will provide a high operational tempo.

OperatorsPresent. The two levels of the Operators Present factor include: present or not present. This factor relates to whether the team of UAV operators is present or not. In the ‘present’ condition, the team of human operators is present in the room while the team supervisor is making decisions. Their role is to monitor the status of their own region of responsibility and to notify the team supervisor when they feel their current team of UAVs is overloaded and the situation in that region is degrading (e.g., number of targets exceed a certain amount, a UAV gets destroyed, many targets of long weapons range, etc.). In the ‘not present’ condition, the operators are not present and the team supervisor must rely solely on information provided on the team supervision displays to make their decisions. The two primary reasons to include this factor are:

  1. To help determine if important information is missing, or is difficult to interpret, from the team supervisor displays. The assumption here is that the team supervisor might ask the operators for information that they feel would help them make a decision but does not appear to be available on their displays.
  2. To understand whether the physical presence of their team hinders or helps supervisor decision making. In particular, whether the ability of the supervisor to prioritize their decisions is impacted when operators, whose problems might be low priority in relation to the overall mission priorities – and who are not necessarily attending to the entire situation – distract the team supervisor from focusing higher priority mission issues.

Dependent Variables

  1. Quantitative
  1. Mission time (mission ends when convoy clears team’s area of responsibility or convoy is destroyed)
  2. Targeting performance:
  3. Targets discovered: % targets discovered = # targets found / # total targets
  4. Target discovery rate: total time to find all discovered targets
  5. Target buffer breach detection performance:
  1. Target buffer breach detection: elapsed time between target scheduling and threat buffer breach detection
  2. Target buffer breach omission errors: # of target buffer breaches not detected
  1. Convoy safety performance:
  1. Convoy health level: convoy begins with certain energy level, which depletes each time an attack occurs (see Appendix A for details of health level calculation)
  2. Convoy safety breaches - target: cumulative time convoy is within threat envelop of a known target across entire mission (threat envelop is the time window when the convoy will be within weapons range of a target (or potential target) - this may or may not result in an attack)
  3. Convoy safety breaches – unknown region: cumulative time convoy is within threat envelop of an unsurveilled region
  4. Convoy attacks: # of times convoy is attacked
  1. Decision making
  1. Number of times each available decision was made during the trials:
  2. Assign spare operator to a region
  3. Remove spare operator from a region
  4. Hold convoy
  5. Resume convoy
  6. Time and mission status for decision taken during the trials (mission status: convoy health, convoy safety level (safe, in threat envelop buffer, in threat envelop, under attack), targeting performance)[ss4]
  7. Number & occurrence pattern of decisions of each type (reactive/proactive):
  1. Reactive: # of decisions made (i.e., a click on one of the decision buttons) after critical incident has occurred (e.g., convoy under attack) or situation deteriorates below a certain threshold (e.g., convoy enters threat envelop)
  2. Proactive: # of decisions made after situation begins to deteriorate (e.g., convoy enters target strike buffer) but before critical incident occurs.
  1. Qualitative
  1. Cognitive Strategies:
  1. Semi-structured interview to include a retrospective video review (see Appendix B for details) eliciting participants to articulate:
  2. their information gathering strategies to making each type of decision (assign/remove spare operator to/from a region, hold/resume convoy)
  3. their information gathering strategies used for the target buffer breach detection task
  4. in particular, where they looked on the displays to gather that information
  1. Communication Analysis: In the trials involving other team members, the team communications (verbal and non-verbal) will be coded using the open coding qualitative analysis coding method (Strauss & Corbin, 1998), with the following information used as the basis for the initial coding scheme:
  1. Information requested by the team supervisor:
  1. what information was requested?
  2. what was the situation when the information was requested?
  3. who was the information requested from?
  4. was the requested information available on the team supervision displays?
  5. how was the information requested (voice only, NVC, voice+NVC?)?
  6. was the requested information provided?
  7. was the provided information accurate?
  8. how was the information provided (voice only, NVC, voice+NVC?)?
  1. Other relevant conversation revealed during the open coding phase of the communication analysis

Table 1 shows the relationship between the research questions and the above study measures.

Table 1. Research questions and their associated measures.

Q# / Research Question / Associated Measures
1 / 1. Do the proposed activity awareness displays enable effective decision making in the supervision of collaborative TST operations? / -Mission time (I.1)
-Targeting performance (I.2)
-Target buffer breach detection performance (I.3)
-Convoy safety performance (I.4)
1.a / 1.a. Do these displays facilitate decision making that result in successful overall mission performance? / -Cognitive Strategies (II.1) with decision making data (I.5.a, I.5.b)
1.b / 1.b. Do these displays facilitate/encourage good decision making strategies? / -Decision making type (I.5.c) (using convoy safety performance data - I.4)
-Time and mission status during decisions (I.5.b)
1.b.i / 1.b.i. Reactive / proactive? / -Decision type (I.5.c)
-Time and mission status during decisions (I.5.b)
1.c / 1.c. Do these displays enable appropriate prioritization of emergent team problems given the overall mission status? / -Time and mission status during decisions (I.5.b) - in particular, convoy safety level for each decision
1.c.i / 1.c.i.Local or global problem solving? Impact on overall mission priorities (e.g., convoy safety) / -Time and mission status during decisions (I.5.b) - in particular, convoy safety level for each decision
-Decision type (I.5.c)
1.d / 1.d. Does operational tempo impact the ability of the proposed displays to facilitate decision making? / -The measures for the above questions (1.a - 1.c) examined under different levels of the Targets factor
2 / 2. Do the proposed activity awareness displays provide the necessary information to make the appropriate decisions? / -Successfully completed mission in acceptable time frame: mission time (I.1), targeting performance (I.2), target buffer break detection performance (I.3), convoy safety performance (I.4) are acceptable (based on thresholds determined during pilot testing)
2.a / a. If so:
2.a.i / 2.a.i. Is this information presented in a form that is easily understandable or accessible when it is needed? / -Semi-structured interviews with retrospective video replay (II.1.a)
-Communication Analysis (I.2), in particular (II.2.a.iiii)
2.a.ii / 2.a.ii. Is this information presented at an appropriate level of detail? / -Semi-structured interviews with retrospective video replay (II.1.a)
-Communication Analysis (I.2), in particular (II.2.a.iiii)
2.b / 2.b. If not:
2.b.i / 2.b.i. What information is missing? / -Semi-structured interviews with retrospective video replay (II.1.a)
-Communication Analysis (I.2), in particular (II.2.a.iiii)

Participants

Twenty-four subjects will be recruited to participate in this study. Participation will be strictly voluntary. If possible, subjects with military supervisory experience will be recruited. The experimenter will explain the study to the subjects and answer any questions about the study subjects might have. Subjects will be required to sign an Informed Consent Form before performing any of the experimental activities.