ABM and the Validation Hurdle- an Illustration

ABMs and the Validation Hurdle - An Illustration

Dr. Alfred Brandstein

Paul J. Wehner

The MITRE Corporation

7515 Colshire Drive

McLean, VA22102, U.S.A

WHAT IS AN AGENT BASED MODEL (ABM)? In order to set the stage, we paraphrase Bonabeau [1] and characterize agent-based modeling as a way of thinking which begins with the microscopic rather than the macroscopic.

WHY IS VALIDATION A CHALLENGE FOR ABMs? Validation – in the traditional sense – requires one to gauge whether or not a simplified representation of reality (i.e., a model) is accurate enough for its intended use. This evaluation is particularly challenging for ABMs because their targeted-application areas do not lend themselves to formal description or direct evaluation.

Generally speaking, ABMs model real-world phenomena that cannot be described by easily solved equations. Physical and engineering examples abound [2]; however, targeted-application areas that contain significant interactions between humans and physical systems are of particular interest to the authors. These areas offer fertile ground for ABMs due to the absence of satisfying macroscopic equations for human behavior.

Since ABMs are often used to explore the causal relationships of complex interactions or to help analysts in their efforts to develop bounded solutions to these challenging application areas, evaluating the ‘appropriate’ degree of accuracy (in the context of intended use) is problematic. Direct evaluation or observation of the targeted-application area may yield discrete results; however, the ability to consider those results “representative” is at best limited due to the degrees of freedom involved.

CAN ABMs CLEAR THE TRADITIONAL VALIDATION HURDLE? In a word, yes. However, we find ourselves questioning ifthe ABMs capable of doing so are sophisticated enough to explore the type of questions key to our interests. For example:

How might the outside world react to…?
How might the system be exploited?
How might we evaluate the impact of new equipment on…?
What are some unexpected consequences of…?
What effects can emerge?
What happens if someone does something irrational?
Why do…?

HOW SHOULD THE VALIDATION HURDLE DIFFER FOR ABMs? If the primary measure of success for an ABM is an evaluation of whether it can and has been used to demonstrate (both within the community and externally) the need for and the consequence of changes, then “accuracy” is likely the wrong measuring stick.

Rather than force fit that construct, we propose applying the following two criteria to determine if an ABM should be deemed “valid”:

Is every model outcome possible?
Is every possible outcome realizable by the model?

AN ILLUSTRATION: For the rest of this paper we concentrate on a specific approach developed by the authors for U.S. Customs and Border Protection’s Office of Border Patrol (CBP’s OBP). Emphasis will be on methodology rather than results.

The Defense Modeling and Simulation Office recommends that validation activities areongoing activities performed as part of the overall development processfor a model/simulation [3]. While we certainly concur, we also contend that the degree of user (i.e., customer) involvement and the transparency of the representation must increase as a function of the perceived complexity of the target-application area; else, the validation hurtle becomes increasingly more challenging, if not insurmountable for ABMs.

Given the recognized complexity of OBP’s targeted-application area (i.e., the Border region), we sought early and continuous involvement of the customer and a modeling construct that supported a high degree of process and representation transparency. OBP responded to our request for customer involvement by co-locating two Border Patrol Agents with our MITRE development team for approximately six months, making others available for consult, and arranging multiple visits to the immediate Border area to witness first hand key aspects of the targeted-application area and speak with local subject matter experts.

The selected modeling construct was an ABM called MANA (Map Aware Non-uniform Automata) [4]. ABM was a natural fit given the high degree of human and system interactions present in the targeted-application area. MANA was selected for the following reasons:

The language offers a GUI that provides a high degree of transparency during model creation and simulation.
The language readily supports development and use in a seminar setting.
Initial models could be constructed in hours.
The resulting models can be run on laptops.
The authors had extensive experience in using MANA in similar settings [5].
An extensive infrastructure exists [6] to evaluate results from this modeling language.

THE DEVELOPMENT/VERIFICATION/VALIDATION PROCESS: The process began and ended withthe development team listening to the customer.

The embedded Border Patrol Agents were a clear and constant window into the targeted-application area we modeled. As they described the Border environment we distilledkey properties of that environment, captured them within the modeling construct, and verified that our representation was true to their experience. By the end of the first day a rudimentary model was running.

Over the next couple of weeks, the model was evolved in this seminar context until the embedded Border Patrol Agents were satisfied that all essential interactions were incorporated. The model wasthen introduced to a wider audience of Border Patrol Agents and additional interactions were identified as essential, captured, and their representation verified. The emerging model was eventuallyco-presented (i.e., developer standing along side Border Patrol Agents) to CBP and OBP senior staff for review and comment.

OBP elected to subject that version of the model to independent verification and validation and CBP tasked MITRE to expand the area of interest modeled. Provisions were incorporated to run the model on massively parallel machines and to link various instantiations of the model together in an attempt to incorporate larger areas. The development process mirrored the process described above and ultimately CBP elected to subject the federated version of the model to independent verification and validation.

The designated verification and validation agent for both efforts was JohnsHopkinsUniversity, Applied Physics Laboratory (JHU APL). In both cases, JHU APL recommended the models for accreditation with limitations; where the expressed limitations largely reinforced the intended uses and stated limitations of the models.

RELATION TO THE OVERALL ANALYTIC PROCESS: It is possible that the analysis of results from these micro-level models, using techniques like Operational Synthesis [7], can be used to design algorithms to be used in deterministic models and macro-level simulations, similar to the ways that Statistical Mechanics evolved from the atomistic theory.

WHAT TO EXPECT: Expected outcomes include the ability to support discussion, debate and decision making. What makes the agent-based modeling approach unique is its transparency; its ability to capture the interactions provided by Subject Matter Experts and, in the case of the MANA based models, the ability to immediately visualize what is happening. Typical uses include demonstrating the consequences of various actions and exploring means of encouraging or discouraging various outcomes. The hallmark of these types of models is that they can produce non-intuitive results in a manner that convinces users that their intuition needs to be honed.

WHAT NOT TO EXPECT: One should be hesitant to drawfirm conclusions about the probability of occurrence for one or more outcomes. This capability, much like the capability to predict with any certainty how an individual or groups of individuals will react to an event, is largely beyond our reach. Moreover, especially in “complex” situations, there is no rigorous method available to determine sampling distributions to determine these probabilities.

REFERENCES:

[1] Bonabeau. “Agent-based modeling: Methods and techniques for simulating human systems”, Proceedings of the National Academy of Science, 99(3):7280--7287, May 2002.

[2] Leamy. “A Cellular Automata Approach for Continuum Mechanics”, manuscript in preparation.

[3] “VV&A Recommended Practices Guide”, Defense Modeling and Simulation Office, RPG Build 3.0, September 2006.

[4] Calligan, Anderson, Lauren. Map Aware Non-uniform Automata, Defense Technology Agency, April 2005

[5] Brandstein, Horne. “Data Farming: A Meta-Technique for Research in the 21st Century”, Maneuver Warfare Science 1998, 93-100, USMC.

[6] Lucas, Sanchez, Brown, Vinyard. “Better Designs for High-Dimensional Explorations of Distillations”, Maneuver Warfare Science 2002, USMC, 17-47

[7] Brandstein. “Operational Synthesis: Applying Science to Military Science,” PHALANX, Vol. 32, No. 4, December 1999.