Fitting methods to paradigms: are Ergonomics methods fit for systems thinking?
Paul M. Salmon1, Guy H. Walker2, Gemma Read1, Natassia Goode1, Neville Stanton3,
1Centre for Human Factors and Sociotechnical Systems,
Faculty of Arts and Business, University of the Sunshine Coast, Maroochydore, QLD 4558, Australia
2Institute for Infrastructure and Environment, Heriot-Watt University, Edinburgh, EH14 4AS, UK
3Transportation Research Group, University of Southampton,
Highfield, Southampton, SO51 7JH, UK.
Abstract
The issues being tackled within ergonomics problem spacesare shifting. Although existing paradigms appear relevant formodern day systems, it is worth questioning whether our methods are. This paper asks whether the complexities of systems thinking, a currentlyubiquitous ergonomics paradigm,are outpacing the capabilities of our methodological toolkit. This is achieved through examining the contemporary ergonomics problem space and the extent to which ergonomics methods can meet the challenges posed. Specifically fivekey areas within the ergonomics paradigm of systems thinking are focussed on: normal performance as a cause of accidents, accident prediction, system migration, systems concepts, and ergonomics in design. The methods available for pursuing each line of inquiry are discussed, along with their ability to respond to key requirements. In doing so, a series of new methodological requirements and capabilities are identified. It is argued that further methodologicaldevelopment is required to provide researchers and practitioners with appropriate tools to explore both contemporary and future problems
Practitioner summary
Ergonomics methods are the cornerstone of our discipline. This paper examines whether our current methodological toolkit is fit for purpose given the changing nature of ergonomics problems. The findings provide key research and practice requirements for methodological development.
Introduction
Structured methods provide the foundationforour discipline (Stanton et al, 2013). Within the realm of cognitive ergonomics, researchers and practitioners have a wide range of methods available for studying aspects of operator, team, and system performance. In the individual operator context these include methods such as cognitive task analysis (e.g. Klein, Calderwood and McGregor, 1986), workload assessment (Hart and Staveland, 1988), situation awareness measurement (e.g. Endsley, 1995), and error identification techniques (e.g. Shorrock and Kirwan, 2002). For teams, they include teamwork assessment (e.g. Burke, 2004), analysis of communications (e.g. Houghton et al, 2006), and team workload assessment (e.g. Helton, Funke and Knott, 2014). More recently methods such as Accimap (Svedung and Rasmussen, 2002), the Event Analysis of Systemic Teamwork (EAST, Stanton et al, 2013), the MacroErgonomic Analysis and Design method (MEAD; Kleiner, 2006), the Functional Resonance Analysis Method (FRAM; Hollnagel, 2012) and Cognitive Work Analysis (CWA; Vicente, 1999) are being applied to analyseoverall systems and theiremergent behaviours.
There is no doubt then that ergonomists have access to a diverse methodological toolkit; however, the systems in which ergonomists operate are becoming increasingly complex andtechnology driven (Grote, Weyer and Stanton, 2014; Dekker, Hancock and Wilkin, 2013; Walker, Salmon, Bedinger, Cornelissen, Stanton, this issue; Woods and Dekker, 2000). Whilst work systems have arguably been complex since the dawn of the discipline, a shift towards the systems thinking paradigm, along with increasing levels of technology and complexity, is beginning to expose the reductionist tendencies of many ergonomics methods.Indeed, an examination of recent papers published in this journal leaves no doubt that the issues currently being tackled are stretching the capabilities of our methods (e.g. Cornelissen et al, 2014; Stanton, 2014; Young, Brookhuis, Wickens and Hancock, 2015; Trapsilawati, Qu, Wickens and Chen, 2015;Walker, Stanton, Salmon and Jenkins, 2010). Example issues include emergence, resilience, performance variability, distributed cognition, and even complexity itself. These issues (or lack of them) are to be found, increasingly prominently, in modern day catastrophes (a major focus of ergonomics research and practice). Typicallythese have numerous contributory factors stretching over multiple people, technologies, organisations environments and time. With these complex problems in mind, it is dangerous to assume that our methods remain fit for purpose simply because we continue to use them.
On top of this is the fact that the problemsthemselves do not appear to be improving as they once were. The statistics across common ergonomics application areas make for sobering reading. In Australia, for example, over 600,000 workers are injured per year with an estimated annual cost of over $60 billion dollars (Safework Australia, 2012a, b). In areas such as road transport, road collisions take the lives of well over 1000 Australians per year and cause approximately 50,000 to be admitted to hospital (Bradley & Harrison, 2008). Even in domains where the level of regulation and control is much higher, such as the aviation industry, there were still 221 serious incidents and around 5,500 incidents reported to the Australian Transport Safety Bureau in 2013 (ATSB, 2013). In the rail industry there were 350 fatalities and 923 serious personal injuries across Australia between 2002 and 2012 (ATSB, 2012).For all of the good ergonomics research and practice achieves, the outcomes that we seek to prevent are still occurring, and in large numbers. This is not of course entirely down to ergonomists and our methods, and clearly the methods being applied are having a beneficial effect otherwise there would not be a business or safety case for continued investment in them. Other issues play a role including the dissemination of ergonomics applications, the integration of ergonomics practices within system design and operation practices, and the gap that exists between ergonomics research and practice (Underwood and Waterson, 2013). Equally though, the numbers suggest that there are likely problems that are proving resistant to ergonomicsmethods. Why is this? Are our methods tackling (successfully) only the deterministic parts of problems, leaving the underlying systemic issues unaddressed? At the very least it is legitimate to question the validity of our existing ergonomics toolkit. Not least because validity is often assumed but seldom tested(Stanton and Young, 1999).
For ergonomics to remain relevant it is imperative that our methods can cope with the problem spaces in which researchers and practitioners work and indeed with the paradigms that are driving this work.We are not alone in expressing these concerns. Dekker (2014), Leveson (2011), Salmon et al. (2011) and Walker et al. (this issue) present an, at times, alarming picture ofthe complexities of modern day systems and the extent to which they are rapidly outpacing the capabilities of our methodological toolkit.In addition, new concepts introduced to better deal with the increasingly complex nature of modern day systems, such as Safety II (Hollnagel, Leonhardt, Licu and Shorrock, 2013), require appropriate methodological support.
The aim of this paper is to explore this by examining the contemporary ergonomics problem space andthe extent to which ergonomics methods can meet the challenges posed. We discussfivekey areas within the highly popular contemporary ergonomics paradigm of systems thinking, when applied to accident analysis and prevention activities, and examinethe ability of existing ergonomics methods to respond to them: normal performance as a cause of accidents (Dekker, 2011; Leveson, 2004; Rasmussen, 1997), accident prediction (Salmon et al, 2014a), system migration (Rasmussen, 1997), systems concepts (Hutchins, 1995; Stanton et al, 2006), and ergonomics in design (Read, Salmon, Lenne & Jenkins,in press).Where our methods are deemed to be lacking, new methodological requirements and capabilities are identified. Whilst we acknowledge that other disciplines and areas of safety science (e.g. resilience, safety II) may possess alternative methodologies that fulfil some of the requirements discussed, the focus of this article is specifically on the ergonomics methods used by ergonomics researchers and practitioners. In line with the topic of the special issue, we see methodological extension, development and integration as an omnipresent issue for our discipline.
Normal performance as a cause of accidents
Systems thinking and itsmethodological implications
Significant progress has been made in understanding accident causation in safety critical systems. Systems models, in particular, are now widely accepted (Leveson, 2004; Rasmussen, 1997) and there are a range of methods that enable accidents to be analysed from this perspective (e.g. Hollnagel, 2012; Svedung and Rasmussen, 2002; Leveson, 2004). This approach has a long legacy in safety science, from the foundational work of Heinrich (1931) through to the evolution of a number of more recent accident causation models and analysis methods (e.g. Leveson, 2004; Perrow, 1984; Rasmussen, 1997; Reason, 1990). Accidents are now widely acknowledged to be systems phenomena, just as safety is (Hollnagel, 2004; Dekker, 2011). Both safety and accidents therefore are emergent properties arising from non-linear interactions between multiple components distributed across a complex web of human and machine agents and interventions (e.g. Leveson, 2004).
It is precisely this form of thinking, and the evolution of it, that brings the methods we use into question. Despite the great progress in safety performance that has been made in most safety critical sectors since theSecond World War,significant trauma still occurs and in some areasprogress may be slowing. Whilst Figure 1 shows a plateauing effect in commercial air transport, data shows a similar trend in otherproblem areassuch as rail level crossings (Evans, 2011). Moreover, in areas such as road transport where the intensity of operations are increasing the global burden is increasing and is projected to increase significantly (WHO, 2014). Leveson (2011) suggests that little progress is now being made and suggests that one reason is that our methods do not fully uncover the underlying causes of accidents.Part of the issue may be that the evolution in accident causation models is not reflected in current accident analysis methods.Another issue that could conceivably play a part is the well document research-practice gap, whereby practitioners continue to use older methodologies that do not reflect contemporary models of accident causation (Salmon et al, In Press; Underwood and Waterson, 2013).
Figure 1 – The pattern of global passenger fatalities per 10 million passenger miles on scheduled commercial air transport since 1993. The graph shows that the precipitous drop in fatality rates since 1945 has, since 2003, levelled off (Source: EASA, 2010).
One of thefundamental advances provided by state of the artmodels centres around the idea that the behaviours underpinning accidents do not necessarily have to be errors, failures or violations (e.g. Dekker, 2011; Leveson, 2004; Rasmussen, 1997). As Dekker (2011) points out, systems thinking is about how accidents can happen when no parts are broken. Normal performance’ plays a role too (Perrow, 1984).This provides an advance over popular models that tend to subscribe to the idea that failure leads to failure (e.g. Reason, 1990). Reason did note that latent conditions can emerge from normal decisions and actions, but to take this idea much further is to describe two key tenets. First, normal performance plays a role in accident causation, and second, accidents arise from the very same behaviours and processes that create safety. In his recent drift into failure model, Dekker (2011) argues that the seeds for failure can be found in “normal, day-to-day processes” (pg. 99) and are often driven by goals conflicts and production pressures. These normal behaviours include workarounds, improvisations, and adaptations (Dekker, 2011), but may also just be normal work behaviours routinely undertaken to get the job done. It is only with hindsight and a limited investigation methodology that these normalbehaviours are treated as failures. Both safety boundaries and behaviour can drift: what is safe today may not be safe tomorrow. It is notable in the Kegworth aviation accident in the UK, the pilots shut down of the left engine would have been the correct action on the previous generation of aircraft for which they were most familiar (Griffin et al, 2010; Plant and Stanton, 2012).
Theoretical advances such as this have important implications for the methodologies applied to understand accidents. We require appropriate methodologies that reflect how contemporary models think about accident causation. The tenets described above provide an interesting shift in the requirements foraccident analysis methodologies. Dekker (2014)argues that practitioners should not look for the known problems that appear in incident reporting data or safety management systems. Instead, he argues, that the focus should be in the places where there are no problems,in other words, normal work. In addition, the burgeoning concept of Safety II I (Hollnagel et al, 2013) argues that safety management needs to move away from attempting to ensure that as little as possible goes wrong to ensuring that as much as possible goes right. A key part of this involves understanding performance when it went right as well as when it went wrong.
This raises critical questions – do our accident analysis methodologies have the capability to incorporate normal performance into their descriptions of accidents? Do we currently incorporate normal performance into accident analyses? And if we do, are we misclassifying it as errors, failures, and inadequacies? Further, should we be investigating and analysing accidents at all or putting those efforts into auditing everyday work providing an opportunity to continuously understand and manage performance variability without waiting for major accidents to occur? If so, do we have appropriate methods to support this?
Accident analysis methods
According to the literature the most popular accident analysis methods are Accimap (Rasmussen, 1997), STAMP (Leveson, 2004) and HFACS (Wiegmann and Shappell, 2003). Accimap accompanies Rasmussen’s now popular risk management framework and is used to describe accidents in terms of contributory factors and the relationships between them. This enablesa comprehensive representation of the network of contributory factors involved. It does this by decomposing systems into six levels across which analysts place the decisions and actions that enabled the accident in question to occur (although the method is flexible in that the number of levels can be adjusted based on the system in question). Interactions between the decisions and actions are subsequently mapped onto the diagram to show the relationships between contributory factors within and across the six levels. A notable feature of Accimap is that it does not provide analysts with taxonomies of failure modes; rather, analysts have the freedom to incorporate any factordeemed to have played a role in the accident in question.
The Systems Theoretic Accident Model and Process method (STAMP) views accidents as resulting from the inadequate control of safety-related constraints (Leveson, 2004), arguing that they occur when component failures, external disturbances, and/or inappropriate interactions between systems components are not controlled (Leveson, 2004; 2011). STAMP uses a ‘control structure’ modelling technique to describe complex systems and the control relationships that exist between components at the different levels. A taxonomy of control failures is then used to classify the failures in control and feedback mechanisms that played a role in the incident under analysis. An additional component of STAMP involves using systems dynamics modelling to analyse system degradation over time. This enables the interaction of control failures to be demonstrated along with their effects on performance.
Although not based on contemporary models of accident causation, the Human Factors Analysis and Classification System (HFACS; Wiegmann and Shappell, 2003) remains highly popular (e.g. Daramola, 2014; Mosaly et al, 2014). HFACS is a taxonomy-based approach that provides analysts with taxonomies of error and failure modes across four system levels based on Reasons Swiss cheese model of organizational accidents: unsafe acts, preconditions for unsafe acts, unsafe supervision, and organizational influences. Although developed originally for use in analysing aviation incidents, the method has subsequently been redeveloped for use in other areas including: mining (Lenné, Salmon, Liu & Trotter, 2012), maritime (Chauvin, Lardjane, Morel, CLostermann and Langard, 2013), rail (Baysari, McIntosh & Wilson, 2008) and healthcare (El Bardissi, Wiegmann, Dearani, Daly, and Sundt, 2007). Later versions of the method have extended the levels to incorporate an ‘external influences’ level which considers failures outside of organisations such as legislation gaps, design flaws, and administration oversights (Chen et al, 2013).
Accident analysis methods and normal performance
A notable shortfall of the latter two methodsis their focus on abnormal behaviours or failures.Both HFACS and STAMP provide taxonomies of error and failure modes that are used to classify the behaviours involved in accident scenarios, which in turn means that there is little scope for analysts to include behaviours other than those deemed to have been failures of some sort.There is no opportunity for analysts to incorporate normal behaviours in their descriptions of accidents – they have to force fit events into one of the error or failure modes provided. The output is a judgment on what errors or failures combined to create the accident under analysis. Whilst this is inappropriate given current knowledge on accident causation, a worrying consequence may be that the normal behaviours that contribute to accidents are not picked up during accident analysis efforts. This may impact accident prevention activities by providing a false sense of security that nothing else is involved and thus nothing needs fixing (apart from error producing human operators). A more sinister implication is that organisations who apply methods such as HFACS may not develop a sufficient understanding of accidents to prevent them. Although the aviation sector routinely monitor normal performance through flight data monitoring systems, arguably they do not run analyses of the role of normal performance in air crashes. Extending methods such as HFACS and STAMP to incorporate analyses of normal performance in accidents is therefore a pressing requirement. The benefits include developing a more holistic view of accident causation that is not entirely based on understanding errors and failures and understanding how normal behaviours lead to system failure.
Accimap, on the other hand, does not use a taxonomy of failure or error modes and so enables analysts to incorporate normal performance and to show its relationship with other behaviours. There isfreedom for analysts to include any form of behaviour in the network of contributory factors. Despite this, Accimap descriptions still tend to incorporate many contributory factors prefixed with descriptors such as ‘failure to’, ‘lack of’ or ending with ‘error’. A pressing question here then is the extent to which the failures described in Accimap analyses actually represent failure or are in fact normal behaviours. Salmon et al (2015) recently examined a sub-set of their own analyses and found examples where contributory factors originally described as failures could be reclassified as normal performance.