credibility [agenda]
worksheet type A
Risk assessment
/ Credibility judgementRef/
Loc / Aspect/
Part / Facet/
Norm / Approval/
Snag
…… / Introduction describes how the assessment arose from a public inquiry / Justified / Provides a reason for taking the assessment seriously
Scope of the assessment defined in terms of types of risk, classes of system and cause of hazard / Complete / Laying out scope systematically, like this, in terms of several dimensions helps with completeness. But also raises the question whether boundaries artificially limit the risks being considered
…… / Statement of the risk model being developed in line with generally accepted guidelines in the industry / Standardised / Using standard methods promotes credibility, although the assessment doesn’t describe the limitations or restrictions associated with the method
…… / Description of how the Hazid led to range of potential accidents, and how the quantitative risk analysis concentrated on generic hazard categories leading to loss of life / Proportionate / Quantification was directed at the worst outcomes, which seems like a good rule of thumb
…… / Description of the operations in question / Knowledgeable, Grounded;
Justified / Describing the operations being assessed provides context and demonstrates knowledge of context. Maybe more could be said about the origins of this knowledge, and how it is likely to change in the future
…… / Reference to a preceding Hazid of the same topic / Justified, Grounded / This provides a sense that this analysis is part of a larger process. Shows some useful grounding.
Categorisation of the identified hazards / Complete, Accessible / Categorising hazards helps give reassurance about completeness, as missing categories ought to be easier to spot than missing specific hazards. It also makes the assessment easier to digest
…… / Description of data collection, including questionnaires, meetings and first-hand observations / Grounded, Knowledgeable,
Integrated, Consultative / This helps the audience to know how dependable the data is, helps give some reassurance that there was consultation, helps give reassurance the methods were scientific, and shows there was some first-hand observation (although it perhaps raises the question as to how representative the sample was)
…… / Description of the development of the risk model / Standardised / Use of a broadly standardised approach helps confer credibility
Description of how fault trees were used to find top event frequencies in each of the main hazard categories, how the main data sources were used and augmented where augmenting data was available, and briefly how gaps in the data were dealt with / Unbiased, Complete, Grounded, Calibrated, Explicit, Honest / Provides an acknowledgement of deficiencies in data and explains measures to cope with this. The use of multiple approaches and testing for robustness helps give reassurance. Doesn’t say explicitly what the problems might be in judging probabilities, however. If the authors’ technical pedigree can be taken as read, the main effort should be devoted to helping the audience understand what can and can’t be done with data and judgment
Admission that limited data was available on major accidents, leading to use of information from a particular accident and ‘general data’ (which is unspecified) / Grounded, Explicit,
Accessible / Again acknowledges limitations in data, and the use of ‘general data’ from which specific estimates can presumably be derived. But it’s perhaps important also to say what’s good about general data (helps you avoid thinking only about narrow classes of event) and bad (doesn’t make distinctions that could be very important). See norms 114 and 115 especially.
Summation of annual fatalities to yield an individual risk of fatality for different categories of person in each operating environment, and an individual risk of personal accident / Grounded, Directed / This aggregation obscures different perceptions of different ways of dying. Maybe this is hard to deal with, but maybe an acknowledgement is important.
…… / Use of influence diagrams / Accessible / These probably assist communication by helping convey the model (and its assumptions) holistically. They also help give the impression of completeness. (How well they can be ‘quantified’ is perhaps an open issue.)
Acknowledgement that some effects in the influence diagram could not be quantified, and recognition that these effects are material / Complete, Honest / This helps show that you’ve avoided the trap that only the quantifiable is taken to be relevant. But it also brings into question the realism and usefulness of producing an overall risk number.
…… / Report of a meeting that decided on risk reducing measures and acknowledgement that potential measures will influence many areas of the risk model; tabulation of likely qualitative effects of risk reducing measures / Iterative / This acknowledges the ‘reflexivity’ problem (knowing the risks changes the risks), but says nothing about the migration problem (new controls introduce new risks) or the compensation problem (people will behave differently)
…… / Results of the quantitative risk assessment, expressed on an ‘annual basis travelling on every voyage ie … a passenger travelling on 3690 trips per year’ / Accessible, Directed, Guiding / This presents a highly aggregated risk number that would lack meaning for most stakeholders. What about comparing it with everyday life risks, looking at voluntarism, controllability and so forth? Maybe this was what was asked for but it doesn’t lend credibility
…… / Argument that these numbers allow comparison of safety performance in specific instances / Directed, Explicit / Directing the analysis specifically at comparing the performance of specific vessels looks appropriate (norm 122). But there is perhaps a tacit assumption that existing risk levels are an appropriate standard. If you’re going to give the status quo a special status you need to say so. Some regulators would argue you shouldn’t. See norm 140.
Admission that certain complexities were identified but excluded because little data was available; also and admission that rare incidents such as contact with low flying aircraft have been omitted. / Complete, Honest, Guiding, Grounded / Acknowledgement of limitations plainly important. But does seem to commit the fallacy of leaving out what is unquantifiable rather than what is irrelevant. It therefore remains unclear what significance the outcome numbers have. The suggestion is that they are pessimistic, but ‘complexities’ may include unfavourable effects as well as favourable ones.
…… / Question set used in data gathering / Grounded / Appending questionnaires helps demonstrate how systematic the study was. But not clear what steps were taken to deal with bias eg due to under-reporting of certain kinds of events
…… / Definitions and frequency arguments given for the main categories / Grounded / The basis for the allocated frequencies is described in detail
Fault trees and influence diagrams / Integrated / Not very clear how the influence diagrams and event trees are combined – or how the influence diagrams influence the final numbers
JUDGMENT
inadequate: there are important aspects of credibility that do not seem to be satisfied
indeterminable: it is not possible to make an overall judgment of credibility
reduced: there are some aspects of credibility that are not fully satisfied
incomplete: there are aspects that contribute to credibility that are unknown
temporary: there are aspects contributing to credibility that might not exist in the future
contingent: there are aspects of credibility that depend on how things turn out
high: there is little of significance to detract from the assessment's credibility /