AppendixBayes factor and the impact of technical errorsin laparoscopic colorectal surgery assessments
Melody Ni
In this appendix, we explain the logic behind usingBayes factor (BF) to assess the impact of a technical error on the sign off assessments within the National Training Programme of Laparoscopic Colorectal Surgeries (NTP).
Belief updating and the Bayes’ rule
We conceptualise the sign-off process within NTP as the process of belief updating, whereby “information is received a piece at a time and integrated into a continuously evolving impression” ([1], p.144). The impression, in our study, corresponds to the subjective belief an NTP assessor forms over the safety of the operation. The information corresponds to a technical error the assessor identified because he or she believed the error was important for safety. We further hypothesised that 1) this belief was sequentially adjusted as the assessor identified an increasing number of technical errors and 2) the updated belief was fundamental to his or her decision over whether to pass the sign-off so that the delegate could embark upon independent practices outside NTP.
The normative framework to capture this belief updating process is the Bayes’ rule [2]. Within this framework, the impression is the hypothesis (H) that the operation was safe; the information is the data(D) that a technical errorhad been identified. A higher probability corresponds to a stronger belief and vice versa. The Bayes’ rule has the following form:
P(D|H) =P(H)P(H|D)/P(D),-(1)
where P(H) is the prior probability or one’s initial belief that the surgery is safe before observing any error; P(D|H) is the conditional probabilityof an error being observed in successful sign offs; P(H|D) is the posterior probability or one’s updated belief that the surgery is safe following identification of an error. P(D) describes how likely the error will be observed regardless of whether or not the surgery is safe. As errors are observed one at a time Bayes’ rule can be applied repeatedly to capture the iterative process of belief updating as more and more errors were identified.
Likelihood ratios and the Bayes’ factor
Since we only considered two possible outcomes of a sign off, pass or fail, a more useful form of Bayes’ rule is the so-called “odds-likelihood ratio” formulation, used to distinguishbetween two mutually exclusive hypotheses, H(successful sign offs for safe surgery performed) and H1 (otherwise, P(H1) = 1-P(H)), based on observationof error (D):
Posterior odds = Prior odds x Likelihood ratios, - (2)
where posterior odds = P(H|D)/(1-P(H|D)), prior odds = P(H)/(1-P(H)), and likelihood ratios = P(D|H)/(1-P(D|H)).The biggest benefit of using the likelihood ratio form instead of (1) is that there is no need to consider the prevalence of error, i.e. P(D).
Bayes’ rule establishes a direct link between one’s subjective beliefs(measured in probability odds) and evidence. The impact of the evidence is measured by the likelihood ratio that compares how likely the data is observed in successful (P(D|H)) versus in unsuccessful sign offs (1-P(D|H)).Bayes’ factor, in its simplest forms, equates the likelihood ratio.
Error and sign off decision
Identification of errors signals lack of safety and decreases the posterior odds of safe surgery. This in turn reduces the chance that assessor would grant a pass to the sign-off submission. Following this logic we can measure the impact of an error on sign off decisions by re-writing Equation 2 into:
Posterior odds of safe sign off being successful
= Prior odds of sign off being successful x Bayes Factor of an error, (3)
The more critical an error the smaller its Bayes’ factor will be and vice versa. Further under the assumption of conditional independence, the Bayes’ framework (Eq. 1 2) dictates that the impact of multiple errors (data) can be measured simply by the multiplication of their respective likelihood ratios. Suppose n number of errors were observed:
Posterior odds of sign off being successful
= Prior odds of sign off being successful x Bayes Factor of error1x Bayes Factor of error2…x Bayes Factor of error1x Bayes Factor of errorn, (4)
Example – how to use Bayes’ factor to estimate sign off success (as if you were an expert assessor)
- Assume a prior probability that the sign off would be successful before observation of any error. We have used 90%in our analysis. This is equivalent to a prior odds of 9 = (0.9/(1-0.9).
- From Table 3 (main text) find out the Bayes’ factor of a significant error in the operative video. For instance the transection error has a Bayes’ factor of 0.20.
- Apply Equation 3 to get the updated probability of successful sign off.
-Posterior odds = Prior odds x Bayes’ factor
-Posterior odds = 9 * 0.2 = 1.8
-Posterior probability = posterior odds/(1+posterior odds) = 1.8/2.8 = 64.3%
Therefore a significant breach in transection would reduce the chance of being signed off from 90% to 64.3%, which is still over the 50% benchmark.
When multiple errors were observed:
Suppose excessive bleeding (BF=0.42) has also been observed.
- Repeat the above process but use the posterior odds as the new prior odds.
-Posterior odds = Prior odds x Bayes’ factor = 1.8 * 0.42 = 0.756
Posterior probability given transection error and excessive bleeding is therefore 43% = (0.756/1.756), in which case the sign off is more likely to be a failure rather than a success (<50%).
Note this is equivalent to estimate the joint BF for the two errors first and then apply Eq 4.
-Joint BF of transection error and excessive bleeding = 0.2*0.42=0.084
-Posterior odds = Prior odds x Bayes’ factor = 9 * 0.084 = 0.756
References:
1.Anderson NH (1981) Foundations of information integration theory, Academic Press, New York
2.Bayes T (1763) An essay towards solving a problem in the doctrine of chances. Phil Trans 53:370-418
1