Improving prognostic accuracy in subjects at clinical high risk for psychosis: systematic review of predictive models and meta-analytical sequential testing simulations
Supplementary Material
32
Supplementary Methods
1. Mathematical details of statistical probabilistic simulations
1.1 Probability of transition to psychosis after knowing the result of one test
The probability of transition (T) to psychosis after knowing that the result of test A (e.g. the CHR assessment) is positive, known as positive predictive value (PPV), is a function of the sensitivity and specificity of test A and the probability of transition before knowing the result of the test:
where p indicates probability, Se sensitivity, and Sp specificity.
The probability of transition after knowing that the result of test A is negative, known as negative predictive value, is also a function of the sensitivity and specificity of the test and the pre-test probability of transition:
Note that for simplicity, these applications of Bayes rule may be expressed using odds1:
and similarly:
where LR+A is the positive likelihood ratio of test A and LR-A is the negative likelihood ratio of the test.
1.2 Probability of transition after knowing the result of several tests
The aforementioned application of Bayes rules may be stepwise combined. The odds of transition after knowing the results of tests A and B, for example, may be derived from the odds of transition after knowing the result of test A and the likelihood ratios of test B, and so on1:
and etc.
1.3 Lower bound of the confidence interval for a test
The lower bound of the confidence interval associated with a test was estimated by first estimating the lower bounds of the binomial-based confidence intervals for the sensitivity and the specificity, and second using these lower bounds to derive the likelihood ratios as specified above.
1.4 Lower bound of the confidence interval for global PPV
In order to obtain a global c (=95%) confidence interval for the global PPV, we created narrower c’ (<95%) confidence intervals for the pre-CHR test probability of transition and for the sensitivities and specificities of the CHR assessment and the 3 tests so that the combination of the lower bounds of the five c’ confidence intervals approached the lower bound of the global c confidence interval.
In the following we describe the derivation of the formula needed to estimate the confidence (c’) of the intervals for the pre-CHR and for the post-test probabilities of transition to psychosis. Please note that for simplicity this derivation assumes that the c' confidence intervals were created using independent and identically normally-distributed variables, and that the c confidence interval was created using the sum of these variables, which is only an approximation.
The formula for the half width of a confidence interval, known as margin of error (ME), is:
where probit is the quantile function of the standard normal distribution and σ2 is the variance.
Thus, the margin of error of the c and c' confidence intervals are:
Note that under the general assumptions of this approximation, the normal variable of the c confidence interval is the sum of the five independent normal variables of the c' confidence intervals so that the variance of the former is the sum of the variances of the latter:
Also, given that the combination of the lower bounds of the former should approach the lower bound of the later, the ME of the later can be written as the sum of the MEs of the former:
We can now isolate c’:
where Φ is the standard normal distribution.
1.5 Sensitivity and specificity of a test conditioned to the results of the previous tests
Sensitivity of test B, conditioned to a positive result in test A, is:
where σA|T is the standard deviation of the result of test A in individuals who will have a psychotic episode, σB|T is the standard deviation of the result of test B in the same individuals, and σA,B|T is the covariance of the results of tests A and B in these individuals.
Similarly, sensitivity of test C, conditioned to a positive result in tests A and B:
Finally, sensitivity of test D, conditioned to a positive result in tests A, B and C:
For simplicity, in this study we have considered that the correlation between any pair of tests is always ρ.
32
2. Improving positive predictive value with complementary assessments
We focused on the combination that yielded the best PPV for both two and three complementary positive tests, as this would be the one to be further validated and potentially applied in clinical practice.2 There are three different options possible to achieve two complementary positive tests: 1) the first and second test are positive, 2) the first and third test are positive, or 3) the second and third are positive. All options result indifferent PPVs and therefore we focused on both the average of the three PPVs, as well as on the minimum of the three PPVs.
32
Supplementary Results
Supplementary Figure 1. Meta-analytical sequential testing simulations diagrams for all 13 combinations of studies.
32
Supplementary Table 1. Number needed to treat in the bounds of the confidence interval of the risk ratio for preventative treatments.
Number needed to treatRR=0.54 / RR=0.34
(lower CI bound) / RR=0.86
(upper CI bound)
Positive CHR assessment and...
... three positive tests / 2 / 2 / 7
... two positive tests / 3 / 2 / 9-10
... one positive tests / 11-18 / 7-13 / 35-60
... no positive tests / 219 / 153 / 719
Negative CHR assessment / 147 / 102 / 483
CI: confidence interval. RR: risk ratio for preventive treatments.
Supplementary Figure 2. Meta-analytical sequential testing simulations diagram using the lower bound of the confidence intervals for the pre-test probability and the sensitivity / specificity of the three-stage tests.
Supplementary Figure 3. Meta-analytical sequential testing simulations diagram assuming correlation between the tests.
32
Supplementary Table 2. Meta-analytical sequential testing simulations: combined probability of transition to psychosis during 36 months of follow-up in individuals with attenuated psychotic symptoms (APS), brief (limited) intermittent psychotic symptoms (BLIPS / BIPS) and genetic risk and deterioration syndrome (GRD).
Probability of transition to psychosisAPS / BLIPS / BIPS / GRD
+ / 99.0% (high) / 99.5% (high) / 95.0% (high)
+, and blood markers
- / 87.6% (high) / 94.2% (high) / 58.3% (medium)
+, and sMRI
+ / 84.1% (high) / 92.4% (high) / 51.2% (medium)
-, and blood markers
- / 28.2% (medium-low) / 47.5% (medium) / 7.2% (low)
+, and Clinical/EEG
+ / 78.5% (medium-high) / 89.4% (high) / 42.0% (medium)
+, and blood markers
- / 21.3% (low) / 38.5% (medium) / 5.1% (low)
Positive CHR test / -, and sMRI
+ / 16.9% (low) / 32.0% (medium) / 3.9% (low)
-, and blood markers
- / 1.5% (low) / 3.4% (low) / 0.3% (low)
- / 2.2% (low) / 5.0% (low) / 0.4% (low)
CHR: clinical high risk; EEG: electroencephalography; sMRI: structural magnetic resonance imaging. As suggested by Clark et al 1, risk was defined as high when probability of transition was >80%, medium when it was 20-80%, and low when it was <20%. Risks between 70% and 80% are further labelled as medium-high, and risks between 20% and 30% as medium-low.
32
Supplementary Figure 4. Meta-analytical sequential testing simulations diagram for individuals with attenuated psychotic symptoms (APS), brief (limited) intermittent psychotic symptoms (BLIPS / BIPS) and genetic risk and deterioration syndrome (GRD).
References
1. Clark SR, Schubert KO, Baune BT. Towards indicated prevention of psychosis: using probabilistic assessments of transition risk in psychosis prodrome. J Neural Transm (Vienna) 2015;122:155-169.
2. Fusar-Poli P, Cappucciati M, Rutigliano G, et al. At risk or not at risk? A meta-analysis of the prognostic accuracy of psychometric interviews for psychosis prediction. World Psychiatry 2015;14:322-332.
32