Supplemental Table 1. Simulation study of ROC analysis methods.
Overall Effect / R(x1,x2) / Model / HRR1 / HRR2 / Area1 / Area2 / ΔArea / ZModest / 0.10 / HRR1=HRR2 / 1.25 (0.11) / 1.26 (0.12) / 0.744 (0.020) / 0.744 (0.020) / 0.000 (0.006) / 0.06 (1.37)
0.10 / HRR2>HRR1 / 1.04 (0.09) / 1.48 (0.13) / 0.734 (0.019) / 0.751 (0.018) / 0.017 (0.008) / 3.27 (0.94)
Strong / 0.10 / HRR1=HRR2 / 1.55 (0.17) / 1.55 (0.17) / 0.781 (0.021) / 0.781 (0.021) / 0.000 (0.012) / 0.01 (1.51)
0.10 / HRR2>HRR1 / 1.08 (0.10) / 2.17 (0.21) / 0.743 (0.019) / 0.800 (0.017) / 0.057 (0.013) / 6.12 (0.92)
Modest / 0.50 / HRR1=HRR2 / 1.35 (0.12) / 1.35 (0.12) / 0.748 (0.019) / 0.748 (0.019) / 0.000 (0.006) / 0.03 (1.40)
0.50 / HRR2>HRR1 / 1.22 (0.10) / 1.48 (0.13) / 0.746 (0.018) / 0.753 (0.018) / 0.012 (0.006) / 2.39 (1.08)
Strong / 0.50 / HRR1=HRR2 / 1.81 (0.19) / 1.81 (0.19) / 0.786 (0.020) / 0.786 (0.020) / 0.000 (0.010) / -0.01 (1.41)
0.50 / HRR2>HRR1 / 1.44 (0.13) / 2.16 (0.21) / 0.757 (0.018) / 0.799 (0.017) / 0.042 (0.011) / 5.28 (1.05)
Modest / 0.90 / HRR1=HRR2 / 1.47 (0.12) / 1.47 (0.12) / 0.750 (0.018) / 0.750 (0.018) / 0.000 (0.003) / -0.01 (1.42)
0.90 / HRR2>HRR1 / 1.42 (0.12) / 1.48 (0.13) / 0.747 (0.019) / 0.750 (0.019) / 0.003 (0.003) / 1.38 (1.36)
Strong / 0.90 / HRR1=HRR2 / 2.11 (0.21) / 2.11 (0.21) / 0.795 (0.018) / 0.795 (0.018) / 0.000 (0.006) / 0.01 (1.47)
0.90 / HRR2>HRR1 / 1.98 (0.19) / 2.16 (0.21) / 0.789 (0.018) / 0.799 (0.018) / 0.010 (0.006) / 2.51 (1.34)
Data are mean (SD); Models where the overall effect is “modest” are simulated so that the joint effect (hazard rate ratio) of the two risk factors (x1 and x2) is ~1.5 per SD, while those where the overall effect is “strong” have hazard rate ratio ~2.2 per SD. R(x1,x2) is the correlation between x1 and x2; HRR1 is the hazard rate ratio per SD of x1; HRR2 is the hazard rate ratio per SD for x2; Area1 is the area under the ROC curve with the present method for x1; Area2 is the area under the ROC curve for x2; ΔArea is Area2–Area1; Z is the test statistic (ΔArea divided by its standard error). Results represent analysis of 1000 replicates for each model; data were generated assuming effects of baseline age, sex and heritage and including these as covariates in addition to x1 and x2.
Supplemental Table 2. Permutation analysis for pair-wise comparisons for Akaike’s Information Criterion (AIC) and ROC AUC by age-groups.
BMI / Waist / Fast gluc. / 2hr gluc / Fast ins / HbA1c / HDL-C / Trigs / SBP / DBP
AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE)
AIC / 0.85 / (0.04) / 0.88 / (0.03) / 0.63 / (0.05) / 0.77 / (0.05) / 0.70 / (0.06) / 0.68 / (0.05 / 0.75 / (0.05) / 0.71 / (0.05) / 0.66 / (0.06) / 0.65 / (0.06)
BMI / 184.4 / 0.17 / <0.01 / 0.11 / <0.01 / 0.01 / 0.03 / <0.01 / <0.01 / <0.01
Waist / 183.8 / 0.81 / <0.01 / 0.06 / <0.01 / <0.01 / 0.02 / <0.01 / <0.01 / <0.01
Fast gluc / 228.1 / <0.01 / <0.01 / 0.02 / 0.31 / 0.46 / 0.08 / 0.24 / 0.64 / 0.78
2hr gluc / 210.6 / 0.02 / 0.03 / 0.03 / 0.20 / 0.20 / 0.63 / 0.25 / 0.14 / 0.05
Fast ins / 221.3 / 0.02 / <0.01 / 0.37 / 0.23 / 0.84 / 0.41 / 0.86 / 0.57 / 0.49
HbA1c / 224.1 / <0.01 / <0.01 / 0.59 / 0.22 / 0.78 / 0.41 / 0.70 / 0.74 / 0.59
HDL / 212.6 / <0.01 / 0.03 / 0.05 / 0.83 / 0.33 / 0.27 / 0.42 / 0.08 / 0.13
Trigs / 216.0 / <0.01 / <0.01 / 0.13 / 0.54 / 0.53 / 0.38 / 0.62 / 0.39 / 0.42
SBP / 223.5 / <0.01 / <0.01 / 0.45 / 0.28 / 0.81 / 0.93 / 0.16 / 0.39 / 0.83
DBP / 224.8 / <0.01 / <0.01 / 0.60 / 0.13 / 0.74 / 0.94 / 0.17 / 0.42 / 0.84
10-14yrs
BMI / Waist / Fast gluc. / 2hr gluc / Fast ins / HbA1c / HDL-C / Trigs / SBP / DBP
AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE)
AIC / 0.63 / (0.05) / 0.64 / (0.04) / 0.64 / (0.05) / 0.74 / (0.04) / 0.63 / (0.05) / 0.68 / (0.04) / 0.62 / (0.05) / 0.55 / (0.05) / 0.54 / (0.05) / 0.53 / (0.05)
BMI / 499.6 / 0.45 / 0.92 / 0.06 / 0.90 / 0.26 / 0.73 / <0.01 / <0.01 / <0.01
Waist / 497.4 / 0.47 / 0.96 / 0.11 / 0.72 / 0.36 / 0.52 / <0.01 / <0.01 / <0.01
Fast gluc / 500.2 / 0.97 / 0.86 / 0.03 / 0.81 / 0.44 / 0.71 / 0.03 / 0.03 / <0.01
2hr gluc / 467.4 / 0.06 / 0.10 / 0.03 / 0.02 / 0.34 / 0.07 / <0.01 / <0.01 / <0.01
Fast ins / 504.9 / 0.61 / 0.48 / 0.66 / <0.01 / 0.22 / 0.85 / 0.01 / 0.03 / <0.01
HbA1c / 495.1 / 0.73 / 0.85 / 0.76 / 0.10 / 0.43 / 0.25 / <0.01 / <0.01 / <0.01
HDL / 510.2 / 0.31 / 0.19 / 0.42 / 0.02 / 0.62 / 0.29 / 0.01 / <0.01 / <0.01
Trigs / 519.9 / 0.01 / <0.01 / 0.06 / <0.01 / 0.07 / <0.01 / 0.16 / 0.57 / 0.10
SBP / 525.8 / <0.01 / <0.01 / 0.03 / <0.01 / 0.03 / 0.01 / 0.05 / 0.40 / 0.23
DBP / 528.3 / <0.01 / <0.01 / <0.01 / <0.01 / <0.01 / <0.01 / <0.01 / 0.06 / 0.20
15-19yrs
BMI / Waist / Fast gluc. / 2hr gluc / Fast ins / HbA1c / HDL-C / Trigs / SBP / DBP
AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE)
AIC / 0.64 / (0.04) / 0.65 / (0.04) / 0.66 / (0.04) / 0.68 / (0.04) / 0.61 / (0.04) / 0.66 / (0.04) / 0.63 / (0.04) / 0.63 / (0.04) / 0.58 / (0.04) / 0.59 / (0.04)
BMI / 561.2 / 0.63 / 0.63 / 0.40 / 0.29 / 0.58 / 0.76 / 0.67 / <0.01 / 0.08
Waist / 561.8 / 0.91 / 0.74 / 0.47 / 0.20 / 0.69 / 0.65 / 0.57 / 0.01 / 0.06
Fast gluc / 560.3 / 0.95 / 0.91 / 0.64 / 0.10 / 0.91 / 0.40 / 0.40 / <0.01 / 0.04
2hr gluc / 553.8 / 0.64 / 0.60 / 0.66 / 0.04 / 0.73 / 0.22 / 0.12 / <0.01 / 0.01
Fast ins / 572.8 / 0.19 / 0.21 / 0.18 / 0.08 / 0.11 / 0.45 / 0.56 / 0.12 / 0.41
HbA1c / 567.0 / 0.61 / 0.68 / 0.56 / 0.38 / 0.55 / 0.39 / 0.38 / <0.01 / 0.05
HDL / 572.7 / 0.33 / 0.35 / 0.28 / 0.13 / 0.99 / 0.59 / 0.91 / 0.02 / 0.16
Trigs / 569.3 / 0.50 / 0.55 / 0.49 / 0.17 / 0.70 / 0.88 / 0.70 / 0.03 / 0.12
SBP / 584.6 / <0.01 / <0.01 / <0.01 / <0.01 / 0.11 / 0.03 / 0.04 / 0.04 / 0.38
DBP / 580.3 / 0.06 / 0.06 / 0.06 / 0.02 / 0.36 / 0.18 / 0.36 / 0.18 / 0.35
5-19yrs
BMI / Waist / Fast gluc. / 2hr gluc / Fast ins / HbA1c / HDL-C / Trigs / SBP / DBP
AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE) / AUC / (SE)
AIC / 0.68 / (0.03) / 0.68 / (0.03) / 0.66 / (0.03) / 0.72 / (0.03) / 0.64 / (0.03) / 0.65 / (0.03) / 0.64 / (0.01) / 0.61 / (0.03) / 0.58 / (0.03) / 0.58 / (0.03)
BMI / 1102.8 / 0.42 / 0.48 / 0.22 / 0.17 / 0.30 / 0.19 / <0.01 / <0.01 / <0.01
Waist / 1101.0 / 0.76 / 0.38 / 0.34 / 0.11 / 0.23 / 0.11 / <0.01 / <0.01 / <0.01
Fast gluc / 1121.3 / 0.36 / 0.33 / 0.05 / 0.61 / 0.77 / 0.68 / 0.09 / <0.01 / <0.01
2hr gluc / 1076.8 / 0.28 / 0.31 / 0.03 / <0.01 / 0.03 / 0.03 / <0.01 / <0.01 / <0.01
Fast ins / 1122.4 / 0.21 / 0.17 / 0.93 / <0.01 / 0.86 / 0.94 / 0.16 / <0.01 / 0.02
HbA1c / 1136.3 / 0.05 / 0.05 / 0.39 / <0.01 / 0.39 / 0.93 / 0.21 / 0.01 / 0.02
HDL / 1134.2 / 0.05 / 0.03 / 0.43 / 0.01 / 0.39 / 0.88 / 0.11 / <0.01 / <0.01
Trigs / 1140.5 / 0.02 / <0.01 / 0.18 / <0.01 / 0.15 / 0.76 / 0.58 / 0.04 / 0.11
SBP / 1160.5 / <0.01 / <0.01 / <0.01 / <0.01 / <0.01 / 0.05 / 0.01 / 0.06 / 0.51
DBP / 1156.6 / <0.01 / <0.01 / 0.01 / <0.01 / <0.01 / 0.10 / 0.04 / 0.13 / 0.53
BMI = body mass index; Fast gluc = fasting glucose; 2hr gluc = 2hr hour glucose; Fast ins = fasting insulin; HDL = high density lipoprotein cholesterol; Trigs = triglycerides; SBP = systolic blood pressure; DBP = diastolic blood pressure. Numbers in italics are the p-values for the tests comparing the ROC AUC (above diagonal) or AIC (below diagonal). All p-values were calculated by randomly permuting the pair of predictor variables. Smaller values of AIC indicate a better fit (stronger prediction).
Methods for Table 2:
To place each pair of variables on a comparable scale, we first standardized them to have a mean of 0 and a standard deviation of 1. If z1 and z2 represent the pair of standardized variables, we next calculate Akaike’s Information Criterion (AIC) for models which contain either z1 or z2, along with age, sex and heritage. The variable which more strongly predicts diabetes is associated with a lower AIC and the difference (ΔAIC=|AIC(z1)–AIC(z2)|) is a measure of the magnitude of the extent to which the stronger predictor is superior. Under the null hypothesis that z1 and z2 are equally strong predictors of diabetes, their values are exchangeable; therefore we used a permutation procedure to assess the statistical significance of a given ΔAIC. A new variable, m, was randomly assigned to each individual and this variable can take values of 0 or 1 with equal probability. Two new variables z*1 and z*2 were then defined as follows:
z*1=z1, z*2=z2 if m=0,
z*1=z2, z*2=z1 if m=1
Proportional hazards models were then fit for z*1 and z*2 and the value ΔAIC* calculated as |AIC(z*1)–AIC(z*2)|. This procedure was repeated 1000 times and the proportion of replicates for which ΔAIC>ΔAIC* was taken as the p-value associated with the observed value of ΔAIC (shown below diagonal). The same procedure was used to obtain an empirical p-value for the difference in the area under the ROC curves (shown above diagonal).