Supplementary
Assessment of Model Performance
We used several criteria to compare the overall diagnostic values of alternative models.
Goodness-of-fit
How effectively a model describes the outcome variable is referred to as its goodness-of-fit. We used following measures of goodness-of-fit. Deviance (D statistic) compares the fit of the saturated model to the fitted model. This will be a small value if the model is good. For purposes of assessing the significance of non-linear terms, the values of D with and without the non-linear terms were compared by computing deviance difference (G statistic). Akaike information criterion (AIC) was used to account for complexity. Difference in AIC >10 was considered significant (1).
Discrimination
Discrimination refers to the ability to distinguish high risk subjects from low risk subjects, and is commonly quantified by a measure of concordance, the c statistic. For binary outcomes, In the survival analysis, discrimination, which is quantified by the Harrell's C statistic, is equivalent to the area under a receiver operating characteristic (ROC) curve for binary dependent variables (2). The Harrell's C statistic measures the probability that a randomly selected person who developed an event, at the certain specific time has a higher risk score than a randomly selected person who did not develop an event during the same specific follow-up interval (3-6). For Harrell’s Cstatistics bias-corrected 95%CIs were estimated with Bootstrap resampling.As a general rule, if ROC = 0.5, it suggests no discriminatory power, if 0.70 ≤ROC<0.80, this is considered acceptable discriminatory power, if 0.80 ≤ROC<0.90, this is considered excellent discriminatory power, if ROC≥0.90, this is considered outstanding discriminatory power (2, 7).
Calibration
Calibration, as it is phrased in reference (6) describes how closely predicted probabilities agree numerically with actual outcomes (7-11). A test very similar to the Hosmer-Lemeshow test has been proposed by Nam and D'Agostino(6) and others (12). We calculated the Nam-D'Agostino χ2 to examine calibration for prediction models(6). As suggested by D'Agostino and Nam, calibration chi-square values greater than 20 (P < 0.01) suggest lack of adequate calibration (6).
Added predictive ability
Traditionally, risk models have been evaluated by using the Harrell’s C statistic, but this method has not enough insensitive in comparing models and is of little direct clinical relevance. New methods have recently been proposed to evaluate and compare predictive risk models. Absolute and relative integrated discriminatory improvement index (IDI) and cut-point-based and cut-point-free net reclassification improvement index (NRI) were used as measures of predictive ability added by ABSI to each traditional anthropometric measure or the Framingham’s general CVD algorithm. Reclassification improvement is defined as an increase in probability category for patients with events and as a decrease for those without event. Net reclassification improvement considers movement between categories in the wrong direction and applies different weights to events and nonevents (13).
Nonlinear associations
Instead of using arbitrary predetermined cut-points, we used restricted cubic splines functions of the ABSI to represent its continuous relationship with the CVD so that the relationships were meaningfully in accordance with substantive background knowledge. As such, we were enabled to capture possible nonlinear associations. Restricted cubic splines function enabled us to flexibly model continuous predictor (ABSI) while allowing us some control over the excessive instability and tendency of spline functions to generate artifactual and uninterpretable features of a curve. Multivariate restricted cubic splines were used with 4 knots defined at, 5th, 25th, 75th, and 95th percentiles (14).
References
1.Akaike H. A new look at the statistical model identification. IEEE Trans Automat Contr. 1974;AC-19:716-23. Eng.
2.Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982 April 1982;143(1):29-36.
3.Hlatky MA. Exercise testing to predict outcome in patients with angina. J Gen Intern Med. 1999 Jan;14(1):63-5. PubMed PMID: 9893094. Pubmed Central PMCID: PMC1496444. Epub 1999/01/20. eng.
4.Harrell FE. Regression modeling strategies: Springer New York; 2001.
5.Harrell FE, Jr., Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996 Feb 28;15(4):361-87. PubMed PMID: 8668867. Epub 1996/02/28. eng.
6.D'Agostino RB, Nam BH. Evaluation of the performance of survival analysis models: Discrimination and Calibration measures. In: Balakrishnan N, Rao C.R., editors. Handbook of Statistics, Survival Methods. 23. Amsterdam, The Netherlands: Elsevier B.V.; 2004. p. 1-25.
7.Hosmer DW, Lemeshow S. Applied logistic regression: Wiley-Interscience; 2000.
8.Hosmer DW, Lemeshow S, May S. Applied survival analysis : regression modeling of time-to-event data. 2nd ed. Hoboken, N.J.: Wiley-Interscience; 2008. xiii, 392 p. p.
9.Steyerberg EW. Clinical prediction models: a practical approach to development, validation, and updating: Springer; 2009.
10.Steyerberg EW, Eijkemans MJ, Harrell FE, Jr., Habbema JD. Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets. Stat Med. 2000 Apr 30;19(8):1059-79. PubMed PMID: 10790680. Epub 2000/05/03. eng.
11.Steyerberg EW, Harrell Jr FE, Borsboom GJ, Eijkemans M, Vergouwe Y, Habbema JDF. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. Journal of clinical epidemiology. 2001;54(8):774-81.
12.Grønnesby JK, Borgan Ø. A method for checking regression models in survival analysis based on the risk score. Lifetime Data Analysis. 1996;2(4):315-28.
13.Pencina MJ, D'Agostino RB, Sr., D'Agostino RB, Jr., Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008 Jan 30;27(2):157-72; discussion 207-12. PubMed PMID: 17569110. Epub 2007/06/15. eng.
14.Royston P, Sauerbrei W. Multivariable modeling with cubic regression splines: A principled approach. Stata Journal. 2007;7(1):45-70.
1