Appendix A – Propensity score quintile analysis of effects of non-response on findings

Nonresponse was seen as a potential confounder in our primary analysis. Because administrative data on team role, VA tenure and clinic location was available for all primary care personnel, we were able to usepropensity score methods to reduce the bias associated with non-response. To do so, we aimed to balance the observed covariates betweenthe respondents and non-respondents using propensity score methods outlined by Rubin and others.[1]

Propensity scores were estimated for each team member using a mixed effects logistic regression model withresponse as the dependent variable, three available factors potentially associated with non-response as independent variables (occupation, VA tenure, and clinic location), and a random effect for VA facility. The estimated propensity scores were then used to group the primary care team members into the five strata based on their predicted probability of response, referred to as“response propensity stratification.”[2]Respondents were weighted by the inverse of the observed rate in that cell.

The primary analysis model was then re-run including these weights (Table A1).Within this model, fully staffed team, team turnover, and having a panel over-capacity remained statistically significant and in the same direction. However, ORs(and their standard errors) for turnover and over-capacity increased substantially from our complete-case models (3.54, 95% CI: 2.77-4.54 and 6.28, 95% CI: 2.44-16.20, respectively); and the OR for fully staffed was much smallerthan in complete-case models (0.17, 95% CI: 0.13-0.21). Additionally, the variable “member of multiple teams” was statistically significant in the propensity-weighted model only (1.89, 95% CI: 1.47-2.43). We conclude thatour initial complete-case findings may be conservative after controlling for non-response.

Table A1: Propensity score analysis findings compared to complete case

Variable / Original Complete-Case ORs (95% CI) / Propensity Weighted
ORs (95% CI) / Change
Fully staffed / 0.55 (0.47-0.65) / 0.17 (0.13-0.21) / ↓
Turnover / 1.67 (1.43-1.94) / 3.54 (2.77-4.54) / ↑
Member of multiple teams / 1.13 (0.95-1.34) / 1.89 (1.47-2.43) / ↑
Panel overcapacity / 1.19 (1.01-1.40) / 6.28 (2.44-16.20) / ↑
Average panel comorbidity / 1.12 (0.84-1.48) / 1.33 (0.29-6.15) / ↔
Works extended hours / 1.14 (0.96-1.34) / 0.82 (0.61-1.10) / ↔

1

[1]D’Agostino RB Jr. Propensity score methods for bias reduction in the comparisonof a treatment to a non-randomized control group. Stat Med. 1998;17:2265-81. [PMID: 9802183]; Rubin DB. Estimating causal effects from large data sets using propensity

scores. Ann Intern Med. 1997;127:757-63. [PMID: 9382394]; Rubin DB, Thomas N. Matching using estimated propensity scores: relating

theory to practice. Biometrics. 1996;52:249-64. [PMID: 8934595]

[2] Little, R.J.A. .Survey Nonresponse Adjustments for Estimates of Means. International Statistical Review, vol. 54 1986 pp. 139-157.