Psychotherapy Improvement – Supplemental Materials 1
Supplemental Materials
Do Psychotherapists Improve with Time and Experience? A Longitudinal Analysis of Outcomes in a Clinical Setting
by S. B. Goldberg et al., 2015, Journal of Counseling Psychology
Supplemental Materials – Table of Contents
Multilevel models (MLMs) predicting change in additional continuous outcomes……..….……2
Multilevel logistic regression models for dichotomous outcomes……..………………..…..……3
MLMs controlling for baseline severity and length of treatment………………………..…..……4
MLMs excluding therapists with less than one year of data……………………………..……….5
MLMs controlling for therapists’ age at first patient in the data set……………….……………..6
MLMs controlling for therapists’ years since beginning graduate school at first patient…….…..7
MLMs examining cases initiating as a potential confound…………………….…………………8
MLMs examining early termination as a potential confound………………………….…………9
MLMs excluding patient-level pre-post d outliers……………………………………….………10
MLMs excluding therapist change in pre-post d across time outliers……………….………...... 11
MLMs excluding therapists with outlying numbers of cases……..…………………….……….12
Psychotherapy Improvement – Supplemental Materials 1
Table S1.
Multilevel Models (MLMs) Predicting Change in Additional Continuous Outcomes
Model Type / Outcome / Predictor / FE [95% CI] / SE / t / p / Int Var / Res Var / ICC / Slp VarUnconditional / Post-test OQ / Pre-test / 0.61 [0.58, 0.64] / 0.016 / 39.07 / < .001 / 0.0021 / 0.183 / .012
Random Intercept / Post-test OQ / Pre-test / 0.61 [0.58, 0.64] / 0.016 / 39.06 / < .001 / 0.0022 / 0.183 / .012
Post-test OQ / Time / 0.0046 [0.0022, 0.0070] / 0.0012 / 3.73 / < .001
Random Slope / Post-test OQ / Pre-test / 0.61 [0.58, 0.64] / 0.016 / 39.18 / < .001 / 1.1 X 10-13 / 0.183 / 5.9 X 10-13 / 0.000046
Post-test OQ / Time / 0.0055 [0.0022, 0.0088] / 0.0017 / 3.17 / .002
Random Intercept / Post-test OQ / Pre-test / 0.61 [0.58, 0.64] / 0.016 / 39.06 / < .001 / 0.0023 / 0.183 / .013
Post-test OQ / Cases / 0.00035 [0.00014, 0.00055] / 0.00011 / 3.23 / .001
Random Slope / Post-test OQ / Pre-test / 0.61 [0.58, 0.64] / 0.016 / 39.15 / < .001 / 0.0011 / 0.182 / .0058 / 5.7 X 10-7
Post-test OQ / Cases / 0.00086 [0.00045, 0.0013] / 0.00021 / 4.16 / < .001
Unconditional / Rate of change / None / 0.00047 / 0.0473 / .0099
Random Intercept / Rate of change / Time / -0.0015 [-0.0027, -0.0026] / 0.00063 / -2.38 / .017 / 0.00055 / 0.0472 / .011
Random Slope / Rate of change / Time / -0.0015 [-0.0029, -0.00014] / 0.00071 / -2.16 / .031 / 0.00048 / 0.0472 / .010 / 3.2 X 10-6
Random Intercept / Rate of change / Cases / -0.00012 [-0.00023, -0.000016] / 0.000054 / -2.26 / .024 / 0.00055 / 0.0472 / .012
Random Slope / Rate of change / Cases / -0.00021 [-0.00037, -0.000053] / 0.00008 / -2.63 / .009 / 0.00047 / 0.0472 / .0099 / 4.3 X 10-8
Note: FE = fixed effect from multilevel models; SE = standard error; t = t-statistic; p = p-value; Int Var = intercept variance; Res Var = residual variance; ICC = intraclass correlation coefficient; Slp Var = random slope variance. n = 6591 patients seen by n = 170 therapists.
Table S2.
Multilevel Logistic Regression Models for Dichotomous Outcomes
Outcome / Predictor / FE [95% CI] / SE / z / pRCI drop / Time / -0.021 [-0.33, -0.0096] / 0.0059 / -3.58 / < .001
RCI drop / Cumulative cases / -0.0017 [-0.0027, -0.00076] / 0.00051 / -3.42 / < .001
Clinical Cut-off / Time / -0.026 [-0.038, -0.014] / 0.0060 / -4.28 / < .001
Clinical Cut-off / Cumulative cases / -0.0019 [-0.0030 , -0.00092] / 0.00052 / -3.70 / < .001
Note: FE = fixed effect from multilevel models; SE = standard error; z = z-statistic; p = p-value; RCI drop = OQ drop equivalent or larger than the reliable change index (Jacobson & Truax, 1991); Clinical cut-off = posttreatment OQ score below clinical cut-off.
Table S3.
Multilevel Models (MLMs) Controlling for Baseline Severity and Length of Treatment
Model Type / Outcome / Predictor / Fixed effect / SE / df / t / z / pRandom Slope / Change in OQ / Time / -0.013 / 0.0043 / 6418 / -3.16 / .002
Random Slope / Change in OQ / Cases / -0.0021 / 0.0005 / 6418 / -4.12 / < .001
Random Slope / Post-test OQ / Time / 0.0055 / 0.0017 / 6418 / 3.16 / .002
Random Slope / Post-test OQ / Cases / 0.00084 / 0.0002 / 6418 / 4.12 / < .001
Random Intercept / Rate of change / Time / -0.0017 / 0.00059 / 6418 / -2.85 / .004
Random Intercept / Rate of change / Cases / -0.00014 / 0.00005 / 6418 / -2.74 / .006
Random Intercept / RCI drop / Time / -0.023 / 0.006 / 6586 / -3.80 / < .001
Random Intercept / RCI drop / Cases / -0.0019 / 0.00052 / 6586 / -3.64 / < .001
Random Intercept / Clinical Cut-off / Time / -0.027 / 0.0063 / 6586 / -4.25 / < .001
Random Intercept / Clinical Cut-off / Cases / -0.002 / 0.00055 / 6586 / -3.62 / < .001
Note: Fixed effects for time and cases from models controlling for baseline OQ and length of treatment. The full sample (n = 6591) was used in analyses. SE = standard error; df = degrees of freedom; t / z = t- or z-statistic (t for MLMs, z for multilevel logistic regression); p = p-value; RCI drop = OQ drop equivalent or larger than the reliable change index (Jacobson & Truax, 1991); Clinical cut-off = posttreatment OQ score below clinical cut-off.
Table S4
MLMs Excluding Therapists With Less Than One Year of Data
Model Type / Outcome / Predictor / Fixed effect / SE / df / t / z / pRandom Slope / Change in OQ / Time / -0.012 / 0.0045 / 5897 / -2.73 / .006
Random Slope / Change in OQ / Cases / -0.002 / 0.00053 / 5897 / -3.70 / < .001
Random Slope / Post-test OQ / Time / 0.0055 / 0.0018 / 5896 / 3.11 / .002
Random Slope / Post-test OQ / Cases / 0.00084 / 0.00021 / 5896 / 3.97 / < .001
Random Intercept / Rate of change / Time / -0.0017 / 0.00066 / 5897 / -2.54 / .011
Random Intercept / Rate of change / Cases / -0.00013 / 0.000055 / 5897 / -2.38 / .018
Random Intercept / RCI drop / Time / -0.021 / 0.0061 / 6024 / -3.38 / .001
Random Intercept / RCI drop / Cases / -0.0017 / 0.00052 / 6024 / -3.24 / .001
Random Intercept / Clinical Cut-off / Time / -0.025 / 0.0063 / 6024 / -4.03 / < .001
Random Intercept / Clinical Cut-off / Cases / -0.0018 / 0.00053 / 6024 / -3.47 / .001
Note: Fixed effects for time and cases from models excluding therapists with less than one year of data available in the data (n = 41 therapists, n = 564 patients), thus analyses included n = 6027 patients. SE = standard error; df = degrees of freedom; t / z = t- or z-statistic (t for MLMs, z for multilevel logistic regression); p = p-value; RCI drop = OQ drop equivalent or larger than the reliable change index (Jacobson & Truax, 1991); Clinical cut-off = posttreatment OQ score below clinical cut-off.
Table S5
MLMs Controlling for Therapists’ Age at First Patient
Model Type / Outcome / Predictor / Fixed effect / SE / df / t / z / pRandom Slope / Change in OQ / Time / -0.015 / 0.0044 / 6338 / -3.30 / .001
Random Slope / Change in OQ / Cases / -0.0023 / 0.00055 / 6338 / -4.21 / < .001
Random Slope / Post-test OQ / Time / 0.0063 / 0.0018 / 6337 / 3.56 / < .001
Random Slope / Post-test OQ / Cases / 0.001 / 0.00022 / 6337 / 4.56 / < .001
Random Intercept / Rate of change / Time / -0.0019 / 0.00064 / 6338 / -2.91 / .004
Random Intercept / Rate of change / Cases / -0.00014 / 0.000055 / 6338 / -2.61 / .009
Random Intercept / RCI drop / Time / -0.024 / 0.006 / 6500 / -3.92 / < .001
Random Intercept / RCI drop / Cases / -0.0019 / 0.00052 / 6500 / -3.69 / < .001
Random Intercept / Clinical Cut-off / Time / -0.029 / 0.0062 / 6500 / -4.74 / < .001
Random Intercept / Clinical Cut-off / Cases / -0.0021 / 0.00053 / 6500 / -4.03 / < .001
Note: Fixed effects for time and cases from models controlling therapist age at first case. Data were unavailable for n = 5 therapists (87 patients), thus patient n = 6,504 for these models. SE = standard error; df = degrees of freedom; t / z = t- or z-statistic (t for MLMs, z for multilevel logistic regression); p = p-value; RCI drop = OQ drop equivalent or larger than the reliable change index (Jacobson & Truax, 1991); Clinical cut-off = posttreatment OQ score below clinical cut-off.
Table S6
MLMs Controlling for Therapists’ Years Since BeginningGraduateSchoolat First Patient
Model Type / Outcome / Predictor / Fixed effect / SE / df / t / z / pRandom Slope / Change in OQ / Time / -0.015 / 0.0045 / 6150 / -3.25 / .001
Random Slope / Change in OQ / Cases / -0.0022 / 0.00055 / 6150 / -3.99 / < .001
Random Slope / Post-test OQ / Time / 0.0063 / 0.0018 / 6149 / 3.52 / < .001
Random Slope / Post-test OQ / Cases / 0.00094 / 0.00022 / 6149 / 4.37 / < .001
Random Intercept / Rate of change / Time / -0.0019 / 0.00065 / 6150 / -2.92 / .004
Random Intercept / Rate of change / Cases / -0.00014 / 0.000054 / 6150 / -2.51 / .012
Random Intercept / RCI drop / Time / -0.024 / 0.0061 / 6310 / -3.87 / < .001
Random Intercept / RCI drop / Cases / -0.0018 / 0.00052 / 6310 / -3.51 / < .001
Random Intercept / Clinical Cut-off / Time / -0.03 / 0.0062 / 6310 / -4.77 / < .001
Random Intercept / Clinical Cut-off / Cases / -0.0021 / 0.00053 / 6310 / -3.94 / < .001
Note: Fixed effects for time and cases from models controlling therapist years since beginning graduate school. Data were unavailable for n = 7 therapists (277 patients), thus patient n = 6,314 for these models. SE = standard error; df = degrees of freedom; t / z = t- or z-statistic (t for MLMs, z for multilevel logistic regression); p = p-value; RCI drop = OQ drop equivalent or larger than the reliable change index (Jacobson & Truax, 1991); Clinical cut-off = posttreatment OQ score below clinical cut-off.
Table S7
MLMs Examining Cases Initiating as a Potential Confound
Model Type / Outcome / Predictor / Fixed effect / SE / df / t / pRandom Intercept / Change in OQ / Cases initiating / 0.0044 / 0.0081 / 2016 / 0.53 / .593
Change in OQ / Time / -0.0033 / 0.0010 / 2016 / -3.32 / .001
Random Slope / Change in OQ / Cases initiating / 0.0018 / 0.0081 / 2016 / 0.22 / .825
Change in OQ / Time / -0.0036 / 0.0012 / 2016 / -2.93 / .003
Random Intercept / Rate of change / Cases initiating / 0.00079 / 0.0016 / 2016 / 0.50 / .617
Rate of change / Time / -0.00049 / 0.00019 / 2016 / -2.54 / .011
Random Slope / Rate of change / Cases initiating / 0.00043 / 0.0016 / 2016 / 0.27 / .785
Rate of change / Time / -0.0005 / 0.00022 / 2016 / -2.22 / .026
Random Intercept / RCI drop / Cases initiating / -0.0016 / 0.0036 / 2016 / -0.44 / .659
RCI drop / Time / -0.0016 / 0.00045 / 2016 / -3.46 / .001
Random Intercept / Clinical cut-off / Cases initiating / 0.004 / 0.0036 / 2016 / 1.11 / .266
Clinical cut-off / Time / -0.0018 / 0.00045 / 2016 / -4.08 / < .001
Notes: Fixed effects for time from models examining the number of cases initiating as a potential confound. Therapists’ number of cases initiating treatment and aggregate outcome was computed across 3-month periods. The order of these 3-month periods served as the predictor variable in these two-level MLMs. Analyses represent outcomes for the full sample of therapists (n = 170). SE = standard error; df = degrees of freedom; t = t-statistic; p = p-value; RCI drop = proportion of patients showing an OQ drop equivalent or larger than the reliable change index (Jacobson & Truax, 1991); Clinical cut-off = proportion of patients with posttreatment OQ score below clinical cut-off.
Table S8
MLMs Examining Early Termination as a Potential Confound
Model Type / Outcome / Predictor / Fixed effect / SE / df / t / pRandom Intercept / Change in OQ / Prop Terminating / -0.085 / 0.10 / 2024 / -0.83 / .405
Change in OQ / Time / -0.0035 / 0.0010 / 2024 / -3.42 / .001
Random Slope / Change in OQ / Prop Terminating / -0.079 / 0.10 / 2024 / -0.78 / .437
Change in OQ / Time / -0.0037 / 0.0012 / 2024 / -3.01 / .003
Random Intercept / Rate of change / Prop Terminating / 0.015 / 0.020 / 2024 / 0.75 / .455
Rate of change / Time / -0.00049 / 0.0002 / 2024 / -2.48 / .013
Random Slope / Rate of change / Prop Terminating / 0.016 / 0.020 / 2024 / 0.82 / .412
Rate of change / Time / -0.00051 / 0.00022 / 2024 / -2.31 / .021
Random Intercept / RCI drop / Prop Terminating / -0.03 / 0.045 / 2024 / -0.66 / .509
RCI drop / Time / -0.0016 / 0.00045 / 2024 / -3.62 / < .001
Random Intercept / Clinical cut-off / Prop Terminating / 0.0082 / 0.044 / 2024 / 0.19 / .853
Clinical cut-off / Time / -0.0018 / 0.00045 / 2024 / -3.97 / < .001
Notes: Fixed effects for time from models examining the proportion of cases terminating prior to session 3 as a potential confound. Therapists’ number of cases terminating prior to session three divided by number of cases initiating (i.e., proportion terminating prior to session 3) and aggregate outcome was computed across 3-month periods. The order of these 3-month periods served as the predictor variable in these two-level MLMs. Analyses represent outcomes for the full sample of therapists (n = 170). SE = standard error; df = degrees of freedom; t = t-statistic; p = p-value; RCI drop = proportion of patients showing an OQ drop equivalent or larger than the reliable change index (Jacobson & Truax, 1991); Clinical cut-off = proportion of patients with posttreatment OQ score below clinical cut-off; Prop terminating = proportion of a given therapist’s cases terminating prior to session 3 during a given 3-month period.
Table S9
MLMs Excluding Patient-Level Pre-Post d Outliers
Model Type / Outcome / Predictor / Fixed effect / SE / df / t / z / pRandom Slope / Change in OQ / Time / -0.013 / 0.0041 / 6375 / -3.03 / .003
Random Slope / Change in OQ / Cases / -0.0021 / 0.0005 / 6375 / -4.14 / < .001
Random Slope / Post-test OQ / Time / 0.0055 / 0.0017 / 6374 / 3.32 / .001
Random Slope / Post-test OQ / Cases / 0.00087 / 0.0002 / 6374 / 4.40 / < .001
Random Intercept / Rate of change / Time / -0.0015 / 0.00061 / 6375 / -2.48 / .013
Random Intercept / Rate of change / Cases / -0.00013 / 0.000052 / 6375 / -2.45 / .014
Random Intercept / RCI drop / Time / -0.021 / 0.0059 / 6543 / -3.62 / < .001
Random Intercept / RCI drop / Cases / -0.0018 / 0.00051 / 6543 / -3.47 / .001
Random Intercept / Clinical Cut-off / Time / -0.026 / 0.0061 / 6543 / -4.32 / < .001
Random Intercept / Clinical Cut-off / Cases / -0.002 / 0.00052 / 6543 / -3.77 / < .001
Note: Fixed effects for time and cases from models with patient-level outliers (on pre-post d) excluded. A total of 45 patients were removed (35 high outliers, 10 low outliers), leaving a sample of n = 6546 patients for analysis. SE = standard error; df = degrees of freedom; t / z = t- or z-statistic (t for MLMs, z for multilevel logistic regression); p = p-value; RCI drop = OQ drop equivalent or larger than the reliable change index (Jacobson & Truax, 1991); Clinical cut-off = posttreatment OQ score below clinical cut-off.
Table S10
MLMs Excluding Therapist Change in Pre-Post d Across Experience Outliers
Model Type / Outcome / Predictor / Fixed effect / SE / df / t / z / pRandom Slope / Change in OQ / Time / -0.012 / 0.0044 / 6363 / -2.85 / .004
Random Slope / Change in OQ / Cases / -0.0020 / 0.00052 / 6363 / -3.82 / < .001
Random Slope / Post-test OQ / Time / 0.0055 / 0.0017 / 6362 / 3.17 / .002
Random Slope / Post-test OQ / Cases / 0.00086 / 0.00021 / 6362 / 4.15 / < .001
Random Intercept / Rate of change / Time / -0.0015 / 0.00063 / 6363 / -2.35 / .019
Random Intercept / Rate of change / Cases / -0.00012 / 0.000054 / 6363 / -2.24 / .025
Random Intercept / RCI drop / Time / -0.021 / 0.0059 / 6527 / -3.64 / < .001
Random Intercept / RCI drop / Cases / -0.0018 / 0.00051 / 6527 / -3.45 / .001
Random Intercept / Clinical Cut-off / Time / -0.026 / 0.006 / 6527 / -4.25 / < .001
Random Intercept / Clinical Cut-off / Cases / -0.0019 / 0.00052 / 6527 / -3.67 / < .001
Note: Fixed effects for time and cases from models with therapist-level change in pre-post d outliers excluded. A total of 4 therapist were removed (1 high outliers, 3 low outliers), leaving a sample of n = 6,530 patients for analysis. SE = standard error; df = degrees of freedom; t / z = t- or z-statistic (t for MLMs, z for multilevel logistic regression); p = p-value; RCI drop = OQ drop equivalent or larger than the reliable change index (Jacobson & Truax, 1991); Clinical cut-off = posttreatment OQ score below clinical cut-off.
Table S11
MLMs Excluding Therapists With Outlying Number of Cases
Model Type / Outcome / Predictor / Fixed effect / SE / df / t / z / pRandom Slope / Change in OQ / Time / -0.015 / 0.0047 / 5348 / -3.18 / .002
Random Slope / Change in OQ / Cases / -0.0027 / 0.00056 / 5348 / -4.89 / < .001
Random Slope / Post-test OQ / Time / 0.0067 / 0.0018 / 5347 / 3.63 / < .001
Random Slope / Post-test OQ / Cases / 0.0012 / 0.00021 / 5347 / 5.45 / < .001
Random Intercept / Rate of change / Time / -0.0021 / 0.00074 / 5348 / -2.85 / .004
Random Intercept / Rate of change / Cases / -0.00035 / 0.00009 / 5348 / -3.83 / < .001
Random Intercept / RCI drop / Time / -0.025 / 0.0068 / 5512 / -3.72 / < .001
Random Intercept / RCI drop / Cases / -0.0042 / 0.00081 / 5512 / -5.22 / < .001
Random Intercept / Clinical Cut-off / Time / -0.032 / 0.0071 / 5512 / -4.47 / < .001
Random Intercept / Clinical Cut-off / Cases / -0.0045 / 0.00086 / 5512 / -5.28 / < .001
Note: Fixed effects for time and cases from models with therapists who had outlying numbers of cases in the data set. A total of 4 therapist were removed (all high outliers), leaving a sample of n = 5,515 patients for analysis. SE = standard error; df = degrees of freedom; t / z = t- or z-statistic (t for MLMs, z for multilevel logistic regression); p = p-value; RCI drop = OQ drop equivalent or larger than the reliable change index (Jacobson & Truax, 1991); Clinical cut-off = posttreatment