Abstract: Aim, Methods, Results, Discussion

Abstract: Aim, Methods, Results, Discussion

Aim: This paper presents analyses of outcome data for 337 therapists treating 14,161 patients over a three-year period to determine if therapists’ effect sizes increased over time. Each therapist treated at least 5 patients in each of their first two years of using outcome measures. Multiple outcome questionnaires were employed. All measures also included a brief alliance scale administered concurrently.

Method: A severity adjusted effect size was calculated for each patient using intake scores and diagnostic group as predictors. The mean severity adjusted effect size for each therapist was calculated for their first and second years of using the outcome tools. This was done using a hierarchical linear model to control for sample size in each year, with a minimum sample of 5 cases in each year. Therapist engagement in receiving feedback was measured by counting the number of times the therapist logged into the online platform to view their results in each of the two years.

Results: Therapists who logged in the view their data at least 24 times in the second year (n=123; 37%) averaged .92 effect size compared to .82 effect size for those seen by therapists who reviewed their results less frequently (n=214; 63%). Login frequency during the first year was not predictive of effect size during the second year.

Discussion: The data provides evidence that effect sizes can trend upwards with measurement and feedback. Therapists’ engagement in receiving feedback appears to increase the likelihood of effect size gain.

Description of Sample

Total ACORN database (as of 8/13/17)

Clinicians submitting data: 10,978

Completed questionnaires: 2807586

Episodes of care: 904385

Selection criteria for study:

Clinicians:

1. Began to use ACORN after 10/1/2011 (date when user login tracking became available): N= 4467
2. Collected data over a minimum of 3 years: N=853 (19% of users)
3. At least 5 cases in each of first 2 years: N=337 (7.6% of users)

Cases (episodes of care):

1. Treated by clinicians meeting inclusion criteria.
2. Began treatment during the first 2 years of clinician participation
3. Intake scores in clinical range.
4. At least two assessments.
5. Duration of treatment one year or less.

14,161 cases the criteria (6103 in year 1, 6365 in year 2)

At risk cases identified within all cases:

1. An expected score calculated for each assessment using intake score, assessment number and time in treatment as predictor variables in a general linear model.
2. Case flagged as at risk if actual score deviated from predicted score by more than the reliable change index of the questionnaire on one or more sessions within the episode.

4237 (34%) cases flagged as at risk (2077 in year 1, 2160 in year 2).

Methods

The mean SAES in year one and year two was estimated at both the therapist level and case level.

1. Calculated the mean SAES for each therapist, using HLM to controlfor sample size and then calculated the unweighted means for all of the therapists.
2. Calculated the mean SAES of the individual cases included in the sample. These results will be different from the mean of the individual therapists.

The level of clinician engagement was determined based the on frequency of logins in each of the first two years, using the following criteria:

1. Low engagement: No logins within the year in question
2. Moderate engagement: 1-23 logins within the year
3. High engagement: At least 24 logins within the year

Results

Means for therapists

Variable / N / Mean / Std Dev / Minimum / Maximum
MeanSAESyear1
logincounty1
MeanSAESyear2
logincounty2
/ 341
341
341
341
/ 0.9429267
55.9970674
0.8774377
49.0498534
/ 0.1804540
96.6765833
0.1659734
72.4921528
/ 0.2691808
0
0.2141067
0
/ 1.3593033
347.0000000
1.4124313
279.0000000

Note: The distribution of results for this sample of therapists is much different from the ACORN sample as a whole. The means for all therapists (5 cases or more) is .8, with an SD of .17. Below is the table from Brown, Simon and Minami (2016). This means that the average results for a therapist in this sample is well above the 75th percentile in the first year. Consideration of regression artifacts suggest that one would expect the results to decline in the second year. Note also that the count of logins decreased from year 1 to year 2.

Following are the frequencies of engagement level for each of the years.

Year 1

High Engagement (N=134; 39.3%)

Moderate Engagement (N=137; 40.2%)

Low Engagement (N=70; 20.5%)

Year 2

High Engagement (N=123; 36.5%)

Moderate Engagement (N=132; 39.2%)

Low Engagement (N=82; 24.3%)

To investigate the association between engagement and change in effect size from year 1 to year 2, we ran a regression analysis using year 1 SAES to predict year 2 SAES for each of the engagement levels. The following graph displays the results.

As expected, therapists with year one effect sizes well below .8 effect size showed relatively large gains in effect size in their second year, but this is balanced by somewhat smaller reductions in effect size for those with year one results well above .8 effect size. For example, a therapist with .6 effect size in year one is expected to show a gain of between .14 and .17 effect size in the second year. On the other hand, a therapist with an effect size of 1.0 in year one is expected to show a decrease of between .09 and .12 depending on engagement level in year two.

There does not appear to be a linear relationship between engagement as measured by log in frequency and the likely effect size. There does appear to be a threshold effect… those who log in at least twice a month on average show better results. However, even within the High Engagement group, the log in frequency count was not correlated significantly with effect size.

Following are the regression formulas derived for each level of engagement:

High Engagement: SAESYear1*0.3596962809+0.5545329971 (R-square=0.109999)

Moderate Engagement: SAESYear1*0.3964279665+0.4976036876 (R-square=0.269302)

Low Engagement: SAESYear1*0.3538682594+0.5294310369 (R-Square= 0.160719)

Following is the table from which the graph was produced.

Project effect size year 2
Effect Size Year 1 / High Engagement (N=123; 36.5%) / Moderate Engagement (N=132; 39.2%) / Low Engagement (N=82; 24.3%)
0.3 / 0.662441881 / 0.616532078 / 0.635591515
0.4 / 0.698411509 / 0.656174874 / 0.670978341
0.5 / 0.734381138 / 0.695817671 / 0.706365167
0.6 / 0.770350766 / 0.735460468 / 0.741751993
0.7 / 0.806320394 / 0.775103264 / 0.777138818
0.8 / 0.842290022 / 0.814746061 / 0.812525644
0.9 / 0.87825965 / 0.854388857 / 0.84791247
1 / 0.914229278 / 0.894031654 / 0.883299296
1.1 / 0.950198906 / 0.933674451 / 0.918686122
1.2 / 0.986168534 / 0.973317247 / 0.954072948
1.3 / 1.022138162 / 1.012960044 / 0.989459774
1.4 / 1.05810779 / 1.052602841 / 1.0248466

Following are the aggregated results using the individual case data included in the study. The mean SAES is each year is above the mean for the entire ACORN sample (.81) Results for this sample remained largely stable both in terms of SAES and number of both assessments and weeks in treatment.

year=1

Variable / N / Mean / Std Dev / Minimum / Maximum
SAES
assessments
weeks
/ 6103
6103
6103
/ 0.8483668
5.1805669
15.0785562
/ 1.0494066
4.2036221
12.7254440
/ -4.3288217
2.0000000
0
/ 4.9699362
39.0000000
52.0000000

year=2

Variable / N / Mean / Std Dev / Minimum / Maximum
SAES
assessments
weeks
/ 6365
6365
6365
/ 0.8443559
5.1054203
14.3985187
/ 1.0241506
4.2321140
12.2718917
/ -2.7498792
2.0000000
0
/ 5.3221665
42.0000000
52.0000000

Overall, 34% of cases were flagged as at risk at some point in their treatment. This percentage was constant from year 1 to year 2. The following are results broken out by whether cases were flagged (signalcase y/n).

For non signal cases, the results were quite good, with SAES > 1.1 on both years.

signalcase=n year=1

Variable / N / Mean / Std Dev / Minimum / Maximum
SAES
assessments
weeks
/ 4026
4026
4026
/ 1.1620362
4.2642822
12.2041729
/ 0.8383528
3.0199668
10.8636688
/ -0.5159804
2.0000000
0.1428571
/ 4.9699362
20.0000000
52.0000000

signalcase=n year=2

Variable / N / Mean / Std Dev / Minimum / Maximum
SAES
assessments
weeks
/ 4205
4205
4205
/ 1.1278098
4.1828775
11.6205877
/ 0.8304784
3.0083284
10.5147546
/ -0.6307183
2.0000000
0.1428571
/ 5.3221665
20.0000000
52.0000000

For signal cases the results are quite different, with SAES less than .3 in each year. The length of treatment (assessments and weeks) is also significantly longer for these cases. The duration averaged 20 weeks in each year compared to 12 weeks for non signal cases.

signalcase=y year=1

Variable / N / Mean / Std Dev / Minimum / Maximum
SAES
assessments
weeks
/ 2077
2077
2077
/ 0.2403584
6.9566683
20.6501823
/ 1.1461200
5.4288221
14.1450638
/ -4.3288217
2.0000000
0
/ 4.6779335
39.0000000
52.0000000

signalcase=y year=2

Variable / N / Mean / Std Dev / Minimum / Maximum
SAES
assessments
weeks
/ 2160
2160
2160
/ 0.2925393
6.9013889
19.8064815
/ 1.1347489
5.5034620
13.5773314
/ -2.7498792
2.0000000
0
/ 4.4941415
42.0000000
52.0000000

The level of engagement for therapists differed over time, as did the relationship between engagement and effect size. For example, in the first year patients treated by a therapist with low engagement had the largest effect size,however ANOVA revealed that none of the mean differences approached significance.

Following are results for patients in year one broken out by the therapist level of engagement in year one.

year=1 engagementy1=High

Variable / N / Mean / Std Dev / Minimum / Maximum
SAES
assessments
weeks
/ 1261
1261
1261
/ 0.8528285
5.5939730
15.6684038
/ 1.0416815
4.8070809
12.8633954
/ -2.4803035
2.0000000
0
/ 4.5137611
38.0000000
52.0000000

year=1 engagementy1=Low

Variable / N / Mean / Std Dev / Minimum / Maximum
SAES
assessments
weeks
/ 2057
2057
2057
/ 0.8784989
4.9747205
15.5496215
/ 1.0739140
4.2504684
13.1451990
/ -3.0333297
2.0000000
0
/ 4.6292329
39.0000000
52.0000000

year=1 engagementy1=Moderate

Variable / N / Mean / Std Dev / Minimum / Maximum
SAES
assessments
weeks
/ 2792
2792
2792
/ 0.8219049
5.1454155
14.4857245
/ 1.0343193
3.8476431
12.3403494
/ -4.3319648
2.0000000
0
/ 4.9707007
37.0000000
52.0000000

The following table displays results in year one by therapist engagement in year one. Likewise, ANOVA confirmed that therapists’ engagement level in year one was not predictive of effect size in year 2.

year=2 engagementy1=High

Variable / N / Mean / Std Dev / Minimum / Maximum
SAES
assessments
weeks
/ 1121
1121
1121
/ 0.8592628
5.5138269
15.3113292
/ 1.0562241
4.3478274
12.4088315
/ -2.5762178
2.0000000
0
/ 5.3245870
36.0000000
51.7142857

year=2 engagementy1=Low

Variable / N / Mean / Std Dev / Minimum / Maximum
SAES
assessments
weeks
/ 2577
2577
2577
/ 0.8662080
5.0911913
15.0698487
/ 1.0118376
4.4225377
12.6087819
/ -2.8627138
2.0000000
0
/ 4.4953824
35.0000000
52.0000000

year=2 engagementy1=Moderate

Variable / N / Mean / Std Dev / Minimum / Maximum
SAES
assessments
weeks
/ 2762
2762
2762
/ 0.8129725
4.9641564
13.5422572
/ 1.0210921
3.9820064
11.9488047
/ -2.6021123
2.0000000
0
/ 4.2894837
42.0000000
52.0000000

However, when looking at results for year two as a function of therapist engagement in year two, an association between high engagement and effect size does emerge.

The SAES for patients seen by high engagement therapists averaged .1 larger effect size compared to the less engaged therapists. Results of ANOVA and Tukey comparisons follow. All comparisons are significant (p<.05).

engagement=High

Variable / N / Mean / Std Dev / Minimum / Maximum
SAES
assessments
weeks
/ 1343
1343
1343
/ 0.9227768
5.3767684
14.6553558
/ 1.0677530
4.1284657
11.9203138
/ -2.5753905
2.0000000
0
/ 4.2933943
36.0000000
51.7142857

engagement=Low

Variable / N / Mean / Std Dev / Minimum / Maximum
SAES
assessments
weeks
/ 2545
2545
2545
/ 0.8251215
4.8188605
14.4273926
/ 0.9872183
4.2197636
12.2525451
/ -2.6439523
2.0000000
0
/ 4.4487049
42.0000000
52.0000000

engagement=Moderate

Variable / N / Mean / Std Dev / Minimum / Maximum
SAES
assessments
weeks
/ 2477
2477
2477
/ 0.8215994
5.2527251
14.2295980
/ 1.0355603
4.2844062
12.4801147
/ -2.7498792
2.0000000
0
/ 5.3221665
41.0000000
51.8571429

The ANOVA Procedure

Tukey's Studentized Range (HSD) Test for effectsize

Note: / This test controls the Type I experimentwise error rate.
Alpha / 0.05
Error Degrees of Freedom / 6362
Error Mean Square / 1.047566
Critical Value of Studentized Range / 3.31527
Comparisons significant at the 0.05 level are indicated by ***.
engagement
Comparison / Difference
Between
Means / Simultaneous 95% Confidence
Limits
High - Low / 0.09766 / 0.01673 / 0.17858 / ***
High - Moderate / 0.10118 / 0.01987 / 0.18248 / ***
Low - High / -0.09766 / -0.17858 / -0.01673 / ***
Low - Moderate / 0.00352 / -0.06420 / 0.07124
Some - Moderate / -0.10118 / -0.18248 / -0.01987 / ***
Some - Low / -0.00352 / -0.07124 / 0.06420

Following are the results of the same ANOVA, this time for non signal cases only. The same .1 effect size advantage to the high engagement therapists is apparent. All Tukey comparisons of high engagement to lower levels of engagement are significant.

engagement=High

Variable / N / Mean / Std Dev / Minimum / Maximum
SAES
assessments
weeks
/ 907
907
907
/ 1.2112287
4.4531422
12.2379902
/ 0.9026879
3.0587981
10.5423349
/ -0.5434547
2.0000000
0.1428571
/ 4.2933943
20.0000000
51.7142857

engagement=Low

Variable / N / Mean / Std Dev / Minimum / Maximum
SAES
assessments
weeks
/ 1679
1679
1679
/ 1.1024041
3.9005360
11.5782353
/ 0.7730642
2.8742403
10.3358116
/ -0.4715525
2.0000000
0.1428571
/ 4.2841454
20.0000000
52.0000000

engagement=Moderate

Variable / N / Mean / Std Dev / Minimum / Maximum
SAES
assessments
weeks
/ 1619
1619
1619
/ 1.1074239
4.3242742
11.3186270
/ 0.8431857
3.0925921
10.6739952
/ -0.6307183
2.0000000
0.1428571
/ 5.3221665
20.0000000
51.5714286
Comparisons significant at the 0.05 level are indicated by ***.
engagement
Comparison / Difference
Between
Means / Simultaneous 95% Confidence
Limits
High - Moderate / 0.10380 / 0.02314 / 0.18447 / ***
High - Low / 0.10882 / 0.02868 / 0.18897 / ***
Moderate - High / -0.10380 / -0.18447 / -0.02314 / ***
Moderate - Low / 0.00502 / -0.06272 / 0.07276
Low - High / -0.10882 / -0.18897 / -0.02868 / ***
Low - Moderate / -0.00502 / -0.07276 / 0.06272

Following are the results for signal cases only. ANOVA confirms that the differences in results are minimal and do not approach significance.

engagement=High

Variable / N / Mean / Std Dev / Minimum / Maximum
SAES
assessments
weeks
/ 436
436
436
/ 0.3227174
7.2981651
19.6841415
/ 1.1339098
5.2551537
13.0137274
/ -2.5753905
2.0000000
0
/ 3.5711481
36.0000000
51.5714286

engagement=Low

Variable / N / Mean / Std Dev / Minimum / Maximum
SAES
assessments
weeks
/ 866
866
866
/ 0.2875261
6.5993072
19.9513362
/ 1.1262449
5.6154164
13.7106299
/ -2.6439523
2.0000000
0
/ 4.4487049
42.0000000
52.0000000

engagement=Moderate

Variable / N / Mean / Std Dev / Minimum / Maximum
SAES
assessments
weeks
/ 858
858
858
/ 0.2822639
7.0046620
19.7224442
/ 1.1447441
5.5023047
13.7358628
/ -2.7498792
2.0000000
0
/ 4.4941415
41.0000000
51.8571429

Discussion

Limitations

There are a number of limitations to this study. The most glaring is the fact that the therapists who met the selection criteria as a group were already significantly more effective that the population of therapists from which they were drawn. This means that the natural tendency of results for many of the therapists will trend downwards due to regression artifacts.

The sample of therapists was drawn from a variety of settings include HMO clinics, non-profit agencies and for profit psychotherapy practices and clinics. Results vary between settings and clinics, indicating that these reported results and may have limited generalizability even within the universe of clinical settings from which the sample of drawn.

Findings

The study sample did provide evidence that over time, the level of therapist engagement in monitoring results was associated with observed effect sizes. The absolute number of therapist log in events during the year was not associated in a linear manner for effect size, but there did appear to be a threshold effect. In others words, logging in often enough made a difference, but logging in a lot more did not predict even better results.

Cases that were not targeted as at risk (non-signal) average large effect sizes and a relatively short length of treatment. The effect sizes for non-signal cases were larger for therapists exhibiting high engagement, while the number of sessions and weeks in treatment did not vary significantly by engagement level.

For signal cases, the effect sizes were much smaller, while the number of sessions and length of treatment were significantly longer. These results did not vary significantly by engagement level. It is encouraging to observe that these patients did tend to remain in treatment much longer than the other cases, but it also draws attention to the challenges of determining how best to improve results for these at-risk cases.

Implications

Need input from authors…