Audit and Feedback Meeting

Audit and Feedback Meeting

Day 1 - December 6th, 2012

Large group presentations: Results from Cochrane review of AF: what do we know and where do we go from here?

Noah’s presentation “Using AF to improve quality of care”

·  Findings from the 2006 Cochrane review

·  2012 update

o  Meta Regression

o  Format, source, frequency, recipient, baseline performance, risk of bias

o  Plus exploratory analyses

o  140 trials – AF as a primary component – most were multifaceted

o  Results:

§  Similar to the previous results – AF seems to work

§  Both verbal + written seems to be more effective

§  Given by a senior colleague more effective than from employer representative

§  Moderately frequent vs. only once

§  Goal plus action plan

§  More effective when the feedback attempted to decrease the behaviour

§  AF more effective for prescribing than other behaviours – why? Need to list exactly what is targeted in context – both of situation (material and social) and of other behaviours.

§  More effective when baseline performance is lower

§  What you were targeting seems to be important – e.g., getting doctors to change their prescribing behaviour---needs more work.

Comments from the group members:

·  Should focus on the middle bullets first – as this is what many AF researchers tick off first.

·  How many behaviours can we look at once?

·  Important to look at the context

·  Try to think about why they may or may not be a modifier

·  Personal vs. interpersonal characteristics of the feedback

·  Issue of sustainability

Susan Michie’s Presentation: “Applying theory to designing AF interventions and evaluations in head to head trials”

Why Theory?

·  More effective, provides a framework to facilitate (accumulation of evidence, communication across research groups), identifies mechanisms of action

MRC Guidance for developing and evaluating complex interventions (Craig et al 2009, BMJ)

What Theories?

·  MRC guidance silent on this

·  NICE’s Behaviour Change evidence review (2008)

o  Identified evidence-based principles of behaviour change (Abraham, Kelly, West, Michie, 2008)

o  No guidance on which theories to use

·  Starting point for selecting theory

o  Understand intervention content

o  Need a method for specifying content (BCTs)

Types of A&F

·  Intensive (individual recipients) AND (verbal format) OR (a supervisor or senior colleague as the source) AND (moderate or prolonged feedback)

·  Non-Intensive (group feedback) NOT (from supervisor) OR (individual feedback) AND (written) AND (containing info about cost or numbers of tests without personal incentives)

·  Moderate (any other combination of characteristics than described in intensive or non-intensive)

o  Problems of categorising by intensity (no theoretical rational, few recommendations for practice offered, mixture of modes of delivery and content)

o  A theory-based approach: Specify content as BCTs to allow theoretically based categorisation & analysis; generate theory-based hypotheses concerning effectiveness

o  Specifying content: 13 papers from AF review; 28 distinct BCTs grouped into goal/standard setting, feedback & action planning

o  What kind of theory would help? Self-regulation theory (Carver & Scheier, 1982)

o  Feedback more effective when goal/target is set

o  Most effective where goal/target & action plan

Head to head trials: On what basis does one select intervention components?

·  Need to have a theory about how AF is working

·  What functions are AF playing?

o  Structure for noticing and reducing discrepancy (target, fb, action plan)

o  Cue to action

o  Reinforcement

o  Social support (how it’s given)

o  Others?

Ensure all behaviour change techniques are identified (within and beyond AF; both intervention and control group)

Summary:

·  Detailed description of intervention a starting pt for identifying mechanisms of action (ie. theory)

·  Might need to draw on more than one formal theory to generate hypotheses about mechanisms.

·  These hypotheses should guide intervention design, optimisation, evidence synthesis and trial design

Group Discussion:

·  Anne Sales: Does feedback have a role in promoting motivation? SM – yes-absolutely

·  Feedback more than once is more effective – how we should apply the theory in terms of multiple cycles of feedback….Susan: want feedback to be unpredictable of when it is coming.

·  Survey response rates are low – survey physicians to see if they are reducing discrepancy first

·  Theories of feedback; theories of goals – should we be going for a theory of AF? Susan: yes – that’s what we are trying to do – a program theory. Can we think about generating a theory of AF from more general theories?

Heather Colquhoun: “A SR of AF interventions: the good, the bad, and the ugly”

·  To whom: 51% given to individuals; rest group (18%), both (16%), unclear (14%); 92% FB given to target person

·  What was given? 79% given about behaviour, 14% on outcomes, 32% other (cost, accuracy of diagnosis, risk data, education, barriers to change) 58% on an individual’s performance, 64% group performance, mostly aggregate (81%)

·  *Comment: Why cost not part of behaviour – it is actionable? Susan – these are outcomes of behaviour – feedback on behaviour is the most effective.

·  Another participant comment: It’s a construct that can be pulled out. Like what we did.

·  Comment: How much were classified as education – correct solution information?

·  36% was graphical; 14% was unclear.

·  89% addressed the behaviour to be changed; 6% no; 5% unclear.

·  Clear comparison in the fb? 74% yes. (mostly others previous performance 68% – very few multiple comparisons)

·  How delivered? Face to face (44%)

·  How much? Total fb : 24% unclear; clear 76% (33/107 given only once, only 27 given more than 4x)

·  Comment: self activated/real time – have it in front of you? Were there any of these?

·  Comment: giving it the same way each time (e.g. 3 X the same way…would there be a difference? )

·  Comment: Anne Sales – new information to people – highly variable; background issue that is important.

·  Lag time between collection of the fb and the provision of the fb not clearly stated (37%); if yes – mostly months.

·  Rational for AF in the trial? 36% was empirical; 28% was intuitive (AF has never been used, but thought it would be a good idea), 26% no rational (just appear in the intervention), 9% used theory to design the intervention.

·  Running list of the articles that included the AF reports – only 7 did this.

·  Summary – some consistency, but also wide variation, reporting is poor, rationale for the intervention is lacking

AFTER LUNCH....

Large Group Discussion – Key Thoughts to Consider for the Meeting:

·  Susan Michie: What might be in a good intervention (includes theory)? The criteria that you would use to evaluate? Nature of the study design for the evaluation?

·  Sylvia Hysong: Better idea of what the priority criteria is (e.g., the 5 things you would want to test because....)

·  Cumulative meta-analysis – shouldn’t discount 140 trials – clearly a lot of theory has gone into it. Don’t want to get into “paralysis by analysis” by spending a lot of time working thru this and not doing new trials.

·  Merrick: Unconfident about the prospects of a detailed predictive theory. But, explicit ideas of how something will work seems like a good idea.

·  Not really sure what the “best bet” looks like

·  Jeremy: Goal of the meeting: same mistakes have been made over the years. If we can start to think about some of the basics of what we should do – this would be helpful – so how to take it forward in the field. This is a “call to action” – to be more effective, rather than just repeating trials.

·  How do we best do it is the real question

·  Jamie Brehaut: the cumulative meta-analysis tells us that we are not getting better – most likely because we don’t have any theory. Very little theory went into the development of these trials –need to develop more thoughtful AF.

·  Susan Michie: 3 research questions that we talked about 1.) Can we design a better AF intervention as a whole? 2.) What is the relative contribution of components within an AF intervention? 3) What are the mechanisms by which the effects are happening?

·  Anne Sales: We don’t want to necessarily come up with a top 5 today, but to leave tomorrow with some possible studies?

·  Even if you don’t think you have theory – you have theory! Need a program theory – some things you deprioritize and some things become more vivid. Can trade off one program theory for another.

·  Plan: To suggest where things could move forward.

Group exercise 1: Susan’s group presentation

·  Goal standard audit

·  Visibility of required behaviour

·  Modifiability of behaviour (co-morbidity of pts, extent to which it is automated, habit strength)

·  Engagement with setting the goal

·  Number of goals

·  Quality of goal

·  SMART goals (achievable, relevant, time bound).

Other group’s additions: Acceptability (fits into above list), actionability, strength of evidence in relation to the guideline (fit with SMART category)

·  Feedback:

·  Competence of deliverer (e.g., how to give fb)

·  Seriousness of outcome

·  Profile/importance of outcome in society

·  Salience/vividness/emotional connotations of fb

·  Recipient seeks fb rather passively receiving it

·  Type/aggregation of behaviours

·  Multiple behaviours

·  Framing(positive vs. negative/gain vs. loss)

·  Complexity of target (different types, draw on theory)

·  Credibility of source

·  Goal orientation (motivation, mastery vs. learning) of recipients

·  Intrusiveness of intervention vs. part of daily routine

·  Visibility of the feedback

·  Timeliness

·  Interpretation, time to review

·  Opportunity to reflect and discuss (e.g. on causes of behaviour, instances of successful performance)

·  Support from management/leaders for feedback given

·  Graded feedback ( starting positively)

·  Multimodality

·  Format re new media

·  Linked to key messages

·  Feedback to teams

·  Longitudinal feedback - to tell a story

Other group’s additions:

·  How feedback is collected (e.g., self-reported), counts vs. priorities (% achieved or appropriateness),

·  Jeremy: cognitive understanding of the AF (comprehensibility of the feedback)

·  Timing, frequency – hard to tease out

·  Context – picking apart things piece by piece isn’t going to work; interventions need to consider if they are patient level, organizational level etc, what are the elements that are modifiable?

·  Action Plan – the group had little to say on this.

·  Involvement with creating action plans

·  Nature of action plans ( e.g. “ if then plans”)

·  Others groups addition:

o  Availability of tools for action

o  Who controls the action? (Anne Sales)

o  Who’s developing the action plan? (Jeremy)

·  Characteristics of the recipients

·  Past experience of AF as more or less useful

·  Alignment with goals, values etc of recipient re. receiving feedback and changing behaviour

·  Knowledge of the goals, and shy they are there

·  allegiance to goals

·  Perceived social comparison with groups/individuals/social norm

·  Trust of the organization

·  Autonomy of recipients

·  Intuitive vs. reflective orientation

·  Learning style

·  Authority of recipient

·  Degree to which recipients are organized/self-regulation

·  Education level/reflectiveness

·  Responsibility of practice

·  Union membership

·  Other group’s additions:

o  Was the recipient a volunteer?

o  Personality traits of the recipients

o  Is feedback given to individual or a group?

o  Time that AF is being delivered

o  Considerations of equity (fairness of comparison)

o  Motivation for change

o  Perceived consequences of achieving the goal

o  Self efficacy (confidence in being able to do it)

·  General/ External aspects

·  Combined with other interventions/elements (how to define these?)

o  Financial, or other consequences of AF

o  Whether AF is evaluated as a process

·  Automation of practice

·  Preparation/anticipation for intervention in advance (e.g., present audit details, and that they will be assessed)

·  Time spent on intervention

·  Engagement in designing AF

·  Alignment with goals/values, etc of recipient

·  Other group’s additions:

o  Comment on automation – control behaviour through system design – via technologies, financial incentives, public reporting – is it different from AF?

o  Patient activation (patient feedback for a physician)

o  Action planning as a co-intervention

o  Prevalence of the condition being audited (e.g., if it’s rare – doesn’t matter how much you get) ---AF may be much better suited for common conditions, as opposed to rare/expensive interventions. Note: Heather added that based on the extraction – only 3 instances where this was reported in the literature.

·  R. Foy’s comment: We’ve been discussing elements of an intervention and other things that are apart of context. Hard to generalize from one study to another if we don’t describe these in our studies. What’s the risk of repeating this work? Number of existing taxonomies describing context.

·  ***Susan: what is AF most relevant for in terms of type of behaviour, situation etc., nuts and bolts of the actual intervention, context in which it happens, mechanisms of action...

·  Mechanism of Action:

·  Relationship between components and outcome (e.g. Curvilinear)

·  Is this really point a mechanism (Anne)? Association between type of AF and the type of outcome. Basically - more is not always better – not necessarily linear.

FOLLOWING SMALL GROUP SESSIONS – RESEARCH AREAS THAT SHOULD BE ADDRESSED TO MOVE THE AF FIELD FORWARD

·  Jill Francis’s Group

·  Discussed the need to have a publication for the design principals of AF

·  Need for reporting guidelines – but then steered away from this idea – they don’t always work that well

·  Discussion about how clear those principals could be. (e.g., could we say “don’t do AF once” ----do we have the evidence to support that?)

·  Leaders of AF – many areas and pieces that can be manoeuvred

·  Need to define AF in terms of its prototypical and discretionary elements – what needs to be there in order to be minimally defined?