SSIA Reablement evaluation workshop April 14th 2010

Objectives for the day were:

To develop an evaluation framework for reablement that

• can be used across Wales

• focuses on outcomes for service users and their families

• also encompasses outputs for organisations in terms of efficient use ofresources

The backgroundwas that reablement schemes have started to develop across Wales but there are no PIs that specifically relate to reablement, and no framework in place to help councils evaluate their services. Can RBA be used to provide such a framework?

At the workshop we started with an exercise – Remember the Future. Participants were asked to:-

•Imagine that it was 5 years from now and that their local reablement programme was truly established.

•Step into the shoes of some service users and write down as clearly and specifically as they could what the programme has done to improve their lives.

Ideas generated should provide a useful way of checking that any evaluation framework is fit for purpose.

The philosopher - Kierkegaard - said “Life must be lived forwards, but can only be understood backwards” and in many ways this is the underpinning rationale for RBA.

We took an overview of Results Based Accountability:

  1. In Population Accountability, we start with the ends we want (outcomes and indicators) and work backward to determine/ establish the means to achieve these ends.
  2. We establish indicator baselines showing where we’ve been and where we’re headed if we stay on our current course (the trend).
  3. Then we consider the story behind the baselines (e.g. the reasons why we might need reablement programmes).
  4. Next we consider all the potential partners who can contribute to improving outcomes.
  5. Then we consider what would help us to do better than the baseline (i.e. improve the trend), including what available research tells us, and what common sense tells us.
  6. Finally, we craft an action strategy that includes ‘no-cost/ low-cost’ actions over a specific period of time.
  7. A report card approach – provides an accessible means of communicating plans and progress against them.
  8. Performance accountability is the domain of services and programmes such as reablement schemes. Performance measures can be classified within three broad categories:

•How much did we do? (Generally the least important)

•How well did we do it?

•Is anyone better off? (Generally the most important)

  1. There is a ‘contribution relationship’ between the 2 types of accountability.
  2. The group identified the population outcomes that their reablement schemes typically contributed to:

Clarifying these outcomes ensures that services have an outward focus, are aligned to broader community goals, but are not held accountable for matters outside their control.,

We went on to identify key performance measures in the 3 categories set out above:

  1. We discussed the importance of a clear data development agenda where data is not readilyavailable to support performance measures. Developing new measures is costly and it is important to prioritise the measures that really need to be in place.
  2. Criteria for identifying headline measures should include the ‘public square’ test – is this a measure you could readily defend in the public square?
  3. Participants prioritised the following list of measures:
  4. Number of referrals and as % total of cases dealt with by ASC - MEASURE OF GROWING IMPACT OF SCHEME
  5. Visits needed and average length of intervention - EFFICIENCY
  6. Time from referral to service commencing - EFFICIENCY
  7. Saved bed days etc. - EFFICIENCY
  8. Unit costs and cost effectiveness - VALUE FOR MONEY
  9. Intensity of support required after reablement - % requiring no support; limited support; same support; increased support - EFFECTIVENESS
  10. % objectives in care plans fully / partially met - EFFECTIVENESS
  11. how long the impact lasts - SUSTAINABILITY
  12. % referred back into service
  13. % maintaining independence after 6 months
  14. User/carer satisfaction with how they are treated and results - SATISFACTION
  15. Staff job satisfaction – STAFF ENGAGEMENT
  16. Case studies (to illustrate the above)

The list needs further refinement. It is important to achieve a balanced set of measures that can help to appraise performance from a number of perspectives, and the list generated is a good starting point. Each measure needs to be developed into a workable indicator.

  1. A report card might be developed to form a basis for evaluation, looking perhaps as follows:

  1. This simple format can help schemes to address key challenges:
  • Who are our customers? End users, the organisation (which is seeking savings from reablement)
  • Are we contributing as we should to relevant population outcomes? Do we need other complementary services to do more or different things to optimise the performance of the reablement programme? EG – if loneliness is a problem that drives individuals into institutional care, whtever the success of their reablement, does more need to be done locally to support social inclusion?
  • Graphs to set out how the scheme is doing against each chosen performance measure.
  • The story behind each of these baselines to contextualise performance
  • Partner evaluation: partnerships are critical to the success of reablement, but are the right partners on board, and is their contribution optimal?
  • What is working – including low cost and imaginative ideas; and what needs to be tried next? Is there learning for other schemes?
  • What is the research and data development agenda that will allow the service to move forward in a progressive and systematic way.

Steve Pitt

April 2010