DESPFRWG 25 09 2012 Agenda Item 2 – DES Post-Program Monitoring Survey
DES Performance Framework Review Working Group
Agenda item: 2
Date of Meeting: 25 September 2012
Title of Discussion Paper: Post-Program Monitoring Survey
Purpose
· To inform the Working Group about the DES Post-Program Monitoring survey and the collection of JSA participant experience data
Background
Post-Program Monitoring Survey
· The Post-Program Monitoring (PPM) survey has operated since 1987 and is primarily used to measure the labour market status of job seekers who received assistance through DEEWR-funded employment programs, including DES, with around 400,000 surveys sent to job seekers each year.
· PPM surveys are sent to DES participants throughout the year with more than 50,000 surveys sent in 2011–12 alone. Two PPM survey types are currently used for DES,
1) Employment Assistance/Post-Placement Support, and
2) Ongoing Support.
· DES achieves a relatively high 37.5 per cent response rate across both DES PPM survey types (see Attachment A: Post-Program Monitoring survey overview for further details).
· During the development of the DES PPM surveys, the department engaged an external consultant with experience in the disability services sector to undertake cognitive testing with a range of DES participants to ensure the appropriateness of the questions asked of the DES client base (see Attachment B: Cognitive Testing for DES PPM survey for further details).
Job Services Australia Participant Experience Measure (KPI 3)
· For Job Services Australia (JSA), the established PPM survey was selected as the most appropriate method to collect data for the JSA Quality KPI measure of Participant Experience. The PPM survey underwent cognitive testing and a pilot test to ensure the appropriateness of the additional questions asked and that they were understood by participants.
· Currently, the scores derived for the JSA KPI3 Participant Experience Measure are based on job seeker responses to questions about the services they’ve received as set out in the JSA Service Guarantee and the job seekers’ extent of satisfaction or dissatisfaction with their Employment Pathway Plan being tailored to their needs. The method for calculating the final scores for the JSA KPI3 Participant Experience Measure were developed by an external consultant, with expertise in developing experience measures (see Attachment C: Job Services Australia KPI3 Participant Experience measure for further details).
Issues
· The production of robust DES provider-level PPM results is currently limited. Any increase to the existing sample sizes will increase survey costs. Given that a number of DES Contracts are relatively small, it cannot be guaranteed that all Contracts will be able to get a statistically reliably and robust assessment through the PPM survey.
Other attachments
· Attachment D: Sample DES PPM survey cover letter
· Attachment E: Sample DES Employment Assistance/Post Placement Support PPM survey
· Attachment F: Sample DES Ongoing Support PPM survey
FOR NOTING AND DISCUSSION
Attachment A: Post-Program Monitoring survey overview
The Post-Program Monitoring (PPM) survey has operated since 1987. The survey is used to measure the labour market status of job seekers who received assistance through DEEWR-funded employment programs with around 400,000 surveys sent to job seekers each year.
Aggregate survey results are used for:
· reporting against the effectiveness indicators for employment services in the Portfolio Budget Statements and the department’s Annual Report
· reporting in the Labour Market Assistance Outcomes report
· input into the KPI 3 (Quality) under the Job Services Australia Performance Framework
· employment program evaluation
· policy development.
The PPM survey is conducted on an ongoing basis and every two weeks a new cohort of participants is selected for inclusion in the survey. This approach makes the large sample easier to administer, and allows seasonal variation to be accounted for. Additionally, it decreases the likelihood of running out of samples as individuals enter and leave the system over the course of the year.
The following process has been adopted to achieve the highest possible response rate:
· A survey is mailed to participants with a reply paid envelope and a letter from the department explaining the purpose of the survey and encouraging response
· Three weeks following the first mail-out, a reminder letter (including a replacement questionnaire and reply paid envelope) is sent to participants who have not yet responded
· Three weeks following the second letter, non-responding participants are called and invited to complete the survey over the telephone.
Special arrangements are in place for visually impaired participants with a paper-based survey sent, allowing for them or a nominee to complete on their behalf. Reminder notices are not sent to visually impaired participants, but they are called sooner if a survey hasn’t been received.
Participants are considered in-scope for inclusion in the survey three months after various events, eg. participation in an activity, exiting a stream, exiting assistance, being placed in a job, etc... though it varies between employment programs. Participants may be included in the PPM survey while they are still receiving assistance from a provider. For DES:
· A one-in-three sample selection is applied with two survey types used:
1. A job seeker will be recorded as being in-scope for a DES Employment Assistance/Post Placement Support PPM survey if they:
o exit from DES, including achieving a 26 week outcome and exiting as a fully independent worker;
o achieve a 26 week outcome and progress into Ongoing Support; or
o reach 12 months in DES assistance without having progressed into Ongoing Support.
2. A job seeker will be recorded as being in-scope for a DES Ongoing Support PPM survey if they:
o exit from DES (and were participating in the Ongoing Support phase); or
o reach 12 months in Ongoing Support.
More than 50,000 DES PPM surveys were sent in 2011–12 with relatively high response rate. More than 37 per cent of Employment Assistance/Post Placement support surveys sent were completed by participants. This level of response is largely consistent across most disability types. PPM surveys sent to Ongoing Support participants have a slightly higher response rate of 41 per cent.
Tables A.1 and A.2 provide further detail.
Table A.1: DES Employment Assistance/Post Placement Support response rates – 2011–12
Disability type / Response rate / Surveyed / Responses / Return to sender / Total in-scope populationPsychiatric / 38.6% / 25,914 / 9,835 / 404 / 83,110
Physical/diverse / 34.7% / 18,126 / 6,172 / 348 / 58,371
Intellectual / 36.5% / 1,998 / 715 / 41 / 6,700
Learning disability / 31.6% / 3,063 / 951 / 56 / 9,960
Sensory/speech / 35.9% / 2,842 / 1,007 / 37 / 9,126
All participants / 37.1% / 45,644 / 16,625 / 790 / 147,075
Table A.2: DES Ongoing Support response rates – 2011–12
Disability type / Response rate / Surveyed / Responses / Return to sender / Total in-scope populationPsychiatric / 40.1% / 2,270 / 898 / 32 / 8,413
Physical/diverse / 38.4% / 1,758 / 665 / 24 / 6,361
Intellectual / 39.5% / 362 / 141 / 5 / 1,337
Learning disability / 35.6% / 481 / 167 / 12 / 1,757
Sensory/speech / 38.5% / 364 / 139 / 3 / 1,283
All participants / 41.2% / 5,586 / 2,268 / 78 / 19,980
Attachment C: Job Services Australia KPI3 Participant Experience measure
Given that the PPM survey is a well-established process, it was decided that gauging a measure of JSA participant experience through the PPM survey was the most appropriate method.
Development
The department engaged an external statistical consultant, with expertise in developing experience measures. The consultant advised on the appropriateness and weighting of the Quality KPI questions to provide the best possible indication of the participant’s experience. To enable calculation of final measures that are equitable to providers and allow the compilation of responses from different streams, statistical methods are used. These sophisticated methods utilise international best practice for these types of measures.
The survey questions and calculations of the final scores were developed by the consultant with input from departmental statisticians. Furthermore, as the survey is used to measure participants’ experiences and is included in the Quality Framework data, detailed investigation about the actual sample design, questionnaire design, participant recall and other methodological factors were taken into consideration. This helps to ensure the collection is undertaken in a uniform manner, on a level playing field, and that results obtained from the PPM Survey are statistically robust and consistent for all providers.
The additional survey questions that were included in the PPM survey underwent a two stage development phase. The development phase involved cognitive testing of the questionnaire and an eight week pilot test of the entire survey process.
Assessment
The objective of the PPM survey is to be able to give accurate feedback of the receipt of services by participants in accordance with the Service Guarantees and Code of Practice on an ESA level.
Providers receive two scores against the Participant Experience measure and are based on survey responses for that Provider, derived from the:
· Participants surveyed at the ESA level that articulate they have received services as set out in the relevant Service Guarantee; and
· Participants surveyed who state their EPP was satisfactorily tailored to their needs.
The two scores are calculated independently and are represented on a 0 to 100 point scale. It is important to note that the scores are calculated through a statistical process, and are not percentages.
The results obtained from the survey are used for comparative purposes allowing providers to benchmark themselves against other providers and the industry as a whole. The scores are also comparable over time, and across Job Services Australia contracts.
Specifically for the KPI3 Participant Experience measure, a ‘top up survey’ is conducted for participants that may have not otherwise triggered a PPM survey to ensure that statistically valid results are available for each provider in each ESA. This survey sampling methodology has also been reviewed by an external statistical consultant.
The number of responses required for a particular provider at the ESA level varies depending on the number of participants who receive services from that provider within the ESA. The department only assesses providers on this measure where the results obtained through the survey are statistically robust.
THE NEXT FEW QUESTIONS ARE ABOUT <PROVIDER NAMEF1 / Did <Provider Name> provide or offer to provide you with any of the following?
(Tick “No, not needed” if not provided and not needed.) / Yes / No / No, not needed
Help to prepare or update your resume / ¨ / ¨ / ¨
Information about job vacancies in your local area / ¨ / ¨ / ¨
Access to computer facilities / ¨ / ¨ / ¨
Advice about how to look for work / ¨ / ¨ / ¨
Access to counselling or other professional support / ¨ / ¨ / ¨
F2 / And did staff at <Provider Name>:
(Tick “No, not needed” if not provided and not relevant for your situation.) / Yes / No / No, not needed
Clearly explain the types of help they will give you? / ¨ / ¨ / ¨
Discuss any jobs you have had in the past? / ¨ / ¨ / ¨
Discuss the types of jobs that you might like in the future? / ¨ / ¨ / ¨
Discuss your skills and education? / ¨ / ¨ / ¨
Discuss extra skills and education that might help you get work? / ¨ / ¨ / ¨
F3 / Did you sign a document called an Employment Pathway Plan (EPP) with
< Provider Name >?
(A sample EPP has been provided on the back of the attached letter.) / ¨ Yes
¨ No è GO TO F5
¨ Don’t know è GO TO F5
F4 / How strongly do you agree or disagree that the Employment Pathway Plan (EPP) you signed with them was tailored to you?
Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don’t know
¨ / ¨ / ¨ / ¨ / ¨ / ¨
F5 / Overall, how satisfied or dissatisfied are you with the assistance they gave you in terms of: / Very
satisfied / Satisfied / Neither satisfied nor dissatisfied / Dissatisfied / Very
dissatisfied
Providing help suited to your circumstances? / ¨ / ¨ / ¨ / ¨ / ¨
Staff treating you as an individual? / ¨ / ¨ / ¨ / ¨ / ¨
Staff treating you with respect? / ¨ / ¨ / ¨ / ¨ / ¨
Overall quality of service? / ¨ / ¨ / ¨ / ¨ / ¨
Page 2 of 6