Evaluation Plan

December 2004

The Minnesota ADRC project is part of a broad strategy to rebalance long-term care in our state. The systems change component of the ADRC grant provides impetus to remove obstacles to services for consumers, county staff, and service providers while increasing opportunities for the provision of high quality information to targeted populations. Other rebalancing strategies taking place within Minnesota include implementing Consumer Directed Community Supports across all waivers, funding innovative LTC projects via Community Service and Community Services Development grants, encouraging long-term care private financing options (long-term care insurance and reverse mortgages), long term care consultation for all income levels, local planning and gaps analysis leading to service development, and expanding affordable assisted living capacity.

Minnesota has chosen to focus a major portion of the ADRC Grant on the creation of a web-based database, MinnesotaHelp.info and decision support tools. Minnesota is evolving a unique version of the resource center concept -- creating a virtual and physical resource network -- known as the MinnesotaHelp Information Network -- to provide people with resources and guidance in long term care planning and decision making. The model being developed for the MinnesotaHelp Information Network consists of Network Portals and the linkages among them. Resource Centers represent the Network Portals where people can access information in all forms -- virtual, telephone, print and face to face. Linkages that connect people and information are both technological and human. Counties, within the State of Minnesota play an important role in the planning and delivery of public long term care services as well as being mandated by state law to provide Long Term Care Consultation and Community Support Plans to all residents of Minnesota who request these services. HennepinCounty was selected to prototype the resource center development for the State of Minnesota for a number of reasons. The County contains one quarter of the state population, it contains rural, urban and suburban areas -- similar to those found throughout the state -- and is working on a major human services redesign that includes a systematic integration of long term care services.
Note: The MN eval plan does not include the following MDS: per FTE providing I&R,per 1000 service population, and # of financial eligibility determinations. Theseelements seem to be a product of the“one-stop” model. The“no wrong door” model that MN has adopted has a multitude of personnel providing I&A (more than AAA and county staff).MN will instead measure the effectiveness of the“no wrong door” model by measuring the decrease, hopefully, in time between intake, eligibility determination, and receipt of services. This element can be found in the Efficiency section.

MN Evaluation Team

  • Hal B. Freshley, PhD- ,651.215.9430
    University of Minnesota Humphrey Institute of Public Affairs, Minnesota Board on Aging.
  • Lois Yellowthunder, PhD ,612.348.9075
    University of Minnesota Department of Family Social Science, Hennepin County Human Services and Public Health Department.
  • Mary Chilvers ,651.215.0187
    Minnesota Board on Aging.

Evaluation Design

  • Mixed method evaluation design.
  • Pre (retrospective) and post-testing of the enhancements are possible – the model and the components pre-date the funding for the enhancements.
  • The MN plan includes a number of different feedback loops designed to ensure CQI.

Evaluation Methodology

Methodology indicated within the evaluation matrix

For each data element, indicate:

Timeline / Data Collection Interval and Frequency

  • Consumer Satisfaction Surveys
  • Senior LinkAge Line® - Each Region sends out a total of 30 surveys per month. In place.
  • Disability Linkage Line™ - Each Provider sends the survey to a minimum of 5 users a week. In place.
  • Survey of Older Minnesotans – annually. In place.
  • MinnesotaHelp.info Satisfaction Survey – available for use by anyone, anytime. In place.
  • Consumer Decision Tool Survey – will be available for use by anyone, anytime. In development.
  • Focus groups – annually, and as needed. In place.
  • MinnesotaHelp.info web tracking – annually. In place.
  • MinnesotaHelp Information Network – annually. In development.
  • MIS (Web tools and Linkage Lines), including client tracking– annually. In place.
  • HSIS(pilot county), including client tracking and LOC determination– annually. In place.

Responsible Entity

  • Responsible entity indicated within the evaluation matrix. Look for “RespPart” in the matrix. RespPart includes responsibilities for product development and/or data collection

Data Sources

  • Data sources indicatedwithin the evaluation matrix

Instruments

  • Senior LinkAge Line® CallCenter Survey, pg. 3
  • Senior LinkAge Line® Community Outreach Survey, pg. 4
  • Information Memorandum #45-04 pertaining to surveys, pg. 5
  • Disability Linkage Line™ Survey, pg. 6
  • Disability Linkage Line™ Survey Protocols, pg. 7
  • MinnesotaHelp.info web survey, pg. 8
  • Consumer Decision Tool web survey (in development), pg. 10
  • Focus Group Tips (focus group facilitation guide), pg. 14
  • Top of Form

The MN eval plan does not include the following MDS: per FTE providing I&R,per 1000 service population, and # of financial eligibility determinations. Theseelements seem to be a product of the“one-stop” model. The“no wrong door” model that MN has adopted has a multitude of personnel providing I&A (more than AAA and county staff).MN will instead measure the effectiveness of the“no wrong door” model by measuring the decrease, hopefully, in time between intake, eligibility determination, and receipt of services. This element can be found in the Efficiency section.

CallCenter Customer Survey

1. The Senior LinkAge Line® person who spoke to me was friendly and courteous.
Yes
No
2. I was able to speak to someone at the Senior LinkAge Line® within one business day of my initial call.
Yes
No (if no, please comment below)
3. I received help with my Medicare, prescription drugs, supplemental coverage or other health insurance issues through the Senior LinkAge Line®.
Yes
No
4. The information I received from the Senior LinkAge Line® was clear and understandable.
Yes
No
5. The information and assistance I received from the Senior LinkAge Line® helped me make a decision or find the service I needed.
Yes
No
6. The information I received from the Senior LinkAge Line® was accurate.
Yes
No (if no, please comment below.)
7. I would recommend the Senior LinkAge Line® to someone else.
Yes
No
8. Are you caring for a person who because of an illness or disability can't care for themselves?
Yes
No
Additional comments are appreciated. If you would like a response to your comments, please include your name and phone number.
______
______
______

Thank you!

For Office Use Only: /


Community Outreach Survey

Recently you received in-person assistance from the Senior LinkAge Line®. Please take a moment to respond to the following questions in order to help us provide a quality service.

  1. How did the Senior LinkAge Line® in-person representative assist you?

Organized bills and statements
Medical Assistance application
Researched a billing error
Help with prescription drug saving options
Compared long-term care insurance options / Compared health insurance options
Submitted claims to health insurance plan
Insurance appeal
Other ______
  1. Were you satisfied with the in-person assistance you received from the Senior LinkAge Line®?
    Yes
    No
  2. Did the in-person assistance received from Senior LinkAge Line® improve your understanding of health insurance benefits?
    Yes
    No
    Not applicable
  3. Did you receive information on protecting Medicare from fraud, abuse and errors?
    Yes
    No
  4. After you called the Senior LinkAge Line®, how long did you wait for a in-person appointment?
    Less than 3 days
    One week
    More than a week
  5. Would you recommend Senior LinkAge Line® in-person assistance to someone else?
    Yes
    No
  6. Additional comments are appreciated. If you would like a response to your comments, please include your name and phone number below.
    ______
    ______

Thank you!

For Office Use Only: /

Bottom of Form

12/1/05

Disability Linkage Line Customer Satisfaction Survey

  1. Were you able to speak to someone at Disability Linkage Line within one business day?

 Yes No (if no, please comment below)

  1. Was the Disability Linkage Line staff that helped you friendly?

 Yes No

  1. Was the information you received from Disability Linkage Line…

a)helpful?

 Yes No

b)correct?

 Yes No

  1. Would you recommend Disability Linkage Line to someone else?

 Yes No

Suggestions for improving Disability Linkage Line: use back or attach another page if needed:




12/1/05

Disability Linkage Line Survey Protocols

1.The goal of the survey is to measure customer satisfaction with Disability Linkage Line’s IR&A service provision. Survey results will be used as an accountability measure to meet federal grant requirements, to demonstrate our successes, and help to plan improvements.

2.Each DLL Provider will send the survey to a minimum of 5 users a week. If survey returns are low (less than 30%), some centers may be asked to send more surveys, or the survey will be changed to improve return rates.

3.Respondents will be selected using the following system.

-the first five callers in the morning of the 1st Monday of the month.

-the first five callers in the afternoon, the 2nd Tuesday of the month.

-the first five callers in the morning the 3rd Thursday of the month.

-the first five callers in the afternoon on the 4th Friday of the month.

1st Monday / 2nd Tuesday / Wednesday / 3rd Thursday / 4th Friday
First 5 in the am / Fist 5 in the pm / First 5 in the am / First 5 in the pm

4.After completing the call, the Disability Linkage Line provider will ask the customer if they are willing to participate in the survey, using the following script. It is important that the script is followed word for word, so that changes in the script can be tracked with response rates.

“Your call has been randomly selected to participate in a customer satisfaction survey. Would you be willing to help us improve this service by taking a one page survey through the mail? The survey is anonymous.”

IF NO: Thank the person for his or her time, and repeat with the next customer.

IF YES: “Can I please have your address to send the survey? [Get the person’s name and address] “Do you need in the survey in an alternate format?”

If you are following up with a caller, let them know that you will send the survey upon completion of follow up. If there are questions regarding the survey, tell them to call Lesli Kerkhoff at 651-634-2281.

5.Respondents will send surveys directly to the DHS. DHS will compile the results, and make them available for secured viewing on line. Please mail the surveys as soon as possible. The surveys must be sent NO LATER than 3 working days from the end of the IR&A service provision.

6.Mailings will be on DLL letterhead. The mailing should include:

-1 page survey with provider ID, year, user type, and staff code…. (see coding sheet)

-1 page cover letter (see customizations below)

-Return postage paid envelope to DHS in Care of Marsha Nadeau (DHS will provide).

-Print-out of service/referrals given to customer.

7.Each letter will have the following customizations. These are the only customizations that should be made.

-Line 1 – insert name of caller after salutation

-Line 2- insert the date of call or one to one assistance

-Line 3 – insert the name of staff who assisted the person

-Signature – staff who sends out survey signs the letter. Signer does not have to be the staff who provided the assistance.

8.Please send Customer Satisfaction Surveys separately and alone, do not include any program brochures, applications, personal notes, etc.

9.It is important that each DLL Provider handles the survey in a similar manner. We welcome your feedback and suggestions, and will work to automate as much of the system as possible. When changes are made to the survey or protocols, DHS will notify all of the DLL Providers through the List-Serv

12/1/05

Section One: Your Feedback
1. / Did you find the site easy to use?
Yes
No
N/A
If not, can you tell us which steps you had trouble with and suggest what would make the site easier to use?
2. / Were you able to find what you were looking for in the site?
Yes
No
N/A
If no, which topic did you choose?
What resources were you looking for?
What county were you looking for?
If no, you can e-mail questions or comments direct to or call 1-800-882-6262.
3. / Which term best describes the performance of this site?
More than acceptable
Acceptable
Slow
N/A
4. / If you found resources that were listed in the wrong place, please tell us the topic you chose and the name of the resource that probably shouldn't be listed there. We use your feedback regularly to correct errors in our topic classifications. Thanks!
Topic Chosen:

Inappropriate Listing Found:

Section Two: Tell Us About Yourself
1. / Age:
Under 17
18 - 29
30 - 49
50 - 64
65 - 84
85+
N/A
2. / Gender:
Male
Female
N/A
3. / Do you have a disability?
Yes
No
N/A
If you are willing, please tell us what it is?
4. / Are you looking for information for: (please check all that apply)
Yourself
Your spouse
Parent
Child(ren) under 18
Child(ren) over 18
Grandchild(ren)
Section Three: Suggestions
1. / Do you have any other suggestions for changes to this site?
2. / Would you like us to contact you once we have resolved this issue?
Yes
No
If yes, what is your contact information?

Top of Form

12/1/05

Consumer Decision Support

Section One: Your Feedback
1. / Did you find the site easy to use?
Yes
No
N/A
If not, can you tell us which steps you had trouble with and suggest what would make the site easier to use?
2. / Were you able to find what you were looking for in the site?
Yes
No
N/A
If no, which topic did you choose?
What resources were you looking for?
What county were you looking for?
If no, you can e-mail questions or comments direct to or call 1-800-882-6262.
3. / The information I received from the Consumer Decision Support Tool helped me make a decision or find the service I needed.
Yes
No
N/A
If no, please explain
4. / Do you think the use of the pie chart was helpful?
Yes
No
N/A
If no, please explain
5. / Do you think it helped you make a better decision about the type of care that is needed?
Yes
No
N/A
If no, please explain
6. / May we contact you in 30 days to see if the services you found were helpful?
Yes
No
N/A
Contact information:
7. / Which term best describes the performance of this site?
More than acceptable
Acceptable
Slow
N/A
8. / If you found resources that were listed in the wrong place, please tell us the topic you chose and the name of the resource that probably shouldn't be listed there. We use your feedback regularly to correct errors in our topic classifications. Thanks!
Topic Chosen:

Inappropriate Listing Found:

Section Two: Tell Us About Yourself
1. / Age:
Under 17
18 - 29
30 - 49
50 - 64
65 - 84
85+
N/A
2. / Gender:
Male
Female
N/A
3. / Optional - please answer if you are comfortable:
Do you have a disability?*
Yes
If Yes, what is your primary disability?
Chemical dependency
Developmental
Mental Health
Physical/Chronic Illness
Sensory (visual or hearing)
Other?
No, I don't have a disability
4. / Optional - please answer if you are comfortable:
What is your race/ethnicity?* (check all boxes that apply)
White/non-Hispanic
Native American
Asian
Hispanic/Latino
African American
Other
*If you used the site on behalf of someone else, please answer these questions about that person.
5. / Are you looking for information for: (please check all that apply)
Yourself
Your spouse/partner
Parent
Child(ren) under 18
Child(ren) over 18
Grandchild(ren)
Family member or friend
Client
Other
Section Three: Suggestions
1. / Do you have any other suggestions for changes to this site?
2. / Would you like us to contact you once we have resolved this issue?
Yes
No
If yes, what is your contact information?

Focus Group Tips

Preparation

  1. Think about the information you need.
  2. Think about who you will obtain information from – their experience, knowledge, work view, emotions.
  3. Think about how you wish to obtain the information – questions.
  4. Pilot questions before finalizing – you may not be asking what you think you are.
  5. Eight to ten participants + four questions = two hours.

Session

  1. Chairs in circle
  2. Refreshments
  3. Introductions – why we are here, what we need, and what the information will be used for.
  4. Circle introductions – beginning with self
  5. If cross talk becomes distracting, gently bring the attention back to the question of use silence until you have the attention of the group.
  6. Do not be afraid of silence; it can help the group focus.
  7. Allow ample opportunity for participants to ask clarifying questions, or add additional information – after each question and after all the questions have been asked.
  8. Allow for input on topics which were not covered by the topics.
  9. Neutrality of the facilitator is essential – be aware of body language, facial expressions, tone of voice. Do not allow yourself to be pulled into the discussion other than clarifying the questions.
  10. Record by using flip chart of note pad. Tape recorders can be used, but often make people uncomfortable and restrict conversation
  11. Be flexible – expect the unexpected.

Write up

  1. ASAP
  2. As much as possible avoid paraphrasing, omissions, etc. Do not add your own clarifications.

12/1/05

Bottom of Form

MinnesotaHelp Information Network Evaluation Plan – Indicators by Desired Systems Change Goals

December 2004

MDS elements in bold

Glossary(in order of appearance) – Network Elements = Information backbone of the Network ( Senior LinkAge Line®, Disability Linkage Line™, and Hennepin County Front Door/Resource Center); Network Portals = doors to information e.g. resource centers, Brookdale Library; RespPart= Entities responsible for product development and/or data collection; MBA = MN Board on Aging (SUA); CDT = web-based consumer decision tools; DSD = DHS Disability Services Division