Minutes from NCI Teleconference on Use and Presentation of NCI Data
4/8/09
NCI staff participating: Sarah Taub, Josh Engler, Karen Auerbach, and Cheryl Sartori from HSRI; Chas Moseley and Nancy Thaler from NASDDDS
States on the call: AL, AR, DE, GA, IL, KY, LA, ME, MO, NJ, NY, NC, OH, OK, PA, SC, TX, VT, WA, WY, MEORC
This is the first in a series of teleconferences. Two states are going to present on how they are using their data: Dana Olsen from PA and Ging Fernandez from NC.
During the second half of the call, we will also discuss the expanded reports we’ll be developing.
Provider Profiles - Presentation by Dana Olsen and Lee Stephens (PA)
Dana Olsen- we have done 2 pieces, one on provider profiling and the other is very user friendly.
For the provider level report, the purpose is to make sure that the residential provider gets to see what their performance is. We base this on NCI data as well as about 70 other questions we ask in PA.
Over the course of the year we conduct about 7000 surveys. The report is available at any time, and it is refreshed every month. The provider gets a good idea of their performance. You need good administrative support to use this information as part of a QM system. We provide TA on the county level on how to use the data for quality improvement.
Lee- the Statewide Summary is a user friendly report. As you can see, there are relatively high percentages of satisfaction in living and work. A little lower in whether people get to go out weekly. We see that we have some work to do on choice, voting and having a key to your own home. Relationships were pretty high. This report was created for people with disabilities and their families to be short, sweet, and easily understood. We send this out through a regional coordinator. We’ll be deciding how they will be disseminated to the public and how they have been received by the public. Usually we present findings at annual meeting. We use the data for QI, not for punitive measures.
Q: Ray Pierce from NY- Does the public have access to these reports?
A: No, providers do and administrative entities do
Q: Ray- Can providers see other providers’ reports?
A: aggregate only for now. Eventually items will be collapsed into “domain” scores that will be available to the public as provider profiles.
Q: Ray- How do you determine sample?
A: Random by administrative entity. We also try to do 1/3 of the population by year. Done by county or county joinder
Q: Ray- how do you fund interviews for 7000 people?
A: State funds and we get federal match. There is a strong support because it is part of our QIS framework.
Discussion about using NCI data to meet CMS Waiver Requirements
HSRI- Meeting CMS waiver requirements is an issue that states have been thinking about and looking into the use of NCI data. On our end, we’re looking at the current set of indicators and how we can tie them into 3.5 waiver applications and renewals.
Lisa from WA- we were recently trying to get a new waiver. CMS was not happy with the measure we were proposing to use, because the family survey response option is stated as “usually or sometimes.” CMS says it needs to be “always” – this cannot be separated out on the current survey.
NC- we did not have any problems with that.
Chas- were they looking for a yes/no?
Lisa- we couldn’t separate always or usually so they said we couldn’t use it.
Sarah- That’s a good example of what we may want to change before the next version.
Sarah- Any other problems that states have had using their NCI data for CMS?
No answers.
Chas- Any suggestions on how to get this aligned with CMS?
NY- I would support NCI aligning with the 3.5 waiver application. Also it would be helpful to try and develop indicators that are consistent with system transformation goals/principles. It would also be helpful to develop empirically-based scales or domain scores (in addition to choice and community inclusion) that could be used to measure performance over time. It would be good to have consistency between what the states are developing and what HSRI uses.
Karen- We’re working on creating scales for each of the family survey domains.
Sarah- We also have funding from NIDRR to do additional analyses on Consumer Survey data. It supplements what we can do with NCI data and we will be exploring the development of additional scales.
Nancy- what specific indicators/measures have you used to provide evidence?
Dana- we will get you a copy of the framework.
NC- we used 2: Do you know your Case Manager and satisfaction with services.
Sarah- a few states have used the family survey for measures such as choice of provider and choice of services. Maybe we can ask states to share what they have used and what has been approved by CMS. We will compile and send out those examples. Also, we have a draft crosswalk of NCI indicators and CMS assurances. We can distribute the crosswalk and states can provide feedback.
Discussion about new reports under development for NCI states
HSRI- We’re planning on still doing summary reports for each survey, and we’re also adding 2 things.
1. An Annual report that highlights major findings, trends, or themes across states. It will be aggregate data by indicator. Chas encourages you to share your thoughts after you receive the draft report.
2. Generating state level reports for each state that compares your state with national results.
Karen- the new report will be sent out annually, it is a nice way to look at results for your state compared to the national average. It starts out with a general overview of NCI, any program changes in the past year, and a list of current indicators. The bulk will be results which will be mostly in chart form.
NY- one thing that we struggled with is that statistical differences may not be significant.
WA- One state or maybe two would be very far ahead which would skew the average for the national average.
Karen- we’re looking at different sample sizes from different states and how to best compute the national average.
Any comments?
Dana- one graph we find useful is one that shows proportions of people in different types of services. It would be good to include some of these demographics in the state reports.
Lisa-Missing how things have changed over time. For states that have been in NCI for a long time, how have things changed?
Karen- we did an aggregate report over 5 years and we didn’t find much change.
Nancy- are provider report cards made public? Is NCI going to be public?
Is there a way to allow NCI to get to a report card level of analysis?
Presentation on Sharing data with families – Maria (Ging) Fernandez, Karen Lukken (NC)
NC-Ging Fernandez- We share our data with developmental centers and LMEs (see handouts for example reports). Each LME gives us 40-50 consent forms so we get enough information. We have a team of 45 interviewers across NC and the majority are family members or consumers themselves. Many are members of CFACs or Arcs. We have trainings at least once a year. We share with interviewers the data from the previous year during their training. The powerpoint handout is an example of a presentation we give at the interviewer training session.
We are especially proud of the brochure (see handout). This was developed using Washington State’s brochure as a model. The handout is not a final version so please do not distribute. We will share the final version with everyone.
Next teleconference will address the topic of sampling.
FOLLOW UP ITEMS
1. Send out a current crosswalk of the NCI measures with the Medicaid assurances referenced in 1915(c) waiver application version 3.5. (see attached)
2. Outline a plan for improving the alignment between the NCI indicators and the Medicaid waiver assurances going forward. Also consider adding/identifying indicators that are consistent with system transformation goals/principles. Need to solicit performance measures that have been approved by CMS. (HSRI will solicit examples from states – WA, NY and others. HSRI/NASDDDS will outline plan for improving alignment and will share with states for feedback.)
3. Establish a small committee to investigate the development of an NCI tool/format to be used in the assessment or tracking of provider performance and, possibly, the future development of provider report cards (NY and others). (HSRI/NASDDDS will convene)
4. Improving the family survey. (CRAFT committee is working on this task)
5. Addressing sampling issues, particularly with respect to sub-state entities and providers. (HSRI will facilitate discussion and present options for addressing sampling concerns.)