Preparing for the Journey: Building Effective Program Monitoring Systems
Welcome everyone. We are pleased that you have joined us today for this webinar title "Preparing for The Journey-Part One, Effective Program Monitoring Strategies Series".
And now it is my pleasure to introduce our speakers: Deanna Khemani, is a Member of the Technical Assistance and Training Team at Social Policy Research Associates. Prior to joining SPR, Deanna served as a Federal Project Officer in Region 6 for the US Department of the Labor, Employment and Training Administration. Diana also has extensive experience in evaluating Workforce Development Programs as a member of SPR Research Unit.
Our second speaker today is Bob Lanter, he is a new Executive Director for the California Workforce Association. He has extensive public sector and nonprofit experience, working in the field of workforce development and job training.
And now I will turn it over to Deanna.
Deanna Khemani:
Thank you all for joining us today. Many of you may know about social policy research associates because of our experience in conducting evaluations on workforce programs, but we also have a special technical assistance in- training unit, that draws upon those 20 years of experience in terms of providing technical assistance to states and local areas and the Federal Government.
We have specialized expertise in the following areas: In governance, employment, retention and advancement, education and training, and also management and operation, and since our inception we have worked really hard with clients to successfully improve program performance and also system design. We just wanted to let you know little bit about our organization. The Office of Regional Management contracted with SPR to provide technical assistance in training for organizations and agencies like yours that participate in federally funded workforce programs and initiatives for this current program.
In that role, basically SPR is providing technical assistance services on behalf of all six ETA regional offices throughout the country and we want to give a special thanks to regions 1, 2 and 6 for their support of this webinar series. As you already know from this day-to-day notice that we have sent out. This webinar is one of its three part series on WIA program monitoring. As you can see from the pictures on this slide, we are using analogy of a journey or a road trip to explore various elements of affected oversight and monitoring. And I am not sure about all of you, but I have had some really wonderful road trips and then I had some that have been complete disasters. I think what makes the road trip more successful is obviously preparation and having realistic expectations. We see oversight and monitoring really aligning with that idea. In terms of you need to prepare for your on-site monitoring and you also need to have realistic expectations for what you can do when you are on-site. So, we plan to layout this map or road map as we call it, to assess kind of where you are out as a State with respect to your oversight and monitoring framework.
We know our analogy is not perfect, but again we hope it explains a little bit of a picture about how you can look at your oversight and monitoring processes. Webinar 1 in title "Preparing for the Journey." In this webinar today, we are going to be talking about the state-wide vision, we will talk about the framework for conducting monitoring, we will talk a little bit about building staff capacity for these efforts, developing customized monitoring tools and then conducting risk assessment and attributes. Our second webinar, which is scheduled for June 6, we will focus on steps you need to take to prepare for and actually conduct your on-site monitoring and in that webinar, we are going to pay particular attention to different quality elements that you want to review when you are on-site.
Our final webinar is scheduled for June 20th. In that webinar, we will discuss communicating your results, once you're completed those desk reviews, your on-site visit, and any risk assessments that you do. What's the difference between a finding and an observation? How do you use the access conference? How do you submit your written reports? We will also talk about some of the most common findings that we see based on discussions that we held. So we know that many of you have systems in place and we hope that this presentation really helps you to identify areas where you can strengthen or improve your oversight and monitoring practices. We are aware that the organizational structures of many of the States on the phone right now will vary. Some of you are single local workforce investment areas whereas others are multiple local workforce investment areas. We try to keep the content of this Webinar broad enough so that all of you regardless of whether you are a single local workforce investment area or have multiple areas that you are responsible for, can gather some information and learn something from this. Because of the differences and how each State is structured, we are going to be using the term local sub-recipient in various places today. And for our purposes, this term refers to any organization or vendor with whom you provide federal funds to operate WIA program. So just keep that in mind as we are talking.
To gather the content for these webinars, we held informal discussions with State Staff responsible for monitoring in 15 states and we also had separate discussions with each of six ETA regional offices. We want to thank all of you on the line, who participated on those calls. Well, we know that they were brief. We know that we did gather some really valuable information about things that we think are working well and are not, with respect to oversight and monitoring and as a result of this discussion, we were able to gather some information about activities and tools being used by States across the Nation and we plan to share or sprinkle some of those highlights throughout this webinar series. I do ask that you hold any questions that you have until the end of the webinar. We will have a separate time for Q&A at the very end. We will be using the chat box for that Q&A discussion.
So, I would like you to do a poll right now and basically we are going to put a poll out here in a minute. The question is "What is the primary reason your state monitors", is it (a) To ensure Federal and State statues in regulations for workforce programs are being followed and there will be no disallowed cost. Is it (b) To identify areas requiring technical assistance. Is it (c) To identify promising practices and share information with workforce system providers and partners, or would you say it is (d) All of the above. We are going to move the poll on to the screen right now if you will make your selection for us, we would appreciate it. Great, I see people are making their selections. I will give you few more seconds let’s see what the answers are.
Sheila, can you pull the answers out for me.
Oh, great, I see here that over 92% of you basically selected (d) All of the above and this is interesting, because this really aligns with the dialogues that we had with many of you at the State and the Federal level. Monitoring should be used for all of these reasons. It should be used to ensure that obviously local sub-recipients are following Federal States statutes and regulations and any training and employment guidance letters. It should also be an opportunity for you to identify technical assistance and possibly deliver some TA while you're working with these local sub-recipients and then finally we hope and this is area where we think maybe states are doing this a little bit less is to use it to identify promising practices and to actually share those practices with your public workforce development system.
We however identified four primary reasons why monitoring is so important, probably the most common reason to monitor is to ensure again that program activities comply with applicable laws regulations and other administrative requirements both at the state federal and local levels. One thing that I want to say here is that in our conversations, we found that many states are really looking at the federal laws and regulations at OMB circulars, but not paying as much attention to their own state plans and local plans and whether local sub-recipients are following those particular guidances as well. The second reason to monitor is to help managers obviously to make decisions about what program activities and services are working well. This management monitoring is based on specific assumptions about what constitutes good or quality programs. You want to make sure that when you monitor sub-recipients that you are looking for what makes up a quality program and looking to see if those attributes actually exist within those programs. For example, you want to see do our local areas asses an individual skills in order to help that customer make a decision about what training or employment options are appropriate for them. A third reason to monitor is for evaluation. This is probably the least used of all of the reasons for monitoring across the states, but we see monitoring again being for evaluation purposes. This does not have to be something really concrete and systematic, but this allows you to look at the impacts of your programs, what's working well and what's not and being able to compare plans versus actual outcomes - and you do some type of analysis of all of that, and then finally monitoring helps to ensure that sub-recipients are meeting obviously their performance targets. We have fiscal and we have participant performance targets within WIA and we want to make sure that our local areas are meeting those requirements. Again, we talked to a lot of federal project officers for this project and we found that a lot of them felt like their states were focusing mainly on compliance.
So again we want to convey how important it is to really spend some time during your monitoring activities, looking at the quality of program operation. Again, these are the four main reasons. There may be others as well. I like this visual because it reminds me a little bit about our journey. You know this reminds of kind of like gears on a car.
We really need to work in coordination across both the federal state and local systems and oversight and monitoring really epitomizes that partnership between the federal state and local partners. DOL basically provides guidance to the national public workforce system, they oversee or monitor what states are doing with respect to implementing that public workforce system and they deliver technical assistance and training both the state and local staff. The state in turn provides policy guidance to local areas or sub-recipients, oversees local operations, and also delivers technical assistance as well, and then finally at the local level they provide the local administrative entities, they oversee their local service providers and contractors, and they have to ensure that they are providing technical assistance to those sub-recipients or those providers to ensure they are operating the program well. We really see dealing this system as the partnership that it really helps everybody involved to understand that quality monitoring and oversight isn't intended to be punitive. It really is intended to be a tool that drives your continuous improvement process and it does so through assessment, analysis, proactive action planning, and sharing promising practices and then identifying those technical assistance needs and actually seeing through that you can help the sub-recipients address those needs.
One thing that came out of our conversations again is that we need to really focus on customers and employers. That is ultimately what the public workforce system is all about. So believe it or not, your federal project officers within ETA are there to help you meet that need. They really do want to see themselves as your partner and help you to make the right improvements in your system. Sheila, can we go back one slide? Okay, you might be missing this one, but as a result what we learned during our discussions, we identified a number of key elements that need to be incorporated into a state monitoring framework. These include to ensure that the state agency staff understand what the federal laws and regulations say about monitoring, also to develop a sharp vision in framework for how you want your state monitoring system to work. We want to build the capacity of state staff to conduct monitoring activities, not just at the state level but also at the local sub-recipient level as well.
Next we want to develop tools that are customized to your policies and procedures at the state level, not just at the federal level. Next we want to incorporate ways to connect to sub-recipient operations. We want to do this through the use of things like risk assessments, desk reviews that are just done in preparation for on-site visits, but are also done more continuously to stay connected to local program operations, and then finally we want to include some kind of formal feedback mechanism into the design. You want to find ways to have a user friendly format in terms of giving feedback to your local sub-recipients and again you want to identify those technical assistance needs and share promising practices.
We have developed a tool that you can use to assess where you are as a state in terms of implementing this process and we have this file available for you under the material section on the right hand side. I would encourage you all to download that file and sit down with your staff that do the monitoring to see kind of where you are with respect to all of these various components that you see here in the state monitoring framework. I would like to turn the presentation over right now to Bob Lanter who will talk about the federal requirements for monitoring.