Equipped for the Future: a Standards Based Approach to Defining and Measuring Results In

Equipped for the Future: A Standards Based Approach to Defining and Measuring Results in the Adult Education and Literacy System

December 12, 2001

Good Morning,

I’m really looking forward to the next two days.

I want to thank Pat, and Kaeli, and Bob and the rest of the Steering Committee for putting together such a strong workshop that I know will provide useful guidance to the National Institute for Literacy and our EFF Assessment Consortium as we move forward in our efforts to develop quality performance assessments for Equipped for the Future.

Mike has talked to you about the Workforce Investment Act and the National Reporting System. As part of setting the policy agenda, I want to talk with you about three things:

How EFF has helped to redefine the results the adult education system aims for – to focus more directly on results that are important to adult learners as well as policymakers at the state and national level.
What our approach so far has been to developing a system to measure those results that is useful for three purposes:

Providing information on learner achievements and mastery that is useful to the learner as well as the teacher throughout the instructional process.
Providing information about what learners can do that is credible to employers, educational institutions, and policymakers, as well as to learners themselves, and
Providing information that is useful for program and system improvement and accountability.

What the implications of our work are for measuring and reporting learning gains using the National Reporting System.

Let me begin with EFF’s definition of results:

The National Institute for Literacy developed Equipped for the Future to try to resolve a major conflict in the field between the goals of the adult education and family literacy program – as expressed by both individual adult learners and the policymakers that established the program – and the means we have available for measuring progress toward those goals.

Our National Adult Literacy and Lifelong Learning Goal -- like the goals for the Workforce Investment Act focus on broad social goods. We want to know that adults are developing the knowledge and skills necessary to:

Gain employment and achieve self-sufficiency (Compete in the global economy, according to the NAAL)
Become full partners in their children’s education, and (again from the NAAL)
Exercise the rights and responsibilities of citizenship.

Those are the goals our students want to achieve – as well as the goals that are important to policymakers – the state governors who set the National Education Goals and the members of Congress who wrote the Workforce Investment Act.

And yet – the measures we have – of learning gains in basic skill content areas – have not been aligned with those goals. We talk about gains without knowing whichknowledge and skills are associated with achieving these goals. We talk about moving from level to level, without knowing what level of a given skill is associated with achievement of these broad goals in individuals' lives.

In fact, our model for defining and assessing results in adult education has been derived from the K through 12 model where we intentionally build -- over 13 years of schooling -- a broad foundation of knowledge and skills that will support effective performance in children's adult lives.

But adults do not come to adult education to build that broad foundation -- they come with very specific goals that are quite congruent with the broad goals for the WIA. And while they might like to be able to build that broad foundation – they don't have the time. As John’s presentation demonstrated – most attend for less than 100 hours.

Adults tend to come to literacy programs for what I think about as just-in-time skills and knowledge – the skills and knowledge they need to overcome a particular barrier they are facing in their lives at that moment: be it getting off of welfare, coping with a job that has been changed by the introduction of technology, trying to understand what the teachers are saying about why their kids aren’t doing well in school. And then they go back to their lives, to use that knowledge. When they come up against another barrier, when they confront another learning need -- they come back -- to learn, once again, what they need to be successful.

When we launched Equipped for the Future as a standards-based adult learning system reform initiative it was with the intention of developing a new definition of results – and a new approach to measuring results -- that was congruent with adult and policymaker goals.

Many of you in this audience know that we began this work by first looking at what adults do in the three primary adult roles reflected in the national goals: Using the same methods that are used in work analysis to identify critical functions and key activities, we developed “maps” for all three roles. Then we identified what effective performance looks like in those roles and used that as a starting point to identify necessary knowledge and skills.

[I should add at this point that our definition of effective performance was shaped by a set of broad purposes for learning, identified in the first phase of our research:

These learner-defined purposes are:

Access to information, so adults can orient themselves in the world;
Voice, so adults can speak with the confidence that they will be heard;
Independent Action, so adults can make decisions without having to rely on others to mediate the world for them; and what we called
Bridge to the future, learning to learn so adults can keep up with the world as it changes, and leave behind their fears of “moving nowhere.”]

Starting with what adults do -- with what they need literacy for -- led us to identify 16 core standards -- a broader range of knowledge and skills than is currently assessed in the federal adult education program. In addition to communication skills, we identified

A core of interpersonal skills that support teamwork in a diverse society,
A core of problem-solving skills that support effective decision making at home and at work, and
A core of lifelong learning skills that enable adults to keep up with change.

EFF proposes that this circle of skills define the domain of adult education and literacy programs. If the aim of our system is in fact to prepare adults to achieve the goals represented in the Workforce Investment Act and the National Goals, then teaching and learning, assessment and reporting need to be grounded in this broader universe of “new basics." Further, teaching and learning, assessment and reporting need to focus not just on what you know – but whether and how well you can use what you know to achieve purposes in your life.

You can see from this definition that when we started to think about assessment for Equipped for the Future standards – our focus on purpose and context had already led us to be thinking about assessing performance of complex tasks.

Before I go on to talk about this aspect of EFF – let me say that our definition of results that matter in the adult education and family literacy system was not developed in an isolated laboratory. It has been constructed through an iterative process: working with over 200 practitioners in more than 50 “laboratories” where teachers have tried out the standards, helped us refine them, and helped us build a framework for assessing performance.

Through their practice we have learned that this definition of results makes sense – that a purposeful and constructivist approach to learning, where learners needs and goals determine not only which of the 16 standards are focused on but what the context for learning should be – leads to better teaching and better results. Learners are motivated by lessons that not only focus on issues they are concerned about, but help them actually develop skills that enable them to do things better in every day life.

Let me turn now to our approach to assessment. The question we posed as we began this work was: What do we need to do in order to be sure that our approach to assessing the construct for each of our standards is firmly grounded in performance of meaningful, real world tasks? How can we be sure that what we are measuring is neither a decontextualized set of discrete knowledge and skills, nor a student's ability to simply perform a task, but rather how well a student can use the standard to perform a task.

The guiding principles we developed, with practitioners who were part of the EFF field research process included the following:

The EFF Assessment Framework must address learning over a lifetime.
The EFF Assessment framework must address a single continuum of performance for all adults- including those with only minimal and those with many years of formal education.
Each level in the EFF Assessment Framework must communicate clearly what an adult at that level can do.
The levels in the EFF Assessment Framework must be explicitly linked to key external measures of competence so that adults and systems can rely on them as accurate predictors of real-world performance.

These principles made clear to us that we had to build our assessment framework on a developmental model of cognition and learning over time. Drawing on cognitive science research on key features that differentiate novice and expert performance, we determined that what we would try to do was to “map,” for each of our standards, an adult continuum of performance that elaborated what performance of the standard looked like all along the continuum – from novice to increasingly expert performance.

We have begun this ambitious work with the help of practitioners and programs in five partner states – Maine, Ohio, Oregon, Tennessee, and Washington. In order to be sure that our developmental model reflects what we see in the performance of actual adult learners, we have asked these practitioners to collect data on learner performance using EFF performance tasks -- learning activities with embedded assessment. The criteria for these tasks includes the following:

Each task:

Explicitly addresses all of the components of performance of the standard
Has a clearly defined purpose
Has clearly defined roles for the learner and the teacher
Represents a meaningful, real-world application of the standard

With the data we have collected over the past two years we have been able to identify the key features of such a developmental continuum for 4 of the standards; By the end of this year, we hope to have sufficient data to elaborate a similar developmental trajectory for eight more.

This brings me to the concluding section of my presentation – how this work connects to efforts to measure performance on the National Reporting System.

I want to begin this section by talking about two more of the principles that have guided our assessment work:

The EFF Assessment Framework must address multiple purposes for assessment.

The EFF Assessment Framework must support a multidimensional, flexible, and systemic approach to assessment.

At the beginning of my talk I mentioned three critical purposes for assessment that an EFF assessment system has to address:

Providing information on learner achievements and mastery that is useful throughout the instructional process.
Providing information about what learners can do that is credible to employers, educational institutions, and policymakers, as well as to learners themselves, and
Providing information that is useful for program and system improvement and accountability.

Through our work with the 81 practitioners who have been part of the data collection process described above, we have developed considerable knowledge about how teachers can use standards-based learning activities with embedded assessment to develop a clearer focus on what students know and what they need to learn in order to carry out important tasks in their lives that require them to use one or more of the EFF Standards.

But our work also has to provide information that is useful for program and system improvement and accountability – in the case of the Adult Education and Family Literacy System – through the National Reporting System.

We have begun our work in constructing the developmental picture of cognition and learning I talked about with the standards that are currently in the National Reporting System – Read, Write, Speak, Listen, Math -- so that we can contribute a more robust, theory-based picture of what performance looks like across each of the NRS levels for ABE and ESL for each of the standards.

We also are in the process of developing a set of performance tasks that practitioners will be able to use to not only assess movement from one level to another on the NRS, but also to assess movement within each level.

We look forward to the guidance that emerges from the work of this BOTA Committee to ensure that the tasks we develop meet the highest criteria for validity.

This brings me to the final principle I want to talk about: support for amultidimensional, flexible, and systemic approach to assessment.

The overriding goal of EFF is adult literacy system reform. In a paper on Understanding Educational Quality Eva Baker identified three criteria for an assessment system to have a positive impact on systemic change and improvement.

All tests measure the same domain
Domain definitions and characteristics are made public.
Improvements on a test for one purpose will transfer to gains assessments intended for other purposes.

We know the key to standards-based system reform is alignment.

In basing our assessments on the standards, we are assuring that all assessments measure the same domain.

In building the kind of cognitive map I talked about…we are beginning the process of making those domain definitions and characteristics public.

In making them public – and the ground of instruction as well as assessment, available to all test developers – we hope to provide the basis for assuring that improvements on tests for one purpose will transfer – not only to gains on assessment for other purposes – but to real differences in adult learners lives. Then our system will truly be able to measure and report results that matter.

Thank You.

Sondra Stein, National Director

Equipped for the Future

National Institute for Literacy

Page 1