Archived Information
Interim Evaluation of the Appalachia Educational Laboratory
I.Overview of Activities and Background
The evaluation visit to Appalachian Educational Laboratory took place on April 12-16, 1999 in Charleston, West Virginia. The background materials were reviewed and the rating sheets completed prior to the visit.
The regional lab (REL) is a part of AEL, a nonprofit corporation serving the four states of Virginia, Tennessee, West Virginia and Kentucky which was established in 1966 in the initial lab and center competition. The specialty area of the REL is rural education. AEL and REL share vision, resources, personnel and a building.
AEL is located in downtown Charleston and has been housed in the same place for over thirty years. The REL, the ERIC Clearing house and other AEL projects are located in this same building. Offices are spread across several floors, but project offices are by and large together.
The vision of AEL is that of “a world that nurtures a lifetime of learning for everyone”. To this end, AEL hopes to establish schools and communities where continuous improvement takes place. They aim to do this by the dissemination of applied research that joins the “wisdom of practice with the knowledge from research”. The projects at AEL aim to bring this objective about by addressing the issues of their stakeholders with strategies that research has found to be effective. They maintain a distinctly regional focus. AEL, according to the current executive director does not aspire to become nationally known. The executive director stated that “ We are terribly rural”. Given this, the service to the four-state region is its first priority. Also, AEL is not, as the executive director notes, an R&D center, but a corporation. As such it only has resources to the extent that it is meeting the needs of its constituency.
The overall work of the lab is directed by a Board of Directors who represent the various stakeholders in the region served by AEL. There are seven representatives per state, including the chief state school officer or designate, a representative for the school administration association, teacher education association and 3 at large representatives. The board is actively involved in setting policy and direction for the lab. In the discussion with the Board of Directors, several members stressed that active involvement did not mean that they micro-managed the lab. The board links the corporation to its constituency in very concrete ways. For example, board members may nominate schools for participation in projects of the lab.
The relationship between the lab (REL) and the corporation (AEL) is both separable yet intertwined. REL and AEL share resources, mission, governance structures and a building. The regional REL is the core of the AEL, comprising 45% of the funding total. REL uses AEL to support its efforts. For example, a math and science consortium in the regional lab might not be engaged in direct involvement with the schools but could call on experts at AEL to support this process and work in REL. AEL supports the programs of the REL. The Executive Director, noting the potential interpretation that REL might look like the marketing tool for AEL remarked that they “never set out to create saleable products but to create solution to the regions’ needs.”
The executive director has been at AEL for some 25 years and is stepping down this summer. They have reorganized the corporate structure somewhat in order to accommodate a transition period. A new director has apparently been named. The transition plan seems to be worked out, although it appears that, as is normal, there is still some change going on in the reorganization plan. There are also a number of key staff who have been there for many years.
The staff communicates via e-mail. The Executive Director noted that he tracks the e-mail of the resident directors in order to keep a heads up on how the work is going. Staff are assigned to projects. On average, staff are assigned to about 3 projects. The lab utilizes consultants where it makes sense to do so to staff a particular project.
II.Implementation and Management
A.To what extent is the regional lab (REL) doing what it is approved to do?
At this interim reporting period, the lab appears to be performing adequately to above expectations in fulfilling the details of the original proposal and modifications, with the possible exception of the rural specialty. The work appears to be completed in a timely fashion, although the original proposal did not provide exact deliverable dates, so this is somewhat difficult to assess. When a project is delayed, the reasons for the delay are usually noted in the quarterly project report. For example, the Kentucky Writing project experienced a minor delay waiting for the evaluation data from the Kentucky Department of Education which was noted and explained in the quarterly report. Similarly, QUEST was delayed due to the leave of absence of a key staff member. Also QUEST began as a project operating in four states, but due to the availability of Goals 2000 dollars in West Virginia, became a project only in that state. This redirection of effort to West Virginia had an impact on studies originally proposed, although this was not as clear to figure out. For example, The Prisoners of Time proposal vanished.
Strengths:
The outstanding strength of this regional lab is its clear vision, focus and unwavering attention to its mission of serving the Appalachian region over the last 33 years. It has been institutionalized in the region.
The ability to effectively collaborate with various stakeholders and to form strategic alliances is another important strength of this regional educational lab. The REL is actively collaborating with other labs, including SEDL, NWREL, as well as state Departments of Education, most intensively with the Kentucky State Department of Education. The staff of the laboratory are excellent in networking and making strategic alliances among the education community. For example, the QUEST project uses this strategic networking ability to successfully join together schools, teachers, and students in an innovative school change process. Another example of effective network is the Comprehensive School Reform network which is linking together schools and others engaged in comprehensive school reform under the comprehensive school reform, Obey-Porter legislation.
The Lab uses previous history and work in the region to leverage its continued work in region. REL capitalizes on its past accomplishments. For example, the current KERA project is a longitudinal offshoot of the original KERA work begun in 1990. This work was started when the legislature and other state officials realized they had not incorporated a feedback cycle in the original legislation. Fortunately, for the study of school reform, the AEL group was able to respond to this need. The data and methods from that study are continued into the present
project now being continued under the REL. Significantly, what started as a “fill in until the legislature fixes this mistake” project has turned into a valuable and singularly important longitudinal study of school reform that is valuable nationally.
The Lab is addressing significant issues in educational change, such as the effects of mandated school reform in Kentucky, and the development of tools and processes for school change and renewal in QUEST and CSRD.
The staff and management create an organizational climate that is supportive of each other and is committed to improving the life chances of children in the region it serves. The organizational climate helps reinforce the belief system and vice versa.
Areas of needed improvement:
Despite an overall positive view of the implementation and management, there are two areas of concern to be noted:
1)There is a lack of integration and consistent networking across projects. Although such things as retreats, Learning Labs and meetings of senior staff on a regular basis do take place, there is still a lack of awareness or use of a product developed in one project in another project. The different projects seem to rarely be customers for each other within REL. Across project communication needs to happen by design, not by chance.
2)Although the transition to a new executive director is well planned, possible deleterious effects of the transition, especially on the value and supportive climate of REL. should be monitored.
Recommendations for improvement:
1)Address issue of project isolation directly by looking for potential areas of cross project collaboration and design a specific program with measurable goals to accomplish this.
2)No specific recommendation, except that other changes suggested for the lab should take into account that there will be a transition period. For example, consider the recommendation to change the lab focus to be more national in scope. Although this may be a good idea, changing the focus at this time may be a terrible idea as the lab, which has been a remarkably stable place, is undergoing a significant change in leadership.
B.To what extent is the REL using a self-monitoring process to plan and adapt activities in response to feedback and customer needs?
The REL excels in using a self-monitoring process to give feedback on customer needs. It uses a variety of methods to gain feedback from its clients. There are customer satisfaction surveys, focus groups, internal reviews, external reviews, quality assurance control systems, advisory panels and a diligent board of governors. In addition, there is typically extensive pilot testing of products, if appropriate to do so as in the case of development of professional development materials. It also has in place a performance review process for staff. The brief task description taken from the technical proposal outlines the approach taken to self-monitoring and is reproduced below. This is an overall strength of the REL.
______
Task 1.2 Performance monitoring
Project performance review every four months
Project advisory committee meetings
Personnel performance review
Task 1.3a Quality assurance
Product quality review board
Complex review requires external review board, simple review does not stand alone.
Products require complex review issue papers, notes from the field, occasional papers, workshop packages
External reviewers must represent substantive expert and someone who is intended audience
Also use expert panels and focus groups early in development
Task 1.3b Evaluation
Formative evaluation - implementation
Summative evaluation - impact
GTR (Goal Tracking Record data collection system customer contacts)
Surveys with Western Michigan States assistance - mandated from RFP
Educator and administrator survey FY 96
SEA/LEA/schools survey
Summative evaluation- impact
overall satisfaction
skills
information sharing
use in planning
use in implementation
indirect benefits
Secondary clients use in implementation
External evaluation with Western Michigan
Third year evaluation or interim evaluation
Task 1.4 Planning
Project planning - internal if <100 days, external > 100 days
Has regular in house evaluations, third-party audits and advisory committee oversight
______
Strengths:
The REL consistently seeks feedback for its products and services from the clients. Descriptions of most products will indicate the type of review it has undergone. They also consistently seek feedback from clients once the product is in the field. The self-monitoring process exceeds expectations. In addition, the feedback is carefully designed to help improve the product. As such, questions are asked that can be analyzed and actually used to alter the product. For example, the CSRD workshop uses a combination of Likert type scale items and open-ended questions to locate the strength and weaknesses of the product or training.
The advisory panels also appear to work as conceived. In our conversation with Fred Hess, a member of the national advisory panel for KERA, he indicated that they met annually as a group, but conferred on a need be basis perhaps three or four other times a year. They also took advantage of AAA and AERA meetings to get together informally. Minutes of the advisory board meeting for KERA indicate a lively exchange about the conduct of specific research ideas. The REL is to be commended for convening this external advisory panel.
Areas of needed improvement:
Feedback from parents and students is not incorporated to the same extent as are other stakeholders.
Recommendations:
Increase attention to feedback from parents and students.
III.Quality
A.To what extent is the REL developing high quality products and services?
The certification by the NDN (National Diffusion Network) of QUILT is one example of external validation of the quality of this product. This was done earlier and not on the current clock, but demonstrates that the organization wants to develop techniques/products that have demonstrated quality that can be validated by a known process.
One measure of the quality of the work is the reputation that the Lab has for excellent, unbiased and useful work. Conversations with various state stakeholders revealed a common thread of trust in the research of AEL and belief that they are unbiased in their approach and fair in their conclusions. The REL’s work represents the gold standard for quality work for these policymakers.
There is a need to consider audience in any discussion of quality. The audience for many of the REL’s reports are primarily policymakers, legislators, and other state stakeholders interested in education. As such, the reports are not intended to be full-blown research reports. Consequently, the reviews of the literature document the research base for products and services, but typically in a very brief way. Also, theory is usually not developed in all that detail, which may be appropriate given the audience.
Another example of somewhat abbreviated treatment of prior research is the QUEST work. For example, other organizational development models that are similar to QUEST are not mentioned. This may be a fruitful literature for the REL to connect to, as there is a fair amount of work on using these models in school change, which may be relevant.
The REL does seek to minimize duplication of effort. For example, it was indicated that there was a request for a meta-analysis of the class size research. The REL did not set out to write a review, but instead to locate a current review and thereby reducing duplication of effort.
In summary, the REL does provide current reviews of the literature as background for work, but this background appears to be presented in an abbreviated form. Publication of materials in refereed journals is not frequent, and does not seem to be a priority for them. When asked about this, it was explained that the REL was not a university R&D Center, but a corporation and that there are different objectives. The emphasis is on kids, not on publications. There does not appear to be a push to publish in refereed journals. Authors do participate in annual meetings of the Anthropology Association and AERA.
The REL uses to good advantage a mixture of qualitative and quantitative methods in its studies. In the qualitative area, the studies rely upon observation, interview and document review and the triangulation of these sources. Traditional quantitative methods include examination of reliability and factor analytic structure of scales used in traditional surveys. These are well-documented approaches and used in a productive combination in the research. Also pilot testing and revision suggest intention to develop quality product.
On the other hand, the study designs seem to change sometimes and one needs to hunt for the justification or discussion. For example, in the Kentucky project that developed writing portfolios, the number of schools to be the base of study changed from 100 in the proposal to about 25 in the report. As noted earlier, given that the audience is not necessarily a research one, it may be appropriate to reduce discussion of methodology, although the study design and research plan should be accessible for the interested reader. There was no readily available consideration of the sample size reduction or whether this was a large enough size for the study at hand. Also, information about the characteristics of the districts in the KERA study were not provided, but this may have been for confidentiality reasons. Similarly, information about the interview protocols were not presented, to the best of my knowledge.
For relevant studies, there is an evaluation standards checklist included which indicates whether specific standards were taken into account in the study. This includes some 30 items such as stakeholder identification, evaluator credibility, practical procedures, rights of human subjects, impartial reporting, and meta-evaluation.
A report may go through several internal review iterations before it is approved, as was the case with several papers in our briefing materials. If appropriate, a methodological review will also be undertaken.