Nunn Consulting, Inc.

LOS ANGELES COLLABORATIVE FOR TEACHER EXCELLENCE

Evaluation Report Year 4

March 2000

LACTE Evaluation Report Year 4 – March 20001

Executive Summary

In 1995, National Science Foundation (NSF) funded Los Angeles Collaborative for Teacher Excellence (LACTE) to improve the preparation of math and science undergraduate students for teaching careers in elementary, middle and senior high schools. NSF believes that by increasing the number of K-12 teachers with backgrounds of math and science, students will improve learning around math and science. The agency has funded several sites as part of their nationwide Collaborative for Excellence in Teacher Preparation initiative. All of these collaboratives have at least one 4-year and at least one 2-year institution of higher education that work together to promote teacher preparation for science and math students. LACTE and the other CETP grantees have the daunting task of increasing interest in teaching among science and math students who undoubtedly have other, often more lucrative, career options. In addition, the collaborative is responsible for providing learning experiences that will prepare students to become effective teachers at the K-12 level. Furthermore, each CETP site must make efforts to encourage the participation of female and ethnic minority students in the teacher preparation programs, so that in the future, K-12 science and math teachers will be reflective of the students in their classrooms.

This report summarizes evaluation activities conducted in 1999 by Nunn Consulting, Inc., a for-profit management-consulting firm. The aim of the outcome evaluation was to assess the effectiveness of LACTE activities in adequately preparing students for teaching careers and to establish mechanisms to document the program’s impact on students, faculty, and institutions for the remainder of the grant. The current Evaluation Consultant assumed responsibilities from the previous consultant, the Evaluation and Training Institute. Evaluation activities for this grant period were focused on becoming acquainted with the LACTE program. The new evaluation team was also able to develop and revise the scope of work and contract, design and revise data collection instruments, collect, analyze and interpret data from the student survey, the “Meet the Teachers” Roundtables, the LACTE Student Symposium and the New Teacher Science Workshop.

This report focuses almost exclusively on the impact that LACTE has had on the preparation of undergraduate students for K-12 teaching careers. Findings reveal that interest in and preparation for teaching careers is increased when students participate in internships, have the opportunity to observe and/or assist in a K-12 classroom, and attend campus student groups meetings. Juniors and seniors are more likely than freshmen and sophomores to have participated in such activities. Also, the results demonstrate that the Student Symposium is often the first point of contact that students have with LACTE. Furthermore, the “Meet the Teachers” Roundtable and the Student Symposium provide LACTE students with an opportunity to learn hands-on teaching techniques and interact with K-12 teachers.

This report also presents results of the New Teacher Science Workshop. Twenty K-12 teachers participated in a summer workshop to learn and practice the inquiry method of teaching. Forty (40%) percent of these teachers were still in training. Participants felt that workshop not only modeled the appropriate way to use the new teaching technique in the classroom, but also provided an opportunity for teachers to network.

More information is needed on the LACTE’s impact on new teachers, LACTE faculty and participating institutions. The evaluation activities planned for the upcoming year are outlined at the end of this report.

Finally, the following recommendations were made to strengthen LACTE in the upcoming year:

Emphasize substantive learning experiences for LACTE students

Involve freshmen and sophomores in LACTE activities early

Ensure that LACTE students obtain core experiences for teacher preparation

Become more strategic with the Student Symposium

Continue teacher training and encourage connection with LACTE and among teachers beyond college.

Table of Contents

Executive Summary

Background

Summary of Evaluation Activities

Evaluation Team – Roles and Responsibilities

Evaluation Scope of Work......

Evaluation Activities

Program Outcomes

Student Survey

Specific LACTE Events

Conclusions and Recommendations

Conclusions

Recommendations

LACTE Evaluation Report Year 4 – March 20001

Los Angeles Collaborative for Teacher Excellence

Evaluation Report Year 4
Background

The Los Angeles Collaborative for Teacher Excellence (LACTE) was funded by the National Science Foundation and established in 1995 to address the issue of enhancing science and math education at the K–12 levels. This collaborative consists of ten institutions that serve the Los Angeles area. The institutions consist of five universities/colleges paired with their feeder community colleges.

Four-Year Institutions / Two-Year Institutions
California State University, Los Angeles / East Los Angeles College
Loyola Marymount University / Santa Monica Community College
California State University, Fullerton / Fullerton Community College
Occidental College / Glendale Community College
California State University, Dominguez Hills / El Camino Community College

The National Science Foundation funded several Collaborative for Teacher Preparation programs throughout the United States. Each collaborative developed its own plan to improve the quality of instruction in K–12 education. In particular, NSF required participating collaborative programs to focus on increasing the number of ethnic minority students who become mathematics and science teachers in elementary, middle and high schools. The Los Angeles Collaborative for Teacher Excellence has identified the following goals for its project:

  • Increase the number of mathematics and science majors from underrepresented groups who plan on teaching as a career
  • Develop a course of study at each institution that integrates mathematics, science and technology in the preparation of undergraduate students
  • Develop an experiential component for future elementary mathematics and science specialists and secondary mathematics and science teachers
  • Establish a support network in each institution for prospective teachers

LACTE has identified four components in which these goals will be accomplished:

Network Development – Activities of this component are focused on building the Collaborative by establishing linkages within and across each participating institution. This includes, but is not limited to, the recruitment of mathematics and science majors; encouraging the participation of science, mathematics, engineering, and education faculty to LACTE; and establishing support networks for both faculty and students. In addition, the Network Development component will be aimed at disseminating information about LACTE sponsored activities outside of the Collaborative.

Curriculum Development – This component focuses on developing courses that meet the various national and state standards in teacher preparation and credentialing. The Collaborative intends to pilot and implement seven new courses at various campuses.

Faculty Development – This component involves monitoring the faculty who will teach the LACTE-sponsored courses, encourage collaboration among LACTE faculty, and provide faculty development workshops that will train faculty on new teaching techniques. The Faculty Development component will also be instrumental in increasing the awareness and ensuring the participation of mathematics and science faculty in teacher preparation. This component will be integrated with the Curriculum Development component.

Work Experience Opportunities – This component is aimed at providing students with valuable field experiences to prepare them for a teaching career. Students will have opportunities to interact with master teachers at the K-12 level and teach at either summer academies for pre-college students, during summer school sessions, and at the California Museum of Science and Industry Saturday science programs.

The Los Angeles Collaborative for Teacher Excellence has been in operation since 1995. The first year of funding was devoted to building the Collaborative. The coordinating team investigated the feasibility of implementing new courses at the participating institutions. Additionally, the Collaborative sponsored a series of workshops for faculty from the mathematics, science and education disciplines and secondary school teachers. The workshops allowed faculty and K-12teachers to share information, experiences, and expertise in a collaborative manner. Examples of LACTE accomplishments and evaluation outcomes are listed below:

LACTE Accomplishments (Years 1 – 4)

  • Organized several conferences, retreats and symposiums in order to develop networks within and outside of the Collaborative
  • Developed LACTE World Wide Web site
  • Compiled LACTE faculty email list
  • Developed the LACTE monthly newsletter
  • Developed guide for institutionalization of LACTE courses
  • LACTE faculty attended and made presentations at professional meetings
  • Held faculty development workshops
  • Established faculty and student recruitment mechanisms at all ten institutions
  • Piloted LACTE courses at all ten institutions
  • Awarded LACTE Merit Scholarships to over 100 students at the participating institutions
  • Established student support groups
  • LACTE students participated in professional conferences and symposiums
  • LACTE students obtained internships, and observed and assisted in K-12 classrooms.

For a more detailed account of LACTE accomplishments, refer to the program evaluation summary findings of the project’s previous evaluation consultant, Evaluation and Training Institute.

LACTE Evaluation Report Year 4 – March 20001

Summary of Evaluation Activities

Evaluation Team – Roles and Responsibilities

Nunn Consulting, Inc., a management consulting firm specializing in nonprofit organizational development, research, and evaluation, conducted the outcome evaluation of the program. The firm has extensive experience in program research and evaluation. Nunn Consulting, Inc. has designed and conducted evaluations of programs funded by the Federal, state and local government agencies, as well as private foundations and educational institutions. The firm has experience in evaluating programs that serve an ethnically diverse population. The staff has a clear understanding of the needs of the target population. As a minority-owned firm, Nunn Consulting possessed the cultural awareness to address sensitive issues facing the target population while maintaining rigorous standards of research design, data collection and participant confidentiality.

The firm’s Principal Evaluator is Dr. Yolanda J. Nunn. Dr. Nunn leads the evaluation team, taking primary responsibility for monitoring project management overall. To this end, Dr. Nunn participated in the CETP – PI Conference in Washington, DC in May 1999.

Dr. Pamela Taylor serves as the Senior Evaluator for this project. Dr. Taylor manages the daily evaluation activities for LACTE. Her duties included meeting and providing updates to the Principal Investigator, attending LACTE Coordinating Committee meetings, overseeing data collection, database design and report writing. In addition, Dr. Taylor participated in the NSF site visit in November 1999.

A research assistant was primarily responsible for creating, maintaining, and updating databases and the creation and revision of evaluation instruments for LACTE.

Evaluation Scope of Work

The primary objectives of the evaluation component are as follows:

  • To monitor and describe program components and activities.
  • To provide administrators with prompt feedback of evaluation results to determine if adequate progress is being made toward project objectives.

Nunn Consulting staff, in conjunction with project staff, conducted all evaluation activities. The evaluation team, while not a part of the LACTE staff, conducted evaluation activities at the program site (with the exception of report writing, data entry and analysis). Evaluators served as an important resource for guiding some aspects of program decision-making. The evaluation team met regularly with program staff to assist with identifying and resolving possible barriers to implementation. All primary data collection instruments for the evaluation were prepared or reviewed by the evaluation team with the review, input, and approval of the Project Director and/or appropriate project staff.

Outcome Evaluation

An outcome evaluation was conducted for the Los Angeles Collaborative for Teacher Excellence. It aimed to answer specific questions regarding the effectiveness of the program and its ability to achieve the stated objectives.

The outcome evaluation focused on establishing mechanisms to document the program’s impact on students, faculty, and institutions for the remainder of the grant period. These methods should also enable LACTE to continue to evaluate its role in improving K-12 teacher preparation in mathematics and science beyond the initial funding.

The outcome evaluation focused on measuring the extent to which the program had a significant impact on students, faculty, and the ten participating institutions. Key outcome evaluation questions are outlined below:

Undergraduate Students

  1. How effective are LACTE courses, internships and work opportunities in preparing students for the teaching profession?
  2. How has participation in LACTE affected student interest in teaching as a career?
  3. What is the level of student involvement in LACTE activities?

LACTE Teachers

  1. What are the experiences of LACTE trained teachers in their schools?
  2. How well do LACTE trained teachers implement hands-on techniques in the classroom?
  3. How are LACTE trained teachers perceived by their colleagues and principals?

Faculty

  1. To what extent has faculty implemented new techniques into their classrooms?
  2. To what extent has faculty produced and disseminated their LACTE course materials?
  3. How are LACTE faculty perceived by their colleagues and administrators?

Participating Institutions

  1. To what extent is each institution involved in disseminating information about LACTE activities within and outside the Collaborative?
  2. To what extent do campus administrators support LACTE activities (perceptions of the program and promotion of LACTE faculty)?
  3. To what extent has the LACTE program been institutionalized at each campus?

Evaluation Activities

The evaluation team concentrated its efforts on becoming acquainted with the LACTE program. The current Evaluation Consultant assumed responsibilities from Evaluation and Training Institute. The Senior Evaluator met regularly with the Principal Investigator to define the goals of the evaluation for the final years of the program. Evaluation activities during 1999 consisted of the following:

  • Establishing rapport with LACTE program staff
  • Reviewing evaluation materials from previous evaluation consultant
  • Developing and revising evaluation scope of work
  • Designing and revising data collection instruments
  • Meeting with Project Coordinator
  • Attendance at LACTE Coordinating Committee meetings
  • Meeting with NSF Project Officers
  • Data collection, data entry and data analysis

Evaluation activities were hindered by two factors: 1) delay in awarding of project funding by NSF and 2) lack of clear expectations for program outcomes from NSF. There was a significant delay in the funding agency’s allotment of project monies for 1999 grant period. Without the proper level of funding, LACTE could not spend money from the evaluation budget and therefore, initially paid for evaluation services through an independent consultant agreement. This contract allowed the evaluation team the time to acquaint itself with LACTE activities. However, it did not allow for full-scale evaluation activities (e.g. site visits, focus groups, interviews etc.) that would have enabled the evaluation team to gather in-depth data about program outcomes. Thus, evaluation methodology during this grant period was limited to instrument design of the LACTE student survey and evaluator observations. Fortunately, program staff’s previous experience with evaluation provided them with the foresight to create their own evaluation tools for LACTE events (e.g., student symposium and teacher workshop).

A lesser factor that impinged upon evaluation activities was the selection of appropriate outcomes for LACTE. During the final years of funding, an emphasis was placed on demonstrating the program’s effectiveness. While it was clear that the evaluation should be outcome-focused, NSF provided little guidance in selecting outcomes that would show the effectiveness of the Collaborative. A good deal of time of the evaluation team was spent developing and revising the scope of work to ensure the evaluation monitored appropriate outcomes.

LACTE Evaluation Report Year 4 – March 20001

Program Outcomes

Student Survey

A survey was distributed to LACTE students at the student symposium in October 1999 and to student groups at each campus in December 1999. A total of 100 surveys were received. Students were asked to indicate their level of participation in LACTE activities (e.g., Meet the Teachers Roundtables, and internships), their experiences with LACTE student groups and internships and the program’s influence on their teaching career. Students were allowed to respond to open-ended questions about LACTE’s impact on their pursuit of a teaching career.

DESCRIPTION OF LACTE STUDENT SURVEY PARTICIPANTS

Survey participants were representative of the general LACTE student population. There were 13 students at the symposium that attended schools outside of the Collaborative.

CAMPUS REPRESENTATION

With the exception of Glendale Community College, responses were almost evenly distributed among students attending two-year and four-year institutions.

Campus / N / %
California State University, Dominguez Hills** / 17 / 20%
California State University, Fullerton** / 2 / 2%
California State University, Los Angeles** / 18 / 21%
East Los Angeles College* / 12 / 14%
El Camino Community College* / 21 / 24%
Fullerton College* / 2 / 2%
Glendale Community College* / 0 / 0%
Loyola Marymount University** / 7 / 8%
Occidental College** / 2 / 2%
Santa Monica College* / 6 / 7%

* Two-year institutions; ** Four-year institutions

DEMOGRAPHIC CHARACTERISTICS

The majority of the students were from ethnic minority backgrounds and an overwhelming number of the students were female.

Ethnicity (N = 89)

Ethnic Background / N / %
American Indian/Native American / 0 / 0%
Asian/Pacific Islander / 15 / 17%
Black/African American / 6 / 7%
Hispanic/Latino / 35 / 39%
White / 24 / 27%
Multiracial / 2 / 2%
Other / 7 / 8%

Gender (N = 88)