Program Evaluation Plan 1
Running head: PROGRAM EVALUATION PLAN
Program Evaluation Plan of
Investigations in
Number, Data, and Space
Ginny Wilburn
College of William and Mary
Educational Policy, Planning, and Leadership
Investigations in Number, Data, and Space
Context
Investigations in Number, Data, and Space is a math curriculum designed to engage students in making sense of mathematical ideas. This program was introduced and implemented to an entire third grade class in the 2008-2009 school year for the first time. The program is a comprehensive math curriculum that emphasizes problem solving, reasoning, communication, making connections, and designing and analyzing representations (TERC, 2008). The program is a K-5 curriculum that incorporates investigation into all units of study.
Several underlying theories framed the development of the program. Mathematical ideas are developed and enhanced through student investigation. Students bring a varying degree of prior knowledge about numbers, shapes, measurements, patterns, and ideas to the classroom. An environment that emphasizes making sense of mathematics enables students to build on prior knowledge by making connections. Once students are able to make sense of mathematical ideas, they learn to become mathematical thinkers. Too often, students are instructed through passive observation. Given the opportunity to collaborate with others and investigate concepts using various materials, students become engaged in their environment. These experiences help translate knowledge to real-world applications that encourages independent reasoning. The opportunity for exploration encourages curiosity. Once a student develops curiosity about math concepts, he/she engages in self assessment through experiences sought out by the student.
The development of the curriculum is based on several guiding principles. Students are supported to make sense of math concepts through guided exploration and investigation. Many of the activities of the programs focus on the development of computational fluency with whole numbers. Activities are designed to make connections between geometry, measurement, data, and early algebra. Reasoning is the cornerstone of each lesson and activity. Students are taught to question their understanding of concepts, test their knowledge, and track their progress. The program promotes the collaboration between students, teachers, and the content. Teachers are trained to investigate and create meaning of math concepts. The interaction between teachers and students provide the teachers with needed information to make effective teaching decisions.
Criteria
The program was developed with specific, measurable intended outcomes in place. The short-term outcomes center on student learning. The outcomes are: increase curiosity about mathematical ideas, increase student motivation, increase computational fluency, increase achievement on benchmark assessments, and utilize the assessments to enhance instruction. The program is based on collaboration between student, teacher, and the content; therefore, the outcome of assessment to guide instruction is a goal of the teacher. These outcomes lead to the longer-term, medium outcomes. The medium results focus on action. The intended outcome of increasing curiosity translates to the student’s ability to engage in self assessment. This meta-cognitive skill is necessary to achieve increased motivation and collaboration. Both of these outcomes lead to an increase in student achievement. The ultimate goal, or long-term outcome, of the program is to develop mathematical thinkers. This is a trait that is honed and developed throughout the student’s mathematical experiences over many years.
Logic Model.
A logic model was created to demonstrate how the interaction of the inputs and activities will result in the development of mathematical thinkers (See Appendix A). The logic model is a tool used to plan and make judgments about the effectiveness and value of the program. The inputs, or resources invested in the program, are teachers, teacher training, manipulatives, curriculum materials, and assessment checklists. The activities, or what we do to serve the client, of the program are investigation games, ten-minute daily math activities, math workshops, and normative assessments for the student and the teacher.. The participants of the program are the clients that the program is designed to reach. Of course, the students are the intended beneficiaries of the program. The program is also designed to communicate the mathematics content and pedagogy to teachers. The collaboration between student and teacher is essential to the effectiveness of the program. The students and the teachers are included as participants in the logic model. The focus of this program evaluation is to make judgments regarding the effectiveness of the processes, or activities, of the program.
Purpose
The purpose for evaluating the Investigations in Number, Data, and Space program is to determine the value and effectiveness of the program in the attempt to improve program practices and procedures. In an age of accountability and the expectation of continuous improvement in public schools, programs must be planned and evaluated thoughtfully. Sanders and Sullins assert, “…any evaluation that is worth doing must ultimately save a school’s time and resources and produce better results.“(2006, p.1) The program must be aligned with the school division curriculum and state standards. The alignment of curriculum, instruction, and assessment is essential to an effective program. The principles forming the foundation of Investigations are directly aligned with the division’s math curriculum
The program evaluation is focused on the activities (outputs) of the logic model. This targeted area provides the means with which to achieve the ends. The outcomes are dependent on the effectiveness of the activities the teachers and students are engaged in each day. The four activities on the logic model are the core elements of the Investigations program. The evaluation of the activities will be analyzed to determine if they are being carried out with fidelity. The evaluation should be informative, timely, and influential to follow the Utility Standards of program evaluation. The evaluation of Investigation activities should provide useful information to improve the quality of the program.
Evaluation Questions
There are four questions to guide the evaluation to identify the program’s strengths and weaknesses (See Appendix B). How consistent are the Investigation games with the school’s overall math curriculum? This question ensures the alignment between the curriculum objectives and program activities. Time should only be devoted to activities that produce the desired outcomes with a high degree of effectiveness. Student outcomes must reflect an alignment between program activities and curriculum.
What topics are covered during the Math Workshops and is sufficient time spent? The content covered during the Math Workshops should be the same as the objectives of the math curriculum or state standards. This question addresses the importance of emphasis. Students need to be given adequate time to develop and practice math concepts and skills. The time spent on Math Workshops should be in alignment with the SOL Test Blueprints. Appropriately placed emphasis helps ensure desired outcomes.
How successful are the Ten-Minute Daily Math Sessions at reinforcing concepts? As noted before, time should only be spent on those activities that are producing the desired results. In order to ensure that this is happening during the program, we need to be able to accurately measure the benefits of the sessions.
In what ways are the normative assessments being used to guide instruction? A critical element to the Investigations program is the use of normative assessments to improve instruction and assist the student in learning to assess their own knowledge and skills. The use of normative assessments, as an essential activity, is for the benefit of both the teacher and the student. Analysis of formal and informal assessment results is needed to target student needs as part of the teaching/learning process. Increased student self-assessment leads to the long-term outcome of developing mathematical thinkers.
Audience
The audiences of the Investigations program evaluation are the math teachers and administration. The purpose of the evaluation is to improve the program. This information would be most useful to those responsible for implementing the program and the administration. Since this is a new program to the school, it is not necessary or appropriate to include parents as an audience. If the purpose of the program evaluation was to determine whether the program should continue or discontinue, then it would be important to include the parents. The results of the evaluation will be useful to the teachers to determine which activities are effectively leading to the outcomes. Time or resources should not be spent on ineffective programs or practices. The results of the evaluation will also be useful to the administration to determine whether the program should continue, be modified, extended to other grade levels, or be discontinued. As a champion to the planning process, the administrator needs to know what is working and what is not working in order to improve programs.
Math Specialist
The Math Specialist will be responsible for leading the study. The person in this position serves as a mentor to the other math teachers at the school. The Math Specialist attends monthly trainings on the latest strategies and practices and provides the math teachers with training. The responsibilities of the Math Specialist coincide with the responsibilities of a program evaluator. In an effort to ensure objectivity of the evaluation, several issues have been considered. The purpose of the program is one in which all are committed. The success of the program equates to the success of all students, teachers, and administration. The results of the evaluation will not affect the Math Specialist in a negative manner. There is no personal stake associated with the Investigations program. The results of the evaluation will be used by the Math Specialist to make necessary changes.
Collection of Data
An Evaluation Work Plan (See Appendix C) assists in the planning of major activities of the evaluation. The focus of this program evaluation plan is on the activities; therefore, the steps in the work plan often overlap. The activities of the Investigations program continue throughout the school year and the fidelity of each activity is monitored through the work plan. Since the activities occur simultaneously throughout the program, it is appropriate to compile results of the quantitative and qualitative data quarterly. A Gantt Chart is a helpful tool to help coordinate and plot evaluation activities throughout the year.
The evaluation plan begins before school starts. The evaluation coordinator will create a checklist to chart the objectives of the Investigation games, objectives of the program curriculum, objectives of the math curriculum for the division, and the SOL Test Blueprints. All of the information is included in one chart to ease teachers’ documentation efforts. This information will be submitted and reviewed quarterly. An assessment chart and self-assessment survey will be created in August and distributed at the start of the school year, after explanation has been shared about the appropriate use of each. The Evaluation Coordinator will compile the comparative data and prepare reports focusing on the alignment of the objectives of the program with that of the curriculum, SOL blueprints, and assessment/remediation efforts. It is important to review the findings with the teachers quarterly to tweak elements of the activities as needed. The intended outcomes of the program depend on the fidelity in which the activities are carried out throughout the school year. The teacher’s perceptions of the effectiveness of the Ten-Minute Daily Math Sessions will be collected through a survey to identify the strengths and weaknesses of this aspect of the program. The findings will be shared with the teachers in March. Once the data is collected and analyzed, the Evaluation Coordinator will share the identifiable relationships revealed through the plan. At the end of the year, suggestions are brainstormed to address the problematic issues identified throughout the evaluation to improve the program the following year.
The Evaluation Work Plan also includes the resources needed to carry out the program evaluation. Due to the fact that the activities, which must be monitored closely, are the focus of the evaluation, the data collection procedures are conducted quarterly. This requires time to educate the teachers about the process of evaluation and their responsibilities to complete the tasks each quarter.
Information Collection and Analysis Chart
Various sources of data are utilized to evaluate the effectiveness of the activities of the Investigations plan. A class by class comparison of objectives from the Investigation games, Math Workshops, division curriculum, and SOL Test blueprints are recorded each quarter by each math teacher to demonstrate alignment between activities and curriculum (See Appendix D). A frequency count quantifies the data to demonstrate consistency. This method of data collection represents a high degree of validity because the information collected through the checklist and frequency chart will provide empirical evidence as to how closely the program activities and the division curriculum are aligned. This method allows for the necessary program documentation, context analysis, and systematic review required, satisfying the accuracy standard of program evaluation. A list of the strengths and weaknesses of the effectiveness of the Daily Math Sessions will be completed by each math teacher (See Appendix E). This qualitative list of teacher perceptions will serve as a basis for determining whether the activity is beneficial and serving to fulfill our outcomes. The utility standards are addressed with this source by identifying the needs of the stakeholders, or teachers. In order to avoid validity issues due to the observable characteristics of this measure, the interpretations of the strengths and weaknesses must be appropriate and supported by data. Lesson plans, a qualitative measure, and assessment charts (See Appendix F), a quantitative tool, are evaluated each quarter to track remediation efforts throughout the teaching/learning process. Students need to be given timely feedback to validate or correct their learning through normative assessments. The validity of these procedures is determined by the accuracy and thoroughness of the data collection. Finally, the 3rd grade SOL Math results will be compared with the assessment charts to determine whether the needs of the targeted students were met. This quantitative measure will be conducted in June and serve as the last piece of data collection for the program evaluation plan. The analysis of SOL results is a valid data source due to its high level of construct and content validity.
Several of the data collection sources in this evaluation plan require informal, qualitative methods. In order to make a valid inference about the effectiveness of the program, the inference must be based on several sources of data. The reliability of the data collection instruments is dependent on the triangulation of data sources. This practice helps lower the level of subjectivity and provides a basis for comparison of data results. Both quantitative and qualitative data are collected to ensure the same construct is being assessed. Triangulation also helps to ensure the data are collected in a feasible, proprietary, and accurate manner.
Conclusions
The Standards for Program Evaluation offer evaluative tools to base the strengths and weaknesses of the proposed evaluation of the Investigations program. The utility of the plan is demonstrated through the broad scope and selection of data sources. Each of these sources are designed to measure the impact of the activities of the program. The stakeholders are identified throughout the program evaluation plan and care is given to careful identify the procedures and justification for each element of the plan. The propriety standards are addressed in the service orientation and by providing a fair assessment. As weaknesses are discovered through data collection, efforts will be made to address them. Valid and reliable data collection methods and tools ensure the accuracy standards. This also leads to the justification of conclusions about the program. The context of the program has been analyzed and procedures are detailed and documented.
Weaknesses can be addressed through consideration of the feasibility standards. Although an evaluation work plan aids the organization of tasks during the evaluation, time is always a difficult issue in the educational setting. The timeline may need to be adjusted by the Evaluator Coordinator to ensure there is sufficient time to accomplish the many tasks. Another issue directly relating to the feasibility of the evaluation is the practical procedures. The Evaluator Coordinator, or Math Specialist, is responsible for the organization, collection, and communication of data. It may be more feasible to delegate responsibilities to others. This was avoided during the creation of the evaluation plan to mitigate bias, whether it be positive or negative, from the math teachers.