The State of Israel / / Ministry of Education, Culture and Sports
NATIONAL AUTHORITY FOR EDUCATIONAL MEASUREMENT AND EVALUATION (RAMA)

Assessment in the Service of Learning

Professor Michal Beller

Director General, RAMA

The National Authority for Measurement and Evaluation in Education

March 2008

Introduction

Three of themain goals of the Israeli education system are: a) the fulfillment of each and every pupil’s intellectual and creative potential; b) the narrowing of achievement gaps in education, and c) value education. Assuming there is a consensus regarding these goals, the question is how would we know whether these goals have been achieved?

How would a parent know whether the education system has provided his or her child with the tools necessary to function in modern society? How would all those involved in the education process (teachers, principals, the different echelons of the education system) know whether he or she has executed his or her role satisfactorily? How would the public know whether the future generation of children has been equipped with the skills needed to meet the challenges of the twenty first century? How can we reassure the public that theconsiderable resources spent on education are being used wisely? And how can we convince decision-makers that the increasing investments in education are worthwhileeven at the expense of other important areas of needs?

The narrowing of achievement gaps is one of the most crucialgoals to be achieved in order to ensure the resilience of Israeli society. How can the gap be detected, and how would we know whether it has narrowed? The detection of gaps should be done by topics (such as violence and the learning environment) and in the different subject areas. It is also important to detect gaps between sectors, cities, settlements, socioeconomic strata, etc.

Valid and professionally constructed measurement tools are needed in order to monitor the achievement of the aims of education. It is known that measurement is a complex issue in the business sector and even more so in the public sector and in the field of education,in particular.The process of learning is unmatched in its complexity due to the differences between students and the variety of pedagogical approaches for achieving educational goals. Similarly, these approaches require appropriate types of measurement and assessment.

Often we tend to look for short, simple, straightforward answers to questions related to educational measurement and assessment. However, a complex educational process should learn to benefit also from complex answers that would provide educators, mainly teachers and principals, with indications of positive and negative directions, and with suggestions as to what is more appropriate and what is less appropriate for specific pupils.

There is no uniform “one size fits all” solution that answers all the needs, and there is no magic solution to the means of measurement. However, the professional, ethical and public dilemmas that accompany the different measurement instruments do not exempt us from facing the challenge and providing an adequate answer.

The National Authority for Measurement and Evaluation in Education (RAMA) wasrecently founded in order to provide a professional response to the issuesofmeasurement and evaluation in the education system. RAMA, from the first day of its founding, has adopted an ideology that is based on two tenets: a) assessment in the service of learning; and b) constructing anassortment of professional solutions that effectively incorporate different components of measurement and assessment (about RAMA – in the box).

About the negative phenomena that accompany high stakes testing

High stakes tests, those tests which are perceived as determining the future of the pupil, the teacher, or the principal, to some extent, are naturally accompanied by negative reactions. Among theseare test anxiety, cheating, attempts to exclude weak pupils, teachers helping pupils during the test, narrowing of the curriculum andfocusing on what is tested, diverting resources (assigning "good" teachers to the classes and subjects that are tested, diverting instructional hours from other subjects, preparing for the test), and abuse of results, etc.

The education system needs to act forcefully in order to minimize these negative phenomena, first and foremost by a cultural change by which measurement and assessment are intended to serve the learning process and not inhibit it. However, these negative phenomena are not, in themselves, a reason to avoid the thoughtfuluse of tests.

About centralization versus decentralization of assessments

Aconstant dilemma facing the education system is setting the balance between local assessment and centralized assessment, in other words, between decentralization and centralization of measurement and assessment. The utopian solution is, of course, to allow maximum freedom to each school to decide on the assessment procedures that best suit it. According to this approach one could also consider giving this freedom to each teacher in the school. However, considerations of accountability, transparency, professionalism, cost benefit, and mainly the capacity to make valid comparisons necessitates that at least a part of the overall assessment be centralized and allow for valid comparisons and systemic monitoring over time.

About "ratings" and "comparisons" – Methodology in a nutshell

Two of the more commonuses of national test scores are ratings ("league tables") and comparisons ("we've gone up", "we've gone down", "we've improved", being a few of the more common expressions). However, these types of ratings, when they are made at the level of the school in an irresponsible manner, may cause great damage. Not every comparison is valid and valuable. For example, is there any meaning in comparing the grades of pupils in Savyon (a prestigious, high SES neighborhood in the center of the country) with those of pupils in Rahat (a Bedouin village in the far south of the country)? Does a grade of 90 on an easy test in a given year imply an improvement over a grade of 75 on a difficult test in a previous year?

RAMA is working to enhance the public's understanding that ratingsonly have meaning, if at all, among relevant populations, and comparisons over years have value only when the rating scales are calibrated.

Comparisons across years that are often carried out between scores on the Meitzav (this is the Hebrew acronym, and in English it is known as GEMS – Growth and Efficiency Measures of Schools) national tests administered to all fifth and eighth grade pupils, and those on the high school matriculation exams (Bagrut) are not valid, as the scoring scales on these tests are not completely calibrated. RAMA is working now to professionally calibrate the Meitzavscores so that the year 2007 will serve as a base year for meaningful comparisons of future Meitzav tests. When RAMA becomes responsible for developing and grading Bagrut exams, it will begin dealing with this subject for these exams also.

Updated assessment and evaluation structure

The format for national assessment was updated in 2007 (not including the matriculation exams – Bagrut[1]). The new format was designed and consolidated by RAMA, in conjunction with various factors at the Ministry of Education and in consultation with school principals and many teachers. In the field of education, this format is intended to provide a suitable, professional answer to all stake holders in the education system: schools and other external bodies.

The new format is intended to maximize the advantages of the current assessment system while providing a solution for its disadvantages and is based on the following principles:

  • Implementing a culture of "assessment for the benefit of learning" – assessment that is intended to support continuous improvement of learning, to align with learning goals and the school vision, and based on the understanding that tests are not a goal in themselves but rather a tool in the service of learning.
  • Meaningful integration between internal and external assessmentand between formative and summative assessment.
  • Maximum decentralization of assessment by supplying professional assessment instruments provided to the schoolsby RAMA.
  • Empowerment of school principals and teachers.
  • Reduction of pressure and the amount of external tests.

The new format includes four components (some are already implemented and some are not): internal school-based assessment, national sample assessment, school-based assessment with external tools (Meitzav) only a small number of which will be administered and checked by RAMA ("external Meitzav ") and most will only serve the school ("internal Meitzav "). The following are details of the components:

School-based assessment

RAMA will help in strengthening the internal assessment that is being carried out in the schools, by providing banks of tasks and items, "off the shelf" tests, diagnostic tests, questionnaires and other tools that will be accompanied by in-service training of school staff.

National sample assessment

National longitudinal trends will be monitored based on a representative, national sample encompassing a small number of classes. This monitoring will give feedback from questionnaires on a variety of educational topics as well as tests in different subjects that will include a wide and in-depth coverage of the curricula. This feedback will serve asan indicator for policymakers interested in examining the extent of success of various instructional methods, changes in the size of educational gaps, etc.

School-based assessment with external tools

The Meitzav is a set of school measures that serve the school principal and his or her staff as a central tool for the monitoring of curricularplanning and for utilization of resources. The new format of the Meitzavintegrates external and internal school assessment.

The development of the Meitzav is based on the understanding that the school is a complex entity that is comprised of many interrelated parts. It is about an organization that includes a learning environment, curricula, achievement, and professional development programs for staff, many relationships (pupils-pupils, teachers-teachers, teachers-pupils, teachers-parents) and more.

The Meitzav in its new format will go back to focusing on its original task, providing school feedback, as listed below:

  • The Meitzav will be carried out towards the end of the school year and the tests will cover the material learned up until that particular point (material could be from the same year and also previous years, specifications for each subject are published on the RAMA internet site).
  • The Meitzav will be given in four subjects – Language Arts (Hebrew/Arabic), Mathematics, English and Science and Technology – and will also include questionnaires related to school climate, pedagogical setting, and more (below referred to as "climate questionnaires").
  • The tests will be administered to pupils in fifth and eighth grades. In Language Arts there will also be a test administered in second grade.
  • There will be a calibration of the grading scales of the Meitzav in a way that will enable valid, future comparisons of the grades across the years. The year 2007 will serve as the base year.

The External Meitzav–As of 2007, the external tests will be administered to a school every other year and they will be composed of a pair of subjects (Two out of the four subjects that are included in the Meitzav). In this way, once every four years a school will be tested externally in a specific subject. Climate questionnaires will be administered on each administration of the external Meitzav. As a consequence, each school will receive a report every other year that will include the achievement scores in the two subjects as well as a reporton the climate and pedagogical setting of the school.

The Internal Meitzav– As of 2007, each school will administer an internal Meitzav in each of the subjects, in those years in which there is no external Meitzav. Schools will not be required to report the results to any external body and they will remain internal to the school. The internal Meitzavwill be based on the integration of three components: a) an objective, external, national test having psychometric properties of reliability and validity developed by RAMA in conjunction with a professional pedagogical committee and with the chief inspectors who ensure that the tests are in alignment with the curricula and the standards of knowledge and understanding that pupils are expected to reach in each subject; b) internal grading by the school staff (with the assistance of a rubric attached to the test) which provides the staff with timely feedback about the mastery of students (or groups of students) in each subject; and c) comparison of the students' achievement in the school with external norms (national, regional, district and sector) that are calculated from the external Meitzav tests administered at the beginning of the following year.

InternationalAssessments

RAMA is also responsible for the international tests, in which many countries participate. The studies based on these international tests allow for a comparison of the achievement of pupils from different countries in a number of subjects, and with regard to other educational issues. In addition, the results of the studies allow for a comparison between sectors and different subgroups of the population within each participating country. The international organizations that develop the tests are among the leaders in the field of assessment and measurement. The tests are translated into different languages and becausethey are professionally constructed they have a high level of reliability and validity. The tests and questionnaires are administered to a representative sample of the population in each country and the scores are reported only at the national level and not at the level of the class or school. Over the last few years Israel has participated in a series of international tests including TIMSS, PISA, PIRLS and SITES.

The results of the studies provide true and reliable information about the education system in Israel from an international perspective. This information may have great importance for policymakers who want to focus on the strong and weak points of the education system in Israel with an objective comparison to other education systems around the world. Furthermore, the participation in international studies allows Israel to learn about new and innovative approaches in the subjects studied and to examine its own curricula in light of thoseimplemented in different countries around the world. Thus, for example, the need for strengthening literacy studies in Science, Mathematics and Language Arts was identified. The studies also provide information about successful models in education from different countries. This can be achieved by comparing countries with high achievement to countries with low achievement levels, and by examining the relationship between student achievement and various background variables (such as education of parents, socio-economic status, student attitudes, etc.). Thus, for example, the world was able to acknowledge the successful education system in Finland where pupils reached the highest level of achievement in the PISA study mainly due to the cultural and social norms there and due to a reform that empowered schools.

The media tends to emphasize the ranking of each country in these studies, and to compare these rankings to previous studiesby presenting the ranking of the countries as the main findings of these international tests. However, the main value of these studies lies in the possibility of each country to compare and check its own achievement of the goals that it set for itself. For example, in Israel, the results of the study allow a comparison between the achievement of pupils who are speakers of Arabic and the achievement of their Hebrew speaking peers and between other sub-groups (for example, boys and girls). In particular, these studies allow us to monitor achievement in Israel across time. This is possible thanks to the cyclical nature of these studies which are carried out in a similar format once every three to five years, maintaining calibration of the scoring scale from one study to the next. More details regarding each of the international tests are available on the tests’ websites of the tests and on the RAMA website.

The "bottom line"

RAMA would like to pass on the message that there is no simple "bottom line" in the field of measurement and assessment; there is not and will never be any "magic number" or single "general conclusion" that can sum up complex educational processes, neither on the level of the individual nor on the system level. RAMA sees its main mission as helping the education system improve through the thoughtfuluse of assessment and measurement. The concept of "assessment in the service of learning" is not just a motto – RAMA sees it as the raison d'etre.

The National Authority for Measurement and Evaluation in Education – RAMA
The National Authority for Measurement and Evaluation in Education (RAMA) is a leading, professional organization of the education system,specializing in educational measurement and assessment. The authority acts as an independent, objective, professional body serving all stakeholders within the education system and without. In order to assure that RAMA can fulfill its mission it was founded (in 2005) as an independent, governmental authority, reporting directly to the Minister of Education, and its activities areanchored in the RAMA article of law.
The vision
RAMA believes that effective processes of measurement and assessment can help in the improvement of education: in identifying gaps between what is and what should be, in aspiring for continuous improvement, in promoting excellence, in attending to students with special needs, and in supporting weak populations.
Measurement and evaluation are a means and not an end, and they are supposed to be part of a ongoing teaching and learning process. RAMA will act to imbed the principle of "assessment in the service of learning": emphasizing the alignment and the feedback that are required between content, standards, instructional methods and assessment.
"Assessment in the service of learning" is a comprehensive approach that supports the integration of internal assessment of schools with external assessment, and also the integration of summative assessment of pupil achievement with formative assessment that supports and promotes instruction.
RAMA is a part of the education system, and as such attaches importance to the constant and direct contact with principals, teachers and parents. RAMA is in daily interaction with the headquarters of the Ministry of Education and with the districts and municipalities, in order to be attentive to their needs and be involved in processes, and in order to contribute to the implementation of the findings. RAMA acts to strengthen itself as a reliable, relevant, professional and objective body. It will carry out and report the assessment and evaluation results fairly and precisely, without fear or bias.
The mission
RAMA's mission is to contribute to the improvement of the education system through:
  • setting up an effective assessment system that is an inseparable part of the learning process;
  • developing a variety of assessment instruments for the continuous monitoring of the development of the education system and for assessing the progress of educational programs;
  • professional training of all those involved in assessment and measurement in the education system;
  • providing findings that are supported by data and tools for scientific analysis to policymakers in the education system, to researchers and interested parties in academia, and to the general public;
  • helping shape national educational policy that relies on findings and facts.
RAMA's activities
The following topics will be included in stages within the responsibilities of RAMA, and will be dealt with in conjunction with the various departments within the Ministry of Education (see table below).
The leading topic / Examples of sub-topics
Large scale assessments, questionnaires and surveys / Development of tests of literacy and basic skills; Meitzav tests, five year plan tests for the Arab sector, tests for the religious sector; international tests; Bagrut tests; tests for special populations; tests and questionnaires that will be defined from time to time.
Formative assessments in the service of learning / Construction of task banks; construction of instruments for formative assessment in the service of teachers.
Research and development / Development of methodologies and statistical instruments for improving the construction of tests and grading; development of a computerized bank of diverse assessment instruments.
Reporting to the public and providing data banks and information / Publication of annual reports; provision of data banks and national information to the public and to the community of researchers.
Training for in-service courses in the field of measurement and assessment / Planning of training programs on topics dealing with measurement and assessment for school assessment coordinators, teachers, principals and district role holders.
Project evaluation and educational interventions / Evaluation of the effectiveness of large-scale national projects and interventions such as the five year plan in the Arab sector, language instruction for new immigrants, remedial education, educational reforms, etc.
RAMA sees schools, teachers and principals as the main consumers of its services, and is in constant contact with the "field" through internet forums and activating a "field team" which provides RAMA with immediate and continuous feedback about its activities.
Further details appear on the RAMA site:

1