Practice Socratic Seminar: “The State of Education in America”
Program for International Student Assessment (PISA)
1. What subject areas are assessed in PISA?
PISA measures student performance in mathematics, reading, and science literacy. Conducted every 3 years, each PISA data cycle assesses one of the three core subject areas in depth (considered the major domain), although all three core subjects are assessed in each cycle (the other two subjects are considered minor subject areas for that assessment year). Assessing all three subjects every 3 years allows countries to have a consistent source of achievement data in each of the three subjects while rotating one area as the primary focus over the years.
Science is the major subject area in 2015, as it was in 2006, since each subject is a major subject area once every three cycles. In 2015, all subjects will be assessed primarily through a computer-based assessment. In addition to the core assessments of science, reading, mathematics, and collaborative problem solving, education systems can participate in an optional financial literacy assessment. The United States will participate in this optional assessment in 2015.
PISA administration cycle
Assessment year / 2000 / 2003 / 2006 / 2009 / 2012 / 2015Subjects assessed / READING
Mathematics
Science / Reading
MATHEMATICS
Science
Problem solving / Reading
Mathematics
SCIENCE / READING
Mathematics
Science / Reading
MATHEMATICS
Science
Problem solving
Financial / Reading
Mathematics
SCIENCE
CPS
Financial
NOTE: Reading, mathematics, and science literacy are all assessed in each assessment cycle of the Program for International Assessment (PISA). The subject in all capital letters is the major subject area for that cycle. A collaborative problem solving (CPS) assessment will be administered in 2015. Financial literacy is an optional assessment for countries. As of PISA 2015, PISA will be administered entirely on computer.
2. What are the components of PISA?
Assessments
PISA 2015 consists of computer-based assessments of students' mathematics, science, and reading literacy, and collaborative problem solving skills. In each participating school, sampled students sit for a two-hour computer-based assessment. Countries can also opt to participate in an assessment of financial literacy.
Questionnaires
In 2015, students will complete a student questionnaire providing information about their background, attitudes towards science, and learning strategies and the principal of each participating school will complete a school questionnaire providing information on the school's demographics and learning environment. New to 2015, PISA will include a teacher questionnaire, to be completed by up to 10 science and 15 non-science teachers per school. There are separate teacher questionnaires that will be administered to science teachers and non-science teachers.
3. How many U.S. schools and students participate in PISA?
Assessment year / Number of participating students / Number of participating schools / School response rate (percent) / Overall student response rate (percent)Original schools / With substitute schools
2000 / 3,700 / 145 / 56 / 70 / 85
2003 / 5,456 / 262 / 65 / 68 / 83
2006 / 5,611 / 166 / 69 / 79 / 91
2009 / 5,233 / 165 / 68 / 78 / 87
2012 / 6,111 / 161 / 67 / 77 / 89
4. How does PISA select a representative sample of students?
Step 1
To provide valid estimates of student achievement and characteristics, PISA selects a sample of students that represents the full population of 15-year-old students in each participating country or education system. This population is defined internationally as 15-year-olds (15 years and 3 months to 16 years and 2 months at the beginning of the testing period) attending both public and private schools in grades 7-12. Each country or education system will submit a sampling frame to the consortium of organizations responsible for the implementation of PISA 2015 internationally. Westat, a survey research firm in Rockville, Maryland, contracted by the OECD, then validates each country or education system's frame.
Step 2
Once a sampling frame is validated, Westat draws a scientific random sample of a minimum of 150 schools from each frame with two replacement schools for each original school, unless there are less than 150 schools, in which case all schools would be sampled. A minimum of 50 schools are sampled for adjudicating entities (e.g., U.S. states that opt to participate separately in 2015). The list of selected schools, both original and replacement, is delivered to each education system's PISA national center. Countries and education systems do not draw their own samples.
Step 3
Each country/education system is responsible for recruiting the sampled schools. They begin with the original sample and only use the replacement schools if an original school refuses to participate. In accordance with PISA guidelines, replacement schools are identified by assigning the two schools neighboring the sampled school in the frame as substitutes to be used in instances where an original sampled school refuses to participate. Replacement schools are required to be in the same implicit stratum (i.e., have similar demographic characteristics) as the sampled school. A minimum participation rate of 65 percent of schools from the original sample of schools is required for a country or education system's data to be included in the international database.
Step 4
After schools are sampled and agree to participate, students are sampled. Each country/education system submits student listing forms containing all age-eligible students for each of their schools using Key Quest, the internationally provided software.
Step 5
Westat carefully reviews the student lists and uses sophisticated software to perform data validity checks to compare each list against what is known of the schools (e.g., expected enrollment, gender distribution) and PISA eligibility requirements (e.g., grade and birthday ranges). The selected student samples are then sent back to each national center. Unlike school sampling, students are not sampled with replacement.
Step 6
Schools inform students of their selection to participate on assessment day. Student participation must be at least 80 percent for a country's/education system's data to be reported by the OECD.
5. Which countries participate in PISA?
Countries and education systems within countries participate in PISA.
· PISA 2015: 75 countries and education systems have signed up to participate.
· PISA 2012: 65 countries and education systems participated.
· PISA 2009: 75 countries and education systems participated (10 of these administered PISA 2009+ in 2010).
· PISA 2006: 57 countries and education systems participated.
· PISA 2003: 41 countries and education systems participated.
· PISA 2000: 43 countries and education systems participated (11 of these administered PISA 2000 in 2001/2002).
6. When is PISA data collected in the United States?
PISA is administered in the fall in the United States, typically in October to November in the year of the assessment. The 2015 data will be collected in October-November 2015.
7. Where can I get a copy of the U.S. PISA reports?
· Performance of U.S. 15-Year-Old Students in Mathematics, Science, and Reading Literacy in an International Context: First Look at PISA 2012
· Highlights from PISA 2009: Performance of U.S. 15-Year-Old Students in Reading, Mathematics, and Science Literacy in an International Context
· Highlights from PISA 2006: Performance of U.S. 15-Year-Old Students in Science and Mathematics Literacy in an International Context
· International Outcomes of Learning in Mathematics Literacy and Problem Solving: PISA 2003 Results from the U.S. Perspective
· Outcomes of Learning: Results from the 2000 Program for International Student Assessment of 15-Year-Olds in Reading, Mathematics, and Science Literacy (PISA 2000)
These and other data products are available athttp://nces.ed.gov/pubsearch/getpubcats.asp?sid=098. The PISAInternational Data Explorer(IDE) (http://nces.ed.gov/surveys/international/ide/) includes data for the United States and other education systems. Users can create their own tables and figures with the IDE.
8. When is PISA next scheduled to be administered?
The next administration of PISA is in 2015. In the United States, data collection will occur in October-November 2015. Results will be reported in December 2016.
9. How does the performance of U.S. students in mathematics and science on PISA compare with U.S. student performance on TIMSS?
Before talking about how the TIMSS results compare with the PISA results, it is important to recognize the ways in which TIMSS and PISA differ.
While TIMSS and PISA both assess mathematics and science, they differ with respect to which students are assessed, what is measured, and the participating countries and educational jurisdictions.
· TIMSS assesses younger students (4th- and 8th-graders) on their knowledge of specific mathematics and science topics and cognitive skills that are closely linked to the curricula of the participating countries. PISA assesses older students (15-year-old students) in mathematics literacy and science literacy, or how well they can apply their knowledge and skills to problems set in real-world contexts.
· While there is some overlap in content, each assessment may have unique topics or different emphases and the nature of the items may differ as well, given their different focuses.
· Not all countries have participated in TIMSS and PISA, or in all administrations of either assessment. Both TIMSS and PISA include developed and developing countries; however, TIMSS has a larger proportion of developing countries participating than PISA because PISA is principally a study of the member countries of the OECD—an intergovernmental organization of developed countries. All 34 OECD countries participate in PISA, but not all of these 34 countries participate in TIMSS.
On TIMSS, students at 4th and 8th grades performed above the TIMSS scale average in both mathematics and science, unlike what we see in PISA in which—in 2012—U.S. 15-year-olds performed below (in mathematics) or not measurably different than (in science) the OECD averages. Five East Asian countries and education systems (Singapore, Korea, Hong Kong-China, Chinese Taipei, and Japan) outperformed the United States in mathematics and science in both TIMSS and PISA.
· Mathematics. The 2011 TIMSS results showed that U.S. students’ average mathematics score for both 4th-graders and 8th-graders were above the TIMSS scale average, which is set at 500 for every administration of TIMSS at both grades. At the 8th grade, students in 6 countries and 5 states or provinces had higher mathematics scores than U.S. students on average; this includes Singapore, Korea, Hong Kong-China, Chinese Taipei, Japan, Russian Federation, Indiana, Massachusetts, Minnesota, North Carolina, and Quebec.
· Science. The 2011 TIMSS results showed that U.S. students’ average science scores for both 4th-graders and 8th-graders were above the TIMSS scale average, which is set at 500 for every administration of TIMSS at both grades. However, students in 8 countries and 4 states or provinces outperformed U.S. students at the 8th grade level: Chinese Taipei, Finland, Japan, Korea, Singapore, Russian Federation, Slovenia, Hong Kong-China, Colorado, Massachusetts, Minnesota, and Alberta, Canada.
10. Can you report PISA results for states?
Yes and no. The U.S. national PISA results are representative of the nation as a whole but not of individual states. Drawing a sample that is representative of all 50 individual states and the District of Columbia would require a much larger sample than the United States currently draws for international assessments, requiring considerable amounts of additional time and money. A state may elect to participate in PISA as an individual education system—as Connecticut, Florida and Massachusetts did in 2012—and in that case a sample is drawn that is representative of that state.
3