Highlights from PISA
Highlights from PISA
Slide 1
Highlights from the Program for International Student Assessment (PISA)
PISA 2012
Slide 2
What is PISA?
International large-scale assessment organized by the Organization for Economic Cooperation and Development (OECD)
NCES is U.S. coordinating and funding organization. PISA Governing Board (PBG) determines policy
Assessment of 15-year-old students
Administered every 3 years 2000
Slide 3
What is PISA?
In 2012, 68 education systems participated in PISA.
Albania, Argentina, Australia, Austria, Belgium, Brazil, Bulgaria, Canada, Chile, Colombia, Connecticut-USA, Costa Rica, Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, Florida-USA, France, Germany, Greece, Hong-Kong-CHN, Hungary, Iceland, Indonesia, Ireland, Israel, Italy, Japan, Jordan, Kazakhstan, Korea, Latvia, Liechtenstein, Lithuania, Luxembourg, Macao-CHN, Malaysia, Massachusetts-USA, Mexico, Montenegro, Netherlands, New Zealand, Norway, Peru, Poland, Portugal, Qatar, Romania, Russian Federation, Serbia, Shanghai-CHN, Singapore, Slovak Republic, Slovenia, Spain, Sweden, Switzerland, Taipei-CHN, Thailand, Tunisia, Turkey, United Arab Emirates, United Kingdom, United States, Uruguay, and Vietnam.
Slide 4
What is PISA?
Assessment subjects: Mathematics, science, reading literacy. Mathematics was main subject assessed in 2012.
Computer-based mathematics and reading literacy. Optional assessment taken by 32 education systems.
Content knowledge, not limited to school-based curricula
PISA assesses applied knowledge/literacy: “How well can students nearing the end of compulsory schooling apply their knowledge to real-life situations?”
Slide 5
What PISA reports
Average scores: Scale of 0-1000 for all domains
Proficiency levels: Percentages of students scoring at 6 levels
Trends: Change between average scores in 2012 and the scores in previous assessment years
Subgroup scores: International (e.g. gender, language spoken in home) and U.S. specific variables (e.g., race/ethnicity)
Slide 6
U.S. PISA Findings
Slide 7
General patterns of U.S. results
Ranked better in reading literacy than in mathematics and science literacy
Below OECD average score in mathematics only
Higher percentage at low proficiency levels than OECD average in mathematics only
Lower percentage of top performers than OECD average in mathematics only
No measurable change in average scores in mathematics, science, or reading literacy
Slide 8
Mathematics literacy
Slide 9
Quantity (25%): Are 15-year-olds able to comprehend multiple representations of numbers, engage in mental calculation, employ estimation, and assess the reasonableness of results?
Uncertainty and Data (25%): Can students use probability and statistics and other techniques of data representation and description to mathematically describe, model, and interpret uncertainty?
Space and Shape (25%): Can students understand perspective, create and read maps, and manipulate 3D objects?
Change and Relationships (25%): Can students model change and relationships with the appropriate functions and equations?
Slide 10
PISA 2012 mathematics literacy content categories
Formulate (25%): Can 15-year-olds recognize and identify opportunities to use mathematics and then provide mathematical structure to a problem presented in some contextualized form in order to formulate situations mathematically?
Employ (25%): Are students able to employ mathematical concepts, facts, procedures, and reasoning to solve mathematically-formulated problems and obtain mathematical conclusions?
Interpret (25%): Can students interpret, apply and evaluate mathematical outcomes in order to determine whether results are reasonable and make sense in the context of the problem?
Slide 11
PISA mathematics literacy proficiency levels
Level 1: Identify information and carry out routine procedures according to direct instructions in explicit situations.
Level 2: Employ basic algorithms, formulae, procedures, or conventions. Capable of direct reasoning and making literal interpretations of the results.
Level 3: Execute clearly described procedures, including those that require sequential decisions. Select and apply simple problem-solving strategies.
Level 4: Work effectively with explicit models that may involve constraints or making assumptions. Capable of reasoning with some insight in straightforward contexts.
Level 5: Work strategically using broad, well-developed thinking and reasoning skills, appropriate linked representations, symbolic and formal characterizations and insight pertaining to these situations.
Level 6: Apply insight along with a mastery of symbolic and formal mathematical operations and relationships to develop new approaches and strategies for attacking novel situations.
Slide 12
Mathematics literacy – Example item
Proficiency Level 1
Question: CHARTS
In January, the new CDs of the bands 4U2Rock and The Kicking Kangaroos were released. In February, the CDs of the bands No One’s Darling and The Metalfolkies followed. The following graph shows the sale of the bands’ CDs from January to June.
Jan: 4U2Rock – 2125; The Kicking Kangaroos – 1650
Feb: 4U2Rock – 2100; The Kicking Kangaroos – 1875; No One’s Darling-250; The Metalfolkies – 1000
Mar: 4U2Rock – 1975; The Kicking Kangaroos – 1575; No One’s Darling-1325; The Metalfolkies – 250
Apr: 4U2Rock – 1800; The Kicking Kangaroos – 1250; No One’s Darling-1600; The Metalfolkies – 500
May: 4U2Rock – 1700; The Kicking Kangaroos – 950; No One’s Darling-1725; The Metalfolkies – 650
Jun: 4U2Rock – 2025; The Kicking Kangaroos – 700; No One’s Darling-1825; The Metalfolkies – 900
Slide 13
Mathematics literacy – Example item
Question: CHARTS – Continued
In which month did the band No One’s Darling sell more CDs than the band The Kicking Kangaroos for the first time?
A No month
B March
C April
D May
SCORING
Correct
C April
Incorrect
Other responses.
Percentage of students answering correctly in 2012:
OECD average: 80%
U.S.: 77%
Slide 14
Mathematics literacy – Example item
Proficiency Level 5
Question: DRIP RATE
Intravenous drips are used to deliver fluids and drugs to patients.
(Picture of a nurse looking at an intravenous drip)
Nurses need to calculate the drip rate, D, in drops per minute for intravenous drips.
They use the formula D equals dv divided by 60n where
d is the drop factor measured in drops per milliliter (mL)
v is the volume in mL of the intravenous drip
n is the number of hours the intravenous drip is required to run.
Slide 15
Mathematics literacy – Example item
Question: DRIP RATE – Continued
Nurses also need to calculate the volume of the intravenous drip, v, from the drip rate, D.
An intravenous drip rate of 50 drops per minute has to be given to a patient for 3 hours. For this intravenous drip the drop factor is 25 drops per milliliter.
What is the volume in mL of the intravenous drip?
Volume of the intravenous drip : ______mL
SCORING
Correct
360 or a correctly transposed and substituted solution.
- 360
- (60 x 3 x 50) 25 [Correct transposition and substitution.]
Incorrect
Other responses.
Percentage of students answering correctly in PISA 2012:
OECD average: 26%
U.S.: 30%
Slide 16
Of 34 OECD Countries, U.S.A. Ranks 27th in Mathematics Literacy
2012 PISA- Mathematics Literacy
1st Korea 554; 2nd Japan 536; 3rd Switzerland 531; 4th Netherlands 523; 5th Estonia 521; 6th Finland 519; 7th Canada 518; 8th Poland 518; 9th Belgium 515; 10th Germany 514; 11th Austria 506; 12th Australia 504; 13th Ireland 501; 14th Slovenia 501; 15th Denmark 500; 16th New Zealand 500; 17th Czech Republic 499; 18th France 495; OECD average 494; 19th United Kingdom 494; 20th Iceland 493; 21st Luxembourg 490; 22nd Norway 489; 23rd Portugal 487; 24th Italy 485; 25th Spain 484; 26th Slovak Republic 482; 27th United States 481; 28th Sweden 478; 29th Hungary 477; 30th Israel 466; 31st Greece 453; 32nd Turkey 448; 33rd Chile 423; 34th Mexico 413
Slide 17
U.S. 15-year-olds OECD average in mathematics literacy
Average higher than U.S. average
Shanghai-CHN 613; Singapore 573; Hong Kong-CHN 561; Taipei-CHN 560; Republic of Korea 554; Macao-CHN 538; Japan 536; Liechtenstein 535; Switzerland 531; Netherlands 523; Estonia 521; Finland 519; Canada 518; Poland 518; Belgium 515; Massachusetts-USA 514; Germany 514; Vietnam 511; Connecticut-USA 506; Austria 506; Australia 504; Ireland 501; Slovenia 501; Denmark 500; New Zealand 500; Czech Republic 499; France 495; OECD average 494; United Kingdom 494; Iceland 493; Latvia 491; Luxembourg 490
Average not measurably different from U.S. average
Norway 489; Portugal 487; Italy 485; Spain 484; Russian Federation 482; Slovak Republic 482; United States 481; Lithuania 479; Sweden 478; Hungary 477
Average lower than U.S. average
Croatia 471; Florida-USA 467; Israel 466; Greece 453; Serbia 449; Turkey 448; Romania 445; Cyprus 440; Bulgaria 439; United Arab Emirates 434; Kazakhstan 432; Thailand 427; Chile 423; Malaysia 421; Mexico 413; Montenegro 410; Uruguay 409; Costa Rica 407; Albania 394; Brazil 391; Argentina 388; Tunisia 388; Jordan 386; Colombia 376; Qatar 376; Indonesia 375; Peru 368
Massachusetts average higher than the U.S. and OECD averages, Connecticut average higher than the U.S. average but not measurably different from the OECD average, and Florida average lower than the U.S. and OECD averages.
Slide 18
State results in mathematics literacy
Massachusetts: 9 education systems scored higher; 12 scored not measurably different; 44 scored lower
Connecticut: 12 education systems scored higher; 14 scored not measurably different; 39 scored lower
Florida: 36 education systems scored higher; 5 scored not measurably different; 24 scored lower
Slide 19
U.S. not measurably different from OECD average in two mathematics content subscales
Subscale
Quantity: U.S. average score 478; OECD average score 495. OECD average significantly higher than U.S. average
Uncertainty and data: U.S. average score 488; OECD average score 493. OECD average not measurably different from U.S. average
Change and relationships: U.S. average score 488; OECD average score 493. OECD average not measurably different from U.S. average
Space and shape: U.S. average score 463; OECD average score 490. OECD average significantly higher than U.S. average
Slide 20
In mathematics literacy, 9 percent of U.S. 15-year-old students scored at proficiency level 5 or above
Percentage at proficiency level 5 or above higher than U.S. percentage:
Shanghai-CHN, Below level 2: 4, Levels 5 and above: 55; Singapore, Below level 2: 8, Levels 5 and above: 40; Taipei-CHN, Below level 2: 13, Levels 5 and above: 37; Hong Kong-CHN, Below level 2: 9, Levels 5 and above: 34; Republic of Korea, Below level 2: 9, Levels 5 and above: 31; Liechtenstein, Below level 2: 14, Levels 5 and above: 25; Macao-CHN, Below level 2: 11, Levels 5 and above: 24; Japan, Below level 2: 11, Levels 5 and above: 24; Switzerland, Below level 2: 12, Levels 5 and above: 21; Belgium, Below level 2: 19, Levels 5 and above: 20; Netherlands, Below level 2: 15, Levels 5 and above: 19; Massachusetts-USA, Below level 2: 18, Levels 5 and above: 19; Germany, Below level 2: 18, Levels 5 and above: 17; Poland, Below level 2: 14, Levels 5 and above: 17; Connecticut-USA, Below level 2: 21, Levels 5 and above: 16; Canada, Below level 2: 14, Levels 5 and above: 16; Finland, Below level 2: 12, Levels 5 and above: 15; New Zealand, Below level 2: 23, Levels 5 and above: 15; Australia, Below level 2: 20, Levels 5 and above: 15; Estonia, Below level 2: 11, Levels 5 and above: 15; Austria, Below level 2: 19, Levels 5 and above: 14; Slovenia, Below level 2: 20, Levels 5 and above: 14; Vietnam, Below level 2: 14, Levels 5 and above: 13; France, Below level 2:22, Levels 5 and above: 13; Czech Republic, Below level 2: 21, Levels 5 and above: 13; OECD average, Below level 2: 23, Levels 5 and above: 13; United Kingdom, Below level 2: 22, Levels 5 and above: 12; Luxembourg, Below level 2: 24, Levels 5 and above: 11; Iceland, Below level 2: 21, Levels 5 and above: 11; Ireland, Below level 2: 17, Levels 5 and above: 11.
Percentage not measurably different from U.S. percentage
Slovak Republic, Below level 2: 27, Levels 5 and above: 11; Portugal, Below level 2: 25, Levels 5 and above: 11; Denmark, Below level 2: 17, Levels 5 and above: 10; Italy, Below level 2: 25, Levels 5 and above: 10; Norway, Below level 2: 22, Levels 5 and above: 9; Israel, Below level 2: 34, Levels 5 and above: 9; Hungary, Below level 2: 28, Levels 5 and above: 9; United States, Below level 2: 26, Levels 5 and above: 9; Lithuania, Below level 2: 26, Levels 5 and above: 8; Sweden, Below level 2: 27, Levels 5 and above: 8; Spain, Below level 2: 24, Levels 5 and above: 8; Latvia, Below level 2: 20, Levels 5 and above: 8; Russian Federation, Below level 2: 24, Levels 5 and above: 8; Croatia, Below level 2: 30, Levels 5 and above: 7.
Percentage at proficiency level 5 or above lower than U.S. percentage
Turkey, Below level 2: 42, Levels 5 and above: 6; Florida-USA, Below level 2: 30, Levels 5 and above: 6; Serbia, Below level 2: 39, Levels 5 and above: 5; Bulgaria, Below level 2: 44, Levels 5 and above: 4; Greece, Below level 2: 36, Levels 5 and above: 4; Cyprus, Below level 2: 42, Levels 5 and above: 4; United Arab Emirates, Below level 2: 46, Levels 5 and above: 3; Romania, Below level 2: 41, Levels 5 and above: 3; Thailand, Below level 2: 50, Levels 5 and above: 3; Qatar, Below level 2: 70, Levels 5 and above: 2; Chile, Below level 2: 52, Levels 5 and above: 2; Uruguay, Below level 2: 56, Levels 5 and above: 1; Malaysia, Below level 2: 52, Levels 5 and above: 1; Montenegro, Below level 2: 57, Levels 5 and above: 1; Kazakhstan, Below level 2: 45, Levels 5 and above: 1; Albania, Below level 2: 61, Levels 5 and above: 1; Tunisia, Below level 2: 68, Levels 5 and above: 1; Brazil, Below level 2: 67, Levels 5 and above: 1; Mexico, Below level 2: 55, Levels 5 and above: 1; Peru, Below level 2: 75, Levels 5 and above: 1; Costa Rica, Below level 2: 60, Levels 5 and above: 1; Jordan, Below level 2: 69, Levels 5 and above: 0; Colombia, Below level 2: 74, Levels 5 and above: 0; Indonesia, Below level 2: 76, Levels 5 and above: 0; Argentina, Below level 2: 66, Levels 5 and above: 0.
Massachusetts and Connecticut had higher percentages than the U.S and OECD; Florida’s percentage was lower than those of the U.S. and OECD.
Slide 21
No measurable change since 2009 in average mathematics literacy scores in more than half of PISA education systems, including U.S.
Of the 62 education systems in PISA 2009 and 2012:
In 33 (including the U.S.) no measurable change in average scores
In 18 average scores increased. Russian Federation was below U.S. and OECD averages in 2003; not measurably different in 2012. Poland was lower than OECD and not measurably different from U.S. averages in 2003; higher than both in 2012.
In 11 average scores declined. Finland declined between 2006-2009 and again between 2009-2012; scored 548 in 2006 and 519 in 2012. Norway was above U.S. average in 2003; not measurably different from U.S. average and below OECD average in 2012
Slide 22
Science Literacy
Slide 23
PISA science literacy proficiency levels
Level 1: Present scientific explanations that are obvious and follow explicitly from given evidence.
Level 2: Use direct reasoning and make literal interpretations of the results of scientific inquiry of technological problem solving.
Level 3: Identify clearly described scientific issues in a range of contexts. Select facts and knowledge to explain phenomena and apply simple models or inquiry strategies.
Level 4: Select and integrate explanations from different disciplines of science or technology and link those explanations directly to aspects of life situations.
Level 5: Use well-developed inquiry abilities, link knowledge appropriately, and bring critical insights to situations. Construct explanations based on evidence and arguments based on critical analysis.
Level 6: Link different information sources and explanations and use evidence from those sources to justify decisions. Demonstrate advanced scientific thinking and reasoning, and use scientific understanding in support of solutions to unfamiliar scientific situations.
Slide 24
Science literacy – Example item
Proficiency Level 2
Question: MARY MONTAGU
Read the following newspaper article and answer the questions that follow:
The History of Vaccination
Mary Mantagu was a beautiful woman. She survived an attack of smallpox in 1715 but was left covered with scars. While living in Turkey in 1717, she observed a method called inoculation that was commonly used there. This treatment involved scratching a weak type of smallpox virus onto the skin of healthy young people who then became sick, but in most cases only with a mild form of the disease.
Mary Montagu was so convinced of the safety of these inoculations that she allowed her son and daughter to be inoculated.
In 1796, Edward Jenner used inoculations of a related disease, cowpox, to produce antibodies against smallpox. Compared with the inoculation of smallpox, this treatment had less side effects and the treated person could not infect others. The treatment became known as vaccination.
Slide 25
Science literacy – Example item
Question: MARY MONTAGU – continued
What kinds of diseases can people be vaccinated against?
A Inherited diseases like hemophilia
B Diseases that are caused by viruses, like polio
C Diseases from the malfunctioning of the body, like diabetes
D Any sort of disease that has no cure
SCORING
Correct
Answer B. Diseases that are caused by viruses, like polio.
Incorrect
Other Responses
Percentage of students answering correctly in 2006:
OECD average: 75%
U.S.: 73%
Slide 26
Science literacy – Example item
Proficiency Level 6
Question: GREENHOUSE
Read the texts and answer the questions that follow.
The Greenhouse Effect: Fact or Fiction?
Living things need energy to survive. The energy that sustains life on Earth comes from the Sun, which radiates energy into space because it is so hot. A tiny proportion of this energy reaches the Earth.
The Earth’s atmosphere acts like a protective blanket over the surface of our planet, preventing the variations in temperature that would exist in an airless world.
Most of the radiated energy coming from the Sun passes through the Earth’s atmosphere. The Earth absorbs some of this energy, and some is reflected back from the Earth’s surface. Part of this reflected energy is absorbed by the atmosphere.
As a result of this, the average temperature above the Earth’s surface is higher than it would be if there was no atmosphere. The Earth’s atmosphere has the same effect as a greenhouse, hence the term greenhouse effect.
The greenhouse effect is said to have become more pronounced during the twentieth century.
It is a fact that the average temperature of the Earth’s atmosphere has increased. In newspapers and periodicals the increased carbon dioxide emission is often stated as the main source of the temperature rise in the twentieth century.
Slide 27
Science literacy – Example item
Question: GREENHOUSE – Continued
Andre persists in his conclusion that the average temperature rise of the Earth’s atmosphere is caused by the increase in the carbon dioxide emission. But Jeanne thinks that his conclusion is premature. She says: “Before accepting this conclusion you must be sure that other factors that could influence the greenhouse effect are constant”.
Name one of the factors that Jeanne means.
SCORING
Correct
Answers that give a factor referring to the energy/radiation coming from the Sun.
Answers that give a factor referring to a natural component or a potential pollutant.
Incorrect
Answers that refer to a cause that influences the carbon dioxide concentration.
Answers that refer to a non-specific factor.
Other incorrect factors or other responses.
Percentage of students answering correctly in 2006: