PISA Truth Effects: the Construction of Low Performance

PISA truth effects: the construction of low performance

Author: Margareta Serder, Malmoe University, Sweden;

Key words: PISA, performed realities, lack of competence, scientific literacy

Network: 23 Politics of Education

This paper is engaged in questions of the realities “performed into being” (Gorur, 2010; Latour, 1999) through knowledge assessment, with a specific interest in PISA and its assessment of scientific literacy. The concern about PISA results accelerated in Sweden, where this study is located, after the latest PISA survey 2012, due to what has been asserted as a historical decrease in performance. During thebroadcasted press conference at the Swedish National Agency for Education held to present the national results, the director-general stated herconfidence in PISA: “Here at the National Agency for Education wetrust PISA” (Ekström, 13-12-03). In line with Gorur (2010), Popkewitz (2011) and many others, I believe there is reason tostress the problematic with this type of unreserved faith in PISA as evidence about educational failure or success. In this paper, I will suggest that low performance is an attribute of the assessment, more than an attribute of school systems or single students. Perhaps surprisingly, I will begin my argumentation in a discussion about microbes:

In The Historicity of Things, Bruno Latour (1999) asks: Where were the microbes before Pasteur? As writes Latour, many would find it commonsensical to answer that the microbes were in the same place before and after Pasteur; that Pasteur did not change the course of events – he only ‘discovered’ a pre-existing phenomenon, and named it microbes. According to Latour,existence presumes experience, and for that, to be given attributes: ‘the mind’ for example has to get the attributes ‘individual’, ‘dis-embodied’ or ‘thinking’ before it can be experienced as such[1]. Therefore, ‘obviously’ thing-like objects for thought, such as microbes – e.g. the ferments in Pasteur’s nineteenth century laboratory in Paris – first needed to be given circumstances, conditions, specific nutrition demands, and not least ways of being detected. Thereafter, they became microbes. And therefore, Latour insists, microbes did not exist before Pasteur “happened” to them (p. 146).

I argue that microbes are not much unlike knowledge, skills and competencies: A thing-like object for thought such as lack of competence (as in a 21st century, OECD-produced assessment about young citizens’ skills for future life) did not exist before a detector had been discovered/invented/constructed (Latour, 1999), making it possible to discern it as something (on detectors, see Knorr Cetina, 1999 and Pickering, 1995). The detector hereis the test-instrument[2]; adjusted to detecting defined competencies by the means of specific test-items within the assessment’s literacy framework.

The ontological argument for Latour lies in what we have learnt to hold for true and not:

In the correspondence theory of truth, the ferments are either out there or not, and if they are out there they have always been out there, and if they are not they have never been there.(Latour 1999, p. 149).

Ipropose that measured competencies are not ‘caused’ by knowledge and learnt skills, but are coming into existence through the means construed to detect them and that lack of competence is linked to the historical event of starting to try to detect a specific competence.In this paper, I aim to 1) trace the production of truth about low performance by looking into how the competencies related to scientific literacy are described and written out in PISA documents and test-items and 2) study how ‘lack of competence’ comes into existence when students are reading, making meaning of and responding to test-items from PISA science (OECD, 2006).

Methodology

The research is based on two different data materials; 1) written PISA documents (frameworks, analyses, test-items) and 2) a 16 hour video-taped material of research-produced situations in which 15-year old students collaboratively engaged in answering PISA test-questions in scientific literacy during a science lesson in the fall of 2010.

In the study, I am interesting myself in how science and performance is discursively and materially produced in the PISA science assessment. I base my analysis on the discourse theory of Foucault (1971), in which knowledge is linked to “being in the truth” (p. 36), that is, to represent what is believed to be true in a specific time, culture or social context. I use the notion of discourse as operating by the repetitive inclusion and exclusion of elements into the discourse; a mechanism through which discursive actions travel and become, or not, part of the discourse. Further, I am trying to adopt the non-realist approach to the contingency of matter from Latour as described earlier.

Documents

The analyzed documents are the PISA assessment frameworks from three of the five surveys (OECD 1999, 2006, 2012) with focus on 2006 in which the scientific literacy framework was defined, and the first and the most recent international report of results and analyses of scientific literacy (OECD 2007, 2013), one OECD report on “strong performers” (2010) and finally the released test-items in scientific literacy from units/24 items).

Student setting

Four 9th grade classes from a Swedish compulsory school participated. They worked in teacher-constructed groups(à 2-4 students) with the instruction to discuss and agree on one common written answer. The written material was also collected. Eleven test-items,(from Acid Rain [S485], Greenhouse [S114] and Sunscreens [S447]), were selected using criteria of relevance to the age group and situation and for long-term use in PISA (see Serder & Jakobsson, 2014).

Analytic procedure

The PISA documents were read through several times in order to conduct a discourse analysis (Kendall & Wickham, 1999) concerned with how science, science knowers, scientific literacy and performance was put forward in the documents.

The transcribed video-material of student discussions wasanalyzed with a focus on how students interacted with the testing material in situations that were categorized as “lack of knowledge”. It centered specific attributes of the test that could be seen to ‘prompt’ low performance (no answers/answers that would rend no score) or experience of low performance.

Expected results from analysis

The documents stresslife-long learning, includingcompetencies such as “willingness to engage”, “flexibility”, “problem solving” and “adaptation” (e.g. OECD, 2006, p. 10-11).

Science seems capable of solving all problems, creating a truthof ‘science and technology’ (ST) as making people take rational decisions, based on scientific arguments. The competent student shares incontestable, scientific values(OECD, 2006, p. 20).

Further, ST contributes to all levels of humanity. Implicitly, there are decisions that are ‘right’ (scientifically informed)while others are ‘wrong’(e.g. socially, ethically or economically informed).The future citizen is incessantly facing challenges that demand scientific competence, who is willing to engage, and who is rational and always acting within this discourse of truth (see e.g. OECD, 2006, s. 24). The student “at risk” (OECD, 2010, p. 29) has no interest in science in this particular manner and does not privilege ST; andwould actually, within this discourse, have a lack.

Explicitly, low performers are described with statements such as “lack the skills needed to function effectively (…) who fail to reach the baseline level of performance (…) therefore risk facing difficulties (…) throughout their lives”. (OECD, 2013, p. 254).

Concerning the attributes of the detector (Knorr Cetina, 1999; Latour, 1999) single words become decisive for the possibility or not to construe answers at the test. Words such as “reference”, “pattern”, “step”, “factor”, “be found” etc. (see Serder & Jakobsson, 2014) are discussed with different meanings in different groups. Therefore, attributes of the test-items can be discussed in relation to students’ non-participation, blaming themselves, questioning of the emerging image of science and answering ‘nonsensical’.

Rickard: the words are so strange…

Jana: what?

Rickard: ”reference substance”…

Jana: yeah…like, what is that?

Cornelia: (looking out through the window)

(silent reading)

Rickard: if only one knew what reference means…?

(Excerpt, Group L, Nov 2010)

Intent of publication

This work is part of a doctoral thesis in progress, intended to be defended in January 2015. Hopefully this text in an elaborated version will be published within the thesis.

References

Ekström, A. (2013). Broadcast from press conference of the Swedish PISA results. 2013-12-03

Foucault, M. (1971). L’ordre du discours. Editions Gallimard.

Gorur, R. (2010). ANT on the PISA Trail: Following the statistical pursuit of certainty. Educational Philosophy and Theory, 43(1), 76-93.

Kendhall, G. & Wickham, G. (1999). Using Foucault Methods. London: SAGE.

Knorr-Cetina, K. (1999). Epistemic Cultures. How the sciences make knowledge. Cambridge, Massachussetts: Harvard University Press.

Latour, B. (1999). Pandora’s Hope: essays on the reality of science studies. Cambridge, Mass.: Harvard University Press.

OECD (2013).PISA 2012 Results: What Students Know and Can Do – Student Performance in Mathematics, Reading and Science (Volume I). OECD Publications.

OECD (2010). Strong performers and successful reformers in education. OECD Publications.

OECD (2007). Competencies for tomorrow’s world. Volume 1. Analysis. OECD Publications.

OECD (2006). Assessing Scientific, Reading and Mathematical Literacy.A Framework for PISA 2006. OECD Publications.

OECD, 1999. Measuring knowledge and skills. A new framework for assessment. OECD Publications.

Pickering, A. (1995). The Mangle of Practice: Time, Agency and Science. University of Chicago Press.

Popkewitz, T. (2011). PISA. In M. A. Pereyra, H.G. Kotthoff & R. Cowen (Eds.),PISA Under Examination: Changing Knowledge, Changing Tests, and Changing Schools (pp. 31-46). Sense Publishers.

Serder, M. & Jakobsson, A. (2014), “why bother so incredibly much?” Student perspectives on PISA science assignments. Cultural Studies of Science Education.DOI: 10.1007/s11422-013-9550-3

1

[1] Which is exactly why the mind also can be experienced as collective, embodied and acting, for instance

[2] In the case of microbes, the detector was the nose of Louis Pasteur.