Beyond the Reach of Teaching and Measurement
Beyond the Reach of Teaching and Measurement:
Methodology and Initial Findings of the Innsbruck Vignette Research
Michael Schratz, Johanna F. Schwarz, Tanja Westfall-Greiter
Contact:
Michael Schratz
School of Education
University of Innsbruck
Innrain 52
A-6020 Innsbruck, Austria
Europe
Abstract
This contribution provides insights into the learning research being conducted at the University of Innsbruck, Austria, where the “Innsbruck Vignette Research” was developed. The research methodology, findings and potentials were presented to a broad international audience for the first time at the ICSEI Conference 2013 in Santiago, Chile. The vignette research was developed in a grant-funded project still in progress and is designed to gain access to students’ learning experiences in the classroom as they occur. The authors frame the research need out ofwhich the methodology developed and describethe theoretical foundations of this particular form of qualitative phenomenologically grounded methodology. The vignette research is illustrated by a hermeneutic phenomenological vignette reading. The significance of the Innsbruck Vignette Research for research into teaching and learning is presented as well as the extended development of the “vignette-driven interview” as a research method. Finally, the relevance of vignette work for teacher education and system development is discussed, including its application thus far in teacher qualification programs and professional learning communities in a nationwide school network of teacher leaders.
Key words: vignette research, phenomenology, learning, lived experience
Beyond the Reach of Teaching and Measurement:
Methodology and Initial Findings of the Innsbruck Vignette Research
Today’s school effectiveness efforts are largely driven by the measurement of learning outcomes both on national and international levels, most predominantly through standardized tests such as TIMSS or PISA. Ideally, such testing provides data not only on the actual learning outcomes measured but also the impact of the school and system regarding socio-economic factors. This information is expected to stimulate school improvement efforts by providing a body of evidence which informs strategically focused measures with a view towards creating better learning outcomes. In this evidence-informed management approach in whichschool quality is articulated in test scores, a common response is fear if schools do not succeed in achieving high scores, particularly when a scheme of reward and punishment and a mental model of uniformity rather than diversity underlie system monitoring. As Biesta (2012) notes, “out of fear of ending up at the bottom end of the league table, we can see schools and school systems transforming themselves into the definition of education that ‘counts’ in systems like PISA, the result of it being that more and more schools and school systems begin to become the same” (p. 10). This growing tendency to uniformity can be encountered nationally due to policy-borrowing (Steiner-Khamsi, 2004) and locally when schools buy into results-based instructional programs promising better scores.
With the focus on testing for purposes of system monitoring it is easy to forget that such measurements cannot yield insights directly transferable into the classroom, where assessment is less about the scores and more about the individual students whose scores are reported. Student populations are becoming exceedingly diverse not only due to demographic, economic and social changes but also in regard toa growing variety of life contexts and ways of life. In all types of schools, school effectiveness efforts have given rise to a paradox: The more diverse classrooms become, the greaterthe tendency towards uniformity through results-based management measures directed atimproving students’ learning outcomes.
In addition, the widespread use of measurement of outcomes has ledto what Biesta (2010) calls “learnification” ofthe educational agenda, referring to “the transformation of the vocabulary used to talk about education into one of ‘learning’ and ‘learners’” (p. 18). The problem in this transformation lies in the underlying individualistic concept at odds with the goals of education and the emptiness of the term “learning”, which “denotes processes and activities … open – if not empty – with regard to content and direction” (ibidem). Similarly, Schratz (2009) sees this “learnification” as an orientation which defines learning from the perspective of teaching (lehrseits).
In German-speaking countries where the education systemsare highly differentiated, this lehrseits perspective has led to an emphasis on individualized instructionand the ideal of so-called “open learning arrangements” (offenes Lernen) as a remedy for the heterogeneity dilemma. Open learning arrangements reduced to the organizational levelof instruction shift students’ attention from the teacher to the material and responsibility from the teacher to the learners. This approach often disguises and thereby fails to transform the traditional “pedagogy of poverty” (Haberman, 1991) into a “pedagogy of plenty” (Hodges, 2001) in which students can gainwhat Dewey (1918, 1936) described as “educative experience”. In the name of “learner autonomy” the students are left largely to themselves and the role of the teacher is often reduced to “coach” or “helper” through aslalom course of materialrather than that of effective educator oriented to educative processes andBildung.
Mecheril and Plößer (2009) define Bildung in contemporary society as a process which fosters participation of individuals in social structures, defines educational institutions and legitimizes them. Current practices in differentiated school systemssuch as those in the German-speaking world, in which differences are created and then problematized through a one-size-fits-all approach in which all must fit into the one size on offer, extrapolates inequity rather than effectively including and responding to social, demographic, ethnic, linguistic, physical and gender differences. Accordingly, there is evidence for Mecheril’s claim that “school produces difference” (2009): Learners adapt to and are formed by the school system rather than school adapting to their diverse personal and social conditions in order to provide greater opportunities and access to higher education. While this is further exacerbated by the so-called “middle-class orientation” and systematic tracking of most European school systems (Arens & Mecheril, 2010), there is also evidence that tracking may have gone underground but it is on the rise in the United States, delineating class and race (Sacks, 2007; Darling-Hammond, 2004). Proactively and reflectively dealing with diversity is the key issue in organizing teaching and learning processes (Schratz, 2003), particularly in contexts where diversity is problematized rather than embraced.
Because international and national large-scale assessments mainly provide data on the system level by collecting statistical data and measuring outcomes after the learning has occurred, they can contribute little to what can be done to improve quality in the classroom, where the teacher’s actions and interventions influence the type and quality of learning experience that leads to measurable learning outcomes. Even on the classroom level, assessments of learningprovide little information for improving quality, because teaching and learning processes have already culminated and come to closure. Such information comes too late for the teacher to act responsively and proactively.
The emphasis on measurable outcomes tends to ignore learning inits nascent state, as the process is set in motion and culminates; what happens in learning processes is rarely the focus of attention, a trend which can even lead to a distorted understanding of what learning is:
“Learning itself comes predominantly into view in the form of results, as in long-term neuronal connections from a neuroscientific perspective or in the building of memory within the framework of cognition theory and its assumptions. The process itself withdraws from our attention both life-worldly and scientifically.” (Meyer-Drawe, 2010, p. 9)
Mitgutsch (2008) reveals the structures of the learning experience and points to learning as a phenomenon which leads a “shadowy existence” (p. 263). If we understand learning as experience (Meyer-Drawe, 2008; 2010) rather than learning as a product out of experience, we see that learning and teaching processes are irrevocably intertwined and codetermining (Schratz, 2009; Schratz, Schwarz, & Westfall-Greiter, 2012). Similarly, Hattie (2011) emphasizes the invisible nature of learning and the OECD’s current “Innovative Learning Environments” (2012) project poses the essential question for evaluating innovation: “What do the learners experience in this learning trajectory?”
A phenomenological approach
Phenomenology as the philosophy of experience is the foundation of lived experience research. The Innsbruck Vignette Researchdiscussed in this contributionis aphenomenological approach to empirical school research which captures the experiences of students in school in an effort to reveal learning as it is set in motion and culminates. The research methodology was developed in Phase 1 of thegrant-funded research project “Personal Learning and Development in Diverse Classroom Communities”[1], which was linked to the Austrian school reform pilot “Neue Mittelschule” [“New Secondary School”(NMS)] initiated in 2008 by the Minister of Education[2]. The initial study (Phase 1)aimed at capturing the experiences of students in everyday school life in order to explore phenomena of learning as constitutive of personal educational processes. By capturing the specific “lived experience” of learners in diverse classroom communities and taking a hermeneutic phenomenological approach to analyzing the data, universal phenomena related to learning and the experience of learning in medias res can be revealed, thereby drawing invisible learning out of its shadowy existence.
The researchers[3] specifically studied diverse classroom communities at 24 school sites across Austria by collecting data at three time points over a one-year period in grade 5 NMS classrooms. Each researcher spent a minimum of two days in the field at three different times (October 2009,January 2010, May 2010) to obtain data, focusing on two learners selected by the teachers in each class[4]. A mix of qualitative data collection instruments was used in order to gain multi-perspective insights into students’ experiences at each school: protocols of lived experience (van Manen, 1990); conversations with the students, their guardians and their teachers; student focus group discussions (Flick, 2005; Morgan, 1998); vignettes; photo documentation by the students and document analysis of learning products selected by the students as well as their teachers.
Co-experiencing and protocoling the lived experiences of the students was a prerequisite in this study for two reasons. Van Manen (1990), who has played a fundamental role in establishing lived experience research in North America, discusses the reflective nature of all descriptions of events and actions:
“All recollections of experiences, reflections of experience, descriptions of experiences, taped interviews about experiences, or transcribed conversations about experiences are already transformations of those experiences. Even life captured directly on magnetic or light-sensitive tape is already transformed at the moment it is captured … they have already lost the natural quiver of their undisturbed existence.” (p. 54)
Vignettes of Lived Experience
In research vignettesare commonly known as fictive case descriptions used in surveys. In our usage as a qualitative, phenomenologically oriented research instrument, the vignette is a dense narrative of a poignant moment, which captures lived experience as it occurs in the classroom. Vignettes in this context are a form of literary non-fiction, initially inspired by van Manen’s (1990) “anecdotes of lived experience”. While van Manen’s anecdotes are based on recall, our vignettes stem from researchers co-experiencing the lived experience of students in the midst of the pedagogic situation, in medias res. In the field, researchers attempt to stay open and particularly attend to pathic elements such as atmosphere, facial and bodily expressions and tone of voice while co-experiencing. Thesedetails are noted by researchers in protocols as a stream of experiential information which then form the basis for writing the vignettes.
Insights from this foundational research should contribute to the growing body of research into the phenomena related to learning. The approach casts seemingly familiar issues in a new light, allowing for a fundamentally alien perspective – that of the learner or the student – so that the fragility inherent in the experience of learning to come into view (Meyer-Drawe, 2008, p. 15). “The impossibility of seeing with the eyes of a child or adolescent forces us to expose ourselves to the alien” (Meyer-Drawe, 2010, p. 11). Schratz (2009), referring to Juul & Jensen (2005, p. 290), advocates sensitivity – the will and ability to respond caringly, empathetically, and reflectively to the self-concepts of learners – as one major prerequisite for an effective approach to learner diversity (cf. Deci & Ryan, 1994). This responsive stance is related to both Muth’s (1962) and van Manen’s (1991, 187f; 2002) notion of “pedagogical tact”.
The stance of the researcher is a central factor to be open to the fundamentally alien. Like teachers, researchers are dependent on learning being revealed in the articulations of students. They are called upon to take on a particular disposition of openness and sensitivity to take notice of it if and when it occurs. Attempting to capture (learning) experiences in statu nascendi means that both learners and researchers are affected by the experience in the midst of the event. Neither can reflect on the experience as it occurs. Rather, learningas event is something that happens, that one undergoes. In the throes of experience, it is impossible to simultaneously be participant and observer. In order to capture such complex data marked by intersubjectivity and interwovenness which the participants themselves are not able to perceive or report in medias res, protocols of lived experience are usedto capture the pathic elements of the experience (cf. van Manen, 2002). AsDenzin (1989) argues, such methods are necessary for understanding the complexities of what is happening in diverse classroom communities:
we need to employ an approach that captures and records the voices of the lived experience … goes beyond mere fact and surface appearances … presents details, context, emotion, and the webs of the social relationship that joins to one another (p. 83).
Lippitz (2003), who has developed a methodological and empirical approach to conducting phenomenological research on childhood and pedagogical ethics in the context of difference and foreignness, argues that it is crucial to “capture experience methodologically... what structure it has and with which validity requirements it can be connected” (Lippitz, 1993, p. 19). Waldenfels (2000), whose recent theorem of “bodily responsivity” is seen as a critical alternative to Husserl’s “concept of intentionality”, recommends extrapolating “what reveals itself, through how it reveals itself” and points tothe need to honor the uniqueness of one’s own and others’ experience while simultaneously consciously ignoring it (Waldenfels, 1992, p. 30). This suspension of one’s own assumptions is essential to ensure rigor and discipline in the data collection. Consequently, researchers involved in the Innsbruck Vignette Research are trained by lead researchers before entering the field.
It goes unsaid that capturing learning experiences for purposes of research is a complex task. Learner utterances such as “Arg!”, “This is so hard!”, or“I get it!” are audible and thus traceable articulations of student learning and understanding, but much of experiential articulation, like knowledge (Polanyi, 1962), is not and cannot be verbalized. Teachers know and recognize these non-verbal articulations – their own excitement when students beam with accomplishment or move fluidlywhile working with full concentration as well as their own discomfort when they noticeslouched shoulders, grimaces and fidgeting. It is this full experience beyond the verbal which vignettes attempt to convey.
To capture data in the field that will yield a vignette, researchers must take a stance that goes beyond mere observation so that the events can be described fully as experiences. A phenomenological approach to empirical research such as the InnsbruckVignette Research sets as prerequisite to using the methodology a particular (phenomenological) way of thinking (Merleau-Ponty, 1958, p. viii) and a particular disposition in the world. Meyer-Drawe, a project mentor and partner, articulated this way of thinking in a research retreat in 2010 when she suggested the discourse be constituted by the things themselves rather than by assumptions, theories and judgments: talking “not about the things but rather out of the things” (nicht über die Sachen, sondern von den Sachen her). When Bakhtin (1993) claims that an event “can be described only participatively” (p. 32) he is pointing to the necessity of being engaged in or “unindifferent” to the event itself. This creates a dilemma for researchers, when the expectation of the scientific community is to maintain objectivity, often manifested as indifference, in order for research to be acknowledged as legitimate and sound science. Yet, following Bakhtin’s argumentation, objectification will not lead to the insights of what it means to be a learner, to actually act as a learner:
… neither theoretical cognition nor aesthetic intuition can provide an approach to the once-occurrent real Being of an event, for there is no unity and no interpenetration between the content/sense (a product) and the act (an actual historical performance) in consequence of the essential and fundamental abstracting-from-myself qua participant in the course of establishing meaning and seeing. (p. 18)
The vignettes are a means for the researchers to participatively, i.e. engaged and unindifferently, capture empirically their own experience of the lived experiences of students being taught at school. In the first phase of the research project, a team of twelve researchers collected data during three field visits at 24 middle schools in the 2009/10 school year. At each school, the researcher focused on two children in a 5th grade class who were initially recommended for the study by their teachers based on their perceptions of difference in gender, ethnicity and achievement. In most cases, these students represented a challenge for the teacher. While present, researchers directed their attention to the events which occurred in the classrooms and attempted to sense particularly pathic moments for the children and capture these lived experiences in protocols (van Manen, 1990). These protocols then served as the basis for writing vignettes.
As a phenomenological text, the vignette is “a disclosure of the world, rests on itself, or rather provides its own foundation” (Merleau-Ponty, 1958, p. xxiii). The experience is the authority, and the “inchoative atmosphere” of phenomenology is part and parcel of phenomenology’s task to “reveal the mystery of the world and of reason” and “not to be taken as a sign of failure” (p. xxiv). Unavoidably, vignette research also entails an aesthetic sensibility. “The vignette should not tell, but rather show. In so doing, it becomes telling, disclosing an encapsulated moment which stands alone and in reference to itself” (Westfall-Greiter & Schwarz, 2012, p. 123). The context emerges from rather than predicating the experience and thereby restricting the reader’s experience to a particular theoretical framework.