Short written introspection: a tool for validating listening comprehension tests

- demonstration of a micro-case

EALTA Conference 2005, Bergen

Joanna Anckar

1

Original Testing Context
Test of listening comprehension
•Administered as part of the Finnish Matriculation Examination (upper secondary school-leaving exam) of French as a Foreign Language
(in spring, 2002; the other parts: reading comprehension, grammar cloze, short essay)
•Level: approx. B1 (after 300-350 hours of instruction)
•45 minutes
•30 multiple-choice items /
3 or 4 options each
Validation study
Modified test: 17 multiple-choice items (sections I&III)+ three open-ended response items (section II)
•Participants: 226 students (mother tongue: Fin/Swe) about to take part in the Matriculation Exam
•Participants divided in quartiles according to success in sections I & II, max p. 17: Low >=6p, Intermediate I 7-8p, Intermediate II 9-11p, Advanced 12+p
Test procedure
Sections I & III:
•Participants listen to text in passages (30 –45 s), select 1 option among 3 after first listening during pause (35 s). After second listening allowed to change their choice of option, leave the original choice.
•At the second listening, participants are asked to motivate their choice of option  introspective responses

Triangulation of validity evidence

Relationship between text comprehension and answers to item & item quality
UNDERSTANDING OF TEXT

/ KEY OPTION
MISUNDERSTANDING OR LACK OF UNDERSTANDING of TEXT / DISTRACTER

= ITEM (probably) MEASURES TEXT COMPREHENSION

= ITEM MEASURES OTHER THINGS THAN/BESIDES TEXT COMPREHENSION

Text passage to items 5 &6

TRANSLATION:

‘Olivier and Martine are both 35 years old. He is a business executive and she is a computer consultant. They have bought an old denture laboratory in the 17th arrondissement. “ We come from southern France my husband and I, and this reminds us a bit of the countryside. Two windows face the street, the others a courtyard and a garden…but when I tried to sleep in a room at the street-side, it was too noisy, between the waste bins in the morning and the rounds of the police. The purchase has demanded big investments – and not only financial ones. I ended up feeling completely exhausted! I lost many kilos, but the worst thing was that my nerves broke down completely… But now, after the difficult beginning, everything seems better. The hardest times are behind us.”’

Item 5

•5. Que raconte Martine?
a) Les bruits de la rue ne la dérangent pas
b) Elle vient d’avoir des problèmes de santé
c) Elle veut repartir en province / •‘What is Martine telling?
a) The noises in the street do not disturb her’
b) She has just had some health problems
c) She wants to go back to the countryside

Item5:
Content analysis

Item 5 Quantitative analysis: selection of options per quartile participants

Item statistics

1

Item 5: Introspection
Types of responses

1

Key option 5b: Introspection

& examples of introspective responses

1

Distracter 5c: introspection

& examples of introspective responses

Item 5: conclusions

Correlation between items 5 and 6

item5 * item6 Crosstabulation

item6 / Total
.00 / 1.00
item5 / .00 / 70 / 39 / 109
1.00 / 69 / 47 / 116
Total / 139 / 86 / 225

Symmetric Measures

Value / Asymp. Std. Error(a) / Approx. T(b) / Approx. Sig.
Interval by Interval / Pearson's R / .049 / .067 / .728 / .467(c)
Ordinal by Ordinal / Spearman Correlation / .049 / .067 / .728 / .467(c)
N of Valid Cases / 225

a Not assuming the null hypothesis.b Using the asymptotic standard error assuming the null hypothesis.

c Based on normal approximation.

Table 1

Item 6

•6. Que dit-elle de leur appartement?
a) Il n’est pas trop cher
b) Il leur évoque des souvenirs
c) Il ne sera jamais prêt
• / ’What is she saying about their apartment?
a) It is not too expensive
b) It evokes some memories for them
c) It will never be ready’

Item 6:
Content analysis

Item 6 Quantitative analysis:
selection of options
per quartile participants


Item & item-total statistics

1

Item 6: Introspection
Types of responses

1

Key option 6b: introspection

& examples of introspective responses

Distracter 6a: introspection

& examples of introspective responses

Distracter 6c: introspection

& examples of introspective responses

1

Conclusive comments on item 6

Conclusions & speculations...

1