Short Written Introspection: a Tool for Validating Listening Comprehension Tests
Short written introspection: a tool for validating listening comprehension tests
- demonstration of a micro-case
EALTA Conference 2005, Bergen
Joanna Anckar
1
Original Testing ContextTest of listening comprehension
•Administered as part of the Finnish Matriculation Examination (upper secondary school-leaving exam) of French as a Foreign Language
(in spring, 2002; the other parts: reading comprehension, grammar cloze, short essay)
•Level: approx. B1 (after 300-350 hours of instruction)
•45 minutes
•30 multiple-choice items /
3 or 4 options each
Validation study
Modified test: 17 multiple-choice items (sections I&III)+ three open-ended response items (section II)
•Participants: 226 students (mother tongue: Fin/Swe) about to take part in the Matriculation Exam
•Participants divided in quartiles according to success in sections I & II, max p. 17: Low >=6p, Intermediate I 7-8p, Intermediate II 9-11p, Advanced 12+p
Test procedure
Sections I & III:
•Participants listen to text in passages (30 –45 s), select 1 option among 3 after first listening during pause (35 s). After second listening allowed to change their choice of option, leave the original choice.
•At the second listening, participants are asked to motivate their choice of option introspective responses
Triangulation of validity evidence
Relationship between text comprehension and answers to item & item quality
UNDERSTANDING OF TEXT/ KEY OPTION
MISUNDERSTANDING OR LACK OF UNDERSTANDING of TEXT / DISTRACTER
= ITEM (probably) MEASURES TEXT COMPREHENSION
= ITEM MEASURES OTHER THINGS THAN/BESIDES TEXT COMPREHENSION
Text passage to items 5 &6
TRANSLATION:
‘Olivier and Martine are both 35 years old. He is a business executive and she is a computer consultant. They have bought an old denture laboratory in the 17th arrondissement. “ We come from southern France my husband and I, and this reminds us a bit of the countryside. Two windows face the street, the others a courtyard and a garden…but when I tried to sleep in a room at the street-side, it was too noisy, between the waste bins in the morning and the rounds of the police. The purchase has demanded big investments – and not only financial ones. I ended up feeling completely exhausted! I lost many kilos, but the worst thing was that my nerves broke down completely… But now, after the difficult beginning, everything seems better. The hardest times are behind us.”’
Item 5
•5. Que raconte Martine?a) Les bruits de la rue ne la dérangent pas
b) Elle vient d’avoir des problèmes de santé
c) Elle veut repartir en province / •‘What is Martine telling?
a) The noises in the street do not disturb her’
b) She has just had some health problems
c) She wants to go back to the countryside
Item5:
Content analysis
Item 5 Quantitative analysis: selection of options per quartile participants
Item statistics
1
Item 5: Introspection
Types of responses
1
Key option 5b: Introspection
& examples of introspective responses
1
Distracter 5c: introspection
& examples of introspective responses
Item 5: conclusions
Correlation between items 5 and 6
item5 * item6 Crosstabulation
item6 / Total.00 / 1.00
item5 / .00 / 70 / 39 / 109
1.00 / 69 / 47 / 116
Total / 139 / 86 / 225
Symmetric Measures
Value / Asymp. Std. Error(a) / Approx. T(b) / Approx. Sig.Interval by Interval / Pearson's R / .049 / .067 / .728 / .467(c)
Ordinal by Ordinal / Spearman Correlation / .049 / .067 / .728 / .467(c)
N of Valid Cases / 225
a Not assuming the null hypothesis.b Using the asymptotic standard error assuming the null hypothesis.
c Based on normal approximation.
Table 1
Item 6
•6. Que dit-elle de leur appartement?a) Il n’est pas trop cher
b) Il leur évoque des souvenirs
c) Il ne sera jamais prêt
• / ’What is she saying about their apartment?
a) It is not too expensive
b) It evokes some memories for them
c) It will never be ready’
Item 6:
Content analysis
Item 6 Quantitative analysis:
selection of options
per quartile participants
Item & item-total statistics
1
Item 6: Introspection
Types of responses
1
Key option 6b: introspection
& examples of introspective responses
Distracter 6a: introspection
& examples of introspective responses
Distracter 6c: introspection
& examples of introspective responses
1
Conclusive comments on item 6
Conclusions & speculations...
1