Multimodal dynamics in interpreter-mediated dialogue
Recent years have witnessed a growing interest in multimodal features of spoken language (Müller et al. 2013), that crucially involve the interaction of the auditory (speech), visual gaze) and kinesthetic (gesture, posture) modalities in construing and communicating meaning.Especially eye-gaze patterns in (semi-)naturalistic settings are receiving increasing attention in multimodal interaction research.
The present studypursues this recent line of research by inquiring into the role of eye-gaze in correlation with other (non-)verbal signals in the specific institutionalized setting of interpreter-mediated dialogue.Studies have shown that synchronization and turn transfer in interpreted dialogues are collaboratively coordinated through verbal and non-verbal means (Wadensjö 1998, Bot 2005, Mason 2012). Especially gaze direction has an important function in signaling conversational attention and facilitating turn takingin interpreted dialogues (Lang 1978, Wadensjö 2001, Bot 2005, Mason 2012).
In this study we aim at an in-depth analysis of the gaze patterns of the participants in interpreted dialogues in relation tospeech, gesture and posture and how they function in the constitution of dialogue management. We collected a corpus of semi-authentic interpreter-mediated interactions by using mobileeye-tracking technology (Gullberg & Kita 2009, Jokinen 2010, BrôneOben 2013, ObenBrône fc.). The data annotation and statistical processing of the gaze data was largely based on the workflow developed in Brône et al. (2013) for face-to-face dialogues.The analysis of gaze patterns is indicative of interpreter’s regulatory role in this type of dialogue. Moreover, the studyprovides insights in the processes of turn-taking, footing (e.g. speaker selection) information management and grounding (or the process of establishing common ground) in interpreted interaction.
REFERENCES
Bot, H. 2005.Dialogue Interpreting in Mental Health, Amsterdam and New York: Rodopi Publishers.
Brône, G. & Oben, B. 2013.InSight Interaction.A multimodal and multifocal dialogue corpus.
Language Resources and Evaluation.
Gullberg, M. & Kita, S. 2009. Attention to speech-accompanying gestures: Eye movements and
information uptake. Journal of Nonverbal Behaviour33: 251-277.
Jokinen, K. 2010. Non-verbal signals for turn-taking & feedback. Proc. of 7th Int. Conf. on Language
Resources & Evaluation (LREC) International Universal Communication Symposium.
Lang, R. 1978. Behavioral aspects of liaison interpreters in Papua New Guinea: some preliminary
observations. In: Gerver, D. & Sinaiko, H. W. (eds.) Language Interpretation and
Communication.New Yotk/London: Plenum Press, 231-244.
Mason, I. 2012. Gaze, positioning and identity in interpreter-mediated dialogues. . In: Baraladi, L.,
Gavioli, C. (eds.) Coordinating Participation in Dialogue Interpreting. Amsterdam/Philadelphia:
Benjamins Publishing, 177-199.
Müller, C. Fricke, E., Cienki, A. & McNeill, D. (Eds.) 2013.Body: Language & Communication. Berlin:
De Gruyter.
Oben, B. & Brône, G. forthc. What you see is what you do. On the relationship between gaze and
gesture in multimodal alignment. Language & Cognition.
Wadensjö, C. 1998. Interpreting as interaction.London and New York: Longman.