Integrative Approaches to Communicative Interaction

A Project Summary

Integrative Approaches to Communicative Interaction

John Trueswell

University of Pennsylvania

We propose to establish an integrative educational foundation for students in eight different Penn graduate groups interested in empirically-based scientific studies of communicative interaction: Anthropology, Biology, Computer Science, Electrical Engineering, Linguistics, Neurology, Philosophy, and Psychology. This program will include two new common courses in mathematical foundations, a new jointly-managed program of summer research projects, a year of educational and research ``cross-training'' for each student, and a series of outside visitors linked to a journal club for IGERT students.

The new Mathematical Foundations courses will be a two-semester sequence, based on practical computer exercises drawn from real problems in the associated disciplines. The cross-training year will allow students to take courses and conduct a research project in a relevant area outside of their core discipline. The summer research program will enable students to learn a variety of perspectives and methods in other disciplines via practical experience. In addition to adding depth, the outside visitor series will provide grounding in relevant research areas not covered at Penn.

Students will learn to do research based on observations of natural behavior (as in ethology, corpus linguistics and clinical observation), as well as research in a laboratory setting (using both behavioral and physiological measures), and research based on algorithmic models of interacting agents. Students' common mathematical foundation will enable them to perform sophisticated analyses of signals, and to develop rigorous, testable models of communicative interaction, whether the object of study is a conversation among friends, a negotiation with a computer to obtain services, or a baboon barking bout.

Our goals are to enable students to do better research within the cooperating disciplines, and also to encourage the development of genuinely new types of cross-disciplinary research.

B Table of Contents (auto-generated in Fastlane)

C Project Description

The project description section contains the following items a through k. Page limitations specified for each item are inclusive of tables, figures, or other graphical data, and must be adhered to.

a)List of Participants

*Computer and Information Science

b)Vision, Goals, and Thematic Basis

We propose a new approach to integrative graduate education and research training on the topic of communicative interaction. Animals, humans and machines commonly interact so as to exchange information. The mechanisms of information exchange are often complex and vary exceedingly – across species, within the human species, and across the artificial systems that humans have invented. However, we believe that concepts and techniques developed for the study of one communications system can often be applied productively to another.

Based on our own cross-disciplinary experience, we feel that researchers in this area will benefit from access to a range of mathematical and computational modeling techniques, a body of substantive knowledge and experimental paradigms from biology, psychology and linguistics, and a set of overarching concepts such as information, learning and evolution. With this background, researchers can analyze and synthesize communicative signals, and explore the psychophysics and physiology of signal perception and production. They can monitor and model the flow of information between individuals and within groups. They can investigate the role of grammatical organization and logical interpretation of symbol sequences in mediating communicative interaction; and they can model the interplay among genetics, culture and individual experience in the development of communications capabilities, whether in the history of an individual, a population or a species.

We recognize that the various disciplines concerned with communicative interaction, at Penn and elsewhere, are quite different from one another, and will remain so. Researchers in these different areas cannot jettison their separate histories and goals, or their distinct sets of external connections – nor would we want them to. However, we strongly believe that greater integration of graduate education and research training across these disciplines, at least for a certain class of students, will lead to faster progress in the existing disciplinary frameworks, and also to the emergence of genuinely new types of research.

Specifically, we propose to establish an integrative educational foundation for students in eight different Penn graduate groups interested in empirically-based scientific studies of communicative interaction. This will include two new common courses in mathematical foundations, a new jointly-managed program of summer research experiences, a year of educational and research “cross-training” for each student, and a set of student-run activities, including a “journal club” with outside speakers, topical workshops, and an annual graduate student conference.

The new Mathematical Foundations courses will be a two-semester sequence, based on practical computer exercises connected to real problems in the associated disciplines. The cross-training year will allow students to take courses and do a research project in an area outside of their core discipline. The summer research program will enable students to learn a variety of perspectives and methods in other disciplines via practical experience.

Students will participate in research based on observations of natural behavior (as in ethology, corpus linguistics and clinical observation), as well as research in a laboratory setting (using both

behavioral and physiological measures), and research based on algorithmic models of interacting agents. Students' common mathematical foundation will enable them to perform sophisticated

analyses of signals, and to model the form and information flow of behavioral sequences, whether the object of study is a conversation among friends, a negotiation with a computer to obtain services, or a baboon barking bout.

c)Major Research Themes

Introduction

Training for empirically-based scientific study of communicative interaction now takes place in a large number of graduate programs at Penn, including Anthropology, Biology, Computer Science, Electrical Engineering, Linguistics, Neurology, Philosophy and Psychology. Relevant research areas include animal communication ([1], [2]), linguistic and non-linguistic communication among humans ([3], [4], [5], [6]), human-computer interaction ([7], [8], [9]), and human communication disorders ([10], [11]).

This distribution of research across disciplines is not unique to Penn, but rather reflects a world-wide pattern, developed over a period as long as the history of the disciplines themselves. In each discipline, researchers have focused on certain scientific or technical problems related to interactive communication, have selected certain aspects of the phenomena for intensive study, and have developed and applied methods and tools that are characteristic of their discipline but may be unfamiliar or inaccessible to outsiders. The boundaries between disciplines are not impermeable, but rather pass ideas and people to varying degrees, like other boundaries between cultural groups. However, despite these patterns of migration and trade, many local differences remain. There are especially large between-discipline differences in the degree and type of computational and mathematical training, the kinds of data deemed relevant, and the relationship between formal models and data.

Data in these disciplines may come from ethological field observation, ethnographic participant-observation, clinical or sociolinguistic interviews, introspection, corpora of transcribed conversations ([12], [13]), controlled behavioral or physiological experiments, or computer simulations. Noteworthy recent developments in experimental tools include eye-tracking machinery for observing attentional direction, and functional neuro-imaging techniques for observing the distribution of brain activity in time and space. Relevant mathematical and computational tools include quantitative analysis of recorded signals, synthesis of signals for perceptual experiments, inferential and exploratory statistics, models of communicating populations, and hand-built or statistically trained grammatical models ([14], [15]). Although these different methods are largely complementary rather than contradictory, researchers often find themselves adapting or re-inventing techniques that have been perfected in other disciplines. Psychologists find themselves doing signal processing, computer scientists wind up doing ethnological observations on task-oriented dialogues, linguists start experimenting with machine learning, neurologists begin doing discourse analysis, and so on. This long-standing cross-disciplinary borrowing reflects the fact that none of the traditional academic disciplines is a good overall fit to the study of communicative interaction. As a result, researchers at all levels struggle as outsiders to understand or re-develop concepts, methods and techniques whose natural home is elsewhere. The effort is needlessly difficult, and there is often an initial period of failure or low-quality progress. What is worse, some interesting strands of research are neglected, because they are cross-disciplinary in their conceptual foundations, not just in their methods.

Building on Penn’s strong research groups within the relevant disciplines, and on a well-established tradition of cooperation among them, we propose to establish a new integrative program for interested students of communicative interaction, across all the participating disciplines. Students in this program, whatever their home discipline, will get a shared mathematical foundation, a cross-disciplinary introduction to key concepts and methods, and participatory training in research across at least two of the cooperating disciplines. This program includes common courses in the mathematical foundations of communication, and a year in educational and research training outside the home discipline (See detailed training program proposal in section C.d below).

Penn is an ideal home for such a graduate training program, with internationally-renowned faculty in all the relevant disciplines, and an excellent record of turning out students who become intellectual leaders themselves. There are also strong intellectual and personal ties among the associated faculty across disciplinary boundaries, with a history of joint advising of students, joint research and publication, joint grant support, and joint development of courses at both undergraduate and graduate levels. The detailed discussion of major research themes will highlight some of these ties. However, this history of cooperation means that the proposed program of integrative graduate education is feasible, not that such a program already exists. For graduate students interested in communicative interaction, the necessary connections across departments and disciplines are now informal, incomplete and sometimes difficult to find and exploit. The purpose of this proposal is to create formal structures, including new courses and a formal system of laboratory cross-training, that will allow us to bring better integrative training to more students more efficiently.

This document has emerged from discussions among forty faculty members in nine departments at Penn, whose research relates to the general theme of communicative interaction, and who want to participate in the program of integrative graduate education and research training here proposed. In the following sections, we describe the relevant parts of the research programs of these forty individuals under five thematic headings in a total of about a dozen pages. As a result of this compression, many interesting and relevant research projects are necessarily neglected, while others are described only briefly. Our goal is to give a picture of the range of research projects connected to this proposal, with enough detail to enable the reader to judge the nature and quality of the work in each area, and with an emphasis on the human and intellectual connections that make the proposed training program both desirable and feasible.

The five thematic headings, and the key participating faculty in each, are:

1. Neurobiological and Field Study of Animal Communication: Cheney, Crawford, Liberman, Seyfarth, Schmidt, Schoenemann
2. Experimental and Clinical Study of Human Communication: Cheney, Clark, Chatterjee, Coslett, Embick, Farah, Gleitman, Grossman, Jha, Joshi, Jung-Beeman,Kearns, Labov, Liberman, Marcus, Sabini, Seyfarth, Trueswell, Thompson-Schill
3. Analysis and Modeling of Language Structure and Use: Bird, Clark, Embick, Joshi, Jung-Beeman, Kendon, Kearns, Kroch, Labov, Liberman, Marcus, Palmer, Pereira, Prince, Sankoff, Saul, Schoenemann, Seyfarth, Silverman, Thompson-Schill, Trueswell, Urban, Weinstein
4. Enhanced Communication Environments and Systems: Badler, Joshi, Kearns, Lee, Palmer, Pereira, Saul, Silverman
5. Fundamentals ooundations of Communicative Interaction: Production and , Perception, and Information Processing: Backus, Brainard, Crawford, Lee, Richards, Saul, Schmidt, Silverman, Sternberg, van der Spiegel.

c.1Neurobiological and Field Study of Animal Communication

IGERT-related research on animal communication occurs mainly in three laboratories. John Crawford studies the neural basis of electric and auditory communication in fish; Marc Schmidt studies the neural mechanisms that underlie song learning in birds; and Robert Seyfarth and Dorothy Cheney study vocal communication and social behavior in nonhuman primates. In addition to the phylogenetic diversity of their subjects, these three research groups use a variety of different techniques to study communication from a number of different perspectives. Nonetheless, they are united by a common approach to the naturalistic study of communication, and by intellectual links to IGERT colleagues in psycholinguistics, neuroscience, and computational biology.

All three groups are committed to an ethological approach that studies communication in its natural, social context wherever possible. In addition, all three investigators make explicit contact with research on human language. This commitment links their research with that of Gleitman, Trueswell, and Liberman, among others. Indeed, the simultaneous study of communication in animals and language in humans is one feature that will make our proposed IGERT group unique among those studying communicative interaction. Finally, all investigators (Crawford and Schmidt in particular) strive to incorporate quantitative, mathematical analyses and computational modeling into their research. The use of such mathematical techniques, now well established in computational linguistics and computer science, is a relatively new and creative addition to the study of animal communication. A focus on computation links Crawford’s and Schmidt’s research with that of Joshi and Marcus, among others, and with the Mathematical Foundations course that forms a central part of our training program. An important goal of IGERT training at Penn will be to produce a new generation of scientists in animal behavior, armed with the necessary mathematical tools to solve complex problems in the neural encoding of communicative interactions.

In Crawford’s laboratory, individuals of weakly electric Pollimyrus fish live in tanks where males defend territories, court females with sonic signals, and females communicate with electrical organ discharges (EODs). Complementing work in this naturalistic setting, Crawford uses operant behavioral methods to study sensory performance, and neurophysiological methods to study the mechanisms that underlie communication. This integration of rich ethological analysis with quantitative studies of perception and neurophysiology is unusual in any species. Three examples illustrate Crawford’s approach.

In their natural habitat, Pollimyrus males are nocturnal, communicating in pitch black water. Females appear to choose mates on the basis of their sounds (Crawford et al. 1997a, b). Under these circumstances, evolutionary theory predicts that male sound production will be energetically expensive (compared, for example, with female EODs); that the acoustic structure and/or rate of male sonic signals will correlate with a male’s physical condition; and that female auditory discrimination will be sensitive enough to distinguish among individual males. In one study, traditional ethological methods are used to test the first two hypotheses (Crawford et al. 1997a, b), and conditioning experiments to test the third. In conditioning experiments, animals respond to synthesized sounds with a burst of EODs, and are trained to change the EOD rate when they detect a sound or perceive a difference between two sounds. Results indicate that subjects’ audiograms closely match the energy in sounds produced by conspecifics (Marvit and Crawford 2000). The best sensitivity is close to 500 Hz, where there is also a prominent peak in the spectrum of conspecific sounds. Pollimyrus are also very sensitive to small differences in tone frequency and in the inter-click intervals of click trains (Marvit and Crawford 2001). These results have led to further work on the underlying computational mechanisms (see below).

How do fish distinguish small differences in auditory signals? In a second area of investigation, anatomical and neurophysiological methods are used to investigate the brain’s processing of auditory stimuli. Thus far, results indicate that the fish’s ear creates a neural code of the time structure of sounds (Fletcher & Crawford 2001). This encoded signal ascends from the ear into the medulla and on to the auditory midbrain, where the representation of acoustic information undergoes a major transformation (Crawford 1993, 1997a, b; Kozloski & Crawford 1998, 2000; Suzuki et al. in press). The transformation is revealed by the emergence of midbrain neurons that are highly selective, firing only in response to particular frequencies or to specific inter-click intervals. Thus, an initial temporal code is used to create a place-code for the period of repetitive sounds (for example, the period of a tone or the fundamental frequency of a complex sound).

How is one neural representation of an auditory signal transformed into another? One computational model (Sullivan 1982; Crawford, 1997a) assumes that temporally synchronized spikes are relayed from the medulla to the midbrain, where input from the medulla branches and excites both a selective and an inhibitory neuron. The inhibitory neuron then produces relatively long- lasting inhibition in the selective neuron followed by rebound depolarization. The inhibitory input acts as a temporal gate, with the preferred interval determined by the time between the initial EPSP and the rebound depolarization. In a third study, Large & Crawford (in press) explore this hypothetical circuit using a dynamic computational model for temporal feature extraction. This model fits the midbrain physiological data closely, and is particularly valuable for generating predictions for new experiments.