Carolyn Penstein Rosé

Carolyn Penstein Rosé

US Citizen

Language Technologies Institute/Human-Computer Interaction Institute

Newell Simon Hall 4531

CarnegieMellonUniversity

Pittsburgh, PA 15213

E-mail:

Homepage:

Phone: (412) 268-3170

Fax: (412) 268-6298

Last Updated: February 15, 2005

0. General Information

Education

Ph.D., Language and Information Technologies, CarnegieMellonUniversity, December 1997.

Thesis advisor: Lori S. Levin

M.S., Computational Linguistics, CarnegieMellonUniversity, May, 1994.

B.S., Information and Computer Science (Magna Cum Laude), University of California at Irvine, June 1992.

Position

[2003-present] Research Computer Scientist, Language Technologies Institute and Human-Computer Interaction Institute, School of Computer Science, CarnegieMellonUniversity

[1997- 2003] Research Associate, Learning Research and DevelopmentCenter, University of Pittsburgh.

Project coordinator in Natural Language Tutoring Group

[1994-1997] Teaching Assistant, Computational Linguistics Program, CarnegieMellonUniversity.

[Summer 1993] Summer Research Internship, Apple Computer, San José, CA.

[1992-1994] Research Assistant, Center for Machine Translation, CarnegieMellonUniversity.

[Summer 1991] Research Internship, Minority Summer Research Internship Program, UC Irvine.

[1990-1992 ] Honors Research, University of California at Irvine.

I. Statement of Career Goals

Overview

My primary research objective is to develop and apply advanced interactive technology to enable effective computer based and computer supported instruction. A particular focus of my research is in exploring the role of explanation and language communication in learning. Thus, one major thrust of my research is in developing and applying language technology to the problem of eliciting, responding to, and automatically analyzing student verbal behavior. However, many of the underlying HCI issues, such as influencing student expectations, motivation, and learning orientation, transcend the specific input modality. This research program involves four primary foci: (1) controlled experimentation and analysis of human tutoring, collaborative learning, and computer tutoring to explore the mechanisms by which effective instruction is accomplished, (2) controlled experimentation and analysis of student interactions with human tutors, peer learners, and computer tutors in order to explore the HCI issues that affect student behavior and motivational orientations, (3) basic research in language technology to enable, facilitate, or study interactions in natural language in learning environments either with computer agents, between humans, or a combination of the two, and finally (4) development of easy-to-use tools for building language interaction interfaces and tutorial environments more generally.

A Historical Perspective

Although my long term goal was always to work in the area of tutorial dialogue, during graduate school I focused on the problem of robust natural language interpretation. I was awarded my Ph.D. in 1997 from the Language Technologies Institute here at CarnegieMellonUniversity. My dissertation research focused on an approach for recovering from interpretation failures resulting from insufficient knowledge source coverage and extragrammatical language (Rosé, 1997; Rosé & Levin, 1998; Rosé, 1999). I always had an affinity for hybrid knowledge based/machine learning approaches (Rosé & Waibel, 1997; Rosé & Lavie, 1997). This work was conducted in the context of a multi-lingual speech-to-speech machine translation project (Woszczyna et al., 1993; Suhm et al., 1994). That context provided a challenging environment in which to explore issues related to robust and efficient natural language understanding. Another focus of my work was on computational modeling of dialogue (Rosé et al., 1995; Qu et al., 1997).

Immediately upon finishing my dissertation research, I accepted a position as a Postdoctoral Research Associate at the Learning Research and Development Center (LRDC), where I worked most closely with Johanna Moore, Kurt VanLehn, and Diane Litman. There I played a very active role in the CIRCLECenter, an NSF funded center pursuing research questions related to the development of tutorial dialogue technology (Rosé et al., 1999; Freedman et al., 2000; Jordan et al., 2001; Vanlehn et al., 2002). An important part of this work was continued research in the area robust language understanding (Rosé, 2000; Rosé & Lavie, 2001; Rosé et al., 2002; Rosé et al., 2003a; Rosé & Hall, 2004; Lavie & Rosé, 2004), in addition to research involving analysis of human tutoring interactions (Rosé et al., 2001b; Rosé et al., 2003b,c; Litman et al., 2004; VanLehn et al., submitted) and evaluation of implemented tutorial dialogue systems (Rosé et al., 2001a; Litman et al., 2004). A recently accepted journal article (Rosé & VanLehn, to appear) and a recently submitted journal article (Rosé, Siler, Torrey, & VanLehn, in preparation) provide an overview of my work from those years at LRDC leading into the work I am doing now.

My Current Work

In October of 2003 I accepted a position as a Research Scientist with a 50%/50% joint appointment in the Language Technologies Institute and the Human-Computer Interaction Institute. The series of studies I ran during my time at LRDC convinced me of the naiveté of assuming that the most important problem in providing effective tutorial dialogue technology was in overcoming the technical challenges. Since coming to CMU I have shifted my emphasis very sharply towards design based on detailed, empirically constructed models of the multiplicity of underlying mechanisms that are at work at the level of the individual student in the midst of human-human interaction. Here I discuss a few selected recent findings and results. A comprehensive list of my other funded projects is found in Section V.

The ONR funded CycleTalk project (PI, Carolyn Rosé, CoPI, Vincent Aleven, HCII) addresses all four of my primary research foci, with an emphasis on exploring the impact of a specific tutorial dialogue strategy, which we refer to as Negotiable Problem Solving Goals (NPSG). NPSG is a tutorial-dialogue based approach to guided exploratory learning involving problem solving goals that are negotiated between tutor and student rather than dictated by the tutor or freely chosen by the student. The NPSG approach to guided exploratory learning is located on a previously untested space on what we call The Exploratory Learning Continuum. In a recent study (Rosé et al., submitted-d), we empirically evaluated the effectiveness of NPSG in relation to two alternative approaches combining tutorial based learning and problem solving, which is the state-of-the-art approach in computer based instruction. The empirical results from this classroom evaluation provide evidence in favor of tutorial dialogue support in exploratory learning environments and provide design recommendations for a new type of tutorial dialogue system based on the NPSG model. In our current work, we are conducting a detailed analysis of the corpus of tutoring dialogues collected as part of that study in order to form more specific hypotheses about the mechanisms at work in the NPSG strategy that account for its effectiveness.

Under the umbrella of the NSF/IERI funded Learning Oriented Dialogue Project (PI, Vincent Aleven, CoPIs Albert Corbett and Carolyn Rosé), I am investigating how characteristics of a learning companion agent influence the behavior and learning of their partner. The rationale for learning companion technology grows out of the successful track record for the collaborative learning paradigm. However, what is known about the mechanisms responsible for its success is largely at the group level rather than at the individual level. And well controlled studies comparing learning across collaborative and non-collaborative settings have been few or non-existent. Even studies presenting evidence about specific effective patterns of interaction have largely provided correlational evidence, and thus do not offer insights on the causal mechanisms at work on the level of the individual student.

My work on this project is focused on investigating previous claims about best practices in learning companion design that have not been subjected to rigorous evaluation. As a key part of this, I am advocating a particular experimental design methodology, which provides a highly controlled way to examine mechanisms by which one peer learner’s behavior influences a partner learner’s behavior and learning. Specifically, it makes use of confederate peer learners who are experimenters acting as peer learners but behaving in a highly scripted way. While this approach lacks the high degree of external validity found in more naturalistic observations of collaborative learning interactions, it provides complementary insights not possible within that framework. The type of insights provided by this type of design are essential for discovering precisely which combination of technological features will ultimately yield the most desirable response from students. By using a controlled experimental approach, we can get specific information about which aspects of the rich interactions are important for achieving the target effect. By using naturalistic collaborative learning and solitary learning as control conditions (Rosé et al., submitted-a) we can measure the extent to which the collaboration provides value as well as how the manipulated collaboration compares in effectiveness to more naturalistic collaborative learning.

My first application of the methodology discussed above has been in the context of the NSF funded Calculategy project (PI, Carolyn Rosé, in collaboration with the EU funded LeActiveMath project headed up by Erica Mellis at DFKI, Saarbrueken). Recent work in the Calculategy project has focused on the instructional value of errors in peer tutoring scenarios. Rosé et al., (submitted-a) presents a controlled investigation of this issue. The empirical investigations reported in this paper were designed with the intention of addressing the following questions: (1) Under what circumstances do the errors that arise during collaborative problem solving interactions have a harmful (or helpful) effect on student learning? (2) How does the accuracy and the level of initiative taking of a peer learner’s contributions affect initiative taking in their partner? (3) What is the relative value of incorrectly worked examples in comparison with correctly worked examples? We did not find evidence that the errors contributed by the confederate peer learners were harmful to student subjects working with them except in the case of students paired with peer learners who contributed with a very high frequency. On the contrary, we found a small but reliable interaction effect in which students paired with peer learners contributing with a low frequency derived some benefit from the errors they were exposed to. An understanding of where errors can be used strategically to stimulate cognitive conflict and student learning may enhance the effectiveness of existing well-established approaches to scaffolding in intelligent tutoring systems. Nevertheless, this is an issue that requires more investigation. Because the majority of the observed learning in this study is explained by correct problem solving, these results do not argue that errors play a large role in student learning relative to correct examples. The weakness of this effect might be explained by a paucity of instructional explanation and help seeking behaviors found in our corpus of collaborative problem solving interactions. We plan to do more investigations along these lines and to use the results to eventually inform the design of a new peer collaborative agent.

One important piece of my work that bridges both the LTI and HCII has been the PSLC funded TagHelper project, the goal of which is to develop and use language technology to support behavioral research (Donmez et al., to appear; Donmez et al., submitted; Rosé et al., submitted-b; Rosé et al., submitted-e). A key focus in this work has been developing techniques for exploiting natural structure in corpora and coding schemes in order to overcome the sparse data problem. This project has afforded me the opportunity to collaborate with technology researchers such as Jaime Carbonell and William Cohen as well as local behavioral researchers such as Bob Kraut and Kenneth Koedinger and especially learning scientists abroad such as Alexander Renkl and his group in Freiburg, Rainer Bromme and Regina Jucks in Muenster, Karen Schweitzer in Heidelberg, Manuela Paechter in Graz, and Frank Fischer and his group in Tuebingen. Pursuing this work is one of my major roles within the interdisciplinary Pittsburgh Sciences of Learning Center.

An important recent result on the TagHelper project has been the development of two techniques that together allow us to train classifiers that achieve human levels of agreement (.7 Kappa or better) on 5 out of 7 dimensions of a multi-dimensional coding scheme developed for a detailed process analysis of the collaborative learning process in a high profile collaborative learning project headed up by Frank Fischer at the KMRC in Tuebingen (Donmez et al., submitted). By taking a more conservative approach of only committing a code on utterances where the classifier is most confident, we can achieve human levels of agreement over 80% of the corpus for one of the remaining dimensions and 96% of the corpus for the other. Beyond the tremendous savings of time these trained classifiers can offer on the analysis of data from further collaborative learning studies within this group, we are jointly working with them on a new type of instructional intervention enabled by this classification technology where collaboration scripts that support the interaction between students are dynamically adapted to the needs of groups based on patterns detected by the classifiers.

My Role in LTI and HCII

I see my role as a bridge between the departments as encompassing more than my research. As part of my mission to bring these two communities into closer contact, I have designed a Conversational Interfaces course ( which was cross-listed in LTI and HCII for Fall of 2004. The twofold goal of this course was to explore the literature addressing the questions raised above as well as to get students from the two departments to work together on projects involving a combination of language technology development on the one side, and design and usability testing on the other side. While the number of students officially taking the course for credit or simply informally participating was small, and while it was a disappointment to me that only LTI students signed up for the course, it was exciting to me to see how the students gained a strong appreciation for the field of HCI and began to think deeply about how these issues related to their own research. Two students from that course decided to continue to pursue these issues with me in an independent study the following semester, which in both cases has lead to a conference submission (Banerjee, Rosé, & Rudnicky, submitted; Rosé et al., submitted-a).

Another forum for building bridges across disciplines is through informal reading and discussion groups. For this reason, another of my ongoing efforts to bring the two departments together has been facilitating the Collaborative Learning Reading and Discussion Group ( Although participation in that group has fluctuated from time to time, it has at times brought together students and sometimes faculty from both departments for discussions of some topics on the frontier between the two communities, such as on-line assessment based on patterns in collaborative discourse or text classification technology for automatically characterizing collaborative learning interactions using a multi-dimensional coding scheme.

Teaching Involvement

Working with students is what I love most about my job here as a Research Scientist. It has been quite a growing experience for me. It’s what I take the most pleasure and the most pride in, and what I most want to learn how to do better. I am pleased to have had the opportunity to advise 2 LTI PhD students (one who has since moved on to a new advisor) and one LTI Master’s student, to have supervised four independent study projects, and to be working with four different students who are working on their proposals with plans to serve on their committees. I have also enjoyed co-teaching courses in both LTI and HCII as well as designing and teaching Conversational Interfaces on my own. I’m looking forward to more teaching opportunities. Currently I am informally working on designing another new course, which will be an undergraduate course in evaluating educational technology to be taught in the Heinz school as part of their educational technology coordinator program. I have also spoken to Bob Kraut about a desire I have to co-design and co-teach an HCII course with him on group dynamics examined through the lenses of collaborative learning and collaborative work. At some later review I hope to be considered for switching to the Tenure track.

II. Publications List

Chapters in Books:

A. Lavie and C. P. Rosé, 2004, Optimal Ambiguity Packing in Context-Free Parsers with Interleaved Unification. In H. Bunt, J. Carroll and G. Satta (eds.), Current Issues in Parsing Technologies, Kluwer Academic Press. (24% of submissions accepted for book publication)

C. P. Rosé and A. Lavie (2001). Balancing Robustness and Efficiency in Unification-augmented Context-Free Parsers for Large Practical Applications. In van Noord and Junqua (Eds.), Robustness in Language and Speech Technology, ELSNET series, Kluwer Academic Press.

C. P. Rosé (1999). A Genetic Programming Approach for Robust Language Interpretation, in L. Spencer et al. (eds.) Advances in Genetic Programming, Volume 3.

C. P. Rosé and A. H. Waibel, 1997, Recovering from Parser Failures: A Hybrid Statistical/Symbolic Approach, in J. Klavans and P. Resnik (eds.), The Balancing Act: Combining Symbolic and

Statistical Approaches to Language Processing, MIT Press. (less than 20% of submissions accepted for book publication)

Y. Qu, B. DiEugenio, A. Lavie, L Levin, and C. P. Rosé (1997). Minimizing Cumulative Error in Discourse Context, In E. Maier, M. Mast and S. LuperFoy (eds.), Dialogue Processing in Spoken Language Systems: Revised Papers from ECAI-96 Workshop, LNCS series, Springer Verlag.

Refereed Journal Papers:

Rosé C. P., & VanLehn, K. (to appear). An Evaluation of a Hybrid Language Understanding Approach for Robust Selection of Tutoring Goals, International Journal of AI in Education

A. C. Graesser, K. VanLehn, C. P. Rosé, P. W. Jordan, and D. Harter, 2001, Intelligent Tutoring Systems with Conversational Dialogue, AI Magazine, Special Issue on Intelligent User Interfaces, Volume 2, Number 4.

Journal Papers in Preparation, Submitted, or in Revision:

Rosé C. P., Siler, S., Torrey, C. & VanLehn, K. (in preparation).Exploring the Effectiveness of Knowledge Construction Dialogues to Support Conceptual Understanding, very soon to be submitted to the International Journal of AI in Education

Rosé, C. P. & Aleven, V. (in preparation). Exploring to Learn and Learning to Explore: Tutorial Dialogue Support in an Exploratory Design Environment, to be submitted to Cognition and Instruction.

Rosé, C. P., Donmez, P., & Cohen, W. (in preparation). TagHelper: Automatic and Semi-Automatic Content Analysis of Corpus Data Applied to Problems inOn-Line Assessment and Educational Research Support, to be submitted to the Journal of Natural Language Engineering special issue on educational applications of natural language processing.

VanLehn, K., Graesser, A., Jackson, G. T., Jordan, P., Olney, A., Rosé, C. P., (in revision). Natural Language Tutoring: A comparison of human tutors, computer tutors, and text. Submitted to the Cognitive Science Journal.

Conference Papers Currently Under Review