Report of Teagle Grant Work 2006-2007
MuhlenbergCollege
July 2007
Project
Muhlenberg is studying the role of capstone experiences and student-faculty research in fostering academic growth and intentional learning, specifically in the senior year. Currently, capstone experiences and opportunities for independent research/study are available for most students but are not a requirement across the curriculum. Our project involves the collection and analysis of student work, survey results and focus group/interview data to make comparisons between students who had these experiences and those who did not. Members of the Teagle committee are Jim Bloom, Kathleen Harring, Trevor Knox, Chris Sistare, Bruce Wightman, and Carol Wilson.
Intentional Learning Rubric
At the October 2006 workshop hosted by Muhlenberg, participants from the five institutions developed a list of fourteen intentional learning outcomes and translated those outcomes into an assessment rubric. Dr. Mary Allen, the consultant retained by the consortium, created a second summary rubric that integrated the fourteen outcomes into six dimensions. Given that these outcomes did not focus on the critical evaluation of ideas or the creation of new knowledge – outcomes that are developed and mastered in capstone experiences and student/faculty research, the Muhlenberg team added two additional outcomes to this integrated rubric (Critical Evaluation and the Creation of New Knowledge). (see Appendix A.)
Essay Prompt Development
Our work during the Fall semester focused on pilot-testing the intentional learning rubric developed at the October workshop. Students in two capstone courses (Advanced Lab in Social/ Personality Psychology and Molecular Biology) wrote essays describing themselves as one of three types of learners and provided evidence to support their classification (see Appendix B). Members of the Teagle committee participated in a grading session to apply our intentional learning rubric to the essays. The results of the session made us realize that our essay prompt did not directly address the outcomes as defined in the rubric. Few student essays tended to single out specific educational experiences, tending instead to parrot the language used in the prompt.Consequently, we developed a new essay prompt (see Appendix C) that directly asked students to describe experiences during their four years at Muhlenberg that 1) led them to view problems and ideas from multiple perspectives, 2) led to connections among different disciplines, 3) led them to an epiphany or to create new knowledge or a new viewpoint, and 4) helped them to critically evaluate claims and ideas.
Data Collection
During the spring 2007 semester we collected data from seniors enrolled in capstone courses including senior seminars taken that semester.Department chairs identified the faculty who taught these courses, and we invited them to participate in the project by assigning the essay to their students and participating in a rubric grading session.
Ten faculty from seven departments and two programs (Anthropology/Sociology, Art, Business, English, Media & Communications, Neuroscience, Philosophy, Psychology, RJ Honors Program) agreed to work with us on the project. We set the following time-table for distribution and collection of the essays:
1) Essay prompt assigned tostudents (week of March 26)
2) Essays collected (week of April 2)
3) Rubric grading session with Teagle team and the ten capstone instructors (week of April 16)
Ninety students completed essays,resulting in a 65% response rate. We randomly sampled essays from each course, giving us a total of 50 essays. Essays were graded based on four learning outcomes (Multiple Perspectives, Makes Connections, Create New Knowledge and Critically Evaluate Knowledge). Each essay was graded by two instructors with members of the Teagle group determining the final grade when graders disagreed. Faculty participants received a $100 stipend and student participants received a nominal gift ($5 coffee card).
The results from the grading session show that,depending on the outcome, 52-64% of the essays were judged as proficient or advanced (e.g., Multiple Perspectives = 64%, Makes Connections = 52%, Create New Knowledge = 60%, Critically Evaluate Knowledge = 55%). Student performance was stronger for the two outcomes that reflected more experiential learning (e.g., Multiple Perspectives and Create New Knowledge), but they had a more difficult time singling out the particular connections they made between disciplines or opportunities for critically evaluation of ideas and information. However, assessing students’ abilities to provide evidence of experiences that relate to each learning outcome does not necessarily mean that the student has (or has not) actually achieved the learning outcome (See Appendix D). While inter-rater reliability was low for exact agreement on a classification (39%), most judgments were one category away or in exact agreement (87%).
We also reviewed the essays to determine the types of educational experiences that students described (see Appendix E). Most students highlighted specific courses that led to the achievement of our learning outcomes, although internships, double majoring in two disciplines, and community service were frequently mentioned. We attribute these results to the way Muhlenberg’s curriculum views learning as mostly a matter of choosing between two kinds of courses: those that fulfill general education and those that meet major requirements.
NSSE Results
At the October workshop, participants identified specific items on the NSSE that indirectly assessed the outcomes we identified as related to intentional learning. We analyzed Muhlenberg’s 2005 NSSE data to test for differences in responses between seniors who reported having had a capstone experience and those who did not report having had a capstone. The results are in Appendix D. Students who reported having had a capstone experience scored higher on many of our intentional learning items. However, what was most noteworthy was that only 65 (36%) ofthe respondents reported having such an experience.In contrast, at least 80% of majors and interdisciplinary programs at Muhlenberg provide a required or optional capstone experience. This disparity between student and faculty perceptions may indicate a need for more intentional articulation of program objectives and learning outcomes by faculty.
Assessment Conferences
Several members of the Teagle committee attended the HEDS Assessment forum in January. We attended sessions on using the Collegiate Learning Assessment (CLA), assessing Senior Theses, Information Literacy assessment and other topics relevant to our work on intentional learning, in addition to the keynote addresses that outlined the Wabash National Study of Liberal Arts Education and the Mellon Assessment Project at HamiltonCollege. The Muhlenberg participants also attended a workshop on creating and using rubrics,giving us more training with assessment rubrics.
Kathy Harring was invited to attend the Hamilton College Mellon Assessment Conference in April. In her presentation at the conference, she outlined Muhlenberg’s Teagle project, shared results from the planning grant focus groups, and described the intentional learning rubric developed by the Teagle collaborators.
2007-2008 Plan
During the Fall 2007 semester, we will collect intentional learning essays from seniors who have not had capstone courses. Our capstone instructors will be convened again to grade these essays using our rubric. In addition, we will be developing interview questions and recruiting and training student interviewers who will be conducting interviews with students who have completed capstone seminars. Interviews will be conducted early in the spring semester.
During the Spring 2008, semester students who had independent research/study experiences will be recruited to write intentional learning essays. We will also analyze our 2005 NSSE data for differences in the intentional learning items as a function of students’ independent research/study experiences.
Appendix A
Intentional Learning Rubric
Learning Outcome / Below BasicBB / Basic
B / Proficient
P / Advanced
A
Multiple Perspectives / Has trouble identifying and/or coping with multiple perspectives. / Demonstrates limited understanding or respect for others’ perspectives or does not systematically analyze their relative merit. / Recognizes the value of considering multiple perspectives. Analyzes differences among perspectives and how they relate to specific phenomena. / Demonstrates sensitivity to others’ perspectives, evaluates the validity of conflicting perspectives, and recognizes subtle differences and their implications.
Makes Connections / Has trouble identifying valid connections between concepts or links across contexts. / Identifies obvious connections between concepts and links across contexts. / Goes beyond the obvious when identifying connections between concepts or links across contexts. / Identifies subtle, sophisticated, and/or creative connections or links and develops insights based on these observations.
Can create new knowledge,
theories, and representations. / Rarely attempts to develop ideas or draw inferences from observations.
Treats all knowledge as static, rather than created. / Often induces general ideas from particular ideas or observations, but cannot explain them well.
Recognizes that knowledge changes. / Able to develop ideas that are new to the student and can explain them to others.
Participates in the process of developing new knowledge. / Able to develop ideas that are new to the world and can explain them to others.
Routinely and actively pursues new knowledge and insight.
Can critically
evaluate knowledge, theories, and representations / Can recognize discrete elements of knowledge, theories, and representations. / Can recognize and synthesize
discrete elements of knowledge, theories, and representations. / Can recognize and synthesize discrete elements of knowledge, theories, and representations.
Can evaluate the effectiveness and coherence of these elements. / Can recognize and synthesize discrete elements of knowledge, theories and representations.
Can evaluate the effectiveness and coherence of these elements in original ways.
Appendix B
Essay on Types of Learners
The following paragraphs describe three types of learners. Please read each one carefully and think about which one is more consistent with the way that you approach your college learning.
1) I tend to choose courses based primarily on how they relate to my current career goals and whether they fulfill major/minor or college requirements. My primary goal is to take the courses I need to fulfill the requirements for graduating with a college degree. While I often enjoy the courses I take, they usually don’t seem to relate to each other in any way I can discern.
2) I tend to choose courses based primarily on how they relate to my current career goals and whether they fulfill major/minor or college requirements. My primary goal is to take the courses I need to fulfill the requirements for graduating with a college degree. I liked many of the courses I took and found the subject matter interesting. I found connections among many of the courses in my major and the courses I took to satisfy college requirements.
3) I tend to choose courses to fulfill my major/minor and college requirements, but I try to find ones that look like they may connect with other courses that I have taken. I always make an effort to integrate and connect the skills and knowledge from different courses and intellectual experiences.
We would like you to write a 300-500 word essay that answers the following questions:
Which of these descriptions best fits you now?
Have you always taken this approach to your learning?
In what ways have you changed as a learner during your college experience?
Give a specific example of an academic experience that you had that represents the kind of learner you consider yourself to be.
Appendix C
ESSAY PROMPT
As a senior, you now have the opportunity to reflect on your learning experiences at MuhlenbergCollege. Consider your academic experiences at Muhlenberg, including traditional courses and non-classroom projects, such as research, internship, studio work, performance, and volunteer work. What experiences encouraged you to view problems and ideas from multiple perspectives? What experiences led to connections among different disciplines? What experiences led you to a new epiphany or to create new knowledge or a new viewpoint? What experiences helped you to critically evaluate claims and ideas? For each prompt, explain and illustrate how such discoveries and revelations changed the way you learn and think.
Please write a 300 – 500 word typed essay addressing these questions and providing specific examples of your experiences.
Your essay is due in class one week from today.
Do not put your name on the essay. However, please indicate the course number for your class in the left-hand corner and your class year (e.g. Junior, Senior).
Appendix D
Results of TeagleEssay Grading
May 2007
Learning Outcome / Below BasicBB / Basic
B / Proficient
P / Advanced
A
Multiple Perspectives / 10% / 26% / 52% / 12%
Makes Connections / 8% / 40% / 34% / 18%
Can create new knowledge,
theories, and representations. / 10% / 29% / 54% / 6%
Can critically
evaluate knowledge, theories, and representations / 10% / 35% / 51% / 4%
N = 50
Appendix E
Types of Experiences by Learning Outcome
Teagle Capstone Student Essays
May 2007
Awareness of Multiple Perspectives
Experience / FrequencyCourses / 13
Internship / 7
Liberal arts education / 4
Double major / 3
Interdisciplinary major / 2
Learning assistant position / 2
Research/Senior project/Thesis / 2
Presidential assistant position / 1
Required class reading / 1
Interdisciplinary minor / 1
Extracurricular activities / 1
Makes Connections Between Disciplines
Experience / FrequencyTaking courses in fields that overlapped / 16
Double major / 9
Liberal art education / 4
Research/Senior project/Thesis / 4
Senior seminar / 3
Extracurricular activities / 1
On-campus job / 1
Internship / 1
Connections within one major / 1
Connections within one minor / 1
Community service / 1
Study abroad experience / 1
Dramaturge for a production in the theatre department / 1
Can Create New Knowledge, Theories, and Representations
Experience / FrequencyCourses / 18
Community service / 8
Internship / 7
Research/Senior project/Thesis / 5
Liberal arts education / 5
Study abroad experience / 4
Major / 4
Capstone class for major / 3
Relationships with faculty / 1
Reading for pleasure / 1
Peer tutoring / 1
Outside lectures and conferences / 1
Extracurricular activities / 1
Performance critiques / 1
Greek Life / 1
Can Critically Evaluate Knowledge, Theories, and Representations
Experience / FrequencyCourses / 9
Internship / 4
Major / 1
Capstone/Senior seminar / 1
Performance critiques / 1
Volunteer work / 1
Professors / 1
Research/Senior project/Thesis / 1
Community service / 1
Liberal arts education / 1
Extracurricular activities / 1
Outside performances / 1
Appendix F
NSSE 2005 Results
Analysis of Intentional Learning Items by Senior Capstone Experience
Question / Had Capstone Course(N = 65) / Did Not Have Capstone Course
(N=118) / Significance
Made a class presentation / 2.69
(.77) / 2.78
(.74) / .45
Working on a paper or project that required integrating ideas or information various sources / 3.57
(.61) / 3.30
(.71) / .01
Included diverse perspectives (different races, religions, genders, political beliefs, etc.) in class discussions or writing assignments / 3.15
(.80) / 2.60
(.68) / .001
Put together ideas or concepts from different courses when completing assignments or during class discussions / 3.18
(.61) / 2.78
(.76) / .001
Talked about career plans with a faculty member or advisor / 3.02
(.86) / 2.69
(.96) / .02
Discussed ideas from your readings or classes with others outside of class (students, family members, co-workers, etc.) / 3.06
(.83) / 2.79
(.69) / .01
Had serious conversations with students of a different race or ethnicity than your own / 2.77
(.95) / 2.45
(.94) / .03
Had serious conversations with students who are very different from you in terms of their religious beliefs, political opinions, or personal values / 3.34
(.78) / 2.93
(.87) / .001
Synthesizing and organizing ideas, information, or experiences into new, more complex interpretations and relationships / 3.37
(.63) / 3.02
(.80) / .001
Making judgments about the value of information, arguments, or methods, such as examining how others gathered and interpreted data and assessing the soundness of their conclusions / 3.17
(.78) / 2.92
(.81) / .04
Applying theories or concepts to practical problems or in new situations / 3.25
(.77) / 3.27
(.79) / .84
Tried to better understand someone else’s views by imagining how an issue looks from his or her perspective / 2.97
(.71) / 2.81
(.86) / .21
Learned something that changed the way you understand an issue or concept / 3.11
(.69) / 2.92
(.78) / .10
Learning effectively on your own / 3.34
(.74) / 3.13
(.80) / .08
Understanding yourself / 3.32
(.79) / 2.96
(.87) / .01
Understanding people of other racial and ethnic backgrounds / 2.32
(.99) / 2.10
(.99) / .15
Solving complex real-world problems / 2.85
(.87) / 2.53
(.97) / .03
Developing a personal code of values and ethics / 2.92
(.85) / 2.59
(.98) / .02
Note: Responses were given on a 4-point scale where higher numbers corresponded to “very often” or “very much” and lower numbers indicated “never” or “very little.”