Enriched Learning Environment Project

Enriched Learning Environment (ELE) Project

Implementation Evaluation

Presented to:

Don Elger

Steve Beyerlein

Submitted by:

Jennifer M. Beller

Audrey E. Townsend

G. Todd Vanek

August 6, 2003

The Assessment and Evaluation Center

Washington State University

P.O. Box 642136

Pullman, WA 99164-2136
509-335-2223
Executive Summary

This report provides the findings of the Enriched Learning Environment (ELE) Project implementation evaluation. Although two basic components of the ELE project were studied, this report focuses primarily on the first year evaluation of the ELE Project.

This evaluation focused on goal attainment, value of change effort/involvement, teamwork/collaboration, and impact on practice to determine the success of the implementation of the ELE project. Interviews were used to obtain information regarding these four factors.

Interaction, collaboration and a supportive environment were key strengths for all involved in the project. The importance of maintaining a time-efficient environment was a key concern for project stakeholders. Student feedback, the use of rubrics, and hands-on learning experiences were also mentioned.

The impact on practice discussion provided numerous recommendations for the next two years of the project. Strengths of the program included the continued collaboration and teamwork with project stakeholders, the building upon the mentoring program, and the creation of an interactive environment in the classroom. Deltas included more collaboration between faculty and students, the need for faculty to share their knowledge and learning experiences with the undergraduate and graduate mentors, and more support from faculty for the mentoring program and Idaho Mindworks. In addition, respondents indicated the need for more understanding on how to implement new ideas generated in the team meetings, more consistency in the implementation of the coaching paradigm, more interactive workshops, more support from departmental administration, and collaboration across departmental lines.

Introduction

The Enriched Learning Environment (ELE) Project is a three-year program designed to identify and investigate pedagogical practices in engineering. In the first year of the project the Assessment and Evaluation Center (AEC) at Washington State University participated in two basic components of the ELE project: 1) mentorship of a social science research project and 2) the first year evaluation of the ELE Project. This report focuses on the implementation evaluation, which was designed to uncover the extent of the project’s start up in relation to project goals.

The social science research project examined metacognition strategies in engineering education and involved the development of a survey of metacognitive engineering problems administered to both faculty and graduate students. The purpose of the questionnaire was to examine whether faculty and/or students believed that these tasks were important in engineering education, were being taught, and were being learned by fellow students. The project resulted in a proceedings publication and presentation to the American Society of Engineering Education.

The evaluation team from the AEC was commissioned to conduct the ELE implementation evaluation and report the results to the project principal investigators. The remainder of this report will focus on a brief examination of how the ELE Project did during the 2002 – 2003 academic year. There is no attempt to measure impact on student learning or to make claims on the effectiveness of the project. Simply stated, this evaluation looks to see if the project is on course with its goals.

Methods

Enriched Learning Environment project principal investigators and the evaluation team from the AEC agreed that interviews (individual and group) would be the best way to get at implementation effectiveness. The evaluation team and two co-principal investigators for the project jointly developed an interview guide to help project personnel understand the progress of the ELE project in its implementation year. The interview guide was constructed over two meetings and is included in AppendixA. The evaluation guide was divided into four areas. Those areas were goal attainment, value of change effort/involvement, teamwork/collaboration, and impact on practice.

Two evaluators from the evaluation team conducted the interviews. Faculty were interviewed individually, graduate and undergraduate mentors were interviewed as groups. Complete responses to interview questions are provided in appendix B. The evaluation team learned several things regarding the methods used to conduct this evaluation and those lessons are presented in appendix C.

The results from the interviews for this evaluation are descriptive in nature and should not be taken as causal evidence for the current status of the ELE project. The data collected and results presented are for information purposes only.

Results

The interviews produced data that provided a deepened understanding of the implementation of the ELE project. The results are presented in the four categories of the interview guide.

Goal Attainment

The first interview question requested the interviewees’ understanding of the ELE project goals. All respondents discussed at least some part of the goals of the ELE Project. Collaborative and interactive learning environments, time and cost effective teaching methods, and a mentoring program for undergraduate engineering students were among the top goals given. In addition, faculty indicated that better assessments to measure student learning, the creation of a coaching paradigm in the classroom, and faculty teaching improvement were important goals of the ELE project.

Value of Change Effort/Involvement

When asked what was engaging for them, interviewees provided a plethora of responses. The main theme across responses was interaction with others. Students noted that interaction with other students was beneficial to their experience. Trying new things and being allowed to make mistakes was also a common response. Several individual responses also provided support for several ELE activities. These individual responses included seeing the importance of giving students useful feedback, the idea of using a rubric (and creating it as a team), the challenges that support personal growth mentally, the hands-on learning opportunities, and the experience of working with many projects.

When confronted with the time and cost effectiveness question, most respondents quickly addressed the time issue. In fact the only cost effective comment was in regards to the idea of collaborative learning. If students work in teams, they can make attempts (and failures) in a group and save resources, which becomes even more cost effective if the attempts are well-documented and passed on. As far as time efficiency, one respondent was very put-off by the assessment “lecture.” The respondent noted, “The lecture went against the idea of the project which is interactive learning.” Respondents noted that workshops were productive, but if the same material were offered again, they would not attend. The workshops were well structured yet did not make attendee participation a priority. Students noted that mentoring is very time consuming but beneficial in creating marketable skills including leadership, experience from working with numerous projects, and teamwork. Students also noted that the program is time efficient as long as student developmental stages are kept in mind so students come prepared for the experience.

Teamwork/Collaboration

The benefits gained through collaboration with others in the teamwork and workshop environments were numerous and uncovered an underlying theme of interaction. Nearly all respondents noted that the interaction was the most beneficial element in ELE activities. Other beneficial elements of the project were the varying levels of expertise represented at the workshops and in the project. The different levels of expertise provided a richer discussion and provided participants with a broader and deeper understanding of ideas. One respondent noted that just listening to other’s ideas was productive.

When interviewees were asked what contributions they made to the ELE project, very different responses were received. Those directly involved with the project (i.e. principal investigators and mentors) reported on their contributions whereas those not directly involved discussed attendance at workshops and meetings. One respondent stated, “My contributions could have been more had implementation been guided better.”

Impact on Practice

When the interviewees were asked the question “To improve your educational practice and scholarship, what sort of ELE project activities would you like to work on in the future”, many suggestions and deltas were given as responses. These responses are covered below in the discussion section under the heading “Deltas.”

Discussion

Some limitations must be considered when reading this report. The first of those limitations is the guarded responses provided by the majority of the respondents. A number of respondents prefaced their responses with “they are doing a good job.” However, as interviews continued, respondents opened up and provided straightforward answers to the questions.

Time was also a limitation. This evaluation was planned, conducted, and reported in less than two months. It was not conducted until after the funding period, which resulted in a push to complete it quickly. This shortened evaluation period does not diminish the validity of the information provided here, however it must be recognized that the depth and breadth of information is limited.

The responses to the question on goal attainment uncovered an interesting phenomena that provided an underlying framework that may help the ELE succeed in the next two years of the project. The perspectives of different faculty suggest there are complimentary sub-goals within the ELE project. The goals may be divided along physical and mental lines. And, although the individuals who are driving the project are focused on one or the other, together they appear to compliment one another. The Idaho Mindworks lab provides the physical environment where students can “try-out” different ideas. The pedagogical changes (coaching, mentoring, collaboration) under investigation by the ELE project provide the mental “skills” that allow students to experiment successfully in the Idaho Mindworks lab. However, there is more that has to happen for the ELE project to make a “change” to the engineering research/teaching culture. One of the most notable threads between interview responses was the idea of change. All levels of participants in the ELE project suggest that changes in the culture of the engineering program must occur. These range from individuals changing their classroom practices, to veteran faculty changing their perspective on teaching and learning. Students noted a positive change in the interaction with other students but would like to see more interaction with faculty. It was suggested that administrators must change funding priorities to allow for a cultural change to take place. Evidence of this need for change comes from students noting the limited interactions with faculty and the faculty’s reluctance to allow students to participate in projects such as Idaho Mindworks and the mentoring program.

An interesting continuum was discovered in the responses to the question on collaboration. Examining a continuum from undergraduates to veteran faculty, there are very different perspectives, even beliefs, about the benefits of collaboration. Students enjoy the collaboration with other students and desire more interaction with faculty. Newer faculty desire interaction on an intellectual level (discussion, evaluations, etc.) and assistance with implementation from senior faculty. Senior faculty noted that it would be difficult to change the autonomy of the PhD, which was amplified by comments from the mentors regarding faculty support of Idaho Mindworks and the mentoring program. “Only a few faculty put graduate students in the Mindworks…others are more protective of their graduate student’s time for their own research needs” was how one student perceived the situation. The cultural change of teaching collaboration will take time. The amount of time to noticeable change could probably be shortened if more faculty modeled the collaborative effort.

Strengths

The biggest strength of the ELE project based on interviewee responses is the teamwork and collaboration components. Faculty interviewees were very impressed with the continued effort by faculty to work together in improving the engineering program. They believed that each person contributing to the project would ensure its success.

The mentoring program is the second strength of the project. The graduate student mentors have indicated that because of the time and resources given to them as undergraduates, the mentoring program gives them a sense of wanting to give something back. The mentoring program also challenges them mentally and allows them to grow as people. And, they feel their understanding of engineering concepts is strengthened. The students interviewed commented that having their mentoring experience on their resume has been very powerful, as it will market them for future employment.

Faculty and students indicated that a third strength of the program is the creation of an interactive environment in the classroom. Student mentors feel that the hands-on experiences are exciting because it is easier to learn by doing, thus connecting the physical world to theory.

Deltas

Deltas are reported based on direct suggestions given by the interviewees when asked the question about impact on practice. Although a strength amongst faculty, collaboration between faculty and students seems to be weak. The students desire more involvement in the planning process rather than just giving them tasks that they are to complete without fully understanding the big picture of the ELE project. This would include discussion of ELE project goals, as most students interviewed were somewhat unclear in this area.

Mentors would like faculty to share their learning experiences and knowledge with the faculty more often. The faculty mentor interaction could come in discussions about big picture issues between faculty and mentors, yearly meetings to address goals, and other ongoing issues. Respondents also suggested that a tiered student mentor program could help more students learn. Mentors suggested that sophomores could mentor freshman, juniors mentor sophomores, etc. and that select ME424 students could mentor ME322 students and select ME322 students could mentor ME 262 students.

In terms of faculty buy-in, faculty and students have mentioned that both the mentoring program and Idaho Mindworks have little support from faculty. It has been indicated that there are few faculty members who allow and support their graduate students in being a part of the mentoring program and Idaho Mindworks.

There seems to be a lack of understanding in how to properly implement ELE project ideas. Faculty would like to see more decisions made followed by immediate action on those decisions. Even when there is something that one doesn’t know how to do, it was suggested to take an idea, try it, and then analyze it for future suggestions instead of just talking about what the faculty wants to get done.

Consistency is lacking in the implementation of the coaching paradigm. One idea brought to light was to videotape and then evaluate the coaching paradigm as it is implemented in the classroom. More support would be greatly appreciated in the implementation of the coaching paradigm by the faculty, rather than a “sink or swim” perspective.

In regards to the workshops, it is suggested to keep them interactive so that participants can become actively involved in putting new ideas into practice. Some past workshops have been too cut and dry with little time for interaction. One respondent suggested that implementation assistance would be key to ensuring the ideas and skills go from the workshop to the classroom. Faculty peer evaluations, open discussions about implementation, and student input would also aid in this process.

There is a perceived lack of support from the Department of Engineering that faculty need to continue to work in gaining. In addition, one respondent suggested that it is time to collect student feedback on the effectiveness of the implementation of the ideas and skills learned through the project. Specifically they would like student feedback on the coaching paradigm. Finally, someone said, “publicize the project outside of the typical scholarly journals.”

In conclusion, the underlying desire of the respondents is for the ELE project to continue to provide an atmosphere (physical and mental) where all (faculty and students) are encouraged to explore. Mistakes are learning opportunities because there is a desire to create a sustainable learning environment using things like pass-down notebooks, project logs, journals, video recordings, and other support of the learning process. With that, the majority of respondents would like to see the push for the teaming and learning efficiency to continue, which will hopefully lead to a culture of learning through coaching, mentoring, and collaboration. That continued collaboration could come in the form of hands-on activities, to include projects that cross department lines, which allows the students to see the connections between the different areas of the engineering field.