During the past year (July 2007- June 2008), the ELIXR program has made positive headway on a number of fronts: Maturing Organizational and Program Planning, Developing Capability and Infrastructure, Case Story Development and Evaluation of Products, Processes and Program. In addition to summarizing the successes and lessons learned in each of these areas, we also list some ongoing Challenges in our quest for long-term sustainability.
Maturing Organizational and Program Planning
The focus of the first year’s efforts was on refining and disseminating the process for case story development and use at the level of individual institutions. In Year 1, small mini-grants were offered to 14 institutions to create 22 discipline-specific case stories organized under six broad themes. Providing institutional grants for individual case stories was necessary in understanding the story development process as well as for enriching initial partnerships on a one-on-basis. However, working at this level would not lead us toward a self-sustaining program, since it does not distribute the initiation and management of sets of case stories on targeted theme topics.
The focus of the second year’s efforts has continued the development and use of case stories, and extended into refining and disseminating the processes at the higher level of sets of case stories organized around particular theme topics of importance to teams of institutions. Therefore, we are developing a new model for initiating and managing collaborations for sets of digital case stories, and moving toward supporting these theme collaborations rather than individual institutions. In Spring 2008, we released a Request for Proposals (RFP) calling for institutional teams to create a theme topic module, a set of at least five digital case stories illustrating an exemplary teaching practice in higher education.
The goal for the theme modules concept is to identify theme leaders who have special expertise and passion as well as connection to an already established community (whether formal, such as Campus Compact if the chosen theme was developing student capabilities through civic engagement, or informal for a theme such as using Knowledge Surveys to support student learning). Working with an existing community of interest provides us with more cost-effective mechanism to engage and coordinate individual institutions to develop and use ELIXR case stories, and to maximize the use of evaluation tools. Where these communities of interest have representation from a diverse set of institutions, this diversity enhances the applicability of the stories across a wider spectrum of institutions in higher education.We believe that supporting theme leaders who already have a community infrastructure to support coordination amongst their institutions give us the best opportunity to position ELIXR to as a self-sustaining program at the conclusion of the grant. In particular, we are now working to engage state or regional leaders in developing theme modules for topics of particular importance for their goals of advancing teaching and learning.
As part of the outreach efforts for the spring 2008 RFP, sixteen people participated in three informational sessions and five proposals were received. The review committee agreed on funding four proposals plus providing advice to an international theme module on academic integrity from Higher Education Quality Council of Ontario. The four new themes and their leaders are as follows:
1) Creative Methods to Teach Creativity, Academy of Art University;
2) Using Assessment Rubrics to Improve Student Learning, University of Colorado, Denver;
3) Using Knowledge Surveys to Support Student Learning, CSU Channel Islands; and
4) Common Management System, University of Missouri, St. Louis.
Capability and Infrastructure Development
Another success has been capability and infrastructure development, and it has taken place on two levels: the program level and at individual institutions. At the system level, the strong focus on building capacity has resulted in a new website - - and a full complement of staff members joining the ELIXR team. The new website has a user-friendly layout and provides easy access to several training resources and the various case stories. The key appointments for staffing include the following: 1) Season Eckardt, ELIXR Program Manager; 2) Tasha Souza, ELIXR Faculty Development Lead; 3) Joel Bennett, Multimedia Development Assistant; and 4) Suzanne Aurilio (temporary staff position), web designer
in addition to the already established team members (Tom Carey, Lou Zweier, and Flora McMartin). The additional staff members have enriched the initiative planning at both conceptual and practical levels and ensured that there is sufficient staff resource to maintain progress on grant objectives.
The ELIXR team has been pleased to note that a few of the 14 individual institutions that initially received small mini-grants are contributing to further capacity building by developing their own internal system for creating case stories, which includes clear communication amongst staff and faculty from digital media and faculty development units. The team at San Francisco State, for instance, has produced 3 case stories in a 9 month period. In turn, the team is now working with one of ELIXR's system partners, Science Education Resource Center (SERC), to help SERC learn about the process of creating a story with the goal that SERC will create a set of case stories for faculty in the geosciences (two SERC stories are already underway).
Building off of the six training materials developed in spring 2007, the ELIXR team has created various forums for discussion and training on the case story creation process for our institutional teams. The ELIXR team has hosted three training webinars that focus on using Pachyderm (multi-media software tool), the application of the First Day of Class stories, and the storytelling process. The archives are available at Other efforts include three storytelling conference calls and in-person visits to all the initial teams. While the in-person visits have been fruitful for relationship building, we recognize that as we increase our scale this in-person guidance will not be feasible. However, the storytelling calls and webinars have helped to develop a sense of community, which will be bolstered in the coming year with more frequent theme meetings online and personalized theme weblog pages.
Case Story Development
Given the initial challenges of starting up the program, including the work-in-progress development of a systematic pathway for the teams to create stories, a majority of the case stories are behind their production schedules by three to five months. With summer approaching, many of the teams are eager to finish their stories and the majority will do so by early August, in time for faculty developers to use the case stories in their 2008-09 faculty workshops and participate in evaluation efforts. To motivate the teams to finish, the ELIXR team has held three “Story Circle” teleconferences this spring with the teams. These calls have provided an important forum for the case builders to receive feedback on the media design process and offer advice on how to address key challenges.
The delay in finishing the stories has resulted in an expansion of the timeline. The deliverable for second grant year, 32 case stories, will come from our newly selected set of themes. These 32 case stories will be finished by May 2009 instead of fall 2008. There will be another call for additional theme leaders in late fall with the aim that the Year 3 case stories will be completed by fall 2009. However, we anticipate that a no-cost extension into 2010 will be necessary to ensure that the deliverables are met and that sustainability is appropriately addressed.
Since the majority of case stories are still being finalized, the focus of evaluation activities for ELIXR has centered on clarifying the evaluation plan, initiation of data collection and the communication of evaluation processes and expectations to uses of ELIXR materials. Evaluation activities shifted in the past year from Evaluation Planning to supporting creation, testing and implementation of Evaluation Tools for measuring and determining the progress of the project. The plan was refined and communicated to the FIPSE program officer in Winter of 2008 (see attached documents: ELIXR Data Collection Plan and ELIXR Data Collection Matrix). The data collection plan moved evaluation efforts from the conceptual into action.
Evaluation of Products, Processes and Program
Evaluation activities included evaluation tool development and testing, focusing primarily on those tools to be used with workshops. The tools used in conjunction with the workshops are central to the evaluation effort in that they are used to evaluate at the three levels making up the ELIXR evaluation: Participation, Faculty Learning and Outcomes and Impact. (The attached documents provide description of each level and how they interact.)
The tools are being pilot tested at the Faculty Learning level. All of the workshop tools have been developed and have undergone face validity testing; they are currently being piloted by Kathy Ross, faculty developer from Indiana University (Kokomo). Indiana University (Kokomo) has agreed to evaluate the case stories and workshop for the First Day of Class at the Faculty Learning level. (Survey instruments have been designed to contain a set of core questions that must be asked for each case story and workshop. Other questions rotate depending on the outcomes for the workshop.) The Faculty Learning level consists of pre surveying of faculty regarding their knowledge of the content in the workshop, followed up by a survey regarding their satisfaction with the workshop. In turn, this is followed by a post survey that measures their learning based on the workshop and the extent to which they have applied that learning to their teaching.
At the Outcome and Impact level, we are currently working on refining the methodology for an evaluation study of use and impacts of the Universal Design case story theme. This case story theme has been developed by leveraging the work of the California State University consortia EnACT program, led by a faculty member Sonoma State University, and funded by the US Department of Education. The evaluation study associated with this story will leverage the data collected regarding student learning and changes in faculty behavior that is being collected for evaluating EnACT and combining it with that collected associated with the case story. Data collection and on-going analysis for this case story will begin in late summer.
A new focus for evaluation is on that of the theme teams, themselves. By the end of summer, an instrument will be developed and administered to the team leaders and team participants to gather their impressions regard the success and challenges associated with working as a team. The results of this survey will be used for formative purposes in refining our internal processes and support mechanisms for theme leaders and their teams.
One of the main challenges with regards to evaluation for this project is the need for effective communication with case story users who are not closely connected to the program. These faculty development leaders may wish to evaluate the effectiveness of their own workshops but may have very limited evaluation expertise and/or resources (indeed, some of our ELIXR partners face the same or similar challenges with regards to evaluation.) We have tried to carefully design the process to help build the capacity of these participants by making the instruments, tools and processes available to them. We have also designed a section of the ELIXR website as an evaluation resource. Here, users find a short description of the evaluation resources available and a description of how to participate. To improve usability, specific tools and descriptions of how to use them are connected to that area of the website where they are used, e.g., workshop tools are found in the section on workshops.
The evaluation of the project depends on the production and use of the case stories. With the set of the first stories almost completed and an impressive set of information on web usage, these accomplishments set the stage for the coming year in collecting data on the use and outcomes of the use of the cases. Since 2007, there have been over eleven thousand views of the five stories, which clearly suggest that there is great amount of interest in using the cases. One of our next steps forward is to create a web tool to gather demographic information.
A plan is also in place to add a “guestbook” function to each story asking people to briefly register themselves and provide comments. This feature will function much like user reviews on learning resource sites such as MERLOT ( asking users to share their impressions of the case story, what elements of the story they saw as most useful and how they plan to use the story. Additionally, we will ask users to submit their email addresses in order that we can follow up with them to see if they would like to participate in evaluation activities.
Evaluating impact is a multi-year effort. This grant period only allows us to evaluate outcomes and impact at a simple level. To get at the larger questions associated with impact, more time will be needed. For example, in the remaining months of the grant period, a faculty member might attend a workshop, revise his or her class, but not offer the class until after the grant period, making it difficult to collect and analyze the impacts of the use of the resources.
The changes to the original evaluation plan relate to identifying ELIXR’s role in lending campuses and users more focused evaluation support in order to improve the evaluation capacity of the source for data collection. We view this as a positive outcome in that not only will we be able to systematically collect the data we require, we will also help our targeted audience improve their own evaluation capacity.
Challenges
The design process is inextricably connected to evaluation, in that we need the case stories to be completed before evaluation can thoroughly begin. Recently, we heard from a team about their perception of the intensive time commitment it takes to produce one case story. While the time of the faculty member involved (who assumed the case author/editor role) was close to the estimated 40 hours, it seemed like much more because it was spread out over a nine to twelve month period (this timeline could be expanded if a faculty member suddenly has other more pressing priorities). Thus, the 'elapsed time' is much greater than if it was a condensed amount of time over two months. One way to ameliorate the situation is to ask the teams to shorten the design process to six months rather than current time period of twelve months. While the longer time period may produce a richer piece, we do not want to jeopardize momentum and tax people’s interest in the project. Also, we have worked on a clearer structure for the new theme leaders, which will guide the theme leaders more effectively, or at least, diminish some of the perceptions about the lengthy design process.
Additionally, we are learning more about the capability development necessary for sustaining this initiative, in particular having the team members embody the appropriate leadership skills in media capability, faculty development, organizational development, and partnership building. In one instance, an instructional designer assigned to the media editor role understood the role as being a ‘worker bee” for the faculty author, rather then serving in an editor role and contributing to decisions about the direction of the video. Another instructional designer struggled with having his faculty member (subject of the case) release the self-made video recordings to him only to discover that the recording was not high quality; he since realized that he does have an expertise in digital media and needs to assert himself more effectively in order that the case story is produced in a dynamic and informative way. Another example is when a theme leader has a passion for the chosen topic and yet the organizing skills are not equally as strong. The skill development can be addressed in some degree with upfront training and guidance and encourage use of the resource materials that have been developed. This is a challenge that could influence long-term institutionalization, and the ELIXR team will give more substantive thought on this issue in the coming year.
Another challenge that we have encountered is identifying ways to fully engage state-wide consortia and state system offices. From the original list of partners, the Science Education Resource Center (SERC) is an active partner in both case story development and evaluation (once the stories are developed). We expect that Oklahoma State Regents for Higher Educationwill develop a theme in 2009 and serve in a peer review capacity for the creativity theme that is being developed in 2008-09. However, the other initial institutional partners - Minnesota State Colleges and Universities System and The University System of Georgia - have played minimal roles with the ELIXR project. We attribute this in large measure to organizational structures, in which close links are lacking between the learning technology functions and the faculty development offices (if any). In addition, to of our initial institutional partners, Coastline Community College and Brigham Young University, had changes in key personnel and were not able to be actively involved to date.