MU315 Research Labs Assessment
Winter Term 2016
Purpose of Assessment
The purpose of this assessment is to ascertain the appropriateness and effectiveness of the Research Lab format in teaching musicological information literacy to MU 315 students. Results will be used to modify future MU 315 information literacy education integration.
Description of Course
MU 315 Music History I is the first in a series of 3 required, consecutive, semester-long courses designed to draw music majors from the beginnings of Western Music through the present day; MU 315 covers this history through the early Baroque. Music majors nearly always complete this course Winter Term of their second year. MU 315 also fulfills a General Education Writing Intensive (WI) credit. In addition to the historical facts and listening skills germane to such a course, MU 315 inspires students to synthesize facts, concepts, and threads of inquiry into a cohesive, whole-minded perception of the course topics.
Wartburg’s Information Literacy Across the Curriculum (ILAC) program is described in terms of three “steps”: Novice Steps, which are courses that lay the groundwork for basic skills and concepts across the ACRL Frames; Stepping Stones, which are first courses in which students perform discipline-specific research by engaging with discipline-specific research sources, strategies, and concepts; and Master Steps, which look at advanced skills and concepts across the Frames and assess whether students can carry their critical thinking skills over into the real world. MU 315 is the Stepping Stones course in the music major.
Related Course Assignment
The main research project in MU 315 is an 8-page research paper, with a minimum of 12 sources, on a topic of the student’s choosing. To prepare for the paper, a Prospectus—which includes a 200-300-word scope, a 10-source bibliography, and a discography—is submitted in advance.
Information Literacy Education Exposure for Music Students Prior to MU 315
Prior to this course, music majors are definitely exposed to general research resources only in IS101: Inquiry Studies; most music majors will also have received general instruction on resource types in EN112: Intermediate Composition, but because of the demands of the music major, students may be taking MU 315 and EN112 concurrently or have transferred in the credit from elsewhere. Other Essential Education courses in which students may interact with Information Literacy are not taken at any certain time, again due to the demands of the music major. Music Therapy majors will have been exposed to more specific resources in early Music Therapy-specific courses prior to taking MU 315, but they will not have received the level of instruction necessary to perform at the level expected in MU 315.
Brief Overview of Librarian Involvement in MU 315
Research Labs grew out of an initial involvement in MU 315, begun in Winter 2012, of a one-shot session of 65 minutes and a draft Prospectus assignment, graded either by the professor or the librarian. While even this limited interaction anecdotally improved the quality of student research, it became apparent quite soon after implementation that it was not sufficient to create the level of research mastery needed.
This amount of information is more logically taught as building blocks, and the opportunity for the librarian to require and assess assignments along the way is invaluable in catching students who previously fell through the cracks. I must also cite Sara Beutter Manus’s article “Librarian in the Classroom”[1]for motivation to embed information literacy across an introductory music course.
Direct comparison of rubric scores from 2014 and 2015 show that moving from a one-shot + assignment format to Research Labs improved scores in skills and perception.
Analysis of student self-reported data on the usefulness of the Research Labs from 2015 led to (1) a revision of the total Research Labs from six to four and (2) a removal of the “draft Prospectus” assignment, sometimes called a Bibliography Sketch—now the Labs function as the building blocks to the Prospectus. These revisions have taken effect this Winter 2016 term.
Analysis of rubric scores from the 2016 Winter Term show what could be an alarming downward trend in the quality of responses on the Prospectus. However, as noted within that chart included below, I believe this is due to the natural adjustment of my strictness in grading based on their apparent increased potential and the fact that the first year you try a new format, you are still learning what you can expect. Now that I have established a baseline, I hope that subsequent years show similar scores, because the lack of 100% perfect scores additionally reflects the stage they are at in this course—they are beginning to learn advanced techniques, but they are by no means masters. I will use the results from the Post-Test (explained below) and conversation with Dr. Survilla about the quality of the final papers to help illuminate whether this is the case.
I found it useful during the first year of Research Labs to collect self-reported data of student knowledge before and after the information literacy interaction. I used this data to help balance the Labs between my own expectations, the students’ desires, and the reality of the students’ abilities. This year, Winter 2016, I am no longer collecting student self-reported data of knowledge and have instead moved to a more objective pre- and post-assessment of student ability and perception, to more easily pinpoint gaps in my pedagogy.
[Attachments]
Learning Outcomes of the Research Labs and their relation to the ACRL Framework
The student will be able to… / ACRL Frame- Use overview articles to the fullest potential of those articles
- Predict the scope of a topic based on analysis of several overview articles
- Understand the importance of conducting overview and in-depth research prior to choosing needs of and searching for articles in musicological research
- Understand the usefulness of books to musicological research based on all aspects, not merely date of publication
- Create a prospectus paragraph summarizing the known knowledge so far and applying analysis of how this knowledge will form into an 8-page paper
- Identify, locate, and use Vogel Library’s overview, book, article, and audio/video sources as applicable to musicological research
- Apply learned information (from overview sources) to choices in keywords while searching for books
- Employ flexibility in using search terms while searching for relevant books
- Create Chicago Style citations for musicological sources
Issues from Winter 2014 and how Research Labs in 2015 addressed these issues
Showing how the Labs themselves were a useful, relevant move.
Percent of Total Students 2014 / Percent of Total Students 2015 / What they did incorrectly / How Research Labs addressed and amended this result in 201567% / 43% / You cited reference articles. / While fewer students cited reference articles, and those that did only cited 1 or 2 (as opposed to many more), this percentage is still too high. I made the mistake of not saying this in the tutorial and spelling it out on the assignment—I only stated it verbally to the students during class.
89% / 4% / You did not do an exhaustive search in the library catalog and/or WorldCat for relevant books. / Only 1 out of 23 students did not fully examine the available books, and this student had already presented issues with completing an assignment to her full potential.
52% / 0% / You found articles, but they are all on different aspects of the topic and are too variant to be used cohesively in this length of paper. / The progressive nature of the Research Labs allows the students time to learn each of these important steps in research so that by the time they reach the Bibliography Sketch (Lab 6), there is little opportunity to fail, because they have been working towards this and receiving help along the way.
33% / 0% / You found articles in JSTOR (or elsewhere) but not RILM.
26% / 0% / You barely looked for articles or didn’t look at all.
70% / 17% / Because of any or all of the above, you did not perceive how to realistically narrow the scope of your topic for a 10-12 page paper. / Four students out of 23 did not write a scope that fully explored their topic. For one student, this was a surprise, since she could have written a great scope, based on the research she had already performed. For another, she had trouble articulating what she meant, which led to a discussion on clarity in language. For the other 2, they were already somewhat underperformers, so this was not a surprise.
63% / 0% / You did not make your Music Online Playlist viewable to me. / This requirement became part of Lab 5 so that it did not affect the Lab 6 grade. Lab 5 is a skill-based assignment, whereas Lab 6 is testing for comprehension and applied skill.
Comparison of rubric grades for Bibliography Sketch/Prospectus from Winter 2014 (one research session), Winter 2015 (6 Research Labs), and Winter 2016 (4 Labs).
The Bib Sketch in 2014 was worth 25 points; the Bib Sketch in 2015 was worth 40 points; the Prospectus in 2016 was worth 25 points (by me—Dr. Survilla also graded it, but I am not looking at her data) but the rubric wording is exactly the same for all 3.
In 2014, only 27 students completed the Bib Sketch; in 2015, 23 students completed the Bib Sketch; and in 2016, 32 students completed the Prospectus. Percentages of students receiving the shown score are used in order to aid in direct comparison.
Excellent / Average / Poor / Analysis of improvement from year to yearScope / Scope shows clear understanding of resources available on topic and reasoning for any narrowing or widening of topic. / Scope shows acceptable understanding of breadth and depth of topic, but further perception was needed to fully elucidate relation of available sources to topic development. / Scope shows marginal to no understanding of breadth and depth of topic and the accompanying sources.
2014 percentage of students who received this score / 7% / 45% / 48% / Because they wrote an initial scope, they knew (for the most part) they should move on to something more detailed for this assignment. Also, because they had plenty of time to gather their resources, the results were better.
2015 percentage of students who received this score / 82% / 9% / 9%
2016 percentage students who received this score / 59% / 22% / 19% / I was surprised by this drop in “Excellent” scores, especially since I felt that these 2 sections were the strongest sections I have worked with in all 5 years, but I do know I was harder on these students than previously because they showed more potential from the beginning. I made many fewer consultations with these students because there were that many fewer who were obviously flailing. In addition, 2015 may have served as the adjustment year in which I learn for myself what I can expect.
Bibliography / The sources chosen for the bibliography were clearly selected as the most appropriate sources from a wide-range of available sources. / The sources chosen for the bibliography show some measure of discernment, but more discernment was needed in selecting appropriate sources for this topic. / The sources chosen for the bibliography do not appear to be carefully selected; the first ten sources found seem to have been those included.
2014 students who received this score (27) / 4% / 44% / 52% / Again, because they were walked through each research step, they had more opportunity to build up their research, rather than cobbling it together quickly.
2015 students who received this score (23) / 57% / 43% / 0%
2016 students who received this score (32) / 53% / 44% / 3% (only 1 person) / This is roughly the same as last year. I think this shows that the students are simply at this level of not being able to fully perceive their sources. It is also a learning point for them to never cite an encyclopedia (as in, they learn it for real by getting this score back).
Discography / The recordings selected for the discography relate to the topic. / The recordings selected for the discography mostly relate to the topic, but some questionable sources were included. / The recordings do not specifically relate to the topic and there is no explanation of why. / Because there was a previous assignment about choosing recordings, they were able (for the most part) to take my comments and make any necessary changes.
2014 students who received this score (27) / 52% / 33% / 15%
2015 students who received this score (23) / 83% / 17% / 0%
2016 students who received this score (32) / 69% / 28% / 3% (1 person who didn’t include one at all) / Again, I’m surprised by the drop in scores, but I think it shows both of what I mentioned above—I was harder on these students and this is a learning point for them. I did notice clear evidence of the students not watching the video tutorials (this is self-monitored), in which they would have learned how to get the “Excellent” score, so I will be brainstorming to myself how to increase this motivation, since there’s no reason, after watching the videos, to not follow the directions.
Results from Pre-Test in 2016
These results show that these students have a basic grasp of sources but will need education to reach the intermediate grasp of resources and search strategies necessary at this level—exactly what I wanted to see, since the Labs speak directly to the education of this intermediate level of skills and concepts. I scored the questions based on whether the student only gave a basic answer or advanced answer. This is why there is more than one result for most questions. You can see the actual questions included after this with examples of what is a “basic” and what is a more “advanced” answer.
MU 315 Research Methods Pre-Test Questions
- Overview
- What is the best use of an entry in a credible overview source that is only a paragraph long?
- Basic answer= get you started on a search
- Advanced answer=read each word closely, look at references, do not cite, etc.
- Books
- Where do you look for a book that was recommended by one of your overview articles?
- Basic answer=library catalog
- Advanced answer=library catalog and WorldCat
- Citations
- Where do you look for an article that was recommend by one of your overview articles?
- Basic answer=in the databases
- Advanced answer=look the journal name up in Journal Finder, see if we carry those years
- Strategy
- If our library catalog shows no results for the first search term you try, what are two different types of actions you can take next?
- Basic answer=Be flexible with your keywords
- Advanced answer=Change the database (to WorldCat)
- Older Sources
- Imagine that you find a book on your topic that was published in 1935. What aspect(s) might make this book a good one to use, even though it seems “old”?
- Basic answer=the facts might still be true or you can use it as a primary source
- Advanced answer=it may be the standard source on that topic in the field—see if other sources refer to it frequently
- Google Books
- What is one problem with using a book found in the Google Books feature?
- Only one answer matters=the books are often not shown in full text.
- Article Strategy
- Why is it important to have a deeper understanding of your topic before you look for articles?
- Only one answer matters=so that you know how to narrow your search terms and look for exactly what you want, rather than being bombarded by thousands of articles that only seem relevant.
- Audio Resources
- What are the 4 main ways the library provides you with access to audio sources?
- CDs, Medici.tv, Music & Performing arts database, or WorldCat/ILL. I also marked if they simply mentioned something about online resources—if they don’t remember the name of the database but do know how to get there, that’s okay with me.
[1]Beutter Manus, S. J. (2009). Librarian in the classroom: An embedded approach to music information literacy for first-year undergraduates. Notes, 66(2), 249-261.