Executive Summary of the ILA/ACRL K-12/Higher Education Information Literacy Pilot Study
Academic Year 2013/14-2014/15
Pam Rees, Chair
In late 2011, the Executive Board of ILA-ACRL approved the creation of the K12/Higher Education ad hoc committee to be chaired by Pam Rees (Grand View University). The charge of the ad hoc committee was “to partner with IASL to develop and implement a framework and action plan for future collaboration”. Concurrently the Executive Board of IASL approved the creation of a committee to work with this committee on concerns related to information literacy. The teacher librarian appointed to chairthe IASL committeewas Kathy Bottaro (Sioux City Public Schools). To facilitate communication between ILA-ACRL and IASL, and to provide a mechanism for establishing cooperative ventures, a K-16 Joint Steering Committee was also formed and jointly chaired. The ILA-ACRL by-laws stipulate that ad hoc committees, such as the K-12/Higher Education Committee, may only be appointed for a maximumof two terms of two years each before permanent action must be taken by ending the initiative, creating a new committee, or reassigning the charge to another committee.In accordance with the by-laws,ILA-ACRL Executive Board reinstated the K-12/Higher Education Committee for a second and final term to end in November 2015.The ILA-ACRL Executive board has been consideringwhat further action should be taken.In early 2015, Kathy Bottaro resigned her leadership role on both committees due to conflicts in her work schedule. No replacement has yet been named.
In the spring of 2012, IASL’s conference planners extended a special invitation to members of the K-12/Higher Education and K-16 Joint Committees to attend and present a session on existing K-12/higher education collaborative initiatives at their spring conference. Members of both committees took advantage of this opportunity as well as the opportunity to meet jointly for the first time. After the session,they also met with teacher librarians to gather their input on issues related to information literacy instruction. The feedback from teacher librarians indicated the need for a better understanding of the information literacy skill levels of students entering college in Iowa. Those present felt that having assessment datato present to their administrators would provide needed evidence demonstrating the necessity for more information literacy instruction at the K-12 levels. Each felt that having access to data was key to gaining more administrative support.As an initial step, ILA-ACRL listserv members were surveyed for feedback on their analysis of freshman information literacy skills. In particular,respondents were asked to reflect on assessment methods or tools being used and if they had the opportunity to assess a significant number of freshmen. Based upon their responses, it was felt that this type of data collection was not being carried out systematically across the state.The K-12/Higher Education Committee decided to coordinate a study to determine how well first year college studentsin Iowa understand information literacy concepts. In addition to providing teacher librarians with desired data, it was also hoped that academic librarians could benchmark their freshman against the state average; which in turn could be helpful in framing their own discussions.
In fall 2012, the K-12/Higher EducationCommittee began recruiting academic institutions to participate in an information literacy pilot study. The pilot eventually enlisted eight confirmed institutions (Briar Cliff College, Central College, Cornell College, DMACC-Ankeny, Drake University, Grand View University, University of Dubuque, and Wartburg College). After resolving some technical and logistical issues the project was fully launched in fall 2013.
The spring prior to that, members of the K-12/Higher EducationCommittee looked at several assessment tools for the pilot study and eventually decided upon an edited version of theninth grade TRAILS test, designed by Kent State University. Although the committee members would like for Iowa students to demonstrate proficiency at the 12th grade level, there was concern about the amount of time that would be required to take that testalong with the overall difficulty of the questions.Specifically, the 12th grade test utilized a problem-based assessment style that would require higher order thinking as well as extra time to read each question. There was also some concern about the library jargon in both the 9th and 12th grade tests. Since one of the pilot participants, the University of Dubuque was already using an edited version of the 9th grade test, with the permission of the Kent State University Libraries the committee elected to use this instrument for at least the first year of the pilot. Dan Chibnall, User Services & Instruction Design Librarian at Grand View University, created a folder on Google Drive where participants easily retrieved an MSWord or Blackboard compatible copy of the test and answers.
Fall 2013 saw the completion of the first year of the pilot. Nearly 1,500 students were tested. Despite the fact that the tool used was the 9th grade level test, the pilot scores were perhaps predictably low. All participating institutions had overall averages in or near the D level; falling between 63-68% (Appendix A). Specifically, 7 of the 26 questions had fewer than 50% fully correct responses at all of the institutions.For a student to have a fully correct response for a multiple answer question, they must choose all correct responses, but they could also select additional responses that were not correct and not be penalized. Had students lost partial credit for added responses, overall scores would have been even lower.
The concepts represented in those low scoring questions were: topic focus, resource selection (twice), resource bias (twice), reading citations, and information integration. These questions are listedin Appendix B of this report with correct responses bolded. Conversely, students did very well with the concepts of broader vs. narrower topics and currency as an element of web site evaluation.
These results were shared during the business meeting of the ILA-ACRL spring conference in 2014. The data was also communicated to the K-16 Joint Committee members via Kathy Bottaro. In addition, members of the K-12/Higher Education Committee distributed the datato their AEA liaisons when possible and to both the University of Iowa and University of Northern Iowa’s library science graduate programs. Additionally, the K-12/Higher Education Chair publicized the results at an AEA Leadership Group meeting to which she was invited. Finally, the results of the study were communicated to Iowa’s national representatives and senators by the President of ILA during advocacy meetings in Washington DC.
During the first year of the pilot, a couple of concerns surfaced regarding the assessment tool and the method of data collection. The first concern related to the scoring of questions with multiple correct answers described above. A discussion centeredon whether to tally only correct answers and disregard incorrect selections or to reduce the number of correct answers by the number of incorrect selections. The second issue involved the method of reporting out the data for these same questions. For comparison purposes the K-12/Higher Education elected to note only fully correct responses instead of an overall composite score for multiple answer questions. In addition to these concerns, there was a little apprehension about the ambiguity of some answers listed for a few of the questions. To resolve these issues, the participants met over the summer to come to a consensus moving forward. There was a strong commitment on the part of several of the institutions to collect longitudinal data, therefore participants ultimately agreed to score and report data for multiple answer questions in the same manner as before. After some debate, participants likewise elected to slightly refocus the ambiguous responses of only a few questions. Only 2 (questions 10 and 17) of the low scoring questions were revised.
The second year of the study was completed in fall 2014. There were still eight institutions participating in the study, but they were not the same eight. For logistical reasons, Drake University and the University of Dubuque both dropped out, but Buena Vista University and St Ambrose University were added.Ultimately, only four libraries submitted data for the 2014-2015 study. The reduction in contributing libraries translated into 550 fewer student participants. Although the cohort changed somewhat, the results of the data analysis was quite similar to that of 2013-14. Scores ranged from an average of between 59.1% and 67.3% or 63.22% overall (Appendix C). During this testing cycle, there were 5 problematic questions as opposed to 7 the previous year. The themes, however, were much the same as before: topic focus, bias (2), and reading citations. The questions that caused the most difficulty in both studies were numbers 5, 11, 13, and 22 from Appendix B but question number 26(Appendix D), which assessed the student’s abilityto identify sourcesfor background information was new.The edits made to questions 10 and 17 when the test was revised the previous summer, may explain why scores on those questions improved. On a more positive note, students in both studies did master a couple key concepts. They could identify the difference between broad and narrow topics and they understood the importance of checking website currency. The institutional scores for those questions ranged between 72-98%.
A comparison of the cumulative data from both studies shows that students had fewer universally problematic questions, but that overall scores actually decreased. In the 2014-13 study, institutional percentages fell from 64.33% to 63.22% in 2014-15. One possible explanation for this may be the impact of having fewer participating libraries in the study.
The 2014-15 data and a brief comparative analysis were shared withthe AEA Leadership Group during their January 2015 meeting. It was also part of the K-12 Higher Education Committee report at the business meeting during the ILA-ACRL 2015 Spring Conference. Additionally, the results were presented during a concurrent session of the IASL 2015 Spring Conference. Finally, a summary of the data was distributed to state representatives at the spring 2015 Legislative Dayheld in the Iowa state capitol building.
While it is too soonto draw any firm conclusions, there are a few definite patterns in the results thus far. Firstly, in general it is evident that students are entering post-secondary education lacking in certain crucial information literacy skills. None of the institutions from either cohort attained an average above a low C and most reported D level averages. In particular,students continue to struggle with concepts such as identifying bias, writing a good thesis statement, and reading basic citation information. Although most academic librarians have noted these difficulties anecdotally, the data now supports those observations.Secondly, the test results weren’t all bad. Students do seem to have a good understanding of a couple of key concepts. They can identify the difference between broad and narrow topics and perhaps more significantly they understand the importance of checking website currency. Thisevidence points to the need for more support, access, funding, and staffing for information literacy instruction at the K-12 level, so that teacher librarians have the opportunity to be as successful with all information literacy concepts as they were with these two. Finally, though the preliminary results are useful,more research is needed to determine the extent of the issues and their consistency over time.
As mentioned earlier in this report,the K-12/Higher Education Committee will cease to exist at the end of November 2015. It is worth noting that over the last several months, the ILA-ACRL Executive Board has been considering how best to continue the committee’s work. To that end, the board appointed a task force to study long range goals and make a proposal. The task force submitted several recommendations, all of which have been ratified by the Executive Board. There are a couple of outside issues that need to be resolved before any formal announcement can be made.However, any decision made by the Executive Board regarding the future of the K-12/Higher Education Committee will likely not go into effect until after the November board meeting. As a result, the K-12/Higher Education Committeeintends to administer the study during the upcoming the fall 2015 semester. Once this project is permanently assigned, that group will decide its future as well as what assessment tool will be used moving forward and whether or not they will shift focus from the Information Literacy standards to the new ACRL information literacy framework.
Appendix A
Appendix B
Questionsfor which fewer than 50% of students selected all correct responses: 2013-2014
5. (1 point)
Consider the topic below as a possible topic for a five- to seven-page research paper. Indicate whether it would be a Good Topic, a Topic Too Broad for this paper, or a Topic Too Narrow for this paper.
Explain why foods fried in transfats have little nutritional value.
*a. Good Topic
b. Topic Too Broad
c. Topic Too Narrow
10. (1 point)
Your group has selected the Arctic National Wildlife Refuge in Alaska as a research topic. As you are researching, you read a fantastic quote attributed to attorney Paul Achitoff of Earthjustice. What should be your next step?
a. Discuss the appropriateness of the quote with the members of your group
*b. Verify that the attorney works for Earthjustice and learn more about the organization
c. Use the quote in your group’s research project because the quote supports the group’s argument
11. (1 point)
You are responsible for writing a paper on the production of electricity in your state. Which resource is least likely to have biased information:
*a. Environmental Protection Agency (
b. A recycling organization (
c. A newsletter written by an electric company
d. The Greenworks Gazette, a sustainability newspaper
e. A coal company (
13. (1 point)
You consult a bibliography to get ideas for resources to write your paper. Within the bibliography you find the citation below. In order to know where in your library to look, you must determine what type of resource this is.
Citation: Gertz, Bill. "Depressions, Recessions, and Inflation." Economic Policy Review 24 (2012) 11: 28-39.
The MLA-style citation refers to what type of resource?
a. book
b. chapter within a book
c. encyclopedia article
d. magazine article
*e. scholarly journal article
17. (1 point, .3 for each response)
You have just been assigned “the influences of Greek and Roman architecture on 19th and 20th century American architecture” as a research topic in your art class. Which of the following would be the best individual(s) to help you focus your topic? [CHOOSE ALL THAT APPLY]
*a. Art professor
b. Interior decorator
*c. Architect
*d. Librarian
e. Bricklayer
19. (1 point, .5 point for each response)
Read the two paragraphs and answer questions 19, 20, and 21
Author A
The latest studies indicate that death rates for smokers are two to three times higher than for non-smokers at all ages. This means that half of all smokers will eventually die as a result of their smoking. If current smoking trends persist, nearly 9% of the world's population will eventually die as a result of tobacco.
Author B
Many people still do not believe smoking causes serious health risks. In addition to respiratory illnesses, secondhand smoke causes 3,000 cancer fatalities each year. If people continue to smoke at the current rate, a huge number of these smokers will bring about their own death. At least five out of every ten will die.
Fact: 50% of people who smoke today will die from it.
Question: Is this fact in Author A's paragraph or Author B's paragraph? [CHOOSE ALL THAT APPLY]
*a. Author A's paragraph
*b. Author B's paragraph
22. (1 point)
The excerpt below comes from a cell phone industry magazine. Does the conclusion illustrate fact,
opinion, or bias?
"A recent survey by the Association for Wireless Telecommunication (AWT) stated that over 114
million people now subscribe to cell phone service. They reported that of the 699 collisions involving
cars with cell phones, 629 of the accidents were related to driver error. Even with this data, there is
no conclusive research at this time that links cell phone use with vehicle collisions."
*a. bias
b. fact
c. opinion
Appendix C