An easy guide to the Information Skills Survey
Introduction.
The CAUL Information Skills Survey can help librarians, academic programme developers, and faculty to get useful information about the information skills of their students, and to demonstrate the effectiveness of programme initiatives.
The survey is designed primarily for use with GROUPS of students – although it can be used for diagnostic purposes with individuals. For diagnostics, there are also additional survey questions available in the CAUL ISS administration manual. Normally these additional questions should be used for diagnostic work.
Why should we use the ISS?
If your university includes information literacy among a set of goals or generic outcomes of undergraduate education, then you have a basis for arguing that your institution needs to support the development of these skills – and of being able to demonstrate that the initiatives are having an effect. Each round of quality audits will ask for additional evidence of attainment of institutional goals, and therefore it will help your institution to prepare for its next quality audit if it begins to monitor the effects of attempts to improve information literacy outcomes.
There is something about numbers that people find convincing. Of course, numbers do not tell you everything there is to know, but when you have credible numbers to put beside other evidence, you have a basis for convincing senior managers in Universities that there is pay off for their support for initiatives to enhance the information skills of students.
When used with other sources of evidence (see below),the ISS allows you to collect evidence for undergraduates in the Faculty of Law, and in the social sciences and related areas.
Frequently asked questions
How can I use the ISS - I do not know anything about statistics?
You can use the ISS if you can add numbers together and you can divide by whole numbers.
Why can I rely on information from such a simple survey?
Information skills are complex and include a range of skills that are used together. This survey asks simple questions. It is perfectly reasonable to wonder why it can be useful, and of course it does not tell the whole picture. However, it is designed to make a valuable contribution to the broader picture.
The reason why the ISS works is that it is built upon the structure of the Australian and New Zealand Information Skills standards. A metaphor may help explain why it works. If you were to try to describe the complexity of the human body, you would have a sound basis if you could identify or describe the skeleton. It tells you about the height of the person, and an analysis of a skeleton can tell you about trauma, about gender and age.
In a similar manner, the ISS gives information about the structure of Information Skills held by a group. You can find out that information in less than 20 minutes, no matter how large the group. Of course, you will want to add more details from observations, but knowledge about the skeleton can give you a good start and can help you to focus further enquiries.
How do I know that the survey is reliable and valid?
There is a plain English explanation of why you can use the ISS with confidence in the executive summary of the ISS Technical manual. If you would like further information you can seek confirmation from a statistician in your planning office who can read and interpret the statistical summaries published in the ISS Technical Manual.
However you need to be aware of two conditions. The first is that you have to use your common sense to make a judgement about the particular situation in which you are working, because the validity of a survey depends upon the context in which it is applied. For example, the ‘law’ version of this survey should only be used with law students, including perhaps those doing business law or criminology. The generic version of this survey was validated using education students. Look at the items and ask if they apply to your students. The survey does assume that students do course work assignments, so if your course has no assignments then it may not apply.
How accurate is this survey?
We asked librarians to assess students’ information skills using the ANZ Information Literacy Standards. They interviewed students and observed them performing information skill tasks for an hour. We found thatstudent self assessments of their information skills using the ISS agreed with the assessments by Librarians. For the generic 20 item scale, a majority out of seven librarians agreed with the results of the student self-reports. This result was obtained notwithstanding some difficulties experienced by some librarians in making assessments due to confusion over the assessment procedure. For the Law version of the ISS, the assessments of 4 out of 5 librarians agreed closely with the student self-assessments.
What other evidence do I need to collect?
To decide if an information literacy programme has worked you should seek three separate sources of evidence. The ISS can be one of these three sources. You should in addition have evidence that the students actually undertook the programme (ie attended and did the exercises); and you should gather some evidence from items in the course assessment that indicate information skills are at the intended levels.
How to I estimate the information skills of a group of students?
The generic ISS will give individual scores across a range from zero to 60. The performance of a group of students can be calculated by taking the average score for the group. To find out the average you need only add the scores for each individual and then divide by the number of students.
Are students likely to think they know more than they do?
Some students may over-estimate their knowledge, but others under-estimate their knowledge. Provided you have a group of at least 15 to 20 students, the average scores for the group are a good guide to the knowledge held by the group as a whole. See the comments below about confidence in the group estimates.
How much of a difference in ISS scores do I need, to know if a programme has worked?
If you are estimating the level of information skills of a group of 20 students, given a reasonable spread of scores of at least 30 (range is 0 to 60) your mean difference estimate will be accurate within a margin of plus or minus 3.5 units. For a group of 100 students, the likely margin for error is plus or minus 1.5 units. These estimates can be relied upon 95% of the time. Hence, if you use the ISS before and again after a programme aimed to build the information skills across a group of 100 students, the average improvement needs to be just one and a halfpoints on a sixty point scale to be sure that there is a real difference. It is three and a half points for 20 students. I expect that you would want a more substantial improvement than that to justify your efforts, so this means that if you get a substantial improvement you can be confident that it is genuine.
Remember that you should use this information in conjunction with other evidence to increase your confidence in the result.
How much difference do I need to know that the programme has worked for an individual student?
The ISS is not designed for use to determine performance of individual students. Difference scores for individuals with any assessment tool, are always problematic, because the errors of estimation for the before and after scores are added together, and you do not have the benefit of smoothing possible individual fluctuations to reduce the size of the composite error. However, for the record, an individual will need to improve by 16 points for you to have confidence that a real difference is demonstrated. Please note that even high quality achievement tests will also require differences of this order to be reliable indicators of change for individual students. Most tests of achievement used in Universities have lower reliabilities and hence much larger estimation errors. I hope that this convinces you that it is much better to focus on the changes in averages for the group.
Are students likely to cheat?
Why would students want to cheat? Remember, you cannot use the ISS to give students a grade – it is not designed for that use. As long as they know that you are using their responses to assess the teaching programme, or to identify what they need to be taught, it will be very unusual for someone to give you false information. Furthermore, provided that you work with findings for the whole group, an individual who tries to mislead you will have very little effect on the overall averages.
Why should I follow the procedures in the administration manual?
Many factors can influence the way people respond to a self-report survey. The conditions under which the survey is administered can maximize the validity of the information, or destroy the credibility of the process in the eyes of the participants. Therefore the way in which the ISS is used affects the utility of the survey.
The statistical evidence on which we base the claims of validity and reliability are derived from use of the survey items in a controlled and specified manner. Therefore, if you follow the administrative guidelines, you can use the published evidence about the validity and reliability of the survey to support your case that your findings have credibility.
Why are the items in the ISS kept secure?
One thing we have learnt from interviewing students, is that the very process of answering the questions in the survey starts them thinking about how they use information. If the questions are freely available, and if students practice answering the questions before they respond in the manner proposed in the administration manual, then we can be sure that their answers will be different to what they would have been if they responded to the items afresh.
Can I use the same items before and after information skills training?
Ideally we would recommend that you use an equivalent form of the survey, but that is not yet available. Provided you are happy to evaluate the total effect of the training including the pre-training survey, you can re-use the same items. If your information skills programme is comprehensive, then the effect is likely to be large compared to the effect of the pre-training survey. Normally students will not even recognise that it is the same survey if there is at least a month between the two uses of the survey. However, if this is a concern, you can do what is called a retrospective pre-test. This process is outlined in the Administration Manual (second edition only).
Where do I go for further information?
There are people in your university who would welcome a chance to work with you on enhancing information skills, including librarians, academic development staff, and members of some Faculties. It could make a big difference to both the support you receive to get started and to the credibility of your efforts if you can gain support of a senior manager in your institution. People to approach could include the senior executive responsible for academic programmes, a faculty Dean, the chief Librarian, or the Head of the Academic Development Section.
Statisticians who work in your university planning department may be willing to advise you on how to prepare and conduct the survey and how to get the results processed efficiently and effectively using standard statistical packages like SPSSX.
Conclusion
You can use the ISS with confidence to help you confirm when your efforts make a difference, and most importantly to help convince University managers that backing your information skills initiatives are worth the money spent!
Updated 3 May 2006
1
An easy guide to the Information Skills Survey