Evaluation of Document Requesting Systems: Final Report
i
Sue Lacey Bryant
North West Health Care Libraries Unit
Evaluation of Document Requesting Systems
An Opportunistic Evaluation of Two Document Delivery Systems to Inform the Strategic Case for a Document Delivery Service for the NHS
Volume one: FINAL REPORT
5
Evaluation of Document Requesting Systems: Final Report
ContentsExecutive summary / Page
1
1. Introduction / 3
1.1 The National Library for Health / 3
1.2 Building the business case / 3
1.3 The evaluation / 4
2. Regional Pilots / 4
2.1 South West / 4
2.2 North West / 5
2.3 Other national document delivery systems / 5
3. The evaluation process / 6
3.1 Conduct of the study / 6
3.2 The NHS copyright license / 7
4. Ideal service offer / 8
4.1 Comparison of the two systems / 8
4.2 Discussion / 10
4.2.1 The service offer / 10
4.2.2 Management requirements / 11
4.2.3 Technical features / 11
5. End user testing / 12
5.1 User scenarios / 12
5.2 Data on usage by end users / 12
6. Usability assessment / 14
6.1 Data on usage by library staff / 14
6.2 Usability criteria / 14
6.3 Findings / 14
6.4 Discussion / 15
7. Evaluation of system administration and management information / 16
7.1 Findings / 16
7.2 Discussion / 19
8. Cost benefit analysis / 20
8.1 Base line data on current practice / 20
8.2 Financial data from both pilots / 20
8.3 Discussion / 22
9. Risks / 23
10. Product summary / 25
11. Conclusions / 26
12. Recommendations / 27
Bibliography / 28
Volume Two: Appendices
5
Evaluation of Document Requesting Systems: Final Report
Executive Summary
Minervation and Sue Lacey Bryant conducted an evaluation of two document delivery requesting systems to inform the development of the strategic business case for the procurement of a national inter-lending and document supply system for the NHS.
1. Recommendations
The report highlights a number of lessons that may be learned from the two pilots:
1. The procurement of a national document delivery system for the NHS to be based on a robust product specification.
2. Should a decision be made to commission a cost benefit analysis to guide the decision to procure a national system, this will need to establish the costs of existing document supply systems, as well as comparing electronic services and including costed benefits to the user (both end-users and library staff).
3. Any further pilot study should be conducted on a systematic basis with a rigorous system for collecting data from the pilot sites. The nature of the data to be collected should be agreed in advance.
4. Comprehensive user testing of all potential systems should be conducted at the earliest opportunity.
5. Emphasis to be placed on the management of user expectations (staff and end-users).
6. Document delivery systems should support different types of user behaviour as motivation for making a request might not always come from searching online resources.
7. One possible solution is either for libraries to have web enabled OPAC using open URL Technology or procuring a single OPAC system.
8. Satisfactory risk management strategies should be put in place to address the issues identified above; and specifically in relation to:
· compliance with current copyright legislation
· operation alongside national core content databases
2. The pilots
The evaluation study, conducted between February and September 2006, compared the functionality and usability of two NHS document delivery pilots.
In the South West, the SWRLIN (South West Regional Library and Information) network piloted the Virtual Document Exchange (VDX) system from Fretwell Downing Informatics (FDI). VDX was piloted by staff from six libraries across the region.
The North West Health Care Libraries Unit (HCLU) piloted usage of Dialog elinks electronic inter-library loans system by library staff in ten sites across the region. Dialog currently supplies clinical databases to the NHS as part of the National Core Content service (NCC).
The South West pilot, and North West acceptability testing, were underway when the contract to evaluate them was awarded. NLH had yet to define its requirements for a national system, and the pilots were taken forward as a means of gaining experience of the usage of such systems, rather than shaped as robust research projects designed to capture the data needed to make a full assessment of the performance of these systems.
3. The evaluation
As a consequence of the opportunistic nature of these pilots, combined with the suspension of both the North West and the termination of the South West pilots in May 2006 the evaluation of these two products was necessarily limited. The North West pilot was suspended dues to concerns regarding compliance with copyright legislation; in the South West a combination of technical difficulties and staff dissatisfaction led to termination of the pilot. Significantly, it has not been possible to conduct a thorough evaluation of two key areas: user experience and cost benefits.
The user experience is arguably the most important element of any such system, critical to the delivery of potential benefits. Hence, the absence of meaningful user data is a significant gap in the assessment.
Further work to conduct a Cost Benefit Analysis to inform a decision on the introduction of a national document delivery system could be carried out using only financial costs and financial benefits. Alternatively, a value could be estimated for those intangible benefits of offering an improved service to users, and freeing up library staff from mundane tasks.
Nevertheless, a detailed evaluation of these two products has been carried out ensuring that these two NHS pilots have yielded invaluable experience on which NLH is able to draw in progressing towards commissioning a national document delivery service for the NHS.
4. The products
The evaluation found little difference between each system in terms of how far they meet the ideal service offer, and usability criteria. Different strengths and weaknesses can be observed.
5. Risks
The study revealed several areas of foreseeable risk that might adversely affect the implementation of a national document requesting system, which must be managed:
5
Evaluation of Document Requesting Systems: Final Report
· Importance of a clear product specification
· Reliance on ATHENS for authentication
· Application across organizational boundaries
· Inter-reliance on other systems
· Licensing agreements (e.g. copyright license)
· Interfacing with other NHS ICT systems (proposed NLH information button)
· End-user engagement
· Library staff engagement
· Project management
5
Evaluation of Document Requesting Systems: Final Report
1.0 Introduction
Minervation and Sue Lacey Bryant conducted an evaluation of two document delivery requesting systems to inform the development of the strategic business case currently being written for the procurement of a national inter-lending and document supply system solution for the NHS.
1.1 The National Library for Health
The vision for the National Library for Health (NLH) is to deliver an efficient, personalized library and information service to clinicians and patients, based on a highly integrated network of library providers, working within a framework of standards[1].
The NHS commits around £1.2 million per annum in direct costs for document delivery[2] and a further unquantified amount (possibly as much as 25% of the cost of administrative staff salaries) on indirect costs.
The NLH single search is beginning to integrate national, regional and local resources/services at a single (virtual) service point. There is now a need to ensure integration of virtual and physical resources to ensure best use of NHS purchased assets. Part of the integration of NHS library services is a requirement to modernize document requesting services that will enable the customers (end-users) to order an article from a local or national collection when an electronic copy is not available
1.2 Building the business case
The strategic business case for a national inter-lending and document supply system must establish the principles underpinning investment in a document delivery service for the NHS and include high level options for the service.
The NLH strategic business case lists the following benefits for a national electronic document delivery system for NHS libraries[3]:
o Supply services that save time, simple to use and access
o Equitable for all, cheap and easy to obtain documents
o Supports clinical governance
o Streamlines library services and links with IT systems
o Provides significant time savings for library staff
o Improved stewardship of NHS resources
o Saving of space used for library archives
o Reduced unit costs, containment of document delivery expenditure
1.3. The evaluation
The evaluation study was conducted between February and September 2006, comparing the functionality and usability of two NHS document delivery pilots: VDX and Dialog eLinks electronic inter-library loans (ILL) systems. No other document delivery systems were evaluated as part of this work.
2.0 Regional pilots
The two document delivery pilots described in this report, were both underway when the contract to evaluate them was awarded in early 2006. NLH had yet to define its requirements for a national system, and the pilots were taken forward as a means of gaining experience of the usage of such systems, rather than shaped as robust research projects designed to capture the data needed to make a full assessment of the performance of these systems. Both pilots were in the very first stages of testing, with a view to preparing for a wider roll-out within each region.
Given that the study is based on an opportunistic decision taken by NLH to evaluate existing pilots rather than the output of a research study designed to assess the usability, efficiency, effectiveness and impact of document supply systems, this report makes pragmatic use of the data available, and seeks to identify further data that would be helpful.
2.1 South West: FDI Virtual Document Exchange (VDX)
In the South West, the SWRLIN (South West Regional Library and Information Network) piloted the Virtual Document Exchange (VDX) system from Fretwell Downing Informatics (FDI). FDI also supply the technology for the federated search functionality of the National Library for Health’s Single Search Engine (SSE).
The original objective of the South West pilot was to test VDX via library staff only. Once staff were satisfied that VDX operated technically, the intention was to roll out to a specific cohort of users who were currently using Knowledge4health, North Bristol NHS Trust’s knowledge portal, which uses the same technology as NLH SSE.
VDX was piloted by staff of six libraries across the region and funding for this pilot came from the NKS.
VDX is used both in the UK, and internationally. The system is designed to be both a user focused resource and a system that reduces administrative tasks (such as the management of document delivery procedures and inter-library loans).
VDX is a web based interface. While inter-library loan requests can either be made by users from any PC with an internet connection, or mediated via the Library in the traditional way, the region decided not to permit end-user access to the document delivering systems, in the first stage of testing preferring to test a model closer to the current manual system of managing inter-library loans. The second, but unfulfilled stage, was to move to user testing.
VDX provides a document supply system that enables both electronic and printed information to be sent either to the library or directly to the user. VDX can search suppliers’ catalogues automatically and send requests for papers, books, etc not held locally to a range of local, regional and national locations.
In the South West the system works in conjunction with SWIMS, the South West Information Management System, a library management system that has been implemented across the region over the last three years (SWIMS is also an FDI product). VDX can search the holding of libraries on this system, and can match this information with library user information. VDX can also search suppliers’ catalogues and union lists of journals.
FDI has recently been incorporated by OCLC/PICA.
2.2 North West: Dialog eLinks
The North West Health Care Libraries Unit (HCLU) secured funding from the NKS to pilot Dialog eLinks. Dialog currently supplies clinical databases to the NHS as part of the National Core Content service (NCC). The North West customer acceptability tested Dialog eLinks by library staff in ten sites across the region.
Dialog eLinks has a web based interface, and can be used from any web-enabled PC. It links between records in the suite of clinical databases currently held within the NCC so that library users can retrieve references while searching and link straight to the full-text of articles in those journal titles purchased by the NHS (either locally, regionally, or nationally).
This pilot also sought to look at how Dialog eLinks allows users to make point-of-search requests for articles that are not available electronically, and to test the rules that determine the sources used, and charges made, to process requests.
The pilot was about to be rolled out across the rest of the North West when it was suspended. See 3.1.
2.3 Other national document delivery systems
Steps were taken to identify other programmes to implement document delivery. `
Public libraries in the UK are already making inroads into developing a national document supply system. Unity UK[4] is a next generation resource sharing and interlibrary loan service from The Combined Regions (TCR)[5] operating in the public library sector. It serves the South West, North West, East Midlands, North East, and Scotland. It brings together the union catalogues of The Combined Regions and LinkUK. This new service uses resource sharing technology from OCLC PICA (incorporating FDI) which also hosts and manages the service. They have expressed their willingness to explore the potential for partnering with the NHS to deliver a national service.
Elsewhere in the world, libraries from different sectors are collaborating on similar systems. See appendix one.
3. The evaluation process
The evaluation was undertaken in four phases:
Phase 1: Preliminary work
o Definition of an ideal service offering
o Definition of use case scenarios