What do teachers, learners and other education advisors want from a web based Educational Portal?

Jocelyn Wishart and Caroline Oades

Paper presented at the British Educational Research Association Annual Conference, Heriot-WattUniversity, Edinburgh, 11-13 September 2003

Abstract

Educational portals aim to streamline access to resources on the Web, but how effective are they for their users? And how could they be improved? This paper will present the research methods involved in an evaluation study of educational portals in the UK and discuss recommendations resulting from the study. Highlighting perceptions among teachers, learners and other educational advisors towards personalisation features such as newsletters and tailored content, it will be of interest to those involved in the development or evaluation of websites for members of the educational community.

As a first step the authors conducted an extensive review of usability evaluations of educational portals across the world. A comparison of methodologies and findings was made and will be presented to inform others embarking on similar research.

With a wide audience encompassing teachers, learners, parents, governors, librarians and many others concerned with educational issues, educational portals have to cater for a vast range of differing information needs. Personalising websites to present the user with a view of the resources tailored to their needs is one possible way of making large portals more useable. This study investigated the potential of a variety of website personalisation features for the education community.

Initial results from task based assignments and focus groups among teachers, adult learners, librarians and other educationalists reveal they have a number of similar information needs together with a broad range of individual requirements of such an educational portal. The study also highlighted problems in that not only do people visit the portal with a variety of needs in mind but also in a variety of roles: teacher, parent, governor etc. The paper considers these in a discussion of methods via which such a web portal may offer optimal benefit to its users.

Additionally, the paper will address the pros, cons and complementarity of the research methods used - the online questionnaire, the task based assignment and the focus group – to obtain the required data.1Introduction

It is a paradox that, with an increasing amount of information available from an increasing number of sources and in a variety of different formats, one of the major problems for users of information lies in obtaining what they need (Ruttenbur, Spickler and Lurie, 2001). Additionally, the amount of time available to most people for information retrieval and use has decreased, which has brought about further difficulties in screening and digesting the information that has been found (Nicholas, 2000).

Time pressure is a particularly relevant issue for teachers, for whom non-contact and planning time is scarce. Recent studies of teachers in the US indicate that average blocks of time available for planning reduced from 45 minutes daily in 2001 to around 30 minutes in 2002 (Fitzgerald, 2001; Fitzgerald and McClendon, 2002). In the UK, an experienced teacher requires two to three hours’ preparation time per subject to write a weekly lesson plan, but with a lack of non-contact time, most of teachers’ planning is done outside normal working hours (Crace, 2002).

As such, conducting lengthy searches for resources of high standard and usability on the web is not given priority; instead, what is required is simple and quick access to quality material (Fullerton et al, 1999).

A number of educational portals have been set up across the world in an effort to provide this quick and easy access to quality sources. They have been subject to evaluation in a variety of ways.

1.1Review of methods for usability evaluations of educational portals

A number of countries have carried out evaluations of their national educational portals. Some rely on electronically recorded information, such as the DutchESS and Finnish Virtual library (Becker et al, 2000) which hosted online questionnaires; others on user feedback. For example, the Canada SchoolNet evaluation conducted by KPMG (KPMG Consulting LP, 2000) comprised an emailed questionnaire to 3,000 users (though only 216 replied), stakeholder interviews and six case studies.

Most combine the two retrieval methods. The UK Resource Discovery Network (RDN) evaluation (Coleman and Amber, 2002) for example, took the form of analysis of web site statistics, an online questionnaire, group interviews with users (both teachers and learners) as well as telephone interviews with staff responsible for implementing RDN within their organisation. Also within the UK, the most well known educational portal, the National Grid for Learning (NGfL), has been evaluated regularly through focus groups and questionnaires, although these studies (Becta 2001, Gravells, 2002) remain unpublished.

The most detailed usability evaluation found was the tripartite evaluation (Fitzgerald, 2000 and 2001 and Fitzgerald and McClendon, 2002) of the GEM gateway in the US. The initial evaluation comprised expert review by three professionals, review by Computer Science undergraduates (31), search tasks for teachers with little Internet experience (11) and an online survey which obtained over 1,000 respondents. The second comprised two focus groups, one of stakeholders (six members of the gateway consortium) and one of site users, (14 teachers), all of whom had used and applied GEM materials at least once. The third evaluation focused on the end users’ perspectives and comprised 51 teachers enrolled on a graduate programme. Each completed a project requiring the use of GEM resources, a written questionnaire and attended a focus group discussion. Three academic librarians also completed the written questionnaire.

The concept of web site personalisation first appears in some of the most recent of these evaluations (Coleman and Amber, 2002, Gravells, 2002) as a potentially desirable feature suggested by users.

1.2Potential of website personalisation for the education community

In a report of the results of the second annual evaluation of the Gateway to Educational Materials, the teachers’ requirements of an educational information gateway were summarised as ‘a system that will supply exact matches for all their topics, with minimal analysis or effort on their part…’ (Fitzgerald, 2001). Additionally, according to a recent report discussing the broad trends in the provision of educational gateways, commissioned by the Australian Council for Educational Research, two key elements of an educational portal are having both a strong sense of user group needs as well as some degree of interactivity (Lonsdale, 2002).

Thus an educational portal offering personalisation features would indeed seem to be an ideal resource. They would enable the portal to cover a vast array of teaching resources and yet deliver appropriate information to the variety of disparate user communities in education, ranging from adult learners and parents through to teachers, educational advisors and researchers.

Two methods of achieving personalisation to improve user experience of a web resource are a) gathering a user-defined profile of information needs and interests and b) tracking user activity to provide details of links that have been visited by others following a similar path.

There are challenges involved in using either method. Encouraging users to define their needs can be difficult when they are not convinced that the time required to set up a profile is worth the effort, thus the benefits of doing so must be very clear and the personalisation procedure very simple (Manber, Patel & Robison, 2000). According to Nielsen (1998), Amazon’s provision of recommended links is successful because it demands no extra effort from the user but even this method cannot be completely accurate. Technology is unable, for example, to match a user’s needs when they are attempting to find information on behalf of someone else.

Site-based bookmarks

Web-based bookmark managers have been available for several years, meeting the need for portable, direct access to favourite websites (Sherman, 1999). The facility to bookmark pages held on an individual website with open access however is barely mentioned in the literature. A recent conference paper makes a brief reference to the development at the Resource Discovery Network of its on-site bookmark facility for external websites (Smith et al, 2002). Known as the ‘Links Basket’, it allows the annotation, but not the storing of links. There is no indication of any recent research having been conducted to evaluate this feature amongst users.

Personalised e - newsletters

A benchmarking exercise involving 40 educational gateways worldwide identified ten gateways offering an e-mail alerting system of some form, the majority despatched on a monthly basis (Campbell, White & Babidge, 2000). Personalised e-newsletters, whose content is based on a user-defined profile, are becoming an important part of relationship marketing, yet over a quarter of them are not opened. (Nielsen, 2002).

A survey undertaken by SCHOLNET, a digital library for the scholarly community, identified that for reasons such as intrusion and control, together with portability, a preferred method for being alerted to new content would be to receive a personalised message when entering the website (2001).

Discussion groups

The benchmarking report mentioned above identified ten gateways offering discussion groups, with the number of groups hosted on each gateway ranging from three to 140 (Campbell, White & Babidge, 2000). The benefits of such groups to the user include the ability to obtain views and knowledge from a wider community than would otherwise be consulted (Crompton & Murchland, 2002). The development of such an online community on the NGfL, in this case specifically for those involved in the education of physics, is viewed by one author as one of the determining factors of the portal’s success in enhancing physics education (Fullick, 1997).

In providing discussion groups for professionals it is important to make new members feel part of the community. Some will become members of more than one group, but will not participate in all of them, instead choosing to ‘lurk’ and still benefit through the gaining of new knowledge from doing so. When starting a new discussion group, it is suggested that much effort be put into encouraging participation, with the facilitator stimulating activity. Structured activities, such as inviting online guests, can take place when community membership reaches a ‘critical mass’, suggested to be at 100 members (Bowes, 2002).

2Methods used

The aim of the research was to investigate the requirements of the different user groups who access educational web portals. Principal objectives were to:

  • evaluate the perceptions that different user groups have of portals
  • establish users’ reactions to personalisation features
  • determine users’ views on concepts such as online communities and discussion groups.

The research was conducted during January through April 2003 by a team of researchers from the Department of Information Science at LoughboroughUniversity.

An online questionnaire was hosted on one of the educational portals studied; however, it was felt that more qualitative feedback on attitudes to personalisation were required than could be obtained from such a questionnaire. Focus groups, as used by the more successful web portal evaluations described in the introduction, were deemed an appropriate supplement. Focus groups are used in order to gain rich data as group discussion helps to reveal consensus views and generates richer responses by allowing participants to challenge or reinforce one another’s views. In particular, facilitated focus groups of 8-10 participants should ensure that the voice of all participants can be heard in an informal and non-intimidating situation whilst providing enough scope for the generation of a range of opinions and discussion. In order to ensure that the groups remained ‘on-topic’, a focus-group question and discussion schedule was drawn up for each group and the group facilitators used this schedule to guide debate. Additionally, to ensure all members of the focus group were familiar with the portal personalisation features under discussion, computer based tasks involving their use were devised for completion by the participants.

2.1Pre-focus group exercises

Focus group participants were asked to complete two sets of exercises before taking part in the discussions. Exercise sheets were provided, onto which participants recorded their views. The aim of the first set, which was completed before arrival, was to gain participants’ initial impressions of a bookmarking feature that stored bookmarks to pages on the portal, mimicking as far as possible the experience of a user being presented with the feature for the first time. In the second, follow-up set of exercises, completed on arrival and before discussion, participants were asked to make use of a personalisation feature that tailored the site view to their role. Questions in both cases elicited perceptions of benefits as well as ease of use.

All comments and multiple choice answers were imported into Excel, where quantitative analysis was performed on the multiple choice answers and qualitative analysis of comments was undertaken by categorisation into three levels of classification.

2.2Focus group discussions

Participation in the focus groups was by self-selection. Advertisements for the focus groups were posted online and in local (to Loughborough) schools and an attendance payment was offered to participants. Five target groups were identified: teachers, librarians, adult learners, parents and governors.

A total of eight focus groups (including one pilot) were held over the period 26th February to 7th March 2003, with 51 participants in all. Of these, 38 were from within the East Midlands region and 13 from other regions in the UK. As can be seen in Table 1, educators accounted for 27 participants and non-educators 24. The educators group consisted of 25 teachers and 2 teaching assistants; four of the eight focus groups held comprised solely members of this user group.

Table 1. Composition of focus groups

No. / %
Teachers / 25 / 49%
Teaching assistants / 2 / 3.9%
Total educators / 27 / 52.9%
Librarians / 6 / 11.8%
Adult learners / 10 / 19.6%
Parents/governors / 8 / 15.7%
Total non-educators / 24 / 47.1%
Total participants / 51 / 100%

Although many of the focus groups were of mixed role composition, none contained an educator/non-educator mix. This was intended so that non-educators would not feel excluded when resources that were only of interest to educators were discussed. The duration of the focus groups was about one hour for non-educators and about 90 minutes for educators, who were asked a number of additional questions regarding resources aimed specifically at meeting their requirements.

The focus group question schedule ranged from use of the bookmarks to a personally tailored web site via discussion of other features that can be personalised such as tailored email alerts, newsletters and discussion groups.

In terms of proficiency in using the web to find information, the median average web proficiency of focus group participants was 3 (on a scale of one to four, where one was beginner and four was very confident). This data is based on self-ranked answers given in the associated online questionnaire. The group with the largest proportion of ‘beginners’ was the adult learners; parents, governors and librarians were proportionately more confident in their ability to find information on the web than the teachers and teaching assistants.

All discussions were minuted by an independent observer, recorded onto tape and then transcribed. Approximately 2000 comments were categorised. Categorisation occurred by importing the comments into Excel before classifying them into the main subject areas outlined in the focus group schedule. Further hierarchical sub-classification was then performed for each group of comments by assigning comments into themes such as usability, functionality, content and design. A final sub-classification of the comments was then made by drawing together consistent themes that arose such as clarity of design, complexity of the login procedure etc.

3. Results from focus group discussions

The results of the focus group data collection are presented in sections according to each of the web site features under discussion.

3.1 Bookmark feature

Following on from exercises involving the use of the bookmark facility, the groups opened with a discussion of its use, whilst the feature was still fresh in participants’ minds. Over two-thirds (69%) of all participants anticipated making use of the bookmark facility again; this rose to over three quarters (89%) of participants in the educator groups. Two educators perceived their bookmarks to be so useful that they expressed disappointment later in the discussion that they were not accessible from another of the sites managed by the same organisation.

This is very positive and reflects an appreciation of the benefits. The downside is that these benefits were not clear to participants before they had used it. Participants’ reasons for not bothering to register were a lack of time to do so; fear of receiving junk mail; a perception that it would not be worth the effort; and a reluctance to give away personal information.

Once they had used the bookmark feature however, participants identified the key benefits to them as being its portability, its potential to save them time as well as its ability to “organise my work with separate pockets for different pieces of information”. The view of one educator, that it “fine tuned the site”, was agreed by many in her group. Negative views principally came from participants who were unable initially to discern any benefit to them over a browser’s ‘Favourites’ facility.

Only one participant (a librarian) made reference to web-based bookmark managers such as that available on My Yahoo. This is perhaps surprising given that these offer the portability and organisational benefits welcomed by the participants. They offer the additional benefit of being able to bookmark actual websites themselves, a function that was not available via the trialled bookmarking facility, which allowed bookmarking of pages containing descriptions of useful sites and associated hyperlinks held on the portal itself.

3.2E-mail alerts vs personalised e-newsletters

Participants were first asked for their views on e-mail alerts, which were defined as ‘a short e-mail, sent to let you know that something new has been added to a website. It is received only by those who have indicated to the website provider that they wish to receive this information’.