End-User Quality of Experience Oriented Adaptive E-learning System
Cristina Hava Muntean and Gabriel-Miro Muntean
Abstract
In the context of new devices and with a variety of network technologies that allow access to the Internet, the providers of e-learning materials have to ensure that the users have a positive experience using their e-learning systems and they are happy to re-use them. Adaptive Hypermedia research aims to provide personalised educational material that ensures a positive learning experience for the end-users. However, user experience is dependent not only on the content served to them, but also on the user perceived performance of the e-learning system. This leads to a new dimension of individual differences between Web users: the end-user Quality of Experience (QoE). We have proposed a solution for Adaptive Hypermedia Systems (AHS) that provides satisfactory end-user QoE through the use of a new QoE layer. This layer attempts to take into account multiple factors affecting QoE in relation to the delivery of a wide range of Web components such as text, images, video, audio.
The effectiveness of our QoE layer has been tested in comparison to a standard educational AHS and the results of these tests are presented in this paper. Different educational-based evaluation techniques such as learner achievement analysis, learning performance assessment, usability survey and correlation analysis between individual student performance and judgment on system usability were applied in order to fully assess the performance of the proposed QoE layer. Results of the tests showed that the use of the QoE layer brought significant improvements in terms of user learning performance, system usability and user satisfaction with the personalised e-learning system while not affecting the user learning achievement.
Keywords
end-user QoE, adaptive hypermedia, e-learning, end-user perceived performance, learning performance
1 Introduction
Extensive research in the area of Web-based adaptive hypermedia has demonstrated the benefit of providing personalized content and navigation support for specific users or user categories. A comprehensive review on the developed adaptive hypermedia systems, the techniques used during the adaptation process and the applicability areas of these systems can be found in Brusilovsky. 2001 and Brusilovsky 1996.
Web users differ in skills, aptitudes, goals and preferences for processing accessed information. They may have different perceptions of the same content and performance factors. Finally, they may have special needs due to disabilities. Therefore, the Web-based Adaptive Hypermedia Systems (AHS) try to capture and analyse these user-related features in order to optimise the user experience with the Web site. A variety of AHS have been applied in the educational area, providing e-learning services. This research area has attracted huge interest due to its capability for facilitating personalized e-learning, its distributed nature and its simplicity of interaction. Several good examples exist in the academic community including ELM-ART II, AHA! and JointZone. These systems build a model of the goals, knowledge and preferences of each individual person and use this model throughout the interaction with the user in order to propose content and link adaptations, which would best suit e-learners. Lately, researchers started to integrate learning styles in the design of a AHS along with the classic learner's features. Several systems providing adaptation to users' learning styles have been created such as INSPIRE and AES-CS.
With the advance in computer and communication technology a variety of Internet access devices (e.g. laptop, pocketPC, PDA, mobile phone) have been launched on the market. The type and capacity of the access device, the network the device operates on, the available bandwidth, the state of the network that may very dynamically over course of session and the complexity of the Web pages delivered, all affect the quality of experience for the end-user. Thus, end-users of educational and training services expect not only high-quality and efficient educational material but also a perfect integration of this material with the day-to-day operational environment and network framework. In this context it is significant to highlight a new problem faced by network-based education over the Internet: providing a good level of end-user perceived Quality of Service (QoS), also called Quality of Experience (QoE).
Currently Adaptive Hypermedia Systems for Education (AHSE) place very little emphasis on QoE and its effect on the learning process. This QoE-unaware approach is perhaps unsuited to a general learning environment (Figure 1) where one can imagine a student with a laptop moving from a low bandwidth home connection, to a higher bandwidth school connection, and potentially to public transport with a mobile connection with a widely varying bandwidth connection. It should be noted that some adaptive hypermedia systems have taken into consideration some performance features (e.g. device capabilities, the type of the access, state of the network, etc.) in order to improve the end-user QoE. For example GUIDE system considers hand-held units as tools for navigation and display of an adaptive tourist guide. INTRIGUE, a tourist information system that assists the user in the organization of a tour, provides personalized information that can be displayed on WAP phones. Merida et al. have considered HTTP protocol, type of the access and the server load in the design of the SHAAD. However, these account for only a limited range of factors affecting performance and do not fully address QoE.
Figure 1. A New E-learning Environment
Therefore, adaptive hypermedia systems should also take into consideration QoE characteristics when the user profile is built and regularly monitor in real-time any change in the system that might indicate variations of QoE. These include changes in the user's operational environment and also modifications of user behaviour, which might possibly indicate dissatisfaction with service (such as an abort action). This would allow for better Web content adaptation that suites varying delivery conditions.
This paper presents an approach that introduces a new QoE-based content adaptation strategy that enhances the functionality of a classic adaptive hypermedia system and aim to improve the end-user QoE. The QoE-based enhancement (QoE layer) measures and analyses various factors that may affect QoE. QoE layer consists of different components (Figure 2) The Performance Monitor measures a variety of performance metrics in order to learn about the Web user's operational environment characteristics, changes in network connectivity between the user's computer and Web server, and assesses the consequences of these changes on the user's QoE. This information is synthesized in the Perceived Performance Model, which proposes strategies for tailoring Web content in order to optimise QoE.
In order to demonstrate the benefits of the proposed QoE layer we have deployed it in the open-source AHA! system creating Quality of Experience-aware AHA!(QoEAHA). In this paper we present results from subjective evaluation in the educational area. The goal of this evaluation was to assess the learning outcome, learning performance, system usability and user QoE when the original AHA! and the QoEAHA systems are used in a low bit rate home-like environment. The results indicated that QoEAHA significantly improves performance and user satisfaction with their experience. The usage of the QoE-layer did not affect the user-learning outcome.
2 Quality of Experience
The term Quality of Experience (QoE) relates to end-user expectations for QoS. QoE is defined by Empirix as the collection of all the perception elements of the network and performance relative to expectations of the users. The QoE concept applies to any kind of network interaction such as Web navigation, multimedia streaming, voice over IP, etc. Depending on the type of application the user interacts with, different QoE metrics that assess the user's experience with the system in term of responsiveness and availability have been proposed. QoE metrics include subjective elements and can be influenced by any sub-system between the service provider and the end-user. ITU-T Recommendation G.1010 provides guidance on the key factors that influence QoS from the perspective of the end-user (i.e. QoE) for a range of applications that involves voice, video, images and text.
In the area of World Wide Web applications, QoE has been also referred as end-to-end QoS or end-user perceived QoS. Measuring end-to-end service performance, as it is perceived by end-users is a challenging task. Previous research (Bhatti et al. 2000, Krishnamurthy et al. 2000, Bouch et al. 2000) shows that many QoS parameters such as download time, perceived speed of download, successful download completion probability, user's tolerance for delay, and frequency of aborted connections factor into user perception of provided quality. Measurement of these parameters may be used to assess the level of user satisfaction with performance. The interpretation of these values is complex, varying from user to user and also according to the context of the user task.
End-user perceived QoS has also been addressed in the area of multimedia streaming. Research such as (Blakowski 1996, Ghinea 1998, Watson 1997) assesses the effect of different network-centric parameters (i.e. loss, jitter, delay), the continuous aspect of multimedia components that require synchronization, or the effect of multimedia clip properties (i.e. frame size, encoding rate) on end-user perceived quality when streaming different type content.
In this paper QoE is addressed only in the area of Web-based AHS with applicability in education. Typical e-learning systems may involve a combination of text, images, audio and video, and their quality of service is based on the combination of all of these rather than any individual component. The educational context also has its own set of requirements and user expectations in terms of learning outcome and it is against these that user perceptual quality will be evaluated.
3 QoE-aware Adaptive Hypermedia System for Education
Starting from a generic architecture of an AHS that consists of a Domain Model (DM), a User Model (UM), an Adaptation Model (AM), and an AHS engine (Wu 2001) we have enhanced the system with a QoE layer that was presented in Muntean 2004a and Muntean 2004b. The QoE layer includes the following new components (see Figure 2): the Perceived Performance Model (PPM), the Performance Monitor (PM), the Adaptation Algorithm (AA) and the Perceived Performance Database (PP DB).
Figure 2. QoE-aware AHSE Architecture
3.1 Performance Monitor
The PM is in charge of monitoring and measuring in real time performance metrics which are then used to infer information regarding user QoE. the performance metrics include download time, round-trip time, throughput and user behaviour-related actions (e.g abort requests). The utility of a session (Bouch et al. 2000) is also calculated and reflects the fact that users become less tolerant to delay as time passes. Tests over high speed connections showed that a 10 sec download time was considered acceptable to 95 % of the participants during the first four Web page accesses, still acceptable for 80 % of the participants during the access of an extra 6 pages, but only for 60 % of accesses over the 11th page were still acceptable (Bouch et al. 2000). Similar tests were performed for download time values between 16 sec and 6 sec and for different type of connections. The conclusion of these results was that the download time should improve over the duration of a session in order to keep acceptable the client's experience with navigation on a web site positive.
The information gathered by the PM during the user access sessions is delivered to the PPM. The mechanism used to measure the performance metrics is based on filtering TCP packets that carry information and monitoring the signals exchanged by the HTTP protocol.
3.2 Perceived Performance Model and Perceived Performance Database
The PPM has the important function of providing a dynamical representation of the user perceived QoE. It models the performance related information in order to learn about the user operational environment characteristics, about changes in network connection and the consequences of these changes on the user's quality of experience. PPM also considers the user's subjective opinion about his/her QoE explicitly expressed by the user. This introduces a degree of subjective assessment, which is specific to each user. The user related information is modelled using stereotype-based technique that makes use of probability and distribution theory (Muntean 2004a) and saved in the PP database.
Finally, the PPM suggests the optimal Web content characteristics (e.g. the number of embedded objects in the Web page, the dimension of the based-Web page without components and the total dimension of the embedded components) that would best meet the end-user expectation related to QoE. PPM aims to ensure that the access time per delivered page, as perceived by the user, respects the user tolerance for delay and it does not exceed the satisfaction zone.
Based on a survey of the current research into user tolerance for delay, three zones of duration that represent how users feel were proposed in (Sevcik 2002): zone of satisfaction, zone of tolerance and zone of frustration. According to a number of studies (Bhatti et al. 2000, Bouch et al. 2000, Servidge 1999, Ramsay et al. 1998) on the effects of download time on users' subjective evaluation of the Web site performance it was indicated that users have some thresholds (user tolerance) for what they consider adequate or reasonable delay. A user is "satisfied" if a page is loaded in less then 10-12 sec, but higher values cause disruption and users are distracted. Any delay higher then 30 sec causes frustration. At the same time it is significant to mention that when the user is aware of the existence of a slow connection, he/she is willing to tolerate a delay that averages 15 sec but does nor exceed 25 sec (Chiu 2001).
3.3 Adaptation Algorithm
The objective of the Adaptation Algorithm (AA) is to determine and apply the correct transformations on the personalised Web page (according to the User Model) in order to match the PPM suggestions on the Web page characteristics. Two types of transformations are considered: modifications in the properties of the embedded components (presented as concepts in the DM) and/or elimination of some of the components. These actions are applied to those components the user is the least interested in as recorded by the UM. The work presented in this paper considers that the Web pages consist of text and images. Since images contribute with the largest quantity of information to the total size of a web page, in this work they were the only ones taken into consideration by this algorithm.
In order to match the PPM suggestion related to the total size of the embedded images, image compression is first applied and, if further reduction is necessary, image elimination is applied. Different compression rates (expressed as percentage) are applied to each image depending on: the total reduction suggested on the total size of embedded images, the image size and user interest in the image as specified in the UM. Thus, if a user is more interested in image A than image B, image A will be reduced less than image B. If one of the computed compression rates cannot be applied to an image (e.g. due to the fact that the quality will be lower than acceptable for the end-users) an image elimination strategy is applied. In the case when an image has to be eliminated, a link to the image is introduced. In this way, if a user does really want to see the image, the link will offer this possibility.The algorithm used for image compression/elimination for the tests in this paper is described in Muntean 2004c. Naturally, the quality of the image relative to its size will depend on the sophistication of the compression technique, as is the decision regarding user perception of the image quality. This is a subject of ongoing research.
Further extension of the algorithm may consider multimedia clips (audio and/or video) that could be embedded in a Web page. For this situation, techniques that involve size and quality adjustments for audio and video can be applied (e.g. for video compression techniques involving frame rate, resolution and colour depth modifications and respectively for audio silence detection and removal technique). These adaptation techniques are studied by the multimedia networking area and they are not addressed in this paper. In addition, a component elimination strategy may be replaced by one of substituting a less bandwidth intensive equivalent for the information eliminated. For example, if video clips could not be supported, images or a sequence of images could be sent instead.