USER BEHAVIOUR IN A VIRTUAL LEARNING MANAGEMENT SYSTEM, AND THE LESSONS TO BE LEARNT
Tay Kwan Ann, Jeffery
National University of Singapore (SINGAPORE)
Abstract
With increased focus on end user experience, it is extremely important to understand their usage behaviour and tune our systems to enhance their experience. Capturing usage information not only helps us understand their behaviour, it benefits other areas as well such as audits, investigation and capacity sizing.
In 2009, NUS built a user behaviour monitoring module into its own home grown Virtual Learning Management System (LMS), which allowed us to track the user's movement within the LMS and understand their usage pattern.
The results gathered busted a fair number of assumptions which we had over students' behaviour.and when we shared this information with other Institutes of Higher Learning, they were surprised as well.
The paper will showcase the expected as well as unexpected findings from this behaviour module as well as how we adapted our LMS to be more student centric.
Keywords: Learning management system, user behaviour
1Introduction
When the National University of Singapore’s (NUS) home-grown learning management system (LMS) – Integrated Virtual Learning Environment (IVLE) – was developed in 1998, functionality took precedence over form. This was de facto for several iterations of IVLE for several years.
In 2008 with the increased adoption of Web 2.0, it seemed we needed to update the IVLE interface and make it more current by incorporating Web 2.0 features such as re-sizable screens, drag-and-drop menus, menu customisation, RSS and selected collaborative features. This new IVLE Web 2.0 interface developed by the NUS Centre for Instructional Technology (CIT) was launched in July 2008. After the launch, we began to receive feedback that the old ‘Web 1.0’ interface where everything was static was better and easier to use.This prompted us to rethink our strategyof changing the IVLE interface to support Web 2.0 features.
From past experience, CIT found that online surveys usually do not tell the true story, so we decided to build a user behaviour monitoring module into IVLE to track the user’s movement within the LMS. This module has several measurement metrics which take into account date and time information for added correlation capabilities.
2Why User Behavior instead of user voice
CIT felt that instead of basing our requirements on the voice of the students and lecturers, it may be more practical to gain insight from their behaviour in IVLE and find ways to improve their experience with IVLE. Additionally users, who gave their time to provide good feedback, usually represent the minority. There are also the additional benefits which these user logs bring such as auditing, security investigation and capacity sizing.
2.1Student Behaviour AssumptionsFrom Students’ Voice
With the experience gathered over the years from speaking to students and from their feedback, CIT made the following assumptions about the student’s behaviour within IVLE. These assumptions provide us with the basis to decide if certain features are worthwhile to develop for students.
- Students prefer to quickly get their information from IVLE and get out as soon as possible. We estimate students would spend about 20 minutes on IVLE
- Students tend to access IVLE between 10pm to 2am.
- Having an eLearning Week generates substantially more load on the system due to the huge number of students accessing at the same time.
- Students tend to access IVLE from home or outside of campus rather than from within campusespecially after office hours.
2.2User Behaviour Measurement Metrics
CIT came up with 7 metrics to measure student behaviour. It is important to note that by themselves the metrics do not reveal much. More data can be mined by cross-referencing the same metrics during different time periods, or cross-referencing different metrics in the same time period.E.g. Total number of active users per hour and Total number of user activities per hour gives us an estimation of the number of activities each active user is performing during this hour, which in turn translates to productivity. The higher the number of activities, the more productive they generally are.
Additionally, by correlating these metrics with the academic calendar, we are able to gain more insight into the behaviour of our users e.g. what is their behaviour when school starts or during exams
The metrics are:
2.2.1Total number of user logins per hour
This metric measures the total number of successful user logins for each hour of the day. Multiple successful logins by the same user will be counted as such. E.g. If only user1 logged in successfully 4 times between 2pm to 3pm, the log will show 4 logins between 2pm to 3pm.
The main purpose of this metric is to give an indication of peak login periods along with their origin, whether it is within the intranet or via the internet.
2.2.2Total number of user activities per hour
This metric measures the total number of user activities for each hour of the day. A user activity is defined as a single action performed by the user. E.g. user1 views the forum and decides to post a thread, this is considered as 2 user activities for the time period.
The main purpose of this metric is to give an indication of the total number of user activities handled by the system along with their origin (intranet or internet).
2.2.3Total number of active users per hour
This metric measures the total number of distinct active users for each hour of the day. An active user is defined as a user who is currently performing an activity on IVLE. E.g. if user1 is viewing the forum between 2pm to 3pm, user1 is considered an active user.
The main purpose of this metric is to give an indication of the total number of active users.
2.2.4Average number of activities per user session
This metric measures the average number of activities per user session per day. The higher the value, the more activities each user does per session.
2.2.5Average duration per user session
This metric measures the average duration of each user session per day. The higher the value, the longer they stay within IVLE and use the available tools.
2.2.6Daily tools usage
This metric measures the usage of the various tools available within IVLE on a daily basis. It provides us with a more detailed breakdown of the user activity so that we can tell which tool generated the most activity during the time period.
3Findings
From the results gathered from the various metrics, it is clear that some assumptions which we held were no longer valid. Also certain interesting snippets of user behaviour surfaced when looking at the various charts.
3.1Students Prefer To Quickly Get Their Information From IVLE But TheyDo NotGet Out As Soon As Possible
Previously, it was thought that students try to spend as short a time within IVLE as possible. They will login to IVLE, perform a few actions e.g. view the forum, download files and then logout. However from the average duration per user session metric (Figure 1), we realized that on a monthly basis the average duration per session is about 55 minutes during school term and 30 minutes during vacation periods. This is slightly more pronounced in December, having an average session duration of 27 minutes.
Cross-referencing this with the average number of activities per user session (Figure 2) metric reveals that the session duration is independent of the number of activities performed.From April 2010 to August 2010, even though the number of activities is high, the average session duration fluctuates. During September 2010 to November 2010, even though the session duration is high, the number of activities per session is low.
We reason that with the trend of keeping both the internet and personal computer on for longer periods, students develop the habit of logging onto a site and leaving it on even if they are not using it.
Figure 1 – Average duration per session
Figure 2 – Average activities per session
3.2Sunday Is TheDay When Students Stay The Longest
Taking the average duration per user session metric based on the day of the week revealed that Sunday is the day when students stay logged onto IVLE the longest (Figure 3). There are minimal differences between the number of activities done during the various days of the week (Figure 4).
This can be attributed to students wanting to prepare for the coming week so as to maximise their learning experience. Alternatively, it could be that they are catching up on the previous week’s work and getting it done before it is due for submission.
Figure 3 – Average duration per session /
Figure 4 – Average activities per session
3.3User Activity Drops As The Term Progresses
Grouping the average number of activities per user session (Figure 5 and Figure 6) by month shows an interesting picture. The number of user activities drop significantly as the term progresses, by as much as 50% between the months of August to September.
This can be attributed to students getting used to the system in the initial phase and once they are used to the system, they are more efficient in getting what they want, resulting in a lower activity rate.
Figure 5 - Average activities per session /
Figure 6 - Average activities per session
3.4ELearning Week Does Not Have Significant Impact On Usage
NUS conducted several eLearning weeks. During the eLearning week, students are supposed to learn for 1 week using purely online tools. All lectures, tutorials and seminars will be done via online tools. One of the main applications used during eLearning week is IVLE.
Tool such as assessment, chatroom, forum, media and workbins were used to replace traditional teaching methods. A total of 260 lectures, 233 seminars and 1049 tutorials were replaced with an online equivalent for the last eLearning week done from 4th to 8th October 2010. Lecturers were given free rein on how they want to conduct their lessons. They could for example use the online chatroomto replace a tutorial session, or use the forum for a more formal replacement for the same tutorial session. They can also use desktop capturing software to record their lecturers and distribute it via the IVLE media and the accompanying lecture notes via the workbin.
The eLearning weeks were conducted on
- 5 to 10 October 2009 for the entire university
- 8 to 13 March 2010 for the faculty of business, and
- 4 to 8 October 2010 for the faculty of arts and social sciences.
The eLearning weeks are marked in figure 5 and 6, and we can see that there is no significant difference between IVLE usage during eLearning week and non e-Learning week periods. From both charts, the average number of activities done during the month where eLearning week falls (October) in is not much higher as compared to the non-eLearning week months (September and November)
3.5Peak Access
Consolidating the Total Number of User Activities Per Hour (Figure 7) metric, it seems our students generally access IVLE after 8pm daily. Additionally, with the exception of Sunday, they are also active on IVLE between 11am to 5pm. This access stems primarily within campus and can be attributed to them accessing IVLE during school hours.Figure 7 also indicates the type of access, access within the intranet is marked yellow and access from the internet is marked blue. We can see that during school hours, the activities done are more or less distributed equally between intranet and internet access. After school hours, the activities are done mostly thru the internet i.e outside of campus /
Figure 7 - Peak access by hour
3.6Workbin Is The Most Accessed Tool
From a grouping of the Daily Tools Usage(Figure 8) metric across the whole time period, the workbin tool is the most heavily used tool amongst all the tools available within IVLE. This is because the workbin is the main tool used to distribute notes, assignments and projects to students. It is also the tool for students to submit their assignments and project documents back to the lecturer.
Figure 8 - Consolidated tool usage
3.7Lesson Plan Is Consistently The Most Accessed Tool Before School Starts
From the Daily Tools Usage metric, we found out that just before the academic year starts, the Lesson Plan tool will be the most heavily used. The Lesson Plan contains information about the module’s lessons during the academic year.
Figure 9 - Tools usage on 5th January 2010 /
Figure 10 - Tools usage on 18th July 2010
Figure 11 - Tools usage on 1st January 2011
4Adaptations to IVLE
From the various findings highlighted in the previous section, the following actions were taken to make IVLE more student centric.
4.1Web 2.0 Interface On Hold
As the number of users wanting a Web 2.0 interface is small, and users are already used to the IVLE interface and experience, we decided to put the Web 2.0 interface redesign on hold. Additionally, the Web 2.0 prototype did not garner much interest. However we employed the development technology’s native Ajax capabilities to build some Ajax functionalities into IVLE.
4.2Desktop Client ForWorkbin submissions
Considering that the workbin tool is the most popular tool used within IVLE, CIT worked on a desktop application which is able to upload files directly to workbin. This is especially useful for students when they need to submit their assignments to their lecturers via the workbin tool. We are also in the progress of enhancing this tool so that students can use this application to download the workbin files as well.
4.3“Live” Update of New Announcements and Friends Messages
Given that the students are staying within IVLE longer, CIT implemented a live update of new announcement and messages from friends. With this feature, whenever there is a new announcement from the modules that the student is taking, or when there is a message from their friend, a small dialog will appear indicating the number of new announcements and messages from friends.
Moving the mouse over this will show a balloon tip with a worded description of the dialog as shown below.
4.4System Procedures
Bearing in mind the usage patterns, CIT changed our system management procedures to minimize any downtime during high usage periods. These include Sundays, after 8pm on weekdays, and just before school starts. This effectively reduces our downtime to the university’s vacation.
5Conclusion
From the findings, we can conclude that Web 2.0 interactivity within a LMS is a good to have, however whether or not it is implemented does not significantly affect the users’ usage of the LMS. It seems what the student need are ways to be more productive within the LMS, be it in terms of tools which reduces their dependency on the LMS or the LMS as a service being there for them when they need it and is always responsive to their requests.