Tracking and Reporting Learner Data Using Web Services:
An LMS Agnostic Approach

Brian Caudill
JCA Solutions

There is currently a need within the industry to track learner data using the SCORM standard data model in a way that is not dependant on any specific LMS. This implementation of an LMS agnostic environment for viewing, in real time, SCORM data reporting and tracking will accomplish 3 goals:

  1. This will increase the number of courseware vendors that use SCORM reporting to the fullest extent, knowing that the reports generated will be usable and accessible.
  2. This will separate the reporting from the LMS so that institutions can select the LMS that delivers training and manages learners in the way that they need while still being able to access advanced reporting, measurement, and tracking data.
  3. This will enable the stake holders of these e-learning programs to easily view and assess learner performance data giving them a reliable measure of training return on investment (ROI).

Project Goal

The goal of this project is to create a web service, which we have named Metis-WS, that is capable of aggregating data from the SCORM 1.2 and SCORM 2004 data models and creating beautiful, extremely usable, and customizable charts and graphs for reporting data generated during the course session. Metis-WS must be able to report on all data model elements used by SCORM 1.2 and SCORM 2004. It must also offer a “quick view” representation of the student’s progress throughout the course session for instructors to assess learner progress. Metis-WS will also function as a run-time data model debugger for viewing, in real time, run-time data transmitted between the LMS and the course.

Metis-WS must be able to work with any course and any Learning Management System. Metis-WS must be able to work in a cross-domain environment. The final product will offer both a hosted solution and a non-hosted, local installation. The non-hosted, local installation must be able to run on servers running on both Windows and Linux environments.

As this system is intended to be useful in a completely standalone environment, no direct dependencies exist between any LMS and the core infrastructure of Metis-WS or Metis-JS. Metis-JS will have binding layers allowing for specific LMS customizations to be tied to the product on a per-LMS basis. From the courseware side, there will be no limitations on the type of course that can be connected to this system, and pre-existing integration will be assisted by complete SCORM conformance, but not in any way limited to this.

Project justification

There is always a difference in the way the course session data is interpreted and reported by the LMS. SCORM is a standard for data transmission and not for data reporting. The first issue with SCORM was that there was no common code representation / interpretation of the logic written within the standard, especially when it came to Sequencing in SCORM 2004. To overcome this issue, several 3rd party vendors, Rustici, Xyleme, and Icodeon invented SCORM 2004 players that are plug and play services that can be plugged into any LMS to enable the proper communication between the course and the LMS. Using this same logic, it is our intent to create both a web service (Metis-WS) and a JavaScript API wrapper (Metis-JS) that provides real time data aggregation and reporting functionality that can be used by content vendors and Learning Management System vendors who are not happy with their current reporting capabilities.

Implementation

The Metis implementation will be a JavaScript file (Metis-JS) that is placed in the SCORM course and a web service (Metis-JS) that is accessible via HTTP. The JavaScript file will intercept the calls from the SCO and forward them to both the Metis Web Service (Metis-WS), where the data can be aggregated, and the LMS (or LMS Cloud service), where the data can be used for SCORM session management. This architecture allows developers to use the Metis system (Metis-WS and Metis-JS) regardless of their development environment and regardless of their deployment LMS.

Figure 1. SCORM defines the data model element names and values that can be used by the course (1) to communicate with the LMS (3). SCORM also defines a standard API object (3) for this communication. Metis will take advantage of this standard communication (1a) in order to intercept the data (1b) being communicated to the LMS and forward this data to the Metis web service (2).

Using the available API will allow the full SCORM data model for both SCORM 1.2 and SCORM 2004 to be communicated from the course to Metis.

Reporting

Creating Reports and Graphs

Reporting in this system is as flexible as the connected data you define for intake. Report wizards will attempt to guide you through a series of pre-defined or easily customized reports that are commonly needed in the SCORM environment for defining useful cross-sections of graphable data. In advanced mode, you can define more complicated aggregations not specifically structured in the data model.

Several graph types are available for data display, and the wizards will automatically choose appropriate types for known data aggregations. Advanced mode will allow you to customize the chart configuration, legend, axes, and other elements of the view.

Chart aggregation will allow for multiple filtered data sets to be plotted on a single chart, whenever appropriate for the chart type.

The Metis system will have 3 default levels of real time reports.

  • Dashboard – a “quick view” graphical representation of the data available for courses, SCOs, and students.
  • Student Watch – watch all data and data subsets for a given student
  • Full Data Mode (debug mode) – see all the data for all courses, SCOs, students.

For some types of data, the system will be capable of generating real-time reports and activity logs, available through the dashboard, certain modules, and the live activity portions of the system. This enables live instructor monitoring of exams as well as a variety of other integration points for third party products and monitoring solutions.

Dashboard

The dashboard is a “quick view” graphical representation of the data available for courses, SCOs, and students. This view is customizable with a variety of graphs and real-time data in a module-based style, and provides drill-down capability directly into most aspects of the system.

Student Watch

Watch each student in real-time as they proceed through the course. Watch what page they are on and what they are doing on that particular page with the with the “over the shoulder” view. This view lets the administrator or teacher see the students session in real time as if they were looking over the students shoulder while the student takes the training.

Heat-map

See where on the screen students have been clicking and what parts of the course they have been interacting with.

Over the Shoulder View

“Over the Shoulder View” provides data about the learner session in an interface that looks as if you were watching the training session over the learners shoulder. The output of this view is similar to that of a screen recorder application. This allows for real-time viewing of learner sessions within the course. The sessions are also archived so they can be played back later to see exactly what the user did while in the training session.

Full Data Mode

The user can also aggregate data into reports, however they like, using the report builder. All reports will be customizable:

  • By Learner
  • By Instructor
  • By Course
  • By SCORM Data Model Element
  • A subset of “most used” elements
  • interactions
  • objectives
  • status
  • score
  • session / total time
  • By Time Frame (day, week, month, year)
  • By Region
  • By IP
  • By Browser

We will also report all SCORM and connection errors so that there will be detailed reports on what SCORM data model elements returned errors from the LMS and what other issues occurred during the user’s session such as low bandwidth alerts and dropped sessions. SCORM error information will be collected using return values from GetLastError(). This will offer valuable information in debugging SCOs.

Outside of SCORM, we will also have the ability to track all the events that occur during the course session including:

  • onMouseOver
  • OnMouseOut
  • OnClick (right and left mouse)
  • Onfocus
  • Key press
  • Key up
  • Key down

Custom user events can also be specified and tracked on a per-course basis.

If these are going to occur and be tracked outside of JavaScript then, for example, within the FLASH plugin, then there may be a need to integrate the content with a customized version of Metis-JS in order for all of these events to be tracked.

There will be two (2) ways to access the data within the Metis-WS. One will be to log in as a registered user and view the data in a browser. In order to view data that has been aggregated into the Metis web service simply log into and view the courses that you are currently tracking. The other way to access the data within Metis-WS will be to use the public Metis-WS Application Program Interface (API) (Metis-WS API) to access the data, and then you, as the developer, can aggregate the data in any format you wish.

This JavaScript implementation functions by using AJAX which sends XMLHTTPRequests from the SCO to the Metis Web Service (Metis-WS).

It is also the intent to have a hosted and non-hosted version or Metis-WS. The hosted version would reside on JCA Solutions web servers and be accessible via HTTP. The second implementation, the local install, could be installed locally to the customers system. This would allow excellent integration into existing Learning Management Systems while maintaining the same look and feel of the reports.