Multichannel Learning System (MLS)
Data Analysis and Collection Plan
Prepared By:
711th HPW/RHAS,
Office of Naval Research (ONR) Reserve Component (RC), and
Lockheed Martin International Training Team (LMITT)
24 September 2014
Table of Contents
1 Introduction 1
1.1 Document Overview 2
1.2 System Description 2
2 Administrative Information 2
2.1 General 2
2.2 Responsibilities 2
2.2.1 MLS Project Sponsor 2
2.2.2 U.S. Air Force Research Lab (AFRL) Warfighter Readiness Research Division (711 HPW/RHA) 3
2.2.3 Office of Naval Research Reserve Component (ONR RC) 3
2.2.4 Lockheed Martin International Training Team (LMITT) 4
2.2.5 GIRAF PM Services GmbH 4
2.2.6 Cranfield University 5
2.2.7 Partner Nations 5
2.3 Deviations from the Test Plan 6
3 Release of Information 6
3.1 Assessment Data 6
3.1.1 Research Ethics & Informed Consent 6
3.1.2 Proprietary Data 7
3.1.3 Release of Information to the Press or Commercial Vendors 7
4 Concept Evaluation Activities 7
5 Evaluation Goals, Objectives and Analysis 9
5.1 Evaluation Criteria 10
5.2 Evaluation Approach 11
5.3 Test limitations and Uncontrolled Variables 11
5.4 Data Analysis 11
5.5 Data Analysis 40
5.6 Data Collection 40
5.7 Analysis Summary Questions 41
6 Reports 41
6.1 Final Reports 41
Figures
Figure 51 -MLS Data Collection Website 41
i
1 Introduction
The Department of State (DoS), Department of Defense (DOD) and other US Government organizations use training and education as a prime tool for achieving strategic objectives, establishing rapport, opening communication, building capacity and maturing relationships with Coalition and international partners. Training is the critical component in the International Security Cooperation Program managed by the Naval Education & Training Security Assistance Field Activity (NETSAFA), United States Marine Corps Security Cooperation Group (MCSCG), Air Force Security Assistance Training Squadron (AFSAT), United States Coast Guard International Affairs and Foreign Policy (USCG/IA), and the U.S. Army Security Assistance Training Field Activity (SAFTA). The Defense Security Cooperation Agency (DSCA) directs, administers and supervises the execution of the Security Cooperation Programs managed by NETSAFA, MCSGC, AFSAT, USCG/IA and SATFA and Defense Institute of Security Assistance Management (DISAM) provides professional education to the staffs of the aforementioned organizations and to the Security Cooperation Offices (SCO).
The Multichannel Learning System (MLS) is a NETSAFA, DISAM and MCSGC research project focused on identifying the best methods for providing distance education that would prepare international military students for a resident training experience in the United States. It will convert selected content from the International Military Student Pre-Departure Briefing (IMSPDB) into multiple formats to support individual learning approaches (i.e., e-publications, mobile applications, videos, and web-based). It has two goals, specifically, (1) identify the best methods for providing distance education in order to prepare international military students for a resident training experience in the United States, and (2) develop several content from the International Military Student Pre-Departure Briefing (IMSPDB) course into multiple formats to support individual learning approaches.
The project will evaluate the use of four learning formats (i.e., e-publication, mobile app, video and web-based) delivered via several channels (i.e., personal computers and mobile devices). The evaluations will (1) identify the best methods for providing distance education for international military students, and (2) evaluate the effectiveness of having multiple learning formats to support Security Cooperation Education and Training Program (SCETP) requirements.
A Concept Evaluation will occur in December 2014 and include both International Military Students (IMS) and International participants. The IMS populations derived from a data pull from the Security Assistance Network (SAN) Web Training database that identifies IMS are enrolled at US Military training sites in the United States during the Concept Evaluation timeframe. US/International partners will identify twenty (20) volunteers to evaluate the IMSPDB content. Each volunteer will evaluate two learning formats in order to assist in identifying the best methods for providing distance education and evaluate the learning effectiveness of the learning approaches.
The MLS Project has the endorsement of the Defense Security Cooperation Agency (DSCA), based on the project’s capability to ensure coalition partners have global access to the Military Services’ state of the art online education and training programs. At the end of this project, the U.S. will have an enhanced operational training capability with participating partner nations, and these capabilities will be integrated into other Security Cooperation Education and Training Programs (SCETP) managed by DSCA.
1.1 Document Overview
This document addresses the evaluation approach to determine if the research project met the intended research goals.
The goals of the MLS research study is to (1) identify the best methods for providing distance education for international military students, and (2) evaluate the effectiveness of having multiple learning formats to support Security Cooperation Education and Training Program (SCETP) requirements.
1.2 System Description
The Multichannel Learning System (MLS) Project will use selected content from the International Military Student Pre-Departure Briefing (IMSPDB). The content will be converted into four different formats (i.e., e-pub, mobile app, video, and web-based) to support individual learning approaches.
Each research volunteer will evaluate two different learning formats, take a demographic survey, and pre- and post-test questionnaire on information related to American and Military Cultures, Culture, Individualism, and Punctuality.
2 Administrative Information
2.1 General
This section describes the MLS Testing & Evaluation Team, the project sponsor, 711th HPW/RHAS, Reserve Component (ONR Program 38 personnel), Defence Academy of the United Kingdom (DAUK), Support Contractors and other US/International Government Personnel.
2.2 Responsibilities
2.2.1 MLS Project Sponsor
The Multichannel Learning System (MLS) is a NETSAFA, MCSCG and DISAM project focused on identifying the best methods for providing distance education that would prepare international military students for a resident training experience in the United States and developing content from the International Military Student Pre-Departure Briefing (IMSPDB) informational course into multiple formats to support individual learning approaches (i.e., web-based, mobile applications, videos, e-Publications, etc.).
2.2.2 U.S. Air Force Research Lab (AFRL) Warfighter Readiness Research Division (711 HPW/RHA)
The U.S. Air Force Research Lab (AFRL) Warfighter Research Readiness Division (711 HPW/RHA) is the US Government Lead for deploying the MLS Testing & Evaluation (T&E) Plan, and assessing the ability of the MLS Project to meet the goals and objective. The T&E Working Group will conduct an evaluation of the learning formats (i.e., e-pub, mobile app, video and web-based) in a multinational environment.
The 711 HPW/RHA will work with the ONR Research Component (ONR RC), all MLS Working Groups (i.e.,, Learning Content and Research Protocol), and GIRAF PM Services GmbH to ensure the Testing & Evaluation Strategy collects the necessary data to ascertain MLS’s ability to support the evaluation of learning formats in the multinational environment.
RHA’s responsibilities include:
a) Promulgate and make changes, as necessary, to this Test Plan;
b) Develop, in consonance with ONR RC and Lockheed Martin’s International Training Team (LMITT) both technical and non-technical evaluation measures;
c) Provide a test plan briefing during appropriate meetings and workshops;
d) Provide advice to the MLS Project Manager, including keeping all of the MLS Management Team, informed of any condition that may affect execution of this assessment;
e) Collaborate with LMITT and other US/International Organizations providing support to the Testing & Evaluation Working Group;
f) Coordinate a data collection and analysis strategy to ensure the evaluation approach is comprehensive-in-nature;
g) Assist in the preparation of the final analysis for inclusion in the MLS Final Report; and
h) Provide recommendations on how the lessons learned will support SCETP requirements.
2.2.3 Office of Naval Research Reserve Component (ONR RC)
The Office of Naval Research Reserve Component (ONR RC) will provide technical and evaluation advice and assistance to the 711 HPW/RHA, MLS Management Team and LMITT as co-lead on the Testing & Evaluation Working Group.
ONR RC will ensure the Testing & Evaluation Strategy identified collects the necessary data to ascertain MLS’s ability to support the m-Learning environment. ONR RC’s responsibilities include:
a) Participate in Testing & Evaluation Working Group meetings, teleconferences, and other activities required to determine the optimum statistical evaluation approaches;
b) Support the analysis and data identification procedures;
c) Conduct analysis of data collected; and
d) Support the development of Testing & Evaluation Briefings.
2.2.4 Lockheed Martin International Training Team (LMITT)
i) Perform as the Principle Advisor on Human Research Protection Program (HRPP) Issues by providing oversight on all data collection, analysis of the test results, and publish appropriate reports in accordance with the Human Research Protection Program (HRPP) Guidelines and the MLS Management Plan;
a) Develop the appropriate Informed Consent Forms and seek Institutional Review Board (IRB) Approvals, as required;
b) Provide input for test plan, scenario, and use case development;
c) Provide input to analysis, assessment plans, and reports;
d) Conduct analysis of data collected; and
e) Incorporate inputs from the Testing & Evaluation Team into the MLS Final Report.
2.2.5 GIRAF PM Services GmbH
GIRAF PM Services GmbH will administer and maintain the MLS Data Collection Portal to ensure the data collected during the evaluations meets US, EU and International Research Protocol Requirements. They will ensure that the data collect by Concept Evaluation participants is in an easy-to-use format for analyses purposes.
A Comma Separated Values (CSV) database will ensure Research Protocol Requirements are fulfilled; specifically,
- Safeguard the database to ensure tight controls in order to ensure no human subject uses another human subject’s PIN to gain unauthorized access.
- If a human subject tries to enter the system without the authorized PIN, Figure 2-1 will appear.
- The Data Collection Portal will check the PIN each time the human subject attempts to enter the portal to (1) confirm the PIN and (2) check the last survey question completed. This ensures that human subjects cannot change responses once provided.
- After the Concept Evaluation has completed, the CSV file will reviewed by the MLS Extramural Investigators before it is provided for analysis to the Testing & Evaluation Team.
- Once the Concept Evaluation analysis has been completed a comprehensive CSV file, containing the data from both Evaluation will be merged and reviewed by the MLS Extramural Investigators before it is provided to the Testing & Evaluation Team for a comprehensive analysis.
- At 180-days, after the comprehensive analysis and report has been delivered, a copy of the CSV file will be provided to DISAM, 711th HPW/RHAS, GIRAF PM Services GmbH, CU@DAUK and the Académie Canadienne de la Défense to support further investigation of the data with other training research projects.
2.2.6 Cranfield University
Participate in all Testing & Evaluation Working Group meetings, teleconferences, and other activities required for determining optimal testing and evaluation processes. The Cranfield University (CU) team are considered ‘subject matter experts’ in the field of Learning Assessments; therefore, they are a representative of the United Kingdom’s Ministry of Defence (MoD) Testing & Evaluation Team. CU’s responsibilities include:
a) Participate in all Testing & Evaluation Working Group meetings, teleconferences, and other activities required to determine the optimum statistical evaluation approaches;
b) Support the 711 HPW/RHA Testing & Evaluation Team in conducting the analysis and data identification procedures;
c) Complete the Collaborative Institutional Training Initiative (CITI) Human Research Protection Program (HRPP) On-Line Training Course since they are conducting additional research related to m-Learning;
d) Conduct analysis of data collected; and
e) Support the development of Testing & Evaluation Briefings, as required.
2.2.7 Partner Nations
Each of the international partners will participate in Testing and Evaluation Working Group. The international representatives will provide inputs on the development of the questionnaire as well as assist in evaluating the initial formats developed by DISAM.
Each international partner will also participate in the Research Protocol Working Group to support the development of the Informed Consent Form and apply to respective Institutional Review Boards (IRB) for approval to participate in the MLS Concept Evaluation.
2.3 Deviations from the Test Plan
711 HPW/RHA Project Lead is authorized to deviate from this Test Plan as the operational situations and Professional Military Judgment (PMJ) dictates as long as it aligns with the MLS Project goals and objectives, keeping the MLS Project Manager and the MLS Management Team updated on what changes have been made and potential impacts.
3 Release of Information
3.1 Assessment Data
The following provides the strategy on the release of MLS Concept Evaluation Assessment Data.
a) Operational assessment data will, as expeditiously as possible, be initially be shared only within the MLS Project Management Team;
b) The release of data will not be provided during the actual conduct of the evaluation until after a valid assessment can be made based on all data collected;
c) All data will be gathered in a means to ensure it protects the rights, privacy and confidentially of all human subjects; and
d) The MLS Final Analysis Report will be released within 90-days of the assessment period, and as part of the MLS Final Report.
3.1.1 Research Ethics & Informed Consent
The Testing & Evaluation Process will meet the guidelines established by the US Department of Defense Directive (DODD) 5400.11 (DOD Privacy Program, DODD 3216.2 (Protection of Human Subjects in DoD-Supported Research, SECNAVINST 3900.39D, 32CFR219 (Common Rule), Canadian Tri-Council Policy Ethical Conduct for Research , European Union (EU) Data Protection Requirements and UK Ministry of Defence JSP 536 (Research Ethics Committee).
a) Each participant must first register, complete a demographics questionnaire and take the pre-test.
b) Once registered, each participant will evaluate two different learning formats of the IMSPDB content (i.e., e-publication, mobile app, video or web-based). Each format contains information related to American and Military Cultures, Cultures, Individualism, Punctuality, Informality, Egalitarianism, Short Term Mentality and Daily Life in America. However, only the Culture, Individualism, and Punctuality are testable.