Usability Comparative Report

531134-LLP-1-2012-1-BG-KA3-KA3MP

The SGSCC (Serious Games for Social &

Creativity Competencies)

Usability comparative report

Title
Deliverable No. / 5.1. / Workpackage No. / WP5
Work Package No. / 5
Status (D: draft; RD: revised draft; F: final) / F
File Name: / Usability comparative report

This project has been funded with support from the European Commission.

This publication reflects the views only of the author, and the Commission cannot be held responsible for any use which may be made of the

Information contained therein.

Version History table

Version no. / Dates and comments
1.0. / 06/29/15 / Aleksandra Đurović / Draft version
1.1. / 06/30/15 / Bojan Velev / Revised version
1.2. / 07/16/2015 / Aleksandra Đurović / Updated draft
1.3. / 07/18/2015 / Aleksandra Đurović / Revised according to received feedback of Manfred Pretis and Karel Van Isacker
1.4. / 07/30/2015 / David Brown / Revised draft
1.5. / 07/30/2015 / Aleksandra Đurović / Final version

1INTRODUCTION

This report sets out to describe piloting methodology and to provide the analysis of the results of usability and evaluation process. In addition, it will include country-specific findings as well as comparative analysis and general conclusions.

This report includes:

• Introduction
• Description of the piloting methodology
• Pilot findings
• Usability Evaluation Schedule of the Desktop Game
• Case studies
• General conclusions

2DESCRIPTION OF THE PILOTING METHODOLOGY

Piloting methodology was outlined in the Usability and evaluation planning document which was accompanied by ten appendices.

Specifically, usability, evaluation and piloting process consisted of two parts. The first part was the analysis of the effects of social competence training as a whole package (educational workshops and educational games (desktop and mobile)) on the development of social competences and creativity. The soft outcome measurement has been carried outusing the pre-test - post-test design, namely, through assessment of the difference between ratings of teachers/trainers and self-assessment of trainees before and after the training. Pre-test training has been used to establish a baseline against which the progress could be observed.

The second part of the piloting process consisted of the usability testing of educational games as well as evaluation of handbook/training materials and the quality of educational workshops. Throughout the piloting usability testing has been carried out in order to identify issues that may prevent the users from interacting successfully with the system. In this process the observational method was used together with the post-test session questionnaire which included questions designed to assess the causes of the failure in the games. Some issues and system and user related events were directly entered in an online bug tracker tool Red Mine ( while the others were directly reported to the developers of the games. The observed issues have been continually used to upgrade the games, so the feedback has been incorporated in the final versions.

In the following section evaluation documents that have been developed or adapted in order to carry out piloting process will be presented.

Three questionnaires have been developed for recording the profile of participants and description of the training (and they were translated in the local languages):

1.1.Background questionnaire (Appendix 1);

1.2.Formfor following implementation of social competence training (Appendix 4)

1.3.Questionnaire for evaluation of training sessions (Appendix 5)

Two questionnaires have been developed in order to estimate the effects of training on development of social competences and creativity (and they were translated in the local languages):

2.1.Questionnaire for self-assessment of social competence and creativity (Appendix 2)

2.2.Questionnaire for evaluation of social competence and creativity (Appendix 3)

Three questionnaires have been developed for evaluation of SGSCC outputs (learning and gaming material) (and they were translated in the local languages):

3.1Questionnaire for evaluation of the handbook and training materials (Appendix 6);

3.2.Post-test questionnaire, to be used after games testing (Appendix 7)

3.3.Engagement ladder concerning evaluation of the desktop and mobile games (Appendix 10).

In addition, Guidelines for observers (Appendix 8) and Recording consent form (Appendix 9) were provided, both translated in the local languages.They have been created as supplementary documents to ensure that the applied methodology follows the ethical guidelines and that the piloting process has been carried out under standardised conditions on each of the pilot sites.

All collected data could subsequently be entered in the Data Entry Form (excel).

2.1Description of the instruments

1. Background questionnaire (Appendix 1)

The Background questionnaire consists of questions regarding the frequency of using email and social networks, learning and playing games on the Internet as well as questions regarding possession of various devices (desktop computer, smartphone…). Conclusions regarding digital competency of the trainees can be indirectly made on the basis of the answers received from this questionnaire.

2.1.2.Formfor following implementation of social competence training (Appendix 4)

This form contains a table with colons reserved for the following information: date & place, a number of the exercise in the Handbook, a number of participants present and comments and evaluation analysis. The last section – Comments and evaluation analysis is partially based on the trainees’ answers on the Questionnaire for evaluation of training sessions, however, it also includes additional relevant information (such as informal feedback or observed nonverbal expression of the trainees).

2.1.3.Questionnaire for evaluation of training sessions (Appendix 5)

This questionnaire consists of four questions, aimed as a tool for the facilitation of the process of getting trainees’ feedback. It is created for group administration. Trainees answer how they like the exercises, their difficulty and usefulness.

2.1.4.Questionnaire for self-assessment of social competence and creativity(Appendix 2)[1]

This questionnaire consists of 10 questions and covers indicators of social competences and creativity. In order to adapt the instrument for trainees from disadvantaged groups, responses are designed as dichotomous, either “good” and “not good” or “easy” and “hard”. Three questions assess communication skills, two questions measure self-control while the conflict resolution, self-esteem, assertiveness and cooperation are assessed by only one item. The tenth question measures creativity.

2.1.5.Questionnaire for evaluation of social competence and creativity(Appendix 3)

This instrument consists of 37 items on a 5-point Likert response scale. It is designed for collecting ratings of teachers/trainers of the trainees regarding social competences and creativity. Answers ranged from “never” to “always”. Moreover, participants have had the option to choose the response “cannot assess”, which was needed in order to evaluate which aspects of social competence cannot be measured by the assessment of teachers/trainers.

Communication skills are assessed by ten items (for instance “Nods/smiles/says “Yes” to show he/she is listening”). Five items measurecooperation skills, such as “Attends to instruction”. Self-control is assessed by four items (e.g. Persists in tasks which require effort”). Conflict-resolution is measured by six items, for example, “Reacts aggressively when he/she is in a conflict situation”. Assertiveness is assessed by seven items, such as “Tells you when treated unfairly”. Finally, three items measureself-esteem(e.g. “Expects to succeed on the task he/she just started doing”). The additional two questions measure creativity. Four items are reversed (three in a category of assertiveness and one in the category of conflict resolution).

2.1.6.Questionnaire for evaluation of the handbook and training materials (Appendix 6)

This questionnaire consists of 10 items on a 5-point Likert response scale (ranging from “strongly disagree” to “strongly agree”). It includes questions regarding the coherency and organisation of the material and suitability for the work with trainees. Moreover, there is space for additional comments of the trainers.

2.1.7.Post-test questionnaire, to be used after games testing (Appendix 7)

This instrument consists of 27 items. Pictorial scale with five smiley faces, which correspond to a 5-point Likert response scale, was used. The items are divided into four groups: items regarding the engagement qualities of the games (6), questions regarding games’ usability (3), questions regardingthe game’s manageability (8) and questions regarding the graphics (10).

2.1.8.Engagement ladder (Appendix 10)

This instrument is adapted from the existing instrument[2]; changes were mostly in the scale being used and the instructions for completion. Engagement, as defined by the authors of the instrument, is multi-dimensional, and encompasses awareness, curiosity, investigation, discovery, anticipation, persistence and initiation. This instrument is in a form of a ladder, and observer writes the name of each participant in the part of the Engagement ladder which corresponds to the level of his/her engagement.

The following scale was used:4 - Fully Engaged: Completely engaged;3 - Mostly engaged: Engagement occurring the majority of the time; 2 - Partly engaged: Emerging engagement but unpredictable; 1 - Emerging & Fleeting: Low & minimal levels of engagement, some evidence of awareness and 0 - No Focus: Inattentive and unresponsive.

2.2Timeline of the piloting and evaluation activities

The chronological steps in evaluation of the social competence training were outlined in the Usability and evaluation planning document(diagram isin the appendix 1).

2.3Data analysis

In order to ensure that the collected data is informative and comparable, the Excel file for entering data from above-mentioned instruments has been developed. In the data analysis Excel and SPSS 20 have been used.

3PILOT FINDINGS

3.1Description of the Pilot Sites

Within this project, seven pilot studies have been conducted in order to carry out usability testing and evaluation. Generally, the piloting was organised in the period from December 2014 to July 2015. In terms of the number of trainees/testers it was planned in the project proposal that there would be 15 testers from every country and 20 from Bulgaria. It can be noted that this number has been surpassed in some instances.

In total, 304 trainees and 67 trainers participated in the piloting. Depending on the pilot site, training sessions were completed weekly, in every two weeks or monthly. The number of training sessions ranged from 4 to 30 (Table 1).Several different organisations from seven countries were included in the piloting process (Austria, Belgium, Bulgaria, Lithuania, Serbia, Turkey and UK) as well as trainers from various backgrounds and trainees with different types of disability and from various disadvantaged groups. The training materials and the process of training were adapted according to the needs of participants. The overview of the number of trainees and trainers in different countries as well as the number of training sessions can be found in the table 1.

Austria / Belgium / Bulgaria / Lithuania / Serbia / Turkey / UK
Number of trainees / 19 / 17 / 123 / 14 / 78 / 32 / 21
Number of trainers / 2 / 6 / 16 / 4 / 14 / 14 / 11
Number of training sessions / 4 / 15 / 20-30 / 15 / 7-13 / 13 / 5

Table 1: The number of participants and training sessions on different pilot sites

The pilot involved trainees from different age groups, however, the majority of trainees belonged to the age group 16-20 (56%) (Figure1). The trainees of 21-30 comprised 17.1% of the whole group and the age group 31-40 composed 18.1% of the trainees who participated in the piloting. On the other hand, trainees who were younger than 16 were less represented in the piloting (4.9%) as well as trainees in the age group 41-50 (3.6%). The smallest number of trainees were older than 50 (0.3%).

Figure 1: Trainees’ age group

In the piloting sample, there were more female participants (54%), however the difference in numbers of female and male participants is relatively small (Figure 2).

Figure 2: Gender of trainees

The pilot involved people with disabilities, which constituted the majority of the trainees’ group (70%) (Figure3)[3].The main target group were people with learning disabilities and they did compromise the majority of the sample. However, some of them have complex disabilities where the main disability is learning disability but it is also combined some other type of impairment. Also, people with Autism spectrum disorder, mild and severe mental disorder, visual and hearing impairmentand mental health problems were also included in the piloting. Apart from the people with learning disabilities, the people from other disadvantaged groups were also involved in the piloting, including students from families from economically deprived areas, one parent families and families with a history of domestic violence, minority groups and youngsters with deviant behaviour.

Figure 3: The type of disadvantaged group of the trainees

It should be also noted that the trainees predominately had some experience in playing games. In fact, 78.8% of 250 participants who tested the games had previously engaged with them.

Moreover, the trainers who were involved in the piloting were predominately specialised in working with people with disabilities and other disadvantaged groups.

After the general description of pilot sites, in the following section the more detailed description of participants and the process of training for each pilot site will be provided. For each pilot site, firstly the profile of participants (trainees and trainers) will be described and then the process of training will be outlined. It should be noted that all trainers received their training before the start of the social competence training.

PILOT SITE - AUSTRIA

PROFILE OF PARTICIPANTS

The profile of trainees

The pilot involved 19 trainees. Eleven trainees belonged to the age group 16-20 and eight trainees belonged to the age group 21-30. Moreover, ten trainees were male while nine of them were female, which leads to the conclusion that there was an equal participation between male and female trainees.

All trainees have disabilities and in this respect they represented a heterogeneous group. Specifically, their disabilities range from mild learning difficulty, a mental health problem to autism spectrum disorder.

The majority of the trainees who participated in the testing of the games have a desktop computer or a laptop (79%), which is not the case with a smart mobile phone or a tablet (68.4%). Furthermore, the majority of trainees have had some experience in playing games (74%) while 42% play them daily. Taking this into account, it can be concluded that ICT skills of the trainees and specifically their skills related to playing games, are at the level which enables them to engage with the games in a productive way.

The profile of trainers

The pilot involved two trainers. One trainer was specialised in working with people with disabilities (people with combined developmental disorders and hyperkinetic disorders, delay of typical development and genetic disorders) while the other trainer was not specialised in this type of work.

DESCRIPTION OF TRAINING

There were four training sessions in total, divided into weekly sessions. The first training session included information about the project, exercises regarding the identification of emotions and evaluation and reflection. The second session entailed repetition of the lesson learnt on the previous session, exercises regarding body language and listening, playing the mobile game and evaluation and reflection. The third session included repetition, exercises regarding the initiation of conversation, using illustrated handbook and desktop game and evaluation and reflection. The last session was mainly dedicated to repetition and discussion of open issues.

PILOT SITE – BELGIUM

PROFILE OF PARTICIPANTS

The profile of trainees

Seventeen trainees participated in the pilot. Nine trainees were male (52.9%) while eight trainees were female (47.1%). With reference to the age of trainees, it should be noted that the majority of them belong to the age group 16-20(41.2%) and 21-30 (35.3%) whereas the minority of them belong to the age group 31-40(5.88%), 41-50(11.8%) or 50 and higher (5.88%). All trainees have learning disabilities with complex disabilities (the main disability is learning disability but is some cases it is also combined with visual, motor or some other type of impairment).

While the majority of them have a desktop computer or laptop (82.35%), it is not the case with a mobile phone or tablet (47.06%). In addition, the large percent of them play games (76.47%).All trainees participated in the online training, which enabled them to engage with the games adequately.

The profile of trainers

There were 6 trainers, among which 50% were female. They have backgrounds in psychology, sociology and they had worked with people with disabilities before.

DESCRIPTION OF TRAINING

The training sessions were held between December 2015 and May 2015. There was an induction training with trainees which was followed by 14 training sessions in total. They lasted approximately 3 to 4 hours split in 45 minutes units. During this training, all exercises from the Handbook were completed. Specifically, mobile games were played by all trainees in the last four training sessions, whereas desktop games were played by 4 people in the last two training sessions. Moreover, additional materials such as videos were used throughout the training.

PILOT SITE – BULGARIA

PROFILE OF PARTICIPANTS

The profile of trainees

In the pilot 123 trainees participated from four cities: Plovdiv, Smolyan, Sliven & Syedinenie.The trainees were heterogeneous regarding age: 23.6% belonged to age group 16 -20, 29.3% to age group 21-30, 39.8% to age group 31-40 and 7.31% to age group 41-50. There were more female trainees (63.4%) than male trainees (36.6%).

The majority of trainees were trainees with disabilities (90.2%) whereas 12 of them (9.8%) were from disadvantaged group (youngsters with deviant behaviour). Specifically, the difficulties that the trainees have are mild learning difficulties, severe learning difficulties, hearing impairment and mobility impairment.

The large percent of trainees have a desktop computer (79%). In contrast, only 39% have a smartphone. However, it is important to note that the majority of trainees (73%) have at least some experience with games. The 25% of trainees play games daily.

The profile of trainers

The pilot involved sixteen trainers: psychologists,social workers, adult trainers and teachers. All trainers were specialised in working with people with disabilities (mild learning difficulties, severe learning difficulties, people with hearing impairment, and people with mobility impairment and youngsters with deviant behaviour).