An Integrated Intelligent Home Environment for the Provision of Health, Nutrition and Mobility Services to the Elderly
Deliverable
D2.4: Definition of the evaluation framework
Date of delivery: July 31, 2010
Contributing Partners: UWE, SMH, SYS, SMARTEX, CSEM, ROBOSOFT, LUT, ANNA, AUTH
Date: 30-July-2010 / Version: v4.1
D2.4: Definition of the evaluation framework1/49
Document Control
Title: / D2.4: Definition of the evaluation frameworkProject: / MOBISERV (FP7 248434)
Nature: / Report / Dissemination Level: Restricted
Authors:
Origin: / UWE
Doc ID: / MOBISERV_D2_4_v4_1.doc
Amendment History
Version / Date / Author / Description/CommentsV1.0 / 27-05-2010 / PC-S / Draft
V1.1 / 02-07-2010 / PC-S, HvdH / Draft
V2.1 / 26-07-2010 / PC-S, HvdH JPV / Draft + review of state of the art
V3.1 / 28-07-2010 / PC-S, HvdH,JPV, SD, DR / Technical Input from Partners
V4.1 / 30-07-2010 / All / Incorporating feedback from reviewer and partners
MOBISERVFP7 248434
D2.4: Definition of the evaluation framework 1/49
Table of contents
Executive Summary
Glossary
1Introduction
1.1Intended Audience
1.2Objectives for defining the evaluation framework
1.3Objectives of this Document
1.4Structure of this report
2State of the art – Evaluation studies
2.1Consideration of the user environment and participation
3Evaluation Methodology
3.1Evaluation Criteria
3.2Evaluation techniques
3.2.1Formative
3.2.2Summative
3.3Evaluation Venues for field trials
3.4Reporting of Evaluation Findings
4Evaluation Format and Foci
4.1Evaluation Criteria
4.2General Aspects to be evaluated for all functions as appropriate
4.3Function specific evaluation criteria
4.3.1System Requirement - Reminder to eat (F_1) Use_Case1
Sub-Functions
Evaluation Criteria
4.3.2System Requirement - Reminder to drink (F_2) – Use_Case2
Sub-Functions
Evaluation Criteria
4.3.3System Requirement - Food inventory (F_5) – Use_Case5
Sub-Functions
Evaluation Criteria
4.3.4System Requirement - Response to call for help from the user (F_6) – Use_Case6
Sub-Functions
Evaluation Criteria
4.3.5System Requirement - Encouragement for exercising (F_8) – Use_Case8
Sub-Functions
Evaluation Criteria
4.3.6System Requirement - Reminder for personal hygiene (F_9) – Use_Case9
Sub-Functions
Evaluation Criteria
4.3.7System Requirement - Facility for carrying things from one room to the next (F_10) Use_Case10
Sub-Functions
Evaluation Criteria
4.3.8System Requirement - Voice/Video/SMS via robot communication with friends and relatives (F_11)- Use_Case_12
Sub-Functions
Evaluation Criteria
4.3.9System Requirement - Diary reminder/management (F_12) – Use_Case13
Sub-Functions
Evaluation Criteria
4.3.10System Requirement - Responding to requests for Weather Information/News (F_13) Use_Case14
Sub-Functions
Evaluation Criteria
4.3.11System Requirement - A mobile screen connected to the front door (F_14) – Use_Case15
Sub-Functions
Evaluation Criteria
4.3.12System Requirement - A tele-medicine/self-check platform (F_17) Use_Case_18
Sub-Functions
Evaluation Criteria
4.3.13Games for Social and Cognitive Stimulation (F_18)
Sub-Functions
Evaluation Criteria
4.3.14System Requirements - Reporting to health professionals (F_19) Use_Case20
Sub-Functions
Evaluation Criteria
5Ethical Considerations
6Appendices
6.1Sample Evaluation Consent Form
6.1.1Usability Test Consent Form
6.2Kompai Preliminary Field Trials
6.2.1Aims of the evaluation
6.2.2Participants
6.2.3Evaluation tasks
6.2.4Initialisation procedures
6.2.4.1Issues to be considered
6.2.4.2To develop before the field test
6.2.4.3Preparation at the test site, before the test
6.2.5Methods and data
6.2.6Ethical issues
6.2.7Safety issues
6.2.8Practical details
7References
Table of Figures
Figure 1. ISO Definition of Usability......
List of Tables
Table 1. User Interface Aspects. from: Kirmani, S., Rajasekaran, S.
Table 2. Common evaluation criteria
Table 3
Table 4
Table 5
Table 6
Table 7
Table 8
Table 9
Table 10
Table 11
Table 12
Table 13
Table 14
Table 15
Table 16
Executive Summary
This document, D2.4: Definition of the Evaluation Framework, presents an in-depth plan for evaluation of the MOBISERV components.
This deliverable is the outcome of task 2.4, prototype and system usability evaluation framework definition. The aim of this task was to define a rigorous and effective framework that will take into consideration all the ethical and practical aspects of conducting user and expert-based usability evaluations. As part of this task we also identified and established appropriate usability criteria in this context and clear protocols and guidelines for ensuring that the outcome of the evaluation is reliable and valid and can be utilised effectively to make refinements and changes to the system being developed.
Our strategy for evaluating the MOBISERV system is based on a two level evaluation procedure: formative evaluation and summative evaluation. The formative evaluation refers to the usability evaluation of each component of the system by the date that will be available in terms of usability and acceptance so as to assure us that by the time each technological component will be integrated to the robotic platform will provide the necessary functionality and will be acceptable by the end users.
The users involved in the evaluation will be both secondary stakeholders and end users.
This evaluation framework augments and compliments the validation plan has been described in detail in D2.2 (MOBISERV Validation Plan). As part of the validation strategy, an extensive (two round) summative evaluation procedure of the integrated MOBISERV system
will be performed so as to investigate if the project meets all the project objectives (both technical and social) and the final produced system is “fit for purpose”.
This outcome of the evaluation conducted as defined in this document will be reported in D2.5 and D2.6.
Glossary
Term / ExplanationMOBISERV / An Integrated Intelligent Home Environment for the Provision of Health, Nutrition and Mobility Services to the Elderly
CON_x / Constraints id
SC_x / Scenario id
F_x / Functional Requirement id
F_x.y / Sub-Functional Requirement id
EV_x_Fx.y / Evaluation Objective id
PRU / Physical Robotic Unit
SHACU / Smart Home Automation and Communication unit
1Introduction
This section describes the intended audience, objectives and use of this deliverable.
1.1Intended Audience
This document will provide the MOBISERVconsortium with the evaluation framework defining the process and procedures by which the MOBISERVcomponents will be evaluated.
1.2Objectives for defining the evaluation framework
In order to ensure that all MOBISERV components are appropriate and usable for the target users for whom they are designed, it is necessary to carry out comprehensive evaluation which addresses all aspects relating to achieving the desired functionality and usability.
The evaluation framework defined in this document provides the structure, and theoretical and pragmatic details to enable the evaluation to be carried out in a scientific, methodical and reliable manner.
1.3Objectives of this Document
As stated in the DoW, this document has the following objectives:
- Defining necessary usability criteria for the components in a number of different contexts and for a range of end-users, with varying levels of different physical and cognitive abilities
- Defining a framework to ensure that usability aspects of prototypes are evaluated throughout the design and development lifecycle.
1.4Structure of this report
Chapter 2 provides a review of previous evaluation studies which provide the underpinning of the definition of evaluation criteria and methodology in Chapter 3.
Chapter 4 is the first draft identifying the mandatory functionality that will be implemented and the corresponding evaluation criteria that can be defined at this preliminary stage prior to development. This chapter will be updated in correspondence with D2.2 at months 13 and 24 where specific details will be available to plan detailed evaluation tests and parameters metrics. As an example of the planning that will be done, the plans for the preliminary field trial of the existing Kompai functionality has been provided in Appendix 6.2. Chapter 5 introduces the ethical issues that will be taken into consideration for evaluations involving user participation.
2State of the art – Evaluation studies
This chapter provides a review of the recent literature on evaluation studies for assistive technology developed for older adults. This literature review was undertaken to provide a theoretical framework to underpin the MOBISERV evaluation methodology and also provide insight into previously recognised and known issues. This will enable the consortium to build on existing research knowledge and expertise.
The review has been organised on the basis of evaluation techniques and gives a brief overview of how the technique was implemented, the context of the evaluation and the findings.
2.1Consideration of the user environment and participation
In their study of exploring how to enable older people to handle future interfaces, Kleinberger et al.[i]consider the challenges of creating interfaces for ambient intelligent systems for assistive technology and the advantages of using an Assistive Living laboratory in order to both train older people to use such technology and test for the acceptability amongst stakeholders and evaluate the technologies fulfilment of its requirements. The results gathered indicate that assistive living technology should be tested within a controlled laboratory environment and further evaluations conducted within the real world domain. Further results suggest that older people must be given time to be trained and informed regarding the use and application of ambient assistive technology, in order to provide unobtrusive assistance within their home. Older people are defined as the most demanding stakeholders who will reject even the most sophisticated and advanced systems if they do not meet their needs, are not accessible or usable.
This suggests that there is an implicit requirement that there should be a clear framework and scaffolding to introduce the technology and functionality prior to ensure a valid and fair evaluation.
Henkemans et al.[ii]investigated user centred design for smart home technologies for medical monitoring to aid independent living. They highlighted the need to involve the end user within the development of eHealth technologies in order to increase the acceptability of such products. Henkemans et al argue that a single focus on technological innovation for this technology without consulting and involving the user results with less consideration for the usability of such technology reduces the likelihood of such devices becoming adopted by older people. Consideration of cognitive and personal requirements of the individual is highlighted as pivotal issues for eHealth development and to take a more technological driven path without consideration for usability aspects of such technology is detrimental.
Henkemans et al recommend the use of a smart home laboratory in order to study and evaluate such technologies in contrast to a traditional lab-based testing environment, for reasons such as natural behaviour with the evaluation participants within a comfortable domestic backdrop. The benefits of using a Smart Home Laboratory in contrast to a traditional laboratory testing environment are listed below:
Awarehome
- Contextualising technologies to be evaluated
- Activities or tasks within the home are different compared to a traditional work environment such as an office or other work place which is the typical design and feel to a lab-based environment.
- A significant difference is the freedom of the user to decide what activities are undertaken, when, who is involved with these activities and how they are defined with the time and space of the home environment.
- The environment provides a means for studying and understanding older people from a wider perspective taking into account sensory, motor and cognitive capabilities and how these issues change over time and their impact on the wider social unit in relation to the environment.
- Reveals opinions, considerations and ideas regarding introducing and developing smart home technology with the end user’s home environment as it becomes available: “iterative evaluation of such technology may increase the probability of acceptance”
- Brings together a multidisciplinary group of researchers whose expertise covers a wide range of relevant knowledge and subject areas which can benefit the design of such technology
Demiris et al.[iii] conducted a participatory evaluation of “smart home” sensor applications within an independent retirement facility. The participants successfully guided the researchers though the evaluation and provided useful and insightful recommendations for the redesign of the system. Acceptability was very positive with interviews and observations used as the main methods as part of the participatory evaluation. Participants did not feel the technology was intrusive or constituted an invasion of privacy, but optimistically embraced the aims of such technology very encouragingly. The authors highlight how such technology will only be accepted by older people if it is seen to empower them and provide a sense of control and active participation with their health care needs. The participatory evaluation approach has been chosen in order for all stakeholders to benefit from an exchange of ideas in regards to the researcher gaining valuable insight and knowledge regarding how requirements and how to fulfil these aspects of the system and in providing the necessary introduction and training with the equipment within the participants own environment as opposed to a strictly laboratory and or simulated environment.
Early user involvement is vital, and as Eisma et al.[iv] stress, the need for participation or involvement of older people with the development of ICT related products targeted for this user group is vital. This paper argues that it is necessary for developers and designers of such systems, to establish and maintain relationships with a diverse range of end users. This study was a part of a wider project, Usable Technology for Older People – Inclusive and Appropriate (UTOPIA) which aims to discover what is different about older people and their relationship with technology and how we include older people most effectively in the development process. The researchers seek to develop and identify a methodology that addresses “user sensitive inclusive design”, an investigation into older people’s lives, their relationship with technology, appropriate means to discover this information and establish guidelines for designing technology and interfaces for older adults.
The methods used are questionnaires, focus groups, workshops and interviews. The results indicate that the use of technology declines with increased age however highly usable products decline at a lesser rate. Positive attitudes towards technology encourage the use of a specific device amongst older adults however this is dependent upon the benefits of such technology being clearly expressed to the end user. Negative attitudes and apprehension towards technology are created by common perception of age and technology, lack of awareness, belief that the product is not suitable for older adults and that the technology is too difficult to engage with.
This study raises issues regarding the means by which the relationship between the developers and older people are established (mail drops, questionnaires, visits etc). In particular the intention of events and the role of participants must be clearly understood, as well as terminology regarding the development life cycle, practical “hands on elements” are made available where possible and social aspects of group gatherings or meeting is encouraged. The authors emphasise that this information will be of great value to industry and lead towards the development of genuine highly usable ICT products that will truly benefit older adults.
Considering appropriate evaluation criteria is an important part of the process. Batavia et al.[v] argue for the need of a comprehensive criteria for assessing the whether an assistive technology satisfies the need of the end user. They employ a focus group procedure in order to identify and prioritize key issues and factors currently overlooked when assessing and evaluating assistive technology. An alternative or modified version of the Delphi method (structured group / panel of experts consulted in the forecasting of the evaluation). Both groups are divided between consumer experts suffering from long term mobility impairments and another group consisting of long term consumer experts with sensory impairments.
Banner et al. gave considerations given to the following experiences:
• difficulty in learning to use the device;
• difficulty in using the device;
• difficulty in maintaining the device;
• difficulty in repairing the device;
• interference with the individual's lifestyle and social activities;
• physical discomfort in using the device;
• physical danger associated with the device;
• social and/or psychological discomfort in using the device ; and,
• inability to afford the device, its repair and maintenance.
The results produced a progressive step towards design, engineering, selection criteria and the concerns of the participants identified and categorised into 17 general factors which will be applicable to MOBISERV. These are listed as follows:
- Affordability—theextent to which the purchase, maintenance, and/or repair of the device causes financial difficulty or hardship to the consumer.
- Compatibility—the extent to which the device will interface with other devices currently and in the future.
- Consumer Repairability—theextent to which the average consumer (or his or her personal assistant) can repair the device if broken, including whether special repair equipment is needed.
- Dependability—the extent to which the device operates with repeatable/predictable levels of accuracy under all conditions of reasonable use.
- Durability—the extent to which the device will continue to be operable for an extended period of time.
- Ease of Assembly—the extent to which the consumer (or his or her personal assistant) can easily assemble the device upon receiving it, including whether it is packaged conveniently.
- Ease of Maintenance—the extent to which the consumer (or his or her personal assistant) can easily maintain the device to keep it operable and safe, including whether it is easy to conduct all required maintenance, cleaning, and infection control procedures.
- Effectiveness—the extent to which the functioning of the device improves the consumer's living situation, as perceived by the consumer, including whether it enhances functional capability and/or independence.
- Flexibility—the extent to which the device is provided with available options from which the consumer may choose.
- Learnability—the extent to which the consumer, upon initially receiving the device, can easily learn to use it and can start using it within a reasonable period of time once assembled, including whether specialized training is required .
- Operability—the extent to which the device is easy to operate and responds adequately to the consumer's operative commands, including whether controls and displays are accessible and whether start-up time for each use is excessive.
- Personal Acceptability—the extent to which the consumer is psychologically comfortable when using the device in public (or in private), including whether the device is aesthetically attractive.
- Physical Comfort—the extent to which the device causes physical pain or discomfort to the consumer.
- Physical Security—the extent to which the device is likely to cause physical harm, including bodily injury or infection, to the consumer.
- Portability—the extent to which the device can readily be transported to and operated in different locations, including whether the length of battery charge and the size and weight of the device permit physical relocation.
- Securability—the extent to which the device can easily be kept within the physical control of the consumer to reduce the likelihood of theft or vandalism.
- Supplier Repairability—the extent to which a local supplier or repair shop can repair the device within a reasonable period of time, including whether replacement parts are readily available and whether the manufacturer must conduct repairs.
While not all of these aspects will have priority at the initial stages of the MOBISERV evaluation, it is vital to be mindful of these from the conceptual design stages as they will help the technical partners to make crucial design choices which will have an impact on these issues.