Using Physical Activity Transitions to Reduce the Perceived Burden of Interruption by Wearable

Using Physical Activity Transitions to Reduce the Perceived Burden of Interruption by Wearable Devices

by

Kelly Jayne Whitacre

B.S., University of Colorado at Colorado Springs, 2005

A thesis submitted to the Graduate Faculty of the

University of Colorado at Colorado Springs in partial fulfillment of the

requirements for the degree of

Master of Science

Department of Computer Science

2009

© Copyright By Kelly Jayne Whitacre 2009

All Rights Reserved

This thesis for the Master of Science degree by

Kelly Jayne Whitacre

has been approved for the

Department of Computer Science

by

______

Dr. Sudhanshu K. Semwal, Chair

______

Dr. Xiaobo Zhou

______

Dr. C. H. Edward Chow

______

Date

Whitacre, KellyJayne (M.S., Computer Science)

Using Physical Activity Transitions to Reduce the Perceived Burden of Interruption by Wearable Devices

Thesis directed by Professor Sudhanshu K. Semwal

Ubiquitous computing devices have become integrated into the consumers’ everyday life, meant to enhance and enrich daily activities by providing automatic and autonomous support. Ill equippedto communicatewith humans, wearable devices proactively communicate through a bombardment of messages regardless of whether or not the user is available to receive them. Causing a disruption in his/her primary task, the message often leaves the consumer with feelings of frustration or anxiety.

This research effectively mediates the delivery of device generated notifications by offering only non-time critical messages during physical activity transitions. During activity transition, the user’s perceived burden is lower because they are actively changing their primary task to prepare for another. This work compares the user’s receptivity to messages delivered during activity transitions and of those delivered at random. An IPhone is augmented with an application that detects physical activity transitionsusing the data acquired from a built-in LIS302DL tri-axial accelerometer. The results suggest that context aware notification mediation can be used to reduce the perceived burden of aninterruption and to augment wearables with this type of mediation would be valuable to the consumer.

CONTENTS

CHAPTER

Table of Tables

List of Tables and Figures

1Introduction

1.1Motivation

1.2Problem Statement

1.3Approach

1.4Contributions

1.5Overview

2Related Work

2.1Cost of Interruption

2.2Modeling Interruption

2.2.1Breakpoints

2.2.2Detecting Interruptibility with Sensors

2.2.3Detecting Interruptibility with Machine Learning

2.2.4Detecting Interruptibility with Natural Breakpoints

2.3Activity Recognition

3Design

3.1Designing an iPhone 3G Application

3.1.1iPhone OS

3.1.2Limitations

3.1.3Designing Views

3.1.4Generating Messages

3.2Calculating Activity Transitions

3.2.1iPhone OS Accelerometer

3.2.2Determining Periods of Activity and Rest

3.2.3Decision Tree Algorithm

3.3Designing an Interruption Study

4Implementation

4.1Hardware Requirements

4.2Tools Used

4.2.1iPhone SDK

4.2.2Programming Language

4.2.3iPhone Developer Portal

4.3iPhone OS Application

4.3.1Determining Activity Transitions

4.3.2Generate, Store, and Deliver

4.3.3View Message

4.3.4User Survey

4.3.5Application Settings

4.3.6Transferring Data off the iPhone 3G

5Results

5.1Verification of Transition Detection Algorithm

5.1.1Evaluating Classification using Precision and Recall

5.1.2Evaluating Classification using ROC

5.2User Experiment

5.2.1Device Placement

5.2.2Control Group vs. AT Group

5.2.3Participant Interview

5.2.4Activity Transition Group

6Future Work

7Conclusions

Works Cited

Appendix A: Activity Transition Trials

Appendix B: User Experiment

Table of Tables

Table 51: Precision and Recall Metrics

Table 52: True and False Positive Rate

Table 53: Receptivity Values for Participant

Table 54: Receptivity Responses

Table 55: Receptivity Responses

List of Tables and Figures

Figure 31: iPhone OS Layers [App09]

Figure 32: Visualizing the Accelerometer Axes

Figure 33 Determining Postural Orientation [Kar06]

Figure 34 Activity Transition Classifier Decision Tree

Figure 41 Isolating the Bodily Component

Figure 42 Calculating SMA and Tilt

Figure 43 Determining Activity Transitions

Figure 44 Generate Random Messages

Figure 45 Store Messages in Notification Array

Figure 46 Deliver Notifications

Figure 47 View Before and After the User Takes Survey

Figure 48 SurveyViewController View

Figure 49 Application Settings

Figure 410 MSThesisReportViewController View

Figure 411 NSMutableURLRequest

Figure 412 PHP Script on Server

Figure 52:

Figure 53zxzx

1

Chapter 1

1Introduction

1.1Motivation

With the increasing popularity of wearable devices, interruptions fromthese devices generate newdemands onthe attention of the consumer. Whether it is e–mail, or telephone calls, upcoming meetings notifications, changes in the stock market or navigation directions, notices on a wearable computer can happen anywhere and at any time. Ubiquitous and mobile devices have become integrated into the consumers’ everyday life, as they do enhance and enrich daily activities by providing automatic and autonomous support. For example, a mobile device could provide a daily reminder to take prescription medicine or an convey an announcement of a newborn baby via a text message from a loved one; however, although proactive messages are at times welcomed, there are times when such interruptions are inappropriate or unwanted. For instance,a user receiving a newborn baby announcement during a critical work related task would most likely view the timing of receiving this message as inopportune. Even though most of the devices come with off or inactive option, many of us forget to turn our devices off while working.

Wearabledevices proactively communicate through a bombardment of messages thereby interrupting the consumer from his/her present task which is most often independent of the device [Che06]. Furthermore, if several devices access the user, this could lead to multiple and simultaneous interruptions that are insensitive to the user’s workload or context [Che04]. Studies have also shown that delivering notifications in this proactive manner runs the serious risk of interrupting the user’s ongoing task creating feelings of frustration and anxiety[Bai08].

Because consumers are not always available to receive messages and because ubiquitous and wearable computing devices are pervasive in nature, users perceive these notifications as disruptive. Disruptive notices can lead to feelings of information overload and “interruption irritability” [HoJ05]. Therefore, there is a need to minimize the perceived burden of proactive messages with the careful management of notifications. Management of notifications should consider the user’s availability by analyzing their context as well as the message content.

1.2Problem Statement

This work addresses the problem of managing the overwhelming interruptions which are generated by wearable computing environments around us. As pointed out earlier, applications such as instant messaging clients and Gmail mail client provide some basic mechanisms for managing interruptions with the communication of user availability; however, this may become inconvenient when a user is busy and neglects to update their status [Che06].

To address this problem, we examine a passive approach for wearable devices to deliver notifications at opportune times while measuring the receptivity of the message upon delivery. Having such a valuable measure, suggests that mobile devices may be improved by the effective mediationof device generated notification delivery.

1.3Approach

Because wearable devices proactively deliver messages that are meant to enhance and aid our everyday lives but often burden us with unwanted and detrimental interruption, a simple augmentation to the device is needed to allow it to recognize opportune times for communication. Motivated by the observation that a transition between two different physical activities may strongly correlate with switching tasks, a time where it is also noted to be of decreased mental load by Ho and Intille[HoJ04,HoJ05], and Kern and Schiele [Ker051,Ker05], this research augments an iPhone3G device with an application that mediates the delivery of non time-critical messages with the detection of four postural activity transitions—sitting to standing, standing to sitting, standing to walking, and walking to standing.A measure of the user’s receptivity of messages delivered during activity transitions and of those delivered at randomis also explored.

1.4Contributions

This work’s main contribution is an iPhone3G application built for the iPhone 2.2.1 operating system. This includes associated source code (implementation and NIB files), resources, Xcode 3.2.1 project and executable. Additionally, associated data files collected for the verification of physical activity transitions and the study of the burden of interruption will be available.

1.5Overview

We start with a discussion of related work in examining the effects of interruption in the field of the Human-Computer Interaction. Next, Chapter 3 presents design considerations for an iPhone application capable of detecting physical activity recognition. In Chapter 4, we discuss the hardware, tools, and implementation detailsrequired for the execution of an activity transition classifier (ATC) and the associated iPhone application. Chapter 5 explains the accuracy results of the ATC and presents a discussion of a set of iPhone application tests by users in a workplace setting.

1

Chapter 2

2Related Work

Over the last several years, there has been a proliferation of research focused on methods for endowing computing devices with an understanding of interruptibility and the perceived burden of interruption to the user. While many ofthe recent interruption studies concentrate on the personal computer [Iqb08,Wal04,Wil06,Wil08,McC03,Bai08]as the producer of interruption, studies have also focused on interruption by wearables[HoJ04,HoJ05,Ker05,Ker051,Oul04,Che06,Che04] device.In this chapter, we provide a discussion of all such studies.

2.1Cost of Interruption

Disruptive notices from wearables or Personal Computerscan lead to feelings of information overload and “interruption irritability”[HoJ05,Wal04]. Delivering notifications in a proactive manner often interrupts the user’s ongoing task. For example, studies have shown that interrupting the user’s task at random moments can cause decreased performance on the task at hand as well as feelings of frustration and anxiety [Bai08].

Furthermore, these interruptions commonly associated with delays, errors, mistakes, and frustrationcan carry with them an additional social and cognitive cost [Bai08,Oul04]. In a social interaction, interruptions delay and distract natural flow of human-to-human conversation and can render actions of people incomprehensible [Bai08]. Similarly, in the cognitive sense, switching attention between information sources or tasks costs at leastseconds, hampering recollection by causing memory to be more susceptible to omission and distortions[Bai08]. In addition, an interruption,which is unrelated to the tasks at hand, takes longer to process andreturn back to original task[Wal04,Iqb08]. The interruption also creates an added memory load making it difficult for the task to be resumed,as user must remember what task was being processed before the interruption [Wal04].

Although the effects of interruption are more often thought of as negative [Wal04] influence on the work being done, there is some evidence that interruption canincrease efficiency, productivity, prevent errors, and even influence behaviorif the interruption aids the current task[Oul04,Wal04] in some way.

2.2Modeling Interruption

Two key factors contribute to the user’s receptivity of interruption. The first includes the selection of the most appropriate time for the user to receive interruption. The moment chosen to gain the user’s attention can drastically alter the user’s receptiveness towards the interruption [HoJ04]. Related to the timing of the notification, the utility of a message or the message contents may also be a factor at the user’s perception of burden of interruption. For instancea critical message might be better suited for immediate delivery, whereas a non-timecritical message might be better received if it was delayed to a later moment[HoJ05,HoJ04].

The medium of interruption or the method by which a message is deliveredhas also been observed to affect the receptivity of interruption[HoJ04,Wal04]. For example, consider an office worker engaged in a critical task. He/she might be more receptive to a SMS message alert rather than if the phone were to ring. Because aurally presented interruptions are thought to be acknowledged more quickly than visual stimuli and as a result more resistant to interruptions than visual ones [Wal04], the visual notification is less likely to disrupt the flow of the current task, perhaps lowering the perceived burden of the interruption[HoJ04]. Furthermore, thermal interruptions have larger detrimental effect than light on disruptiveness and performance. Additionally, motion as a notification system is effective compared with static items as traveling motions,such as a visual stimuli, are more disruptive than anchored motions [Wal04].

2.2.1Breakpoints

An emerging body of empirical research is now beginning to probe how manipulatingthe time at which a notification is delivered relative to the executionof the ongoing task impacts costs of interruption. Notifications have shown reduced interruption cost if delivered at moments of lower mental workloads or breakpoints. A breakpoint represents the moment of transition between two observable, meaningful units of task execution reflecting transitions in perception or action[Iqb08]. Recent studies in the field of Human-Computer Interaction have employed techniques to identify breakpoints with the use of sensors and sensors accompanied by machine learning. Moreover, the novel work of Wilson and Miller [Wil08,Wil06] in which the observation of natural breakpoints were exploited, is explored as the work takes the research in the direction of using sensors for pinpointing if it is a good time to interrupt or not.

2.2.2Detecting Interruptibility with Sensors

One approach to detect and differentiate breakpoints in user tasks is to use statistical models that map user interaction with physiological measures of the heart’s electrocardiogram and the brain’s electroencephalogram as shown in[Che06]. Similarly, McCrickard and Chewer [McC03] use physical or biomedical sensors to infer workload characteristics; yet, they are augmented with sensors that monitor eye gaze to detect breakpoints. Along the same lines, Bailey and Iqbal [Bai08]use an eye tracking system to collect pupil dilation data, a reliable measure of workload, to align task models for the determination of the best time to deliver notifications. Sawhney and Schmandt [Saw00]use auditory cues of current activities and conversations in the room to determine breakpoints.

2.2.3Detecting Interruptibility with Machine Learning

Additionally, machine learning techniques have been used to learn relatively accurate models for predicting the interruptibility of users. This is accompanied by the incorporation of sensor data which the techniques quite rely on. For example, Horvitz et al.[Hor08,Hor03]monitors the activity of a user interacting with different client devices augmented with event sensing and abstraction which sense computer events from the operating systems and applications executed on their clients accompanied bya microphone to report visual pose and nearby conversation, a camera to capture nearby conversations, and online appointment information to extract meeting properties. With sensor data, a Bayesian head tracking system, audio signal processing, learned models, and decision graphs Horvitz et al. [Hor08,Hor03]is able to delay a message during higher mental loads until a user is more susceptible for delivery. Furthermore, Gievska and Sibert [Gie05]created a framework based on an interruption taxonomy and Bayesian Belief Networks learning algorithm to aid interface designers selecting the most appropriate timing for interruption.

Kern and Schiele [Ker051,Ker05]use a Bayes’ rule learning technique for the classification of activity transitions, showing that the period of activity transitions is one when mental load is low, with the aid of acceleration, audio, and location sensors. Ho and Intille[HoJ04,HoJ05] also rely on activity transitions to mediate interruptions; however, Ho and Intille use accelerometers as its sole sensor to classify activity transitions and a C4.5 decision tree learning algorithm to deliver messages during transition. Our work also uses the accelerometer to identify the activity transition as is described next.

2.2.4Detecting Interruptibility with Natural Breakpoints

Alternatively, rather than attempting to build a model of the user’s mental workload, slow-growth notification—a novel method of notifying users through the use of gradual changes in the visual representation of a message, relies on the user’s own internal interruption model. Because the point at which the user tends to notice the slow-growth notification corresponds to a time at which they are more ready to be interrupted, Wilson and Miller [Wil08,Wil06]exploit these naturalbreakpoints with slow-growth notification to reduce the perceived burden of interruption.

2.3Activity Recognition

This work employs techniques presented by Ho and Intille[HoJ04,HoJ05], and Kern and Schiele [Ker051,Ker05]using activity transitions as an opportune time to deliver messages to the user. Similar to Ho and Intille [HoJ04,HoJ05]an accelerometer is used to collect movement data to aid in the classification of activity transitions; however, Ho and Intille [HoJ04,HoJ05]used two accelerometers in their study, one ankle mounted and the other thigh mounted.

Chapter 3

3Design

To ease the burden of the sensors and the ubiquitous device on the user and to promote use, a single triaxial accelerometer built into the waist mounted wearable is employed. As a result, the iPhone OS was chosen as the wearable device to augment with an activity transition classifier. Also similar to Ho and Intille [HoJ04,HoJ05], and Kern and Schiele [Ker051,Ker05], a machine learning algorithm is augmented with accelerometer data to determine an activity transition. However, unlike Ho and Intille [HoJ04,HoJ05], and Kern and Schiele [Ker051,Ker05], a basic decision tree learning technique is used as the decision-support aid. This is because the data describing the user’s context is discrete and the memory on the iPhone device is limited. In this chapter, we discuss the how we design an iPhone 3G application with a focus on the iPhone OS and limitations that come with it.,

3.1Designing an iPhone3G Application

3.1.1iPhone OS

The iPhone OS is comprised of an operating system and technologies used to create advanced applications that respond to touch events and display high quality graphics[App09]. It is currently supported on only two devices—iPhone, an internet-connected multimedia smartphone[1],and iPod Touch, a portable media player, personal digital assistant, and Wi-Fi mobile platform[2]. Acting as intermediary between the hardware and applications that appear on the screen; the iPhone OS interact with the appropriate drivers so the applications never have to interact directly with the hardware [App09]. In addition to the basic application infrastructure, iPhone OS gives you access to Multi-Touch events, the hardware accelerometers, camera features, and more [App09].

Figure 31: iPhone OS Layers [App09]

The iPhone OS features can be viewed as a set of layers, which are illustrated in [App09]. Most applications should be designed from the top layer down usinghigher-level frameworks over lower-level frameworks whenever possible. The Cocoa touch layer provides the key infrastructure needed for implementing graphical and event-driven applications via three frameworks, UIKit, Foundation, and Address Book UI. The frameworks provide a core set of features including application management, standard system views and controls support, graphics and windowing support, touch event handling, user interface management, access to acceleration data, camera features, user’s photo library, device name and model information, and support for text and web content [App09].