Evaluation Framework for Pervasive Computing Systems 3
1 Introduction 3
2 Related Work 6
2.1 Sensor network 7
2.1.1 Sensor node and platform 8
2.1.2 Connectivity and Sensors Networking 10
2.2 Middleware 11
2.3 Applications 13
2.3.1 Bat Ultrasonic Location System 14
2.3.2 Smart homes 14
2.3.3 Exploratorium 15
2.3.4 e-Science 16
2.3.5 iHospital 16
2.3.6 Automotive SPUR 16
2.3.7 Outdoors deployment 17
2.4 Evaluations of Existing Pervasive Computing Systems 18
Evaluation methodology 18
2.4.1 18
2.4.2 Evaluation parameters 19
3 Taxonomy of pervasive systems 19
3.1 Architecture 24
3.1.1 Distribution 24
3.1.2 Mobility of distributed pervasive systems 25
3.1.3 Infrastructure of mobile distributed pervasive systems 26
3.2 Modulability 27
3.3 Geographic Span 28
3.4 Purpose 29
3.5 Integration 30
4 Reference Scenario 31
4.1 Classical scenarios 32
4.2 Comprehensive reference scenario 33
4.3 Scenario analysis 38
5 Evaluation Parameters 40
5.1 Aspects and Parameters 41
5.2 System-centric Parameters 44
5.2.1 Quantifiable System Centric Parameters 44
5.2.1.1 System performance 44
5.2.1.2 Communication performance and cost 44
5.2.1.3 Software footprint 44
5.2.1.4 Power profiles 45
5.2.1.5 Data storage and manipulation 45
5.2.1.6 Quality of context 45
5.2.1.7 Programming efficiency 46
5.2.1.8 Reliability and fault-tolerance 46
5.2.1.9 Scalability 46
5.2.1.10 Adaptivity and self-organization 47
5.2.1.11 Maintainability 47
5.2.1.12 Proactivity 47
5.2.2 Non-quantifiable Characteristics 48
5.2.2.1 Node-level characteristics 48
5.2.2.2 Service and Application characteristics <another name?> 48
5.2.2.3 Context characteristics 49
5.2.2.4 Security and Privacy 49
5.2.2.5 Economical considerations 49
5.2.2.6 Knowledge representation 49
5.2.2.7 Architectural characteristics 50
5.2.2.8 Adaptivity characteristics 50
5.2.2.9 Standardization characteristics 50
5.2.2.10 Projective and backward compatibility 51
5.3 User-centric Parameters 51
5.3.1 Quantifiable usability parameters through measurement 52
5.3.1.1 Effectiveness 52
5.3.1.2 Performance 52
5.3.1.3 Learnability 52
5.3.1.4 Efficiency 52
5.3.1.5 Error 53
5.3.2 Quantifiable parameters through surveys, focus groups and interview 53
5.3.2.1 Customization 53
5.3.2.2 Explicitness 53
5.3.2.3 Modality 54
5.3.3 Non-quantifiable Characteristics 54
5.3.3.1 Acceptance 55
5.3.3.2 Satisfaction 55
5.3.3.3 Usefulness 55
6 Discussion: Design Methodology of Pervasive Computing Systems 57
7 Conclusion And Future Work 58
Reference 59
Evaluation Framework forFor Pervasive Computing Systems:
AUTHOR 1
University of FloridaAffiliation 1
______
A More than a decade and a half after the debut of pervasive computing, plenty of prototypesprototype systems, sample applications, and interactive modalities have emerged. However, there is from research laboratories around the world. Researchers, however, found themselves lost amid all the developments, as they cannot reach a lack of consensus about the best approaches inwhen creating suchpervasive computing systems, or even how to draw fair comparisons, and the compare different proposals. The future direction of pervasive computing remains fuzzy at best. To address these issues, we developdefine a performance evaluation framework for pervasive computing systems. Based upon extensiveon extensively survey ofon published literature and our own experience in building the Gator Tech Smart House, we establish the taxonomy, a common reference scenario and identify case scenarios and parameters of interest. The taxonomy generates a reference framework when evaluating vastly diversified pervasive computing systems; the reference scenario serves as common case scenarios give researchers a tool to gauge identify the shortcomings of systemsthe designed system at the specification level; and the list ofidentified parameters allows for impartial evaluation of the implementations. To address the diversity and complexity of the pervasive computing systems, they are first classified into different categories which address and focus on different aspects of pervasive computing. Within each aspect, we further identify the related parameters that can be characterized and measuredTogether these three components define applicable criteria, characteristics, functionalities and a list of parameters of interest when evaluating different categories of pervasive computing systems.
______
1 Introduction
Recent advent of embedded systems, wireless communications, and sensor technology has been the major force in pushing the technology of pervasive computing forward. The components for computations and communications are much smaller, more capablepowerful and more power- aware than before, allowing the vision of invisible computation that weaves into the fabric of daily life a goal within the reach. The development of network technology and utilization of large-scale data processing and storage also provide a fertile ground for pervasive computing to draw inspirations from.
Many of the early projects attempt to find killer applications for pervasive computing, and/or to get a basic understanding on some novel concepts such as contexts as well as identify the necessary system supports. There were many application oriented research projects that have tried to address certain existing issues or to explore possible solutions toward certain design parameters. Others try to demonstrate the applicability of computers and sensors in many practical applications, or explore how computers and humans can interact in an intelligent environment. During these exercises, most research groups designed their own middleware, programming model and placed emphasis on certain aspects of pervasive computing systems. Well into the second decade of progress in pervasive computing, we as a community continue to see new middleware and programming models being proposed and prototyped. There is a lack of consensus regarding these different approaches, as to what the more advantageous solutions are, or even how to compare these different proposals.
We have observed quite a few full scale deployments in deployment in controlled environment such as Gator Tech Smart House, Aware home, smart presentation room using Gaia, Smart Kindergarten and experimental deployment in real-world environment, such as iHospital system in Denmark over the last few years. These deployments move us one step closer to the day when pervasive computing technology can actually make significant impacts in the real world. Accordingly, we have seen an increase on deployment-oriented publications, and lesson learned papers, however, they are content with mostly qualitative evaluations with minimal quantitative data.
The bottom line is, fifteen years since Weiser’s vision was unveiled, and there are minimal large-scale practical system deployments that actually created impacts. Once the initial excitement about novel applications subsided, there is a general feeling of loss when it comes to identifying the killer applications and there is minimal impact materialized from this promising technology of the future on a world eagerly expecting.
Without solid foundation for evaluations, there is no way to know where we are today, and where we should be heading tomorrow. Surveying the research literatures from the past tells that repetitive efforts have been spent on reinventing the wheels and peddling in place. In other words, there exists an uncertainty in measuring real achievements and advances in the field. The need for a quantitative measurement, a benchmark, a collection of common set scenarios is imminent. Can we objectively compare one middleware to another, measure one deployment against another, and are able to identify their relative strengths and weaknesses? Can we declare with confidence that certain system performs x percent faster than others, while consuming y percent more resources?
One of the difficulties in evaluating pervasive computing systems is their openness and diversity. The extremely diverse domain of applications and the corresponding huge differences in users’ need and expectations make selecting evaluation criteria non-trivial. The challenge, therefore, is to find a common set of scenarios that can be used as the showcase to demonstrate what the pervasive computing technology can really bring to the table, but also serves as a reminder and a ruler to our research community about where our efforts should be directed. These scenarios give the research community a measuring gauge, so we can work together to identify the goals of our research efforts and the hurdles between where we are and where pervasive computing technology should head towards.
When conducting the extensive survey of the research literatures published in the last decade, we collect scenarios and premises of actual deployments that were proposed and demonstrated by various research groups in the field. We are able to identify the primary performance parameters specified during the design and implementation of these projects. By surveying and analyzing the collected information, and combined with our own experience gained when establishing on Gator Tech Smart House [Helal et al. 2005], we were able to come up with the taxonomy of pervasive computing systems. We further present a common set scenario for pervasive computing systems, and subsequently based on the analysis of these scenarios, identify major parameters that should be measured when conducting performance evaluations for these systems.
<need a paragraph stating the concept of performance has been expanded in pervasive computing.>
Performance evaluation in computing systems is not a new idea, as numerous evaluation and simulation methods have long been proposed and widely adopted in both hardware and software design and implementations. Many of these methods provide inspirations to solutions that would be suitable when applying to pervasive computing systems.
Throughout the years, as the computing paradigms have shifted from mainframe to personal computer, from stand-alone to networked, from stationary to mobile, and again from visible, established computing devices to invisible, interweaved computing environment, so have the definition of performance evaluation. The most predominant parameters that define the performance of these computing systems have also shifted, some parameters such as MFLOPS, response time, processing time, throughput, bandwidth, transaction per second, and power consumption rate have become obsolete, while others to be the staple of the evaluation, albeit in a less definitive manner, but as a part of the bigger picture. It is not feasible to bluntly apply old methods designed for hardware, database or distributed systems directly on pervasive computing systems.However, it is not feasible to bluntly apply these methods for hardware, database or distributed systems to pervasive computing systems. The heterogeneity, dynamicity, and heavily context dependent behaviors of pervasive computing systems require new thinking when performance is assessed and evaluated. We have surveyed some of the applicable evaluation methodologies from related fields, and explain how they influence our design decisions when designing our test bed. A summary of this effort is provided in the Related Work section.
Our framework for evaluating the performance of various pervasive computing systems utilizes a three-facet approach. The first facet is to establish taxonomy of pervasive computing systems based on our survey of proposed and prototyped systems and research projects. This taxonomy anchors a reference framework when evaluating vastly diversified pervasive computing systems, both in terms of the domain of applications as well as the architecture adopted in their implementations. The second facet is to create common case scenarios. This set of scenarios outlines the practical applications as well as futuristic visions of what pervasive computing technology can bring to the table, and can serve as a checklist of goals and functionalities for system designers to consider during both the design and implementation stages. The common case scenarios give researches a tool to identify the shortcomings of the designed system at the specification level. The third facet is to identify critical parameters for evaluating pervasive computing systems. The list of parameters of interest allows designers to decide what and how to evaluate the implemented system.
We first present a brief summary of all surveyed publications in Section 2. It is followed by descriptions of the taxonomy for pervasive computing systems, which generate a reference framework for evaluating vastly diversified systems intended for different application domains. With this framework anchored in Section 3, we continue in Section 4 by identifying common set scenarios that can be used to gauge if each system supports critical functionalities crucial for its application. Once the primary functionalities are identified, Section 5 offers a list of parameters of interest, which allow researchers to quantitatively or qualitatively evaluate pervasive computing systems. We wrap up in Section 6 with the conclusion and brief glimpse of the future work.
Pervasive computing is a broad area that encompasses a wide range of research topics including distributed computing, mobile computing, sensor networks, human-computer interaction, and artificial intelligence. mostBy combining and integrating knowledge and technologies from these , our research community is seeing an increasing number of pervasive computing applications highlighting different features. However, in our extensive survey of the research literatures, our finding is that although many research groups have performed some evaluation or testing infocuses on very specific certain areas from a specific facet,have been found onfofthe as a whole Hence, ibreak a into different layers, and use a vertical view to examine technologies at each layer. Figure X shows the architecture of a typical pervasive computing system.aA is used to organize this section: we start from the layer plays a active role in this fieldthen movingeobserveto probehow various differencesdesigns inofthe we close the section by introducing ofand of pervasive computing systems both in experimental environments asnd Layer LayerSensor Platform LayerPhysical Layerwith anAs important has beeninforIn this subsection, wWe examineeach of the perspectivefacets of sensor network technologywireless networking during the past three years underin[?] final currentthatbeen achieved the Berkeley team has released their [Polastre et al. 2005]Many groups are working with Motes either as the basis for other projects or to enhance the sensor platform itself. Intel and Berkeley have worked together on iMote [Nachman et al. 2005], a Bluetooth-enabled version of the wireless sensor node. College of the Atlantic collaborated with Berkeley to use wireless sensor networks for habitat monitoring on Great Duck Island [14Mainwaring et al. 2002].
many unrelated areasin adoptused [15Greenberg and Fitchett 2001]The extremely the isarethey only supportSome groups have workeda more modular tThe [16O'Flynn et al. 2005] [17Edmonds 2005],19will bewildefrom become the major design considerationstermong sleepingrepeatitionin a long time intervals. atto applicationskeyWhile aarehave been systemand .,of these haveare fullyof the platforms ,issue ofa pervasive space, sinceasmeansusedfor the specific purpose ofto makeing mAnother issue with sensor platforms is . aAspace incorporates a variety of applications which make use of dand and process data with different characteristics. A sensor for one application area may not fit into another due to face-recognitionvideo camera occupies higher provide supports inofnetwork protocols and processing units, withan integrated are constrainedto a (runsning)does not allow for in which applications can be developed using service compositionNnetworkingPNumerous works have been done on performance in the general sense, for example, [Scalable ModelingKwon and Performance Evaluation of Wireless Sensor NetworksAgha 2006] [Varsheny and Bagrodia 2004][Performance Evaluation of Routing Protocols for Ad Hoc Wireless NetworksBoukerche 2004]A less researched area isnetworkingingtoin .[Scalable information dissemination for pervasive systems:implementation and evaluationWilliamson2006] has workproposed on propose to use gossiping, class of probabilistic routing protocol, to disseminate context information throughout the environment. They also established aA has been described. In this framework, atTo derive conductaonfthe colleagueset al [Rehman et al. 2007],258Research work has been done in analyzing and evaluating these software architectures. Back in the 90s, when pervasive computing research is at its early stage, there have been a number of methods and techniques developed for the evaluation of software architectures for distributed and mobile systems, for example: Simone and Kazman [De Simone and Kazman, 1995] have proposed SAAM (Software Architecture Analysis Method, a analytical method for evaluating software architectures; Kazman et al have also developed ATAM (Architecture Trade-off Analysis Method, [The Architecture Tradeoff Analysis Method]), a structured technique for understanding the tradeoffs inherent in the architectures of software intensive systems; Clements describes ARID (Active Reviews for Intermediate Designs, [5]), a method for reviewing preliminary software designs for suitability in its intended usage context and environment.