Legion: Build your own virtual super computer with Silverlight
By Daniel VaughanLegion is a grid computing framework that uses the Silverlight CLR to execute user definable tasks. It provides grid-wide thread-safe operations for web clients. Client performance metrics, such as bandwidth and processor speed, may be used to tailor jobs. Also includes a WPF Manager application.
Contents
· Introduction
· Background
· Long live Clog
· Legion Agent UI
· Legion Manager
· Components Overview
· Grid Tasks
· Configuration
· Agent Performance Metrics
· Prime Finder Task
· Agent Task Execution
· Client-side Mutual Exclusion
· Expiring Collections
· Saving and retrieving data using the Isolated Store
o Saving data using the Isolated Store
o Retrieving data using the Isolated Store
· How to write a Grid Task
· Future enhancements
· Conclusion
· Special Thanks
· References
· History
Introduction
And he asked him, What is thy name? And he answered, saying, My name is Legion: for we are many.
Mark 5:9 (KJV)
I have always been a fan of distributed computing. With the advent of Silverlight, I felt there was an exciting opportunity to create a Grid computing framework utilizing the client-side Silverlight CLR. That is how Legion emerged.
Legion is a Grid Computing framework that uses the Silverlight CLR to execute user definable tasks. Legion uses an ASP.NET application and web services to download tasks, upload result data, and provide grid-wide thread-safe operations for web clients or agents. Multiple tasks can be hosted at once, with Legion managing the delegation of tasks to agents. Client performance metrics, such as bandwidth and processor speed, may be used to tailor jobs for clients. Legion provides a management service and WPF application that is used to monitor the Legion grid.
I have deployed Legion to a demonstration server here so you can see it in action. (http://www.orpius.com/Silverlight/Legion)
Figure: Legion system overview.
Background
While Grid computing is not new, there has been a resurgence of interest in it due mainly to organisations looking to capitalise on their underutilised IT infrastructure, to perform computationally intensive business related tasks.
Aside from large organisations jumping on the bandwagon, public interest in Grid computing has of course been stimulated by large scientific projects, such as the Seti@home project. After performing some initial research for this article I became enthused by what a volunteer grid computer is capable of achieving, and what, by offering a framework like Legion may achieve. The results of some volunteer Grids have been remarkable. For example, the Smallpox Research Grid Project managed to find 44 strong treatment candidates for smallpox; an, as yet, incurable disease. Vaccination for Smallpox ended in the 1980s, but there are fears that the virus may re-emerge. (Wikipedia, 2007; The Inquirer, 2003)
The allure of harnessing unused computing power is strong, and so too is the notion of many users contributing otherwise wasted clock cycles for the benefit of humanity. One reason why so many Grid projects have garnered so many participants, is the feeling of belonging to a shared mission; it's free, and users gain a feeling of contributing to a good cause, just by running an application in the background on their computer.
Could it be that we are on the cusp of a web paradigm shift, where web browsers share the load of the web server? Opening the door to smaller start-ups with little investment in infrastructure, providing an unlimited capacity; serving any number of clients. Such a capability has traditionally been the domain of monolithic companies with massive data centres. What would that do? Well, it would radically transform the web; providing a level of democratisation never before seen. Taking this idea further, imagine, if you will, what a peer to peer WebTorrent protocol might bring. Web servers and web browsers alike, serving as seeds and distributing the load. We are venturing outside the scope of the article a little, but I raise these ideas to encourage in the reader a feeling of opportunity. For that is what having a CLR in the browser now affords us. An opportunity, not only to create nice UI, but also to move to a less server-oriented application model. Not merely a thin-client model, but rather a distributed processing model.
I believe security to be the main challenge in providing client-centric peer to peer volunteer computing. Protecting a user's privacy is paramount, and it's risky sending sensitive information, which may be eavesdropped or modified by some rogue client. Likewise, trusting the results from an inherently untrustworthy source presents an issue. Clearly mechanisms are required to address these issues.
Likewise, providing secure cross domain browser support will bring a lot of opportunity to decentralise and share services more readily between organisations. Microsoft and others have proposed some techniques. Yet, it was inevitable that doing so within existing browser capabilities requires an assortment of hacks. Hacks, such as using IFrames to communicate across domains. While preventing cross domain browser communication seems reasonable, with today's ubiquity of webservices, we can quite easily send client data from server to server, and thus across domains, which effectively circumvents the protection mechanism anyway.
Unlike other distributed computing projects, Legion allows users to participate by simply viewing a webpage. The shift to cloud computing, and web based application, may have fostered a growing reluctance in users to download and install applications. Forcing users to download software to participate, while improving the dependability of the grid (we'll discuss the pitfalls of a browser based Grid later), must surely decrease the overall participation rate. Users prefer not to install software. It's a hassle.
Legion, on the other hand allows us to bring new 'volunteers' to our grid by means of providing enticing content. Thus, we are no longer dependent on a single motivating cause to bring participation. It provides developers with a system that will deploy and execute modules written using any .NET Desktop/Silverlight CLR compatible languages. The beauty of Legion is that it does not require user installation of any Legion software. It relies solely on the Silverlight CLR in the user's browser.
I've heard many comparisons between Silverlight, Flash, Java applets etc. Most comparison seems to resolve around what an old project manager of mine called "pointy clicky things."; pure UI. Well, in this article my aim is to take the focus away from being solely UI, and take more of a look at what having a .NET CLR in the browser might mean.
Long live Clog
Clog, unbeknownst to me, was a prelude to this project, and turned out to be invaluable in its development. Without Clog for Silverlight working on the concurrency elements of this project would have been made extremely difficult. I have, as it were, been eating my own dog food. It made debugging with multiple clients a breeze. Using the Log4NetLogStrategy I had logging coming in from the website, the Agents and even the WPF Manager at the same time. It would have been next to impossible to debug multiple clients executing concurrently. And using trace statement would have been awful. Clog is pretty cool, even if I do say so myself. I've made several important updates to its code base, which I will be publishing soon.
Legion Agent UI
On the client-side, Legion is hosted within a Silverlight module that automatically controls the downloading and execution of tasks.
The Agent visualizer that is provided may be customized to suit. I may provide a more discreet interface at a later stage. As can be seen, the Silverlight module is hosted on an HTML page, and will happily share this space with other content; be it HTML or perhaps some other interactive Silverlight animation to keep the user enthralled and the browser open.
The interface displays the task name, as it is defined in the server-side config, the number of tasks completed, stored on the clients machine in Isolated Storage, and the percentage complete of the current executing task.
Legion Manager
The Legion Manager is a WPF application that allows monitoring of the grid. It provides a summary of the progress of all tasks, and the current grid capacity; including available processing power in megaflops and total bandwidth for all connected clients.
Components Overview
Legion consists of three main components. The GridComputing and GridComputing.Management components use the Desktop CLR. The third, GridComputing.Silverlight, is Legion's main client-side component, and is executed within the Silverlight CLR. The main types from each component are shown below.
Figure: Legion component overview.
The WPF Manager is a WPF application that communicates with the server-side GridComputing module to monitor the Legion grid. It consumes GridSummary instances that are created by the GridManager.
Agent and Client instances are passed to the grid web services as part of each request from the management and agent components; and they encapsulate the location data and performance metrics of callers.
The MasterTask and SlaveTask provide the respective server and client task implementations. Jobs are created by a MasterTask and dished out to SlaveTasks by the GridManager.
The GridMonitor is a tool to provide mutual exclusivity for Agents. The GridSync encapsulates the location of the code region, and the GridMonitor uses a web services to request exclusive access the region.
Agents communicate with Legion on the server-side using the JsonGridService. When working with Silverlight 1.1 we use JSON JavaScript Object Notation. You can find a little more info on working with Silverlight and web services in one of my previous articles.
Figure: JsonGridService class diagram.
The Grid service is mostly a wrapper for the GridManager with some additional error handling thrown in. The following is a brief overview of the JsonGridService methods:
· Register: Lets the server know that the calling Agent is ready to be connected to the network, and that it is available for new tasks. Returns a unique identifier that will be used in future calls to identify the Agent.
· StartNewTask: Retrieves a TaskDescriptor containing information regarding a new SlaveTask and the Job data required to run the task. This is called periodically by the Agent on the completion of each task.
· UpdateJobProgress: Lets the server know where the client is up to in its processing of a SlaveTask.
· JoinTask: Called when the SlaveTask completes on the client-side. This allows the result of the task to be processed by the MasterTask and joined with other results from other clients. This is the endpoint for a client's SlaveTask.
· Disconnect: Lets the server know that the Agent will no longer be processing tasks.
· Download100KB: Used when measuring client bandwidth. It simply returns a large object that brings up the SOAP message size to approximately 100 KB.
· LockEnter, LockUpdate, LockExit: Used to provide mutual exclusion for client-side code blocks. When an Agent has exclusive access, it periodically polls using the LockUpdate method to inform the LockManager that it is still active (that the client browser has not been closed).
Grid Tasks
When creating a new Legion task, we create a server-side master task, and a client-side slave task. A custom task is created by extending the MasterTask class in the main GridComputing module, and by extending the SlaveTask class in the GridComputing.Silverlight module. The role of the MasterTask is to split the grid computing activity into bite size chunks, and hand these off to client-side Agents. The client-side SlaveTask is the point of concurrency; this is where we place the code that we want executed by multiple clients. When a SlaveTask completes its work, the result is sent back to the MasterTask to be merged with the results of other SlaveTasks.
Figure: MasterTask class diagram.
The client-side static TaskRunner is responsible for the instantiation, asynchronous execution, and provision of Job data for the SlaveTask. The TaskRunner is also the main point of interaction between the Silverlight UI and Legion.
Figure: Silverlight Agent module - class diagram of TaskRunner and SlaveTask.
In order to execute a client-side task (a SlaveTask) we require the SlaveTask Type name, and the Job data. The Job data is created by the MasterTask, while the TypeName is defined in the server-side config as slaveType. The TaskDescriptor and child Job instance are returned from the JsonGridService when the Agent makes a call to JsonGridService.StartNewJob(Agent agent).
Figure: TaskDescriptor and Job class diagram.
Configuration
Legion uses a configSection to define the tasks to be run. First we declare the section within the configSections element, as shown in the following excerpt.
configSections
section name="Grid"
type="Orpius.GridComputing.Configuration.GridConfigSection"
requirePermission="false"/>
<!-- Client Logging section -->
section name="ClientLogging"
type="Orpius.Logging.Configuration.ClientLoggingConfigSection"/>
.
.
.
</configSections
Then we declare our grid tasks within the section, as shown in the following excerpt.
Grid
Tasks
<!--
slaveType: The name of the Silverlight task type.
Please note that Version, Culture,
and PublicKeyToken are required.
-->
add name="PrimeFinder"
type="Orpius.GridComputing.Tasks.PrimeFinderMaster,
Orpius.GridComputing.Tasks.PrimeFinder"
slaveType="Orpius.GridComputing.Tasks.PrimeFinderSlave,
Orpius.GridComputing.Tasks.PrimeFinder.Silverlight,
Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/>
add name="MutexExample"
type="Orpius.GridComputing.Tasks.MutexExampleMaster,
Orpius.GridComputing.Tasks.MutexExample"
slaveType="Orpius.GridComputing.Tasks.MutexExampleSlave,
Orpius.GridComputing.Tasks.MutexExample.Silverlight,
Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/>
</Tasks
</Grid
It is important to note that the slaveType attribute of each task must include the entire signature of the assembly. If we fail to include any part, such as the Version or Culture attributes, Silverlight will be unable to resolve the download path for the assembly.
Agent Performance Metrics
To measure the connection bandwidth we merely call a web service that returns a result message that is approximately 100 kilobytes. We do this once when the connection is established.
/// <summary>
/// Measures the bandwidth of the connection
/// with the server by calling a web service