Changing a Web Vanguard – Windows Live Hotmail
Executive Summary
This case study documents how the Windows Live Hotmail development team successfully utilized Visual Studio Team Edition for Software testers to build one of the most ambitious new releases of their iconic Web-based email service.
Introduction
Windows Live Hotmail is a popular Web-based, consumer email service from Microsoft. One of the pioneers of Web-based software, Windows Live Hotmail has seen its user base grow from about 8 million customers in 1997 to over 260 million customers worldwide. Customers from around the world rely on Windows Live Hotmail for their email; making Windows Live Hotmail one of the busiest Web sites in the world.
Windows Live Hotmail customers enjoy a great deal of functionality – all users are given 2GB of space for their email, extensive security and safety features, as well as integration with Windows Live Messenger and Windows Live Spaces.
Never a team to rest on its laurels, in late 2005 the Windows Live Hotmail development team announced the plans for an ambitious re-design and re-architecture of their iconic product. This new release of Windows Live Hotmail promises to deliver revolutionary functionality and integration with the rest of Windows Live.
The challenge the Windows Live Hotmail development team faced was how to ensure what they were building would scale to meet the demands of millions and millions of customers from around the world.
To meet this challenge, the development team incorporated Visual Studio Team Edition for Software Testers into their daily development cycle. Specially, they relied on the unique Web Testing and Load Testing capabilities of Visual Studio Team Edition for Software Testers to ensure that they were reaching their performance and scalability goals.
Visual Studio Team Edition for Software Testers is a member of the Visual Studio Team System family of products and its enables developers and testers to validate the quality, performance and scalability of their applications. Visual Studio Team Edition for Software Testers includes technology to perform unit testing, Web functional testing and load testing. To simulate the most demanding stress tests, Visual Studio Team Edition for Software Testers can enlist help from a complementary Visual Studio Team System product, the Visual Studio 2005 Load Test Agent. Visual Studio Team Edition for Software Testers is able to deploy tests to the computers that have the Load Test Agent installed; those machines will execute the tests on behalf of Visual Studio Team Edition for Software Testers and report the results back.
Now let’s have a look at the challenges that the Windows Live Development team faced, and the solutions that Visual Studio Team Edition for Software Testers provided.
The Operations Environment
The operating environment that hosts Windows Live Hotmail is one of the most advanced data centers in the world. To go into great detail about how this data center is architected would be beyond the scope of this document. What we can say about his environment is that it is broadly categorized into front door servers and behind-the-scenesservers. This logical architecture is illustrated in Figure 1.
The front door servers are responsible for the first contact with Windows Live Hotmail customers. Once a session is established, they are responsible for making requests on behalf of the customer to the appropriate behind-the-scenes server. Based on prior experience, the deterministic factor in the overall performance of Windows Live Hotmail would depend on the front door servers; that being the case, the focal point of the performance and scalability testing would be on these servers.
The goal established by the Windows Live Development team for the front door servers was that each server to be able to support 215,000concurrent clients. However, to validate this goal, the team had to understand what their baseline was.
Establishing a Baseline
One of the biggest challenges that any development team will face when doing performance and scalability testing is determining what their baseline capacity is. Simply put, what value should they use to determine if they are making progress against their goals?
Visual Studio Team Edition for Software Testers provides a unique solution to this problem with its goal-based load testing pattern.
Generally speaking, a load test is executed by simulating the actions of a number of users. The exact number of users a load test will simulate and how those users are introduced is what Visual Studio Team Edition for Software Testers calls a load test pattern.There are three load test patterns that Visual Studio Team Edition for Software Testers provides –constant, step and goal-based. These are illustrated in Figure 2.
Figure 2
Constant and step load testing patterns are fairly self explanatory. A constant load pattern is simply one where the number of simulated users is constant. This pattern is ideal when testing the absolute resiliency of a system while under stress. A step pattern is one where the number of users is increased at regular intervals of time. This pattern is arguably a more realistic way to simulate the demands a system might face.
The goal-based load pattern is unique in the sense that the number of simulated users is changed – increased or decreased – based on a metric that the tester chooses. In the case of Windows Live Hotmail, prior experience told them that the front-door servers tended to be CPU bound. In other words, the computations that the front-door servers are doing were keeping the utilization of their processors high. This utilization would ultimately determine the performance and scalability of Windows Live Hotmail as a system.
To build an accurate baseline for the performance of the front-door servers, the Windows Live Hotmail team used a goal-based load test pattern that used the processor utilization percentage as the metric to determine whether the number of users should be increased or decreased. Figure 3 shows an example of a load test configured to use a goal-based pattern against this metric.
Figure 3
The metrics are calculated from the actual Windows Performance Counter values on the front-door servers. In the example shown in Figure 3, theWindows Performance Counter being monitored is the % Processor Time.The High End of the pattern is set to 90 and the Low End is set to 70. This means that Visual Studio Team Edition for Software Testers will increase or decrease the number of users as necessary in order to keep the processor utilization between 90 and 70 percent. The other settings in the goal-base pattern determine how many users are increased or decreased at a time, and what the maximum number of users should be.
By using this goal-based load test pattern, the Windows Live Hotmail development team was able to determine what the baseline performance of their system was. With a goal of keeping the processor utilization in a functional range of 70 and 90 percent, the number of users that were being simulated in the load test determined whether they had reached their goal of supporting 215,000 users per front-door server.What we will look at next are the actual tests that were part of the load test and how often these tests were run.
Web Testing in Visual Studio Team Edition for Software Testers
A LoadTest in Visual Studio Team Edition for Software Testers is really just a container for a collection of functional tests; a load test is used to amplify the effect of a single test to create stress on the system being tested. Typically, a Load Test will contain one or more Web Tests, although it’s possible to execute other test types as well such as Unit Tests or Generic Tests.
Visual Studio Team Edition for Software Testers can produce Web Tests in a number of ways. The most generic way is to record the use of a Web application as a series of HTTP requests. This can be done by launching the Web Test Recorder and clicking through the application just as a user would. This script then becomes the basis of the test. However, in order to have the finest grained control over how your Web application is tested Visual Studio Team Edition for Software Testers provides a .NET object model that can be used to programmatically create these Web tests. The Windows Live Hotmail team used this approach to hand-craft their Web Tests; Figure 4 illustrates an example of a Web Test built in C# using this approach.
Figure 4
In order to accurately simulate how a customer might use Windows Live Hotmail, or any other Web application, Visual Studio Team Edition for Software Testers provides an attribute called think time. This value reflects the amount of time a customer might pause between each request, or each action. These pauses are important in scalability testing because they may reveal race conditions and deadlocks that would not occur if all the requests are sent in a uniform sequence. Figure 5 illustrates how the ThinkTime attribute can be set in a coded Web Test.
Figure 5
A Load Test makes use of the ThinkTime property to simulate Web traffic. This can be used as an absolute value and applied to all requests, or used as the midpoint for a normalized distribution curve. Using think times enabled the Windows Live Hotmail development team to more accurately simulate the traffic they would see in production.Controllers, Agents and Rigs
Given that the current version of Windows Live Hotmail can see upwards of 200 million unique visitors in a month, the load testing that the Windows Live Hotmail development team performed had to simulate a great deal of virtual users.
Simulating virtual users is itself a computationally intensive activity. A given machine can only simulate a certain number of virtual users before the management overhead becomes overwhelming. In order to test the scalability requirements of Windows Live Hotmail, the team utilized the controller-agent architecture provided by Visual Studio Team Edition for Software Testers.
Visual Studio Team Edition for Software Testers has the ability to coordinate the activities of a group of machines to simulate virtual users and load. In order to generate the amount of load they needed for a realistic load test, the Windows Live Hotmail team used a group of physical machines for their load testing.
In any group of computers, Visual Studio Team Edition for Software Testers relies on the presence of one controller and one or more agents. Collectively this group is called a rig. An agent is part of the rig that is used to run tests and generate simulated load. The controller is part of the rig that is used to coordinate the agents and collect test results. The controller and agent functionality is only available as a separate product – the Visual Studio Team System Load Agent.
Visual Studio Team Edition for Software Testers can be used to contact a controller and initiate a test run that will be executed by the agents.Figure 6 illustrates how Visual Studio Team Edition for Software Testers can be used to contact a controller and determine how many agents it has available for a given load test.
Figure 6
With an understanding of what kind of testing the Windows Live Hotmail development team conducted, we can turn our attention towards how frequently the testing was done, and how the results were interpreted.
Performance and Scalability Testing in the Development Process
Performance in any Web-based application is important, but in the case of such a high profile Web application like Windows Live Hotmail, it is absolutely crucial. That being the case, the performance and scalability of the system was monitored with every build produced.Figure 7 illustrates the process the Windows Live Hotmail development team followed.
As soon as a build is produced, it is automatically deployed to the test environment. This test environment uses the same hardware as the actual deployment environment used to host Windows Live Hotmail.
A custom tool written by the Windows Live Hotmail development team polls for the presence of new builds; as soon as one is detected, it uses the Visual Studio Team Edition for Software Testers command line interface to trigger a new test run.
The Windows Live Hotmail development team chose to monitor performance in three different ways. First, using the Web and Load Testing capabilities of Visual Studio Team Edition for Software Testers, they monitored the ISAPI Requests/Second performance counterat a constant CPU usage rate of 75%. Secondly, they used the profiling capabilities of Visual Studio Team Edition for Software Developers to determine the exact behavior of their code; and finally, custom written code in their application to report back performance characteristics.
Once a test run is complete, the results are examined by hand. Most importantly, the delta between the current test run results and the previous one are calculated. Most importantly, the difference in actual execution time is calculated – any major differences at this level are considered the most serious of potential bugs.
Figure 7
The ability of Visual Studio Team Edition for Software Testers to generate a realistic amount of load against the Windows Live Hotmail Beta implementation and its ability to integrate into the development life cycle has enabled the team to continue to innovate while maintaining and exceeding their performance and scalability goals. Visual Studio Team Edition for Software Testers has become a critical tool for the Windows Live Hotmail development team as it pushes for the public launch of a new Windows Live Hotmail.
Conclusion
Windows Live Hotmail is one of the busiest Web applications in the world; and it is undergoing its most ambitious revision to date. The Windows Live development team responsible for this iconic Web property had to make sure that they delivered the highest quality product possible. To do this, one of the key technologies that they depended on was Visual Studio Team Edition for Software Testers.
Rajeev Shukla, one of the team members responsible for Windows Live Hotmail, had the following to say about his team’s experience with Visual Studio Team System.
”With Visual Studio Team System, we have been able to improve our test result accuracy by at least a factor of 20% over the results we got from our previous tools. Now the capacity results produced in the test lab are very much in-line with what the operations team measures on the live site. This helps the whole team stay synchronized and make improvements to Windows Live Hotmail as a whole.”
Providing Web-based email to millions of customers around the world is probably not a scenario that most enterprise IT environments would share with the Windows Live development team. But, having to deliver a fast, scalable, highly-visible Web-based application to a group of customers with high-expectations is probably a challenge that both groups share. As demonstrated by the Windows Live development team, Visual Studio Team Edition for Software Testers provides a number of solutions for that challenge; these solutions can easily be leveraged for other environments as well.
Additional Resources