Customer Solution Case Study
/ County Saves Nearly $1 Million, Experienced Zero Downtime on Virtualized Systems
Overview
Country or Region: United States
Industry: Government—Local
Customer Profile
Snohomish County covers 2,090 square miles in Washington and is home to 711,100 residents. It has about 2,670 employees.
Business Situation
Snohomish County needed to replace its aging server computer fleet while adhering to budget constraints and making the environment easier for its limited IT staff to manage.
Solution
The county deployed the Windows Server 2008 R2 operating system with Hyper-V technology and Microsoft System Center data center solutions to virtualize and improve management of its server environment
Benefits
· Saved 50 percent in hardware costs, consolidated servers by 35 percent
· Eliminated downtime, boosted availability
· Sped provisioning and recovery
· Supported green initiative, infrastructure advancements / “We used Microsoft virtualization technologies to create a more advanced, reliable infrastructure that county employees can depend on, while minimizing costs and optimizing IT staffing resources.”
Aaron Reardon, Executive, Snohomish County
With shrinking IT budgets and staff cuts, Snohomish County was struggling to update and ensure reliability of aging servers. It was behind schedule for replacing servers and spending too much time provisioning and restoring them. By deploying the Windows Server 2008 R2 operating system with the Hyper-V virtualization technology and Microsoft System Center data center solutions, Snohomish County consolidated physical servers by 35 percent and cut server racks in half, reducing power consumption and hardware costs. It expects to save nearly U.S.$1 million during its seven-year computer refresh cycle. The county eliminated downtime from outages and maintenance, boosting availability. Provisioning new servers takes 30 minutes instead of more than six weeks, and IT staff easily back up countywide data and recover failed systems 75 percent faster, while creating a more advanced infrastructure.
Situation
Snohomish County, located on Puget Sound in Washington, has more than 711,000 residents. Like most local governments in the United States, Snohomish County has been facing decreasing revenues for several years. As a result, it implemented significant budget cuts. Since 2007, the county has reduced its IT budget by 23 percent and IT workforce by 27 percent. Just nine IT workers manage the data center, which was comprised of about 300 physical servers that are accessed by approximately 2,670 employees across 28 business groups.
“Not only was it time-consuming to maintain all these computers, but also many were aging and needed to be replaced. We had a U.S.$300,000 annual hardware refresh budget, but, because of funding cuts, we didn’t have the staff needed to keep up with the replacement schedule. The backlog of replacement candidates was growing, and we were in danger of pushing refresh dates out so far that the computers would begin to break down,” says Randy Manley, Systems Manager for Snohomish County. “Our IT team was already finding it challenging to ensure continuity, and, with an increasingly aging computer fleet, we knew that our ability to provide reliable service would suffer even more.”
Availability and Downtime Issues
Only about five percent of the applications used by Snohomish County employees were on high-availability clusters. The rest were subject to downtime because of hardware failures and scheduled maintenance. “We didn’t have enough budget to put every application on a hardware cluster. Anytime there were hardware failures or maintenance that required powering down physical devices, employees couldn’t access those applications. We were usually able to get applications back up and running within four hours, but that was a long time for employees whose work was interrupted,” says Manley.
Snohomish County IT staff spent a fair amount of the unplanned downtime on troubleshooting. “Before we could begin fixing a problem, we had to figure out what broke down and why,” explains Eric Henry, Systems Engineer for Snohomish County. If a server needed to be restored completely, it could easily take 8 to 12 hours for IT staff to identify the issue and get the server running again from its backup system.
Lengthy Provisioning of Underutilized Servers
Provisioning new server computers was time-intensive. First, Snohomish County had to procure the hardware, which could take six weeks to arrive. Then a dedicated IT staff person had to spend about two days configuring and installing the server in a rack.
“We provisioned about three new physical servers every month. It was a constant process of building and racking servers and adding new hardware, and it consumed a lot of resources,” says Henry.
The servers were drastically underutilized. “Our business units have ownership over their applications and servers. When a group received approval for a new application, it would also buy a new server to run it on, even if just a dozen employees were using it. The servers generally ran at just two to three percent of CPU utilization. This would mean a large quantity of dollars would be left on the table, unused inside our server farm,” says Michael Cruz, Enterprise Technology Architect for Snohomish County.
With so many new servers coming on board, Snohomish County was reaching the limits of its data center space. “We ended up building out a new data center, largely to accommodate known and anticipated server growth. We allotted room for 60 server racks—which was three times more than the 20 racks we needed at the time—so that we wouldn’t run out of space several years down the road,” says Manley.
All these servers and the equipment needed to support them consumed a lot of power, which was contrary to the county’s emphasis on environmental sustainability. “Snohomish County is very serious about reducing our environmental footprint. The IT department wanted to lower the amount of energy that our data center required, but that was hard to do when our need for servers kept growing,” says Manley.
By late 2008, its server replacement backlog had reached a critical point and Snohomish County had to figure out how to procure new servers without exceeding its budget or its staffing limitations. It also wanted to make it easier for the IT department to manage and ensure reliability of its server infrastructure.
Solution
Snohomish County considered two options for resolving its server backlog issues. “We could spend time identifying and replacing the oldest computers and slowly building out our physical server farm, or we could look at new technologies such as virtualization, which we had heard could help speed server deployments and save money while also making the data center easier to manage. Virtualization seemed much more prudent since it could deliver much greater returns over the long term,” says Cruz.
Virtualization Vendor Selection
Snohomish County evaluated server virtualization solutions from VMware and Microsoft—specifically the Hyper-V virtualization technology that is a feature of the Windows Server 2008 operating system. “VMware was cost-prohibitive and about three times more expensive than Microsoft. Although at the time, VMware ESX had more features, Hyper-V had everything we needed and we knew that the product would evolve. And, after we realized how using Microsoft System Center Virtual Machine Manager could help us simplify virtual machine management, the Microsoft solution became even more attractive,” says Manley. The IT staff was particularly interested in the System Center Virtual Machine Manager physical to virtual (P2V) machine capability, which it ultimately used to convert about 30 physical servers to virtual servers running Hyper-V.
Snohomish County was also reassured by the level of support it could receive from Microsoft Services Premier Support. “We already had a great relationship with Microsoft Services Premier Support. Microsoft always provided very responsive service and that was really important to us as we considered moving to a whole new type of technology and a new way of operating,” Henry said.
According to Manley, “Cost is a huge consideration when budgets and staffing levels are shrinking. When we looked at the cost savings that Microsoft had over VMware, plus its track record of superior service and strong virtualization and management products, it was clear that Microsoft was the best solution for us.”
Virtualization Deployment
Snohomish County began its transition to virtualization in June 2008 by building test servers running Windows Server 2008 Datacenter with the beta edition of Hyper-V. As the test systems proved successful, the IT team moved them into production along with System Center Virtual Machine Manager 2008. By March 2009, Snohomish County had begun its production deployment with Windows Server 2008 Datacenter with Hyper-V and System Center Virtual Machine Manager 2008. It upgraded to Windows Server 2008 R2 Datacenter with Hyper-V and System Center Virtual Machine Manager 2008 R2 in September 2009 to take advantage of features such as Live Migration, which enables the county to move virtual machines between Hyper-V hosts with no downtime and reduce the planned downtime associated with routine system maintenance, as well as quick storage migration and rapid provisioning.
“We chose the Datacenter edition of Windows Server 2008 R2 for the county’s main data center because it allowed an unlimited number of licenses for virtual guests on a single physical host. We were able to put 64 virtualization servers on one physical server, which made it very cost-effective,” says Cruz. Snohomish County also uses Windows Server 2008 R2 Enterprise in remote offices, such as its Medical Examiner and Parks departments and the Snohomish County Paine Field Airport. The Enterprise version allows for four virtual guests, which is sufficient for those environments.
As of March 2011, Snohomish County has deployed 170 servers running Hyper-V on 12 physical servers. This includes 69 production virtual machines that run on a four-node hardware cluster and that use the Windows Server 2008 R2 failover clustering feature for high availability. Another 63 virtual machines run in a test environment, and approximately 21 virtual machines reside within the Snohomish County perimeter network on a three-node cluster. The virtual machines run on HP ProLiant DL785, HP ProLiant BL460c, HP ProLiant BL480c, and HP ProLiant DL380 computers.
Snohomish County runs a variety of workloads using the Hyper-V virtualization technology, ranging from web, print, file, and network management services to Microsoft SQL Server 2008 database management software. The county is planning to upgrade from Microsoft Exchange Server 2007 and Microsoft Office SharePoint Server 2007—which run on physical servers—to Microsoft Exchange Server 2010 and Microsoft SharePoint Server 2010, and it will run these programs on virtualization servers.
“Our plan is to virtualize most of our existing environment and use Hyper-V for new servers. So far we’ve virtualized about 50 percent of our infrastructure. We hope to reach 75 percent virtualization by December 2012,” says Manley.
The Snohomish County virtualization solution also includes Microsoft System Center Configuration Manager 2007 R3, which the IT staff uses to update virtual machines and physical hosts; Microsoft System Center Operations Manager 2007 R2, which it uses to monitor the health and performance of its infrastructure; and Microsoft System Center Data Protection Manager 2010, which it uses to back up approximately 18 terabytes of data and restore failed systems. Snohomish County had been using Legato NetWorker (now EMC NetWorker) for backup. “We’re halfway through the process of transitioning to System Center Data Protection Manager. We like that the Microsoft product does disk-to-disk-to-tape backup because that helps us speed backup and recovery; and the way it works so seamlessly with other Microsoft products, such as Hyper-V and Microsoft SQL Server, makes operations easier,” says Henry.
Benefits
By using Microsoft virtualization and management technologies, Snohomish County is able to realize substantial cost savings, consolidate physical servers, and boost availability of its server infrastructure. The county has accelerated IT operations, from provisioning to backup and recovery, improved its environmental sustainability efforts, and freed up resources to focus on strategic IT projects.
“In today’s difficult economy, we must look for continued cost savings. We used Microsoft virtualization technologies to create a more advanced, reliable infrastructure that county employees can depend on, while minimizing costs and optimizing IT staffing resources,” says Aaron Reardon, Snohomish County Executive.
Saved 50 Percent in Hardware Costs, Consolidated Servers by 35 Percent
By moving to servers running Hyper-V, Snohomish County has reduced its hardware costs. “Because we don’t have to replace all our aging physical servers, we’re saving about $150,000 per year. That’s close to a 50 percent savings and will amount to around $1 million over our normal refresh cycle,” says Manley.
Snohomish County estimates that it also saves another $10,000 annually on licensing for System Center data center solutions. “For example, instead of licensing System Center Configuration Manager agents on every physical server, we now have one license for each host on the virtual infrastructure. It’s definitely a better deal,” notes Cruz.
The IT department no longer has difficulties keeping up with hardware replacements. “In the past, our staff didn’t have time to replace all the computers that we had budgeted for refresh. But now that provisioning virtual servers is so simple and fast, we can easily handle hardware replacements. We’ve almost totally eliminated the server refresh backlog and the risk of older machines breaking down and interrupting employees’ work due to hardware failures,” adds Manley.
The county has significantly reduced its number of physical servers. “We went from about 300 servers to about 195, a 35 percent decrease. And we cut our server racks in half. Instead of 20, we only need 10,” says Manley. Now that the county can run virtual machines for different business groups on the same physical server, it regularly achieves server utilization rates of 40 percent, which is more than a 100 percent increase.
“Virtualization was essential to achieving maximum value for our server hardware investment,” adds Cruz.
Snohomish County doesn’t need the extra data center space that it had allocated for server growth. “We’re looking at repurposing the data center. For instance, we could lease some of the space to agencies in other counties to help offset revenue deficiencies,” says Manley.