Customer Solution Case Study
/ Financial Services Firm Improves File Server Availability by 40 Hours a Year
Overview
Country or Region: United States
Industry: Financial services
Customer Profile
Equifax provides consumer and business credit intelligence, fraud detection, and other services. The US$2 billion company is headquartered in Atlanta, Georgia, and employs 6,500 people.
Business Situation
Equifax wanted to reduce server downtime to maintain excellent customer service. It also wanted to reduce server maintenance work so that its IT staff could focus on creating new products.
Solution
Equifax is upgrading its servers to the Windows Server 2012 operating system to take advantage of transparent failover and other features that support high availability and IT efficiency.
Benefits
· Improve file server availability by up to 40 hours annually
· Create more time for innovation
· Reduce server needs by up to 30 percent
· Improve work-life balance / “Windows Server 2012 delivers a revolutionary change in how we perform cluster updates.… We expect to eliminate up to 40 hours annually of planned and unplanned downtime for our file server infrastructure.”
Ron Francois, Technology Vice President, Workforce Solutions, Equifax
Equifax provides consumer and business data and other financial services to thousands of businesses. Its IT systems must be available at all times. To eliminate the downtime associated with updating server clusters, Equifax upgraded to the Windows Server 2012 operating system, which provides transparent file server failover. With this capability, Equifax can reduce maintenance-related downtime for applications that depend on its file server infrastructure and improve availability. It can also relieve IT staff of the onerous update work, which frees them to focus on new product development. Additionally, by reducing the time that the IT staff needs to work late, the company promotes a healthy work-life balance and boosts employee retention. Equifax expects to trim file server costs by up to 30 percent a year, due to the higher server efficiency enabled by transparent file server failover.
Situation
Businesses of all sizes rely on Equifax for consumer and business credit intelligence, portfolio management, fraud detection, decision-making technology, marketing tools, and much more. Equifax maintains data on more than 572 million consumers, 81 million businesses, and 200 million employee files worldwide—an amount of data equivalent to 15 United States Libraries of Congress. It generates 158 billion credit score updates each month, or 60,000 updates per second.
With this kind of workload hitting its data center, server availability is of utmost importance to Equifax. The Equifax IT organization commits to 99.85 percent application uptime to Equifax business units, with a goal of moving to 99.95 percent availability. However, maintaining dozens of file server clusters running the Windows Server 2008 R2 operating system required periodic downtime to apply security updates. The IT team had a difficult time scheduling these updates to server clusters, especially the 10 or 12 that ran a batch processing application for the Equifax Workforce Solutions business called TALX Unified Batch Alert (TUBA). TUBA performs high-speed batch processing in the fulfillment of certain Equifax services.
“Our customers are becoming more and more demanding in terms of availability,” says Bryan Garcia, Chief Technology Officer at Equifax. “Our SLAs [service level agreements] allow us about 20 minutes of downtime a month. However, one cluster node failure would involve one to two hours of downtime and consume our allotment for the whole month. We were experiencing about 20 of these cluster failures annually across all of our file server clusters in the Workforce Solutions business.”
Routine server updates, and responding to unscheduled cluster failures, also required a great deal of extra work by the server team and also the applications teams, which had to restart the 40 to 50 applications running on the updated cluster. “Agility is one of our biggest business objectives,” Garcia says. “Our business has been growing both organically and by acquisition, and this growth drives a lot of new product creation. However, the IT organization was pulled away from new product development when an outage occurred.”
Employee job satisfaction was also affected by cluster updates. “Rising customer availability requirements were pushing more of our work into nighttime hours, which was affecting job satisfaction,” says Ron Francois, Technology Vice President for Workforce Solutions at Equifax. “No one wants to be here at night updating and restarting servers.”
Solution
When Equifax IT staff members learned about the high-availability features in the Windows Server 2012 operating system, they asked to join the Microsoft Rapid Deployment Program (RDP) for the product. The company was specifically interested in the transparent failover capability in Windows Server 2012 and wanted to try the failover clustering improvements with its TUBA application. “We have been in a number of Microsoft RDPs over the years, and they’ve always been great experiences,” Garcia says. “Microsoft is right there side by side with us and open to suggestions and feedback about the product.”
Test Transparent Failover with TUBA
For the Windows Server 2012 RDP, Equifax set up three TUBA clusters—for development, quality assurance, and production—all running Windows Server 2012 Datacenter. The Datacenter edition provides operating system licenses for all virtual machines on a licensed host server.
Equifax chose HP ProLiant DL380 Gen8 servers for its Windows Server 2012 RDP, because the team wanted to test Windows Server 2012 on the latest and most powerful hardware. “For the x86 processor market, we’ve standardized on HP servers,” Francois says. “The Gen8 server has features that enable the operations team to log and retrieve internal hardware errors regardless of the operating system state. This is a great thing for our operations staff, which has access to much more data when a server has problems.”
Equifax worked with Oakwood Systems Group, a St. Louis, Missouri–based member of the Microsoft Partner Network, during the RDP. Oakwood worked with Equifax when it implemented Microsoft System Center 2012, and the experience had been positive. Oakwood built the RDP hardware and software environment, using the Hyper-V technology in Windows Server 2012 to create virtual machines in each host server for hosting virtualized file server clusters.
“The ease and speed with which Windows Server 2012 installed was notable,” says Christopher Meyers, Senior Consultant at Oakwood Systems Group. “The Server Manager interface makes configuring the operating system easy and efficient.”
With the transparent file server failover capability in Windows Server 2012, a workload running on an active-active cluster remains available even when the particular cluster node that it is running on fails unexpectedly. Further, if a hardware or software failure occurs on a cluster node, transparent failover lets file shares fail over to another cluster node without interrupting server applications as they read and write data onto the file shares. Transparent failover delivers higher server utilization and thus lower server costs, because organizations like Equifax have the ability to share capacity across all nodes in a cluster and need fewer backup servers.
“With the tested application running a full load, we successfully failed over the cluster resource multiple times without error,” Meyers says. “The active cluster node was taken down with a hard power-down without application impact. The application owners were very impressed.”
By eliminating application interruptions during security updates, Francois’s team can simply notify application teams when an update will happen but doesn’t need them to be involved. “The ability to perform updates to file servers as a notification rather than a scheduled event will revolutionize how we operate,” Francois says.
Equifax is running its Windows Server 2012–based TUBA clusters in production and plans to upgrade other application clusters to the new operating system when Windows Server 2012 is released to manufacturing.
Test Other Improvements
During the RDP, Equifax also tested the new Server Message Block 3 (SMB3) feature in Windows Server 2012, which enables organizations to store application data on file shares and take advantage of the new SMB3 protocol. “Our TUBA clusters were I/O [input/output] performance-bound when accessing information on file servers, and the SMB2 protocol allowed only one I/O connection per process,” Garcia says. “SMB3 spreads that connection across multiple I/O links, so if I have four-gigabit connections, all four connections can be used to significantly improve performance and throughput.”
Equifax also evaluated Chkdsk runtime improvements in Windows Server 2012. Chkdsk (short for check disk) is a command on Windows-based computers that displays the file system integrity status of hard disks and can fix logical file system errors. The file system in Windows Server 2012 quickly self-heals more issues online without requiring Chkdsk to run. This reduces Chkdsk execution frequency. “We run very large volumes on our file servers, and each may have multiple terabytes of disk space,” Garcia says. “Prior to Windows Server 2012, if we had a problem and ran Chkdsk, it could take many hours before the file server returned to operational status. With Windows Server 2012, the server returns to an operational status quickly while Chkdsk performs the validation in the background.”
Benefits
By upgrading to Windows Server 2012, Equifax will be able to significantly reduce maintenance-related downtime and work, which frees the IT staff to spend more time on new initiatives. The company will save money on servers and give its IT staff a better work-life balance.
Improve File Server Infrastructure Availability by Up to 40 Hours Annually
Equifax can use transparent failover in Windows Server 2012 to apply updates to clusters without scheduling downtime. “Windows Server 2012 delivers a revolutionary change in how we perform cluster updates,” Francois says. “We won’t have to schedule application shutdowns, fail over workloads to different nodes, and bring in the application teams to reconnect their applications. We expect to eliminate up to 40 hours annually of planned and unplanned downtime for our file server infrastructure.”
With Windows Server 2012, Equifax IT can continue to meet its 99.85 percent uptime commitments and, over time, raise its SLA. “In the information service business, it’s a big competitive differentiator if we don’t have to ask our customers for a Saturday night service outage while our competition does,” Francois says.
Create More Time for Innovation to Fuel Business Growth
With less time consumed by cluster updates, the Equifax server and application teams have more time to focus on the innovation needed to fuel business growth. “There are only 24 hours in the day, and ensuring service availability always goes to the top of the list,” says Garcia. “If we can reduce the time devoted to service availability, we can focus more effort on innovation. Our development team’s priority is creating new products, not working on support issues. With Windows Server 2012, we can be even more forward-thinking as an IT team.”
Reduce New Server Deployments by Up to 30 Percent Annually
Equifax will also be able to trim the number of servers it buys by upgrading to Windows Server 2012. “With Windows Server 2008 R2, if I needed five file servers, I had to buy 10—five in my active cluster and five sitting there as insurance,” Francois says. “With Windows Server 2012, if I need five file servers, I only have to buy seven, with five in the active cluster and only two as insurance. We now have shared capacity across all the nodes, giving us a server efficiency of about 80 percent rather than our current 50 percent efficiency. With this kind of efficiency, we stand to reduce new server deployments by 20 to 30 percent annually.”
Improve Work-Life Balance
With Windows Server 2012, Equifax can also deliver a better work-life balance for its busy IT staff. “Anything I can do to minimize the number of people involved in a maintenance event delivers a big benefit, both in more efficient use of resources and in staff morale and retention,” Francois says. “Rising maintenance requirements were pushing us deeper into off-hours work, which is not when people want to be here. Any product that promotes better work-life balance means a lot to me as someone who runs a data center.”
Windows Server 2012
Windows Server drives many of the world’s largest data centers, empowers small businesses around the world, and delivers value to organizations of all sizes in between. Building on this legacy, Windows Server 2012 redefines the category, delivering hundreds of new features and enhancements that span virtualization, networking, storage, user experience, cloud computing, automation, and more. Simply put, Windows Server 2012 helps you transform your IT operations to reduce costs and deliver a whole new level of business value.
For more information, visit:
www.microsoft.com/en-us/server-cloud/windows-server/2012-default.aspx