Data Center Design
Christopher Geyer
Department of Computer Science
University of Wisconsin Platteville
Platteville, WI 53818
Abstract
Businesses today are gathering data at an ever increasing rate. Today’s IT departments are being called on to store this data for use and manage the servers that track the data. When a company’s needs expand beyond a small data server, many will look to either an outsourced data storage location or building their own data center. Some of us may have idealized views of a “server room”, but don’t know what needs to be taken into account when planning to build one. Power, cable routes, cooling, and physical space are all concerns that need to be addressed during the planning phase of a data center.
Introduction
The use of data centers are an integral part of our lives. When ever we use a credit card, visit a website, pay at a tollbooth, or use a phone a data center is being utilized. Many programs that use databases access this information from a data center. Even with the staggering use of data centers many continue to disregard their importance or understand what is entailed in a data center.
Data centers are highly secure, fault-resistant facilities housing equipment that connect to telecommunications networks. The facilities accommodate servers, switches, routers, and modem racks. Data centers support corporate data bases, web sites, provide locations for CLECs, ISPs, ASPs, Web hosting companies, DSL providers and other IT services. [5]
Why is it important for Computer Science majors to have an understanding of data center design? People going into the industry may find themselves in a position to influence decisions on how to implement their company’s new data center or influence changes being made to an existing data center. Also, knowing where and on what your software will be running on can give insight in the development process and can take away false assumptions.
Data center design is an extremely large topic covering many different areas of study. The primary elements of data center design include the RLU definitions, the site, the command center, cable management, network infrastructure, environmental controls, and power. The primary rule of a data center is “Form follows function.” [1]
Elements of a Data Center
RLU Definitions
A rack location unit definition is a way to standardize measurements for equipment that will go into a data center. RLUs allow for flexibility to configure and reconfigure racks to meet changing demands, budgets, and further optimize space in a data center. Equipment within the racks, where the equipment is held, specify equipment space in increments of U’s, where one U is 1” high, 19” in width and can vary in depth.
RLU definitions specify parameters for a given type of equipment to be properly placed in a data center. The naming convention for an RLU definition is RLU-X where X is an alpha or alpha numeric unique identification code. The first parameter of a RLU definition is the weight of the equipment. The second parameter is power; this includes circuit requirements, outlet type and quantity, and maximum watt intake. The third parameter is cooling in BTUs per hour to be cooled. The next parameter is physical space in number of U’s and depth in inches. The fifth parameter is bandwidth; this includes the type of connections required (Cat6, Multi-mode Fiber etc.) and quantity. The last parameter is functional capacity; this includes such specifications as storage capacity and/or gigaflops of processing power.
The Site
The site is the location where the data center is to be built. This location can be within an existing structure or a new facility. When considering the site of a data center the following should be taken into consideration: proximity to the organization, proximity to internet backbone, possible natural disasters (such as flooding, earthquakes, tornados etc.), and location to major roadways.
Structural aspectual should be considered when choosing the site. The use of a raised floor can impact the specifications for the site. When dealing with a raised floor ceiling height matters. The average raised floor takes up two feet of vertical space and current recommendations for a data center include a three foot raised floor. So, in a ten foot room the space has been decreased to seven feet. The remaining seven feet do not leave much room for cable tracks at the ceiling. A more appropriate height is 12’ giving 9’ with a 3’ raised floor and plenty of room for the cable tracks. Basement data center locations near water are not a good idea. Periodic flooding can create problems within the datacenter. There must be a pathway for equipment to be moved in and out of the data center. Narrow hallways can make equipment transfer difficult along with sharp turns and long flights of stairs. Make sure the floor where the data center is to be located is rated for the estimated load. If the floor is not rated for the estimated load, serious structural problems can occur where the raised or sub floor collapses or deforms.
The Command Center
The command center is an area to control and oversee the data center. This entails security surveillance, environmental monitoring of the data center, supporting equipment, and analyzing of logical infrastructure. The command center is connected to the data center, providing a location for staff to over see functional aspects and provide support and prevent potential problems.
Cable Management
Cable management is a structure designed to organize and layout cable flow through the data center. The use of cable management is very important to the success and efficiency of a data center. Cable management involves the use of ceiling cable trays providing a pathway for data lines from their origin to their destination throughout the data center. This adds ease of installation, maintenance, future expansion, and separation from power and HVAC (Heating, Ventilation, and Air Conditioning). A raised floor is used to manage power lines through the data center.
Network Infrastructure
The network infrastructure includes switches, routers, and support equipment to provide the core logical infrastructure for the data center. This equipment allows the servers to communicate with other equipment within the data center and outside of the data center.
Each rack needs to be supplied with appropriate connection media such as: Cat6 copper, multi-mode fiber, and/or single-mode fiber. The equipment held within the rack dictates what needs to be run to the rack. Most server use Cat6 but some WAN devices use Single-mode fiber.
While expansion is good, too many of the wrong media type cable can cause problems of its own. This wastes budgets and can make it harder to get the appropriate media to a given location. Only run the expected media types.
Environmental Controls
The environmental controls include HVAC (or Heating, Ventilation, and Cooling), humidity control, and air filtration. The data center must keep the environment hospitable for the proper operation of equipment held within it.
Power
Power includes power distribution, power stability controls, and redundancy. Adequate power needs to be supplied not only to the building but to the individual equipment as well. The power distribution units should include plenty of room for expansion. This room does not have to be occupied until it is needed. Surge suppression needs to be implemented to protect from power surges. In the event of a lighting strike the power can be directed away from the sensitive equipment.
Proper grounding of equipment is necessary to prevent ESD damage to equipment. Without proper grounding static electricity has no where to go. Cable layout should place the power in the raised floor, away from data lines to prevent current induction into the CAT6 and other copper data lines
Power redundancy should include multiple forms of redundancy. This is necessary to create a continues clean power source. The first form is battery-feed UPS units which systems directly connect to for immediate power failure. The second is power generators for extended power failure protection. Unstable power creates unexpected down time and degrades the life of the equipment that it is connected to.
Criteria
Budgetary
An important question to ask, and should be the first question asked, is “What is the available budget?” [1] While this question may seem simple it is very important and can be very complex. The data center cost must be separated from server and staff costs, also their must be separation of the budget vs. the running budget. The next question to ask is “Can the scope of the project be achieved with the current budget?” [1] It is important to recognize early on if current budget constraints make the project unfeasible. The next question is “What are the actual funds needed to create the data center?” [1] A static number must be set to base preliminary cost calculations. The last question is, “How will funds be distributed & can they be redistributed?” [1] The elements of the data center must be allocated individual costs: the site, command center, cable management, network infrastructure, environmental controls, and power must be allocated some funds to begin the project. With these individual costs each project group will know what costs they can work within.
Physical Constraints
Physical Constraints are also very important. The available space and weight of equipment need to be determined. This covers how many U’s of rack space are required for the equipment and estimated weight of the equipment. Power requirements are used to estimate power peak intake and nominal along with backup power uptime requirements for the data center. The cooling includes requirements for both redundancy and adequate BTU output for the equipment; the cooling system should be selected to output 60% of total capacity. The redundancies for cooling can include backup power for the cooling system and additional cooling units. The last physical constraint is bandwidth, this term does not just mean capacity of the internet pipe but the connections and type of connections inside the data center. This can very greatly depending on what services are being held in the data center but a primary consideration must be expandability.
System Availability Profiles
The system availability profiles are used to determine the importance each system held within the data center. The first step is the categorization of the systems in a data center. Each system must be categorized from mission-critical (needing 24x7x365 day uptime) to non-critical systems. This can be broken down into degrees of uptime importance depicted on a graph showing the level of uptime importance over time. The system availability profiles also include device redundancies which are the number of supplemental systems (or systems that can handle the tasks of others during failure/overload). The profiles include power redundancies which include the number of feeds from different parts of the power grid, number of UPS systems (universal power systems), and backup power generators. The next part of the profile is the cooling redundancies. This includes the number of extra HVAC (or Heating, Ventilation, and Cooling) units in the event of one or more failing or above normal load. The last part of the profile is the network redundancies. This includes the number of redundant network components and connections to alternate ISPs in the event of connection failure. A common error is the want to categorize all systems with the same level of critical importance.
Security
Good security is vital to a data center. A system is most venerable at its weakest point; physical access can be a very big chink in a server’s armor. A unique identification security system should be used to ensure physical access with accountability. A good example would be pin-codes access doors or RF Badges to restrict access to the data center and supporting systems.
Levels of Access should be used depending on how elaborate the data center is. Different areas should have different security levels; in particular to co-location scenarios, where some customers demand segregated security for their equipment. Restricting access in levels keeps personnel in the locations they work in and limits the damage in the event a security code is stolen to the area that code has access to.
The best physical security is biometric. While this security is more expensive than traditional security is has the added advantage that users do not have to remember codes and can not loose keys/badges.
Monitoring in a data center can add value to a security system. Data centers tend to have personnel in them all hours of the day, which can be hard to manage. The use of surveillance equipment can ensure a deeper level of security and accountability.
Expandability
Looking at the past we can look to the future. Technology changes continuously making versatility a primary focus in a data center. At one point in time a single computer occupied the space of an entire data center. That same space can be now occupied by thousands of servers today.
It is important to have an expandable data center to be able to meet future needs. One way to create expandability is to create more RLUs than current needs dictate. While servers continue to occupy less space and increase processing power, trends show computing needs are still growing faster than computing power. Another way to create expandability is to create extra room within the power distribution units. As computer technology continues to consolidate down, power requirements continue to rise and more servers are able to be occupying the same space causing increased power draw in a highly dense area. Also, HVAC needs to be expandable. With power requirements raising more BTUs are given off, which can strain a HVAC system that was previously adequate. Initial requirements should put the cooling systems at 60% capacity, leaving room for expansion. Over cooling is a possible problem when trying to create expansion in a data center. This happens when a HVAC system is too big for a given area. This causes the unit to run below optimal capacity, making the unit build up ice and not function correctly. Physical space is the most obvious key expansion. Room on the data center floor should be created for the sole purpose of expansion. If possible in the budget, lay down the infrastructure for those future racks. The creation of the infrastructure when the data center is being built will be much cheaper than doing so later on. This includes network cable trays, HVAC infrastructure, and power.