Dell™ PowerEdge™1650:
Rack Impacts on Cooling
for High Density Servers
Enterprise Systems Group (ESG)
Dell White Paper
By Paul Artman, David Moss, and Greg Bennett
August 2002
Contents
Introduction 3
Server Internal Cooling 4
Rack Impact on Server Cooling 5
Thermal Data 6
Conclusion 8
Appendix- Test Pictures 10
August 2002 Page 11 Dell Enterprise Systems Group
Section 1
Introduction
The growth of the Internet fueled the need for rack dense servers as service providers (ISPs/ASPs), dot-coms, enterprise customers, or any organization leveraging Internet technologies all struggled with the same fundamental issue – data center space. Even as the dot.com bubble burst and the economy weakened, rack dense servers remained popular and have continued to gain share with respect to the rest of the server market (rack servers currently account for approximately 40 percent of the overall market[1], and account for over 50 percent of Dell sales). The primary reason for this growth is that data center space is either scarce, expensive, or both for most organizations; so whether customers build their own data centers or lease space from a service provider, companies must maximize their return by deploying as many servers as possible in the smallest space possible.
These factors have made 1U and 2U[2] servers particularly attractive. Moving forward, servers will get even denser with the advent of Server Blades and Modular Blades. With this increased density, however, comes increasing power and thermal concerns as data center managers struggle with the ability to power and cool these rack-dense configurations.
This paper provides some general guidance regarding data center design, deployment, and cooling, and uses the PowerEdge 1650, Dell’s dual-processor 1U server, as a basis for examining the impact of power and cooling in an extremely rack dense environment.
August 2002 Page 11 Dell Enterprise Systems Group
Section 2
Server Internal Cooling
The ultimate temperatures seen by internal server components will vary from server to server depending on the configuration, application, position in the rack, position in the data center, the amount of cabling, etc. Dell servers are designed to cool from front to back and are tested to meet elevated temperatures exceeding what is commonly found even in the worst-case locations in a data center. All Dell servers are designed for a 35° C (95° F) inlet temperature (into the front server surface) at maximum component power dissipations. This means that when run at full load, internal components are maintained below their recommended guidelines, or below the more stringent guidelines imposed by Dell.
In a redundantly cooled system, the components meet these temperature requirements even in the event of a fan failure. With processors, servers are usually designed to cool to meet the requirements of future processor speeds, up to the maximum speed expected (based on the Intel specification). So, for a server component to exceed allowable operating temperatures, the server must be operating at maximum power (a maximized application, maximum processor speed) in an environment exceeding 35° C (95° F). Since most data centers are cooled to the low 20° C (68° F) range, there should be significant margin. The remainder of this white paper quantifies some of the variables introduced by the data center that may affect that margin.
Section 3
Rack Impact on Server Cooling
The effect of installing a rack full of high-density servers is a moderate rise in system component temperature. This white paper discusses the impact of racking servers on system component temperatures compared with bench top temperature measurements. In addition, the impact and variance in different rack locations is investigated. Finally, the reasons for increased component temperatures are discussed.
High-density servers often have reduced system airflow due to the added impact of the rack, cables, and cable management arm. Factors for system-reduced airflow include the following:
· Blockage due to cable management arms
· Blockage due to cables
· Rack doors
In addition, there can be temperature variance in inlet temperature to the rack due to the following:
· Low data center cooling flow rates
· Poor flow balancing under the raised floor
· Proximity to perforated tile location resulting in temperature variance from rack to rack
· Temperature variance in the rack (top to bottom)
In summary, component temperatures can be higher due to decreased flow rate through the server and increased temperature in front of the server. To try to quantify cumulative effects within a rack full of PowerEdge 1650 1U servers, Dell ran a series of thermal tests.
August 2002 Page 11 Dell Enterprise Systems Group
Section 4
Thermal Data
A Dell PowerEdge Rack 4210 42U (2m) rack was loaded with 29 PowerEdge 1650 1U servers to determine the impact on system component temperatures of installing multiple servers in a rack. The servers were configured with dual Intel Pentium III 1.26GHz with 512KB L2 cache processors. In addition to the servers, the rack contained a 1U keyboard/monitor tray and three 3U UPSs. The servers were cabled with dual NICs, dual power cords, and keyboard, video, mouse (KVM) cables for each server. This represents an above average amount of cabling for a 1U server (see pictures of test environment in Appendix). Cable management arms were also installed on all servers. For the testing, the front and rear doors of the cabinet were closed, and the side covers were in place. The test was conducted in a lab with no raised floor at room temperature (~70° F/21° C).
From this testing, three generalities were determined:
· Server inlet temperature could vary by as much as 6° C (11° F), depending on location up and down the rack. Temperature increased toward the top.
· Dense rear cabling reduces server flow rates. Reduced flow rates could raise component temperatures up to 5°C (9°F).
· Component temperature increases were not uniform; component temperatures at the rear of the server increased more than components at the front.
Inferences about these three points:
· Higher Top Location Entrance Temperature
Within any enclosure, whether it is a rack enclosure or a room enclosure, there is always a natural tendency toward a gradient of lower cool air to higher warm air. Outside the rack, the gradient might be minimized or virtually eliminated by the high velocity air shooting up from the floor tiles. Inside the rack, each server imparts a small amount of heat to the surrounding air and surfaces (known as convection and radiation). Because of the natural tendency toward a temperature gradient, the air surrounding and being drawn into the servers will be higher toward the top of the rack. With the lack of a raised floor in the Dell lab test, the observed 6°C variance up the rack may have been exaggerated due to a room gradient outside the rack. It is important to note that the flow rate through the floor tiles should be at least equal to the cumulative flow rate expected through the servers. If it is not, air drawn into the top servers (furthest from the cooling tiles) is possibly pre-heated air. It is also important to note that attempts to expel air from the top of a rack should be closely scrutinized. Apart from providing venting at the top of the rack, Dell does not recommend other means of expelling air from the top of the rack without a complete analysis on the effect to each server component. In some cases this has been known to raise server temperatures by causing enough backpressure in the rack to reduce the flow rates through the servers.
· Heavy cable blockage reduces server flow rates
Most 1U servers should not be as densely cabled as the test case, so they should not exhibit as much of the 5°C temperature increase seen in the test. Because of the large amount of cabling relative to the height of a 1U server, 2U and greater servers should see less of an impact due to cabling.
· Rear Components Experience Greater Temperature Increases
When the overall flow rate through the server is decreased by added flow resistance (cables, rack doors, etc), the air does not pick up the heat as efficiently. The air flowing through a server is heated by and increases in temperature as it passes each successive component or heatsink. The resulting air temperature increase at any point in the server is inversely proportional to how fast the air is moving through the server. If the speed is cut in half, the increase in the air temperature doubles at any point within the server. For instance, if the air is heated up 1 degree as it passes the first component and is 5 degrees higher as it passes the last, it should heat up 2 degrees past the first and 10 degrees past the last component if the flow rate were halved. So, on a relative basis, the increase in the air temperature around the rear components is larger than around the front components for a server with a decreased flow rate.
Section 5
Conclusion
At least three factors inherent in data centers can lead to an increase of internal server temperatures:
· Decreased flow rate through the server due to cabling and rack doors
· Uneven temperatures in front of each rack:
o Proximity to nearest cooling tile
o Imbalance below raised floor
· Increased temperature gradient from bottom to top in a rack
This paper discussed and quantified the first and third factors. The second factor is data center specific.
It is important to note that the component temperature increases discussed in this paper are considered extreme: systems dissipating maximum power (at full Intel speed/thermal specification), greater than average number of cables in a rack dense platform (1U), and location at top of the rack. Based on the results in this paper, a system operating in the top space in a rack and blocked by heavy cabling might see temperatures as high as 11°C more than a bench test of the same system. A system like the PowerEdge 1650 is rated to a bench test inlet temperature of 35°C (95°F) and will maintain reliable component temperatures even with an internal fan failure. An equitable comparison of the 35°C (95°F) bench test would be a server with maximized operation, operating in the top of a closed rack, with heavy cabling, where the temperature at the bottom of the rack is 24OC (75OF). Most data centers will measure temperatures much lower at the bottom of each rack.
Suggestions for optimizing cooling within a data center include:
· Data centers should be set up with hot and cold aisles. In other words, racks should be set up back-to-back (hot aisle) and front-to-front (cold aisle).
· Care should be taken to ensure uniformity of temperature in the front of each rack by balancing flow rates under and through the floor.
· Perforated tiles should be used in cool aisles. Different perforation levels (e.g., 25%, 50%, etc.) can help balance airflow within a data center. Hot aisles should not contain perforated tiles; this lowers the cooling ability of the cool aisle.
· If there are discrepancies between advertised system operating temperatures, systems with lower operating temperatures should be located lower in the rack.
· The addition of rack fans or fan trays is not recommended. In some cases, additional top mounted rack fans have actually impeded server thermal performance, but this may not be the case in every environment.
· The servers are designed to expel their own heat, therefore the data center should optimize the efficiency at which that expelled heat is picked up and transported to the HVAC.
NOTE: The above suggestions are guidelines only. Results will vary by specific data center attributes and server configurations. Dell recommends running tests to help to determine ways to optimize cooling within a particular data center environment.
THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND.
Dell and PowerEdge are trademarks of Dell Computer Corporation. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products. Dell disclaims proprietary interest in the marks and names of others.
©Copyright 2002 Dell Computer Corporation. All rights reserved. Reproduction in any manner whatsoever without the express written permission of Dell Computer Corporation is strictly forbidden. For more information, contact Dell.
Information in this document is subject to change without notice.
August 2002 Page 11 Dell Enterprise Systems Group
Section 6
Appendix- Test Pictures
The picture above is a close up of the back of the rack with the cable management arms. It also shows the thermocouples attached to a one of the servers in the rack to determine the test results.
August 2002 Page 11 Dell Enterprise Systems Group
[1] Based on market data from the IDC Quarterly Server Tracker, Q1 2002
[2] A “U” is a widely accepted form of rack measurement. It represents 1.75-inches and refers to the height of a server/storage system or the rack itself.