Data Center Standard(Attach C)

(Attach C)

State of Iowa Data Center Standard

October 8, 2009

Purpose:

To provide a data center standard that protects critical computing infrastructure from risks associated with loss of power, fire, unmanaged temperature, or unauthorized access.

Overview:

This standard is intended to apply to all State of Iowa data centers as defined below. The intent of this standard is to reduce risk and increase the longevity of critical network assets.

Several Iowa agency network engineers conducted research and toured both government and private data centers to provide state agencies with the following data center standard practices and best practices.

Scope:

For the purpose of this standard, all State of Iowa participating agencies, boards or commissions operating a data center facility will ensure the proper management, risk mitigation, redundancy, and reliability of the following data center areas:

  • Power
  • Physical Security
  • HVAC
  • Fire Suppression
  • Cable Management

Agencies will be required to comply with the provisions as stated in the standard practice section of this standard no later than June 30, 2010.The Technology Governance Board TGB has the authority to determine entity compliance or non-compliance of this standard. Failure to comply with this standard will result in a review by the TGB.

Updates:

This document will be reviewed at least every two years and updated as needed.

Definitions:

Selected terms used in the Data Center Standard are defined below:

  • Agency - means any agency as listed in Iowa Code Chapter 8A Section 201 paragraph 4.
  • Best Practice–is a technique, method, process, or activity that is believed to be effective at delivering a particular outcome. Best practices noted in this document are viewed as recommendations, not requirements.
  • Critical IT infrastructure– is defined by business service restoration within 72 hours in an agency’s disaster recovery plan.

1

Data Center Standard(Attach C)

  • Data Center –is a facility dedicated to the purpose of securing data and systems and is used to house network server systems and associated components. It includes networked servers, controlled access, environmental controls such as air conditioning and fire suppression, power and electrical systems, and networking equipment. The threshold of what facilities is considered to be a data center is provided below:

Space Type / Typical Site Infrastructure System Characteristics
Localized data center / Typically use under-floor or overhead air distributionsystems and a few in-room computer room airconditioner (CRAC) units. CRAC units in localized data centers are more likely to be air cooled and have constant-speed fans and are thus relatively low efficiency. Operational staff is likely to be minimal, which makes it likely that equipment orientation and airflow management are not optimized. Air temperature and humidity are tightly monitored. However, power and cooling redundancy reduce overall system efficiency.
Mid-tier data center / Typically use under-floor air distribution and in-room CRAC units. The larger size of the center relative to those listed above increases the probability that efficient cooling, e.g., a central chilled water plant and external storage central air handling units with variable speed fans, is used. Staff at this size data center may be aware of equipment orientation and airflow management best practices. However, power and cooling redundancy may reduce overall system efficiency.
Enterprise-class data center / The most efficient equipment is expected to be found in these large data centers. Along with efficient center cooling, these data centers may have energy management systems. Equipment orientation and extensive airflow management best practices are most likely external storage implemented. However, enterprise-class data centers are designed with maximum redundancy, which can reduce the benefits, gained from the operational and technological efficiency measures.
  • Environmental Stability – refers to the controls for fire suppression, temperature, humidity, and air quality.
  • Networking and data cabling – terminology pertaining to the installation and maintenance of twisted-pair and optical fiber cabling.
  • Physical Security–describes both measures that prevent or deter attackers from accessing a facility, resource, or information stored on physical media and guidance on how to design structures to resist various hostile acts.
  • Power and Electrical Systems –terminology relating to reliable, conditioned powerthat is provided for computer and networking systems located within a data center.
  • Standard Practice – is a technique, method, process, or activity that is believed to be effective at delivering a particular outcome. Standard practices noted in this document are viewed as requirements, not recommendations.
  • Visitor – Any non-authorized state personnel, non-authorized vendors, or the general public using or touring State of Iowa facilities.

Data Center Standard Practices:

State of Iowa data center standard practices require that:

  1. The following physical security practices be implemented:
  2. Barriers shall exist that restrict access to data center rooms;
  3. Physical access shall be restricted to selected personnel, with an auditable physical security process using security card access. If a security card system is not present, room(s) shall be secured by key or keypad system. A key system shall have an audited checkout process;
  4. Access shall be restricted to employees and vendors who need to maintain equipment or infrastructure in the room(s). An escort is required for all visitors and vendors to the room(s). In addition, visitors and vendors shall be given a physical access token (badge or access device) that identifies visitors as non-employee(s);
  5. Whenever practical, critical IT infrastructure as designated by the director in consultation with the State CIO, should reside inside data centers. It is not the intent to apply this standard to non-critical servers, network infrastructure or communication assets located inside of unimproved utility closets;
  6. If the site is subject to Payment Card Industry (PCI) rules and requirements, video cameras shall be used to monitor sensitive areas. Recorded video shall be retained for a minimum of three months.
  7. The following environmental stability practices be implemented:
  8. Smoke detectors and sprinkler systems or clean agent fire suppression gaseous systems are required;
  9. Monitoring, alarming and alerting shall be in effect in case fire and all fire suppression systems must be installed and maintained in accordance with local fire code;
  10. Air handling equipment must supply sufficient cooling and humidity controls to meet the most restrictive equipment cooling and humidity specifications of the equipment residing within the data center;
  11. Storage of flammable or combustible materials (e.g. wood, cardboard and corrugated paper, plastic or foam packing materials, flammable liquids or solvents) shall not be allowed in the room(s).
  12. The following power and electrical system practices be implemented:
  13. All devices, including servers, networking equipment, etc., shall be protected by conditioned power and suitable UPSsufficient to maintain power until power is restored through commercial power or generator backup;
  14. Cabinets and racks shall be properly grounded, in accordance with existing commercial building grounding and bonding standards.
  15. The following networking and data cabling practices be implemented:
  16. Data cabling shallbeinstalled and tested in accordance with industry standards and best practices listed in the ANSI/TIA-568 family of Telecommunications Standards;
  17. Data cabling routed outside of cabinets shall be protected and contained, using solutions such as cable trays, flexible conduit, J-hooks, etc.;
  18. Data cabling routed within or between bayed cabinets shall be done in a manner so as to not inhibit air flow through the cabinet. Cabling within a cabinet shall be dressed in such a way as to enhance air flow through the cabinet;
  19. Twisted-pair and fiber panels shall be labeled, and all cables shall be labeled at both ends, including twisted-pair and fiber patch cords;
  20. Cabling, cable lengths, and terminations shall meet current BICSI cabling and termination standards.
  21. Waivers to the standard may be granted using current Iowa Administrative Code Chapter 25, Section 11-25.6 (8A).

Data Center Best Practices:

State of Iowa data center best practices recommend that:

  1. The following physical security practices be implemented:
  2. Video camera surveillance and security escorts should be considered in cases where large data centers contain sensitive information;
  3. Gates or gate-like systems should be used above dropped ceilings and below raised floors to deny access into false floor/ceiling space;
  4. Biometric identification systems and processes are recommended for access to highly sensitive areas of a data center;
  5. Where possible, mantraps should be established to segment areas of the data center, with location-based access only;
  6. Limit or avoid windows in the room(s);
  7. Food and drink should not be allowed.
  8. The following environmental stability practices be implemented:
  9. Redundant cooling is recommended. N+1 or outside air should augment cooling systems. Use of outside air should be considered to help economize cooling;
  10. A clean agent fire-suppression system such as FM-200 is recommended, where possible;
  11. Monitoring, alarming, and alerting shouldbe in effect for instances of temperature and humidity thresholds and failures;
  12. Monitoring, alarming, and alerting are recommended for water detection;
  13. Blanking panels should be placed in cabinets to help direct air flow through rack-mounted devices;
  14. Temperature and humidity range requirements should be measured at multiple entry points on equipment racks, and at the ventilation output ducts.
  15. The following power and electrical system practices be implemented:
  16. Power availability should be 100 percent and should guide decisionmaking on UPS and power distribution;
  17. Monitoring, alarming, and alerting should be in effect for instances of UPS thresholds and failures, and power or breaker failures;
  18. Room-level PDUs should be protected by room UPS;
  19. Cabinet-level PDUs should be protected either by room or cabinet UPS;
  20. The following networking and data cabling practices be implemented:
  21. Data cabling installers should make a best effort to maintain neat and easily identifiable cabling systems, in order to support debugging and documentation efforts;
  22. Data cabling exterior to a cabinet should be routed through overhead cable trays, where possible, and twisted-pair and fiber cabling should be segregated within such trays;
  23. Data cabling installers should test all new, installed cables, and test results should be provided to the customer in electronic form.

Data Center Standard Appendices

Related Reference Materials:

When implementing this standard please reference the following materials used to create best and standard practices:

  1. Physical security practices:
  2. ANSI/BICSI-002 (Release December 2009)
  3. Data Center Physical Security Checklist (SANS) (See Appendix A page 5)
  4. 19 Ways to Build Physical Security into a Data Center (CSO) (See Appendix B page 9)
  5. Let's get physical: Data center security (searchCIO) (See Appendix C page 12)
  6. Environmental stability practices:
  7. Local Temperature Control in Data Center Cooling (ZDNet)
  8. Power and electrical system practices:
  9. ANSI/NECA/BICSI-607 (Release August 2009)
  10. J-STD-607-A
  11. Guidelines for Specification of Data Center Power Density (APC)
  12. Crash Course: Data Center Power (PowerManagement) (See Appendix D page 14)
  13. Networking and data cabling:
  14. ANSI/TIA/EIA-568-B series
  15. ANSI/TIA/EIA-569-B
  16. ANSI/TIA/EIA-942
  17. Siemon Network Cabling Standards Guide (Siemon) (See Appendix E page 18)
  18. Building Industry Consulting Service International (
  19. Data Center Facility Definitions (See Appendix F)

Appendix A. Data Center Physical Security Checklist (SANS)

Data Center Physical Security Checklist retains full rights

8D FDB5 DE3D F8B5 06E4 A169 4E46

This checklist is not a comprehensive physical security checklist. It merely provides a reasonablestarting point in regards to physical security for a data center.Always obtain written permission from proper management before performing security testing of any kind. Ensure that allthe testing performed (physical penetration, fire control, social engineering) is outlined explicitly in the permission receivedfrom management.Data Center Management may require that a Non-Disclosure Agreement be signed because of the potential exposure ofsecurity procedures. This checklist, as designed, only covers the physical aspects of your security setup. You will needother checklists to secure networks, operating systems, applications and other potential targets.

Using the checklist

The checklist is broken into two sections, property and people. Property includes, but is not limited to the building,infrastructure, servers, laptops and data. People is further broken down into users and outsiders. Users are employees,clients and others who need access to business data. Outsiders are those who are not directly employed by thebusiness. Cleaning crews, security guards, and service engineers are examples of outsiders.

Property Section - Place a check by each item that passes.

1.1 Site Location

____ 1.1.1 Natural Disaster Risks

The site location SHOULD be where the risk of natural disasters are acceptable. Natural Disasters include but are notlimited to forest fires, lightning storms, tornadoes, hurricanes, earthquakes and floods.

____ 1.1.2 Man-Made Disaster Risks

The Site Location SHOULD be located in an area where the possibility of man-made disaster is low. Man-made disastersinclude but are not limited to plane crashes, riots, explosions, and fires. The Site SHOULD NOT be adjacent to airports,prisons, freeways, stadiums, banks, refineries, pipelines, tank farms, and parade routes.

____ 1.1.3 Infrastructure

The electrical utility powering the site SHOULD have a 99.9% or better reliability of service. Electricity MUST be receivedfrom two separate substations (or more) preferably attached to two separate power plants. Water SHOULD be availablefrom more than one source. Using well water as a contingency SHOULD be an option. There MUST be connectivity tomore than one access provider at the site.

____ 1.1.4 Sole purpose

A data center SHOULD NOT share the same building with other offices, especially offices not owned by the organization.If space must be shared due to cost then the data center SHOULD not have walls adjacent to other offices.

1.2 Site Perimeter

____ 1.2.1 Perimeter

There SHOULD be a fence around the facility at least 20 feet from the building on all sides. There SHOULD be a guardkiosk at each perimeter access point. There SHOULD be an automatic authentication method for data center employees(such as a badge reader reachable from a car). The area surrounding the facility MUST be well lit and SHOULD be freeof obstructions that would block surveillance via CCTV cameras and patrols. Where possible, parking spaces should be a

minimum of 25 feet from the building to minimize damage from car bombs. There SHOULD NOT be a sign advertisingthat the building is in fact a data center or what company owns it.

____ 1.2.2 Surveillance

There SHOULD be CCTV cameras outside the building monitoring parking lots and neighboring property. ThereSHOULD be guards patrolling the perimeter of the property. Vehicles belonging to data center employees, contractors,guards, and cleaning crew should have parking permits. Service engineers and visitor vehicles should be parked invisitor parking areas. Vehicles not fitting either of these classifications should be towed.

____ 1.2.3 Outside Windows and Computer Room Placement

The Site Location MUST NOT have windows to the outside placed in computer rooms. Such windows could provideaccess to confidential information via Van Eck Radiation and a greater vulnerability to HERF gun attacks. The windows also cast sunlight on servers unnecessarily introducing heat to the computer rooms.Computer rooms SHOULD be within the interior of the data center. If a computer room must have a wall along an outsideedge of a data center there SHOULD be a physical barrier preventing close access to that wall.

____ 1.2.4 Access Points

Loading docks and all doors on the outside of the building should have some automatic authentication method (such as abadge reader). Each entrance should have a mantrap (except for the loading dock), a security kiosk, physical barriers(concrete barricades), and CCTV cameras to ensure each person entering the facility is identified. Engineers andCleaning Crew requiring badges to enter the building MUST be required to produce picture ID in exchange for the badgeallowing access. A log of equipment being placed in and removed from the facility must be kept at each guard desklisting what equipment was removed, when and by whom.Security Kiosks SHOULD have access to read the badge database. The badge database SHOULD have pictures of eachuser and their corresponding badge. Badges MUST be picture IDs.

1.3 Computer Rooms

____ 1.3.1 Access

There SHOULD be signs at the door(s) marking the room as restricted access and prohibiting food, drink, and smoking inthe computer room. There SHOULD be an automatic authentication method at the entrance to the room (such as abadge reader). Doors should be fireproof. There SHOULD only be two doors to each computer room (one door withoutwindows is probably a violation of fire code).Access should be restricted to those who need to maintain the servers or infrastructure of the room. Access should berestricted to emergency access only during moratoriums for holidays. Service Engineers MUST further go to the NOC toobtain access to computer room badges.

____ 1.3.2 Infrastructure

Computer Rooms should be monitored by CCTV cameras. Each computer room SHOULD have redundant access topower, cooling, and networks.There should be at least an 18" access floor to provide for air flow and cable management. Computer rooms should haveair filtration. Computer rooms should have high ceilings to allow for heat dispersal.(Level, 1)

____ 1.3.3 Environment

Each computer room SHOULD have temperature between 55 and 75 degrees Fahrenheit and a humidity of between 20and 80 percent.(Safeguarding, 5:2) Environmental sensors should log the temperature and humidity of the room andreport it to the NOC for monitoring and trend analysis(Level, 1).

____ 1.3.4 Fire Prevention

There SHOULD be a Halon or other total flooding agent solution in place in each computer room. There MUST be fireextinguishers located in each computer room. There MUST be emergency power off switches inside each computerroom. There MAY be respirators in computer rooms. There MUST NOT be wet pipe sprinkler systems installed.