Proposal for virtualization business continuity plan.

As part of the COOP (Continuity of Operations Plan) in 2010, a number of architectural changes in infrastructure were developed to improve service availability and to protect data.

  • Fire suppression system installed in Memorial Library Computer Center and Morris Hall fiber room.
  • Replacement of air conditioners in Memorial Library Computer Center to address air restriction problems, frequent equipment failures and to reduce energy consumption.
  • Secondary data center was built in McEloy Center. This data center runs on a separate emergency generator than ML in event of a failure with that emergency power source.
  • A redundant network core switch/router equipment was installed in McElroy for network redundancy.
  • A first phase of fiber was run from data closets to McElroy for network redundancy.
  • A secondary Storage Area Network was moved to McElroy to mirror data with ML data center.
  • Data backups were migrated from tape backup to a disk backup system and located in McElroy for off-site backup storage.
  • Most servers and services were virtualized onto a VMWare ESX environment and designed to be failed over from ML data center to McElroy data center (or vice versa) in case of a failure in either facility.
  • Servers and services that cannot be virtualized have been designed with redundancy wherever possible and those servers and services split between ML and McElroy data centers.
  • A redundant 10 Gbps fiber was installed by the State of Minnesota for WAN and Internet redundancy from the MSU campus to the state-wide network and to the Internet.
  • An agreement was reached with Scott County that they will provide co-location space in one of two data centers within their existing facilities in Scott County and will allow MSU to utilize direct fiber that is in place to connect the MSU campus to the Scott County data center at no charge.

Although the business continuity plan is nearly in place, several things need to be done in order to fully achieve redundancy and disaster recovery business objectives.

  • Server equipment, storage and data backup need to be moved from McElroy to Scott County in order to achieve a recommended distance of 13.5 miles or greater between data centers. Prior Lake would be 50 miles and would provide adequate distance in event of a very weather or power disaster.
  • Some virtual servers and services can be moved to cloud-based PaaS (platform as a service), and SaaS (software as a service) to provide distance and fault-tolerance.
  • Additional fiber needs to be installed to provide better campus network redundancy in event of a damaged fiber within the tunnel system, mechanical space or the building’s distribution network.

What will this plan achieve? Some critical services will be able to remain active and operational in event of a campus disaster, widespread power disruption, or major network outage. This redundancy will be in place without the need for ITS staff or many other entities being involved in moving equipment or services and involving other companies to help in event of a disaster. In effect, service redundancy will be in place and ready for a disaster in case one would happen at a time when staff may not be available to do this work. Additionally, it will provide critical data backup and network backup in event of a more localized disaster on campus.

Co-location with Scott County will provide other benefits as well. The dedicated 10 Gbps fiber link will provide high speed dedicated bandwidth between data centers that will be as fast as if the equipment were on the same computer floor. Extremely low network latency over the direct fiber will allow SAN mirroring and very quick failover of servers and data. Finally, Scott County has a separate point-of-presence with the OET State of Minnesota network which will allow network traffic to be routed from Scott County to the Internet in case the MSU campus becomes inaccessible to the rest of the world. All of this is achievable at extremely low cost to MSU thanks to the partnership with Scott County.

What is needed in order to complete this plan?

1)In order to co-locate with Scott County, all equipment must be reduced to two racks which includes UPS equipment.

  1. Purchase blade server technology to reduce physical server space.
  2. Purchase a more-dense SAN to reduce physical server space.
  3. Purchase rack-mounted UPS sized for the equipment in the rack.
  4. Migrate virtual server hosts from VMWare ESX to Microsoft Hyper-V to reduce software costs. NOTE: one ESX host may have to remain in place to cover VMs that don’t support Hyper-V.

2)Fiber optics need to be purchased to interface the terminated fiber on each end.

3)Routing design will have to be done to accommodate the network move.

4)A network switch/router and firewall will need to be purchased for the Scott County equipment racks.

5)An agreement will need to be in place with OET (State of Minnesota) to allow service traffic to be routed through the Scott County egress.

Several services and virtual servers can be moved to cloud services such as Azure and Amazon Web Services pending contractual approval, although we don’t have an idea of the impact or savings that will occur using these services.