Information Technology Services (ITS)

DATA CENTRE TRANSITION PROGRAMME PLAN PAPER

Introduction

The current Data Centre facilities used byBSO ITShave suffered six major outages in recent years as a direct result of issues with power supplies and cooling within these facilities. The risk of service failure has resulted in significant work being carried out to minimise the impact of outages in the legacy facilities and to better address known risks. The siting of regional Datacentres on hospital sites will neverprovide the levels of availability or priority required by critical IT services and so BSO have been engaged in procuring new Data Centre facilities within the Shared Public Sector Datacentre Project.

The contract for the new Data Centres has now been signed with BT and the Final Business Case has been approved. The FBC approves funding for a full refresh and replacement of the BSO datacentre IT infrastructure in order to minimise the risk posed to the services during migration.

Progress

The Shared Public Sector Data Centre (SPSDC) contract was signed in September between DFP and BT as a result of the procurement exercise completed by the public sector. The contract “Go-Live” date is August 29th 2016 for two facilities based in Belfast. The first of these is in Telephone House in May Streetandthe secondis atKnock exchange. Both sites are currently in use as data centre facilities for the NICS. The new facilities will provide colocation for the IT equipment of all tenants of the SPSDC. Each tenant will continue to managetheir own equipment and services. This contract is not for a managed IT service rather the provision of managed datacentre facilities to be shared by all tenants. The new datacentres will have improved availability embedded into them; this isoutlined in the table below.

Description / Existing Data Centres / SPSDC
Availability / Tier 2 A guarantee of 99.749% availability (22 hours of downtime per year) only after all the mitigations have been put in place to manage known risks. / Tier 3 A guarantee of 99.982% availability (94 minutes downtime per year) Contractually the availability for power supply is set to 100% in the new contract
Location / Hospital sites / Dedicated Data Centre provider
Scale / 40 racks across two sites / 500 racks across 2 sites
Power / 120 Kva per site fed from a single substation with a single point of failure
Shared generator between IT and critical health infrastructure / 1 Mva per site from diverse dual substations for each site and no SPOF
Dedicated N+1 generators at each site
Power Utilisation Efficiency & Cost / 2.2 PUE (multiplier for power used on IT alone to host the infrastructure.) 14p per unit / 1.4 PUE
9.9p per unit
Cooling / Inefficient air conditioners without protected power / Efficient air condition on protected power (UPS)
Security / No dedicated security function / Official level security procedures in place accredited by CPNI and to ISO 27001
Engineering / Shared facilities team for entire BHSCT putting data centre priority below care of patients / Dedicated engineering resource contractually bound to respond to any incident in relation to the data centre
Service Levels / No enforceable SLA / Strict SLA based on and ISO 20000-1 accredited service and the availability of power, cooling and humidity control.
Business Continuity & Disaster Recovery / Best endeavours by BSO ITS staff & facilities staff / ISO 22301 accredited supplier with in house BCDRP function and testing regime. Pre-approved Planned Preventative Measures schedule agreed with Service Management Board annually.
Value for money / Current spend on electricity per year is £350,000 for the BSO ITS facilities / Expected cost for the complete managed service to be in the region of £257,000 for both the managed facility and the electricity.

Once the facilities are made available inAugust 2016, BSO can begin to populate the two Data centres with the new infrastructure required to host HSC services. A Memorandum of Understanding (MOU) is currently being agreed between Enterprise Shared Services and the various client organisations which will define how the new facilities are cooperatively managed. BSO will use the provisions of the contract as a member of the Authority and will be represented on the Service Management Board which will manage the Data Centres for its customers over the course of the contract.

BSO ITS are actively engaged with BT as part of the Data Centre design and build phase as key stakeholders within that development project which has just completed its mobilisation phase. The next phase of the Datacentre Project involves BT procuring the services and equipment required to upgrade the facilities to tier 3 and provide the required capacity. There are a few key risks facing that project which will be discussed later.

BSO ITS have also been actively engaged with HP in designing a new infrastructure to replace our existing IT equipment and aim to have that design finalised by February 2016. This timeframe is in order to allow an order to be placed in April do that equipment can be built and delivered in line with planned availability of the new datacentre facilities. Capital funding for the new equipment has been made available by HSCB in 16/17 FY.

The key elements within the infrastructure design are:

  • Compute
  • Storage
  • Network
  • Security
  • Backup
  • Cloud Platform

Current and Future Work

The transition to the new Shared Public Data Centres is set to cover the following key areas

  • Market Engagement & research (Completed)
  • Data Centre Procurement (Completed)
  • Data Centre Design, Build and Testing (In Progress)
  • Infrastructure Design (in Progress)
  • Infrastructure Purchase and Install
  • Services Rationalisation in order to ensure systems and services are based on current technologies (in Progress)
  • Infrastructure Testing
  • Services Migration
  • Services Testing
  • Service Delivery Enhancements

A detailed project plan is included in the appendix with timescales included.

Draft Programme Structure

The Capital funding withinthe approved FBC allows BSO ITS to recruit a Band 8A and two Band 5’s. This will not allow the staff to be recruited until the 16/17 FY and the transition the new datacentres could be accelerated by providing these posts now using a bid for non- recurrent revenue. This will allow these staff to focus on the work of rationalising the current environment and development of the private cloud platform which is being designed into the new infrastructure and which BSO customers will use after migration.

The capital funded staff resource is approved for three years with the FBC, which covers the migration of the services through “Go-Live” and early life support of the new systems and handover to Business as Usual.

Risks

The current risk register is included in the appendix but at a high level the following are the various risks associated with the project. The risks outlined below are to be raised by BSO ITS for discussion at the next Programme Board meeting for the SPSDC. It should be noted that there are significant financial penalties for BT in relation to late delivery of the new facilities so the onus is on BT to manage and mitigate the risks described.

Data Centre Programme Risks

  • BT has undertaken to enhance NIE Power provision to the Knock Facility to provide a second power supply to the facility from a separate power substation from the existing supply. This involves digging work on the Newtownards Road.
  • Planning permission around the deployment of new power infrastructure within Knock Exchange.

Specific risks around the Migration of the NICS IT Assist equipment exist. These risks create dependencies between the migration plans of both anchor tenants (BSO and IT assist)

  • Risksto on-going IT assist service delivery due to development required on the existing sites may cause delays to the implementation phase of the data centre builds which affect our ability to deploy servers within Knock or Tele house.
  • Issues with IT assist deployment within the data centres such as a decant to another site or a delay in deployment may have a financial impact on BSO by reducing the volume of racks being used to be lower than predicted in the financial model.This would increase the perrack cost to BSO.
  • On-going IT assist consolidation of existing servers occurring after BSO deployment and causing risk to service delivery or change in financial cost predictions.
  • On-going IT assist consolidation of existing servers occurring after BSO deployment and causing reconfiguration work within the data centre that affects the ability of the service management board to control the layout and structure of the facilities to maximise efficiency of the contract.
  • Lack of senior management representative from IT assist onthe Service Management board affecting its ability to make strategic decisions quickly and with authority

Due to the nature of the contract with BT, IT Assist and BSO ITS have a shared aimin ensuring the solution is as financially advantageous as possible. The volume of racks utilised by the Authority has a direct impact on price. For this reason, BSO ITS will continue to work closely with representatives from IT Assist to ensure that there are no delays to the deployment of IT Assist infrastructure within the new facilities.

Next Steps

BSO ITS will continue to follow Programme and Project best practice and aim to have the formal programme structure for transition in place by February 2016 as well as draft copies of the relevant documentation produced. If possible, the resources for the project will be recruited at an accelerated timescales, and the utilisation of an apprentice for project support should allow some of the timescales to be shortened. The band 5 staff would be used to backfill the more experienced staff in the existing infrastructure team to allow them to concentrate on the rationalization, design build and migration to the new infrastructure.

The project team aims to have the design of the replacement infrastructure completed in February 2016to place an order for all the equipment to be shipped and installed as soon after Go Live as possible. The order should not be placed any earlier than this in order to make sure the equipment supplied is as up to date as possible. Work can then begin on thoroughly testing the environment and eventually migrating services. Work to rationalise the existing infrastructure will continue when resource is available.

A communication strategy is being developed but continued stakeholder engagement will be key to ensuring the programme is a success. The programme structure is designed to and in this transition by ensuring buy in from the relevant organisations.

Appendix 1 – Project Plan

Appendix 2 – Risk Registers

BSO

SDC

BT

1