Department of Health and Human Services
Centers for Medicare & Medicaid Services
Office of Information Services
CMS Testing Framework Overview
Version 1.1
May 18, 2011
Centers for Medicare & Medicaid Services May 18, 2011
Table of Contents
1. Introduction 1
1.1 Purpose 1
1.2 Scope 1
1.3 Audience 2
1.4 Document Organization 2
2. CMS Testing Framework Overview 3
2.1 Business Application Testing Overview 4
2.2 Infrastructure Testing Overview 7
2.3 Categories of Testing 8
2.4 Deliverables 10
2.5 Reviews 11
2.6 Roles and Responsibilities 13
2.7 Testing Tools 15
2.8 Test Data 15
3. Development Testing 16
3.1 Unit Testing 16
3.2 Application Integration Testing 16
3.3 Section 508 Testing 16
4. Validation Testing 17
4.1 Business Application Validation Testing Functions 17
4.1.1 System Testing 17
4.1.2 Functional Testing 17
4.1.3 End-to-End Integration Testing 17
4.1.4 User Acceptance Testing 18
4.1.5 Regression Testing 18
4.1.6 Section 508 Testing 18
4.2 Infrastructure Validation Testing Functions 18
4.2.1 Infrastructure Testing 18
4.2.2 Infrastructure Regression Testing 19
4.2.3 Application Regression Testing 19
4.2.4 Section 508 Testing 19
5. Implementation Testing 20
5.1 System Acceptance Testing 20
5.2 Performance & Stress Testing 20
5.3 Initial ST&E 21
5.4 Final Integration Testing 21
5.5 Initial Contingency Planning Testing 22
6. Operational Testing 23
6.1 Production Ready Testing 23
6.2 Monitoring & Reliability Testing 23
6.3 Operational ST&E 23
6.4 Audits 24
6.5 Operational Contingency Planning Testing 24
Acronyms 25
List of Figures
Figure 1: Type of Figure: Testing Function per Environment 6
List of Tables
Table 2: Role of ILC Framework Deliverables in the CMS Testing Framework 10
Table 3: Role of ILC Framework Reviews in the CMS Testing Framework 12
Table 4: CMS Testing Framework Roles & Responsibilities 13
CMS Testing Framework Overview - Version 1.1 ii
Centers for Medicare & Medicaid Services May 18, 2011
1. Introduction
The Centers for Medicare Medicaid Services (CMS) has identified the need for integrated, standardized testing life cycle processes and guidelines to be defined within the CMS Integrated IT Investment and System Life Cycle Framework (hereafter simply the “ILC Framework”). These testing guidelines will describe a framework of testing functions to be performed during the Development, Test, Implementation, and Operations and Maintenance (O&M) Phases of the ILC Framework in the CMS development, test, implementation, and production environments to enable the consistent delivery of high quality, production-ready CMS business applications and infrastructure.
1.1 Purpose
The purpose of the CMS Testing Framework Overview (hereafter simply the “CMS Testing Framework”) is to establish a consistent, repeatable CMS testing life cycle process and framework for business application and infrastructure testing functions, which will reduce business risk by promoting more predictable testing actions and results.
The CMS Testing Framework will help establish, define, and organize guidelines for testing new and existing CMS business applications and infrastructure prior to their deployment to a data center’s production environment. The CMS Testing Framework establishes standard terminology, definitions, structure, deliverables, reviews, roles and responsibilities, and support tools for CMS business application and infrastructure testing functions to facilitate efficient, responsive, and secure use and operation of CMS business applications and infrastructure.
Utilizing the CMS Testing Framework will reduce CMS and Office of Information Services (OIS) organizational risk, facilitate better resource need forecasts, improve testing schedules, and lower the incidence of reactive break/fix episodes.
The CMS Testing Framework will guide the testing standards for all contractor task orders supporting CMS business applications and infrastructure. Through contractor compliance with these standards, CMS will ensure alignment by contractors with the ILC Framework.
1.2 Scope
This CMS Testing Framework identifies and describes the various testing functions that may be performed for a CMS business application or infrastructure project during the project’s ILC Framework phases of Development, Test, Implementation, and O&M. This document also identifies the various deliverables and reviews prescribed by the ILC Framework and their specific role in the CMS Testing Framework.
This CMS Testing Framework clarifies organizational roles and responsibilities in support of business application and infrastructure testing, and briefly addresses testing tools and test data.
1.3 Audience
This document is intended for use by the executive leadership of CMS, CMS personnel, and CMS contractors/subcontractors responsible for the ownership, management, definition, development, implementation, maintenance and support of CMS business applications and infrastructure.
1.4 Document Organization
This document is organized as follows:
Table 1: Document Organization
Section / Purpose /Section 1: Introduction / Defines the purpose, scope, audience, and organization of this document.
Section 2: CMS Testing Framework Overview / Provides an overview of the business application and infrastructure testing functions, deliverables, reviews associated with the testing framework, as well as roles and responsibilities, testing tools, and test data.
Section 3: Development Testing / Defines the Development Testing functions for CMS business applications.
Section 4: Validation Testing / Defines the Validation Testing functions prescribed for CMS business applications and infrastructure.
Section 5: Implementation Testing / Defines the Implementation Testing functions prescribed for CMS business applications and infrastructure.
Section 6: Operational Testing / Defines the Operational Testing functions prescribed for CMS business applications and infrastructure.
Acronyms / Provides a list of acronyms used in this document.
CMS Testing Framework Overview - Version 1.1 5
Centers for Medicare & Medicaid Services May 18, 2011
2. CMS Testing Framework Overview
The CMS Testing Framework is comprised of numerous testing functions that may be conducted during the life cycle of a given business application or infrastructure project, based on the specific circumstances of the project.
During the Planning Phase of a business application or infrastructure project’s life cycle, the project team shall determine for each of the testing functions prescribed in the CMS Testing Framework if the testing function is required or not required for the given project. Each of the prescribed testing functions should be documented in the project’s Project Process Agreement (PPA), along with the following information:
· lifecycle phase during which the testing function will be performed;
· name of the organization that will lead the testing;
· name and role of the key organization(s) to participate in and/or support the testing;
· any caveats or expectations if the testing function is required for the project, or a justification for why the testing function is not required for the project.
Determination of the specific testing functions to be performed for a project will be dependent on, but not limited to, the following:
· type of project (i.e., business application or infrastructure);
· acquisition, development, and/or maintenance approach;
· whether or not the CMS business application or infrastructure is new, is experiencing a major change, is experiencing a maintenance change, or if the change is due to an emergency problem correction;
· intended use and audience for the business application or infrastructure; and/or
· project risk(s) and/or information security risk level.
Testing functions may be performed iteratively and repeatedly for a particular project based upon the project implementation process.
The project’s overall testing approach/strategy shall be appropriately documented in a Test Plan(s), along with a detailed description of each of the planned tests. The Test Plan(s) should describe how the CMS Testing Framework will be applied to the project, and identify any deviations from the prescribed CMS Testing Framework. Key aspects of the testing approach should be documented in the test plan, such as content, methodology, prioritization, and progression of testing activities.
For example, the project’s testing methodology should identify the order by which the selected testing functions are to be performed during the life cycle, identify if some testing functions are to be combined for testing efficiencies, and/or identify how the selected testing functions will be performed (i.e., the testing methods). The testing methods may include (but are not exclusive to) white box testing[1], black box testing[2], positive testing[3], and negative testing[4], influenced by factors such as project cost, schedule, risk, and architecture. From an architecture perspective, for example, an application implemented using a Service Oriented Architecture (SOA) would include a focus on testing the scenarios of how a service is used by the CMS business application. As much as possible, reuse of test plans, test cases, test scripts, and test data should be considered.
2.1 Business Application Testing Overview
The CMS Testing Framework provides guidance about “what” testing is necessary for CMS business applications built on mainframe and mid-tier platforms, but not “how” that testing should be performed (i.e., a testing methodology). This does not preclude CMS from defining the deliverable and its format. The CMS Testing Framework includes readiness reviews, which are control gates to exit one environment and to enter another environment.
Figure 1 depicts a conceptual view of the CMS Testing Framework for business application testing that identifies the following:
· the four ILC Framework phases during which business application testing is a primary activity (i.e. Development Phase, Test Phase, Implementation Phase, and O&M Phase);
· the four categories of testing described previously in section 2.1 aligned with the corresponding lifecycle phases (i.e., Development Testing, Validation Testing, Implementation Testing, and Operational Testing);
· the respective business application testing functions aligned with the four main categories of testing:
o Unit Testing, Application Integration Testing, and Section 508 Testing associated with Development Testing;
o System Testing, End-to-End Integration Testing, Regression Testing, Functional Testing, UAT, and Section 508 Testing associated with Validation Testing;
o System Acceptance Testing, Initial ST&E, Initial Contingency Planning Testing; Performance & Stress Testing, and Final Integration Testing associated with Implementation Testing; and
o Production Ready Testing, Operational ST&E (Annual Security Control Testing), Operational Contingency Planning Testing, Monitoring & Reliability Testing, and Audits associated with Operational Testing;
· the supporting testing environments (i.e., Development Environment, Test Environment, Implementation Environment, and Production Environment); and
· the three primary ILC Framework readiness reviews during which business application testing is a key contributing factor (i.e., Validation Readiness Review (VRR), Implementation Readiness Review (IRR), and Operational Readiness Review(ORR)).
As identified in Figure 1, the testing functions possibly performed for a new or modified CMS business application align with corresponding phases of the ILC Framework. This diagram, however, does not represent the specific sequence of testing activities, although one may infer a general flow by reading the diagram from left to right.
CMS Testing Framework Overview - Version 1.1 5
Centers for Medicare & Medicaid Services May 18, 2011
Figure 1: Type of Figure: Testing Function per Environment
CMS Testing Framework Overview - Version 1.1 5
Centers for Medicare & Medicaid Services May 18, 2011
2.2 Infrastructure Testing Overview
The CMS Testing Framework also provides guidance about “what” testing is necessary for new or modified infrastructure installed and configured on mainframe and mid-tier platforms, without prescribing the specific testing methodology to use. Within the CMS Testing Framework, infrastructure is deemed anything that is not a CMS business application. Therefore, infrastructure includes hardware, system software, data communications, and many other items that support CMS business applications running on the mainframe and mid-tier platforms.
Examples of infrastructure system software include an operating system such as Solaris, a database management system such as DB2, or Web server software used to host CMS Intranet Web applications. A change to infrastructure may include, for example, an upgrade of an operating system to a more current version; installation of a patch to Commercial Off-the-Shelf (COTS) software or utility; a change to a driver, utility, or firmware; or an upgrade to a board or Central Processing Unit (CPU).
The CMS Testing Framework includes readiness reviews, which are control gates to exit one environment and to enter another environment. Figure 2 depicts a conceptual view of the CMS Testing Framework for infrastructure testing that identifies the following:
· the three ILC Framework phases during which infrastructure testing is a primary activity (i.e. Test Phase, Implementation Phase, and O&M Phase);
· the three categories of testing described previously in section 2.1 aligned with the corresponding lifecycle phases (i.e., Validation Testing, Implementation Testing, and Operational Testing);
· the respective infrastructure testing functions aligned with the three categories of testing:
o Infrastructure Testing, Infrastructure Regression Testing, Application Regression Testing, and Section 508 Testing associated with Validation Testing;
o System Acceptance Testing, Initial ST&E, Initial Contingency Planning Testing; Performance & Stress Testing, and Final Integration Testing associated with Implementation Testing; and
o Production Ready Testing, Operational ST&E (Annual Security Control Testing), Operational Contingency Planning Testing, Monitoring & Reliability Testing, and Audits associated with Operational Testing;
· the supporting testing environments (i.e., Test Environment, Implementation Environment, and Production Environment); and
· the two primary ILC Framework readiness reviews during which infrastructure testing is a key contributing factor (i.e., IRR and ORR).
As identified in Figure 2, the testing functions possibly performed for new or modified infrastructure align with corresponding phases of the ILC Framework. This diagram does not represent the specific sequence of testing activities, although one may infer a general flow by reading the diagram’s testing function boxes from left to right.
2.3 Categories of Testing
The CMS Testing Framework encompasses four main categories of testing during the integrated IT investment and system life cycle: Development Testing, Validation Testing, Implementation Testing, and Operational Testing.
· Development Testing – A set of testing functions performed within a development environment for a CMS business application. These testing functions will confirm: