SHARE:New Mexicostatewide Human Resource, Accounting, and Management Reporting System

Attachment 1A

Infrastructure Optimization .

Recommendations for:

SHARE:New MexicoStatewide Human Resource, Accounting, and Management REporting System

Executive Overview...... 3

Introduction...... 10

Technology Recommendations...... 11

Discussion of Current Environment ...... 11

Performance...... 13

Business Continuity...... 21

Manageability...... 26

Information Lifecycle Management...... 29

Data Security...... 31

Conclusion...... 34

Appendix A - Oracle Solutions and Services...... …………..35

Appendix B - Completed Discovery Questionnaire.....…………..43

Appendix C - Draft Database Architecture...... 55

Executive Overview

The State of NM (SoNM) SHARE project team and Oracle have jointly engaged in a technology assessment. This is a complimentary program designed to help our strategic customers fully leverage their investment in technology and utilize it according to industry best practices. The purpose of this document is to present findings and high-level recommendations to help the SoNM meet its business objectives.

In June and July of 2008, the Oracle and SoNM teams jointly analyzed the SHARE infrastructure in an effort to optimize the environment and create a short, medium, and long-term management strategy.

The Assessment focused on the following areas:

Performance
Business Continuity (High Availability Disaster Recovery)
Manageability
Information Lifecycle Management
Data Security

Oracle performed a comprehensive assessment of the current environment, then identified and prioritized a list of actionable and executable recommendations for SoNM to improve their infrastructure strategy. This effort provides detailed technology and architecture recommendations, and an implementation roadmap.

The Assessment team spent one day conducting in-person interviews with SoNM’s subject matter experts to fully understand architecture, technology, policies and practices of the current PeopleSoft environment. Following the discovery workshop, the Oracle team, with the assistance of the SHARE team, spent the next 4 weeks analyzing the discovery findings, researching solutions, and formulating recommendations. Our methodology towards solutioning included a 3-sided approach:

Including a perspective of the trends, best peer practices, regulation and compliance issues.
Leveraging SoNM’s currently owned products, existing systems and processes, and in-house technology skills.

Providing a phased implementation roadmap that allows for modular successes and sustainable long-term growth.

The following tables summarize the recommendations for each technology area, which pains they address and highlights of some of the key business drivers and values gained.

SHARE Solution Recommendations
Current Pains / Solution / Technology Recommendations / Business Value
Performance

Performance is unacceptable

Most calls come from users not being able to login – application is unavailable

NMDOT expansion concerns

Application Deadlocks are common

Storage configuration is a concern (currently RAID 5)

Configuration disparity between Development / Test / QA / Production

Processes that run efficiently in test take many hours in production

Dev / Test / QA / Production environments exist but they’re not used in the traditional way

A Training environment does not exist and is needed

No formalized Change Management Process

Database testing is problematic due to resource constraints

Batch processing takes long

NM owns Partitioning but it is currently not being used

Reports often snowball into a queue of long running reports that have to be manually canceled

Citrix environment is unstable, often not accessible or slow (being addressed by ABBA technologies)

Juniper load balancers can be, and have been, a bottleneck (being addressed by ABBA technologies)

Investigate andre-design storage architecture

Utilize holistic tools to proactively determine bottlenecks from the end-user desktop to the backend database

Formalize and document Service Level Agreements

Establish a Configuration Management Database and establish system baselines

Upgrade to PeopleTools 8.48 or above

Implement a complete Production Support infrastructure – dedicated Development / Test / QA / Training environments

Establish an Enterprise-wide Change Management Process including thorough testing

Implement Partitioning

SHARE is running at optimal rates increasing user productivity and end-user satisfaction

Ability to quickly respond and proactively fix problems, increasing user productivity

Enable SHARE to accommodate growth and new user requirements (NM DOT)

Business Continuity

Most help desk calls come from users not being able to login, application is unavailable

No coordinated effort for backing up the complete system

A clear uptime SLA does not exist: The expectation is 24 x 7, 365, but realistically, the model is “6 – 16’s;” 6a.m. to 10p.m, Monday – Saturday

roswell-a was intended to be a Q/A server but is now serving a mission critical service as a file-based report repository

It is unknown how long it would take to recover from a complete service loss

Single Points of Failure (SPFs):
DS4800
Roswell-a
Process Schedulers
Tucumcari

Failing process can cause orphan records which have to be manually deleted

Any major datacenter outages are long to recover

At one time, the system was so overloaded that the database crashed

Citrix environment is unstable, often not accessible or slow (The system is currently being redesigned to address these issues)

Formalize and document Service Level Agreements

Implement a modern grid architecture strategy

Utilize automatic data replication and data recovery tools

Eliminate single points of failure

Modernized DR plan with regular testing

Automate error prone and time consuming maintenance operations

Implement a Master Process Scheduler

Ability to meet future high availability business requirements (24/7) and service level agreements

Ensuring that SHARE will quickly be available after a major outage

Manageability

No automated configuration management

No complete system central control

Developers doing their own code backup

A standard development methodology does not exist

No coordinated effort for backing up the system

Dev / Test / QA environments exist, but they are not used in the traditional way

Limited formalized Change Management Process

Oracle Databases are inconsistent across environments

People can change the system without a tracking mechanism in place

On the technical side, there are no formal documents that travel with the process

NMDOTs system will impact size on Projects (Project tables are larger in general and can grow exponentially)

No monitoring of the Web / Application Tier

Database has not been patched for over a year due to Maximus requirements

. /

Utilize DB automation tools that provide enterprise view of the system

Create a configuration management database & leverage current tools to automate configuration tracking

Design and implement a structured approach to software development and release management

Implement Partitioning

Have separate & dedicated environments closely reflect production

Utilize diagnostics and tuning tools that determine where bottlenecks are and proactively monitor performance

Freeing up highly skilled resources to work on higher value tasks

Prevention of user mistakes

Highly trained staff completing manual tasks faster

Prevention of user mistakes

Proactively avoiding potential production problems

Ability to respond and fix problems more quickly increasing user productivity

Maintaining control over the software development lifecycle (SDLC) and ensuring consistency across applications

Enforcing standards results in reduced complexity and troubleshooting costs

Information Lifecycle Management

No archival system place

The complete database is replicated throughout SHARE’s different environments

1 – 5% a month data growth

NM owns Partitioning but it is currently not being used

NMDOT Project tables will add significantly more data

Future plans for Enterprise Performance Management (EPM) demand an archival strategy

Formally document data retention policies

Implement Partitioning to facilitate and automate archiving

Redesign storage tier to make use of lower cost storage devices for historical data

Define retention policies

Define the Data Classes

Create Storage Tiers for the Data Classes

Create Data Access and Migration Policies

Define and Enforce Compliance Policies

Ensure SHARE’s ability to scale and meet changing business requirements

Enable EPM reporting needs by keeping historical data close to production data but eliminating any performance impact

Lower TCO of data retention at the same time raising the ROI at historical data

Data Security

SQL Developer and customer queries are often run directly against the production database without testing

Production data is replicated to all environments (Dev / Test / QA)

Developers develop in the production database

Implement sensitive information handling policies that include development, storage and transmission

Implement database encryption and backup tape encryption

Implementing a masking policy to hide any sensitive data

Compliance with regulatory and privacy laws

Ensure protection of sensitive data

Performance Recommendations:

In-depth study of the i/o subsystem and storage design
In-depth study of the WebLogic thread issues
Implement a tool to measure real-user performance

Upgrade to PeopleTools 8.48 or greater
Formalize and Document Service Level Agreements
Establish holistic monitoring procedures
Establish a Configuration Management Database
Implement a complete Production Support infrastructure – dedicated Development / Test / QA / Training environments
Establish an Enterprise-wide Change Management Process including thorough testing
Implement Partitioning (you own it, use it)

Business Continuity Recommendations:

Formalize and Document Service Level Agreements
Design, Test, Implement a modern Business Continuity plan
Automate error prone and time consuming maintenance operations
Eliminate Single-Points-of-Failure, particularly at the storage tier
Implement a Master Process Scheduler
Maintain distinct support environments

Manageability Recommendations:

Utilize DB automation tools that provide enterprise view of the system
Create a configuration management database & leverage current tools to automate configuration tracking
Design and implement a structured approach to software development and release management
Implement Partitioning
Have separate & dedicated environments closely reflect production
Utilize diagnostics and tuning tools that determine where bottlenecks are and proactively monitor performance

Information Lifecycle Management Recommendations

Formally document data retention policies

Implement Partitioning to facilitate and automate archiving

Redesign storage tier to make use of lower cost storage devices for historical data

Define the Data Classes
Create Storage Tiers for the Data Classes
Create Data Access and Migration Policies
Define and Enforce Compliance Policies

Data Security:

Implement sensitive information handling policies that include development, storage and transmission
Implement database encryption and backup tape encryption
Implement a masking policy to hide sensitive data as it is moved from environment to environment

Based on the aforementioned business priorities of SoNM, pre-requisites, typical implementation timelines and SoNM’s bandwidth capabilities the following roadmap is proposed.

We recognize implementation of any recommendations discussed in this paper might only be possible after further analysis and support. The Oracle team stands ready to continue working with in the SoNM developing specific project plans, ROI analysis, building proof of concepts and continued partnership and mutual education around key IT areas.

We look forward to receiving your impressions and response to our findings and trust you will find the Assessment process valuable to you.

Introduction

The State of New Mexico (SoNM) and Oracle have jointly engaged in a technology assessment. This is a complementary program designed to help our strategic customers fully leverage their investment in technology and utilize it according to industry best practices. The purpose of this document is to present findings and recommendations, high-level solutions and roadmaps to help the SoNM meet its business objectives.

The Oracle Team would like to mention that we recognize that vendors are not often invited to participate directly with the SoNM on a strategic level, and we very much appreciate this opportunity to assist with this critically important initiative.

The Oracle Team would like to thank all the employees of the SoNM who spent time with us and provided information necessary to create this document. We fully enjoyed sharing ideas and discussing IT needs in an open, collaborative environment.

State of New Mexico Team members

Name / Title / Role / Email / Phone
John Prihoda / SHARE Development Director / / 505.476.7629
Kelly Mentzer / Deputy Director Enterprise Operations & Support Services Division / / 505.827.2105
George Robinson / PeopleSoft Development Manager / / 505.827.3917
David Emery / Database Administrator / / 505.827.2137
Alex Canett / PeopleSoft Administrator / / 505.827.3969
Karla Fernandez / PeopleSoft Technical Analyst / / 505.827.3686

Oracle Team members

Name / Title / Role / Email / Phone
Chris LeBaron / Account Manager / / 801.803.1602
Mike Baraiolo / Regional Vice President, Midwest / / 702.592.1525
Jeremy Forman / Principal Solution Architect / / 602.333.9037
Chris Wilson / Consulting Sales Representative / / 916.716.6900
Stephen Stewart / Consulting Technical Director / / 865.805.7067
Byron Pearce / Consulting Solution Manager / / 817.271.4664

Technology Recommendations

This section gives detailed analysis and recommendations on how to best address the technological focus areas and topics that lead to a comprehensive approach to SHARE infrastructure optimization. Through the assessment discovery and interview process, the Oracle team captured and analyzed SoNM’s current capabilities, challenges, goals and visions. The recommendations discussed in this section include both technology and policy initiatives. It is important to highlight that investments in technology are not enough, and a robust strategy for infrastructure optimization includes People, Process, and Technology.

Discussion of the current environment:

From the SHARE web site:

The SHARE project is the State of New Mexico's PeopleSoft enterprise resource planning (ERP) system implementation. SHARE will combine the State's accounting, human resource, payroll, and purchasing functions into an integrated statewide system. The Project's goal is to develop processes for the State's basic financial and human resource functions that are economical, efficient, and effective.

SHARE will enable the State of New Mexico to achieve more timely accountability to the public, more strict compliance with statutes and laws related to fiscal matters, reduction of duplicate effort, and improve relations with those supplying the State with goods and services.

The SHARE system is an implementation of the PeopleSoft Finance (FIN) and Human Capital Management (HCM) modules. Within the HCM module, Payroll and Time and Labor are the mission critical modules, with Recruitment being extremely important but not considered mission critical. The mission critical function within the Finance module is the complete process of paying vendors. Although this is a mission critical function, it does have a little more leeway in terms of downtime than does Time and Labor or Payroll.

The SHARE system is founded on the PeopleSoft Pure Internet Architecture (PIA):

The SHARE system is owned by the Department of Finance and Administration (DFA), but infrastructure and database supportare provided by the Department of Information Technology (DOIT). Clear Service Level Agreements do not exist in terms of system performance, business continuity, or from the SHARE system to the end user, however the original project plan does call out for an SLA requiring 99.7% System Availability for data center network, power, and environmental controls. This SLA is open to interpretation because it does not specify the PeopleSoft system components themselves, but it does imply that the SHARE system will be available for just less than 364days per year (or 363 days, 21.6 hours). This SLA is between DOIT and DFA, but, again, nothing is specific to the SHARE end user and their experience.

At present, the Department of Transportation has requested specific changes and augmentations to the SHARE system to support their unique business requirements. At the time of this writing, it is believed that the SHARE system will acquire and incorporate a PeopleSoft solution developed for and used by the Wyoming Department of Transportation to support specific transportation business processes. The criticality of this project is extremely high due to the Federal dollars associated with the success of the provided functionality.

The currentSHARE environment, including all nodes, physically consists of approximately 17 IBM servers divided into 22 logical servers, one IBM DS4800 Storage System (other storage infrastructure was not provided), and four Juniper DX 3250-S-2C load balancers.

The production PeopleSoft server environment is as follows:

2 IBM p5 550Q – Application & Web Servers (taos & gallup)

1 IBM x336 – File Server (nambe)

2 IBM x336 – Citrix Servers (cerrillos & chama)

2 IBM p5 570 with 4 Logical Partitions (LPAR)

2 Database Servers (santafe-a, santafe-b)

1 Report Repository (roswell-a)

1 Enterprise Manager Grid Control (hatch)

The Development / Test / QA PeopleSoft environments are as follows:

2 IBM p5 550Q – Application & Web Servers (clovis & deming)

2 IBM x336 – Non-Production Files Servers (pecos & pojoaque)

2 IBM x336 – Non-Production Citrix Servers (chimayocordova)

1 IBM x336 – UPK Server (alamos)

1 IBM x336 – Upgrade Server (vegas)

2 IBM p5 570 with 3 Logical Partitions (LPAR)

3 Database Servers (roswell-a, roswell-b, hobbs)

Recommendations

1) Performance (& Scalability)

The SHARE system supports approximately:

24000 Time Entry Users

4000 Full-time Financial (FIN) Users

1000 Full-time Human Resource (HR) Users

It is projected that over time, the number of HR users will remain flat but the number of FIN users will increase; considerable growth is not expected. In general, aside from a couple days a week, FIN experiences a higher level of concurrency than does HR.