How to reduce downtime and risk during your Oracle 11i Upgrade

David Joffe

Introduction

Companies upgrading from a 10.7 or 11.0 release of Oracle applications to an 11i (11.5.x) version face a variety of significant challenges. IT staff and functional process owners must learn about new features and technology and decide how to utilize them. They must design new business processes, port customizations to conform to changes in the 11i data model, plan and execute internal test cycles, and conduct user training sessions. Organizations hope that the result of these efforts is that the company can run in a more efficient and cost-effective manner. Though some problems are inevitable in a project of this magnitude, implementation teams hope that any frustrations experienced by internal operations, customers, and suppliers after the upgrade goes live are minimal.

After beginning 11i upgrade projects, many companies quickly realize that in addition to making sure that the end state system works, transitioningfrom the old release to the new 11i release is a significant effort in itself. Many companies have found that the conversion process can take days or even weeks when it is time to go live. A prolonged period of downtime while the upgrade is in progress causes negative business impacts and creates a poor first impression for users who already will be challenged to adopt a new system.

This white paper lists proven, specific tips for reducing the amount of downtime that you will need to incur while upgrading your Oracle instance to 11i, and for eliminating risks that something will go wrong during the go live transition.

Assumptions

There are two major ways that companies have transitioned from pre-11i releases of Oracle to 11i. Reimplementation involves starting out with a clean 11i instance, patching it to the level that you want, creating all setup data that you need in it, migrating all customizations to that instance, and then developing and deploying custom scripts to migrate transactional data from your old applications instance to the new one when it is time to go live. The Upgrade methodrefers to following Oracle-provided processes and running Oracle-provided scripts to convert your instance from your current release to the newer release.

Although some of the techniques described in this document would be useful even for those who are using the Reimplementation approach, the primary focus of this document is on the Upgrade approach.

This white paper assumes that the reader has some familiarity with basic Oracle Upgrade concepts. If not, you might find it useful to skim Oracle’s Upgrading Oracle Applications Release 11i manual before proceeding.

Categories of Downtime Reducing Techniques

The tips for reducing downtime are divided in this document into the following categories:

  • Organization, Planning, and Project Management
  • Executing AutoUpgrade, the 11i Maintenance Pack, and other Patches
  • DBA and Infrastructure considerations
  • Module Specific Techniques
  • Deploying customizations and custom setups
  • Certifying the upgraded environment
  • Customizing Oracle’s Upgrade and Patch scripts

A.Organization, Planning, and Project Management

Tip: Define an acceptable amount of downtime

At the start of your project, you will not have enough information to accurately estimate how much downtime you will incur. You cannot rely solely on other companies’ experiences because there are many variables unique to each Oracle implementation. Significant factors affecting how much downtime you will have include: the amount of data in your current system, which modules you use, the quality of your hardware, the extent of your customizations, and the amount of personnel you have available to reduce the downtime. Nevertheless, it is extremely useful to think about how much downtime you can tolerate as early as possible in your project lifecycle.

The amount of effort you need to spend on reducing downtime will be very different if you can only tolerate being down for a weekend as opposed to an entire week. Your fundamental approach to tackling the conversion might also change.

The amount of tolerable downtime can influence your project’s go live date as well. For example, if you absolutely cannot be down for an entire workday, but you do not think that you can reduce the downtime below 72 hours with the resources you have available, then you might choose to go live on a holiday weekend where the impact of a third day of downtime might not be as severe to your operations.

Tip: Start Upgrade work as early as possible

It is advisable to start your first upgrade practice run as soon as you have committed to doing the project. Although it will not encompass every detail from your ultimate go live plan, completing this first upgrade run will be a valuable educational experience for your project team and give you a much better idea of how much downtime you might incur and what it will take to streamline the process.

Completing this first upgrade run also has the added benefit of producing an environment that will be very useful to your entire project team. Having an 11i environment that was produced via upgrade of your own production system will be much more valuable for other analysis, design, development, and testing than using an empty Oracle 11i instance or the fabricated “Vision” environment that Oracle provides.

Tip: Establish ownership of managing the transition

Many of your project team members will be involved in reducing downtime and ensuring a smooth transition in some way. The process requires system administrators, DBAs, and functional and technical applications experts. It is useful to designate one person as the downtime coordinator or transition manager for your project. As downtime can be reduced by different team members via different technical, functional, or organizational approaches, it is important to have a clearly defined single point of contact for tracking progress and facilitating key cross-team decisions.

Tip: Define high level cutover phases

Oracle’s Upgrade manuals describe several hundred tasks that need to be executed in order to upgrade your system. It divides these tasks into six stages that it labels Category 1 thru Category 6 (more information about these tasks in a later section). For most companies, these are not the only steps you will need to perform in order to complete your transition.

As such, it is useful to define meaningful global phases that describe the tasks that need to be done in easily understandable terms for communication and optimization purposes. One such categorization is listed in the chart below.

Phase
/
Phase Description
Advance / Tasks that can be done “out of the critical path” without any downtime of the old or the new system. These tasks might be done days, weeks, or months before the actual system downtime.
Rampdown / Tasks that are done during the last 24-48 hours that the old system is still up. This would include some of Oracle’s Category 2 and 3 steps such as final runs of certain programs and clearing of open interface tables. You might also choose to cease using certain modules / functionality during this phase even though downtime has not started. No system modifications that stop the environment from functioning on its current release should be done at this point.
PreUpgrade / Downtime officially starts here. End users are restricted from accessing the system and additional Oracle Category steps are performed.
AutoUpgrade / Execute Oracle’s AutoUpgrade process which takes you from the current release to the base 11i release (11.5.1)
11i Maintenance Pack / Execute Oracle’s latest maintenance pack (e.g. 11.5.10) to upgrade your system to the latest 11i point release.
Database Upgrade / Perform database upgrade. As many 10.7 / 11.0 users are still on a pre-9i release of the database, and 11.5.9+ is only certified on 9i, many applications upgrade plans also involve doing a database upgrade. Depending on your current environment, the database upgrade might precede the applications upgrade
Additional Patches / Execute consolidated upgrade patches, family packs, and one off patches that you might require on top of the base maintenance pack.
Customization deployment / Deploy any customizations to the database (if required); Perform any manual setups that are needed
Audit / Business users audit the system to make sure that the data in the upgraded environment is as expected. This might involve the execution of standard Oracle or custom reports in “before” and “after” environments to make sure that the data ties out.
Acceptance Test / It is wise to perform a few critical transactions in the upgraded system before opening up the system to the entire user community, especially if you have a significant number of customizations. If the Audit and the Acceptance test have passed, then you are ready to “go live” (open up your system to the end user community)
Ramp Up / After you are a go, you might choose to start certain programs / enable access to different sets of users in a phased approach

Tip: Create and maintain a detailed cutover task list

Even in the earliest trial runs of your upgrade, it is extremely useful to record a detailed list of each and every step that you perform to create your 11i environment. Decide how you want to maintain your detailed go live plan. DBAs and technical and functional applications owners will ultimately have to perform dozens of steps in a specific sequence in order to complete your upgrade successfully. It is helpful to have a single place where every single step that needs to be performed is recorded.

The “Upgrade Assistant Spreadsheet” provided by Oracle is a good starting point for your detailed go live plan. It will contain all of the steps from the core Oracle Upgrade manual. But it needs to be augmented with many steps based on each company’s unique requirements.

You might use a spreadsheet or Microsoft Project for tracking your go live tasks. Or you can make or use a simple database application for maintaining them. In any case, your go live plan should be easily accessible. People will realize that additional go live steps or step details are needed both during practice transition executions and during functional test cycles. It is imperative that people record such steps as soon as they are noticed so that steps are not missed in the most critical go live run.

Whatever method you use for maintaining your detailed go live plan, you should consider recording for each task, its:

  • Go Live Phase
  • Step #
  • Detailed step instructions
  • Dependent steps
  • Step executor
  • Expected step duration (from previous run)
  • Expected start and end time
  • Actual duration
  • Actual start and end time
  • “WHO” columns (created by, creation date, last updated by, last update date)

Tip: Regularly measure and report on the progress of downtime reduction

Establish regular meetings to review the project team’s progress in achieving downtime goals. Review key issues that are outstanding and review key post mortems from previous practice runs. Look into how long each high level go live phase is taking and decide which one(s) have the most opportunity for improvement.

Tip: Practice the transition as many times as necessary

You should of course expect that each practice run of transitioning your system from 10.7/11.0 to 11i is faster than the previous one. In addition to tuning long running steps, people executing the tasks should become more familiar with what they have to do, so you should have less wasted time and fewer errors. As much as possible, try to make sure that the people who execute tasks in your practice runs are the same ones who will execute those tasks in the go live runs.

Tip: Synchronize cutover testing and functional testing

You should align your functional test plans with your cutover or transition test plans. Suppose your organization has decided that it needs to conduct three formal test cycles or conference room pilots before going live. If that is the case, you should plan on executing three corresponding transition tests to produce the upgraded environments that you will use for those three test cycles. It is important and instructive to do this. When you succeed, you will prove to yourself that you can build a working environment when you go live. If you repeat your functional test cycles using the same environment as a base each time, you have not necessarily proven that your go live will be successful. Doing the final transition test and functional test consecutively in the same fashion that you will use when going live will also help you to better understand the resources needed to make your go live a success.

Tip: Utilize experienced personnel

Upgrade projects are most certainly a time to utilize your personnel who are most experienced with Oracle Applications, the internal business operations of your organization, and with executing complex systems projects. In particular, you will want to utilize the best applications DBAs, performance tuners, and technical and functional module experts that you have available.

If you need help from outside consultants, insist on getting resources who have participated in 11i upgrade projects before.

Tip: Define a rigid freeze process

Your 11i upgrade project should have a well-defined freeze process. You should have clear milestones that result in increasingly strict controls being placed on the scope of your project. In addition to system stability, the impact on downtime and the smoothness of your transition should be considered when accepting changes and bug fixes late in your project cycle. It is particularly important that your final practice run of transitioning the system should be IDENTICAL with what you plan to do in the go live run. If you accept new customizations, bug fixes, or Oracle patches after that final practice run, you will be adding more uncertainty to the success of your go live run.

Tip: Apply changes before or after downtime where possible

People’s first instinct when attempting to reduce downtime is to make steps faster. In addition to that, you should constantly be on the lookout for tasks that do not need to be done during the primary downtime window. Upon careful review, you should be able to find technical and functional tasks that can be done without risk while your system is up. Specific examples of such tasks are given in later sections of this paper.

Tip: Plan for uncertainty

No matter how many times you practice and how much you plan, you will discover that it is extremely difficult to know exactly how much downtime you will have in your go live run. The oracle patching processes themselves consist of tens of thousands of individual scripts. Data in your current system is changing all of the time until go live. Even if your team is extremely thorough, some new unforeseen error might occur when you go live. As such, you should expect that the actual execution of your go live time might be off by a few hours from your best estimate. You should make sure that your team understands what needs to be done if the go live upgrade is faster or slower than what you expected.

Tip: Have a written fallback plan

After investing months of effort into an upgrade project, your team might not want to think too hard about what to do if some catastrophic error happens during your go live run. However, it is necessary to be prudent. Insist that your team has a written plan on how to “re-activate” your old version of the system in case of an emergency. Ensure that proper backups are taken so that you have integrity within your Oracle Applications system as well as between that system and any external systems that your organization also uses.

B.Executing AutoUpgrade, the 11i Maintenance Pack, and other Patches

To upgrade from 10.7 or 11.0 to 11i, Oracle instructs you to run several large conversion programs. Oracle divides this process into several chunks. “AutoUpgrade” is the piece that converts your system from 10.7 or 11.0 into the base 11i (11.5.1) release. You then use adpatch to apply the “11i Maintenance Pack” that takes you to the specific point release (11.5.9, 11.5.10, 11.5.10.2, etc) that you want to upgrade to. Then you will want and most likely need to apply a set of additional patches on top of that point release.

Note: You might wonder why Oracle makes you run several conversion programs instead of one. Organizing it this way enabled Oracle to freeze its AutoUpgrade code so that they do not have to change that when new releases of 11i come out. Basically this structure reduces the amount of permutations of conversions from source to target releases that Oracle has to support, but adds complexity for its upgrading customers.

This section discusses tips on how to improve your execution of AutoUpgrade, the 11i Maintenance Pack, and other patches. In some cases, it refers to Oracle manuals or metalink notes that give very helpful information. A consolidated list of all such notes is given at the end of the paper.