Reducing Patch Downtime in Oracle EBS

Note: This content majorly based on the article Top 7 Ways of Reducing Patching Downtimes for Apps by Steven Chan and few other supporting articles.

1. Overview

Patching an Oracle Applications environment is a major maintenance activity which will generally account to significant percentage of planned downtime of production environments. This patching could be as simple as applying a one-off patch to fix a specific bug to as huge as applying a maintenance pack. In this document we will discuss various options that can be utilized to reduce the patching downtime of Oracle Applications environments.

2. Techniques to Reduce Patching Downtime

Oracle Applications provides various features that can utilized to decrease the amount of downtime required to perform patching. Some of these options require investment of extra resources in terms of extra servers and storage, while others require efficient usage of existing adpatch options and other ad utilities with proper analysis & planning.

While adopting these methods one should note that not all methods will decrease the patching time. Patch downtime majorly depends on the type of patch that we are applying, the kind of activity that the patch is performing and the patch levels of existing environment. One should also note that patching activity depends equally on both database tier resources and on application tier. Careful analysis of patching bottlenecks is vital in deciding the right mix of techniques to ensure maximum reduction of patching downtime.

It is also mandatory to note that the one can only decrease the patch downtime but not eliminate it. Very few patches can be applied as hot patches with any downtime. Often, patch README files do not mention whether a patch can be applied as a hot patch or not. One should do a complete analysis of patch activity to identify whether a patch can be as a hot patch or not.

Below are the techniques recommended by Oracle to decrease the patching downtime…

2.1 Using Staged Application File system

The Staged Application File System approach allows us to perform as many changes as possible in an offline Apps environment, and defers taking down production environment only for the final database patches tasks. Using this approach, one can apply new patches to an exact clone of production E-Business Suite environment. This can be done while production system is still running. The staged Applications environment is then used to run database updates and APPL_TOP changes into production environment.

Perquisites

  1. Topology – A staged Applications system should duplicate the topology of your Production system. Each physical APPL_TOP of your Production system must exist in your staged system.
  2. Patch Level –11.5.10 or AD.I.1 is required. For environments prior to 11.5.10 – patch 3285255

Steps Involved

Below are the high level steps involved with this process.

  1. Prepare Source Environment
    1. Maintain Snapshot information
    2. Ensure minimum patch levels as per pre-requisites
  2. Prepare Staged environment (Should be exact replica of PROD topology)
  3. Apply necessary patches to Staged environment
  4. Update Production system
    1. Update production database
    2. Update production APPL_TOP
  5. Post patching steps
    1. Export patch history on Staged environment
    2. Import patch history on Production environment.

Only step 4 requires production environment to be down and steps 4-a and 4-b can be executed in parallel. This will ensure that overall time spent required to perform patching is minimal.

Pros

  1. Can be used for patches of any size and complexity.
  2. Significant reduction for patch down time for huge patches.

Cons

  1. Requires double hardware resources – number of servers and storage.
  2. Complexity involved in the initial setup phase.

Evaluating the benefit

Below example will help us understand how Staged Applications environment will help us in reduced down time.

Example 1:

Below is an example of a one-off patch application. This patch is majorly DB bound and does little activity on the Application file system.

From this report, we can observe that 20% of total patch time is spent on Administration phase which will involve tasks like forms generation and reports generation.

These reports can be generated on the Staged Application file system (R12 Only) before hand to assess the exact amount of time that can be gained by using the Staged Application File system approach.

Phase Name Start Time End Time Elapsed Jobs
Seq

5/12/2011 9:45

5/12/2011 9:45

0:00:00

2
Tab

5/12/2011 9:45

5/12/2011 9:45

0:00:02

2
tbm: Create

5/12/2011 9:45

5/12/2011 9:45

0:00:00

2
Pls

5/12/2011 9:45

5/12/2011 9:45

0:00:01

5
Vw

5/12/2011 9:45

5/12/2011 9:45

0:00:13

8
Plb

5/12/2011 9:45

5/12/2011 9:45

0:00:09

11
daa+52

5/12/2011 9:45

5/12/2011 9:45

0:00:02

1
Dat

5/12/2011 9:45

5/12/2011 9:45

0:00:06

1
dat+1

5/12/2011 9:45

5/12/2011 9:45

0:00:12

2
upg+70

5/12/2011 9:45

5/12/2011 9:55

0:09:57

1
Administration

5/12/2011 9:56

5/12/2011 9:57

0:00:20

5
Administration

5/12/2011 9:57

5/12/2011 10:00

0:03:08

23
Total

5/12/2011 9:45

5/12/2011 10:00

0:15:10

63

References:

2.2 Use a shared application-tier file system

Creating a multi-node system with a shared application tier file system saves patching time in multiple Application Node environments, as we apply patch only once – on the primary node. This ensures minimal downtime as all adpatch actions are performed only once.

This diagram is a sample representation of a Shared Application Tier file system environment in R12 environment.

References

  • Sharing the Application Tier File System in Oracle Applications Release 11i [ID 233428.1]
  • Sharing The Application Tier File System in Oracle E-Business Suite Release 12 [ID 384248.1]
  • How To Apply Patches On Shared Application Tier File System Environment [ID 745580.1]

2.3 Use Distributed AD

Distributed AD is a parallel processing feature that can reduce downtime by efficiently utilizing all the available resources on a shared application file system. AD Administration and Auto Patch run on the primary node and direct workers running on that node and other nodes in the system. The AD Controller utility controls and monitors the actions of the workers that you specify.

This parallel processing makes use of managers, which direct the actions of worker processes. The manager assigns each worker a processing job and monitors its progress. When a worker completes a job, the manager assigns it another until all jobs are complete. AD Administration and AutoPatch can be directed to distribute processing tasks across multiple remote machines in a multi-node system. This type of parallel processing operation is called Distributed AD. It further reduces the time to complete a maintenance task by utilizing the processing capabilities of all of the nodes in the system.

Because the AD workers create and update file system objects as well as database objects, Distributed AD can only be used on systems that are using a Shared Application Tier File System to ensure that files are maintained in a single, centralized location.

Pre-requisites

  • AD.H or higher
  • Shared Application File system

Pros

  • Increased utilization of OS resources on all nodes of Shared Application systems
  • Works efficiently in environments where the middle tier resources are a bottleneck for activities like forms generation.

Cons

  • Complex to implement. Requires manual intervention in adpatch operations
  • Will only work for application tier operations. If DB tier is having resource shortage, performance improvement will not be significant.
  • Assigning more number of workers is not always a solution if the number of workers is not the bottleneck. Proper care must be taken while deciding the number of workers.

References

2.4 Merge multiple patches using AD Merge Patch

Merging patches saves time because the AutoPatch overhead of starting a new session is eliminated for those patches that are consolidated.  Duplicate linking, generating or database actions are run once only.  If two patches update the same file, AD Merge Patch will save time by applying only the latest one.  Patches can — and should — be merged with their listed prerequisite patches.

Some guide lines while considering Merge Patch:

  • Oracle DBA (AD) patches may change the AutoPatch utility itself. Always merge AD patches separately from non-AD patches.
  • Merge patches with their listed prerequisite patches to make patch application easier.
  • Individual patches can be merged with aggregated patches such as minipacks, maintenance releases, and release update packs.
  • Merged patches can be merged with other patches. This may be useful when maintaining a “current” patch which can be applied to other systems in order to bring them up to the same level.
  • Merging a patch performs any necessary system-specific character set translation.

2.5 Run AD Patch in Non-Interactive Mode

Automating the patching process by applying patches in non-interactive mode will help us reduce the delays introduced by manual intervention. This process allows you to store the responses to the patching prompts in a defaults file, and then specify the name of this file when you run AutoPatch. This is particularly helpful when applying huge number of patches.

Reference:

  • AD Command Line Options for Release R12 [ID 1078973.1]

2.6 Effective use of AD Patch options

AD Patch provides various options which can be used effectively to avoid repetitive tasks performed by AD Patch. These options will have greater benefit particularly if we are applying multiple patches separately. Alternatively we can also use merge feature of ad patches to avoid repeated execution of ad patches. Below are some of the options which can be used to optimize the adpatch runtime.

nocompiledb

This option will stop compilation of database INVLID objects after every patch.

nomaintainmrc

This option will stop maintenance of MRC schema operations after every patch.

nocompilejsp

This option will suppress compilation of jsp files.

nogenerateportion

Suppresses generate driver actions like genform, genpll…

3. Special Consideration for NLS patches

If the Oracle Applications environment is enabled with multiple NLS languages, NLS translation versions of all patches needs to be applied. This will significantly increase the overall downtime of the environment and thus needs to be handled appropriately. Below are the two most prominent techniques used for with NLS patches to reduce the downtime.

3.1 Merge NLS (Translation) Patches and Apply Them During Uptime

If there are multiple patches for multiple languages, merge all US patches into a single patch. Then, merge the NLS translation patches for each active language into a single patch for each language. Apply the US patches first during downtime. Then, the merged NLS translation patches can be applied during uptime.

3.2 Translation Synchronization

The Translation Synchronization patch feature provides a quick way to synchronize existing translations with the American English file versions on given Applications instance. By applying just one patch for each language, once can bring your translations up to current Applications patch level. You can also choose to get the latest translations to bring your translations up-to-date.

The basic concept involves preparing a single manifest file for your Applications instance.  This manifest reflects the language software content for all the active languages on your current Applications system.  The manifest will be used to generate customized patches for your system.

This feature reduces the installation time and complexity for synchronizing language installations. It also provides a simple way for you to update your existing translations, after major patching activities.

Prerequisites

  • 11i environments – 11i.AD.I.2 with patch 5837664 or 11i.AD.I.6
  • R12 environments – R12.AD.A.1

References

  • Requesting Translation Synchronization Patches [ID 252422.1]

4. Considerations for Database Tier

Often database tier operations of adpatch consume majority of the downtime. To ensure that this down time is reduces one should optimize the initialization parameters of database for patching specific workload. This will ensure that the database performance is optimally tuned for different kind of workloads like patching and normal operations. For example: Increasing the SGA /PGA sizing related parameters.

5. References

Leave a comment