elcarOnOsdnaH: May 2012

Wednesday, May 30, 2012

Patch Oracle EBS/Apps R12 from 12.1.1 to 12.1.2

As promised earlier, I recently applied the patch EBS R12, 12.1.1 to 12.1.2.
In this post, I am going to share my experience which may help you to plan your upgrade.
Preparation -
To patch EBS/Apps from 12.1.1 to 12.1.2, the required patch is 7303033. However before one can apply this patch there is a prerequisite patch R12.AD.B.DELTA.2 i.e. 8502056, that one needs to apply.

So ultimately it boils down to the following patches

1. Apply Pre-requisite patch R12.AD.B.DELTA.2 i.e. 8502056

2. Apply 12.1.2 RUP 2 i.e. 7303033

3.Apply the latest consolidated online help Patch 7303032(Optional)

4. Post Patch Process

However, before we start there are two pre-checks one needs to perform as follows.

Pre-check 1 -

The minimum required version of OracleAS 10g should be 10.1.3.4

(R12.1.1 comes with 10.1.3.4)

[root@appsnode appl]# sudo su - oracle

[oracle@appsnode ~]$ cd /u01/app/oracle/visr12/apps/apps_st/appl/

[oracle@appsnode appl]$ . ./APPSvisr12_appsnode.env

[oracle@appsnode appl]$ cat $CONTEXT_FILE | grep ias

<ias_version oa_var="s_ias_version">10.1.3.4.0</ias_version>

Pre-check 2 -

The minimum required version of OracleAS 10g Forms and Reports should be 10.1.2.3.0

(R12.1.1 comes with 10.1.3.3)

[oracle@appsnode appl]$ cat $CONTEXT_FILE | grep forms_version

<forms_version oa_var="s_forms_version">10.1.2.3.0</forms_version>

1. Apply Pre-requisite patch R12.AD.B.DELTA.2 i.e. 8502056

Execute the following command to generate your instance specific installation instructions

1. Source the Applications environment file

2. Run the Patch Application Assistant by entering "admsi.pl".

Patch Location - /home/oracle/R1212/8502056

Pre-install Tasks -

You must shut down all Application tier services before you start the patch application

Instructions for running AD Grants

Run the adgrants.sql script as a user that can connect as SYSDBA to grant privileges to selected SYS objects and create PL/SQL profiler objects.

Usage:

1. Create $ORACLE_HOME/appsutil/admin on the database server.

2. Copy adgrants.sql (UNIX) from this patch directory to

$ORACLE_HOME/appsutil/admin.

3. Set the environment to point to ORACLE_HOME on the database server.

4. Use SQL*Plus to run the script:

$ sqlplus /nolog

SQL> @$ORACLE_HOME/appsutil/admin/adgrants.sql apps

-- Stop the application tier

[oracle@appsnode appl]$ . ./APPSvisr12_appsnode.env

[oracle@appsnode appl]$ cd $ADMIN_SCRIPTS_HOME

[oracle@appsnode scripts]$sh adstpall.sh apps/apps

-- Enable Maintenance Mode

[oracle@appsnode 8502056]$

adadmin defaultsfile=defaultsfile=/u01/app/oracle/visr12/apps/apps_st/appl/admin/visr12/adalldefaults.txt logfile=adadmin.log

AD Administration Main Menu

--------------------------------------------------

1. Generate Applications Files menu

2. Maintain Applications Files menu

3. Compile/Reload Applications Database Entities menu

4. Maintain Applications Database Entities menu

5. Change Maintenance Mode

6. Exit AD Administration

Connecting to APPS......Connected successfully.

Change Maintenance Mode

----------------------------------------

Maintenance Mode is currently: [Disabled].

Maintenance mode should normally be enabled when patching

Oracle Applications and disabled when users are logged on

to the system. See the Oracle Applications Maintenance

Utilities manual for more information about maintenance mode.

Please select an option:

1. Enable Maintenance Mode

2. Disable Maintenance Mode

3. Return to Main Menu

Enter your choice [3] : 1

PL/SQL procedure successfully completed.

Commit complete.

Successfully enabled Maintenance Mode.

-- Apply patch

This patch contains the following unified driver file to be applied with AutoPatch:

u8502056.drv (This is the unified driver)

-- This will take 30-60 minutes

[oracle@appsnode 8502056]$

adpatch defaultsfile=/u01/app/oracle/visr12/apps/apps_st/appl/admin/visr12/adalldefaults.txt logfile=/home/oracle/patchapply_8502056.log driver=u8502056.drv workers=8

2. Apply 12.1.2 RUP 2 i.e. 7303033

Execute the following command to generate your instance specific installation instructions

1. Source the Applications environment file

2. Run the Patch Application Assistant by entering "admsi.pl"

-- Apply patch

This patch contains the following unified driver file to be applied with AutoPatch: u7303033.drv

-- This patch took around 30 hours for me as it failed twice and my system was running slow.

[oracle@appsnode 7303033]$

adpatch defaultsfile=/u01/app/oracle/visr12/apps/apps_st/appl/admin/visr12/adalldefaults.txt logfile=patch_apply_7303033.log driver=u7303033.drv workers=4

Issues -

Issue 1 -

Patch Hung after few hours

After approx 8 hours into the patch, patch was stuck with product AR and was not progressing at all. Hence I took a look with adctrl and found that worker 1 was stalled. I decided to stop worker and restart the failed job but it didn't improve the situation so I have to manually stop the application of the patch and re-initiate it again.

Issue 2-

Few fmx files failed to compile

When got the warning for failed execution, I choose to Yes and continued with the patch and then manually generated the plls/fmx files using adadmin to resolve the issue.

Issue 3 -

AutoPatch error:

The following ORACLE error:

ORA-03113: end-of-file on communication channel

occurred while executing the SQL statement:

SELECT control_code, status FROM fnd_install_processes ORDER BY worker_id

Telling workers to quit...

I checked the database following this error and found it went down without any apparent error. I restarted the DB and re-initiated the patch

Post Patch Process

Update database tier nodes with the Release 12.1.2 code level.

After you have upgraded to Release 12.1.2, perform the following post-update steps to update all database tier nodes with the code level, provided by Release 12.1.2:

-- Run the admkappsutil.pl utility to create the file appsutil.zip in the <INST_TOP>/admin/out directory.

[oracle@appsnode /]$perl $AD_TOP/bin/admkappsutil.pl

[oracle@appsnode ~]$ cd /u01/app/oracle/visr12/inst/apps/visr12_appsnode/admin/out/

[oracle@appsnode out]$ ll

-rw-r--r-- 1 oracle oinstall 2745685 May 25 07:46 appsutil.zip

-- Copy this appsutil.zip file to $ORACLE_HOME/appsutil

[oracle@appsnode out]$scp appsutil.zip appsdbnode:/u01/app/oracle/visr12/db/tech_st/11.1.0/appsutil

[oracle@appsdbnode 11.1.0]$ mv appsutil appsutil.old

[oracle@appsdbnode 11.1.0]$ unzip -o appsutil.zip

-- Run AutoConfig on the <RDBMS ORACLE_HOME>.

[oracle@appsdbnode visr12_appsdbnode]$ pwd

/u01/app/oracle/visr12/db/tech_st/11.1.0/appsutil/scripts/visr12_appsdbnode

[oracle@appsdbnode visr12_appsdbnode]$ ./adautocfg.sh apps/apps

Database tier and application tier:

Re-run adpreclone.pl on the database tier and the application tier.

[oracle@appsdbnode visr12_appsdbnode]$ perl ./adpreclone.pl dbTier

Redwood Shores, California, USA

Oracle Applications Rapid Clone

Version 12.0.0

adpreclone Version 120.20.12010000.2

Enter the APPS User Password:

Running:

perl /u01/app/oracle/visr12/db/tech_st/11.1.0/appsutil/bin/adclone.pl java=/u01/app/oracle/visr12/db/tech_st/11.1.0/appsutil/jre mode=stage stage=/u01/app/oracle/visr12/db/tech_st/11.1.0/appsutil/clone component=dbTier method=CUSTOM dbctx=/u01/app/oracle/visr12/db/tech_st/11.1.0/appsutil/visr12_appsdbnode.xml showProgress

APPS Password :

Beginning database tier Stage - Thu May 31 10:42:50 2012

/u01/app/oracle/visr12/db/tech_st/11.1.0/appsutil/jre/bin/java -Xmx600M -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/u01/app/oracle/visr12/db/tech_st/11.1.0/oui -classpath /u01/app/oracle/visr12/db/tech_st/11.1.0/lib/xmlparserv2.jar:/u01/app/oracle/visr12/db/tech_st/11.1.0/jdbc/lib/ojdbc6.jar:/u01/app/oracle/visr12/db/tech_st/11.1.0/appsutil/java:/u01/app/oracle/visr12/db/tech_st/11.1.0/oui/jlib/OraInstaller.jar:/u01/app/oracle/visr12/db/tech_st/11.1.0/oui/jlib/ewt3.jar:/u01/app/oracle/visr12/db/tech_st/11.1.0/oui/jlib/share.jar:/u01/app/oracle/visr12/db/tech_st/11.1.0/oui/jlib/srvm.jar:/u01/app/oracle/visr12/db/tech_st/11.1.0/jlib/ojmisc.jar oracle.apps.ad.clone.StageDBTier -e /u01/app/oracle/visr12/db/tech_st/11.1.0/appsutil/visr12_appsdbnode.xml -stage /u01/app/oracle/visr12/db/tech_st/11.1.0/appsutil/clone -tmp /tmp -method CUSTOM -showProgress

APPS Password :

Log file located at /u01/app/oracle/visr12/db/tech_st/11.1.0/appsutil/log/visr12_appsdbnode/StageDBTier_05311042.log

\ 50% completed

Completed Stage...

[oracle@appsnode scripts]$ perl ./adpreclone.pl appsTier

Redwood Shores, California, USA

Oracle Applications Rapid Clone

Version 12.0.0

adpreclone Version 120.20.12010000.5

Running:

perl /u01/app/oracle/visr12/apps/apps_st/appl/ad/12.0.0/bin/adclone.pl java=/u01/app/oracle/visr12/apps/tech_st/10.1.3/appsutil/jdk mode=stage stage=/u01/app/oracle/visr12/apps/apps_st/comn/clone component=appsTier method= appctx=/u01/app/oracle/visr12/inst/apps/visr12_appsnode/appl/admin/visr12_appsnode.xml showProgress

APPS Password :

method defaulted to CUSTOM

Beginning application tier Stage - Thu May 31 05:06:42 2012

/u01/app/oracle/visr12/apps/tech_st/10.1.3/appsutil/jdk/bin/java -Xmx600M -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath /u01/app/oracle/visr12/apps/tech_st/10.1.3/lib/xmlparserv2.jar:/u01/app/oracle/visr12/apps/tech_st/10.1.3/jdbc/lib/ojdbc14.jar:/u01/app/oracle/visr12/apps/apps_st/comn/java/classes:/u01/app/oracle/visr12/apps/tech_st/10.1.3/oui/jlib/OraInstaller.jar:/u01/app/oracle/visr12/apps/tech_st/10.1.3/oui/jlib/ewt3.jar:/u01/app/oracle/visr12/apps/tech_st/10.1.3/oui/jlib/share.jar:/u01/app/oracle/visr12/apps/tech_st/10.1.3/oui/jlib/srvm.jar:/u01/app/oracle/visr12/apps/tech_st/10.1.3/jlib/ojmisc.jar oracle.apps.ad.clone.StageAppsTier -e /u01/app/oracle/visr12/inst/apps/visr12_appsnode/appl/admin/visr12_appsnode.xml -stage /u01/app/oracle/visr12/apps/apps_st/comn/clone -tmp /tmp -method CUSTOM -showProgress

Log file located at /u01/app/oracle/visr12/inst/apps/visr12_appsnode/admin/log/StageAppsTier_05310506.log

/ 20% completed

Completed Stage...

-- Disable the maintenance mode on Apps tier

[oracle@appsnode 8502056]$

adadmin defaultsfile=defaultsfile=/u01/app/oracle/visr12/apps/apps_st/appl/admin/visr12/adalldefaults.txt logfile=adadmin.log

AD Administration Main Menu

--------------------------------------------------

1. Generate Applications Files menu

2. Maintain Applications Files menu

3. Compile/Reload Applications Database Entities menu

4. Maintain Applications Database Entities menu

5. Change Maintenance Mode

6. Exit AD Administration

Connecting to APPS......Connected successfully.

Change Maintenance Mode

----------------------------------------

Maintenance Mode is currently: [Enabled].

Maintenance mode should normally be enabled when patching Oracle Applications and disabled when users are logged on to the system. See the Oracle Applications Maintenance Utilities manual for more information about maintenance mode.

Please select an option:

1. Enable Maintenance Mode

2. Disable Maintenance Mode

3. Return to Main Menu

Enter your choice [3] : 2

PL/SQL procedure successfully completed.

Commit complete.

Successfully disabled Maintenance Mode.

Start the Application Tier -

-- Start the services on Apps tier -

[oracle@appsnode scripts]$ pwd

/u01/app/oracle/visr12/inst/apps/visr12_appsnode/admin/scripts

[oracle@appsnode scripts]$ ./adstrtal.sh apps/apps

[oracle@appsnode scripts]$ ./adopmnctl.sh status

You are running adopmnctl.sh version 120.6.12010000.3

Checking status of OPMN managed processes...

Processes in Instance: visr12_appsnode.appsnode.localdomain

---------------------------------+--------------------+---------+---------

ias-component | process-type | pid | status

---------------------------------+--------------------+---------+---------

OC4JGroup:default_group | OC4J:oafm | 16264 | Alive

OC4JGroup:default_group | OC4J:forms | 16176 | Alive

OC4JGroup:default_group | OC4J:oacore | 16095 | Alive

HTTP_Server | HTTP_Server | 16043 | Alive

Check the Version from Back End -

[oracle@appsdbnode ~]$ cd /u01/app/oracle/visr12/db/tech_st/11.1.0/

[oracle@appsdbnode 11.1.0]$ . ./visr12_appsdbnode.env

[oracle@appsdbnode 11.1.0]$ sqlplus apps/apps

SQL> select RELEASE_NAME from fnd_product_groups;

RELEASE_NAME

--------------------------------------------------

12.1.2

You just have patch Oracle EBS/Apps R12 to new code level.

Next Step - Patch 12.1.2 Apps to 12.1.3

Apply PSU 5 for 11gR2 Grid and RAC (11.2.0.2)

Oracle released the Patch Set Update 5 for 11g Grid Infrastructure and RAC recently. This PSU contains some imp bug fixes and new enhancements so I decided to apply the latest Grid Infra Patch Set Update on 11.2.0.2( Both CRS and RAC are on same version) in my test env.

During the analysis, the important facet that catches me was, "Do Patchset Updates (PSU's) Change the Oracle Release Version/Fifth Digit?"

Answer was Big NO.

As per MOS -
The PSU does NOT change the database version . That means PSU 10.2.0.4.x /10.2.0.5.x, 11.1.0.7.x, 11.2.0.1.x /11.2.0.2.x ( where x is the fifth digit ) does NOT change the 5th digit of the Oracle Database Server version.
After applying the PSU "opatch lsinventory" still shows the version as 10.2.0.4.0/10.2.0.5.0 for Oracle 10g R2, 11.1.0.7 for Oracle 11g R1 and 11.2.0.1 /11.2.0.2 for Oracle 11g R2
PSU's also will not change the version of oracle binaries (like sqlplus, exp/imp etc.)

Note : It is also applicable for the Oracle Enterprise Manager Grid Control and EM Agent

In-fact it was the supposed to be my last note in this post but turned out to be the first one!!

Its time, I run you thru the actual process step by step

Noes - racnode2, racnode2
GI Home - 112020
RAC Home - 11202
ASM - Yes

Another imp note is that when you apply PSU, you can choose to either keep HAS Stack up or down, I actually choose to keep the stack down to avoid any unpleasant situation that may arise...

[root@racnode1 ~]# cd /u01/app/11.2.0.2/grid/bin/

[root@racnode1 bin]# ./crsctl stop crs

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'racnode1'

CRS-2673: Attempting to stop 'ora.crsd' on 'racnode1'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'racnode1'

[root@racnode2 ~]# cd /u01/app/11.2.0.2/grid/bin/

[root@racnode2 bin]# ./crsctl stop crs

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'racnode2'

Pre-requisite :

Apply latest opatch patch on RAC nodes (OPatch utility version should be 11.2.0.1.5 or later or otherwise PSU application will fail)

Patch number - p6880880_112000_LINUX.zip

ON NODE1 -

[root@racnode1 grid]# pwd

/u01/app/11.2.0.2/grid

[root@racnode1 grid]# sudo su - oracle

[oracle@racnode1 ~]$ cd /u01/app/11.2.0.2/grid

[oracle@racnode1 OPatch]$ . oraenv

ORACLE_SID = [oracle] ? +ASM1

The Oracle base for ORACLE_HOME=/u01/app/11.2.0.2/grid is /u01/app/oracle

[oracle@racnode1 OPatch]$ ./opatch lsinventory

Invoking OPatch 11.2.0.1.1

Oracle Interim Patch Installer version 11.2.0.1.1

Oracle Home : /u01/app/11.2.0.2/grid

Central Inventory : /u01/app/oraInventory

from : /etc/oraInst.loc

OPatch version : 11.2.0.1.1 -- Version Before OPatch patch was applied

OUI version : 11.2.0.2.0

OUI location : /u01/app/11.2.0.2/grid/oui

Log file location : /u01/app/11.2.0.2/grid/cfgtoollogs/opatch/opatch2012-05-22_12-09-22PM.log

Patch history file: /u01/app/11.2.0.2/grid/cfgtoollogs/opatch/opatch_history.txt

Lsinventory Output file location : /u01/app/11.2.0.2/grid/cfgtoollogs/opatch/lsinv/lsinventory2012-05-22_12-09-22PM.txt

--------------------------------------------------------------------------------

Installed Top-level Products (1):

Oracle Grid Infrastructure 11.2.0.2.0

There are 1 products installed in this Oracle Home.

There are no Interim patches installed in this Oracle Home.

Rac system comprising of multiple nodes

Local node = racnode1

Remote node = racnode2

--------------------------------------------------------------------------------

OPatch succeeded.

ON NODE2 -

[root@racnode2 bin]# sudo su - oracle

[oracle@racnode2 ~]$ cd /u01/app/11.2.0.2/grid/OPatch/

[oracle@racnode2 OPatch]$ . oraenv

ORACLE_SID = [wiustg2] ? +ASM2

The Oracle base for ORACLE_HOME=/u01/app/11.2.0.2/grid is /u01/app/oracle

[oracle@racnode2 OPatch]$ ./opatch lsinventory

Invoking OPatch 11.2.0.1.1

Oracle Interim Patch Installer version 11.2.0.1.1

Oracle Home : /u01/app/11.2.0.2/grid

Central Inventory : /u01/app/oraInventory

from : /etc/oraInst.loc

OPatch version : 11.2.0.1.1 -- Version Before OPatch patch was applied

OUI version : 11.2.0.2.0

OUI location : /u01/app/11.2.0.2/grid/oui

Log file location : /u01/app/11.2.0.2/grid/cfgtoollogs/opatch/opatch2012-05-22_12-10-59PM.log

Patch history file: /u01/app/11.2.0.2/grid/cfgtoollogs/opatch/opatch_history.txt

Lsinventory Output file location : /u01/app/11.2.0.2/grid/cfgtoollogs/opatch/lsinv/lsinventory2012-05-22_12-10-59PM.txt

--------------------------------------------------------------------------------

Installed Top-level Products (1):

Oracle Grid Infrastructure 11.2.0.2.0

There are 1 products installed in this Oracle Home.

There are no Interim patches installed in this Oracle Home.

Rac system comprising of multiple nodes

Local node = racnode2

Remote node = racnode1

--------------------------------------------------------------------------------

OPatch succeeded.

ON NODE1 -

[oracle@racnode1 OPatch]$ echo $ORACLE_HOME

/u01/app/11.2.0.2/grid

[oracle@racnode1 OPatch]$ unzip /home/oracle/oracle11202/Opatch_patch112015/p6880880_112000_LINUX_OPatch112015.zip -d /u01/app/11.2.0.2/grid

Archive: /home/oracle/oracle11202/Opatch_patch112015/p6880880_112000_LINUX_OPatch112015.zip

creating: /u01/app/11.2.0.2/grid/OPatch/oplan/

inflating: /u01/app/11.2.0.2/grid/OPatch/oplan/README.html

inflating: /u01/app/11.2.0.2/grid/OPatch/oplan/README.txt

replace /u01/app/11.2.0.2/grid/OPatch/docs/FAQ? [y]es, [n]o, [A]ll, [N]one, [r]ename: A

inflating: /u01/app/11.2.0.2/grid/OPatch/docs/FAQ

[oracle@racnode1 OPatch]$ ./opatch lsinventory

Oracle Interim Patch Installer version 11.2.0.3.0

Oracle Home : /u01/app/11.2.0.2/grid

Central Inventory : /u01/app/oraInventory

from : /u01/app/11.2.0.2/grid/oraInst.loc

OPatch version : 11.2.0.3.0 -- Version after OPatch patch applied

OUI version : 11.2.0.2.0

[root@racnode1 ~]# unzip /home/oracle/oracle11202/Opatch_patch112015/p6880880_112000_LINUX_OPatch112015.zip -d /u01/app/oracle/product/11.2.0.2/dbhome_1
Archive: /home/oracle/oracle11202/Opatch_patch112015/p6880880_112000_LINUX_OPatch112015.zip
creating: /u01/app/oracle/product/11.2.0.2/dbhome_1/OPatch/oplan/
inflating: /u01/app/oracle/product/11.2.0.2/dbhome_1/OPatch/oplan/README.html
replace /u01/app/oracle/product/11.2.0.2/dbhome_1/OPatch/docs/FAQ? [y]es, [n]o, [A]ll, [N]one, [r]ename: A

ON NODE2 -

[oracle@racnode2 OPatch]$ echo $ORACLE_HOME

/u01/app/11.2.0.2/grid

[oracle@racnode2 OPatch]$ unzip /home/oracle/oracle11202/Opatch_patch112015/p6880880_112000_LINUX_OPatch112015.zip -d /u01/app/11.2.0.2/grid

Archive: /home/oracle/oracle11202/Opatch_patch112015/p6880880_112000_LINUX_OPatch112015.zip

creating: /u01/app/11.2.0.2/grid/OPatch/oplan/

[oracle@racnode2 OPatch]$ ./opatch lsinventory

Oracle Interim Patch Installer version 11.2.0.3.0

Oracle Home : /u01/app/11.2.0.2/grid

Central Inventory : /u01/app/oraInventory

from : /u01/app/11.2.0.2/grid/oraInst.loc

OPatch version : 11.2.0.3.0 -- Version after OPatch patch applied

OUI version : 11.2.0.2.0

One-off Patch Conflict Detection and Resolution.

opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ./13653086
export PATH=$PATH:<GI_HOME>/OPatch
opatch auto <UNZIPPED_PATCH_LOCATION>
opatch auto <UNZIPPED_PATCH_LOCATION> -oh <GI_HOME>
opatch auto <UNZIPPED_PATCH_LOCATION> -oh <ORACLE_HOME>, <ORACLE_HOME2>

Patch 13653086 : applied on Tue May 22 14:15:01 IST 2012
Unique Patch ID: 14538982
Patch description: "Grid Infrastructure Patch Set Update 11.2.0.2.5 (includes DB PSU 11.2.0.2.5) #2"
Created on 2 Feb 2012, 02:06:04 hrs PST8PDT

[root@racnode2 ~]# unzip /home/oracle/oracle11202/Opatch_patch112015/p6880880_112000_LINUX_OPatch112015.zip -d /u01/app/oracle/product/11.2.0.2/dbhome_1
Archive: /home/oracle/oracle11202/Opatch_patch112015/p6880880_112000_LINUX_OPatch112015.zip
creating: /u01/app/oracle/product/11.2.0.2/dbhome_1/OPatch/oplan/
inflating: /u01/app/oracle/product/11.2.0.2/dbhome_1/OPatch/oplan/README.html
replace /u01/app/oracle/product/11.2.0.2/dbhome_1/OPatch/docs/FAQ? [y]es, [n]o, [A]ll, [N]one, [r]ename: A

NOTE -
The fastest and easiest way to determine whether you have one-off patches in the Oracle home that conflict with the PSU, and to get the necessary conflict resolution patches, is to use the Patch Recommendations and Patch Plans features on the Patches & Updates tab in My Oracle Support.

Before beginning patch application, check the consistency of inventory information for GI home and each database home to be patched. Run the following command as respective Oracle home owner to check the consistency.

<ORACLE_HOME>/OPatch/opatch lsinventory -detail -oh $ORACLE_HOME

Generate ocm.rsp file -

Since we apply the patch in silent mode, one needs to create ocm.rsp file which will be required during patch application. Generate the ocm.rsp as follows

[root@racnode1 11.2.0.2]# export PATH=$PATH:/u01/app/11.2.0.2/grid/OPatch

-- To generate ocm.rsp file

#$ORACLE_HOME/OPatch/ocm/bin/emocmrsp -no_banner -output /home/oracle/ocm.rsp

The OCM configuration response file (/home/oracle/ocm.rsp) was successfully created.

-- To Patch GI Home

The utility must be executed by an operating system (OS) user with root privileges (usually the user root), and it must be executed on each node in the cluster if the GI home or Oracle RAC database home is in Non-shared storage.

The utility should not be run in parallel on the cluster nodes.

Depending on command line options specified, one invocation of Opatch can patch the GI home, one or more Oracle RAC database homes, or both GI and Oracle RAC or Oracle RAC One Node database homes of the same Oracle release version. You can also roll back the patch with the same selectivity.

Add the directory containing the opatch to the $PATH environment variable.

You can choose following methods to apply patches selectively to different Homes at same time or separately.

1. To patch GI home and all Oracle RAC database homes of the same version at same time

2. To patch only the GI home

3. To patch one or more Oracle RAC database homes

I choose to apply patch to GI Home and RAC Home separately to have better control and easy of troubleshooting.

Applying Patch on GI Home -

ON NODE1 -

[root@racnode1 112025_GI_PSUPatch]# opatch auto /home/oracle/112025_GI_PSUPatch -och /u01/app/11.2.0.2/grid

Executing /usr/bin/perl /u01/app/11.2.0.2/grid/OPatch/crs/patch112.pl -patchdir /home/oracle -patchn 112025_GI_PSUPatch -och /u01/app/11.2.0.2/grid -paramfile /u01/app/11.2.0.2/grid/crs/install/crsconfig_params

opatch auto log file location is /u01/app/11.2.0.2/grid/OPatch/crs/../../cfgtoollogs/opatchauto2012-05-22_14-01-14.log

Detected Oracle Clusterware install

Using configuration parameter file: /u01/app/11.2.0.2/grid/crs/install/crsconfig_params

OPatch is bundled with OCM, Enter the absolute OCM response file path:

/home/oracle/ocm.rsp

Successfully unlock /u01/app/11.2.0.2/grid

patch /home/oracle/112025_GI_PSUPatch/13653086 apply successful for home /u01/app/11.2.0.2/grid

patch /home/oracle/112025_GI_PSUPatch/13343424 apply successful for home /u01/app/11.2.0.2/grid

ACFS-9300: ADVM/ACFS distribution files found.

ACFS-9312: Existing ADVM/ACFS installation detected.

ACFS-9314: Removing previous ADVM/ACFS installation.

ACFS-9315: Previous ADVM/ACFS components successfully removed.

ACFS-9307: Installing requested ADVM/ACFS software.

ACFS-9308: Loading installed ADVM/ACFS drivers.

ACFS-9321: Creating udev for ADVM/ACFS.

ACFS-9323: Creating module dependencies - this may take some time.

ACFS-9327: Verifying ADVM/ACFS devices.

ACFS-9309: ADVM/ACFS installation correctness verified.

CRS-4123: Oracle High Availability Services has been started.

ON NODE2 -

[root@racnode2 112025_GI_PSUPatch]# cd /u01/app/11.2.0.2/grid/OPatch/
[root@racnode2 OPatch]# opatch auto /home/oracle/112025_GI_PSUPatch -och /u01/app/11.2.0.2/grid

Executing /usr/bin/perl /u01/app/11.2.0.2/grid/OPatch/crs/patch112.pl -patchdir /home/oracle -patchn 112025_GI_PSUPatch -och /u01/app/11.2.0.2/grid -paramfile /u01/app/11.2.0.2/grid/crs/install/crsconfig_params opatch auto log file location is /u01/app/11.2.0.2/grid/OPatch/crs/../../cfgtoollogs/opatchauto2012-05-22_14-33-40.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/app/11.2.0.2/grid/crs/install/crsconfig_params
OPatch is bundled with OCM, Enter the absolute OCM response file path: /home/oracle/ocm.rsp

Successfully unlock /u01/app/11.2.0.2/grid patch /home/oracle/112025_GI_PSUPatch/13653086 apply successful for home /u01/app/11.2.0.2/grid patch /home/oracle/112025_GI_PSUPatch/13343424 apply successful for home /u01/app/11.2.0.2/grid ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.

ACFS-9315: Previous ADVM/ACFS components successfully removed.

ACFS-9307: Installing requested ADVM/ACFS software.

ACFS-9308: Loading installed ADVM/ACFS drivers.

ACFS-9321: Creating udev for ADVM/ACFS.

ACFS-9323: Creating module dependencies - this may take some time.

ACFS-9327: Verifying ADVM/ACFS devices.

ACFS-9309: ADVM/ACFS installation correctness verified.

CRS-4123: Oracle High Availability Services has been started.

Applying Patch on RAC Home -

ON NODE1 -

[root@racnode1 ~]# which opatch

/u01/app/oracle/product/11.2.0.2/dbhome_1/OPatch/opatch

[root@racnode1]# opatch auto /home/oracle/112025_GI_PSUPatch -oh /u01/app/oracle/product/11.2.0.2/dbhome_1

Executing /usr/bin/perl ./crs/patch112.pl -patchdir /home/oracle -patchn 112025_GI_PSUPatch -oh /u01/app/oracle/product/11.2.0.2/dbhome_1 -paramfile /u01/app/11.2.0.2/grid/crs/install/crsconfig_params
opatch auto log file location is /u01/app/oracle/product/11.2.0.2/dbhome_1/OPatch/crs/../../cfgtoollogs/opatchauto2012-05-22_16-13-57.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/app/11.2.0.2/grid/crs/install/crsconfig_params
OPatch is bundled with OCM, Enter the absolute OCM response file path:
/home/oracle/ocm.rsp
patch /home/oracle/112025_GI_PSUPatch/13653086/custom/server/13653086 apply successful for home /u01/app/oracle/product/11.2.0.2/dbhome_1
patch /home/oracle/112025_GI_PSUPatch/13343424 apply successful for home /u01/app/oracle/product/11.2.0.2/dbhome_1

ON NODE2 -

[root@racnode2 ~]# export PATH=$PATH:/u01/app/oracle/product/11.2.0.2/dbhome_1/OPatch

[root@racnode2 ~]# which opatch

/u01/app/oracle/product/11.2.0.2/dbhome_1/OPatch/opatch

[root@racnode2]# opatch auto /home/oracle/112025_GI_PSUPatch -oh /u01/app/oracle/product/11.2.0.2/dbhome_1

Executing /usr/bin/perl /u01/app/oracle/product/11.2.0.2/dbhome_1/OPatch/crs/patch112.pl -patchdir /home/oracle -patchn 112025_GI_PSUPatch -oh /u01/app/oracle/product/11.2.0.2/dbhome_1 -paramfile /u01/app/11.2.0.2/grid/crs/install/crsconfig_params

opatch auto log file location is /u01/app/oracle/product/11.2.0.2/dbhome_1/OPatch/crs/../../cfgtoollogs/opatchauto2012-05-22_15-38-25.log

Detected Oracle Clusterware install

Using configuration parameter file: /u01/app/11.2.0.2/grid/crs/install/crsconfig_params

OPatch is bundled with OCM, Enter the absolute OCM response file path:

/home/oracle/ocm.rsp

patch /home/oracle/112025_GI_PSUPatch/13653086/custom/server/13653086 apply successful for home /u01/app/oracle/product/11.2.0.2/dbhome_1

Check Inventory on Both Nodes -

[oracle@racnode1 OPatch]$ ./opatch lsinventory -detail

Oracle Home : /u01/app/11.2.0.2/grid

Central Inventory : /u01/app/oraInventory

Interim patches (2) :

Patch 13343424 : applied on Tue May 22 14:20:00 IST 2012

Unique Patch ID: 14538982

Patch description: "Database Patch Set Update : 11.2.0.2.5 (13343424)"

[oracle@racnode2 OPatch]$ ./opatch lsinventory -bugs_fixed | egrep 'PSU|PATCH SET UPDATE'

13343447 13653086 Tue May 22 14:47:17 IST 2012 GI PSU 11.2.0.2.5 (INCLUDES DATABASE PSU 11.2.0.2.
13396479 13653086 Tue May 22 14:47:17 IST 2012 11.2.0.2 GI PSU 5 HAS-CRS TRACKING BUG
13396487 13653086 Tue May 22 14:47:17 IST 2012 11.2.0.2 GI PSU5 USM TRACKING BUG
JAN 2012 PSU
13653086 13653086 Tue May 22 14:47:17 IST 2012 GRID INFRASTRUCTURE PATCH SET UPDATE 11.2.0.2.5
(INCLUDES DB PSU 11.2.0.2.5)

As you can now see that the PSU is reflecting in inventory properly.

Post Patch Steps -

Start your database/instances/services and check that they are running fine...

Tuesday, May 29, 2012

11g Agent Crashes After Start-up / Time Zone Issue With Agent /

Time Zone Skew Between OEM & Agent

I recently came across the issue that agent was removed from one of our cluster and was re-installed later on. However, during the re-installation the timezone was not selected properly. The timezone has to be named timezone but was not set correctly. Due to this, the moment the agent starts, it crashes.

We tried to correct the Timezone at OS level but still the issue didn't get resolved coz the entry in emd.properties was still having the old value.

So the value in emd.properties files needs to get updated. Following is the way to correct it...

[oracle@racnode1 config]$ echo $TZ
+05:30

[oracle@racnode1 config]$ emctl config agent getTZ
EM Configuration issue. /u01/app/oracle/product/11.2.0/dbhome_1/racnode1_ORCL not found.

[oracle@racnode1 config]$ export TZ=IST-5:30IST
[oracle@racnode1 config]$ echo $TZ
IST-5:30IST

[oracle@racnode1 config]$ emctl config agent getTZ
EM Configuration issue. /u01/app/oracle/product/11.2.0/dbhome_1/racnode1_ORCL not found.

[oracle@racnode1 config]$ /u01/app/oracle/product/11.2.0/agent11g/bin/emctl config agent getTZ
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
GMT+05:30

[oracle@racnode1 config]$ export ORACLE_HOME=/u01/app/oracle/product/11.2.0/agent11g
[oracle@racnode1 config]$ export PATH=$ORACLE_HOME/bin:$PATH
[oracle@racnode1 config]$ cd $ORACLE_HOME/bin

[oracle@racnode1 bin]$ ./emctl resetTZ agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Updating /u01/app/oracle/product/11.2.0/agent11g/sysman/config/emd.properties...
----- Mon Apr 9 16:16:21 2012::tzOffset for +05:30 is 330(min), but agent is runnning with tzOffset 390(min)
-----
----- Mon Apr 9 16:16:21 2012::trying again after waiting for 1 sec to account for daylight transition
-----
----- Mon Apr 9 16:16:21 2012::tzOffset for +05:30 is 330(min), but agent is runnning with tzOffset 390(min)
-----
resetTZ failed.
The agentTZRegion in:
/u01/app/oracle/product/11.2.0/agent11g/sysman/config/emd.properties
is not in agreement with what the agent thinks it should be.
Fix your environment.
Pick a TZ value that corresponds to time zone settings listed in:
/u01/app/oracle/product/11.2.0/agent11g/sysman/admin/supportedtzs.lst

-- check the correct value from following file, find out the calcutta tz setting and set it in TZ env variable
[oracle@racnode1 bin]$ vi /u01/app/oracle/product/11.2.0/agent11g/sysman/admin/supportedtzs.lst

[oracle@racnode1 bin]$ export TZ=Asia/Calcutta

[oracle@racnode1 bin]$ ./emctl resetTZ agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Updating /u01/app/oracle/product/11.2.0/agent11g/sysman/config/emd.properties...
Successfully updated /u01/app/oracle/product/11.2.0/agent11g/sysman/config/emd.properties.

Since you have successfully modified the Agent configuration, now its the time to modify the settings in the OEM Repository.

-- Login as the em repository user and run the script:
SQL> exec mgmt_target.set_agent_tzrgn('racnode1.localdomain:3872','Asia/Calcutta')
SQL> commit;

Log in to the repository database as SYSMAN and run (again) -
SQL> select timezone_region from mgmt_targets where target_name ='racnode1.localdomain:3872';
'Asia/Calcutta'

The above query confirms that TZ setting is correct now in OEM.

Try to restart the agent and give few minutes to sync up. It issues should have resolved now!!

Sometimes, you may have to clear state of the agent and secure it again.

Monday, May 28, 2012

Multi-Node Installation of Oracle EBS (Apps) R12

This post describes how to install Oracle EBusiness Suite AKA Oracle Apps, R12 in multi node environment on linux.

EBS software version - 12.1.1

OS - RHEL 5.5 32 bit

DB Node - appsdbnode.localdomain, require around 45-50 GB for Fresh installation and around 120 GB for Vision DB installation

Application Node - appsnode.localdomain, requires around 30-35 GB for application tier file-system

Staging (over NFS) - 192.168.1.14:/var/ovs/mount/600E49F9AFBA4C82B97000DA02088110/iso_pool/FinalAppsInstaller

Since we are doing multiple node installation, we have to make sure that both nodes have installers available to them.

So one way, either you make two copies, one on each individual hosts or use some shared drive mechanism.

I used NFS to stage the installers and mount NFS mounts on local mount points.

In this case I am creating a stage on 192.168.1.14 host.

You can download the software from http://edelivery.oracle.com

Following is the list of files you need to download and unzip in your staging location in order to install EBS

B53824-01_1of4.zip

B53824-01_2of4.zip

B53824-01_3of4.zip

B53824-01_4of4.zip

V15564-01_1of3.zip

V15564-01_2of3.zip

V15564-01_3of3.zip

V15565-01_1of2.zip

V15565-01_2of2.zip

V15566-01_1of3.zip

V15566-01_2of3.zip

V15566-01_3of3.zip

V15567-01_1of3.zip

V15567-01_2of3.zip

V15567-01_3of3.zip

V15568-01_1of3.zip

V15568-01_2of3.zip

V15568-01_3of3.zip

V15569-01_1of3.zip

V15569-01_2of3.zip

V15569-01_3of3.zip

V15570-01_1of3.zip

V15570-01_2of3.zip

V15570-01_3of3.zip

V15571-01_1of3.zip

V15571-01_2of3.zip

V15571-01_3of3.zip

V15572-01_1of3.zip

V15572-01_2of3.zip

V15572-01_3of3.zip

V15573-01_1of3.zip

V15573-01_2of3.zip

V15573-01_3of3.zip

V15574-01_1of3.zip

V15574-01_2of3.zip

V15574-01_3of3.zip

V15575-01.zip

V15576-01_1of3.zip

V15576-01_2of3.zip

V15576-01_3of3.zip

[root@appsdbnode ~]# showmount -e 192.168.1.14
Export list for 192.168.1.14:
/var/ovs/mount/600E49F9AFBA4C82B97000DA02088110/iso_pool/FinalAppsInstaller *

[root@appsdbnode ~]# service nfs restart
Shutting down NFS mountd: [FAILED]
Shutting down NFS daemon: [FAILED]
Shutting down NFS quotas: [FAILED]
Shutting down NFS services: [FAILED]
Starting NFS services: [ OK ]
Starting NFS quotas: [ OK ]
Starting NFS daemon: [ OK ]
Starting NFS mountd: [ OK ]

[root@appsdbnode ~]# service portmap restart
Stopping portmap: [ OK ]
Starting portmap: [ OK ]

[root@appsdbnode ~]# mount.nfs 192.168.1.14:/var/ovs/mount/600E49F9AFBA4C82B97000DA02088110/iso_pool/FinalAppsInstaller /appsinstall/
[root@appsdbnode ~]# cd /appsinstall/
[root@appsdbnode appsinstall]# ll
total 1024
-rwxrwxrwx 1 root root 754 Apr 26 20:42 filelist.txt
drwxrwxrwx 70 root root 3896 Apr 26 20:17 oraAppDB
drwxrwxrwx 14 root root 3896 Apr 26 20:32 oraApps
drwxrwxrwx 5 root root 3896 Apr 26 20:20 oraAS
drwxrwxrwx 9 root root 3896 Apr 26 19:26 oraDB
drwxrwxrwx 4 root root 3896 Apr 26 20:38 R12.1.2Patches
drwxrwxrwx 6 root root 3896 Apr 26 20:41 startCD
-rwxrwxrwx 1 root root 512668 Apr 26 20:33 unzip

Once unzipped you need to navigate to startCD/Disk1/rapidwiz directory.

Here you will be able to see rapidwiz executable

# ./rapidwiz

Select the Oracle Applications Release 12.1.1

Select suitable option if you need to download updates

Select [New Configuration]

Select the port pool, I have chosen the default i.e. 0, for my env.

Enter DB host info along with user and group

Select [Suite Licensing]

Install the languages if you need multi-lingual support

Select the Internationalization Settings

Enter Application Node info along with owner and group

Review the info you entered for hosts. You can also add Additional Nodes here for Application tier file system

Now the OUI will perform the pre-checks before the actual installation kicks in

Review the Validation Report. check the failed ones and click on Retry

Installation will kick in, keep an eye on it. It will usually take around couple of hours depending on the speed of your machine

Once installation completes, the post install validation box will reappear. Make sure everything is green before going further

At the end, click on Finish to end the installation of DB tier.

Next step is to install the application tier on appsnode host.
Before you start the install of the App tier you need to copy the config file from DB tier to apps tier.
To do this there are two ways
1. Copy <sid>_<host>.xml file aka context file to appsnode
2. start the install by issuing ./rapidwiz and provide the DB details to copy the config from DB

One can copy context file as follows

[oracle@appsdbnode appsutil]$ cd /u01/app/oracle/visr12/db/tech_st/11.1.0/appsutil
[oracle@appsdbnode appsutil]$ scp conf_visr12.txt appsnode:/home/oracle/
conf_visr12.txt 100% 13KB 12.8KB/s 00:00

click on [Saved configuration] option, and enter the path copied context file

check the validation report to make sure everything is healthy.

Click on [Next]

Review the progress of the installation

After install was finished, my post install checks failed as my HTTP server failed to start hence all other dependencies failed

Cause -
After a bit of research I found out that the start failed due to the fact it failed to locate on library component and fix is the create the link for that missing component

Solution -
[root@appsnode ~]# ln -s /usr/lib/libgdbm.so.2.0.0 /usr/lib/libdb.so.2

[oracle@appsnode scripts]$ adstrtal.sh apps/apps

[oracle@appsnode scripts]$ adopmnctl.sh status
You are running adopmnctl.sh version 120.6
Checking status of OPMN managed processes...
Processes in Instance: visr12_appsnode.appsnode.localdomain
---------------------------------+--------------------+---------+----------------------
ias-component | process-type | pid | status
---------------------------------+--------------------+---------+----------------------
OC4JGroup:default_group | OC4J:oafm | 31135 | Alive
OC4JGroup:default_group | OC4J:forms | 31070 | Alive
OC4JGroup:default_group | OC4J:oacore | 30984 | Alive
HTTP_Server | HTTP_Server | 30933 | Alive

After manually start the opmn stack, I click on [Retry] on validation page of installer and there you go. Everything came up

click on [Finish] to end the install

Post install, I tried to login to Apps console using following URL.
URL Format : http://<hostname.domain>:<port_pool_no>

URL - http://appsnode.localdomain:8000
use default credentials - sysadmin/sysadmin to login console

That's it. You have now successfully installed the EBS R12.1.1 successfully.

Next -
Apply 12.1.2 or 12.1.3 RUP to Apps R12 (12.1.1)

Wednesday, May 30, 2012

Patch Oracle EBS/Apps R12 from 12.1.1 to 12.1.2

Apply PSU 5 for 11gR2 Grid and RAC (11.2.0.2)

Tuesday, May 29, 2012

11g Agent Crashes After Start-up / Time Zone Issue With Agent / Time Zone Skew Between OEM & Agent

Monday, May 28, 2012

Multi-Node Installation of Oracle EBS (Apps) R12

11g Agent Crashes After Start-up / Time Zone Issue With Agent /

Time Zone Skew Between OEM & Agent