Tuesday, February 12, 2013

OUI-15038: Unable to execute rule 'CheckCentralInventoryLocation' OR OPatch failed to locate Central Inventory

This happend during application of the CPU aka SPU i.e. SPU Jan 2013. While running the OPatch on the oracle home we hit the wall.
As a standard practice we run the pre-requisite checks on any Oracle Home / DB which needs to be patched and we ran into folliwng errors.

Output Log
Step is being run by operating system user : 'oracle'
Run privilege of the step is : Normal 
This is Provisioning Executor Script

Input PropertyFilePath is /u01/app/oracle/agent11g/agent11g/EMStage/PAF/Job_d457e08c75eb0142e043257a9597dd42/properties
Input directiveScriptFilePath is /u01/app/oracle/agent11g/agent11g/EMStage/PAF/Job_d457e08c75eb0142e043257a9597dd42/runPrereq.pl
Input directiveTypeProperty is NAME_Executor
Input componentPath is null
Input directivePath is Directive
Input componentFilename is null
Directive Type is SUB_Perl
Dir to CD is /u01/app/oracle/agent11g/agent11g/EMStage/PAF/Job_d457e08c75eb0142e043257a9597dd42
Final command line to execute is $PERL_HOME/perl "/u01/app/oracle/agent11g/agent11g/EMStage/PAF/Job_d457e08c75eb0142e043257a9597dd42/runPrereq.pl" -debug "false" -oh "/u01/app/oracle/product/11.2.3/db_1" -stageLoc "/u01/app/oracle/agent11g/agent11g/EMStage" -entryPoint "oracle.opatch.em.apply" -patches "14841409" -usePrereqComp "false" -isRac "false" -racNodes "${racNodes}" ${cleanPatches} -patchOpts "''" -applyBundle "${applyBundle}" -entryPointForPatchset "oracle.patchset.db_Custom" -isPatchset "false" -sqlScript ""defaultSqlScript""
The output of the directive is:
Mon Feb 11 03:54:03 2013 - Arguments: -debug false -oh /u01/app/oracle/product/11.2.3/db_1 -stageLoc /u01/app/oracle/agent11g/agent11g/EMStage -entryPoint oracle.opatch.em.apply -patches 14841409 -usePrereqComp false -isRac false -racNodes  -patchOpts '' -applyBundle  -entryPointForPatchset oracle.patchset.db_Custom -isPatchset false -sqlScript defaultSqlScript
Oracle Home:/u01/app/oracle/product/11.2.3/db_1
Stage Location:/u01/app/oracle/agent11g/agent11g/EMStage
Entry Point:oracle.opatch.em.apply
Is RAC :false
Entry Point For Patchset:oracle.patchset.db_Custom
Is PATCHSET :false
OUI and OPatch present in the ORACLE_HOME:/u01/app/oracle/product/11.2.3/db_1 will be used for running the prerequisite checks.
Mon Feb 11 03:54:03 2013 - Checking if the patch has sqls and the required metadata...

Mon Feb 11 03:54:03 2013 - Path is: /u01/app/oracle/agent11g/agent11g/EMStage/14841409/14841409
Mon Feb 11 03:54:03 2013 - STATUS: Patch '14841409' has required metadata to apply sql.

Checking for the latest OPatch ...
Oracle Interim patch Installer location:/u01/app/oracle/product/11.2.3/db_1/OPatch
Oracle Interim patch Installer location:/u01/app/oracle/product/11.2.3/db_1/OPatch
OPatch version command:/u01/app/oracle/product/11.2.3/db_1/OPatch/opatch version -oh "/u01/app/oracle/product/11.2.3/db_1".
OPatch version:11.2.0.3.3.

Starting execution of Prerequisites...
Total No of checks: 21
Performing check for CheckUserAdminPrivilege
Is the user ADMIN?
Check complete. The overall result of this check is: Passed

Check complete: Passed
========================================================
Performing check for CheckOracleHome
Is Oracle Home valid?
Check complete. The overall result of this check is: Passed

Check complete: Passed
========================================================
Performing check for CheckIfOHLockedForPatching
Is the Oracle Home locked?
Check complete. The overall result of this check is: Passed

Check complete: Passed
========================================================
Performing check for CheckJDK
Is JDK present in the Oracle Home?
Check complete. The overall result of this check is: Passed

Check complete: Passed
========================================================
Performing check for CheckOraInstLocation
Is given oraInst.loc file valid?
Check complete. The overall result of this check is: Passed

Check complete: Passed
========================================================
Performing check for CheckCentralInventoryLocation
Is Central Inventory location valid?
Check complete. T
he overall result of this check is: Not executed <<<<

Check complete: Not executed <<<<
OUI-15038: Unable to execute rule 'CheckCentralInventoryLocation' from ruleset 'OPatchChecks'. The underlying test 'oracle.opatch.opatchprereq.OPatchChecks.checkCentralInventoryLocation' has thrown an unhandled exception [null]. If the exception results in a failure of the test and the exception needs to be propagated, then encapsulate the exception in the result object and return the result.
Recommendation: Check if the oraInst.loc specified is correct.
========================================================
Performing check for CheckCentralInventoryForOH
Is ORACLE_HOME registered with the Central Inventory?
Check complete. The overall result of this check is: Not executed <<<<

Check complete: Not executed <<<<
OPatch failed to locate Central Inventory.
Possible causes are:
    The Central Inventory is corrupted
    The oraInst.loc file specified is not valid.
Recommendation: Check if the oraInst.loc specified is correct or use OUI AttachHome feature to attach the home to the Central Inventory.
========================================================

Check complete: Failed <<<<
Problem: Read-Write Session could not be created for this Central Inventory.
Recommendation: Check if there are proper permissions for the Central Inventory. Also, check if some other instance has locked the given Central Inventory.
]
Check complete. The overall result of this check is: Failed <<<<

So, clearly the issue was with the central inventory location. So I decided to point to central inventory for OPatch. and ran the check with following flag

#./opatch prereq CheckConflictAmongPatchesWithDetail -phBaseDir /home/oracle/14841409

This also failed with following error
OPatch failed to locate Central Inventory.
Possible causes are:
    The Central Inventory is corrupted
    The oraInst.loc file specified is not valid.
Recommendation: Check if the oraInst.loc specified is correct or use OUI AttachHome feature to attach the home to the Central Inventory

So something is amiss and decided to look into the inventory.xml. However, the inventory looks fine and still something not right. Now if you read the above error carefully it also tells you, how to fix it!!

Now I had two ways, possibly, to fix the problem.
1. Either to re-create the central inventoy (which looks preeety dicy to me as this was shared server with other application groups)
2. OR to simpley try to reattach the home and see if that fix the problem.

I decided to go with option 2 and again I have two choices to fix it.

1. Either run the attachHome script from Problem ORACLE_HOME 
OR
2.  run the following command to fix it.
Command -
./runInstaller -silent -invPtrLoc=/u01/app/oracle/product/11.2.3/db_1/oraInst.loc ORACLE_HOME="/u01/app/oracle/product/11.2.3/db_1" ORACLE_HOME_NAME="OraDb11g_home1"

However there is a problem with above command. To use it I need to have the name of ORACLE_HOME. Hence in the case where my inventory is corrupted, it is very difficult to find out the correct name for ORACLE_HOME.
So I simply decided to run the attachHome from ORACLE_HOME
#$ORACLE_HOME/oui/bin/attachHome.sh

And when I ran the opatch again the issue was resolved.

Wednesday, February 6, 2013

OEM Agent Not Starting Up On Oracle RAC After Restart

 Not sure what went wrong, but after installation of agent on RAC Cluster, the agent was crashing out.

Log shows...
emagent.trc 
FATAL_ERROR::500|ORA-20603: The timezone of the multiagent target
(ODB,rac_database)is not consistent with the timezone (America/Detroit) reported by other agents.
2013-02-04 19:01:49,073 Thread-1099983168 ERROR upload: number of fatal error exceeds the limit 3
2013-02-04 19:01:49,073 Thread-1099983168 ERROR upload: agent will shutdown now

emagent.log 
 Unsuccessful Upload attempts for XML file exceeds specified limit=3, Agent will shutdown (00851)
2013-02-04 18:47:57,213 Thread-1383669120 EMAgent abnormal terminating (00704
)

So far looks like something wrong with Time zone between OMS and agent. But upon reviewing following file, the thought immediately went away.

emdctl.log
 2013-02-05 04:18:11,186 Thread-2402549120 WARN  http: nmehl_connect_internal: connect failed to (hostb:3872): Connection refused (error = 111)
2013-02-05 04:18:11,186 Thread-2402549120 ERROR main: nmectla_agentctl: Error connecting to
https://hostb:3872/emd/main. Returning status code 1
2013-02-05 04:22:05,803 Thread-3880828288 WARN  http: nmehl_connect_internal: connect failed to (hostb:3872): Connection refused (error = 111)
2013-02-05 04:22:05,803 Thread-3880828288 ERROR main: nmectla_agentctl: Error connecting to
https://hostb:3872/emd/main. Returning status code 1
2013-02-05 04:23:06,873 Thread-1536675200 WARN  http: nmehl_connect_internal: connect failed to (hostb:3872): Connection refused (error = 111)
2013-02-05 04:23:06,874 Thread-1536675200 ERROR main: nmectla_agentctl: Error connecting to
https://hostb:3872/emd/main. Returning status code 1

So looks like the root.sh on hostb was not ran!!! So its the time to run it and see how it goes.

Latest after root.sh execution
2013-02-05 06:29:19,293 Thread-1116105024 ERROR pingManager: nmepm_pingReposURL: Cannot connect to https://oms:7799/em/upload/: retStatus=-1
2013-02-05 06:29:19,298 Thread-1116105024 WARN  http: nmehl_readAgentKey: File access failure
2013-02-05 06:29:19,298 Thread-1116105024 ERROR ssl: Open wallet failed, ret = 28759
2013-02-05 06:29:19,298 Thread-1116105024 ERROR ssl: nmehlenv_openWallet failed
2013-02-05 06:29:19,298 Thread-1116105024 ERROR ssl: Error initializing SSL
2013-02-05 06:29:19,298 Thread-1116105024 ERROR http: 11: Unable to initialize ssl connection with server, aborting connection attempt: ret -1

Upon careful review of this, I realized the agent was trying to upload data on 7799 port , which is my management port and upload port is 4900. So thats is there. I need to fix the port to upload it on 4900

On HostA Agent URL         : https://hosta:3872/emd/main
Repository URL    : https://oms:4900/em/upload

On HostB
Agent URL         :
http://hostb:3872/emd/main
Repository URL    : https://oms:7799/em/upload/

So one has to modify the $AGENT_HOME/sysman/config/emd.properties

There is an entry REPOSITORY_URL=https://oms:7799/em/upload

I have to modify this to reflect the port as follows
REPOSITORY_URL=
https://oms:4900/em/upload
Once done I have to clear out the pending upload files...
Files to remove -
$AGENT_HOME/sysman/emd/state/*
$AGENT_HOME/sysman/emd/upload/*

Once done do clearstate agent
$AGENT_HOME/bin/emctl clearstate agent

Try to start the agent now..
$AGENT_HOME/bin/emctl start agent a
nd now, the agent comes back up fine..