Friday, June 21, 2013

Do You Really Need oraclehomeproperties.xml file ??!!


Recently, when we were trying to apply a SPU Apr2013 patch to our Oracle Cluster Database, albeit thru OEM Deployment Procedure, during analyze step we hit the following error. The error was indicating that oraclehomeproperties.xml file was missing from the 2, out of 3 hosts missing. while digging further we figured out following..

Info about oraclehomeproperties.xml File --

    This file contains the details about the node list, the local node name, and the CRS flag for the Oracle Home.
    In a shared Oracle Home, the local node information is not present.
    This file also contains the following information:
    GUID — Unique global ID for the Oracle Home
    ARU ID — Unique platform ID. The patching and patchset application depends on this ID.
    ARU ID DESCRIPTION — Platform description

NOTE : The information in oraclehomeproperties.xml overrides the information in inventory.xml. This file is located under $ORACLE_HOME/inventory/ContentsXML

Other folders
Folder Name    Description
Scripts     Contains the scripts used for the cloning operation
ContentsXML     Contains the details of the components and libraries installed
Templates     Contains the template files used for cloning.
oneoffs     Contains the details of the one-off patches applied

I did check the other cluster to check the existance and the contents for the same file and found that all nodes in cluster DB home has this file and each one contains different GUID. hmmm.. So somehow the cluster I intend to patch doesn't contain this file. I spend lots of time on MOS to figure out how to recreate this one and couldn't find much info.

Suddenly, thought came across the mind and I decided to copy the file from one node to other remaining two nodes of the cluster and tried to apply the patch and would you believe it. My analyze step completed successfully.

So the lesson learned here is that oraclehomeproperties.xml file contains the info about GUID and ARU ID which informs patching process about platform and ID of the node.

Wednesday, June 12, 2013

Cloning Oracle 11g Agent On AIX/LINUX


I was, recently, trying to setup agent on some old Oracle legacy servers and run into issues of packages and utilities missing like, gzip, wget not present and wget present was not supporting https.
This were the hosts were nightmare as no monitoring was available and upgrading them with latest patches and technology levels were not ventured due to the risk of breaking them.

So DBAs have to find the way round on how to install agents on them and bring them under OEM umbrella. 

Normally, you can do the install of an agent using following well known methods.
1. Push
2. Pull
3. Using Setup / Silent Install
4. Clone

We decided to go for cloning as other options were not working for us. 
Following is the method one can use to clone agent.  

1. Identify the host where you have agent up and running, this will provide you source install.


2. Zip the Agent Oracle home that you want to clone (for example, agent.zip).
oracle@xxxxxxxx> zip -r agent11g.zip  ./agent11g/

3. Perform a file transfer (ftp,scp) of this zipped Oracle home onto the destination host where you want to install the cloned Agent (for example, ftp agent.zip).

4. In the destination host, unzip the Agent Oracle home (for example, unzip agent.zip).
oracle@xxxxxxxx> unzip  agent11g.zip

5.Remove all the files from the location $AGENT_HOME/sysman/emd/collection in the destination host

6. On the destination host, Go to $Agent_ORACLE_HOME/oui/bin/ directory and execute the following command: 
./runInstaller -clone -forceClone ORACLE_HOME=<full path of Oracle home> ORACLE_HOME_NAME=<Oracle home name> -noconfig -silent
Where ORACLE_HOME is the unzipped Agent Oracle home on the destination host


oracle@xxxxxxx /u01/app/oracle/agent11g/agent11g/oui/bin> ./runInstaller -clone -forceClone ORACLE_HOME=/u01/app/oracle/agent11g/agent11g ORACLE_HOME_NAME=agent11ghome1 -noconfig -silent
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 17343 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2013-06-06_11-42-34AM. Please wait ...oracle@hofdvorc02 /u01/app/oracle/agent11g/agent11g/oui/bin> Oracle Universal Installer, Version 11.1.0.8.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.

You can find the log of this install session at:
 /oracle/server/oraInventory/logs/cloneActions2013-06-06_11-42-34AM.log
.................................................................................................... 100% Done.


Installation in progress (Thursday, June 6, 2013 11:42:40 AM CDT)
............................................................     60% Done.
Install successful

Linking in progress (Thursday, June 6, 2013 11:42:43 AM CDT)
.                                                                61% Done.
Link successful

Setup in progress (Thursday, June 6, 2013 11:43:01 AM CDT)
.......................                                         100% Done.
Setup successful

End of install phases.(Thursday, June 6, 2013 11:43:03 AM CDT)
Starting to execute configuration assistants
The following configuration assistants have not been run. This can happen because Oracle Universal Installer was invoked with the -noConfig option.
--------------------------------------
The "/u01/app/oracle/agent11g/agent11g/cfgtoollogs/configToolFailedCommands" script contains all commands that failed, were skipped or were cancelled. This file may be used to run these configuration assistants outside of OUI. Note that you may have to update this script with passwords (if any) before executing the same.
The "/u01/app/oracle/agent11g/agent11g/cfgtoollogs/configToolAllCommands" script contains all commands to be executed by the configuration assistants. This file may be used to run the configuration assistants outside of OUI. Note that you may have to update this script with passwords (if any) before executing the same.

--------------------------------------
WARNING:
The following configuration scripts need to be executed as the "root" user.
#!/bin/sh
#Root script to run
/u01/app/oracle/agent11g/agent11g/root.sh
To execute the configuration scripts:
    1. Open a terminal window
    2. Log in as "root"
    3. Run the scripts

The cloning of agent11ghome1 was successful.

7. Once install is done, Execute the following script to run the Agent Configuration Assistant (agentca): 
$Agent_ORACLE_HOME/bin/agentca -f

oracle@xxxxxxx /u01/app/oracle/agent11g/agent11g/bin> ./agentca -f

Stopping the agent using /u01/app/oracle/agent11g/agent11g/bin/emctl  stop agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Running agentca using /u01/app/oracle/agent11g/agent11g/oui/bin/runConfig.sh ORACLE_HOME=/u01/app/oracle/agent11g/agent11g ACTION=Configure MODE=Perform RESPONSE_FILE=/u01/app/oracle/agent11g/agent11g/response_file RERUN=TRUE INV_PTR_LOC=/u01/app/oracle/agent11g/agent11g/oraInst.loc COMPONENT_XML={oracle.sysman.top.agent.10_2_0_1_0.xml}
Perform - mode is starting for action: Configure


Perform - mode finished for action: Configure

You can see the log file: /u01/app/oracle/agent11g/agent11g/cfgtoollogs/oui/configActions2013-06-06_11-45-26-AM.log

Stopping the agent using /u01/app/oracle/agent11g/agent11g/bin/emctl  stop agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Stopping agent ... stopped.

Running Agent Addon Configuration using /u01/app/oracle/agent11g/agent11g/perl/bin/perl /u01/app/oracle/agent11g/agent11g/sysman/install/AddonConfig.pl
Arguments passed

Configuring Addon from xml : oracle.sysman.plugin.virtualization.agent.11_1_0_1_0.xml

Running Command : /u01/app/oracle/agent11g/agent11g/oui/bin/runConfig.sh ORACLE_HOME=/u01/app/oracle/agent11g/agent11g ACTION=configure MODE=perform RERUN=true  RESPONSE_FILE=/u01/app/oracle/agent11g/agent11g/vt_responsefile COMPONENT_XML={oracle.sysman.plugin.virtualization.agent.11_1_0_1_0.xml}
 Setting the invPtrLoc to /u01/app/oracle/agent11g/agent11g/oraInst.loc

perform - mode is starting for action: configure

perform - mode finished for action: configure

You can see the log file: /u01/app/oracle/agent11g/agent11g/cfgtoollogs/oui/configActions2013-06-06_11-46-22-AM.log

Agent Addon Configuration done

Starting the agent using /u01/app/oracle/agent11g/agent11g/bin/emctl  start agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Agent is already running

oracle@xxxxxxx /u01/app/oracle/agent11g/agent11g/bin> ./emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 11.1.0.1.0
OMS Version       : 11.1.0.1.0
Protocol Version  : 11.1.0.0.0
Agent Home        : /u01/app/oracle/agent11g/agent11g
Agent binaries    : /u01/app/oracle/agent11g/agent11g
Agent Process ID  : 21163
Parent Process ID : 21118
Agent URL         : http://xxxxxxxxxxxxxxxxxxxxxx:3872/emd/main/
Repository URL    : https://xxxxxxxxxxxxxxxxxxxxxxxx:4900/em/upload/
Started at        : 2013-06-06 11:46:24
Started by user   : oracle
Last Reload       : 2013-06-06 11:46:24
Last successful upload                       : (none)
Last attempted upload                        : (none)
Total Megabytes of XML files uploaded so far :     0.00
Number of XML files pending upload           :       20
Size of XML files pending upload(MB)         :    20.04
Available disk space on upload filesystem    :    65.34%
Last attempted heartbeat to OMS              : 2013-06-06 11:46:27
Last successful heartbeat to OMS             : unknown
---------------------------------------------------------------
Agent is Running and Ready



8. Run the newly cloned Agents root.sh script 
$Agent_ORACLE_HOME/root.sh as root user
 root@xxxxxxx /u01/app/oracle/agent11g/agent11g> root.sh

Now one needs to try and test a upload from agent home to OEM. 



oracle@xxxxxxxxxxx /u01/app/oracle/agent11g/agent11g/bin> ./emctl upload agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload error: uploadXMLFiles skipped :: OMS version not checked yet. If this issue persists check trace files for ping to OMS related errors.

If you come across above errors, one has to clear the previous state of the agent. This happens when you have upload files pending in collections, upload or state directory. Also when you check status of the agent , try to look at lines highlighted in RED, that will tell you that something is not right. If the agent is configured correctly, it will show some data. 

Now lets try to fix the issue on hand.

oracle@xxxxx /u01/app/oracle/agent11g/agent11g/bin> ./emctl stop agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Stopping agent ... stopped.
oracle@xxxxxxx /u01/app/oracle/agent11g/agent11g/bin> ./emctl clearstate agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
EMD clearstate completed successfully
oracle@xxxxx /u01/app/oracle/agent11g/agent11g/bin> ./emctl secure agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Agent is already stopped...   Done.
Securing agent...   Started.
Enter Agent Registration Password :
Securing agent...   Successful.
oracle@xxxxxxx /u01/app/oracle/agent11g/agent11g/bin> ./emctl start agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Starting agent ..... started.
oracle@xxxxxx /u01/app/oracle/agent11g/agent11g/bin> ./emctl upload agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

Now if you check on OEM targets page, you should see the newly configured host available for monitoring.