Troubleshooting ASM Startup Issue in 11g Grid Infrastructure
In 11g when one places the ASM spfile, OCR or VoteDisk on ASM diskgroup, thus exists a tight interlink between these components. ASM plays very important role when starting grid infrastructure stack, let it single node or multi node i.e. RAC.
With Single Node Grid Infrastructure, the database using ASM instance has to register it self with CSS.
So when one cannot start ASM for some reason, the question will be how to trouble shoot the issue since everything is tightly integrated to each other. So first lets look at ASM startup sequence.
When Grid Infrastructure starts, Oracle will try to locate the ASM parameter file, to start CSSDAgent in following sequence.
- First it will look into GPNP Profile to find the parameter with name "asmdiskstring"
- Profile is usually located under <GRID_HOME>/gpnp/profiles/peer
- file - profile.xml
Here the issue can be if the above mentioned file is not found then ASM will fail to start. So make sure your profile reflects the correct value.
- If the above step doesn't reflect any value then, the next look up will be done in GRID_HOME/dbs folder and if located the file will start using pfile found.
There is a caveat here. what if the gpnp profile reflect the the proper file which exists on ASM but cannot be opened due to corruption. In this case again ASM will fail to start.
Solution to these issue will be to start ASM with transient parameter file as follows ( this is for 2 node RAC )
Use Case - On 2 node RAC only one node is healthy and another node is having problem with CRS stack start up.
1- Create a new ASM pfile
ora_+ASM1.ora
+ASM1.asm_diskgroups='DATA','FRA'#Manual Mount
+ASM2.asm_diskgroups='DATA','FRA'#Manual Mount
*.asm_diskstring='/dev/oracleasm/disks/*'
*.asm_power_limit=5
*.diagnostic_dest='//u01/app/11.2.0.3/grid/log'
*.instance_type='asm'
*.large_pool_size=12M
*.remote_login_passwordfile='EXCLUSIVE'
2- Start up the ASM instance+ASM1.asm_diskgroups='DATA','FRA'#Manual Mount
+ASM2.asm_diskgroups='DATA','FRA'#Manual Mount
*.asm_diskstring='/dev/oracleasm/disks/*'
*.asm_power_limit=5
*.diagnostic_dest='//u01/app/11.2.0.3/grid/log'
*.instance_type='asm'
*.large_pool_size=12M
*.remote_login_passwordfile='EXCLUSIVE'
on the first node
$ export $ORACLE_SID=<asm instance name>
$ export $ORACLE_HOME=<full path of the asm home>
$ sqlplus / as sysdba
sql> startup pfile=<the full pathname of ora_+ASM1.ora>
$ export $ORACLE_SID=<asm instance name>
$ export $ORACLE_HOME=<full path of the asm home>
$ sqlplus / as sysdba
sql> startup pfile=<the full pathname of ora_+ASM1.ora>
This will start ASM Instance on node 1.
3) Recreate the spfile
SQL> create spfile='+DATA' from pfile='/u01/app/11.2.0.3/grid/dbs/init+ASM1.ora'; File created.
SQL> sho parameter spfile;
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string +DATA/<host>/asmparameterfile
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string +DATA/<host>/asmparameterfile
/registry.253.768413123
Note - Pls make sure that you check your gpnp profile reflecting its new value.
4) Restarted the OHAS/CRS stack on node
Connect as root user:
Connect as root user:
# /u01/app/11.2.0.3/grid/bin/crsctl stop crs
Make sure that all the processes are exited and then start crs
# /u01/app/11.2.0.3/grid/bin/crsctl start crs
After some time do the health check on crs stack that it is started successfully.
# /u01/app/11.2.0.3/grid/bin/crsctl check cluster -all
