Restarting the Cluster
Emergency Restart
Use the following method when something has gone wrong with your RAC enironment, and the decision has been taken to restart everything (i.e. database, ASM, the whole cluster) ASAP.
Down
Log onto node 03 as root (03 is used as an example).
Run standard checks beforehand:
/u01/app/12.1.0/grid/bin/crsctl check cluster
/u01/app/12.1.0/grid/bin/crsctl status resource –t
Next, bring down the cluster as follows:
If you want to shut down the cluster on both nodes, run the following:
/u01/app/12.1.0/grid/bin/crsctl stop cluster –all –f
If you want to shut down the cluster on a single node (03, in this example), run the following:
/u01/app/12.1.0/grid/bin/crsctl stop cluster -n db03 -f
(NOTE: we are using the ‘force’ option because of the fact that there is something wrong with the environment at present, and so need to bring it down ASAP)
Confirm everything has come down by running standard checks again:
/u01/app/12.1.0/grid/bin/crsctl check cluster –all
/u01/app/12.1.0/grid/bin/crsctl status resource –t
Doesn’t hurt to confirm DB and ASM are definitely down:
(especially if we are restarting the cluster because of issues)
ps –ef | grep smon
You can also bring down HA (High Availability) at this point:
# crsctl stop has
NOTE: this command will stop HA on the local node only.
Therefore - depending on your situation (you might be doing the restart only on one node, for example) – you will need to log onto node 04 to stop it there.
Up
Start everything back up again:
If you took down HA, then simply restarting this will bring up the rest of the cluster:
# crsctl start has
If you didn’t take down HA, then run this command to bring up the clusterware:
/u01/app/12.1.0/grid/bin/crsctl start cluster –all
Confirm everything has come up again:
crsctl check cluster –all
crsctl status resource –t
Alternative Commands
NOTE: You can also use the following commands to restart CRS, which achieve the same result as the above “crsctl start/stop cluster” commands:
crsctl check crs
crsctl stop crs
crsctl start crs
Controlled Restart
If you are taking down the cluster as part of some maintenance activity (i.e. not an emergency scenario), then it’s better to take the cluster components down using srvctl commands as described in this section.
From [1] – taking the whole cluster down in one go using CRSCTL as in the previous section "can lead to the database instances being stopped similar to shutdown abort, which requires an instance recovery on startup. If you use SRVCTL to stop the database instances manually before stopping the cluster, then you can prevent a shutdown abort, but this requires that you manually restart the database instances after restarting Oracle Clusterware."
Therefore, it is better to use the method below:
Down
Database
Log onto node 03 as user ‘oracle’.
srvctl status database -db ism_dbname
srvctl stop database -db ism_dbname
srvctl status database -db ism_dbname
Alternatively, you can do this one node at a time (e.g. if one node is already down, as we have currently with rpap2)
srvctl status instance -db ism_dbname –instance rpap1
srvctl stop instance -db ism_dbname –instance rpap1
srvctl status instance -db ism_dbname –instance rpap1
-- only if you’re doing both nodes:
srvctl status instance -db ism_dbname –instance rpap2
srvctl stop instance -db ism_dbname –instance rpap2
srvctl status instance -db ism_dbname –instance rpap2
ASM
Log onto node 03 as user ‘grid’.
srvctl status asm -n db03
srvctl stop asm -n db03
srvctl status asm -n db03
-- only if you’re doing both nodes:
srvctl status asm -n db04
srvctl stop asm -n db04
srvctl status asm -n db04
Nodeapps
Still logged on as ‘grid’:
srvctl status nodeapps -node db03
srvctl stop nodeapps -node db03
srvctl status nodeapps -node db03
-- only if you’re doing both nodes:
-- only if you’re doing both nodes:
srvctl status nodeapps -node db04
srvctl stop nodeapps -node db04
srvctl status nodeapps -node db04
CRS
Log onto node 03 as ‘root’.
/u01/app/12.1.0/grid/bin/crsctl check cluster –all
/u01/app/12.1.0/grid/bin/crsctl status resource –t
If you are taking down the clusterware on both nodes:
/u01/app/12.1.0/grid/bin/crsctl stop cluster –all
If you are taking down the clusterware on one node only:
/u01/app/12.1.0/grid/bin/crsctl stop cluster -n db03
Check everything is now down.
/u01/app/12.1.0/grid/bin/crsctl check cluster –all
/u01/app/12.1.0/grid/bin/crsctl status resource –t
HA
You can also bring down HA (High Availability) at this point:
# crsctl stop has
NOTE: this command will stop HA on the local node only.
Therefore - depending on your situation (you might be doing the restart only on one node, for example) – you will need to log onto node 04 to stop it there.
Up
In order to bring everything up again, follow sthe steps in the “Emergency Restart” (Up) section. The steps will be the same.
Reference
Oracle® Real Application Clusters
Administration and Deployment Guide
12c Release 1 (12.1)
E48838-09
Additional - shutting down one node.
ps -ef | grep pmon
This will show the instances running on this machine/node
srvctl status database -db datadb -v
This will show the status of the database named and this instances associated with that database and which server said instance is running on.
srvctl status instance -db datadb -v
This will show the status of the instances running on the named database and any services associated with that instance
This will show the status of the instances running on the named database and any services associated with that instance
srvctl stop service -db ods -service ods_1 -instance ods1
This will stop the service named ods_1 associated with instance ods1 which uses database ods
srvctl stop service -db datadb -service prodcat, datadb_1 -instance datadb1
srvctl stop service -db datadb -service prodcat, datadb_1 -instance datadb1
NB - you may want to disable 'has' e.g. for patching, otherwise the services will come back up following a reboot.
This will stop the list of services linked to instance datadb1 which uses databases datadb
