In an emergency situation where you have very little time to shut
down a RAC cluster.
“crsctl stop has” would bring
the node down but this does a shutdown abort and can cause issues – especially
on a production environment, although oracle does not give any warnings, so
best avoided.
So for each node on RAC that needs to be closed - stop Oracle
applications like Enterprise Manager and Golden Gate (outside scope of this
doc). Note on a multi node RAC env, for example, there maybe GG processes on two nodes but not
on the other nodes.
Please check for acfs file systems - outside of scope of this doc.
+++++++++++++++++++++
Run crsctl stat res -t – copy output, make sure at the end the same
command outputs the same.
You will need to set the environment depending on the step.
After
each step check status.
++++++++++++++++++++
emctl stop agent
++++++++++++++++++++
Oracle
user – RDBMS env.
srvctl stop listener -n node1
++++++++++++++++++++
The
instance shutdown depends on how the instance is managed – we are automatic.
[oracle@server01 ~]$ srvctl config database -db oradb -all
Database unique name: abp
…..
Management policy: AUTOMATIC
So –
we are using policy managed so :
$ srvctl
stop instance -db oradb -node node1 (replace db and node name as
approp.)
Stop
ALL instances on node you want to stop.
++++++++++++++++++
ASM
env. i.e. Grid
When
instances are down you can then stop ASM
srvctl stop asm -n node1
++++++++++++++++++
srvctl stop nodeapps -node1 -f
++++++++++++++++++
In
this command all CRS related process will be stopped. This is the only command
which needs to be executed by "root" user on all database
nodes.
crsctl stop crs
+++++++++++++++++
Check
cluster down :
crsctl stat res -t
+++++++++++++++++
Startup
– this in reverse but using start.
++++++++++++++++++++
More
here :
**NB**
Disable autostart of CRS in order to be able to perform server-maintenance and CRS is NOT to be restarted after a reboot of the server (as user root)
Root]
# . oraenv
ORACLE_SID = [+ASM] ? +ASM1
The Oracle base has been set to /u01/app/oracle
# crsctl disable crs
CRS-4621: Oracle High Availability Services autostart is disabled.
Further
notes --
Create
blackout on OEM and stop relevant apps – outside scope of doc.
Run
a ‘ps -ef | grep smon’ to check what instances are running.
Take
output from ‘crsctl stat res -t’
Set
environment to appropriate oracle home and begin shutdown of node.
[oracle@server01 ~]$ srvctl stop listener -n server01
[oracle@server01 ~]$ srvctl stop service -db oradb -node server01
[oracle@server01 ~]$ srvctl stop instance -db oradb -node server01
[oracle@server01 ~]$ srvctl stop service -db ods -node server01
[oracle@server01 ~]$ srvctl stop instance -db ods -node server01
[oracle@server01 ~]$ srvctl stop asm -n server01 -f
[oracle@server01 ~]$ srvctl stop listener -listener ASMNET1LSNR_ASM -node server01
[oracle@server01 ~]$ srvctl stop nodeapps -node server01 -f
PRCR-1014 : Failed to stop resource ora.net1.network
PRCR-1065 : Failed to stop resource ora.net1.network
CRS-2670: Unable to start/relocate 'ora.net1.network' because
'ora.qosmserver' has a stop-time 'hard' dependency on it
CRS-0245: User doesn't have enough privilege to perform the
operation
--will run further investigation on this.
Logout
So up the priv to root for this node – need to be root to stop crs
anyway.
[root@server01 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/grid
[root@server01 ~]# srvctl stop nodeapps -node server01 -f
PRCC-1017 : ons was already stopped on server01
PRCR-1005 : Resource ora.ons is already stopped
[root@server01 ~]# crsctl stat res -t | more
--------------------------------------------------------------------------------
Name
Target State
Server
State details--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE usnympmtaora03b STABLE
OFFLINE OFFLINE server01 STABLE
……etc
ora.oradb.ec.svc
1
ONLINE ONLINE
usnympmtaora03b STABLE3 OFFLINE OFFLINE STABLE
4 ONLINE ONLINE usnyspmtaora02b STABLE
ora.oradb.frm.svc
1 ONLINE
ONLINE
usnyspmtaora02b STABLE
[root@server01 ~]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability
Services-managed resources on 'server01'
CRS-2673: Attempting to stop 'ora.crsd' on 'server01'CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'server01'
CRS-2673: Attempting to stop 'ora.chad' on 'server01'
CRS-2673: Attempting to stop 'ora.ods_gg_data.odsggdata.acfs' on 'server01'
CRS-2673: Attempting to stop 'ora.umb_gg_data.umbggdata.acfs' on 'server01'
CRS-2677: Stop of 'ora.ods_gg_data.odsggdata.acfs' on 'server01' succeeded
CRS-2673: Attempting to stop 'ora.ODS_GG_DATA.ODSGGDATA.advm' on 'server01'
CRS-2677: Stop of 'ora.ODS_GG_DATA.ODSGGDATA.advm' on 'server01' succeeded
CRS-2677: Stop of 'ora.umb_gg_data.umbggdata.acfs' on 'server01' succeeded
CRS-2673: Attempting to stop 'ora.UMB_GG_DATA.GGDATA.advm' on 'server01'
CRS-2677: Stop of 'ora.UMB_GG_DATA.GGDATA.advm' on 'server01' succeeded
CRS-2673: Attempting to stop 'ora.proxy_advm' on 'server01'
CRS-2677: Stop of 'ora.chad' on 'server01' succeeded
CRS-2677: Stop of 'ora.proxy_advm' on 'server01' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'server01' has completed
CRS-2677: Stop of 'ora.crsd' on 'server01' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'server01'
CRS-2673: Attempting to stop 'ora.crf' on 'server01'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'server01'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'server01'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'server01'
CRS-2677: Stop of 'ora.drivers.acfs' on 'server01' succeeded
CRS-2677: Stop of 'ora.crf' on 'server01' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'server01' succeeded
CRS-2677: Stop of 'ora.storage' on 'server01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'server01'
CRS-2677: Stop of 'ora.asm' on 'server01' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'server01'
CRS-2677: Stop of 'ora.mdnsd' on 'server01' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'server01' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'server01'
CRS-2673: Attempting to stop 'ora.evmd' on 'server01'
CRS-2677: Stop of 'ora.ctssd' on 'server01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'server01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'server01'
CRS-2677: Stop of 'ora.cssd' on 'server01' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'server01'
CRS-2677: Stop of 'ora.gipcd' on 'server01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'server01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@server01 ~]# crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
Down as expected.
STARTUP…
Some of the startup is ran when we start nodeapps but for completeness.
[root@server01 ~]# crsctl start crs
. oraenv to one of the databases…
[oracle@server01 ~]$ srvctl start nodeapps -node server01
[oracle@server01 ~]$ crsctl stat res -t
[oracle@server01 ~]$ srvctl start asm -n server01
[oracle@server01 ~]$ srvctl start listener -listener ASMNET1LSNR_ASM -node server01
[oracle@server01 ~]$ srvctl start instance -db oradb -node server01
[oracle@server01 ~]$ srvctl status database -db oradb -v
[oracle@server01 ~]$ srvctl start instance -db ods -node server01
[oracle@server01 ~]$ srvctl status database -db ods -v
[oracle@server01 ~]$ srvctl start listener -n server01
[oracle@server01 ~]$ emctl status agent
[oracle@server01 ~]$ emctl start agent
[oracle@server01 ~]$ crsctl stat res -t (check everything running)
[oracle@server01 ~]$ crsctl stat res -t | grep 04x | wc -l (compare with start)
No comments:
Post a Comment