Stopping a node in Oracle RAC - 12c


In an emergency situation where you have very little time to shut down a RAC cluster.
“crsctl stop has” would bring the node down but this does a shutdown abort and can cause issues – especially on a production environment, although oracle does not give any warnings, so best avoided.

So for each node on RAC that needs to be closed - stop Oracle applications like Enterprise Manager and Golden Gate (outside scope of this doc). Note on a multi node RAC env, for example, there maybe GG processes on two nodes but not on the other nodes.

Please check for acfs file systems - outside of scope of this doc.

+++++++++++++++++++++

Run crsctl stat res -t  – copy output, make sure at the end the same command outputs the same.

You will need to set the environment depending on the step.

After each step check status.

++++++++++++++++++++

emctl stop agent

++++++++++++++++++++

Oracle user – RDBMS env.

srvctl stop listener -n node1

++++++++++++++++++++

The instance shutdown depends on how the instance is managed – we are automatic.


[oracle@server01 ~]$ srvctl config database -db oradb -all

Database unique name: abp

…..

Management policy: AUTOMATIC


So – we are using policy managed so :

$ srvctl stop instance -db oradb -node node1 (replace db and node name as approp.)

Stop ALL instances on node you want to stop.

++++++++++++++++++

ASM env. i.e. Grid

When instances are down you can then stop ASM

srvctl stop asm -n node1

++++++++++++++++++

srvctl stop nodeapps -node1 -f

++++++++++++++++++

In this command all CRS related process will be stopped. This is the only command which needs to be executed by "root" user on all database nodes.

crsctl stop crs

+++++++++++++++++

Check cluster down :

crsctl stat res -t

+++++++++++++++++

Startup – this in reverse but using start.

++++++++++++++++++++

More here :




**NB**

Disable autostart of CRS in order to be able to perform server-maintenance and CRS is NOT to be restarted after a reboot of the server (as user root)

Root] # . oraenv

ORACLE_SID = [+ASM] ? +ASM1
The Oracle base has been set to /u01/app/oracle

# crsctl disable crs
CRS-4621: Oracle High Availability Services autostart is disabled.


Further notes  --

Create blackout on OEM and stop relevant apps – outside scope of doc.

Run a ‘ps -ef | grep smon’ to check what instances are running.

Take output from ‘crsctl stat res -t’

Set environment to appropriate oracle home and begin shutdown of node.

[oracle@server01 ~]$ srvctl stop listener -n server01

[oracle@server01 ~]$ srvctl stop service -db oradb -node server01

[oracle@server01 ~]$ srvctl stop instance -db oradb -node server01

[oracle@server01 ~]$ srvctl stop service -db ods -node server01

[oracle@server01 ~]$ srvctl stop instance -db ods -node server01

[oracle@server01 ~]$ srvctl stop asm -n server01 -f

[oracle@server01 ~]$ srvctl stop listener -listener ASMNET1LSNR_ASM -node server01

[oracle@server01 ~]$ srvctl stop nodeapps -node server01 -f

PRCR-1014 : Failed to stop resource ora.net1.network

PRCR-1065 : Failed to stop resource ora.net1.network

CRS-2670: Unable to start/relocate 'ora.net1.network' because 'ora.qosmserver' has a stop-time 'hard' dependency on it

CRS-0245:  User doesn't have enough privilege to perform the operation

--will run further investigation on this.

Logout

So up the priv to root for this node – need to be root to stop crs anyway.

[root@server01 ~]# . oraenv

ORACLE_SID = [root] ? +ASM1

The Oracle base has been set to /u01/app/grid

[root@server01 ~]# srvctl stop nodeapps -node server01 -f

PRCC-1017 : ons was already stopped on server01

PRCR-1005 : Resource ora.ons is already stopped

[root@server01 ~]# crsctl stat res -t | more

--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       usnympmtaora03b          STABLE
               OFFLINE OFFLINE      server01          STABLE
……etc

ora.oradb.ec.svc
      1        ONLINE  ONLINE       usnympmtaora03b          STABLE
      3        OFFLINE OFFLINE                               STABLE
      4        ONLINE  ONLINE       usnyspmtaora02b          STABLE
ora.oradb.frm.svc

      1        ONLINE  ONLINE       usnyspmtaora02b          STABLE


[root@server01 ~]# crsctl stop crs

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'server01'
CRS-2673: Attempting to stop 'ora.crsd' on 'server01'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'server01'
CRS-2673: Attempting to stop 'ora.chad' on 'server01'
CRS-2673: Attempting to stop 'ora.ods_gg_data.odsggdata.acfs' on 'server01'
CRS-2673: Attempting to stop 'ora.umb_gg_data.umbggdata.acfs' on 'server01'
CRS-2677: Stop of 'ora.ods_gg_data.odsggdata.acfs' on 'server01' succeeded
CRS-2673: Attempting to stop 'ora.ODS_GG_DATA.ODSGGDATA.advm' on 'server01'
CRS-2677: Stop of 'ora.ODS_GG_DATA.ODSGGDATA.advm' on 'server01' succeeded
CRS-2677: Stop of 'ora.umb_gg_data.umbggdata.acfs' on 'server01' succeeded
CRS-2673: Attempting to stop 'ora.UMB_GG_DATA.GGDATA.advm' on 'server01'
CRS-2677: Stop of 'ora.UMB_GG_DATA.GGDATA.advm' on 'server01' succeeded
CRS-2673: Attempting to stop 'ora.proxy_advm' on 'server01'
CRS-2677: Stop of 'ora.chad' on 'server01' succeeded
CRS-2677: Stop of 'ora.proxy_advm' on 'server01' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'server01' has completed
CRS-2677: Stop of 'ora.crsd' on 'server01' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'server01'
CRS-2673: Attempting to stop 'ora.crf' on 'server01'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'server01'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'server01'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'server01'
CRS-2677: Stop of 'ora.drivers.acfs' on 'server01' succeeded
CRS-2677: Stop of 'ora.crf' on 'server01' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'server01' succeeded
CRS-2677: Stop of 'ora.storage' on 'server01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'server01'
CRS-2677: Stop of 'ora.asm' on 'server01' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'server01'
CRS-2677: Stop of 'ora.mdnsd' on 'server01' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'server01' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'server01'
CRS-2673: Attempting to stop 'ora.evmd' on 'server01'
CRS-2677: Stop of 'ora.ctssd' on 'server01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'server01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'server01'
CRS-2677: Stop of 'ora.cssd' on 'server01' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'server01'
CRS-2677: Stop of 'ora.gipcd' on 'server01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'server01' has completed

CRS-4133: Oracle High Availability Services has been stopped.

[root@server01 ~]# crsctl stat res -t

CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.

Down as expected.

STARTUP…

Some of the startup is ran when we start nodeapps but for completeness.

[root@server01 ~]# crsctl start crs

. oraenv to one of the databases…

[oracle@server01 ~]$ srvctl start nodeapps -node server01

[oracle@server01 ~]$ crsctl stat res -t

[oracle@server01 ~]$ srvctl start asm -n server01

[oracle@server01 ~]$ srvctl start listener -listener ASMNET1LSNR_ASM -node server01

[oracle@server01 ~]$ srvctl start instance -db oradb -node server01

[oracle@server01 ~]$ srvctl status database -db oradb -v

[oracle@server01 ~]$ srvctl start instance -db ods -node server01

[oracle@server01 ~]$ srvctl status database -db ods -v

[oracle@server01 ~]$ srvctl start listener -n server01

[oracle@server01 ~]$ emctl status agent

[oracle@server01 ~]$ emctl start agent

[oracle@server01 ~]$ crsctl stat res -t (check everything running)

[oracle@server01 ~]$ crsctl stat res -t | grep 04x | wc -l (compare with start)



Grid / Oracle 19c - on one Node.

Notes re Oracle grid / rdbms installation on one node.

The oracle-base website is excellent - taken that as a template to work from :


Download Software
Download the Oracle software from OTN or MOS depending on your support status.
Hosts File
The "/etc/hosts" file must contain a fully qualified name for the server.
<IP-address>  <fully-qualified-machine-name>  <machine-name>
For example.
127.0.0.1       localhost localhost.localdomain localhost4 localhost4.localdomain4
192.168.56.107  ol7-19.localdomain  ol7-19
Set the correct hostname in the "/etc/hostname" file.
ol7-19.localdomain
Oracle Installation Prerequisites
Perform either the Automatic Setup or the Manual Setup to complete the basic prerequisites. The Additional Setup is required for all installations.
Automatic Setup
If you plan to use the "oracle-database-preinstall-19c" package to perform all your prerequisite setup, issue the following command.
# yum install -y oracle-database-preinstall-19c
It is probably worth doing a full update as well, but this is not strictly speaking necessary.
# yum update -y
It's worth running the all the YUM commands listed in the manual setup section. Depending on the OS package groups you have selected, some additional packages might also be needed.
If you are using RHEL7 or CentOS7, you can pick up the PRM from the OL7 repository and install it. It will pull the dependencies from your normal repositories.
# yum install -y https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/oracle-database-preinstall-19c-1.0-1.el7.x86_64.rpm

Manual Setup
If you have not used the "oracle-database-preinstall-19c" package to perform all prerequisites, you will need to manually perform the following setup tasks.
"I would check these file anyway just to confirm - otherwise the installer will complain; make sure the server is built with enough swap space too"
cd /mnt/resource or cd /mnt
ls
swapfile is the name of the Swap File
Enable Swap File for use:
sudo swapon -f swapfile
Run free-m to see Memory and Swap Space statistics
Swap: Should show some space allocated under total space

sudo swapon /swapfile


If no swapfile in /mnt/resource then create file "sudo fallocate -l 1G swapfile"
mk swap file: "mkswap /swapfile"

When complete if not part of Linux build disable SELinux and Firewalld - otherwise possible issues when you try to connect using sqlplus.

Add the following lines to the "/etc/sysctl.conf" file, or in a file called "/etc/sysctl.d/98-oracle.conf".
fs.file-max = 6815744
kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
kernel.panic_on_oops = 1
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500


Run one of the following commands to change the current kernel parameters, depending on which file you edited.
/sbin/sysctl -p
# Or
/sbin/sysctl -p /etc/sysctl.d/98-oracle.conf
Add the following lines to a file called "/etc/security/limits.d/oracle-database-preinstall-19c.conf" file.
oracle   soft   nofile    1024
oracle   hard   nofile    65536
oracle   soft   nproc    16384
oracle   hard   nproc    16384
oracle   soft   stack    10240
oracle   hard   stack    32768
oracle   hard   memlock    134217728
oracle   soft   memlock    134217728

Someone in the comments suggested you might need to add the previous lines into the "/etc/security/limits.conf" file also for CentOS7. This is definitely not needed for OL7, but worth considering if the installer gives prerequisite failures for these settings.

yum install -y oracle-database-preinstall-19c
yum update -y

yum install -y bc   
yum install -y binutils
yum install -y compat-libcap1
yum install -y compat-libstdc++-33
#yum install -y dtrace-modules
#yum install -y dtrace-modules-headers
#yum install -y dtrace-modules-provider-headers
yum install -y dtrace-utils
yum install -y elfutils-libelf
yum install -y elfutils-libelf-devel
yum install -y fontconfig-devel
yum install -y glibc
yum install -y glibc-devel
yum install -y ksh
yum install -y libaio
yum install -y libaio-devel
yum install -y libdtrace-ctf-devel
yum install -y libXrender
yum install -y libXrender-devel
yum install -y libX11
yum install -y libXau
yum install -y libXi
yum install -y libXtst
yum install -y libgcc
yum install -y librdmacm-devel
yum install -y libstdc++
yum install -y libstdc++-devel
yum install -y libxcb
yum install -y make
yum install -y net-tools # Clusterware
yum install -y nfs-utils # ACFS
yum install -y python # ACFS
yum install -y python-configshell # ACFS
yum install -y python-rtslib # ACFS
yum install -y python-six # ACFS
yum install -y targetcli # ACFS
yum install -y smartmontools
yum install -y sysstat

# Added by me.
yum install -y unixODBC
yum install xorg-x11-xauth xorg-x11-fonts-* xorg-x11-font-utils xorg-x11-fonts-Type1

--handy to check if display working ok

yum install xclock

Change password for Oracle

[root@server01 ~]# passwd oracle
Changing password for user oracle.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.


Add grid user manually

[root@server01 ~]# useradd -u 54322 -g oinstall -G dba,oper grid

Change password for user grid.

[root@server01 ~]# passwd grid

New password:
Retype new password:
passwd: all authentication tokens updated successfully.

Copy s/w onto server :

[root@server01 media]# pwd
/u99/media

[root@server01 media]# ls -la
total 8460848
drwxr-xr-x. 2 oracle oinstall       4096 Feb 26 05:13 .
drwxr-xr-x. 4 root   root           4096 Feb 26 05:10 ..
-rw-r--r--. 1 oracle oinstall  556240981 Feb  3 02:16 191004_fbo_ggs_Linux_x64_shiphome.zip
-rw-r--r--. 1 oracle oinstall 3059705302 Jan 29 03:01 LINUX.X64_193000_db_home.zip
-rw-r--r--. 1 oracle oinstall 2889184573 Jan 29 03:54 LINUX.X64_193000_grid_home.zip
-rw-r--r--. 1 oracle oinstall 2043088041 Jan 29 06:06 p30464035_190000_Linux-x86-64.zip
-rw-r--r--. 1 oracle oinstall  115653541 Jan 29 05:51 p6880880_190000_Linux-x86-64.zip


Make grid home directory (as grid) and copy the grid installation zip file into there:

/u01/app/19.0.0/gridhome

cd to grid home directory
cp /u99/media/LINUX.X64_193000_grid_home.zip .

  unzip LINUX.X64_193000_grid_home.zip


Now that the system is registered with ULN, it can install the software.
Log in as root and run:



# yum install oracleasm-support oracleasmlib oracleasm-`uname -r`

[root@server01 etc]# oracleasm configure –i

Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.
Default user to own the driver interface []: grid
Default group to own the driver interface []: oinstall
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done


[root@server01 etc]# oracleasm init
Creating /dev/oracleasm mount point: /dev/oracleasm
Loading module "oracleasm": oracleasm
Configuring "oracleasm" to use device physical block size
Mounting ASMlib driver filesystem: /dev/oracleasm
[root@server01 etc]# oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes


Check to see what disk(s) available for ASM via “pvdisplay” command and “fdisk –l” (or check with your infrastructure team)

[root@server01 dev]# fdisk /dev/sdf

Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x843d9ee5.

The device presents a logical sector size that is smaller than
the physical sector size. Aligning to a physical sector (or optimal
I/O) size boundary is recommended, or performance may be impacted.

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-2145386495, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-2145386495, default 2145386495):
Using default value 2145386495
Partition 1 of type Linux and of size 1023 GiB is set

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.


Add oracleasm diskgroup – name is arbitrary e.g. DATA1 or DATA001

[root@server01 dev]# oracleasm createdisk DATA1 /dev/sdf1

Writing disk header: done
Instantiating disk: done

[root@server01 dev]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...

[root@server01 dev]# oracleasm listdisks
DATA1

[root@server01 rpm]# pwd
/u01/app/19.0.0/gridhome/cv/rpm

[root@server01 rpm]# export CVUQDISK_GRP=oinstall
[root@server01 rpm]# rpm -iv cvuqdisk-1.0.10-1.rpm
Preparing packages...
cvuqdisk-1.0.10-1.x86_64

Create /u01/app/oraInventory directory with full owner and group permission as grid and oracle need to see it i.e chmod 770 on that directory.

Run the gridsetup from the grid home to install grid

--insert graphic here


Create /u01/app/oraInventory directory with full owner and group permission as grid and oracle need to see it i.e chmod 770 on that directory.

Once complete check clustering working ok.

[grid@server01 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       server01           STABLE
ora.LISTENER.lsnr
               ONLINE  INTERMEDIATE server01           Not All Endpoints Re
                                                             gistered,STABLE
ora.asm
               ONLINE  ONLINE       server01           Started,STABLE
ora.ons
               OFFLINE OFFLINE      server01           STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       server01           STABLE
ora.diskmon
      1        OFFLINE OFFLINE                               STABLE
ora.evmd
      1        ONLINE  ONLINE       server01           STABLE
-------------------------------------------------------------------------------

Install Oracle DB s/w

# Unzip software.

cd /path/to/temp_area e.g. /u99/media/db_install/

unzip -oq /path/to/software/LINUX.X64_193000_db_home.zip

Once installation is complete you can delete this directory.

# Interactive mode.

./runInstaller

Just do a s/w only install.

Run the scripts when prompted.

As a root user, execute the following script(s):

        1. /u01/app/oraInventory/orainstRoot.sh
        2. /u01/app/oracle/product/19.0.0/dbhome_1/root.sh

Once the s/w is on the server, run the dbca command and build database(s) as appropriate.

Run the latest patches :

Apply latest OPatch patch to both grid and oracle home.

  
[root@uszw2scc2ora01 OPatch]# ./opatchauto apply /u99/media/30501910

OPatchauto session is initiated at Thu Feb 27 05:41:55 2020

System initialization log file is /u01/app/19.0.0/gridhome/cfgtoollogs/opatchautodb/systemconfig2020-02-27_05-41-58AM.log.

Session log file is /u01/app/19.0.0/gridhome/cfgtoollogs/opatchauto/opatchauto2020-02-27_05-42-04AM.log
The id for this session is GJR1

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.0.0/dbhome_1
Patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/dbhome_1


Verifying SQL patch applicability on home /u01/app/oracle/product/19.0.0/dbhome_1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.0.0/dbhome_1


Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.0.0/gridhome
Patch applicability verified successfully on home /u01/app/19.0.0/gridhome


Preparing to bring down database service on home /u01/app/oracle/product/19.0.0/dbhome_1
Successfully prepared home /u01/app/oracle/product/19.0.0/dbhome_1 to bring down database service


Bringing down database service on home /u01/app/oracle/product/19.0.0/dbhome_1
Following database has been stopped and will be restarted later during the session: clipapp
Database service successfully brought down on home /u01/app/oracle/product/19.0.0/dbhome_1


Bringing down CRS service on home /u01/app/19.0.0/gridhome
Prepatch operation log file location: /u01/app/grid/crsdata/uszw2scc2ora01/crsconfig/hapatch_2020-02-27_05-43-26AM.log
CRS service brought down successfully on home /u01/app/19.0.0/gridhome


Start applying binary patch on home /u01/app/oracle/product/19.0.0/dbhome_1
Binary patch applied successfully on home /u01/app/oracle/product/19.0.0/dbhome_1


Start applying binary patch on home /u01/app/19.0.0/gridhome
Binary patch applied successfully on home /u01/app/19.0.0/gridhome


Starting CRS service on home /u01/app/19.0.0/gridhome
Postpatch operation log file location: /u01/app/grid/crsdata/uszw2scc2ora01/crsconfig/hapatch_2020-02-27_05-56-01AM.log
CRS service started successfully on home /u01/app/19.0.0/gridhome


Starting database service on home /u01/app/oracle/product/19.0.0/dbhome_1
Database service successfully started on home /u01/app/oracle/product/19.0.0/dbhome_1


Preparing home /u01/app/oracle/product/19.0.0/dbhome_1 after database service restarted
No step execution required.........


Trying to apply SQL patch on home /u01/app/oracle/product/19.0.0/dbhome_1
SQL patch applied successfully on home /u01/app/oracle/product/19.0.0/dbhome_1

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:uszw2scc2ora01
SIDB Home:/u01/app/oracle/product/19.0.0/dbhome_1
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /u99/media/30501910/30489632
Reason: This patch is not applicable to this specified target type - "oracle_database"

Patch: /u99/media/30501910/30655595
Reason: This patch is not applicable to this specified target type - "oracle_database"


==Following patches were SUCCESSFULLY applied:

Patch: /u99/media/30501910/30489227
Log: /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-02-27_05-43-58AM_1.log

Patch: /u99/media/30501910/30557433
Log: /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-02-27_05-43-58AM_1.log


Host:uszw2scc2ora01
SIHA Home:/u01/app/19.0.0/gridhome
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /u99/media/30501910/30489227
Log: /u01/app/19.0.0/gridhome/cfgtoollogs/opatchauto/core/opatch/opatch2020-02-27_05-49-02AM_1.log

Patch: /u99/media/30501910/30489632
Log: /u01/app/19.0.0/gridhome/cfgtoollogs/opatchauto/core/opatch/opatch2020-02-27_05-49-02AM_1.log

Patch: /u99/media/30501910/30557433
Log: /u01/app/19.0.0/gridhome/cfgtoollogs/opatchauto/core/opatch/opatch2020-02-27_05-49-02AM_1.log

Patch: /u99/media/30501910/30655595
Log: /u01/app/19.0.0/gridhome/cfgtoollogs/opatchauto/core/opatch/opatch2020-02-27_05-49-02AM_1.log



OPatchauto session completed at Thu Feb 27 06:02:25 2020
Time taken to complete the session 20 minutes, 30 seconds