DB_RECOVERY_FILE_DEST out ot space - PRCR-1079 / CRS-5017 / ORA-03113


Strange error on starting one of our databases today - why oracle can't just say the recovery file destination is 100% full I have no idea but anyway :


[oracle@ora01 ~]$ srvctl start database -db oradb
PRCR-1079 : Failed to start resource ora.cng.db
CRS-5017: The resource action "ora.oradb.db start" encountered the following error:
ORA-03113: end-of-file on communication channel
Process ID: 129203
Session ID: 259 Serial number: 59108
. For details refer to "(:CLSN00107:)" in "/u01/app/oracle/product/12.2.0/diag/crs/ora01/crs/trace/ohasd_oraagent_oracle.trc".

CRS-2674: Start of 'ora.oradb.db' on 'ora01' failed

From the alert log, trace files hit 100% on the recovery file dest size - more info from Mr Burleson.


http://www.dba-oracle.com/p_db_recovery_file_dest_size.htm

As this was RAC one node used srvctl to mount the database.

[oracle@ora01 ~]$ srvctl start database -db oradb -o mount

[oracle@ora01 ~]$ . oraenv


ORACLE_SID = [] ? oradb


The Oracle base has been changed from /u01/app/oracle/product/12.2.0 to /u01/app/oracle
[

oracle@ora01 ~]$ sqlplus / as sysdba

SQL> show parameter db_recovery_file_dest_size;


NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest_size           big integer 8192M


SQL> alter system set db_recovery_file_dest_size = 16192M;

System altered.

SQL> alter database open;
Database altered.

Script from the Burleson website :

SQL>
col name     format a32
col size_mb  format 999,999,999
col used_mb  format 999,999,999
col pct_used format 999

select
name,
   ceil( space_limit / 1024 / 1024) size_mb,
   ceil( space_used / 1024 / 1024) used_mb,
   decode( nvl( space_used, 0),0, 0,
   ceil ( ( space_used / space_limit) * 100) ) pct_used
from
    v$recovery_file_dest
order by
   name desc;



NAME                                  SIZE_MB      USED_MB PCT_USED
-------------------------------- ------------ ------------ --------
+DATA                                  16,192        7,729       48


Just need to ensure the archive logs are cleared down so this does not happen again.

https://www.durham.computer/2019/04/a-very-quick-script-to-clear-down-rman.html


**NB**

Unless the "Archive Destination" is set to use the db_recovery_file_dest the archiver will use +DATA until it it is exhausted and won't max out at 12G; however the recovery area will still be used by the control files etc (see below) and can still reach 100%


SQL> archive log list
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            +DATA
Oldest online log sequence     9750
Next log sequence to archive   9751
Current log sequence           9752


SQL> show parameter db_reco

 NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest                string      +DATA
db_recovery_file_dest_size           big integer 12G



SQL> l
  1* SELECT * FROM V$RECOVERY_AREA_USAGE

FILE_TYPE PERCENT_SPACE_USED PERCENT_SPACE_RECLAIMABLE NUMBER_OF_FILES  CON_ID
------    ------------ ------------------------- --------------- ----------
CONTROL FILE                           .13       0               1          0
REDO LOG                              4.91       0               3          0
ARCHIVED LOG                             0       0               0          0
BACKUP PIECE                           .09       0               1          0
IMAGE COPY                               0       0               0          0
FLASHBACK LOG                            0       0               0          0
FOREIGN ARCHIVED LOG                     0       0               0          0
AUXILIARY DATAFILE COPY                  0       0               0          0

8 rows selected.


SQL>/ (script above)

NAME                                  SIZE_MB      USED_MB PCT_USED
-------------------------------- ------------ ------------ --------
+DATA                                  12,288          632        6







The ss utility to monitor network connections

The Linux ss command has proved very useful, it is used to dump socket statistics.

Something I have only come across very recently.

Shows information similar to netstat but can display more TCP and state information than other tools.

https://www.tecmint.com/ss-command-examples-in-linux/

SQL Server - quick notes.

I am currently being asked to look after more and more SQL Server boxes so with a couple of hours spare thought I would build SQL Server on Linux via Oracle Virtual Box (might one day try VM Workstation). I had already built SQL Server on Linux ages ago and had no issues but can only think newer versions of Red Hat have changed somewhat - thought it would be really easy 😕

However, after talking to the team, the general opinion was that SQL Server was more suited to Windows and that would be the better option. Anyway, after fighting Red Hat for what seemed an age (disabled SE Linux and IPV6 thinking that was the issue) - SQL Server just would not play.

Configure SELINUX=disabled in the /etc/selinux/config file: # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # ...
Reboot your system. After reboot, confirm that the getenforce command returns Disabled.

Disable IPV6
# sysctl -w net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.all.disable_ipv6 = 1
# sysctl -w net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6 = 1


Firewalld needs to be disabled too.

[root@usnyspmtaoem01 ~]# systemctl stop firewalld
[root@usnyspmtaoem01 ~]# systemctl status firewalld


SQL Server was happy enough to start up but I had no joy with getting connected - can only think this was an issue with ipv6 but never found the reason, I reckon there is some bug issue that needs to be fixed that I would love to get to the bottom of but just don't have the time. Eventually gave up and went down the Windows road and downloaded Windows Server 2019 and an evaluation copy of SQL Server 2017.



Relatively straight forward, but when trying to connect hit a few problems. Disabled the Windows Firewall (would not normally do this but machine already behind a firewall anyway, should have just created a rule to "punch through" on 1433).  Also, had to enable "SQL Server and Windows Authentication mode" as the server was only set up with Windows Authentication and not applicable for my test environment. Even then had to enable the SA account as this is disabled by default for security -  everything working

Just need to build another server and get high availability working now - more to follow.


SA user enabled
Enable SQL Server and Windows Authenticaion mode