Unable To mount ACFS after Reboot


Built a stand alone grid infrastructure and added an ACFS file system.

However, following a reboot, I was unable to start the acfs cluster.

Eventually, found this document on Metalink.

 Unable To mount ACFS after Reboot On Standalone GI [SIHA] (Doc ID 2348133.1)

 Standalone setup does not start ACFS automatically, so create rc.local script to load the ACFS drivers well before the OHASD service starts. ACFS drivers should get loaded before ASM starts.

 To prove this worked, I ran a shutdown on ASM, started acfs i.e. "acfsload start" and then started ASM and et voila. Need to add the "acfsload start" to the startup scripts otherwise will need to do this following every reboot.

[oracle@localhost ~]$ sqlplus / as sysasm
SQL*Plus: Release 12.2.0.1.0 Production on Tue Jan 22 04:19:45 2019
Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
SQL> shutdown abort
ASM instance shutdown
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

If the server is using HA i.e. RAC One Node or Oracle restart you could run a :

crstl stop has 

 then :

[oracle@localhost ~]$ lsmod | grep oracle
oracleasm              59173  1

--- No acfs.

[oracle@localhost ~]$ su
Password:

[root@localhost oracle]# . oraenv
ORACLE_SID = [+ASM] ?
The Oracle base remains unchanged with value /u01/app

[root@localhost oracle]# acfsload start

ACFS-9391: Checking for existing ADVM/ACFS installation.
ACFS-9392: Validating ADVM/ACFS installation files for operating system.
ACFS-9393: Verifying ASM Administrator setup.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9154: Loading 'oracleoks.ko' driver.
ACFS-9154: Loading 'oracleadvm.ko' driver.
ACFS-9154: Loading 'oracleacfs.ko' driver.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
ACFS-9156: Detecting control device '/dev/ofsctl'.
ACFS-9322: completed

Now :

[root@node01 ~]# lsmod | grep oracle

oracleacfs              4815623  1
oracleadvm            785504  7
oracleoks               638647  2 oracleacfs,oracleadvm
oracleasm              59173  1

Restart ASM or "crsctl start has" if the server is HA enabled.

[oracle@localhost ~]$ sqlplus / as sysasm

SQL*Plus: Release 12.2.0.1.0 Production on Tue Jan 22 04:21:26 2019
Copyright (c) 1982, 2016, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> startup;
ASM instance started
Total System Global Area 1140850688 bytes
Fixed Size                  8629704 bytes
Variable Size            1107055160 bytes
ASM Cache                  25165824 bytes
ASM diskgroups mounted
ASM diskgroups volume enabled

SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

Check diskgroups are ok via asmcmd, if not run a "volenable --all" to ensure the volumes are enabled.

ASMCMD> volinfo --all
Diskgroup Name: NFPSADWGGDATA
         Volume Name: NFPSADWGG
         Volume Device: /dev/asm/nfpsadwgg-236
         State: ENABLED
         Size (MB): 306176
         Resize Unit (MB): 512
         Redundancy: UNPROT
         Stripe Columns: 8
         Stripe Width (K): 1024
         Usage: ACFS
         Mountpath: /acfsmounts
Using this information :

mount -t acfs /dev/asm/nfpsadwgg-236 /acfsmounts

 Check successfully mounted

$ df -h | grep acfs

Filesystem                                                   Size  Used Avail Use% Mounted on
/dev/asm/nfpsadwgg-236                            299G  4.6G  295G   2% /acfsmounts


Check anything that is using acfs is working e.g. Golden Gate.

See also this metalink doc when acfs not running on OCI cluster :

OCI Backup issue - DCS-10001:Internal error encountered: Mount point:/opt/oracle/dcs/commonstore is not mounted or does not exist (Doc ID 2513385.1)

No comments:

Flashback database

I don't really like using flashback, if given the choice would use pluggable databases and clone the pdb prior to doing any work and if you hit a problem clone the pdb back to it's original state (but keep that for another day).

Here are the steps for flashback of a database. The DB needs to be in achieve log mode and forced logging enabled. Verify that the flash recovery area is setup and large enough for the upgrade.

NB - This process is slightly different if using DataGuard please see - 

https://www.thegeekdiary.com/how-to-create-and-drop-guaranteed-restore-point-in-oracle-data-guard/


You don’t need to enable flashback on the whole database, but I did it anyway. The restore point needs to be guaranteed.

Verify Flashback status, size and location:

select flashback_on, LOG_MODE, FORCE_LOGGING, status from v$database, v$instance;

show parameter DB_RECOVERY

show parameter FLASHBACK


Steps before the upgrade:

ALTER DATABASE FLASHBACK ON;

CREATE RESTORE POINT before_upgrade GUARANTEE FLASHBACK DATABASE;

Steps after the upgrade:

DROP RESTORE POINT before_upgrade;

ALTER DATABASE FLASHBACK OFF;



Steps if need to flashback the Database:

su – oracle

sqlplus / as sysdba;


select current_scn from v$database;

shutdown immediate;

startup mount;

select * from v$restore_point;

flashback database to restore point before_upgrade;

alter database open resetlogs;

Check flashback space usage:

select * from v$flash_recovery_area_usage;

select name, time, GUARANTEE_FLASHBACK_DATABASE, STORAGE_SIZE from v$restore_point;

More detailed instructions (when running an application upgrade and you hit problems) :

Instruction Steps
Task Steps
Reference/Screenshots
1.     
Verify Flashback Recovery status, size and location:

If db_recovery_file_dest and db_recovery_file_dest_size is blank, add them before proceeding.
select flashback_on, log_mode, force_logging, status from v$database, v$instance;
show parameter db_recovery;
show parameter flashback;
2.     


3.     
Steps before the Application upgrade:
Note: Save value for current_scn to compare if recovery is performed
alter database flashback on;
select current_scn from v$database;
create restore point "before_upgrade" guarantee flashback database;
4.     


5.     
Steps after the Application upgrade:
Note: Only perform these steps if upgrade has been tested and declared successful
drop restore point "before_upgrade";
alter database flashback off;
6.     


7.     
If Application upgrade was not successful and a roll back is required, perform these steps to roll back database changes:



Note: Compare SCN from step 3 to ensure flashback was successful. SCN’s may not match, but should be close in range

Replace db_name with your RAC database name
For single database instance:
sqlplus / as sysdba;
shutdown immediate;
startup mount;
flashback database to restore point "before_upgrade";
alter database open resetlogs;
select current_scn from v$database;

For multi-node RAC instances:
srvctl stop database -db db_name
export ORACLE_SID=db_name_1
sqlplus / as sysdba;
startup mount;
flashback database to restore point "before_upgrade";
alter database open resetlogs;
select current_scn from v$database;
shutdown immediate;
quit;
srvctl start database -db db_name
8.     


9.     
Check flashback space usage:
select * from v$flash_recovery_area_usage;
select name, time, guarantee_flashback_database, storage_size from v$restore_point;
10.   







No comments:

Removing node from RAC Cluster















 **Host names etc changed.

Stop all Process running on the Database home     




srvctl stop instance -d rpap_gdc2 -n prsodb02

srvctl disable  instance -d rpap_gdc2 -n prsodb02



RELOCATE ALL SERVICES HAVING NODE 2 AS PREFERRED OR AVAILABLE


srvctl modify  service -s customer_pdb.scc.local -d rpap_gdc2  -modifyconfig  -preferred "rpap1"

srvctl modify  service -s lms_pdb.scc.local -d rpap_gdc2  -modifyconfig  -preferred "rpap1"

srvctl modify  service -s osmm.scc.local -d rpap_gdc2  -modifyconfig  -preferred "rpap1"

srvctl modify  service -s pdb1p_pdb.scc.local -d rpap_gdc2  -modifyconfig  -preferred "rpap1"

srvctl modify  service -s batch_service -d rpap_gdc2  -modifyconfig  -preferred "rpap1"



Ensure that the instance to be removed from the database is neither a preferred nor an available instance of any service



srvctl config service –s  customer_pdb.scc.local -d rpap_gdc2 

srvctl config service –s lms_pdb.scc.local -d rpap_gdc2 

srvctl config service –s  s osmm.scc.local -d rpap_gdc2 

srvctl config service –s pdb1p_pdb.scc.local -d rpap_gdc2 

srvctl config service –s s batch_service -d rpap_gdc2 





DELETE THE DATABASE INSTANCE USING DBCA SILENT MODE


dbca -silent -responseFile deleterpap2.rsp




Verify that the dropped instance's redo thread has been removed by using SQL*Plus on an existing node to query the GV$LOG view. If the redo thread is not disabled, then disable the thread. For example:

 ALTER DATABASE DISABLE THREAD 2;

Verify that the instance has been removed from OCR by running the following command, where db_unique_name is the database unique name for your Oracle RAC database:

 srvctl config database -d rpap_gdc2



UPDATE INVENTORY ON prsodb02 TO REFLECT NODE TO BE DELETED


Run the following command from $ORACLE_HOME/oui/bin on the node that you are deleting to update the inventory on that node:

 ./runInstaller -updateNodeList ORACLE_HOME= /u01/app/oracle/product/12.1.0.1/dbhome_1 "CLUSTER_NODES={prsodb02}" -local



DEINSTALL ORACLE HOME ON prsodb02


Deinstall the Oracle home—only if the Oracle home is not shared—from the node that you are deleting by running the following command from the Oracle_home\deinstall directory:

 deinstall -local



UPDATE THE INVENTORY ON prsodb01 To REFLECT REMAINING NODE


 


Run the following command from the $ORACLE_HOME/oui/bin directory on any one of the remaining nodes in the cluster to update the inventories of those nodes, specifying a comma-delimited list of remaining node names and the name of the local node:

  ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/oracle/product/12.1.0.1/dbhome_1

"CLUSTER_NODES={ prsodb01}"  LOCAL_NODE= prsodb01







 


REMOVE GRID HOME




Stop and DISABLE THE listener process


srvctl stop listener -l listener  -n prsodb02

 srvctl disable listener -l listener  -n prsodb02

UPDATE INVENTORY ON prsodb02 TO REFLECT NODE TO BE DELETED


On the node you want to delete, run the following command as the user that installed Oracle Clusterware from the Grid_home/oui/bin directory where node_to_be_deleted is the name of the node that you are deleting:

  ./runInstaller -updateNodeList ORACLE_HOME= /u01/app/12.1.0.1/grid "CLUSTER_NODES={

prsodb02}" CRS=TRUE -silent -local



DEINSTALL GRID HOME ON prsodb02


For a local home, deinstall the Oracle Clusterware home from the node that you want to delete, as follows, by running the following command, where Grid_home is the path defined for the Oracle Clusterware home:

  Grid_home/deinstall/deinstall –local



UPDATE THE INVENTORY ON prsodb01 To REFLECT REMAINING NODE




On any node other than the node you are deleting (except for a Leaf Node in an Oracle Flex Cluster), run the following command from the Grid_home/oui/bin directory where remaining_nodes_list is a comma-delimited list of the nodes that are going to remain part of your cluster:

./runInstaller -updateNodeList ORACLE_HOME== /u01/app/12.1.0.1/grid  "CLUSTER_NODES={ prsodb01}" CRS=TRUE -silent



REMOVE DELETED NODE FROM OCR


From any node that you are not deleting, run the following command from the Grid_home/bin directory as root to delete the node from the cluster:

  crsctl delete node -n prsodb02

Run the following CVU command to verify that the specified nodes have been successfully deleted from the cluster:

  cluvfy stage -post nodedel -n prsodb01 -verbose



If you remove a cluster node on which Oracle Clusterware is down, then determine whether the VIP for the deleted node still exists, as follows:

 srvctl config vip -node prsodb02



If the VIP still exists, then delete it, as follows:

  srvctl stop vip -node prsodb02

  srvctl remove vip -node prsodb02
No comments:
Subscribe to: Posts (Atom)