Introduction
This post details the steps carried to add three disks to a DATA diskgroup in ASM.
The three disks were 500Gb each in size (approx).
After the RFC / CAB was raised, ‘presented’ the disks to the two database servers.
The next steps were to add the new storage to ASM, thereby making them
available to Oracle.
available to Oracle.
IMPORTANT:
Most of the steps here can be carried out by logging onto the database server as ‘oracle’,
followed by ‘sqlplus / as sysdba’
However, the step of actually adding the disks needs to be done using
‘sqlplus / as sysasm’.
Therefore – for consistency, and to keep things simple – it is advisable to carry out all of
the steps in this document by logging onto the DB server as the ‘grid’ user, and then onto
the ASM instance using sqlplus / as sysasm
Step 1 – check the space currently in ASM
Most of the steps here can be carried out by logging onto the database server as ‘oracle’,
followed by ‘sqlplus / as sysdba’
However, the step of actually adding the disks needs to be done using
‘sqlplus / as sysasm’.
Therefore – for consistency, and to keep things simple – it is advisable to carry out all of
the steps in this document by logging onto the DB server as the ‘grid’ user, and then onto
the ASM instance using sqlplus / as sysasm
Step 1 – check the space currently in ASM
Log onto DB server (either one) as grid.
sqlplus / as sysasm
set lines 120
col name format a25
col group_number format 999 heading GROUP
col name format a20
select GROUP_NUMBER
,NAME
,ROUND(TOTAL_MB/1024,1) TOTAL_GB
,ROUND(((TOTAL_MB - FREE_MB)/1024),1) USED_GB
,ROUND(FREE_MB/1024,1) FREE_GB
,ROUND((100-(FREE_MB/TOTAL_MB)*100),2) "% Used"
from V$ASM_DISKGROUP
order by group_number
e.g.
SQL> /
GROUP NAME TOTAL_GB USED_GB FREE_GB % Used
----- -------------------- ---------- ---------- ---------- ----------
1 DATA 3366.7 2679.4 687.3 79.59
2 FRA 959.8 473.3 486.5 49.32
SQL>
Step 2 – Check that the disks have been presented to the OS:
db01 /dev/asmdisks > pwd
/dev/asmdisks
cdepld2prsodb01 /dev/asmdisks (rpap1) > ls -l
total 0
brw-rw----. 1 grid asmadmin 253, 8 Mar 11 14:43 asm1
brw-rw----. 1 grid asmadmin 253, 6 Mar 11 14:43 asm10
brw-rw----. 1 grid asmadmin 253, 32 Mar 11 14:43 asm11
brw-rw----. 1 grid asmadmin 253, 31 Mar 11 14:39 asm12
brw-rw----. 1 grid asmadmin 253, 34 Mar 11 14:39 asm13
brw-rw----. 1 grid asmadmin 253, 30 Mar 11 14:39 asm14
brw-rw----. 1 grid asmadmin 253, 9 Mar 11 14:43 asm2
brw-rw----. 1 grid asmadmin 253, 7 Mar 11 14:43 asm3
brw-rw----. 1 grid asmadmin 253, 11 Mar 11 14:43 asm4
brw-rw----. 1 grid asmadmin 253, 10 Mar 11 14:43 asm5
brw-rw----. 1 grid asmadmin 253, 12 Mar 11 14:43 asm6
brw-rw----. 1 grid asmadmin 253, 3 Mar 11 14:43 asm7
brw-rw----. 1 grid asmadmin 253, 5 Mar 11 14:43 asm8
brw-rw----. 1 grid asmadmin 253, 4 Mar 11 14:43 asm9
cdepld2prsodb01 /dev/asmdisks (rpap1) >
Make sure the new ones have the same ownership and permissions as the existing ones.
→there used to be 11, now there are14.
Step 3 – Check which disks ASM can actually see
Set lines 200 pages 999
col disk_group format a15
col DISK_NAME format a15
col path format a50
select g.name disk_group
,d.DISK_NUMBER
,d.name disk_name
,d.PATH
,d.TOTAL_MB
,d.header_status
from v$asm_disk d,
v$asm_diskgroup g
where d.GROUP_NUMBER = g.GROUP_NUMBER (+)
order by g.name, d.disk_number
1 select g.name disk_group
2 ,d.DISK_NUMBER
3 ,d.name disk_name
4 ,d.PATH
5 ,d.TOTAL_MB
6 ,d.header_status
7 from v$asm_disk d,
8 v$asm_diskgroup g
9 where d.GROUP_NUMBER = g.GROUP_NUMBER (+)
10* order by g.name, d.disk_number
SQL>
SQL>
SQL> /
DISK_GROUP DISK_NUMBER DISK_NAME PATH TOTAL_MB HEADER_STATU
--------------- ----------- --------------- ------------------------- ---------- ------------
DATA 0 DATA_0000 /dev/asmdisks/asm1 492500 MEMBER
DATA 1 DATA_0001 /dev/asmdisks/asm2 492500 MEMBER
DATA 2 DATA_0002 /dev/asmdisks/asm3 492500 MEMBER
DATA 3 DATA_0003 /dev/asmdisks/asm4 492500 MEMBER
DATA 4 DATA_0004 /dev/asmdisks/asm5 492500 MEMBER
DATA 5 DATA_0005 /dev/asmdisks/asm6 492500 MEMBER
DATA 6 DATA_0006 /dev/asmdisks/asm11 492500 MEMBER
FRA 0 FRA_0000 /dev/asmdisks/asm10 245700 MEMBER
FRA 1 FRA_0001 /dev/asmdisks/asm7 245700 MEMBER
FRA 2 FRA_0002 /dev/asmdisks/asm8 245700 MEMBER
FRA 3 FRA_0003 /dev/asmdisks/asm9 245700 MEMBER
0 /dev/asmdisks/asm12 0 CANDIDATE
1 /dev/asmdisks/asm14 0 CANDIDATE
2 /dev/asmdisks/asm13 0 CANDIDATE
14 rows selected.
SQL>
→the highlighted ones are the new ones that have not been added to a diskgroup yet.
Step 4 – Add the first disk
SQL> l
1 ALTER DISKGROUP DATA
2 ADD DISK '/dev/asmdisks/asm12'
3* NAME DATA_0007 rebalance power 0
SQL> /
Diskgroup altered.
SQL>
Step 5 – Run the two queries again:
SQL> /
GROUP NAME TOTAL_GB USED_GB FREE_GB % Used
----- -------------------- ---------- ---------- ---------- ----------
1 DATA 3847.7 2680.8 1166.9 69.67
2 FRA 959.8 475.1 484.7 49.5
SQL>
And:
SQL> /
DISK_GROUP DISK_NUMBER DISK_NAME PATH TOTAL_MB HEADER_STATU
--------------- ----------- --------------- ------------------------- ---------- ------------
DATA 0 DATA_0000 /dev/asmdisks/asm1 492500 MEMBER
DATA 1 DATA_0001 /dev/asmdisks/asm2 492500 MEMBER
DATA 2 DATA_0002 /dev/asmdisks/asm3 492500 MEMBER
DATA 3 DATA_0003 /dev/asmdisks/asm4 492500 MEMBER
DATA 4 DATA_0004 /dev/asmdisks/asm5 492500 MEMBER
DATA 5 DATA_0005 /dev/asmdisks/asm6 492500 MEMBER
DATA 6 DATA_0006 /dev/asmdisks/asm11 492500 MEMBER
DATA 7 DATA_0007 /dev/asmdisks/asm12 492500 MEMBER
FRA 0 FRA_0000 /dev/asmdisks/asm10 245700 MEMBER
FRA 1 FRA_0001 /dev/asmdisks/asm7 245700 MEMBER
FRA 2 FRA_0002 /dev/asmdisks/asm8 245700 MEMBER
FRA 3 FRA_0003 /dev/asmdisks/asm9 245700 MEMBER
0 /dev/asmdisks/asm14 0 CANDIDATE
1 /dev/asmdisks/asm13 0 CANDIDATE
14 rows selected.
SQL>
Step 6 – Check ASM alert log
/u01/app/grid/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log
Check no errors.
e.g.
Wed Mar 11 14:57:30 2015
SQL> ALTER DISKGROUP DATA
ADD DISK '/dev/asmdisks/asm12'
NAME DATA_0007 rebalance power 0
Wed Mar 11 14:57:30 2015
NOTE: GroupBlock outside rolling migration privileged region
NOTE: Assigning number (1,7) to disk (/dev/asmdisks/asm12)
NOTE: requesting all-instance membership refresh for group=1
NOTE: Disk 7 in group 1 is assigned fgnum=8
NOTE: initializing header (replicated) on grp 1 disk DATA_0007
NOTE: initializing header on grp 1 disk DATA_0007
NOTE: requesting all-instance disk validation for group=1
Wed Mar 11 14:57:32 2015
NOTE: skipping rediscovery for group 1/0xc6785ea9 (DATA) on local instance.
Wed Mar 11 14:57:32 2015
NOTE: requesting all-instance disk validation for group=1
Wed Mar 11 14:57:32 2015
NOTE: skipping rediscovery for group 1/0xc6785ea9 (DATA) on local instance.
Wed Mar 11 14:57:32 2015
GMON updating for reconfiguration, group 1 at 12 for pid 39, osid 6241
Wed Mar 11 14:57:32 2015
NOTE: group 1 PST updated.
Wed Mar 11 14:57:32 2015
NOTE: membership refresh pending for group 1/0xc6785ea9 (DATA)
Wed Mar 11 14:57:32 2015
GMON querying group 1 at 13 for pid 21, osid 7028
NOTE: cache opening disk 7 of grp 1: DATA_0007 path:/dev/asmdisks/asm12
GMON querying group 1 at 14 for pid 21, osid 7028
Wed Mar 11 14:57:34 2015
SUCCESS: refreshed membership for 1/0xc6785ea9 (DATA)
Wed Mar 11 14:57:34 2015
SUCCESS: ALTER DISKGROUP DATA
ADD DISK '/dev/asmdisks/asm12'
NAME DATA_0007 rebalance power 0
NOTE: Attempting voting file refresh on diskgroup DATA
Step 7 – check DB alert log
Again, check for any errors:
e.g.
Wed Mar 11 14:57:34 2015
NOTE: Assigning number (1,7) to disk (/dev/asmdisks/asm12)
SUCCESS: disk DATA_0007 (7.3915951733) added to diskgroup DATA
Wed Mar 11 14:58:23 2015
Step 8 – check if rebalancing is taking place:
SQL> select * from v$asm_operation ;
GROUP OPERA PASS STAT POWER ACTUAL SOFAR EST_WORK
EST_RATE EST_MINUTES
EST_RATE EST_MINUTES
----- ----- --------- ---- ---------- ---------- ---------- ---------- ---------- -----------
ERROR_CODE CON_ID
-------------------------------------------- ----------
1 REBAL REBALANCE WAIT 0
0
1 REBAL COMPACT WAIT 0
0
SQL>
→the above is correct, since we have ‘disabled’ automatic rebalancing by using the ‘power 0’ syntax.
Step 9 – add the remaining disks
SQL> l
1 ALTER DISKGROUP DATA
2 ADD DISK '/dev/asmdisks/asm13'
3* NAME DATA_0008 rebalance power 0
SQL> /
Diskgroup altered.
SQL>
SQL> l
1 ALTER DISKGROUP DATA
2 ADD DISK '/dev/asmdisks/asm14'
3* NAME DATA_0009 rebalance power 0
SQL> /
Diskgroup altered.
SQL>
SQL> /
GROUP NAME TOTAL_GB USED_GB FREE_GB % Used
----- -------------------- ---------- ---------- ---------- ----------
1 DATA 4809.6 2680.8 2128.8 55.74
2 FRA 959.8 475.9 483.8 49.59
SQL>
SQL> /
DISK_GROUP DISK_NUMBER DISK_NAME PATH TOTAL_MB HEADER_STATU
--------------- ----------- --------------- ------------------------- ---------- ------------
DATA 0 DATA_0000 /dev/asmdisks/asm1 492500 MEMBER
DATA 1 DATA_0001 /dev/asmdisks/asm2 492500 MEMBER
DATA 2 DATA_0002 /dev/asmdisks/asm3 492500 MEMBER
DATA 3 DATA_0003 /dev/asmdisks/asm4 492500 MEMBER
DATA 4 DATA_0004 /dev/asmdisks/asm5 492500 MEMBER
DATA 5 DATA_0005 /dev/asmdisks/asm6 492500 MEMBER
DATA 6 DATA_0006 /dev/asmdisks/asm11 492500 MEMBER
DATA 7 DATA_0007 /dev/asmdisks/asm12 492500 MEMBER
DATA 8 DATA_0008 /dev/asmdisks/asm13 492500 MEMBER
DATA 9 DATA_0009 /dev/asmdisks/asm14 492500 MEMBER
FRA 0 FRA_0000 /dev/asmdisks/asm10 245700 MEMBER
FRA 1 FRA_0001 /dev/asmdisks/asm7 245700 MEMBER
FRA 2 FRA_0002 /dev/asmdisks/asm8 245700 MEMBER
FRA 3 FRA_0003 /dev/asmdisks/asm9 245700 MEMBER
14 rows selected.
SQL>
→all looks good. ASM alert log:
No errors:
DB Alert log:
Also no errors:
Wed Mar 11 15:07:33 2015
NOTE: Assigning number (1,8) to disk (/dev/asmdisks/asm13)
SUCCESS: disk DATA_0008 (8.3915951736) added to diskgroup DATA
Wed Mar 11 15:08:53 2015
NOTE: Assigning number (1,9) to disk (/dev/asmdisks/asm14)
SUCCESS: disk DATA_0009 (9.3915951738) added to diskgroup DATA
Wed Mar 11 15:10:03 2015
Step 10 – Run queries on Node 2
Should return the same results as on Node 1:
SQL> !uname -a
Linux db02.scc.local 2.6.32-431.11.2.el6.x86_64 #1 SMP Mon Mar 3 13:32:45 EST 2014 x86_64 x86_64 x86_64 GNU/Linux
SQL> /
GROUP NAME TOTAL_GB USED_GB FREE_GB % Used
----- -------------------- ---------- ---------- ---------- ----------
1 DATA 4809.6 2681.3 2128.3 55.75
2 FRA 959.8 476.5 483.2 49.65
SQL>
And:
SQL> !uname -a
Linux db02.scc.local 2.6.32-431.11.2.el6.x86_64 #1 SMP Mon Mar 3 13:32:45 EST 2014 x86_64 x86_64 x86_64 GNU/Linux
SQL> /
DISK_GROUP DISK_NUMBER DISK_NAME PATH TOTAL_MB HEADER_STATU
--------------- ----------- --------------- ------------------------- ---------- ------------
DATA 0 DATA_0000 /dev/asmdisks/asm1 492500 MEMBER
DATA 1 DATA_0001 /dev/asmdisks/asm2 492500 MEMBER
DATA 2 DATA_0002 /dev/asmdisks/asm3 492500 MEMBER
DATA 3 DATA_0003 /dev/asmdisks/asm4 492500 MEMBER
DATA 4 DATA_0004 /dev/asmdisks/asm5 492500 MEMBER
DATA 5 DATA_0005 /dev/asmdisks/asm6 492500 MEMBER
DATA 6 DATA_0006 /dev/asmdisks/asm11 492500 MEMBER
DATA 7 DATA_0007 /dev/asmdisks/asm12 492500 MEMBER
DATA 8 DATA_0008 /dev/asmdisks/asm13 492500 MEMBER
DATA 9 DATA_0009 /dev/asmdisks/asm14 492500 MEMBER
FRA 0 FRA_0000 /dev/asmdisks/asm10 245700 MEMBER
FRA 1 FRA_0001 /dev/asmdisks/asm7 245700 MEMBER
FRA 2 FRA_0002 /dev/asmdisks/asm8 245700 MEMBER
FRA 3 FRA_0003 /dev/asmdisks/asm9 245700 MEMBER
14 rows selected.
SQL>
Step 11
Now that we have added all the disks, we can invoke a disk rebalance manually.
Current situation shows that the disks are imbalanced in this diskgroup:
SQL> select DISK_NUMBER
2 ,ROUND(TOTAL_MB/1024,2) TOTAL_GB
3 ,ROUND(FREE_MB/1024,2) FREE_GB
4 ,ROUND((TOTAL_MB - FREE_MB)/1024,2) USED_GB
5 ,ROUND(((TOTAL_MB - FREE_MB)/TOTAL_MB)*100,0) PERCENT_USED
6 from v$asm_disk
7 where GROUP_NUMBER = &1
8 order by 1
9
SQL>
SQL> /
Enter value for 1: 1
old 7: where GROUP_NUMBER = &1
new 7: where GROUP_NUMBER = 1
DISK_NUMBER TOTAL_GB FREE_GB USED_GB PERCENT_USED
----------- ---------- ---------- ---------- ------------
0 480.96 .16 480.8 100
1 480.96 0 480.96 100
2 480.96 .04 480.91 100
3 480.96 .07 480.89 100
4 480.96 .17 480.79 100
5 480.96 .05 480.91 100
6 480.96 414.49 66.47 14
7 480.96 425.86 55.09 11
8 480.96 425.93 55.03 11
9 480.96 425.95 55 11
10 rows selected.
SQL>
The four highlighted disks are the ones we’ve just added (plus another one I did earlier).
Kick off a manual rebalance:
SQL> alter diskgroup DATA rebalance power 5 ;
Diskgroup altered.
SQL>
We can see that a rebalance is now in progress:
SQL> select * from v$asm_operation;
GROUP_NUMBER OPERA PASS STAT POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES
------------ ----- --------- ---- ---------- ---------- ---------- ---------- ---------- -----------
ERROR_CODE CON_ID
-------------------------------------------- ----------
1 REBAL REBALANCE RUN 5 5 620 260870 2886 90
0
1 REBAL COMPACT WAIT 5 5 0 0 0 0
0
SQL>
When the rebalance has completed, we can see that all of the disks are now
fully balanced:
fully balanced:
SQL> select DISK_NUMBER
2 ,ROUND(TOTAL_MB/1024,2) TOTAL_GB
3 ,ROUND(FREE_MB/1024,2) FREE_GB
4 ,ROUND((TOTAL_MB - FREE_MB)/1024,2) USED_GB
5 ,ROUND(((TOTAL_MB - FREE_MB)/TOTAL_MB)*100,0) PERCENT_USED
6 from v$asm_disk
7 where GROUP_NUMBER = &1
order by 1 8
9
SQL>
SQL> /
Enter value for 1: 1
old 7: where GROUP_NUMBER = &1
new 7: where GROUP_NUMBER = 1
DISK_NUMBER TOTAL_GB FREE_GB USED_GB PERCENT_USED
----------- ---------- ---------- ---------- ------------
0 480.96 167.8 313.16 65
1 480.96 167.82 313.13 65
2 480.96 167.82 313.13 65
3 480.96 167.82 313.14 65
4 480.96 167.83 313.13 65
5 480.96 167.83 313.13 65
6 480.96 167.84 313.12 65
7 480.96 167.84 313.12 65
8 480.96 167.84 313.12 65
9 480.96 167.83 313.13 65
10 rows selected.
SQL>
Notes
This was for a system using 2Tb disks - had to set the compatibility to 12.2 for both asm and rdbms see post on this.
[root@aodw01 ~]# lvmdiskscan
/dev/mapper/NFPSADWGGDATA01S [ <300.04 GiB]
/dev/mapper/NFPSADWORADATA03S [ 2.00 TiB]
/dev/mapper/NFPSADWORADATA04S [ 2.00 TiB]
/dev/mapper/NFPSADWORADATA02S [ 2.00 TiB]
/dev/mapper/NFPSADWORADATA01S [ 2.00 TiB]
/dev/mapper/NFPSADWORARECV01S [ 2.00 TiB]
/dev/mapper/NFPSADWGGDATA01S [ <300.04 GiB]
/dev/mapper/NFPSADWORADATA03S [ 2.00 TiB]
/dev/mapper/NFPSADWORADATA04S [ 2.00 TiB]
/dev/mapper/NFPSADWORADATA02S [ 2.00 TiB]
/dev/mapper/NFPSADWORADATA01S [ 2.00 TiB]
/dev/mapper/NFPSADWORARECV01S [ 2.00 TiB]
Create oracleasm disk for each of these
[root@usnympmtaodw01 disks]# oracleasm createdisk NFPSADWGGDATA /dev/mapper/NFPSADWGGDATA01S
Writing disk header: done
Instantiating disk: failed
Writing disk header: done
Instantiating disk: failed
Had to manually chown oracle:oinstall on the disk under /dev/oracleasm/disk - need to check why
--- END ---
No comments:
Post a Comment