Durham Oracle

Introduction

This post details the steps carried to add three disks to a DATA diskgroup in ASM.

The three disks were 500Gb each in size (approx).

After the RFC / CAB was raised, ‘presented’ the disks to the two database servers.

The next steps were to add the new storage to ASM, thereby making them
available to Oracle.

IMPORTANT:

Most of the steps here can be carried out by logging onto the database server as ‘oracle’,
followed by ‘sqlplus / as sysdba’

However, the step of actually adding the disks needs to be done using
‘sqlplus / as sysasm’.

Therefore – for consistency, and to keep things simple – it is advisable to carry out all of
the steps in this document by logging onto the DB server as the ‘grid’ user, and then onto
the ASM instance using sqlplus / as sysasm

Step 1 – check the space currently in ASM

Log onto DB server (either one) as grid.

sqlplus / as sysasm

set lines 120

col name format a25

col group_number format 999 heading GROUP

col name format a20

select GROUP_NUMBER

,NAME

,ROUND(TOTAL_MB/1024,1) TOTAL_GB

,ROUND(((TOTAL_MB - FREE_MB)/1024),1) USED_GB

,ROUND(FREE_MB/1024,1) FREE_GB

,ROUND((100-(FREE_MB/TOTAL_MB)*100),2) "% Used"

from V$ASM_DISKGROUP

order by group_number

e.g.

SQL> /

GROUP NAME TOTAL_GB USED_GB FREE_GB % Used

----- -------------------- ---------- ---------- ---------- ----------

1 DATA 3366.7 2679.4 687.3 79.59

2 FRA 959.8 473.3 486.5 49.32

SQL>

Step 2 – Check that the disks have been presented to the OS:

db01 /dev/asmdisks > pwd

/dev/asmdisks

cdepld2prsodb01 /dev/asmdisks (rpap1) > ls -l

total 0

brw-rw----. 1 grid asmadmin 253, 8 Mar 11 14:43 asm1

brw-rw----. 1 grid asmadmin 253, 6 Mar 11 14:43 asm10

brw-rw----. 1 grid asmadmin 253, 32 Mar 11 14:43 asm11

brw-rw----. 1 grid asmadmin 253, 31 Mar 11 14:39 asm12

brw-rw----. 1 grid asmadmin 253, 34 Mar 11 14:39 asm13

brw-rw----. 1 grid asmadmin 253, 30 Mar 11 14:39 asm14

brw-rw----. 1 grid asmadmin 253, 9 Mar 11 14:43 asm2

brw-rw----. 1 grid asmadmin 253, 7 Mar 11 14:43 asm3

brw-rw----. 1 grid asmadmin 253, 11 Mar 11 14:43 asm4

brw-rw----. 1 grid asmadmin 253, 10 Mar 11 14:43 asm5

brw-rw----. 1 grid asmadmin 253, 12 Mar 11 14:43 asm6

brw-rw----. 1 grid asmadmin 253, 3 Mar 11 14:43 asm7

brw-rw----. 1 grid asmadmin 253, 5 Mar 11 14:43 asm8

brw-rw----. 1 grid asmadmin 253, 4 Mar 11 14:43 asm9

cdepld2prsodb01 /dev/asmdisks (rpap1) >

Make sure the new ones have the same ownership and permissions as the existing ones.

→there used to be 11, now there are14.

Step 3 – Check which disks ASM can actually see

Set lines 200 pages 999

col disk_group format a15

col DISK_NAME format a15

col path format a50

select g.name disk_group

,d.DISK_NUMBER

,d.name disk_name

,d.PATH

,d.TOTAL_MB

,d.header_status

from v$asm_disk d,

v$asm_diskgroup g

where d.GROUP_NUMBER = g.GROUP_NUMBER (+)

order by g.name, d.disk_number

1 select g.name disk_group

2 ,d.DISK_NUMBER

3 ,d.name disk_name

4 ,d.PATH

5 ,d.TOTAL_MB

6 ,d.header_status

7 from v$asm_disk d,

8 v$asm_diskgroup g

9 where d.GROUP_NUMBER = g.GROUP_NUMBER (+)

10* order by g.name, d.disk_number

SQL>

SQL> /

DISK_GROUP DISK_NUMBER DISK_NAME PATH TOTAL_MB HEADER_STATU

--------------- ----------- --------------- ------------------------- ---------- ------------

DATA 0 DATA_0000 /dev/asmdisks/asm1 492500 MEMBER

DATA 1 DATA_0001 /dev/asmdisks/asm2 492500 MEMBER

DATA 2 DATA_0002 /dev/asmdisks/asm3 492500 MEMBER

DATA 3 DATA_0003 /dev/asmdisks/asm4 492500 MEMBER

DATA 4 DATA_0004 /dev/asmdisks/asm5 492500 MEMBER

DATA 5 DATA_0005 /dev/asmdisks/asm6 492500 MEMBER

DATA 6 DATA_0006 /dev/asmdisks/asm11 492500 MEMBER

FRA 0 FRA_0000 /dev/asmdisks/asm10 245700 MEMBER

FRA 1 FRA_0001 /dev/asmdisks/asm7 245700 MEMBER

FRA 2 FRA_0002 /dev/asmdisks/asm8 245700 MEMBER

FRA 3 FRA_0003 /dev/asmdisks/asm9 245700 MEMBER

0 /dev/asmdisks/asm12 0 CANDIDATE

1 /dev/asmdisks/asm14 0 CANDIDATE

2 /dev/asmdisks/asm13 0 CANDIDATE

14 rows selected.

SQL>

→the highlighted ones are the new ones that have not been added to a diskgroup yet.

Step 4 – Add the first disk

SQL> l

1 ALTER DISKGROUP DATA

2 ADD DISK '/dev/asmdisks/asm12'

3* NAME DATA_0007 rebalance power 0

SQL> /

Diskgroup altered.

SQL>

Step 5 – Run the two queries again:

SQL> /

GROUP NAME TOTAL_GB USED_GB FREE_GB % Used

----- -------------------- ---------- ---------- ---------- ----------

1 DATA 3847.7 2680.8 1166.9 69.67

2 FRA 959.8 475.1 484.7 49.5

SQL>

And:

SQL> /

DISK_GROUP DISK_NUMBER DISK_NAME PATH TOTAL_MB HEADER_STATU

--------------- ----------- --------------- ------------------------- ---------- ------------

DATA 0 DATA_0000 /dev/asmdisks/asm1 492500 MEMBER

DATA 1 DATA_0001 /dev/asmdisks/asm2 492500 MEMBER

DATA 2 DATA_0002 /dev/asmdisks/asm3 492500 MEMBER

DATA 3 DATA_0003 /dev/asmdisks/asm4 492500 MEMBER

DATA 4 DATA_0004 /dev/asmdisks/asm5 492500 MEMBER

DATA 5 DATA_0005 /dev/asmdisks/asm6 492500 MEMBER

DATA 6 DATA_0006 /dev/asmdisks/asm11 492500 MEMBER

DATA 7 DATA_0007 /dev/asmdisks/asm12 492500 MEMBER

FRA 0 FRA_0000 /dev/asmdisks/asm10 245700 MEMBER

FRA 1 FRA_0001 /dev/asmdisks/asm7 245700 MEMBER

FRA 2 FRA_0002 /dev/asmdisks/asm8 245700 MEMBER

FRA 3 FRA_0003 /dev/asmdisks/asm9 245700 MEMBER

0 /dev/asmdisks/asm14 0 CANDIDATE

1 /dev/asmdisks/asm13 0 CANDIDATE

14 rows selected.

SQL>

Step 6 – Check ASM alert log

/u01/app/grid/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log

Check no errors.

e.g.

Wed Mar 11 14:57:30 2015

SQL> ALTER DISKGROUP DATA

ADD DISK '/dev/asmdisks/asm12'

NAME DATA_0007 rebalance power 0

Wed Mar 11 14:57:30 2015

NOTE: GroupBlock outside rolling migration privileged region

NOTE: Assigning number (1,7) to disk (/dev/asmdisks/asm12)

NOTE: requesting all-instance membership refresh for group=1

NOTE: Disk 7 in group 1 is assigned fgnum=8

NOTE: initializing header (replicated) on grp 1 disk DATA_0007

NOTE: initializing header on grp 1 disk DATA_0007

NOTE: requesting all-instance disk validation for group=1

Wed Mar 11 14:57:32 2015

NOTE: skipping rediscovery for group 1/0xc6785ea9 (DATA) on local instance.

Wed Mar 11 14:57:32 2015

NOTE: requesting all-instance disk validation for group=1

Wed Mar 11 14:57:32 2015

NOTE: skipping rediscovery for group 1/0xc6785ea9 (DATA) on local instance.

Wed Mar 11 14:57:32 2015

GMON updating for reconfiguration, group 1 at 12 for pid 39, osid 6241

Wed Mar 11 14:57:32 2015

NOTE: group 1 PST updated.

Wed Mar 11 14:57:32 2015

NOTE: membership refresh pending for group 1/0xc6785ea9 (DATA)

Wed Mar 11 14:57:32 2015

GMON querying group 1 at 13 for pid 21, osid 7028

NOTE: cache opening disk 7 of grp 1: DATA_0007 path:/dev/asmdisks/asm12

GMON querying group 1 at 14 for pid 21, osid 7028

Wed Mar 11 14:57:34 2015

SUCCESS: refreshed membership for 1/0xc6785ea9 (DATA)

Wed Mar 11 14:57:34 2015

SUCCESS: ALTER DISKGROUP DATA

ADD DISK '/dev/asmdisks/asm12'

NAME DATA_0007 rebalance power 0

NOTE: Attempting voting file refresh on diskgroup DATA

Step 7 – check DB alert log

Again, check for any errors:

e.g.

Wed Mar 11 14:57:34 2015

NOTE: Assigning number (1,7) to disk (/dev/asmdisks/asm12)

SUCCESS: disk DATA_0007 (7.3915951733) added to diskgroup DATA

Wed Mar 11 14:58:23 2015

Step 8 – check if rebalancing is taking place:

SQL> select * from v$asm_operation ;

GROUP OPERA PASS STAT POWER ACTUAL SOFAR EST_WORK
EST_RATE EST_MINUTES

----- ----- --------- ---- ---------- ---------- ---------- ---------- ---------- -----------

ERROR_CODE CON_ID

-------------------------------------------- ----------

1 REBAL REBALANCE WAIT 0

1 REBAL COMPACT WAIT 0

SQL>

→the above is correct, since we have ‘disabled’ automatic rebalancing by using the ‘power 0’ syntax.

Step 9 – add the remaining disks

SQL> l

1 ALTER DISKGROUP DATA

2 ADD DISK '/dev/asmdisks/asm13'

3* NAME DATA_0008 rebalance power 0

SQL> /

Diskgroup altered.

SQL>

SQL> l

1 ALTER DISKGROUP DATA

2 ADD DISK '/dev/asmdisks/asm14'

3* NAME DATA_0009 rebalance power 0

SQL> /

Diskgroup altered.

SQL>

SQL> /

GROUP NAME TOTAL_GB USED_GB FREE_GB % Used

----- -------------------- ---------- ---------- ---------- ----------

1 DATA 4809.6 2680.8 2128.8 55.74

2 FRA 959.8 475.9 483.8 49.59

SQL>

SQL> /

DISK_GROUP DISK_NUMBER DISK_NAME PATH TOTAL_MB HEADER_STATU

--------------- ----------- --------------- ------------------------- ---------- ------------

DATA 0 DATA_0000 /dev/asmdisks/asm1 492500 MEMBER

DATA 1 DATA_0001 /dev/asmdisks/asm2 492500 MEMBER

DATA 2 DATA_0002 /dev/asmdisks/asm3 492500 MEMBER

DATA 3 DATA_0003 /dev/asmdisks/asm4 492500 MEMBER

DATA 4 DATA_0004 /dev/asmdisks/asm5 492500 MEMBER

DATA 5 DATA_0005 /dev/asmdisks/asm6 492500 MEMBER

DATA 6 DATA_0006 /dev/asmdisks/asm11 492500 MEMBER

DATA 7 DATA_0007 /dev/asmdisks/asm12 492500 MEMBER

DATA 8 DATA_0008 /dev/asmdisks/asm13 492500 MEMBER

DATA 9 DATA_0009 /dev/asmdisks/asm14 492500 MEMBER

FRA 0 FRA_0000 /dev/asmdisks/asm10 245700 MEMBER

FRA 1 FRA_0001 /dev/asmdisks/asm7 245700 MEMBER

FRA 2 FRA_0002 /dev/asmdisks/asm8 245700 MEMBER

FRA 3 FRA_0003 /dev/asmdisks/asm9 245700 MEMBER

14 rows selected.

SQL>

→all looks good. ASM alert log:

No errors:

DB Alert log:

Also no errors:

Wed Mar 11 15:07:33 2015

NOTE: Assigning number (1,8) to disk (/dev/asmdisks/asm13)

SUCCESS: disk DATA_0008 (8.3915951736) added to diskgroup DATA

Wed Mar 11 15:08:53 2015

NOTE: Assigning number (1,9) to disk (/dev/asmdisks/asm14)

SUCCESS: disk DATA_0009 (9.3915951738) added to diskgroup DATA

Wed Mar 11 15:10:03 2015

Step 10 – Run queries on Node 2

Should return the same results as on Node 1:

SQL> !uname -a

Linux db02.scc.local 2.6.32-431.11.2.el6.x86_64 #1 SMP Mon Mar 3 13:32:45 EST 2014 x86_64 x86_64 x86_64 GNU/Linux

SQL> /

GROUP NAME TOTAL_GB USED_GB FREE_GB % Used

----- -------------------- ---------- ---------- ---------- ----------

1 DATA 4809.6 2681.3 2128.3 55.75

2 FRA 959.8 476.5 483.2 49.65

SQL>

And:

SQL> !uname -a

Linux db02.scc.local 2.6.32-431.11.2.el6.x86_64 #1 SMP Mon Mar 3 13:32:45 EST 2014 x86_64 x86_64 x86_64 GNU/Linux

SQL> /

DISK_GROUP DISK_NUMBER DISK_NAME PATH TOTAL_MB HEADER_STATU

--------------- ----------- --------------- ------------------------- ---------- ------------

DATA 0 DATA_0000 /dev/asmdisks/asm1 492500 MEMBER

DATA 1 DATA_0001 /dev/asmdisks/asm2 492500 MEMBER

DATA 2 DATA_0002 /dev/asmdisks/asm3 492500 MEMBER

DATA 3 DATA_0003 /dev/asmdisks/asm4 492500 MEMBER

DATA 4 DATA_0004 /dev/asmdisks/asm5 492500 MEMBER

DATA 5 DATA_0005 /dev/asmdisks/asm6 492500 MEMBER

DATA 6 DATA_0006 /dev/asmdisks/asm11 492500 MEMBER

DATA 7 DATA_0007 /dev/asmdisks/asm12 492500 MEMBER

DATA 8 DATA_0008 /dev/asmdisks/asm13 492500 MEMBER

DATA 9 DATA_0009 /dev/asmdisks/asm14 492500 MEMBER

FRA 0 FRA_0000 /dev/asmdisks/asm10 245700 MEMBER

FRA 1 FRA_0001 /dev/asmdisks/asm7 245700 MEMBER

FRA 2 FRA_0002 /dev/asmdisks/asm8 245700 MEMBER

FRA 3 FRA_0003 /dev/asmdisks/asm9 245700 MEMBER

14 rows selected.

SQL>

Step 11

Now that we have added all the disks, we can invoke a disk rebalance manually.

Current situation shows that the disks are imbalanced in this diskgroup:

SQL> select DISK_NUMBER

2 ,ROUND(TOTAL_MB/1024,2) TOTAL_GB

3 ,ROUND(FREE_MB/1024,2) FREE_GB

4 ,ROUND((TOTAL_MB - FREE_MB)/1024,2) USED_GB

5 ,ROUND(((TOTAL_MB - FREE_MB)/TOTAL_MB)*100,0) PERCENT_USED

6 from v$asm_disk

7 where GROUP_NUMBER = &1

8 order by 1

SQL>

SQL> /

Enter value for 1: 1

old 7: where GROUP_NUMBER = &1

new 7: where GROUP_NUMBER = 1

DISK_NUMBER TOTAL_GB FREE_GB USED_GB PERCENT_USED

----------- ---------- ---------- ---------- ------------

0 480.96 .16 480.8 100

1 480.96 0 480.96 100

2 480.96 .04 480.91 100

3 480.96 .07 480.89 100

4 480.96 .17 480.79 100

5 480.96 .05 480.91 100

6 480.96 414.49 66.47 14

7 480.96 425.86 55.09 11

8 480.96 425.93 55.03 11

9 480.96 425.95 55 11

10 rows selected.

SQL>

The four highlighted disks are the ones we’ve just added (plus another one I did earlier).

Kick off a manual rebalance:

SQL> alter diskgroup DATA rebalance power 5 ;

Diskgroup altered.

SQL>

We can see that a rebalance is now in progress:

SQL> select * from v$asm_operation;

GROUP_NUMBER OPERA PASS STAT POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES

------------ ----- --------- ---- ---------- ---------- ---------- ---------- ---------- -----------

ERROR_CODE CON_ID

-------------------------------------------- ----------

1 REBAL REBALANCE RUN 5 5 620 260870 2886 90

1 REBAL COMPACT WAIT 5 5 0 0 0 0

SQL>

When the rebalance has completed, we can see that all of the disks are now
fully balanced:

SQL> select DISK_NUMBER

2 ,ROUND(TOTAL_MB/1024,2) TOTAL_GB

3 ,ROUND(FREE_MB/1024,2) FREE_GB

4 ,ROUND((TOTAL_MB - FREE_MB)/1024,2) USED_GB

5 ,ROUND(((TOTAL_MB - FREE_MB)/TOTAL_MB)*100,0) PERCENT_USED

6 from v$asm_disk

7 where GROUP_NUMBER = &1

order by 1 8

SQL>

SQL> /

Enter value for 1: 1

old 7: where GROUP_NUMBER = &1

new 7: where GROUP_NUMBER = 1

DISK_NUMBER TOTAL_GB FREE_GB USED_GB PERCENT_USED

----------- ---------- ---------- ---------- ------------

0 480.96 167.8 313.16 65

1 480.96 167.82 313.13 65

2 480.96 167.82 313.13 65

3 480.96 167.82 313.14 65

4 480.96 167.83 313.13 65

5 480.96 167.83 313.13 65

6 480.96 167.84 313.12 65

7 480.96 167.84 313.12 65

8 480.96 167.84 313.12 65

9 480.96 167.83 313.13 65

10 rows selected.

SQL>

Notes

This was for a system using 2Tb disks - had to set the compatibility to 12.2 for both asm and rdbms see post on this.

[root@aodw01 ~]# lvmdiskscan

/dev/mapper/NFPSADWGGDATA01S [    <300.04 GiB]
/dev/mapper/NFPSADWORADATA03S [       2.00 TiB]
/dev/mapper/NFPSADWORADATA04S [       2.00 TiB]
/dev/mapper/NFPSADWORADATA02S [       2.00 TiB]
/dev/mapper/NFPSADWORADATA01S [       2.00 TiB]
/dev/mapper/NFPSADWORARECV01S [       2.00 TiB]

Create oracleasm disk for each of these

[root@usnympmtaodw01 disks]# oracleasm createdisk NFPSADWGGDATA /dev/mapper/NFPSADWGGDATA01S
Writing disk header: done
Instantiating disk: failed

Had to manually chown oracle:oinstall on the disk under /dev/oracleasm/disk - need to check why

--- END ---

Durham Oracle

How to add disks to ASM

edb360 - reporting tool