ASM rebalance Power
Whenever there is a disk added/removed/resized, ASM starts to rebalance the data on the disks in the disk group. The power with which it does the rebalancing is the rebalancing power. The rebalance power defaults to the value of the ASM_POWER_LIMIT initialization parameter (default value = 1). The higher the limit, the faster a rebalance operation may complete. Lower values cause rebalancing to take longer but consume fewer processing and I/O resources. This leaves these resources available for other applications, such as the database. Beginning with Oracle Database 11g Release 2 (11.2.0.2), if the COMPATIBLE.ASM disk group attribute is set to 11.2.0.2 or higher, then the range of ASM_POWER_LIMIT parameter is 0 to 1024. All previous versions have the parameter range of 0 to 11.
The disk group rebalance operation has three phases:
1. Planning
2. File extents relocation
3. Disk compacting
It is possible to adjust this parameter dynamically, however adjusting ASM_POWER_LIMIT only affects future rebalances. It does not affect an in progress rebalance. The asm_power_limit can be found in the v$asm_operation view.
SQL> show parameter limit NAME TYPE VALUE ------------------------------------ ----------- ------ asm_power_limit integer 1
Changing the rebalancing power of an ongoing operation
Let us drop a disk from the diskgroup DATA :
SQL> alter diskgroup DATA drop disk data_disk01; Diskgroup altered.
Whenever there is a rebalancing in the disk group, a rebalancing process ARBn will start :
SQL> select * from v$asm_operation;
GROUP_NUMBER OPERA STAT POWER ACTUAL SOFAR EST_WORK EST_RATE
------------ ----- ---- ---------- ---------- ---------- ---------- ----------
1 REBAL WAIT 1 0 0 0
ASM alert log :
Fri Dec 26 16:55:08 2008 NOTE: starting rebalance of group 1/0x8c6014a7 (DATA) at power 1 Starting background process ARB0 ARB0 started with pid=14, OS id=316 Fri Dec 26 16:55:08 2008 NOTE: assigning ARB0 to group 1/0x8c6014a7 (DATA)
As we can see, the power of the rebalancing is 1, as it defaults to the parameter ASM_POWER_LIMIT. After running a rebalance command, the running rebalance process will be stopped and new rebalance slave processes will be spawned depending on POWER used.
SQL> alter diskgroup dg1 rebalance power 8; Diskgroup altered.
Check for the 8 new rebalance slave processes spawned (ARB0,….,ARB7)
SQL> select * from v$asm_operation;
GROUP_NUMBER OPERA STAT POWER ACTUAL SOFAR EST_WORK EST_RATE
------------ ----- ---- ---------- ---------- ---------- ---------- ----------
1 REBAL RUN 8 8 0 407 0
ASM alert.log :
Fri Dec 26 16:55:11 2008 SQL> alter diskgroup dg1 rebalance power 8 NOTE: stopping process ARB0 Fri Dec 26 16:55:11 2008 NOTE: rebalance interrupted for group 1/0x8c6014a7 (DG1) Fri Dec 26 16:55:11 2008 NOTE: PST update: grp = 1 NOTE: requesting all-instance PST refresh for group=1 Fri Dec 26 16:55:11 2008 NOTE: PST refresh pending for group 1/0x8c6014a7 (DG1) SUCCESS: refreshed PST for 1/0x8c6014a7 (DG1) Fri Dec 26 16:55:14 2008 NOTE: starting rebalance of group 1/0x8c6014a7 (DG1) at power 8 Starting background process ARB0 Starting background process ARB1 ARB0 started with pid=14, OS id=1628 Fri Dec 26 16:55:15 2008 Starting background process ARB2 ARB1 started with pid=15, OS id=1920 Fri Dec 26 16:55:15 2008 Starting background process ARB3 ARB2 started with pid=16, OS id=1964 Fri Dec 26 16:55:15 2008 Starting background process ARB4 ARB3 started with pid=17, OS id=1664 Fri Dec 26 16:55:15 2008 Starting background process ARB5 ARB4 started with pid=18, OS id=1836 Fri Dec 26 16:55:15 2008 Starting background process ARB6 ARB5 started with pid=19, OS id=1972 Fri Dec 26 16:55:15 2008 Starting background process ARB7 ARB6 started with pid=20, OS id=1800
How much should I increase the power ?
The rebalance power can be adjusted dynamically, i.e. during the rebalance. If the rebalance with the default power is ‘too slow’, the power can be increased. How much? To answer that questions, we need to understand the I/O load, the I/O throughput and most importantly the I/O limits the system can take. If we don’t know that, the power can be increased to 5 (with ‘ALTER DISKGROUP … REBALANCE POWER 5;’). We can then check if that makes a difference. Should we go any higher with the rebalance power? Again, as long as we are not adversely impacting the database I/O performance, we can keep increasing the power. But I haven’t seen much improvement beyond power 30. Note that the power can go up to 11 with disk groups with COMPATIBLE.ASM < 11.2.0.2 and up to 1024 for disk groups with COMPATIBLE.ASM=>11.2.0.2.
The testing is the key here. We really need to test the rebalance under the regular production load, with different values for the power. There is no point testing with no databases running or on a system with different storage characteristics