Oracle ASM Diskgroup Rebalancing Under Heavy Production Load

Executive Summary

ASM rebalancing is one of the most misunderstood background operations in Oracle Database. When a disk is added, dropped, resized, or fails, ASM redistributes extents across the disk group to maintain uniform I/O distribution. Under heavy production load, this redistribution competes for the same storage array cache, HBA/FC queue depth, and interconnect bandwidth as the database workload. The result is a cascading performance degradation: elevated read/write latency, buffer cache pressure, log file sync spikes, and — in severe cases — ORA-00600 internal errors when concurrent DDL collides with rebalance metadata updates.

This guide provides a field-tested methodology for diagnosing, bounding, and resolving ASM rebalance contention in production. It covers the diagnostic queries that matter, the exact ALTER DISKGROUP commands to throttle and control rebalance, and a step-by-step runbook that starts with safety checks and ends with post-rebalance validation.

Key takeaways:

• Rebalance is I/O-bound, not CPU-bound. Throttle with POWER, not with OS-level ionice or nice.
• v$ASM_OPERATION is your single source of truth for progress and rate.
• Always estimate before you execute: v$ASM_ESTIMATE projects work, rate, and minutes.
• POWER 0 pauses a rebalance; it does not cancel it. The operation resumes on the next REBALANCE POWER n command or instance restart.
• In 23ai, rebalance operations are more observable and safer under concurrency, but only if the ASM home is patched to the latest Release Update.

---

1. Overview

ASM rebalancing follows a well-defined lifecycle from trigger to completion. Understanding this flow is essential for diagnosing where contention occurs and which control lever to pull at each stage.

Stage details:

1. Trigger — ADD/DROP/RESIZE DISK, disk failure, or power change initiates an automatic rebalance.
2. Estimate — ASM computes the extent redistribution plan; v$ASM_ESTIMATE exposes projected work, rate, and minutes.
3. Plan — ASM builds the rebalance plan in memory (kfgb) and assigns extents to the new disk layout.
4. Execute — RBAL/ARBx processes move extents at the configured POWER level; I/O contention peaks here.
5. Monitor — v$ASM_OPERATION tracks SOFAR/EST_WORK/EST_RATE; adjust POWER dynamically.
6. Complete — Operation state transitions to DONE; disk group returns to BALANCED state.

---

2. Diagnostic Checklist

2.1 Identify Active Rebalance Operations

Run on the ASM instance (or gv$asm_operation on RAC ASM):

SELECT group_number,
       operation,
       state,
       power,
       actual_power,
       sofar,
       est_work,
       ROUND(sofar / est_work * 100, 2) AS pct_complete,
       est_rate,
       est_minutes,
       start_time
FROM   v$asm_operation
ORDER  BY group_number;

Interpretation:

• state = RUNNING — rebalance is actively moving extents.
• state = WAITING — rebalance is queued or paused (POWER 0).
• state = DONE — operation completed; verify with v$asm_diskgroup.
• actual_power < power — ASM is throttling due to I/O errors or resource constraints.

2.2 Check Disk Group Balance State

SELECT name,
       state,
       type,
       total_mb,
       free_mb,
       usable_file_mb,
       offline_disks,
       rebalance_power
FROM   v$asm_diskgroup;

2.3 Estimate Rebalance Work Before Making Changes

-- Run on ASM instance before ADD/DROP/RESIZE
SELECT group_number,
       operation,
       est_work,
       est_rate,
       est_minutes,
       power
FROM   v$asm_estimate;

If v$asm_estimate is empty, force an estimate:

ALTER DISKGROUP DATA REBALANCE POWER 1 NOWAIT;

Then re-query v$asm_estimate.

2.4 Check Disk-Level I/O Distribution

SELECT dg.name              AS diskgroup,
       d.name               AS disk_name,
       d.total_mb,
       d.free_mb,
       d.reads,
       d.writes,
       d.read_time,
       d.write_time
FROM   v$asm_disk d
JOIN   v$asm_diskgroup dg ON dg.group_number = d.group_number
WHERE  dg.state = 'MOUNTED'
ORDER  BY d.write_time DESC;

2.5 Check ASM Alert Log

# ASM alert log location
$ORACLE_BASE/diag/asm/+asm/+ASM1/trace/alert_+asm1.log

# Grep for rebalance-related messages
grep -i "rebal\|ORA-00600\|ORA-15032" alert_+asm1.log | tail -50

2.6 Check OS-Level I/O Contention

# Check I/O wait and per-disk utilization
iostat -x 5 10

# Check ASM background processes (RBAL, ARBx)
ps -ef | grep -E "arb|rbal" | grep -v grep

# Check storage array latency (if available)
sar -d 5 10

2.7 Check for Concurrent DDL / Metadata Operations

-- Check for active DML and structural DDL on the database side
SELECT sid, serial#, username, command, status, sql_id, elapsed_time
FROM   v$session
WHERE  status = 'ACTIVE'
AND    command IN (2, 6, 7, 12, 15, 28, 85)  -- INSERT, UPDATE, DELETE, DROP TABLE, ALTER TABLE, RENAME, TRUNCATE
ORDER  BY elapsed_time DESC;

---

3. Step-by-Step Resolution Runbook

Step 0: Safety Checks

Before touching anything, verify the following:

-- 1. Confirm you are on the ASM instance
SELECT instance_name, host_name, version
FROM   v$instance;

-- 2. Verify disk group redundancy and state
SELECT name, state, type, total_mb, free_mb
FROM   v$asm_diskgroup
WHERE  state = 'MOUNTED';

-- 3. Confirm no existing rebalance is already running
SELECT group_number, operation, state, power
FROM   v$asm_operation;

-- 4. Verify ASM_POWER_LIMIT
SHOW PARAMETER asm_power_limit;

# 5. Verify ASM instance health and attributes
asmcmd lsdg
asmcmd lsattr -G DATA -l rebalance_power

Safety rules:

• Never run ALTER DISKGROUP ... REBALANCE while another rebalance is active.
• Never drop a disk that is the last online copy of a redundancy group.
• Ensure a recent RMAN backup exists for all databases using the disk group.
• For NORMAL/HIGH redundancy disk groups, verify no disks are offline:

  SELECT name, mode_status FROM v$asm_disk WHERE mode_status != 'ONLINE';

• Confirm the ASM home is patched to the latest Release Update before starting any rebalance in a security-hardened environment.

Step 1: Identify the Trigger

Check the ASM alert log and operation history:

grep -i "rebal\|adding disk\|dropping disk\|offline" alert_+asm1.log | tail -100

Common triggers:

• ALTER DISKGROUP DATA ADD DISK '/dev/sd*'
• ALTER DISKGROUP DATA DROP DISK DATA_0001
• Disk failure causing automatic offline and rebalance
• ALTER DISKGROUP DATA RESIZE ALL
• Manual power change: ALTER DISKGROUP DATA REBALANCE POWER 8

Step 2: Assess Impact on Production

-- Check rebalance progress and rate
SELECT group_number,
       operation,
       state,
       power,
       actual_power,
       sofar,
       est_work,
       ROUND(sofar / est_work * 100, 2) AS pct_complete,
       est_minutes,
       est_rate
FROM   v$asm_operation;

If est_minutes is high and production is suffering, proceed to Step 3.

Step 3: Estimate the Work

-- Estimate the cost of the pending operation
SELECT group_number, operation, est_work, est_rate, est_minutes
FROM   v$asm_estimate;

If the estimate is not available, force a new estimate:

ALTER DISKGROUP DATA REBALANCE POWER 1 NOWAIT;

Then immediately re-query v$asm_estimate.

Step 4: Throttle the Rebalance

The single most effective control is the POWER parameter. It controls the number of parallel ARBx processes (0–11).

-- Throttle down to reduce I/O contention (e.g., from 8 to 2)
ALTER DISKGROUP DATA REBALANCE POWER 2;

-- Or pause entirely (POWER 0 stops the rebalance)
ALTER DISKGROUP DATA REBALANCE POWER 0;

Important: Setting POWER 0 pauses the rebalance but does not cancel it. The operation resumes at the next REBALANCE POWER n command or instance restart.

For a controlled, time-boxed rebalance:

-- Run at high power during a maintenance window
ALTER DISKGROUP DATA REBALANCE POWER 8 WAIT;

-- Run at low power during peak hours
ALTER DISKGROUP DATA REBALANCE POWER 1 NOWAIT;

Step 5: Monitor Progress

-- Poll every 60 seconds
SELECT group_number,
       operation,
       state,
       power,
       actual_power,
       sofar,
       est_work,
       ROUND(sofar / est_work * 100, 2) AS pct_complete,
       est_minutes,
       est_rate
FROM   v$asm_operation;

# Watch ASM background processes
watch -n 5 "ps -ef | grep -E 'arb|rbal' | grep -v grep"

Step 6: Handle Errors (ORA-00600, ORA-15032)

If you encounter ORA-00600 during rebalance:

1. Do not panic. The rebalance may have paused. Check v$asm_operation.
2. Collect diagnostics:

   asmcmd lsdg
   grep -i "ORA-00600" alert_+asm1.log | tail -20

3. Check for concurrent DDL. ORA-00600 during rebalance is frequently caused by concurrent ALTER DISKGROUP or DROP DISK operations colliding with metadata updates.
4. Pause the rebalance:

   ALTER DISKGROUP DATA REBALANCE POWER 0;

5. Resume at lower power:

   ALTER DISKGROUP DATA REBALANCE POWER 1;

6. If ORA-15032 is raised (ALTER DISKGROUP statement failed), check the underlying cause:

   SELECT * FROM v$asm_operation;
   SELECT name, state, offline_disks FROM v$asm_diskgroup;

   The most common cause is a disk that went offline during the rebalance. Bring it back online:

   ALTER DISKGROUP DATA ONLINE DISK DATA_0003;

7. Escalate to Oracle Support with the incident number and trace files if the error persists. Use the ORA-00600 lookup tool in My Oracle Support to identify the specific internal assertion.

Step 7: Post-Rebalance Validation

-- Confirm no active operations
SELECT group_number, operation, state
FROM   v$asm_operation;

-- Confirm disk group is BALANCED
SELECT name, state, type
FROM   v$asm_diskgroup;

-- Verify disk I/O distribution is uniform
SELECT d.name,
       d.total_mb,
       d.free_mb,
       d.reads,
       d.writes
FROM   v$asm_disk d
JOIN   v$asm_diskgroup dg ON dg.group_number = d.group_number
WHERE  dg.state = 'MOUNTED'
ORDER  BY d.name;

# Verify with asmcmd
asmcmd lsdg
asmcmd lsattr -G DATA -l rebalance_power

Step 8: Update Baseline and Configuration

• Set a sensible default ASM_POWER_LIMIT for your environment (typically 1–4 for production).
• Document the expected rebalance duration for each disk group.
• Schedule disk additions/drops during maintenance windows.
• Consider using disk-group-level rebalance power for finer control.
• In 23ai, leverage the enhanced rebalance observability (v$ASM_OPERATION enhancements) and the REBALANCE ... NOWAIT semantics to avoid blocking foreground operations.

---

📚 Official Documentation & Technical References

Oracle Documentation

• Oracle Documentation — Administering ASM Disk Groups
• Oracle Documentation — V$ASM_OPERATION
• Oracle Documentation — V$ASM_ESTIMATE
• Oracle Documentation — ALTER DISKGROUP

My Oracle Support

For the latest ASM rebalance tuning and disk group administration guidelines, refer to My Oracle Support (MOS).

Related Resources

• Oracle Automatic Storage Management Administrator’s Guide

---

Need Expert Help?

ASM rebalance incidents under production load are time-critical and high-risk. If you need hands-on assistance from certified Oracle engineers, contact DBPros.Net today.

• Contact Us
• View Our Services