Emergency ASM Diskgroup Recovery: Troubleshooting ORA-15063 with kfed and amdu

The Nightmare Scenario: ORA-15063 on Diskgroup Mount

During server reboots, SAN firmware updates, or unexpected storage failovers, an Oracle DBAs’ worst nightmare is attempting to mount an ASM diskgroup and receiving:

ORA-15032: unhandled error raised in ASM instance
ORA-15063: ASM discovered an insufficient number of disks for diskgroup DATA

When ORA-15063 occurs, the ASM instance cannot read the Allocation Unit (AU) 0 metadata header on the physical storage devices.

⚠️ GOLDEN RULE OF ASM DISASTER RECOVERY:
DO NOT run kpartx, fdisk, mkfs, or dd commands on unmountable ASM disks to “test” them. Overwriting Allocation Unit 0 destroys the ASM file directory and metadata allocation maps.

---

🛠️ Essential Recovery Toolset: kfed & amdu

Oracle provides two low-level, command-line diagnostic utilities inside $ORACLE_HOME/bin:

1. kfed (Kernel Files Metadata Editor): Reads, dumps, and repairs ASM disk metadata headers directly from block devices without requiring the ASM instance to mount the diskgroup.
2. amdu (ASM Metadata Dump Utility): Scans raw storage devices, parses ASM metadata trees, and extracts intact database datafiles (.dbf) from unmounted or damaged diskgroups directly to local disk.

---

Step 1: Diagnosing Header Corruption with kfed

Run kfed read against the suspect disk device path to inspect Block 0 (the Disk Header):

# Read Disk Header Block 0 using kfed
$ORACLE_HOME/bin/kfed read /dev/mapper/mpath_data_01 block=0

Healthy Disk Header Output:

kfbh.type:               1 ; 0x000: KFBTYP_DISKHEAD
kfvd.grpname:         DATA ; 0x028: Diskgroup Name
kfvd.dskname:      DATA_01 ; 0x048: ASM Disk Name
kfbh.endian:             1 ; 0x004: Little-Endian

Corrupted Disk Header Output:

kfbh.type:               0 ; 0x000: KFBTYP_INVALID  <-- HEADER CORRUPTED
kfvd.grpname:              ; 0x028: 

---

Step 2: Repairing Header Corruption from Backup or Replica

If the header block (kfbh.type = KFBTYP_INVALID) was zeroed out by a rogue OS script, but the rest of the disk data remains intact:

Option A: Restore Header from AU1 Backup Header using kfed repair

ASM maintains a secondary metadata header backup in Allocation Unit 1 (AU1). You can repair corrupted AU0 header blocks using the AU1 backup header:

# 1. ALWAYS take a raw dd safety backup of AU0 before writing changes!
dd if=/dev/mapper/mpath_data_01 of=/tmp/mpath_data_01_au0.bak bs=1M count=2

# 2. Repair AU0 header block from secondary AU1 backup header using kfed
$ORACLE_HOME/bin/kfed repair /dev/mapper/mpath_data_01

# 3. Re-verify header layout and checksums using kfed
$ORACLE_HOME/bin/kfed read /dev/mapper/mpath_data_01 block=0 | grep -E "KFBTYP|grpname|dskname"

---

Step 3: Extracting Datafiles using amdu (When Diskgroup Won’t Mount)

If the diskgroup cannot be mounted and kfed repairs fail, use amdu to bypass ASM completely and extract your production .dbf datafiles:

# Run AMDU Discovery and Extract Files from Diskgroup DATA
$ORACLE_HOME/bin/amdu -diskstring '/dev/mapper/mpath_*' -extract DATA

What amdu Produces:

• report.txt: Detailed report on all discovered ASM allocation units and metadata structures.
• DATA_*.f files: Recovered database datafiles (e.g., DATA_256.f maps to system01.dbf).

Once extracted, rename the .f files back to their original names and mount the database using standard RMAN or SQL*Plus:

-- Rename Extracted Datafile in Control File
ALTER DATABASE RENAME FILE '+DATA/orcl/datafile/system.256.1092837' TO '/u01/app/oracle/oradata/system01.dbf';

---

🔍 Prevention & ASM Header Backup Strategy

Prevent future ORA-15063 emergencies by backing up your ASM disk headers automatically after every storage change:

# Backup ASM Metadata Headers for All Diskgroups
$ORACLE_HOME/bin/asmcmd md_backup /u01/app/oracle/asm_header_backup.bak

---

📚 Official Documentation & Technical References

• Oracle Automatic Storage Management Administrator’s Guide: Troubleshooting Oracle ASM Diskgroups — Official documentation on diskgroup mount issues and diagnostic utilities.
• ORACLE-BASE: ASM Metadata Utilities (kfed, amdu, kwkf) — Technical guide to low-level ASM recovery utilities.

---

Need emergency assistance recovering unmountable ASM diskgroups or restoring damaged datafiles with amdu? Request Emergency DBA Support or explore our Enterprise Database Audits.