Oracle Linux 8 UEK Memory Starvation & KVM vCPU Overcommit Pitfalls

1. Executive Summary

2. Process Flow: Memory Starvation Cascade

Trigger Point: Step 2 — Balloon Inflation. Once the KVM balloon inflates beyond the guest’s available headroom, the cascade becomes self-reinforcing and typically requires manual intervention.

3. Diagnostic Checklist

Run these commands in order on the KVM host and the guest. Capture output before making any changes.

3.1 Host-Level Diagnostics

# 1. Check vCPU overcommit ratio (host)
echo "=== Physical Cores ==="
lscpu | grep -E "^CPU\(s\)|^Thread|^Core|^Socket|^NUMA"

echo "=== Total vCPUs allocated to all guests ==="
for vm in $(virsh list --name); do
  vcpus=$(virsh vcpucount "$vm" --live 2>/dev/null | grep current | awk '{print $3}')
  echo "$vm: $vcpus vCPUs"
done

echo "=== Overcommit Ratio ==="
total_vcpus=$(for vm in $(virsh list --name); do virsh vcpucount "$vm" --live 2>/dev/null | grep current | awk '{print $3}'; done | awk '{s+=$1} END {print s}')
physical_cores=$(lscpu | grep "^CPU(s):" | awk '{print $2}')
echo "Total vCPUs: $total_vcpus / Physical Cores: $physical_cores"
echo "Ratio: $(echo "scale=2; $total_vcpus / $physical_cores" | bc)"

# 2. Check memory balloon status for each guest
for vm in $(virsh list --name); do
  echo "=== $vm Balloon ==="
  virsh dommemstat "$vm" --live
  virsh dumpxml "$vm" | grep -A5 "<memballoon>"
done

# 3. Check host memory pressure
echo "=== Host Memory Pressure ==="
free -h
cat /proc/pressure/memory

# 4. Check NUMA topology on host
echo "=== Host NUMA Topology ==="
numactl --hardware

# 5. Check KVM steal time accounting
echo "=== Steal Time (host-side) ==="
top -b -n1 | grep -E "steal|st" | head -5

3.2 Guest-Level Diagnostics (Oracle Linux 8 UEK)

# 6. Check guest memory allocation
echo "=== Guest Memory ==="
free -h
cat /proc/meminfo | grep -E "MemTotal|MemFree|MemAvailable|SwapTotal|SwapFree|HugePages_Total|HugePages_Free"

# 7. Check NUMA topology exposed to guest
echo "=== Guest NUMA ==="
numactl --hardware
cat /sys/devices/system/node/node*/meminfo 2>/dev/null | head -20

# 8. Check UEK kernel version
echo "=== UEK Kernel ==="
uname -r
# Expected: 5.15.0-200.131.27.el8uek.x86_64 or newer

# 9. Check kswapd and memory reclaim activity
echo "=== kswapd Activity ==="
ps aux | grep kswapd | grep -v grep
cat /proc/vmstat | grep -E "pgscan|pgsteal|kswapd"

# 10. Check CPU steal time in guest
echo "=== Guest CPU Steal Time ==="
top -b -n1 | grep -E "%Cpu|st"
mpstat -P ALL 1 3 | grep -E "CPU|steal"

# 11. Check OOM killer history
echo "=== OOM Killer Events ==="
dmesg -T | grep -i "oom-killer" | tail -20
journalctl -k --since "24 hours ago" | grep -i "out of memory" | tail -20

# 12. Check Oracle alert log for memory errors
echo "=== Oracle Alert Log (last 100 lines) ==="
tail -100 $ORACLE_BASE/diag/rdbms/$ORACLE_SID/$ORACLE_SID/trace/alert_$ORACLE_SID.log | grep -E "ORA-04031|ORA-4031|KGH"

# 13. Check Oracle memory parameters
echo "=== Oracle Memory Parameters ==="
sqlplus -S / as sysdba <<'EOF'
SET LINESIZE 200
COLUMN name FORMAT A40
COLUMN value FORMAT A30
SELECT name, value
FROM v$parameter
WHERE name IN ('memory_target', 'memory_max_target', 'sga_target', 'sga_max_size', 'pga_aggregate_target', 'use_large_pages', 'large_pages_size')
ORDER BY name;
EXIT;
EOF

# 14. Check HugePages configuration
echo "=== HugePages ==="
grep -i hugepages /etc/sysctl.conf /etc/sysctl.d/*.conf 2>/dev/null
cat /proc/sys/vm/nr_hugepages
cat /proc/sys/vm/nr_overcommit_hugepages

3.3 SQL Diagnostic Queries

-- Run as SYSDBA on the affected instance
SET LINESIZE 200
SET PAGESIZE 100

-- 1. Check SGA memory allocation failures
SELECT component, current_size/1024/1024 AS current_mb,
       min_size/1024/1024 AS min_mb, max_size/1024/1024 AS max_mb
FROM v$sga_dynamic_components
WHERE current_size != max_size;

-- 2. Check PGA memory usage
SELECT name, value/1024/1024 AS value_mb
FROM v$pgastat
WHERE name IN ('aggregate PGA target parameter', 'total PGA allocated', 'maximum PGA allocated');

-- 3. Check memory-related wait events
SELECT event, total_waits, time_waited/100 AS time_waited_sec
FROM v$system_event
WHERE event LIKE '%memory%' OR event LIKE '%KGH%'
ORDER BY time_waited DESC;

-- 4. Check ORA-04031 occurrences in alert log
SELECT COUNT(*) AS ora_04031_count
FROM v$diag_alert_ext
WHERE message_text LIKE '%ORA-04031%'
  AND originating_timestamp > SYSDATE - 7;

-- 5. Check current memory allocation
SELECT * FROM v$memory_dynamic_components
WHERE component IN ('SGA Target', 'PGA Target', 'DEFAULT buffer cache', 'SHARED POOL');

4. Step-by-Step Resolution Runbook

⚠️ CRITICAL SAFETY NOTICE: The following steps modify production systems. Execute in a maintenance window. Always take a full VM snapshot and database backup before proceeding.

Step 0: Safety Checks & Pre-Flight Validation

# 0.1 Verify you have current backups
echo "=== Backup Verification ==="
rman target / <<'EOF'
LIST BACKUP SUMMARY;
EXIT;
EOF

# 0.2 Verify VM snapshot capability
virsh snapshot-list --domain "$VM_NAME" 2>/dev/null || echo "No snapshots exist - create one first"

# 0.3 Check current database status
sqlplus -S / as sysdba <<'EOF'
SELECT instance_name, status, database_status FROM v$instance;
SELECT name, open_mode FROM v$database;
EXIT;
EOF

# 0.4 Verify maintenance window approval
echo "MAINTENANCE WINDOW: $(date)"
echo "APPROVAL REQUIRED: Confirm change ticket # before proceeding"

Step 1: Stop Oracle Database Cleanly

# 1.1 Shut down the database gracefully
sqlplus -S / as sysdba <<'EOF'
SHUTDOWN IMMEDIATE;
EXIT;
EOF

# 1.2 Verify shutdown completed
ps -ef | grep pmon | grep -v grep || echo "PMON not running - database is down"

# 1.3 Stop Oracle listener
lsnrctl stop

Step 2: Configure HugePages on the Guest

# 2.1 Calculate required HugePages based on SGA size
# Formula: HugePages = (SGA_TARGET + 1GB overhead) / HugePage_Size
# Example: SGA_TARGET=64GB, HugePage_Size=2MB
# HugePages = (64 * 1024 + 1024) / 2 = 33280

SGA_TARGET_GB=64
HUGEPAGE_SIZE_MB=2
HUGEPAGES_TOTAL=$(( (SGA_TARGET_GB * 1024 + 1024) / HUGEPAGE_SIZE_MB ))
echo "Calculated HugePages: $HUGEPAGES_TOTAL"

# 2.2 Set HugePages in sysctl
cat >> /etc/sysctl.d/99-oracle-hugepages.conf <<EOF
vm.nr_hugepages = $HUGEPAGES_TOTAL
vm.nr_overcommit_hugepages = 64
vm.hugetlb_shm_group = 54321
EOF

# 2.3 Apply sysctl settings
sysctl -p /etc/sysctl.d/99-oracle-hugepages.conf

# 2.4 Verify HugePages allocation
grep -E "HugePages_Total|HugePages_Free" /proc/meminfo

Step 3: Configure Oracle Database Memory Parameters

-- 3.1 Set memory parameters in spfile
ALTER SYSTEM SET sga_target=64G SCOPE=SPFILE;
ALTER SYSTEM SET sga_max_size=64G SCOPE=SPFILE;
ALTER SYSTEM SET pga_aggregate_target=16G SCOPE=SPFILE;
ALTER SYSTEM SET use_large_pages='ONLY' SCOPE=SPFILE;
ALTER SYSTEM SET large_pages_size='2MB' SCOPE=SPFILE;
ALTER SYSTEM SET memory_target=0 SCOPE=SPFILE;
ALTER SYSTEM SET memory_max_target=0 SCOPE=SPFILE;

-- 3.2 Verify changes
SHOW PARAMETER sga_target;
SHOW PARAMETER use_large_pages;
EXIT;

Step 4: Pin vCPUs to Physical Cores (Host Level)

# 4.1 Identify physical core topology
lscpu -e

# 4.2 Determine optimal vCPU pinning
# Example: 16 vCPUs on a 32-core host with 2 NUMA nodes
# Pin vCPUs 0-7 to cores 0-7 (NUMA node 0)
# Pin vCPUs 8-15 to cores 16-23 (NUMA node 1)

# 4.3 Apply vCPU pinning via virsh
virsh vcpupin "$VM_NAME" 0 0
virsh vcpupin "$VM_NAME" 1 1
virsh vcpupin "$VM_NAME" 2 2
virsh vcpupin "$VM_NAME" 3 3
virsh vcpupin "$VM_NAME" 4 4
virsh vcpupin "$VM_NAME" 5 5
virsh vcpupin "$VM_NAME" 6 6
virsh vcpupin "$VM_NAME" 7 7
virsh vcpupin "$VM_NAME" 8 16
virsh vcpupin "$VM_NAME" 9 17
virsh vcpupin "$VM_NAME" 10 18
virsh vcpupin "$VM_NAME" 11 19
virsh vcpupin "$VM_NAME" 12 20
virsh vcpupin "$VM_NAME" 13 21
virsh vcpupin "$VM_NAME" 14 22
virsh vcpupin "$VM_NAME" 15 23

# 4.4 Verify pinning
virsh vcpupin "$VM_NAME" --live

Step 5: Set Memory Balloon Floor (Host Level)

# 5.1 Set minimum balloon to prevent over-inflation
# This prevents KVM from reclaiming memory below the guest's minimum
virsh qemu-monitor-command "$VM_NAME" --hmp \
  "balloon 131072"  # 128 GB in MB - set to guest's minimum

# 5.2 Disable ballooning entirely (recommended for production)
virsh dumpxml "$VM_NAME" > /tmp/vm_backup.xml

# Edit the XML to set balloon to 'none'
sed -i 's/<memballoon model="virtio">/<memballoon model="none">/' /tmp/vm_backup.xml
virsh define /tmp/vm_backup.xml

# 5.3 Verify balloon configuration
virsh dumpxml "$VM_NAME" | grep -A3 "<memballoon>"

Step 6: Align NUMA Topology (Host Level)

# 6.1 Configure NUMA node mapping in VM XML
# Edit the VM XML to expose NUMA topology matching host
virsh edit "$VM_NAME"

# Add the following to <cpu> section:
# <cpu mode='host-passthrough' check='none'>
#   <numa>
#     <cell id='0' cpus='0-7' memory='68719476736' unit='KiB'/>  <!-- 64 GB -->
#     <cell id='1' cpus='8-15' memory='68719476736' unit='KiB'/>  <!-- 64 GB -->
#   </numa>
# </cpu>

# 6.2 Verify NUMA topology in guest after restart
numactl --hardware

Step 7: Configure cgroup v2 Memory Limits (Host Level)

# 7.1 Create cgroup for the VM
mkdir -p /sys/fs/cgroup/kvm/$VM_NAME

# 7.2 Set memory limit (128 GB = 137438953472 bytes)
echo "137438953472" > /sys/fs/cgroup/kvm/$VM_NAME/memory.max

# 7.3 Set memory swap limit (0 to disable swap for VM)
echo "0" > /sys/fs/cgroup/kvm/$VM_NAME/memory.swap.max

# 7.4 Attach VM processes to cgroup
for pid in $(pgrep -f "qemu.*$VM_NAME"); do
  echo $pid > /sys/fs/cgroup/kvm/$VM_NAME/cgroup.procs
done

# 7.5 Verify cgroup settings
cat /sys/fs/cgroup/kvm/$VM_NAME/memory.max
cat /sys/fs/cgroup/kvm/$VM_NAME/memory.current

Step 8: Restart and Validate

# 8.1 Restart the VM
virsh shutdown "$VM_NAME"
virsh start "$VM_NAME"

# 8.2 Wait for VM to boot and verify
sleep 60
ssh root@"$GUEST_IP" "uname -r && free -h && numactl --hardware"

# 8.3 Start Oracle Database
sqlplus -S / as sysdba <<'EOF'
STARTUP;
SELECT instance_name, status FROM v$instance;
EXIT;
EOF

# 8.4 Verify HugePages are being used
grep -E "HugePages_Total|HugePages_Free" /proc/meminfo
# HugePages_Free should be close to 0 (all allocated to SGA)

# 8.5 Verify no ORA-04031 errors
tail -50 $ORACLE_BASE/diag/rdbms/$ORACLE_SID/$ORACLE_SID/trace/alert_$ORACLE_SID.log | grep -E "ORA-04031" || echo "No memory errors found"

# 8.6 Monitor for 15 minutes
echo "Monitoring for 15 minutes..."
for i in $(seq 1 15); do
  sleep 60
  echo "=== Minute $i ==="
  mpstat -P ALL 1 1 | grep -E "CPU|steal"
  free -h | grep Mem
done

Step 9: Post-Validation & Documentation

# 9.1 Collect final metrics
echo "=== Final Metrics ==="
echo "--- Host ---"
virsh vcpupin "$VM_NAME" --live
virsh dommemstat "$VM_NAME" --live
echo "--- Guest ---"
ssh root@"$GUEST_IP" "free -h && cat /proc/meminfo | grep HugePages && mpstat -P ALL 1 1 | grep steal"

# 9.2 Document changes
cat > /tmp/memory_fix_changes.txt <<EOF
CHANGE TICKET: [TICKET_NUMBER]
DATE: $(date)
CHANGES APPLIED:
1. HugePages configured: $HUGEPAGES_TOTAL pages of 2MB
2. Oracle memory parameters: SGA=64G, PGA=16G, use_large_pages=ONLY
3. vCPU pinning: vCPUs 0-7 → cores 0-7, vCPUs 8-15 → cores 16-23
4. Memory balloon: disabled (model=none)
5. NUMA topology: 2 cells, 64GB each
6. cgroup v2 memory limit: 128GB, swap disabled
VERIFIED BY: [DBA_NAME]
EOF

# 9.3 Update runbook documentation
echo "Update your enterprise runbook with these changes."

5. Troubleshooting Quick Reference

Common Failure Signatures

Key Metrics to Monitor

📚 Official Documentation & Technical References

Oracle Documentation

• Oracle Linux 8 UEK Documentation: Oracle Linux 8 Unbreakable Enterprise Kernel (UEK) Release Notes
• Oracle Database 19c Memory Architecture: Oracle Database Concepts - Memory Architecture
• Oracle Database 23ai Memory Management: Oracle Database 23ai - Managing Memory
• Oracle Linux 8 Documentation: Oracle Linux 8 Documentation

KVM / Virtualization References

• Red Hat KVM Virtualization Tuning Guide: KVM Virtualization Tuning and Optimization Guide
• Kernel Virtual Machine (KVM) - Memory Ballooning: libvirt Domain XML - Memory Balloon

6. Need Expert Assistance?

Memory starvation issues in virtualized Oracle Database environments can be subtle and environment-specific. If you’re experiencing persistent ORA-04031 errors, unexplained CPU steal time, or OOM-killer events in your Oracle Linux 8 UEK KVM guests, our team of certified Oracle and Linux engineers can help.

• Contact Our Team — Get a free initial assessment of your virtualized Oracle Database environment
• View Our Services — Explore our database performance tuning, virtualization optimization, and 24/7 production support offerings
• Emergency Support — 24/7 emergency response for production outages

DBPros.Net — Enterprise Database & Systems Excellence Since 2008.