Linux Kernel I/O Scheduler Tuning for High-IOPS Database Workloads (mq-deadline vs. kyber vs. none)

Executive Summary

Modern Linux kernels (5.0+) implement the multi-queue block layer (blk-mq), which replaced the legacy single-queue I/O stack. The old noop, deadline, and cfq schedulers are gone, replaced by three production schedulers: mq-deadline, kyber, and none. For enterprise database workloads — particularly Oracle Database on NVMe or SSD — the I/O scheduler selection directly impacts latency, throughput, CPU overhead, and IOPS stability.

The single most common root cause of unexplained database I/O latency spikes on flash storage is the kernel I/O scheduler performing unnecessary request merging, reordering, and dispatch throttling on devices that already manage deep native queues. Oracle’s async I/O (libaio / io_uring) and direct I/O (FILESYSTEMIO_OPTIONS=setall) bypass the page cache and expect the block layer to be a low-overhead pass-through. When mq-deadline or kyber sits in the path, it can add 100–500 µs of latency per I/O and cause CPU spin under high IOPS.

This article provides a diagnostic checklist, a step-by-step resolution runbook, and validation guidance using fio, iostat, and Oracle AWR / v$filestat so you can confidently select, apply, and persist the correct scheduler for your storage tier.

1. Overview — I/O Scheduler Selection Process

The following process flow summarizes the end-to-end workflow covered in this guide. Each step is detailed in the sections that follow.

Scheduler Comparison at a Glance

Scheduler	Best For	Core Mechanism	Database Workload Verdict
mq-deadline	HDD / rotational, mixed workloads	Per-request deadline (read 500 ms, write 5 s), read-priority dispatch, request merging	Default for HDD-backed Oracle; safe but adds overhead on flash
kyber	NVMe/SSD with mixed latency-sensitive traffic	Token-bucket queue-depth control per priority class (read 2 ms, write 10 ms targets)	Good middle ground for flash with mixed OLTP + batch; higher CPU than none
none	NVMe/SSD high-IOPS, Oracle direct/async I/O	Pass-through — no merging, no reordering, no dispatch throttling	Recommended for Oracle on flash/NVMe — lowest latency, lowest CPU

Key Tunables

Scheduler	Sysfs Tunable	Default	Recommendation
mq-deadline	deadline_read_expire	500 ms	Leave at default unless read latency spikes
mq-deadline	deadline_write_expire	5000 ms	Leave at default; do not lower below 1000 ms
mq-deadline	deadline_writes_starved	2	Increase to 4 if writes starve reads
mq-deadline	fifo_batch	16	Lower to 8 for latency-sensitive OLTP
kyber	read_lat_nsec	2000000 (2 ms)	Lower to 1000000 for strict read SLA
kyber	write_lat_nsec	10000000 (10 ms)	Lower to 5000000 for redo-log-like writes
none	—	—	No tunables; device handles queuing

---

Diagnostic Checklist

Run these commands before making any change. Record all output — you will need it for the rollback plan and for before/after comparison.

1. Identify Storage Type

# Rotational flag: 0 = SSD/NVMe, 1 = HDD
lsblk -d -o NAME,ROTA,SCHED,MODEL,SIZE

# Show full topology with mount points
lsblk -o NAME,ROTA,SCHED,TYPE,MOUNTPOINT

# NVMe-specific inventory
lspci | grep -i nvme
nvme list 2>/dev/null || true

# Check if device is SAN/array-backed (virtio, mpath, etc.)
lsblk -d -o NAME,TRAN,MODEL

2. Inventory Current Scheduler Settings

# Show scheduler for every block device (brackets = active)
for d in /sys/block/*/queue/scheduler; do
  dev=$(basename "$(dirname "$(dirname "$d")")")
  printf "%-12s %s\n" "$dev" "$(cat "$d")"
done

# Queue depth and request limits
for q in /sys/block/*/queue/nr_requests /sys/block/*/queue/max_sectors_kb; do
  echo "$q = $(cat "$q")"
done

# Kernel and distro version
uname -r
cat /etc/os-release | head -4

3. Capture I/O Latency and Throughput Baseline

# Device-level stats (watch avgqu-sz, await, %util)
iostat -x -m 1 /dev/sda /dev/nvme0n1

# fio random read — 4k, high queue depth (OLTP data file pattern)
# WARNING: Use a dedicated test device or a file on a test filesystem.
fio --name=randread --ioengine=libaio --rw=randread --bs=4k \
    --size=4G --numjobs=16 --iodepth=64 --runtime=60 --time_based \
    --direct=1 --filename=/dev/nvme0n1 --group_reporting

# fio random write — 8k, moderate queue depth (redo log pattern)
fio --name=randwrite --ioengine=libaio --rw=randwrite --bs=8k \
    --size=4G --numjobs=4 --iodepth=32 --runtime=60 --time_based \
    --direct=1 --filename=/dev/nvme0n1 --group_reporting

4. Oracle-Specific I/O Health Check

-- Average read/write latency per datafile (ms) — run during peak load
SELECT file#,
       phyrds,
       phywrts,
       ROUND(readtim  / NULLIF(phyrds, 0), 2) AS avg_read_ms,
       ROUND(writetim / NULLIF(phywrts, 0), 2) AS avg_write_ms,
       ROUND(avgiotime, 2)                     AS avg_io_ms
FROM   v$filestat
ORDER  BY avg_io_ms DESC
FETCH FIRST 20 ROWS ONLY;

-- I/O calibration (run during a quiet window; requires CALIBRATE_IO privilege)
SET SERVEROUTPUT ON
DECLARE
  lat  NUMBER;
  iops NUMBER;
  mbps NUMBER;
BEGIN
  DBMS_RESOURCE_MANAGER.CALIBRATE_IO(
    num_physical_disks => 8,
    max_latency        => 20,
    max_iops           => iops,
    max_mbps           => mbps,
    actual_latency     => lat
  );
  DBMS_OUTPUT.PUT_LINE('Max IOPS: ' || iops);
  DBMS_OUTPUT.PUT_LINE('Max MBPS: ' || mbps);
  DBMS_OUTPUT.PUT_LINE('Latency:  ' || lat || ' ms');
END;
/

-- Confirm async I/O is in use (should be > 0)
SELECT name, value
FROM   v$sysstat
WHERE  name LIKE '%async I/O%';

5. Decision Matrix

Storage Type	Oracle I/O Mode	Recommended Scheduler
NVMe (local or array)	direct + async	none
SATA/SAS SSD	direct + async	none
HDD / rotational	any	mq-deadline
SAN / array with battery-backed cache	any	none (array handles ordering)
Virtualized (VMware/KVM) paravirtual device	any	none (hypervisor handles queuing)
Mixed flash with strict read SLA + batch writes	direct + async	kyber (tuned read_lat_nsec)

---

Step-by-Step Resolution Runbook

Step 0: Safety Checks

1. Confirm change window and approval — scheduler changes are low-risk but can cause a brief I/O pause on busy devices.
2. Verify backups — ensure RMAN backup or storage snapshot exists for all Oracle data files, control files, and redo logs.
3. Test on non-production first — replicate the workload on a staging host with identical kernel and storage.
4. Record current state — save the output of every command in the Diagnostic Checklist to a file:

   ./diagnostic_checklist.sh > /tmp/io_scheduler_baseline_$(date +%F).log

5. Ensure out-of-band console access — if the device is the root/boot device and you are remote-only, confirm IPMI/iDRAC/console access before changing the scheduler.
6. Do not change the scheduler on the root device of a remote-only server without console access — a misconfiguration can prevent boot.

Step 1: Identify Storage Type

lsblk -d -o NAME,ROTA,SCHED,MODEL,TRAN

• ROTA=0 → flash/NVMe → target scheduler is none (or kyber if latency tuning is needed).
• ROTA=1 → HDD → target scheduler is mq-deadline.
• TRAN=mpath or TRAN=virtio → array/hypervisor-managed → none.

Step 2: Capture Baseline

Run the full Diagnostic Checklist (Section 3 and 4) and store the output. Generate an AWR report for the current peak hour:

-- Generate AWR snapshot pair and report
EXEC DBMS_WORKLOAD_REPOSITORY.CREATE_SNAPSHOT();
-- Wait 60 minutes of peak load, then:
EXEC DBMS_WORKLOAD_REPOSITORY.CREATE_SNAPSHOT();

-- Use awrrpt.sql to produce the report
@?/rdbms/admin/awrrpt.sql

Step 3: Select the Scheduler

Apply the Decision Matrix from the Diagnostic Checklist. For the vast majority of Oracle-on-flash deployments, the answer is none.

Step 4: Apply the Runtime Change

# NVMe device
echo none > /sys/block/nvme0n1/queue/scheduler

# SATA/SAS SSD
echo none > /sys/block/sda/queue/scheduler

# HDD (rotational)
echo mq-deadline > /sys/block/sda/queue/scheduler

# Verify the active scheduler (brackets show the active one)
cat /sys/block/nvme0n1/queue/scheduler
# Example output: [none] mq-deadline kyber

Note: The change is immediate and affects all I/O on that device. Oracle does not need a restart — subsequent I/Os use the new scheduler.

Step 5: Persist the Setting via udev

Create /etc/udev/rules.d/60-io-scheduler.rules:

# NVMe devices -> none
ACTION=="add|change", KERNEL=="nvme[0-9]*n[0-9]*", ATTR{queue/scheduler}="none"

# SCSI/SATA SSDs (non-rotational) -> none
ACTION=="add|change", KERNEL=="sd[a-z]+", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="none"

# Rotational disks -> mq-deadline
ACTION=="add|change", KERNEL=="sd[a-z]+", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="mq-deadline"

# Device-mapper multipath -> none
ACTION=="add|change", KERNEL=="dm-*", ATTR{queue/scheduler}="none"

Reload and trigger:

udevadm control --reload-rules
udevadm trigger

Alternative — kernel boot parameter:

# /etc/default/grub
GRUB_CMDLINE_LINUX="... elevator=none"

# Regenerate grub config
grub2-mkconfig -o /boot/grub2/grub.cfg   # RHEL/Rocky
# or
update-grub                                  # Debian/Ubuntu

Caution: elevator=none applies to all block devices, including HDDs. Use the udev rule approach if you have mixed storage.

Step 6: Validate the Change

Re-run the same fio benchmarks and Oracle queries from the baseline:

# Re-run fio (same parameters as baseline)
fio --name=randread --ioengine=libaio --rw=randread --bs=4k \
    --size=4G --numjobs=16 --iodepth=64 --runtime=60 --time_based \
    --direct=1 --filename=/dev/nvme0n1 --group_reporting

-- Re-check file latency
SELECT file#,
       ROUND(readtim  / NULLIF(phyrds, 0), 2) AS avg_read_ms,
       ROUND(writetim / NULLIF(phywrts, 0), 2) AS avg_write_ms
FROM   v$filestat
ORDER  BY avg_read_ms DESC;

-- Compare AWR top wait events (expect lower 'db file sequential read' / 'log file parallel write')
SELECT event, total_waits, time_waited_micro
FROM   v$system_event
WHERE  event IN ('db file sequential read', 'db file scattered read',
                 'log file parallel write', 'log file sync')
ORDER  BY time_waited_micro DESC;

Success criteria:

• Average read latency in v$filestat is equal or lower than baseline.
• fio 4k random read IOPS is equal or higher; latency (clat p99) is equal or lower.
• CPU %iowait and %system (kernel time) are reduced.
• No increase in log file sync wait times.

Step 7: Monitor and Rollback Plan

• Monitor for 7 days — watch iostat -x 1, v$filestat, and AWR snapshots.
• Rollback trigger — if average read latency increases by more than 20% or log file sync degrades, revert immediately.
• Rollback procedure:

# Restore previous scheduler (e.g., mq-deadline)
echo mq-deadline > /sys/block/nvme0n1/queue/scheduler

# Remove or comment out the udev rule
vi /etc/udev/rules.d/60-io-scheduler.rules
udevadm control --reload-rules
udevadm trigger

---

📚 Official Documentation & Technical References

Oracle Documentation

• Oracle Database 19c Performance Tuning Guide
• Oracle Database 23ai Reference — V$FILESTAT

My Oracle Support

For the latest Oracle-specific I/O scheduler recommendations and I/O calibration guidance, search My Oracle Support for relevant knowledge documents. MOS Doc IDs are subject to change; verify current documentation via the MOS search interface.

Related Resources

• Linux Kernel Documentation — blk-mq sysfs interface
• Kernel source: block/mq-deadline.c, block/kyber-iosched.c

---

Need Expert Help?

I/O scheduler misconfiguration can silently degrade database performance, cause erratic latency spikes, or increase I/O wait times under storage stress. DBPros.Net’s enterprise database engineers can audit your Linux I/O stack, Oracle AWR metrics, and storage configuration to right-size your scheduler, queue depth, and filesystem I/O options.

• Contact our team
• Explore database services

DBPros.Net — Enterprise Database & Systems Expertise.