Oracle RAC for ERP: Architecture, Installation & Rolling Maintenance

1. Overview & Executive Summary

Oracle Real Application Clusters (RAC) provides the foundation for high availability (HA) and horizontal scalability across enterprise ERP environments running SAP, Oracle EBS, PeopleSoft, and Ellucian Banner. However, improper node placement, interconnect misconfigurations, and uncoordinated maintenance can turn a multi-node cluster into a single point of failure under peak transaction loads.

This guide details the end-to-end operational runbook for deploying, isolating, and maintaining Oracle RAC clusters for high-volume enterprise applications. It covers interconnect jumbo frame validation, SCAN VIP management, Application Continuity (AC) implementation, service-level workload isolation, and rolling maintenance via opatchauto.

Process Flow

---

2. Diagnostic Checklist

Run these diagnostic commands from any cluster node to verify Grid Infrastructure (GI), RAC interconnect, and service health before initiating changes.

2.1 Clusterware Health & Service Status

# Check overall Grid Infrastructure stack status
crsctl check crs

# Verify status of all cluster resources across all nodes
crsctl status resource -t

# Verify SCAN VIP and Listener status
srvctl status scan
srvctl status scan_listener

Expected output: All CRS services (crsd, ocssd, evmd) showing ONLINE on all nodes; SCAN listeners active on their assigned nodes.

2.2 Private Interconnect & Jumbo Frame MTU Validation

# Verify private network interface configuration in Grid Infrastructure
oifcfg getif

# Test private interconnect MTU (Jumbo Frames 9000 bytes) between Node 1 and Node 2
ping -I eth1 -M do -s 8972 node2-priv

Expected output: 0% packet loss. If ping fails with Frag needed and DF set, MTU 9000 is not consistently configured across OS interfaces and network switches.

2.3 OCR & Voting Disk Integrity

# Verify Voting Disk location and status
crsctl query css votedisk

# Check Oracle Cluster Registry (OCR) integrity
ocrcheck

Expected output: Voting disks show ONLINE; ocrcheck reports Validation succeeded.

2.4 RAC Database Services & Node Alignment

# Check ERP database services allocation across nodes
srvctl status service -db erpdb

# Inspect current service configuration
srvctl config service -db erpdb

Expected output: Preferred and available nodes match your planned ERP workload isolation topology (e.g., OLTP preferred on Nodes 1-2, Batch/Reporting preferred on Node 3).

---

3. Step-by-Step Resolution Runbook

Step 0: Safety Checks

⚠️ CRITICAL: Confirm the following before initiating any maintenance:

1. Verify RMAN Catalog & Controlfile Autobackup:

rman target /
RMAN> SHOW ALL;

2. Confirm Cluster Health: Ensure crsctl check crs returns clean on all nodes.
3. Verify Application Continuity is configured on client connection strings so sessions drain transparently during maintenance.

---

Step 1: ERP Service Isolation & Workload Management

Mixing high-concurrency OLTP queries with batch reporting on the same RAC instance leads to Cache Fusion block pinging (gc cr block busy, gc buffer busy acquire). Isolate application tiers using dedicated database services created via SRVCTL.

# 1. Create OLTP service preferred on Node 1 & Node 2, available on Node 3
srvctl add service -db erpdb -service erp_oltp \
  -preferred erpdb1,erpdb2 -available erpdb3 \
  -tafpolicy BASIC -failovertype TRANSACTION -failovermethod BASIC \
  -clbgoal SHORT -rlbgoal SERVICE_TIME

# 2. Create Batch/Job Submitter service preferred on Node 3
srvctl add service -db erpdb -service erp_batch \
  -preferred erpdb3 -available erpdb1,erpdb2 \
  -clbgoal LONG -rlbgoal THROUGHPUT

# 3. Start the services
srvctl start service -db erpdb -service erp_oltp
srvctl start service -db erpdb -service erp_batch

---

Step 2: Configure Application Continuity via SRVCTL

For Clusterware-managed RAC databases, service properties must be configured using srvctl modify service rather than DBMS_SERVICE PL/SQL calls. Application Continuity enables in-flight transactions to replay on surviving RAC nodes without returning session errors (ORA-03113, ORA-03135) during node maintenance.

# Configure Application Continuity attributes on the Clusterware-managed service
srvctl modify service -db erpdb -service erp_oltp \
  -failovertype TRANSACTION \
  -failover_restore LEVEL1 \
  -commit_outcome TRUE \
  -failoverretry 30 \
  -failoverdelay 1

Verify service configuration in Grid Infrastructure:

srvctl config service -db erpdb -service erp_oltp

Expected output: Shows Failover type: TRANSACTION, Failover restore: LEVEL1, and Commit outcome: TRUE.

---

Step 3: Execute Rolling Maintenance with OPatchAuto

opatchauto automates rolling Release Update (RU) patching. It stops CRS on the target node, drains database services, applies Grid Infrastructure and Database binary patches, relinks binaries, and restarts the stack without downtime for the overall cluster.

# Log in as root on Node 1
sudo su -

# 3.1 Pre-check the patch bundle against GI and DB homes
$ORACLE_HOME/OPatch/opatchauto apply /u01/app/patches/<PATCH_NUMBER> -analyze

# 3.2 Execute rolling patch application on Node 1 only
$ORACLE_HOME/OPatch/opatchauto apply /u01/app/patches/<PATCH_NUMBER> -oh $ORACLE_HOME

# 3.3 Verify CRS restarted cleanly on Node 1
crsctl check crs
crsctl status resource -t

Repeat Step 3 on Node 2 and Node 3 sequentially after verifying Node 1 has re-joined the cluster and resumed service handling.

---

Step 4: Post-Maintenance SQL Datapatch & Verification

After all nodes have been patched rolling:

# Run datapatch from Node 1 to apply SQL catalog scripts
cd $ORACLE_HOME/OPatch
./datapatch -verbose

Verify patch status in the database dictionary:

SELECT patch_id, version, action, status, description
FROM dba_registry_sqlpatch
ORDER BY action_time DESC;

Expected output: STATUS = 'SUCCESS' for the latest patch ID across all containers.

---

📚 Official Documentation & Technical References

Oracle Documentation

• Oracle Real Application Clusters Administration and Deployment Guide, 23ai
• Oracle Clusterware Administration and Deployment Guide, 23ai
• Oracle Database High Availability Overview and Best Practices

My Oracle Support

For official My Oracle Support RAC architecture and OPatchAuto troubleshooting guidelines, consult My Oracle Support:

• Doc ID 2485457.1: AutoUpgrade Tool Instructions and Download

Related Resources

• Oracle Maximum Availability Architecture (MAA) Best Practices for RAC

---

Need Expert Help?

Designing, scaling, and maintaining Oracle RAC clusters for mission-critical enterprise applications requires specialized architecture experience. DBPros.Net’s certified Oracle Certified Masters and Principal Engineers can help you:

• Audit & Optimize RAC interconnect performance, ASM storage, and Cache Fusion contention
• Design & Implement Application Continuity (AC) and workload-isolated service topologies
• Execute zero-downtime rolling OPatchAuto and major-version RAC upgrades
• Harden Grid Infrastructure security and clusterware resilience 24/7/365

Contact DBPros.Net Today | Explore Our Enterprise Services