Oracle RAC 19c Rolling OPatchAuto Upgrades Without Node Downtime

Executive Summary

Oracle RAC environments can be patched with OPatchAuto in rolling mode, allowing each node to be patched sequentially while the rest of the cluster continues serving traffic. The critical discipline is to treat the Grid Infrastructure home and the RAC database home as separate patching units, validate patch applicability before starting, and never interrupt OPatchAuto once node cycling has begun.

This guide is written for enterprise DBAs and database engineers who need a repeatable, low-risk process for applying Oracle 19c Release Updates, Release Update Revisions, and selected one-off patches to RAC environments.

Overview

OPatchAuto automates the rolling patch workflow by stopping only the resources on one node at a time. For a RAC database, this means an individual instance or clusterware stack is briefly restarted, but the database as a whole stays available. Applications that use connection pools, Fast Application Notification, and service failover will see a short service drain and then recover automatically.

The overall process for a RAC 19c rolling patch is:

Rolling mode does not mean a node will never be restarted. It means there is no full-cluster, all-nodes-at-once outage. Each node will be briefly unavailable during its patch window, while the remaining nodes carry the workload.

Diagnostic Checklist

Run these checks before starting any patch work. This gives you a baseline to compare against after the patch is complete.

# 1. Confirm cluster and RAC health
crsctl status resource -t
crsctl check cluster -all
crsctl check crs

# 2. Check database instance and service status
srvctl status database -d PROD
srvctl status service -d PROD

# 3. Verify active version and OCR integrity
crsctl query crs activeversion
ocrcheck

# 4. Review OPatch version and inventory
$GI_HOME/OPatch/opatch version
$ORACLE_HOME/OPatch/opatch version
$ORACLE_HOME/OPatch/opatch lsinventory -detail

# 5. Check patch applicability and conflicts
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -ph /u01/stage/<patch_id>
$ORACLE_HOME/OPatch/opatch prereq CheckApplicable -ph /u01/stage/<patch_id>

Run the following SQL from any RAC instance to establish the database baseline:

sqlplus -S / as sysdba <<'SQL'
SET PAGESIZE 100
SET LINESIZE 200

COLUMN instance_name FORMAT A15

SELECT inst_id, instance_name, status
FROM   gv$instance
ORDER  BY inst_id;

SELECT name, open_mode, database_role
FROM   v$database;

SELECT inst_id, service_name, COUNT(*)
FROM   gv$session
GROUP  BY inst_id, service_name
ORDER  BY inst_id, service_name;
SQL

Step-by-Step Resolution Runbook

Step 0: Safety Checks

Do not start the patch until every item below is confirmed.

1. All nodes must be online and healthy.
   Fix any down node, offline resource, or reconfiguration issue before patching.

2. No conflicting maintenance should be running.
   Avoid online DDL, backup jobs, Data Guard switchovers, and large batch jobs during the patch window.

3. Capture the current configuration.

   crsctl status resource -t
   srvctl config database -d PROD
   srvctl status database -d PROD

4. Verify free disk space.
   OPatchAuto creates backup and log files in the Oracle home and staging location.

   df -h $GI_HOME $ORACLE_HOME /u01/stage

5. Verify OPatch version.
   The OPatch binary must meet the minimum version required by the patch README.

6. Confirm the patch is rolling-patchable.
   Review the patch README and run opatch prereq CheckApplicable. If the patch is not rolling-patchable, OPatchAuto will not be able to apply it without a full outage.

7. Have a rollback plan.
   The primary rollback path is opatchauto rollback. Do not rely on manual file copies of a Grid Infrastructure home as your rollback strategy.

Step 1: Stage and Validate the Patch

Download the patch from Oracle Support and extract it to the same path on every RAC node. Identical staging paths avoid confusion and make OPatchAuto log analysis easier.

# Example staging path
mkdir -p /u01/stage
unzip -o /u01/stage/<patch_id>.zip -d /u01/stage

Confirm the patch directory is present on every node:

for node in rac1 rac2 rac3; do
  ssh ${node} "ls -l /u01/stage/<patch_id>"
done

Run patch conflict and applicability checks on every node. A node that passes prerequisites on one RAC member may fail on another if the local inventory differs.

$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -ph /u01/stage/<patch_id>
$ORACLE_HOME/OPatch/opatch prereq CheckApplicable -ph /u01/stage/<patch_id>

Step 2: Patch the Grid Infrastructure Home

Run OPatchAuto against the Grid Infrastructure home first. Patching GI after the database home can create version mismatches between Oracle Clusterware and database instances.

Set the environment for the grid home and run:

export GI_HOME=/u01/app/19.0.0/grid
export GI_BASE=/u01/app/grid
export PATH=$GI_HOME/bin:$PATH

$GI_HOME/OPatch/opatchauto apply /u01/stage/<patch_id> -oh $GI_HOME

OPatchAuto will:

• Detect all RAC nodes.
• Stop the clusterware stack on one node.
• Apply the patch to the GI home on that node.
• Execute root scripts.
• Restart the clusterware stack.
• Move to the next node.

Do not interrupt this process. Monitor the logs in a separate session:

tail -f $GI_HOME/cfgtoollogs/opatchauto/opatchauto-$(date +%Y-%m-%d)*.log

If OPatchAuto prompts for root execution, ensure the invoking user has the required sudo or root access. The utility manages root scripts automatically; you should not need to run them manually.

Step 3: Validate the Grid Infrastructure Home

After the GI patch completes, verify every node is back online and the patch inventory is consistent.

$GI_HOME/OPatch/opatch lsinventory -patch_level
crsctl query crs activeversion
crsctl status resource -t
ocrcheck
srvctl status asm -a

Confirm that:

• All cluster resources are ONLINE.
• ASM instances are running on all nodes.
• The OCR integrity check reports no errors.
• Every node shows the same patch ID in opatch lsinventory.

Step 4: Patch the RAC Database Home

Once GI is validated, patch the RAC database home.

Set the environment for the database home and run:

export ORACLE_HOME=/u01/app/oracle/product/19.0.0/dbhome_1
export ORACLE_BASE=/u01/app/oracle
export PATH=$ORACLE_HOME/bin:$PATH

$ORACLE_HOME/OPatch/opatchauto apply /u01/stage/<patch_id> -oh $ORACLE_HOME

For a RAC database home, OPatchAuto uses rolling mode by default. It will:

• Stop one database instance at a time.
• Apply the patch to the database home binaries.
• Start the instance.
• Move to the next instance.

The database remains open for the entire operation. Services should fail over to surviving instances if they are configured correctly.

Step 5: Post-Patch Database Validation

After the database home patch completes, run the following checks.

$ORACLE_HOME/OPatch/opatch lsinventory -patch_level
srvctl status database -d PROD
srvctl status service -d PROD

Then verify the database from SQL:

sqlplus -S / as sysdba <<'SQL'
SET PAGESIZE 100
SET LINESIZE 200

COLUMN instance_name FORMAT A15

SELECT inst_id, instance_name, status
FROM   gv$instance
ORDER  BY inst_id;

SELECT name, open_mode, database_role
FROM   v$database;

SELECT inst_id, service_name, COUNT(*)
FROM   gv$session
GROUP  BY inst_id, service_name
ORDER  BY inst_id, service_name;
SQL

Also check the alert log for each instance:

tail -n 200 $ORACLE_BASE/diag/rdbms/<db_unique_name>/<instance_name>/trace/alert_<instance_name>.log

Look for ORA-00600, ORA-07445, or unexpected ORA- errors after instance restart.

Step 6: Rollback and Recovery

If a patch fails or must be reverted, use OPatchAuto rollback. Rollback also runs in rolling mode for RAC homes.

# Roll back the GI home patch
$GI_HOME/OPatch/opatchauto rollback <patch_id> -oh $GI_HOME

# Roll back the database home patch
$ORACLE_HOME/OPatch/opatchauto rollback <patch_id> -oh $ORACLE_HOME

If the apply or rollback process is interrupted, resume it instead of starting a second operation:

$ORACLE_HOME/OPatch/opatchauto resume

Do not run opatch manually in the same home while opatchauto is running. Manual OPatch commands can corrupt the Oracle inventory and make recovery significantly more difficult.

Operational Notes

• Patch the GI home before the RAC database home.
• Do not run OPatchAuto against the GI and database homes at the same time.
• Use a maintenance window even though the process is rolling. Unexpected issues still need time to resolve.
• Use FAN/ONS and connection pooling so application sessions drain cleanly when an instance is stopped.
• Keep OPatchAuto logs from every node for audit and Oracle Support troubleshooting.
• If Oracle Support confirms a patch is not rolling-patchable, schedule a full outage and use the documented non-rolling procedure instead of forcing it.

📚 Official Documentation & Technical References

• Oracle Database 19c Documentation
• Oracle OPatchAuto & Database Patching Guide
• Oracle Clusterware Administration and Deployment Guide
• Oracle RAC Administrator’s Guide
• Oracle Database Upgrade Guide

---

Next Steps

Need help planning, validating, or executing an Oracle RAC 19c rolling upgrade? Contact DBPros.Net or explore our database services for hands-on support.