Automated SSL/TLS Certificate Lifecycle Management with Custom Internal CAs for Air-Gapped DB Clusters

1. Overview

In air-gapped database clusters—where no external network connectivity exists—managing SSL/TLS certificates for Oracle Database instances is a critical yet often manual and error-prone task. Without a public CA, enterprises must operate their own internal Certificate Authority (CA) to issue, renew, and revoke certificates. Manual processes lead to expired certificates, service outages, and security compliance gaps. This article presents a production-ready approach to automate the entire certificate lifecycle using a custom internal CA, OpenSSL, Oracle Wallets, and scheduled scripts. We cover the architecture, step-by-step implementation, diagnostic checks, and troubleshooting for Oracle Database 19c and 23ai environments. By the end, you will have a resilient, automated system that ensures your database connections remain secure and uninterrupted.

2. Diagnostic Checklist

Before implementing automation, assess the current state of your certificate infrastructure. Use the following commands to gather critical information.

2.1 Check Oracle Wallet Contents

# List certificates in the wallet (requires orapki)
orapki wallet display -wallet /u01/app/oracle/wallet/root -pwd <wallet_password>

2.2 Verify Certificate Expiry

# Extract certificate details from wallet
orapki wallet display -wallet /u01/app/oracle/wallet/root -pwd <wallet_password> -cert | openssl x509 -noout -dates

2.3 Check Database Listener SSL Configuration

# View listener.ora SSL settings
lsnrctl status <listener_name>

2.4 Validate Current SSL Connections

-- Query v$session to see if SSL is in use
SELECT COUNT(*) FROM v$session WHERE network_service_banner LIKE '%SSL%';

2.5 Inspect Alert Log for SSL Errors

# Search for SSL-related errors in the alert log
grep -i "SSL\|TLS\|wallet" $ORACLE_BASE/diag/rdbms/*/*/trace/alert_*.log

If any certificate is expired or nearing expiry, proceed to the resolution runbook.

3. Step-by-Step Resolution Runbook

Step 0: Safety Checks

• Backup all existing wallets and configuration files (wallet, sqlnet.ora, listener.ora).
• Verify you have root or appropriate OS privileges on all nodes.
• Document the current certificate serial numbers and expiry dates for rollback.
• Test the automation scripts in a non-production environment first.
• Ensure the CA host is physically and logically isolated from external networks.

Step 1: Set Up the Internal Certificate Authority

On a dedicated, air-gapped host (e.g., ca-host), create a root CA and an intermediate CA using OpenSSL.

# Create directory structure
mkdir -p /opt/ca/{root,intermediate}/{certs,crl,csr,newcerts,private}
chmod 700 /opt/ca/root/private /opt/ca/intermediate/private

# Generate root CA private key (RSA 4096)
openssl genrsa -aes256 -out /opt/ca/root/private/ca.key.pem 4096

# Generate root CA certificate (valid 10 years)
openssl req -x509 -new -key /opt/ca/root/private/ca.key.pem \
  -days 3650 -sha256 -out /opt/ca/root/certs/ca.cert.pem \
  -subj "/C=US/O=MyOrg/CN=MyOrg Root CA"

# Generate intermediate CA key
openssl genrsa -aes256 -out /opt/ca/intermediate/private/intermediate.key.pem 4096

# Generate intermediate CA CSR
openssl req -new -key /opt/ca/intermediate/private/intermediate.key.pem \
  -out /opt/ca/intermediate/csr/intermediate.csr.pem \
  -subj "/C=US/O=MyOrg/CN=MyOrg Intermediate CA"

# Sign intermediate CA with root CA
openssl ca -config /opt/ca/root/openssl.cnf -extensions v3_intermediate_ca \
  -days 3650 -notext -md sha256 \
  -in /opt/ca/intermediate/csr/intermediate.csr.pem \
  -out /opt/ca/intermediate/certs/intermediate.cert.pem

Create the OpenSSL configuration files (openssl.cnf) for both root and intermediate CAs, specifying appropriate basicConstraints, keyUsage, and crlDistributionPoints.

Step 2: Generate Server and Client Certificates

For each database node, generate a server certificate with the node’s hostname and IP in the SAN.

# On CA host, generate a private key for the DB server
openssl genrsa -out /opt/ca/intermediate/private/db1.example.com.key.pem 2048

# Create a CSR
openssl req -new -key /opt/ca/intermediate/private/db1.example.com.key.pem \
  -out /opt/ca/intermediate/csr/db1.example.com.csr.pem \
  -subj "/C=US/O=MyOrg/CN=db1.example.com"

# Sign the certificate with the intermediate CA
openssl ca -config /opt/ca/intermediate/openssl.cnf \
  -extensions server_cert -days 365 -notext -md sha256 \
  -in /opt/ca/intermediate/csr/db1.example.com.csr.pem \
  -out /opt/ca/intermediate/certs/db1.example.com.cert.pem

Repeat for client certificates (e.g., for application users) with appropriate extensions.

Step 3: Deploy Certificates to Oracle Wallets

Transfer the certificate, private key, and CA chain to the database host. Then create or update the Oracle wallet.

# On DB host, create a new wallet (or use existing)
orapki wallet create -wallet /u01/app/oracle/wallet/root -pwd <wallet_password> -auto_login

# Add the server certificate and private key
orapki wallet add -wallet /u01/app/oracle/wallet/root -pwd <wallet_password> \
  -cert /tmp/db1.example.com.cert.pem -key /tmp/db1.example.com.key.pem

# Add the intermediate CA certificate (trusted)
orapki wallet add -wallet /u01/app/oracle/wallet/root -pwd <wallet_password> \
  -trusted_cert -cert /tmp/intermediate.cert.pem

# Add the root CA certificate
orapki wallet add -wallet /u01/app/oracle/wallet/root -pwd <wallet_password> \
  -trusted_cert -cert /tmp/ca.cert.pem

Update sqlnet.ora to point to the wallet:

WALLET_LOCATION = (SOURCE = (METHOD = FILE) (METHOD_DATA = (DIRECTORY = /u01/app/oracle/wallet/root)))
SSL_CLIENT_AUTHENTICATION = TRUE
SSL_CIPHER_SUITES = (TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384)

Restart the listener and database to apply changes.

Step 4: Automate Certificate Renewal

Create a renewal script that checks certificate expiry and, if within a threshold (e.g., 30 days), generates a new certificate and updates the wallet.

#!/bin/bash
# /opt/scripts/renew_cert.sh
WALLET_DIR="/u01/app/oracle/wallet/root"
WALLET_PWD="<wallet_password>"
CA_HOST="ca-host"
CERT_DAYS_THRESHOLD=30

# Get current certificate expiry
CERT_EXPIRY=$(orapki wallet display -wallet $WALLET_DIR -pwd $WALLET_PWD -cert | openssl x509 -noout -enddate | cut -d= -f2)
CERT_EXPIRY_EPOCH=$(date -d "$CERT_EXPIRY" +%s)
NOW_EPOCH=$(date +%s)
DAYS_LEFT=$(( (CERT_EXPIRY_EPOCH - NOW_EPOCH) / 86400 ))

if [ $DAYS_LEFT -le $CERT_DAYS_THRESHOLD ]; then
    echo "Certificate expires in $DAYS_LEFT days. Renewing..."
    # 1. Generate new private key and CSR LOCALLY on DB host (Key never leaves DB node)
    openssl req -new -newkey rsa:2048 -nodes \
        -keyout /tmp/$(hostname -f).key.pem \
        -out /tmp/$(hostname -f).csr.pem \
        -subj "/C=US/O=MyOrg/CN=$(hostname -f)"
    
    # 2. Copy ONLY the CSR to the CA host for signing
    scp /tmp/$(hostname -f).csr.pem $CA_HOST:/opt/ca/intermediate/csr/
    
    # 3. Request CA host to sign the CSR
    ssh $CA_HOST "openssl ca -config /opt/ca/intermediate/openssl.cnf -batch -days 365 -in /opt/ca/intermediate/csr/$(hostname -f).csr.pem -out /opt/ca/intermediate/certs/$(hostname -f).cert.pem"
    
    # 4. Fetch ONLY the signed public certificate back
    scp $CA_HOST:/opt/ca/intermediate/certs/$(hostname -f).cert.pem /tmp/
    
    # 5. Update Oracle wallet
    orapki wallet remove -wallet $WALLET_DIR -pwd $WALLET_PWD -cert "$(hostname -f)"
    orapki wallet add -wallet $WALLET_DIR -pwd $WALLET_PWD \
        -cert /tmp/$(hostname -f).cert.pem -key /tmp/$(hostname -f).key.pem
    # Reload wallet in Oracle (see Step 5)
    sqlplus / as sysdba <<EOF
        ALTER SYSTEM SET ENCRYPTION WALLET CLOSE;
        ALTER SYSTEM SET ENCRYPTION WALLET OPEN IDENTIFIED BY "$WALLET_PWD";
EOF
    echo "Certificate renewed and wallet reloaded."
else
    echo "Certificate valid for $DAYS_LEFT days. No action needed."
fi

Schedule this script via cron on each DB node:

0 2 * * * /opt/scripts/renew_cert.sh >> /var/log/cert_renew.log 2>&1

Step 5: Reload Wallet Without Downtime

For Oracle Database 19c and later, you can reload the wallet dynamically using ALTER SYSTEM commands. For older versions, you may need to restart the listener and database instances. The script above demonstrates the dynamic approach.

Step 6: Monitor and Alert

Set up monitoring to check certificate validity and alert on anomalies. Use a simple script that queries the wallet and sends alerts via email or a central monitoring system.

#!/bin/bash
# /opt/scripts/check_cert.sh
WALLET_DIR="/u01/app/oracle/wallet/root"
WALLET_PWD="<wallet_password>"
ALERT_EMAIL="dba@example.com"

CERT_EXPIRY=$(orapki wallet display -wallet $WALLET_DIR -pwd $WALLET_PWD -cert | openssl x509 -noout -enddate | cut -d= -f2)
CERT_EXPIRY_EPOCH=$(date -d "$CERT_EXPIRY" +%s)
NOW_EPOCH=$(date +%s)
DAYS_LEFT=$(( (CERT_EXPIRY_EPOCH - NOW_EPOCH) / 86400 ))

if [ $DAYS_LEFT -le 14 ]; then
    echo "WARNING: Certificate expires in $DAYS_LEFT days" | mail -s "DB Certificate Expiry Alert" $ALERT_EMAIL
fi

Add this to cron as well.

Step 7: Handle Revocation and Rotation

In case of compromise, revoke the certificate using the CA’s CRL mechanism. Update the wallet to remove the revoked certificate and add a new one. Ensure the CRL is distributed to all nodes.

📚 Official Documentation & Technical References

• Oracle Database Security Guide – Managing Wallets
• Oracle Database Net Services Reference – SSL Configuration
• Oracle Database 23ai New Features – Automatic Wallet Management
• My Oracle Support (MOS) Note 401251.1 – “Configuring SSL for Client Authentication and Encryption With Self Signed Certificates On Both Ends Using orapki”
• MOS Note 1629906.1 – “How To Create a Wallet via ORAPKI in Fusion Middleware”
• MOS Note 1367988.1 – “Using ORAPKI Utility to Create a Wallet with Third Party Trusted Certificate”

Need Expert Assistance?

Automating certificate lifecycle management in air-gapped environments is complex. If you need help designing, implementing, or troubleshooting your Oracle Database security infrastructure, our team of certified DBAs and security engineers is ready to assist. We offer comprehensive services including certificate automation, wallet management, and 24/7 support.

Contact Us | View Our Services