Enterprise Apache Tomcat & ORDS Java Runtime Upgrade Runbook: Keystore Migration & JVM Hardening

Executive Summary: The Overlooked Mid-Tier Runtime Hazard

Enterprise mid-tier architectures powering Oracle REST Data Services (ORDS), Ellucian Banner 9 Student/Faculty Self-Service (SSB), and custom Spring/Jakarta microservices almost universally rely on Apache Tomcat or standalone Java web engines.

While database administrators and security operations centers (SOC) diligently patch Oracle Database PSUs and operating system kernels, the underlying Java Runtime Environment (JRE / JDK) powering Tomcat is routinely neglected for months—or years.

Security audits consistently uncover Tomcat services running on unpatched JDK versions that expose the host to:

1. Remote Code Execution (RCE) through memory safety flaws in older Java HotSpot JIT compilers.
2. Cryptographic Downgrade & Man-in-the-Middle (MITM) risks caused by outdated TLS cipher suites and deprecated SunJSSE handshake implementations.
3. Application Outages During Ad-Hoc Upgrades: When sysadmins hastily upgrade Java without a structured runbook, Tomcat instances crash due to missing private Certificate Authority (CA) certificates in the new cacerts keystore, generating breaking PKIX path building failed: unable to find valid certification path exceptions.

This guide provides a resilient, defensive runbook designed to help operations and blue teams discover running JVMs, stage new JDK releases side-by-side, automate enterprise CA certificate migration, and safely restart Tomcat or ORDS without operational disruption.

---

Architecture: Tomcat & ORDS JVM Execution Lifecycle

Understanding how Apache Tomcat and standalone ORDS discover and load the Java environment prevents configuration drift across enterprise Linux systems:

+-------------------------------------------------------------------------+
| Enterprise Linux Host (RHEL / Oracle Linux / Rocky Linux)               |
|                                                                         |
| +---------------------------------------------------------------------+ |
| | systemd Service Unit: tomcat.service / ords.service                 | |
| | Environment="JAVA_HOME=/u01/app/java/jdk-17.0.12"                   | |
| +---------------------------------------------------------------------+ |
|                                   | (invokes)                           |
|                                   v                                     |
| +---------------------------------------------------------------------+ |
| | Apache Tomcat Runtime: $CATALINA_HOME/bin/catalina.sh               | |
| |                                                                     | |
| | Sources $CATALINA_BASE/bin/setenv.sh:                               | |
| |  - JAVA_HOME / JRE_HOME path definition                             | |
| |  - JVM Sizing: -Xms4g -Xmx4g -XX:+UseG1GC                           | |
| |  - Security Keystore: -Djavax.net.ssl.trustStore                    | |
| +---------------------------------------------------------------------+ |
|                                   | (spawns JVM)                        |
|                                   v                                     |
| +---------------------------------------------------------------------+ |
| | Java Virtual Machine (JVM): $JAVA_HOME/bin/java                     | |
| |                                                                     | |
| | Resolves Root Trust: $JAVA_HOME/lib/security/cacerts                | |
| | (LDAP / Active Directory / Oracle DB TCPS / SAML Identity Provider) | |
| +---------------------------------------------------------------------+ |
+-------------------------------------------------------------------------+

Key operational takeaways:

• Never rely on the OS default /usr/bin/java: System-level RPM package updates can alter the OS symlink unexpectedly, leading to mismatched JVM parameters or unsupported bytecode versions.
• Pin the JDK explicitly in bin/setenv.sh or systemd: Tomcat evaluates bin/setenv.sh before spawning Catalina, ensuring consistent runtime versions regardless of user environment variables.
• Preserve Internal PKI: Every modern enterprise Tomcat instance talks to internal LDAPS servers, SAML/OIDC federations, or Oracle Database listeners over TCPS (mutual TLS). The new Java install ships with a stock Mozilla certificate bundle; internal root and intermediate CAs must be migrated.

---

Phase 1: Pre-Flight Audit & Environment Discovery

Before introducing new binaries or scheduling maintenance, audit the active runtime parameters.

1. Identify Running Java Processes and Active JDK Paths

Run these non-intrusive discovery commands on the target host as the root or tomcat user:

# Locate active Tomcat / ORDS Java processes and inspect command-line arguments
ps -ef | grep -E "catalina.startup.Bootstrap|ords.war" | grep -v grep

# Inspect the exact binary path executing the process
PID=$(pgrep -f "catalina.startup.Bootstrap" | head -n 1)
ls -l /proc/$PID/exe

# Extract the current JAVA_HOME and startup flags directly from the live environment
cat /proc/$PID/environ | tr '\0' '\n' | grep -E "^(JAVA_HOME|JRE_HOME|CATALINA_)"

2. Verify Tomcat Version and JVM Capabilities

Execute Catalina’s built-in version inspection tool:

# Switch to the Tomcat service account
su - tomcat

# Run catalina version check
$CATALINA_HOME/bin/catalina.sh version

Example Output:

Using CATALINA_BASE:   /opt/tomcat
Using CATALINA_HOME:   /opt/tomcat
Using CATALINA_TMPDIR: /opt/tomcat/temp
Using JRE_HOME:        /usr/lib/jvm/jdk-11.0.18
Using CLASSPATH:       /opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar
Server version: Apache Tomcat/9.0.86
OS Name:        Linux
OS Version:     5.15.0-205.149.5.1.el9uek.x86_64
Architecture:   amd64
JVM Version:    11.0.18+10-LTS
JVM Vendor:     Oracle Corporation

3. Inventory Internal Certificates in Existing Keystores

Inspect the current keystore to see if internal CAs were manually injected:

# Set existing Java home path based on discovery
OLD_JAVA_HOME="/usr/lib/jvm/jdk-11.0.18"

# List certificates non-interactively (default password is 'changeit')
$OLD_JAVA_HOME/bin/keytool -list -keystore $OLD_JAVA_HOME/lib/security/cacerts \
  -storepass changeit | grep -iE "corporate|internal|incommon|ad_ca|db_root"

If custom aliases appear, you must execute the automated keystore migration in Phase 3.

---

Phase 2: Side-by-Side JDK Deployment (Zero Modification to Running Stack)

Never overwrite an existing JDK directory in-place. Deploying side-by-side allows for instant, deterministic rollback if an unexpected library dependency fails.

1. Create Enterprise Java Installation Directory

# Create standardized middleware java directory
sudo mkdir -p /u01/app/java
sudo chown -R root:root /u01/app/java

2. Extract the New Java SE JDK Release

Obtain the latest CPU release (e.g., Oracle JDK or Eclipse Temurin LTS 17.0.12 or 21.0.4) and extract it:

cd /tmp
# Extract tarball into target directory
sudo tar -xzf openjdk-17.0.12_linux-x64_bin.tar.gz -C /u01/app/java/

# Verify binary execution and permissions
/u01/app/java/jdk-17.0.12/bin/java -version

3. Verify Ownership & File Permissions

Ensure the Tomcat service user (tomcat or ords) can read and execute the new binaries:

# Grant standard read and execute permissions across the JVM tree
sudo chmod -R 755 /u01/app/java/jdk-17.0.12

---

Phase 3: Automated Enterprise Keystore Migration

When Java is upgraded, enterprise connections to LDAP, Active Directory, internal microservices, and Oracle Database SSL/TCPS endpoints will fail if internal root and intermediate certificates are missing.

Here is an automated Bash script to diff and copy all non-standard certificates from the old cacerts file into the new cacerts file:

#!/usr/bin/env bash
# ==============================================================================
# Script: migrate_java_cacerts.sh
# Purpose: Compares old vs new Java cacerts and imports custom enterprise CAs
# ==============================================================================
set -euo pipefail

OLD_JAVA="/usr/lib/jvm/jdk-11.0.18"
NEW_JAVA="/u01/app/java/jdk-17.0.12"
STOREPASS="changeit"

# Determine cacerts path (Java 8 uses jre/lib/security; Java 9+ uses lib/security)
get_cacerts_path() {
  local jh="$1"
  if [ -f "$jh/lib/security/cacerts" ]; then
    echo "$jh/lib/security/cacerts"
  elif [ -f "$jh/jre/lib/security/cacerts" ]; then
    echo "$jh/jre/lib/security/cacerts"
  else
    echo "ERROR: cacerts not found in $jh" >&2
    exit 1
  fi
}

OLD_CACERTS=$(get_cacerts_path "$OLD_JAVA")
NEW_CACERTS=$(get_cacerts_path "$NEW_JAVA")

echo "--> Source Keystore: $OLD_CACERTS"
echo "--> Target Keystore: $NEW_CACERTS"

# Backup target cacerts before any modifications
cp "$NEW_CACERTS" "${NEW_CACERTS}.backup.$(date +%F_%H%M%S)"

# Extract all alias names from source keystore
TMP_DIR=$(mktemp -d)
trap 'rm -rf "$TMP_DIR"' EXIT

"$OLD_JAVA/bin/keytool" -list -keystore "$OLD_CACERTS" -storepass "$STOREPASS" \
  | grep "trustedCertEntry" | awk -F',' '{print $1}' > "$TMP_DIR/old_aliases.txt"

"$NEW_JAVA/bin/keytool" -list -keystore "$NEW_CACERTS" -storepass "$STOREPASS" \
  | grep "trustedCertEntry" | awk -F',' '{print $1}' > "$TMP_DIR/new_aliases.txt"

# Find aliases in old keystore missing from new keystore
grep -Fxvf "$TMP_DIR/new_aliases.txt" "$TMP_DIR/old_aliases.txt" > "$TMP_DIR/migrate_aliases.txt" || true

MIGRATE_COUNT=$(wc -l < "$TMP_DIR/migrate_aliases.txt")
echo "--> Found $MIGRATE_COUNT custom enterprise certificate(s) to migrate."

while IFS= read -r alias; do
  [ -z "$alias" ] && continue
  echo "    Exporting and importing: $alias"
  
  # Export cert from source
  "$OLD_JAVA/bin/keytool" -exportcert -alias "$alias" \
    -keystore "$OLD_CACERTS" -storepass "$STOREPASS" \
    -file "$TMP_DIR/cert.der" > /dev/null 2>&1
    
  # Import cert into target
  "$NEW_JAVA/bin/keytool" -importcert -noprompt -alias "$alias" \
    -keystore "$NEW_CACERTS" -storepass "$STOREPASS" \
    -file "$TMP_DIR/cert.der" > /dev/null 2>&1
done < "$TMP_DIR/migrate_aliases.txt"

echo "--> Migration completed successfully."

---

Phase 4: Tomcat & ORDS Configuration Updates

With the new JDK deployed and internal certificates staged, configure the application runtime.

Approach A: Standalone Tomcat via bin/setenv.sh (Recommended)

Tomcat’s standard initialization script looks for $CATALINA_BASE/bin/setenv.sh. This keeps environment parameters decoupled from system-level profile scripts.

Create or edit $CATALINA_BASE/bin/setenv.sh:

# /opt/tomcat/bin/setenv.sh
# ------------------------------------------------------------------------------
# 1. Target JDK Runtime
# ------------------------------------------------------------------------------
export JAVA_HOME="/u01/app/java/jdk-17.0.12"
export JRE_HOME="/u01/app/java/jdk-17.0.12"

# ------------------------------------------------------------------------------
# 2. Modern JVM Memory & GC Configuration (16GB RAM Server Profile)
# ------------------------------------------------------------------------------
# Pin Heap Size to prevent fragmentation (-Xms equals -Xmx)
CATALINA_OPTS="-Xms4096m -Xmx4096m"

# Modern G1 Garbage Collector (recommended for Tomcat / ORDS workloads)
CATALINA_OPTS="$CATALINA_OPTS -XX:+UseG1GC"
CATALINA_OPTS="$CATALINA_OPTS -XX:MaxGCPauseMillis=200"
CATALINA_OPTS="$CATALINA_OPTS -XX:InitiatingHeapOccupancyPercent=45"

# Security & Headless Flags
CATALINA_OPTS="$CATALINA_OPTS -Djava.awt.headless=true"
CATALINA_OPTS="$CATALINA_OPTS -Dfile.encoding=UTF-8"

# Prevent DNS caching TTL infinity for database failover
CATALINA_OPTS="$CATALINA_OPTS -Dnetworkaddress.cache.ttl=60"

export CATALINA_OPTS

Ensure correct file permissions:

chmod 750 /opt/tomcat/bin/setenv.sh
chown tomcat:tomcat /opt/tomcat/bin/setenv.sh

Approach B: Modern systemd Unit Override

If your Tomcat instance is managed natively by systemd, utilize drop-in configuration overrides rather than editing the vendor service file directly:

# Open systemd override editor for tomcat
sudo systemctl edit tomcat.service

Add the following override block:

[Service]
# Clear existing JAVA_HOME definition
Environment="JAVA_HOME="
# Set updated JAVA_HOME path
Environment="JAVA_HOME=/u01/app/java/jdk-17.0.12"

Reload the systemd daemon:

sudo systemctl daemon-reload

Approach C: Oracle REST Data Services (Standalone Mode)

If running ORDS in standalone mode (ords --config /etc/ords/config serve):

1. Inspect the service wrapper or /etc/systemd/system/ords.service.
2. Update the environment variable or binary invocation:

[Unit]
Description=Oracle REST Data Services
After=network.target

[Service]
Type=simple
User=ords
Group=ords
Environment="JAVA_HOME=/u01/app/java/jdk-17.0.12"
Environment="PATH=/u01/app/java/jdk-17.0.12/bin:/usr/local/bin:/usr/bin"
ExecStart=/opt/ords/bin/ords --config /etc/ords/config serve
Restart=always

[Install]
WantedBy=multi-user.target

---

Phase 5: Service Execution & Verification Checklist

Schedule a minimal maintenance window or coordinate behind an upstream load balancer (HAProxy, F5, Nginx, or Caddy) to perform a rolling update.

1. Stop the Service & Terminate Zombie Threads

sudo systemctl stop tomcat.service

# Confirm the process has completely terminated
ps -ef | grep catalina | grep -v grep || echo "Tomcat process terminated cleanly."

If the Java process hangs due to unclosed database connections or stuck threads, issue a kill before starting the new engine:

# If PID persists after 30 seconds:
sudo pkill -9 -f "catalina.startup.Bootstrap"

2. Start the Service Under New JVM

sudo systemctl start tomcat.service

3. Verify Active JVM and Runtime Health

Execute these verification checks immediately following startup:

# Check 1: Verify systemd service status
sudo systemctl status tomcat.service --no-pager

# Check 2: Inspect running process command line to confirm new JDK path
PID=$(pgrep -f "catalina.startup.Bootstrap" | head -n 1)
ls -l /proc/$PID/exe

# Check 3: Review catalina.out for clean startup and JVM version banner
tail -n 100 /opt/tomcat/logs/catalina.out | grep -E "Server version|JVM Version|Command line argument"

Expected Log Confirmation:

04-Oct-2026 21:15:02.112 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           17.0.12+7-LTS
04-Oct-2026 21:15:02.113 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:            Eclipse Adoptium
04-Oct-2026 21:15:02.113 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         /opt/tomcat
04-Oct-2026 21:15:03.450 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [3824] milliseconds

4. Health Check Application Endpoints

Validate connectivity and ensure no SSL trust handshake errors occurred:

# Test local HTTP listener
curl -I -s http://127.0.0.1:8080/ | head -n 5

# Test ORDS / Database endpoint
curl -I -s http://127.0.0.1:8080/ords/ | grep -E "HTTP/|Content-Type"

# Monitor for any PKIX trust errors in catalina.out
grep -i "PKIX" /opt/tomcat/logs/catalina.out || echo "Zero SSL/TLS trust store errors detected."

---

Rollback Procedure: Instant Recovery

If an untested enterprise web application or third-party servlet encounters bytecode or library incompatibilities under the new Java release:

1. Revert Configuration:

   # Revert setenv.sh back to previous JDK
   sed -i 's|/u01/app/java/jdk-17.0.12|/usr/lib/jvm/jdk-11.0.18|g' /opt/tomcat/bin/setenv.sh

2. Reload & Restart:

   sudo systemctl daemon-reload
   sudo systemctl restart tomcat.service

3. Verify Rollback:

   PID=$(pgrep -f "catalina.startup.Bootstrap" | head -n 1)
   ls -l /proc/$PID/exe

   Total rollback execution time is under 30 seconds.

---

📚 Technical References

• Apache Tomcat 10 Documentation: RUNNING.txt — Official runtime parameters, environment scripts, and systemd integration standards.
• Oracle REST Data Services Installation and Configuration Guide — Official standalone and servlet deployment guides.
• Oracle Java SE Support Roadmap & Release Matrices — Support lifecycles for Java 8, 11, 17, and 21.