When a major-version Grid Infrastructure upgrade fails partway through, Oracle gives you a documented recovery path: the “Online Terminate Upgrade” downgrade procedure. It’s well documented, it runs clean, and it tells you it succeeded.

But in a real 2-node RAC test cluster, following that exact procedure left the cluster in a state where CRS simply refused to come back up even though the downgrade reported success at every step.

This post walks through exactly how that happened, with real commands and real errors, and what actually fixed it.

• Nodes: node1, node2
• Source Version: Oracle 19c GI
• Target Version: Oracle 26ai (23.x) — grid_26
• OS: Linux (systemd-based)

Step 1: Upgrade Node1 — Completes Successfully

rootupgrade.sh was run on node1 first, as usual for a rolling GI upgrade. It completed cleanly with no issues — node1 was now fully on 26ai.

Step 2: Upgrade Node2 — Interrupted Mid-Run

rootupgrade.sh was then started on node2. Partway through — specifically at step 15 of 16 (UpgradeNode) — the script was interrupted with Ctrl-C (repeated SIGINT).

This timing matters. By step 15, the cluster stack had already restarted under the new grid_26 home, and OCR keys had already been rewritten during the earlier step 14 (UpgradeCluster). So this wasn’t an early, clean abort — it was an interruption after the important internal state changes had already happened.

Step 3: Follow the Documented Downgrade Procedure

With node2 now broken, the next move was Oracle’s documented recovery path — “Downgrading Oracle Grid Infrastructure After a Failed Upgrade on First Node”:

https://docs.oracle.com/en/database/oracle/oracle-database/26/cwlin/downgrading-oracle-grid-infrastructure-after-failed-upgrade-on-first-node.html

Ran the downgrade starting on node1 — the node that had already completed its upgrade successfully:

clscfg -nodedowngrade -h node1
rootcrs.sh -downgrade -online

The command completed cleanly and reported:

CLSRSC-591: successfully downgraded Oracle Clusterware stack on this node

Everything about the output said this worked.

Step 4: Restart CRS — It Doesn’t Come Up

From the 19c home:

crsctl start crs

Result:

PROC-00035: Cannot perform cluster registry operation due to invalid version
OCRUTL: u_version_compatible: Versions are not compatible.
Prognode Ver [19.0.0.0.0] Persist Ver [23.0.0.0.0]

CRS refused to start. The software was correctly rolled back to 19c — but something inside the cluster registry still thought otherwise.

Step 5: Confirm It With ocrdump

ocrdump

SYSTEM.version.hostnames.node1 = 23.0.0.0.0   <- wrong, should be 19c
SYSTEM.version.hostnames.node2 = 19.0.0.0.0   <- correct, node2 never completed

There it was. Node1’s own version stamp inside OCR was still at 23.0.0.0.0, despite the software layer being back on 19c. The downgrade had reverted the binaries but never touched this internal version stamp.

Why This Happens

The “Online Terminate Upgrade” downgrade path is designed for upgrades that fail before OCR gets migrated to the new format. In this case, OCR had already been migrated during step 14 — before the interrupt even happened at step 15. So the downgrade correctly rolled back the software, but had no mechanism to roll back OCR’s already-committed version stamp for the node that had completed its upgrade.

CRS checks this stamp on startup — it compares its own version against what’s recorded for it in OCR (SYSTEM.version.hostnames.<node>), and refuses to initialize on a mismatch. That’s the PROC-35.

Step 6: The Only Fix — Restore OCR From Backup

crsctl stop crs                              # all nodes
crsctl start crs -excl -nocrs                # one node, from 19c home
ocrconfig -showbackup                        # locate a pre-upgrade backup
ocrconfig -restore <backup file>
ocrcheck                                     # verify integrity
ocrdump                                      # confirm version stamp reverted
crsctl stop crs                              # exit exclusive mode
crsctl start crs                             # cluster-wide, normal mode

After the restore, ocrdump confirmed SYSTEM.version.hostnames.node1 back at 19.0.0.0.0, matching node2. CRS came up cleanly cluster-wide.