Regional-DR build runbook (the proper way)

Stand up ODF Regional-DR (async RBD mirroring) between two air-gapped OCP clusters, from an ACM hub, using oc + GitOps only. Build in the order below. Each phase has a do-it-right note, a verify, and the pitfall it avoids. Doing Phase 3 correctly is what saves you days.

Components

Prerequisites (fix these before you start)

• ACM hub + 2 managed ODF clusters, all in one ManagedClusterSet.
• Non-overlapping pod + service CIDRs across the two clusters. Decide at install time.
• Mirrored operators in the air-gap catalog: submariner, odf-multicluster-orchestrator, odr-hub-operator, odr-cluster-operator.
• ODF healthy on both clusters; RBD block pool present.
• Know your air-gap catalog name (here: cs-redhat-operator-index-v4-20). You will need it.

Why non-overlapping CIDRs matter: overlap forces Submariner Globalnet, and ODF StorageClusterPeer uses raw ClusterIPs that Globalnet cannot NAT — peering never completes. Fixing this later means rebuilding a cluster. Plan it up front.

Phase 1 — Submariner

Do it right:

• Broker with globalnetEnabled: false (CIDRs don’t overlap).
• One gateway node per cluster, labeled submariner.io/gateway=true.
• SubmarinerConfig: cableDriver: libreswan, airGappedDeployment: true, subscriptionConfig.source = your air-gap catalog, channel stable-0.24.

Pitfall — gateway-node pod scheduling (this one is subtle):

• Pods running on the Submariner gateway node cannot reach the remote cluster. OVN’s gateway router SNATs their egress to the node IP, which no longer matches the IPsec tunnel selector, so their packets never enter the tunnel.
• Keep DR-critical pods off the gateway node: ocs-operator, rook-ceph-rbd-mirror, the ODF provider/client pods.
• Best practice: a dedicated gateway worker with no ODF workload. On 3-node compact clusters, at least check placement (oc get pods -n openshift-storage -o wide vs the gateway node) and reschedule offenders (cordon gateway, delete pod, uncordon).

Verify:

• oc get submariner -n submariner-operator ... connection status connected both ways.
• Cross-cluster pod→pod from a non-gateway pod succeeds.

Phase 2 — ODF MirrorPeer (storage peering)

Do it right:

• Install odf-multicluster-orchestrator on the hub.
• Create a MirrorPeer (type: async, schedulingIntervals: ["5m"]) referencing both StorageClusters.
• The orchestrator drives StorageClusterPeer → Peered, exchanges S3 secrets, and enables RBD pool mirroring automatically.

Pitfall — the peering gRPC to the remote provider (:50051) is issued by ocs-operator. If it sits on the gateway node, StorageClusterPeer = Failed (see Phase 1). Move it off.

Verify:

Phase 3 — Ramen hub operator + S3 trust (the step most builds get wrong)

Do these before creating any DRCluster, in this order.

3a. Install ODR Hub Operator

• CSV odr-hub-operator on the hub. It creates cm ramen-hub-operator-config in openshift-operators.

3b. Point Ramen at the air-gap catalog

• Ramen’s drClusterOperator.catalogSourceName defaults to redhat-operators.
• In air-gap the mirror catalog has a different name, so the auto-installed odr-cluster-operator subscription later fails ResolutionFailed.
• Two options:
  1. Edit cm ramen-hub-operator-config → drClusterOperator.catalogSourceName: cs-redhat-operator-index-v4-20, then delete the ramen-hub pod to reload.
  2. Cleaner/GitOps-able: create a CatalogSource literally named redhat-operators in openshift-marketplace pointing at your mirror index, so the default just works.

Config reload: Ramen reads its config only at startup. oc rollout restart is reverted by OLM — instead oc delete pod <ramen-hub-pod>.

3c. Make Ramen trust the NooBaa route cert — the RIGHT way

Key facts (this is why the naive fix fails):

• Ramen’s S3 client uses path-style + Go’s system cert pool. It does not use the profile caCertificates field for object-store validation (that field only feeds the Velero path).
• The S3 endpoint is the external NooBaa route https://s3-openshift-storage.apps.<cluster>.<domain>, served by the cluster’s default ingress cert (issuer CN=ingress-operator@<ts>, per-cluster, self-signed — not your lab CA).
• So the CA must go into Ramen’s system trust, not the profile.

Steps:

1. Get each cluster’s ingress CA: oc --kubeconfig <c> get cm default-ingress-cert -n openshift-config-managed -o jsonpath='{.data.ca-bundle\.crt}'
2. Add both CAs to cm user-ca-bundle (openshift-config) — the one referenced by proxy/cluster .spec.trustedCA.
   • GitOps: user-ca-bundle is Argo-managed (hub/cluster-trust/user-ca-bundle.yaml). Edit it in Git, not with oc — Argo self-heal reverts manual oc edits.
3. CNO merges it into openshift-trusted-cabundle, which ramen-hub mounts.
4. Delete the ramen-hub pod (loads system roots once at startup; won’t auto-detect new certs — documented ODF known issue).

Verify trust before going further — from the ramen-hub pod:

curl -s -o /dev/null -w '%{ssl_verify_result}\n' \
  https://s3-openshift-storage.apps.<c>.<domain>/<bucket>      # want 0 (trusts)

3d. S3 profiles + secrets

• Create ramen-format secrets s3secret-<c> (keys AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY) in openshift-operators, from each cluster’s ODF S3 creds (the ODF NooBaa odrbucket secret).
• Add both profiles to ramen-hub-operator-config s3StoreProfiles: s3ProfileName, s3Bucket, s3CompatibleEndpoint (the route), s3Region, s3SecretRef.
• Keep any caCertificates base64 on one line (YAML wrapping corrupts []byte).
• Delete the ramen-hub pod to reload.

Phase 4 — DRClusters + DRPolicy

Only after Phase 3 trust check returns 0.

DRCluster/wl     spec.region: region-wl      s3ProfileName: s3profile-wl-...
DRCluster/wl-dr  spec.region: region-wl-dr   s3ProfileName: s3profile-wl-dr-...
DRPolicy/dr-policy-5m   drClusters: [wl, wl-dr]   schedulingInterval: 5m

• Distinct regions ⇒ async / Regional-DR. (Same region = Metro/sync.)
• Creating the DRPolicy auto-installs odr-cluster-operator on both spokes.

Pitfall — never oc delete a DRCluster that is mid-reconcile. Its finalizer hangs it in Terminating, Ramen takes the delete code path, and validation freezes. If stuck:

oc patch drcluster <n> --type merge -p '{"metadata":{"finalizers":[]}}'   # then recreate

Verify (spoke operator install can take ~10 min):

Phase 5 — Protect a workload (DRPlacementControl)

• Create a DRPlacementControl (+ Placement) referencing dr-policy-5m and the app’s PVC label selector.
• Ramen then mirrors those PVCs and can fail the app over to the secondary.

GitOps ownership map

Upgrade caution: an ODF operator upgrade wipes the runtime Ramen config (s3 profiles, CA, catalog name). Re-apply them after every ODF upgrade. Documented ODF known issue.

Failure → cause → fix (quick reference)

Debugging notes

• Ramen logs only the aws-sdk wrapper (RequestError: send request failed) even at -zap-log-level=debug — it hides the real x509 cause.
• Reproduce the real S3 error with boto3 (path-style + CA = OK confirms the recipe).
• Set Ramen debug via the CSV odr-hub-operator (OLM reverts plain deployment patches).