L6 — Data protection & DR
Two independent tiers: backup/restore (everything → MinIO) and live DR (Regional-DR for workloads + ACM hub failover). Deep build steps: regional-dr-build.md.
Backups → MinIO
| What | Mechanism / schedule | Retention |
|---|---|---|
| Cluster resources + PVs | OADP/Velero + kopia; Schedule daily-cluster 03:00 |
7d |
| etcd | bastion cron 01:00 etcd-backup.sh (both clusters) |
7d |
| Vault raft snapshot | vault-0 cron 01:30 | 14d |
| GitLab | gitlab VM cron 02:00 | 14d |
| Postgres HA | pgBackRest → MinIO pg-backups (WAL + full/diff) |
— |
Buckets/users (creds in Vault): velero-*/velero, loki-*/loki, acm-observability/obs, etcd-backups+vault-snapshots+gitlab-backups/infra-backup.
Tier 1 — Regional-DR (wl ↔︎ wl-dr, storage)
Active/standby ODF RBD async mirroring, driven from the ACM hub, all via oc + GitOps.
| Layer | What | Readiness signal |
|---|---|---|
| Submariner | libreswan IPsec tunnel (globalnet off) between gateway nodes | connection connected |
| MirrorPeer | ODF mirrorpeer-wl-wldr → StorageClusterPeer Peered, RBD mirroring |
phase ExchangedSecret |
| Ramen | odr-hub-operator (hub) + odr-cluster-operator (spokes) |
— |
| DRPolicy | dr-policy-5m (5m interval), DRClusters wl + wl-dr |
Validated=True + replicationID |
- GitOps: DRClusters/DRPolicy (
hub/regional-dr), DR-hub operators (hub/regional-dr-operators), rbd-mirror off-gateway placement + SSA self-heal (spoke/base/odf/storagecluster.yaml), ingress-CA trust (*/cluster-trust). - Out-of-band (runtime): Ramen s3 profiles/secrets + air-gap catalog — regenerated by
scripts/reintegrate-ramen-dr.sh. - One-command re-establish:
scripts/reintegrate-regional-dr.sh— ordered Submariner→MirrorPeer→ramen→DRPolicy with readiness gating + air-gap guards baked in.
Failover drill (proven, zero data loss): patch DRPC action=Failover → app + PVC restore on wl-dr with matching marker sha. Gotchas: pre-create the target namespace; for failback never delete the old-primary PVC (scale workload to 0, reach PeerReady=True, then Relocate).
Tier 2 — ACM hub failover (hub ↔︎ hub-dr)
Backup/restore of the ACM control plane.
| Piece | Where |
|---|---|
| Active hub backup | BackupSchedule hub-backup-schedule (useManagedServiceAccount=true) → MinIO |
| hub-dr posture | passive restore-passive-sync (continuously syncs the hub’s backups) |
| DR operators on hub-dr | GitOps hub/regional-dr-operators (so restored DR CRs have CRDs) |
| MSA auto-import | auto-import-account MSA tokens ride the backup → hands-off spoke re-import |
Failover drill (proven): fresh backup → activation restore on hub-dr → wl+wl-dr re-register (klusterlet bootstrap flips to hub-dr) → fail back (re-apply the original hub’s klusterlet manifests) → hub-dr back to passive. Gotcha: failback re-import needs the target hub’s <cluster>-import manifests (default AutoImportStrategy=ImportOnly skips them).
Backup/restore drills (all passed)
| Drill | Result |
|---|---|
| Namespace restore (wl) | sha256 identical after ns delete + restore |
| CNV VM restore (wl) | 30 Gi VM disk via datamover; guest sha256 identical |
| etcd point-in-time (wl) | pre-snapshot state intact; see wedges in Operations |
| Full cluster-loss (wl) | reinstall + ACM reimport + GitOps reconverge; RTO ≈ 2h |
| Postgres pgBackRest restore | marker recovered from MinIO to isolated instance; live cluster untouched |
GitOps vs. out-of-band
GitOps: all DR CRs, operators, placements, ACM-backup chain. Out-of-band (scripted): Ramen runtime config, the ordered Submariner/MirrorPeer/DRPolicy handshake. See Rebuild Phase 3.