Rebuild runbook (reference)

Rebuild any/all of the 4 clusters from GitLab GitOps. MinIO + bastion + Vault are kept (they hold state/secrets). Run in order; scripts/*.sh are idempotent.

What GitOps does vs. doesn’t

GitOps rebuilds	Out-of-band (below)
All Day-2 config: `hub/` (~30 components) + `spoke/{base,overlays}`, DR operators, ACM-backup chain	Cluster installs (Day-0); secret VALUES (Vault); operator-runtime config (Ramen s3)

Phase 0 — Day-0 install (per cluster)

install-config refs: hub /root/ocp420-backup, wl /root/spoke-backup, hub-dr/wl-dr /root/<c>-backup/install-config.reference.yaml.
⚠️ Non-overlapping CIDRs: wl 10.128/14+172.30/16; wl-dr 10.132/14+172.31/16.
3-node compact, FIPS. agent create image → boot masters → wait for install-complete.

Phase 1 — GitOps bootstrap (per cluster)

Install OpenShift GitOps operator → oc apply -f bootstrap/root-app.yaml.
Seed gitops-repo deploy-token secret (Vault apps/gitlab/argo-deploy).

Phase 2 — Out-of-band seeds (in order)

Step	Command
Vault k8s-auth	`scripts/reintegrate-vault-k8s-auth.sh <hub\\|spoke>`
Node labels	`scripts/label-odf-nodes.sh`, `label-infra-nodes.sh`
ODF KMS	pre-stage `ocs-kms-token` (fresh Vault token, policy `odf-kms`) before the ODF wave
ACS	init bundle from hub Central

Phase 3 — Regional-DR + hub-DR

Task	How
Regional-DR (wl↔︎wl-dr)	`scripts/reintegrate-regional-dr.sh` — ordered Submariner→MirrorPeer→ramen→DRPolicy with readiness gating + air-gap guards
Ramen runtime only	`scripts/reintegrate-ramen-dr.sh` (called by the above)
Hub DR	GitOps `hub/acm-backup` provides OADP DPA/BSL + BackupSchedule (`useMSA=true`)

If a pull-model app is stuck OutOfSync

Cause	Fix
Credential-less `gitops-repo` secret	Seed deploy-token creds (Phase 1.2)
Long-running Argo sync op pins stale manifests	Recreate the app: finalizer-clear + `oc delete app --cascade=orphan` (ManifestWork recreates fresh)
StorageCluster won’t take a field	SC carries `ServerSideApply=true`; remove `last-applied-configuration` annotation

Reality check

Automated: all Day-2 config + DR reintegration (GitOps + 4 idempotent scripts).
Still manual (inherent): the 4 agent-based cluster installs (Phase 0).
Host note: keep etcd on fast storage (see Foundation) or the rebuild flaps on API timeouts.