Operations (reference, day-2 runbooks)

Reach Vault nodes via the host (ssh ze@dl385-2 'ssh root@192.168.130.21 ...'). Values: Credentials.

Vault

HA nodes (.21–.23) auto-unseal via the transit node (.24).

After the transit node (.24) reboots, unseal it manually:

ssh ze@dl385-2 'ssh root@192.168.130.24 \
  "VAULT_ADDR=https://127.0.0.1:8200 vault operator unseal \
   $(jq -r .unseal_keys_b64[0] /root/transit-init.json)"'

Until then, the HA nodes cannot auto-unseal.

Mirroring (add operator / upgrade)

Edit ~/ocp420-airgap/mirror/ops/imageset-ops.yaml on dl385-2 — keep the full imageset.
Rerun oc-mirror phase1 on host → scp to bastion:/root/mirror-staging → phase2 on bastion.
Re-apply regenerated IDMS/ITMS to clusters and commit to git in the same motion (Argo self-heal reverts a lagging direct apply).
Restart cs-redhat catalog pods; verify packagemanifest count ~29.

Recovery: mirror-to-mirror direct from dl385-2 → registry.airgap.lab:8443.

Quay storage

Blobs on the 1 TB vdb bind-mount (/root/mirror-staging/quay-storage). On push failures, check df + Quay /health/instance first — a full Quay reports disk_space=false → cascading “manifest unknown”/500. Grow: virsh blockresize ... staging.qcow2 <SIZE> + xfs_growfs.

Storage

hub/hub-dr: LVMS. wl/wl-dr: ODF (deviceset count:1 replica:3, Vault-KMS encryption).
Label storage nodes cluster.ocs.openshift.io/openshift-storage=.

Observability

Logs: Observe → Logs (COO logging-view UIPlugin). Traces (wl): Observe → Traces.
ACM MCO Grafana on hub shows all clusters. Alerts → Wazuh SIEM webhook.
Perses: use the Monitoring UIPlugin perses.enabled: true — don’t hand-create a Perses CR.

etcd restore wedges (if a point-in-time restore stalls)

Wedge	Fix
Old kube-apiserver keeps serving on recovery node	Cycle its static-pod manifest (mv out, wait 45s, mv back)
haproxy frozen, VIP dead	`crictl stop` every haproxy container on all masters
etcd refuses revision cut at 1 member	Temporarily set `useUnsupportedUnsafeNonHANonProductionUnstableEtcd: true`, let members rejoin, revert

Post-restore API status objects are stale — verify with on-node crictl ps + manifest presence.

Credential rotation

Completed 2026-07-10 — all exposed creds rotated (new values in Vault; git history values now invalid). Rotate again if anything new lands in history.