Air-Gapped OpenShift BFSI Lab — Overview
Platform-as-code + operations docs for an air-gapped OpenShift 4.20.26 BFSI lab: 4 clusters (2 primary + 2 DR), FIPS, on one KVM host dl385-2, libvirt net ocpnet (192.168.130.0/24), domain airgap.lab.
Secrets policy: no credential values in this repo or these docs. Everything is referenced by its Vault path or on-host file. See Credentials.
Read in layers (build order)
Each layer depends on the one below it. Rebuild from the bottom up.
| # | Layer | What it delivers |
|---|---|---|
| L0 | Foundation | KVM host, libvirt, networking, DNS, storage |
| L1 | Bastion & air-gap services | Registry, MinIO, GitLab, Vault, FreeIPA, mirroring |
| L2 | Cluster provisioning | Day-0 agent installs — the 4 clusters |
| L3 | GitOps bootstrap | Day-1 — root app, ACM pull model |
| L4 | Platform services | Day-2 — ACM, storage, security, observability |
| L5 | Identity & integration | Keycloak SSO, OIDC, LDAP, APISIX |
| L6 | Data protection & DR | Backups, Regional-DR, hub failover, drills |
| L7 | Applications | Demo apps, mesh, tracing, middleware VMs |
Reference shelf (spans all layers): Inventory · Access · Credentials · Gotchas · Operations · Rebuild runbook
At a glance
| Cluster | Role | Domain | API/Ingress VIP |
|---|---|---|---|
| hub | ACM management (active) | hub.airgap.lab | .5 / .6 |
| hub-dr | ACM passive/restore hub | hub-dr.airgap.lab | .57 / .58 |
| wl | Workload (primary) | wl.airgap.lab | .7 / .8 |
| wl-dr | Workload DR (secondary) | wl-dr.airgap.lab | .62 / .63 |
Design principles
| Principle | How |
|---|---|
| Air-gapped delivery | oc-mirror v2 → Quay on bastion (registry.airgap.lab:8443) |
| GitOps everything | Argo on hub syncs hub/; spokes via ACM pull model |
| Secrets out of git | ESO pulls from Vault apps/; cert-manager from Vault PKI; ODF KMS in Vault |
| FIPS + PCI-DSS | FIPS on all clusters + infra VMs (GitLab excepted); Compliance Operator ocp4-pci-dss |
| Two-tier DR | Regional-DR (wl↔︎wl-dr, storage) + ACM hub backup/restore (hub↔︎hub-dr) |
The two cross-cutting distinctions
Everything here sorts along two axes — keep them in mind:
- GitOps-managed vs. out-of-band — most config is GitOps; a handful of runtime/secret bits are scripted (see each layer’s “GitOps vs out-of-band” note and Rebuild).
- Cluster role — hub / hub-dr / wl / wl-dr each get a different slice.