Gotchas (reference)
Hard-won failure modes → fixes, grouped by area. Symptom in the left column.
Host / etcd
Cluster-wide API context deadline exceeded, controllers wedge, Argo slow |
etcd starved by host disk I/O contention (many VMs on one HDD-RAID; r_await 50–171 ms). Move etcd/master disks to fast/dedicated storage; reduce VM density |
Install / mirror / OLM
| FIPS install rejects SSH key |
Use RSA (Ed25519 rejected); FIPS openshift-install |
| Install hangs |
Remove openshift/99-master-tang-luks-root.yaml before agent create image |
| Operators silently disappear after a mirror run |
Shared catalog tag is one overwrite target — always mirror the full imageset (~30 pkgs); verify packagemanifest count ~29 |
| Delta mirror corrupt |
Never delete/rsync the shared cache or .history; recover via mirror-to-mirror from dl385-2 |
| Fresh install ImagePullBackOff x509 |
Missing Quay CA in user-ca-bundle (cached images mask it) |
| Bundle unpack stuck (DeadlineExceeded) |
Delete the unpack job+cm in openshift-marketplace AND the subscription |
CA trust
| Pods can’t reach MinIO/SSO (x509) |
Missing Airgap CA in user-ca-bundle (needs BOTH Airgap + Quay CAs) |
Ramen s3ListFailed / send request failed |
Ramen uses system trust — add each cluster’s ingress CA to user-ca-bundle (GitOps */cluster-trust), then restart ramen-hub |
ACM pull model / Argo
| 0 spoke apps generated |
Missing GitOpsCluster CR (+ appset RBAC + acm-placement CM) |
| App “Synced” but resources Missing, counts oscillate |
The Application has a deletionTimestamp (stuck Terminating). Clear finalizer + oc delete app --cascade=orphan (ManifestWork recreates fresh) |
| GitOps change won’t apply though sync = Succeeded |
Stale in-flight sync op pins old manifests, OR a stuck op / monitoring drift-hijack. Recreate the app; use finite retry.limit |
| StorageCluster field won’t apply via Argo |
Client-side-apply lock — SC carries ServerSideApply=true; remove its last-applied-configuration annotation |
| CR “Pending deletion” gates a sync forever |
CR stuck with foregroundDeletion finalizer — clear the finalizer |
Regional-DR (see DR)
| rbd-mirror / peering breaks after churn |
rbd-mirror pod on the Submariner gateway node (OVN-SNAT). Now GitOps-pinned off-gateway; if seen, cordon gateway + reschedule |
Submariner status error but data flows |
Health-check pinger false-fails on the gateway node — disable connectionHealthCheck (baked into reintegrate-regional-dr.sh) |
| DRPolicy won’t validate |
Wrong Ramen catalogSourceName (must be cs-redhat-operator-index-v4-20); or DRCluster stuck Terminating (clear finalizer) |
| Failover restore fails “namespace not found” |
Pre-create the target namespace on the failover cluster |
| Failback (relocate) wedged |
Never delete the old-primary PVC; scale workload to 0, reach PeerReady=True, then Relocate |
| Re-protect fails “PVC bound to different PV” |
Stale S3 metadata from a force-deleted DRPC — re-protect under a new DRPC name |
Storage / observability / mesh
| ODF OSDs all on one node |
Deviceset count:1 replica:3 (not 3×1); avoid flexibleScaling |
| TempoStack wedged (status frozen, no pods) |
delete+recreate the TempoStack CR to force a fresh reconcile |
| OTel no traces |
Exporter must be OTLP HTTP :4318 (not :4317); Instrumentation CR must exist before pod CREATE |
| Ambient mTLS silently fails |
Set CA_TRUSTED_NODE_ACCOUNTS: istio-system/ztunnel on istiod |
Authoring
Don’t author YAML via nested ssh "..." heredocs (local $var expansion, nested quotes, flow chars). Write → scp → run. Validate with oc kustomize / python3 ast. |
pgrep/pkill -f self-matches ssh wrappers — use [b]racket patterns |