Keystone Gateway

Troubleshooting

Use the items below to identify common Keystone Gateway issues, understand likely causes, and apply the recommended resolution.

Handle logs and diagnostics safely

Gateway-owned structured logs redact recognized sensitive keys and, for string/error values, high-confidence credential representations, including nested structured attributes. This protection is defense in depth, not a complete secret detector: arbitrary opaque secrets, request/configuration objects, and raw error or message strings may still contain sensitive data. Keep credentials out of anything passed to logging, use safe contextual errors, and inspect logs, exporter output, and startup stderr before sharing them. Startup errors are a separate diagnostic path and are not covered by the structured-log redactor. See customer-safe diagnostics for the collection procedure.

Licensing

Invalid license

Cause

Symptoms. Service startup fails with gateway license validation failed, or /readyz/lease operations fail after expiry.

Likely causes. Expired, malformed, tampered, untrusted-signature, wrong Gateway ID, or unlisted replica ID.

Resolution

Diagnosis. Check journalctl -u keystone-gateway -n 100, node-id --config ..., and the safe /v1/license projection when running. Do not send the raw signature or secrets.

Resolution. Install a valid license naming the configured IDs and restart. See licensing.

Escalation information to collect. Version, node-ID output, timestamp/time source status, redacted logs, and license metadata.

License issued to the wrong node

Cause

Symptoms. Startup validation fails despite an otherwise valid file.

Likely causes. gateway_id or ha.replica_id differs from the license claims.

Resolution

Diagnosis. Compare node-id JSON with the license’s gateway_id and nodes values.

Resolution. Correct a configuration error or request a replacement license. Do not edit a signed document.

Escalation information to collect. Both identity values and redacted license metadata.

Leases and capacity

Capacity exhaustion

Cause

Symptoms. Acquire returns HTTP 409 capacity_exhausted; product startup cannot obtain capacity.

Likely causes. All declared units are held by live leases, or HA reconciliation is over capacity.

Resolution

Diagnosis. Query authenticated /v1/entitlements, inspect lease rejection metrics and /v1/ha.

Resolution. Release unused leases, wait for expiry, reduce requested counts, or obtain sufficient capacity. Do not delete state to free capacity.

Escalation information to collect. Entitlement projection, component, correlation ID, lease rejection metrics, and HA status.

Failed lease renewal

Cause

Symptoms. Client enters degraded state or becomes unlicensed at its recorded expiry.

Likely causes. Network failure, TLS/auth failure, standby target, store failure, expired license, or clock issue.

Resolution

Diagnosis. Check client logs, /readyz, Gateway logs using correlation ID, connectivity to both endpoints, and error code. not_active, store_unavailable, and rate_limited are retryable; preserve the same idempotency key.

Resolution. Restore connectivity/authentication or route to a ready peer. A lease cannot be revived after expiry; acquire a new one.

Escalation information to collect. Client/Gateway version, error code, correlation ID, endpoint test, readiness, and time status.

Telemetry

Telemetry failure

Cause

Symptoms. Collector has no recent data or exporter errors appear in logs.

Likely causes. Incorrect endpoint/protocol, CA/header, DNS/firewall, or collector outage.

Resolution

Diagnosis. Validate telemetry configuration and network path; distinguish it from /readyz and lease operations.

Resolution. Repair Collector connectivity/configuration. Telemetry failure does not stop Gateway licensing.

Escalation information to collect. Redacted telemetry configuration, exporter logs, protocol/endpoint, and readiness result.

High availability

HA peer failure

Cause

Symptoms. peer_connected is false, split-brain warning is true, or a node becomes degraded_active.

Likely causes. Peer process/network loss, peer TLS identity failure, clock skew, or partition.

Resolution

Diagnosis. Query /v1/ha on both nodes; check both peer ports and certificate target/CN, then logs and time synchronization.

Resolution. Restore peer transport. Expect convergence; if both sides issued leases, new acquisitions remain blocked until capacity is no longer exceeded. See Active/Standby.

Escalation information to collect. Both HA projections, peer connectivity tests, clock status, correlation IDs, and redacted logs.

Confirm