Troubleshooting¶

Release binding: v0.1.0-enterprise-ready · AMI ami-010806d4d3445660e · 2026-05-19

The appliance fails closed by design: manifest tamper and missing GPUs stop services rather than degrading silently.

Instance will not boot (AWS)¶

The most common cause is a missing kms:CreateGrant on the CMK for the launching principal, or for the instance profile role when one is attached. Without the grant the instance fails with an EBS volume decryption error. See Known Boundaries for the exact IAM policy.

Services do not start¶

Check /health:

manifest: tampered means the signed manifest failed verification. Services will not start. This is fail-closed by design; investigate the image before proceeding.
The GPU preflight refuses to start on a host without a compatible NVIDIA GPU. CPU-only hosts are not a supported serving target.

`redis: unavailable`¶

Restart in dependency order:

sudo systemctl restart redis
sudo systemctl restart hx-engine
sudo systemctl restart hx-gate

hx-gate does not automatically reconnect after a Redis restart in v0.1.0 (known gap, P3), so restart the gate explicitly.

502 from the gate¶

The engine is down or unreachable on loopback.

sudo systemctl restart hx-engine

The engine rebuilds the GPU index from disk on startup; allow seconds to minutes depending on corpus size.

429¶

The tenant hit its sliding-window rate limit. Back off honoring Retry-After, batch requests, or raise the tenant's limit. See Rate Limits.

503 under load¶

Admission control returned 429/503 with Retry-After rather than allowing the GPU to run out of memory. Size up the instance or reduce concurrency. A 503 can also mean an operator-set CU quota is exhausted for the tenant (see Metering) or EXACT-path signing is unavailable.

409 on `/v1/query/exact`¶

No raw corpus is registered for the namespace. Register one through the corpus registry first.

Top-K ordering differs after a reboot¶

Expected behavior: after a reboot, top-K results overlap approximately 9 of 10 with pre-reboot results for the same query, with scores preserved; ordering of tied or near-tied scores may shift with GPU index rebuild order. Filter by score rather than by position. See Known Boundaries.

`verdict: passthrough` or `weak`¶

The Oracle measured the entry and chose the safe representation. passthrough means compression does not pay for this data; weak means the representation is lossy but structure-preserving. Neither is an error. See Concepts.

Logs and support¶

sudo journalctl -u hx-engine -f
sudo journalctl -u hx-gate -f

The audit log is append-only JSONL at /var/log/hx-gate/audit.jsonl. Generate a redacted support bundle with:

hx-sdp support-bundle create --out /var/lib/hx-sdp/support/support-bundle.tar.gz

Support bundles redact secrets, keys, and customer data. Send them through Support; do not post them publicly.