Skip to content

Troubleshooting

Common issues and solutions when working with HX-SDP.

Authentication errors

401 Unauthorized — Missing or invalid API key

Cause: No X-Api-Key or Authorization: Bearer header, or the key does not match any active tenant.

Fix:

  1. Verify the key is correct (keys start with hx_live_).
  2. Confirm the tenant is active: GET /gate/admin/tenants.
  3. If the key was rotated, use the new key.
# Test your key
curl -v https://gate.holonomx.com/health \
  -H "X-Api-Key: $HX_API_KEY"

403 Forbidden — Namespace not authorized

Cause: The tenant's API key does not have access to the requested namespace.

Fix: Add the namespace to the tenant's ACL:

curl -X POST https://gate.holonomx.com/gate/onboard/add-namespace \
  -H "Authorization: Bearer $SERVICE_KEY" \
  -H "Content-Type: application/json" \
  -d '{"tenant_id": "acme-corp", "namespace": "staging"}'

Rate limiting

429 Too Many Requests

Cause: Request volume exceeded the tenant's rate limit window.

Fix:

  1. Implement exponential backoff (see Rate Limits).
  2. Check X-RateLimit-Remaining headers proactively.
  3. Batch operations where possible (put_cores_batch).
  4. Upgrade to a higher tier for increased limits.

Quota errors

503 Service Unavailable — CU quota exhausted

Cause: Monthly CU allocation is fully consumed.

Fix:

  1. Check usage: GET /gate/admin/usage/{tenant_id}.
  2. Upgrade tier via Console or API.
  3. Wait for the billing period to reset (1st of month, UTC).

Compression issues

verdict: passthrough — Data stored uncompressed

Cause: Compression ratio fell below the configured threshold (HOLONOMIX_MIN_COMPRESSION_RATIO, default 1.2).

This is expected behavior for data that doesn't compress well (random noise, already-compressed data, very small payloads below MIN_VALUE_BYTES).

To investigate:

result = hx.put("test-key", data)
print(f"Verdict: {result.verdict}")
print(f"Ratio: {result.compression_ratio:.2f}")
print(f"Compressed: {result.compressed}")

verdict: weak — Low fidelity compression

Cause: The Oracle determined that compression introduces significant error.

Options:

  1. Increase max rank: Set HOLONOMIX_MAX_RANK higher (e.g., 32 → 64).
  2. Tighten tolerance: Lower HOLONOMIX_COMPRESSION_TOL (e.g., 1e-8 → 1e-10).
  3. Accept it: For many applications, weak verdict data is still usable.

Connection issues

502 Bad Gateway — Engine unreachable

Cause: HX-Gate cannot reach the HX-Engine backend.

Fix:

  1. Verify the engine is running: curl http://engine:8000/health.
  2. Check HX_GATE_ENGINE_SERVICE_KEY matches the engine's HOLONOMIX_API_KEYS.
  3. Confirm network connectivity between gate and engine containers.
  4. Check engine logs for startup errors.

Connection timeout

Cause: Operation took longer than the client timeout.

Fix:

  1. Increase client timeout: HolonomiX(url=..., timeout=60.0).
  2. For large payloads, verify HOLONOMIX_MAX_UPLOAD_BYTES is sufficient.
  3. For GPU operations, the first request may cold-start the CUDA context — retry.

Data issues

404 Not Found — Key does not exist

Cause: The key was never stored, was deleted, or the namespace is wrong.

Fix:

  1. Verify the namespace: keys are scoped per namespace.
  2. List keys: GET /v1/list/{namespace}?prefix=....
  3. Check if the key was deleted (tombstone retention is 30 days).

Version mismatch

Cause: Requested a specific version that doesn't exist.

Fix: Omit the version parameter to get the latest, or list available versions via GET.

Performance

Slow PUT operations

Possible causes:

  • Large payload with high-rank decomposition → expected, TT-SVD is $O(n \log n \cdot r^2)$
  • Engine CPU-bound → check if GPU is available (HOLONOMIX_DEVICE=cuda)
  • Network latency → colocate gate and engine

Slow similarity queries

Possible causes:

  • Large namespace with many entries → top-K scans all candidates
  • High bond dimensions → core contractions scale as $O(d \cdot r^2)$
  • Use metadata filters to narrow the candidate set

Logs

Gate audit log

Location: HX_GATE_AUDIT_LOG_PATH (default: /var/log/hx-gate/audit.jsonl)

tail -f /var/log/hx-gate/audit.jsonl | jq .

Each line contains: timestamp, tenant_id, method, path, status, CUs, latency_ms.

Engine logs

docker logs hx-engine --tail 100 -f

Set HOLONOMIX_LOG_LEVEL=debug for verbose output.