Operations Guide¶

Release binding: v0.1.0-enterprise-ready · AMI ami-010806d4d3445660e · 2026-05-19

Service topology¶

Three systemd services. Only hx-gate binds a routable interface; the engine and Redis are loopback-only appliance internals.

redis-server : 127.0.0.1:6379
hx-engine    : 127.0.0.1:8000
hx-gate      : 0.0.0.0:8443 (TLS)

sudo systemctl status hx-engine hx-gate redis
sudo systemctl restart hx-engine

Operational rules:

Do not modify /opt/hx-sdp after image bake; it is covered by the signed manifest.
Environment configuration lives under /etc/hx-sdp; customer state lives under /var/lib/hx-sdp.
Expose only hx-gate TCP 8443 to trusted client networks.

Appliance commands¶

hx-sdp status
hx-sdp doctor
hx-sdp gpu-preflight
hx-sdp validate --mode smoke
hx-sdp support-bundle create --out /var/lib/hx-sdp/support/support-bundle.tar.gz

Health¶

curl -sk https://<host>:8443/health -H "Authorization: Bearer <API_KEY>"

Field	Healthy	Degraded
`engine`	`ok`	`unavailable` or `error`
`redis`	`ok`	`unavailable`
`gpu`	`ok`	`not_detected` or `oom`
`manifest`	`verified`	`tampered` (services will not start)

Metrics¶

Prometheus exposition at GET /metrics on the gate. Engine metric names carry the holonomix_ prefix; a representative operator set:

Metric	Type	Meaning
`holonomix_requests_total`	counter	API requests by endpoint
`holonomix_request_duration_seconds`	histogram	Request latency
`holonomix_query_total`	counter	Served queries by namespace, retrieval class, and metric
`holonomix_query_duration_seconds`	histogram	Query latency by namespace and retrieval class
`holonomix_query_class_transitions_total`	counter	Governor class transitions
`holonomix_receipt_verifications_total`	counter	ML-DSA-65 verify outcomes
`holonomix_auth_failures_total`	counter	Auth middleware rejections
`holonomix_compression_ratio`	histogram	Per-entry compression ratio at ingest
`holonomix_vram_total_bytes`	gauge	VRAM in use
`holonomix_index_entries`	gauge	Rows in the serving index
`holonomix_memory_pressure_pct`	gauge	Memory pressure
`holonomix_vram_budget_exceeded_total`	counter	VRAM guard rejections

The full list is self-describing at /metrics. Retrieval-class alerting is covered on the Retrieval Classes page.

Recommended alerts¶

Condition	Threshold	Action
Health endpoint non-200	3 consecutive	Check service status and logs
VRAM above 90 percent	sustained	Larger instance or reduce corpus
Disk above 80 percent	sustained	Expand the volume or prune data
Any `holonomix_receipt_verifications_total{result="rejected"}`	any	Integrity event; see the retrieval-classes runbook

Logs and audit¶

sudo journalctl -u hx-engine -f
sudo journalctl -u hx-gate -f
sudo journalctl -u redis -f

The gate's audit log is append-only JSONL at /var/log/hx-gate/audit.jsonl; ship it to your SIEM.

Reboot¶

A clean reboot preserves all data. On boot: manifest verification runs fail-closed, services start in order (redis, hx-engine, hx-gate), and the engine rebuilds the GPU-resident index from persisted state.

Backup (AWS reference)¶

This section applies to an AWS image explicitly included in an agreed deployment. It is retained as a dated technical reference, not a marketplace purchase route.

Snapshot the EBS volume manually or with AWS Backup; recommended retention 7 daily and 4 weekly. Restore by launching a fresh instance from the AMI and attaching a volume created from the snapshot as the root device. Durability is snapshot-based; the recovery point is the snapshot interval.

Operator configuration¶

Customer-visible configuration on the appliance is a small, stable subset; set it under /etc/hx-sdp and restart the affected service.

Variable	Default	Meaning
`HX_GATE_DEFAULT_RATE_LIMIT_REQUESTS`	100	Gate-wide default requests per window
`HX_GATE_DEFAULT_RATE_LIMIT_WINDOW_S`	60	Gate-wide default window seconds

Per-tenant limits and quotas are set through the onboarding API and override the defaults. See Admin and Tenants.

Upgrade¶

hx-sdp upgrade precheck --payload /path/to/payload
hx-sdp upgrade install --payload /path/to/payload
hx-sdp upgrade postcheck
hx-sdp upgrade rollback

For an agreed pilot using the documented Google Cloud image, the reference pattern is blue/green VM replacement through Terraform; see the archived Google Cloud image reference. Use the upgrade procedure supplied for your deployed release.