HX-AEIR on Google Cloud¶

HX-AEIR (Agent Evaluation Incident Reconstruction) is a private Google Cloud VM appliance for validating, signing, storing, exporting, and independently verifying evidence used to reconstruct agent evaluation incidents. It runs inside your Google Cloud project, uses a customer-local ML-DSA-65 signing key, and requires no HolonomiX runtime service for normal issuance or verification.

Distribution and validation boundary

This page documents the validated customer-project behavior of HX-AEIR 1.0.1. Private cross-project image validation does not establish Google Cloud Marketplace approval, a product license, Marketplace-owned image-copy validation, or Producer Portal acceptance. Deploy only through a distribution path explicitly authorized for your organization.

HX-AEIR records and signs supplied evidence. It does not execute, sandbox, monitor, block, or contain evaluated agents. Runtime prevention by HX-AEIR is NONE.

Deployment model¶

Component	Description
Product	HX-AEIR
Runtime location	Customer-controlled Google Cloud project
Compute	Single Shielded Compute Engine VM
Availability model	Single zone and single VM; no automatic failover
Terraform baseline	Terraform `>=1.9.8,<2.0.0`; Google provider `7.43.0`
Default machine	`e2-standard-2`
Default disk	Separate 50 GiB `pd-balanced` boot/data disk
Public address	None by default
Administrative ingress	Optional IAP-only SSH with OS Login
Application ingress	No HTTPS firewall rule unless explicit CIDRs are supplied
Production authentication	Mutual TLS on TCP 443
Front end	Nginx TLS, route, and request boundary
Application transport	Group-restricted Unix socket at `/run/hx-aeir/hx-aeir.sock`
Signing algorithm	ML-DSA-65
Signing-key model	Customer-local key created only by an explicit root-authorized action
Runtime vendor dependency	None for issuance, export, or verification
Evidence custody	Customer-controlled disk, backups, exports, and retention policy

Prerequisites¶

Before deployment, prepare:

a dedicated Google Cloud project with billing and the required Compute Engine, IAM, IAP, OS Login, Service Usage, and Resource Manager APIs available;
an approved identity able to create the reviewed networks, subnetworks, firewall rules, addresses, disks, VM, and dedicated service account;
authorized access to the exact reviewed Terraform package and qualified image for the intended release;
a region and matching zone, machine type, disk policy, and capacity plan;
an approved IAP and OS Login administrative path;
exact HTTPS client CIDRs if direct network ingress is required;
a server certificate, matching private key, DNS name, client CA, and client certificates for mutual TLS;
a governed issuer identifier and separate owners for signing-key, TLS, evidence, backup, and recovery custody;
a protected external location for the pinned public key and initialization record; and
evidence-retention, export, backup, restore, and destruction decisions.

The Terraform defaults do not assign a public IPv4 address, do not permit web ingress, enable VM deletion protection, leave signing-key initialization disabled, and retain the boot disk after out-of-band instance deletion. A normal Terraform destroy can still delete the separately managed disk unless your approved state-retention process preserves it.

Optional CMEK configures Google Cloud persistent-disk encryption. It does not create, wrap, replace, or manage the HX-AEIR ML-DSA-65 signing key.

Deploy the appliance¶

Current delivery is Private Appliance or a scoped pilot; Google Cloud Marketplace distribution is planned. Request HX-AEIR access if your organization has not received an authorized package. The current-release checklist identifies the materials to obtain before deployment.

Use the exact deployment package supplied through your approved channel. If your organization receives a Marketplace listing, verify the listing and entitlement in your own tenant before deployment; this page is not evidence that a listing or license is active.

Review all inputs and the generated plan before applying:

terraform version
terraform init
terraform validate
terraform plan -out=hx-aeir.tfplan
terraform apply hx-aeir.tfplan

The maintained package creates a custom-mode VPC, one regional subnet, a dedicated service account with no project IAM grants, optional IAP-only SSH ingress, optional CIDR-restricted HTTPS ingress, an optional reserved public address, one persistent boot/data disk, and one Shielded VM.

After apply, record the following in your controlled change record:

instance name, zone, machine type, and deployment timestamp;
resolved image self-link and numeric image ID;
canonical release-bundle SHA-256 reconstructed from the image labels;
disk name, type, size, and actual source-image ID;
VPC, subnet, firewall rules, and internal or approved external address;
dedicated service-account identity and its effective IAM grants;
deletion-protection, ingress, and initialization settings; and
Terraform plan, apply result, state location, and operator approval.

The package checks that the created disk came from the same image incarnation resolved during deployment and that the image labels bind the expected canonical release digest. Do not replace those checks with a mutable image family or unpinned artifact.

Inspect the resulting VM from an authorized workstation:

gcloud compute instances describe "$INSTANCE_NAME" \
  --zone="$ZONE"

gcloud compute disks describe "$BOOT_DISK" \
  --zone="$ZONE"

Confirm that the VM has no public address unless one was explicitly approved, HTTPS ingress is absent or limited to approved CIDRs, Secure Boot/vTPM/integrity monitoring are enabled, project-wide SSH keys and serial-console access are disabled, and the runtime service account has no unintended roles.

Connect for administration¶

When IAP SSH is enabled, connect through your approved IAP and OS Login path:

gcloud compute ssh "$INSTANCE_NAME" \
  --zone="$ZONE" \
  --tunnel-through-iap

Do not add direct TCP 22 ingress for convenience. Root administrators are inside the appliance trust boundary and can replace application, evidence, or key material; restrict and audit that access accordingly.

First boot and signing custody¶

The neutral image is intentionally keyless. First boot does not silently generate a signing key, mint bearer or admin API keys, or place credentials in serial output. The application remains unavailable until a customer explicitly initializes signing custody.

Before initialization, inspect rather than repeatedly restarting services:

sudo systemctl status \
  hx-aeir.service \
  hx-aeir-reconcile.service \
  nginx.service \
  --no-pager

Initialize a customer-local key and start the application:

sudo /usr/local/sbin/hx-aeir-initialize \
  --issuer-id 'org:customer-controlled' \
  --start

Use an issuer identifier governed by your organization. To keep it out of shell history, place it in a root-readable regular file through your approved secret-handling process and use --issuer-id-file instead.

Initialization validates an existing complete key rather than replacing it. Capture the emitted key ID and public-key fingerprint through an independently controlled channel. Do not begin production issuance until the public key is pinned outside the VM and signing-key recovery has been exercised.

Configure mutual TLS¶

Production default is mutual TLS. Place the server certificate, matching private key, and client CA on the VM through an approved administrative channel, then activate them:

sudo /usr/local/sbin/hx-aeir-configure-tls \
  --certificate /root/tls/server.crt \
  --private-key /root/tls/server.key \
  --server-name aeir.customer.example \
  --client-ca /root/tls/client-ca.crt \
  --enable-hsts

The command validates certificate dates, hostname and purpose, private-key match, client-CA purpose, generated Nginx policy, and durable activation before reload. Validate the active Nginx configuration afterward:

sudo nginx -t
sudo systemctl status nginx.service --no-pager

Before TLS activation, Nginx is loopback-only. --allow-unauthenticated-clients is an explicit exception for a private deployment behind a separately trusted authentication proxy; it is not guidance for anonymous public access.

Verify health and key identity¶

Check the backend locally over its Unix socket:

sudo curl --fail \
  --unix-socket /run/hx-aeir/hx-aeir.sock \
  http://localhost/healthz

Then test the customer-facing mutual-TLS path from an authorized client:

curl --fail --silent --show-error \
  --cacert "$SERVER_CA" \
  --cert "$CLIENT_CERT" \
  --key "$CLIENT_KEY" \
  "https://$HX_AEIR_HOST/healthz"

Confirm that health reports the expected product, release, AEIR profile, receipt envelope, and signing-key fingerprint, and that runtime_prevention_by_hx_aeir is NO. A network request without an accepted client certificate must be rejected before it reaches the application.

There is no HTTP public-key endpoint. Copy the public key from /etc/hx-aeir/keys/ml-dsa-65.public through an authenticated administrative channel. For example:

gcloud compute ssh "$INSTANCE_NAME" \
  --zone="$ZONE" \
  --tunnel-through-iap \
  --command='sudo cat /etc/hx-aeir/keys/ml-dsa-65.public' \
  > pinned-ml-dsa-65.public

chmod 0600 pinned-ml-dsa-65.public

Compare its fingerprint with the independently retained initialization record, then store the key separately from exported packages. A public key delivered beside an untrusted package is not an authoritative trust anchor.

API boundary¶

The customer-facing base URL is https://$HX_AEIR_HOST. Mutual TLS protects all externally reachable routes unless a reviewed private authentication proxy is the explicit boundary.

Method	Endpoint	Purpose
`GET`	`/healthz`	Report bounded appliance health and signing identity
`POST`	`/v1/aeir/receipts`	Validate, sign, and persist a strictly linked receipt/package
`POST`	`/v1/aeir/verify`	Verify a supplied receipt or package and return a deterministic verdict
`POST`	`/v1/aeir/export`	Export the immutable package for an exact receipt digest

Interactive API documentation and OpenAPI routes are disabled. Unknown fields are rejected. The default maximum request body is 16 MiB, and evidence-file values in API requests must be strict base64 strings.

HX-AEIR does not issue application bearer keys. Authentication is enforced at the customer-controlled Nginx mTLS boundary or by the explicitly configured trusted private upstream.

Sign required authorization and preflight documents¶

Receipt issuance requires signed authorization and preflight documents. The public HTTP surface does not create them; an authorized appliance operator uses the active release CLI with organization-reviewed JSON document bodies.

HX_AEIR_CLI=/opt/hx-aeir/current/venv/bin/hx-aeir

sudo -u hx-aeir "$HX_AEIR_CLI" issue-document \
  --document-type evaluation_authorization_manifest \
  --document-id "$AUTHORIZATION_DOCUMENT_ID" \
  --issued-at "$ISSUED_AT" \
  --issuer-id "$AUTHORIZATION_ISSUER_ID" \
  --body /var/lib/hx-aeir/import/evaluation-authorization-body.json \
  --output /var/lib/hx-aeir/import/evaluation-authorization.json

sudo -u hx-aeir "$HX_AEIR_CLI" issue-document \
  --document-type independent_preflight \
  --document-id "$PREFLIGHT_DOCUMENT_ID" \
  --issued-at "$ISSUED_AT" \
  --issuer-id "$PREFLIGHT_ISSUER_ID" \
  --body /var/lib/hx-aeir/import/independent-preflight-body.json \
  --output /var/lib/hx-aeir/import/independent-preflight.json

Set ISSUED_AT to an approved RFC 3339 timestamp, pass each governed document issuer explicitly, and use governed document identifiers. The body files are JSON objects controlled by your profile and evidence policy; issue-document wraps and signs them as hx.aeir.document/v1 envelopes. Include those outputs under the required keys in linked_receipts when preparing the issuance request. Inspect exact command arguments with "$HX_AEIR_CLI" issue-document --help.

A receipt-issuance request contains payload, linked_receipts, optional authorized_disclosure_source, and optional evidence_files. Missing or inconsistent authorization, preflight, prior-revision, source, or evidence linkages fail closed.

Verification accepts value plus any required link, source, and evidence objects. Export accepts the exact receipt_digest in sha256: form returned by successful issuance. Do not infer request structures from unrelated products or older releases.

Issue, verify, and test tamper rejection¶

Submit a schema-valid issuance request:

curl --fail --silent --show-error \
  --cacert "$SERVER_CA" \
  --cert "$CLIENT_CERT" \
  --key "$CLIENT_KEY" \
  -H 'Content-Type: application/json' \
  --data-binary @issue-request.json \
  "https://$HX_AEIR_HOST/v1/aeir/receipts" \
  --output issuance-response.json

The response contains receipt_digest, package_digest, receipt, and the issuance-time verification report; it is not itself an evidence package. Build the verification request with the response's receipt and the same required linkage/source/evidence material. For example, with jq:

jq -n \
  --slurpfile issued issuance-response.json \
  --slurpfile request issue-request.json \
  '{
    value: $issued[0].receipt,
    linked_receipts: ($request[0].linked_receipts // {}),
    authorized_disclosure_source: ($request[0].authorized_disclosure_source // null),
    evidence_files: ($request[0].evidence_files // {})
  }' > verify-request.json

curl --fail --silent --show-error \
  --cacert "$SERVER_CA" \
  --cert "$CLIENT_CERT" \
  --key "$CLIENT_KEY" \
  -H 'Content-Type: application/json' \
  --data-binary @verify-request.json \
  "https://$HX_AEIR_HOST/v1/aeir/verify" \
  --output verify-result.json

Preserve the original response. Change one signed field only under value in a separate copy, save it as tampered-verify-request.json, and submit the same verification call. A well-formed tampered request may return HTTP success with a negative cryptographic verdict, so evaluate the response body rather than treating transport success as verification success.

Export an immutable package¶

Create the export request from the exact issuance digest, then export the package:

jq '{receipt_digest}' issuance-response.json > export-request.json

curl --fail --silent --show-error \
  --cacert "$SERVER_CA" \
  --cert "$CLIENT_CERT" \
  --key "$CLIENT_KEY" \
  -H 'Content-Type: application/json' \
  --data-binary @export-request.json \
  "https://$HX_AEIR_HOST/v1/aeir/export" \
  --output exported-package.json

sha256sum exported-package.json

Store the package digest, receipt digest, release identifier, export timestamp, and authorization record in your system of record.

Verify offline¶

Move the exported package, the independently pinned public key, and any required authorized disclosure source to an isolated verifier. Install the verifier only from the reviewed release artifact and verify that artifact through your approved software-supply-chain process.

No network connection to HolonomiX, Google Cloud, or the appliance is required:

hx-aeir verify exported-package.json \
  --public-key pinned-ml-dsa-65.public \
  --format json

Exit status meanings are:

Exit	Meaning
`0`	Verification completed with a positive result
`2`	Verification completed with a negative result
`1`	Command or input processing failed

Preserve the JSON report with the reviewed evidence set. The verification report also carries an overall status of VALID, PARTIAL, REQUIRES AUTHORIZED SOURCE PACKAGE, or INVALID; the Evidence and Offline Verification guide documents each status and the full check set. For a raw receipt, also provide linked receipts with --links and evidence bytes with --evidence-map; for a derivative package requiring source authorization, add --authorized-source.

Verification establishes supported schema and semantic checks, canonical digests, ML-DSA-65 signatures under the pinned key, exact package manifests, available evidence hashes, and required linkages. It does not prove that omitted telemetry never existed, that supplied statements are factually complete, that runtime controls prevented behavior, or that a third party certifies the incident conclusion.

Validate reboot persistence¶

Keep the client-side export and VM-side service observations separate. On the authorized mTLS client, record the original package hash and retain export-request.json:

sha256sum exported-package.json

On the VM, record the boot ID and complete health response, including the signing fingerprint, then reboot:

cat /proc/sys/kernel/random/boot_id
sudo curl --fail \
  --unix-socket /run/hx-aeir/hx-aeir.sock \
  http://localhost/healthz
sudo reboot

After the VM is reachable again, reconnect through IAP and verify durable recovery on the VM:

cat /proc/sys/kernel/random/boot_id
sudo systemctl status \
  hx-aeir-reconcile.service \
  hx-aeir.service \
  nginx.service \
  --no-pager

sudo curl --fail \
  --unix-socket /run/hx-aeir/hx-aeir.sock \
  http://localhost/healthz

Confirm that the boot ID changed, reconciliation succeeded, the application and Nginx remain enabled and active, and the signing fingerprint is unchanged. From the authorized mTLS client, confirm mutual TLS and HSTS still apply and export the same receipt digest to a second file:

curl --fail --silent --show-error \
  --cacert "$SERVER_CA" \
  --cert "$CLIENT_CERT" \
  --key "$CLIENT_KEY" \
  -H 'Content-Type: application/json' \
  --data-binary @export-request.json \
  "https://$HX_AEIR_HOST/v1/aeir/export" \
  --output exported-package.after-reboot.json

sha256sum exported-package.json exported-package.after-reboot.json
cmp --silent exported-package.json exported-package.after-reboot.json

Repeat pinned-key offline verification against exported-package.after-reboot.json and repeat tamper rejection. Retain the before/after boot IDs, health responses, package hashes, comparison result, and verification reports in the acceptance record.

Operational acceptance checklist¶

Before production use, confirm:

Runtime architecture¶

mTLS client or trusted private upstream
                  |
               TCP 443
                  |
        Nginx TLS/request boundary
                  |
  /run/hx-aeir/hx-aeir.sock (Unix socket)
                  |
    unprivileged hx-aeir application
          |                    |
 /var/lib/hx-aeir       /etc/hx-aeir/keys
 evidence store          customer ML-DSA-65

Nginx accepts only GET /healthz and POST /v1/aeir/*, applies the 16 MiB request limit, and returns 404 for other paths. The application is restricted to AF_UNIX under the shipped systemd policy and cannot open an IP listener. Application access logging is disabled; bounded service errors may still reach the system journal.

A normal evidence flow is:

The customer prepares authorization, preflight, incident, linkage, and optional evidence material.
HX-AEIR validates syntax and cross-document semantics.
Canonical JSON and evidence digests are computed deterministically.
The customer-local ML-DSA-65 key signs the receipt and package.
Immutable objects and append-only indexes are persisted on the customer disk.
A verifier receives the package, independently pinned public key, and required linked/source material and verifies offline.

Evidence does not transit a vendor runtime service. Customer-selected networking, snapshots, backups, log sinks, exports, and support sharing can move data and must be governed separately.

Signing, storage, and recovery posture¶

Path	Purpose
`/opt/hx-aeir/releases`	Immutable application releases
`/opt/hx-aeir/current`	Atomic link to the active release
`/opt/hx-aeir/previous`	Atomic link to the rollback release
`/var/lib/hx-aeir`	Evidence store, imports, exports, and durable application state
`/etc/hx-aeir/keys`	Customer-local ML-DSA-65 key material
`/run/hx-aeir/hx-aeir.sock`	Ephemeral group-restricted application socket

Evidence, signing keys, TLS state, and software releases have separate custody and lifecycle rules. Normal backups exclude the signing private key. A backup that explicitly includes it preserves issuance continuity but becomes highly sensitive. A data-only restore requires the same complete signing key to exist at the destination.

Pin prior public keys for historical verification. Replacing or losing a private key affects future issuance continuity; it does not make prior packages independently verifiable without the corresponding retained public key.

Backup and restore¶

A normal backup contains the evidence store, non-secret appliance configuration, public signing key, key metadata, and release metadata. It excludes the signing private key and all TLS private-key material:

sudo /usr/local/sbin/hx-aeir-backup \
  --output /var/backups/hx-aeir/hx-aeir-data.tar.gz

Create a continuity backup only when its destination is encrypted, tightly access-controlled, and approved for signing-key custody:

sudo /usr/local/sbin/hx-aeir-backup \
  --output /var/backups/hx-aeir/hx-aeir-continuity.tar.gz \
  --include-signing-key

The command prints the archive SHA-256. Record that digest outside the VM through an independently controlled channel. TLS certificates and private keys are never included and must be recovered separately.

For an approved recovery test, copy the archive to a root-controlled path on an isolated compatible appliance and obtain the expected digest from the independent record:

sudo /usr/local/sbin/hx-aeir-restore \
  --backup /root/restore/hx-aeir-continuity.tar.gz \
  --sha256 "$EXPECTED_BACKUP_SHA256" \
  --start

sudo /usr/local/sbin/hx-aeir-initialize --start

Restore validates the outer digest, safe archive structure, internal manifest, signer inputs, stored packages, signatures, disclosure-source availability, and exact store closure before committing data. On a pristine recovery appliance, restore --start starts and health-checks the application but does not enable it or start Nginx. The following initialization command validates the restored complete key without replacing it and enables both services for reboot persistence. A continuity archive carrying a different key is refused when a complete target key already exists; --replace-signing-key is reserved for an explicitly governed signer transition that also updates verifier pins.

After restore and initialization, re-establish TLS separately, compare the fingerprint to the expected pin, audit the store, export and independently verify selected packages, reboot, and repeat health and verification checks. A backup is not considered recoverable until this isolated exercise succeeds.

Upgrade and rollback¶

Use only an independently qualified canonical bundle with an approved SHA-256 and compatibility statement. Before an upgrade, create and verify a continuity backup, record current release and key identity, confirm capacity and health, and define rollback criteria.

sudo /usr/local/sbin/hx-aeir-upgrade \
  --bundle /root/releases/hx-aeir-appliance.tar.gz \
  --sha256 "$EXPECTED_RELEASE_SHA256"

Afterward, validate service state, Nginx, health, representative export, and pinned-key offline verification before issuing new production records:

systemctl status hx-aeir.service --no-pager
sudo curl --fail \
  --unix-socket /run/hx-aeir/hx-aeir.sock \
  http://localhost/healthz
sudo nginx -t

Rollback switches to the one recorded previous immutable release without replacing the evidence store or signing key:

sudo /usr/local/sbin/hx-aeir-rollback

Do not roll back to software that is incompatible with current data or security policy. Do not bypass hx-aeir-reconcile.service, edit release links manually, or delete transaction markers to force startup.

Uninstall and cloud cleanup¶

The safe appliance uninstall removes application binaries and service integration while retaining evidence, signing/configuration material, and backups:

sudo /usr/local/sbin/hx-aeir-uninstall

A complete local purge is explicit and irreversible from that VM:

sudo /usr/local/sbin/hx-aeir-uninstall \
  --purge-data \
  --purge-signing-key \
  --purge-backups \
  --remove-account \
  --yes

Before purge or Terraform destroy, export required packages, preserve the public keys needed for historical verification, create and test any authorized archive, and record retention or destruction approval.

VM deletion protection is enabled by default and requires a reviewed change before destroy. Decide whether the managed disk must be retained before changing Terraform state. Then account separately for the VM, disk, reserved address, VPC, subnet, firewall rules, service account, snapshots, backups, KMS grants, DNS, certificates, monitoring records, exported packages, and external key escrow.

Cloud deletion does not erase copied archives, snapshots, log sinks, or externally retained cryptographic material. Record the final resource inventory and verify that no unintended listener, instance, disk, address, or service-account grant remains.

Security and shared responsibility¶

HX-AEIR ships with explicit signing-key initialization, restrictive key permissions, Unix-socket-only application traffic, hardened systemd policy, Nginx TLS 1.2/1.3 and mutual TLS, bounded requests and timeouts, Shielded VM controls, OS Login, optional IAP-only SSH, no module-granted project role for the VM identity, durable fail-closed reconciliation, exact release manifests, and a redacting support-bundle tool.

These controls are not a certification or guarantee against compromise. Customers remain responsible for project and root access, certificate lifecycle, public-key pinning, signing-key backup protection, evidence authorization and provenance, ingress policy, vulnerability review, monitoring, capacity, retention, and disclosure decisions.

HX-AEIR is not a sandbox, firewall, identity provider, policy engine, endpoint detector, or runtime containment mechanism. External systems and operators perform prevention, monitoring, response, and containment.

Support diagnostics¶

Before escalation, collect bounded local diagnostics without repeatedly restarting a failed appliance:

systemctl status \
  hx-aeir.service \
  hx-aeir-reconcile.service \
  nginx.service \
  --no-pager
sudo nginx -t
sudo curl --fail \
  --unix-socket /run/hx-aeir/hx-aeir.sock \
  http://localhost/healthz

Generate a local support bundle only when needed:

sudo /usr/local/sbin/hx-aeir-support-bundle \
  --output /var/backups/hx-aeir/hx-aeir-support.tar.gz

The tool is designed to exclude evidence bodies and private keys and to redact common secrets and identifiers, but redaction is defense in depth rather than authorization to transmit. Extract the archive in a controlled environment, inspect every file, apply your disclosure policy, and send it only through the approved support channel supplied with your distribution agreement. Never include signing keys, TLS private keys, credentials, regulated evidence, or unsanitized customer data.

Support URLs, hours, severity definitions, response targets, and escalation terms are contractual inputs. This technical page does not create or imply a service level.

Release and validation boundary¶

The HX-AEIR 1.0.1 customer workflow has been exercised from a privately shared image into an independent customer-style Google Cloud project with no public VM address, IAP-only administration, a no-role runtime identity, explicit key initialization, mutual TLS, issuance, positive and negative verification, immutable export, isolated offline verification, and reboot persistence.

That result validates the documented private cross-project candidate behavior. It does not claim Marketplace approval, licensed Marketplace image admission, Marketplace-owned image-copy validation, or Producer Portal completion. Those are separate publication and commercial gates.