HX-Provenance for Vertex AI

Deploy in your own project from the Google Cloud Marketplace listing.

Documentation set

Goal	Guide
Deploy and understand the architecture	This deployment and operations guide
Create and verify a first receipt	Quickstart
Integrate a packet producer	Packet integration reference
Monitor, upgrade, retain, or remove	Operations, upgrades, and removal
Diagnose a failure	Troubleshooting
Review trust boundaries and custody	Security and data handling
Contact HolonomiX safely	Support

HX-Provenance for Vertex AI pairs a private HX-Provenance VM appliance with an internal Google Cloud adapter. A workload writes a bounded answer packet to Cloud Storage. The adapter reads the exact finalized object generation, hashes those bytes, validates the packet, and asks the appliance to sign the digest together with generation-bound context.

The solution does not replace Vertex AI, host a model, or automatically intercept every Vertex AI request. Your workload decides which outputs become packets, constructs their producer commitments, and writes them to the configured bucket.

Architecture

Open the editable SVG · Download the 2400 × 1600 PNG

The customer-owned AI workload is deliberately outside the Marketplace deployment boundary: the package does not create Vertex AI resources or an automatic exporter. Eventarc uses its managed Pub/Sub transport to invoke the internal adapter at least once. Invalid or oversized packets produce deterministic, create-only rejection evidence and bypass both the appliance and BigQuery. Valid packets use private, CA- and hostname-verified HTTPS to request an ML-DSA-65 receipt; the adapter independently verifies that receipt before creating evidence and indexing accepted metadata.

All runtime resources are designed to run in the customer Google Cloud project. The appliance has no external IP and needs no HolonomiX call-home connection for receipt issuance or verification.

What the deployment package creates

Resource	Purpose
Compute Engine VM	Private HX-Provenance signing and verification appliance
Dedicated VPC and subnet	Private path between the adapter and appliance
Private Cloud DNS zone and record	Appliance hostname matching its TLS certificate SAN
Cloud Run service	Internal, one-instance/one-concurrent-request adapter
Eventarc trigger	Delivers generation-bearing artifact object-finalize events
Artifact bucket	Receives Vertex AI answer packets under the configured prefix
Evidence bucket	Stores create-only signed receipt JSON and deterministic packet-rejection JSON
BigQuery dataset and table	Indexes receipt and source-generation metadata
Four Secret Manager containers	Admin key, ML-DSA-65 public key, public-key fingerprint, and deployment CA certificate
Dedicated service accounts	Separate appliance, adapter, and trigger identities; optional packet producer
Firewall rules	Permit private adapter HTTPS and IAP administration only

The buckets use uniform bucket-level access, public access prevention, object versioning, and configured lifecycle eligibility. The appliance uses Shielded VM controls, OS Login, blocked project SSH keys, and no public network interface.

Processing and receipt bindings

For each accepted finalize event, the adapter:

1. preserves (bucket, object name, generation) from the event;
2. reloads and downloads that exact generation with generation-match preconditions;
3. validates the packet under configured byte, depth, node, string, and source-count bounds;
4. computes SHA-256 over the exact downloaded packet bytes;
5. derives event_key from the bucket, object name, and generation;
6. submits artifact_id = gcs-finalized:<event_key>, the packet-object digest, and exact signed context to the appliance;
7. independently verifies the returned receipt before storage; and
8. creates evidence with a create-only object-generation precondition and MERGEs one index row by event_key.

Signed context includes the packet schema and ID; packet creation time; provider and model; producer-declared content, prompt, output, and source-set commitments; source IDs; adapter version; source bucket, name, generation, and event ID; event_key; and the independently computed packet-object SHA-256.

Before setting self_verified to true or writing new evidence, the adapter verifies:

• a supported receipt schema and ML-DSA-65 / FIPS 204 declaration;
• the ML-DSA-65 signature over the canonical signed body;
• exact equality of the submitted artifact ID, digest, and signed context, including source generation;
• the embedded public key's SHA-256 against the receipt fingerprint;
• equality of the embedded key to separately loaded pinned public-key bytes; and
• equality of that pinned key's digest to a separately loaded expected fingerprint.

The hx.verifiable-ai.rag-packet/v1 parser validates the shape and format of producer commitments, but it intentionally does not recompute their nested preimages. The receipt proves those values were included in the generation-bound processing request; it does not independently prove that they truthfully describe the prompt, output, model, or retrieval event. The packet-object SHA-256 remains the adapter's independently computed commitment.

Prerequisites

You need:

• a Google Cloud project with billing enabled;
• permission to procure and deploy VM products from Google Cloud Marketplace;
• permission to create Compute Engine, VPC, Cloud DNS, Cloud Run, Eventarc, Cloud Storage, BigQuery, Secret Manager, IAM, and service-account resources;
• Vertex AI access for the workload identity that invokes your selected model;
• IAP and OS Login access for appliance administrators;
• an available RFC1918 subnet CIDR and appliance address; and
• organization policies compatible with the listed resources.

The deployment module enables these Google Cloud APIs in the customer project:

aiplatform.googleapis.com
bigquery.googleapis.com
cloudresourcemanager.googleapis.com
compute.googleapis.com
dns.googleapis.com
eventarc.googleapis.com
iam.googleapis.com
iap.googleapis.com
pubsub.googleapis.com
run.googleapis.com
secretmanager.googleapis.com
serviceusage.googleapis.com
storage.googleapis.com

The immutable adapter image is built and published separately, so Artifact Registry enablement is a publisher-side image pipeline concern rather than a service enabled by this customer deployment module. Google Cloud infrastructure, Vertex AI, Cloud Run, Cloud Storage, BigQuery, networking, logging, and related usage charges are separate from the HX-Provenance software fee.

Deploy after Marketplace publication

Use this flow only after the distinct product has verified external identities, a signed release package, successful Producer Portal preview and buyer-flow evidence, and live documentation.

1. Open HX-Provenance for Vertex AI in Google Cloud Marketplace.
2. Select the customer project, region, and Compute Engine zone.
3. Choose a unique deployment prefix.
4. Confirm the dedicated subnet CIDR and private appliance address do not overlap existing routes.
5. Select the appliance machine type and boot-disk size. The baseline is e2-standard-2 with a 30 GiB balanced persistent disk.
6. Optionally select a packet-producer service account. It receives only artifact-bucket object creation access.
7. Review the deployment preview, including IAM changes.
8. Deploy, retain the outputs, and wait for the bootstrap status command to print READY before uploading packets.

Record the deployment output values rather than inferring resource names:

export PROJECT_ID="your-project-id"
export REGION="us-central1"
export ZONE="us-central1-a"
export APPLIANCE="deployment-output-appliance-name"
export APPLIANCE_HOSTNAME="deployment-output-private-hostname"
export ARTIFACT_BUCKET="deployment-output-artifact-bucket"
export EVIDENCE_BUCKET="deployment-output-evidence-bucket"
export BQ_TABLE="project.dataset.receipts"
export ADAPTER_SERVICE="deployment-output-adapter-service"
export EVENTARC_TRIGGER="deployment-output-eventarc-trigger"
export ADMIN_SECRET="deployment-output-admin-secret"
export PUBLIC_KEY_SECRET="deployment-output-public-key-secret"
export PUBLIC_KEY_SHA256_SECRET="deployment-output-public-key-sha256-secret"
export CA_CERTIFICATE_SECRET="deployment-output-ca-certificate-secret"

Validate the deployment

Confirm the VM is private and Shielded VM protections are enabled:

gcloud compute instances describe "$APPLIANCE" \
  --project="$PROJECT_ID" \
  --zone="$ZONE" \
  --format='yaml(name,hostname,networkInterfaces,shieldedInstanceConfig,serviceAccounts)'

Confirm the baked first boot, isolated bootstrap, manifest, and services through IAP:

gcloud compute ssh "$APPLIANCE" \
  --project="$PROJECT_ID" \
  --zone="$ZONE" \
  --tunnel-through-iap \
  --command='sudo test -f /etc/hx-provenance/.first-boot.done &&
    sudo test -f /etc/hx-provenance/.vertex-ai-marketplace-bootstrap-v1.done &&
    sudo systemctl is-active hx-provenance.service nginx &&
    sudo /opt/hx-provenance/bin/verify_manifest.py \
      --manifest /opt/hx-provenance/MANIFEST.sha256 \
      --root /opt/hx-provenance'

Confirm the adapter and trigger exist:

gcloud run services describe "$ADAPTER_SERVICE" \
  --project="$PROJECT_ID" --region="$REGION"

gcloud eventarc triggers describe "$EVENTARC_TRIGGER" \
  --project="$PROJECT_ID" --location="$REGION"

Do not upload packets until the bootstrap marker, manifest, and service checks pass.

Secret bootstrap and custody

Terraform creates four empty Secret Manager containers. After the image's baked first boot generates the appliance admin key and per-instance ML-DSA-65 keypair, the isolated VM startup bootstrap creates a deployment-local CA and matching private-DNS leaf certificate. It then adds secret versions for exactly:

• the admin key;
• the raw ML-DSA-65 public key;
• the lowercase SHA-256 fingerprint of that public key; and
• the CA certificate.

These values are published by the VM after first boot, not passed through Terraform variables or stored in Terraform state. The ML-DSA-65 signing private key, CA private key, and TLS leaf private key remain on the VM. Bootstrap reads versions/latest before adding a version, avoids duplicate versions on retry, and marks readiness only after all four latest values match local material.

Cloud Run uses versions/latest for the active coordinated set. Do not print secret values into shell history, shared logs, tickets, screenshots, or CI output. See Security and data handling for rotation and custody requirements.

Connect a Vertex AI workload

The integration boundary is one JSON object under the artifact prefix, which defaults to packets/. The workload—not the appliance—creates this packet.

A minimal valid packet is:

{
  "schema": "hx.verifiable-ai.rag-packet/v1",
  "envelope": {
    "packet_id": "rag-answer-20260101-000001",
    "created_utc": "2026-01-01T00:00:00Z"
  },
  "content": {
    "task": "rag-answer",
    "model": {
      "provider": "vertex-ai",
      "model": "gemini-model-id"
    },
    "prompt": {
      "prompt_sha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
    },
    "retrieval": {
      "k": 2,
      "source_set_sha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
      "sources": [
        {
          "id": "source-id-1"
        },
        {
          "id": "source-id-2"
        }
      ]
    },
    "output": {
      "output_sha256": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc"
    }
  },
  "content_sha256": "dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd"
}

The repeated hex values are format examples only. Replace them with lowercase SHA-256 commitments computed by your workload.

Validation requires:

• exact schema hx.verifiable-ai.rag-packet/v1 and content.task: rag-answer;
• valid UTF-8 JSON with an object root, no duplicate keys, and no non-finite numbers;
• a real RFC3339 UTC created_utc ending in Z or +00:00;
• bounded ASCII identifiers for packet, provider, model, and sources;
• lowercase 64-hex SHA-256 strings for all four commitments;
• a non-negative integer retrieval.k within the source limit;
• no more source objects than retrieval.k; and
• unique source IDs.

Extra JSON fields are permitted but remain subject to the configured bounds. The default limits are 4 MiB per packet, JSON depth 24, 100,000 JSON nodes, 1,000,000 characters per string, and 256 sources. Deployment variables can lower or raise these values only within the package's validated ranges.

Use a unique object name for each logical packet and retain its finalized source tuple:

export OBJECT_NAME="packets/rag-answer-20260101-000001.json"
gcloud storage cp packet.json \
  "gs://${ARTIFACT_BUCKET}/${OBJECT_NAME}"

The adapter processes only JSON objects under the configured prefix. Each finalized generation has its own identity, but producer truthfulness and sensible object naming remain customer responsibilities.

Retrieve the receipt and index row

The safest discovery path is BigQuery because the index stores the complete source tuple and receipt URI:

bq --project_id="$PROJECT_ID" query --use_legacy_sql=false \
  --parameter="source_bucket::${ARTIFACT_BUCKET}" \
  --parameter="source_name::${OBJECT_NAME}" \
  "SELECT event_key, source_bucket, source_object_name, source_generation,
          packet_id, packet_object_sha256, receipt_uri,
          receipt_object_generation, receipt_object_sha256,
          receipt_public_key_sha256, self_verified, receipt_issued_utc
   FROM \`${BQ_TABLE}\`
   WHERE source_bucket = @source_bucket
     AND source_object_name = @source_name
   ORDER BY ingested_utc DESC"

Do not derive receipt paths from packet IDs. With the default receipt prefix, the object name is:

receipts/<first-two-event-key-hex>/<64-hex-event-key>.receipt.json

To derive event_key, encode the bucket and object name as UTF-8. Start with the exact domain separator HX-GCS-FINALIZED-EVENT-V1 followed by one NUL byte. Append the bucket byte length as an unsigned 8-byte big-endian integer and then the bucket bytes; append the object-name length and bytes the same way; append generation as an unsigned 8-byte big-endian integer. SHA-256 the result. The Eventarc event ID is not part of this key.

Given an authoritative generation from the index or finalize event:

export GENERATION="1234567890123456"
EVENT_KEY="$(
  python3 - "$ARTIFACT_BUCKET" "$OBJECT_NAME" "$GENERATION" <<'PY'
import hashlib
import sys

bucket = sys.argv[1].encode("utf-8")
name = sys.argv[2].encode("utf-8")
generation = int(sys.argv[3], 10)
if not 0 < generation < 2**63:
    raise SystemExit("generation must be a positive signed 64-bit integer")
canonical = bytearray(b"HX-GCS-FINALIZED-EVENT-V1\0")
for value in (bucket, name):
    canonical.extend(len(value).to_bytes(8, "big", signed=False))
    canonical.extend(value)
canonical.extend(generation.to_bytes(8, "big", signed=False))
print(hashlib.sha256(canonical).hexdigest())
PY
)"
RECEIPT_OBJECT="receipts/${EVENT_KEY:0:2}/${EVENT_KEY}.receipt.json"
gcloud storage cp \
  "gs://${EVIDENCE_BUCKET}/${RECEIPT_OBJECT}" \
  receipt.json

Use the configured receipt prefix instead of receipts/ if your deployment changed it.

Retry, rejection, and idempotency behavior

Eventarc delivery remains at least once. The serialized v1.0.0 deployment uses a bounded 600-second Cloud Run request deadline and provides application idempotency as follows:

• event_key deterministically binds bucket, object name, and generation;
• receipt names are derived from event_key;
• receipt upload uses a create-only generation precondition;
• a retry finds, generation-pins, parses, and independently re-verifies existing canonical receipt bytes rather than requesting a new timestamped receipt;
• a create race succeeds only when the existing bytes are identical;
• BigQuery uses MERGE ... ON event_key and leaves a matching row unchanged; and
• permanent packet validation or packet-size-limit failures create deterministic, create-only rejections/<first-two-event-key-hex>/<event-key>.rejection.json evidence, then return 2xx so Eventarc does not repeatedly deliver poison input.

A rejection acknowledgement is sent only after the rejection object persists. Failure to create or re-read rejection evidence remains a retryable 5xx. Rejection bytes contain only the event key, generation-specific source tuple, Cloud Storage size/checksum metadata, a schema, and a fixed outcome code. Delivery event IDs, timestamps, adapter versions, parser details, packet hashes, and configurable policy limits are omitted so retries and later adapter revisions converge on identical evidence. Once present, a validated rejection is authoritative for that finalized generation.

These controls make retries converge for this one-instance/one-concurrent-request deployment. They do not change Eventarc's delivery contract. Monitor failed deliveries and reconcile artifact generations against receipt, rejection, and index records.

Verify a receipt offline

Use verify_receipt.py from the release-matched, hash-checked verifier package distributed with the signed release. Do not copy a verifier from the appliance VM.

Obtain the raw public key and expected fingerprint through authorized Secret Manager access or another independent trusted channel:

gcloud secrets versions access latest \
  --project="$PROJECT_ID" \
  --secret="$PUBLIC_KEY_SECRET" \
  > issuer-ml-dsa-65.pub

EXPECTED_FINGERPRINT="$(
  gcloud secrets versions access latest \
    --project="$PROJECT_ID" \
    --secret="$PUBLIC_KEY_SHA256_SECRET" \
    | tr -d '\r\n'
)"
[[ "$EXPECTED_FINGERPRINT" =~ ^[0-9a-f]{64}$ ]]

From the release verifier directory:

python3 -m venv verifier-venv
. verifier-venv/bin/activate
python -m pip install --requirement requirements.txt
python verify_receipt.py \
  --receipt receipt.json \
  --public-key issuer-ml-dsa-65.pub \
  --expected-fingerprint "$EXPECTED_FINGERPRINT" \
  --artifact packet.json

Use --artifact-sha256 instead of --artifact when a separate trusted workflow supplied the packet digest. Both normal issuer-verification inputs—--public-key and --expected-fingerprint—are required. Explicit --integrity-only mode is weaker: it checks against the embedded key but does not establish issuer identity.

Verify through the appliance

Retrieve the deployment CA, then open the IAP tunnel in a separate terminal:

gcloud secrets versions access latest \
  --project="$PROJECT_ID" \
  --secret="$CA_CERTIFICATE_SECRET" \
  > deployment-ca.crt

gcloud compute start-iap-tunnel "$APPLIANCE" 443 \
  --project="$PROJECT_ID" \
  --zone="$ZONE" \
  --local-host-port=127.0.0.1:8443

Use the private hostname in the HTTPS URL so hostname verification checks the certificate SAN:

jq -n \
  --slurpfile receipt receipt.json \
  --arg fingerprint "$EXPECTED_FINGERPRINT" \
  '{receipt: $receipt[0], pinned_public_key_sha256: $fingerprint}' \
  > verify-request.json

curl --fail --silent --show-error \
  --resolve "${APPLIANCE_HOSTNAME}:8443:127.0.0.1" \
  --cacert deployment-ca.crt \
  -H 'Content-Type: application/json' \
  --data @verify-request.json \
  "https://${APPLIANCE_HOSTNAME}:8443/v1/provenance/verify" \
  | tee verification.json

jq -e '.verified == true and .key_pinned == true' verification.json

The verification endpoint can return HTTP 200 for a rejected receipt. Automation must check the JSON verdict.

Export and verify an evidence bundle

Evidence-bundle export requires authorized access to the admin-key secret:

(
  set +x
  umask 077
  AUTH_HEADER="$(mktemp)"
  cleanup() {
    rm -f "$AUTH_HEADER"
    unset ADMIN_KEY
  }
  trap cleanup EXIT
  trap 'exit 130' INT
  trap 'exit 143' HUP TERM

  ADMIN_KEY="$(
    gcloud secrets versions access latest \
      --project="$PROJECT_ID" \
      --secret="$ADMIN_SECRET"
  )"
  printf 'Authorization: Bearer %s\n' "$ADMIN_KEY" > "$AUTH_HEADER"
  unset ADMIN_KEY

  jq -n \
    --slurpfile receipt receipt.json \
    '{receipt: $receipt[0], artifact_id: $receipt[0].body.extras.artifact_id, fmt: "zip"}' \
    > bundle-request.json

  curl --fail --silent --show-error \
    --resolve "${APPLIANCE_HOSTNAME}:8443:127.0.0.1" \
    --cacert deployment-ca.crt \
    --header @"$AUTH_HEADER" \
    -H 'Content-Type: application/json' \
    --data @bundle-request.json \
    "https://${APPLIANCE_HOSTNAME}:8443/v1/provenance/evidence-bundle" \
    > bundle-response.json

  jq -e '.verified == true' bundle-response.json
  jq -r '.bundle_b64' bundle-response.json \
    | base64 --decode > evidence-bundle.zip
)

Use the release-matched bounded bundle verifier:

python verify_bundle.py \
  --bundle evidence-bundle.zip \
  --public-key issuer-ml-dsa-65.pub \
  --expected-fingerprint "$EXPECTED_FINGERPRINT"

verify_bundle.py supports json-dir, ZIP, and tar.zst bundles. It verifies signed manifest integrity, exact membership, hashes and sizes, signer equality, and contained receipts without extracting archive entries. Its entry-count, per-entry, total-byte, and compression-ratio limits are configurable downward or within documented safe ranges. As with receipt verification, --integrity-only is an explicit weaker mode.

Operations

Check service status and recent appliance logs:

gcloud compute ssh "$APPLIANCE" \
  --project="$PROJECT_ID" \
  --zone="$ZONE" \
  --tunnel-through-iap \
  --command='sudo systemctl status hx-provenance.service nginx --no-pager;
    sudo journalctl -u hx-provenance.service --since "30 minutes ago" --no-pager;
    sudo journalctl -t hx-vertex-ai-bootstrap --since "30 minutes ago" --no-pager'

Inspect adapter logs without exporting packet contents:

gcloud run services logs read "$ADAPTER_SERVICE" \
  --project="$PROJECT_ID" --region="$REGION" --limit=100

Monitor appliance service state, Eventarc delivery failures, adapter 5xx responses, immutable packet rejection objects, artifact generations without receipt or rejection evidence, receipt create conflicts, signer-fingerprint changes, and TLS certificate expiry.

Upgrade and signer continuity

Treat each Marketplace image as immutable. Validate a replacement before moving packet production.

Active adapter secret references use versions/latest. Before replacing an appliance, record the current numeric versions of the public-key and public-key-fingerprint secrets and supply both as previous_public_key_version and previous_public_key_sha256_version. Keep allow_data_destroy=false throughout normal rotation. First set allow_appliance_replacement=true and apply without changing the image. Then update the verified image identity and apply the appliance replacement. After bootstrap reports READY, restore allow_appliance_replacement=false and apply immediately. This narrowly lowers only VM deletion protection; it never enables bucket or BigQuery purge controls. The replacement bootstrap adds new active secret versions without mutating or destroying old versions.

The previous pair is accepted only to re-verify already stored immutable receipts during the bounded rotation. New issuance must verify against the active pair. Rotate admin and TLS material as one coordinated appliance/secret update; partial sets fail closed. Retain historical public keys and fingerprints for as long as historical receipts must remain verifiable. Never transfer an ML-DSA signing key or TLS private key through support channels.

Remove the deployment

Before deletion:

1. stop packet producers;
2. wait for in-flight Eventarc deliveries;
3. export required receipts, bundles, index rows, public keys, and fingerprints;
4. decide whether to retain or purge versioned bucket objects and BigQuery data; and
5. document final signing-key custody.

allow_data_destroy=false is the safe data-retention default: Cloud Run deletion protection and nonempty protected data stores block destructive teardown. Appliance deletion protection remains enabled unless the appliance-only replacement switch or the destructive purge switch is explicitly active. An intentional purge requires keeping allow_appliance_replacement=false, setting allow_data_destroy=true, applying that change, confirming retention and legal requirements, and only then destroying the deployment. Once destruction begins, the purge is irreversible and removes the VM boot disk with its on-disk private keys.

Claim boundaries

HX-Provenance verifies cryptographic commitments and signatures. It does not independently establish:

• that a model output is factually correct, safe, unbiased, or policy compliant;
• that producer-supplied packet fields truthfully describe events before packet creation;
• that a model, prompt, retrieval source, or artifact was authorized;
• that every Vertex AI request was captured;
• that a receipt satisfies a particular legal or regulatory requirement; or
• that an embedded public key belongs to an expected issuer without external pinning.

Customers retain responsibility for packet truthfulness, retention, monitoring, and key custody. See Security and data handling for the trust boundary and Support for troubleshooting and contact requirements.