Skip to content

Security and data handling

HX-Provenance for Vertex AI is designed to run in the customer Google Cloud project. HolonomiX does not require routine access to workload packets, receipts, evidence, API keys, signing keys, or deployed Google Cloud resources.

Trust boundary

Customer project
  Vertex AI workload and packet producer
  Cloud Storage artifact bucket
  Eventarc
  internal serialized Cloud Run adapter
  private Cloud DNS zone
  private HX-Provenance VM
  Secret Manager
  Cloud Storage evidence bucket
  BigQuery receipt index

Google Cloud control-plane and managed-service boundaries still apply. No HolonomiX call-home dependency for issuance or verification does not mean the deployment is disconnected from the Google APIs used by its configured workflow.

The adapter is a trusted signing delegate. It can read scoped packet generations, authenticate to the appliance, request receipts for accepted packets, create evidence objects, and mutate the receipt-index dataset. Its identity and runtime therefore require the same monitoring and change control as other evidence-producing services.

Data flow

Data Location and handling
Vertex AI request and output Controlled by the customer workload and Vertex AI configuration
Answer packet Stored in the artifact bucket; exact finalized generation is read with generation-match preconditions
Packet-object digest SHA-256 computed by the adapter over the exact generation bytes
Producer commitments Strictly formatted packet fields copied into signed context; nested preimages are not recomputed by the adapter
Signed source context Source bucket, object name, generation, event ID, event_key, packet digest, packet claims, and adapter version
Receipt Independently verified by the adapter, then created in the evidence bucket under an event-key-derived name
Packet rejection Deterministic create-only evidence for permanent parser or byte-limit failure; binds source tuple and object metadata without packet contents
Receipt index Source tuple, receipt metadata, commitments, and verification result MERGEd into BigQuery by event_key
Appliance admin key Generated on the VM and published to a deployment-specific Secret Manager container after baked first boot
ML-DSA-65 secret key Generated at first boot and retained on the customer VM
Public verification material Raw public key and lowercase fingerprint published to separate deployment secrets
TLS trust material Deployment CA certificate published to Secret Manager; CA and leaf private keys remain on the VM
Logs Retained by customer-configured Compute Engine and Google Cloud logging controls

The adapter sends the packet-object digest and extracted signed context to the appliance. It does not send the full packet body. The original packet remains in Cloud Storage until customer retention or deletion policy removes it.

Packet validation boundary

The adapter accepts only hx.verifiable-ai.rag-packet/v1 with content.task exactly rag-answer. It rejects malformed UTF-8 JSON, duplicate object keys, non-finite numbers, missing required objects, malformed identifiers, invalid UTC timestamps, malformed commitments, repeated source IDs, and invalid retrieval.k relationships.

Required commitments are lowercase 64-hex SHA-256 values. created_utc must be a real RFC3339 UTC value. Packet, provider, model, and source identifiers are bounded ASCII values. Configurable limits bound object bytes, JSON depth, aggregate nodes, individual strings, and source count. The deployment defaults are 4 MiB, depth 24, 100,000 nodes, 1,000,000 characters per string, and 256 sources.

These checks constrain syntax and resource use. The adapter intentionally records nested_commitments_recomputed: false; it does not receive or recompute every prompt, output, content, or source-set preimage. Customers remain responsible for producer logic and truthful packet metadata.

Generation binding and application idempotency

For a finalized Cloud Storage object, event_key is SHA-256 over a domain-separated binary encoding of bucket UTF-8 bytes, object-name UTF-8 bytes, and the positive 64-bit generation. The adapter creates artifact_id = gcs-finalized:<event_key> and includes the complete source tuple and key in signed context.

Metadata and content reads both carry a generation-match precondition, and the returned generation and byte length are checked. This prevents a delayed event from silently reading a later generation at the same object name.

Receipt names use the event key:

<receipt-prefix><first-two-event-key-hex>/<event-key>.receipt.json

Receipt writes use a create-only generation precondition. A retry first generation-pins and independently re-verifies the existing canonical receipt bytes. A create race is accepted only when the existing bytes are identical. BigQuery uses a parameterized MERGE keyed by event_key; matching rows are unchanged. The Cloud Run adapter is serialized to one instance and one concurrent request, with a 600-second request deadline containing the configured bounded downstream operation budgets, so these controls provide application idempotency for this deployment.

Deterministic packet parser and packet-size-limit failures do not enter an unbounded retry loop. The adapter writes canonical, create-only rejection evidence under rejections/<first-two-event-key-hex>/<event-key>.rejection.json and acknowledges with 2xx only after persistence succeeds. The record binds the finalized source tuple and Cloud Storage size/checksum metadata without packet contents. It omits delivery IDs, timestamps, adapter versions, parser details, packet hashes, and configurable policy limits so repeated delivery and later revisions converge on identical bytes. An existing canonical rejection is authoritative for that finalized generation. Persistence failure, malformed existing evidence, or a differing-byte collision remains retryable and fail closed.

Eventarc delivery is still at least once. Operators must monitor delivery failures and reconcile artifact generations, evidence, and index rows.

Independent receipt verification

The adapter does not trust a successful appliance HTTP response by itself. Before new evidence storage or self_verified becomes true, it verifies:

  • supported outer and signed-body receipt schemas;
  • ML-DSA-65 and FIPS 204 declarations;
  • a GENERIC retrieval class for this adapter path;
  • the ML-DSA-65 signature over canonical signed-body bytes;
  • the exact submitted artifact ID and packet-object digest;
  • the exact submitted signed context, including source generation and event_key;
  • the embedded public key's digest against the receipt-declared fingerprint;
  • the embedded key against separately fetched pinned public-key bytes; and
  • the separately fetched key's digest against a separately fetched expected fingerprint.

New issuance must verify against the active public-key/fingerprint pair. During a coordinated rotation, one explicitly configured previous pair can be used only to re-verify an already stored create-only receipt. Verification or coordination failures stop processing and allow Eventarc retry.

Network and TLS controls

The deployment uses:

  • a dedicated custom-mode VPC and subnet;
  • no external IP on the appliance;
  • internal-only Cloud Run ingress;
  • Direct VPC egress from the adapter;
  • a private Cloud DNS zone attached to the deployment VPC;
  • one appliance A record pointing to the reserved private IP;
  • HTTPS on port 443 from the deployment subnet to the appliance;
  • IAP source range 35.235.240.0/20 for administrative SSH and HTTPS;
  • OS Login, blocked project-wide SSH keys, and disabled serial-port access; and
  • private Google API access from the subnet.

After baked first boot, the VM creates a deployment-local P-256 CA and an HTTPS leaf certificate. The VM hostname, adapter URL, private DNS record, leaf common name, and sole DNS SAN use the same private appliance hostname. Bootstrap verifies the leaf chain and hostname before nginx activation.

The adapter loads the CA certificate from Secret Manager and performs ordinary certificate-chain and hostname validation. There is no insecure bypass setting. A private network limits reachability; the matching SAN and customer-deployment CA authenticate the appliance endpoint.

Review organization firewall, VPC Service Controls, DNS, egress, and service-perimeter requirements before deployment. The supplied subnet-to-appliance rule can be narrowed further when the customer's networking model provides a more specific adapter source range.

Identity and IAM

Identity Required access
Appliance service account roles/secretmanager.secretVersionAdder and roles/secretmanager.secretAccessor on each of the four deployment bootstrap secrets
Adapter service account roles/secretmanager.secretAccessor on those four secrets; roles/storage.objectViewer on artifacts; roles/storage.objectCreator and roles/storage.objectViewer on evidence; dataset roles/bigquery.dataEditor; project roles/bigquery.jobUser
Optional packet-producer service account roles/storage.objectCreator on the artifact bucket only
Eventarc trigger service account roles/eventarc.eventReceiver and adapter roles/run.invoker
Cloud Storage service agent roles/pubsub.publisher for the storage event transport
Deployment operator Permission to create and remove solution resources and bindings
Appliance administrator IAP and OS Login access according to customer policy

The adapter is not granted Vertex AI model invocation permission. Model calls belong to the customer workload identity. The optional producer grant is creation-only; it is not required when the customer supplies a different authorized upload path.

Evidence-bucket creation and viewing are separate grants. The adapter cannot overwrite or delete evidence objects through these roles. BigQuery MERGE requires both dataset data editing and project job execution.

Review inherited permissions after deployment and avoid default service accounts or project-wide roles where narrower identities are available.

Key and secret bootstrap

Terraform creates four empty Secret Manager containers and grants only the access required to populate and consume them. No secret payload enters Terraform state.

After baked first boot, the VM:

  1. verifies the shipped appliance payload and completes first-boot setup;
  2. generates the appliance admin key and ML-DSA-65 keypair;
  3. stores the signing private key under the root-owned VM data path;
  4. creates the deployment-local CA and matching private-DNS leaf identity;
  5. installs the TLS leaf certificate and key for nginx;
  6. validates the public-key/fingerprint relationship and TLS hostname; and
  7. publishes the admin key, public key, public-key fingerprint, and CA certificate as Secret Manager versions.

Publication reads each versions/latest value before adding a version, does not add a duplicate when it already matches, and verifies the new latest value. Old versions are retained rather than mutated or destroyed. The ready marker is written only after all four values are coordinated.

The ML-DSA signing private key, CA private key, and TLS leaf private key never leave the VM. Never place private keys, API keys, or Secret Manager values in Terraform variables, source control, packet objects, BigQuery, support tickets, screenshots, or ordinary CI logs.

Rotation and historical verification

Active adapter references use versions/latest for all four secrets. The provider reads them sequentially, validates the loaded material, and replaces its in-process cache only after the complete load succeeds; Secret Manager does not return the four values as one atomic bundle. During rotation, mixed latest values can fail authentication, TLS, or pin checks and remain retryable until publication converges. Before replacing the appliance, record the active numeric public-key and fingerprint version IDs. Configure both previous_public_key_version and previous_public_key_sha256_version during the replacement update; setting only one is rejected.

The replacement VM advances active latest values by adding versions. One previous key/fingerprint pair remains available to re-verify existing receipts, while new issuance requires the active pair. Rotate admin and TLS material as one coordinated appliance and secret update. Until the set is coherent, authentication, TLS, or pin checks fail closed.

A new appliance normally has a new signing identity. Retain historical public keys and fingerprints for the entire historical receipt-verification period. Define and test backup, restore, revocation, and replacement procedures before production use; do not move a signing private key through an ordinary support channel.

Offline verification boundary

The signed release distributes release-matched verify_receipt.py and bounded verify_bundle.py tools. Use them from a hash-checked release package with both --public-key and --expected-fingerprint obtained through trusted channels. Do not copy verification code from the VM.

Explicit --integrity-only mode relies on the embedded key. It can show internal signature integrity but cannot establish who issued the evidence. Normal issuer verification deliberately requires external key and fingerprint inputs.

The bundle verifier does not extract archives. It validates the signed manifest, exact membership, hashes, sizes, signer equality, and contained receipts while enforcing configurable entry-count, per-entry, total-byte, and compression-ratio limits.

Cryptographic scope

Receipts use ML-DSA-65 as standardized by FIPS 204. SHA-256 commitments bind signed receipt fields to exact packet-object bytes or to producer-declared values.

Successful verification establishes that the signed body has not changed, the signature verifies under the pinned issuer key, the external fingerprint matches that key, and the checked artifact or context binding matches the receipt. It does not establish factual truth, safety, authorization, completeness, or legal sufficiency of the underlying AI output.

Storage, retention, and deletion

The artifact and evidence buckets use object versioning and public access prevention. Lifecycle eligibility and BigQuery partition expiration are configured through retention_days, but customers must ensure those settings satisfy their own retention and legal obligations. No bucket retention lock or WORM policy is configured. "Create-only" describes adapter write behavior; it does not prevent lifecycle deletion or deletion by an authorized customer administrator.

Customers must define artifact and receipt retention, object-version lifecycle, BigQuery export, encryption requirements, legal hold, deletion, public-key retention, and signer-continuity backup. Deleting a live object from a versioned bucket can leave prior generations; verify all generations when erasure is required.

Safe Terraform teardown defaults to allow_data_destroy=false, which makes protected resources and nonempty stores block destructive removal through this package. It does not disable configured lifecycle deletion or prevent an authorized customer administrator from deleting data. An intentional Terraform purge requires a separate explicit apply with allow_data_destroy=true before destroy. Export required evidence and public verification material first.

Logging and support data

The appliance records operational service events; Cloud Run records adapter requests and failures; firewall logging is enabled for deployment rules. Raw packet bodies are not required in ordinary diagnostics.

Before sending diagnostics to HolonomiX, remove Secret Manager values, API keys, authorization headers, private signing and TLS keys, raw prompts and model outputs, retrieved text, regulated records, sensitive receipts, and unrelated packet or bucket content. Use an agreed secure transfer channel when sensitive artifacts are necessary.

Shared responsibility

HolonomiX is responsible for supplied appliance and integration code, release artifacts, documented interfaces, and product security fixes.

The customer is responsible for:

  • project, organization, IAM, network, DNS, and service-perimeter policy;
  • Vertex AI model, prompt, retrieval, and safety configuration;
  • packet construction, nested commitment computation, unique identifiers, and producer truthfulness;
  • choosing validated input bounds suitable for the workload;
  • secret, signing-key, public-key-pin, CA, and coordinated rotation custody;
  • storage, retention, backup, deletion, and legal-hold policy;
  • log access, monitoring, updates, and vulnerability remediation;
  • retaining historical verification material; and
  • deciding whether receipts meet contractual, legal, regulatory, or audit requirements.

The deployment does not automatically capture every Vertex AI request and does not provide factual, safety, legal, or policy assurance.

Security reporting

Report suspected product vulnerabilities privately to [email protected]. Do not include live credentials, private keys, or customer data in the first report.

Public policy: https://holonomx.com/security

Return to the deployment and operations guide or see Support.