Troubleshooting¶

Start with the finalized source tuple and UTC time window. Preserve existing packet, receipt, rejection, and index evidence while you isolate the failing component.

Safe triage context¶

Set names and identifiers only. Do not place secret values in environment variables that will be copied into a ticket.

export PROJECT_ID="your-project-id"
export REGION="us-central1"
export ZONE="us-central1-a"
export APPLIANCE="deployment-appliance-name"
export ADAPTER_SERVICE="deployment-adapter-service"
export EVENTARC_TRIGGER="deployment-eventarc-trigger"
export BQ_TABLE="project.dataset.receipts"

Capture:

source bucket, object name, and generation;
event_key, when available;
UTC start and end time;
deployment prefix and product version;
expected and observed outcome; and
sanitized resource status and logs.

Never send API keys, Secret Manager values, authorization headers, private keys, regulated data, packet bodies, or unsanitized receipts through ordinary email.

Triage map¶

Symptom	First check	Next section
Bootstrap never reaches `READY`	VM first-boot and bootstrap journals	Bootstrap
No receipt or rejection	Object prefix, generation, Eventarc target	No outcome
Rejection evidence appears	Packet schema and configured bounds	Rejected packet
Adapter returns 5xx	Private TLS, secrets, evidence write, BigQuery IAM	Adapter 5xx
Receipt exists without an index row	BigQuery IAM and retry	Missing row
Appliance connection fails	DNS, SAN, CA, VPC egress, firewall	Private HTTPS
Offline verification fails	Original bytes, release tool, issuer pins	Verification
IAP fails	IAM, firewall source range, OS Login	IAP

Bootstrap never reaches `READY`¶

Check the instance and startup state:

gcloud compute instances describe "$APPLIANCE" \
  --project="$PROJECT_ID" --zone="$ZONE" \
  --format='yaml(name,status,networkInterfaces,serviceAccounts,shieldedInstanceConfig)'

gcloud compute ssh "$APPLIANCE" \
  --project="$PROJECT_ID" --zone="$ZONE" --tunnel-through-iap \
  --command='sudo systemctl status hx-provenance-first-boot.service \
      hx-provenance.service nginx --no-pager;
    sudo journalctl -u hx-provenance-first-boot.service --since "60 minutes ago" --no-pager;
    sudo journalctl -t hx-vertex-ai-bootstrap --since "60 minutes ago" --no-pager;
    sudo test -f /etc/hx-provenance/.first-boot.done && echo FIRST_BOOT_DONE;
    sudo test -f /etc/hx-provenance/.vertex-ai-marketplace-bootstrap-v1.done && echo READY'

Confirm:

the immutable image contains the required first-boot service and files;
the appliance service starts before the integration bootstrap;
the appliance service account can add and access versions in all four secret containers;
the VM can reach Google APIs through Private Google Access;
private DNS and the configured hostname are valid; and
generated public-key/fingerprint and TLS material pass local consistency checks.

Terraform creates empty secret containers. The VM publishes their first values only after first boot. Do not inject secret values through Terraform state.

No receipt or rejection appears¶

Describe the finalized object:

gcloud storage objects describe "gs://BUCKET/OBJECT.json" \
  --format='yaml(name,bucket,generation,size,crc32c,md5Hash,timeCreated,updated)'

Confirm:

the object name starts with the configured prefix, normally packets/;
the name ends in .json;
the generation is positive and still readable;
the Eventarc event is an object-finalize event;
the trigger targets the expected internal adapter;
the trigger identity can invoke the adapter; and
the adapter can read that exact generation.

gcloud eventarc triggers describe "$EVENTARC_TRIGGER" \
  --project="$PROJECT_ID" --location="$REGION"

gcloud run services logs read "$ADAPTER_SERVICE" \
  --project="$PROJECT_ID" --region="$REGION" --limit=100

Search by source tuple and time. Do not upload a differently named copy until you know whether the original generation remains pending; a new name creates a separate provenance event.

Packet is rejected¶

Permanent JSON, schema, and configured packet-limit failures produce:

rejections/<first-two-event-key-hex>/<event-key>.rejection.json

Check that:

schema is exact hx.verifiable-ai.rag-packet/v1;
content.task is exact rag-answer;
created_utc is a real RFC3339 UTC timestamp;
JSON contains no duplicate keys or non-finite numbers;
all four commitments are lowercase 64-hex SHA-256 strings;
packet, provider, model, and source IDs are bounded ASCII identifiers;
retrieval.k is a bounded non-negative integer;
source count does not exceed k; and
source IDs are unique.

The rejection is authoritative for that finalized generation. Correct the producer and upload a new generation. Do not delete or overwrite rejection evidence before retention and incident review.

Adapter returns 5xx¶

A 5xx indicates a failure that might succeed on retry. Common causes are:

source generation disappearance or precondition failure;
missing or incoherent Secret Manager material;
private DNS, firewall, CA-chain, or certificate-hostname failure;
rejected appliance credentials or unhealthy appliance service;
receipt signature, exact-context, key, or fingerprint mismatch;
a create-only evidence collision with different bytes;
inability to create or re-read rejection evidence; or
Cloud Storage or BigQuery permission failure.

Use a narrow log window and source tuple. Correct the cause; Eventarc can redeliver the same event. Do not weaken TLS, modify stored evidence, or rename the packet as a retry mechanism.

Receipt exists but no BigQuery row appears¶

Receipt creation and index MERGE are separate. A retry re-reads and verifies the existing receipt, then retries the index operation.

Confirm the adapter has dataset roles/bigquery.dataEditor and project roles/bigquery.jobUser, then query by event key:

export EVENT_KEY="64-lowercase-hex-event-key"
bq --project_id="$PROJECT_ID" query --use_legacy_sql=false \
  --parameter="event_key::${EVENT_KEY}" \
  "SELECT event_key, source_bucket, source_object_name, source_generation,
          receipt_uri, receipt_object_generation, self_verified, ingested_utc
   FROM \`${BQ_TABLE}\`
   WHERE event_key = @event_key"

Preserve the receipt. Do not request another receipt or delete evidence to force a new row.

Receipt path is unknown¶

Query BigQuery by source bucket, object name, and generation. Do not infer a path from packet_id.

With the default prefix and known event key:

receipts/<first-two-event-key-hex>/<event-key>.receipt.json

The Packet integration reference documents exact event-key derivation.

Repeated delivery is observed¶

Eventarc is at least once. Repeated adapter invocations are expected to converge on one receipt or rejection object and one BigQuery key.

Investigate only when you see:

different bytes competing for one create-only evidence name;
a source tuple mismatch;
repeated 5xx after the original transient issue is resolved; or
a receipt that repeatedly fails re-verification or index MERGE.

Preserve conflicting evidence and logs for incident review.

Adapter cannot connect to the appliance¶

Confirm:

private Cloud DNS is attached to the deployment VPC;
the appliance A record points to its reserved private IP;
the adapter URL uses the private hostname, not the IP address;
the certificate SAN matches that hostname;
the active CA-certificate secret matches the appliance CA;
Direct VPC egress uses the intended subnet;
the subnet-to-appliance port 443 firewall rule exists; and
hx-provenance.service and nginx are active.

There is no supported insecure TLS mode. Do not disable hostname or CA verification or add an external VM IP.

Verification fails¶

Use the signed release's verifier and issuer material obtained independently:

python verify_receipt.py \
  --receipt receipt.json \
  --public-key issuer-ml-dsa-65.pub \
  --expected-fingerprint "$EXPECTED_FINGERPRINT" \
  --artifact packet.json

Common causes:

packet or receipt bytes changed;
unsupported receipt schema;
artifact digest mismatch;
malformed canonical signed data;
signature failure;
wrong issuer public key; or
wrong expected fingerprint.

--integrity-only is deliberately weaker and cannot establish issuer identity. Do not obtain the verifier from the appliance VM.

For the appliance endpoint, HTTP 200 is not the verdict. Automation must inspect the response body and require both the verification and key-pin result.

Rotation causes signature, authentication, or TLS failure¶

All four active secret references use versions/latest. The adapter validates a completed load before replacing its cached set, but Secret Manager does not return one atomic four-value bundle.

Confirm the active public key and fingerprint match each other and the appliance signer. Confirm the CA certificate matches the appliance leaf and private hostname. During appliance replacement, both previous public-key version inputs must refer to the corresponding historical pair.

Do not delete historical public-key versions or weaken validation. Complete or roll back the coordinated rotation.

IAP access fails¶

Confirm:

the administrator has IAP tunnel and OS Login permissions;
the IAP firewall rule permits 35.235.240.0/20 to ports 22 and 443;
project-wide SSH keys remain blocked;
the VM is running; and
organization policy permits IAP and OS Login.

Do not add an external IP as a workaround.

Escalate safely¶

If the issue remains, follow Support. Include names, identifiers, timestamps, expected and observed behavior, minimal reproduction steps, and sanitized logs. For an exposed key, forged receipt, unexpected signer change, or suspected vulnerability, stop affected producers, preserve evidence, restrict access, and contact [email protected] privately.

Troubleshooting¶

Safe triage context¶

Triage map¶

Bootstrap never reaches READY¶

No receipt or rejection appears¶

Packet is rejected¶

Adapter returns 5xx¶

Receipt exists but no BigQuery row appears¶

Receipt path is unknown¶

Repeated delivery is observed¶

Adapter cannot connect to the appliance¶

Verification fails¶

Rotation causes signature, authentication, or TLS failure¶

IAP access fails¶

Escalate safely¶

Bootstrap never reaches `READY`¶