Operations, upgrades, and removal¶

Operate the appliance, adapter, and evidence stores as one generation-bound pipeline. Alert on missing outcomes and signer changes, not only service uptime.

Record deployment outputs¶

Keep the Marketplace deployment outputs in your approved configuration system. They contain names and URIs, not secret values.

export PROJECT_ID="your-project-id"
export REGION="us-central1"
export ZONE="us-central1-a"
export APPLIANCE="deployment-appliance-name"
export ARTIFACT_BUCKET="deployment-artifact-bucket"
export EVIDENCE_BUCKET="deployment-evidence-bucket"
export BQ_TABLE="project.dataset.receipts"
export ADAPTER_SERVICE="deployment-adapter-service"
export EVENTARC_TRIGGER="deployment-eventarc-trigger"
export PUBLIC_KEY_SECRET="deployment-public-key-secret"
export PUBLIC_KEY_SHA256_SECRET="deployment-public-key-sha256-secret"

Also retain the private appliance hostname, four secret-container names, image identity, product release, and deployment prefix.

Routine health checks¶

Appliance¶

gcloud compute instances describe "$APPLIANCE" \
  --project="$PROJECT_ID" \
  --zone="$ZONE" \
  --format='yaml(name,status,hostname,networkInterfaces,shieldedInstanceConfig,serviceAccounts)'

gcloud compute ssh "$APPLIANCE" \
  --project="$PROJECT_ID" \
  --zone="$ZONE" \
  --tunnel-through-iap \
  --command='sudo test -f /etc/hx-provenance/.first-boot.done &&
    sudo test -f /etc/hx-provenance/.vertex-ai-marketplace-bootstrap-v1.done &&
    sudo systemctl is-active hx-provenance.service nginx &&
    sudo /opt/hx-provenance/bin/verify_manifest.py \
      --manifest /opt/hx-provenance/MANIFEST.sha256 \
      --root /opt/hx-provenance'

The VM must have no external IP. Do not add one to simplify administration.

Adapter and Eventarc¶

gcloud run services describe "$ADAPTER_SERVICE" \
  --project="$PROJECT_ID" \
  --region="$REGION" \
  --format='yaml(name,uri,template.scaling,template.maxInstanceRequestConcurrency)'

gcloud run services logs read "$ADAPTER_SERVICE" \
  --project="$PROJECT_ID" \
  --region="$REGION" \
  --limit=100

gcloud eventarc triggers describe "$EVENTARC_TRIGGER" \
  --project="$PROJECT_ID" \
  --location="$REGION"

Do not export packet bodies, authorization headers, or secret values into log aggregation or tickets.

Evidence and index¶

Query recent accepted receipts:

bq --project_id="$PROJECT_ID" query --use_legacy_sql=false \
  "SELECT event_key, source_bucket, source_object_name, source_generation,
          receipt_uri, receipt_object_generation, self_verified,
          receipt_public_key_sha256, ingested_utc
   FROM \`${BQ_TABLE}\`
   WHERE ingested_utc >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 24 HOUR)
   ORDER BY ingested_utc DESC"

Every accepted row must report self_verified = true. BigQuery does not contain rejected packets; inspect the rejections/ prefix separately when reconciling source generations.

Reconcile generation outcomes¶

For each finalized packet generation, expect exactly one authoritative branch:

Branch	Evidence	BigQuery
Accepted	One create-only `receipts/...receipt.json`	One row keyed by `event_key`
Permanently rejected	One create-only `rejections/...rejection.json`	No row
Pending or transient failure	Neither authoritative object yet	No guaranteed row

Investigate:

artifact generations with neither receipt nor rejection evidence;
receipt evidence without an index row after Eventarc retry opportunities;
a BigQuery row whose source tuple differs from the finalized object;
differing-byte create conflicts;
repeated adapter 5xx responses;
unexpected signer fingerprints; and
TLS certificate expiry or coordinated-secret load failures.

Do not treat at-least-once Eventarc delivery itself as duplication. One source tuple can produce repeated invocations while converging on one evidence object and one index key.

Track signer identity¶

Record the active public key and fingerprint versions without printing their values:

gcloud secrets versions list "$PUBLIC_KEY_SECRET" \
  --project="$PROJECT_ID" \
  --format='table(name,state,createTime)'

gcloud secrets versions list "$PUBLIC_KEY_SHA256_SECRET" \
  --project="$PROJECT_ID" \
  --format='table(name,state,createTime)'

Alert on an unplanned active fingerprint change. Retain historical public keys and fingerprints for as long as their receipts must remain verifiable.

The ML-DSA-65 signing key, deployment CA private key, and TLS leaf private key stay on the appliance. Never export them into Secret Manager, Terraform state, support bundles, or ordinary backup systems.

Upgrade and replace the appliance¶

Marketplace images are immutable. Validate a new image identity before replacement.

Stop new packet production or establish a documented maintenance window.
Drain or reconcile in-flight Eventarc deliveries.
Export required receipts, public keys, fingerprints, and index records.
Record the current numeric versions of both the public-key and fingerprint secrets.
Set previous_public_key_version and previous_public_key_sha256_version together.
Keep allow_data_destroy=false.
Set allow_appliance_replacement=true and apply without changing the image.
Update only the independently verified image/release identity and apply the replacement.
Wait for the new bootstrap status command to print READY.
Verify service health, manifest integrity, active public-key/fingerprint agreement, private TLS, and a new test receipt.
Restore allow_appliance_replacement=false and apply immediately.
Resume packet production after acceptance.

The previous key pair is accepted only when re-verifying an existing immutable receipt during bounded rotation. New issuance must verify against the active pair.

Do not use allow_data_destroy=true for an ordinary appliance replacement.

Rotate coordinated material¶

The adapter reads versions/latest for the admin key, public key, fingerprint, and CA certificate. It replaces its in-process set only after a complete load passes consistency checks. Secret Manager does not publish the four values atomically.

For controlled rotation:

preserve current numeric public-key and fingerprint versions;
update appliance signing material and matching public verification material as one controlled event;
rotate admin and TLS material with the appliance update;
wait for all four latest versions to agree with the appliance;
verify private DNS, SAN, CA, authentication, signature, and fingerprint checks; and
retain historical public verification material.

Mixed latest values fail closed and can cause retryable adapter failures. Do not weaken TLS or pin checks to make a partial rotation succeed.

Retention and evidence export¶

The buckets are versioned and use age-based lifecycle eligibility. The BigQuery table uses partition expiration. These are deletion policies, not retention-lock or WORM guarantees.

Before changing retention_days:

confirm legal, contractual, and audit requirements;
inventory object versions and BigQuery partitions;
export required receipts, bundles, index rows, keys, and fingerprints;
record which policy version authorized deletion; and
verify historical receipts remain independently verifiable.

Authorized evidence export should include the source tuple, packet bytes or trusted digest, canonical receipt, receipt object generation and hash, issuer public key, expected fingerprint, verifier release, and verification result.

Remove while retaining data¶

The safe default is allow_data_destroy=false. Stop producers, reconcile in-flight events, and export required verification material. Terraform protection and nonempty stores are expected to block destructive teardown while retained data remains.

Document any resources intentionally retained outside Terraform management. A partial destroy without an ownership plan can leave costs or inaccessible evidence.

Intentionally purge the deployment¶

Purge is high impact and irreversible once destruction starts.

Stop producers and in-flight delivery.
Export all evidence and historical verification material required by policy.
Obtain the required retention, legal, and change approvals.
Confirm allow_appliance_replacement=false.
Set allow_data_destroy=true and apply that change.
Review the destructive plan and affected object versions, table data, VM disk, and keys.
Destroy only after explicit operator confirmation.
Preserve the change record and final evidence inventory.

Destroying the VM boot disk removes on-disk signing and TLS private keys. Purging versioned buckets and BigQuery data removes customer evidence. If purge is cancelled before destruction, restore allow_data_destroy=false and apply immediately.

Operational checklist¶

See Troubleshooting for failure modes and Support for escalation requirements.