Gf Reapi Cell Ac Nuke Key

gf-reapi-cell AC Nuke-Key Runbook

Use this runbook when a single gf-reapi-cell action-cache entry is suspected to be poisoned and the operator needs to remove that entry without clearing the tenant’s whole AC namespace.

This is an action-cache runbook only. It does not delete CAS blobs, restart the cell, mutate RustFS, touch Attic publication, change runner labels, or promote the proof-local cell into durable production RBE.

Inputs

Required facts:

  • GF_REAPI_STORE_ROOT for the cell instance being operated.
  • REAPI instance_name: default, system, or spoke-<slug>.
  • action_digest as <sha256>/<size_bytes>.
  • Matching row in ${GF_REAPI_STORE_ROOT}/audit/ac-writes.jsonl.
  • Operator reason and operator identity.

The audit row is required by default because it identifies the writer subject, token ID, worker image digest, platform digest, and action digest. Use --allow-missing-audit-row only for an explicitly documented emergency override.

Dry-Run

just gf-reapi-ac-nuke-key nuke \
  --store-root "$GF_REAPI_STORE_ROOT" \
  --instance-name spoke-alpha \
  --action-digest "$ACTION_DIGEST" \
  --operator "$USER" \
  --reason "poisoned AC entry from incident TIN-XXXX"

The default is dry-run. It prints the exact AC entry path, quarantine tombstone path, matching audit-row count, event-log path, and backup path that would be used.

Execute

just gf-reapi-ac-nuke-key nuke \
  --store-root "$GF_REAPI_STORE_ROOT" \
  --instance-name spoke-alpha \
  --action-digest "$ACTION_DIGEST" \
  --operator "$USER" \
  --reason "poisoned AC entry from incident TIN-XXXX" \
  --execute

Execution does three bounded mutations:

  • copies the AC entry into ${GF_REAPI_STORE_ROOT}/audit/ac-nuke-key-backups/
  • removes exactly ${GF_REAPI_STORE_ROOT}/instances/<instance_name>/ac/<hash>-<size>.pb
  • writes a server-enforced quarantine tombstone at ${GF_REAPI_STORE_ROOT}/instances/<instance_name>/ac-quarantine/<hash>-<size>.json

The default quarantine window is 3600 seconds. While the tombstone is valid, direct UpdateActionResult returns FailedPrecondition, and Execute returns the execution result but refuses to populate the quarantined AC key. Expired tombstones are removed by the next write attempt.

The tool appends structured events to ${GF_REAPI_STORE_ROOT}/audit/ac-nuke-key-events.jsonl.

Rollback

Use the backup_path printed by the execute command.

just gf-reapi-ac-nuke-key rollback \
  --store-root "$GF_REAPI_STORE_ROOT" \
  --instance-name spoke-alpha \
  --action-digest "$ACTION_DIGEST" \
  --backup-path "$BACKUP_PATH" \
  --operator "$USER" \
  --execute

Rollback restores the backed-up .pb entry and removes the quarantine tombstone unless --keep-quarantine is passed. It refuses to overwrite an existing AC entry unless --overwrite is explicit.

Synthetic Canary Drill Record

2026-05-19 local contract drill:

  • created a synthetic AC entry under instances/spoke-alpha/ac/
  • created a matching synthetic accepted row in audit/ac-writes.jsonl
  • ran dry-run and verified no mutation
  • ran nuke --execute and verified exact entry removal, backup creation, quarantine tombstone creation, and completed event-log row
  • ran rollback --execute and verified AC entry restoration plus tombstone removal
  • verified missing-audit default rejection leaves the AC entry intact
  • verified symlink AC entries are refused

Contract command:

just gf-reapi-ac-nuke-key-contract-check

This drill closes the operator mechanics for W2.4/TIN-1465. It does not close W2.5 chaos testing, durable CAS/AC authority, retention/query/dashboard surfaces, tenant quota/fairness, or broad/default RBE safety.

GloriousFlywheel