Justfile Commands

The project Justfile provides recipes for common development, build, and deployment tasks. Run just --list to see all available recipes or just <recipe> --help for usage details.

The intended developer entrypoint is direnv allow or nix develop, then just info to confirm substrate state. For Bazel dogfood, use just bazel-build-cached only after just cache-contract-strict passes. Raw local bazel build, raw local bazel test, bazelisk, and --config=ci remain compatibility or debug paths only. Current main proves shared cache acceleration here, not full remote offload for every developer workload.

Proxy

Recipes for the compatibility SOCKS5 proxy path. The preferred operator model is direct or tailnet-private access to the honey cluster via ~/.kube/kubeconfig-honey.yaml and context honey.

Development

General development workflow recipes.

just bazel-external-input-authority-package-gate --package <path> validates the non-secret W3/TIN-1468 package for a future durable Bazel external-input mirror. The package must name a dedicated endpoint and bucket, scoped GF_EXTERNAL_INPUT_MIRROR_* credential environment variables, a dedicated mirror_prefix, mirror layout, restore proof, read-only consumer exposure, retention/maintenance behavior, quota policy, observability, and authority separation. It rejects Civo, RustFS, current Attic/cache/state buckets, inline secrets, protected prefixes, and template sentinels. The package is readiness evidence only; the durable authority contract remains no-live-durable-authority until live restore proof updates covered_inputs.

just bazel-external-input-authority-package-template [output] writes the expected package shape with posture=candidate_only, replace-me, and example.invalid sentinels. It must fail the package gate until an operator fills in real non-secret backend details, then the live-readiness guard must still reject it until the posture is deliberately promoted to proof_ready. Do not commit credentials into the package.

just nix-flake-source-input-manifest inventories the locked GitHub archives in flake.lock, including each owner/repo, rev, narHash, original ref, and archive URL. just nix-flake-source-input-authority validates docs/contracts/nix-flake-source-input-authority.json: current first-party CI is classified as authenticated-live-GitHub-only, not durable authority. The gate requires setup-flywheel and nix-job to keep Nix github.com access-token plumbing, requires every GitHub flake input to remain rev+narHash pinned, and fails if the contract claims durable coverage before mirror, restore, retention, provenance, auth, and consumer exposure evidence exists.

just nix-flake-source-mirror-package --source-manifest <manifest.json> --source-root <archives> --mirror-root <mirror> packages already-materialized flake source archives into a provider-neutral local mirror root. The package records archive_sha256 for tarball bytes separately from the locked Nix narHash; it is a local byte/provenance primitive, not live durable source authority. just nix-flake-source-mirror-verify checks the package bytes and sidecars, and just nix-flake-source-mirror-restore reconstructs a local source archive root plus nix-flake-source-archive-manifest.json for the future live restore proof.

just nix-flake-source-authority-package-template writes the non-secret package shape for the future dedicated Nix flake source archive authority. The template uses posture=candidate_only, replace-me, and example.invalid sentinels, so just nix-flake-source-authority-package-gate must reject it until an operator fills in real backend details. The package uses scoped credential env names such as GF_NIX_FLAKE_SOURCE_MIRROR_ACCESS_KEY_ID and must not reuse Attic, OpenTofu state, Bazel distdir, Bazel remote cache, or RBE CAS/action-cache surfaces. Passing the package gate is still not live durable authority; the status stays authenticated-live-GitHub-only until mirror placement, restore, retention, provenance, auth-boundary, and consumer-exposure evidence exists.

just bazel-distdir-full-package-proof runs the TIN-1468 full-candidate package mechanics proof: materialize every entry in docs/contracts/bazel-external-input-mirror-candidates.json, package the verified bytes into the provider-neutral mirror layout, verify --all-candidates, restore --all-candidates, and emit bazel-distdir-full-package-proof-evidence.json. The scheduled workflow runs on tinyland-nix-heavy and mirrors evidence to the Actions step summary. Artifact upload is opt-in through GF_UPLOAD_BAZEL_DISTDIR_FULL_PROOF_ARTIFACTS=true so GitHub artifact quota does not mask the proof result. This remains package/restore mechanics only: it is not live durable mirror storage, CAS/action-cache authority, or broad/default RBE.

just bazel-distdir-full-package-proof-contract-check is the bounded local guard used by just check. It verifies the script, workflow, and docs wiring without downloading the full candidate archive set.

just bazel-distdir-mirror-live-proof --package <package.json> --mirror-root <mirror-root> --all-candidates is the live TIN-1468 proof harness. It first revalidates the non-secret authority package and local mirror package, then uses the package’s scoped GF_EXTERNAL_INPUT_MIRROR_ENDPOINT, GF_EXTERNAL_INPUT_MIRROR_REGION, GF_EXTERNAL_INPUT_MIRROR_ACCESS_KEY_ID, and GF_EXTERNAL_INPUT_MIRROR_SECRET_ACCESS_KEY environment variables to upload the package to the selected bucket/prefix, download it back into a fresh mirror root, verify the bytes, restore a local BAZEL_DISTDIR, and write bazel-distdir-mirror-live-proof-evidence.json. The matching manual workflow, Bazel Distdir Mirror Live Proof, dogfoods tinyland-nix-heavy, does not use GitHub-hosted runners, and mirrors evidence into the Actions step summary instead of depending on artifact quota. Passing this proof is candidate live authority evidence; contract promotion remains a separate reviewed change.

just bazel-distdir-mirror-live-readiness is the fail-closed preflight for that live proof. It validates the non-secret authority package, requires posture=proof_ready, checks that the scoped GF_EXTERNAL_INPUT_MIRROR_* environment names are present, verifies endpoint/region values match the package, and writes redacted bazel-distdir-mirror-live-readiness-evidence.json. Missing package or missing credentials is a readiness blocker, not runner capacity or Bazel proof failure.

just bazel-distdir-mirror-github-readiness checks the GitHub-side injection surface before dispatching the live proof. It lists visible GitHub variable and secret names only, never values. Endpoint and region may be GitHub variables or secrets, but the access-key and secret-key names must be GitHub secrets. RustFS, Attic, Civo, OpenTofu state, broad AWS, Bazel remote cache, and future RBE CAS/action-cache credentials are reported as substitute names and do not count as TIN-1468 readiness.

just e3-external-input-authority-status renders the read-only E3/TIN-1447 close-gate status from GitHub workflow metadata, the non-secret authority package gate, and the same GitHub variable/secret-name readiness surface. It reports the W3.4 vendor-mode nightly streak plus the latest on-demand green vendor proof, the Bazel Distdir Full Package Proof nightly streak, whether a proof_ready authority package exists, scoped mirror injection readiness, and whether a reviewed Bazel Distdir Mirror Live Proof has passed on main. Because operator review is not inferable from Actions metadata, pass reviewed live proof IDs explicitly, for example --reviewed-live-proof-run-id 123456789. The command is intentionally non-mutating: it reads run metadata and visible GitHub name lists only, never secret values, and performs no S3, Bazel, Kubernetes, Attic, OpenTofu, or RBE mutation.

just bazel-vendor-mode-canary runs the W3.4/TIN-1470 vendor-mode canary: stage verified distdir inputs for the Linux x64 Node toolchain and the hermetic_launcher prebuilt stubs, invoke bazel vendor with a bounded scratch-disk and timeout guard, and, on success, consume the generated vendor directory for //:deployment_bundle. The default full scope is the lockfile-authoritative E3 close signal; GF_VENDOR_MODE_SCOPE=production passes --ignore_dev_dependency for operator investigation without rewriting MODULE.bazel.lock. GF_VENDOR_MODE_DISTDIR_INPUTS can override the staged input set for focused investigations; the legacy single-input GF_VENDOR_MODE_DISTDIR_INPUT remains accepted. The scheduled canary runs on tinyland-nix-heavy and requires a 192Gi scratch preflight because current full-scope proofs materialize roughly 170Gi before cleanup. It also passes GF_VENDOR_MODE_BAZEL_HOST_JVM_XMX to Bazel as an explicit host JVM heap envelope. The workflow writes evidence JSON to the Actions step summary and job log, and attempts artifact upload on a best-effort basis because GitHub artifact quota must not hide the canary result. A green canary is mirror completeness evidence only; it is not durable mirror storage, CAS/action-cache authority, or broad/default RBE.

just bazel-vendor-mode-canary-contract-check is the bounded local guard used by just check. It verifies the script, workflow, and docs wiring without vendoring the external repository graph.

First-Party CI Dogfood

just dogfood-contract-audit enforces the rule that GloriousFlywheel’s own merge-blocking validation, security scans, Bzlmod/Bazel canaries, RBE proof surfaces, and runner-status workflows must dogfood shared tinyland-* capability-class runners. The source repo has a zero-exception hosted-runner rule for first-party CI, publication, status, and release workflows. The audit fails direct hosted runs-on scalars, multi-line runs-on lists, matrix/env hosted-runner literals, and any attempt to add a configured hosted-runner exception. Queue pressure, billing, artifact quota, or runner availability issues must be fixed in the shared runner substrate or handled as best-effort evidence, not hidden behind a GitHub-hosted fallback.

Nix

Recipes for Nix-based builds and maintenance.

OpenTofu

Infrastructure-as-code recipes for planning and applying changes. Each recipe takes a stack name argument (e.g., attic, gitlab-runners, runner-dashboard).

Note: tofu-plan expects a {ENV}.tfvars file (default dev.tfvars) in the stack directory. ENV=dev and ENV=prod are logical deployment environments and can both map to the same physical cluster, such as honey. For arc-runners, the local operator path also auto-includes {ENV}-policy.tfvars before local overrides and {ENV}-extra-runner-sets.tfvars when present, so the committed ARC baseline and additive lanes like tinyland-nix-heavy join the same local plan/apply path. For overlay deployments, create your tfvars there or use the per-stack Justfiles directly.

Current backend note:

• just tofu-init <stack> follows the active backend family declared in that stack’s backend.tf
• all four active stacks now use backend "s3" on the environment-owned honey state path
• TOFU_BACKEND_CONFIG_FILE, TOFU_BACKEND_CONFIG_DIR, and TOFU_BACKEND_S3_* are now the primary local init paths across those active stacks
• TF_HTTP_* remains compatibility-only for legacy or archived HTTP-backend repair paths
• tofu-apply runs the saved-plan guard by default and fails before apply if the plan deletes/replaces protected runner namespaces, namespace policy, cache PVCs/buckets, or introduces repo-scoped ARC registration URLs or repo-shaped runner label drift; set GF_TOFU_PLAN_GUARD_ALLOW=1 only for a reviewed maintenance exception
• for arc-runners, tofu-apply also runs scripts/arc-local-apply-source-guard.sh, which refuses dirty or non-origin/main local source; use the managed Deploy ARC Runners workflow for ordinary capacity changes and set GF_ARC_RUNNERS_LOCAL_APPLY_ALLOW=1 only for a reviewed break-glass local apply
• when a stack flips to backend "s3", the same root entrypoint can use a matching backend HCL file or the live TOFU_BACKEND_S3_* environment path
• if config/backends/<stack>-<env>.hcl exists, just tofu-init <stack> now treats it as the implicit local default without requiring TOFU_BACKEND_CONFIG_DIR
• the live direction is environment-owned S3-compatible state on honey

just tofu-preflight <stack> is the shortest local check before init. It validates:

• config/organization.yaml
• the resolved ENV cluster context
• the resolved kubeconfig path and local context presence
• {ENV}.tfvars for the chosen stack
• for arc-runners, the baseline {ENV}-policy.tfvars file when present
• for arc-runners, the optional {ENV}-extra-runner-sets.tfvars file when present
• the currently configured backend-init path
• when the active family is s3, the TOFU_BACKEND_S3_* environment path and state-key resolution

just tofu-backend-audit runs the same preflight logic across all four active stacks and prints the current backend mode, backend ref, and first blocking reason for each one. This is the fastest way to confirm whether Slice 1 is blocked on local setup drift or on one shared backend-authority decision.

just tofu-state-authority-proof is the live runtime check for the proven RustFS-backed S3 candidate on honey. It reads the cluster-managed credentials secret, opens a bounded port-forward to attic-rustfs-openebs, runs signed aws s3api reads, and prints the current bucket and object-key view. Use this when the question is “does the environment-owned S3 state authority actually exist and answer now?” rather than “can the current local stack init path run?”

just tofu-state-authority-deep-check [stack] is the stronger protected-apply guard. It uses the cluster-managed RustFS secret and a bounded port-forward, but signs S3 requests without requiring AWS CLI. It checks the RustFS Deployment and ready pod, inspects /data/<bucket> and /data/.rustfs.sys/buckets/<bucket> when kubectl exec is allowed, verifies the S3 API bucket view, optionally requires the stack’s canonical state object, GETs/parses that state object as JSON with --read-state-objects, and writes/reads/deletes a temporary sentinel object. Use this before protected OpenTofu mutation when RustFS bucket-index reliability is in question.

just rustfs-bucket-index-rca [args...] captures the incident/RCA evidence around RustFS bucket-index reliability without restarting RustFS. It prints the workload shape, selected pod, RustFS process/version evidence, PVC/job state, bootstrap and lifecycle logs, recent RustFS logs, namespace events, and the state authority deep check. Pass --scratch-probe to create and delete an isolated scratch bucket that proves normal API/disk coherence without touching the real OpenTofu state objects. The default tofu-state bucket probe checks the known GloriousFlywheel state keys. Non-default buckets are bucket-only checks unless you pass explicit keys. Pass --strict-scratch-disk-markers with --scratch-probe when the probe should fail if scratch bucket disk markers are missing after create or still present after API delete. Lifecycle Job log reads are bounded by default with --job-log-timeout 3s and --job-log-tail 160. The RCA also captures transient rustfs-bucket-ensure-* Job and Pod names, logs, labels, annotations, owner references, and event-only pod names because those objects can appear in the event window immediately before a bucket-index recurrence, then be TTL-cleaned before the incident artifact is collected. Large bucket directories are sampled rather than fully enumerated, and expensive du summaries are skipped on bucket-sized paths, so an attic bucket with many chunk directories cannot consume the canary window before bucket-index and NAR evidence is captured. Example:

just rustfs-bucket-index-rca --bucket attic \
  --state-key opentofu/states/mail-honey.tfstate

The same probe runs through the self-hosted RustFS State Authority Canary workflow on main, on demand, and hourly while RustFS remains the interim state authority. That workflow runs on the shared tinyland-nix-operator dogfood lane because it uses kubeconfig-backed operator probes and bounded port-forwards; it is not allowed to fall back to hosted runners or the generic Nix overflow lane. It publishes the interim HA-readiness output, read-only attic bucket-index RCA output, Attic NAR integrity output, strict scratch bucket-index RCA output, and read-only HA state candidate inventory as artifacts. The Attic evidence steps run with if: always() so tofu-state failure does not prevent capture of cache-read evidence, and stderr is folded into the uploaded artifacts so exact FAIL: lines are preserved outside the Actions log. The scratch RCA path uses --strict-scratch-disk-markers, so a green canary means the scratch bucket was visible through both the S3 API and RustFS disk markers during create, and the disk markers were gone after API delete. The state RCA path also reads and JSON-validates the known OpenTofu state object bodies without logging contents. It is current coherence evidence for known RustFS debt, not a strict HA gate and not Bazel RBE or BCR authority. A NO_LIVE_HA_STATE_CANDIDATE inventory verdict should be treated as captured TIN-1012 evidence, not as a canary failure.

just rustfs-repair-surface-inventory [args...] captures the live RustFS repair/admin surface without starting a repair, creating a scratch bucket, restarting the pod, or running Attic publication. It records the deployment shape, selected pod, image, process command line, top-level rustfs CLI help, safe non-config rustfs info output, installed admin/client binary candidates, and recent repair-relevant logs. Use it for TIN-1147 evidence before deciding whether the next path is a signed RustFS heal probe, a RustFS upgrade/topology change, or backend replacement. A green inventory does not restore trusted Attic writes and does not promote RustFS to HA state or RBE CAS/action-cache authority.

just rustfs-admin-status-probe [args...] sends one status-only request to the signed admin endpoint /rustfs/admin/v3/background-heal/status through a temporary port-forward. It uses the deployed RustFS credentials but does not call /heal/, create scratch buckets, restart RustFS, or run Attic publication. A green probe proves admin-auth observability for TIN-1147; it is not a non-restart repair proof and does not restore trusted Attic writes.

just rustfs-heal-semantics-audit [args...] audits the tagged RustFS source tree, defaulting to /private/tmp/rustfs-v1.0.0-beta.1-source. It does not call live /heal/ endpoints. It also does not port-forward, restart RustFS, create buckets, or run Attic publication. It verifies the status endpoint remains observability-only and records the current source finding that /rustfs/admin/v3/heal/{bucket} parses HealOpts but builds the channel request with create_heal_request, whose dry_run field is None; the heal processor then defaults missing dry_run to false. Treat that endpoint as a mutating repair path unless a future source audit proves otherwise.

just rustfs-bucket-meta-semantics-audit [args...] audits the tagged RustFS source tree, defaulting to /private/tmp/rustfs-v1.0.0-beta.1-source. It does not call live bucket-metadata admin endpoints, port-forward, restart RustFS, create buckets, or run Attic publication. It verifies that /rustfs/admin/v3/export-bucket-metadata is tied to the current bucket API view through list_bucket/get_bucket_info, not an independent disk-marker reindex path. It also records that /rustfs/admin/v3/import-bucket-metadata is a mutating archive import path that can call make_bucket with force_create; the current handler does not persist the accumulated imported metadata config updates. Treat export/import-bucket-metadata as unproved for TIN-1147 until a future controlled live proof shows otherwise.

just rustfs-trusted-publication-gate-check [args...] validates rustfs-trusted-publication-backend-gate.json, the static TIN-1147 stop/go gate for restoring trusted Attic publication. The gate accepts only three proof paths: non-restart RustFS repair/reindex, RustFS upgrade/topology fix, or backend replacement. It explicitly rejects restart-only recovery, green canary-only coherence, source-only admin-route existence, dry-run assumptions, ARC dispatch evidence, RBE proof evidence, and OpenTofu state-only HA proof as substitutes for trusted Attic publication backend evidence.

just attic-backend-replacement-package-gate --package <path> validates the non-secret package for the backend_replacement path in the TIN-1147 trusted Attic publication gate. The package must keep the trusted publication gate closed, name only replacement-backend endpoint and credential environment variables, reject inline secret material, reject the current attic-rustfs-openebs and cache service endpoints, and document scratch object proof, representative small-check/medium-check publication proof, read compatibility for existing Attic objects, retention, restore, maintenance, observability, rollback, and quarantine. It is a static package gate only; it does not run live kubectl, OpenTofu, RustFS repair, or Attic publication commands.

just attic-backend-replacement-package-template [output] writes the expected Attic backend replacement package shape. The generated template contains replace-me and example.invalid sentinels and must fail just attic-backend-replacement-package-gate --package <output> until an operator fills in real non-secret backend details. Do not commit credentials into the package; it may name ATTIC_BACKEND_REPLACEMENT_* environment variables but not their values.

just rustfs-upgrade-topology-candidate-check [args...] validates rustfs-upgrade-topology-candidate.json, the non-secret TIN-1152 packet for evaluating RustFS 1.0.0-beta.4 through the TIN-1147 upgrade/topology path. The packet records upstream release and compare links, preserves the current beta.1 rollback digest, records the selected Docker Hub beta.4 manifest/platform digests, and keeps trusted Attic publication disabled until operator-approved post-upgrade tofu-state, attic bucket-index, NAR integrity, and representative small-check/medium-check publication evidence exist.

just rustfs-upgrade-topology-proof-plan-check [args...] validates rustfs-upgrade-topology-proof-plan.json, the source-owned, non-mutating proof plan for the upgrade-topology path. The plan narrows the eventual live change to the rustfs_image field in tofu/stacks/attic/honey.tfvars, preserves the beta.1 rollback digest, rejects Civo as an endpoint or fallback, requires just tofu-plan-guard attic, and keeps trusted Attic publication disabled until post-upgrade tofu-state, bucket-index RCA, NAR integrity, and representative small-check/medium-check publication evidence clear the current NoSuchBucket, curl 18, and size_download=0 failure classes.

just rustfs-upgrade-topology-plan-guard [plan-json] validates the tofu show -json output for the future upgrade-topology maintenance-window plan. It is stricter than the generic just tofu-plan-guard attic: it accepts only digest-pinned beta.1 -> upgrade-topology candidate RustFS container image updates for the live Deployment and, if present, the drained legacy StatefulSet template. It rejects Secret data changes, selector changes, PVC/storage/service drift, delete/create actions, wrong image direction, and plans that do not include the live module.rustfs[0].kubernetes_deployment_v1.rustfs[0] image update. The managed Deploy Attic Stack workflow uses the same guard when manually dispatched with plan_scope=rustfs_upgrade_topology. That proof scope may only continue past expected-red RustFS state authority during action=plan; all applies still require strict state authority and post-apply candidate-image verification.

just tofu-state-ha-readiness [args...] checks the current S3 state authority against the HA implementation gate. It is expected-red for the current RustFS singleton unless run with --expect-interim, because the current path is one RustFS Deployment replica on a bumble-bound OpenEBS ZFS ReadWriteOnce PVC. TIN-1002 captured the candidate plan; TIN-1012 owns making this strict gate green. Use it as evidence that the state path is guarded but not yet an HA authority.

just tofu-state-ha-readiness --expect-interim

just ha-state-candidate-proof [args...] is the TIN-1013 proof harness for a candidate that is meant to replace the current guarded RustFS singleton. It requires --endpoint-package <path> and validates that package through just ha-state-endpoint-package-gate before any S3 or OpenTofu action runs. The package supplies the approved endpoint, region, and non-state scratch bucket. Credentials still come through TOFU_HA_STATE_ACCESS_KEY and TOFU_HA_STATE_SECRET_KEY or matching command line flags. The harness refuses direct endpoint, region, or scratch bucket values that do not match the package. It intentionally does not inherit TOFU_BACKEND_S3_* or broad AWS_* variables, because those can refer to the active interim state backend or a wider cloud credential than TIN-1026 allows. By default it proves list-buckets, head-bucket, and bounded object write/head/read/delete against the packaged non-state scratch bucket. It refuses the active tofu-state bucket and the protected attic, arc-runners, gitlab-runners, and runner-dashboard state keys.

Use --run-disposable-tofu --use-lockfile to add the disposable backend proof: the harness creates a temporary root module with backend "s3", initializes OpenTofu against a scratch key with native S3 lockfile mode enabled, applies an output-only state, verifies a no-op plan, changes the state, restores the first object body, and verifies the restored state again. The proof fails early if the repo-managed OpenTofu binary is older than the lockfile-capable version required by the selected HA contract. Use --keep-scratch-bucket --checkpoint-file <path> before a pod restart or node-maintenance event, then rerun with --verify-existing --from-checkpoint <path> after the event to prove the scratch bucket/object still answer. New checkpoints record the endpoint package digest, and checkpoint verification refuses a different endpoint package file. This command is OpenTofu state-authority hardening; it is not Bazel RBE, BCR, CAS, or action-cache authority.

just ha-state-candidate-static-gate --contract <path> validates the written contract that must exist before any future backend is treated as the HA OpenTofu state authority. The gate requires endpoint, credential, recovery, locking, failure-behavior, observability, bucket-divergence response, proof order, and protected-migration fields. It rejects the current attic-rustfs-openebs singleton, Sting local-path storage, Attic/Bazel cache surfaces, and the active tofu-state bucket as final-state candidate contracts. Use just ha-state-candidate-static-gate-self-test for the offline fixture guard.

just ha-state-selected-candidate-static-gate validates the selected non-secret contract artifact at docs/contracts/ha-opentofu-state-managed-s3-candidate.json. That artifact selects a managed or appliance S3-compatible state service as the next proof target. It does not prove a live endpoint, migrate protected state, or turn the current RustFS singleton into HA.

just ha-state-endpoint-package-gate --package <path> validates the non-secret TIN-1026 endpoint package that must exist before just ha-state-candidate-proof runs against a real service. The package must name a non-placeholder HTTPS endpoint, region, network audience, state-only credential source and rotation owner, TOFU_HA_STATE_* injection variables, scratch bucket and scratch policy, protected state denials, recovery behavior, maintenance proof method, state-locking behavior, observability, proof commands, and authority separation. The proof commands must include the disposable OpenTofu proof with --use-lockfile, so endpoint readiness cannot skip the locking requirement. The gate rejects the current attic-rustfs-openebs singleton, in-cluster nix-cache.svc endpoint, active tofu-state bucket, inline secret fields, and broad cache/RBE authority claims. Use just ha-state-endpoint-package-self-test for the offline fixture guard. This is endpoint readiness, not a protected state migration.

just ha-state-endpoint-readiness --endpoint-package <path> is the operator preflight for TIN-1026. It validates the same non-secret package, then checks that scoped proof credentials are present as TOFU_HA_STATE_ACCESS_KEY and TOFU_HA_STATE_SECRET_KEY without printing their values. If TOFU_HA_STATE_ENDPOINT, TOFU_HA_STATE_REGION, or TOFU_HA_STATE_SCRATCH_BUCKET are set, they must match the package. The package path can also come from TOFU_HA_STATE_ENDPOINT_PACKAGE. A missing package, unedited template, mismatched override, or missing scoped credential returns TIN-1026_NOT_READY. A green result is only permission to run the scratch and disposable proof; the readiness command does not touch Kubernetes, S3, or OpenTofu state.

just ha-state-github-secret-readiness [args...] is the GitHub-side TIN-1026 preflight. It lists visible GitHub secret names only, never values, and checks for scoped TOFU_HA_STATE_ACCESS_KEY and TOFU_HA_STATE_SECRET_KEY authority. RUSTFS_*, ATTIC_*, and broad AWS_* names are reported when visible but do not count as HA state proof credentials. By default it checks repo-level secrets for tinyland-inc/GloriousFlywheel; pass --org <org> or --environment <name> when the proof runner intentionally relies on org or environment secrets. Org secret-name listing requires org admin or Actions secrets permission; an inaccessible org inventory is reported as unknown rather than silently ignored. A green result is only GitHub secret-name readiness; it does not validate the endpoint package or touch Kubernetes, S3, or OpenTofu.

just ha-state-endpoint-package-template [output] writes the exact endpoint package JSON shape expected by the gate. The generated file intentionally contains replace-me and example.invalid sentinels and must fail just ha-state-endpoint-package-gate --package <output> until an operator fills it with real non-secret endpoint, policy, retention, maintenance, and observability details. Do not commit credentials into the package; credentials come from TOFU_HA_STATE_ACCESS_KEY and TOFU_HA_STATE_SECRET_KEY or another secure injection path.

just k8s-secret-last-applied-audit --context <context> audits live Kubernetes Secrets for kubectl.kubernetes.io/last-applied-configuration annotations that still embed data or stringData. It reports only namespace/name and key names, never annotation values. A candidate TIN-1026 proof credential source is not hygiene-clean while this audit is red; remove the annotation or recreate the Secret through a path that does not persist secret material in object metadata before using it for TOFU_HA_STATE_*.

just k8s-secret-last-applied-cleanup-plan --context <context> renders a reviewable shell plan for the same findings. The plan removes only the kubectl.kubernetes.io/last-applied-configuration metadata annotation from each affected Secret with kubectl annotate ... last-applied-configuration-. It may name affected Secret keys for review, but it does not print Secret values and it does not execute the commands.

just ha-state-candidate-inventory [args...] is the read-only live candidate classifier for TIN-1012. It checks known object-store and storage surfaces, including the current attic-rustfs-openebs state path, staging S3-compatible test candidates, TCFS/SeaweedFS services, Sting local-path classes, and Longhorn presence. The normal evidence mode exits 0 even when no HA candidate is present so operators can capture the reality. Pass --fail-without-candidate when you need a blocking gate. This inventory only classifies possible OpenTofu state authority inputs; it is not Bazel RBE, BCR, CAS, or action-cache proof.

TOFU_HA_STATE_ACCESS_KEY=... \
TOFU_HA_STATE_SECRET_KEY=... \
just ha-state-candidate-proof \
  --endpoint-package ./endpoint-package.json \
  --run-disposable-tofu \
  --use-lockfile

Restart or node-maintenance proofs should use a kept checkpoint and explicit cleanup. The same endpoint package file content must be used for verification and cleanup, because the checkpoint records the package digest.

just ha-state-candidate-proof \
  --endpoint-package ./endpoint-package.json \
  --keep-scratch-bucket \
  --checkpoint-file /tmp/gf-ha-state-checkpoint.json

# Restart or maintenance event happens here.

just ha-state-candidate-proof \
  --endpoint-package ./endpoint-package.json \
  --verify-existing \
  --from-checkpoint /tmp/gf-ha-state-checkpoint.json

just ha-state-candidate-proof \
  --endpoint-package ./endpoint-package.json \
  --cleanup-checkpoint \
  --delete-scratch-bucket \
  --from-checkpoint /tmp/gf-ha-state-checkpoint.json

just attic-nar-integrity-check [args...] verifies the live Attic cache path that Nix uses after narinfo resolution. It mints a short-lived read-only token from the in-cluster Attic signing secret, fetches <cache>/<hash>.narinfo, then downloads the advertised NAR body and fails if the HTTP stream closes early or transfers zero bytes. Use it when Nix reports Transferred a partial file from the Attic substituter, because cache metadata can remain present while the RustFS-backed NAR body path is broken.

just attic-nar-integrity-check --store-hash 8iv5j0f2difw6wg9vwj9r2raacb08fkv

just tofu-state-contract <stack> prints the current state name and the proven ENV=dev or ENV=honey S3 key when one is locked. Use this before writing or reviewing backend files so you do not silently fall back to the wrong object name during the remaining cutovers.

just tofu-state-audit checks the known legacy GitLab state paths for the four active stacks individually and reports which ones actually exist, along with their Terraform version and serial. This replaces the older blind project-level state listing assumption, which does not work against the current GitLab API shape used by the archived legacy mirror.

just arc-runtime-audit inspects the live ARC runtime on the active cluster context and prints:

• which runner sets are live
• the AutoscalingRunnerSet phase for each runner set
• the live minRunners / maxRunners caps for each runner set
• the active runner pod count for each runner set
• the live listener-config min_runners / max_runners caps when present
• whether listener caps have drifted away from the AutoscalingRunnerSet
• whether live runner pods disagree with status.currentRunners
• the current CPU and memory envelope for each lane
• the current nodeSelector and tolerations
• the live ATTIC_SERVER value when present
• whether tinyland-nix-heavy exists yet
• whether the heavy lane is accidentally depending on storage-biased bumble instead of the currently admitted ARC payload surface
• active runner job pod placement by scale set label
• stale idle EphemeralRunnerSet blockers when a listener is missing while no-job runners are still registered
• recent active runner logs for ARC broker/session continuity errors such as Socket Error: TryAgain, acquirejob failed, cancelled-job handoff messages such as Job message not found, invalid job assignment, or lost server communication
• a kubectl top nodes snapshot when metrics are available

Use this after an arc-runners rollout to confirm the live cluster matches the repo contract instead of trusting stale runtime state. It is also the repeatable operator guard for TIN-620-style runner/CNI continuity checks.

The script also supports a bounded strict mode:

• bash ./scripts/arc-runtime-audit.sh honey arc-runners arc-systems --fail-on-listener-cap-drift
• bash ./scripts/arc-runtime-audit.sh honey arc-runners arc-systems --fail-on-listener-cap-drift --fail-on-active-listener-gap
• bash ./scripts/arc-runtime-audit.sh honey arc-runners arc-systems --fail-on-runner-count-drift
• bash ./scripts/arc-runtime-audit.sh honey arc-runners arc-systems --fail-on-runner-session-drift
• bash ./scripts/arc-runtime-audit.sh honey arc-runners arc-systems --fail-on-stale-idle-listener-blocker

Add --emit-drift-tsv to any strict invocation to also print machine-readable drift-tsv lines (set, reason, spec caps, listener caps) for each drifting set; just arc-prove-listener-caps consumes these to separate transient listener recreation from persistent drift after a managed apply, and just arc-reap-zombies --scale-set <name> [--dry-run] is the bounded manual counterpart of the managed apply’s pre-quiesce idle-leaked EphemeralRunner reap.

The listener-cap strict mode exits nonzero when an idle lane is missing listener config or when an idle listener-config cap disagrees with the matching AutoscalingRunnerSet. Add --fail-on-active-listener-gap for post-apply proofs that must also fail when config is missing while active runners still exist. Without that active-gap flag, strict listener-cap mode intentionally does not fail on missing config or cap drift while runners are still active, because ARC can defer listener replacement until the in-flight pods drain. The runner-count strict mode exits nonzero when live runner pods disagree with status.currentRunners. The runner-session strict mode exits nonzero when active runner pod logs contain known ARC broker/session continuity errors. If the log excerpt shows cancelled-job handoff evidence on a max-1 operator lane, first verify the replacement GitHub job is still queued and the pod has no Runner.Worker process, then delete only the affected EphemeralRunner. Do not use that cleanup as a generic queue-drain shortcut or hosted-runner escape hatch.

The stale-idle listener blocker strict mode covers the TIN-1647 failure shape: maxRunners > 0, no listener-config, and one or more running no-job EphemeralRunner objects owned by an EphemeralRunnerSet for the same capability lane. It is non-mutating. To turn its cleanup hint into an operator action, first capture GitHub runner API output and pass it through --github-runners-json; the audit only prints an EphemeralRunnerSet delete command when every candidate runner is verified as busy=false with GitHub status online or offline. Before deleting the set, also confirm the candidate pods have no Runner.Worker process. Never delete an assigned runner or use this as hosted-runner fallback.

just arc-network-continuity-audit is the companion TIN-620 classifier for the Kubernetes evidence around a lost-runner incident. It reads events, node conditions, and ARC runner pod status, then separates:

• API/CNI continuity evidence such as no route to host, connection refused, i/o timeout, kube-root-ca registration drift, and configmap/secret cache sync failures
• kubelet eviction or pressure evidence such as Evicted, DiskPressure, and ephemeral-storage
• node readiness or network-unavailable evidence

Use this after GitHub reports The self-hosted runner lost communication with the server so the incident is classified before blaming cache, auth, overlay, or downstream repo code. The bounded strict modes are:

• just arc-network-continuity-audit --runner-name <pod> --fail-on-network-drift
• just arc-network-continuity-audit --runner-name <pod> --fail-on-eviction
• just arc-network-continuity-audit --fail-on-node-pressure

just arc-listener-queue-drift --repo <owner/name> --run-id <run-id> is a read-only TIN-620 diagnostic for the queue/scale gap that arc-runtime-audit cannot see by itself. It correlates queued GitHub Actions jobs with live ARC AutoscalingRunnerSet status and listener pods. When multiple owner-overlay scale sets advertise the same workflow-facing label, --repo makes the diagnostic prefer the scale set whose githubConfigUrl matches the queued repo’s exact repo or owner/org scope. Use --fail-on-drift to exit nonzero when a queued job maps to a scale set with maxRunners > 0, a running listener, and zero current/pending/running ARC runner activity. That state means the problem is not a simple scale-set cap or raw cluster size issue; it is a listener/session/broker handling symptom.

just arc-shared-label-capacity-audit joins live Helm release values with ARC AutoscalingRunnerSet status and groups scale sets by workflow-facing tinyland-* labels. Use it for TIN-627-style checks where several owner overlays intentionally publish the same shared label through distinct ARC registration identities.

The audit is read-only. It prints the scale sets, owner scopes, current runner counts, per-scale-set max total, resource envelopes, and placement for each matching label. A warning on a label means ARC is enforcing maxRunners per scale set only; it is not a global concurrency cap for that shared label.

Optional strict flags are available for policy experiments, but should not be used as default product gates until a real global-capacity controller or policy decision exists:

• just arc-shared-label-capacity-audit --fail-on-cross-owner-label
• just arc-shared-label-capacity-audit --fail-on-multi-scale-label
• just arc-burst-capacity-audit --fail-on-warning
• just arc-burst-capacity-audit --fail-on-honey-pod-pressure
• just arc-burst-capacity-audit --fail-on-missing-overflow-pvc

just arc-diagnostic-self-tests runs the offline regression fixtures for arc-network-continuity-audit, arc-runtime-audit, arc-listener-queue-drift, arc-shared-label-capacity-audit, and arc-burst-capacity-audit. It is part of just check and CI validation, so API/CNI versus eviction classification, broker/session log classification, listener-cap drift, queue owner matching, shared-label capacity warnings, Honey pod-slot pressure, active runner job attribution, included runner placement reasons, and fast-local DinD scratch PVC regressions remain covered without requiring cluster access. The placement section is where Unschedulable / Too many pods failures should show up during a shared-label fanout before an operator reaches for a live scale-set mutation. The overflow-saturation section is the follow-on classifier: it calls out the product-relevant case where a Honey-bound baseline lane has pod-slot-blocked runners while the matching Sting compute-expansion lane is already at its source-owned maxRunners value and the Sting node still has pod headroom. Treat that as a reviewed cap/storage-envelope decision, not as proof that the cluster is out of CPU or memory.

The burst audit also prints Shared Label Queue Pressure. That section joins the included label report, active EphemeralRunner job repositories, and not-ready runner pod scheduler messages. Use it when a scarce capability lane, such as tinyland-nix-heavy, has pendingEphemeralRunners while its declared slot is occupied by another repository. A queued-behind-active-runner-capacity classification is queue/admission evidence; a scheduler-resource-pressure classification means Kubernetes is reporting resource pressure such as Insufficient ephemeral-storage and should be read before raising ARC maxRunners.

The burst audit also prints JIT Runner Assignment Traps. That section keeps same-label overflow incidents from being misread as generic resource exhaustion. offline-no-job-cleanup-candidate means the runner has no visible GitHub job assignment and its pod or ARC runner is not ready; verify the GitHub runner is offline/not busy before deleting the EphemeralRunner. assigned-job-at-risk means ARC has already attached a GitHub job to the runner, but Kubernetes has not made the pod usable; do not delete it unless the GitHub job is cancelled or explicitly declared stale. idle-no-job-runner means the runner is ready and unassigned, so leave it alone because ARC may still hand it work.

For Nix runner pods, the included placement rows also print storage-guidance when Sting reports Insufficient ephemeral-storage. That warning names the implementation boundary operators must check before raising caps: Nix runner root/store behavior needs a proved fast-local root/workdir model before Sting raw NVMe can count as usable Nix runner capacity. The first source-owned TIN-1400 model is tinyland-nix-compute-expansion with per-pod local-path-sting-fast-ephemeral PVCs for /nix and /home/runner/_work; do not lower heavier Nix lane requests or move heavier lanes until their exact storage model is proved live. The TIN-1600 correction lowers only the PVC-backed tinyland-nix-compute-expansion runner root ephemeral-storage request to 1Gi while leaving the PVC sizes intact. The TIN-1649 memory correction separately raises that same shared lane to a 4Gi memory request and 16Gi memory limit after dogfood OOM evidence.

The managed Deploy ARC Runners path has a separate productionization contract. It must run on an operator/control-plane lane outside the shared consumer labels it quiesces, use quiesce-arc-warm-runner-set.sh --freeze-max-runners so existing listeners cannot refill the lane during drain, and keep a cap snapshot that can be restored if quiesce or tofu apply fails. The managed workflow owns best-effort failure restore and gives active shared jobs a bounded 20-minute drain window before failing closed. On success it generates and guards a fresh post-quiesce apply plan, restores caps from source tfvars targets before proving listener caps, and avoids applying a stale pre-quiesce plan that cannot see the manual max-freeze drift. The pre-quiesce snapshot is only a failure rollback; after a successful apply it may contain the old cap that the source change intentionally replaced. tinyland-nix-heavy is only the bootstrap fallback; once tinyland-nix-operator is live, set ARC_DEPLOY_RUNNER_LABEL to that label.

just runner-capacity-model-check is the committed, offline guard for the live runner namespace ResourceQuota and LimitRange values. It parses the ARC and GitLab honey.tfvars files, derives the modeled burst envelope, and fails if a quota is too loose to serve as a real admission backstop or too small for the largest modeled runner pod. This is in-namespace capacity hygiene, not a cross-overlay global shared-label policy, and it does not claim every lane can run at its individual max concurrently when the quota is intentionally the finite machine-envelope stop.

just runner-scale-contract-check keeps the committed ARC and GitLab runner scale/placement policy explicit. It fails if ARC scale-to-zero minima drift from zero, if runner storage envelopes become implicit, or if default ARC/GitLab runner placement silently moves back to storage-biased bumble before the TIN-613 live remediation decision is explicit.

just kubelet-imagefs-capacity-audit reports node-level kubelet capacity that is not visible from durable storage size alone:

• Kubernetes ephemeral-storage capacity and allocatable values
• current scheduled pod request and limit envelopes
• kubelet summary rootfs, imagefs, and containerfs capacity, available bytes, and used bytes when the kubelet summary endpoint is available
• Ready, DiskPressure, and NetworkUnavailable status per node
• warnings when filesystem availability is below the configured threshold even if DiskPressure=False

The live audit can also read recorded fixtures through --nodes-json, --pods-json, and --summary-json-dir. just kubelet-imagefs-capacity-audit-self-test uses those fixture inputs to keep healthy, warning, and critical threshold behavior covered without cluster access.

Use this for TIN-613-style incidents where a node such as bumble has large OpenEBS/ZFS durable storage but a much smaller kubelet root/image filesystem. The default mode is read-only and non-failing:

• just kubelet-imagefs-capacity-audit
• just kubelet-imagefs-capacity-audit --node bumble
• just kubelet-imagefs-capacity-audit --fail-on-critical
• just kubelet-imagefs-capacity-audit-self-test

just honey-runner-workdir-audit inspects the default honey runner hosts (honey-am-1 and honey-am-2, or hosts you pass explicitly) over SSH and reports:

• visible repo workdirs under the runner _work/ root
• stale .git/index.lock files
• sample non-writable files that can break actions/checkout
• sample ownership mismatches
• largest workspace directories

Use this for checkout failures that happen before downstream repo code runs, such as EACCES unlink errors in persistent _work/* paths.

just honey-runner-workdir-remediate <host> <repo> [--mode unlock|remove] [--apply] is the bounded remediation pair for that audit:

• dry-run by default
• scoped to one repo workdir on one runner host
• unlock restores owner write bits so the tree can be inspected or removed
• remove restores owner write bits and deletes the contaminated repo workdir

Use this only after the affected runner has been stopped or drained. See Honey Runner Workdir Contract for the lifecycle boundary and escalation rules around that flow.

just honey-runner-workdir-reconcile sits between those two commands. It scans the selected honey runner hosts, classifies contaminated repo workdirs, and then:

• prints a bounded remediation plan when exactly one repo workdir is dirty on a host
• stops at escalation when more than one repo workdir is contaminated on the same host
• can run the safe single-repo remediation automatically when used with --apply --confirm-drained

Use this as the default operator entrypoint after a checkout-failure audit when the question is “which hosts can I recover safely right now without widening cleanup past the contract boundary?”

just honey-runner-host-lifecycle <host> [status|drain|start|restart] is the bounded lifecycle surface for one honey runner host root. It:

• discovers the runner root under /home/jess/am-runners/<host> unless you override it
• prefers svc.sh when present and falls back to run.sh when needed
• writes a repo-owned drain marker file before stop/restart actions
• verifies that the matching runner process actually stops or starts within a bounded grace window

Examples:

just honey-runner-host-lifecycle honey-am-2 status

just honey-runner-host-lifecycle honey-am-2 drain

just honey-runner-host-lifecycle honey-am-2 start

Use --local --runner-root /tmp/honey-am-2 when you want to validate the launcher contract against a local fake runner root without touching a real host.

just honey-runner-checkout-triage starts one step earlier when you already have the failing GitHub Actions run id or run URL. It:

• pulls the run metadata and full log with gh
• extracts honey host and repo-workdir targets from the checkout failure lines
• runs the bounded host reconcile surface for those hosts
• previews the targeted bounded repo remediation for the extracted workdir
• prints the follow-on host lifecycle commands for drain and restart
• can apply that bounded remediation only when --apply --confirm-drained is supplied and the reconcile result still says each host is a safe one-repo candidate

Examples:

just honey-runner-checkout-triage \
  https://github.com/Jesssullivan/scheduling-bridge/actions/runs/24525417273

just honey-runner-checkout-triage \
  --repo Jesssullivan/scheduling-bridge \
  24525417273 \
  --parse-only

Use --parse-only when you want the repo-owned run/log archaeology without touching the remote honey hosts from the current shell.

just orgwide-enrollment-scoreboard is the live reporting path for the orgwide enrollment contract. It scans the configured owners from config/orgwide-enrollment-scoreboard.json, inspects recent non-fork repos with workflows through the GitHub API, and prints:

• real runner authority on the default branch
• template consumers by mode
• named authority exceptions when non-default branches are still relevant

Use just orgwide-enrollment-scoreboard --format json when you want to feed the result into another report or compare the current live census with the earlier research baselines.

just orgwide-enrollment-queue is the follow-on execution surface for that scoreboard. It combines the live scoreboard with config/orgwide-enrollment-queue.json and prints:

• runner-authority-blocked repos whose counted self-hosted proof is currently dead because the repo exposes zero accessible runners or equivalent reachability failure
• related issue and proof-policy metadata for owner-boundary blockers, including closure canary dispatch policy, required assigned-job proof, and evidence that explicitly does not count
• hybrid-authority cleanup repos that already have real authority but still need explicit policy or convergence cleanup
• template-contract cleanup repos that should stay visible but not be confused with runner enrollment
• platform-prereq repos whose promotion story is blocked on missing shared runner reachability or similar authority debt
• real promotion candidates that are ready for actual rollout energy
• repos that should stay hosted and out of the active migration metric

Use just orgwide-enrollment-queue --format json when you want the queue in a machine-readable form for issue updates or PM rollups. Use repeated --repo owner/name arguments for a scoped live queue around a small owner-boundary decision.

Use just orgwide-enrollment-queue-contract-check when changing queue policy fields. It keeps the #407, #412, and #413 closure-proof guardrails from collapsing into queued-job or repo-shaped-runner claims.

just arc-runner-taxonomy-guard is a source-config guard for the ARC runner stack. It inspects committed literal runner_label assignments and extra_runner_sets entries. It fails if new runner lanes use repo-scoped GitHub URLs in committed core stack config, runner labels outside the shared tinyland-* capability namespace, or project-identity labels such as dell-7810-*.

The guard does not inspect live cluster residue. Live compatibility debt stays tracked through the enrollment queue and operator issues.

just arc-runner-residue-audit is the read-only live classifier for ARC runner residue. It queries Helm releases and ARC AutoscalingRunnerSet objects and groups them as shared capability lanes, implementation-overlay-owned lanes, Jess-rehomed compatibility lanes, standalone compatibility lanes, or unknown repo scopes.

just arc-runner-rehome-manifest-check validates config/arc-runner-residue-rehome.json, the static selected-disposition manifest for the current Jess personal-boundary rehome path. It does not pull or mutate state.

just arc-runner-residue-rehome-plan renders that manifest for operator review. Use --format commands to print dry-run tofu state mv command skeletons after the core and overlay state files have been pulled from initialized stack checkouts. The rendered commands are local state-file operations; they are not a remote-backend apply path.

just arc-runner-residue-state-check validates pulled OpenTofu state JSON files against the same manifest and prints only resource addresses. In the default pre-move phase it expects selected source addresses to exist in core state and destination addresses to be absent from overlay state. Use --phase post-move after local state-file movement and before any state push. Do not push either state file unless the post-move checker reports summary: 0 blockers.

just bazel-dogfood-tranche-status is the live operator path for tranche-1 Bazel productization proof. It reads config/tranche-proof-status.json, queries the current GitHub workflow state for the bounded proof bundle, and prints:

• the current source-repo Bazel proof on GloriousFlywheel
• the lab benchmark/cache-evidence canary
• the canonical scheduling-kit package-lane proof
• the tinyland-inc/scheduling-kit mirror-integrity proof
• the explicit scheduling-bridge publish dry-run proof
• the intentional tinyland.dev hybrid-by-policy boundary

Use just bazel-dogfood-tranche-status --format json when you want the bundle in a machine-readable form for issue updates or program rollups.

That same bundle now has an Actions workflow path through Tranche Proof Status. The workflow dogfoods tinyland-nix, renders the command on main, on demand, and on a daily schedule, publishes text and JSON artifacts, and writes the text view into the workflow run summary.

The source-repo tranche proof itself now has a separate routine workflow path through Source Bazel Proof on main, on demand, and on a daily schedule. That keeps the bounded Bazel/cache-first package from riding inside the broader Platform Proof runner-contract matrix.

The Actions workflow now prefers a GitHub App path when TRANCHE_PROOF_GH_APP_CLIENT_ID and TRANCHE_PROOF_GH_APP_PRIVATE_KEY are set as GitHub Actions secrets. With those in place, the workflow mints bounded owner-scoped read tokens for tinyland-inc and Jesssullivan instead of relying on one broad PAT. TRANCHE_PROOF_GH_APP_ID remains a compatibility fallback if the client-id secret is not present yet.

Tranche-proof workflow runs now require that GitHub App path and fail fast if those App secrets are missing, instead of silently falling back to a broad PAT. Local shells can still provide TRANCHE_PROOF_GH_TOKEN_<OWNER> overrides such as TRANCHE_PROOF_GH_TOKEN_TINYLAND_INC and TRANCHE_PROOF_GH_TOKEN_JESSSULLIVAN when you want to inspect the bundle outside Actions without reusing the Actions App flow.

The GF REAPI Cell Proof workflow reuses the same GitHub App secret pair for private consumer checkouts. For supported owners (tinyland-inc and Jesssullivan), it mints a repository-scoped token for the requested consumer_repository with contents: read; private proof dispatches should set require_consumer_app_token=true so missing App credentials fail before the consumer checkout rather than being misclassified as RBE target evidence. Public consumer proofs should leave that input false so the checkout stays on the workflow’s normal GITHUB_TOKEN path.

When the App permission update is still blocked, the proof workflow can be dispatched with --consumer-checkout-authority repo-scoped-deploy-key or --consumer-checkout-authority owner-scoped-secret instead of --require-consumer-app-token. The deploy-key path is preferred when a read-only deploy key can be installed on the consumer repo; it requires GF_REAPI_CONSUMER_CHECKOUT_SSH_KEY_TINYLAND_DEV or GF_REAPI_CONSUMER_CHECKOUT_SSH_KEY_MASSAGEITHACA. The token path requires a fixed repo secret for the owner: GF_REAPI_CONSUMER_CHECKOUT_TOKEN_TINYLAND_INC or GF_REAPI_CONSUMER_CHECKOUT_TOKEN_JESSSULLIVAN. Use repository-scoped read credentials only; do not replace this with a broad PAT or a free-form workflow input token. The checkout credential only gets the consumer workspace to the next proof gate; passing checkout is not RBE evidence until the proof artifact shows nonzero remote execution.

The same proof workflow can stage the private tinyland-schemas v0.2.4 archive through --tinyland-schemas-private-handoff. That path mints a GitHub App token scoped to tinyland-inc/tinyland-schemas, downloads the GitHub codeload tag archive matching the BCR-recorded archive sha256 and prefix, verifies its sha256, and exposes it through BAZEL_DISTDIR for the proof run. This is private proof-run distdir staging, not durable mirror authority, repository-cache retention, CAS/action-cache authority, or broad/default RBE.

For arc-runners, local tofu-plan, tofu-refresh, and tofu-destroy also accept GHCR_USERNAME and GHCR_TOKEN. When those are set, the root operator path passes them through as ghcr_username and ghcr_token so OpenTofu can create or rotate an explicitly configured GHCR pull secret. The current honey runtime does not use imagePullSecrets on the ARC lanes by default, so public-image plans can leave those variables unset without causing artificial drift.

just tofu-backend-scaffold <stack> creates the expected local backend file in config/backends/. Once that file exists, it becomes the implicit local default for just tofu-init <stack>. Preflight will still fail until you replace the example placeholder values with a real current backend endpoint and credentials.

just tofu-backend-scaffold-s3 <stack> writes the same local filename, but with the migration-prep S3-compatible example instead of the active HTTP one. That is preparation only. Preflight will reject that file until the matching stack backend.tf also switches to the s3 backend family.

If you are repairing a legacy generic HTTP backend path and already have the coordinates exported in your shell, just tofu-backend-materialize-http <stack> will capture those TF_HTTP_* values into the local backend file for that stack.

If you already have the target-direction S3-compatible coordinates exported in your shell, just tofu-backend-materialize-s3 <stack> will capture those TOFU_BACKEND_S3_* values into the local backend file for that stack. That still does not change the active stack backend family by itself.

For the current proven honey baseline, that helper now uses the live dev key map instead of assuming <stack>-<env>.tfstate:

• attic -> attic/terraform.tfstate
• arc-runners -> arc-runners/terraform.tfstate
• gitlab-runners -> tinyland-infra/gitlab-runners/terraform.tfstate
• runner-dashboard -> tinyland-infra/runner-dashboard/terraform.tfstate

If you are preparing a different environment or a non-baseline layout, set TOFU_BACKEND_S3_KEY explicitly before running the helper.

If you still rely on legacy GitLab HTTP state and you know the real gitlab.project_id, just tofu-backend-materialize-gitlab-legacy <stack> will write the local backend file for you from config/organization.yaml plus TF_HTTP_PASSWORD. That is compatibility-only; the four active stacks on current main now use backend "s3".

Bazel

Build system recipes for the Bzlmod-based build. The normal GloriousFlywheel story is direnv allow or nix develop, just info, then just cache-contract-strict, then just bazel-build-cached once the shell is attached to a real shared cache. That proves cache-backed local execution with shared acceleration, not full remote execution.

Kubernetes

Cluster inspection and debugging recipes. All commands route through the proxy when HTTPS_PROXY is set, but the preferred path is direct or tailnet-private access to honey.

For the current Tinyland on-prem rollout, the cache-oriented helper defaults now use the live nix-cache namespace.

App (Runner Dashboard)

Recipes for the SvelteKit runner-dashboard application.

Runners

Shortcut recipes for the GitLab Runner infrastructure stack (gitlab-runners).

Cache Platform

Shortcut recipes for the cache platform stack (tofu/stacks/attic/). This stack deploys the complete Nix binary cache platform: CNPG Operator, PostgreSQL cluster, RustFS S3-compatible object storage, Attic API server, GC worker, DNS records, cache init job, warming CronJob, and optional Bazel remote cache.

The stack directory is named attic for state backend compatibility, so just tofu-deploy attic also works.

Docs

Recipes for the documentation site.

TeX

Recipes for building the research document.

Related

• Environment Variables — variables consumed by these recipes
• Configuration Reference — organization.yaml used by tofu recipes
• Pipeline Overview — CI equivalents of local recipes