Watch-Store Bootstrap

Watch-Store Bootstrap

attic watch-store is the mechanism that turns Nix builds into incrementally cached operations. Rather than pushing the entire closure to the binary cache after the build completes, watch-store pushes each store path as it appears in /nix/store, making partial results available to future builds immediately.

How watch-store Works

When started as a background process, attic watch-store <cache-name> opens an inotify (Linux) or FSEvents (macOS) watch on /nix/store. Each time Nix finalizes a new store path — meaning the derivation built successfully and its output hash is registered — watch-store detects the new entry and pushes it to the named Attic cache.

The push is asynchronous relative to the build. Nix continues building the next derivation in the dependency graph while the previous one uploads. This overlap means that for most builds, the upload cost is hidden behind computation time.

Incremental Caching and Partial Failures

The key advantage of watch-store over end-of-build push is resilience to partial failures. Consider a complex derivation tree with 80 intermediate outputs:

  • End-of-build push: If the build fails at output 60, zero outputs are cached. The next pipeline rebuilds all 60 from scratch.
  • watch-store: Outputs 1 through 59 are already in the cache when the failure occurs. The next pipeline substitutes those 59 outputs instantly and resumes from output 60.

This property is especially valuable for long-running Rust compilations (like the attic client itself) and for Nix builds that involve many independent intermediate derivations (system libraries, toolchains, compiler bootstraps).

The Bootstrap Problem

watch-store depends on the attic CLI binary. The attic CLI is itself a Nix derivation — building it from source involves compiling a substantial Rust codebase with Cargo, which can take 30-60 minutes on CI runners.

This creates a circular dependency: to push derivations incrementally, we need attic; to get attic quickly, we need it to already be cached; to cache it, we need to have pushed it with attic.

Bootstrap Sequence

The bootstrap logic resolves this circular dependency with a two-phase approach:

Phase 1: Attempt Substitution Only

nix build github:NixOS/nixpkgs/nixos-24.11#attic-client \
  --no-link \
  --print-out-paths \
  --max-jobs 0

The --max-jobs 0 flag tells Nix to fetch from configured substituters (binary caches) but never build locally. If the attic client is already in the cache, this completes in seconds. If not, it fails immediately — there is no 60-minute wait.

In GitHub Actions, the shared Nix wrapper adds the workflow GITHUB_TOKEN as github.com fetcher auth. That token only raises the GitHub API rate limit for flake source resolution; Attic cache reads and writes still use the configured netrc-backed ATTIC_TOKEN.

Phase 2a: Client Available (Common Case)

If phase 1 succeeded, the command prints the store path containing bin/attic. The CI job:

  1. Authenticates with attic login ci <server-url> <token>.
  2. Starts the background watcher with attic watch-store <cache-name> &.
  3. Proceeds to the main nix build step. All new derivations stream to the cache in real time.
  4. In after_script, stops watch-store and logs the push count.

Phase 2b: Client Not Available (First Pipeline)

If phase 1 failed (no client is substitutable), the CI job skips Attic publication for that job and leaves the build result unchanged. It does not compile Attic locally as a side effect of validation. Seeding the client is a separate warm-cache or runner-image responsibility.

CI Lifecycle Integration

The watch-store lifecycle maps to GitLab CI’s job phases:

CI Phase Action
before_script Discover Attic server endpoint. Attempt client substitution. If available, authenticate and start watch-store in the background.
script Run nix build normally. Derivations are pushed incrementally by the background watch-store process.
after_script Terminate watch-store. Log the number of store paths pushed. Run a final delta push when the client is available so top-level outputs and their closures are cached.

The final explicit push in after_script is not strictly necessary when watch-store is running, but it guards against edge cases where watch-store might miss a path (for example, if the process is terminated before it processes the last batch of events).

Monitoring

watch-store logs its activity to stderr. In CI job logs, look for:

watch-store pushed 47 store paths incrementally

A high number indicates many cache misses were filled during the build. A low number (or zero) means most derivations were already cached. Both are normal — the first indicates a productive cache-warming run, and the second indicates a mature cache.

GloriousFlywheel