FlakeHub Cache vs Self-Hosted Attic: Evaluation

Executive Summary

Recommendation: Hybrid — FlakeHub for public flake distribution, Attic for internal binary cache. This avoids vendor lock-in for the performance-critical CI cache path while leveraging FlakeHub’s zero-config distribution for external consumers.

Candidates

FlakeHub Cache (Determinate Systems)

Managed binary cache service integrated with FlakeHub flake registry.

Attic (Self-Hosted)

Self-hosted Nix binary cache with chunk-level deduplication.

Comparison

Performance

For CI runners in the same cluster as Attic, self-hosted wins on latency by 10-20x. This matters for builds with hundreds of store path fetches.

Cost

For a small team (<5 devs, <10 runners), FlakeHub is cheaper when ops time is valued at >$50/hr. For larger deployments or teams that already manage Kubernetes, Attic’s infrastructure cost is negligible.

Features

Risk

Hybrid Architecture (Recommended)

                       External Consumers
                             |
                        FlakeHub
                    (public distribution)
                             |
                    flakehub-publish.yml
                    (push to main / tags)
                             |
            +----------------+----------------+
            |                                 |
       CI Runners                        Dev Machines
            |                                 |
     Attic (cluster-local)          Attic (via tailnet)
     (fast read/write)              (fast read, occasional write)
            |                                 |
     Shared /nix/store PVC          Local /nix/store
     (Longhorn RWX)                 (attic use gloriousflywheel)

How It Works

1. CI builds use Attic exclusively — cluster-local latency, chunk dedup, no egress costs, no SaaS dependency in the critical path.

2. FlakeHub publishing happens post-merge via GitHub Actions workflow (already implemented in .github/workflows/flakehub-publish.yml). Uses OIDC auth, no secrets needed.

3. External consumers pull from FlakeHub as a flake input. They never interact with Attic.

4. Dev machines use Attic over the tailnet for fast cache reads. Optionally also use FlakeHub as a flake input for published versions.

Why Not FlakeHub-Only?

• CI runners are in the same cluster as Attic — 5ms vs 200ms latency
• Chunk-level dedup saves storage for incremental Nix builds
• No SaaS dependency in CI critical path (builds work during FlakeHub outage)
• Data sovereignty (build artifacts stay in-cluster)
• Free (no per-org subscription for cache)

Why Not Attic-Only?

• FlakeHub provides semantic versioning and public discovery
• External consumers shouldn’t need Attic access or signing keys
• FlakeHub handles CDN, availability, and global distribution
• flake.lock resolution works natively with FlakeHub URLs

Migration Path

Current state: Attic deployed but not fully operational (#185). FlakeHub publishing automated (#198).

Phase 1: Restore Attic (Issues #185, #167)

• Fix Attic deployment in cluster
• Verify CI runners can push/pull from Attic
• Set up garbage collection cron
• Monitor cache hit rates

Phase 2: Validate Hybrid

• Confirm FlakeHub publishing works end-to-end
• Verify external consumer flow (flakehub.com/f/tinyland-inc/GloriousFlywheel/*)
• Measure CI cache hit rates with Attic
• Compare build times with/without cache

Phase 3: Optimize

• Tune Attic GC retention policy
• Add Prometheus metrics for cache hit/miss rates
• Consider FlakeHub Cache as fallback if Attic is down (optional)
• Document cache strategy in runbook

Decision Record