Comprehensive Build Infrastructure Plan

IaC Monorepo - Full Stack Package Caching

Date: 2026-02-05 Status: Planning Phase Repository: attic-cache (dogfood example)

Executive Summary

This plan consolidates research from 4 parallel investigations to create a unified build infrastructure supporting:

Key Answer: Yes, Attic can cache ALL standard Nix packages - musl builds, web servers, flakes, devShells, cross-compiled binaries. Attic’s chunk-level deduplication makes it especially efficient for variant builds (glibc/musl/static) that share common code.

Part 1: Attic for Standard Nix Packages

What Attic Can Cache

musl/Static Build Example

# flake.nix
{
  packages.x86_64-linux = {
    # Standard glibc
    myapp = pkgs.myapp;

    # Static musl (single binary)
    myapp-static = pkgs.pkgsStatic.myapp;

    # Cross-compiled aarch64 static
    myapp-aarch64 = pkgs.pkgsCross.aarch64-multiplatform-musl.pkgsStatic.myapp;
  };
}

# Build and cache
nix build .#myapp-static --out-link result
attic push main result

# Verify static linking
file result/bin/myapp
# Output: ELF 64-bit LSB executable, statically linked

devShell Caching

# Make devShell cacheable
packages.x86_64-linux.dev-shell =
  self.devShells.x86_64-linux.default.inputDerivation;

# Build and cache devShell dependencies
nix build .#dev-shell && attic push main result

watch-store for Comprehensive Caching

# Cache ALL store paths created during a build
attic watch-store main &
nix build .#package
nix develop -c echo "Shell deps cached"
kill %1

Recommended Cache Organization

# Multiple caches for different retention needs
attic cache create main --public              # General (90 days)
attic cache create ci --public                # CI builds (30 days)
attic cache create static --public            # musl/static (180 days)
attic cache create devshells --public         # Dev environments (60 days)

Part 2: Bazel Remote Caching

bazel-remote Module

New module at tofu/modules/bazel-cache/ with:

Resource Requirements:

Client Configuration

# .bazelrc
build --remote_cache=grpc://bazel-cache.prod.example.com:9092
build --remote_upload_local_results=true
build --remote_download_minimal

# CI-specific
build:ci --remote_cache=grpc://bazel-cache.bazel-cache.svc.cluster.local:9092

rules_nixpkgs Integration

# MODULE.bazel
bazel_dep(name = "rules_nixpkgs_core", version = "0.13.0")
bazel_dep(name = "rules_nixpkgs_cc", version = "0.13.0")

# .bazelrc
common --host_platform=@rules_nixpkgs_core//platforms:host
common --incompatible_enable_cc_toolchain_resolution

Supported Toolchains: C/C++, Go, Python, Rust, Haskell, Node.js

Part 3: RPM, NuGet, and Pulp Options

Comparison Matrix

Recommendations

1. Start with GitLab Package Registry - Already available, supports NuGet/npm/PyPI
2. Add Nexus OSS - If you need RPM or unified management
3. Add Galaxy NG (Pulp) - For Ansible collections/roles

Nexus OSS Deployment

# tofu/modules/nexus-repository/main.tf
resource "helm_release" "nexus" {
  name       = "nexus"
  repository = "https://sonatype.github.io/helm3-charts/"
  chart      = "nexus-repository-manager"
  namespace  = var.namespace

  set {
    name  = "persistence.storageClass"
    value = "longhorn"
  }

  set {
    name  = "nexus.resources.requests.memory"
    value = "4Gi"
  }
}

Galaxy NG (Pulp) for Ansible

# tofu/modules/galaxy-ng/main.tf
resource "helm_release" "galaxy_ng" {
  name       = "galaxy"
  repository = "https://pulp.github.io/pulp-operator"
  chart      = "pulp"
  namespace  = var.namespace

  values = [yamlencode({
    pulp_settings = {
      galaxy_enable_api_access_log = true
    }
    content_origin = "https://galaxy.prod.example.com"
  })]
}

Part 4: IaC Monorepo Dogfooding

attic-cache as Reference Implementation

This repository demonstrates:

attic-cache/                    # IaC Monorepo Example
├── MODULE.bazel                # Bzlmod dependencies
├── flake.nix                   # Nix tools + containers
├── .bazelrc                    # Bazel configuration
│
├── tofu/
│   ├── modules/                # Reusable OpenTofu modules
│   │   ├── BUILD.bazel         # Module validation targets
│   │   ├── bazel-cache/        # NEW: Bazel caching
│   │   ├── nexus-repository/   # NEW: RPM/NuGet
│   │   └── galaxy-ng/          # NEW: Ansible
│   └── stacks/
│       └── attic/              # Self-deploying infrastructure
│
├── build/                      # Custom Bazel rules
│   └── tofu/
│       └── rules.bzl           # tofu_validate, tofu_fmt
│
└── tests/                      # Validation tests
    └── BUILD.bazel

OpenTofu Validation with Bazel

# tofu/modules/BUILD.bazel
load("@rules_tf//tf:defs.bzl", "tf_module")

tf_module(
    name = "bazel_cache",
    srcs = glob(["bazel-cache/*.tf"]),
    providers = ["kubernetes", "helm"],
)

tf_module(
    name = "nexus_repository",
    srcs = glob(["nexus-repository/*.tf"]),
    providers = ["kubernetes", "helm"],
)

Affected Target Analysis

# .gitlab-ci.yml
bazel:affected:
  script:
    - |
      # Only validate changed modules
      target-determinator \
        -targets="//tofu/..." \
        origin/main HEAD > affected.txt

      if [ -s affected.txt ]; then
        bazel build $(cat affected.txt)
      fi

Part 5: Implementation Phases

Phase 1: Current (Attic Foundation)

Status: In Progress

• Attic deployed on dev-cluster
• MinIO storage configured
• GitLab CI integration
• Production deployment (prod-cluster)
• Document musl/static caching patterns

Deliverables:

• Attic serving at attic-cache.prod.example.com
• Example flake.nix with static builds
• CI template for cache push

Phase 2: Bazel Caching (Weeks 1-3)

Goals:

• Deploy bazel-remote with MinIO backend
• Integrate rules_nixpkgs for toolchains
• Add OpenTofu validation rules

Tasks:

Phase 3: Package Repositories (Weeks 4-6)

Goals:

• Evaluate GitLab Package Registry for NuGet
• Deploy Nexus OSS if needed
• Deploy Galaxy NG for Ansible

Tasks:

Phase 4: Full Integration (Weeks 7-10)

Goals:

• Affected target analysis in CI
• Self-hosting validation (dogfood)
• Organization-wide documentation

Tasks:

Part 6: Resource Summary

Total New Infrastructure

Storage Allocation (MinIO)

Current MinIO: 800Gi (sufficient)

Part 7: Quick Reference

Cache URLs (Target State)

Client Configuration

Nix (nix.conf):

substituters = https://attic-cache.prod.example.com/main https://cache.nixos.org

Bazel (.bazelrc):

build --remote_cache=grpc://bazel-cache.prod.example.com:9092

NuGet (nuget.config):

<add key="Internal" value="https://gitlab.com/api/v4/groups/GROUP_ID/-/packages/nuget/index.json" />

Ansible (ansible.cfg):

[galaxy]
server_list = org_galaxy, galaxy

[galaxy_server.org_galaxy]
url = https://galaxy.prod.example.com/api/

Appendix A: Research Documents

Appendix B: Decision Log

Appendix C: Related Projects