Building a Pull-Through Container Registry Cache for a 5-Node Kubernetes Homelab

The Problem: Every Node Pulls From the Internet, Every Time

A 5-node cluster with no shared image cache means every node independently hits Docker Hub, ghcr.io, and quay.io for every image it doesn’t already have on local disk. An audit of the cluster turned up 253 running containers across 93 distinct images — and Docker Hub, the registry with the strictest anonymous-pull rate limits, accounted for the most redundant traffic: postgres:16-alpine pulled independently on eight different occasions, postgres:16 on seven, bitnami/kubectl on six.

One node in particular makes this worse. kube-worker4 is a baremetal HP Proliant with a passively-cooled Tesla T4, and it power-cycles nightly on a Wake-on-LAN schedule to save power. Every morning it wakes up with an empty local image cache and has to cold-pull everything it needs from scratch before its GPU workloads can actually start.

The fix is a pull-through registry cache: one shared cache backed by NAS storage, with every node’s container runtime transparently redirected to it for the registries that matter most.

---

Architecture

Component	Choice	Why
Cache engine	Zot (single replica)	OCI-native, no database dependency, small footprint. A distribution/Harbor-style registry would work too, but Zot’s on-demand sync extension is a closer match for “cache what’s requested” rather than “mirror everything up front.”
Backing storage	NFS, dedicated export	Persistent across pod restarts and node reschedules; a StatefulSet with a network-backed PVC, not local disk.
Node-to-cache routing	containerd hosts.toml per registry	The standard, non-deprecated way to redirect specific registries to a mirror, per node, without touching image references anywhere in the cluster.
Cache reachability	Fixed NodePort, not ClusterIP	containerd runs on the host, outside the pod network — it cannot resolve cluster-DNS service names. Every node talks to the cache at http://127.0.0.1:<nodePort>, which works regardless of which node the cache pod currently lives on.

Here’s the full request path, from a pod needing an image to that image landing on disk:

flowchart TD
    subgraph Node["Any Kubernetes Node"]
        Pod["Pod scheduled,<br/>needs image X"]
        Kubelet["kubelet"]
        Containerd["containerd<br/>(host process)"]
        HostsToml["/etc/containerd/certs.d/&lt;registry&gt;/hosts.toml<br/>redirects to 127.0.0.1:NodePort"]
    end

    subgraph Cache["Registry Cache (StatefulSet, 1 replica)"]
        Zot["Zot"]
        Decision{"Already cached<br/>&amp; fresh?"}
        Storage[("NFS-backed storage")]
    end

    subgraph Upstream["Upstream Registries"]
        Hub["Docker Hub"]
        Ghcr["ghcr.io"]
    end

    Pod --> Kubelet --> Containerd --> HostsToml --> Zot
    Zot --> Decision
    Decision -->|yes, cache hit| Storage
    Decision -->|no, sync on demand| Hub
    Decision -->|no, sync on demand| Ghcr
    Hub --> Storage
    Ghcr --> Storage
    Storage --> Zot
    Zot -->|image layers| Containerd

    style Zot fill:#4c8bf5,color:#fff
    style Storage fill:#2d9d5f,color:#fff

If the mirror fails for any reason — the cache is down, or it can’t authenticate to pull a private image — containerd’s hosts.toml falls through to a direct pull from the origin registry automatically. The cache is additive. Nothing breaks if it’s unavailable; pulls just stop being fast.

---

Deploying the Cache

Zot via Helm, pinned to a manually-provisioned PV/PVC on the NFS export rather than the cluster’s default dynamic StorageClass — this cache is deliberately isolated from other NFS-backed workloads:

persistence: true
pvc:
  create: false
  name: registry-cache-pvc

podSecurityContext:
  fsGroup: 999
securityContext:
  runAsUser: 999
  runAsGroup: 999

service:
  type: NodePort
  nodePort: 30500

mountConfig: true
configFiles:
  config.json: |-
    {
      "storage": { "rootDirectory": "/var/lib/registry" },
      "extensions": {
        "sync": {
          "enable": true,
          "registries": [
            { "urls": ["https://registry-1.docker.io"], "onDemand": true },
            { "urls": ["https://ghcr.io"], "onDemand": true }
          ]
        }
      }
    }

Two settings matter more than they look:

• pvc.create: false — the chart defaults to dynamically provisioning a PVC via the default StorageClass. That’s wrong for a manually-created NFS PV; without this, the cache silently ends up on the wrong backing storage.
• runAsUser: 999 — the chart’s own values.yaml doesn’t document a securityContext field in its comments at all. It’s there in the templates, just unmentioned. Without it, Zot’s container UID doesn’t match the NFS export’s ownership and every write fails.

---

Redirecting containerd: Three Bugs in One Feature

This is where most of the actual debugging time went. Getting hosts.toml recognized correctly turned out to involve three distinct, real containerd bugs — reproduced identically across containerd 1.7.24, 1.7.27, and 2.2.1, so none of this is specific to this cluster.

Bug 1 — the default generated config is broken. Running containerd config default produces a config_path with two colon-separated directories:

config_path = '/etc/containerd/certs.d:/etc/docker/certs.d'

This is silently ignored. No error, no warning — pulls just skip the mirror entirely and go straight to the origin. The fix is a single path only:

config_path = '/etc/containerd/certs.d'

Bug 2 — containerd 2.1+’s Transfer Service has its own, separate config_path. Since containerd 2.1, CRI image pulls go through a newer “transfer service” by default, and that plugin (io.containerd.transfer.v1.local) reads its own config_path, entirely independent from the CRI image plugin’s registry config. Fixing one without the other means the mirror is silently bypassed regardless of what the CRI plugin’s config says.

Bug 3 — a GPU driver install can silently shadow the whole registry config. Both GPU-bearing nodes in this cluster have /etc/containerd/conf.d/99-nvidia.toml, dropped in by the NVIDIA container toolkit setup, and imported via:

imports = ["/etc/containerd/conf.d/*.toml"]

That file redeclares the entire CRI registry block, including config_path = "" — overriding any edit made to the main config file. Editing /etc/containerd/config.toml looked correct and did nothing. The only way to catch this was containerd config dump, which shows the actual effective, merged configuration rather than the contents of any one file on disk:

$ containerd config dump | grep -A2 registry
[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = ""   # <- main file says something else; this is what's actually active

If either GPU node’s NVIDIA toolkit ever gets reinstalled, this file likely regenerates and the mirror silently breaks again. Worth a spot-check after any driver update, not something to assume is permanent.

Verifying It Actually Works

ctr images pull is not a valid test — it bypasses certs.d entirely, via a different resolver path than the one kubelet actually uses. The only reliable check: schedule a real pod requesting a tag that’s never been pulled anywhere, then watch the cache’s own logs.

$ kubectl logs -n registry-cache zot-0 | grep sync
"trying to get updated image by syncing on demand" repository=library/alpine reference=3.19.1
"sync: syncing image" remote=registry-1.docker.io repo=library/alpine
"successfully synced image" repo=library/alpine reference=3.19.1

That’s unambiguous. A pod reaching Running isn’t — it might just be serving an image that was already sitting on local disk from before the mirror existed.

One More Thing: Restarting containerd Is Safer Than It Looks

Every one of these fixes required a containerd restart to take effect, including on the cluster’s sole control-plane node. That’s a reasonable thing to be nervous about — but containerd’s shim architecture makes it lower-risk than it sounds. Running containers are managed by independent containerd-shim-runc-v2 processes that stay alive across a daemon restart and simply reattach once the daemon comes back. Restart counts for etcd, kube-apiserver, kube-scheduler, and kube-controller-manager were identical before and after restarting containerd on the control-plane node — zero disruption, confirmed by comparing restart counts, not just assuming.

---

Warming the Cache Before It’s Needed

The NodePort redirect solves cold pulls for anything requested after a node is up. It doesn’t help the first request after kube-worker4 wakes from its nightly power cycle — that pull is still cold, cache or no cache, if nothing has asked for that image yet.

For that, a small CronJob timed to fire shortly after the node’s Wake-on-LAN wake window runs crictl pull (not ctr — same bypass problem as above) against the handful of images backing the node’s KEDA-scaled-to-zero GPU workloads, using the host’s existing containerd socket:

containers:
  - name: prepull
    image: busybox:1.36
    command:
      - sh
      - -c
      - |
        for img in \
          docker.io/linuxserver/dolphin:latest \
          docker.io/ollama/ollama:latest \
          ghcr.io/ai-dock/stable-diffusion-webui:latest-cuda \
        ; do
          /host/usr/bin/crictl --runtime-endpoint unix:///run/containerd/containerd.sock pull "$img"
        done
    volumeMounts:
      - { name: crictl-bin, mountPath: /host/usr/bin/crictl }
      - { name: containerd-sock, mountPath: /run/containerd/containerd.sock }

These are real multi-gigabyte emulator and Stable Diffusion images — the first test run needed a 900-second activeDeadlineSeconds after the default 300 seconds wasn’t enough. Because it runs through the same containerd/hosts.toml path as everything else, this pre-pull warms the shared cache too, not just that one node’s local disk — so the very first real user request of the day is also fast.

---

Keeping It Self-Healing

A minimal DaemonSet reconciles the hosts.toml files on every node every five minutes, so a future node rebuild picks up the mirror configuration automatically without a manual step:

containers:
  - name: mirror-config
    image: busybox:1.36
    command:
      - sh
      - -c
      - |
        mkdir -p /host/certs.d/docker.io /host/certs.d/ghcr.io
        while true; do
          cp /config/docker.io-hosts.toml /host/certs.d/docker.io/hosts.toml
          cp /config/ghcr.io-hosts.toml /host/certs.d/ghcr.io/hosts.toml
          sleep 300
        done

It deliberately does not touch config_path or the NVIDIA conf.d shadowing issue — that part is genuinely node-specific and needs a restart plus a config dump verification, which isn’t something to automate blindly onto a control-plane node.

---

Results

• All 5 nodes — including the control-plane node — pull through the shared cache for docker.io and ghcr.io, verified end-to-end via cache logs, not just pod status.
• kube-worker4’s nightly wake now pre-warms its GPU workload images before real traffic arrives, instead of cold-pulling multi-gigabyte images on first request.
• Private ghcr.io images (no registry credentials configured on the cache yet) fall through to a direct pull exactly as before — the cache is additive, so adding a new upstream registry can’t regress anything that already worked.
• Three real, previously undocumented-by-me containerd bugs, found by actually reading containerd config dump output instead of trusting the config file on disk.

The single most useful debugging habit here: never trust that an edited config file is the config that’s actually running. containerd config dump — or the equivalent “show me the effective, merged state” command for whatever system you’re touching — is the difference between a fix that looks right and a fix that works.