containerd 1.x is done: what breaks moving to containerd 2.0 before Kubernetes 1.36

The Kubernetes v1.35 release blog spells it out plainly: 1.35 is the last minor version that still supports nodes running containerd 1.x. Before upgrading to 1.36, every node has to move to containerd 2.0+. On paper that reads like one more line in the deprecation section of the release notes. In practice it’s a separate cluster-wide rollout with an incompatible config format, renamed plugins, and different GPU behavior — and it’s worth planning ahead of time, not the week before the control-plane upgrade.

Table of contents

Why this isn’t a routine minor bump

containerd 2.0 shipped back in late 2024 as the project’s first major release since 1.0, and it has lived alongside the 1.7 branch ever since — most managed clusters (EKS, GKE, AKS) kept 1.7 as the safe default until the last possible moment. Kubernetes’ version skew policy no longer guarantees that coexistence: 1.35 is the last release tested and supported against containerd 1.x, and 1.36 requires 2.0+ on every node that joins it. That turns the migration from an optional “we’ll upgrade the runtime eventually” into a hard prerequisite for the next control-plane upgrade.

The problem isn’t the binary itself — swapping the containerd package via apt/yum takes seconds. The problem is that 2.0 doesn’t read the old config the way 1.7 did, part of the CRI plugin is structured differently, and GPU workloads on CDI can start behaving differently without a single line in the logs if you don’t check it beforehand.

A separate complication: the symptoms of a botched migration almost never show up on the node itself. containerd starts, systemctl status containerd is green, kubelet connects to the CRI socket and reports the node Ready — from the control plane’s point of view, everything looks fine. The real problem surfaces on the next pod the scheduler places on that node: ImagePullBackOff because a private registry didn’t get picked up, a pod missing its GPU device despite nvidia.com/gpu in its limits, or an unexpectedly different sandbox_image being pulled fresh for every new pod. That’s exactly why checking containerd config dump before a node takes production traffic is cheaper than debugging the incident afterward.

config.toml version 2 → version 3: an incompatible format

containerd is configured through /etc/containerd/config.toml, with a version field in the header. Version 1 hasn’t been read for a long time; version 2 was the default across the entire 1.x branch. containerd 2.0 requires version 3 — and that’s not a cosmetic number bump, it’s a different plugin path schema.

The main change: the monolithic CRI plugin io.containerd.grpc.v1.cri, which in 1.x handled both image pulls and container execution, is split in 2.0 into two independent plugins with their own IDs:

# containerd 1.7, config version 2
[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.k8s.io/pause:3.10"
  [plugins."io.containerd.grpc.v1.cri".containerd]
    default_runtime_name = "runc"
  [plugins."io.containerd.grpc.v1.cri".registry]
    config_path = "/etc/containerd/certs.d"

# containerd 2.0, config version 3
version = 3

[plugins."io.containerd.cri.v1.images"]
  [plugins."io.containerd.cri.v1.images".pinned_images]
    sandbox = "registry.k8s.io/pause:3.10"

[plugins."io.containerd.cri.v1.runtime"]
  [plugins."io.containerd.cri.v1.runtime".containerd]
    default_runtime_name = "runc"

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/etc/containerd/certs.d"

If you just swap the package and leave the old config.toml untouched, containerd 2.0 won’t fail to parse it — version 2 is still accepted — but plugins under the old io.containerd.grpc.v1.cri.* IDs silently won’t find their settings, and containerd will fall back to defaults instead of whatever you explicitly set: your sandbox_image, a custom registry.config_path, runtime limits. The node comes up, kubelet connects to the CRI socket — and the behavior is quietly different, until someone notices a private registry stopped resolving.

The right sequence isn’t editing config.toml by hand from memory — it’s generating a fresh default and merging your values into it:

containerd config default > /etc/containerd/config.toml.new
diff /etc/containerd/config.toml /etc/containerd/config.toml.new

The diff on a real node usually shows 30-40 changed lines even with minimal customization — that’s expected, not a reason to back out.

What to check with containerd config dump

containerd config default shows the default — what you’d get with no config file at all. To see what containerd actually applied after startup (defaults merged with your config.toml), you need a different command:

containerd config dump | grep -A5 'plugins."io.containerd.cri.v1'

Three things worth verifying line by line before switching a node into a production node pool:

• sandbox_image — if it doesn’t match what was set under 1.7, kubelet will pull a different pause image for every new pod scheduled on that node.
• registry.config_path — the path to per-registry TLS/auth config; a mismatch can leave the node unable to authenticate against a private registry, so pods on that one node start failing with ImagePullBackOff.
• containerd.runtimes — the list of registered container runtime classes (runc, kata, gvisor, and so on); a plugin that didn’t pick these up is a node that can’t run pods with the matching runtimeClassName.

Separately: some plugins that were deprecated-but-working in 1.7 (the legacy CRI v1alpha2 API, the old docker-schema1 image format, the shim adapter for runtime v1) are fully removed in 2.0, not just disabled by default. If you have a custom runtime handler or a CI runner that explicitly depends on one of these paths, containerd config dump simply won’t show it — and it’s cheaper to find that out before switching the node than after.

CDI and GPUs: different device-plugin behavior

There’s a separate trap for GPU nodes. In containerd 1.7, CDI (Container Device Interface) — the mechanism the NVIDIA device plugin uses to inject GPU devices into a container — was an opt-in feature, enabled explicitly with enable_cdi = true under the CRI plugin section. Without that flag, the NVIDIA device plugin fell back to the older injection mechanism via a runtime hook.

In containerd 2.0, CDI injection is part of the runtime by default, not an opt-in flag, and the path to CDI specs (cdi_spec_dirs) moved out of the CRI plugin section into the runtime section itself. The practical consequence: if a GPU node’s old config never set enable_cdi explicitly (i.e. the team relied on the 1.7 default — CDI off, legacy hook injection), after moving to 2.0 the behavior flips to CDI automatically, and an NVIDIA device plugin version built around the explicit flag can end up conflicting on device mount paths with what containerd itself now does.

A practical checklist for a GPU node pool: before switching, pin the NVIDIA device plugin version known to be compatible with CDI-by-default (check the device plugin’s own changelog for any mention of containerd 2.0), and run at least one smoke pod with nvidia.com/gpu: 1 in its limits on the canary node before switching the whole GPU pool.

containerd 1.7 vs containerd 2.0: what actually changed

What it takes to roll this out with zero downtime

You need a node pool you can update in parts (a managed node group in EKS/GKE/AKS, or your own set of nodes behind a single kubectl cordon cycle), and at least one node you won’t mind breaking as a canary. The strategy isn’t “upgrade containerd on every node at once” — it’s splitting the node pool by containerd version and rolling in batches, starting with a canary group.

#!/usr/bin/env bash
set -euo pipefail

NODE="$1"

echo "== cordon & drain: $NODE =="
kubectl cordon "$NODE"
kubectl drain "$NODE" --ignore-daemonsets --delete-emptydir-data --timeout=180s

echo "== backup current config (needed for rollback) =="
ssh "$NODE" "sudo cp /etc/containerd/config.toml /etc/containerd/config.toml.bak"

echo "== replace containerd package =="
ssh "$NODE" "sudo apt-get install -y --only-upgrade containerd.io"

echo "== regenerate config: v2 -> v3 =="
ssh "$NODE" "sudo containerd config default | sudo tee /etc/containerd/config.toml.new"
ssh "$NODE" "sudo systemctl stop containerd"
ssh "$NODE" "sudo mv /etc/containerd/config.toml.new /etc/containerd/config.toml"
ssh "$NODE" "sudo systemctl start containerd"

echo "== verify CRI socket reachable by kubelet =="
ssh "$NODE" "sudo crictl --runtime-endpoint unix:///run/containerd/containerd.sock info | grep -q '\"status\": \"true\"' || (echo 'CRI socket not ready' && exit 1)"

echo "== uncordon =="
kubectl uncordon "$NODE"
echo "== done: $NODE =="

Rollout order across the node pool:

1. Canary node group — 1-2 nodes, isolated from production traffic with a nodeSelector/taint, updated first and watched for at least a day.
2. Smoke test on canary — a regular workload pod (no GPU) plus, on GPU nodes, a separate pod with nvidia.com/gpu in its limits — both should start and pass readiness with no manual intervention.
3. Splitting the node pool by containerd version — until the rollout finishes, part of the pool runs 1.7 and part runs 2.0 at the same time; this is compatible with the current control-plane version (verified up through 1.35), but shouldn’t last longer than the rollout itself needs.
4. Batch waves — 10-20% of nodes at a time, with a pause between waves to check metrics (ImagePullBackOff, pod restarts, GPU pod alerts) rather than rolling the whole pool at once.

Rolling back if the canary fails

Rolling out in waves assumes from the start that rollback is a routine step of the plan, not an exceptional situation. If a canary node shows ImagePullBackOff after moving to 2.0, a GPU pod doesn’t see its device, or containerd config dump doesn’t match expectations — the node gets rolled back the same way it was updated, not patched in place under incident pressure:

kubectl cordon "$NODE"
kubectl drain "$NODE" --ignore-daemonsets --delete-emptydir-data --timeout=180s
ssh "$NODE" "sudo apt-get install -y --allow-downgrades containerd.io=1.7.*"
ssh "$NODE" "sudo systemctl stop containerd"
ssh "$NODE" "sudo cp /etc/containerd/config.toml.bak /etc/containerd/config.toml"
ssh "$NODE" "sudo systemctl start containerd"
kubectl uncordon "$NODE"

The one condition this rollback depends on is a backup of the version-2 config.toml, taken before running containerd config default on the node for the first time (cp /etc/containerd/config.toml /etc/containerd/config.toml.bak as the very first line of the rollout script, before the package upgrade). That’s also why it’s worth keeping the old node pool mixed with the new one for a while: as long as part of the fleet is still running the proven 1.7 config, the team has time to root-cause a canary failure without the pressure of “the whole cluster already moved, there’s nowhere to roll back to.”

How to verify a node is ready

After uncordon, the minimal check is that kubelet actually sees the node through the new CRI socket and the containerd version in Node.status.nodeInfo has updated:

kubectl get node "$NODE" -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}'
# expect containerd://2.0.x, not containerd://1.7.x

Next, run a test pod pinned to that exact node (nodeName in the manifest) and confirm it starts and logs what’s expected:

kubectl run canary-check --image=busybox --overrides='{"spec":{"nodeName":"'"$NODE"'"}}' \
  --restart=Never -- sh -c 'echo ok'
kubectl logs canary-check

On a GPU node, separately run a pod with nvidia.com/gpu: 1 and confirm nvidia-smi inside the container actually sees the device: the CDI behavior gap between 1.7 and 2.0 usually shows up here, not on regular workloads.

Bottom line

The move to containerd 2.0 is disguised as a minor detail of the Kubernetes 1.36 upgrade, but it’s really a separate rollout with its own config format, a split CRI plugin, and different default CDI behavior for GPUs. The difference between “works correctly, invisibly” and “works incorrectly, invisibly” isn’t running a new binary — it’s checking containerd config dump line by line and testing on a canary node before rolling it out to the whole pool. It’s cheaper to spend a day on that checklist now than to debug a private-registry or GPU incident in production after the control plane has already moved to 1.36 and there’s officially no path back to 1.x.