Security update: Encrypted overlay networks in Moby and Mirantis Container Runtime

During routine work on the Moby project, Mirantis discovered a set of security issues affecting all versions of Docker Engine which include the Swarm mode feature. This includes Mirantis Container Runtime and other projects downstream of Moby.

While most users are not directly impacted, these issues could impact some Swarm mode users’ overlay network traffic that was meant to be encrypted. 

We have created the following CVEs to track and identify these vulnerabilities:

• CVE-2023-28841: Encrypted overlay network traffic may be unencrypted

• CVE-2023-28840: Encrypted overlay network may be unauthenticated

• CVE-2023-28842: Encrypted overlay network with a single endpoint is unauthenticated

These CVEs have been patched via coordinated security updates released April 4, 2023.

CVE-2023-28840 and CVE-2023-28841 may affect users with encrypted Swarm overlay networks on Red Hat Enterprise Linux 8 or Red Hat Enterprise Linux 9 (or derivatives such as CentOS, Rocky, and Alma).

CVE-2023-28840 can additionally affect any platform depending on how the iptables rules are laid out. All firewalld configurations which allow Swarm overlay networks to function are known to be affected.

CVE-2023-28842 affects users using encrypted overlay networks with only a single node (machine) attached, which we expect to be rare in production deployments.

Patches for the three CVEs are available in Mirantis Container Runtime 20.10.16 and 23.0.3. All users should update, and should strongly prefer the patches over any temporary mitigations.

To determine whether Swarm overlay networks are in use, you can use the following command (which requires jq):

docker network inspect $(docker network ls -q) | jq -r '.[] | select(.Scope=="swarm" and .Driver=="overlay") | select(.Containers // {} | with_entries(select(.key | test("-sbox$") | not)) | length > 0) | .Name'

This will print the name of any overlay network in active use on the local node. This will include the default ingress network (and any user-defined ingress networks), but only if they have a container attached (as the -sbox suffixed ‘containers’ are in fact virtual).

To assess an entire cluster, the command should be run across every node on the cluster, and the results aggregated. While Swarm-scoped networks are replicated across the entire cluster, the ‘actively used’ status is local to each node, and should be queried individually.

If you know that you are using overlay networks, you should ensure the VXLAN port (UDP 4789 by default, and called “Data Path Port” in docker info) is not accessible to untrusted network traffic. If encrypted overlay networks are in use over an untrusted network, ensure that IPSec ESP (protocol 50) is allowed through the boundary firewall.

If a temporary mitigation is required, users of Red Hat Enterprise Linux 8 and 9 (and derivatives, such as CentOS, Rocky, and Alma) can take the following alternate steps:

• Check if the xt_u32 kernel module is available (e.g. modprobe xt_u32).

• If it is unavailable:

  • On Enterprise Linux 8, install kernel-modules-extra and restart dockerd.

  • On Enterprise Linux 9, discontinue use of overlay networks where Swarm-level encryption is necessary for security.

• Update to MCR 20.10.16 or 23.0.3 as soon as practical.

The timeline below details the steps Mirantis has taken to fix and disclose these vulnerabilities:

• Issues first identified and confirmed March 8th, 2023

• Upstream partners alerted March 9th, 2023

• Patch is code-complete and reviewed on March 22nd, 2023

• Patch completes internal validations on March 23rd, 2023

• Timeline and content for disclosure finalized with partners on March 30th, 2023

• Disclosed and patched today, April 4th, 2023

If you have questions about these issues, we encourage you to contact us.