k0s 1.28 released
Team k0s proudly presents: k0s 1.28 with Kubernetes 1.28 built in.
Of course, the biggest thing in this brand-new version 1.28 of k0s is support for1.28 Kubernetes, a.k.a Planternetes.
But as per usual, the team has not been just working on updating the embedded Kubernetes components. We’ve also packed in some noteworthy new features and fixes. In this post, we dive into a few of the more exciting new features and enhancements in version 1.28 of k0s.
Enhanced k0s autopilot
k0s autopilot, bundled initially into our earlier 1.24 release, lets operators easily keep clusters up-to-date by applying an update
Planobject. The update
Plancontains all the information needed to perform an automated update, orchestrated by the cluster itself.
Building on top of this, 1.28 now supports scheduled updates that follow a specific update channel. So how does this work?
# The folowing fields configures updates to happen only on Tue or Wed at 13:00-15:00
planSpec: # This defines the plan to be created IF there are updates available
The general idea is that each minor version will have its own dedicated update channel. That lets you very easily keep up-to-date with frequent patch updates — updates that both upstream Kubernetes and k0s provide. Once you’re ready to hop on a newer minor release, just update the
UpdateConfigto follow a different channel. For those who like living on the edge, we’re also providing channels for beta releases and a
To keep things easy, the update server “protocol” is very simple. Each channel only has a simple
index.yamlthat contains info about the latest version thatchannel provides. This enables, for example, building your own update servers for airgapped environments. A reference implementation can be found athttps://github.com/k0sproject/update.
ipset and kernel version incompatibility
As some of you might remember, about a year ago, team k0suncoveredan
iptablesversion incompatibility bug. Then a few weeks ago, some of our integration tests started to fail unexpectedly when running on the newer 6.2 Linux kernel. While we started to debug and triage those tests, we pretty soon realized that we’d again stumbled over something very similar to the iptables saga.
In the Linux kernel, there’s a framework called
ipsetwhich manages sets of addresses within the kernel. Those sets can be used for lightning fast matching on rules. Naturally, in the world of Kubernetes, there are a few things that interface with
ipset, like kube-proxy and some CNI providers.
As with Kubernetes, and containers in general, the main philosophy is that each container comes with all the tools and libraries they need bundled within the container image. While that works pretty well when considering run-of-the-mill applications, it poses some challenges for system level things. In this particular case, kube-proxy bundles one version within the image and a CNI provider (Calico in our case) might bundle some other version. Why is this a problem? Well, in most cases it is not and should not be as the Linux kernel has very strong backwards compatibility guarantees. However, on kernels >= 6.2, there’s a new bitmask parameter for
ipset, which newer ipset versions support and use, but which older versions don’t know how cope with. This leads to a failure, with the error message:
Kernel and userspace incompatible: settype hash:ip with revision 6 not supported by userspace.
The kube-proxy image was shipping
ipsetv7.17, whereas the calico-node image was shipping
ipsetv7.15. That fails on newer kernels, if kube-proxy uses IPVS. Now rules set up with v7.17 — rules that utilize the new bitmask parameter — fail on loading by the older v7.15 version. 💥
The fix in itself is easy for k0s. Wejust(okay, harder than it looks) need to ensure that ALL system components bundle the same compatible version of ipset. But this raises the question: how can system admins ensure all low-level kernel and user space tooling IS actually compatible between all system components? The longer (and painful)answer is: by testing EVERY single update — k0s, Kubernetes upstream, Linux kernel, or other components should be rigorously tested BEFORE hitting a production environment.
The short (and painless) answer is by using a well-tested Kubernetes distro like k0s. :) We believe sysadmins should not need to worry about stuff like this.
All this brings up one other major thing the team has been working on in the scope of the 1.28 release.
Extended OS testing matrix
As you know, one of the promises of k0s is that it runs on ANY Linux setup. As k0s bundles in ALL the needed dependencies, there’s really nothing a cluster admin needs to set up as a prerequisite before running k0s. But how can we really be sure that is the case? With rigorous testing.
We’ve now extended our OS testing matrix to cover the following Linux distros and versions:
Centos: 7, 8 and 9
Debian: 10, 11 and 12
Fedora CoreOS: 38
Oracle: 7.9, 8.7, 9.1
RHEL: 7, 8 and 9
Rocky: 8 and 9
Ubuntu: 20.04, 22.04 and 23.04
Not only do we test different OS versions, but we also test different configurations of k0s. The matrix “dimensions” also cover different CNI providers and different kube-proxy modes (iptables vs. IPVS). So the total number of permutations we test is going up fast: right now, we test 84. We’re also planning to add tests for arm64, which will double the amount of permutations permutations.
For each OS and configuration combination we run the full Kubernetes conformance test suite. That gives us pretty good guarantees that everything is working on that combination as expected.
As you see, we take our promise of “runs anywhere” very seriously.
Starting with the 1.28 release, we now build a full signed SBOM of k0s build artifacts. This has several benefits to the users:
Enhanced Security: By providing a full signed SBOM, k0s enables users to verify the authenticity of their Kubernetes distribution, reducing the risk of using tampered or malicious components.
Transparency: Users gain insight into the components and dependencies included in the k0s distribution, improving transparency and aiding in vulnerability management.
Compliance: SBOMs are valuable for organizations striving to meet compliance requirements, such as those outlined by regulatory bodies or internal policies.
Going forward we’re also planning to sign ALL the build artifacts to further enhance the security of the build pipeline.
In case you’ve missed it, we launched a k0suser surveylast week. By answering this survey, you’ll help us prioritize features that mean the most to YOU, and in general, you’ll help us to shape the future of k0s.