You may recall I mentioned in my previous post that I'm the upstream for Adélie Linux' easy-kernel. You may also recall that I mentioned that it was time to start cutting a 6.6-series release.
Over the Christmas period, I've set myself to trying to get a 6.6-series kernel out. The reasons for this are threefold:
- The last kernel of the year is usually chosen for longterm support - they're a good target to patch against as they usually don't undergo massive structural changes.
- Christmas stresses me out in a way that hacking on the second-lowest-level software running on your Linux system simply doesn't.
- I'm going to need to rip off the bandaid about the loss of kernel-space data deduplication at some point, and now is the time. 6.6 brings many new features that make it worth it.
While I do the work on preparing the patchset, I'm going to be writing this document as a sort of a living testament to some design decisions - things I miss, things I'm looking forward to, complaints about the Linux ecosystem, and in the hopes of providing a starting point if you want to one day play around with modifications to the kernel.
Working remotely
It's the way of the future.
I couldn't find the power brick for my ThinkPad X1 Carbon Gen 4 before leaving for Christmas, so the next-best option was my MacBook Pro running Ventura. I don't have one of the ThunderBolt-to-DisplayPort gadgets, so its 1280x800 display was going to have to do.
The MacBook has a quirk, however. It really doesn't like connecting to ssh on a port that isn't the ssh port (port 22) - and I really don't like opening standard service ports to the outside world. Some trial and error showed that two Ubuntu boxes running on this network were perfectly capable of such shenanigans, so I'm now logged into my home system through a System76 meerkat 3 NUC, loaned for the low low cost of upgrading it from Ubuntu 18.04 to Ubuntu 22.04. The Meerkat 3 has more compute power than the ThinkPad that I do all my Adélie work on, and ... I'm honestly now contemplating seeing if there's a way to get one to replace that ThinkPad.
In any case, the work on kernel-mc is done on the same machine it always is, and every reasonable precaution has been taken to ensure that a bad actor, or a bored engineer who's had a little too much Christmas cheer, can't release something using my good name on my own server.
Linux 6.6
What does it get us?
My previous release series is based on the Linux kernel 5.15. This version has support until sometime in 2026, at least. I've looked at whether I can also release a 3.4 or some such, to support 386 machines. The answer is: maybe.
When we look at a new kernel version, we have to ask what we get in exchange for abandoning the familiar and proven.
In the case of Linux 6.6, the primary thing we get is a new longterm, at the start of its support cycle. We also get a few other interesting things, such as:
- Support for drivers written in Rust, a memory-safe language that is seeing some uptake among the youth.
- Apple Silicon support (kind of) - what upstream doesn't provide is covered by the Asahi Linux project, and I believe that my patchset is not fundamentally incompatible with their modifications.
- A number of bug fixes and performance improvements on ARM, AArch64, PowerPC, MIPS, and RISC-V.
- Support for new devices, and improved handling of newer CPUs.
The main thing we lose is support for data deduplication in kernel space, presented in this USENIX paper, as the upstream doesn't support newer than kernel 5.18. This means that we get to play with the newer uksmd technology by Oleksandr Natalenko - but I'm still not convinced that having your memory deduplication code be killable as a user process, especially under memory pressure, is the best idea.
Choosing Patches
Who do we trust with something as important as the Kernel?
There is no point in hiding the "secret sauce" behind kernel-mc. We're all in this together, it's all open source, and you could just ... look in the tarball, or read the README file anyway.
I'm not a kernel hacker in the same way as some people are. As far as programming goes, I'd prefer to be doing as little of it as possible in kernel space. The Good Lord gave us Python and Visual BASIC for a reason, after all. And Satan, for his part, decided to visit on us the Bourne Shell.
I understand enough about kernel code to do rudimentary audits of other people's work, and to tweak some values that are set in stone at compile time; a particular example is the vm.swappiness value which Dr. Con Kolivas addresses in a simple patch customarily called swap-sucks . Dr. Kolivas is right - swap does suck! With modern machines having their fixed disks provided by SSDs which can be damaged by intense paging of memory, he's only become more right. But in the days where swap was backed by rotational storage with poor random access characteristics, overly aggressive paging of memory into swap space had a very real performance benefit.
Dr. Kolivas no longer maintains his excellent kernel patchset; somewhere along the way he lost his passion for kernel work (working as a physician during a global pandemic will do that to you), and he decided that instead of churning out sub-par code, he should allow others to take the place he once held. Ironically, the EEVDF in Linux 6.6 is based on similar design principles to his Brain Fuck and Multiple Queue Skiplist Schedulers. Sometimes, it takes upstream a little time to catch up.
So, if I'm not a proper kernel hacker, and I'm releasing a proper kernel patchset, the code has to come from somewhere.
The first thing I have to say is that the genpatches project for gentoo distribution kernels is a godsend. As far as I'm concerned, every single patch in genpatches-base should be mainlined, and in most cases, they'd prefer to do that than to just maintain a fork of the kernel. Basically all of Genpatches is used in kernel-mc - and ... knowing how many users run the gentoo-sources package, I'm comfortable with that. Given enough eyeballs, all bugs, after all, are shallow.
The second thing to say is that the genpatches project is incomplete, in the same way that Wikipedia is. There's always new bugs to find and squash, and if there's a bug that hasn't made it in to genpatches yet, others may have fixed it. Laurent Bercot provides a small patch - a one-character change - that allows the kernel to build on x86 where gcc may be overenthusiastic about warning generation. This is the essence of a good patch - the smallest possible change that fixes a problem satisfactorily and reliably. Likewise an existing patch in genpatches provides an option - you can switch it on, and any firmware the kernel loads will be explicitly printed to console. I don't consider that option useful, so I have made the patch smaller, to simply print any firmware load regardless. This makes kernel debugging simpler, believe it or not.
My kernel patchset used to share some commonality with the post-factum kernel, by Oleksandr Natalenko, and the -ck kernel, by Dr. Con Kolivas. As further issues have arisen that need attention, and some kernel forks have taken on such divergence from mainline as to make the first step in troubleshooting "use a vanilla Linux kernel", kernel-mc has taken on its own distinct identity and mission.
The 6.6 series kernel-mc has patches from Con Kolivas, Oleksandr Natalenko, and Mike Pagano, among others, as well as contributions from Intel, AMD, Meta, Red Hat Linux, and IBM. Each of these people or corporations has seen an issue that they can remedy and done so - ejected it into the aether in the hopes that it will prove useful to others. The changes have been audited by me, because if it has my name on it I'm making damn sure it's not shipping brand-new exploits.
It needs a codename.
Hail to the King, baby.
One of the things that is often overlooked in the Linux Kernel is a small addition to the Makefile which, as far as I can tell, has no impact whatsoever on the object vmlinux or the functioning of the kernel, and is thus only of interest to bored nerds.
---------- on concorde ~/dev/adelie-easy-kernel $ tar -xf linux-6.6.tar.xz
---------- on concorde ~/dev/adelie-easy-kernel $ grep NAME linux-6.6/Makefile
NAME = Hurr durr I'ma ninja sloth
Patches to the linux-4.14 series included the KAISER pagetable isolation patches - a component of hardening the kernel against the heavily-publicised Meltdown vulnerabilities. All -mc series 4.14 kernels shipped with these backported features, and thus it seemed the ideal name was KAISER und König - a joke that perhaps few would get - but one that could hide in a Makefile innocently for many years. The original codename for the Linux 4.14 release was Petit Gorille - which I presume was some other developer's private joke.
Linux 6.6 has the codename Hurr durr I'ma ninja sloth - which is honestly just uninspiring, but whoever put it there obviously earned the ability to. With the release of linux-6.6.6, on which the first 6.6-mc kernel is built, it seemed an obvious time for some mayhem. The -mc series 6.6 kernels are thus named Hard Rock Hallelujah - that one's hyperlinked for a reason, in case you don't keep up with affairs in the old country.
---------- on concorde ~/dev/adelie-easy-kernel $ head -n6 linux-6.6-mc1/Makefile | tail -n5
VERSION = 6
PATCHLEVEL = 6
SUBLEVEL = 6
EXTRAVERSION = -mc1
NAME = Hard Rock Hallelujah
While I wouldn't typically indulge in such churlish merriment, the opportunity comes but once in a lifetime for a major software project to produce such a perfect joke; and as 6.6.6 notably reverted some bugs introduced in 6.6.5, including one that led to potential filesystem corruption and another that could cause network connectivity issues, I wasn't going to waste it.
Those of you who are of a particularly superstitious bent can wait this one out, if you're uncomfortable with the numerology - but thankfully, as any Bible historian can tell you - this one isn't actually the number of the beast anywhere but in the public perception.
Hard Rock Hallelujah
In conclusion...
This entry is a small guide through the thoughts I have preparing a kernel patchset that is to be used as a distribution kernel. Notwithstanding its affiliation with Adélie Linux, I intend kernel-mc to remain a kernel that does not sacrifice portability and correctness, that pursues the few extra fixes that haven't landed in mainline yet, that provides a few extra tweaks to give a little extra oomph to your computer. Any computer.
You can run kernel-mc as Linux/m68k. I'm not joking. If you want to build it to run on a Mac IIvx or something, I want to see that! If you want to run Gentoo on a Raspberry Pi and use my kernel, I want to see that. Show me your neofetch output! If you are bringing up a brand new board, I want kernel-mc to be a suitable base for that, and I want you to submit your code to the mainline guys as soon as you can. We're all stronger when we work together.
I finished up work on this patchset on Christmas Day, in my timezone, as it turned out - the write-up took a little longer to polish. But in the spirit of gemutlichkeit and collaboration and unlimited figgy pudding;
Merry Christmas - and to all, a good night!
Kernel 6.6-mc1 is available at this link - it must be applied to a linux-6.6.tar.xz from kernel.org to function.