Linux kernel headers are the unstable, constantly-changing, internal API of the kernel. This includes internal kernel structures (for example,
[…]task_struct) as well as helper macros and functions. Unlike the UAPI headers used to build userspace programs that are stable and backward-compatible, the internal kernel headers can change at any time and any release. While this allows the kernel unlimited flexibility to evolve and change, it presents some difficulties for code that needs to be loaded into the kernel at runtime and executed in kernel context.My solution to the problem is to embed the kernel headers within the kernel image itself and make it available through the sysfs virtual filesystem (usually mounted at /sys) as a compressed archive file (/sys/kernel/kheaders.tar.xz). This archive can be uncompressed as needed to a temporary directory. This simple change guarantees that the headers are always shipped with the running kernel.
This change has been added to version 5.2 of the Linux kernel.
I’m a bit baffled and probably missing something important, but how’s shipping a kernel module that contains a compressed archive any better than shipping the compressed archive itself?
Oibaf,
It’s kind of a minor non-issue for most users, but when you have a need to compile against the kernel headers for the current kernel, retrieving those headers from the kernel itself could prove more reliable/accurate in some circumstances than downloading them via a side channel especially if using a 3rd parties kernel.
For me, given that I always compile kernel modules along with the kernel itself, I won’t have a use for this feature.
You still have to modprobe the correct kernel module, which has to be shipped with the distro you’re using. If the module can be shipped, so can the archive itself.
Oibaf,
The use case would be someone building their own kernel modules. Think something like nvidia building the kernel driver for your machine. Obviously you can download the headers separately, but if they were included in the kernel you could eliminate that step and increase confidence that they are in fact the correct headers/structures.
Hypothetically if the kernel was modified by a 3rd party and didn’t take care to distinguish the uname, the official headers could actually be wrong for the running kernel. It’s not something most people have to worry about.
@Alfman
But they’re not included in the kernel as in being part of the kernel image, they’re included in the set of modules that have to be installed alongside the kernel image. If one fails to install that specific module, no headers will be found.
I still don’t see how’s that any different than installing the kernel headers alongside the kernel image.
I must be missing something for sure, I don’t think I know better than Linus about this. 😀
Oibaf,
The article says you have the choice to build in kernel or as a module.
If it’s built as a module then you are correct that the module would have to be loaded dynamically in order for it to work. Maybe it is a clumsy solution, but that’s the solution given rather than distributing the headers as an archive.
I don’t know that linus is the one who personally approved this commit, but in any case he’s not a god. He can and does make mistakes and linux has plenty of evidence of that through the years. We could probably do a better job in many areas, but at the end of the day his kernel will always be more popular, haha.
The developer of this patch explained that this is not useful for compiling kernel modules, but for compiling eBPF tracing programs for the Linux kernel (“tools” in eBPF parlance) , and that is its intended usage. eBPF is Linux response to Solaris’ DTrace.
Although at first he also wanted to be able to compile kernel modules, it appears that it needed the same compiler version that was used for compiling the kernel and also had to include some binaries in the tgz, so that made it impractical. Such limitations doesn’t apply to its usage for eBPF tools.
Antartica_,
I saw that too, he wanted to include the entire build chain, compiler and all. But in the context of a typical distro this is not only complete overkill, but the conclusion is factually wrong (in part because your using the build tools provided by the distro, which is already compatible with the kernel and modules provided by the distro). We regularly build modules from kernel headers with the distro’s own tool chains. It’s worked this way for decades, and bundling the kernel headers with the kernel shouldn’t suddenly break that.
I think the motivation for much of this work is to eliminate the dependency on the distro’s own tooling and replacing it with a pure linux solution (and by linux here I mean core linux kernel support excluding OS resources). But that’s what’s getting them into trouble. I can see why shifting responsibilities away from distros and into linux is controversial, and there may even be a chance this opens the door down the line to using this approach to sort of usurp userspace functionality by bundling the kernel with blobs that run in userspace. As Oibaf argues, the technical benefit is kind of flimsy because from a distro’s point of view they can already do what this patch does.