I’m sure we can all have a calm, rational discussion about this, so here it goes: zlib-rs, the Rust re-implementation of the zlib library, is now faster than its C counterparts in both decompression and compression.
We’ve released version 0.4.2 of zlib-rs, featuring a number of substantial performance improvements. We are now (to our knowledge) the fastest api-compatible zlib implementation for decompression, and beat the competition in the most important compression cases too.
↫ Folkert de Vries
As someone who isn’t a programmer, looking at all the controversies and fallout around anything related to Rust is both fascinating and worrying. Fascinating because Rust clearly brings a whole slew of improvements over established and older languages, and worrying because the backlash from the establishment has been wildly irrational and bordering on the childish, complete with tamper tantrums and the taking of balls and going home. It shouldn’t surprise me that people get attached to programming languages the same way people get attached to operating systems, but surprisingly, it still does.
If Rust not only provides certain valuable benefits like memory safety, but can also be used to create implementations that are faster than those created in, say, C, it’s really only going to be a matter of time before it simply becomes an untenable position to block Rust from, say, the Linux kernel. Progress has a tendency to find a way, especially the more substantial the benefits get, and as studies show, even only writing new code in memory-safe languages provides substantial benefits. In other words, more and more projects will simply switch over to Rust for new code where it makes sense, whether Rust haters want it or not.
There will be enough non-Rust code to write and maintain, though, so I don’t think people will be out of a job any time soon because they refuse to learn Rust, but to me as an outsider, the Rust hate seems to grow more and more irrational by the day.
If you have a code base as large as Linux kernel is then it’s nearly an impossible goal to rewrite it in another language. Now as for glueing two different languages, toolchains, cultures and i guess generations into such project. IMHO that is a rather bad idea. There are already consequences of that and that is after years of fiddling with Rust inside Linux, result for now mostly being useless by itself in terms of meaningful functionality, one Linux maintainer argued he won’t do Rust in any form when confronted with Rust, after that another Rust proponent went haywire on social media and some apparently resigned from such efforts altogether. Linus called out the behaviour of such individuals and at the same time took a stance he will force Rust merge request and basically saying Linux maintainers have no say in this. That resulted in original Linux maintainer to stop maintaining that portion of Linux kernel, now that part is down to one maintainer. All this happened due to Apple hardware driver that although nice in theory in reality has no real impact on anything beyond some niche group of Linux users. All in all a lot of ego fuelled witch hunt alike toxicity and rage quitting involved. Then there is another prominent Linux maintainer that argues although he has little experience with Rust he is in support of Rust due to C having a lot of memory related safety issues. I mean OK but if prominent Linux maintainer has no real experience with Rust, how is it then expected for all of this to work? You have this Frankenstein kernel on where left side doesn’t know what the right side is doing? And how exactly is that a good thing? On top of all of this you have social media pressure on where the sentiment is C should die and Rust is the future and some commentators out there just don’t understand on why Linux wasn’t already rewritten in Rust, beyond that wasting no effort whatsoever in making it happen. So all in all unless some big IT company commits and does something useful, like some meaningful device driver, then it’s just not worth it, on contrary it will only hurt Linux just like it already did.
A lot to unpack above.
Let’s just be honest about what Linus has said though:
– No maintainer has to accept Rust code into the code that they maintain unless they want to
– No maintainer has the power to veto code OUTSIDE of the code that they maintain
– If you do not want to work with Rust code, your opinion about Rust code does not count
– If a maintainer is up for it, mixing C and Rust code within a subsystem is fine
So the biggest correction to the above is that Linus did NOT tell a maintainer that he had to take on Rust code in the system that they maintained. He told them that they did not have the power to veto who called into their code and how that code is used and that they NEVER had that power even if all the code was in C.
Actual events were that a Rust dev wrote Rust code to interface to the dma code (C code) in the Linux kernel. It was not part of the dma system. It called into it. The dev was committed to maintaining it. The dma maintainer vetoed it. An unrelated Rust dev tried to take the fight to social media, was smacked down by Linux, and quit in protest (head of Asahi Linux). Linus told the dma maintainer that they had overstepped their authority and merged the Rust code that had been vetoed. The C code maintainer quit. Both of those that left have been replaced.
A huge fraction of “the Linux kernel” is drivers. That is where Rust is being used today. For example, the Apple Silicon GPU driver has been written in Rust. To make using Rust with Linux easier, a bunch of wrapper / interface code is being written. Where that code touches C, there has been conflict.
Writing a device driver in Rust does not “rewrite Linux in Rust” in any way. It is Rust code working with C code. All the core code is still C.
Today, “new” code is being written in Rust. We are getting close to where some of that “new” code may be a new design for something that was previously done in C. At some point, Rust will replace C code.
All that said, “core” Linux code is unlikely to be Rust for some time. For one thing, Rust does not target all the platforms it needs to yet. The GCC/Rust project may be the solution for this but it is far from ready.
The “prominent Linux maintainer” mentioned is Greg HK, widely regarded as the #2 guy in Linux and maintainer of the stable kernels.
There seems to be a lot of dishonesty and exaggeration around Rust in the kernel and the arguments are often more social or political than technical. But I will let readers decide where the “ego fueled” “toxicity” is to be found. What I have seen is the Rust devs asking to be judged by more technical criteria. Linux and GHK seem to be saying the same.
Exactly, Linus told the maintainer he has no power over deciding if Rust code will be merged or not. It will hence eventually just end up being in there. True that currently we are talking only about Apple Silicon GPU driver, a hobby device driver to serve as a proof of concept. As for one maintainer of some area speaking one language and another one a different one, for that not being a real issue, on where AFAIK neither Linus or Greg Kroah-Hartman are fluent with Rust. In my opinion that is living on a pipe dream, on how that won’t cause any real issues down the road. So if they plan to force Rust on Linux then at minimum IMHO Linux foundation would need to sponsor, for any interested maintainer to get some sort of education and certification, for being fluent with both C and Rust. In real life that means around three to five years of education if you had some previous programming experiences.
“Exactly, Linus told the maintainer he has no power over deciding if Rust code will be merged or not. It will hence eventually just end up being in there.”
That ship sailed when Torvalds decided to let Rust4Linux into the kernel. That was back in 2020. Rust has been in the kernel for years now. The time to throw a tantrum over mixed code bases by Hellwig was in 2020, not now. (Which would probably have ended the same way.)
I think Torvalds is in the right here. The “controversial” rust code is a common library written for Rust drivers to use to send their data through the DMA system. Currently any driver in Rust has to write that code themselves to be able to use DMA. It makes sense to have one common component for Rust code to send data over DMA. To be crystal clear, this Rust DMA library exists outside of the C DMA code and translates between Rust and the C DMA subsystem. It will not be inside the C DMA code. (Torvalds also has no plans to force C maintainers to accept Rust code inside their C code bases.)
Hellwig’s NACK was based on not wanting to have any Rust in Linux. Which is too little, too late. He thought he could hamper the effort by forbidding Rust code to use a common library to talk to his C DMA subsystem. His “strongest” argument, that he didn’t want to be restricted by outside code preventing him from breaking changes in the DMA subsystem, was bunk. C code is allowed to break Rust code. The Rust folk accept that and have declared that they are responsible for repairing the breakage on the Rust side.
If C DMA changes the internal structures, it makes more sense to have one unified DMA library in Rust. That way the breakage only needs to be fixed once and not in every driver that has written their own DMA implementation. It makes maintenance less of a burden.
It’s IMHO rather naive to say C code is allowed to break Rust code in Linux and after for that to become the problem of somebody else to fix it. This is not how successful big projects do it, for being a capable maintainer you are responsible to fix it and can’t just leave it dangling. Moreover this is not new argument, as this is an ongoing debate from the get go and it was never properly addressed. If you would really do it like that, then once we scale things up, beyond some hobby proof of concept device driver, inner frustration and clash is bound to happen. The maintainer that stepped down hence had a valid argument and Linus will need to deal with that one way or another. Saying that this problem will get sorted out by ignoring it, it won’t. Once and if we scale things up it will only get worse.
Geck,
I’m laughing so hard right now because you have a history of defending unstable ABIs, and suddenly now it’s a problem when it comes to rust. C code has been allowed to break other C code in other parts of the kernel, Rust doesn’t change any of that. Yes C code can break Rust code, but this is nothing new and breaking changes have been the status quo in linux long before Rust came along.
Ironically I agree this is one of the notorious problems with unstable ABIs. However it’s extremely hypocritical of you to only admit this in the context of Rust code. Rust devs didn’t create this problem, they’re playing by the same rules and you are criticizing them for it. It’s a hypocritical argument.
Thom Holwereda,
It should be no surprise that I agree with you about the importance of memory safety. But I’m skeptical that progress has a tendency to “find a way”. The fact that we can do better doesn’t imply that we actually will. Incumbents often benefit from a power imbalance favoring them. This definitely seems to be the case here. While memory unsafe languages have had a troubled track record for decades, this often gets written off (if not denied) and there are many who consider unsafe languages “good enough” despite their longstanding faults..
I feel that technology leaders should embrace memory safety, but without their enthusiastic buy in they can intentionally or unintentionally make it an uphill battle for progress. In the worst case they can deliberately sabotage goodwill efforts to replace unsafe languages with safe ones. Reasons for this can be varied and may be personal. Regardless of the reasoning though it’s not an even playing field. Technology with the most merit doesn’t always win especially when the incumbent is perceived to be good enough.
I’d say that “progress finds a way” does happen… but it’s prone to happening in forms like EGCS, where you essentially had the GCC ecosystem migrating to something analogous to the out-of-tree Linux downstream Android used to be built around. (Until Stallman admitted he’d been too tyrannical in his management of GCC, opened up, and blessed EGCS as what would go on to become GCC 2.95.)
It’s just a question of how much water has to pile up behind the dam before a new route is found.
ssokolow (Hey, OSNews, U2F/WebAuthn is broken on Firefox!),
In your analogy, it seems a bit fortuitous that Stallman changed his mind. Sometimes tech leaders can be too stubborn to change.
You are right it could happen, but I think it depends on whether enough important people start to call out unsafe legacy languages as not good enough, which doesn’t seem to be happening yet. I concede a shift in attitude could be easier to recognize in hindsight because critical mass doesn’t suddenly happen overnight.
When that happens, you get the OpenOffice/LibreOffice split. In that case, Oracle had a contract with IBM riding on their continued ability to offer an alternative license to build IBM Lotus Symphony on top of. (That’s why Oracle gave it to the Apache Foundation as a last-ditch thing to try to keep the free labour flowing in.)
Well, Securing tomorrow’s software: the need for memory safety standards did just get posted to the Google Security Blog. It’s basically a call to action standing in support of the recent ACM article, It Is Time to Standardize Principles and Practices for Software Memory Safety.
Combine that with the calls for more memory safety from various government agencies and it looks like we’re at risk of an “Oh, you’re stubborn… not more stubborn than new procurement regulations” situation. (Which wouldn’t suprise me. Things like systemd and Flatpak are basically people saying “You ignored our use-cases for 20 years. We’re tired of waiting. Don’t like it, compete!” Greybeards seem to have a tendency to dig in their heels until they become irrelevant. I’m reminded of that saying about how science advances one funeral at a time.)
I’d say the latest controversy helped Linux, not hurt it.
As LeFantome wrote, Linus Torvalds’ final word on the matter was a clarification of what maintainers could and couldn’t do regarding Rust code. This was overdue, but is a good thing.
Hellwig resigned as the maintainer of the DMA mapping helper, which is a good thing, and he will continue to maintain other areas of the kernel with a clear understanding of what the limits of his authority is. That’s also a good thing.
The clarification Torvalds gave means that contributors of drivers in Rust can be confident that their work won’t be ‘blocked’ by maintainers with an irrational hatred of said language. This helps Linux.
Does even Linus speak Rust or are the Rust additions like “black boxes” to him? Parts of the kernel somebody else will take over to maintain and develop?
The question with the millions of lines of code in Linux is how much of the C code is basically a black box to Torvalds. Sure, Torvalds breathes C. He sure knows the moving parts of his kernel. Does he know the nitty gritty of every part? Not very likely. There is a reason he trusts on his lieutenants.
Also Torvalds is not a static entity. Who is to say if he is or is not looking into how Rust works? Even to get a basic understanding of what has been bolted on to Linux. It’s an ongoing affair and people are never too old to learn something new.
C’mon, it’s like saying Linus or his first lieutenant are perfectly capable of writing poems in English or Chinese and if they can’t then that they are not too old and can always learn. Pipe dream.
Geck,
Kind of ageist of you to argue that. They’re no longer kids, but even older folks can and do learn. I’m not saying everyone’s open to learning new programming languages, but if they are willing then it isn’t a pipe dream. Sheesh what an argument.
“…it simply becomes an untenable position to block Rust from, say, the Linux kernel.”
I would turn this round and ask why it matters so much to the rust brigade that one operating system kernel adopts their language. There are no shortage of other open source projects like Redox, and MS is re-writing bits of Windows in rust.
The Boeing 737 MAX debacle was because nobody was willing to countenance a new airframe in response to new engine technology – something that would not even be a consideration before. Boosters of this new wunder-language should be chomping at the bit at the opportunity to rewrite the entire world from scratch, not trying to enter old projects that already have enough on their plate maintaining themselves.
…because saying “There’s Redox” is like saying “Don’t like our airline’s ‘no X policy’? You can always design and build your own plane and then found your own airline.”
Linux isn’t “a kernel”. Linux is an ecosystem of drivers and compatible userland software and it’s unreasonable to expect people to start from scratch on integrating that.
I’m reminded of Drew Devault’s The reckless, infinite scope of web browsers.
…and Bryan Cantrill’s comment about what a pain it was to emulate the interaction between Linux’s vfork and SIGCHLD semantics in Solaris-descended OSes accurately enough to reliably run Go binaries under ABI compatibility.
ssokolow (Hey, OSNews, U2F/WebAuthn is broken on Firefox!),
I appreciate the point you are making, but at the same time there’s a lot of legacy baggage in linux and arguably linux C compatibility requirements may impede better/safer kernel designs. So the suggestion to start clean isn’t completely without merit.
I would use the “linux is a kernel” spiel, but I won’t because I think your broader point is of much deeper importance. I have little doubt a new kernel/OS design can make better use of rustlang than linux could, but there’s the huge caveat that they’re starting so far behind linux as to face the prospects of never catching up. Linux has so much ancillary code including drivers/software/etc that it’s hard to conceive of a way for new projects to realistically come up to speed regardless of technical merit, As you suggested, adding rust to linux may be the more viable path forward, albeit less ideal.
It’s kind of like having an open source reactos, many of us are theoretically very interested in that, but nobody believes it will ever catch up.
I agree… but you don’t outcompete the incumbent by doing the same thing but marginally different, from the customer’s perspective. You overturn the table. You find an un-served/under-served niche.
That’s essentially why Rust saw so much interest so early compared to languages like D and Zig and Hare which are trying to be “C but better” or “C++ but better”. It’s why git won out over Subversion (CVS, but better).
Even if you had, I’d have come back with my “People think in concepts. Force them to call it LinuxOS or Blurf or whatever if you want, but the point they intend to communicate won’t change.” spiel.
That aside, I agree. It’s why I’m hoping that Servo’s turn toward being less of a competitor to Blink or WebKit or Gecko that’s perpetually on the back foot and more in the direction of being suitable for being an open-source competitor to sciter or a lightweight competitor to CEF works out for them.
ssokolow (Hey, OSNews, U2F/WebAuthn is broken on Firefox!),
Hey now, I’m actually a fan of Dlang.
IMHO they did a great job making a better version of C++! The main detractor is that it’s garbage collected. If it had used compile time safety constructs like rustlang does, then I think more developers would prefer dlang to rustlang. Alas, being a GC managed language means dlang is not a suitable replacement for C especially for things like operating systems.
While I only use it for windows software, I do like C#. I like how it implements advanced constructs in a clean way (compared to say C++). But there too, it’s not suitable for low level system code. I use C++ a lot. It was/is a good successor to C, but it also inherited many of C’s flaws. If you were only ever a C++ developer, you might not notice this, but after using other modern languages and then going back to C++ it’s legacy footprint becomes more apparent. I could enumerate many examples, but I imagine we’re already on the same page.
Yeah, that’s another example, we could probably find them everywhere.
I’m not sure about “legacy code base in given language to maintain” factor any more. LLM are quite a game changer in this area and for codebases sufficiently covered by unit tests it could become a non-issue
Have you actually tried to use LLMs to convert codebases?
I’m not talking about converting single classes or files, or even subdirectories. But entire codebases?
They would have assumptions that are correct in one language, but very difficult to replicate in the other.
At best case, you’d have things similar to “Java in C++” where many junior coders do. (Using objects when simple structs are available, dynamically allocating everything, and so on). It is technically correct, but nobody loves it.
Worst case? You’ll introduce very subtle bugs that would be hard to catch. (Yes, even rust programs can have bugs).
But the most expected case is that it will just not work.
And those unit tests? They won’t be enough. Rust does not even have a stable ABI, or calling conventions:
https://github.com/rust-lang/rfcs/issues/600#issuecomment-290103101
And is this performance difference due to any inherent characteristics of rust and/or its compiler, or is it just that the existing zlib C code is inefficient?
The original zlib is already pretty optimized, but also favors portability over extreme performance.
zlib-rs authors are gauging their work to more optimized C implementations though, from Google and others.
So the existing zlib C code is clearly not inefficient, and some of the performance gains are pretty substantial.
Yes,
There are “low hanging fruits” that will make zlib (or similar algorithms) faster at the expense of portability.
Specifically, making use of vector operations (SIMD, like SSE, AVX, NEON), making better use of cache locality, or using CPU specific instructions for CRC and similar calculations.
I have looked at their benchmarks, and they seem to compare with other portable ones. But as far as I know Intel has their own compression library: ISA-L, and Apple has: Compression. There are probably others from vendors that are highly specific for chips.
(I will be lazy and won’t do a benchmark on those).
bert64,
It’s a fair question. Obviously different implementations can have drastically different performance profiles and language is not the differentiator there. But nevertheless I think rustlang engineers have demonstrated a solid ability to run and optimize code effectively and genuinely be competitive with C.
The rust language itself has some design properties, that gives the compiler great optimization possibities. For example, for most of the variables the lifetime is known to the compiler even over function borders.
This itself results in lot’s of optimization possibilities for the compiler.
Like always, the newcomer is standing on the shoulders of the giant achievements of prior languanges. As a C++ developer I see lot’s of ideas that came from C++ and are now properly implemented in rust by default, because rust did not need to be compatible to legacy code.
The biggest challange of rust is going to be the long term compability. Are they going to plan for a long term compatibility with old code? At least last year, the rust handbook contained code examples that the newest rust compiler did not compile again. The rust developers need to figure out how to give some guarantees in this direction.
Kretikus,
Can you cite an example? Old and unstable features may break but I thought rustlang was supposed to be stable now. I’d be curious to see what the counterexamples are.
It was to long ago, to cite a specific example, I think there was already a github issue for this somewhere.
As far as I remember, one example had something to do with error handling. It could not convert from one error type to another error type.
Let’s regard this as an old issue and outdated documentation, which happens on projects of this scale.
I hope they can keep the promise of keeping the language stable, Even for decades.