With the recent Rust in Linux events in the last couple of days, it’s a good time to write up Rust in illumos. Both to spread the word a bit and also to set expectations for both sides (Rust and illumos/OpenIndiana devs) what is currently possible and what work would need to be invested to make things smooth. And also to let the Rust community know about what illumos people were talking about.
What most of the talk currently is about, are the technical details. But we must not leave the social aspects out of it. Software distributions are not made by lone walkers but by groups of people. Bringing in a new language means facilitating change. And that means there are more topics to discuss than just API design. We are talking about impacts on the whole software lifecycle.
↫ Till Wegmüller (Toasty)
I try to steer clear of all the Rust-related drama, mostly because it’s outside of my wheelhouse, but also because I don’t think anything I can highlight here will help anyone get anywhere or solve anything. In this particular case, there’s no drama, and it’s just a good ol’ discussion about what Rust as a programming language can contribute to the illumos community and code.
I can’t program so here my useful contributions end.
I don’t think it’s unreasonable to roughly correlate redundancy of C vs Rust and Xorg vs Wayland. For me Rust introduction to the linux kernel is as high of a priority as fixing Wayland’s big issues, but it seems both the lack of rust adoption and Wayland’s problems are equally ignored mostly for arbitrary reasons.
So while I understand avoiding the Rust debate related to the Linux kernel, it’s reasonable, it does however seem contradictory stance to take given how much Wayland gets pushed. Maybe I’m being unfair.
I think part of the pushback against rust is… its a fairly ugly verbose language to some eyes approaching the complexity of C++. It requires you to know what that ugly syntax means also… which is both good and bad.
https://matklad.github.io/2023/01/26/rusts-ugly-syntax.html
While C on the other hand is very simple … too simple but that is what people are used to.
cb88,
Yeah, I love the safety concepts, they’ve done a great job…but I can’t deny I wish the syntax were different. Personally I wish they had engineered rust around d-lang, which itself did a good job cleaning up c and c++.
Used to it, yes, but C has some terrible traits too. Even if we disregard the issue of robust software, the global namespace is terrible in this day and age, and frankly I miss not having OOP. These were added into C++ of course, but the template syntax is messy and meanwhile it doesn’t fix some of C’s worst traits like the preprocessor, Include files and forward declarations. These are the worst and C++ kept them to be compatible, ugh. New languages not only improve on features, but they’re cleaner and more pleasant to work with as well. When I go to C# or Dlang they’re a breath of fresh air next to C/C++ even though I use the later much more often.
For better or worse large swaths of our software and operating systems are still built on legacy languages. Developers of new applications are more keen to embrace new languages, but there’s so much preexisting code still based on C/C++ that won’t realistically get ported anytime soon.
As a Java programmer, I am the blind talking about color here: I like the concept of RUST, I hate the syntax, I am sad that there is still no stable/good/rich UI toolkit for — or else I would actually try it.
Although I don’t think its needed for the Kernel space: Those components are most static, well encapsulated and (hopefully) audited by millions of talented programmers. I think RUST can shine more in the application development, where a small team continuously just add features, highly dynamically, no code reviews, much complexity. This is the space (like Java) where the guidance and safety matters most.
The blog post seems to be in fact not focusing on the technical details of Rust usage, as much as on the conundrum of software packaged and delivered by a distribution vs. software packaged and installed by a 3d part tool (think npm, composer, pip, cargo, etc…).
As far as I can tell, this is an interesting discussion, which has been going on for a while, and has found no real answer so far – immutable os images and flatpaks/snaps, despite the latter two being heavily despised by many users, have in fact brought even more possibilities and confusion to the table.
It is smart to acknowledge that the needs of a distribution builder are not the same needs of a software developer, the same way the needs of a sysadmin managing a server dedicated to running a specific set/version of apps are not the same needs of a desktop user, the only common need probably being to keep the system secure. No single solution is probably ever going to be the best for all users and usecases; striving for an ecosystem where different means of installing software exist, while insuring that security fixes can quickly percolate through all of them, and at the same time minimizing the disk+memory bloat due to having multiple copies of the same versions of shared libraries is all we can realistically hope for.
I’d say immutable is despised on the desktop also… its fine for appliances and consoles devices though.
I think Rust demands new operating systems like Redox, Theseus, or something that is small enough to be rewritten in one “big bang” like HarveyOS/r9. I do not know about Illumos but as for Linux I’m glad they seem to be failing to bring Rust onboard: those rust developers are freed up to volunteer for a new project instead. Creative destruction!
Obviously as readers of OSNews we would rather see new operating systems than just the same old Unix based ones perpetuated.
It is not a zero sum game. I want both.
Also, Rust in the Linux kernel is going to mean more Rust programmers. More Rust programmers mean more side-projects in Rust–including operating systems.
I use Linux every day. Windows is starting to use Rust in places. I do not want Linux falling behind. I want it in Haiku as well. And elsewhere.
Not that I am a Rust zealot. If we can get something better, let’s put that everywhere instead. Other comments here have complained of the Rust syntax and, well, my eye agrees with them. But let’s have higher productivity with greater concurrency and fewer security bugs. Right now, Rust seems like a great way to get that.
tanishaj,
It’s about time start looking beyond C for system code!
To what Squizzler was saying, one of the issues with the “add rust to X” approach is that it typically requires rust code to use C interfaces, which sets (or continues) a bad precedent. I feel C interfaces, while good for providing commonality, are bad for limiting the use of modern innovation at the API level. So for example across a C API boundary you may need to build a wrapper with C primitives, you have to kludge the namespaces, you can’t use strongly typed templates,, you can’t use interfaces without typecasting, and moreover you have to abandon more advanced language features such as memory leak testing.
I see no good answer to this, it’s basically not realistic to replace C interfaces in established code bases. The work is too great, and the politics don’t align, therefor the use of C interfaces is out of necessity.
To your point that you want both, I see what you mean and it is a logical approach. However I do worry this intermediate rust+c hybrid model (assuming it actually succeeds) could end up becoming a rather permanent development. Future generations might look back with regret that we didn’t go further, much like the inadequacies of ipv4 so many of us still have to live with to this day even though Network Address Translation was only supposed to be a temporary fix.
I hear you.
That said, there is no reason that the more expressive language cannot build its own richer APIs even if it has to rely on the less advanced C APIs for compatibility. If I understood the recent Linux Filesystem API dust-up which Ted Tso, what the Rust folks were trying to do was create richer Rust APIs that better encoded the semantics of the C API so that most of the Rust code could call into these more expressive APIs rather than having to call into the C API directly. You could do the same thing in reverse in sub-systems written in Rust where you have the more powerful Rust API and then expose a lowest common denominator C API for compatibility.
One would hope that, if Rust ( or whatever superior language ) delivered on its promises, more and more of the OS would be written in Rust that is being called from Rust instead of the situation today where Rust is primarily calling into larger bodies of code written in C and nothing is really calling into the Rust code (yet). That could certainly take a long time as nobody is going to want to ( or even should ) rewrite all that C code anytime soon. That said, major systems do get rewritten. There are new schedulers fairly often it seems. The memory management code has been massively reworked before. Filesystems of course appear with regularity. These all feel like things that, at some point, could be replaced with superior Rust versions not because the were “rewritten” in Rust but because they were replaced by implementations of newer designs that chose Rust for that task.
tanishaj,
Well, it’s not that I object to your approach but it adds workload and I don’t have the confidence that it will be used. It’s like building a new OS with novel APIs alongside a linux compatibility API. Even assuming your new API has merit, it does not follow that any 3rd party software will actually use it. Your OS will only be supported using the linux compatibility API. And then to add salt to injury the burden will be on your operating system to be compatible with linux’s implementation of some feature innovation they copied from your operating system because the linux API is the one that virtually 100% of software developers will target.
Well, even assuming we’re ok with this, creating and supporting two APIs using primitives that don’t translate directly into each other is a nuisance that might end up creating a lot of duplication of work. Just think of all the work that the compiler’s doing when we call “async” methods. Ok great, I love this innovation BTW! But now we have to additionally create a C interface that necessarily throws that innovation away and works completely differently to be C compatible. Not only do we have to write more code, but we have to debug it too. Yeah, I guess it’s doable, but at what cost and at what benefit?
Honestly my vote would be to just embrace the innovation and leave C compatibility behind. Innovation warrants a clean break from the past, otherwise it gets hampered by the past. I see this as a negative for your proposal. But then I still realize that many projects can’t and won’t break away from the past like I’m suggesting. It’s a conundrum.
Yet all of these get rewritten within the constraints of old code & models. One example is that linux kernel lacks proper asynchronous support, both at the API level as well as at the driver level. Incidentally this is why some hung processes on linux can’t be killed until the input they’re waiting on times out. If you want non-blocking file operations you have no choice but to use threads. It doesn’t have to be this way, windows has nonblocking operations for example, but linux was designed to clone unix where that wasn’t a thing. Today the linux kernel is extremely dependent on blocking threads. There is an asynchronous POSIX API, which is supported on linux, but in order to support linux they actually have to spawn POSIX threads behind the scenes to marshal the asynchronous requests from userspace to the kernel and back. So while they successfully mapped one API to another, the amplification of resources needed to do it means that nobody who actually cares about scalability actually wants to use it.
This has nothing to do with rust per say, but I bring it up to highlight how decisions made eons ago continue to impact our modern technology. And I think this message is relevant to our discussion about the future of rust in C codebases. Adding rust code to a C kernel might not lead the way to a kernel optimized to use rust features effectively. Let’s even say that we did this for every line of C code (not likely given current linux politics, but just hypothetically)… I think we’d end up with a kernel written in rust but designed around C primitives. At least for me, this is not the goal we had in mind.
There’s no doubt that a rust implementation from the ground up would be cleaner for the future, however I still have to concede that replacing C code this way might not be a realistic option. As messed up as the rust+C hybrid model is, it may be the only way to phase out our dependence on C. I don’t have a better answer.
It’s after the edit window closed, but I would replace “messed up” with “non-ideal”.