Guest post by Redox, a Unix-like operating system written in Rust, recently rewrote its kernel:
Since August 13, the kernel is going through a complete rewrite, which makes the kernel space ultra-small (about the size of L4). Everything which can run outside the kernel in practice, will do so.
It is almost complete and will likely be merged in the coming week.
The reasons cited for the rewrite include memory management, concurrent design, SMP support, and 64-bit by default.
One has to wonder why it wasn’t 64-bit by default in the first place
Because there are embedded systems out there that are 32-bit only?
I thought Rust was supposed to be completely safe and basically magical in its ability to prevent bugs.
You can still write code that is incorrect or has a flawed design. Rust is designed to eliminate common errors in dealing with memory pointers, one subset of bugs. It doesn’t make a flawed algorithm work, just makes it a bit harder to accidentally botch an implementation of good design.
Rust isn’t really any more “magic” than say Python. But unlike Python it is usable for programming traditionally left to C. Which is pretty unique.
Edited 2016-11-01 05:00 UTC
DrJohnnyFever,
While it’s true python shouldn’t generate segfaults or experience memory corruption or anything like that, which are some of the most dreaded kinds of faults to track down. I’d argue that rust is safer than python on account of verifying code at compile time. Python won’t even tell you until run time that you are using an undeclared variable. It will detect the error and throw an exception, but in a system that’s designed not to fail, it’s not great that the earliest detection of a fault is at the time of execution.
To deal with things like VM mappings you need to use unsafe code even in Rust. Once you do that all of Rust’s safeties are essentially off (at least for that piece of code).
Don’t believe RUST was design aimed as a VM framing. Just don’t do that [with].
kwan_e,
I’m not familiar with this particular code, but it should be explained that rust supports two modes of operation, both safe and unsafe.
When in safe sections of code, which is the default, rust enforces integrity by verifying the correctness of each state transition, which is what makes rust robust. And the fact this this is done at compile time allows us to build safe zero cost abstractions. However when you are developing new primitives, you may have to jump into unsafe sections to perform operations (like raw/untyped memory access) that haven’t been encapsulated by a safe call or library.
Most programs wouldn’t have to do this, but when your implementing an OS or other system interface code, you may need some primitives that aren’t implemented by the standard libraries, and rust lets you do that by letting you explicitly disable code safety checks.
“Unsafe code” in rust is essentially “normal code” in other static languages. One benefit of explicitly requiring unsafe to be tagged as such is that in a large project with millions of lines of code the scope of unsafe code is significantly reduced, making it much easier to find/audit/fix.
Edited 2016-11-01 15:33 UTC
Surely, “zero cost abstractions” shouldn’t need a library to encapsulate it. Or, the first thing to do is to leverage these “zero cost abstractions” to make unsafe code safe.
kwan_e,
That’s exactly how the unsafe code sections in rust are used.
Do you take issue that developers need “unsafe sections” at all? The trouble with that is that it assumes that everything would be built in to the language from the get go. But satisfying everyone’s niche requirements with a general purpose framework would result in a bloated framework becoming overspecialized.
Granted, there’s always going to be a debate as to where to draw the line, should a framework satisfy 90% of users, 95%, 99? 100% is not realistic, there will always be someone who needs to go outside of existing work. It makes sense to me that operating system code is in this set where developers may need to define new specialized primitives.
Also, keep in mind that for all strings and abstract data type primitives in the standard library, their safety against corruption stems from the efforts of the developers who implemented them. Rust, at least for today, doesn’t have any kind of AI to validate unsafe code. It’s sort of like a “boot strapping” problem. Maybe the next generation of languages could try to tackle that, but in doing so, they might have to contend with Gödel’s incompleteness theorem.
https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleten…
Edited 2016-11-02 10:22 UTC
Also, it isn’t obvious from what I’ve written, but only the original primitives at the bottom of the stack would require unsafe sections, not the entire framework. For example, if you build a framework using entirely pre-existing rust primitives, you wouldn’t need to write any unsafe code yourself.
The most common use of “unsafe” code in rust is actually linking to existing C libraries simply because they are so pervasive.
I don’t have an issue with it at all. I’m just having a laugh at people and languages that try to promise the impossible.
I love me some “zero cost abstractions”, and even some “negative cost abstractions”, to force compile time errors if I can help it.
kwan_e,
Well, I’d like for it to become standard practice for software. For better or worse, many applications have transitioned to “managed” languages like java, .net, and a plethora of scripting languages, which protect from corruption at run time, but are not zero cost abstractions.
Edited 2016-11-02 14:04 UTC
It’s literally WYSIWYG assembly, even though it’s using high level constructs like classes and methods and the compiler will still scream at you if you get anything wrong.
https://www.youtube.com/watch?v=zBkNBP00wJE
kwan_e,
I know Ada is derived from pascal, but not much else. Does Ada have anything to protect developers from heap management bugs that are possible in pascal?
It’s funny that each community has it’s own dominant computer language. Obviously C tops the charts for operating systems since it’s unix debut. Java is big for enterprise apps. Ada is a standard for government contracts. FORTRAN is big in the scientific community. Javascript is the leader for web browsers. PHP leads website programming.
These all have historical reasons for becoming dominant in their respective domains, but to someone being introduced to software field for the first time, it must seem highly arbitrary, I imagine
I’ve never done Pascal outside of high school, so I’m not familiar with heap management bugs of Pascal.
Ada has to work in constrained environments, so its memory management is much stricter than general heaps. It uses storage pools, and you can define your own and specify which storage pool to use when allocating.
You could look at it that way. But web browsers, JVMs and Javascript engines are all written in C++, with the newer ones fully utilizing the newest C++xx features available to them.
Outside code written by big companies like Google, Facebook, Microsoft, Apple, what I usually see on the small to medium size companies it more C compiled with C++ compiler than anything else.
The Rust compiler uses LLVM for the backend, so it gets some optimisations (although apparently not as much as Clang, because LLVM better understands how Clang does things). Of course, that also means compilation isn’t fully self-hosted, although I think focussing on the frontend is a pragmatic use of effort.
Anyway, maybe it’ll turn out to be a fad, but I wouldn’t write it off just yet. It seems like Mozilla is pretty serious about getting it into Firefox.
I don’t doubt that Rust can do zero cost abstractions. I think it’s capabilities has just been oversold a bit. My code snippet was just to point out it was already possible to do low level safe programming with zero-to-negative cost high level abstractions.
And reading up on Rust’s apparently Turing complete generics, I don’t really see why people complain about metaprogramming. It’s obviously a useful technique, in and outside of Rust, and certainly not beyond the skills of your average programmer. But somehow Rust is touted as having magical compile time checking not possible in other languages, when it actually is possible.
kwan_e,
It’s not quite the same, rust actually does perform static safety checks that no other static language currently can. C++ templates don’t get you all the way there. While they can help produce correct code, they still don’t check that the code is actually being used correctly.
STL is not thread safe, it’s up to you as the developer to make it so. You have to decide where and when to add locks. Adding locks on every method by default would incur too much overhead, not adding locks by default means the developer could make a mistake. Rust can actively tells you that there is a race condition at compile time, C++ doesn’t know any better.
In C++, it’s easy to accidentally do the wrong thing like deleting an object twice or deleting it while there is still a reference. C++ templates can give us “smart” ref-counted object types. But if you were to add them to every object as a precaution to prevent accidental deletion, it wouldn’t be zero cost as it requires both more memory as well as more CPU at run time. Making it thread safe would add serialization overhead on SMP machines. Rust’s compile time code scanning detects and prevents even subtle violations from compiling, C++ doesn’t have a scanning engine capable of detecting these violations. A sophisticated lint tool might bring C++ pretty close though, which would no doubt be preferred by fans of C++
In a world where we now know it may be faster just to operate on copies of data, or tiling your data to be usable by task-based concurrency, rather than sharing data, I see little value in being able to check code for such data race violations. And I do wonder whether Rust is sending the wrong message and causing people to go for suboptimal designs because they can rely on the compiler for detecting data races, instead of spending a little time properly designing the isolation of tasks such that data races are not even a concern. C++ also has futures/promises and packaged tasks to make it easier to do parallel work without sharing resources. Admittedly, async_exec is a poor-man’s executor, but it’s suitable for quite a lot of basic tasks.
In a way, you can statically make sure things aren’t shared by deleting copy constructors from your design. That way, it prevents you, at compile time, from introducing races in those circumstances. You can use compile time checks to, say, make sure lambda functions do not contain state. Trying to coerce lambdas into a normal function pointer will fail if a lambda contains state. Tasks can enforce non sharing by enforcing any resources passed to it are r-value references – meaning that the resources are either created in-situ, or moved to it, making it solely owned by that task.
Edited 2016-11-04 08:45 UTC
kwan_e,
I started responding point by point. But wow, you wrote a lot. So here’s an abbreviated response
I agree, but this highlights the point I’m trying to make: Rust can flat out tell you when your assumptions are wrong. C++ will happily compile unsafe algorithms and it doesn’t know any better. The responsibility lies on the C++ developer to make sure the right choices were made, otherwise an executable can end up corrupting it’s own memory. I agree that you CAN make safe choices in C++, but C++ won’t tell you when you’ve failed to do so. And history makes clear that anytime you leave it to the developer, eventually bad assumptions will lead to mistakes. That’s a strength Rust has over C++.
I answered your last paragraph first because it’s the most relevant to this article.
I don’t understand the issue here. I can implement something such that only safe uses are allowed at compile time. Then someone using that code unsafely (ie, outside of the contract) will fail to compile. That is functionally the same as C++ telling you when you’ve failed to make the right choice. C++ provides that ability by proxy in the language. Rust provides that ability directly in the compiler. But the fact is the C++ language is designed such that those contracts can be expressed in the language as a library solution. You may as well be saying that the problem with C++ is it doesn’t provide those checks in the form of syntactic sugar and that it has to suffer through library implementations of those checks.
Sure, a user can then choose to do something like reinterpret_cast, but that’s the same as someone in Rust declaring their code to be unsafe.
kwan_e,
Thread safety is a perfect counterexample.
Edit:
We might have to just disagree. Oh well, no big deal…maybe we can just point and laugh at a common enemy, like PHP or Perl
Edited 2016-11-04 18:50 UTC
That seems to be the only counterexample. And I outlined various compile time ways to make it a non-issue – like by preventing sharing in the first place. Yes, it’s not a compiler intrinsic, you have to enforce it through SFINAE contracts (and possibly being a bit dickish, like using SFINAE and static_assert to block the use of shared pointers or mutexes etc in the same translation unit). If someone can write something to calculate a SHA1 to warn about consistency at compile time, they can do this as well. But it’s still a capability that isn’t beyond C++. I mean, did you even know about the SHA1 thing before someone figured out a way to do it? If not, then how can you know it is beyond the capability of C++?
Incidentally, I just remembered Microsoft’s C++ AMP introduced a keyword “restrict” to only apply to functions that are code that is passed to a GPU that does throw a compile error if you reference any global state. And I understand there’s a proposal for a “pure function” that does a similar thing. http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3744.pdf
* Side note about the point about Rust not being able to prevent from making a bad design (even though I’m not sure it was the case with the Redox kernel) is interesting, since you can use C++ to prevent your from making a bad design.
Edited 2016-11-05 01:22 UTC
kwan_e,
Yes, I was aware templates are turing complete. It essentially means you can translate code from any turing complete language into C++ templates (within the limited confines of C++ template I/O)
https://en.wikipedia.org/wiki/Template_metaprogramming
Ironically, this means you could theoretically write a new C++ compiler capable of scanning for interthreaded safety issues, port the entire compiler into a C++ template, and then parse another program for issues using the templated version of the C++ compiler! Ouch
Edited 2016-11-05 02:53 UTC
Some crazy person can do it. I mean, the SHA1 thing is based on compile-time manipulation of constexpr strings – char arrays – as though they were runtime strings. It will just take hours to compile.
Edited 2016-11-05 04:25 UTC
kwan_e,
You really can’t think of any languages that are worse than Rust? I don’t believe you
We’re not talking about perfection, of course there are things I dislike about rust, they just hadn’t come up since we were discussing safety.
I don’t consider memory faults to be the worst thing about C++ either, IMHO I would say one of the uglier aspects of C/C++ programming is the pre-prosessor and header files – that’s something even devote C/C++ coders usually seem to agree upon. Most modern languages will have better ways to handle symbolic imports better than C/C++.
http://dlang.org/pretod.html
Edited 2016-11-05 06:27 UTC
I mean in the context of the discussion about languages that can verify things at compile time. Some depend heavily in the compiler. Others allow library solutions. In the end, they have all achieved roughly similar ends, even if the solutions don’t look the same.
kwan_e,
This isn’t a fair comparison because the compiler based safety tells you when code is bad. The library solution requires developers to anticipate faulty uses of the library and explicitly block them. However this isn’t foolproof and can inadvertently block legitimate/safe uses that the library developers didn’t anticipate or else didn’t have a way to detect because C++ lacks the context for. A future revision to C++ could fix this, but for now it’s a limitation.
Edited 2016-11-05 17:37 UTC
How is that different from if a Rust developer wants to do something legitimate that the compiler writer didn’t anticipate, so they declare their code unsafe to get around all of the checks? Which they do, especially in kernel development like Redox?
The point of libraries is that if it does block a legitimate use, it can be changed easily. Especially with a generic library where the contracts are SFINAE, it’s a lot easier to design around the library at the interface than changing your legitimate design to get around the compiler. You can’t change a compiler from within your library source, you can only disable safety checks. I would argue that makes a language as safe as its unsafe parts, regardless of how much checking is done in the safe parts.
Edited 2016-11-05 23:57 UTC
kwan_e,
The difference is that in Rust, the library primitives don’t need to prove/disprove the safety outside of themselves because it’s a feature of the language.
With no similar process in C++, the burden necessarily lies with template code to make a determination about safety, however it’s forced to do so without the benefit of knowing any of the context in which it’s used (like the threads we’re talking about, or whether or not the user’s algorithms belong to a safe variety). Therefor a template is forced to make assumptions. These assumptions will result in either false positives or false negatives when the assumptions are broken. We can’t fix the assumptions without changing C++.
For the sake of argument, can you point to an example C++ library that will only compile if it’s usage in a program is safe?
Do the safety assumptions break once you go multithreaded?
Does the library throw errors inside of legitimately safe contexts?
I assert that the only way to solve this in C++ is for the library to be programmed for the worst case scenario and use locking internally at every single invocation. However doing so adds potentially unnecessary run time costs for developers who don’t need the locking because their code is already safe (say because they’re already doing locking correctly themselves outside the library).
You could provide two versions of the library, one with locks and one without, but C++ can’t prove that the class without locks is being used correctly by the developer. The fact that Rust can is an advantage.
Edited 2016-11-06 03:25 UTC
At this point, I think we’re just going to be discussing in circles, because I think library implementations of compile time checks is legitimate, and other people think only compiler intrinsics “count” and other methods “don’t count”.
Well, no, since there are lock-free concurrent libraries out there now. Like I said, sometimes you can solve a problem by bypassing it completely. In those cases, you don’t need to warn about errors because their lock-free nature prevents it at runtime.
kwan_e,
Ok, but you have to admit that restricting certain functions of your classes for one context can makes it less useful for another context. While your restrictions may be well intentioned, they’re based on assumptions that the contexts being blocked are dangerous. However C++ templates have no mechanism for determining whether a context is actually dangerous or not.
A C++ template has no clue if it’s being called with a held lock. It has no clue if that lock is the only one needed to protect from concurrent access. It has no clue if an object is referenced by another thread. Without this information, C++ cannot help your template to block only the dangerous contexts, so you end up blocking the functionality in all contexts. Can we please agree on this?
Also, I don’t think you realize how difficult it is to block dangerous code in C++. Say I’m a user of your “safe” C++ class library. As the library author, you have no idea how I’m going to use your class. Do you incur a performance penalty to safely support multi threaded access? Do you support single threaded access and leave safety to the user? These are both valid approaches…but with C++ you have no way to enforce the correct usage of your library, and that’s the problem!
I create a class with a reference to your class, nothing special at all, but once I pass my object into a new thread suddenly your class faces the possibility of reentrancy/concurrency. Again you have no way of knowing or enforcing what I do at compile time…the C++ template is powerless over what the developer does. I could even accidentally cause your object to be destructed in two threads causing serious corruption, and there’s little your “safe” C++ library could do about it at compile time to prevent it.
You know I’m right, so can we please finally agree?
https://blog.rust-lang.org/2015/04/10/Fearless-Concurrency.html
Even here it’s touting the benefits of library solutions. And “ownership”? Isn’t that what I’ve been talking about? It sounds like, at least here, Rust’s supposed memory safety is through enforcing the ownership of data by a library – exactly what I’ve been saying is possible in C++. And preventing sharing is apparently one of the main ways Rust avoids these problems – just like I have been saying.
More and more, it really does seem Rust afficionados have built up its magical abilities too much.
That’s not fair, I’ve been very patient and very responsive. Rust isn’t magical, but it was developed with safety in mind whereas C++ was not. C++ templates are powerful, but the language will need some enhancements before it can do the same thing.
Edited 2016-11-06 07:17 UTC
If you don’t mind my asking, whenever you say “C++ can’t…”, which version of C++ are you automatically thinking of? Because I’m talking at a minimum C++11, since Rust came about that time. Through some of your counter-examples, I can’t help but think we’re talking at different versions of C++, or at least when you try to set up a situation that I would agree be impossible with C++98.
This discussion sort of reminds me of The Sandman when Dream had to go to hell to retrieve his helm and he battles a demon for it through a battle of stories.
“I am hope…”
Edited 2016-11-06 09:06 UTC
I think what you are misunderstanding is that adding these restrictions would somehow prevent a second thread from accessing them. Hint: they don’t. Secondly, what you propose disables functionality from C++ because of your assumption that they might be used dangerously, but you also end up blocking safe & correct use as well, which is what I’ve been saying all along.
You’ve accused me of this: “it really does seem Rust afficionados have built up its magical abilities too much.”, but I’ve given numerous reasons to substantiate my claims. Yet you keep insisting that C++ has no safety issues even though it doesn’t solve these problems, so maybe C++ is the one with magic
We’re running out of time, so I’m going to end this here leaving the onus on you to find a C++ library that can protect resources and can’t be used unsafely AND doesn’t require runtime overhead. If you can’t do that, then I’ll let it speak for itself. I hope that perhaps we can find some more common ground in the future, Cheers
Edited 2016-11-06 17:47 UTC
Considering the very Rust doc page I linked to talks about reference counted resources and ownership semantics enforced by mutexes, I’m claiming Rust doesn’t achieve your claims either. There’s still runtime overhead, because Rust doesn’t elide locking, and you still need to use Arc<T>s to guarantee to meet the ownership requirements. They are not zero overhead.
Show me where Rust says “if we prove your code correct, we completely elide locking and compile away the Arc<T>”.
Here are the OFFICIAL DOCS:
https://doc.rust-lang.org/std/sync/struct.Mutex.html
“data.lock().unwrap()”
“Acquires a mutex, blocking the current thread until it is able to do so.
This function will block the local thread until it is available to acquire the mutex. Upon returning, the thread is the only thread with the mutex held. An RAII guard is returned to allow scoped unlock of the lock. When the guard goes out of scope, the mutex will be unlocked.”
Tell me where it says the code in “lock()” and “unwrap()” is compiled away if it doesn’t detect a data race.
Conversely, I could easily write a type trait for my class which can compile away the internal locking mechanism if it isn’t used in a thread.
Please show me Rust documentation that actually proves data races are eliminated that doesn’t rely on ownership semantics (which C++ has, because Rust took that from C++). Because the OFFICIAL DOCS talk about memory safety because of ownership semantics. It does what I’ve been saying all along – eliminate data races by eliminating sharing.
Edited 2016-11-06 23:44 UTC
Are there plans to support OpenGL or Vulkan? Supporting any of these is mandatory for modern user interfaces.