New OpenBSD kernel security feature

Submitted by Ryan Freeman 2017-06-14 OpenBSD 38 Comments

Theo de Raadt unveiled and described an interesting new kernel security feature: Kernel Address Randomized Link.

Over the last three weeks I’ve been working on a new randomization feature which will protect the kernel.

The situation today is that many people install a kernel binary from OpenBSD, and then run that same kernel binary for 6 months or more. We have substantial randomization for the memory allocations made by the kernel, and for userland also of course.

However that kernel is always in the same physical memory, at the same virtual address space (we call it KVA).

Improving this situation takes a few steps.

About The Author

Thom Holwerda

Follow me on Mastodon @[email protected]

38 Comments

2017-06-15 6:45 am

codifies
I’ve never seemed to own hardware that BSD wants to play with, which is a real shame, my hardly bleeding edge broadwell laptop, with intel wireless card either crashes with one, distro, or with another doesn’t seem to recognise the wifi card (no ether port) leaving me stuck…

the last time I asked help on a BSD board was years ago I got three responses demanding I call it BSD and not bsd in a fairly rude manner and no help…

I guess if few people are using it, the its not a malware target…

2017-06-15 7:50 am

unclefester
I’ve run FreeBSD on dektop hardware without a hitch. It is trivial to install using a GhostBDSD ISO. GhostBSD also runs in Virtualbox.

Edited 2017-06-15 07:52 UTC
2017-06-15 8:35 am

grandmasterphp
OpenBSD works fine with almost all intel wireless cards.

http://man.openbsd.org/iwn.4

However you do need to use the fw_update tool to load the firmware.

http://man.openbsd.org/fw_update.1

Or copy it via a usb stick.

They have a clear list of devices that work and those that don’t.

Edited 2017-06-15 08:39 UTC
2017-06-15 2:04 pm

unix_joe
Give it a try. Broadwell has been working on OpenBSD for a year and a half now. Broadwell is the most recent hardware I’ll buy for that exact reason (i3, i7 NUC, X250 fwiw).

There’s no hardware configuration or tweaking needed at all. Install, reboot, and you have Xorg running with xenodm to login. It’s the simplest operating system to use.

It’s the newer Intel releases that are still a work in progress.

2017-06-15 2:00 pm

Bill Shooter of Bul Platinum Prime
Yet, I’ve never found a great use for it in the tech stack. Suggestions? Maybe as a router and/or firewall?

Software I’d run on an application server isn’t *BSD friendly.

2017-06-15 3:20 pm

Flatland_Spider
Firewall, router, IPSEC VPN, or, now, load balancer are the situations I could see using it.

They have quite a few technologies that make it really nice for network devices. They have CARP for failover, PF for firewall, Relayd for load balancing and proxying services, OpenIKED for IPSEC VPNs.

Then there is OpenSMTPD for email. This isn’t limited to OpenBSD, so it can be run on most Unix-like platforms.
2017-06-15 6:03 pm

darknexus
Yet, I’ve never found a great use for it in the tech stack. Suggestions? Maybe as a router and/or firewall?

Indeed. It makes an excellent Firewall if you know how to set everything up. Also makes a nice web proxy.
2017-06-16 7:03 am

grandmasterphp
I use it as a Desktop instead of Linux.

http://sohcahtoa.org.uk/openbsd.html (this isn’t mine but I basically do more or less the same thing).

Most open source applications are either in packages or ports.

The only thing I can’t get working is Dropbox and Sublime Text and I can live with Atom.

Edited 2017-06-16 07:04 UTC

2017-06-16 1:46 pm

Bill Shooter of Bul Platinum Prime
Not going to work for me, I prefer a number of Linux only technologies in my workstation.

2017-06-17 11:19 am

grandmasterphp
such as out of interest?

2017-06-15 2:14 pm

dionicio
The Smart, Diligent People we’re used to.
2017-06-15 3:55 pm

kwan_e
When you’re so far ahead of the curve and you don’t know what else to do.

“Fuck it, let’s randomize the kernel!”

2017-06-15 7:36 pm

Megol
When you’re so far ahead of the curve and you don’t know what else to do.

“Fuck it, let’s randomize the kernel!”

LOL!

2017-06-15 4:31 pm

Alfman verbose=1
Address randomization and ASLR don’t serve a purpose in languages that don’t have exploitable memory leaks in the first place. Clearly the purpose is to mitigate the consequences in the aftermath of these vulnerabilities, but they’re still there and this fails to address the underlying causes.

Despite all the real security problems with C based operating systems, it still has magnitudes more development resources than anything else. Although we have a few OS projects experimenting with robust programming languages to enforce safe memory usage at compile-time, unless the big players start seriously pushing them over the incumbent C kernels, then I’m not sure that any competitor can achieve the critical mass needed to become mainstream.

2017-06-15 4:51 pm

dionicio
You’re right, but a smaller sitting duck is always welcomed
2017-06-15 4:55 pm

dionicio
Isn’t this common practice at secure embedded devices?
2017-06-15 7:49 pm

Megol
Address randomization and ASLR don’t serve a purpose in languages that don’t have exploitable memory leaks in the first place. Clearly the purpose is to mitigate the consequences in the aftermath of these vulnerabilities, but they’re still there and this fails to address the underlying causes.

Despite all the real security problems with C based operating systems, it still has magnitudes more development resources than anything else. Although we have a few OS projects experimenting with robust programming languages to enforce safe memory usage at compile-time, unless the big players start seriously pushing them over the incumbent C kernels, then I’m not sure that any competitor can achieve the critical mass needed to become mainstream.

Enforcing safety at compile time is pretty hard. Linear types and things derived from them are AFAIK the most useful technique, but it creates extra work for programmers. Having partially compile time and partially run-time checking is easier and what most safe languages do.

A light-weight safe programming language with manual memory management (and mechanisms to make management automatic in most cases) for systems programming would be nice. The hard thing is detecting dangling pointers with as little overhead as possible.

While I agree with you the world we live in ATM systems are generally programmed in C and just rewriting everything isn’t realistic.

2017-06-15 9:31 pm

Alfman verbose=1
Megol,

Enforcing safety at compile time is pretty hard. Linear types and things derived from them are AFAIK the most useful technique, but it creates extra work for programmers. Having partially compile time and partially run-time checking is easier and what most safe languages do.

A light-weight safe programming language with manual memory management (and mechanisms to make management automatic in most cases) for systems programming would be nice. The hard thing is detecting dangling pointers with as little overhead as possible. [/q]

Zero cost secure abstractions is the motivation for languages like rust. In principal, any place you’d traditionally consider a run-time check, you could instead employ compiler technology that verifies leaky paths are not possible. We human programmers can accomplish the same thing in any language by carefully following code paths by hand, but after a certain complexity all human programmers become fallible, and this is where safe languages have the advantage.

[q]While I agree with you the world we live in ATM systems are generally programmed in C and just rewriting everything isn’t realistic.

I hear you, and yet until we address it, our software will continue to be vulnerable to memory corruption.

Edited 2017-06-15 21:31 UTC

2017-06-16 7:26 am

Brendan
Hi,

I hear you, and yet until we address it, our software will continue to be vulnerable to memory corruption.

..and then after we address it, our software will continue to be vulnerable to compiler bugs and “row hammer” and malicious devices plugged into a gaping “thunderbolt” holes and…

– Brendan

2017-06-16 2:07 pm

Alfman verbose=1
Brendan,

..and then after we address it, our software will continue to be vulnerable to compiler bugs and “row hammer” and malicious devices plugged into a gaping “thunderbolt” holes and…

Yep, bugs are always possible without formal proofs anyways, but we should still make an effort to fix what we can.

With firewire/thunderbolt, that’s what you get when you expose the system’s inner bus to peripherals. USB protocols are clearly safer by design (although I suspect a USB fuzzer would expose a lot of holes in our USB stacks).

Regarding hardware corruption, that’s tough. We can mitigate it by adding ECC and redundancy to everything. A fully redundant CPU could mitigate soft CPU errors, but not systematic ones where both CPUs have the same fault. You linked a video about open source CPUs, which could help detect systematic faults. Having redundant CPUs from different manufacturers could catch both types of errors, but it would be very expensive and largely wasteful for events that (I presume) are extremely rare. It could make sense in more mission critical roles though.

2017-06-16 5:01 pm

Megol
Megol,

Enforcing safety at compile time is pretty hard. Linear types and things derived from them are AFAIK the most useful technique, but it creates extra work for programmers. Having partially compile time and partially run-time checking is easier and what most safe languages do.

A light-weight safe programming language with manual memory management (and mechanisms to make management automatic in most cases) for systems programming would be nice. The hard thing is detecting dangling pointers with as little overhead as possible.

Zero cost secure abstractions is the motivation for languages like rust. In principal, any place you’d traditionally consider a run-time check, you could instead employ compiler technology that verifies leaky paths are not possible. We human programmers can accomplish the same thing in any language by carefully following code paths by hand, but after a certain complexity all human programmers become fallible, and this is where safe languages have the advantage.

[/q]

The problem is that the abstraction isn’t zero cost, not for the compiler and not for the programmer.

Rust is a good example, it uses ownership types, borrowing of data and reference counting GC. It is also very verbose, complicated and not especially high performance. Nothing about the security checking is really novel and the big thing about it is very good compile time checking of constraints. To get there it removes a lot of features otherwise considered essential for a modern language.

The programmer have to work harder to do a certain task in Rust, some tasks “have to” be coded as unsafe which removes the main reason to use it.

Using reference counting for garbage collection is a slow way to do it, less real-time than scanning collectors, less capable (RC in itself can’t collect cyclic structures) and worse performing. It is hard to reason about execution timing and it have space overheads too.

[q]While I agree with you the world we live in ATM systems are generally programmed in C and just rewriting everything isn’t realistic.

I hear you, and yet until we address it, our software will continue to be vulnerable to memory corruption.

Yes. But even using safe languages aren’t enough IMHO, making programmers handle exceptional cases correctly is also important – it doesn’t matter if memory is only accessed by an entity that have the rights if it writes the wrong data.

But that is another story.

2017-06-16 6:24 pm

Alfman verbose=1
Megol,

The problem is that the abstraction isn’t zero cost, not for the compiler and not for the programmer.

Rust is a good example, it uses ownership types, borrowing of data and reference counting GC. It is also very verbose, complicated and not especially high performance. Nothing about the security checking is really novel and the big thing about it is very good compile time checking of constraints. To get there it removes a lot of features otherwise considered essential for a modern language. [/q]

You are right, nothing is novel about static security checking, human programmers have been doing it since the beginning. If this could produce consistent results then it would be fine, however human programmers are faulty and increasingly so as complexity rises, which is why safe languages are advantageous.

The programmer have to work harder to do a certain task in Rust, some tasks “have to” be coded as unsafe which removes the main reason to use it.

You need to use “unsafe” sections in rust for the same reason you need to use “asm” sections in C, to build new primitives. You can consider this a limitation and I won’t argue that, but the vast majority of code can be safe.

Using reference counting for garbage collection is a slow way to do it, less real-time than scanning collectors, less capable (RC in itself can’t collect cyclic structures) and worse performing. It is hard to reason about execution timing and it have space overheads too.

I’m not sure why you mention this because rust doesn’t make you use garbage collection or reference counting. If you use RC or not is up to you, like with C. Also objects are cleaned up deterministicly. Rust tracks objects just as C does, the difference is that the ownership of rust objects are explicitly declared in code such that rust can prove correctness, whereas in C the ownership is ambiguous and a human programmer has to keep track of object ownership in one’s mind.

[q]Yes. But even using safe languages aren’t enough IMHO, making programmers handle exceptional cases correctly is also important – it doesn’t matter if memory is only accessed by an entity that have the rights if it writes the wrong data.

I agree it’s not going to verify the programmer’s logic, but it is enough to prevent the segfaults and pointer errors that address randomization is designed to mitigate.

Regarding “checked exceptions”, that opens up a whole can of worms I don’t want to get into, haha. But in principal run time exceptions handling is orthogonal to compile time pointer verification.

Edited 2017-06-16 18:30 UTC
2017-06-16 9:36 pm

Megol
Megol,

You are right, nothing is novel about static security checking, human programmers have been doing it since the beginning. If this could produce consistent results then it would be fine, however human programmers are faulty and increasingly so as complexity rises, which is why safe languages are advantageous.

[/q]

As I wrote in the first comment I’d prefer a safe language with allocation and de-allocation of memory under programmer control. That is dangling pointers, free-after-free and use-after-free errors would be caught either at compile time or run time. That is enough to be a safe language memory vise.

I have experimented with ownership with borrowing etc. earlier (before Rust existed) and at least for me it isn’t worth the trouble. It is harder for a programmer to reason about ownership without a lot of hand-holding by the compiler, something Rust doesn’t provide. IMHO of course.

You need to use “unsafe” sections in rust for the same reason you need to use “asm” sections in C, to build new primitives. You can consider this a limitation and I won’t argue that, but the vast majority of code can be safe.

Unless that changed after I last looked at it Rust programmers used unsafe constructs for non-trivial datatypes. IIRC.

Other safe languages with unsafe code support generally use it for very primitive constructs. Semaphores etc.

I’m not sure why you mention this because rust doesn’t make you use garbage collection or reference counting. If you use RC or not is up to you, like with C. Also objects are cleaned up deterministicly. Rust tracks objects just as C does, the difference is that the ownership of rust objects are explicitly declared in code such that rust can prove correctness, whereas in C the ownership is ambiguous and a human programmer has to keep track of object ownership in one’s mind.

Reference counting is part of the Rust standard unless (like Swift) later language revisions have made extreme changes. No one is forced to use RC, sure. But avoiding using Rc and Weak references in practice means even more unsafe code. Just look at the FAQ where implementing a graph structure with internal cycles lists ways to do that (in order):

. Rc, Weak references.

. unsafe code.

. Vectors and indices – essentially creating a unmanaged memory area!

. using UnsafeCell which is another type of unsafe code.

[q]

I agree it’s not going to verify the programmer’s logic, but it is enough to prevent the segfaults and pointer errors that address randomization is designed to mitigate.

Regarding “checked exceptions”, that opens up a whole can of worms I don’t want to get into, haha. But in principal run time exceptions handling is orthogonal to compile time pointer verification.

Exceptions can be beautiful. They are however often a PITA.
2017-06-16 11:07 pm

Alfman verbose=1
Megol,

As I wrote in the first comment I’d prefer a safe language with allocation and de-allocation of memory under programmer control. That is dangling pointers, free-after-free and use-after-free errors would be caught either at compile time or run time. That is enough to be a safe language memory vise. [/q]

Sure, in rust allocation and deallocation are under programmer control. This level of control is one of the reasons rust is suitable for low level kernel work whereas a lot of “managed languages” are not.

I have experimented with ownership with borrowing etc. earlier (before Rust existed) and at least for me it isn’t worth the trouble. It is harder for a programmer to reason about ownership without a lot of hand-holding by the compiler, something Rust doesn’t provide. IMHO of course.

The thing is it’s not something that was invented for the benefit of the programming language, short of using full blown garbage collection we *must* do it implicitly (in our heads) to obtain correct memory access semantics. Making it explicit allows the language to verify that actual usages matches intended usage. I get that some C programmers might not like having to declare pointer ownership, but it is still something that we technically have to reconcile in our heads.

Unless that changed after I last looked at it Rust programmers used unsafe constructs for non-trivial datatypes. IIRC.

Depends what you mean, but generally if you are just combining existing primitives, then you can do so without using unsafe code.

[q]Reference counting is part of the Rust standard unless (like Swift) later language revisions have made extreme changes. No one is forced to use RC, sure. But avoiding using Rc and Weak references in practice means even more unsafe code. Just look at the FAQ where implementing a graph structure with internal cycles lists ways to do that (in order):

If you want to criticize what’s included in the standard library, I won’t object. There are things I don’t like about libc and the C++ STL too, however this isn’t normally relevant for kernel code because kernel code usually replaces the standard library with something else; the same is true with rust.

Edit: I’m not trying trying to sell anyone on rust per say, so it’s fine if you don’t like it. The main point I want to put out is that the continued use of C is impeding the state of software security. With the lessons of the past several decades, we can do better than this.

Edited 2017-06-16 23:14 UTC
2017-06-17 12:21 am

kwan_e
I agree it’s not going to verify the programmer’s logic, but it is enough to prevent the segfaults and pointer errors that address randomization is designed to mitigate.

I don’t think segfaults are the problem. If something segfaults, that means the memory protection set up by the OS has worked. It’s when it doesn’t segfault that is worrying.
2017-06-17 12:39 am

Alfman verbose=1
kwan_e,

I don’t think segfaults are the problem. If something segfaults, that means the memory protection set up by the OS has worked. It’s when it doesn’t segfault that is worrying.

Both should be worrying IMHO. A direct segfault is arguably the best outcome and is certainly the easiest to debug! However it’s still possible the fault created a latent condition where the heap continues to be used in its corrupted state for a while before segfaulting later. This is very difficult to debug because you can’t directly track down the segfault to the faulty code that caused it

2017-06-16 11:12 am

kwan_e
ATM systems are generally programmed in C

Well, as more people move to electronic cash, we won’t need ATMs anymore. Crisis averting.

2017-06-16 1:49 pm

Bill Shooter of Bul Platinum Prime
So, the solution to insecure computers in charge of money, is other insecure computers in charge of money.

Got it.

2017-06-16 2:11 pm

Alfman verbose=1
Bill Shooter of Bul,

So, the solution to insecure computers in charge of money, is other insecure computers in charge of money.

Got it.

I assumed kwan_e was being sarcastic, I thought it was funny
2017-06-16 2:18 pm

kwan_e
I wouldn’t call it sarcastic. It’s more self-deprecating as it’s making fun of me misparsing that sentence.

2017-06-16 4:22 pm

Megol

ATM systems are generally programmed in C

Well, as more people move to electronic cash, we won’t need ATMs anymore. Crisis averting.

[q]

😀

(For anyone confused ATM in this case = At The Moment)

2017-06-16 2:41 pm

agentj
Address randomization and ASLR don’t serve a purpose in languages that don’t have exploitable memory leaks in the first place. Clearly the purpose is to mitigate the consequences in the aftermath of these vulnerabilities, but they’re still there and this fails to address the underlying causes.

Despite all the real security problems with C based operating systems, it still has magnitudes more development resources than anything else. Although we have a few OS projects experimenting with robust programming languages to enforce safe memory usage at compile-time, unless the big players start seriously pushing them over the incumbent C kernels, then I’m not sure that any competitor can achieve the critical mass needed to become mainstream.

C is still one of the most feasible choices for OS kernel programming. Those magical “safe” languages from utopia-land always require crap load of runtime written in “evil” C/C++ and assembly.

2017-06-16 3:34 pm

Alfman verbose=1
agentj,

C is still one of the most feasible choices for OS kernel programming. Those magical “safe” languages from utopia-land always require crap load of runtime written in “evil” C/C++ and assembly.

C has the distinction of being the pioneer, and it’s ubiquitous support makes it the defacto standard for virtually all low level work, but it’s not actually that unique in terms of it’s capabilities. Ada, modula 2, or pascal, etc could also be used.

Most C programmers could probably switch to Dlang without much trouble since it was meant to build on C’s familiarity while fixing C’s most notorious problems.

As you know java and .net (and a plethora of scripting languages) add safety through the runtime checks, however developments in modern static languages like rust have eliminated the need for runtime checks and can enforce the same safe memory semantics at compile time, making them suitable for low level & real time work.

If it weren’t for C’s incumbent advantages, and given it’s deficiencies, it wouldn’t be getting much attention if it were invented today. However since we already have so many mission critical C codebases and interfaces, we’re kind of stuck with it as our fundamental building block.

Edit:

If you disagree, do you mind providing an example of something C can do that others cannot?

Edited 2017-06-16 15:37 UTC

2017-06-16 7:49 pm

Zan Lynx
If your system has any program written in “safe” language combined with a buffer-overflow language, then randomization is still useful.

ROP attack code needs access to machine instructions that are already in executable memory. It is a lot easier to do ROP if you know where to find the code you need.

And by ROP I also mean to include things such as function / virtual function pointer overwrites.

Randomized kernel addresses are good for this too because bugs do happen. If it was ever possible to copy data into the kernel through a buggy write() call, for example, and then trick the kernel into using that memory as a structure of some kind with a function pointer, randomized addresses would make that a lot harder. And the attack probably only gets one try, because getting a bad address would crash the system.
2017-06-17 3:43 am

AER
What about Ada?

2017-06-17 4:22 am

Alfman verbose=1
allanregistos,

What about Ada? [/q]

I mentioned Ada it in one post. It wasn’t very accessible outside of government contractors until it was way too late to change the market (C based operating systems were already dominant). I’ve never heard of anyone using it in my circles. Cobal yes, but not ada.

Here’s a good article about the Ada’s popularity plateau:

https://www.nap.edu/read/5463/chapter/3

[q]Hopes for broad commercial adoption of Ada have not been realized, however. Its commercial use has been eclipsed by other languages, such as C, then C++, and, most recently, Java. DOD’s inclusive approach in the development of the language, as well as its promotional campaigns in support of Ada, do not appear to have been successful in fostering adoption of the language beyond defense and other mission-critical applications.

During Ada’s lifetime, DOD’s position in the software market has shifted. Although DOD still has an influence, its share of the market has diminished-not because DOD’s need for software has decreased, but rather because the size of the commercial software market has exploded, generating a corresponding increase in investments in commercial software technology. DOD made significant investments to develop Ada (both Ada 83 and Ada 95)1 and mandated its use on certain DOD projects. The DOD requirement to use Ada appears to have been beneficial for custom software that has no commercial counterparts (e.g., weapon systems and performance-critical C3I software). On the other hand, this policy has frequently been counterproductive in application areas that have strong commercial support. In these areas, DOD’s policy has inhibited DOD from taking advantage of existing commercial infrastructure written in or for other languages.

Interestingly this seems to apply to other languages as well, they often are safer but the industry support for them is so poor that it becomes a pretty big disadvantage.

Most of us know C’s safety record is atrocious, but its preexisting codebase is so enormous that changing it is unpalatable, kind of like a “too big to fail” situation with software.

2017-06-17 6:05 am

sakeniwefu
The main reason they ignore “the problem” is that they are C programmers with deep knowledge of UNIX and very good ones at that. They don’t want to throw it away and start building something in Rust today only to be told next year to switch to Mildew++.

They took 44BSD(something that worked) and slowly turned it into a more reliable system that still does everything you need. They keep innovating and making the system better and better – they call it “polishing a turd”.

Every once in a while, Silicon Valley kids come, create a new language, build a secure micro kernel that can do nothing, argue about it for a year or two and then their ADHD leads them to greener pastures after it becomes a maintenance nightmare that no amount of stimulants can fix.

The real value is in mature systems. Windows, Linux, macOS. Not some toy project.

Of course, then the problem you run into is that you get Byzantine security features designed by a corporate committee of pointy heads attached to a patchwork of legacy code that originally followed toy project standards.

OpenBSD makes security easy and fortunately some of it leaks to mainstream OSs. Rewriting all your legacy code in Rust is not easy. Having to negotiate with your OS so that it allows you to write a file or else just disable everything and broadcast its contents to the Internet is not easy.

2017-06-17 3:56 pm

Alfman verbose=1
sakeniwefu,

The main reason they ignore “the problem” is that they are C programmers with deep knowledge of UNIX and very good ones at that. They don’t want to throw it away and start building something in Rust today only to be told next year to switch to Mildew++. [/q]

There’s no reason we have to switch every year. Obviously you don’t want to switch all the time, but C’s run has been going on for many decades already, we can put off consideration of safer languages indefinitely, but then we’ll always be behind the curve with software security, and that’s a real shame.

They took 44BSD(something that worked) and slowly turned it into a more reliable system that still does everything you need. They keep innovating and making the system better and better – they call it “polishing a turd”.

Haha, you could call it that

[q]Every once in a while, Silicon Valley kids come, create a new language, build a secure micro kernel that can do nothing, argue about it for a year or two and then their ADHD leads them to greener pastures after it becomes a maintenance nightmare that no amount of stimulants can fix.

The real value is in mature systems. Windows, Linux, macOS. Not some toy project.

Popularity is what determines if it’s a toy or not (just think how DOS was arguably a “toy” – many years behind alternatives, yet the PC bundling made it the most popular business platform). Many of us who have worked on these kernels do see the technical problems and we acknowledge they can be done better, however it’s extremely difficult to match the resources of incumbent technologies, which impedes merit based advancement for alternatives.

The difficultly does not lie in developing better operating systems since we have the benefit of hindsight. The difficulty lies in overcoming this catch-22 where operating systems have more resources because they are more popular and operating systems are more popular because they have more resources.

I’ll concede, switching from C is an uphill battle. If C was starting at the same starting line as modern languages, then it would undoubtedly loose in favor of those that don’t have it’s share of problems. However in reality C doesn’t start at the starting line, it starts near the finish line and consequently wins a lot of races despite having all of it’s problems.

Edited 2017-06-17 16:00 UTC