Linux kernel developers discuss dropping x32 support

20 Comments

1. 2018-12-12 11:48 pm

   FlyingJester

   x32 is a really decent idea, but I think it has been hurt by a lack of promotion from distros.

   I also kind of think that it’s time has passed, as increasing memory requirements have made the 64-bit pointer size more pervasively important.

   Edited 2018-12-12 23:49 UTC
• 2018-12-13 1:30 am

  JLF65

  It’s time hasn’t passed at all. The primary use WASN’T programs that only needed 32-bit pointers, it was programs that assumed all pointers were 32 bits that were also a bitch to clean up. A good example of that would be Gens/GS, the Sega Genesis emulator. This emulator was such a bitch to clean that the authors gave up trying and are now rewriting the 64-bit version from scratch. x32 would allow a “64 bit” version without needing to change the pointer code, which would tide people over until the rewrite was done.
• 2018-12-13 3:35 am

  panzi

  Also I wonder if it might make sense on some embedded platforms? Or what processors are common there?
• 2018-12-13 8:25 am

  mattias1

  I am quite sure keeping x32 makes the move from 32 Bit ARM to AARCH64 easier for many embedded systems.
• 2018-12-13 1:06 pm

  ahferroin7

  x32 is x86-only. It’s completely independent from the AArch64-IPL32 support, just like it’s completely unrelated to the MIPS n32 support.
• 2018-12-13 7:01 pm

  FlyingJester

  According to the LKML post, this is absolutely not the case.
• 2018-12-13 7:03 pm

  FlyingJester

  How is the existing x86-on-amd64 layer not sufficient for that?
• 2018-12-14 2:35 am

  JLF65

  Depending on how old it is, you may need libs that are no longer supported. I ran into that with Gens/GS – you’d need to compile it on a 32-bit system with the latest libraries to get a newer 32-bit version to run under the 32-bit libs. And compiling 32-bit is a pain – they simply tell you to chroot to a 32-bit install, or use a 32-bit VM. x32 would allow you to compile as 64-bit on a 64-bit OS with the latest libs and bypass all that hassle.
• 2018-12-13 8:29 am

  kwan_e

  You can still have 64-bits (or rather, at least 48-bits) of storage available. You just need to page memory in and out of your 32-bit address space. Really, only a select few processes on any system needs a truly flat 64-bit address space.

  I mean, we’re talking about 32-bit address space per application on a 64-bit system, not at all like a 32-bit space on a 32-bit system. Even a memory hungry process like a Firefox browsing context is only taking up 1GB at most on my system. You’d only really hit the limit with a badly designed small Java program.

  Interestingly, one of my previous jobs was on mainframe software that had to capture memory dumps really quickly. That involved being hooked into a running/crashing process via a 24-bit address space – 16MB – and trawl through potentially 64-bit address spaces for writing out process state relevant memory to disk, all under one second on a production machine.

  You can definitely address 64-bits of address space without 64-bit pointers.
• 2018-12-14 12:05 am

  zima

  You’d only really hit the limit with a badly designed small Java program.

  Do you have anything in particular in mind? ( http://www.osnews.com/thread?665828 ? )
• 2018-12-13 11:29 am

  dsmogor

  I guess it was waiting for some industry endorsement (in the embedded) and that haven’t happened. Specifically if Intel wasn’t interested what can you do.

  Not every decent idea catches on.
• 2018-12-13 1:09 pm

  ahferroin7

  The issue isn’t so much lack of support from distros as how buggy it was and how software has to quite often be rewritten to the same level that is seen for porting to real 64-bit just to support it. The lack of support from distros is a direct consequence of those two facts.

  It really comes down to the fact that the approach x32 takes is problematic in and of itself (compare this for example to AArch64 IPL32 support (not merged yet) and MIPS64 n32 ABI support, both of which are exponentially cleaner).
• 2018-12-13 2:11 pm

  Alfman verbose=1

  ahferroin7,

  The issue isn’t so much lack of support from distros as how buggy it was and how software has to quite often be rewritten to the same level that is seen for porting to real 64-bit just to support it. The lack of support from distros is a direct consequence of those two facts.

  Can you elaborate on what was buggy? It shouldn’t require much more than a recompile (assuming the distros support it). I’d like to understand what you are referring to, however my current impression is that the main motivation for removing it is simply that it’s an extraneous mode that doesn’t get used very much.
• 2018-12-13 2:18 pm

  ahferroin7

  Anything that calls system calls directly (which is actually more than most people think) may need manual changes to work correctly, as does anything that makes assumptions about the sizes of types.

  strace and other tools that trace system calls from userspace have consistently had issues with x32 because of how it handles the system calls.

  Almost everything in the kernel that cares about syscall numbers (for example, the auditing infrastructure and seccomp) has issues too, and anything that uses those interfaces needs special care to work properly.

  Pretty much, the bugs and code comparability issues arise from the x32 ABI not matching either 64-bit or 32-bit x86 for structure sizes consistently, as well as having a whole slew of it’s own independent syscall numbers.
• 2018-12-13 7:08 pm

  Alfman verbose=1

  ahferroin7,

  Anything that calls system calls directly (which is actually more than most people think) may need manual changes to work correctly, as does anything that makes assumptions about the sizes of types.

  strace and other tools that trace system calls from userspace have consistently had issues with x32 because of how it handles the system calls.

  Almost everything in the kernel that cares about syscall numbers (for example, the auditing infrastructure and seccomp) has issues too, and anything that uses those interfaces needs special care to work properly. [/q]

  These things don’t qualify as “buggy” to me though. Relying on direct programming could break on PPC and ARM too, but it doesn’t mean that those platforms are buggy. When you, as a dev, choose to bypass standard libraries on any platform, you implicitly become responsible for maintaining your code on new platforms/architectures. It’s not the fault of the architecture that the application developer chose not to use portable code. Honestly I don’t think anyone should label this as a shortcoming of x32 any more than any other architecture (including x86_64).

  [q]Pretty much, the bugs and code comparability issues arise from the x32 ABI not matching either 64-bit or 32-bit x86 for structure sizes consistently, as well as having a whole slew of it’s own independent syscall numbers.

  I’ll grant you data type sizes can be problematic when it comes to data structures, although we should recognize that it’s not really specific to x32. C has a long history of ambiguity in this area, which is why programmers should not assume on types having specific sizes.

  https://stackoverflow.com/questions/18901080/why-is-the-sizeofint-si…

  Arguably this exposes improper data type usage by programmers. The best advice is for programmers to learn to use size specific data types as needed since this will allow their code to be more portable in general.

  http://www.cplusplus.com/reference/cstdint/
• 2018-12-13 7:31 pm

  ahferroin7

  These things don’t qualify as “buggy” to me though. Relying on direct programming could break on PPC and ARM too, but it doesn’t mean that those platforms are buggy. When you, as a dev, choose to bypass standard libraries on any platform, you implicitly become responsible for maintaining your code on new platforms/architectures. It’s not the fault of the architecture that the application developer chose not to use portable code. Honestly I don’t think anyone should label this as a shortcoming of x32 any more than any other architecture (including x86_64). [/q]
  Normally, I’d agree. The problem is that unlike similar sub-arches like AArch64-ILP32 and MIPS64 n32, x32 is completely inconsistent in how it’s handling system call numbers. They don’t map linearly, they don’t consistently get updated as needed, etc.

  Also, it’s not the fault of the application developer that they’ve been functionally forced to bypass the system libraries because the system libraries refuse to provide functions for interacting with the interfaces they need. Needing to make direct system calls is unfortunately normal on Linux because of glibc and other libc implementations not reliably providing interfaces for them, sometimes even for years after the system call has been introduced.

  I’ll grant you data type sizes can be problematic when it comes to data structures, although we should recognize that it’s not really specific to x32. C has a long history of ambiguity in this area, which is why programmers should not assume on types having specific sizes.

  Again, it’s not anywhere near as much of a problem with other similar sub-arches. AArch64-ILP32 and MIPS n32 both use type size rules consistent with their purely 32-bit architecture variants (ARM OABI and EABI and MIPS o32 respectively), while x32 is consistent with neither i486 or x86_64 when it comes to type widths (some are like i486, others are like x86_64). This ironically ties in with why the system call tables are all fubar on x32, they use some of the x86_64 system calls directly instead of just having their own dedicated set.

  [q]Arguably this exposes improper data type usage by programmers. The best advice is for programmers to learn to use size specific data types as needed since this will allow their code to be more portable in general.

  This is great advice in theory, except size-specific types aren’t reliably portable. Using them is perfectly fine if you’re only ever running on one platform, but the moment you care about portability, things get complicated, because not everyone uses the same C or C++ standard revision.

  Even aside from all of this, there have been numerous other bugs over the years. There’s at least two extant ones I know of, a buffer overrun in the select() API on x32 (which has a proposed fix submitted but not yet committed), and the fact that glibc’s get_phys_pages() function doesn’t work correctly on x32 (because it assumes that certain structure fields aren’t longer than the `long` type). There are probably others I don’t know of right now, I stopped caring actively about x32 a few years back because it was just too much of a pain to keep a system working with it that did anything more complicated than building software.
• 2018-12-13 9:55 pm

  Alfman verbose=1

  ahferroin7,

  Normally, I’d agree. The problem is that unlike similar sub-arches like AArch64-ILP32 and MIPS64 n32, x32 is completely inconsistent in how it’s handling system call numbers. They don’t map linearly, they don’t consistently get updated as needed, etc. [/q]

  I guess we could debate whether x86_64 should have embraced the same syscalls as i386. I’m sure the linux kernel mailing lists would have some insight into this decision somewhere, although at this point it’s ancient history.

  Also, it’s not the fault of the application developer that they’ve been functionally forced to bypass the system libraries because the system libraries refuse to provide functions for interacting with the interfaces they need. Needing to make direct system calls is unfortunately normal on Linux because of glibc and other libc implementations not reliably providing interfaces for them, sometimes even for years after the system call has been introduced.

  While I understand the argument, I can’t say this has been my experience in practice. Do you have a specific example of normal userspace code being forced to program directly to syscalls?

  I do remember one time when I had to rely on syscalls directly, but to be fair it was a temporary condition with a relatively new syscall. However I maintain that using syscalls directly is a hack, not something I would ever expect to be portable (or even good practice for a single architecture).

  This ironically ties in with why the system call tables are all fubar on x32, they use some of the x86_64 system calls directly instead of just having their own dedicated set.

  I’m open to being convinced otherwise, but I would think it should use the x86_64 syscalls. IMHO it makes more sense to think of x32 as an operating mode of x86_64 than a new unique architecture or x86.

  This is great advice in theory, except size-specific types aren’t reliably portable. Using them is perfectly fine if you’re only ever running on one platform, but the moment you care about portability, things get complicated, because not everyone uses the same C or C++ standard revision.

  Hmm. I disagree, they are very widely available in part because they’re defined by the C99 standard. In the rare event you’ve got a non-compliant compiler, you can still define them yourself.

  https://en.wikibooks.org/wiki/C_Programming/stdint.h

  stdint.h is a header file in the C standard library introduced in the C99 standard library section 7.18 to allow programmers to write more portable code by providing a set of typedefs that specify exact-width integer types, together with the defined minimum and maximum allowable values for each type, using macros[1] . This header is particularly useful for embedded programming which often involves considerable manipulation of hardware specific I/O registers requiring integer data of fixed widths, specific locations and exact alignments. stdint.h (for C or C++), and cstdint (for C++) can be downloaded or quickly created if they are not provided.

  [q]Even aside from all of this, there have been numerous other bugs over the years. There’s at least two extant ones I know of, a buffer overrun in the select() API on x32 (which has a proposed fix submitted but not yet committed), and the fact that glibc’s get_phys_pages() function doesn’t work correctly on x32 (because it assumes that certain structure fields aren’t longer than the `long` type). There are probably others I don’t know of right now, I stopped caring actively about x32 a few years back because it was just too much of a pain to keep a system working with it that did anything more complicated than building software.

  These are exactly the kinds of bugs I was initially inquiring about

  I do understand that there just isn’t much motivation for developers to use x32 over full x86_64. Only someone who cares an awful lot about memory/cache efficiency would care about the difference to not use x86_64. That’s not many people these days, haha.

  Edited 2018-12-13 21:58 UTC
• 2018-12-14 12:31 pm

  ahferroin7

  Do you have a specific example of normal userspace code being forced to program directly to syscalls? [/q]
  Depends on what you mean by ‘normal’ userspace code. `gettid()` is probably the best example of a generic system call that has no glibc wrapper, `futex()` is another one. Most of the stuff that is persistently not covered by glibc is usually not used much, but that doesn’t mean there aren’t uses for it. `renameat2()` is probably the best example of something that fits your requirement, as it provides something that’s generally useful to most userspace code dealing with file operations (namely, the ability to atomically fail a rename if the target exists). Note that I’m ignoring stuff here that has it’s own library (like `seccomp()`).

  [q]I’m open to being convinced otherwise, but I would think it should use the x86_64 syscalls. IMHO it makes more sense to think of x32 as an operating mode of x86_64 than a new unique architecture or x86.

  The problem is that x32 is a distinct ABI. Using all of the x86_64 syscalls isn’t an option because it forces use of 64-bit pointers and wide integers in many places, which x32 is trying to avoid.

  Note that I’m not saying that it’s a bad thing that x32 reuses code from the x86_64 syscalls, I’m saying it’s a bad thing that it uses the exact same syscall numbers directly for some of it’s syscalls. IOW, certain syscalls don’t have a dedicated syscall number for x32, because they get called using the x86_64 syscall number instead. This makes maintenance of the ABI a serious pain in the arse because it means that the syscall numbers are sparse instead of just being a linear array.
• 2018-12-14 5:10 pm

  Alfman verbose=1

  ahferroin7,

  Depends on what you mean by ‘normal’ userspace code. `gettid()` is probably the best example of a generic system call that has no glibc wrapper, `futex()` is another one. Most of the stuff that is persistently not covered by glibc is usually not used much, but that doesn’t mean there aren’t uses for it. `renameat2()` is probably the best example of something that fits your requirement, as it provides something that’s generally useful to most userspace code dealing with file operations (namely, the ability to atomically fail a rename if the target exists). Note that I’m ignoring stuff here that has it’s own library (like `seccomp()`). [/q]

  Is there a reason you wouldn’t use pthreads, which is explicitly portable, instead of linux specific features like “gettid”?

  The goal of glibc is to provide a standard portable C interface. The maintainers are deliberately hitting the breaks on linux specific functionality that’s not portable. Consequently if you want portable software, you probably should stick to glibc functionality.

  Here’s a very good article about this very thing:

  https://lwn.net/Articles/655028/

  On the face of it, adding a new system-call wrapper should be a simple exercise. The kernel has already defined an API for the system call, so it is just a matter of writing a simple function that passes the caller’s arguments through to the kernel implementation. Things quickly get more complicated than that, though, for a number of reasons, but they all come down to one root cause: glibc is not just a wrapper interface for kernel-supplied functionality. Instead, it provides a (somewhat standard-defined) API that is meant to be consistent and independent of any specific operating system.

  There are provisions for adding kernel-specific functions to glibc now; those functions will typically fail (with errno set to ENOSYS) when called on a kernel that does not support them. Examples of such functions include the Linux-specific epoll_wait() and related system calls. As a general rule, though, the glibc developers, as part of their role maintaining the low-level API for the GNU system, would like to avoid kernel-specific additions.

  This concern has had the effect of keeping a lot of Linux system-call wrappers out of the GNU C Library. It is not necessarily that the glibc developers do not want that functionality, but figuring out how a new function would fit into the overall GNU API is not a straightforward task. The ideal interface may not (from the glibc point of view) be the one exposed by the Linux kernel, so another may need to be designed. Issues like error handling, thread safety, support on non-Linux systems, and POSIX-thread cancellation points can complicate things considerably. In many cases, it seems that few developers have wanted to run the gauntlet of getting new system-call wrappers into the library, even if the overall attitude toward such wrappers has become markedly more friendly in recent years.

  Nevertheless, I agree sometimes software can benefit from non-standard linux-specific features. In these cases I think it makes a whole lot of sense to isolate all those features in their own source files and enable or emulate them using if-def blocks. This way all of the non-portable code remains in one spot and is easy to identify & update.

  [q]The problem is that x32 is a distinct ABI. Using all of the x86_64 syscalls isn’t an option because it forces use of 64-bit pointers and wide integers in many places, which x32 is trying to avoid.

  I think this is incorrect. Just because the userspace process is using 32bit pointers doesn’t mean that the 64bit ABI can’t be used setting the high bits to zero. I’m looking at all of the syscalls you’ve mentioned and it looks like they share the same syscalls between x86_64 and x32. I don’t see a problem.

  Edited 2018-12-14 17:15 UTC
2. 2018-12-14 12:20 am

   cpcf

   That headline gave me a heart attack!