Rethinking the OS

Every hard-core OS aficionado has done it: Laid out a grand scheme for creating the perfect OS. Taking all the best features and attributes from the OSes we love, and making sure to assiduously avoid the pitfalls of the OSes we don’t. Maybe our goals were modest, and we just wanted a slightly tweaked version of an existing OS. But sometimes we’re feeling ambitious, and we have large, creative ideas for revolutionizing computing. Long-time OSNews reader and contributor J. Scott Edwards just couldn’t help himself, and he has set about to not only plan, but to try to build his dream OS.

This is my third attempt at writing this all down. The first was too detailed. The second was still too long. This time I’ve decided to just lay out my ideas, and try to keep them clear and brief. This is more or less just an overview. There are just too many details to try to squeeze them all in here. Many of the ideas are not really new, some have been talked about before. I’m just trying to figure out how to put them together into one system.

I’ve been working with computers for 30 years and they certainly haven’t gotten any easier. In fact I’m pretty sure they have gotten harder. True, I’m doing things now that I never dreamed of doing then. But, it feels like, while they’ve become 1,000 times more powerful, it seems like they have also become 1,000 times more complex.

I’ve tried almost every major operating system out there and I always seem to run into difficulties. So I have been thinking: “What if we started over from scratch and re-thought how an operating system is put together?”. One of the first operating systems I ever dealt with was Exec 8 on a Univac 1108. It contained most of the concepts used by all of today’s major operating systems. Now you are probably thinking “it didn’t have a GUI!”. That’s very true. What I’m talking about are the underlying concepts: processes, files, how programs run and interact, things like that.

I have been brainstorming for some time now, and I have some ideas that I would like to share. They are just ideas right now, some or all of them may never work. It is possible that I have failed to think of some aspect of an idea that makes implementation infeasible. But I feel the need to get them out there for several reasons. One, I am hoping that if they are published somewhere they will quit going around and around in my head and I can get some sleep. Two, I don’t want a scene like the one with Microsoft and SenderID

One last thing I would like to say up front. I am going to mention some software that I use, as examples, such as Linux, OpenBSD, gcc, K3B, etc. Please don’t take that as my disparaging that particular piece of software. I am just using what I am most familiar with. I still think it is excellent software, which I use everyday and will continue to use for the foreseeable future.

So without further ado, I guess I’ll jump right in with both feet.

Simplicity

I have been reading the book Better, Faster, Lighter Java and although the author is talking specifically about Java, he might as well be talking about computing in general.

Chapter 2 is titled “Keep It Simple” and explains the value of simplicity. It even goes on to say that “Simplicity may be the core value”. I wish I could reproduce or link to the first section of chapter 2 because I think it applies to all of computing.

So that is my primary goal is to keep the entire system as simple and elegant as possible. When complexity is inescapable, contain it in one place, and don’t allow it to infect the rest of the system.

Flat Files and Relational Databases

The biggest thing that I can think of that adds complexity to computer programs is storing the data. It seems to me that any non-trivial program has a good portion of the code just to mess around with files. There are configuration files and data files, which most of the time cannot be used directly by the program, and have to be interpreted and converted into the internal format of the application.

Another alternative to storing data in flat files is to store them in a database, most commonly a relational database. While this has several advantages over the flat files, if using a relational database, the program has to do queries on the database and then usually reformat the result into the program’s internal data structures. This is really messy when using Object Oriented programming. There are of course Object Oriented databases, but I am not aware of any that have gained wide acceptance, especially ones that are Open Source like MySQL and PostgreSQL.

My idea is to have all of the information stored on the disk in the native Object format of the program. That way instead of having to constantly convert and interpret data, the application can just access that object directly. And when that data is needed in a flat file format, you have a converter App (object) that can access the internal data and convert it to a flat file type of format.

For example, let’s say you have some compressed (with Ogg Vorbis or whatever) audio objects on your computer. And you want to burn an audio CD which can be played in a normal audio CD player. You would create a playlist object and connect the output (more on this later) to the input of a Ogg Vorbis converter object and then into the Audio CD burning object.

Where Is It Stored?

Another problem that I would like to solve is the “where is that data?” or “what did I do with that file?” problem. Just yesterday I was looking for a spreadsheet that had some information I needed in it. At one time I had it stored on my desktop machine. But I’ve changed that machine several times in the past year and reloaded different operating systems several more times. So I finally put it on my file server. But when I looked for it, I didn’t see it anywhere. Worse yet I couldn’t remember exactly what it was named. I had to do many searches before I finally found the stupid thing. I probably wasted 20 minutes looking for it and as time went by I started worrying that I had lost it and wondering when (if?) I had backed it up. I just don’t need this kind of stress!

My thought on that is to remove the distinction between different partitions, disk drives, file servers, CD and DVD media, etc. No more C:, D: and E: partitions. No more host7:/shared… No more mounting the CD drive. It’s all just have one big object storage space. If it’s out there the machine can find it. If it’s public information it can even be obtained over the Internet. If it was stored on a tape, CD or DVD backup and that media is not in the drive, the OS will tell you on what media that object was stored and you can find that media and insert it and the App can continue without even knowing what happened. If you don’t have access to that media at that time you should have the option of stopping that App right where it is and restarting it later at the same exact point when you have the media (more on this later).

**It seems that Google, Apple, and others are moving toward making this a reality on today’s operating systems, so that’s good news.

Relating to Each Other

Another problem with Flat Files is the lack of information about the data in the file. The BeOS had a database built into the file system with was nice addition, and I understand that Microsoft’s Longhorn was going to have some sort of database built in before it was scaled back. I also recently discovered that the Linux ext3 filesystem allows additional “attributes” to be added to a file.

But I would like to go beyond all of that. My idea is to take an object as defined in current languages like C++ and Eiffel and add some additional information to it. Information such as when the object was created, who created it, md5 checksum, some protection bits, the usual meta-data that a file would have in the OS now.

In addition to those, the most significant addition is the reference table. This is a table maintained by the OS which lists all of the objects which reference that object. That provides a couple of benefits. First it allows you to easily find any object that references a given object. Second it allows one to determine which objects are still in use on a system. If no other active object on a system refers to an object (other than the directory) it could be moved to back up storage and deleted from the active system.

For example, going back to the music example, all of the music objects would reference the artist that created them. All of the Beatles tracks would reference the Beatles Artist object. The Beatles Artist object would not specifically have references to songs because songs would be only one type of object that could refer to the Beatles. We wouldn’t want to encumber an Artist object with explicit references to every possible type of object that could refer to it. We don’t even know what every type of object that could reference it even is. And if we did that we would have to change the Artist class every time some other class was created that referred to an artist class. Instead every object will just have a generic list that can point to any type of object (the “ANY” class in Eiffel). This list will be automatically maintained by the OS and updated as the program runs. And when you want to find all of the songs you have by The Beatles, you simply get the Beatles object and search it’s reference list for objects of the class “Track”.


Versions

I have also been considering keeping versions of each object when that object is modified. So instead of actually changing that object, it clones it and then modifies the clone. That way if you have made a mistake and need to go back, or if you want to see what changes have been made you can view the differences. I.E. you have built-in CVS.

To go along with this there should be some mechanisms for cleaning up old objects that are no longer required. For instance if you have been modifying some source code and you had version 7 that worked and then you made several changes in versions 8, 9, 10, 11, and 12. And 9, 10 and 11 had bugs that you don’t want to keep. When you are finished you should be able to remove those unneeded versions.

Disk space is extremely cheap now. I just checked and you can buy a 200 Gigabyte hard drive for just over $100. And Sony’s Blu-ray discs will be able to hold 23 Gigabytes of data per layer. Given that kind of storage space I think saving multiple copies of your work just makes sense.

Security

Security is one aspect that I haven’t gotten completely figured out. The things I have thought about are as follows:

  • Different parts of an object can have different levels of security. For example if you had a Person object, some of the info might be public, for example the person’s name and e-mail address. Then other parts might be accessible by a smaller group such as your co-workers, for example your cell phone number. Then other information may only be accessible by a select few, like your Social Security Number.
  • All non-public data, when not in RAM, will be encrypted. When that data is stored on any media or transferred over the network it will be encrypted.
  • There needs to be some sort of group mechanism that allows people to be grouped, similar to the Unix/Linux group mechanism.
  • There should be different levels of security as well. For example, if you are connected as yourself, you can access some data just based upon your login authentication. Then a higher level of security that even though you are logged in as you, you have to put in another pass phrase to access that specific data. Possibly an even higher level of security that uses one of those dongles or something. Mac OS X does this to a certain extent with regular access and “administrator” access to install software and make major configuration changes. I would like to have these multiple levels within a single person’s account. For example, you may have a normal level of security where you are writing a report on global warming. But you would want to have a higher security level for a object that contains your Social Security number. I want to incorporate something like SE Linux which defines “policies” of who can do what. For example there should not be one all powerful user who can read any object on the system, even the ones containing your SSN.

RAM Caching

I also want to change the way RAM is used. I want to use the main RAM simply as a fast cache for the objects that are stored on the Hard Disk. (Or really if there was no hard disk, objects loaded from the network.) So that the whole image of a running application is kept up to date as possible on the disk. As an object is executing in RAM, it should be copied back to disk as soon as possible.

There are several benefits to this scheme. One is that if the power is lost, the loss is less significant. Which brings me to one of the other ideas…

More Continuity

Another thing I want to try is to make applications have more continuity. It seems like applications now still work in the same way that they did on batch processing systems years ago. They start up completely new each time read their configuration files, inputs, do their processing and write their outputs and stop.

Instead of starting up from nothing each time and then being terminated as if it never existed. I think it would be better if (at least some) applications kept their state intact. It would be more like you turned them on and off.

For example, I wonder if a word processing program couldn’t be made more like a typewriter? Say you have a piece of paper in the typewriter and you are in the middle of typing something. You get a phone call and get interrupted and have to fly to Istanbul, you shut it off (assuming it’s an electric typewriter) and leave it. Six months later you come back, turn it back on and it’s right where you left it.

While it’s true that most word processors have a “recent documents” feature that you can re-open the document it is usually not exactly the same thing. What if the power fails, can you easily get back to where it was?

I think it should be possible to have applications that you can just turn on and off and they are just like you left them.

Deeper Down

Most of what I have discussed so far is at a higher level, really on top of the operating system. In fact my initial plan for doing a proof of concept is to build the above Object store on top of Linux and/or one of the BSD’s.

But then I want to take the Object paradigm further down. I would like to have all of the operating system be object oriented, except the very core. I envision having something like a Java Virtual Machine at the core. When the machine powers up it loads the virtual machine, most likely with something like a Just In Time compiler for speed. Everything after that is object oriented.

What’s in a Language

I know that some people disagree that Object Oriented programming has delivered what it was supposed to. But I have found that if you use the right OO language it has significant benefits to productivity.

Ok, I’m going to break one of my tenets and get down on C++ a bit here. I know it’s possible to write good code in C++. It’s possible to write good code in Assembly Language. The thing is the amount of difficulty involved. I have worked on many C++ projects, but I have never worked on one where I felt that it was productive. Even the projects I have done all by myself in C++ have been disasters. I have found that usually one of two things happens, the developers stick to a small subset of the language so it’s really just C with a few objects thrown in here and there. Or it becomes so complex most people have difficulty understanding it and a nightmare to add features or maintain. I remember on one project a developer complaining that to add one simple feature he had to modify 42 different files. When code becomes that complex is a bad thing. In his book Thinking in C++, Bruce Eckel even says that C++ is more complicated than Ada. There are many sites on the internet about the problems with C++, for example: Why C and C++ Are Bad. In my opinion a programming language should make it as simple as possible to write code that is well designed and easy to maintain. I feel C++ falls short in this area.

I know 99% of the programmers in the world are going to tune out right now. But in my experience I have found that Eiffel and the Design by Contract an absolute boon to programming.

Let me give an example of where I think Design by Contract would be a major improvement. One of the biggest causes of security holes in software is the buffer overrun. And I think the fault lies in the C language, it does not check anything. And programmers are not perfect, they make mistakes and C allows them to make mistakes. I believe if you use a language like Eiffel, that checks for error conditions, software would be more secure. (One caveat here is that you could not turn off these checks off, they must always be in place. I would gladly trade a slight performance hit for security.)

Now that I’ve said all of that, I want to say that Eiffel is not exactly what I want to program this in. It is the best language that I have used. It embodies simplicity, correctness, and ease of programming. Programs are easier to debug and need less debugging. But I would like to change the programming paradigm as well.

One think I would like to do is have the documentation embedded in the programming language. (Now I’m sure that any remaining programmers are tuning out) I think there needs to be some way that the documentation and help are programmed right in the programming language. It should not be a separate thing. I hate documentation as much as the next programmer, here you’ve gone to this incredible amount of work writing this great program, and now you have to go back and document it. Which may be as much or more work than writing the program. It sucks! So I think it would be better if creating the documentation and help were built right into the program, in the same place. Then it would be done at the same time as the program.

And along those lines, I think it should be such that you can embed other types of things besides text in the program itself. I think you should be able to just put a drawing, a chart, or even a picture right in the source code. I think the source code should just be another object which can contain any other types of objects.

One thing that occurred to me about storing the code as objects instead of text is that you could display/edit the code in your preferred format. If you like the curly brace scene as seen in the C family of languages, you could view the program like that:

 if (a == b) { c++; } 
It could even prevent all those arguments over where the curly braces go:
 if (a == b) { c++; } 
And if you prefer an easier to read syntax:
 if a = b then c := c + 1; end 


Object Editor

Because everything is stored as an object, there needs to be simple editor. Something very similar to a text editor, that allows you to view and manipulate objects. It should work on any object (assuming you have permission to view and edit that object). It of course won’t be able to view or edit parts of the object that are encrypted, unless you have the key.

For example: many programs have a configuration file, now they will have a configuration object and you can change the configuration this way. Similar to editing a configuration file with a text editor now.

And the OS Has Continuity Too

I think the same thing should be possible with the whole operating system that is done with the apps. It should have more saved state when it starts up. It should basically be restored to where it was when you turned it off.

My memory is so bad that I pretty much have to leave my machines running all the time. That has been one nice thing about Linux that I have been able to leave it running for months. Then when I come back it’s just as I left it. The bad thing is when the power fails. It usually takes me an hour to get some semblance of what I was doing. And then it is never exactly everything I had going on. I just can’t see why it’s not possible to have the machine almost in the same state it was in when the power was lost. Obviously if you are in middle of typing something, some stuff is going to be lost. But why do we have to start over from nothing?

Modules and Software On Demand

One of the problems I’ve had on several occasions in Linux is the installation of software. Oft times, I have wanted to install a piece of software to do one function. But when I went to install it say via RPM, it had a dependency list a mile long. Sure “apt” and “yum” in most cases can handle the dependency thing. But it still bugs me to have all of that additional stuff, that I never use, on my hard drive wasting space.

As an example the other day I had done a minimal installation of SuSE on a machine (because it had very little disk space). I wanted to use K3B just to create data CD’s and DVD’s. However, when I tried to install it, it wanted MP3 libraries, Ogg Vorbis libraries, FLAC libraries, the list was quite long. I did not need any of this stuff on that machine, it was completely superfluous.

I think the software should be set up in smaller modules which will work independently. I further think that only a minimal amount of software should be installed on the system when it is started up. Then when you need to perform a certain task that software is installed automatically (with your authorization of course) and ONLY the software you actually require gets installed.

I alluded to this before, but I want to mention it again. I would like to develop a mechanism for independent modules to communicate with each other each other. Similar to the way in Unix/Linux you can hook the standard input and standard outputs together in a pipe. But have other types of interfaces (audio/video/whatever) and connect them at run time to perform some other function. For example if you had say, a filter plug-in, instead of just being able to use that plug-in inside the audio editor, you could just use the plug-in at the OS level and plug it between the output of some device and the mixer. Or you perhaps you could plug it into the mixer.

Can’t Change Everything

While it would be tempting to try to change everything about a computer to see if it could be improved upon. I have decided there are limits to what makes sense.

The hardware is well, still “hard”. It would be expensive to change the hardware and it would make it impossible for anyone without the special hardware to use the software. Making the whole point of the open source software moot. So we have to stick with the existing hardware.

I feel like the same holds true for the networking. It doesn’t make sense to use something other than TCP/IP. TCP/IP seems to be flexible enough to support everything I have envisioned. Then we can communicate with the rest of the world. That can only be a good thing.

Endian-ness

One of the problems that I have not decided how to handle yet is the endian-ness of different machines and data. Since the machine has access to the class definition of each object, it could automatically convert data to match the current processor.

The dilemma is which one should it be stored in? Or should it always be stored on disk in the format for the CPU the disk is connected to? It is tempting to store the information always in one format, say big-endian to match the network byte order. But then since the most common machines (x86) are little-endian, they would have to always have to take the hit.

Another alternative, would be to do as is done now and store the data on the hard drive in the native order for the CPU. This has speed benefits, the data only needs to be reversed when it’s going over the network.


User Interface

I haven’t worked very much on the details for the GUI. I would certainly like to go beyond the 2D interface. Sun’s Looking Glass project looks kind of fun. It will be interesting to see what happens there.

I had envisioned something beyond that, where you didn’t really have windows but more like different rooms, like a music room, a library, a tv room, etc. But as I said I haven’t put as much effort into this as the underlying pieces.

One thing I would like to come out of having software built out of separate modules that could be connected is that an app could have different interfaces without having to change the app.

For example, an application would have a control connection to it. It would not have the interface coded into the app itself. Then you could connect an interface, either GUI or command line, to it. Depending upon your preference.

Where to Start?

This is the big question. Should I take the top down approach or a bottom up approach? I read once that “Real programmers do it middle out!”. Or perhaps start at both ends and see if I can meet up in the middle somewhere.

Actually my current plan is to build the Object store on top of an existing OS. Find out if it is going to work at all and how well it works. In fact I don’t see any reason that one couldn’t build practically the whole OS on top of another OS, similar to User Mode Linux.

One other thing I plan to investigate is how well the object paradigm works at lowest levels of the operating system. It seems like a natural fit, but I could be wrong. One of the things in Linux that seems kind of strange to me, although it seems to work pretty well, is the simulating of a SCSI device for IDE and SATA drives. It does seem like it’s extra work.

It appears to me that came from a program such as ‘cdrecord’ that wanted to talk to a SCSI device. Although I’m sure there are other good reasons. But in Object Oriented land it would seem like the application (cdrecord) could be written to talk to a generic object (CD-Drive), which could be instantiated as either a SCSI or IDE object. So I want to investigate this further to see if that would be possible or if I’m up in the night.

I also plan to use as much of existing Open Source as possible. It makes no sense to have to write all device drivers from scratch (and it would take years). So I want to see what drivers can be used from Linux or any of the *BSD’s and perhaps wrap them with an object interface. This seems like the beauty of Open Source, that you don’t have to start completely from scratch. At a minimum you can at least look at how someone else has done something and try to improve upon it. Standing on the shoulders of giants if you will.

Wrap Up

Well, this is already longer than I had intended it to be. And there are more details that I would really like to include. I have started a project at SourceForge for further details. It will be at http://nwos.sourceforge.net/ when I get it set up. I intend to put up copies of my hand scribbled notes, some drawings and some more details in the next few days.

As I stated in the introduction, none of this may actually work. Or it may be so excruciatingly slow that it’s unusable. On the other hand with processors now running over 3 GHz, I cannot imagine that would be the case. Seriously, when I’m balancing my checkbook the CPU is probably not too burdened. I know sometimes (like rotating photographs) that you need all the speed you can get. But in the general case I would gladly trade some horsepower for some ease of use and reliability.

In the end this is really all about the famous “Itch I have to Scratch”. Since I deal with computers all day, almost everyday, I constantly wonder “does this have to be this complicated?” and “is there a better way?”. Perhaps there isn’t, perhaps computers, difficulty, and complexity all go hand in hand. But I just have to see for myself. I really cannot give up the idea that there must be a simpler way. It keeps haunting me. Therefore, I have to prove to myself that it either can or can’t be done.

86 Comments

  1. 2004-12-17 7:20 pm
  2. 2004-12-17 7:26 pm
  3. 2004-12-17 7:26 pm
  4. 2004-12-17 7:27 pm
  5. 2004-12-17 7:29 pm
  6. 2004-12-17 7:32 pm
  7. 2004-12-17 7:38 pm
  8. 2004-12-17 7:40 pm
  9. 2004-12-17 7:41 pm
  10. 2004-12-17 7:54 pm
  11. 2004-12-17 7:58 pm
  12. 2004-12-17 7:58 pm
  13. 2004-12-17 8:03 pm
  14. 2004-12-17 8:06 pm
  15. 2004-12-17 8:22 pm
  16. 2004-12-17 8:28 pm
  17. 2004-12-17 8:31 pm
  18. 2004-12-17 8:44 pm
  19. 2004-12-17 9:00 pm
  20. 2004-12-17 9:10 pm
  21. 2004-12-17 9:12 pm
  22. 2004-12-17 9:22 pm
  23. 2004-12-17 9:28 pm
  24. 2004-12-17 9:47 pm
  25. 2004-12-17 9:53 pm
  26. 2004-12-17 10:46 pm
  27. 2004-12-17 11:05 pm
  28. 2004-12-17 11:05 pm
  29. 2004-12-17 11:10 pm
  30. 2004-12-17 11:12 pm
  31. 2004-12-17 11:18 pm
  32. 2004-12-17 11:41 pm
  33. 2004-12-17 11:44 pm
  34. 2004-12-17 11:49 pm
  35. 2004-12-18 12:01 am
  36. 2004-12-18 12:33 am
  37. 2004-12-18 1:01 am
  38. 2004-12-18 1:02 am
  39. 2004-12-18 1:07 am
  40. 2004-12-18 1:38 am
  41. 2004-12-18 2:41 am
  42. 2004-12-18 3:23 am
  43. 2004-12-18 3:37 am
  44. 2004-12-18 4:59 am
  45. 2004-12-18 5:08 am
  46. 2004-12-18 6:48 am
  47. 2004-12-18 7:55 am
  48. 2004-12-18 8:30 am
  49. 2004-12-18 8:50 am
  50. 2004-12-18 10:05 am
  51. 2004-12-18 10:29 am
  52. 2004-12-18 10:51 am
  53. 2004-12-18 12:15 pm
  54. 2004-12-18 12:16 pm
  55. 2004-12-18 12:44 pm
  56. 2004-12-18 12:57 pm
  57. 2004-12-18 1:28 pm
  58. 2004-12-18 3:26 pm
  59. 2004-12-18 3:31 pm
  60. 2004-12-18 4:03 pm
  61. 2004-12-18 4:17 pm
  62. 2004-12-18 4:52 pm
  63. 2004-12-18 6:07 pm
  64. 2004-12-18 6:12 pm
  65. 2004-12-18 8:16 pm
  66. 2004-12-18 8:33 pm
  67. 2004-12-18 10:30 pm
  68. 2004-12-19 1:13 am
  69. 2004-12-19 3:51 am
  70. 2004-12-19 10:07 am
  71. 2004-12-19 5:43 pm
  72. 2004-12-19 6:35 pm
  73. 2004-12-19 7:42 pm
  74. 2004-12-19 7:45 pm
  75. 2004-12-19 8:24 pm
  76. 2004-12-20 1:01 am
  77. 2004-12-20 6:06 pm
  78. 2004-12-22 8:26 am
  79. 2004-12-22 2:54 pm
  80. 2004-12-22 3:01 pm
  81. 2004-12-22 4:31 pm
  82. 2004-12-24 2:30 pm
  83. 2004-12-24 2:33 pm
  84. 2004-12-24 2:58 pm
  85. 2004-12-24 3:04 pm
  86. 2004-12-24 3:08 pm