This article offers feature suggestions to budding OS developers looking for that neat edge.Conventionally, each file on a computer system has a type. Different operating systems typically had different ways of working out the logical type of a file. UNIX uses a special flag to see if a file was ‘executable’, otherwise it is ‘data’. Therefore, applications on UNIX have a variety of strategies for determining the file type. Some programs use the file extension as a hint (.jpeg is probably an image, for example) whereas others use more advanced heuristics (e.g. the UNIX ‘file’ command) sometimes called ‘magic’. MS-DOS and MS-Windows use the file extension to determine the type (.exe for executables, .txt for text files etc). More recent operating systems have embraced more recent classification systems such as MIME. In BeOS, for example, a server scans files and marks the MIME type of them in an attribute (basically caching the guess from the UNIX-style ‘file’ heuristics) so that it can easily be looked at later. Almost all Internet content is marked with a MIME type (web pages, files received via email etc) and this can clearly indicate to the receiving computer the intended type of a file.
Now let me introduce something slightly novel and new! Imagine that our new hobby operating system supports multi-stream files. Each stream in a file is a certain MIME type. Some of these streams might be generated programmatically on-the-fly, and basically encapsulate conversion. These programmatic streams allow plug-in filters and converters to be integrated into the operating system and used by programs. Let us explore what this might mean and how it might be used by programs:
The operating system knows the actual MIME type of a file. It also has a list of programmatic stream plug-ins associated with each MIME type to handle the interpretation of a stream of one type into that of another. An obvious plug-in would be to filter text/html so that it was text/plain, stripping out the markup. Another plug-in might convert image/jpeg to image/gif. Another might extract the soundtrack of a movie file and present it as audio/wav.
An application/x-idx file might have a programmatic stream of type text/gnu-makefile, while another programmatic stream might convert text/gnu-makefile streams into application/bash shell scripts. Therefore the operating system can string together these streams transparently, making an application/x-idx file executable in at a bash prompt.
Encodings and character sets might also be incorporated into the same system, for on-the-fly conversion between various sets. A code snippet:
FILE *f = fopen_ex(“ftp://stop.blogging.com/varfar/myos.zip/documents/filesystem.pdf”,”rb”,”text/html;charset= iso-8859-1″);
A file selector dialog might have the file type set as a MIME type and display all documents that match a specific MIME type that the application understands without the application having to understand the conversion process.
Most conversions are lossy – they remove information and meta-data, downgrading the quality of a source. Therefore, it is anticipated that almost all programmatic streams might be read-only. Many programmatic plug-ins’ might be easiest to implement if they support sequential access only. For this reason, it makes sense for the file selector dialog to be able to filter out streams with inappropriate access (e.g. only display application/ogg streams that are writeable and support random access). However, the opportunity is there for super plug-ins that facilitates content stored as one type to be completely edited as another, e.g. reading and writing to an application/ms-word file with an application that only understands application/x-multipart-html.
Let me extend the idea even further. Not only might a file be multi-streamed, it might also be multi-part! Many file formats encapsulate multiple content parts – a movie has both image frames and a soundtrack; a rich document has text, drawing and images embedded; a mail message might contain a plain-text version, a rich text version or two and many attachments. An archive contains multiple files inside.
Such files could be presented to programs as directories. You could now use your favorite drawing program to edit the individual frames of a movie!
Directories themselves can have streams too. In the case of a virtual directory being generated from a single file, the directory’s stream would be the actual physical file (and programmatic streams that convert it). A conventional directory might have a text/html stream the serves the index.html file that it contains; another directory stream might provide a tar file of it and its children.
Conversion libraries on most operating systems abound. Typically these libraries are fragmented, often unwieldy, often exposing technical details, and not pervasively used.
A new hobby operating system has the opportunity to provide a uniform interface between applications and these conversion filters, in a way that is minimum effort for application programmers and in a way that is extendable (by providing more conversion filter plug-ins) without requiring extra code nor recompiling or re-linking of all applications. Placing this conversion into the way that files are provided to applications is, in my opinion, an excellent way to achieve this.
Programmatic streams do not expose many parameters to tweak conversion and therefore do not completely replace the need for dedicated, detailed conversion libraries for dedicated programs. But they do further the usability of the average program in the normal operating scenario.
If you would like to see your thoughts or experiences with technology published, please consider writing an article for OSNews.
already such thing exists, like blob in your database, a compressed filesystem, the pk4 files id uses with doom3 and many more.
but what if your stream increases in size? and decreasing a stream size and try to not fragment your container file. try resizing a jpeg that you put as a blob in your database.
maybe we end up with a definition for a filesystem with predefined block sizes.
So it’s a filesystem level conversion? (presumably with some way of stringing together conversion pipelines). I guess it’d be nice to read out an ogg file as an mp3 and have that transparent to the app
Good idea. I think you could have a userland app to do the same, but doing it at that level makes sense too (basically you’ve got conversions and metadata – right?).
Er, just to clarify, I mean metadata sounds like your multiple streams of data (which ReiserFS can do), so you’d just need some way to do conversions and to string them together.
Yes ! Do it !
Will it be the first ReiserFS plugin ?
Multiple streams are also supported in windows 2000 and above. It’s used for for example keeping an html file + it’s images together in explorer.
In BFS metadata has a datatype (int32 / string / etc) and is indexed
This is the kind of idea we need to make a better system before Longhorn comes out. Keep them coming!
I’m sure I read this article a couple of weeks ago.
Quite sure it isn’t 😉
Maybe you are thinking of this previous article? I am doing a bit of a mini-series here:
http://www.osnews.com/story.php?news_id=7907
br,
Will.
This features were in OS/2’s HPFS filesystem.
There may be several possible ways of converting one file type to another, all of which may be lossy, and a number of which may have significant advantages in different situations.
Utilising hierarchies (like directory hierarchies) for files (as mentioned) seems a sensible way of reducing conversion headaches, but many files are limited in supporting such hierarchies (or perhaps more complicated for this endevour, employ almost-but-not-quite-hierarchy structures), and importantly, many more are ambigious in regards to exactly what their hierarchies are.
XML may be nice in that it has a clear hierarchy, but whether this hierarchy is actually a good conceptualisation of the file’s structure may depend on the XML file format in question (and perhaps on XML’s own structure).
And of course some files, like sound clips, are simply are not hierarchical in nature, they are contineous.
It would be nice to have a universal hierarchy that extends from the root to the most primitive data types, but it doesn’t exist. Once you leave the filesystem and enter the file, you are definately not in Kansas anymore.
This idea has been around as long as userland file servers have been. In fact, a good implementation of userland file servers can, in fact, do this.
Doing it with userland file servers also has another advantage: The user can pick what file gets converted to what type and where it gets placed. File /tmp/x.gif mounted to /home/x.jpg and the gif-to-jpeg file server could be applied. It can then be accessed with traditional file access calls instead of specialized ones with MIME type selectors. This allows any program to use the file, which is converted on the fly by userland file servers, without having to worry about MIME types. The gif-to-jpeg server could also work as a jpeg-to-gif so the conversion goes back the other way when the application saves it.
This would allow an image editor to support one image format (a native format created for the OS that is easily manipulated) but, through the use of userland file servers, read and write to any format.
Go back to the drawing board and come up with something new.
Mac has had multiple streams since its beginnings. Originally there were two, a standard data stream and the so-called resource fork, which is essentially a list of tagged data blocks organized as a basic database. It’s fully writable, unlike Windows resources which are embedded in a single stream of a specific (PEF) file type.
Later versions added additional streams. They aren’t tagged as to content type and haven’t been widely used AFAIK.
OS 9 added NEXT-style packages, special directories containing collections of files that are presented as a single file to the user. Some applications already use this (rather than binarily embedding graphics in text, store them as separate files in a package as standard graphic files.)
The package scheme is much simpler than file system level streams and lets you use standard file types and access the “streams” easily (as individual files). If more elaborate typing is needed, add subdirectories or an application/package defined XML file to define it.
Another advantage is that copying a package is the same as copying a directory, and all the contents usually come with it. Copying a multistream “file” from Mac to Windows, for example, usually causes some streams to be lost or appear as separate, unusable files that don’t get copied back correctly.
Interesting point Anonymous, and not something I was aware of.
How would a scheme that does mounting e.g. /tmp/x.gif mounted as /home/x.jpg do it’s automounting?
I mean, when I ask to open a file of type image/jpeg, how would the system know to offer x.jpg unless it had been manually mounted by the user?
And what if x.jpg already existed as a true file?
I am not sure how a ‘mounting’ approach helps hide the file type. It seems, from your description, to rely upon something somewhere associating one virtual filename with a programmatic stream parsed from some real file. This approach of all files being able to be interpretted as sensible other types is all about making the filesystem do that mounting transparently and on-demand.
I disagree with such a conversion system at file system level. For example the html to plain text conversion. If I have text editor and I open a html file with it as plain text, I modify it and want to save it. How should it know on how to save the modified text file back to html. That is the point of this conversion system, isn’t it? html and plain text are 2 completely different types. Of course you can convert it from one to another, but it isn’t equal. Making this conversion transparent to applications brings a lot of problems with it. I suggest to keep conversion an explicit task for the program, those programs have much more information and knowledge to do the conversion, and if some extra information is needed, it could always ask the user. In other words, you try to automate a process which isn’t possible all the time.
I could go into how the mounting, as you put it, would work but I don’t think that’s really the issue. If you’d use an OS with userland file servers, you’d see how it would work clearly.
“I mean, when I ask to open a file of type image/jpeg, how would the system know to offer x.jpg unless it had been manually mounted by the user? ”
That was the point: The user does the mounting. Doing on-the-fly conversions via APIs is just stupid. Operating systems like Windows or Mac OS X have huge APIs as it is. Adding one more to the mix isn’t helpful. Doing this via already known file system operators makes it easy. It could also be applied to any exsiting operating system (if it had userland file servers, and most don’t).
“I am not sure how a ‘mounting’ approach helps hide the file type. It seems, from your description, to rely upon something somewhere associating one virtual filename with a programmatic stream parsed from some real file. This approach of all files being able to be interpretted as sensible other types is all about making the filesystem do that mounting transparently and on-demand.”
But the mounting in your method isn’t done transparently, you must request it via the fopen_ex function. With file server-based mounting, the file type is hidden from the program. And there is no reason this couldn’t be done with a file server-based approach.
As for on-demand, a file server could be created, applied to the completely file system that automatically creates any possible formats for any given file. That is, for x.gif it creates x.jpg, x.png, x.tiff, and so on in the same directory, or use the file as a directory as itself.
All I was saying is your idea isn’t new, as it claims to be. It’s at least 10 years old.
Or how about another method not using fopen (or the like) or file servers? This, again, could be done on any OS. Create a function like ‘int gif-to-from-jpeg(int fd, int mode).’ A file descriptor is passed to the function and another one returned. Any time GIF data is read from the file descriptor, it’s returned as JPEG data. If JPEG is written to the file descriptor, it’s written to the actual file as GIF data. The mode variable would modify how the paths work, i.e. if it’s gif->jpeg and jpeg->gif or the other way around. Plan 9 already does this for applying SSL encryption to network streams (See http://www.cs.bell-labs.com/magic/man2html/2/pushssl ). Very handy stuff.
Why stop at conversions of formats, while we are at it? Why not apply “stacking” of file servers to audio and video strams? This, of course, implies all audio and video streams are presented as files. Live effects could be applied to any audio stream, with recorders pushed in between the effects outputting to various files. Or various audio streams could be brought together to form a new stream that is actually pushed to the speakers via a play file server that is added to the very end of the stack. I bet sound studios and radio stations would love this.
You see, by building around an abstract like files and file servers, amazing things can be built without much low-level work. Cool stuff, huh?
As you point out Anonymous, the approach you outline – mounting as separate filenames – is not quite the same as presenting a single filename as multiple streams, each of some specific mime type and some possibly converted on-the-fly programmatically.
The conversion I outlined is transparent to applications in the sense that the mime-type in the fopen_ex is known at compile-time by the application programmer – they know how to read files of a certain mime-type – and the fact that they are reading files of another type is transparent to them at runtime.
Reiser4 plugins and Hurd translators can do this today. The main problem is: Reiser4 is *one* of many Linux filesystems and no distro up to the present time did commit to only one filesystem to build their userspace over. Ditto for the Hurd — we would need a Debian GNU ^2 to ditch the linux-like stuff and try to build a REAL competing Hurd stuff using the advantages of the Hurd. Maybe even implementing reiser4 as the default FS?! THAT would be great. Hmmmmmm…. maybe if L4Hurd gets a little bit more stable?…
You’re still missing the point that it is nothing new. It’s just how you are applying the idea, through a function instead of file servers.
There are also advantages to my method. It can be done on existing operating systems and work with existing applications on those operating systems. Both methods still have the application reading one specific file type and the true file type being hidden, thus archiving the same goal.
This sound very much like content negotioation in HTTP/1.1.
So what we want is something like gnome-vfs, merged with a http-1.1/webdav server and stream transformer plugins.
file://localhost:/home/media/music/foo%20-%20bar.mp3 is exposed to the file manager as a mime audio/mp3. The user symlinks this to ~/.playlists/current/.
The application reading the playlist however requests the resource as audio/raw.
The resource is thus serverd thorugh the audio/mp3 to audio/raw plugin.
Simple and powerfull.
See now Will you made me go off and search the Syllable forums. I knew I’d seen this idea from you before! It’s interesting to see how you’ve expanded on your original ideas. I don’t suppose there’s any chance of an implementation atop AFS is there?
no… don’t insert all this stuff down into the lower levels of the OS. taht’s what MS does with windows (and does so insecurely – but that’s a different issue) – it’s short sighted and it goes out of date quickly. the OS should aim to only implement the most trend-agnostic smallest building blocks .. from which upper layers can then build fancy intelligent media handling capabilities… even some of the TCP/IP protocol is too fancy in my opinion.
I’m really growing tired of the “this isn’t new” responses every time someone suggests an idea or project they have been working on. Sure, maybe it isn’t 100% new, but maybe it is a twist on an old idea or perhaps it was just a mental exercise for the author. How many ideas are entirely original anyway? Not very damn many. Just because it isn’t 100% new does not mean it is worthless!!!
Because when the author states “Now let me introduce something slightly novel and new!” even though nothing could be further from the truth, a lot of credibility flies out the window. It shows that the author is looking for fame instead of actually trying to innovate. It shows the author didn’t do his homework and probably hasn’t been in the operating system world all too long.
This answer seems better to me: because the name of the site is OS “news”, and not OS “olds” ?
Now, seriously, Anonymous (IP: —.dsl.emhril.ameritech.net) has a point: it has to be new because it was presented as new. But look, it isn’t: ext2+hurd and reiser4 have the same principles… just write the translators/plugins.
And then, you can present them as “news”.
When I read this thread, I know why Linux simply will never get there. If someone comes up with a nice (but not specially new idea). Some people will argument: in this or that way, we already have it, invent something new.
And so that idea stays a toy of linux literates, while such a simple trick could bring it too real users lands(users as in the 80% that uses windows because they don’t need and can’t change rcY.X files). Example We never needed a Database Filesystem, but now microsoft simply makes it and suddenly we all need it.
It’s always easy to say: Can’t be done, shouldn’t do it. For a change find a reason to do it: Linux can grow with positive feedback, not with negative feedback.
Happy linuxing
Exactly — who cares if someone came up with this idea ten years ago. Windows is that it popularises common ideas. I mean, Windows has people thinking that Microsoft invented the application bar.
Popularisation is everything, and that means stable usable software. Hopefully this won’t just be an osnews column, but the beginnings of an application. Good luck.
The author thinks it is new, and it is not.
Several other posters have pointed out other implementations of this. The oldest one I am aware of is the original Macintosh filesystem implementation. It also seemed to be the most logical.
A very important thing to consider is that something like this is not terribly useful unless it is used and maintained by all applications which access the filesystem. It is also most useful if this is transparent to the user.
This means that the “streams” or what-not always appear to be one entity. Allowing the user to alter this (directly or indirectly) breaks things (ever run DOS scandisk on a FAT partition using long filenames?).
It should be done, but it will take more than just another implementation of existing technology. Unfortunately it will take coordination and cooperation, which seem to be difficult outside of a commercial effort.
A very important thing to consider is that something like this is not terribly useful unless it is used and maintained by all applications which access the filesystem. It is also most useful if this is transparent to the user.
That is why only one application should access the FS, the file manager. Everything else should just handle what is fed to it. Note the implication that “everything elses” doesn’t get to choose streams.
Golly, hadn’t expected so many people to get so upset. I probably would be better off not replying 😉
This osnews mini-series is a series to give people developing hobby operating systems – for fun – ideas that take their os from a kernel to a ‘system’.
There are many similiar approaches already out there as people are keen to point out pointedly. But I have yet to see an actual operating system – hobby or otherwise – where an application can ask the user to select a text file and for the user to – transparently, not necessarily even understood by the user – select some richer text file instead. Oh well, maybe I am not ‘well travelled’.
It is all well and good for a particular filesystem to implement ‘translators’ or others; but personally, I’d favour an implementation at a higher abstraction such as within the VFS layer. There is no reason this approach to transparent on-demand conversion couldn’t work on top of the most basic filesystems, or even remote ones.