Submitted by Back when Windows Vista was still known as Windows Longhorn, the operating system contained a very interesting and promising feature, a feature promoted as one of the ‘pillars’ of Longhorn: WinFS. WinFS was a storage subsystem for Windows, based on a relational database, that could contain whatever data you wanted to put in it. Thanks to the relational properties of the database, you could then create relationships between data, or let the computer do that for you.WinFS would allow programmers to do all sorts of wild and radical things with data in their applications, and back in 2003, during the Professional Developers Conference, Microsoft showed a video demonstrating what could possibly be done with WinFS. This video, called IWish [.wmv], can still be downloaded from Microsoft’s website.
Sadly, the video only showed what you could do with WinFS in the future, because the version of WinFS that shipped with the Longhorn test releases back then was, well, disturbingly not useful. It did not do a whole lot of useful stuff back then, and to make matters worse, it was the world’s worst resource hog ever. Disabling WinFS would turn your Longhorn build from a crippled snail into a fairly usable operating system – testament to its yet unoptimised nature.
We all know what happened to WinFS after its audience-woowing days of 2003. It was first slated for release after the final release of Windows Vista, as an add-on, but later on, in 2006, it was cancelled altogether. Parts of WinFS ended up in Microsoft SQL Server and ADO.Net, but it would no longer be delivered as a stand-alone product or a Windows component.
Since then, we haven’t heard a whole lot on whatever happened to WinFS and its associated technologies – that is, until a few days ago. Jon Udell has published an interview with Quentin Clark, who led the WinFS team from 2002 until 2006, when he joined the SQL Server team as a general manager.
One of the burning questions that rang through forums everywhere was whether WinFS was a filesystem or not. Clark explains that it depends on your viewpoint:
People would often ask me if WinFS was a file system, and I’d struggle with the answer to that, because, well, you know, from a certain standpoint the answer is yes. The stuff I saw in the shell, was it in the WinFS filesystem? Well, OK. But there are no streams inside the database. So from a user perspective, those files were “in” the filesystem. But from an API perspective it was more nuanced than that. I could still use the Win32 APIs, get some file, open it, and from that point forward the semantics were exactly like NTFS. Because it was NTFS at that point.
Clark explains that you can look at SQL Server 2008, ADO.NET, and VS 2008 SP1 today and trace its lineage right back to WinFS. The schemas of WinFS and the required technology, which were used to store properties of objects, were shelved, because they are not needed any more. The WinFS APIs are now part of ADO.Net as the entity framework. Finally, “What’s getting delivered as part of VS 2008 SP1 is an expression of that, which allows you to describe your business objects in an abstract way, using a fairly generalized entity/relationship model.”
Clark details a whole set of new features and ideas that try to marry the database world with the filesystem world, and how they are trying to do this in a step-by-step fashion. According to Clark, this is exactly what lacked during the Longhorn days of WinFS. “That’s kind of where we got tripped up in the Longhorn cycle. We were building too much of the house at once. We had guys working on the roof while we were still pouring concrete for the foundation.”
Clark remains hopeful about the future of integrated storage. “But I do at some point want to see that place in my heart fulfilled around the shared data ecosystem for users, because I believe the power of that is enormous,” he explains, “I think we’ll get there. But for now we’ll let the concrete dry, and get the framing in place, and then we’ll see how the rest of the house shapes up.”
I think that having a system like BeServed for every *nix out here is more feasable than WinFS coming out for Windows.
At the very least it would be here sooner with enough developer interest.
some interesting projects:
http://www.gnome.org/~seth/storage/
http://www.glscube.org/
http://relfs.sourceforge.net/
http://tech.inhelsinki.nl/dbfs/
http://beagle-project.org/
Too bad all of those are more or less dead ends. I would love something like this to come to fruition, so that we can finally have the kind of actions and applications Microsoft touted back in 2003 and Longhorn.
Of course, Beagle isn’t dead, but Beagle is not a relational database-based “filesystem” – it’s an indexer and query tool. Also nice, but in essence it’s got as much to do with WinFS as a spoon has to do with a pig.
WinFS is like Beagle – neither of them are a real filesystem (WinFS sits on top of NTFS), and both use indexing and so on to offer an alternative way of finding and using your data. The WinFS model is more advanced, I agree, but Beagle is working right now.
The other projects I linked against offer many concepts and code that /could/ be used to create an awful new tag-based or database-like filesystem. Maybe they are dead end just because people don’t care about it.
Interesting thread here, and I’m giving up my ability to mod in this article to point out there is a new kid on the block (according to my copy of LXF105 any way, website looks like its been around awhile–maybe under a different name?) called Recoll.
http://www.lesbonscomptes.com/recoll/
Supposedly it uses less resources to do what beagle does and is faster as well….time will tell.
–bornagainpenguin
I also have a development project aiming to implement a relational filesystem, and the relative desktop environment build around it: http://lobotomy.sf.net .
I’ve not so much time to dedicate the project, actually it is just a (not enterely working) proof, but sometime I can release some new code and, above all, some new addiction to the specifications.
If someone has feedback, it is always welcome 😉
Look one article down to
“Interview: Kevin Musick, BeServed; Haiku Code Drive 2008
Written by Thom Holwerda on Sat 17th May 2008 19:19 PST”
Microsoft should drop their ambitions to code that they are not equipped to organize and develop and turn to people who can. Or go with ZFS.
Please, people, read the article PROPERLY before spouting NONSENSE. ZFS, Beagle, Spotlight – they’re all awesome, but have absolutely NOTHING to do with the technologies behind WinFS or the original WinFS project that led to their creation.
ZFS is an advanced filesystem that allows for things like volume snapshots and copy-on-write. Indexers and query tools like Beagle and Spotlight index the data on your hard drive, and expose that index to the user through what is in essence a glorified search dialog. All these technologies do their job well, they’re useful, and I wouldn’t want to live without Spotlight on my Mac.
WinFS has little to do with searching and indexing, and more to do with managing. WinFS allowed you to set relations between objects, and use those relations to manage and organise your data – either manually, or automatically via applications. To achieve this goal, it used a relational database, as described rather well in the interview.
Please, people, I know it’s fashionable to discredit anything Microsoft does, but the ideas behind WinFS were sound, and the goals ambitious. They failed a lot of times to bring these concepts to your doorstep, but it seems, judging by the words of Clark, that they heave learned from these errors and are now working at in a different way: ensure a solid base, and build up the house from there.
In the interview, you can clearly read that the idea of bringing WinFS’ features to the desktop are not quite dead just yet. And you can be anti-Microsoft all you want, but I’m excited about that.
That’s true. If you want a semantic desktop in the *nix world, have a look at nepomuk http://nepomuk.kde.org/
As KDE4 will be ported to Windows, MacOS, there is a chance that we will see the the ideas of the original
WinFS come true, regardless what Microsoft decides to do with WinFS in the future.
Nepomuk integration is one of the most exciting features of KDE4. OTOH I’m worried about implementing semantic attributes at such a high level. I mean, what happens if you log into, say, icewm, or console, instead of KDE, and move or rename a file? Can Nepomuk detect this change correctly the next time you log into KDE? In Windows the problem doesn’t arise because there’s only one desktop environment, but in GNU/Linux it seems that semantic features should be implemented below any DE, maybe with FUSE. I guess there must be a good answer, because people are actually using KDE4, but I haven’t found it.
I don’t know if the nepomuk developers have addressed this, but one way of doing it in Linux could be to use inotify to trigger some kind of update method in such an event. This would work even at the CLI level.
Spot on! Sebastian Trueg, main nepomuk-kde author, is working on this issue right now and his solution is indeed based on inotify. See:
http://websvn.kde.org/trunk/playground/base/nepomuk-kde/filewatch/
Given the KDE schedule, I suppose that this is KDE 4.2 material. Note that this is very very badly needed as currently if you copy a file even inside konqueror/dolphin, the metadata is lost, see:
https://bugs.kde.org/show_bug.cgi?id=161403
Re GNOME integration, the great thing about Nepomuk is it’s an abstract, open specification, of which nepomuk-kde is just the kde implementation, but nothing prevents GNOME devs to implement a nepomuk-gnome the way they like, based on their own frameworks etc.
I was wondering that too. Maybe the Gnome team should create their own semantic backend, if they already don;t have one, and work with the KDE team to standardize the protocols so that interaction between both services would be flawless. Better yet fdo can maybe create it a standard for all desktops to use. I’m not a fan of KDE’s UI, but the technology can’t be denied. Gnome needs to get in the game.
//As KDE4 will be ported to Windows, MacOS, there is a chance that we will see the the ideas of the original //
KDE 4 will be ported to Windows and OS X? Please pass the water bong.
I think people are just getting fedup with Microsoft announcing products and features that never come to fruition.
Microsoft constantly goes competition bowling with press releases.
Iphone – surface, is the most recent example I can think of… Microsoft seems to feel that if you don’t have your own thunder then try and steal everyone elses AND/OR announce a non existent product before a real existing launch to stupefy your competitions customers. Years later while they wait to buy Microsofts previous announcement MS can sell them something with half the promised features when the competition had already a fully implementable solution — sometimes driving the competition out of business in the process!
I have all the respect in the world for the WinFS team and the dream that could have been but the PR fiasco that was Vista was happily hyped and fueled by MS before the release and now they wonder why people are pissed when it’s all Scots mist?
http://www.downloadsquad.com/2006/11/07/windows-longhorn-concept-vi…
That previous video of longhorn showing the leopard like smooth transitions and accelerated graphics actually made what would become Vista an interesting OS and yet all we end up getting is an encumbered XP with a pretty skirt?
Once Microsoft actually starts delivering on their promises THEN maybe they’ll regain some respect from the tech community.
It’s funny, people say Steve Jobs has a personal reality distortion field generator — and its bloody true!
But in all honesty, I think it’s standard equipment for everybody working in Microsofts public relations department!
Wow! I want that as my OS. If that was what vista looked like now, I would go back to windows in heartbeat. Why would they not try to at least meet half of those goals? The funniest thing was the tag line at the end, “Simply the best windows we ever built”, yeah because it doesn’t exist. Rather sad really.
Please see video that I linked, the one showing WinFS team demoing and talking about WinFS beta 1.
“Not equuiped to organize and develop..” ??? Really? How did they manage to get it to beta 1 then?
The decision to stop working on it (for now) had nothing to do with teams’ (in)ability to deliver.
Edited 2008-05-18 16:34 UTC
Microsoft also released, on Channel9, another video of WinFS, showing beta 1 release.
Here:
http://channel9.msdn.com/showpost.aspx?postid=106356
Looks quite OK to me for beta 1, especially given that the big deal for the team at that point was to provide full backward compatibility.
Edited 2008-05-18 16:19 UTC
Stupid peaple didn’t understand Windows Search – which indexes in low I/O priority and only a few folders – and so they disable it thinking it slows down Windows, but the same stupid people are complaining because WinFS – that’s heavier and more bloated – has been removed from Vista
Edited 2008-05-18 18:00 UTC
Stupid peaple didn’t understand Windows Search which indexes in low I/O priority and only a few folders, but the same stupid people are complaining because they want WinFS that’s heavier and more bloated
Umm…do you even understand what WinFS was all about?
There might be some truth to what he said though… doing the metadata updates in WinFS probably would probably have worsened the overhead of certain filesystem writes.
In the list of features for Windows 7… that will be dropped in the middle of the development process (just like in every iteration of Windows Microsoft has promised to deliver it into). This guys are sickening…. and it’s even worse to see people buying into anything Microsoft says.
“I’m going to attribute a statement to Microsoft that no one there has said and then declare my utter disgust with that imagined statement.”
No one is releasing any concrete information about Win7 features.
I pretty much expect that when a new version of Windows is announced, it’ll be out at least a year or two after they said and that some features will be dropped.
Maybe they’ve finally learned their lesson and will actually deliver on time and with whatever features they promise for Windows 7. I’m not holding my breath though
last i heard it was put on the back burner because they were wasting too much time trying to do fancy things rather than make an OS.
As I understand it, WinFS is a database on top of NTFS? Wouldn’t it be possible if SUN released a database on top of ZFS, in a similar vein? We know that SUN has excellent engineers with outstanding technical excellence. SUN could surely implement WinFS-esque features on top of ZFS? Wouldnt be awesome with WinFS based on ZFS instead? The best of two worlds?
As Sun tend to GPL things now and, as Microsoft have said themselves they “don’t agree” with the GPL, I think that any Sun Database FS on top of ZFS will probably released for Solaris or general *nix. Sun would be foolish indeed to give technology away to Microsoft, look what nearly happened to Java.
What happened to Java? You mean MS hijacked it, and called i J++, only to be sued, and then later released it as C#?
SUN tried to standardise Java, but got hindered by Microsoft. MS doesnt want that.
http://www.openmalaysiablog.com/2007/09/microsoft-conde.html
I got a new laptop a couple of weeks ago and I have spent most of my time with it in Linux. The other day I decided I wanted to play a game that I’ve had a for a while, Battlefield 1942. I installed it in Vista. To get it to work with widescreen, you have to edit a couple of files. I edited those files and go it to work. Then, for some other reason I decided to uninstall it, re-install it. My settings remained. These were settings in a file, not in a registry. So I uninstalled it again, this time going to the C:\Program Files entry and shift+deleted the directory. Then I re-installed the game. The settings were still there.
I went through windows explorer to see those files, it wouldn’t show them to me. I enabled hidden and system files…still wouldn’t show them to me. I ran a dir /a in DOS and nothing.
I had Python installed and did an os.walk(‘.’) and sure enough those files showed up in Python but not in Windows.
I think I managed to get a corrupt filesystem with less than 3 hours using Vista. I had files that I couldn’t navigate to using Explorer or DOS but I could with Python. I actually had to edit the files using python commands which wasn’t too bad, just a couple of string replacements but still….ridiculous.
Interesting to note:
He talks about one of those “neat” “super” ideas that spilled over into other places: letting you have “stream” column type that gives you a file handle to a db item. Then he goes on and on about how people want these win32 api semantics for these objects, how it’s so great and super that it provides these.
Now where I have hear this before? Something about providing a “file” interface for every object. Hm… dang.. let me rack my brain a little more… something about “everything is a file”. I just don’t know.
I think any time someone from microsoft starts throwing around the word “super” and “great”, it means two things:
– Bill! Bill! come see my project. SUPER!
– I just ripped off an idea that’s been around for years and years, and finally figured out how to shim it windows in the most bizarre manner possible.
Ok, I really try to avoid posting to osnews, coz I’m always too sarcastic, but I just couldn’t help myself. Like the site, though, keep up the good work Thom!
Everything is a file….
There is a mainframe operating system which was released in 1974 where everything IS a file.
AND it has:
– Virtual Machine architecture
– An API (SCL) which can be run line-at-a-time, as command files or COMPILED (the API is the interface – facilities only now becoming available with ‘Dynamic Languages’)
– SCL inherently has positional and/or keyword parameter specification (MY_FUNCTION(START=0, STOP=99, REPLY=result))
– Cascade loading so that code is only brought into memory when first accessed
– Separation of programs into ‘pure’ and ‘writeable’ areas (NO code corruption at run-time)
….and lots more
And it now runs on commodity x86 hardware (via emulation)!
See http://en.wikipedia.org/wiki/ICL_VME