The article I’m about to link to, by Oliver Reichenstein, is pretty terrible, but it’s a good way for me to bring up something I’ve been meaning to talk about. First, the article: “Apple has been working on its file system and with iOS it had almost killed the concept of folders – before reintroducing them with a peculiar restriction: only one level! With Mountain Lion it brings its one folder level logic to OSX. What could be the reason for such a restrictive measure?” So, where does this crusade against directory structures (not file systems, as the article aggravatingly keeps stating) come from?
I have honestly never seen a single person have any issues with directories, nested or no, and as old as the concept might be, the people I interact with seem to be able to handle it just fine. Contrary to the broad strokes made in the article, nested folders really aren’t all that complicated conceptually, especially not when compared to, say, the mouse (indirect manipulation), the concept of applications (totally arbitrary), and dozens of other accepted computing concepts.
In fact, a complete absence of a regular accessible directory structure seems to create more problems than it supposedly solves. Applications on the poster child for this concept, iOS, are islands, and the files within them are not easily shared, which leads to all sorts of different problems. Then there’s the issue of not knowing where a certain file is; instead of having your own structure, tailor-made for you because you created it in the first place, you now have to remember which application created what file. And what happens when it’s in an application you deleted? It’s all needlessly complicated.
It seems to me that directory structures are something that some people desperately want to be difficult, but in reality, really aren’t. It’s a conceptual problem that, in reality, doesn’t exist. I’ve never heard anyone say “Instead of tucked away in folders I can remember, I want all my files to be in one big mess of a pile that only a searching tool can make sense out of, so that I, consequently, have to remember each and every file name or parts of its content”. I also have never heard people say “I wish all my files were strictly tied to individual applications, making it very hard to move them around or take them with me.”
All this talk about directory structures as if they were the seed of the devil annoys me, because it seems like the success of iOS makes it a legitimate position, as if iOS’ success is 100% – or even 10% – based on how it doesn’t handles files. The fact that virtually everyone I know with an iPhone or iPad emails files to themselves to get them on those devices illustrates just how far we’ve regressed.
Emailing files to get them on devices? Seriously?
Directory structures are not evil, and they’re not hard to understand just because some people believe they should be. Like cupboards, Tupperware, boxes, closets, pockets, wallets, and about a gazillion other products, directories allow us to organise our stuff, and that’s a concept as old as time itself. Throwing them away just because Steve Jobs hated them is not only short-sighted, it’s destructive, and makes computing more complicated, not less.
Right, so this paragraph wasn’t supposed to be here, but this just hit me the moment I wanted to press publish: making files harder to move around? Making it harder to find them? Making it harder to take them with you? It’s been staring me in the face this whole time.
A good commentary, Thom. This is the second level of file-based vendor lock-in after the proprietary file format thing just doesn’t work like back in the days any more.
It is now wonder that it comes from cell phones. It reminds me well of my feature phone that can run all these nifty Java ME applications, but I am not allowed to upload them myself.
And what about opening a directory with 15 MP3 files (an album)? Should I remember all tune file names? And search for them individually to add them to my playlist? How convenient is that?
Always reinventing the wheel. Always inventing non-existent problems. Always trying to look “inovative”. It reminds those who always try to “reinvent” the email concept. I just wish the Linux folks (and MS) don’t jump into this bandwagon like blind sheeps.
You of course search stored data about your data.
In this case things like ID3 tags.
No, that’s terrible too. The correct counter point, is that you shouldn’t be looking at any folder for music. The music app should already know about all of your media files and organize them into albums accessible by pictures.
I think the user you replied too was kidding 😉
(Who would search for data about his data, encoded in binary information…LOL)
Regarding your tune files unaccessible, this is not a problem as long as you don’t want to copy them, back them up or move them to a different drive. You’ll argue that you shouldn’t copy copyrighted material, but this is neither the real life, nor what the users want. They don’t want to lose functionality.
I’m playing a little devils advocate, I don’t actually believe what I’m arguing…
But Apple has purposefully always made it difficult to copy music files from devices.
What if that’s not how I want to organize my music?
Right!
I’m a ballroom dancer, and I usually want to organize by dance-category (slow waltz, foxtrot, tango, quickstep Viennese waltz). Then by other things (exact tempo, etc.)…
You know, that works with ID3 (and such) tags, too …probably more swift, even. And with added benefit category “branches” being non-exclusive.
I don’t agree, really. I think files themselves are an awful abstraction for user data, and a hierarchical store makes them even worse. It forces you to generate names for everything, and to think of where to store things. That is a job that should be left to the computer! Why should there be a name for every picture in a thousands-long collection? Either I will have to cook it up, or put up with a serial number, or some such.
I think that the current trend of showing data as content, rather than files, organized by metadata (Ã la music manager or photo app)is the way to go. And it is better to keep it separate from file system management. I’ve seen MANY people TOTALLY FAIL at understanding the difference between pictures or music as named files in the explorer, with OS-provided previews and views, or as objects in their media manager, with different names (if any) and organized in a totally different fashion.
Now, the way filing-by-metadata is implemented in iOS SUCKS A LOT. It is one thing to have a sea of data units accessed by content, and one, very different one, to force you to only access them with the app they were created with, or maybe some specialist app.
Why is it so difficult to add or retrieve an attachment in an email message on an iPhone? For example, how do you attach four pictures to an email message? And why can’t I see any data other than pics when I plug the phone into a computer? Of course, the answer is: because Apple thinks you are an idiot, and a thief, and a cash cow that they want shepherded through their channels (iTunes, appStore). It is a pity that such nice devices with such a beautifully made OS are so restrictive due to such a poor conception of the human being, but that is the way it is, and it makes me want to flee to Android.
Thom – BeOS live queries. This is how I used to find files. It was rare for me to traverse file structures. MacOS X Spotlight, I don’t need to know where a file is. Windows, well, on 7 I have a few folders pinned that have my useful files in them. Do I know how to use a file system? Yes. Do I feel compelled to complicate my file storage? No.
Even if I myself do used BeOS and still does use Haiku, it’s not rare for me to traverse file structures.
For a simple reason: I don’t create the whole set of files and folders, and often the structure itself is part of a “larger thing”, not just a bunch of disconnected files but a way to structure the relation between them, too.
Like… source tree.
Anyone having to work on a large flat (all in one folder) source code knows how stupid is it. And, in such case, no smart search tool can recreate the missing bits, because these bits, this missing structure, has *semantic* value, not just a technical arrangement.
Wait.
Dumbification/beingidiotproof/ourwayorthehighway?
Oh.
Okay, then. Who am I to think I can actually do better job than a computing device. Or just want to try it, to keep control on the way I use these tools, while clearly these tools deserve to have a dumber user compliant with their *innovating* way.
Please update my iBrain, I’m ready to “think different” (and grammatically, wrong, BTW).
Go.
Charge me (pun intended 😉 )
Edited 2012-07-25 23:29 UTC
You do realise that this one-level directory restriction applies only to documents saved in iCloud, don’t you? Saving files on your Mac hard disk continues to work the way it always has.
I do.
I also realized that’s it’s done voluntary to prepare people to drop local storage in favor of cloud one, and indeed losing multiple level hierarchy is one feature that needs to be less visible to end-user to ease this move.
Last but not least, I realized the argument that it’s better that way is uncomplete: it’s better for cloud companies in term of profit and storage complexity management, but it’s a feature regression for end-users.
Not that I care that much: I don’t plan to use an Apple technology anytime soon, and if I use cloud storage one day, it will be as crash plan storage, with signed encrypted backup files.
For the share everywhere everytime everyone, OwnCloud or similar is fine enough, thank you.
right!
searching for files without a hierarchical structure is not a new concept, and Apple didn’t invent it. It has been fermenting in the heads of researchers for a while, and it has actually been integrated into some applications already.
I must agree that simply removing structure just makes things more complicated. we’ve seen in apps like notmuch that using only tags can work fantastically for some people, but fails utterly for others. maybe the combination of hierarchy and chaos is the best option (think thunderbird 3+, BeFS …)
files in such an environment can be searched by file name, mime type, ownership, permissions, arbitrary tags, size, content, age, in fact any data or metadata you can think about a file.
are you using “find” a lot on linux to execute commands over files selectively? imagine having that sort of nit-picking functionality in any command that accesses files. (e.g. rm tag:naughty_content age:”more than 3 weeks” type:avi; mplayer tag:naughty_content type:avi age:”15 minutes”)
Yeah. It’s been a staple of DOS and CP/M as well as obscenely obsolete versions of MacOS (then known generically as System Software) before it. Too bad Apple can’t be granted a shiny new patent on this wonderful new… uh… step back in time, functionality, technology, flexibility, and organization.
But, thy can..
All you have to do is to tag “…using XML” or “…over the internet” then any old technology is suddenly a patentable invention.
Damn… sadly, you’re probably right.
I have a few questions regarding this new Document Library thing :
– It seems to still allow folders (would be a fucking huge mess otherwise), but only one level. This seems really dumb. Like if you keep your contracts or receipts, it makes a *lot* of sense to have a contracts folder and inside it at least one folder per year (and maybe then per-month).
– How does it handle files that can be used by multiple apps : html, photos/images, etc… ?
Oh and I think I’ve never heard anybody complain about how having folders is hard. I think it’s one of the easiest computing concept to understand for non-tech people. It’s actually like in real life. You’ll have a shelf with a “receipt” folder and then in it you’ll have on folder per year or whatever.
Now, I’m all for adding tagging and metadata capabilities to filesystem with a nice UI so you can easily have a file in multiple folders and search/filter them more easily. But removing the whole folder concept seems really dumb.
People don’t have problems with hierarchical directory structures that they create themselves (like in their own home folder or ‘My Documents’ directory).
However, a lot of people have problems when they find themselves in another directory on the disk/system/network, and don’t know how to get back to their stuff. And Microsoft has been making this harder and harder with each release of Windows, with Libraries being the epitome of ‘hide things behind abstractions so that no one knows where things are actually stored’.
No one had issues with directories in MS-DOS. No one had issues with directories in Windows 3.x. Things got a little confusing with Windows 95 and the introduction of ‘My Documents’. then things stayed ‘normal’ throu Win 98, ME, and XP. things moved with Vista. Then, suddenly things went wonkers with Win 7 where everything that was under My Documents was moved up a level, Libraries were added, and things just got confusing.
But, the real confusion started with the introduction of ‘Desktop’ as a magical pseudo-top level directory, which is actually in the middle of the file system. We’d all be so much better off if that was never introduced, and everyone had to deal with the actual layout on the disk for everything. I believe that’s the root of all the ‘issues’ with confusion with directories.
You have deftly expressed exactly how we got into this “dumbification” mess.
Joe Sixpack had no trouble understanding files and directories with Windows 3.x and DOS. As soon as the Windows ’95 obfuscation appeared, things went south, and we have never recovered.
phoenix,
“However, a lot of people have problems when they find themselves in another directory on the disk/system/network, and don’t know how to get back to their stuff.”
You hit the nail on the head. In fact this is one of the best points that’s been brought up; users can obviously understand their own directory structures. The main aspect that might be genuinely confusing is being exposed to rather arbitrary and scary system directories (including C:\ on windows).
“And Microsoft has been making this harder and harder with each release of Windows, with Libraries being the epitome of ‘hide things behind abstractions so that no one knows where things are actually stored’.”
It is hard to use because it lacks consistency and solid points of reference. Windows puts system directories at the root, but that’s not what a user wants to see. From a user standpoint, the root directory should be initially empty and take on whatever files / hierarchy the user creates there.
The system files, nor their associated hierarchy should ever need to be displayed to normal users. Placing users at the root (even if only a virtual root) makes it that much harder to get genuinely lost since users would be intimately familiar with their root directories.
Novice users needn’t be intimidated by any pre-existing system hierarchy, advanced users should be able to override the configuration, and everyone should be able to work with directories & archives without any silly device directory limitations.
Of course for any paradigm to be successful it must be adhered to consistently in software, which is far easier to do with a new platform than an existing one.
I agree. As a Windows user from Windows 95 to Windows XP (~1997 to late 2006) and a Linux user from late 2006 to now and the foreseeable future, Windows Vista’s “Library” concept has got to be the most fucked up, confusing, retarded, bullshit new “feature” of ANY OS. Period. Well, besides Metro and GNOME 3, anyway.
But seriously… whoever decided to put that feature in Windows needs to be shot.
I find libraries quite useful, because I actually understand how something works without throwing a hissy fit everytime something changes.
The first macintosh while in development didn’t have the notion of files … it had the idea of a “scrapbook” until steve jobs took over development.
Edited 2012-07-26 07:26 UTC
Raskin’s mac was nothing like what it ended up becoming. Pointing that out seems to lend creditability to the idea that files and folders are a bad idea by relating it to the insanely great product that the first mac was, but in reality, that was one of several good decisions the team made prior to launch that lead to its success. The first mac did have files and directories, so if you think it was a good product then those must not be that bad either.
It not whether I like the first mac, but I think that everyone thinks of files and directories as something essential to computers, when maybe they aren’t.
I keep on talking about Design decisions and lock in on here, but it is mostly ignored.
Once you make a design decision about a piece of software as fundamental as directories and files, you essentially lock in that idea.
REST has a different paradigm … while normally implemented with web services, it could be implemented in other ways (for example, apparently it can be implemented via Email – I have a hard time imagining it but It can).
It is different way of accessing information than files. I know fundamentally lower down somewhere there is going to be a files storing the information or the the values the information was computed on, however that is abstracted away from the user.
It is another paradigm of accessing information.
I think my main argument is, why were files and directories chosen in the first place? Is there a better more modern way of abstracting this out.
Most people at work, when they want to find something out use Google or Wikipedia … there is no real concept of a file containing information, there is a stream of relevant information, more akin to picking another person’s brain.
Edited 2012-07-27 17:52 UTC
I understand how they work also as I am sure many other OSNews users do; regardless, they are more of a hindrance than a help. To echo others: they add an additional unnecessary layer of abstraction to the process of finding and accessing a file.
I think that the Filesystem Hierarchy Standards of Linux and the BSDs are smart and logical; the full text of “man hier” should be in any *nix for dummies book. I completely agree with you about Windows. In 3.x I knew where my stuff was. In 9x I knew where my stuff was. In XP I mostly knew where my stuff was. In NT6.x I have no fucking clue what the actual directory structure is and it pisses me off. BeOS, Haiku, and OS X make a good compromise between traditional *nix layouts and the “everything for this program in one lump” layout pioneered by DOS, with /boot/apps/ and /Applications/ respectively.
I never had a problem with XP’s “Documents and Settings”, it was obviously labelled and followed a logical structure for user folders.
Even if made slightly more long winded from Win2k to XP, you still had built in support for overwriting the “Default User” profile with your chosen candidate; avoiding the creation of scripts and policies for things like adding bunch of networked printers configured for double sided printing. This is no longer possible.
I hope someone can point out the benefit of the mess I see in Windows 7, I’ve met plenty of “IT Professionals” who are not even aware of the many possible locations for user data now.
Edited 2012-07-26 16:44 UTC
I agree about W7 libraries. It sounded nice when I first read about them, but now that I have them forced on me, I think they are incredibly confusing.
The problem is the way they mix the physical layout of files on disk with a misguided abstraction of a parallel hierarchical view which is meant to help you, but totally fails.
Why would you need an easy way to find your files when you can just pay for them through itunes like a good boy?
I use a mix of hand structured directories, and Windows 7 native search for organizing my stuff. This way, I can have my structured stuff – like software projects in perfect order, and never need to remember where I put a word document (they just go to an archive, which I can easily search).
But the phone model is completely broken. You can neither organize, nor search them. They are all belonging to apps. Even worse in WP7. For example if you want to attach a PDF from the web to an email, there is no way to do it. The PDF will automatically belong to Adobe Reader, and it can only view or delete them. The email app will not even look at those. And the PC transfer application (Zune) will not touch anything other than multimedia. You can share the link, though
This may sound contradictory, but I don’t think think it’s the concept of folders [née directories] that’s hard for ‘normal’ users to grasp, it’s the concept of hierarchies. Technical people like to organise things in tree-like hierarchies, but (with all due respect), we don’t grasp just how unnatural this is for ordinary people.
Like a books in a bag. Or money in a wallet inside a pocket?
Yeah, hierarchies are so unnatural.
“Like a books in a bag. Or money in a wallet inside a pocket?”
No, like a bag inside a book, which is inside a bag… or a wallet inside a pocket, which is in turn inside another wallet…
I’m not necessarily defending Apple’s decision regarding one level of folders in iCloud, but it’s not hard to see how restricting it to one level make it easier for ordinary people to understand. People put (real, paper) documents inside a (real, manilla) folder, but it’s a rare person that puts a (real) folder in a folder.
Edited 2012-07-26 23:20 UTC
wallet inside a pocket, which is in turn inside another wallet
The restrictive nature of real world is limiting sometimes. But you can hide entire human with a wallet in a wallet if the latter is big enough.
A person who can’t understand it is clearly suffer from serious mental disorder.
So who are these “ordinary” people who can’t live in our hierarchical world?
Who just put a document in a folder on the shelf in a room in a house in a town in a country in a planet in a galaxy in the universe.
One level is clearly not enough.
They are the people with over 100 files on their desktops. And they outnumber us at least 100 to 1.
It’s simple. People aren’t smart enough to use computers. They can’t start their own projects with their own meaningful file structure. If you paint a texture for a game project, you’ll just have to live with it being right next to the wedding photos. Like web design? Too bad, your index.html is just going to have to sit next to all your other html files, including other index pages. You’re not smart enough to program anyway, go back to watching Pit Boss you dummy.
You’re just not smart enough to use a computer, let alone use it to do actual work.
At least, that’s what this nut-case thinks. I doubt Apple would ever go so far as to remove your access to your own files. Too many art “geeks” depend on them. I mean think about it, when you develop software, a website, or even a movie, you know you Oliver is simply off his rocker. Apple depends on those “Geeks” he speaks so poorly about to produce content for Apple to publish, websites for Macs to read, and programs that run on their computers and i-devices. That’s why they can’t even take your console away, because they know their developers still need it.
There’s only so much iOS-ification you can do to a computer before it becomes too much of a hassle to develop for, so you can count on this just being a fantasy.
I deal with a lot of Windows users in my day job.
People understand directory hierarchies. But Windows makes ‘Desktop’ the root of a pseudo-filesystem, and ‘My Documents’ the root of another pseudo-filesystem, both of which are only accessible from icons in Windows Explorer.
When people have to deal with the actual filesystem on the disk, they get confused, because C: is the real root of the filesystem, and ‘Desktop’ and ‘My Documents’ are buried several levels deep.
I’m a firm believer that all the ‘help’ MS provides to users is actually hindering more then helping.
If you provide users access to the filesystem, they’ll figure things out. Hide things behind layers upon layer of abstractions to ‘simplify’ things, and you’ll just confuse them.
Agreed, 1000%!
The fact that they’re “buried” is not so bad. Hell, look at Linux and its typical directories:
/: Root file system
/home/user: User’s home directory
/home/user/Document’s: User’s documents
/home/user/Music: User’s music
/home/user/Pictures: User’s pictures
/home/user/Videos: User’s videos
…etc…
They’re “buried” (not in the top level directory), but it’s relatively simplified and makes perfect sense to me… completely logical and easy to remember. Windows Vista, as shitty as it was, did make some improvements here though, I have to admit (“C:\Users\User Name” instead of “C:\Documents and Settings\User Name”
But you’re wrong about C: being the “real” root of the file system. It’s only the root of what is most commonly the system drive in Windows and DOS before it, which is usually set up to be a whole disk but can sometimes be a smaller partition. Every hard drive partition, CD/DVD-ROM disc and USB drive has its own root file system. In UNIX/Linux, there is one virtual filesystem under which *everything* resides.
Indeed, Linux is guilty of the the same sort of confusion, and more so when you look at system files! I still admire the Amiga way of arranging things. It keeps each device separate, so the highest point in the file system is just a list of devices. This also corresponds to the desktop, which contains all the attached drives and can hold shortcuts, but isn’t actually a directory. It’s just a fancy list of devices. Very intuitive and very quick to grasp.
Luckily a typical low-skill computer user of one of the mainstream Linux distros, you practically *never* have to leave your home directory and enter such system territory as /etc and /boot. You might need /etc if you’re dealing with daemons, but I doubt that an inexperienced user will even know what that is let alone need to run them.
Similarly, with Linux’s stronger separation from root from users, a user most likely won’t have to worry about totally screwing up their entire OS with malware so it won’t boot, and is even more unlikely to need to lurk around in /boot. Even then, with a clean segregation of / and /home partitions, a fix without losing any of your personal settings is just a reinstall away, keeping your existing /home untouched.
Windows is certainly not any cleaner with its C:\WINDOWS, C:\WINDOWS\SYSTEM and C:\WINDOWS\System32 directories, among others. In my opinion, it’s much more of a mess.
I will say though, that I’ll take something like /etc/hosts over C:\WINDOWS\System32\drivers\etc\hosts any day. [Drivers? Really? WTF?]
No. Linux is almost always very straightforward, with none of the “pseudo” weirdness found in post 3.x Windows and without the wholesale obfuscation of system internals found in OSX.
The basic difference between the hierarchal models of Linux/Unix systems and DOS is is very simple:
– with Linux/Unix, all devices, files and folders are contained within the root partition;
– with DOS, all partitions/devices exist together at the root level, and one does not contain another.
If you are confused by system files, just don’t look at them!
Which is all well and good until something doesn’t quite work. There are plenty of tools and things out there which need to have their configurations manually set in text files – some of them are in /bin, some are in /usr/bin, some are in the home directory and so on. For example, on my Ubuntu 10.04 box I can’t save changes to my graphics card settings without editing the text file manually because something funky is going on with the Nvidia configuration tool. I’ve done it several times, yet I still couldn’t tell you which file I need to edit or where it is until I spend a while trawling through forums and FAQs. The same for SMB mounts of my NAS. They’re not reliable unless I mount them by hand, and so to have them available at boot means wading through the system files.
I appreciate that that’s a little OT, and that with Linux all the folders are basically as you see them, but it doesn’t stop it from being confusing / messy. Again, comparing it to the Amiga way of handling drives, mounting media is quite unintuitive, with things like a CD drive having nothing to distinguish it from any folder on the hard drive. IMHO there shouldn’t be a filesystem higher than the root of any drive, and having the root of a CD at /mnt/cd0 or whatever just doesn’t make sense in the whole desktop/folders paradigm.
Honestly, I don’t know the last time I had to set up system files. Configuring SSH or MPD maybe? I doubt that a typical person coming from Windows would even know what a daemon is as I mentioned, and I wish them luck on somehow managing to download some kind of malware to destroy things like /boot as I mentioned to truly wreck a system, Windows-style. UNIX doesn’t make it easy.
For those people who *are* likely to play around by experimenting heavily and screw up OS installations, well, chances are they’re intermediate to advanced users, and at least have an understanding on the Linux filesystem to get things back up again if needed. For those users who do hear of some interesting daemons that they’d like to learn to use, well, a couple commands for start/restart/stop and editing a file or two under /etc is really not that difficult, and by the time they learn of it they’ll probably already feel at home in the OS.
I don’t recall ever having to modify a program’s configuration that placed its config files in /bin or /usr/bin. If the configuration is not within a configuration window of the program itself, it’s somewhere under /home/user. If it’s a daemon, it’s somewhere under /etc, or for some web-based daemons it can be accessed as a GUI by logging into a web page on the machine running it. Simple as that.
If configurations files are in /bin or in /usr/bin, something is probably wrong. “Bin” is short for binary, so, only binary files (apps) should reside in such directories. User-specific configuration files are usually hidden in the user’s home directory.
See how these names/organization make sense?
However, global configuration files are usually found in the /etc directory. This directory name is probably held over from the early Unix days. It doesn’t seem intuitive, but it is easy to guess how the name evolved, and it makes sense when thinking of the name in that regard.
By the way, aside from the intentional “pseudo” obfuscation, Windows has similar problems with cryptic system directory/file names. Anyone who has been “dumbified” will have the same trouble navigating Windows system internals.
It’s much worse when a “tardified” Apple user has to go into the OSX system internals. Such users have to contend with two sets of system directories: the hidden, cryptic Unix bottom layer, and the tard-oriented top layer.
I think we’ve found the “root” of the problem.
If one has trouble remembering things like file paths, all one has to do is write down the path somewhere. Or, one could record the path into a file and give it a name that one can remember (and, of course, put it in a directory where one can remember it).
/etc stands for Editable Text Configurations or “Et Cetera,” because it’s not a binary, a bootloader, a library, /var-iable size data, or anything else the top level directories contain.
Actually Windows does have a single virtual file system, you just don’t usually see it. You can also mount drives as folders without drive letters in Windows.
As for your example of /home/user/music etc. Where do you think that came from in Linux?! Hint – it wasn’t there in Redhat 6 but it was there in Windows 95!
No it wasn’t. “My Music” and “My Pictures” were introduced in Windows 98.
As a pure single-user operating system, Windows 95 didn’t have shit for storing personal user files until Win95-OSR2. Windows 98, still single-user though adding kludges in an attempt to expand, finally brought the “My Documents” folder concept to Windows users who bought retail versions instead of OEM.
In Linux, the closest comparison to My Documents was… drumroll… /home. Every user (not just one, unless there is only one user set up) has their own directory to store all of their personal files, and permissions at the file system level to allow or restrict other users on the system from accessing their files.
I’m sure it was common for people to organize their home directory with documents, downloads, music, video, etc. subdirectories long before similarly-named directories have started coming with new installations of more recent distros. I used to have all of those, in lower case, before distros started conveniently adding them by default while screwing things up and making the first letter capital (certainly not optimal on an OS whose file system fully respects the capitalization of file names and has a nice command line interface). I personally preferred it when my home directory came clean on a freshly installed system, so I didn’t have to later clean it up by nuking the caps.
You have a valid point but you have to see how other people use their computers, I have seen many people place important files directly in the Root of the file System, whether is Windows C or Mac’s HD. Although others have point out something valid and is that it shouldn’t be hardcoded. Yes I know it’s only ICloud but the article spawned a interesting debate by calling it Anti-n Directory Structure Movement. I firmly believe that when Apple finalize the iOS Mac OS merging there will be some kind restriction maybe not to the iPhone iPad point but still some limitation in folder creation, maybe at the end we will have only smart folders temporary folders with no depth.
The basic difference between the hierarchal models of Linux/Unix systems and DOS is is very simple:
– with Linux/Unix, all devices, files and folders are contained within the root partition;
– with DOS, all partitions/devices exist together at the root level, and one does not contain another.
Other than a couple of exceptions (GoboLinux, etc.), Linux/Unix perpetrates no “obfuscation” of the type that phoenix explains is inherent in post Windows 3.x systems.
With Linux/Unix and DOS, the organization makes sense and nothing is “pseudo” (but some things are “sudo”).
OSX is possibly the worst offender, in that it tries to completely hide the system directories from the user, which additionally makes things more complex in regards to how the system files are actually handled internally.
Of course, GoboLinux also hid the system directories from the user, but it was much more open and straightforward about it. Also, the GoboLinux method went further than OSX in coordinating the hidden and non-hidden sections, and the GoboLinux links system certainly was much better suited to an eventual elimination of the hidden section.
And yet not a long time ago there was an article documenting how little sense it has, lots of weird historical baggage… http://www.osnews.com/story/25556/Understanding_the_bin_sbin_usr_bi…
(and of course religious war in the comments)
Edited 2012-08-02 00:07 UTC
As long as you can convince the users that /usr has absolutely nothing to do with them! 🙂
Quite right. As long as there is a possibility of an A: or a D: drive, conceptually the file system begins below C: even if it is invisible in practical terms. One of the great tragedies in computing history is that when MS … was inspired by CP/M they didn’t take the opportunity to clean up THAT mess. And here we are, still stuck with it 40 years later.
I beg to differ. Given how much revenue OSX generates compared to iOS one might very well say that it wouldn’t hurt Apple much to ditch entire platform apart from the fact that OSX is the development platform for iOS. The latter could be remedied by porting XCode to Windows.
I don’t think there is any intrinsic need for Apple to continue supporting UNIX desktop (or server). Of course, for the time being the OSX is development platform and testbed for technologies but we’ve also seen that Apple provides special hardware for developers and noone else. In the end of the day we could only have idevices and development system for the subscribed developers and Apple’s revenues largely maintained at the same level (but profit margin could be even higher since they wouln’t need to support millions of end users).
They would lose ownership of the “whole widget” if they gave development away to Windows users, and especially if they gave up their PC platform.
Sure, they could survive financially, maybe even for years after the bad press died down… but Apple would be a body without a head if they did that. The magic behind the Apple experience that so many users enjoy would not survive.
However as a Windows user, I would welcome that development. Microsoft is pretty good about making sure Windows stays a decent development platform. They might even help Apple cut off its own head by working with them to build those compiling options and Apple libraries directly into Visual Studio. Then Apple wouldn’t have to worry about Xcode either. They could just be a dumb phone and gadget manufacturer like everybody else. It would be sad, but I’d get over it pretty quick.
Edited 2012-07-29 04:27 UTC
Mostly agree with you in that I prefer having my own directory structures. But the one time I actually like having all of my files dumped into one folder is with mp3s. Ironically, the pisser is that the one app that doesn’t allow me to organize files easily this way is iTunes.
Seems like Apple is hellbent on ditching organizing things in folders, except the one time I actually don’t want it
Interestingly, I absolutely always use the iTunes-style of directory structure for my MP3s. To transfer files to other devices (e.g. my phone, other computers), it’s so nice to be able to navigate to them in my file manager, select the album I want to copy and drag it over. I like it so much, that when I developed my own music player (for a custom jukebox) I made it use the same directory structure based on the contents of the ID3 tags.
Have never been an ‘album’ sort of guy myself, so I usually have about 5-6 tracks per artist at the most, with a few exceptions here and there. Therefore, trying to store them in some kind of logical directory structure seems like a waste of time for me. Of course, there are others who prefer the folder structure, so it’s always nice to have options. But Apple has never been about giving people the option to choose one way or the other.
You are telling me that if I upgrade to Mountain, I will not be able to have folders inside folders!?
Please tell me I got this article wrong.
Yes, you did read it wrong (although I think Thom’s summary doesn’t help). Only when saving documents in iCloud are you restricted to one-level folder hierarchies. When saving documents on your Mac, everything works as it did before.
Organizing files with meta data or by having applications track their files works fairly well until a person needs to transfer data to another computer/person. If I want to e-mail a file or archive a development project or (S)FTP something then meta-data and one-level folders get in the way. I don’t think directory trees are going anywhere because nothing better has come around to replace them. There are too many cases where people need to separate their data into groups independent of applications.
Problem? Most people don’t organize anything and live in a complete mess…
Additionally, I think the author really speaks about “most people” and excludes advanced users (formerly: average users), power users and whatever he means by “geeks” (maybe those who can actually operate a computer).
He writes:
In general, documents belong to an app. While there are often several apps that can use the same document, we usually have a preferred app for each document type.
This is, in my opinion, debatable in two ways:
1st: Users don’t use programs at all. They just “open files” and “do stuff”. Many users also confuse actual programs with files. You cannot make them understand what a hierarchy is. Things they could do in real life are predicted to fail when attempted using a computer.
2nd: The author misses that depending on use, file types may be opened with different programs, e. g. a HTML file for editing with a code editor, for viewing with a web browser. The same may be true for images, e. g. one program to view (or slideshow through) a collection of files, another program to edit them, or maybe even more programs, depending on what edit task may be better in this or that program. The “one program” association doesn’t work here.
Maybe I’m wrong and that’s just a new Apple-specific thinking. (I’ve been a Mac user, but such thoughts never occured to me.)
Accessing a file’s content means opening it. Basically there are two kinds to view at this “problem”: One aspect is to “call a file” (which then opens the hopefully correct program and loads the file). This is done from some sort of file manager, library or database view — whatever represents files (carrying content) to the user. Another aspect is to start the program desired and then select the file from within it. Depending on individual work ergonomy, the one or the other aspect might look as the “most natural” one. But I think I’m already talking too much poweruser-like here again. 🙂
Directories are a natural solution to the problem of clutter. It’s quite synonymous to real life organisation skills, so while the on screen execution might pose a slight learning curve, the *concept* should hardly be foreign to anyone.
Even if someone wants to keep all their files (or directories in this case) in one big pile, let them…I couldn’t care less what others do, but that’s a terrible reason to deny me the ability to use directories. The two approaches are not mutually exclusive, and actually they both enhance one another.
In other words, let us choose what’s best for ourselves. Of course user opinion seems to be very unimportant to corporations these days.
But it isn’t… Directories in conventional file systems simply don’t work the same as real world “directories”. There are massive differences. For simplicity, lets just say we are comparing directories to drawers in a desk:
1. There is only one way to get to the top left drawer of my desk. It is impossible for me to open the bottom right drawer and look inside and end up seeing the contents of the top left one…
2. If I put a file in the top left drawer of my desk, it cannot also be the drawer below it. And I don’t mean a copy – I mean the same damn file!
3. I can’t open a drawer and end up looking at the contents of a drawer in someone else’s desk.
4. I can’t accidentally delete a drawer.
5. When I put a container of some kind (which holds things) into a drawer, it is invariably not another drawer…
I’m just saying it is an imperfect analogy to the real world, and most of the imperfections are actually intentional features. They are useful of course (symlinks, network shares, etc.), but we can’t just pretend that it is all simple – it isn’t. We made it complicated over time – admittedly to fulfill real needs, but that doesn’t mean it is without complexity.
You are right. They are not mutually exclusive. But Apple seems to take the approach that the best way to “teach” a new behavior to its users is to thrust it upon them and see if it sticks. I guess this is because they do not do user acceptance research the way most companies do. Anyway, they are wrong sometimes – well see if this is one of those times. My gut says that their users will have no problem adjusting to this…
You can choose for yourself. You can choose not to buy an Apple computer, can’t you? And then if Microsoft does the same thing, you can always run Linux. And if enough geeks rage and end up doing this then all those corporations will wake up and put things back the way they should be right?
The problem with this is Microsoft and Apple don’t sell products for geeks anymore. They haven’t for some time, we are simply too small a market to matter to them…
galvanash,
“But it isn’t… Directories in conventional file systems simply don’t work the same as real world “directories”. There are massive differences. For simplicity, lets just say we are comparing directories to drawers in a desk:”
I disagree, when speaking about the *concept* of a directory, it’s pretty close to how people already organise things in the real world. An office can have file cabinets, labelled drawers, labelled files, individual documents in the files. A disk directory is a very good representation of these abstractions, and I dare say anyone already familiar with the physical equivalents would automatically be comfortable with an identically structured disk directory. Do you really think otherwise?
Heck, if you wanted to loose a level of abstraction, you could get ridiculously literal and render graphical representations of unique rooms, cabinets, drawers, files, etc. It would show the exact same information as the directory layout, would that make you happy? I think most people are mature and educated enough to understand directories without office pictographs to correlate to the physical world.
All your points seem to revolve around an implementation of the directory abstraction rather than the concept of a directory itself. Please note I’m defending the concept of a directory, not a specific implementation.
“They are useful of course (symlinks, network shares, etc.), but we can’t just pretend that it is all simple – it isn’t. We made it complicated over time – admittedly to fulfill real needs, but that doesn’t mean it is without complexity.”
Well, that’s true, but the debate about whether directories should be overloaded for other purposes is different from the debate about whether they should exist. Even without any of those “features” you brought up, directories still have an undeniably useful role in allowing users to organise files.
“You can choose for yourself. You can choose not to buy an Apple computer, can’t you?”
In the case of apple, yes, but it does nothing to dispel criticism of their practices.
Sure its close – I’m not denying that. It is a good analog even. It has served well for for the last 30 years or so and still does. But it does have complexities, and many of them shatter the analogy. Sure, none of this is a show stopper – most people figure it out if they are inclined to bother.
Yes actually. I think that for your average user, a filing cabinet is where you put things you never need, a drawer is where you put things you rarely need. The things you use all the time you put on top of your desk or in arms reach at least – you keep them close by, not hidden away.
Computers are really really good for a particular thing, in fact I think that this is their primary reason for existence: they are good at finding things almost instantly if you can tell them enough about the thing to make it distinct from all the other things. Computers are essentially information retrieval devices, and yet we use them as if they are filing cabinets (opening up folders, looking for things…)? Does that make sense to you?
But they don’t. Not really. The do have an undeniably useful role in allowing computers to organize files, but for users? They are just one way of skinning the cat.
All a directory path is is a unique identifier into a block storage space – it is an index. It is THE index. For the computer they are critically important. But all it tells users is where things are in an imaginary hierarchical space that in reality does not exist.
I dont’ want to have to remember where things are… If I know what I am looking for, or when I made it, or who gave it to me, or what I made it with, I can find it quite easily. All directories really do is hide things from my eyes.
I am not, btw, arguing for the elimination of hierarchical file systems at all. What I am saying is that if the computer can show you what you need, when you need it, you don’t actually need to make the hierarchy visible to users. The don’t in fact need it for normal day to day stuff. It can be like a filing cabinet – the place you put stuff you only need in a crisis (i.e. backups).
A system that largely eliminates the need to manage and orginize files is imo a good thing. It is doable. It is not dumbification. It is arguably better for most people. Maybe not everyone, but certainly many. It is at least worth trying. I’m still waiting for someone to build one that works though
Just because that is how you might use your computer it does not mean that that is how every other person on the planet does.
Business computing would grind to a halt if user created directories were removed. If Microsoft tried this it would finally be the year of the Linux desktop, and if Apple tried this they would soon be left with dwindling iPhone users as the Mac faithful migrated to a Linux desktop and a Linux based phone (like Android).
For every directory that can be hidden, there is someone else that had to create it; and they need a means of doing so.
That is so unrealistic. MS would probably backtrack and a lot of ios and mac users are so whipped that they would just go along with it.
Yeah, “finally”, like it was supposed to be so many times already…
Bullshit. Just start throwing folders and documents in the trash without giving it much thought. You can nearly empty the drawer in no time, no doubt having the potential of throwing out some things you didn’t mean to. And you have until the garbage men come to get it out of the recycle bin/trash/whatever. Drink some booze and you’ll be even more likely to f–k up and not even realize it. It’s not really any different from a person dealing with a computer file system who actually understands the concepts… f–kups happen, whether real or virtual.
Edited 2012-07-26 07:45 UTC
I was going to respond with an argument about differing definitions of “accidentally”, but I can’t… I’m laughing too hard at the mental image of me drunkenly destroying my desk. Thanks for that
That’s just a mental image for you?
I’m getting too old for this shit
.
Edited 2012-07-26 08:26 UTC
When you’re done with the desk, can you come do my office?
Funny a few years ago I was thinking that virtually infinite folder depth was a pain in the posterior. When you are using your own computer you can surf your folder structure really easy but if your are doing tech service or using some shared storage or using someone else computer is horrible.
When I was working at a tech service I used to organize people’s file structure as a bonus when the job was pretty straight forward and people used to thank me a lot for that because they could find their files really easy and identify garbage files and duplicated content.
I actually do that myself and I am Software Engineer I don’t think I qualify as dumb enough to use a computer, is just that I like to keep things simple.
Edited 2012-07-26 00:40 UTC
netcat or NFS are my preferred ways of moving files around internally.
But I’m not a mac user.. Neevvaahh!! :p
I have files inside of files all over the place. It’s the only way I can find anything. Some of my files go 6 deep (gaming stuff).
Apparently nothing. I would certainly expect an information architect” to know the difference between file system, presentation and cloud service.
So we’re smart enough to be at the top of the food chain for a long time, fly to the moon, create computers and stuff but not smart enough to grasp a folder structure? Come on.
I don’t know what he’s trying to say. My browser only has one file menu.
That’s funny. The operating systems I use regularly (including Windows) have had this, or a similar, feature for a loooong time. Maybe it’s just a revelation to Mac users. It is handy though.
Sure, I guess. If your definition of cross-device is as narrow as “between OSX , iPad and iPhone”.
uh, yes it does. It even supports more devices than iCloud.
My conclusion from this article is “file management on Mac must really have sucked the devils cojones before”.
The article is just painful to read. It’s full of stuff like – I didn’t think feature X was useful, but then Apple did it and it’s awesome. The guy is an “information architect” – whatever the fuck is that – and he can’t grasp folders … color me unimpressed.
Hmmm… I don’t know when they introduced it, but my iBook from 2003 running Tiger has that feature. Maybe it just wasn’t turned on for that user, or maybe he’s talking about Macs a decade ago…
Hmmm… I don’t know when they introduced it, but my iBook from 2003 running Tiger has that feature. Maybe it just wasn’t turned on for that user, or maybe he’s talking about Macs a decade ago… [/q]
2003 is a “decade ago” (almost)
+1 about the Amiga filesystem though!
Well, that’s what I meant
anyone who’s found the easy-access shortcuts in the left column of a Finder window as a new feature must only have experience of Macs at least as old as mine. My Mac experience has a hole in it from OS 8 to OSX 10.4, so I’m not sure when OSX got it, but it must be a decade ago…
What do you mean you can’t nest folders in folders in Mountain Lion? I have Mountain Lion and I am doing it right now. If you mean in Launch Pad, yes, you only have one level, but as far as your directory structure for your data, those still work the same way they always did.
It’s in iCloud that you can only nest up to 1 folder, not only that only the application that saved the file has access to it, so you can’t just create an email and attach a file, the app is the one that needs to initiate the email (i’m assuming via API’s).
So if you save a file to the iCloud it’s more of a hassle to share it and you can’t open it from another program.
Well, that last bit is not entirely clear. If Spotlight is set up to index the local copy that iCloud keeps in ~/Library, then opening the file up in another application becomes entirely possible. Failing that, no doubt the makers of the Found app are hard at work adding iCloud to their list of sources.
And as has been pointed out above, you can open the iCloud Open dialog and drag out the file to your desktop or wherever in your file hierarchy you like. Work on it with a dozen different apps, then drag it back in if you like. I’ll let you know when ML is downloaded & installed.
It’s a free offsite backup and syncing facility that works with Apple’s own software. If you don’t like it, don’t use it. The rest of the Mac still works just like it always has. Talk about first-world problems …
“Free”? It requires Apple device to sign up (while with MS Skydrive for example this works basically with any browser, on any platform)
Technically, a “hiearchical set of directories” is just a view over things anyway. So.. really, there is not “directory” structure. It’s really just a view that allows us to make better sense of things.
Orphan files FTW!
Are you saying that on the deepest view of reality, there is just a rapidly spinning piece of metal displaying fluctuating patterns of magnetization?
Say it isn’t so! Don’t take my file system away from me! Pleeeease!!!!
This isn’t about file systems or hierarchical directory management. It isn’t about limiting nesting to one folder. It isn’t about vendor locking either. Thom, you are missing the point entirely.
The most profound thing you wrote in your article was this:
Yes. Seriously.
Geeks will rant and rage, but facts are facts. People use email, yes EMAIL to store files. Not dumb people, and not a few of them. LOTS of them. They do this on purpose. Microsoft found this behavior was overwhelmingly common when they researched user behavior, and Apple (as they always do) came to roughly the same conclusions doing whatever voodoo they do internally. Ask any corporate email admin – they fight with this constantly…
Why? Because they discovered, through their own personal experience, that it is the easiest and most reliable way to make sure they can find the file when they need it, regardless of what computer they are on or where they are.
Email uses completely different semantics for finding things… The primary attributes are different.
File systems are all about “where” things are. You can of course augment things with searches, metadata, whatever, but its all about location, location, location. Where is the heart of the matter, and no amount of metadata can really change that.
Email has no “where”. It doesn’t matter. There is only “who”, “what”, and “when”. Email allows one to blissfully ignore where they put things… Everything in just “in their email”.
So what is it about email, considering how bad all us geeks know it really is as a file storage system, that makes people use it anyway?
1. Its gloriously flat.
2. It has extremely rich metadata.
3. It is primarily organized by ownership and chronology, i.e. “who” and “when”.
4. It is location independent.
5. It is platform independent.
6. It is device independent.
7. It is network accessible using a fundamental protocol that is supported by just about any device you can think of.
Cloud storage IS email. Its just email for file storage. Does it make sense to model a device independent internet storage service after a conventional file system? No. You model it after the thing that people are already using to do the same thing… Email.
ps. Yes, I know IMAP supports hierarchical folders. But the only people that use them are power users and geeks. Folders in email are, primarily, metadata that categorized things in an exclusive manner. Something may be in a folder, but the folder has a meaning attached to it. Almost no one organizes their email, because they simply don’t need to…
ps.ps. Yes, I know ONE of you is going to reply declaring that I am completely wrong because you meticulously organize your email all the time… Please don’t bother – I doubt I could fill a small stadium with the lot of you. Sorry, but your behavior is completely atypical.
ps.ps.ps. I’m not saying I like this trend by the way, I’m still on the fence myself. But I do see the rationale behind it – it isn’t just “dumbing things down”. I’m just saying you need to appreciate the argument for why they are moving file storage in this direction.
galvanash,
“Geeks will rant and rage, but facts are facts. People use email, yes EMAIL to store files. Not dumb people, and not a few of them. LOTS of them. They do this on purpose.”
I think you missed Thom’s point entirely. He wasn’t criticising email, he was criticising that users had to use email to work around limitations of the device.
I think you missed my point…
Yes, in his particular example that is what they are doing, but I’m not talking about users using email to get around device limitations – many of them do this (and have for years) as a way to get around having to remember where they put things… In fact as a way to avoid the complexities of file systems. And they like it.
iCloud (and the Document Library thing) are modeled after email because people do this – it is in fact largely emulating the way email works.
I’m not saying that hierarchical directories are not useful at all. I’m saying that they are not terribly useful when what you are building is file storage that:
1. Only serves the purpose of storing user generated data files.
2. Is meant to be directly accessible remotely by the applications generating those files.
3. There is only one root… You cannot nest file systems. By design.
What purpose do directories serve in this scenerio? They are for categorization, nothing more. And primarily it is exclusive categorization, i.e. the way most peoples minds work.
In a conventional file system, directories are indexes – they are critical for system operation. Their primary purpose is not to assist users in organizing their things – it is to reduce scope (make things easier to find to organizing them into smaller subsets so that it is not necessary to search through everything to find something). But they are user visible indexes. They create vast trees of information that the user must navigate through to get to things… We all got used to them – but lets not forget that they were not created for us. They are first and foremost a performance optimization…
Email does not work like this. It is indexed, but the indexes are not user visible. You don’t get to things by remembering locations, you just search for them (by whatever metadata you choose) in a completely flat storage space. It does not attempt to subdivide the problem space, a complexity introduced to assist performance that is not really needed anymore.
Many users like this, actually prefer it. Not geeks, but your average joe computer user. Again, I’m not saying this is the “right way” to do things – but it is a perfectly valid and sane approach to file management, and it has tangible advantages.
“I think you missed my point…”
Oh I got it, it just doesn’t seem relevant to what you were responding to, but I’ll let you and Thom sort it out if he cares.
And for the umpteenth time, tagging & meta data are not mutually exclusive to hierarchies. You keep implying that it’s gotta be one or the other, but it’s not. Even with the email example, people today already use both directories and metadata tagging together, they are non the worse for it (I personally use both). I get that you love meta data, great for you, and it is great for me too. But allowing others to use directories in no way impedes the use of metadata. I’m getting awfully tired of pointing this out.
It should be your choice to use directories or not for yourself. I don’t care in the least whether you create another directory in your lifetime, why are you so adamant that others shouldn’t be allowed to? Are you more interested in preaching the one true way than coming up with a practical solution that can suit both our needs?
I know that. I even stated so in another post. It doesn’t matter though. I don’t look at this problem relative to the status quo – to me it is a fundamental thing. You can’t just tack on metadata and call it a day – it doesn’t actually solve anything.
The problem is not that users use directories to organize files. The problem is that users organize files. It is totally pointless… It serves absolutely no real purpose. It is a distraction. It is stupid. It is nonsensical.
Yet I do it every day. So do you. Why? Because my computer is designed to make me think that I need to. Adding search didn’t make me stop doing it. Computers thrust the concept of “file management” on me at every turn, you can’t escape it – it is primal to modern computer interfaces. The only way to get rid of it is to expunge it, otherwise it will never truly go away.
Anyway, I’m not interested in an argument about it. I have my notions, you have yours. I said my piece – to each his own.
I’m going to extrapolate what you’re saying to the real world. Basically you think that nothing should be organized a long as it tagged labelled, which basically amounts to a massive wall of documents covered in post-it notes labeling the various pages with strings showing connecting all the related documents. Those kinds of rooms are the rooms of crazy people.
All information in the world is organized in hierarchies simply because it would be too distracting to shove all information everywhere at a person.
Look at a document. We have paragraphs, headings and subheadings for a reason; they’re not strictly necessary, but we use them to make things more manageable. Same thing with folders.
Edited 2012-07-26 10:50 UTC
Ugh.. Im not opposed to hierarchies. Im just saying your computer and its applications can do a much better job of defining them than you can. LET IT.
Hierarchies are great for browsing lots of things. So use them for that – the system can define them for you (by size, by types, by arbitrary metadata, whatever). The thing that is wrong about directories (from a UI perspective) is that they are location specific taxonomy, and they are exclusive. It is an artificial limitation – it is taking power away from you and making you waste braincells.
We even have stupid workarounds, like symbolic links, complicated things that serve no real purpose other than a UI crutch. They shouldn’t be necessary – the only reason that they exist is because we use hierarchies as the primary means of organized storage. We all concern ourselves with something that in reality doesn’t even matter – “where is my file?”. Why the hell should it matter where your file is – what matters is if you can get to it easily when you need to.
Im just saying why not let your computer keep things organized for you? That way you can, I don’t know, go do real work.
Edited 2012-07-26 16:13 UTC
galvanash,
“The thing that is wrong about directories (from a UI perspective) is that they are location specific taxonomy, and they are exclusive. It is an artificial limitation – it is taking power away from you and making you waste braincells.”
Oh you are hopeless, maybe this will help: DIRECTORIES ARE NOT EXCLUSIVE TO TAGGING AND METADATA!!! You seem to be wilfully ignorant of this fact and every single one of your arguments thus far has relied upon that ignorance to make a case in favour of metadata and against directories. You prefer tagging, fine, however you don’t speak for me or anyone else. The big pile approach is a regression for millions of users and professionals. It’s an arbitrary decree that computers shouldn’t enable us to organise our digital files as we would in the physical world, all because you can’t spare the brain power.
“Im just saying why not let your computer keep things organized for you? That way you can, I don’t know, go do real work.”
To the extent that works, then sure, but nobody here has argued against that.
Read DeadFishMan’s post:
http://www.osnews.com/thread?528371
If a user uses a device to download music and movies, then metatags should work great. No user brain cells wasted here. Though it is still not an argument against *allowing* for directories. Permitting both is a simple solution that works for everyone, there’s no reason to get authoritarian over how other users choose to work.
Edited 2012-07-26 17:27 UTC
Yeah I know, Einstein said once that he never learned by memory any telephone number because brain connections are scarce, that’s what telephone guides are for.
For the 2nd time – I know that already.
How do applications access files? By directory path. Its baked into the file system. Its baked into the APIs. Its very, very hard to change. The point is you don’t have to change it – you just have to make it difficult for users to muck up.
If you build a system that is designed to manage the taxonomy of files with metadata, and build applications that are designed to use this taxonomy to access the files, how do you maintain system integrity when uses can just willy nilly change the primary key of a file behind the systems back? Sure, you could use some sort of other metadata and graft it on top and let the system use that instead of the primary key – but that is horribly inefficient…
You end up with a system where only the user really knows how to find anything – the system has to do all searching because it cannot establish any meaningful rules as to where things should be put. We have that now – it sucks. Every hard drive looks different. Things are invariably a mess. Sure, you can find your stuff – but no one else can.
Im sorry but it is not as simple as grafting on metadata. We have already done that – its been done for years. It hasn’t worked, because it is not a fundamental change – the primary mechanism for accessing files is still by directory path, and we let users play with it to their hearts content.
Look, there are compromises that can be made – I get that. But I am trying to convince you that you don’t need directories, not that you can’t have them. Hierarchal storage is efficient. It works. There is no need at all to get rid of it. What is needed is to make it impossible for inexperienced users to break system defined storage hierarchies, once you do that the number of interface paradigms you can use to present things to users drastically increases, because from the computers point of view things stay where they are put.
If you graft some form of hierarchial metadata (which can be easily done with simple tags) onto the filesystem to let user organize things for their own purposes, that is perfectly fine. But it shouldn’t be the primary key!
A system admin or developer need not worry about what I am describing. File systems will more than likely always offer hierarchal storage – but you don’t need to actually expose that to your average user at all. In fact, what I am saying is that it is counter-productive to do that – it creates an insurmountable obstacle to improving the status quo. You cannot reliably create a system like I am describing if you let users muck with the primary keys…
WHERE things should be stored should be left up to the system. User file organization and and the primary access mechanism for files are two completely orthogonal concepts. THEY HAVE NOTHING TO DO WITH EACH OTHER! The first is a completely arbitrary concept that matters to know one but the user, the 2nd is quite the opposite.
galvanash,
“How do applications access files? By directory path. Its baked into the file system. Its baked into the APIs. Its very, very hard to change.”
That’s a problem with the implementation, not a problem with the directory concept. I completely agree that it’s difficult to enhance the paradigm on existing platforms. Even if you provide the API’s, legacy software and conventions will be a sore point and it’s unlikely for a unified paradigm to emerge without compromises. These wouldn’t be a problem for a new platform, but that may not be a luxury we have now.
As such, we should be asking ourselves ways we can get there, but not by eliminating organised directories.
“how do you maintain system integrity when uses can just willy nilly change the primary key of a file behind the systems back? Sure, you could use some sort of other metadata and graft it on top and let the system use that instead of the primary key – but that is horribly inefficient…”
I’m confident all the technical problems like indexing multiple criteria are solvable, and actually many file systems already use surrogate keys. In linux for example we have an inode which serves as a primary key. The user doesn’t handle these directly of course because of the file naming abstraction. I’m happy to discuss technical designs, if you want to, but it shouldn’t be relevant to a user.
“But I am trying to convince you that you don’t need directories, not that you can’t have them. Hierarchal storage is efficient. It works. There is no need at all to get rid of it.”
Is it possible that we have nothing more to argue about?
“What is needed is to make it impossible for inexperienced users to break system defined storage hierarchies, once you do that the number of interface paradigms you can use to present things to users drastically increases”
I don’t think the number of paradigms is as important as being usable. But I agree with hiding OS details from the user at least by default. Normal non-admin users should never have to see or deal with system directories, so there’s no reason to needlessly distract users with them.
As an example: On android, I find it gnawing that I need to navigate through the root directory full of system directories just to get to the sd card. It’s not difficult, but those steps seem unnecessary and ugly to me, especially having to scroll down to find “/mnt/”. Each application seems to implement it’s own file browser, so there isn’t really a clear way for android to fix it now.
“If you graft some form of hierarchial metadata (which can be easily done with simple tags) onto the filesystem to let user organize things for their own purposes, that is perfectly fine. But it shouldn’t be the primary key!”
It’s how extfs already works. Each directory is in essence an index pointing to inodes, for other meta data you could just add more indexes that work in a similar way. Or you could rebuilt the FS from the ground up, or you could stick the metadata in an SQL database…the implementation shouldn’t really matter to the user. If I can navigate my directories and search my metadata, I’m happy.
“You cannot reliably create a system like I am describing if you let users muck with the primary keys…WHERE things should be stored should be left up to the system.”
Primary keys can be surrogate keys, it isn’t anything that’s insurmountable and in fact we can’t change the inodes directly in linux. Again, I’m confident we can solve all technical problems.
Inodes don’t completely solve this problem (as I understand them to work). Yes, you can use one to get to a file – but they do not have back references. Getting the inode of a file does not give you a way to get a list of its “siblings”. For that you need a path so you can identify the directory it is in, and as soon as your users start moving files around you are back to the same problem… I think btrfs fixes the no backreference thing, but it is still pretty new.
I doesn’t really change my point though. As long as the hierarchy of the file system (as viewed by the OS and its applications) is user mutable, it creates an obstacle. There needs to be an immutable hierarchy for the system to work with. Once you have that you can present things to the user however you want to.
“I doesn’t really change my point though. As long as the hierarchy of the file system (as viewed by the OS and its applications) is user mutable, it creates an obstacle. There needs to be an immutable hierarchy for the system to work with. Once you have that you can present things to the user however you want to.”
It’s very good to be moving on to technical subject matters.
I really don’t see a technical reason a hierarchy must be immutable, that’s not an existing restriction on any FS I’m aware of. Ext3 fs can move files with no problem whatsoever. (btw it shouldn’t be a forgone conclusion that ext3 is the best approach)
You brought up the issue of retrieving file names from inodes. I don’t believe linux or ext3 offer such functionality for uncached files. However if necessary it would be very easy to add a field on the file inode pointing back to inodes of directories which contain it.
Whenever one moves a file, the directory entry would obviously have to be changed as before, and now the file inode would need to point to the new directory inode(s). There are certainly other approaches we could try too, but this seems to be the path of least resistance for ext3.
Then comes the new user tagging information. This could either be a layer on top of ext3 (like a separate relational database), or the new metadata/indexes could be incorporated into the file system itself. Using a separate layer might help keep the ext3 FS backwards compatible, but either way should be feasible in implementing the APIs.
Is there a specific problem that you see?
Not immutable… user immutable.
What I mean by that is that ideally a user should be able to tag, categorize, colorize, hierarchically organize, move within their hierarchy, copy, etc. – basically do your everyday run of the mill file operations that make them happy if they so choose – but none of these operations should ever change the actual location of the file in the systems storage hierarchy. Its storage location isn’t completely immutable, but it is only mutable by the OS – not even applications can move files. It becomes a privileged operation.
Lets pick a complex operation. You have an HTML file that you have manually transformed into XML. You now want to tell the system that its type has changed. You would normally do that by just changing the file extension (or its type metadata) and you are done.
I envision that something like this would involve the following (conceptually speaking)
1. The system is made aware of the change – the OS is always notified on such changes (for metadata it it tracks internally) since it has to be asked to make the change anyway.
2. Since it is no longer an HTML document the file is disassociated with programs that only handle HTML. This is done by removing it from the systems HTML storage directory and moving it to the XML storage directory. Essentially the file is moved from one place to another. File extensions to signify type are completely optional, type is denoted by location – not extension. The actual names of the files, however, are immutable and system assigned (like an inode). User visible filenames become metadata.
All applications registered for XML files will now see the file, because they _always_ work with files from the storage location the OS reports for applications that work with XML file. An applications view into storage is based on what types of files it handles – it sees all files of those types autmatically and can present them to the user when needed. The OS doesn’t have to keep track of what applications are registered for what files, because it is defining where the files are stored by type. All an application has to do to say it handles XML files is ask the OS for a view in the XML storage space.
Anyway, if the _user_ has metadata in this scenerio (such as a storage “folder”) tied to the file, that might have to be updated so that it points to the new file (depending on technical details of the file system). But a type change is an uncommon operation.
Common operations, like the user moving the file from one “folder” to another don’t required _any_ system level change – because that is user level metadata that the system doesn’t care about and changing it does not actually change the files location. Its invisible to applications. Most such metadata is user level, the system primarily only concerns itself with type and its associated metadata. File location becomes irrelevant to applications – they just ask the OS for views.
I understand and appreciate the argument for using a surrogate key for this. But I personally like the idea of it being human readable for system admins and developers, i.e. the OS defines how and where files are stored – and that is where they will be. I see the ideal solution is that directories, which are never exposed directly to users, carry human readable conventional names as they always did. But file access would be based on inode (or some other surrogate key), and the root view of the file system (for admins, developers, etc.) would only concern itself with this key. Filenames would still exist and still be very useful (for both admins and users), but at the application API level everything is handled by the files key, not its name. And the key would never change.
No questions asked, no deviations, no exceptions. All user generated XML files will always be stored in /Users/Whoever/XML. In other words when you log in as root you see that _actual_ storage system, but when you log in as a normal user you simply see /Users/Whoever/[whatever-primary-oranizational-view-you-prefer].
You are, conceptually at least, giving control of the existing directory structure over the the OS – all you have to worry about is creating a metadata layer so that users can have their own preferred views into that structure. You don’t actually need a new file system to do this – pretty much any existing file system will do. You just need to add some kind of metadata database to manage all the new metadata.
What does this buy you:
1. Want to find all the mp3 files a user has. No indirect lookups required – ask the OS for a view on mp3 files and you get them all. Nothing but a directory lookup. Ditto any other type or combination of types.
2. Most user level operations don’t require any type of system level change. Things stay where they are put unless they need to be moved for system level reasons.
3. Applications can easily create customized and optimized views of files. They can leverage custom user level metadata to do lots of neat things to make peoples lives easier. They can get creative again.
4. Unlike (imo) failed attempts at this type of concept, applications don’t own files at all, but they can own metadata. Files can be shared amongst applications without them conflicting with each other.
As far as how the metadata goes, that is what I was thinking more or less.
galvanash,
“basically do your everyday run of the mill file operations that make them happy if they so choose – but none of these operations should ever change the actual location of the file in the systems storage hierarchy.”
“actual location” as a separate concept from “user location” doesn’t make sense to me. I see no reason the “actual location” shouldn’t be the “user location”. I’m definitely tripping over your terminology, but I’ll try to parse what you mean…
“Since it is no longer an HTML document the file is disassociated with programs that only handle HTML. This is done by removing it from the systems HTML storage directory and moving it to the XML storage directory.”
“All user generated XML files will always be stored in /Users/Whoever/XML.”
Where did the concept of system’s HTML & XML storage directory come from? What purpose does that have? If you want to track files by their type, why not simply create an index on type instead of changing their “actual location”?
“Common operations, like the user moving the file from one ‘folder’ to another don’t required _any_ system level change – because that is user level metadata that the system doesn’t care about and changing it does not actually change the files location.”
If your storing path/filename information in a database instead of in the underlying ext3 file system, that’s true. However you still need some kind of index, and that duplicates the functionality of the file system. You might as well map the database to inodes directly and completely bypass ext3 filenames if they don’t serve any purpose other than being a transient identifier.
“I understand and appreciate the argument for using a surrogate key for this. But I personally like the idea of it being human readable for system admins and developers, i.e. the OS defines how and where files are stored – and that is where they will be. I see the ideal solution is that directories, which are never exposed directly to users, carry human readable conventional names as they always did.”
What you are describing is every file type is hard coded to be located in a folder matching it’s file extension, but it perplexes me why that would be more useful for either the file system itself, or the administrator than just using the “user’s location” directly. The intermediate file system representation you’ve chosen isn’t terribly useful, I question why bother having it at all.
“1. Want to find all the mp3 files a user has. No indirect lookups required – ask the OS for a view on mp3 files and you get them all…”
Ok, but what good is this to an administrator who can’t see any of the identifying metadata? The only thing I could ascertain from a raw directory listing is how many files the user has and how big they are. I’d have to open every single file to check what it contains.
“2. Most user level operations don’t require any type of system level change…”
I don’t think “system level changes” were ever a problem to start with.
The indirection will add some overhead when accessing files and running integrity checks. No biggie though.
“3. Applications can easily create customized and optimized views of files. They can leverage custom user level metadata to do lots of neat things to make peoples lives easier. They can get creative again.”
In terms of UI the backend really shouldn’t matter.
I’d prefer stronger FS integration myself, and your intermediate representation strikes me as weird, but of course it could be made to work.
That is the very heart of the matter though… A piece of data has to live somewhere – it has to have an address so to speak. In Linux this it ultimately an inode, but my example is meant to be conceptual, not literal.
I was using existing file system paradigms to describe the goal. Ultimately, all it boils down to is that the system (as in the OS) has a meaningful view into the users storage area – but not simply in the sense that it know all the inodes of their files – it needs to know certain things about those files and their relationships with each other… In my example type information. I was using directory structure as an index, but it can be any type of index.
The real objective is to move “user” metadata out of the realm of the OS – meatadata, such as arbitrary file organizations and relationship mappings that don’t mean anything at all to the OS or its applications. Push it out of the OS and into the hands of the user in a way that allows them (and applications) to create their own views into this information, while maintaining a completely coherent view for the OS to work with.
A simple list of inodes doesn’t cut it. The OS has relationships it has to manage too. Current file systems use directory structure for this – I was just trying not to stray too far from the existing paradigm. Ultimately it is an implementation detail – you simply want to _expose_ it as a directory structure and make it human readable (for admins and developers).
Again, just an example – not meant literally. The point is an index IS a location and vice versa. Indexes resolve to an inode, so do paths. The difference is in current filesystems a file can only have 1 path (excluding tricks like hard links and what not) and users can alter it at will.
When I say “actual location”, I merely mean system maintained – as in the user cannot alter it. Think of it in database terms… The system manages tables, users and apps only work with views.
I was only trying to give an off the cuff example of _why_ you would want to push user metadata out of the system realm. I agree it is completely contrived and not a very useful example. I suspect from the system level you would want lots of indexed views into block storage for different purposes – the real point is keeping them coherent, and to do that users cannot alter them.
You give the impression that you think inodes are fine for the systems purposes, because they are unique, don’t change, and users cannot alter them. That is all true, but inodes are not enough – you also need metadata and you need to establish relationships between files.
Right now the problem is Windows, OSX, Linux, whatever… They all use directory hierarchy as metadata. I.e. config files go in /etc. That is metadata. We keep users from mucking it up by locking them out of it unless they have permissions.
We need the exact same thing for USER data… User data needs system level metadata too. It lets the OS and apps do a whole lot of very useful things they cannot do efficiently now. The reason we don’t have that in current file systems is users can alter file paths, thus destroying the metadata…
That is all I am really getting at. I don’t have a master plan on how to build it right – but I do understand why it is needed.
If you have root access you can just use their metadata… In other words if you have access to the file you would have access to the metadata (since the same user owns both). I’m not trying to keep the OS or admins from seeing users metadata, I’m trying to keep users from altering metadata that the system generates and uses for dealing with their files.
That is where I totally disagree. If I am a developer and I want to make a program that works with some particular document type…
I do not want to make the user have to go find the files I work with for me, I dont’ want to go searching for them… I just want to ask the OS to hand them to me. I (as the application) can present them to the user in a better manner than the OS can – because I know how I am going to be using them. It boils down to being application centric, not document centric.
Establishing conventions for this, i.e. “My Documents”, is not good enough… It needs to be strictly enforced and maintained.
I think at this point you’re getting too wrapped up in implementation ideas, which may be good or bad in themselves but either way are distracting from the big picture. i.e. What are the contracts you want the system to make with the user regarding the safety, security and accessibility of their data? Musings on possible behind-the-scenes implementations is way, way down the list by comparison.
..
Incidentally, representing a single piece of data in multiple file formats is an especially lousy example to use. REST already figured out the correct answer: you continue to present that information as a single resource, publicly announce which representations it is available in along with any hints as to which are optimal and which are lossy, and let the client specify which of those representations it wants. The actual conversion processes – be they manual or automatic, cached or on-the-fly – are then implementation details.
The one useful point that does arise from this sort of example is that a traditional filesystem cannot natively support this sort of presentation, lacking 1. the ability to cache multiple representations at a single location, and 2. any means for the user to negotiate or specify type requirements when accessing data at that location. Whereas a metadata-driven storage system could do stuff like this in its sleep.
I agree actually. I was just trying to make an example in the context of existing file system paradigms. Frankly, I have not through it all the way through.
That was kind of what I was trying to get at. I know what I want, but not how it should work exactly.
In a nutshell I want to OS to manage and maintain coherent views into use data, so that as an application developer I can simply ask for them. I want the OS to expose metadata facilities that I can use to index this data, but in a way that the system and other applications can leverage if they want or need to. And I want to let users (if they choose to) create and manage their own metadata for their own purpose.
How it should work… I don’t know exactly. But I am a firm believer in application centric interfaces and this kind of plumbing really is quite critical if you want to be able to do them right (and still maintain data democracy)
galvanash,
“The problem is not that users use directories to organize files. The problem is that users organize files. It is totally pointless… It serves absolutely no real purpose. It is a distraction. It is stupid. It is nonsensical.”
Right, go tell that to your doctor, accountant, etc. Seriously do you hear yourself? As a developer I have technical reasons that I need directories, but even if that was not the case I would rather not merge all my client’s data in a single hodge-podge directory with metadata only to search for them. I’m advocating a methodology that combines the best of both worlds, you’re advocating a methodology that removes my choice. Why does my having a choice bother you so fundamentally?
“Yet I do it every day. So do you. Why? Because my computer is designed to make me think that I need to. Adding search didn’t make me stop doing it. Computers thrust the concept of ‘file management’. Your only referring to specific implementation issues that stem from evolutionary designs and legacy interfaces…The only way to get rid of it is to expunge it, otherwise it will never truly go away.”
You are still complaining against existing implementation problems as a reason to eliminate the directory concept all together, that’s an illogical jump. I’m saying both tagging and directories should be designed from the ground up as well integrated equals on a new platform. If that’s the case, you could throw all your files into one top level directory to your heart’s content. Giving me the ability to use directories doesn’t harm your ability not to, but removing my ability to use directories does harm my ability to organise my files as I see fit.
In other words, your proposition is biased towards your needs, mine is inclusive of both our needs as far I as can honestly discern.
One man argues that if you’re to get all aspects of an OS and it’s applications to use only an e-mail like organisation of files using only meta-data then you’d need to do away with regular user access to the file system…
otherwise some applications will use the file system as the default means for file access and therefore some files won’t have/need the meta-data and we stay in the situation we’re in now.
Another man argues that he doesn’t want to live in that world, it’d be totally impractical for doing his job. The file system needs to exist for so many functions.
Both are correct and I hope both would agree that in an ideal situation, anyone looking to implement an OS without regular user access to the file system would would implement it well enough to make it worth everyone’s time (by saving it) but still make access to the file system a trivial root/administrator function for those that need it.
BTW, I have no problem finding files. Name, dates, extension, location etc. is more than enough for me but I understand why people like the e-mail concept.
I appreciate the 40k foot view.
By the way, the controversy isn’t over exposing users to the native file system per say (most users shouldn’t have to be aware of the native file system in my opinion). It is over whether user directories should be totally flat, with all of a user’s files in a single directory, or if users should be allowed to place files in separate directories to organise them.
The phrase you want is “make-work”. Like recompiling a kernel to add a new driver; itself an exercise that has only recently fallen out of the category of “positive ways to spend one’s life” (and even so this probably only applies to the mainstream distros).
A simpler way to put your argument might be: “I am not my computer’s secretary”.
The problem is, the typical computer nerd/geek is a pathological hoarder: always happy to absorb shiny new toys, but never, ever throws away what they already own, no matter how clapped out or broken down it might be.
I suspect a lot of folks here, while enthusiastic about computers, just don’t know much history beyond their own direct experience, so don’t appreciate just why the proto-greybeards invented the terms ‘primary storage’ and ‘secondary storage’ in the first place. Disk drives – and, by extension, files and folders and whatever other related abstractions you care to mention – are just a workaround for the historical limitations of primary (currently RAM) storage: i.e. severely restricted capacity and limited persistence. As a workaround, those early ‘users’ invented mechanisms for serialising live program data so it could stored away in a more capacious and durable (but too slow for live use) system, and transferred back to main memory the next time it was needed. I daresay if those old valve-based guys had invented high-speed, high-capacity holographic memory instead, such a split would never have occurred. All data would spend its entire lifetime as live objects within permanently running processes, with any cross-process data exchange conducted via IPC, and we wouldn’t even be having this argument today.
Folks are now so used to the myriad kludges and abstractions layered on over 60-odd years of ad-hoc evolution, they just assume it’s how things ought to be and never think to doubt or challenge those assumptions. Don’t get me wrong: organising data hierarchically is a mighty fine tool for certain types of problems (e.g. it makes a lot of sense for stuff like OS and application files), but for user data it is an absolute tyranny.
Straight off the top of my head: consider backups. Backups are an evil, brain-damaged solution to a very real problem: how to ensure the safety of the user’s data? There was a time, way back when, that the traditional ‘copy to external media/server/whatever’ strategy was unavoidable: the technology of the time was too limited to permit anything better. Yet the problem still exists today, at least a decade after that is no longer a hard limitation. Recast the problem in modern terms that better reflect users’ actual needs and expectations – redundancy, replication, historical timeline, easy access from more than one hardware device – and you start to see what the real problem is: data being bound to a single point in space and time. And the only way to break that bind is by decoupling the user from the physical storage process and location.
Suddenly, all sorts of ghastly convoluted crap collapses into a far simpler and elegant conceptual model. Not that there aren’t all sorts of practical problems still to be solved, but at least now you know where you want to be in the near future, not merely fighting to stay where you were 20 years ago.
(Incidentally, once user data storage and presentation is fully decoupled, you can provide any view onto that data you like, even old-skool hierarchical if that’s what tickles your fancy. Ah, the joys of good abstraction. And anyone here who isn’t aware that a file system is itself a thick abstraction over a bunch of other thick abstractions – half of which don’t even have anything to do with actual files if you’re on the likes of Unix or Plan9 – needs a healthy clout from the LART stick; and I do mean that.)
Good luck with effecting any sort of change though. There are all too many folks who’ll fight to the death to maintain a nasty, ugly, crippled status-quo where they are the undisputed kings than sweep it away to stand shoulder-to-shoulder all the little folk in a brave new world. OSS/*nix folks should be taking the lead in shaping the future to everyone’s best interests. Instead they bicker and whine and refuse to let go of what they already know and step into the unknown, allowing the big self-serving vendors like Apple and MS all the time they need to shape it primarily to suit themselves.
It’s a bit frustrating.
Ah… Someone who both appreciates where I’m coming from AND knows what a LART stick is. Kindred spirit
hhas,
Thank you for that insightful post. I agree with your ideas. I’ve given the same thoughts to the duality of documents on storage and documents on display. That duality isn’t inherent, it was evolutionary just like you said. I think there are some platforms that tried to treat all documents as active documents and eliminate the notion of opening & saving them. It’s unfortunately such projects have been relegated to obscure niche usage since I think it’s a nice alternative.
“Don’t get me wrong: organising data hierarchically is a mighty fine tool for certain types of problems (e.g. it makes a lot of sense for stuff like OS and application files), but for user data it is an absolute tyranny.”
I still have to disagree with you here though, even once we’ve expelled the disk-saved/memory-loaded duality, which was invented for the computer’s sake so to speak, it hasn’t eliminated clutter nor the user’s desire to keep certain documents organised in relative hierarchies, metadata only takes you so far and works much better in some cases than others.
People used hierarchies even before computers came on the scene because it was a natural approach for them to organise data – it’s not like they were invented for the computer’s own sake. They exist in all kinds of “human” contexts; my favourite computer part supplier, newegg, organises their website products in a hierarchy. That wasn’t for the benefit of their web server, or their developers. No, it was for my benefit as a user. In addition to hierarchies, they also provide great metadata searching too, proving that both approaches are complementary. We should my own computer not permit me to work this way if I want to?
“Good luck with effecting any sort of change though. There are all too many folks who’ll fight to the death to maintain a nasty, ugly, crippled status-quo where they are the undisputed kings than sweep it away to stand shoulder-to-shoulder all the little folk in a brave new world. OSS/*nix folks should be taking the lead in shaping the future to everyone’s best interests.”
I certainly hope you are not referring to me, because I agree with you. The status-quo really does impede progress sometimes. I don’t feel user directories do that since they’re entirely optional and don’t interfere with metadata.
I see no reason why parent-child and sibling relationships couldn’t be expressed via metadata as well. Point is, users should neither be constrained by it nor forced to maintain it themselves. Heck, from a user’s POV a file path is just degenerate metadata, e.g. /usr/local/bin/foo tells me that ‘foo’ is an executable tool that I installed myself. By its own nature a file path also implies that certain hierarchical relationships exist, but whether these are unique, optimal, helpful, necessary or even appropriate isn’t something that can be answered without a lot more expert insight. Insight that the system should automatically provide as much as possible so the user (who has better things to do with their life) doesn’t have to work at it themselves.
Once properly decoupled, users and applications can express and view whatever shaped graph(s) they like; they’re all equal in the display system’s eye. And it can be left to self-organise by default, according to whatever criteria the storage system deems most appropriate to a given data type; any additional organisational advice that a user might wish to contribute are merely icing on top.
It works better once you start thinking about how to decompose application functionality as well. Consider an email client, which consists of a functional core that knows how to talk POP/IMAP/SMTP, a big old data silo (which may be local, remote, or spread over both), and a bit of user interface for interacting with the both of them. Recast this though, and the core component stays the same, while the data store and UI become parts of an expert system for providing one ‘intelligent’ view onto a specific type of user data – essentially just one more plugin extension to the general-purpose data storage system you already have. You eliminate the closed-off data silo, along with all the dog-work needed to implement its foundation, benefiting both users – who have much freer access to their data – and developers – who can leverage far more shared architecture, leaving them much more time to invent other cool toys, go down the pub, etc.
Don’t worry; they shall be known by their actions, not by mere words. Folks who really want to effect change will put their backs into it. Folks who stand to lose power and status and are unwilling to accept that will battle against it. While the majority will just bring popcorn, of course.
A one-armed straitjacket is still a straitjacket; you’ve just gotten so used to living with it that it’s never occurred you could take it off at any time.
Treating one particular idiom/organisation/representation as a special case that is hard-wired into the storage system requires a duplication of effort, and will colour and constrain everything else that the system might want to do. Whereas if you treat your much loved tree structures as soft or ‘virtual’ relationships, you can express them just as easily within your general metadata-driven framework without imposing any of those disadvantages.
Meanwhile, since the system now has unimpeded control over how and where everything is physically stored, it can make whatever rule-driven arrangements it deems best: e.g. an application (executable plus various resources) may be written out locally as an old-school file system hierarchy for efficient performance; your favourite Rush tracks are replicated across your local devices for easy access; the hilarious cat pictures you just took yourself are replicated to your online personal cloud space for safe-keeping and perhaps from there onto your favourite public sharing sites as well; etc. While your ‘smart viewers’ can display that info from anywhere, to anywhere, in whatever way(s) you like.
hhas,
You’ve got some great ideas.
“I see no reason why parent-child and sibling relationships couldn’t be expressed via metadata as well. Point is, users should neither be constrained by it nor forced to maintain it themselves.”
I didn’t mean to imply that a directory structure cannot be considered a kind of metadata. I’d point out that even today, even though we think of directories as being special, it really IS metadata for the file (which is represented as inode inside the linux kernel, and not by the filename metadata), it just happens to be the only indexed metadata available to us in *nix. That’s what’s restrictive. We don’t need to get rid of the directory metadata, we just need to add more indexed metadata types. “Directory” metadata would be optional, like other metadata types. It should not be treated specially. As long as it’s available, that’s what’s important.
“Once properly decoupled, users and applications can express and view whatever shaped graph(s) they like”
I’m not sure if using a full graph instead of a tree would be more confusing for users. It might break alot of recursive directory operations that are possible now. For example, deleting a directory and it’s children could be extremely dangerous. However I’m not opposed to the idea and I’d like to see it implemented.
“It works better once you start thinking about how to decompose application functionality as well”
I’ve often wished we had a way to treat all data the same regardless of what it was. Emails shouldn’t be managed under a separate room or filing system than any of my other documents. If I want to create a directory (you probably despise that terminology, can we substitute “container”?) to hold a specific set of emails and other documents, I should be able to – nothing is special about the email. I might even like to do the same thing on a more fine grained approach, using the filing system for email addresses & contacts.
All applications on such a platform should use the standard data types so contacts don’t have to be “synchronized”, they’d simply be shared.
I think I’m loving it
“A one-armed straitjacket is still a straitjacket; you’ve just gotten so used to living with it that it’s never occurred you could take it off at any time.”
It’s not a straitjacket though because it’s optional. Having a hammer doesn’t mean I always have to use it. Just because a hammer can’t do everything doesn’t mean it’s not worth having.
“Treating one particular idiom/organisation/representation as a special case that is hard-wired into the storage system requires a duplication of effort, and will colour and constrain everything else that the system might want to do.”
I never really wanted it to be special, I wanted the concept of hierarchies to be integrated along with other forms of metadata. There’s no reason the computer should not be able to arrange things in hierarchies when that’s what the user explicitly wants to do, which is what I’ve been trying to point out since my first post.
“Meanwhile, since the system now has unimpeded control over how and where everything is physically stored, it can make whatever rule-driven arrangements it deems best:”
I agree with the idea, however the “problem” was never with directories in the first place, it is the implementation of them. AFS is a great example of a file system that transparently handles local/remote storage without regards to a file’s path. I agree very strongly that a filing system’s organization should not drive (or be driven by) it’s storage requirements.
FWIW, I do have a rather eclectic background (amongst other things, I studied art, not CS). One of its benefits is being able to bring a refreshingly unique perspective to the same old set of problems. OTOH, one of its drawbacks is not owning nearly enough brains to put any proposed solutions into practice myself.
On this occasion, I’ll let you in on my secret: I’ve had quite a bit of experience at wrapping my head around multiple mixed paradigms simultaneously. For example, the idea that you can present a ‘virtual’, idealised tree (or other shaped) representation of data which is bound together by relationships, not containment, is something I’ve already seen extensively done by the Apple Event Object Model (the elegant but widely misunderstood RPC+query-driven foundation of ‘AppleScriptable’ applications). And the notion that you can decouple the roles of data identification, data representation, and the physical data storage behind it is taken directly from my time designing and implementing RESTful HTTP interfaces for a distributed application (REST being another very elegant and also widely misunderstood conceptual model for communicating between data management systems at a very high level).
So mostly all I’m really doing here is taking a bunch of ideas I’ve previously seen conceived and used elsewhere by folks far smarter than me, and synthesising a nice solution to the current predicament out of them. Like I say, it’s good to know something of history. (And, hey, on the off-chance somebody here makes their future billions after being inspired by my waffle here, I hope they remember theirs too.;)
So yes, I quite agree they are great ideas… I just can’t take personal credit for them is all.
For anyone still reading – and with profuse apologies in advance for mad length – I’ve been giving the topic a bit more thought. Being an IPC weenie, I will tend to cast any given solution in terms of “I know, let’s use some IPC.” (And now we have two problems.) So I’m coming at everything from a “it’s a communication problem” perspective, just so you’re aware.
First, let’s reiterate the root requirements: how to ensure a user’s data is safe, secure and quickly and easily accessible – and all of this in a highly heterogenous, networked environment – without them having to do any of the tedious clerical work that involves?
If we phrase this in terms of a contract between computers and users, then straight off the top of my head:
1. This has to work across:
– individual user devices (not just general-purpose PCs but also phones, tablets, AV systems, games consoles, and anything else you can think of)
– home and business intranet-based NAS and server boxes
– big internet-based commercial cloud storage services.
2. It’ll require behaviours such as:
– automatic replication and redundancy (including all the challenges that come with synchronising user changes across all data stores)
– full revision history
– security and trust (not only must accidental data loss never occur, but all data must be heavily protected against malicious access)
– transparency and automation-based ease of use (since it must cater to users from all walks of life).
Not a comprehensive list by any means, but enough to illustrate. It’s no small request either, so any attempt at addressing it is going to have to work extremely hard at keeping a lid on complexity.
Sounds daunting, I know, but this is actually something that the early Unix world had a fantastic knack for: take a large, complicated, scary problem and solve it by the highly creative, insightful and relentlessly parsimonious application of what few crazily limited resources were available to the task. It’s only later, as growing wealth and maturity generate comfort and complacency, that such careful habits become non-essential, and the old talent and tricks for efficient, effective problem solving are forgotten or lost in the middle-age spread.
For example, think about what the hierarchical Unix file system [plus its expanded and refined Plan9 descendent] represents. From a default perspective (such as the one Thom is seeing), it’s just a simple 1:1 representation of the contents of one or more disk-type storage devices wired to the local system.
However, those old Unix guys were terrific at spotting opportunities for reusing concepts and behaviours, so it also works as a collection of endpoints onto various services – device drivers, network-mounted drives, etc. – thanks to stuff like Unix domain sockets, which are in turn only a tail’s shake away from Internet sockets. Imagine the increased load and complexity on early Unix had they simply ploughed in with a wealth of tools and manpower at their disposal, creating a completely new interaction model as each new requirement presented itself. (They still missed some opportunities – hence Plan9 – but for a first crack at the problem it was pretty damn good.)
This parsimonious, holistic approach to all-over system design seems to have been all too often squandered or forgotten over the years (e.g. the failure of Linux DEs to follow Unix Philosophy [1]), but this is surely the right time to get back to it. Like it or not, the entire concept of what a computing system is and what it should be is rapidly and radically changing.
Just as the original primary (RAM) vs secondary (disk) storage kludge has created all sorts of awkward divisions and clumsiness, so too has local vs remote storage. We’ve one protocol for accessing local data (the hierarchical file system) and a whole plethora of them for accessing remote data (RESTful HTTP being just one example). These are divides drawn along technological lines – originally out of necessity, but now through inertia, laziness and lack of vision. Why people use technology has got lost due to microscopic focus on how they currently do it, with no regard to whether it’s still optimal or not.
Such well-worn ruts may have become familiar and comfortable – and even a position of power for those most practised in negotiating them, – but they are going to become a liability that even the most conservative nerds/geeks will not be able to afford to ignore.
What’s needed is to step all the way back and try slicing the entire problem along new and different (even novel) lines, i.e. according to [ideal] usage. And then redefine the technology so that in future it fits the way that users should interact with their data, rather than forcing users to adapt themselves to the current technology with all its legacy baggage and myriad complexities and faults.
Change is already underway, of course, but even from my largely uneducated viewpoint it looks completely piecemeal with no clear overarching strategy. For example, Mountain Lion’s Core Data (data storage) framework can now use iCloud as its backing store, but this is at the application level, and just one particular framework on one particular OS using one particular cloud. In fact, ML now has no less than three different ways to interact with iCloud.
Now, I am all for ‘let a thousand flowers bloom’ as far as research projects go, but when it comes to production systems to be used by millions, a coherent overarching strategy is absolutely essential if complexity is to be managed at all. Such as: pushing the functionality as far down in the system as it’ll possibly go (i.e. right down in the bowels of the OS, alongside the file and network subsystems), defining [open] standards and protocols to be used all the way throughout, and ruthlessly consolidating and eliminating duplication of effort and needless redundancy (c.f. Unix’s ‘everything is now a file’ idea that instantly provided all clients with a powerful, clearly defined IPC mechanism essentially for free).
Obviously, for OSes and applications, the traditional device-centric patterns work well as a whole, providing a more than adequate benefit-vs-drawback ratio, so they will no doubt continue to rely on the existing hierarchical file system to manage their own resources.
OTOH, what’s needed for user data is a user data-centric, not device-centric, approach, and that means decoupling user data management from the nitty-gritty implementation-dictated details of file systems, databases, etc. and trying as much as possible to create a single, standard interaction model for accessing user data regardless of how and where it is stored.
The more you think like this, the more you realise just how far beyond the local file system this goes. For example, what is LDAP if not a half-assed reinvention of the ‘Unix file system as namespace’ principle? And what are the ‘everything is a file’ and REST interaction philosophies, if not two sides of the same damn coin? All this and more is just crying out for massive consolidation.
So the HFS will still be required; it just won’t be something users interact with directly any more. Instead of accessing file and network subsystems directly, userland processes will talk to a single standard ‘data management’ subsystem whenever they need to read or write user data. Once that decoupling is completed, the system is free to deliver on all of the wonderful promises made in the contract above. Plus of the file system-imposed problems currently bedevilling users (backup hell, iOS data exchange, etc.) simply cease to exist!
Ultimately then, it’s all a matter of good communication. Admittedly, before the mechanical machine-to-machine challenges are addressed some additional effort may still be needed the geek-to-geek front. But with luck Thom &co will become believers yet…;)
[1] http://www.faqs.org/docs/artu/ch01s06.html
[2] http://www.osnews.com/thread?528519
hhas,
Still good ideas, but I think you should take a closer look at NT’s DFS and open source AFS to consider the work already done to separate virtual organization from physical organization. The separation has largely been achieved in these examples, what they are lacking is the integration of indexed metadata.
Since we’re throwing ideas around:
We might also want to consider how a “global file system” would work, it’d obviously make use of local caching and maybe sophisticated conflict resolution (like in source control). However it would put an end to the need to “email” files, you’d only have to give the other user a secure link into the global file system and they could access it and possibly even work on it with you (avoiding the all too common usecase of emailing back and forth).
The TL;DR version…
Imagine you have two pneumatic tubes sitting next to each other. One leads to local (file system) storage, the other to remote (network) storage. Data is stored by inserting it into one or other of these tubes.
Today’s systems require that a client decide which of the two tubes they should drop their data down. Furthermore, these tubes are slightly different shapes, so the client has to do some extra wrangling to fit their data into a particular tube.
This arrangement is acceptable enough when the computer is the client. Machines are great at performing tedious, fiddly, repetitive tasks according to a predefined set of rules over and over again without error or omission – it’s what they were designed to do.
However, the same approach really sucks for human clients. People already have far better things to do with their time than micromanage thousands of resources across multiple locations with perfect diligence and accuracy, never mind doing it all via such a primitive, labor-intensive interface.
Yet the only reason humans are having to work with the same set of tubes as the computer is that back when all this technology was being invented, this was the absolute best that could be achieved with the seriously limited resources of the time. Nowadays, we have a wealth of resources spilling out of our ears, but we’re still doing things the same old way because nobody’s yet bothered to build something better.
What users should instead be presented with is a single, standard tube into which they throw all of their data without any additional fiddling or fuss. That tube then leads into a ‘user data management’ subsystem, and it’s up to that automated system to determine which of the two original tubes is appropriate, and perform any additional fiddling required to store and retrieve the data using that. (And change-track, replicate, synchronize, secure, categorise, translate, share, etc, too.)
For this to work though, you must totally decouple the user from the physical means and methods of data storage. Because the very concept of ‘physical’ containment – which is what the hierarchical file system is built on – becomes absolutely meaningless in an environment where data may exist in any location [1] at any time, and frequently in several places at once!
—
[1] Some kinds of data might not even end up in a file system, but in [e.g.] a relational/non-relational database instead. All sorts of fascinating new possibilities could arise once the ‘single user data pipe’ abstraction is in place; computer scientists should be falling over themselves to explore it.
The STL;SDR version:
The goal is to replace the current imperative model of [user] data storage and management with a declarative one.
It’s dependent on network access, which make it an online storage.
Aka exact same pros and cons than cloud storage.
What will happened when user don’t have access to network? Exactly like they did when their company network goes down: launch a locally stored game or simply take a (long) pause.
A progress, in fact. I’m convinced! 🙂
It syncs to local copies, just like dropbox. Yes, the master store is on the internet, but you can work offline quite easily and re sync when you have access again.
Now it’s synced with local storage.
But in the future, with always more throughput?
The local / remote sync will becomes a cache or offline feature, up to the point that without access, no game.
It’s already happened with… video games!
For the same exact end objective.
Which is making money by charging you to access your own data.
Because what you describe is perfectly achievable on a local storage area, and nothing, litterally nothing forbid the same to be implemented locally. Just add a mail server and use the always working 127.0.0.1 network address, and you will have the same exact feature you describe. Or install a cheap NAS device with email server on it. Or install a cheap NAS device with OwnCloud on it.
But that’s not the aim.
The aim is to push people to store data outside their own network, in order to be able to charge them for both storage *and* access cost to their own data.
The aim is that people don’t own anymore their computing environment, but rent it. The sign is written everywhere, that’s the aim, the new business plan.
Every cloud storage providers are after that new market, and the ones in a position to push a little bit harder users to do it earlier than latter are using their position to eventually degrade the current local storage situation. Yes, I’m pointing Apple and Microsoft’s operating systems current trend here.
I see. The solution to making things easier for non-tech types is making them harder and more expensive… Do you think your average american can manage what you just described without help from someone else? Do you think that even if they could they would want to? There are thousands of people who would trip over themselves to throw money at Apple or Microsoft to make managing and accessing their data easier, your ability to accomplish the same thing with duck-tape and bubble gum doesn’t mean anything to those people and never will.
Besides, do you use email? Unless you own an ISP or run your own mail server you are paying someone to access your own data. And if you use free email you are STILL paying someone, just with your eyeballs instead of your wallet. Welcome to capitalism!
You don’t get my point. If the actual purpose is to make things easier for average user, it’s perfectly achievable with local storage alone.
My point is that the move is not about making it easier but about lying that it can *only* be easier *their* way, via remote service you can’t hope to have any control on ever, not talking about being charging.
I’m all for moving toward a more smart data storage and retrieval paradigm, I just fail to see why doing it on a remote machine will magically make it easier but the same on the user’s machine can’t.
For today mobile devices, which comes with far limited storage capacity than personal computers, okay, but today the laters have both large enough power and capacity to do it well all locally.
Plus why any new feature should first remove another feature that prove usefull to power-users (not just tech guys are power users these days, mind you, it’s not 90’s anymore…)? Can’t progress be to *add* a new feature, not replace one with a new one that I’m not the only one to think it’s a regression (you do undertsand that the “today” hierarchical storage is perfectly able to handle the one folder only case, right?
That’s was the point(s).
But you miss it, probably too busy to make me look like some IT conservative and in the same time claiming that the average american is not able to understand a hierarchy and that there is no money motivation behind these post-file push.
PS : yep, I know, I use a free email service. But, mind you, my *local* emailer fetch any new message and store it *locally*. If people ever discover that this well-known free email service is abusing his position, I wont lost my emails history.
Not sure that people who store theirs files online can all say the same already.
Anyway. I’m not against cloud storage, or dumb-proof, I’m just against enforcing on every user. And will always take action to avoid it.
“Right, so this paragraph wasn’t supposed to be here, but this just hit me the moment I wanted to press publish: making files harder to move around? Making it harder to find them? Making it harder to take them with you?”
First of all, as people already pointed out, this one lever folder is an iCloud thing, nothing has changed in OS X. You can simply move stuff in and out of iCloud to your local Mac, Dropbox or anywhere else.
So with regards to iCloud:
Harder to move around: You can move them to where you want to.
Harder to find them: Why? If you can only go one directory deep you know pretty well where a file is. Spotlight can find it for you and apps only show their own files, so it’s kind of hard not to find your file.
Harder to take them with you: Not really, your files are in the cloud so you can access them everywhere you are connected.
The problem with directories and files in general is that people do not take the time to organize them. It doesn’t matter if they are flat or heirarchical.
Helper apps like iTunes and Lightroom can make organizing and working with specific file types easier, but the more effort you put into ensuring the metadata is correct, the better they work for you. In Lightroom, I have nested keywords Places>California>Los Angeles>City Hall or People>Family>Name Name. If you dump all of the keywords under 1 level, you’ll be scrolling forever. Same with files.
I prefer metadata searching, but I also made my own extensible “universal directory structure” which meets all of my needs. I actually spent time researching online to see how others did it and then categorizing all of my documents and other user created files. Once I found patterns, the directory structure became apparent. If your machine is a real work horse, then you need to interact with your directory structure frequently.
As to the comment that mail is “gloriously flat”, I use folders to organize my email, too. There is no way someone who works out of their email can get away from that, using default metadata.
The big difference is between those who work on a PC and those who have a web/ mail device.
Welcome back to the early 1980s. Over-simplifying to the point of the exact opposite of progress… Apple. Introducing the all-new shiny flat file system. Think Different.
Or should I say, Think Antiquated.
It seems that Apple just wants to make their users even dumber (as if they haven’t made them braindead fucking retarded enough by now anyway) to further restrict what they can do on their systems, locking them down even further. Apple in 2012… who else would to go to such extremes? The GNOME guys are still trying to play catch up, because so far no one has mastered the art of dumbing shit down as well as Apple. They always reach new lows in regression, both technically and by design, that I would never imagine were possible. And for that, I sure as hell won’t commend them.
Personnally, I don’t understand the love of hierarchical structure… When I am looking for a picture, I am looking for a topic, a place, or a person. What we need is a filesystem-as-a-database, not a stiff, 1-dimensional hierarchy. When organizing stuff, I always stopped to wonder “should I organize it this way or that way? Geographical subfolders inside temporal folders or the opposite? “Album” subfolders inside “Artist” folders? Yeah but what about compilations? And where do I put genres?”
Directory structures are shitty. Sorry.
Edited 2012-07-26 07:12 UTC
it is time for new things
and yes, folders do confuse ALL new computer users (special older people) because they are to abstract to them.
when they get use to them, than it is ok but there must be some more logical way to deal with files/objects.
Filesystem with files and folders _concept_ was broken as soon as we we start to use email: suddenly you have _files in email_ ?!?!!! WTF? “I know than I have file on my C: disk but where are stored files in emails??” – this was start of breaking concept of folders and files…
it is time for new things
btw access to bare filesystem will be always mandatory for content creators: programers, web developers, musicians, photographers, CAD users…
Show me the science. Parroting the party line is fun and all, but I want the science. Without it, this is all hot air.
I second that – as soon as I explain that folders are like a filing cabinets (with folders inside folders being like folders in a filing cabinet) then they realise that they can organise the files easily. It is amazing how I too hear the ‘anti-directory’ brigade and ignore the fact that people have been doing the same thing in the real world with filing cabinets for years but when it comes to the computer then apparently it is ‘all too hard and complex’. Oh and for a fair chunk of them they lack organisational skills but in the real world rather than changing the dynamics of the whole organisation to cater for their disorganisation the manager will tell them to get their act together. Apparently when people come to computers all the expectations of organisation go out the window because a computer is some sort of magical device that magically does stuff for people whilst ignoring the fact that a computer is little more than a glorified calculator that only does what it is told – garbage in garbage out.
Edited 2012-07-26 08:09 UTC
Thom, I’m totally on your side on this issue, but still he does have a point.
I’ve spent quite some time instructing professional computer users. Many of them do seem to have a problem with hierarchical structures (or just don’t see the benefit).
Just not sure if the problem is that hierarchical structures are too abstract.
Since in my experience most of those who seem to have problems are either younger (say under 30) or older (say over 50), I feel the real problem is with poor or lacking computer education in schools.
Perhaps the problem is (just thinking here) that it simply takes energy to work with and set up hierarchies in a meaningful way. And ‘modern’ or new computer users have grown up with this fatalistic image that anything computer ‘just works’ (or just doesn’t work) and they just don’t see the benefit of spending any energy on that.
I mean, this really seems to be something specific to computer users. I’ve never met a biologist that didn’t get taxonomies or any office worker that couldn’t exactly describe his/her position in the company hierarchy…
but I think a valid point might be that *even if*
the older people among new computer users find
folders or hierarchical directories ‘hard to abstract to’
…the current ‘young generation’ of new and not so new computer user will make up the huge majority of older computer users in the future!
and they’ll have grown up with this ever changing computing landscape, and be naturally more adaptable to different computing paradigms (in various directions), hopefully including variations on our current favourite file systems and folder/directory trees..
mistersoft,
“the current ‘young generation’ of new and not so new computer user will make up the huge majority of older computer users in the future! and they’ll have grown up with this ever changing computing landscape, and be naturally more adaptable to different computing paradigms (in various directions)”
Agreed. There’s one aspect troubles me though, the move to lock down computing devices (assuming the trend continues unabated), could spawn a widening gap between consumer and developer-capable devices. It is especially worrying to me that many kids will only be exposed to vendor-locked devices with crippled kernel and userspace programmability. This is a stark contrast to our own exposure to computers growing up.
If we don’t fight against closed computing today, we could very well end up in a future where consumers are all running restricted devices, and in order to reach any significant portion of them developers have no choice but to participate in walled gardens controlled by oligopolists. Hopefully everyone agrees that future should be avoided.
Edited 2012-07-26 15:23 UTC
My mother has been using a computer for over 15 years and still doesn’t get the concepts of folders/directories. She continues to create new ones with an increasing number of underscores to get them to the front of her exploding “My Documents” folder.
I’ve seen this happen before to others. I think phasing out the file system is a GREAT idea. The problem is that we need a next-generation solution for power users too, because, yeah, emailing documents is a mess.
A worthy call, Thom. But let’s recall that your own article starts with “I have honestly never seen a single person have any issues with directories, nested or no, and as old as the concept might be, the people I interact with seem to be able to handle it just fine.”
You’ve been to university, you know that the plural of “anecdote” is not “data”, so for you to suddenly demand “science” after basing your entire argument on your own perceptions is really way out of line.
regarding facts: I work as instructor with dozen of people in few companies in last 8 years. I know what I am talking – there should be better way than this.
Obviously it is a problem since Apple try to solve it with “All My Files” option in Finder.
and I am not going to fight with bunch of IT experts if hierarchical file system is best way. I already said:
IT professionals (content creators) will continue to use it! Other users, office and casual, have no need for it! And whole idea with files start to fell apart as soon as you have file in Mail client and not in file system (hidden from user – you should start complain back in 1993.
)!
beside, IT professionals should thinking in this direction http://www.youtube.com/watch?v=zumdnI4EG14 , and not “if hierarchical file system is best”…
Edited 2012-07-27 14:26 UTC
It started out with no “directory structure” to organize your documents in, only tags allowed.
It kept evolving until almost all features of “filesystem-based” folders were back in.
Reason is simple: some people like searching for stuff, some people like browsing for it.
I think that having folder-based browsing adds extra value as well, even for people who like searching more.
Take a shared-file-dump scenario:
– you can learn which “types of content” exist without having to search (drill-down through folders === get a list of tags)
– you learn how to tag your content (by deciding where to save it) instead of creating 25 variations of the same tag to express the same concept
The main problem for this is that different people would naturally organize content in different ways, and might have a hard time adapting to other people’s views. And some are messy by nature.
I would rather that Apple finally integrated a useful file manager in their OS. It would relieve me from the most painful shortcoming of the Mac.
To me that is a bogus problem, or – at least – it is located somewhere else.
I’m pretty sure that the whole iOS system does in fact use directory structure. Only the user part doesn’t.
It’s all about treating your user as a total f%$##@ moron, really. They think you’re too stupid to navigate through simple directory structure. I think they might be true considering the way that the whole Apple infrastracture works. Closed source, closed app store, closed platforms, closed hardware, etc.
They don’t want you to mess around, tinker and … generally THINK. iThink …
I see this whole misery around file system hierarchies to be very much pointless and generally a waste of time. There are two important things to keep in mind in this topic:
1. Average users have pictures/videos, a few documents, applications and games. For the latter two, they don’t care how and where they are stored and what they contain. For the first two, they need two locations (folders, libraries, whatever you want to call them) an be done with it.
2. More involved users, who also sometimes develop, create content (graphics, videos, applications, etc.) plus advanced users and developers (basically geeks and coders). They have a million different projects, a trillion files with mixed type, content and purpose. They simply can’t live with a library where you dump everything and then search for files when you need them. It just can’t work. E.g. I have projects that have thousands of files associated to them, which don’t have any meaning from another project’s point of view, they simply belong under their parent.
Without a hierarchical organization, people in category 2 would always be in pain and suffering. That can’t be. If you cause pain and suffering to those who make the lives of the people in category 1 enjoyable, then you’ll find out soon enough that you can’t just ignore them.
I’m not against simplifying file system use for average users. By all means, do that. But I’m arguing against the killing of the hierarchical file system paradigm, simply because of far-reaching practical reasons, the most important being: we don’t yet know a better way to do it. We can’t just jump into a very much inferior option just for the sake of “evolution”.
From the linked text:
Well yes, if all you do is write quick notes and blog texts and such, there’s not much more you’d need. But come on, think of the children –
– I mean think of the other people, you know, those who are not you, and do more with their computing devices (I mean the full range here) than yourself.
Back in the 90’s, when computer was just starting to make a real dent into all companies, i was working with a lot of older people for which fiddling with a computer was new.
And one of the most obvious concept that a lot of them couldn’t grasp was … directory structure.
It struck me because i considered it to be so simple. But the result was in front of my eyes : most of them would simply let their files be stored “anywhere” on the computer. That is, where the dialog/window would put it. They would only care about the file name.
The “default folder”, quite often, was at the root of C:. Heck, this would even be the case of some IT people and computer teachers !
Later on, when searching for such file,
if, by any chance, it was not in the presented “default” folder, they could not find it, let alone search for it.
And then history repeated when my mother acquired a computer. Same problem : what are these “directories” ? It took her more than a full year to really understand and use them properly.
So, now, my understanding is :
NO, Directory structure is NOT obvious.
At least, certainly not to non-geekies, non-engineer, well, you know, “normal” people, >90% of planet.
Of course, normal people can still learn, and cope with it. There’s a hell of complexity they can more or less manage in many areas, including computing.
But they really don’t care. They would very much prefer a system which would “store these things somewhere” and “find them when they need it”. The easier, the less “managing” they have to do, the better.
Sadly, us, computer guys, we are often the last people to ask for User Interaction Interface : we tend to reproduce what we already know perfectly, which itself was built around technical limitations of our early days.
Just take someone from “outside” to have a fresh look, and it becomes obvious : so many “features” we take for granted are in fact technical limitations pushed into user territory. We manage them. We’re accustomed to. So it seems “the way it must be”.
But the reasons have vanished…
Edited 2012-07-26 10:37 UTC
Still remember the old good days when computing was typing or loading a program (now called app), when you have to face the “tabula rasa” and succeed, or fail, no matter what, but the sensation of doing something and the feeling of “creating” was nothing similar to what we can do today.
In fact, you were learning how computers work and how to operate with them, and of course it wasn’t for anyone (in price too). With the socialization of the computers a lot of people gained access to them, just for work, and in the internet era with the entertainment and content sharing and consumption things go dumber and dumber. It’s true, my mother can open skype and talk to my children, and read a newspaper, but don’t aske her to do something more, and my 11-yo nephew can upload a silly video to youtube and have fun with friends, and do some schoolwork and stuff like that without knowing anything about the computer.
The price of socialization of the computers as Clive Sinclair predicted (and loved, as he sells a lot of them with this argument).
So my point is that for people that aren’t interested in going further and only consume content or use the computer just for fun, maybe one folder hierarchy it’s enough or even harder to understand ()… but there are a lot of people that born with computers, or begin to use computers with the hierarchy folders system in use, or have a lot of projects, for working, for organizing, for whatever, why to go in the opposite direction? Would be the OSX of the future encapsulated in a one-folder ?? 🙂
have fun.
If you’re not doing serious work, or you’re not continuously under pressure, and don’t have a family, I can see why you don’t consider folders a bad idea.
If you are, however, you’ll put things in the wrong folders, fail to clean up from folders, forget about folders existing, forget whether you put the GUI / database interaction task in the GUI or database folder (because it could be both).
Soon you’ll have an ungodly mess which it would take hours to clean up. Hours you don’t have, because you want to finish five important tasks by the end of the day, so you can go home on time and spend the evening with your children. Rinse and repeat the next day, only this time the mess is bigger.
The only real solution to this is to either give each employee time for menial tasks (seriously?? You think management is going to provide time for file system organising for £30+ per hour staff?), or get computer systems that automate the task by using clever, automated tagging and searching.
I know what I’d chose.
Ah yes, the pinnacle of quality arguments: insinuating that those who don’t agree with you have no life, no job and no family.
And having a single-level tag cloud with hundreds of tags is of course much better and easier.
Sorry but tags and metadata does not magically solve this problem.
Tagging and searching is not mutually exclusive to folders. I really don’t know where this misconception comes from.
Edited 2012-07-26 12:46 UTC
My friend works for a local College in the art & design dept.
When the students first arrive that have no concept of organisation of files. This is because up and till that moment their files do not have to be used by others and there is no collaboration. Each student in High School worked on isolated individual tasks.
The first problem they hit is collecting files together for print. They just have a mass of source, development and final files mashed together. They just don’t understand how to organise files so others can work with them. This is childs play compared to then getting them to do simple HTML websites that require a structure to work.
Then you get a few of them together and you have collision of messy or no existent file systems and often restricted shared storage.
File systems are there not just for a single person’s needs, but for the whole collaborative effort.
Killing collaborative effort seems to just be another arm of Apple’s self mutilation of their devices content creation abilities.
It all seems to be in the name of ‘Usability’, but usability isn’t about never asking the user to learn something. That mentality and combined with the insanity of over simplification eventually leads to less capable software and less capable users.
I don’t mean it is the snobby tech nerd sense, that if you can’t do Y you aren’t a ‘Real’ user. I just mean that like so many things, you become more capable by learning, not by removing every need to learn.
Edited 2012-07-26 11:54 UTC
Wow. In this entire thread, discussing the potential elimination of the most important empowerment computers can provide the individual, the word ‘political’ does not occur even once!
That most important empowerment? The ability to consciously organize, archive, categorize, and OWN information, entirely under the control of the individual. This is an intensely political capability.
Yet no one here sees this push (which it is, and has been going on for many years) to remove hierarchical filesystems from the reach of average users, as politically motivated? You all *seriously* think this is just some accidental and ill-advised conceptual mistake? Some kind of UI fad, that will go away once the CEOs and ‘technology directors’ at Apple and Microsoft come to their senses?
Or apparently, a few of you agree with the author, that the worst case is that the anti-file-system push is possibly profit motivated. Closed systems, proprietary data formats, in little App-islands, is just for more profit? Because giant corporations, with all their political connections and ideological agendas, never do anything for *power*, only for money?
Seriously?
The terms ‘DRM’ and ‘copyright’ also don’t appear in this thread so far. So none of you seem to be even aware of how the powers that be use the DRM/copyright fake issue as a lever to screw down controls on free exchange of information. Which directly relates to the file system issue, since you can’t easily exchange things you can’t easily locate, group, hold and manipulate.
I say ‘fake issue’ since the very same business entities who’ve been screaming about ‘content piracy’ for years and how something must be done about it (eg MS’s Longhorn etc), turned out to have been behind the development and free distribution of most file sharing software. They created the problem, AS THEY INTENDED.
Problem, reaction, solution; the Hegelian Dialectic. To achieve an objective that would be unpopular, first manufacture a ‘crisis’ to which there is only one apparent solution – what you wanted in the first place. This is how politics has always worked, and probably always will.
The people who would control us all are very good at it.
For quite some time, they’ve been pretending that ‘hierarchical file systems’ are really a handicap to users, and we’d all be better off if the major computer OSs presented no such capability to the user.
It’s a deceitful lie. What they really want, is take way the user’s ability to accumulate, organize and manipulate large volumes of complex information.
Because a computer is an extension of the mind. And the last thing TPTB want, is for the public’s mind to be greatly extended. In fact they find the public’s awakening cognitive power very uncomfortable, and would like to give it a hierarchical filesystem lobotomy.
Incidentally I do think there are more capable ways to deal with data than the traditional folders & file tree structure. But that’s another story, for the future.
Take a peek in the end of that linked article. The author basically ends with the notion that; “Of course all this BLOWS far too much to be of any use for WORK or anything remotely important.”
All those “brilliant” “innovations” of “usability” for managing your personal collections of whatever.
– Save a picture of a monkey from the interwebs and you can find it in the pictures library along with your family pictures. Isn’t that neat.
– If you are a super brilliant geek, you can set up a folder for monkey pictures and another for family pictures. Unfortunately, you can’t have monkeys folder in the family folder, so pictures of your knuckle dragging parents have to go into the monkeys folder.
p.s.
Mailed a public_key file to myself to get it into my Android tablet (for use with ssh). Gmail app didn’t know what it was and didn’t let me save it. How convenient.
Yup, on Android what you could have done is used Dropbox and run a FTP server and pushed the file. All you needed to do then is go to the application and added it by browsing to the location you saved it…
Folders are just not that complex. I used to be the fixit guy for most of my friends and family who are not all that computer literate. They all use folders.
A point I often make is that there is often very little need for deep folder hierarchies as far as the user is concerned. And that is the behavior I see for most regular PC users. They simply don’t have the data needed for really deep hierarchies.
Pictures\myvacation – one level deep
Projects\2010\class\projectname – three levels deep
It rarely gets more complicated than that for the average user.
I applaud all these operating systems for throwing in some default folders (pictures, music, documents…) because it gives most users a starting point at least.
The problem with folders comes in maintaining them. People get lazy and don’t want to create a folder. I do it all the time. I’d rather the OS make some effort to help you keep it organized. Suppose I just copy a bunch of pictures to my Picture directory. Perhaps that would be a nice way for the file manager to ask ‘Do you wish to play these in a new folder?’ I know a popup dialog would be annoying, so I’m not suggesting that
I remember with Vista and they supposed database file system, I thought to myself… what is the demand here? I get the theoretical vision, but I didn’t see how it would play out. For real server work…. people already use real data bases. For things people have a lot of (pictures, music…) people already have dedicated index/search players.
It really is trying to solve a problem that doesn’t exist.
I think its better to keep folders, enhance it with search and help auto-organize.
I can see advantages and disadvantages to both directory structures and tagging approaches but I fail to understand the rationale to choose one over the other. They are not mutually exclusive.
I only started to see the benefits of metadata with the advent of MP3 and the huge libraries of songs that everybody has. My files are all inside ~/Music but while many of them are neatly stored within subdirectories following the Artist/Album/Song.mp3 scheme, the large majority of them are simply scattered all over the place.
The reason for the mess is that while I’ll bother to create subdirectories for entire albums when I rip them from the original media or download the entire album from somewhere I’ll hardly take the time to do the same things for those artists whose work I couldn’t care less other than that one song that I downloaded from YouTube using a converter or whatever.
While I could fix it to have it almost 100% neatly stored within directories following the above scheme using something like EasyTag leaving just those odd files with messed up tags out, I don’t really see the point. It is too much work to manage my music collection that way when database-like music managers such as Amarok do a stellar job, including letting you check your collection for the date the songs were added, the most or least played songs, etc. It would work with a few hundreds files but most people these days have a few thousands files, filling several gigs of their hard drives.
Same goes for pictures. Pretty much all photo management utilities use EXIF tags in one way or another to help one to categorize their photos by date, location, etc. At the very least, it will provide a way to create a timeline to visualize these pictures chronologically.
In both use cases, tagging mostly works because the tags are generated automatically (either by ripping the album or using the EXIF metadata info added by the camera) and that is a very important distinction to make. If the heavy lifting of filling metadata were to be left for the user to do, I doubt that most people would be able to find anything at all on their archives.
Yeah, metadata based search would help somewhat on this regard but I bet that it would be cumbersome as well to look for anything older than a couple of months on these users hard drives.
KDE is trying hard to make it work with its semantic desktop without too much success. It should be noted that while Nepomuk can use Strigi to gather metadata from the filesystem automatically to work more or less like Google Desktop does, it actually lets users create their own tags for their files but in practice I don’t see it being used much.
On the other hand, while directory structures might require the user to think a little bit about organizing their files up front, it actually pays off in the end when working collaboratively or simply when handling one’s files to transfer them to other devices, upload them or whatever.
Sure, database-like content management for music and pictures will let the user upload songs and/or pictures to their cellphones or whatever but that’s it. Even people with little interest in computers will have other kinds of files that they care about such as spreadsheets, letters, notes, etc for which there is no database-like file manager application and that’s where the file manager comes handy.
One can simply open the file manager, traverse the directory structure that, again, makes sense to his/her way of organizing things, can select one or several files at once and open it with the preferred app, delete it, copy it to their cell phones (at least, the non-Apple variety), copy it to their Dropbox folder, drag it into the open media player to create a quick playlist, etc.
Where the directory structure falls flat is when one needs a file to, e.g., be in two different directories. Symlinks help somewhat but it is not a perfect solution. Here I think that tagging on top of the traditional filesystem structure would make sense. Both approaches are complementary to each other IMHO.
Please accept my sincerest apologies for the huge post!
“I have honestly never seen a single person have any issues with directories, nested or no,”
Personally I have seen plenty and I have torn my hair out on many occasions trying to get people to understand where their stuff is. The fault is not theirs, the technology needs to be upgraded. It’s irritating for you, and me, that stuff is designed for the majority and that this makes the life of the minority more complex or difficult but that is the right way round surely.
The answer is more simple, now files come with meta-information, so, instead of having the folder /images/school/ you have tags or meta-info in the file that makes them easy to organize, remember that prolly the main reason for directories to exists is to easily find the information that is classified in each folder, now, that is not necessary, faster processors plus meta-information is making it more easy.
1100101101000101101010100010101000101000100101011010
0010101100100101100100101010101100010100010010000010
1000101010110100010101100100010010110101001010100110
0110100110101001010101010010101001010010101010010110
1010101010101010101001010001001010100010101011100100
0101001001010000101010101010001010111000111010100011
Forget folders, forget files – it’s all just ones
and zeroes, flying around in the matrix, m-a-a-n. You
humans and your arbritrary distinctions.
There are no actual ones and zeroes. Even that is an abstraction. There are microscopic pieces of disk material that are magnetized or demagnetized. And the transistorized equivalent in flash memory …
Please don’t take this as me advocating against the use of folder hierarchies (I think a combination folder/metadata tags approach is best), but there are at least two very real problems that they cause for end users.
1. Sharing documents with others via folder hierarchies alone absolutely sucks.
I deal with this problem every day at work. Unless those involved can agree on the specific document types they will be adding to the folders, and thus the specific folder categories that they need, they will always lose things. This is due to the simple fact that personal information management strategies are almost never the same, and placing things into location-based categorical hierarchies means that user x has to navigate exactly to where user y put a document if he wants to retrieve it. If he ever finds the document, he makes a copy of it to keep in his own folder structure because he doesn’t trust that he’ll be able to locate it again. Now there are two documents to update and keep in sync. Ugh.
2. Computers necessitate that ordinarily unstructured information be worked into a hierarchy.
Multiple studies have been done on information workers, even before the proliferation of desktop computers, that illustrate the need to create information by taking notes, etc., but not necessarily needing to keep that information. In fact, there is much evidence to suggest that it’s simply the act of taking notes that helps people process and digest information, but that in the case of information workers notes are rarely referenced after they’re taken. They’re kept, sure, but in relatively unorganized piles. They’re very rarely ever filed. But, computers require that the user make a decision as to where to put this type of ephemeral information. That’s pointless. And, after a while it creates a mess in the personal file hierarchy. Eventually, users will feel compelled to clean up and reorganize. That’s a waste of time.
Edited 2012-07-27 05:09 UTC
I fully agree with that article. The whole iOS and the “lack of directory structure” is taking a serious step backwards for computing. I experienced this first hand with my iPad.
I tried out a trial animation program on the iPad. I created a lengthy animation in the trial version, and decided I am going to purchase the full version. So I bought and installed the full version, but my animation didn’t appear in the full version.
After searching for over an hour I couldn’t figure out how to tell the full version to open my animation from the trial version. I emailed the software vendor, and they gave me a 20 step process that involved syncing with iTunes, un-plugging the iPad, copying files around on the iMac in hidden directories, then plugging my iPad back in and syncing again with the iPad. It worked, but what a step backwards for something that should be so damn simple!!!!
Long live the Desktop PC!!!
I really like having a hierarchical filesystem and find not having it available in iOS limiting. I like to move compatible documents through various programs that have there own strengths. The diversity allows me to do my best work, because I can take advantage of the best tool for any particular aspect of my work. The document centric world makes that kind of workflow easier, where as the app centric world is moving in a direction that prevents it. Not only is my iPad a walled garden, but the apps are becoming ivory towers within it.
Apple has done some tricks in the past that I have liked to simplify the clutter of files. I like Apples notion of making apps a bundle that appears as a file…. but in reality it can be opened to reveal a hierarchy of sub files within. Some more complex formats like Keynote employ the same technique to hide that they are in-fact, a configuration of files. It is also nice having the ability to flatten the view of the filesystem through the “All My Files” view. However, I have far too many documents to manage them effectively with just that flat view of my files.
I like the notion of moving everything into meta-data… but I would still like the filesystem metaphor to be available as a view. Gmail comes to mind as a good example of someone doing this already.
Directories and files are just a representation, and an unrealistic one at that, of how files and applications are stored on disk. I see no reason why it can’t be replaced by another representation or none.
I only deal with files during backup and syncing and with iOS I don’t need to even bother with that. On my desktop I use applications to access files or search.
I think thats why i never ever would by an Apple product. Used to have a imac G3 but i switched back to a pc.
Of course one can see it as a an innovation of sorts but i always hoped that one day there would be a filesystem that can handle tons of files in a folder.
To make a folderless approach is bold, but i assume that most users will not even care or know.
Glad i don’t use any of these devices and just have a pc.
I have for a long time disliked the current file system paradigm which has almost universal use in computer operating systems. I prefer a file system that more closely resembles real life filing systems and paradigms. For example, how often in real life filing systems do we put folders inside other folders inside other folders? Also, I dislike the restriction of having to have unique file names. Again this does not mimic real life situations. For example I can purchase several copies of a book and place them in a bookshelf, without the bookshelf spitting all but one of the books out for having the same name. I’m sick of having to have numerous files with similar names to support having multiple versions of a file.
Going back to the folder issue, I am also sick of having to traverse these complex directory trees in order to find anything. I believe that the folder concept should be implemented as meta data, allowing a file to “reside” in as many folders/categories and you like.