One of the innovations that the V7 Bourne shell introduced was built in shell wildcard globbing, which is to say expanding things like
↫ Chris Siebenmann*,?, and so on. Of course Unix had shell wildcards well before V7, but in V6 and earlier, the shell didn’t implement globbing itself; instead this was delegated to an external program,/etc/glob(this affects things like looking into the history of Unix shell wildcards, because you have to know to look at theglobsource, not the shell).
I never knew expanding wildcars in UNIX shells was once done by a separate program, but if you stop and think about the original UNIX philosophy, it kind of makes sense. On a slightly related note, I’m currently very deep into setting up, playing with, and actively using HP-UX 11i v1 on the HP c8000 I was able to buy thanks to countless donations from you all, OSNews readers, and one of the things I want to get working is email in dtmail, the CDE email program. However, dtmail is old, and wants you to do email the UNIX way: instead of dtmail retrieving and sending email itself, it expects other programs to those tasks for you.
In other words, to setup and use dtmail (instead of relying on a 2010 port of Thunderbird), I’ll have to learn how to set up things like sendmail, fetchmail, or alternatives to those tools. Those programs will in turn dump the emails in the maildir format for dtmail to work with. Configuring these tools could very well be above my paygrade, but I’ll do my best to try and get it working – I think it’s more authentic to use something like dtmail than a random Thunderbird port.
In any event, this, too, feels very UNIX-y, much like delegating wildcard expansion to a separate program. What this also shows is that the “UNIX philosophy” was subject to erosion from the very beginning, and really isn’t a modern phenomenon like many people seem to imply. I doubt many of the people complaining about the demise of the UNIX philosophy today even knew wildcard expansion used to be done by a separate program.
Mandatory prof Kernighan on the whole one-program-one-action thing and very cool pipelining demos:
https://www.youtube.com/watch?v=tc4ROCJYbm0
I strongly recommend his UNIX: A History and a Memoir. Tons of tales about how specific UNIX utilities came to be.
The UNIX philosophy is alive for the ones who want it. Tons of utilities remain alive and maintained in Linux and the BSDs.
Case in point – I write some quite complex automations for Azure in Bash and I prefer to use curl or wget and the REST APIs vs. Azure CLI.
Azure CLI is so heavy and so slow and eats so much RAM that it actually using plain REST saves me from requiring more RAM in the machines that run the automations.
I can’t find the article now but, a year or so ago, I went through a blog post and the author claims having done faster (an order of magnitude) parse of a huge (few TB) dataset using basic awk, sort than natively via SQL.
Sometimes I prefer to use the basic tools because I know they will be there. Not everyone using my scripts can install applications, so I know I can count on awk being installed, but I can’t always count on jq. Usually, there will be curl or wget, so using REST makes my automations compatible out-of-the-box with BSD, rather than counting on the user installing Azure CLI, etc..
Thanks to the magic of the open source world and people being as different as they are, we can live the raw pure UNIX way if we want. Or count on modern combined tools if we wish. For example, I have full access to my email on my self hosted server from my HP 712, point-to-point, in my “vintage VLAN”. Different rules apply connecting from the Internet. All thanks to the power of mixing and matching UNIX tools. =)
Or… https://www.terminal.shop
There’s really for something for everyone these days!
Shiunbird,
We take it all for granted today, but in those years it was revolutionary.
Ahh, I absolutely hate shell programming, haha. I would go strait to any real programming language for all but the most trivial of commands. I don’t mind shells executing commands as they were designed to do in the beginning, but IMHO cramming programming capabilities into them has created some of the most hamfisted & awkward programming languages we have. Even simple algorithms and structures have to get translated into obtuse shell form. Just because we can doesn’t make them a good choice. But to each their own 🙂
SQL databases can be optimized. However I do find that getting data into a relational database in the first place can take time and create bottlenecks. When that’s the case external tools that compute results directly on source data can eliminate tons of data transfer/conversion overhead. If you don’t need a database or the power of SQL, then go ahead and use simpler tools instead 🙂
We actually went from a C# application doing tons of string concatenations to call Linux commands to Bash scripts, which was DEFINITELY the correct move. =)
I use Bash a lot to automate system administration tasks, and this is probably the ideal use case for it. Also, I do a lot of one liners to go through text/parse output of commands, as second nature already – tons and tons of pipes. But it’s nothing that I’d recommend to someone else. I just grew into it naturally.
It seems that we are in agreement – there is always the best tool for the job. And “best” depends on the operator, too. Sometimes better to use the solution you know best than doing a shitty job with the “theoretically ideal” solution.
We have choice these days, and computers are fast enough to run most things just fine.
Shiunbird,
Everyone’s entitled to an opinion, I say to each their own 🙂
But I can assure you that you will not convince me that bash is good for programming of anything significant. If you are only using it to call other command line tools that’s one thing, but the native programming facilities are awful and I do think the majority of programmers would feel the same way.
That’s true, computers can be fast enough to make inefficient approaches viable. However my main criticism of bash scripting isn’t the spawning tons of children and opening up lots of pipes, but rather how painful it is to implement advanced programming algorithms and structures natively in bash on it’s own merits. As a professional programmer, why would I bother using tools that are both beneath my skill set and harder to use?
I agree with you about how cool unix pipes are though.
I prefer when the program being launched performs the expansion.only when it’s appropriate rather than the shell presuming that * inside of command line parameters should match files. IMHO DOS did this better since commands could explicitly decide if that was the appropriate thing to do, which seems superior to me.
Consider something like
Today on a unix system what this does is ambiguous and it depends on the contents of the current working directory.
If there are no files/directory matching test, then the command is executed as is printing “test*”. But if there is a file or directory matching test, then you’ll get output like “test test.c test.html” despite the fact that echo has nothing to do with files. This is just an illustrative example but consider other applications where * makes sense however the tool is meant to operate beyond the domain of the local file system, maybe the tool performs math, matches text in a file, runs sql commands, or it’s for a filespec on a remote system…
Most of the time this will execute correctly because “othermachine” does not exist as a local file, but if coincidentally or maliciously the local file does exist then it will be expanded and the wrong rsync command gets executed. This expansion can actually make things unnecessarily dangerous in some cases of user error:
Since the shell performs the expansion without cp being able to see that a destination file operand is missing, cp doesn’t know any better than to accept the expansion as a source and destination, which is wrong.
Of course I understand that the * can/should be escapped so the shell won’t replace it, but nevertheless I find such inconsistencies to be extremely annoying and this design makes for bad outcomes. I find it would be much better to let the tool decide if/when local file expansion is appropriate…alas the convention is already set in stone and all our commandline tools assume the shell is responsible for expanding wildcards. It’s way too late to fix this.
The discussion about doing “email the UNIX way” goes a bit deeper. Some of the “UNIX way” here is multi-user..
Having all this power on a desktop ( nevermind phone ) is not how it always was. UNIX was not designed as a single-user desktop. Mail on UNIX was designed with the expectation that relatively simple computers with little storage would be working as terminals to much more powerful computers with “lots” of storage. The mail system in UNIX ran on a central computer where everybody would have their emails stored. A Mail Transfer Agent (MTA), like Sendmail, would fetch, send, and organize mail by user account on the central server as local files in a standard layout and format. The email client that each user would use on their terminal was just a simple front-end to this system.
On a desktop system where even “root” and “user” are the same person, none of this local complexity makes much sense and is in fact a bit contrary to the UNIX philosophy. Mail clients that can directly interact with remote servers are a better reflection of the UNIX philosophy on a modern desktop.
Mail/mailx (POSIX mail) can be configured to interact with remote SMTP servers directly without needing a local MTA like Sendmail. Older versions of mailx may lack this option though.
I do not know much about dtmail. The man page talks about mailx but it is not clear if it uses mailx under the hood or just uses the same commands and directory structure. I suspect the latter. In that case, or if mailx lacks SMTP options, dtmail will require a local MTA.
My favourite email client (or Mail User Agent – MUA) from back in the day was ELM which was actually created at HP (quite likely on HP-UX though I used it on Sun workstations and early Linux).