Command-line interactive programs in UNIX shell-scripts

Like it or not, but sooner or later you realize that you’ll have to write shell-scripts to administer UNIX. And among these scripts there certainly will be those to cooperate with interactive applications such as telnet, ftp, su, password, ssh. But it means the end of the admin’s quiet life because while dealing with interactive programs one often come across numerous hidden traps which doesn’t usually happen with ordinary sh-scripts. Though fortunately or may be not, but most of these problems generally
turn up within first five minutes of the work under the script. The symptoms typically look like that author can’t pass the authentication from the script. At first you feel confused because usual pipe constructions such as:

$ echo luser && echo TopSecret | telnet foo.bar.com

fail you and the problem which seemed so plain on the face of
it grows into “mission impossible”. Yet, it isn’t all
so very hopeless and quite a simple solution of the problem will
come quickly enough for many of the dialogue applications include
built-in mechanisms of handling scripts. See, for example,
standard FreeBSD ftp-client:

$ echo '$FILEPUT' | ftp -N ftprc [email protected]

This command will get ftp to connect with the foo.bar.com
hosh with the name of luser and start FILEPUT
macro described in the file ftprc. Besides the macro
there must be also described the host, the login and user’s
password in this file:

$ cat ftprc
machine foo.bar.com
	login luser
	password TopSecret

macdef FILEPUT
	binary
	cd /tmp
	put some_usefull_file.bin
	bye
		<-- Attention! There is a new-line symbol at the end of the macro.
$

If for some reason the dialogue application doesn't support
the built-in scripts then you'll easily find it's freeware
counterpart capable to act automatically. Really you are not the
first to run into such a problem 🙂

Another possibility to persuade an interactive program to do
work by itself, without a user, is to redirect its standard input.
For example, much simplified script to start Oracle database may
look like this:

#!/bin/sh

su - oracle -Ó /oracle/bin/svrmgrl <
Yet we can't work like that with all kinds of applications.

For instance, it isn't suitable for telnet, through the
"problem of user's authentication" mentioned above. The

difficulty to debug scripts at the moment of authentication

complicates matters. So, it's clear that to find the way out we

need some special solution. Let's don't break good traditions and

google the Internet. Going through different search systems

brings us some fruits in the form of indistinct mumbling about

the untimely closed I/O data streams, TTYs and PTYs (pseudoterminals)

and all the rest of it. But in all this incongruous abundance

you'll certanly find the links to
expect
It's just what is wanted: the tool, which is traditionally

used to communicate automatically with interactive programs. And

as it always occurs, there is unfortunately a little fault in it:

expect needs the programming language TCL to be present.

Nevertheless if it doesn't discourage you to install and learn

one more, though very powerful language, then you can stop your

search, because expect and TCL with or without TK have

everything and even more for you to write scripts.
If there is no expect installed in your system you

should install it. On FreeBSD you'd better do it by using port-system:
# cd /usr/ports/lang/expect
# make install clean
As a result expect and all it needs will be

downloaded from the Internet and installed on your system.
Now we can work with the TCL and expect. As for the

problem of intercative programs, the expect-script for a short

telnet-session with host foo.bar.com (let it be SCO

UnixWare-7.1.3), under login of luser with password TopSecret
can look like that:
#!/usr/bin/expect

spawn telnet foo.bar.com
expect ogin {send luser\r}
expect assword {send TopSecret\r}
send "who am i\r"
send "exit\r"
expect eof
By the way, the README file of the expect says there

is a libexpect library that can be used to write

programs on C/C++ which allows to avoid the use of TCL itself.

But I'm afraid, this subject is beyond this article. Besides

authors of expect themselves seem to prefer expect-scripts

to the library.

However, if in spite of all the attractiveness of the

foregoing method and all the arguments of the expect

authors (see FAQ) you have made up your mind not to use expect,

then you are either too lazy or entirely poisoned by Perl. Well,

in this case your salvation lies in the installation of the

corresponding Perl-module (http://sourceforge.net/projects/expectperl),

which is supposed to support all the functions of the original expect,

On FreeBSD you can carry it out by the installation from ports:
# cd /usr/ports/lang/p5-Expect
# make install clean
Now our example with telnet-session will be like that:
#!/usr/bin/perl

use Expect;

my $exp = Expect->spawn("telnet foo.bar.com");
$exp->expect($timeout,
        [ 'ogin: $' => sub {
                            $exp->send("luser\n");
                            exp_continue; }
        ],

        [ 'assword:$' => sub {
                                $exp->send("TopSecret\n");
                                exp_continue; }
        ],
        '-re', qr'[#>:] $'
);
$exp->send("who am i\n");
$exp->send("exit\n");
$exp->soft_close();
If I an mistaken and your attachment to Perl isn't very strong

you can take the opportunity of using Python as a corresponding

module pexpect is written for it (http://pexpect.sourceforge.net).

It's clear that Python language should be installed on the system

beforehand, otherwise the FreeBSD ports will help you again:
# cd /usr/ports/lang/python
# make install clean 
And the same for the pexpect module:
# cd /usr/ports/misc/py-pexpect
# make install clean
The script of our telnet-session in Python will be like that:
#!/usr/local/bin/python

import pexpect

child = pexpect.spawn('telnet foo.bar.com');

child.expect('ogin: ');
child.sendline('luser');

child.expect('assword:');
child.sendline('TopSecret');

child.sendline('who am i');
child.sendline('exit');

child.expect(pexpect.EOF);

print child.before;
Certainly, if for some reason Python doesn't suit you either

you can install, let us say, PHP language. Well, I think you

realize that the searching of suitable solution can go on for a

long time and may be only MS Visual Basic will be lacking in the

list of results. So, I believe the time has already approached to

put it all aside and come to 
to the Point.
Well, now I'll tell you what really takes place when we start

interactive applications from shell scripts. Though the last

sentence is a bit in the stile of a conclusive speech of Hercules

Poirot we are rather far from the end of narrative. In fact we

are at its beginning. So let's forget everything suggested by expect
and its worthy clones.
At first we'll try to make some rough model to simulate the

expect-like programs. Let's use sh-scripts with all the standard

UNIX tools trying to get our point and the target of this article.

It's essential to note that all the experiments are valid for

FreeBSD and I can't guarantee they'll give the same results on

other operating systems.
To simplify out difficult task and not to fight with pipes

mentioned in the first examples of the article let's make to FIFO-files:

one for the standard input (in.fifo) and the other for the output

(out.fifo) leaving alone the standard error-flow for now:
$ mkfifo in.fifo
$ mkfifo out.fifo
Then with the redirection its I/O to in.fifo and out.fifo let

us execute unsafe telnet of which, I'm afraid, you are already

sick and tired because of numerous examples on expect, perl and

python:
$ telnet -K localhost > out.fifo < in.fifo
A little step aside: as all the experiments are hold on the

FreeBSD box, when we attempt to connect with the FreeBSD telnet

server, SRA secure login mechanisms are automatically involved to

secure telnet-session:
$ telnet localhost
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Trying SRA secure login:
User (luser):			<-- server waits for login, no new-line printed by server

To avoid it and return telnet its traditional

behavior let's start telnet with the -K option:
$ telnet -K localhost
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

FreeBSD/i386 (unity) (ttyp1)

login:				<-- server waits for login, no new-line printed by server
So let our first script (test_1.sh) consist of the following

lines:
#!/bin/sh

mkfifo in.fifo
mkfifo out.fifo

telnet -K localhost > out.fifo < in.fifo 
After executing the script:
$ ./test_1.sh &
we can begin our experiments. Pay attention to the &

parameter which brings the script to the background. It's done

for our possibility to continue working with the same terminal

during the experiments. For example, let's try to read something

from the out.fifo and to write something in the in.fifo:
$ cat out.fifo &
The & parameter plays the same above-mentioned role.

Though, unfortunately the command cat will not show

anything on the screen. Perhaps, it is caused by the blocks on

reading/writing during the work with the FIFO-files: writing may

be blocked till there is no one to read data from FIFO; and

reading is blocked too till the other side isn't ready to write

in it. May be, the whole process is hampered by our in.fifo,

where there is nothing written. Let's check our guess by sending

a newline symbol into the input-channel:
$ echo > in.fifo
Either our guess has been correct or the true reason lies

somewhere else but the miracle has happened! The long-awaited

results of the command cat out.fifo have at last

appeared on the terminal:
$ Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

FreeBSD/i386 (unity) (ttyp1)


login: login:
The only thing that is a little out of the way is the twice-repeated

login:. Actually, that was the natural reaction of the

telnet server on the newline symbol which was sent by echo

> in.fifo command. So let's modify the command echo

> in.fifo to avoid the additional newline

character:
$ echo -n > in.fifo
May be, it will be even better to use the following

combination:
$ cat > in.fifo &
In the future it will prevent the closing of the input stream,

which is certainly interpreted by telnet server as the connection

breakup.
Well, now we can go on with our research but first we must get

rid of all the background processes, which were caused by using

&. Now run fg command and then send ^C to finish the

process:
$ fg
./test_1.sh
^C[2]  + Done                          cat out.fifo 
The next step will be an attempt to implement the main

functions of expect by filtering the output (out.fifo)

to find the necessary data and sending an answer:
expect request {send answer}
At first glance something like this seems suitable:
$ cat out.fifo | grep request && echo answer
Though in our case this chain won't work correctly even at the

grep point. The matter is the grep is designed

to print strings in accordance with the pattern. So it's quite

natural that before comparing every new line with the pattern grep

always waits for the end either of line ('\n') or file ('\0').

This particular feature makes it impossible to intercept "login:"

with the help of the grep because "login:"
isn't followed by the newline ('\n') character (the system still

waits for user to enter his login-name on the same line). This is

shown above on the example with telnet -K localhost.

Thus grep will be waiting till the end of time for the end

of line (EOL) and at last on telnet-server timeout it'll see

just the end of file (EOF).

It's clear another way is needed to deal with these unfinished

lines. As a possibility the dd command can be used

instead of cat. In circle the dd will send data

character by character to the grep. I mean the following

construction, which is shown in the example of already working

expect.sh script:
#!/bin/sh

while :; do
        dd if=out.fifo bs=1b count=1 2>/dev/null | grep $1
        if [ $? -eq 0 ]; then
                # Match found
                echo "$2" > in.fifo
                exit 0
        fi
        # Match not found, let's play again
done
The script can be started in this way (files in.fifo and out.fifo

should be already created):
$ ./expect.sh "request" "answer"
So the time has come to gather everything together in one

script to make our traditional telnet-session work automatically:
#!/bin/sh

mkfifo out.fifo in.fifo

telnet -K localhost 1> out.fifo 0< in.fifo &

cat > in.fifo &
cat out.fifo > out.fifo &

pid=`jobid`

./expect.sh "ogin" "luser"
./expect.sh "word" "TopSecret"

sleep 1
echo 'who am i > /tmp/test.txt' > in.fifo
sleep 1
echo "exit" > in.fifo

rm out.fifo in.fifo
kill $pid
After executing this script we may get, in the case of

success, the following output:
$ ./test_2.sh
login:
Password:
Connection closed by foreign host.
And as the war trophy there will appear a file /tmp/test.txt

to confirm the succeeding of our experiment:
$ cat /tmp/test.txt
luser             ttyp3    May 10 16:39 (localhost)
But if something was wrong, the command kill may be

used for each process left after the failed experiment.
Unfortunately, this script is quite unstable: it very much

depends on the value of the sleep parameter and the

reply speed of the telnet-server. So the data don't always come

in time to be properly filtered by dd and grep it

causes script hang-ups. And you must kill these processes. It's

also clear that such a construction doesn't work everywhere, for

example, on Linux I didn't got any acceptable results. So maybe

the adepts of Linux will succeed in it.
As for me
I decided to go on and try to find a tool, which can be used

as an expect-substitute for the pure shell without any

superstructure as TCL, Perl or Python. I realized it must be

written in C and ported for as many operating systems as possible.

Well, let's google! And after some time such a program is found.

It's pty-4.0 written by Daniel J. Bernstein in 1992. But as far

as I can see it haven't been developed after this release.
After a while I even succeeded to compile the source code of

pty-4.0 into binary executable. And some parts of it began to run.

But by that moment I have realized that it's much easier to write

my own program than to sort out the old one.
And why not to write? So I set about studying the problem more

carefully. Before long I found out that the most convenient way

to communicate with interactive applications was really to

imitate a terminal for them. And the place of pseudoterminals in

the structure of the future program was determined clearly

enough, in spite of the contradictive mumbling of specialists

from Internet about expect and PTY-sessions. Next, it

also because clear that it makes everything easy to start

applications under the control of the PTY-sessions inside some

kind of shell, for example TCL, Perl and others. Though nothing

prevent us from using C and pure sh interpreter.
As a result of all this in a couple of weeks I had a working

version of empty (http://www.sourceforge.net/projects/empty)

which allows to start interactive programs and communicate with

them using FIFO-files. For example, the FreeBSD telnet-session in

the sh-script for empty will look like that:
#!/bin/sh

empty -f -i in.fifo -o out.fifo telnet -K localhost
empty -w -i out.fifo -o in.fifo -t 5 "ogin:" "luser"
empty -w -i out.fifo -o in.fifo -t 5 "assword:" "TopSecret"
empty -s -o in.fifo 'who am i > /tmp/test.txt'
empty -s -o in.fifo 'exit'
Well, it's much shorter then the buggy test_2.sh script and

works quite stable on BSD, Linux and Solaris. Besides, it doesn't

require TCL, Perl or Python. I admit there aren't so many

functions in the program yet as there are in other expect-like

tools, but I hope everything is still ahead.



If you would like to see your thoughts or experiences with technology published, please consider writing an article for OSNews.