automate command line linux

Steve Stonebraker
June 28, 2011

How can you automate command line linux ?

Using Pexpect of course!

Pexpect is a pure Python module that makes Python a better tool for controlling and automating other programs. Pexpect is similar to the Don Libes `Expect` system, but Pexpect as a different interface that is easier to understand. Pexpect is basically a pattern matching system. It runs programs and watches output. When output matches a given pattern Pexpect can respond as if a human were typing responses. Pexpect can be used for automation, testing, and screen scraping. Pexpect can be used for automating interactive console applications such as ssh, ftp, passwd, telnet, etc. It can also be used to control web applications via `lynx`, `w3m`, or some other text-based web browser. Pexpect is pure Python. Unlike other Expect-like modules for Python Pexpect does not require TCL or Expect nor does it require C extensions to be compiled. It should work on any platform that supports the standard Python pty module.

Download and Installation

Download the current version here from the SourceForge site here: pexpect current

The Pexpect tarball is a standard Python Distutil distribution. Running the following commands should get you a working Pexpect module. Note that you have to have root access to install a site package.

 wget http://pexpect.sourceforge.net/pexpect-2.3.tar.gz
 tar xzf pexpect-2.3.tar.gz
 cd pexpect-2.3
 sudo python ./setup.py install




Examples


Under the distribution tarball directory you should find an  "examples" directory. This is the best way to learn to use Pexpect. See  the descriptions of Pexpect Examples.
 

topip.py


This runs `netstat` on a local or remote server. It calculates some  simple statistical information on the number of external inet  connections. This can be used to detect if one IP address is taking up  an excessive number of connections. It can also send an email alert if a  given IP address exceeds a threshold between runs of the script. This  script can be used as a drop-in Munin plugin or it can be used  stand-alone from cron. I used this on a busy web server that would  sometimes get hit with denial of service attacks. This made it easy to  see if a script was opening many multiple connections. A typical browser  would open fewer than 10 connections at once. A script might open over  100 simultaneous connections.
 

hive.py


This script creates SSH connections to a list of hosts that you  provide. Then you are given a command line prompt. Each shell command  that you enter is sent to all the hosts. The response from each host is  collected and printed. For example, you could connect to a dozen  different machines and reboot them all at once.
 

script.py


This implements a command similar to the classic BSD "script"  command. This will start a subshell and log all input and output to a  file. This demonstrates the interact() method of Pexpect.
 

fix_cvs_files.py


This is for cleaning up binary files improperly added to CVS. This  script scans the given path to find binary files; checks with CVS to see  if the sticky options are set to -kb; finally if sticky options are not  -kb then uses 'cvs admin' to set the -kb option.
 

ftp.py


This demonstrates an FTP "bookmark". This connects to an ftp site;  does a few ftp tasks; and then gives the user interactive control over  the session. In this case the "bookmark" is to a directory on the  OpenBSD ftp server. It puts you in the i386 packages directory. You can  easily modify this for other sites. This demonstrates the interact()  method of Pexpect.
 

monitor.py


This runs a sequence of commands on a remote host using SSH. It runs a  simple system checks such as uptime and free to monitor the state of  the remote host.
 

passmass.py


This will login to each given server and change the password of the  given user. This demonstrates scripting logins and passwords.
 

python.py


This starts the python interpreter and prints the greeting message  backwards. It then gives the user iteractive control of Python. It's  pretty useless!
 

rippy.py


This is a wizard for mencoder. It greatly simplifies the process of  ripping a DVD to Divx (mpeg4) format. It can transcode from any video  file to another. It has options for resampling the audio stream;  removing interlace artifacts, fitting to a target file size, etc. There  are lots of options, but the process is simple and easy to use.
 

sshls.py


This lists a directory on a remote machine.
 

ssh_tunnel.py


This starts an SSH tunnel to a remote machine. It monitors the connection and restarts the tunnel if it goes down.
 

uptime.py


This will run the uptime command and parse the output into variables.  This demonstrates using a single regular expression to match the output  of a command and capturing different variable in match groups. The  grouping regular expression handles a wide variety of different uptime  formats.

Overview

Pexpect can be used for automating interactive applications such as ssh, ftp, mencoder, passwd, etc. The Pexpect interface was designed to be easy to use. Here is an example of Pexpect in action:

   # This connects to the openbsd ftp site and
   # downloads the recursive directory listing.
   import pexpect
   child = pexpect.spawn ('ftp ftp.openbsd.org')
   child.expect ('Name .*: ')
   child.sendline ('anonymous')
   child.expect ('Password:')
   child.sendline ('[email protected]')
   child.expect ('ftp> ')
   child.sendline ('cd pub')
   child.expect('ftp> ')
   child.sendline ('get ls-lR.gz')
   child.expect('ftp> ')
   child.sendline ('bye')

Obviously you could write an ftp client using Python’s own ftplib module, but this is just a demonstration. You can use this technique with any application. This is especially handy if you are writing automated test tools.

There are two important methods in Pexpect — expect() and send() (or sendline() which is like send() with a linefeed). The expect() method waits for the child application to return a given strong. The string you specify is a regular expression, so you can match complicated patterns. The send() method writes a string to the child application. From the child’s point of view it looks just like someone typed the text from a terminal. After each call to expect() the before and after properties will be set to the text printed by child application. The before property will contain all text up to the expected string pattern. The after string will contain the text that was matched by the expected pattern. The match property is set to the re MatchObject.

An example of Pexpect in action may make things more clear. This example uses ftp to login to the OpenBSD site; list files in a directory; and then pass interactive control of the ftp session to the human user.

   import pexpect
   child = pexpect.spawn ('ftp ftp.openbsd.org')
   child.expect ('Name .*: ')
   child.sendline ('anonymous')
   child.expect ('Password:')
   child.sendline ('[email protected]')
   child.expect ('ftp> ')
   child.sendline ('ls /pub/OpenBSD/')
   child.expect ('ftp> ')
   print child.before   # Print the result of the ls command.
   child.interact()     # Give control of the child to the user.

Special EOF and TIMEOUT patterns

There are two special patterns to match the End Of File or a Timeout condition. You you can pass these patterns to expect(). These patterns are not regular expressions. Use them like predefined constants.

If the child has died and you have read all the child’s output then ordinarily expect() will raise an EOF exception. You can read everything up to the EOF without generating an exception by using the EOF pattern expect(pexpect.EOF). In this case everything the child has output will be available in the before property.

Lists if patterns

The pattern given to expect() may be a regular expression or it may also be a list of regular expressions. This allows you to match multiple optional responses. The expect() method returns the index of the pattern that was matched. For example, say you wanted to login to a server. After entering a password you could get various responses from the server — your password could be rejected; or you could be allowed in and asked for your terminal type; or you could be let right in and given a command prompt. The following code fragment gives an example of this:

   child.expect('password:')
   child.sendline (my_secret_password)
   # We expect any of these three patterns...
   i = child.expect (['Permission denied', 'Terminal type', '[#\$] '])
   if i==0:
       print 'Permission denied on host. Can't login'
       child.kill(0)
   elif i==2:
       print 'Login OK... need to send terminal type.'
       child.sendline('vt100')
       child.expect ('[#\$] ')
   elif i==3:
       print 'Login OK.'
       print 'Shell command prompt', child.after

If nothing matches an expected pattern then expect will eventually raise a TIMEOUT exception. The default time is 30 seconds, but you can change this by passing a timeout argument to expect():

   # Wait no more than 2 minutes (120 seconds) for password prompt.
   child.expect('password:', timeout=120)

Find the end of line — CR/LF conventions

Matching at the end of a line can be tricky. In general the $ regex pattern is useless. Pexpect matches regular expressions a little differently than what you might be used to. The $ matches the end of string, but Pexpect reads from the child one character at a time, so each character looks like the end of a line. Pexpect can’t do a look-ahead into the child’s output stream. In general you would have this situation when using regular expressions with a stream. Note, pexpect does have an internal buffer, so reads are faster than one character at a time, but from the user’s perspective the regex pattern test happens one character at a time.

The best way to match the end of a line is to look for the newline: “\r\n” (CR/LF). Yes, that does appear to be DOS-style. It may surprise some UNIX people to learn that terminal TTY device drivers (dumb, vt100, ANSI, xterm, etc.) all use the CR/LF combination to mark the end of line. UNIX uses just linefeeds to end lines in files, but not when it comes to TTY devices! Pexpect uses a Pseudo-TTY device to talk to the child application, so when the child application prints a “\n” your TTY device actually sees “\r\n”. TTY devices are more like the Windows world. Each line of text end with a CR/LF combination. When you intercept data from a UNIX command from a TTY device you will find that the TTY device outputs a CR/LF combination. A UNIX command may only write a linefeed (\n), but the TTY device driver converts it to CR/LF. This means that your terminal will see lines end with CR/LF (hex 0D 0A). Since Pexpect emulates a terminal, to match ends of lines you have to expect the CR/LF combination.

   child.expect ('\r\n')

If you just need to skip past a new line then expect (‘\n’) by itself will work, but if you are expecting a specific pattern before the end of line then you need to explicitly look for the \r. For example the following expects a word at the end of a line:

   child.expect ('\w+\r\n')

But the following would both fail:

   child.expect ('\w+\n')

And as explained before, trying to use ‘$’ to match the end of line would not work either:

   child.expect ('\w+$')

So if you need to explicitly look for the END OF LINE, you want to look for the CR/LF combination — not just the LF and not the $ pattern.

This problem is not limited to Pexpect. This problem happens any time you try to perform a regular expression match on a stream. Regular expressions need to look ahead. With a stream it is hard to look ahead because the process generating the stream may not be finished. There is no way to know if the process has paused momentarily or is finished and waiting for you. Pexpect must implicitly always do a NON greedy match (minimal) at the end of a input {### already said this}.

Pexpect compiles all regular expressions with the DOTALL flag. With the DOTALL flag a “.” will match a newline.

Beware of + and * at the end of input

Remember that any time you try to match a pattern that needs look-ahead that you will always get a minimal match (non greedy). For example, the following will always return just one character:

   child.expect ('.+')

This example will match successfully, but will always return no characters:

   child.expect ('.*')

Generally any star * expression will match as little as possible.

One thing you can do is to try to force a non-ambiguous character at the end of your \d+ pattern. Expect that character to delimit the string. For example, you might try making thr end of your pattrn be \D+ instead of \D*. That means number digits alone would not satisfy the (\d+) pattern. You would need some number(s) and at least one \D at the end.

Matching groups

You can group regular expression using parenthesis. After a match, the match parameter of the spawn object will contain the Python re.match object.

Debugging

If you get the string value of a pexpect.spawn object you will get lots of useful debugging information. For debugging it’s very useful to use the following pattern:

try:

   i = child.expect ([pattern1, pattern2, pattern3, etc])

except:

   print "Exception was thrown"
   print "debug information:"
   print str(child)

It is also useful to log the child’s input and out to a file or the screen. The following will turn on logging and send output to stdout (the screen).

   child = pexpect.spawn (foo)
   child.logfile = sys.stdout

Exceptions

EOF

Note that two flavors of EOF Exception may be thrown. They are virtually identical except for the message string. For practical purposes you should have no need to distinguish between them, but they do give a little extra information about what type of platform you are running. The two messages are:

   End Of File (EOF) in read(). Exception style platform.
   End Of File (EOF) in read(). Empty string style platform.

Some UNIX platforms will throw an exception when you try to read from a file descriptor in the EOF state. Other UNIX platforms instead quietly return an empty string to indicate that the EOF state has been reached.

Expecting EOF

If you wish to read up to the end of the child’s output without generating an EOF exception then use the expect(pexpect.EOF) method.

TIMEOUT

The expect() and read() methods will also timeout if the child does not generate any output for a given amount of time. If this happens they will raise a TIMEOUT exception. You can have these method ignore a timeout and block indefinitely by passing None for the timeout parameter.

   child.expect(pexpect.EOF, timeout=None)