— Expect —

A Tool For Automating

Interactive Programs

Don Libes
libes@nist.gov

National Institute of Standards and Technology

Expect: Introduction and Overview Don Libes 1

 Expect – Why?

• A tool for automating interactive programs

• tip, telnet, ftp, passwd, su, rogue . . .

% telnet medlib.iaims.georgetown.edu
Trying 141.161.42.1 ...
Connected to medlib.iaims.georgetown.edu.
Escape character is ‘^]’.
UNIX(r) System V Release 4.0 (medlib)
login: medlib
Enter Your Last Name: mulroney
Password:XXXXXXXX
... menu ...
Enter a number or Q to QUIT: 4

Expect: Introduction and Overview Don Libes 2

 Talk Overview

• Motivation
• Basics, Examples
• Patterns and Actions
• Autoexpect
• Total automation vs partial automation
• Converting line-oriented apps to window-apps
• Background: CGI, cron, etc
• Security
• Esoterica
• URLs and other pointers

Expect: Introduction and Overview Don Libes 3

 Expect Commands Are Simple

• Start an interactive process

spawn ftp tribble.cme.nist.gov

• Send a string
send "dir\r"
send "get $file\r"

• Wait for a response

expect "Password:"
expect "200*ftp>"

• Pass control from script to user

interact

Expect: Introduction and Overview Don Libes 4

 Example – /bin/passwd

spawn /bin/passwd $user

expect "Password:"
send "$password\r"
expect "Password:"
send "$password\r"
expect eof

Expect: Introduction and Overview Don Libes 5

 Another Example: Dial a Modem

• Connecting to a remote site through a modem

spawn tip modem

expect "connected"
send "ATZ\r"
expect "OK"
send "ATD1234567\r"
expect "CONNECT"
send "\r\r"
expect "login:"
send "library\r"
interact

Expect: Introduction and Overview Don Libes 6

 Tcl – A Scripting Language

• Tcl: a shell-like extensible interpreter by Ousterhout, Winter

1990 USENIX. Example:

set temperature 20

if {$temperature < 50} {

puts "It’s pretty cold."
} elseif {$temperature > 70} {
puts "It’s really hot."
} else {
puts "It feels like spring!"
}

Expect: Introduction and Overview Don Libes 7

 A While Loop

while {$temperature < 60} {

puts "$temperature is still chilly!"
heat 10 min
}
puts "Ah, that’s warm enough."

• break: break out of a while loop

• continue: continue a while loop from the beginning
• return: return from this procedure

Expect: Introduction and Overview Don Libes 8

 More Password Automation

• Changing passwords on accounts on multiple machines

foreach host $hostlist {

spawn rlogin $host
expect "$prompt"
send "/bin/passwd\r"
expect "Old password:"
send "$oldpass\r"
. . .
}

• Actual script is parameterized

Expect: Introduction and Overview Don Libes 9

 Passmass – Already Written!

• Handles different access methods (telnet, rlogin, etc.)

• Handles different password programs (passwd, yppasswd)
• Handles different user names, prompts, host equivalencing
• No special knowledge of daemons, password formats, encryption

Expect: Introduction and Overview Don Libes 10

 In The Same Way . . .

• Expect easily controls:

• VMS systems
• Printers
• Modems
• Pagers
• Routers
• Servers
• Black boxes
• ...and more
• Expect is also useful for testing
• Software
• Hardware
• Test Suites: Cygnus (DejaGnu), X/Open, VSC4 (test suite for XPG), NIST

Expect: Introduction and Overview Don Libes 11

 Actions

• Tcl’s if command has an action

if {$temp == 100} {puts "It’s really hot!"}
if {$temp == 100} {
puts "It’s really hot!"
}
• Expect’s action work the same way
expect "100" {puts "It’s really hot!}
expect "100" {
puts "It’s really hot!
}

Expect: Introduction and Overview Don Libes 12

 Pattern/Action

expect "pattern" action "pattern" action

expect {
"pattern1" action1
"pattern2" action2
"pattern3" {
action3a
action3b
action3c
}
}

Expect: Introduction and Overview Don Libes 13

 Waiting For Different Responses

expect {
"does not exist" exit
"password: " {
send "$password\r"
# more stuff can done here
}
}

• Actions can include expect commands

• Simple actions do not need braces, example: exit

Expect: Introduction and Overview Don Libes 14

 Waiting For Different Responses – part 2

• Host equivalencing produces different prompts

expect {
"$shellprompt" {
send "/bin/passwd\r"
}
"Password:" {
send "$password\r"
exp_continue
}

Expect: Introduction and Overview Don Libes 15

 More On Patterns

• Prompts can include variables

expect "$shellprompt"
• Glob patterns (Shell-style)
expect -gl "catch a falling *"
• Regular expressions
expect -re "(login|Username):"
• Exact strings
expect -ex "catch a falling *"

Expect: Introduction and Overview Don Libes 16

 Mixed patterns

expect {
-re "3.*ftp>" action3
-re "2.*ftp>" action2
-gl "ftp>" actionDefault
}

• Expect’s internal pattern matching strategy is intuitive

• Loop until match
• Match patterns in order
• Idle while waiting for more input
• Expect is event-loop compliant

Expect: Introduction and Overview Don Libes 17

 Anchors

• ^ matches the beginning of the buffer

• $ matches the end of the buffer
• Valid for -gl and -re
• Not for -ex

Expect: Introduction and Overview Don Libes 18

 Keywords

expect eof action

expect timeout action

set timeout 60
set timeout -1 ;# no timeout

• These are implicit in every expect command. Consider:

expect "foo"

Expect: Introduction and Overview Don Libes 19

 Timeout Example

• Host equivalencing produces different prompts

expect {
"$shellprompt" {
send "/bin/passwd\r"
}
"Password:" {
send "$password\r"
exp_continue
}
timeout {
puts "timed out!"
continue
}
Expect: Introduction and Overview Don Libes 20
 Alternatives To Expect

• Automating passwd – the hacker approach

• Get source (if possible) and modify command-line argument handling.
• If no source...
• Encrypt passwords
• Lock/read/write password database
• God forbid any of these change
• NIS
• Kerberos
• Shadow passwords
• Rechange, retest, redebug...
• Expect – the solution for the rest of us
• Re-usable on passwd, telnet, others

Expect: Introduction and Overview Don Libes 21

 Partial Automation

• Sometimes, it is inappropriate to totally automate

spawn telnet $host

expect "login:" {send "$name\r"}
expect "Password:" {send "$password\r"}
expect "$prompt" {send "cd $dir\r"}
interact

• interact works in both directions

Expect: Introduction and Overview Don Libes 22

 Example: fsck, The File System Checker

• fsck: a typical vital program with a poor interface

• fsck -y or fsck -n (that’s it for programmability!)

while {1} {
expect {
eof break
-re "UNREF FILE.*CLEAR.*?" {send"y\r"}
-re "BAD INODE.*FIX.*?" {send "n\r"}
-re "? " {interact}
}
}

Expect: Introduction and Overview Don Libes 23

 Interact patterns/actions

interact pattern action pattern action . . .

while {1} {
expect {
eof break
-re "UNREF FILE.*CLEAR.*?" {send"y\r"}
-re "BAD INODE.*FIX.*?" {send "n\r"}
-re "? " {
interact "+" return
}
}
}

Expect: Introduction and Overview Don Libes 24

 Example: Adding Commands To ftp

• Better than fsck in terms of programmability

• But not much! No reliability.

interact {
"~g\r" {get_current_directory}
"~p\r" {put_current_directory}
"~l\r" {list_current_directory}
}

Expect: Introduction and Overview Don Libes 25

 list_current_directory

proc list_current_directory {} {
send "dir\r"
# expect commands to read directory

foreach file $list {

if {$isdirectory} {
send "cd $file\r"
list_current_directory
send "cd ..\r"
}
}
}

Expect: Introduction and Overview Don Libes 26

 Feedback

• expect_out contains results of a match

• expect_out(buffer) contains entire match plus things skipped
• expect_out(0,string) contains entire match
• expect_out(1,string) contains submatch 1
• expect_out(2,string) contains submatch 2
• expect_out(3,string) . . .
• and so on

expect -re "a*((ab)*|b*)"

Expect: Introduction and Overview Don Libes 27

 Feedback example

expect {
"ld password:" {
send "$oldpass\r"
} "assword*:" {
send "$newpass\r"
} -re "(.*)\n" {
showerr "$expect_out(1,string)"
} eof {
showerr "passwd died unexpectedly"
}

Expect: Introduction and Overview Don Libes 28

 Lots of Other Features

• Global patterns
• expect_before
• expect_after
• Multiple processes
• Example: testing two programs
• Example: program1 -> program2 -> program1

Expect: Introduction and Overview Don Libes 29

 Tk — An X11 Extension To Tcl

• Tk commands are simple

• Buttons
button $gbut -text "Get File" -command get
button $pbut -text "Put File" -command put
• Bindings
bind $window <Button3> swap-directory
• Lots of other widgets, Motif-style
• Window
• Scrollbar Get File
• Radio button
• Check button Put File
• Canvas
• Etc

Expect: Introduction and Overview Don Libes 30

 Expectk – Script Driven

Expectk
Expect Tk original
Tcl Xlib process

script

• Add scrollbars, buttons, etc. to existing programs

• Or completely cover them up.
• No changes are required to original programs.
• Ergo, no testing of changes is necessary.

Expect: Introduction and Overview Don Libes 31

 Example – tkpasswd – A Tk GUI for passwd

Expect: Introduction and Overview Don Libes 32

 tkpasswd — Performance Metrics

sort
rand

passwd

password
databases

help
dictionaries

Expect: Introduction and Overview Don Libes 33

 How To Avoid Learning Expect

• Autoexpect – Watches you interact and generates an Expect script

generated
Expect
script

interactive processes

Expect: Introduction and Overview Don Libes 34

 Usage

• Similar in feel to the “script” command:

% script
script started, file is typescript
... interact ...
script done, file is typescript
%

% autoexpect
autoexpect started, file is script.exp
... interact ...
autoexpect done, file is script.exp
%

Expect: Introduction and Overview Don Libes 35

 No guarantees

• Autoexpect has to guess about certain things

• Timing
• Changing Behavior
• Occasionally it guesses wrong
• But these spots are easy to fix
• It actually does a very good job
• has some neat heuristics
• even experts use it
• Good news
• very easy to use
• free & well documented
• nothing easier
• Bad news
• You have to stop what you’re doing

Expect: Introduction and Overview Don Libes 36

 Character graphic automation

• xterm
• send: output appears in xterm for user to see
• expect: reads user keystrokes
• xterm itself takes care of everything else: character graphics, select, mouse, scroll
• good for applications that demand an xterm
• tkterm
• Expect and Tk take care of everything
• essentially a terminal widget
• with enhanced expect command
• good for Curses-based applications
• For most applications, tkterm is the way to go
• much more flexible

Expect: Introduction and Overview Don Libes 37

 Background

• Expect works fine in the background

• cron, at, batch
• CGI
• Telnet daemon
• Storing passwords in scripts
• Not storing passwords in scripts
• Storing passwords in scripts anyway

Expect: Introduction and Overview Don Libes 38

 CGI

• http://expect.nist.gov/cgi.tcl

Expect: Introduction and Overview Don Libes 39

 Backend CGI Script

cgi_title "Password Change Acknowledgment"

cgi_import name
cgi_import old
cgi_import new1
cgi_import new2
spawn /bin/passwd $name
expect "Old password:"
send $old
expect "New password:"
send $new1
expect "New password:"
send $new2
puts "Password Changed!"
Expect: Introduction and Overview Don Libes 40
 Passwords in Scripts

• Generally bad
• but so useful
• easy to avoid in Expect

Expect: Introduction and Overview Don Libes 41

 Prompt at start-up

# prompt user for password

set password [getpass]

# go into background
if {[fork]} exit
disconnect

# everything hereafter is in background

sleep ...
spawn telnet ...
expect ...
send ...
Expect: Introduction and Overview Don Libes 42
 Variations

• Example: password unknown to script or changed

while {1} {
send "$password\r"
expect {
"sorry" {
find-human
set passwd [getpass-from-human]
}
"$prompt" break
}
}

Expect: Introduction and Overview Don Libes 43

 Telnet Daemon

• Secure scripts can be done with file permissions

• More secure with physically secure machines
• put in locked room
• turn off all daemons
• Expect script as telnet daemon
• Users telnet to secure machine which then supplies password for them

Expect: Introduction and Overview Don Libes 44

 Widely Used

“Expect has become a necessary tool for system administration. In a short time,
we have used Expect in six areas and have cut out seven hours a week in tedious
and repetitive tasks.”
—Thomas Naughton, Hull Trading Company

“Expect is a lifesaver for a project that I am currently involved with. I have only
been working with Expect for the last couple of days, but it has already shaved
about 6 months off of the completion time of the project.”
—Ron Young, System Computing Services, University of Nevada

“Thanks for making my life easier. This program has really helped me shorten
the cycle time for software Q.A. Expect is like a dream come true for automa-
tion. My productivity has really increased.”
—Brian F. Woodson, 3Com NSD Software Q.A.

“Thanks for Expect. It just made an impossible project possible.”

—Bruce Barnett, GE Corporate Research and Development Center

“I figure we saved about $35K last year (Jan-Dec94) that was directly attribut-
able to Expect. The indirect benefits drive that figure to more like $75K.”
—John Pierce, Chem Dept, UC San Diego

Expect: Introduction and Overview Don Libes 45

 Expect Is Freely Available

• Easy to get
• Cost: Free
• URL: http://expect.nist.gov
• Easy to install
• Portable
• UNIX: GNU-style configure
• Windows: ports from Cygnus and Berkeley
• Mac: sorry
• Well documented
• Numerous published papers
• Comprehensive man pages
• Exploring Expect (O’Reilly), ISBN: 1-56592-090-2
• Commercial Support Available
• Scriptics
• Cygnus Software
• Computerized Processes Unlimited

Expect: Introduction and Overview Don Libes 46