Unix Shell Scripting
Version no
Table of Contents
Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Module 7 Module 8 Module 9 Module 10 Module 11 Module 12 Module 13 Module 14 Module 15 Module 16 Getting started General Purpose Utilities Working with Directories & Files The Shell vi Editor File permissions File Comparison The process Filters Advanced Filters Introduction to shell scripting User inputs and expressions Conditions and loop Some more scripts Communication Utilities System Administration 3 11 21 44 52 70 77 81 91 108 131 147 158 178 187 191
Module 1. Getting started
Overview What is UNIX Features of Unix Evolution of Unix Flavors of Unix Unix architecture Signing into Unix Unix commands
What is UNIX?
What is UNIX? UNiplexed Information and Computing System
Features of Unix
Features Multi-user Multi-tasking Hierarchical directory structure Portability Drawback Lack of GUI Difficult operating system to learn
Worded commands & messages Many UNIX commands have short names
Evolution of Unix
Developed by : Dennis Ritchie and Ken Thompson Where : Bell Telephone Laboratories When : 1969
Flavors of Unix
Solaris (Sun Microsystem) HP-UX (Hewlett Packard UniX) AIX (Advanced Interactive eXecutive ) by IBM Most popular is Linux which is very strong in network and internet features.
Unix Architecture
Applications/ Commands Shell Kernel Hardware
Signing onto Unix
Every user of a UNIX system must log on to the computer using an existing account. Login name is to be entered at login prompt.
login: user1
Password: $
Unix Commands
Unix commands are entered at command prompt ($) $ ls All unix commands must be entered in lowercase. Between command name and its options, there must always be a space. $ ls -l To cancel the entire command before u press Enter, use Del key.
10
Module 2. General Purpose Utilities
Overview banner cal date Who echo passwd bc script
11
banner : display message in poster form
12
$ banner hello $ banner Hello Unix $ banner Hello Unix
cal : calendar
13
$ cal $ cal 7 2008 $ cal 1752
date: Display System date
$ date $ date +%a $ date +%A $ date +%b $ date +%B $ date +%d $ date +%D
14
who : Login details
$ who $ who H $ who am i
15
echo : Display Messages
16
$ echo Welcome To Unix $ echo Welcome To Unix
passwd: change password
17
$ passwd
bc :calculator
$ bc 12+5 17 12*12; 2^3 144 8 <ctrl-d> $
18
script: Record your session
$ script Script started, file is typescript $ _
19
$ exit Script done, file is typescript
Few other commands
20
1. 2. 3. 4. 5.
clear tty uname logname exit
Module 3 . Working with Directories & Files
Overview
21
Unix File Structure & its features Types of Files Rules for filenames Directory Handling Commands pwd, mkdir, cd, rmdir File Handling Commands cat, ls, cp,mv,rm, ln, wc Absolute path & Relative path Setting alias Inode
Unix File Structure
Unix treats everything it knows and understands as a file. Unix File system resembles an upside down tree.
22
/
bin
(root)
lib dev usr tmp etc mnt
user1
user2
bin
Features of Unix file Structure
23
It has a hierarchical file structure Files can grow dynamically Files have access permissions All devices are implemented as files.
Types of Files
Unix files are categorized into : Ordinary Files Directory files Device files
24
Rules for filenames
Filename can consist of:
Alphabets and numerals Period (.), hyphen (-) and underscore (_)
25
Filename can consist of upto 255 characters. Files may or may not have extensions
Sample Tree structure
projects/
26
reports/
graphs/
backups/
r_jan
r_feb
r_mar
b_jan
b_feb
g_jan
g_feb
backup/
backup
pwd command : Present working directory
$ pwd $ pwd
27
mkdir command : Make directory
$ mkdir [option] [directory_name] $ mkdir projects $ mkdir graphs backup $ mkdir p projects/graphs $ mkdir m 700 reports
28
cd command : Change Directory
$ cd [directory_name] $ $ $ $ cd graphs cd projects/reports cd .. cd
29
rmdir command : Remove Directory
$ rmdir [options] [directory_name] $ $ $ $ rmdir graphs rmdir reports graphs backups rmdir backups/backup rmdir p backups/backup
30
cat command: create new file
Creates file with the specified name and can add data into it. $ cat > r_jan This is report of January month. <ctrl+d> $ cat r_jan $ cat file1 file2 > r_jan $ cat file1 file2 >> r_jan
31
The cat command
(Contd )
32
Displays the contents of the file with numbering $ cat n [file_name] Display $ at end of each line $ cat e [file_name]
ls command : listing files & directories
$ ls [option] [directory/file] Options to ls -a -l -i -r -s -t -F -R -d displays hidden files also long listing if files showing 7 attributes of a file displays inode number Reverse order while sorting Print size of each file, in blocks Sort by modification time Marks executables with * and directories with / Recursive listing of all files in sub-directories List directory entries
33
The ls command (Contd )
Examples: $ ls $ ls d* $ ls [dk]* $ ls d? $ $ $ $ $ ls ls ls ls ls -l -F -i -r t or -a
34
ls -rt
Absolute path and Relative path
Absolute path Relative path
35
File & directory related commands
/ home/ training/ projects/ reports/ graphs/ g_jan backups/ g_feb
(Contd..)
36
backup/ backup r_jan r_feb r_mar b_jan b_feb
The cp command
$ cp [option] [source] [destination]
37
Options to cp: -i : -r : Examples:
Prompt before overwrite Recursive copying
$ cp r_jan reports $ cp -i r_jan reports
mv command: Renaming & moving files
$ mv [option] [source] [destination] Options to mv: -i :
38
Prompt before overwrite
Examples: $ mv b_jan newfile $ mv file1 file2 newdir $ mv olddir newdir $ mv -i b_jan newfile $ mv b_jan newdir $ mv b_jan newdir/
The rm command
$ rm [option] [file/directory] Options to rm: -i : Confirm before removing -r : Recursive deletion -f : Forceful deletion Examples: $ rm r_jan $ rm -i r_feb $ rm -f r_mar $ rm r backups
39
Setting alias for commands
40
$ alias $ alias rm='rm -i' $ alias cls=clear
$ unalias cls $ unalias -a
Count words using wc
$ wc [option] [file_name]
Options to wc: -l : Display no. of lines -w : Display no. of words -c : Display no. of characters
41
Examples: $ wc new_link 3 12 59 new_link $ wc -l new_link 3 new_link
File links
Soft link or symbolic link or symlink
$ ln -s [source_path] [destination_path]
42
Hard link
$ ln [source_path] [destination_path]
Inode
Inode contains
File type (executable, block special etc) Permissions (read, write etc) Owner Group File Size File access, change and modification time File deletion time Number of links (soft/hard) Extended attribute such as append only or no one can delete file including root user (immutability)
43
Module 4. Shell
Overview What is Shell Unix shells Redirection System Variables .profile file
44
What is shell?
Hardware Kernel Different shells (Various Unix Commands)
45
Unix shells
46
Bourne shell (sh) C shell (csh) Korn Shell (ksh)
Redirection
1. Standard input (<) $ wc < emp.lst 1. Standard output (>) $ ls > listing $ cat >> file_name 1. Standard error (2>) $ cat emplist 2>errorlogfile
47
Connecting Commands with Pipes
48
$ who | wc l $ who > emp.lst $ wc l emp.lst
$ ls | wc l >fcount
tee command
$ who | tee users.list
49
$ who | tee users.list |wc l $ who | tee /dev/tty | wc -l
Unix System variables
System variables PATH Purpose
50
The set of directories the the shell will search in the order given, to find the command HOME Set of users home directories MAIL The directory in which electronic mail is sent to you is places PS1 The primary prompt string PS2 The secondary prompt string SHELL It sets the path name of the shell interpreter TERM Identifies the kind of terminal you are using LOGNAME Displays the username
.profile
51
This file is present in home your directory. It contains the script to be executed during login time.
Module 5 . The vi editor
Overview
Introduction to vi editor Moving between 3 modes of vi editor Input Mode commands Navigation Moving between the lines and scrolling pages Ex mode commands Delete text Replacing and changing text Yanking, pasting and joining line Pattern search and replace Customizing vi Abbreviating text Multiple file editing in vi
52
Introduction to vi
vi( short for visual editor) is an editor available with all versions of unix. It allows user to view and edit the entire document at same time. Written by Bill Joy Its case-sensitive
53
How to Invoke vi session
54
$ vi newfile $ vi
Three modes of vi editor
i, I, o, O, r, R, s, S, a, A
55
Input Mode <Esc>
Command Mode
<Esc> :
<Enter>
ex Mode
Input Mode Commads
Input text i Inserts text to left of cursor I Inserts text at the beginning of the line Append text a Appends text to right of cursor A Appends text at the end of the line Opening a new line o Opens line below the cursor O Opens line above the cursor
56
Input Commands (contd.)
Replacing text r ch R s S Replaces single character under cursor with ch Replaces text from cursor to right Replaces single character under cursor with any number of characters Replaces entire line
57
Navigation
k h l
58
Moving between the lines and scrolling pages
Moving between the lines G Goes to end of the file nG Goes to line number n Scrolling page ctrl+f Scrolls page forward ctrl+b Scrolls page backward ctrl+d Scrolls half page forward ctrl+u Scrolls half page backward
59
Ex mode commands: Save file and quit
Saving & Quitting :w Saves the files & remains in the editing mode :wq Saves & quits from editing mode :x Saves & quits from editing mode :q! Quitting the editing mode without saving any changes Writes selected lines into specified file_name :w <file_name> Save file as file_name :n1,n2w <file_name> Writes lines n1 to n2 to specified file_name :.w <file_name> Writes current line into specified file :$w <file_name> Writes last line into specified file name
60
Commands for Deleting text
x dw dd db d0 d$ nx ndw ndd Deletes a single character Deletes a word ( or part of the word) Deletes the entire line Deletes the word to previous start of the word Deletes current line from cursor to beginning of line Deletes current line from cursor till end of line Deletes n characters Deletes n words Deletes n lines
61
Yanking, pasting and joining line
Yanking/Copying text yw Yanks word from cursor position yy Yanks current line y0 Yanks current line from cursor to beginning of line y$ Yanks current line from cursor till end of line nyy Yank the specified number of lines Paste p Join J
62
Pastes text
Joins following lines with current one
Pattern search
Command /pattern ?pattern n N Purpose Searches forward for pattern Searches backward for pattern Repeats the last search command Repeats search in opposite direction
63
Building Patterns for searching
Character * [] ^ $ \< \> Examples: ^finance finance$ [a-d]ing team\> Wing* [^p]art Meaning Zero or more characters Set or range of characters Matches pattern towards beginning of line Matches pattern towards end of line Forces match to occur only at beginning of word Forces match to occur at end of word
64
Search and Replace
Command Purpose :s/str1/str2 Replaces first occurrence of str1 with str2 :s/str1/str2/g Replaces all occurrences of str1 with str2 :m,n s/str1/str2 /g Replaces all occurrence of str1 with str2 from lines m to n :.,$ s/str1/str2/g Replaces all occurrence of str1 with str2 from current line to end of file Example: :s/director/member/g
65
Customizing vi
:set number/set nu :set nonumber/set nonu :set wrapmargin=20 :set ignorecase :set ai :set showmode :set autowrite Sets display of line numbers ON Set no number Set wrap margin equal to 20 Ignorecase while searching Set auto indent Shows the working mode Automatically writes when moves to another page
66
Abbreviating text
Command :abbr <abbr> <longform> :abbr :una <abbr> Purpose an abbreviation is defined for longform lists currently defined abbreviation Unabbreviates the abbreviation
67
Example: :abbr pspl pragati software pvt. ltd.
.exrc file
$ vi .exrc set nu set ignorecase set showmode set wrapmargin=60
68
Multiple file editing in vi
69
Command
vi file1 file2 :n :n! :rew :rew! :args :f
Purpose
Loads both files in vi for editing. Permits editing of next file Permits editing of next file without saving current file Permits editing of first file in buffer Permits editing of first file without saving current file Displays names of all files in buffer Displays name of current file
Example: $ vi file1 file2 file3 $ vi *.c
Module 6 . File permissions
Overview Permission for file and directories The chmod command Octal notation umask (default file and directory permission)
70
Permissions for files and directories
Three types of permissions 1. Read For files : cat, vi For directories : ls 1. Write For files : cat > file_name, vi For directories : mkdir, rmdir, mc 1. Execute For files : ./filename For directories : cd
71
The chmod command
$chmod [category] [operation] [attributes] [file/directory]
Category u: g: o: a: Operation +: - : =: Attributes r: w: x: read write execute assign revoke absolute user group others all
72
Example of chmod
Giving user execution permission $ chmod u+x report.txt $ ls l report.txt -rwxrw-r-- 1 user1 group1 320 Jun 26 23:26 report.txt For others remove read permissions $ chmod o-r report.txt $ ls -l report.txt -rwxrw---- 1 user1 group1 320 Jun 26 23:26 report.txt Give absolute permissions to group as read and execute $ chmod g=rx report.txt $ ls -l report.txt -rwxr-x--- 1 user1 group1 320 Jun 26 23:26 report.txt
73
Octal notation
$ chmod [octal_notation] [file/directory] Permissions 4 2 1 OctalNotation Read Write Execute
74
Example of chmod using octal notations
Original permissions $ ls-l report.txt -rwxr-x--- 1 user1 group1 320 Jun 26 23:26 report.txt Assigning permissions using octal notations $ chmod 664 report.txt After assigning permissions $ ls -l report.txt -rw-rw-r-- 1 user1 group1 320 Jun 26 23:26 report.txt
75
umask (default file and directory permissions)
Default umask $ umask 0002 Default file permissions $ touch report.txt $ ls -l report.txt -rw-rw-r-- 1 user1 group1 320 Jun 26 23:41 report.txt Change umask $ umask 066 File permissions with umask changed $ touch newReport.txt $ ls -l newReport.txt -rw------- 1 user1 group1 320 Jun 26 23:42 newReport.txt
76
Module 7 . File comparision
Overview Comparing files using cmp command Comparing files using comm command Comparing files using diff command
77
Comparing files using cmp command
$ cmp [file1] [file2] $ cat file1 CDROM CPU FLOPPY DISK HARD DISK KEYBOARD MONITOR PRINTER
$ cmp file1 file2
78
$ cat file2 CDROM CPU HARD DISK KEYBOARD MONITOR MOUSE PRINTER
file1 file2 differ: byte 13, line 3
comm command : finding what is common
$ comm [file1] [file2] Output for comm command is $ comm file1 file2 CDROM CPU FLOPPY DISK HARD DISK KEYBOARD MONITOR MOUSE PRINTER
79
diff command: convert one file into another
$ diff [file1] [file2] $ diff file1 file2 3d2 < FLOPPY DISK 6a6 > MOUSE
80
Module 8 . The process
Overview Process status ps Mechanism of process creation Executing jobs in background Job control kill command Scheduling jobs for later execution
81
ps : Process status
$ ps [options] Options to ps: -a : Display all user processes -f : Display Process ancestry -u : Display process of a user -e : Display system processes Example: $ps PID 1032 1074 $ps a
82
TTY TIME pts/1 00:00:00 pts/1 00:00:00
CMD ksh ps
Process status
$ ps -u user1 $ ps -f UID PID user1 1032 user1 1275
ps
(Contd )
83
PPID C STIME TTY TIME 1031 0 09:10:02 pts/1 00:00 1032 0 09:10:02 pts/1 00:00
CMD bash ps f
Mechanism of Process Creation
84
Three phases are involved in creation of process : fork exec wait
Executing jobs in background
In order to execute a process in background, just terminate the command line with & $ sort o emp.lst emp.lst & 550 # jobs PID
85
nohup : Log Out Safely
86
nohup ( no hangup) command, when prefixed to a command, permits execution of the process even after user has logged out.
$ nohup sort emp.last &
Job Control
Commands used for job control: bg, fg, kill, jobs $cat > test this is example of suspended process [1]+ Stopped $ jobs [1]+ Stopped $ bg %cat cat >test & $ fg %cat continued cat >test
87
cat >test
kill command: Terminate a process
$ kill 1346 $ kill 1346 121 400 $ kill -9 1346
$ kill $! # kills last background job
88
Scheduling jobs for later Execution
at command
$ at 15:15 echo Hello > /dev/tty <ctrl+d> Options: l ( list ) : -r (remove ) : View list of submitted jobs Remove submitted job
89
Scheduling jobs using cron
cron lets you schedule jobs so that they can run repeatedly. $ crontab -l 00-10 17 * * * echo Hello > /dev/tty day of week (0 - 6) (Sunday=0) month (1 - 12) day of month (1 - 31) hour (0 - 23) min (0 - 59)
90
Module 9 . Filters
Overview pr head tail cut paste sort uniq tr grep egrep fgrep
91
Data file
$cat -n emp.lst 10 | A.K.Sharma | Director | Production | 12/Mar/1950 | 70000 11 | Sumit Singh | D.G.M | Marketing | 19/Apr/1943 | 60000 12 | Barun Sen | Director | Personnel | 11/May/1947 | 78000 23 | Bipin Das | Secretary | Personnel | 11/Jul/1947 | 40000 50 | N.k.Gupta | Chairman | Admin | 30/Aug/1956 | 64000 43 | Chanchal | Director | Sales | 03/Sep/1938 | 67000
92
pr : Paginating files
$ pr [option] [file_name] Options to pr: -d Double spaces input. -n displays line numbers. -o n offset lines by n spaces -h Displays header as specified instead of file name. Example: $pr emp.lst $pr -dn emp.lst $pr h "Employee Details" emp.lst
93
head: Displaying the beginning of a file
$ head [option] [file_name] Options to head: -n Displays specified numbers of lines
94
Example: $ head emp.lst $ head -n 6 emp.lst | nl
tail : Displaying the end of the file
$ tail [option] [file_name] Options to tail: -n Displays specified numbers of lines Example: $ tail emp.lst $ tail -6 emp.lst $tail +10 emp.lst
95
cut : Slitting a file vertically
$ cut [option] [file_name] Options to cut: -c : cutting columns -f : cutting fields -d : specify delimeter Example: $ cut -c 1-4 emp.lst $ cut -d "|" -f1 emp.lst $ cut -d "|" -f2,4 emp.lst
96
paste : Pasting files
$ paste [option] [file_name] Options :
-d specify delimeter for pasting files
97
Example: $ paste empno empname $ paste -d :" empno empname
sort : Ordering a file
$ sort [option] [file_name] Options: -n sorts numerically -r Reverses sort order -c Check whether file is sorted +k Starts sorting after skipping kth field -k Stops sorting after kth field -o File Write result to File instead of standard output -t Specify field separator
98
sort : Ordering a file
Example:
(Contd..)
99
$ sort emp.lst $ sort -t "|" -k2 emp.lst $ sort -t "|" -k2 emp1.lst -o emp1.lst $sort -t"|" -k 3,3 -k 2,2 emp.lst
uniq : Locate repeated and no repeated lines 100
$ uniq [file_name] Options: -d : selects only one copy of the repeated lines -c : displays the frequency of occurrence of all lines -u : selects only non-repeated lines Examples: $ cut -d"|" -f 4 emp1.lst > departments $ sort departments | uniq $ sort departments | uniq -d $ sort departments | uniq -c
tr : translating characters
$ tr [options] [expression1] [expression2] [standard_input] $tr '|/' '~~' < emp.lst Changing case for text $tr [a-z] [A-Z] < emp.lst Deleting characters $tr -d '|/' < emp.lst
101
grep : Searching for pattern
$ grep [options] [file_name(s)] Simple search $grep sales emp.lst $grep d.g.m. emp.lst $grep 'jai sharma' emp.lst Ignoring case $grep -i "SALES" emp.lst
102
grep : Searching for pattern
Deleting specified pattern lines $grep -v "sales" emp.lst Displaying line numbers $grep -n "sales" emp.lst Counting lines containing pattern $grep -c sales emp.lst Displaying filenames $grep -l sales *
(Contd )
103
Regular Expressions
Significance Matches zero or more occurrence of previous character . Matches a single character [pqr] Matches a single character p, q or r [a-r] Matches a single character within range a r [^pqr] Matches a single character which is not p, q or r ^pattern Matches pattern at beginning of line pattern$ Matches pattern at end of line \<pattern Matches pattern at beginning of word pattern\> Matches pattern at end of word Symbols *
104
grep : Searching for pattern
(Contd )
105
Searches for a pattern only at the beginning of a word and not anywhere on the line $grep "\<man" emp.lst Searches for a pattern only at the end of a word and not anywhere on the line $grep "man\>" emp.lst Using metacharacters $grep sa[kx]s*ena emp.lst $grep ag[agr][ra]r*wal emp.lst
egrep : Extended grep
$grep -e sengupta -e dasgupta -e gupta emp.lst $egrep "sengupta|dasgupta|gupta" emp.lst $egrep "(sen|das|)gupta" emp.lst
106
fgrep : Fixed string grep
Taking patterns from a File $cat -n pattern.lst 1 sales 2 gupta $fgrep -f pattern.lst emp.lst
107
Module 10 : Advanced Filters
Overview
sed and awk
108
sed- stream editor Introduction to awk Formatting output with printf Logical and relational operators Number processing The -f option The BEGIN and END section Positional parameters and shell variables Built-in variables Making awk interactive using 'getline' statements Arrays Functions The if statement Looping constructs
sed: Stream EDitor
sed is a multi-purpose tool which combines work of several filters. Designed by Lee McMohan. It is used for performing non-interactive applications
109
sed instruction
$ sed [options] address action [file_name]
110
Line addressing Print 3rd line $head -n 3 emp.lst | tail -n 1 $sed '3p' emp.lst Print only 3rd line $sed -n '3p' emp.lst Print only last line $sed -n '$p' emp.lst
111
Using multiple instructions ( -e ) Print 3rd and 6th line $sed -n -e'3p' -e'6p' emp.lst Print 3rd to 6th line $sed -n -e '3,6p' emp.lst
Context addressing
112
$ sed -n '/gupta/p' emp.lst $ sed -n -e'/gupta/p' -e'/sharma/p' emp.lst $ sed -n -e'/gupta/,/sharma/p' emp.lst $ sed -n '/ag[agr][ar]r*wal/p' emp.lst
Writing selected lines to a file
113
$sed -n '/director/w dlist' emp.lst $sed -n '/dirctor/w dlist > /manager/w mlist > /executive/w elist' emp.lst
Text editing
114
$sed '$i\ 1000|jitesh sharma' emp.lst $sed '$a\ 1000|jitesh sharma' emp.lst $sed '/director/d' emp.lst
Substitution
[address]s/string1/string2/flag $ sed 's/ | / : /' emp.lst $ sed 's/ | / : /g' emp.lst $ sed '1,5s/ | / : /g' emp.lst
115
Introduction to awk
116
Authors : Aho, Weinberger , Kernighnan Use : Pattern scanning and processing language Unlike other filters, awk operates at field level
awk
$ awk <options> address {action} <file(s)> $ awk '/director/ { print }' emp.lst $ awk F"|" '/sales/ {print $2,$3,$4,$6}' emp.lst $ awk F | NR==3, NR==6 {print NR, $2, $3, $6 } emp.lst
117
Formatting output with printf
$ awk -F "|" /director/ { printf("%3d %-20s %-12s %d\n", NR, $2, $3, $6) }' emp.lst
118
The logical and relational operators
Logical And Logical Or $ && ||
119
awk -F "|" '$3=="director" || $3=="chairman" { print }' emp.lst
$ awk -F "|" '$6 > 7500 { print }' emp.lst
Number processing
$ awk -F "|" '$3=="director" { printf %d %d \n", $6, $6*0.15 }' emp.lst
120
$ awk -F "|" '$3=="director" && $6>6700{ kount++ printf "%d %s \n", kount, $2 }' emp.lst
The f option
$ cat empawk.awk $3=="director"||$6>7500 {printf"%-20s %-12s %d \n", $2,$3,$6 } $ awk -F"|" -f empawk.awk emp.lst
121
The BEGIN and the END sections
$ cat -n empawk2.awk 1 BEGIN { 2 printf "\n\t\tEmployee abstract\n\n" 3 } 4 $6 > 7500 { 5 kount++; tot += $6 6 printf "%3d %-20s %-12s %d\n", kount, $2, $3, $6 7 } 8 END { 9 printf "\n\t\tThe average basic pay is %6d\n", tot/kount 10 } $ awk F| f empawk2.awk emp.lst
122
Positional parameters and shell variables
$ $6 > 7500 $ awk F| f empawk2.awk mpay=7800 emp.lst (change empawk2.awk line number 4 as $5>mpay )
123
Built-in variables
$ awk 'BEGIN { FS="|" ; OFS="~"} > $5~/5[25]$/ {print $1,$2,$3,$5}' emp.lst $ awk 'BEGIN {FS="|" } > NF!=6 { > print "Record No ", NR, " has", NF, " fields"}' emp.lst $ awk '$6>6000 > { print FILENAME, $0 }' emp.lst
124
Making awk interactive using getline statement
$ cat -n empawk3.awk 1 BEGIN { 2 printf "\nEnter the cut-off basic pay : " 3 getline var < "/dev/tty" 4 printf "\n\t\tEmployee abstract\n\n" 5 } 6 $6 > var { 7 printf( "%3d %-20s %-12s %d \n", ++kount, $2, $3, $6) 8 } $ awk F| f empawk3.awk emp.lst
125
Arrays
$ cat -n empawk4.awk 1 BEGIN { FS = "|" 2 printf ("\n%46s\n", "Basic Da Hra Gross" ) 3 } 4 /sales|marketing/ { 5 da = 0.25 * $6; hra = 0.50 * $6; 6 gp = $6+hra+da; 7 tot[1] +=$6 ; tot[2] += da; 8 tot[3] +=hra; tot[4] += gp 9 kount++ 10 } 11 END { 12 printf "\n\t Average %5d %5d %5d %5d\n", \ 13 tot[1]/kount, tot[2]/kount, tot[3]/kount, tot[4]/kount 14 }
126
Functions
$ awk '{ print length()}' emp.lst $ awk 'BEGIN{ print sqrt(144)} $ awk 'BEGIN{ print int(100.987)}' $ awk 'BEGIN{ print system("date")}' $ awk 'BEGIN{ print system("clear")} $ awk 'BEGIN{ print index("pragati software","gati")}
127
The if statement
128
if ($6 < 6000) { hra = 0.50 * $6 da = 0.25 * $6 } else { hra=0.40*$6 da=1000 }
Looping with for
$ awk -F"|" '{ > kount[$3]++} > END{ > for(design in kount) > print(kount[design],design) > }' emp.lst
129
Looping with while
cat -n newline.awk BEGIN{ FS="|" } { newline("-",50); printf("%d %s", $1, $2); } function newline(ch, num){ printf "\n"; i=1 while(i<num){ printf("%c",ch); i++; } printf("\n"); } $ awk -f newline.awk emp.lst $
130
Module 11 . Introduction to shell scripting
Overview Why shell scripting? When not to use shell scripting? Shell as an interpreter Writing your first shell script (first.sh) Different ways to execute a shell script The sha bang statement Some basics of scritping The export statement Comments in shell script
131
Why shell scripting?
Automating commonly used commands Performing system administration and trouble shooting Creating simple application
132
When not to use shell scripts?
1. Resource intensive task when speed is a factor 2. Heavy math operations 3. Portability 4. Structured programming 5. Subcomponents with interlocking dependencies. 6. Extensive file operations 7. Multi dimensional arrays 8. Data structures like linked lists or trees 9. Generate or manipulate graphics or GUIs. 10. Direct access to system hardware. 11. Socket programming 12. Libraries to interface with legacy code 13. Proprietary or closed source software
133
Shell as an interpreter
Compilers i.e., C/C++ Windows Prg1.c Compilation Prg1.obj Linking Prg1.exe Prg1 Prg1.out Renaming Prg1 Renaming Prg1 Prg1.o Unix Prg1.c Interpretation Prg1.sh Interpreter i.e., Shell
134
Writing your first shell script (first.sh)
$ echo "Hello World of UNIX shell scripts.
135
Different ways to execute a shell script
1. 2. 3. 4. 5. shell_name scriptName scriptName ./scriptName /FQPN/scriptName . ./FQPN/scriptName
136
* FQPN Full Qualified Path Name, e.g., /home/redhat/scripts/
shell_name scriptName
Run this script by using any of the following commands $ bash scriptName $ ksh scriptName $ csh scriptName
137
scriptName
To run the script like a command 1) Set the path in PATH variable 2) Set execute permission for the script Run this script by using the following command $ scriptName
138
./scriptName
This method can be used for that particular directory Requires execution permission Run this script by using the following command $ ./scriptName scripts
139
outbox first.sh
inbox
first.sh
first.sh
/FQPN/scriptName
This method of running the script requires execution permission It bypasses the PATH Run this script by using the following command $ /FQPN/scriptName $ /home/scripts/first.sh / etc home scripts first.sh
140
. ./FQPN/scriptName
This method of running the script does not requires execution permission This method bypasses the PATH. It honors PATH user specifies. Run this script by using the following command $ . ./FQPN/scriptName
141
The sha bang statement
Specifies which kind of interpreter should get followed #!/bin/bash #!/bin/ksh #!/bin/csh #!/bin/more #!/bin/rm
142
magicScript.sh
1. #!/bin/rm 2. echo executing this script
143
The export statement
1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. echo $$ A=10 echo $A ksh echo $$ echo $A exit export A ksh echo $A A=90 echo $A exit echo $A export n A
144
first.sh
1. 2. 3. 4. 5. 6.
(contd )
145
#!/bin/bash echo "Hello world of UNIX Shell Script" echo "process id of your shell is: $$ echo value of A is $A A=500 echo value of A is $A
Comments in shell scripts
1. 2. 3. 4. 5. 6. 7. 8. 9. 10. #!/bin/bash ################################ # Pragati Software Private Limited # Purpose : This is first shell script. ################################ echo "Hello world of UNIX Shell Script" echo "process id of your shell is: $$ echo value of A is $A A=500 echo value of A is $A
146
Module 12 . User inputs and expressions
Overview Using read Command line arguments Special parameters used by the shell Using set Using shift exit and exit status of the commands Computations : expr Various quotes on shell prompt Arithmetic operations using 'let'
147
Using read
$ cat -n search.sh echo -e "Enter filename : \c" read filename echo -e "Enter pattern :\c" read pattern grep $pattern $filename
148
Command line arguments
$ cat -n searchPattern.sh echo "Program : $0" echo "Number of arguments specified is $#" echo "The arguments are $*" grep $1 $2
149
Creating USAGE messages
$ cat -n usageDemo.sh if [ $# -ne 2 ] ; then echo "USAGE : addnum.sh <num1> <num2>" else echo "Addition is : `expr $1 + $2`" fi
150
Special parameters used by the shell
Shell parameters Significance $1,$2, $# $0 $* $@ $? $$ $! Positional parameters representing command line argument Number of arguments specified in command line Name of executed command Complete set of positional parameters as a single string Each quoted string treated as separate argument Exit status of last command PID of current shell PID of the last background job
151
Using set
set assigns its positional parameters to the positional parameters $ set 123 456 789 $ echo \$1 is $1 ,\$2 is $2 ,\$3 is $3
152
Using shift
$ cat -n shiftDemo.sh 1 #!/bin/ksh 2 NO_ARGS=$# 3 echo .Number of arguments passed $NO_ARGS. 4 echo "Argument 1 is $1" 5 echo "Argument 2 is $2" 6 echo "Argument 3 is $3" 7 echo "Argument 4 is $4" 8 echo "Argument 5 is $5" 9 echo "Argument 6 is $6" 10 echo "Argument 7 is $7" 11 echo "Argument 8 is $8" 12 echo "Argument 9 is $9" 13 shift 2 14 echo "Argument 10 is $8" 15 echo "Argument 10 is $9"
153
exit and exit status of command
exit echo $?
154
Computations : expr
$ cat -n compute.sh #!/bin/ksh A=500 B=20 echo "Two values are $A and $B" ADD=`expr $A + $B` #ADD=$((A+B)) echo "Addition is $ADD" SUB=`expr $A - $B` echo "Subtraction is $SUB" MULT=`expr $A \* $B` echo "Multiplication is $MULT" DIV=`expr $A / $B` echo "Addition is $DIV"
155
Various quotes on shell prompt
"" (double quotes) '' (single quotes) `` (grave accent, back or reverse quotes)
156
Arithmetic operations using let
157
$ let sum=A+B $ echo $sum $ let mult=$A*$B $ echo $mult
Module 13 . Conditions and loops
Overview Logical operators ( && and || ) The if condition Using 'test' and '[ ]' to evaluate expression String comparison operator File comparison operator The case statement The while loop The until loop The for loop The break statement The continue statement
158
Logical operators ( && and || )
$ grep director emp.lst && echo Pattern found
159
$ grep managet emp.lst || echo Pattern not found
The if condition
1. if [ condition is true ] ; then statements fi 1. if [ condition is true ] ; then statements else statements fi 1. if [ condition is true ] ; then statements elif [ condition is true ] ; then statements else statements fi
160
ifSearch.sh
$ cat -n ifSerach.sh 1 #!/bin/ksh 2 echo -e "Enter filename : \c" ; read filename 3 echo -e "Enter pattern :\c" ; read pattern 4 grep $pattern $filename 5 GREP_STATUS=$? 6 if [ $GREP_STATUS -eq 1 ] ; then 7 echo "Pattern not found" 8 fi 9 if [ $GREP_STATUS -eq 2 ] ; then 10 echo "File not found" 11 fi
161
Using test and [ ] to evaluate expressions
$ x=5;y=7;z=7.2 $ test $x eq $y ; echo $? $ test $x lt $y ; echo $? $ test $z gt $y ; echo $? $ test $z eq $y ; echo $? Shorthand for test $ [ $z eq $y ] ; echo $?
162
String comparison operators
Operators s1=s2 s1!=s2 -n str -z str str s1= =s2 Meaning String s1=s2 String s1 is not equal to s2 String str is not a null String String str is a null String String str is a assigned and null String String s1= =s2(korn and bash only)
163
File comparison operators
Operators -f file -r file -w file -e file -d file -e file -L file Meaning File exist and is a regular file File exist and is readable File exist and is writable File exist and is executable File exist and is directory File exist (korn and bash only) File exist and is a symbolic link
164
fileSearch.sh
$ cat -n fileSearch.sh 1 echo -e "Enter file name\c" ; read filename 2 if [ -e $filename ] ; then 3 echo "Enter pattern" ; read pattern 4 grep $pattern $filename 5 GREP_STATUS=$? 6 if [ $GREP_STATUS -eq 1 ] ; then 7 echo "Pattern not found." 8 fi 9 else 10 echo "File not found." 11 fi
165
elifTest.sh
$ cat -n elifTest.sh 1 A=500 2 B=20 3 echo "Two values are $A and $B" 4 echo -e "Enter your choice \n 1)Addition\n2)Subtraction\n3)Multiplication \n4)Division\n" 5 read CH 6 if [ $CH -eq 1 ] ; then 7 echo "Addition is `expr $A + $B`" 8 elif [ $CH -eq 2 ] ; then 9 echo "Subtraction is `expr $A - $B`" 10 elif [ $CH -eq 3 ] ; then 11 echo "Multiplication is `expr $A \* $B`" 12 elif [ $CH -eq 4 ] ; then 13 echo "Division is `expr $A / $B`" 14 fi
166
The case statement
case condition in 1)statements ;; ------*)statements ;; esac
167
case test
$ cat -n caseTest.sh 1 A=500 2 B=20 3 echo "Two values are $A and $B" 4 echo -e "Enter your choice \n 1)Addition\n2)Subtraction\n3)Multiplication \n4)Division\n" 5 read CH 6 case "$CH" in 7 1) echo "Addition is `expr $A + $B`" ;; 8 2) echo "Subtraction is `expr $A - $B`" ;; 9 3) echo "Multiplication is `expr $A \* $B`" ;; 10 4) echo "Division is `expr $A / $B`" ;; 11 *) echo "Invalid option" 12 esac
168
Matching multiple patterns
$ cat -n multimatch.sh 1 echo "Do you wish to continue [y/n]" 2 read ch 3 case "$ch" in 4 y|Y) 5 echo " $ch is selected" 6 ;; 7 n|N) 8 echo " $ch is selected" 9 ;; 10 esac
169
The while loop
Syntax :while condition is true do commands done
170
whileDemo.sh
1 #!/bin/ksh 2 PATTERN_NOT_FOUND=10 3 FILE_NOT_FOUND=20 4 ch='y' 5 while [ $ch = 'y' -o $ch = 'Y' ] 6 do 7 echo -e "Enter filename : \c" ; read filename 8 echo -e "Enter pattern :\c" ; read pattern 9 grep $pattern $filename 2>/dev/null 10 GREP_STATUS=$? 11 if [ $GREP_STATUS -eq 1 ] ; then 12 echo "Pattern not found...." 13 fi 14 if [ $GREP_STATUS -eq 2 ]; then 15 echo "File not found..." 16 fi 17 echo "Do you want to continue [y/n]?" ; read ch 18 done
171
The until loop
Syntax :until condition is true do commands done
172
untilDemo.sh
$ cat -n untilDemo.sh 1 #!/bin/bash 2 until [ $var == end ] 3 do 4 echo "Input variable #1 " 5 echo "(end to exit)" 6 read var1 7 echo "variable #1 = $var1" 8 done
173
The for loop
Syntax : for variable in list do commands done
174
forDemo.sh
1. for planet in Mercury Mars Saturn do echo $planet done 1. PLANETS=Mercury Mars Saturn for planet in $PLANETS do echo $planet done 1. for((i=0;i<5;i++)) do echo $i done
175
breakDemo.sh
$ cat -n breakDemo.sh 1 LIMIT=10 2 a=0 3 while [ "$a" -le "$LIMIT" ] 4 do 5 a=$((a+1)) 6 if [ "$a" -gt 5 ];then 7 break # Skip entire rest of loop. 8 fi 9 echo -n "$a " 10 done
176
continueDemo.sh
$ cat -n continueDemo.sh 1 LIMIT=20 # Upper limit 2 echo "Printing even numbers from 1 to 20 " 3 a=0 4 while [ $a -le "$LIMIT" ] 5 do 6 let a=a+1 7 REM=`expr $a % 2` 8 if [ $REM -ne 0 ] 9 then 10 continue # Skip rest of this particular loop iteration. 11 fi 12 echo "$a" 13 done
177
Module 14 . Some more scripts
Overview Block redirection Block commenting Arrays Functions
178
Block redirection (output to file)
$ cat -n blockRedirectionDemo.sh 1 #!/bin/ksh 2 i=1 3 while [ $i -lt 10 ] 4 do 5 echo $i 6 let i=i+1 7 done>outfile.sh 8 if [ -f outfile.sh ] 9 then 10 echo "File exit" 11 else 12 echo "File does not exits" 13 fi
179
Block redirection (input from file)
$ cat -n readFile.sh 1 while read line 2 do 3 echo $line 4 done<emp.lst
180
Block commenting
$ cat -n blockComment.sh 1 echo "Block comment" 2 <<BLOCKCOMMENT 3 Hi Hello 4 this I can not see 5 BLOCKCOMMENT 6 echo "End of Comment"
181
Arrays
$ cat -n arrayDemo.sh 1 #!/bin/bash 2 arr[0]=zero 3 arr[1]=one 4 arr[2]=two 5 arr[3]=three 6 arr[4]=four 7 echo ${arr[0]} 8 echo ${arr[1]} 9 echo ${arr[2]} 10 echo ${arr[3]} 11 echo ${arr[4]}
182
Declare variable as array
$ cat -n declare_array.sh 1 #!/bin/bash 2 declare -a arr 3 for((i=0;i<10;i++)) 4 do 5 arr[$i]=$i 6 done 7 for((i=0;i<10;i++)) 8 do 9 echo ${arr[$i]} 10 done
183
Functions
function-name ( ) { command1 command2 ..... ... commandN }
184
Simple function
$ cat -n calc.sh 1 add(){ 2 echo "Enter num1:" 3 read num1 4 echo "Enter num2:" 5 read num2 6 echo "Addtion is `expr $num1 + $num2`" 7 } 8 add
185
Passing parameters to the function
$ cat -n parameterPassing.sh 1 add(){ 2 num1=$1 3 num2=$2 4 5 echo "Addition is `expr $num1 + $num2`" 6 } 7 8 add 10 30
186
Module 15. Communication utilities
Overview
The write and wall command Controlling messages using mesg Sending mails
187
The write and wall command
$ wall Hi all ctrl+d $ wall < file $ write redhat Hi redhat ctrl+d
188
Controlling messages using mesg
$ tty /dev/tty1 $ mesg < /dev/tty2 is y $ mesg n < /dev/tty2 $ mesg < /dev/tty2 is n $ mesg is y
189
Sending mails
$ mail training@pragatisoftware.com Subject: Hi This is just a short note to say hello. I don't have anything else right now. . Cc: ctrl+d $ mail
190
Module 16. System Administration
Overview
root : super users login Administrator privileges Starting up and shutting down the system Disk management find command Backups File compression User administration File system administraton
191
root : Super user s login
192
login: root password: # Prompt of root is #
su : Acquiring super user status
193
$ su Password:*****<enter> # pwd /home/local #prompt changes, but directory doesnt
Administrator s privileges
194
# passwd # date
Starting Up the System
1. Kernel is Loaded 2. Kernel then starts spawning further processes, most important is init ( PID =1). 3. init spawns further processes. init becomes parent of all shells. 4. Unix system can be set up in number of modes( Run-levels) that are controlled by init. Single-user mode Multi-user Mode
195
Shutting Down the system
196
# shutdown -g2 # shutdown -g0 # shutdown -g0 i6
Managing Disk Space
Disk free space # df
197
Disk Usage # du
find : Locating Files
find path_list selection_criteria action
198
# find / -name newfile.sh print # find . mtime -2 -print
Expressions used by find
Selection Criteria -name flname -user uname -type f -type d -group gname -atime +x -mtime +x -newer flname Selects file flname Selects file if owned by uname Selects file if it is an ordinary file Selects file if it is a directory Selects file if owned by group gname Selects file if access time is > x days Selects file if modification time is > x days Selects file if modified after flname Significance
199
Expressions used by find
(contd..)
200
Action -exec cmd {} \; -ok cmd {} \;
Significance Executes Unix command cmd Executes Unix command cmd, after user confirmation Prints selected file on standard output
-print
Backing up files using tar
Creating tar file
$ tar -cvf backup.tar *
201
Extracting tar file
$ tar -xvf backup.tar
Compression and decompression of files
zip and unzip zip newFile.zip filename unzip newFile.zip
202
gzip and gunzip gzip filename gunzip filename
User Administration
For user management, Unix Provides following command:
useradd usermod userdel
203
# useradd u 210 g dba c RDBMS d /home/oracle s /bin/ksh m oracle # usermod s /bin/csh oracle # userdel oracle
File System Administration
fsck : File System Checking # fsck /dev/user1
204