Shell Scripting 101
Updated September 12, 2003
Created August 25, 2003

Autogenerated Site Map
Search this Site!:

The true power lies with programs ran from the command line not the gui.

Here are the tools! The fast track to shell scripting.

First a few basics:

How to set a variable:
MYVAR=rememberthis
MYVAR="remember this"
MYVAR=3

Do not place a space around the equal sign. A space on the right
will mean to execute the item on the right as a regular executable.
Also this setting a variable and running a command in this manner
will forget the variable change after the command completes.
LANG=C make

Adding to a variable
MYVAR="remember this"
MYVAR="Can you $MYVAR"
MYVAR="${MYVAR} is cool stuff"

MYVAR="This is"
MYVAR="${MYVAR}runtogether"

The following are the same:
echo $MYVAR
echo ${MYVAR}

Arithmetic and Variables:
COUNT=3
COUNT=`expr $COUNT + 1`
echo $((++COUNT))

The following is a combination, it adds/subtracts to
the variable AND presents the value of the variable:
Pre-increment/decrement
echo $((COUNT++))
echo $((COUNT--))

Post-increment/decrement
echo $((++COUNT))
echo $((--COUNT))

The above is usually useful for some kind of loop statement (for/while/...)
or control statement (if / test / [ test ] / ...).

unset will forget a variable
so will an empty assignment:
MYVAR=

Set a variable for 1 command, then forget that variable change, example:
date
TZ=UTC date
date

Also you can use semicolons to separate each command:

date; TZ=UTC date; date
date; (TZ=UTC date); date
date; (TZ=UTC; date); date

The above demonstrates the short method (#1), but also presents the
parenthesis which will cause any variables that are set to be forgotton.

Testing a variable to see if it is set or is set to null or doesn't exist:

if [ -z "$MYVAR" ]; then
echo MYVAR is not set
else
echo Contents of MYVAR are $MYVAR
fi

if [ -n "$MYVAR" ]; then
echo MYVAR is non-empty
echo Contents of MYVAR are $MYVAR
else
echo MYVAR is not set
fi

Common Name	Standard In	Standard Out	Standard Error
Short Name	stdin	stdout	stderr
Assumed Redirector	<	>	2>
Explicit Redirection	0<	1>	2>
File Descriptor	&0	&1	&2

<
So as you see above you can redirect input with <.

>
You can redirect ouput with the >. Keep in mind that
this method will erase the initial contents of the file
you are redirecting output to.

>>
If you wish to append output to an existing file, then use
the double greater than sign >>

<<
This is a special redirector which creates a "here-document".

<<EOF
contents goes here
EOF

The terminating delimeter (EOF) with no trailing spaces must appear
in the leftmost column unless the dash "-" is used. If the dash is
used then tabs will be ignored and the EOF can appear away from the
left column.



	<<-EOF

	contents

	EOF

& - The Ampersand
The ampersand will place a task in the background and can be controlled
with jobs, bg, fg, %1, %1 &, CTRL+z, kill %1.

| - The Pipe
The pipe is perhaps one of the most important command line tools. This
allows you to join together several programs to work on one task. This
can be done at the command line even without the assistance of a script.

cat myfile | grep keepstring

Result Code - $?
Most all programs return an result code also called error code. You can
see the result code of the last program by typing:

echo $?

Often the result code is stored in a variable as follows:

RESULT=$?

&& and || - Logical AND and Logical OR
These two rely on the result code of the previous program. If success
(result code of 0) then the code behind && will run. On failure
(result code of non-zero) then the code behind the || will run.

myprog && DoProgOnSuccess || DoProgOnFailure

{}
This controls program flow. Here is an example with an if statement

myprog && {
echo trythis
DoProgOnSuccess
echo ok did this
} || {
echo trythat
DoProgOnFailure
echo ok did that
}

# - Comment
A comment begins with a pound sign and is in effect until the end of that line

()
The parenthesis can group some commands together. The parenthesis can also
help capture stdin, stdout, stderr, and other file descriptors. Also the
parenthesis runs in a sub shell. When the sub shell is completed, then
the environment of the sub shell is discarded. For example the following
command will cd to the newdir and do a directory listing there - when
the parenthesis finishes, we will still be in our original directory, not
in newdir.

pwd
(cd newdir; ls)
pwd

\ - Escape
This allows you to escape a character (i.e. take away the special meaning of that
character). Often if you want to have a special character not have it's special
meaning, then you need to escape that character. Here is an escaped backtick:

\`

`` and $() - Command Substitution
Command substitution is really useful as follows:

seq 1 10
This will give me the numbers 1 through 10: 1 2 3 4 5 6 7 8 9 10. However,
I can provide this as input to a for loop without having to type out the
numbers myself:

for X in `seq 1 10`; do
echo $X
done

The two methods are interchangeable:

for X in $(seq 1 10); do
echo $X
done

If you will have nested backticks then the inner ones will need to be escaped. Each
time you nest the backticks another level then more escapes need to be added:

for X in `seq \`expr 5 - 4\` 10`; do
echo $X
done

The above executes as follows, expr 5 - 4 leaves an answer of 1. seq 1 10 (remember
the 1 came from `expr 5 - 4`) will give us 1 2 3 4 5 6 7 8 9 10. Then the for loop
will cycle through each of these numbers in turn.

Special device nodes:
/dev/null
This has many names, the bit bucket, trash, null... Whatever you want to call it,
if you redirect output to /dev/null then that output is no more.

/dev/zero
This provides a source of zero (true null, not the character 0). This has its uses.

Common Commands and sample uses:

Grep is a filter. You can keep or throw out certain output. Useful in pipes or by itself:

cat myfile | grep keepstring
cat myfile | grep -v ThrowThisStringOut

grep keepstring myfile
grep -v ThrowThisStringOut

Use -i to do a case insensitive filter:

cat myfile | grep -i keepstring
cat myfile | grep -vi ThrowThisStringOut

Sed is useful for substitutions. g stands for global:

cat myfile | sed -e 's/ReplaceThis/WithThat/'
cat myfile | sed -e 's/ReplaceThis/WithThat/g'

Sed can print certain line numbers. This will print line 10:

cat myfile | sed --silent 10p

Awk is great for printing a certain column. This prints column 5 of the output of ps ax:

ps ax | awk '{print $5}'

Control Statements:

if; then; elif; else; fi

if [ "$MYVAL" -lt 5 ]; then
echo less than 5
fi

if [ "$MYVAL" -lt 5 ]; then
echo less than 5
else
echo not less than 5
fi

for; do; done

for X in a b c 1 2 3; do echo $X; done

Loops (for, while, until) can make good use of "continue" and "break". While loops
can make good use of the system command "read".

while; do; done

Also "read" will populate the variable REPLY. Here I use a read with a while loop.

cat myfile | while read; do echo $REPLY; done

"read" will populate variables you give names for (Here I use variables ONE, TWO, and OTHERS).
Note that the input line to read is broken up by the field separators (default is whitespace and
is defined by the special variable which is named "IFS" which has a default variable of SPACE (" "),
TAB (\t), and NEWLINE (\n).

So if you give names for 3 variables but have 5 words of data, then the following will happen:

echo one two three four five | read ONE TWO OTHERS
echo $ONE
one
echo $TWO
two
echo $OTHERS
three four five

Notice that all the extra data is lumped into the last named variable.

cat myfile | while read ONE TWO OTHERS; do echo $ONE; echo $TWO; echo $OTHERS; done

while :; do sleep 10; done
(colon ":" means true, so while true do sleep 10, done)

"continue" will skip the rest of the current loop (while, for, until, etc) and will
start on the next iteration of the loop

while :; do echo Doing stuff; if [ $? -eq 0 ]; then continue; fi; echo Do other stuff when not continuing; done

"break" will exit the loop now (at this point).

while :; do echo Doing stuff and the break will let us out; if [ $? -ne 0 ]; then break; fi; done

FAIL=0
while [ $FAIL -eq 0 ]; do echo Doing stuff; if [ $? -ne 0 ]; then FAIL=1; fi

(cd SourceDir && tar -cf - .)|(cd DestDir && tar -xvf -)

Regex:
^ Beginning
$ End
. any one item
* previous item repeated 0 or more times
[0-9] match a single digit number
[0-9]* match a number with 0 or more digits

Sed grouping:
Each set of parenthesis can be called with the number \1, \2, \3, and so on.

cat myfile | sed -e 's/href="$.*$"/\1/'

In Sed, a ampersand "&" on the right hand side will repeat what was matched
from the left hand side:

cat myfile | sed -e 's/This line/& was too short so I added more text./'

System Commands:
while loop
read command
for loop
seq gives a sequence of numbers. Here you can pad with blanks or my favorite, leading zeros:
seq -f "%4g" 7 15
seq -f "%04g" 7 15

Awk can also reformat for you to give you leading blanks or zeros:

echo 5 | awk '{printf ("%04g", $1)}'

Also there is a command called printf:

printf "%04g" 5
printf "%04g" `echo 5`

date
date "+%Y%m%d%H%M%S"
date "+%Y%m%d-%H%M%S"
date "+%Y%m%d-%H%M.%S"

ls
ls -ltr
ls -ltr | grep ^d
ls -ltr | grep ^-
ls -ltr | grep ^l

find
find .
find . -type f
find . -type d
find . -type f -printf "%TY%Tm%Td%TH%TM%TS\n"
find . -type f -exec grep -Hi searchstring {} \;

head -n 5
tail -n 10
sort
sort -r
uniq

When printing, you can send straight to the printer with lpr:

cat myfile | lpr

But often you miss out on a few lines at the head and foot of each page, so use enscript instead:

enscript myfilename --margins=20:20:20:20

Variables

VARIABLE=mydata
echo $VARIABLE
echo ${VARIABLE}and other stuff

Functions

FunctionName() {
echo function contents goes here
}

Other important stuff:
man
info
tee
script
expect
less
more

Anything special that makes up a script file?
It is common to have the following line as the very first line in the script file:
#!/bin/bash

When you first create your script it is not executable. Mark it executable, then you can run it:

chmod +x myscript

./myscript

How do I run my script? It doesn't want to run:
Either of the following will let sh or bash run your script:
sh myscript
bash myscript

Isn't there a better way to run my script?
Of course there is:
./myscript

The above ./myscript doesn't work, what's wrong?
Likely you don't have the execute permission set on your script file:
chmod +x myscript

Now you can run your script with the following command
./myscript

I remember the old 8.3 file format where the only files I could run were .exe, .com, or .bat.

Unix/Linux is not bound by what extension a file has. For the most part file extensions don't mean anything in Unix/Linux. To help determine what type a file is, use the file command:
file myfile

Which editor should I use?
I recommend VI because it is very powerful. You can also use pico or any other editor you like.

What is a sub shell? A sub shell is a 2nd copy of your command interpreter (bash)
which often takes a copy of your environement variables. When this 2nd copy
of your command interpreter finishes running, then it's environment is discarded.
This means that variables you set in a sub shell will not hold their value once
the sub shell is discarded. If you wish to hold the modified value of the variable
then you need to find a method that sets the variable outside of the sub shell.

What creates sub shells:
()

What does not create a sub shell:
functions

You now have enough information to be able to read and understand man bash. Please read "man bash" now.
If all else fails then check out man bash or info bash. Or use man or info on the other
program you have problems on. If you need more examples (LEARN BY EXAMPLE) then give this a try:

cd /etc
find . -type f -exec grep -Hi SearchString {} \;

If you're ready for more details or advanced details then move to shell scripting 102.

Search this Site!:

Search this site powered by FreeFind

Homepage: http://www.cpqlinux.com
Site Map: http://www.cpqlinux.com/sitemap.html

Shell Scripting 101 Updated September 12, 2003 Created August 25, 2003

Shell Scripting 101
Updated September 12, 2003
Created August 25, 2003