Tips and Tricks for Bash
Getting an "Argument list too long" error using Linux?
This is because you are passing too many arguments to an executable.
For example, using *.html below results in too many arguments to /bin/rm:
rm *.html
bash: /bin/rm: Argument list too long
One solution is to use Bash control loops to iterate through arguments.
for HTMLFILE in *.html
> do
> rm -f $HTMLFILE
> done
Alternatively, the find command can be used to manipulate large data sets.
The following find command is identical in result to the for loop shown above.
bash$ find ./ -maxdepth 1 -name '*.html' -exec rm -f {} \;
The Bash & (ampersand) is a builtin control operator used to fork processes. From the Bash man page, "If a command is terminated by the control operator &, the shell executes the command in the background in a subshell".
If logged into an interactive shell, the process is assigned a job number and the child PID is displayed. The job number below is one.
bash$ sleep 30 &
[1] 3586
Note that when a process is forked, the child PID is stored in the special variable $!
bash$ echo $!
3586
You can terminate the job by its job number like so:
bash$ jobs
[1]+ Running sleep 30 &
bash$ kill %1
[1]+ Terminated sleep 30
bash$
Arrays in Bash are quite simple. Here are some array examples to get you going. The official Bash documentation has more details and examples.
Initialize an entire array:
bash$ DAYS=(mon tue wed thu fri sat sun)
There are two ways to reference an entire array:
bash$ echo ${DAYS[@]}
mon tue wed thu fri sat sun
bash$ echo ${DAYS[*]}
mon tue wed thu fri sat sun
Initialize array element
bash$ ARRAY[0]="Fedora"
bash$ ARRAY[1]="RedHat"
bash$ ARRAY[2]="CentOS"
Display an element
bash$ echo ${ARRAY[0]}
Fedora
Get the length of an array:
bash$ echo ${#ARRAY[@]}
3
Get the length of an array value based on index:
bash$ echo ${ARRAY[0]}
Fedora
bash$ echo ${#ARRAY[0]}
6
Get the subset of an array through trailing substring extraction:
bash$ echo ${ARRAY[@]:0}
Fedora RedHat CentOS
bash$ echo ${ARRAY[@]:1}
RedHat CentOS
bash$ echo ${ARRAY[@]:2}
CentOS
For floating point math calculations in Bash, use /usr/bin/bc.
A simple calculator program might look something like this:
#!/bin/bash
echo "scale=2; $@" | /usr/bin/bc -l
exit $?
Alternatively, a calculator function:
calc()
{
echo "scale=2; $@" | /usr/bin/bc -l
return $?
}
If you are looking for more, here are other ways to do math in Bash.
Bash Color Escape Codes
Echo (echo -e) the following escape codes inside \e[ESCCODEm to colorize text in Bash:
Make sure to use echo -e to enable interpretation of backslash escapes:
bash$ echo -e "This is red->\e[00;31mRED\e[00m"
Remove Color
Echo \e[00m to remove text color modifications:
bash$ echo -n '\e[00m'
Some examples of iterating through files and looping through file content in Bash. For details, see the Bash man page or the Bash reference manual.
Loop Through Files
for FILE in *; do
# do something with $FILE
echo "File: $FILE"
done
For more control, use /usr/bin/file
for FILE in $(find ./ -name *.html -type f); do
# do something with $FILE
echo "HTML File: $FILE"
done
Loop Through Lines in a File
while read LINE; do
# do something with $LINE
echo "Line: $LINE"
done < /etc/hosts
Loop Through Words in a File
for WORD in $(cat /etc/hosts); do
# do something with $WORD
echo "Word: $WORD"
done
Loop Through Characters in a File
while read -n1 CHAR; do
# do something with $CHAR
echo "Character: $CHAR"
done < /etc/hosts
Below are examples of the Bash for loop in action. If you need more help, see the official Bash documentation for an introduction to Bash programming.
Examples
Echo all jpg files in the current folder:
for JPG in *.jpg; do
echo $JPG
done
Count a list of numbers:
for NUM in $(seq 1 10); do
echo $NUM
done
One liners work as well:
for n in $(seq 3); do echo $n; done
Iterate through the process ID's for httpd and kill them:
for PID in $(ps -C httpd | awk '/httpd/ { print $1 }'); do
kill -TERM $PID
done
List arguments to a function named showarg:
showarg() {
for ARG in $@; do
echo $ARG
done
}
Loop through folders and back them up using tar:
DATE=$(date +'%Y-%m-%d')
for DIR in /etc /var /root; do
tar -czvf /backups/$DIR_$DATE.tgz $DIR
done
Here are some example BASH functions. For more information, see the online BASH Function documentation or the BASH man page.
Declaring Functions
A BASH function must first be declared. One of the two methods can be used to declare a BASH function:
Method #1: function name compound-command
function say_goodbye {
echo "goodbye"
}
Method #2: name() compound-command
say_hello() {
echo "hello"
}
Using Functions
Just call the name as you would any other simple command:
say_hello
hello
Passing Variables
Pass variables into the function by space delimited fields and accept parameters using $1, $2 and so on.
my_function() {
echo "$1"
}
Example:
my_function hello this is a test
hello
To expand all variable parameters inside the function, use $@
log() {
echo "[$(date)]: $@"
}
log this is a test
[Sun Jun 22 10:21:26 PDT 2008]: this is a test
Return Codes
Functions return an exit status to their caller:
subtest() {
return 1
}
The function will return 1:
subtest || echo "Returned: $?"
Returned: 1
Since the function returned 1, it failed the test of || (or).
Use Bash builtins (let, expr) for integer math and bc (a GNU numeric processing language) for floating point arithmetic in your bash scripts. Here are some examples.
Integer Math in Bash
The expr builtin can be used as a simple integer calculator. Results are rounded to the nearest integer and floating point is unknown. BE sure to escape the multiplication asterisks (*) to avoid Bash expansion.
bash$ expr 1 + 1
2
bash$ expr 3 \* 2
6
bash$ expr 6 / 3
2
bash$ expr 6 % 3
0
bash$ expr 3 / 2
1
bash$ expr 3 / 6
0
bash$ expr 6 \* 3.5
expr: non-numeric argument
Instead of expr(), you can also echo the output of $(( )) or $[ ].
bash$ expr 1 + 1
4
bash$ echo $((2 + 2))
4
bash$ echo $[2 + 2]
4
Use the bash builtin let for quick manipulation of bash variables.
bash$ NUM=41
bash$ let NUM+=1
bash$ echo $NUM
42
Floating Point Arithmetic in Bash
Using floating point in bash scripts requires an external calculator like GNU bc. Pipe your request to bc and note that escaping is not needed for quoted asterisks.
bash$ echo "3.8 + .4" | bc
4.2
bash$ echo '6 * 1.5' | bc
9.0
If all input values are integers, the bc option scale must be defined if you expect a floating point result.
bash$ echo '2 / 5' | bc
0
bash$ echo 'scale=2; 2 / 5' | bc
.40
You can also use the bash here string <<< to accomplish the same as a pipe of echo to bc:
bash$ echo 'scale=2; 2 / 5' | bc
.40
bash$ bc <<< 'scale=2; 2 / 5'
.40
Or, use bc -l to evoke the standard (but not default!) mathlib and see the result in floating point at max scale:
bash$ bc -l <<< '10.5 / 1'
10.50000000000000000000
Call the bash builtin variable $RANDOM to produce a random integer between 0 and 32767.
bash$ echo $RANDOM
194
bash$ echo $RANDOM
7601
bash$ echo $RANDOM
17576
For more control, use let to manipulate the number range. Examples below.
For a single digit Integer:
let R=$RANDOM%10; echo $R
For a number between 0 and 99
let R=$RANDOM%100; echo $R
For a number between 1 and 100
let R=$RANDOM%100+1; echo $R
Bash supports various string operations. Here are some examples to get you started quickly. For detailed information, see the official Bash documentation.
String Assignment
Below are a few ways to assign a string variable.
bash$ OS=Linux
Use quotes to encapsulate strings with spaces:
bash$ OS='CentOS 4.6'
Or, use double quotes to allow for variable expansion:
bash$ DISTRO="Fedora"
bash$ OS="$DISTRO Linux"
bash$ echo $OS
Fedora Linux
String Length
Determining the length of your string is easy:
bash$ PROG="Bash"
bash$ echo ${#PROG}
4
You can also use expr:
bash$ expr length $PROG
4
Substring Extraction
Below are some examples of substring expansion in the form of ${string:position} and ${string:position:length}. String indexing starts at zero!
bash$ PROG="Bash"
bash$ echo ${PROG:0}
Bash
bash$ echo ${PROG:1}
ash
bash$ echo ${PROG:1:2}
as
Testing Strings
Test if a string is of length 0
[ -z $STRING ]
Test if length of string is not zero:
[ ! -z $STRING ]
[ -n $STRING ]
Test if strings are equal.
[ $STRING1 == $STRING2 ]
Remember to use quotes if the string has spaces or escape characters (newlines).
[ "$STRING1" == "$STRING2" ]
The 'not equal; operator is !=
[ $STRING1 != $STRING2 ]
Do you want to catch control-c keyboard interrupts in your Bash program? Use the Bash builtin trap command to catch system signals. The following runs control_c() when a user interrupts the main() section with a Control-C (SIGINT).
#!/bin/bash
cleanup()
# example cleanup function
{
rm -f /tmp/tempfile
return $?
}
control_c()
# run if user hits control-c
{
echo -en "\n*** Ouch! Exiting ***\n"
cleanup
exit $?
}
# trap keyboard interrupt (control-c)
trap control_c SIGINT
# main() loop
while true; do read x; done
Need to clear your Bash history?
Use the Bash builtin history command:
history -c
To stop the writing of your Bash history to a file when you log out:
unset HISTFILE
See the Bash man page for details.
Find that your shell is logging you out after a certain period of inactivity? You can tweak the number of seconds before the logout, or disable auto logout completely. In the instructions below, zero (0) disables auto logout or replace 0 with number for timeout seconds.
Disable Auto Logout in bash or sh
bash$ export TMOUT=0
Disable Auto Logout in csh or tcsh
tcsh% set autologout=0
You need to move a binary, but are unsure of it's dependencies? This script uses ldd and rpm to list the RPMs and standalone libraries needed for a dynamically linked executable.
#!/bin/bash
# myneeds.sh [FILE]
FILE=${1:?"error: no file argument"}
#parse the library listing
LIBS=$(ldd $FILE | sed -r 's:[^/]*(/.*)\ .*:\1:' | grep -e '^/' | sort | uniq)
# query the rpm database
rpm -qf $LIBS | sort | uniq -c
exit $?
Examples
bash$ ./myneeds.sh /usr/bin/mutt
1 cyrus-sasl-lib-2.1.22-8.fc8.x86_64
1 db4-4.6.21-1.fc8.x86_64
1 e2fsprogs-libs-1.40.4-2.fc8.x86_64
7 glibc-2.7-2.x86_64
1 gnutls-1.6.3-2.fc8.x86_64
1 keyutils-libs-1.2-2.fc6.x86_64
4 krb5-libs-1.6.2-14.fc8.x86_64
1 libgcrypt-1.2.4-6.x86_64
1 libgpg-error-1.5-6.x86_64
1 libidn-0.6.14-4.x86_64
1 libselinux-2.0.43-1.fc8.x86_64
2 ncurses-5.6-12.20070812.fc8.x86_64
1 zlib-1.2.3-14.fc8.x86_64
bash$ ./myneeds.sh /opt/apache2/bin/httpd
1 e2fsprogs-libs-1.40.4-2.fc8.x86_64
1 expat-2.0.1-2.x86_64
1 file /opt/apache2/lib/libapr-1.so.0 is not owned by any package
1 file /opt/apache2/lib/libaprutil-1.so.0 is not owned by any package
7 glibc-2.7-2.x86_64
1 sqlite-3.4.2-3.fc8.x86_64
So you need to backup all the MySQL databases on your system? This is easy with MySQL's mysqldump. The quickest solution is to
dump all databases to one ugly file:
mysqldump -u root -p --all-databases > $(hostname)-everything.sql
A more organized approach is to loop through all your databases and dump individual files. A Bash script example:
#!/bin/bash
read -p "Enter MySQL user: " -s DBUSER
read -p "Enter MySQL $DBUSER password: " DBPASS
DATE=$(date +'%m%d%Y')
# dump all databases to current path
for DB in $(echo "show databases" | mysql -u$DBUSER -p$DBPASS | grep -v Database); do
mysqldump -u$DBUSER -p$DBPASS $DB > $DB-$DATE.sql || exit $?
gzip $DB-$DATE.sql || exit $?
done && exit $?
Sources
MySQL.com: MySQL AB :: MySQL 5.0 Reference Manual :: 8.13 mysqldump A Database Backup Program
Have you ever seen this error when using tar?
tar -czf etc.tgz /etc
Removing leading `/' from member names
Tar is removing the leading / from the archive file, and warning you about it. Although you can redirect STDERR to /dev/null, doing so can result in missed errors. Instead, use tar with the -P or --absolute-names switch. They do the same thing: leave the leading / in the archived files.
tar -czPf etc.tgz /etc
When you untar the archive without -P, the leading / will still equate to your current working directory. Use the -P when untarring to restore from archive to the absolute path name. For example:
The following creates ./etc (dot, slash, etc)
tar -xzf etc.tgz
This overwrites /etc (slash, etc)!
tar -xzPf etc.tgz
Unset PROMPT_COMMAND
First, make sure that PROMPT_COMMAND is not set, since PROMPT_COMMAND is commonly used to set dynamic xterm titles. The contents of $PROMPT_COMMAND is executed before every primary prompt ($PS1) is displayed.
unset PROMPT_COMMAND
Echo The Right Escape Codes
Change your xterm title in Bash by echoing the title within the following escape sequences:
* Start Xterm Title: \033]0;
* End Xterm Title: \077
Notice that echo is used with -n (no trailing newline) and -e (enable interpretation of backslash escapes).
unset PROMPT_COMMAND
echo -ne "\033]0;TITLE GOES HERE\007"