Bash Notebook

Tips and Tricks for Bash

An Explanation of .bashrc and .bash_profile

Both the ~/.bashrc and ~/.bash_profile are scripts that might be executed when bash is invoked. The ~/.bashrc file gets executed when you run bash using an interactive shell that is not a login shell. The ~/.bash_profile only gets executed during a login shell. What does this all mean? The paragraphs below explains interactive shells, login shells, .bashrc, .bash_profile and other bash scripts that are executed during login.

Login Shells (.bash_profile)

A login shell is a bash shell that is started with - or --login. The following are examples that will invoke a login shell.

sudo su -
bash --login
ssh user@host

When BASH is invoked as a login shell, the following files are executed in the displayed order.

  1. /etc/profile
  2. ~/.bash_profile
  3. ~/.bash_login
  4. ~/.profile

Although ~/.bashrc is not listed here, most default ~/.bash_profile scripts run ~/.bashrc.

Purely Interactive Shells (.bashrc)

Interactive shells are those not invoked with -c and whose standard input and output are connected to a terminal. Interactive shells do not need to be login shells. Here are some examples that will evoke an interactive shell that is not a login shell.

sudo su
bash
ssh user@host /path/to/command

In this case of an interactive but non-login shell, only ~/.bashrc is executed. In most cases, the default ~/.bashrc script executes the system's /etc/bashrc.

Be warned that you should never echo output to the screen in a ~/.bashrc file. Otherwise, commands like 'ssh user@host /path/to/command' will echo output unrelated to the command called.

Non-interactive shells

Non-interactive shells do not automatically execute any scripts like ~/.bashrc or ~/.bash_profile. Here are some examples of non-interactive shells.

su user -c /path/to/command
bash -c /path/to/command

For more information, see the bash man page.

Argument List Too Long

Getting an "Argument list too long" error using Linux?

This is because you are passing too many arguments to an executable.
For example, using *.html below results in too many arguments to /bin/rm:

rm *.html
bash: /bin/rm: Argument list too long

One solution is to use Bash control loops to iterate through arguments.

for HTMLFILE in *.html
> do
> rm -f $HTMLFILE
> done

Alternatively, the find command can be used to manipulate large data sets.
The following find command is identical in result to the for loop shown above.

bash$ find ./ -maxdepth 1 -name '*.html' -exec rm -f {} \;

Bash & (Ampersand)

The Bash & (ampersand) is a builtin control operator used to fork processes. From the Bash man page, "If a command is terminated by the control operator &, the shell executes the command in the background in a subshell".

If logged into an interactive shell, the process is assigned a job number and the child PID is displayed. The job number below is one.

bash$ sleep 30 &
[1] 3586

Note that when a process is forked, the child PID is stored in the special variable $!

bash$ echo $!
3586

You can terminate the job by its job number like so:

bash$ jobs
[1]+ Running sleep 30 &
bash$ kill %1
[1]+ Terminated sleep 30
bash$

Bash Arrays

Arrays in Bash are quite simple. Here are some array examples to get you going. The official Bash documentation has more details and examples.

Initialize an entire array:

bash$ DAYS=(mon tue wed thu fri sat sun)

There are two ways to reference an entire array:

bash$ echo ${DAYS[@]}
mon tue wed thu fri sat sun
bash$ echo ${DAYS[*]}
mon tue wed thu fri sat sun

Initialize array element

bash$ ARRAY[0]="Fedora"
bash$ ARRAY[1]="RedHat"
bash$ ARRAY[2]="CentOS"

Display an element

bash$ echo ${ARRAY[0]}
Fedora

Get the length of an array:

bash$ echo ${#ARRAY[@]}
3

Get the length of an array value based on index:

bash$ echo ${ARRAY[0]}
Fedora
bash$ echo ${#ARRAY[0]}
6

Get the subset of an array through trailing substring extraction:

bash$ echo ${ARRAY[@]:0}
Fedora RedHat CentOS
bash$ echo ${ARRAY[@]:1}
RedHat CentOS
bash$ echo ${ARRAY[@]:2}
CentOS

Bash Calculator

For floating point math calculations in Bash, use /usr/bin/bc.
A simple calculator program might look something like this:

#!/bin/bash
echo "scale=2; $@" | /usr/bin/bc -l
exit $?

Alternatively, a calculator function:

calc()
{
   echo "scale=2; $@" | /usr/bin/bc -l
   return $?
}

If you are looking for more, here are other ways to do math in Bash.

Bash Colors

Bash Color Escape Codes

Echo (echo -e) the following escape codes inside \e[ESCCODEm to colorize text in Bash:

Make sure to use echo -e to enable interpretation of backslash escapes:

bash$ echo -e "This is red->\e[00;31mRED\e[00m"

Remove Color

Echo \e[00m to remove text color modifications:

bash$ echo -n '\e[00m'

Bash File Loops

Some examples of iterating through files and looping through file content in Bash. For details, see the Bash man page or the Bash reference manual.

Loop Through Files

for FILE in *; do
  # do something with $FILE
  echo "File: $FILE"
done

For more control, use /usr/bin/file

for FILE in $(find ./ -name *.html -type f); do
  # do something with $FILE
  echo "HTML File: $FILE"
done

Loop Through Lines in a File

while read LINE; do
  # do something with $LINE
  echo "Line: $LINE"
done < /etc/hosts

Loop Through Words in a File

for WORD in $(cat /etc/hosts); do
  # do something with $WORD
  echo "Word: $WORD"
done

Loop Through Characters in a File

while read -n1 CHAR; do
  # do something with $CHAR
  echo "Character: $CHAR"
done < /etc/hosts

Bash For Loop

Below are examples of the Bash for loop in action. If you need more help, see the official Bash documentation for an introduction to Bash programming.

Examples

Echo all jpg files in the current folder:

for JPG in *.jpg; do
  echo $JPG
done

Count a list of numbers:

for NUM in $(seq 1 10); do
  echo $NUM
done

One liners work as well:

for n in $(seq 3); do echo $n; done

Iterate through the process ID's for httpd and kill them:

for PID in $(ps -C httpd | awk '/httpd/ { print $1 }'); do
  kill -TERM $PID
done

List arguments to a function named showarg:

showarg() {
  for ARG in $@; do
    echo $ARG
  done
}

Loop through folders and back them up using tar:


DATE=$(date +'%Y-%m-%d')
for DIR in /etc /var /root; do
  tar -czvf /backups/$DIR_$DATE.tgz $DIR
done

Bash Functions

Here are some example BASH functions. For more information, see the online BASH Function documentation or the BASH man page.

Declaring Functions

A BASH function must first be declared. One of the two methods can be used to declare a BASH function:

Method #1: function name compound-command
function say_goodbye {
  echo "goodbye"
}

Method #2: name() compound-command
say_hello() {
  echo "hello"
}

Using Functions

Just call the name as you would any other simple command:

say_hello
hello

Passing Variables

Pass variables into the function by space delimited fields and accept parameters using $1, $2 and so on.

my_function() {
  echo "$1"
}

Example:

my_function hello this is a test
hello

To expand all variable parameters inside the function, use $@

log() {
  echo "[$(date)]: $@"
}

log this is a test
[Sun Jun 22 10:21:26 PDT 2008]: this is a test

Return Codes

Functions return an exit status to their caller:

subtest() {
return 1
}

The function will return 1:

subtest || echo "Returned: $?"
Returned: 1

Since the function returned 1, it failed the test of || (or).

Bash Math

Use Bash builtins (let, expr) for integer math and bc (a GNU numeric processing language) for floating point arithmetic in your bash scripts. Here are some examples.

Integer Math in Bash

The expr builtin can be used as a simple integer calculator. Results are rounded to the nearest integer and floating point is unknown. BE sure to escape the multiplication asterisks (*) to avoid Bash expansion.

bash$ expr 1 + 1
2
bash$ expr 3 \* 2
6
bash$ expr 6 / 3
2
bash$ expr 6 % 3
0
bash$ expr 3 / 2
1
bash$ expr 3 / 6
0
bash$ expr 6 \* 3.5
expr: non-numeric argument

Instead of expr(), you can also echo the output of $(( )) or $[ ].

bash$ expr 1 + 1
4
bash$ echo $((2 + 2))
4
bash$ echo $[2 + 2]
4

Use the bash builtin let for quick manipulation of bash variables.

bash$ NUM=41
bash$ let NUM+=1
bash$ echo $NUM
42

Floating Point Arithmetic in Bash

Using floating point in bash scripts requires an external calculator like GNU bc. Pipe your request to bc and note that escaping is not needed for quoted asterisks.

bash$ echo "3.8 + .4" | bc
4.2
bash$ echo '6 * 1.5' | bc
9.0

If all input values are integers, the bc option scale must be defined if you expect a floating point result.

bash$ echo '2 / 5' | bc
0
bash$ echo 'scale=2; 2 / 5' | bc
.40

You can also use the bash here string <<< to accomplish the same as a pipe of echo to bc:

bash$ echo 'scale=2; 2 / 5' | bc
.40
bash$ bc <<< 'scale=2; 2 / 5'
.40

Or, use bc -l to evoke the standard (but not default!) mathlib and see the result in floating point at max scale:

bash$ bc -l <<< '10.5 / 1'
10.50000000000000000000

Bash Random Numbers

Call the bash builtin variable $RANDOM to produce a random integer between 0 and 32767.

bash$ echo $RANDOM
194
bash$ echo $RANDOM
7601
bash$ echo $RANDOM
17576

For more control, use let to manipulate the number range. Examples below.

For a single digit Integer:

let R=$RANDOM%10; echo $R

For a number between 0 and 99

let R=$RANDOM%100; echo $R

For a number between 1 and 100

let R=$RANDOM%100+1; echo $R

Bash String Examples

Bash supports various string operations. Here are some examples to get you started quickly. For detailed information, see the official Bash documentation.

String Assignment

Below are a few ways to assign a string variable.

bash$ OS=Linux

Use quotes to encapsulate strings with spaces:

bash$ OS='CentOS 4.6'

Or, use double quotes to allow for variable expansion:

bash$ DISTRO="Fedora"
bash$ OS="$DISTRO Linux"
bash$ echo $OS
Fedora Linux

String Length

Determining the length of your string is easy:

bash$ PROG="Bash"
bash$ echo ${#PROG}
4

You can also use expr:

bash$ expr length $PROG
4

Substring Extraction

Below are some examples of substring expansion in the form of ${string:position} and ${string:position:length}. String indexing starts at zero!

bash$ PROG="Bash"
bash$ echo ${PROG:0}
Bash
bash$ echo ${PROG:1}
ash
bash$ echo ${PROG:1:2}
as

Testing Strings

Test if a string is of length 0

[ -z $STRING ]

Test if length of string is not zero:

[ ! -z $STRING ]
[ -n $STRING ]

Test if strings are equal.

[ $STRING1 == $STRING2 ]

Remember to use quotes if the string has spaces or escape characters (newlines).

[ "$STRING1" == "$STRING2" ]

The 'not equal; operator is !=

[ $STRING1 != $STRING2 ]

Bash Trap Control C

Do you want to catch control-c keyboard interrupts in your Bash program? Use the Bash builtin trap command to catch system signals. The following runs control_c() when a user interrupts the main() section with a Control-C (SIGINT).

#!/bin/bash
 
cleanup()
# example cleanup function
{
  rm -f /tmp/tempfile
  return $?
}
 
control_c()
# run if user hits control-c
{
  echo -en "\n*** Ouch! Exiting ***\n"
  cleanup
  exit $?
}
 
# trap keyboard interrupt (control-c)
trap control_c SIGINT
 
# main() loop
while true; do read x; done

Bash Variable Variable

So, you want a Bash variable variable? You know, a variable that contains a variable name. No problem - Use indirect expansion or eval. Below are some examples and options.

Bash Indirect Expansion for Variable Variables

First, lets say you have a set of variables that contain integers.

bash$ ONE=1
bash$ TWO=2
bash$ THREE=3

Next, you want to store the variable name inside a variable called NUM.

bash$ NUM=TWO

This is how you expand $NUM to give you the value of the variable name it is storing.

bash$ echo ${!NUM}
2

Again, if you change $NUM to THREE:

bash$ NUM=THREE

You can expand $NUM to get the value of the variable name stored in $NUM:

bash$ echo ${!NUM}
3

Here is an example that provides you with the number of days in the current month.

#!/bin/bash
 
THISMONTH=$(date +'%b')
 
Jan=31 Mar=31 May=31 Aug=31 Oct=31 Dec=31
Apr=30 Jun=30 Sep=30 Nov=30
Feb="28 or 29"
 
echo "${!THISMONTH} days in $THISMONTH"
exit $?

Eval for Bash Variable Variables

You can alternatively use eval to accomplish variable variables:

bash$ FIVE=5
bash$ NUM=FIVE
bash$ eval echo \$$NUM
5

Here is an eval example that uses a variable as part of a variable name:

bash$ COLOR_RED=FF0000
bash$ COLOR_BLUE=0000FF
bash$ COLOR_GREEN=00FF00
 
bash$ MYCOLOR=RED
bash$ eval echo \$COLOR_$MYCOLOR
FF0000

Clear and Disable Bash History

Need to clear your Bash history?
Use the Bash builtin history command:

history -c

To stop the writing of your Bash history to a file when you log out:

unset HISTFILE

See the Bash man page for details.

Disable auto logout

Find that your shell is logging you out after a certain period of inactivity? You can tweak the number of seconds before the logout, or disable auto logout completely. In the instructions below, zero (0) disables auto logout or replace 0 with number for timeout seconds.

Disable Auto Logout in bash or sh

bash$ export TMOUT=0

Disable Auto Logout in csh or tcsh

tcsh% set autologout=0

How to Be Faster at the Linux Command Line

Want to be faster at the Linux command line interface? Since most Linux distributions provide Bash as the default CLI, here are some Bash tricks that will help cut down the amount of typing needed to execute commands. Feel free to comment and share your own speed tricks.

Control-R Through Your History

This is my most used shortcut. Hit Control-R and begin to type a string. You immediately get the last command in your Bash history with that string. Hit Control-R again to cycle further backwards in your history.

For instance, type the following and hit Enter.

grep root /etc/passwd

Then hit Control-R and begin to type 'grep'.

Control-R
(reverse-i-search)`gre': grep root /etc/passwd

When you see the original command listed, hit Enter to execute it. Alternatively, you can also hit the Right-Arrow to edit the command before running it.

Use History Expansion

Bash's command history can be referenced using the exclamation mark. For instance, typing two exclamation marks (!!) will re-execute the last command. The next example executes date twice:

date
!!

If you are interested in more than just the last command executed, type history to see a numbered listing of your Bash's history.

history
  39 grep root /etc/passwd
  40 date
  41 date
  42 history

Since grep root /etc/passwd is command number 39, you can re-execute it like so:

!39

You can also reference Bash's history using a search string. For instance, the following will run the last command that started with 'grep'.

!grep

Note, you can set the number of commands stored in your history by setting HISTSIZE.

export HISTSIZE=1000

You can also wipe your history clear with the -c switch.

history -c

Use History Quick Substitution

Historical commands can be edited and reused with quick substitution. Let's say you grep for 'root' in /etc/passwd:

grep root /etc/passwd

Now, you need to grep for 'root' in /etc/group. Substitute 'passwd' for 'group' in the last command using the caret (^).

^passwd^group

The above command will run:

grep root /etc/group

Use Vi or Emacs Editing Mode

You can further enhance your abilities to edit previous commands using Vi or Emacs keystrokes. For example, the following sets Vi style command line editing:

set -o vi

After setting Vi mode, try it out by typing a command and hitting Enter.

grep root /etc/passwd

Then, Up-Arrow once to the same command:

Up-Arrow
grep root /etc/passwd

Now, move the cursor to the 'p' in 'passwd' and hit Esc.

grep root /etc/passwd
               ^

Now, use the Vi cw command to change the word 'passwd' to 'group'.

grep root /etc/group

For more Vi mode options, see this list of commands available in Vi mode. Alternatively, If you prefer Emacs, use Bash's Emacs mode:

set -o emacs

Emacs mode provides shortcuts that are available through the Control and Alt key. For example, Control-A takes you to the beginning of the line and Control-E takes you to the end of the line. Here is a list of commands available in Bash's Emacs mode.

Use Aliases and Functions

Bash allows for commands, or sets of commands, to be aliased into a single instruction. Your interactive Bash shell should already load some useful aliases from /etc/profile.d/. For one, you probably have ll aliased to ls -l.

If you want to see all aliases loaded, run the alias Bash builtin.

alias

To create an alias, use the alias command:

alias ll='ls -l'

Here are some other common aliases:

alias ls='ls --color=tty'
alias l.='ls -d .* --color=auto'
alias cp='cp -i'
alias mv='mv -i'

Note that you can also string together commands. The follow will alias gohome as cd , then run ls. Note that running cd without any arguments will change directory to your $HOME directory.

alias gohome='cd; ls'

Better yet, only run ls if the cd is successful:

alias gohome='cd && ls || echo "error($?) with cd to $HOME"'

More complex commands can be written into a Bash function. Functions will allow you to provide input parameters for a block of code. For instance, let's say you want to create a backup function that puts a user inputted file into ~/backups.

backup() {
  file=${1:?"error: I need a file to backup"}
  
  timestamp=$(date '+%m%d%y')
  backupdir=~/backups
  
  [ -d ${backupdir} ] || mkdir -p ${backupdir}
  cp -a ${file} ${backupdir}/$(basename ${file}).${timestamp}
  return $?
}

Like the example above, use functions to automate small, daily tasks. Here is one I use to set my xterm title.

xtitle() {
  unset PROMPT_COMMAND
  echo -ne "\033]0;${@}\007"
}

Of course, you can use functions together with aliases. Here is one I use to set my xterm title to 'MAIL' and then run Mutt.

alias mutt='xtitle "MAIL" && /usr/bin/mutt'

Finally, to ensure that your custom aliases and functions are available each login, add them to your .bashrc.

vim ~/.bashrc

Libraries needed for binary

You need to move a binary, but are unsure of it's dependencies? This script uses ldd and rpm to list the RPMs and standalone libraries needed for a dynamically linked executable.

#!/bin/bash
# myneeds.sh [FILE]
FILE=${1:?"error: no file argument"}

#parse the library listing
LIBS=$(ldd $FILE | sed -r 's:[^/]*(/.*)\ .*:\1:' | grep -e '^/' | sort | uniq)

# query the rpm database
rpm -qf $LIBS | sort | uniq -c
exit $?

Examples

bash$ ./myneeds.sh /usr/bin/mutt
1 cyrus-sasl-lib-2.1.22-8.fc8.x86_64
1 db4-4.6.21-1.fc8.x86_64
1 e2fsprogs-libs-1.40.4-2.fc8.x86_64
7 glibc-2.7-2.x86_64
1 gnutls-1.6.3-2.fc8.x86_64
1 keyutils-libs-1.2-2.fc6.x86_64
4 krb5-libs-1.6.2-14.fc8.x86_64
1 libgcrypt-1.2.4-6.x86_64
1 libgpg-error-1.5-6.x86_64
1 libidn-0.6.14-4.x86_64
1 libselinux-2.0.43-1.fc8.x86_64
2 ncurses-5.6-12.20070812.fc8.x86_64
1 zlib-1.2.3-14.fc8.x86_64
bash$ ./myneeds.sh /opt/apache2/bin/httpd
1 e2fsprogs-libs-1.40.4-2.fc8.x86_64
1 expat-2.0.1-2.x86_64
1 file /opt/apache2/lib/libapr-1.so.0 is not owned by any package
1 file /opt/apache2/lib/libaprutil-1.so.0 is not owned by any package
7 glibc-2.7-2.x86_64
1 sqlite-3.4.2-3.fc8.x86_64

Quick and Dirty MySQL Backup

So you need to backup all the MySQL databases on your system? This is easy with MySQL's mysqldump. The quickest solution is to
dump all databases to one ugly file:

mysqldump -u root -p --all-databases > $(hostname)-everything.sql

A more organized approach is to loop through all your databases and dump individual files. A Bash script example:

#!/bin/bash
read -p "Enter MySQL user: " -s DBUSER
read -p "Enter MySQL $DBUSER password: " DBPASS
DATE=$(date +'%m%d%Y')
# dump all databases to current path
for DB in $(echo "show databases" | mysql -u$DBUSER -p$DBPASS | grep -v Database); do
   mysqldump -u$DBUSER -p$DBPASS $DB > $DB-$DATE.sql || exit $?
   gzip $DB-$DATE.sql || exit $?
done && exit $?

Sources

MySQL.com: MySQL AB :: MySQL 5.0 Reference Manual :: 8.13 mysqldump A Database Backup Program

Removing leading `/' from member names

Have you ever seen this error when using tar?

tar -czf etc.tgz /etc
Removing leading `/' from member names

Tar is removing the leading / from the archive file, and warning you about it. Although you can redirect STDERR to /dev/null, doing so can result in missed errors. Instead, use tar with the -P or --absolute-names switch. They do the same thing: leave the leading / in the archived files.

tar -czPf etc.tgz /etc

When you untar the archive without -P, the leading / will still equate to your current working directory. Use the -P when untarring to restore from archive to the absolute path name. For example:

The following creates ./etc (dot, slash, etc)

tar -xzf etc.tgz

This overwrites /etc (slash, etc)!

tar -xzPf etc.tgz

XTerm Title

Unset PROMPT_COMMAND

First, make sure that PROMPT_COMMAND is not set, since PROMPT_COMMAND is commonly used to set dynamic xterm titles. The contents of $PROMPT_COMMAND is executed before every primary prompt ($PS1) is displayed.

unset PROMPT_COMMAND

Echo The Right Escape Codes

Change your xterm title in Bash by echoing the title within the following escape sequences:

* Start Xterm Title: \033]0;
* End Xterm Title: \077

Notice that echo is used with -n (no trailing newline) and -e (enable interpretation of backslash escapes).

unset PROMPT_COMMAND
echo -ne "\033]0;TITLE GOES HERE\007"