Linux

Bash Baby Steps Part 2

Input, Output and Error Redirections

Normally commands executed on GNU/Linux command line either produce output, require input or throw an error message. This is a fundamental concept for shell scripting as well as for working with GNU/Linux’s command line in general.

Every time, you execute a command, three possible outcomes might happen. The first scenario is that the command will produce an expected output, second, the command will generate an error, and lastly, your command might not produce any output at all:

What are we most interested in here is the output of both ls -l foobar commands. Both commands produced an output which by default is displayed on your terminal. However, both outputs are fundamentally different.

The first command tries to list non-existing file foobar which, in turn, produces a standard error output (stderr). Once the file is created by touch command, the second execution of the ls command produces standard output (stdout).

The difference between stdout and stderr output is an essential concept as it allows us to a threat, that is, to redirect each output separately. The > notation is used to redirect stdout to a file whereas 2> notation is used to redirect stderr and &> is used to redirect both stdout and stderr. The cat command is used to display a content of any given file. Consider a following example:

Replay the above video few times and make sure that you understand the redirection concept shown.

 Quick Tip:When unsure whether your command produced stdout or stderr try to redirect its output. For example, if you are able to redirect its output successfully to a file with 2> notation, it means that your command produced stderr. Conversely, successfully redirecting command output with > notation is indicating that your command produced stdout.

Back to our backup.sh script. When executing our backup script, you may have noticed an extra message display by tar command:

tar: Removing leading `/' from member names

Despite the message’s informative nature, it is sent to stderr descriptor. In a nutshell, the message is telling us that the absolute path has been removed thus extraction of the compressed file not overwrite any existing files.

Now that we have a basic understanding of the output redirection we can eliminate this unwanted stderr message by redirecting it with 2> notation to /dev/null. Imagine /dev/null as a data sink, which discards any data redirected to it. For more information run man null. Below is our new backup.sh version including tar’s stderr redirection:

#!/bin/bash

# This bash script is used to backup a user's home directory to /tmp/.

user=$(whoami)
input=/home/$user
output=/tmp/${user}_home_$(date +%Y-%m-%d_%H%M%S).tar.gz

tar -czf $output $input 2> /dev/null
echo "Backup of $input completed! Details about the output backup file:"
ls -l $output

After executing a new version of our backup.sh script, no tar stderr message will be displayed.

The last concept to briefly cover in this section is a shell input. Apart of the above stdout and stderr descriptors bash shell also features input descriptor name stdin. Generally, terminal input comes from a keyboard. Any keystroke you type is accepted as stdin.

The alternative method is to accept command input from a file using < notation. Consider the following example where we first feed cat command from the keyboard and redirecting the output to file1.txt. Later, we allow cat command to read the input from file1.txt using < notation:

Functions

The topic we are going to discuss next is functions. Functions allow a programmer to organize and reuse code, hence increasing the efficiency, execution speed as well as readability of the entire script.

It is possible to avoid using functions and write any script without including a single function in it. However, you are likely to end up with a chunky, inefficient and hard to troubleshoot code.

 Quick Tip:The moment you notice that your script contains two lines of the same code, you may consider to enact a function instead.

You can think of the function as a way to the group number of different commands into a single command. This can be extremely useful if the output or calculation you require consists of multiple commands, and it will be expected multiple times throughout the script execution. Functions are defined by using the function keyword and followed by function body enclosed by curly brackets.

The following video example defines a simple shell function to be used to print user details and will make two function calls, thus printing user details twice upon a script execution.

The function name is user_details, and function body enclosed inside curly brackets consists of the group of two echo commands. Every time a function call is made by using the function name, both echo commands within our function definition are executed. It is important to point out that the function definition must precede function call, otherwise the script will return function not found error:

As illustrated by the above video example the user_details function grouped multiple commands in a single new command user_details.

The preceding video example also introduced yet another technique when writing scripts or any program for that matter, the technique called indentation. The echo commands within the user_details function definition were deliberately shifted one TAB right which makes our code more readable, easier to troubleshot.

With indentation, it is much clearer to see that both echo commands below to user_details function definition. There is no general convention on how to indent bash script thus it is up to each individual to choose its own way to indent. Our example used TAB. However, it is perfectly fine to instead a single TAB use 4 spaces, etc.

Having a basic understanding of bash scripting functions up our sleeve, let’s add a new feature to our existing backup.sh script. We are going to program two new functions to report a number of directories and files to be included as part of the output compressed the backup file.

#!/bin/bash

# This bash script is used to backup a user's home directory to /tmp/.

user=$(whoami)
input=/home/$user
output=/tmp/${user}_home_$(date +%Y-%m-%d_%H%M%S).tar.gz

# The function total_files reports a total number of files for a given directory. 
function total_files {
        find $1 -type f | wc -l
}

# The function total_directories reports a total number of directories
# for a given directory. 
function total_directories {
        find $1 -type d | wc -l
}

tar -czf $output $input 2> /dev/null

echo -n "Files to be included:"
total_files $input
echo -n "Directories to be included:"
total_directories $input

echo "Backup of $input completed!"

echo "Details about the output backup file:"
ls -l $output

After reviewing the above backup.sh script, you will notice the following changes to the code:

  • we have defined a new function called total_files. The function utilized the find and wc commands to determine the number of files located within a directory supplied to it during the function call
  • we have defined a new function called total_directories. Same as the above total_files function it utilized the find and wc commands however it reports a number of directories within a directory supplied to it during the function call
 Quick Tip:Read manual pages, if you wish to learn more about find, wc and echo command’s options used by our backup.sh bash script. Example: $ man find

Once you update your script to include new functions, the execution of the script will provide a similar output to the one below:

$ ./backup.sh 
Files to be included:19
Directories to be inlcuded:2
Backup of /home/linuxconfig completed!
Details about the output backup file:
-rw-r--r-- 1 linuxconfig linuxconfig 5520 Aug 16 11:01 /tmp/linuxconfig_home_2017-08-16_110121.tar.gz

Numeric and String Comparisons

In this section, we are going to learn some basics of numeric and string bash shell comparisons. Using comparisons, we can compare strings ( words, sentences ) or integer numbers whether raw or as variables. The following table lists rudimentary comparison operators for both numbers and strings:

Bash Shell Numeric and String Comparisons
Description Numeric Comparison String Comparison
Shell comparison example: [ 100 -eq 50 ]; echo $? [ “GNU” = “UNIX” ]; echo $?
less than -lt <
greater than -gt >
equal -eq =
not equal -ne !=
less or equal -le N/A
greater or equal -ge N/A

 

After reviewing the above table, let’s say, we would like to compare numeric values like two integers 1 and 2. The following video example will first define two variables $a and $b to hold our integer values.

Next, we use square brackets and numeric comparison operators to perform the actual evaluation. Using echo $? command, we check for a return value of the previously executed evaluation. There or two possible outcomes for every evaluation, true or false. If the return value is equal to 0, then the comparison evaluation is true. However, if the return value is equal to 1, the evaluation resulted as false.

Using string comparison operators we can also compare strings in the same manner as when comparing numeric values. Consider the following example:

If we were to translate the above knowledge to a simple bash shell script, the script would look as shown below. Using string comparison operator = we compare two distinct strings to see whether they are equal.

Similarly, we compare two integers using the numeric comparison operator to determine if they are equal in value. Remember, 0 signals true, while 1 indicates false:

#!/bin/bash

string_a="UNIX"
string_b="GNU"

echo "Are $string_a and $string_b strings equal?"
[ $string_a = $string_b ]
echo $?

num_a=100
num_b=100

echo "Is $num_a equal to $num_b ?" 
[ $num_a -eq $num_b ]
echo $?

Save the above script as eg. comparison.sh file, make it executable and execute:

$ chmod +x compare.sh 
$ ./compare.sh 
Are UNIX and GNU strings equal?
1
Is 100 equal to 100 ?
0
 Quick Tip:Comparing strings with integers using numeric comparison operators will result in the error: integer expression expected. When comparing values, you may want to use echo command first to confirm that your variables hold expected values before using them as part of the comparison operation.

Apart from the educational value, the above script does not serve any other purpose. Comparisons operations will make more sense once we learn about conditional statements like if/else. Conditional statements will be covered in the next chapter, and this is where we put comparison operations to better use.

Conditional Statements

Now, it is time to give our backup script some logic by including few conditional statements. Conditionals allow the programmer to implement decision making within a shell script based on certain conditions or events.

The conditionals we are referring to are of course, if, then and else. For example, we can improve our backup script by implementing a sanity check to compare the number of files and directories within a source directory we intend to backup and the resulting backup file. The pseudocode for this kind of implementation will read as follows:

IF the number of files between the source and destination target is equal THEN print the OK message, ELSE, print ERROR.

Let’s start by creating a simple bash script depicting a basic if/then/else construct.

#!/bin/bash

num_a=100
num_b=200

if [ $num_a -lt $num_b ]; then
    echo "$num_a is less than $num_b!"
fi

For now the else conditional was deliberately left out, we will include it once we understand the logic behind the above script. Save the script as, eg. if_else.sh and execute it:

Lines 3 – 4 are used to initialize an integer variables. On Line 6 we begin an if conditional block. We further compare both variables and if the comparison evaluation yields true, then on Line 7 the echo command will inform us, that the value within the variable $num_a is less when compared with the variable $num_b. Lines 8 closes our if conditional block with a fi keyword.

The important observation to make from the script execution is that, in the situation when the variable $num_a greater than $num_b our script fails to react. This is where the last piece of the puzzle, else conditional comes in handy. Update your script by adding else block and execute it:

#!/bin/bash

num_a=400
num_b=200

if [ $num_a -lt $num_b ]; then
    echo "$num_a is less than $num_b!"
else
    echo "$num_a is greater than $num_b!"
fi

The Line 8 now holds the else part of our conditional block. If the comparison evaluation on Line 6 reports false the code below else statement, in our case Line 9 is executed.

 Exercise:Can you rewrite the if_else.sh script to reverse the logic of its execution in a way that the else block gets executed if the variable $num_a is less than variable $num_b?

Equipped with this basic knowledge about the conditional statements we can now improve our script to perform a sanity check by comparing the difference between the total number of the files before and after the backup command. Here is the new updated backup.sh script:

#!/bin/bash

user=$(whoami)
input=/home/$user
output=/tmp/${user}_home_$(date +%Y-%m-%d_%H%M%S).tar.gz

function total_files {
        find $1 -type f | wc -l
}

function total_directories {
        find $1 -type d | wc -l
}

function total_archived_directories {
        tar -tzf $1 | grep  /$ | wc -l
}

function total_archived_files {
        tar -tzf $1 | grep -v /$ | wc -l
}

tar -czf $output $input 2> /dev/null

src_files=$( total_files $input )
src_directories=$( total_directories $input )

arch_files=$( total_archived_files $output )
arch_directories=$( total_archived_directories $output )

echo "Files to be included: $src_files"
echo "Directories to be included: $src_directories"
echo "Files archived: $arch_files"
echo "Directories archived: $arch_directories"

if [ $src_files -eq $arch_files ]; then
        echo "Backup of $input completed!"
        echo "Details about the output backup file:"
        ls -l $output
else
        echo "Backup of $input failed!"
fi

There are few additions to the above script. Highlighted are the most important changes.

Lines 15 – 21 are used to define two new functions returning a total number of files and directories included within the resulting compressed backup file. After the backup Line 23 is executed, on Lines 25 – 29 we declare new variables to hold the total number of source and destination files and directories.

The variables concerning backed up files are later used on Lines 36 – 42 as part of our new conditional if/then/else statement returning a message about the successful backup on Lines 37 – 39only if the total number of both, source and destination backup files is equal as stated on Line 36.

Here is the script execution after applying the above changes:

$ ./backup.sh 
Files to be included: 24
Directories to be included: 4
Files archived: 24
Directories archived: 4
Backup of /home/linuxconfig completed!
Details about the output backup file:
-rw-r--r-- 1 linuxconfig linuxconfig 235569 Sep 12 12:43 /tmp/linuxconfig_home_2017-09-12_124319.tar.gz

 

Comment here