Skip to content

Beginner · Linux for DevOps · topic 8 of 53 · 35 min

Bash scripting, text tools and cron

Explanation

A bash script is a text file of commands that runs from top to bottom, so a task you do by hand can run the same way every time. You use variables, quoting, if, loops, functions and arguments, start every script with set -euo pipefail, and join small tools like grep, awk, sed, sort and uniq with pipes. cron runs scripts on a schedule such as "30 2 * * *".

Why script?

Any task you do by hand twice is a candidate for a script. A script never forgets a step, never mistypes a path at 2 am, and can run inside a CI pipeline or on a schedule. Bash scripts are the glue of DevOps: they back up databases, rotate logs, deploy releases and check health.

The first lines

bash
#!/usr/bin/env bash
set -euo pipefail

echo "Starting backup on $(hostname)"
The shebang line picks the interpreter; set -euo pipefail makes the script stop on errors.

The first line, starting with #!, is the shebang. It tells Linux which program runs the script. Make the file executable with chmod +x backup.sh and run it with ./backup.sh.

  • set -e: stop the script as soon as a command fails (exits non-zero).
  • set -u: treat an unset variable as an error instead of an empty string.
  • set -o pipefail: a pipeline such as a | b fails if ANY command in it fails, not just the last.

Variables and quoting

bash
APP="orders-api"
RELEASE_DIR="/opt/$APP/releases"
TODAY="$(date +%F)"            # command substitution: 2026-10-19
echo "Deploying $APP to $RELEASE_DIR on $TODAY"
echo 'Single quotes print $APP literally'
No spaces around =. Double quotes expand variables; single quotes do not.
Careful: Always put variables in double quotes: "$FILE", not $FILE. If FILE contains a space, such as "monthly report.csv", the unquoted version becomes two separate arguments and rm $FILE tries to delete two wrong files. A free tool, ShellCheck, finds this and many other script bugs.

Exit codes, if and test

bash
if [ -f /etc/nginx/nginx.conf ]; then
  echo "nginx is configured"
fi

if [ "$ENV" = "production" ] && [ "$#" -lt 2 ]; then
  echo "usage: deploy.sh production VERSION" >&2
  exit 1
fi

if curl -fsS http://localhost:3000/healthz > /dev/null; then
  echo "healthy"
else
  echo "health check failed" >&2
  exit 1
fi
if runs a command and checks its exit code. [ ] is the test command: -f file exists, = compares strings, -lt compares numbers.

Loops, functions and arguments

bash
for server in web1 web2 web3; do
  ssh "$server" "sudo systemctl restart orders-api"
done

count=0
while [ "$count" -lt 3 ]; do
  count=$((count + 1))
done

log() {
  echo "[$(date +%T)] $1"
}

log "script name: $0, first argument: $1, all arguments: $@, count: $#"
$1, $2 are arguments; "$@" is all of them; $# is how many; $0 is the script name.

Pipes and redirection

SymbolMeaningExample
|Send the output of one command into the nextcat app.log | grep ERROR
>Write output to a file (overwrite)ls > files.txt
>>Append output to a fileecho done >> deploy.log
2>&1Send errors (stream 2) to the same place as output (stream 1)./job.sh > job.log 2>&1
<Read input from a filemysql shop < schema.sql

The text toolbox

bash
grep -i "error" app.log                 # lines containing error, any case
grep -v "healthz" access.log            # lines NOT containing healthz
cut -d: -f1 /etc/passwd                 # first field, split on :
awk '{print $1}' access.log             # first column (client IP) of each line
sed -i 's/staging/production/g' app.env # replace text in the file
sort | uniq -c | sort -rn | head -n 5   # count repeats, biggest first

# top 5 IP addresses hitting the site
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -n 5

# delete .tmp files older than 2 days, safe with spaces in names
find /tmp/build -name "*.tmp" -mtime +2 -print0 | xargs -0 rm -f
Small tools joined by pipes answer most log questions in one line.

A real backup script

bash
#!/usr/bin/env bash
set -euo pipefail

DB_NAME="${1:-orders}"                      # first argument, or "orders"
BACKUP_DIR="/var/backups/postgres"
STAMP="$(date +%F-%H%M)"
FILE="$BACKUP_DIR/$DB_NAME-$STAMP.sql.gz"

mkdir -p "$BACKUP_DIR"
pg_dump "$DB_NAME" | gzip > "$FILE"
echo "Backup written to $FILE"

# keep 7 days of backups
find "$BACKUP_DIR" -name "*.sql.gz" -mtime +7 -delete
With pipefail, a failed pg_dump stops the script instead of leaving an empty .gz file that looks fine.

Debugging a script

When a script misbehaves, run it with bash -x script.sh. The -x option prints every command, with variables already filled in, just before it runs, so you can see exactly where it goes wrong. Paste the script into ShellCheck as well: it points out missing quotes, unused variables and commands that behave differently from what you expect.

cron: running on a schedule

cron runs commands at set times. Edit your schedule with crontab -e and list it with crontab -l. Each line has five time fields, then the command:

output
# m   h   dom  mon  dow   command
  30  2   *    *    *     /opt/scripts/backup.sh >> /var/log/backup.log 2>&1
  */15 *  *    *    *     /opt/scripts/health-check.sh
  0   9   *    *    1-5   /opt/scripts/report.sh
Minute, hour, day of month, month, day of week (0 or 7 is Sunday). Daily at 02:30, every 15 minutes, and 09:00 on weekdays.

* means every value. A list like 1,15 means those values. A range like 1-5 means 1 to 5. */15 means every 15th value, counted from the field’s first value. When both day of month and day of week are restricted, cron runs when EITHER matches, which surprises many people.

Careful: Common cron mistakes: (1) The script works in your shell but fails in cron, because cron has a tiny PATH and no profile. Use absolute paths like /usr/bin/pg_dump. (2) Output disappears; redirect it with >> file 2>&1. (3) A % in a crontab line means "new line" and must be escaped with a backslash. (4) Forgetting the server time zone, often UTC on cloud servers, so 02:30 UTC is 08:00 in India.

Need help?

Stuck on a word or an idea? Search this course, or every course, and jump straight to the explanation or a worked question.

Try:

Watch the lesson

Bash scripting, text tools and cron · 1/8

Bash scripts

A bash script is a file of commands that runs the same way every time. It is the glue of DevOps.

Prefer a video by a teacher? Find free videos on “bash scripting tutorial for devops set -euo pipefail cron” on YouTube ↗

In real life: Nightly database backups for a college ERP

A college in Chennai runs its student ERP on PostgreSQL. The admin used to take backups by hand "when he remembered". Now a 12-line bash script dumps and compresses the database, names the file with the date, deletes backups older than seven days, and copies the newest one to cloud storage. A crontab line runs it at 02:30 every night and writes output to a log. Once a month the team restores the latest backup on a test machine to prove it works.

output
30 2 * * * /opt/scripts/backup.sh erp >> /var/log/backup.log 2>&1
The crontab line that runs the backup every night at 02:30.

Practice (0/3 done)

Task 1 · Question

Which crontab line runs a script every day at 02:30?

Task 2 · Code · Python

Write cron_runs(expr, minute, hour, day, month, weekday) that says whether a five-field cron expression runs at that moment (weekday 0 = Sunday). Each field can be * (any value), a number, a comma list like 1,3,5, or */n (every n-th value counted from the field’s first value: 0 for minute, hour and weekday, 1 for day and month). Treat all five fields as AND. (Real cron uses OR when both day fields are restricted; the tests here never restrict both.)

Task 3 · Practice

Write a bash script check_disk.sh that takes a usage limit as its first argument (default 80), checks the usage percentage of the root filesystem "/" using df, prints "OK: 63% used" or "ALERT: 91% used", and exits with code 1 on ALERT so a pipeline or cron wrapper can notice. Use set -euo pipefail and quote your variables.

Quick quiz

Question 1

1.What does set -o pipefail change?

Question 2

2.Which redirection sends both normal output and errors into job.log?

Question 3

3.FILE="monthly report.csv". Why is rm $FILE dangerous?

Question 4

4.What does the cron line "0 9 * * 1-5 /opt/report.sh" do?

Question 5

5.Which pipeline lists the 5 most frequent client IPs, assuming the IP is the first column of access.log?

Read more (free)

Open a question to see the answer and why. The first ones are this topic's own quiz, so try the quiz above before you peek.

What does set -o pipefail change?

Answer: A pipeline fails if any command in it fails, not just the last one

Without pipefail, pg_dump failing in "pg_dump | gzip" would be hidden because gzip succeeded.

Which redirection sends both normal output and errors into job.log?

Answer: ./job.sh > job.log 2>&1

> job.log sends output to the file, then 2>&1 sends errors to the same place.

FILE="monthly report.csv". Why is rm $FILE dangerous?

Answer: Without quotes it splits into two arguments, "monthly" and "report.csv"

Unquoted variables are split on spaces. Always write rm "$FILE".

What does the cron line "0 9 * * 1-5 /opt/report.sh" do?

Answer: Runs at 09:00 Monday to Friday

Minute 0, hour 9, any day of month, any month, weekdays 1 to 5 (Monday to Friday).

Which pipeline lists the 5 most frequent client IPs, assuming the IP is the first column of access.log?

Answer: awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -n 5

awk picks the IP, sort groups equal lines, uniq -c counts them, sort -rn puts the biggest first.

What does $(hostname) do inside a script?Related

Answer: Runs hostname and puts its output there

Command substitution.

From Bash scripts: variables, quoting, arguments and exit codes

Why use set -o pipefail?Related

Answer: So a failure in any part of a pipeline makes the pipeline fail

Otherwise only the last command's status counts.

From Bash scripts: conditions, loops and safe habits

In "sort names.txt | uniq -c", what does the pipe do?Related

Answer: Sends sort's output to uniq -c as input

A pipe connects stdout of the left command to stdin of the right one.

From Reading text: grep, less, pipes and redirection

A script works in your terminal but fails from cron with "command not found". Most likely?Related

Answer: cron uses a minimal PATH; use full paths

Set PATH or use absolute paths.

From Scheduling with cron and crontab

Pass the quiz (60% or more) or mark this topic as done when you are ready.

Your progress is saved on this device only.

Next: Linux networking for DevOps →

Next steps

  1. 1 more topic in “Linux for DevOps” before its exam: Linux networking for DevOps.
  2. Module 2 exam: Linux for DevOps · 13 questions · 25 min, without help. 5 topics of this module still to finish first.
  3. Then the next module, Git in depth. 53 topics left in the course before the final exam.

← Previous: Processes, signals and systemd services