Skip to content

Instantly share code, notes, and snippets.

@dan4thewin

dan4thewin/bash4thewin.md

Last active April 27, 2025 20:20
Show Gist options
  • Save dan4thewin/a7c4f072ed09953b3f9b9c2a5d241bd0 to your computer and use it in GitHub Desktop.
Save dan4thewin/a7c4f072ed09953b3f9b9c2a5d241bd0 to your computer and use it in GitHub Desktop.
Download ZIP
Bash idioms and tools
Raw
bash4thewin.md

This page collects a set of bash scripting idioms and tools for easy reference.

Improved Error Detection: set -e and set -u

With -e (errexit), bash exits immediately if an unchecked command exits with a nonzero value. A checked command is part of a while, until, if, elif, or an expression with && or ||. Everything else is unchecked. For a pipeline, only the last command is considered. Adding "or no-op", || :, ensures a command is safe, e.g., do-something | grep 'may_not_match' || :. NB. Calling a bash function in a checked context effectively disables -e for all commands in the function. Use the optional shellcheck, check-set-e-suppressed, to alert on this.

With -u (nounset), bash exits immediately if there is an unset variable. To permit an optional variable, use ${var:-}, e.g., ${1:-} for an optional function argument.

Idioms

idiom description
${var:-default} value of var if var is set, default otherwise
${var:+alt_value} alt_value if var is set, nothing otherwise, e.g., ${PORT:+-p $PORT}
${var#pattern} value with matching prefix removed
${var##pattern} value with longest matching prefix removed
${var%pattern} value with matching suffix removed
${var%%pattern} value with longest matching suffix removed
${var%%$'\n'*} the first line (value with everything after the first newline removed)
${var,,} value converted to lower case
printf '%(%s)T' -1 print epoch time, seconds since 1970-01-01 00:00:00 UTC
printf -v var '%(%s)T' -1 set var to epoch time
$EPOCHSECONDS epoch time, available since bash v5.0
do_something 2>&- discard command output to stderr
local -n var_ref=$1 pass by reference, access a variable in the caller
${0##*/} script name without directory, same as basename $0
${0%/+([^/])} directory containing the script, needs shopt -s extglob
files=(glob); ${#files[*]} count of files matching glob pattern, needs shopt -s nullglob
var=$(<a_file) set var to contents of a_file
${var:0:1} the first character of the value
${var%%*(/)} value with zero or more trailing slashes removed, needs shopt -s extglob
$'\t' a literal tab character
((var++,1)) increment var; return status is always 0

Longer examples

With -e beware of ending a function with an && or || expression unless the function represents a predicate.

#!/bin/bash

# GOOD
function log1 () {
    if (( VERBOSE )); then
        echo "$*"
    fi
}

# BAD
function log2 () {
    (( VERBOSE )) && echo "$*"
}

set -eu

VERBOSE=0
log1 "this works"
log2 "this trips -e"
echo done  # never reached

The following needs the associative array declared with -A and lastpipe enabled or it fails because of an unbound variable.

#!/bin/bash
# shellcheck disable=SC2034

setup_bash () {
	# catch unexpected errors
	set -eu
	# allow the tail of a pipe to set values in this shell
	shopt -s lastpipe
}

# Snapshot running services (pid != 0) with name and start time
# Use sed to convert stanzas to tabular data
# Populate a dictionary named by $1 with the key/value pairs
load_start_times () {
	local -n dict=$1
	local pid start id

	(systemctl show -p Id,MainPID,ExecMainStartTimestampMonotonic \*; echo) |
	sed -n '/^$/{x; s/^\n//; s/\n/\t/g; /^[1-9]/p; b}; s/.*=//; H' |
	while read -r pid start id; do
		dict[$id]=$start
	done
}

main () {
	setup_bash
	local -A start_times
	load_start_times start_times
	echo "${start_times[cron.service]}"
}

main "$@"

Combining sleep and wait with background processes provides simple concurrency.

#!/bin/bash
# demonstrate child processes that repeatedly do something while parent is alive
# while kill -s 0 "$parent" 2>&-; do something; done

# Function to snapshot memory usage of a pid to a file once each SAMPLE_INTVL
function sample_statm () {
    local parent=$1 pid=$2
    exec > "$pid".statm
    cd /proc/"$pid"
    while kill -s 0 "$parent" 2>&-; do
        echo -n "$EPOCHSECONDS "
        cat statm
        sleep "$SAMPLE_INTVL"
    done
}

function get_max_memory () {
    local file=$1 column=$2
    local -n _max=$3 _time=$4
    local -a f
    _max=0
    while read -ra f; do
        if (( f[column] > _max )); then
            _max=${f[$column]} _time=${f[0]}
        fi
    done < "$file"
}

function main () {
    set -eu
    DURATION=${DURATION:-60}
    SAMPLE_INTVL=${SAMPLE_INTVL:-1}
    local -i start_time=$EPOCHSECONDS
    local pid

    if (( ! $# )); then
        echo "usage: $0 pid [pid...]" >&2
        exit 1
    fi

    sleep "$DURATION" &
    local sleep_pid=$(jobs -p %sleep)
    for pid in "$@"; do
        sample_statm "$sleep_pid" "$pid" &
    done

    # Of the background processes, all but sleep self-terminate.
    # At shell exit, kill sleep.  The "or no-op" preserves the prior exit value.
    trap 'kill %sleep 2>&- || :' EXIT
    if ! wait -n; then
        echo "child error; exit 1" >&2
        exit 1
    fi

    local -i max_memory max_time
    for pid in "$@"; do
        get_max_memory "$pid".statm 2 max_memory max_time
        echo "$pid: max RSS: $max_memory at $(( max_time - start_time )) seconds"
    done
}

main "$@"

Trace files illuminate the sometimes vexing nuances of errexit.

# capture a trace of the current script
# $1 is an optional directory to contain the output file
# opens filehandle 4 for append
function start_trace () {
    local dir=${1:-.}
    mkdir -p "$dir"
    exec 4>> "$dir/xtrace"
    BASH_XTRACEFD=4
    PS4='+$$@$LINENO: '
    set -x
    date >&4
    declare -p >&4
}

sed

sed is a non-interactive editor that uses regular expressions to transform text. Many bash scripts use sed making it worthwhile to cover less-used sed functions and idioms.

function/idiom description
b label branch to label; if label is omitted, branch to end of script.
h H copy/append pattern space to hold space.
g G copy/append hold space to pattern space.
x exchange the contents of the hold and pattern spaces.
n N read/append the next line of input into the pattern space.
sed -n '/pat/{h; n; G; p}' print a matching line and the following line in reverse order
ls | sed 'h; s/foo/bar/; x; G; s/\n/ /; s/^/mv /' | sh rename files
sed -n '/^$/{x; s/^\n//; s/\n/\t/g; p; b}; H' convert stanzas to tab-separated values (input must end with a blank line)

sed examples

    # handle bash-like kvp, comments, and reasonable quoting
    sed -rn -e 's/^\s*//; /^\w+=./!d; s/=/\t/;' \
            -e "/^\w+\t[^'\"]/{s/#.*//; /['\"]/d; s/\s*$//; p; b}" \
            -e "/^\w+\t'/{s/'//; s/'.*//p; b}" \
            -e 's/"//; s/\\"/\n/g; /"/!d; s/".*//; s/\n/"/g; p' "$file" |
    while IFS=$'\t' read -r key val; do
            dat[$key]=$val
    done
operation description
s/^\s*// remove leading whitespace
/^\w+=./!d drop line unless the beginning matches at least one word char followed by = and one more char
s/=/\t/ replace = with tab
/^\w+\t[^'"]/{ enter block if line begins with \w+\t followed by one non-quote char
    s/#.*// remove # and anything after it
    /['"]/d drop line if it has any quote char
    s/\s*$// remove trailing whitespace
    p print the line
    b skip the rest of the script
} end the block
/^\w+\t'/{ enter block if line begins with \w+\t followed by a single-quote char
    s/'// remove a single-quote char
    s/'.*//p remove ' and anything after it; print the line only if the substitution succeeded
    b skip the rest of the script
} end the block
s/"// by elimination the line must match /^\w+\t"/; remove a double-quote char
s/\\"/\n/g replace any "escaped" double-quote with a newline char; undo this later
/"/!d drop the line unless there is a closing double-quote char
s/".*// remove " and anything after it
s/\n/"/g replace any newline char with a double-quote to undo the earlier replacement
p print the line

NB. The table above discards the semicolons and quotes necessary in the bash script.

Tools

ShellCheck

Available from https://www.shellcheck.net/ and packaged in many Linux distributions, shellcheck runs lint-like checks for common bash pitfalls. The following additon to the top of a script enables all optional checks except two:

# Enable all optional shellcheck checks except the
# two that encourage superfluous braces and quoting.
# shellcheck enable=all
# shellcheck disable=SC2248,SC2250

Kcov

kcov provides line-based code coverage reports for bash scripts and many other languages. Find it at https://simonkagstrom.github.io/kcov/ or packaged from your Linux distribution.

ShellMetrics

From https://github.com/shellspec/shellmetrics, this tool measures cyclomatic complexity for shell scripts.

BASH Debugger

Single step through a bash script, set breakpoints, show a backtrace - all this and more from https://bashdb.sourceforge.net/
NB. For versions of bash newer than 5.0, try building from the matching branch of the git repo.

Further Reading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment