Introduction
# Shell Scripting
Write reliable, maintainable bash scripts. Covers argument parsing, error handling, portability, temp files, parallel execution, process management, and self-documenting scripts.
## When to Use
- Writing scripts that others (or future you) will run - Automating multi-step workflows - Parsing command-line arguments with flags and options - Handling errors and cleanup properly - Running tasks in parallel - Making scripts portable across Linux and macOS - Wrapping complex commands with a simpler interface
## Script Template
```bash #!/usr/bin/env bash set -euo pipefail
# Description: What this script does (one line) # Usage: script.sh [options] <required-arg>
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" readonly SCRIPT_NAME="$(basename "$0")"
# Defaults VERBOSE=false OUTPUT_DIR="./output"
usage() { cat <<EOF Usage: $SCRIPT_NAME [options] <input-file>
Description: Process the input file and generate output.
Options: -o, --output DIR Output directory (default: $OUTPUT_DIR) -v, --verbose Enable verbose output -h, --help Show this help message
Examples: $SCRIPT_NAME data.csv $SCRIPT_NAME -v -o /tmp/results data.csv EOF }
log() { echo "[$(date '+%H:%M:%S')] $*" >&2; } debug() { $VERBOSE && log "DEBUG: $*" || true; } die() { log "ERROR: $*"; exit 1; }
# Parse arguments while [[ $# -gt 0 ]]; do case "$1" in -o|--output) OUTPUT_DIR="$2"; shift 2 ;; -v|--verbose) VERBOSE=true; shift ;; -h|--help) usage; exit 0 ;; --) shift; break ;; -*) die "Unknown option: $1" ;; *) break ;; esac done
INPUT_FILE="${1:?$(usage >&2; echo "Error: input file required")}" [[ -f "$INPUT_FILE" ]] || die "File not found: $INPUT_FILE"
# Main logic main() { debug "Input: $INPUT_FILE" debug "Output: $OUTPUT_DIR" mkdir -p "$OUTPUT_DIR"
log "Processing $INPUT_FILE..." # ... do work ... log "Done. Output in $OUTPUT_DIR" }
main "$@" ```
## Error Handling
### set flags
```bash set -e # Exit on any command failure set -u # Error on undefined variables set -o pipefail # Pipe fails if any command in the pipe fails set -x # Debug: print each command before executing (noisy)
# Combined (use this in every script) set -euo pipefail
# Temporarily disable for commands that are allowed to fail set +e some_command_that_might_fail exit_code=$? set -e ```
### Trap for cleanup
```bash # Cleanup on exit (any exit: success, failure, or signal) TMPDIR="" cleanup() { [[ -n "$TMPDIR" ]] && rm -rf "$TMPDIR" } trap cleanup EXIT
TMPDIR=$(mktemp -d) # Use $TMPDIR freely — it's cleaned up automatically
# Trap specific signals trap 'echo "Interrupted"; exit 130' INT # Ctrl+C trap 'echo "Terminated"; exit 143' TERM # kill ```
### Error handling patterns
```bash # Check command exists before using it command -v jq >/dev/null 2>&1 || die "jq is required but not installed"
# Provide default values NAME="${NAME:-default_value}"
# Required variable (fail if unset) : "${API_KEY:?Error: API_KEY environment variable is required}"
# Retry a command retry() { local max_attempts=$1 shift local attempt=1 while [[ $attempt -le $max_attempts ]]; do "$@" && return 0 log "Attempt $attempt/$max_attempts failed. Retrying..." ((attempt++)) sleep $((attempt * 2)) done die "Command failed after $max_attempts attempts: $*" }
retry 3 curl -sf https://api.example.com/health ```
## Argument Parsing
### Simple: positional + flags
```bash # Manual parsing (no dependencies) FORCE=false DRY_RUN=false
while [[ $# -gt 0 ]]; do case "$1" in -f|--force) FORCE=true; shift ;; -n|--dry-run) DRY_RUN=true; shift ;; -o|--output) [[ -n "${2:-}" ]] || die "--output requires a value" OUTPUT="$2"; shift 2 ;; --output=*) OUTPUT="${1#*=}"; shift ;; -h|--help) usage; exit 0 ;; --) shift; break ;; # End of options -*) die "Unknown option: $1" ;; *) break ;; # Start of positional args esac done
# Remaining args are positional FILES=("$@") [[ ${#FILES[@]} -gt 0 ]] || die "At least one file is required" ```
### getopts (POSIX, short options only)
```bash while getopts ":o:vhf" opt; do case "$opt" in o) OUTPUT="$OPTARG" ;; v) VERBOSE=true ;; f) FORCE=true ;; h) usage; exit 0 ;; :) die "Option -$OPTARG requires an argument" ;; ?) die "Unknown option: -$OPTARG" ;; esac done shift $((OPTIND - 1)) ```
## Temp Files and Directories
```bash # Create temp file (automatically unique) TMPFILE=$(mktemp) echo "data" > "$TMPFILE"
# Create temp directory TMPDIR=$(mktemp -d)
# Create temp with custom prefix/suffix TMPFILE=$(mktemp /tmp/myapp.XXXXXX) TMPFILE=$(mktemp --suffix=.json) # GNU only
# Always clean up with trap trap 'rm -f "$TMPFILE"' EXIT
# Portable pattern (works on macOS and Linux) TMPDIR=$(mktemp -d 2>/dev/null || mktemp -d -t 'myapp') trap 'rm -rf "$TMPDIR"' EXIT ```
## Parallel Execution
### xargs -P
```bash # Run 4 commands in parallel cat urls.txt | xargs -P 4 -I {} curl -sO {}
# Process files in parallel (4 at a time) find . -name "*.csv" | xargs -P 4 -I {} ./process.sh {}
# Parallel with progress indicator find . -name "*.jpg" | xargs -P 8 -I {} sh -c 'convert {} -resize 800x600 resized/{} && echo "Done: {}"' ```
### Background jobs + wait
```bash # Run tasks in background, wait for all pids=() for file in data/*.csv; do process_file "$file" & pids+=($!) done
# Wait for all and check results failed=0 for pid in "${pids[@]}"; do wait "$pid" || ((failed++)) done [[ $failed -eq 0 ]] || die "$failed jobs failed" ```
### GNU Parallel (if available)
```bash # Process files with 8 parallel jobs parallel -j 8 ./process.sh {} ::: data/*.csv
# With progress bar parallel --bar -j 4 convert {} -resize 800x600 resized/{/} ::: *.jpg
# Pipe input lines cat urls.txt | parallel -j 10 curl -sO {} ```
## Process Management
### Background processes
```bash # Start in background long_running_command & BG_PID=$!
# Check if still running kill -0 $BG_PID 2>/dev/null && echo "Running" || echo "Stopped"
# Wait for it wait $BG_PID echo "Exit code: $?"
# Kill on script exit trap 'kill $BG_PID 2>/dev/null' EXIT ```
### Process supervision
```bash # Run a command, restart if it dies run_with_restart() { local cmd=("$@") while true; do "${cmd[@]}" & local pid=$! log "Started PID $pid" wait $pid local exit_code=$? log "Process exited with code $exit_code. Restarting in 5s..." sleep 5 done }
run_with_restart ./my-server --port 8080 ```
### Timeout
```bash # Kill command after 30 seconds timeout 30 long_running_command
# With custom signal (SIGKILL after SIGTERM fails) timeout --signal=TERM --kill-after=10 30 long_running_command
# Portable (no timeout command) ( sleep 30; kill $$ 2>/dev/null ) & TIMER_PID=$! long_running_command kill $TIMER_PID 2>/dev/null ```
## Portability (Linux vs macOS)
### Common differences
```bash # sed: macOS requires -i '' (empty backup extension) # Linux: sed -i 's/old/new/g' file.txt # macOS: sed -i '' 's/old/new/g' file.txt # Portable: sed -i.bak 's/old/new/g' file.txt && rm file.txt.bak
# date: different flags # GNU (Linux): date -d '2026-02-03' '+%s' # BSD (macOS): date -j -f '%Y-%m-%d' '2026-02-03' '+%s'
# readlink -f: doesn't exist on macOS # Portable alternative: real_path() { cd "$(dirname "$1")" && echo "$(pwd)/$(basename "$1")"; }
# stat: different syntax # GNU: stat -c '%s' file # BSD: stat -f '%z' file
# grep -P: not available on macOS by default # Use grep -E instead, or install GNU grep ```
### POSIX-safe patterns
```bash # Use printf instead of echo -e (echo behavior varies) printf "Line 1\nLine 2\n"
# Use $() instead of backticks result=$(command) # Good result=`command` # Bad (deprecated, nesting issues)
# Use [[ ]] for tests (bash), [ ] for POSIX sh [[ -f "$file" ]] # Bash (safer, no word splitting) [ -f "$file" ] # POSIX sh
# Array check (bash only, not POSIX) if [[ ${#array[@]} -gt 0 ]]; then echo "Array has elements" fi ```
## Config File Parsing
### Source a config file
```bash # Simple: source a key=value file # config.env: # DB_HOST=localhost # DB_PORT=5432
# Validate before sourcing (security: check for commands) if grep -qP '^[A-Z_]+=.*[;\`\$\(]' config.env; then die "Config file contains unsafe characters" fi source config.env ```
### Parse INI-style config
```bash # config.ini: # [database] # host = localhost # port = 5432 # [app] # debug = true
parse_ini() { local file="$1" section="" while IFS='= ' read -r key value; do [[ -z "$key" || "$key" =~ ^[#\;] ]] && continue if [[ "$key" =~ ^\[(.+)\]$ ]]; then section="${BASH_REMATCH[1]}" continue fi value="${value%%#*}" # Strip inline comments value="${value%"${value##*[![:space:]]}"}" # Trim trailing whitespace printf -v "${section}_${key}" '%s' "$value" done < "$file" }
parse_ini config.ini echo "$database_host" # localhost echo "$app_debug" # true ```
## Useful Patterns
### Confirm before destructive action
```bash confirm() { local prompt="${1:-Are you sure?}" read -rp "$prompt [y/N] " response [[ "$response" =~ ^[Yy]$ ]] }
confirm "Delete all files in /tmp/data?" || die "Aborted" rm -rf /tmp/data/* ```
### Progress indicator
```bash # Simple counter total=$(wc -l < file_list.txt) count=0 while IFS= read -r file; do ((count++)) printf "\rProcessing %d/%d..." "$count" "$total" >&2 process "$file" done < file_list.txt echo "" >&2 ```
### Lock file (prevent concurrent runs)
```bash LOCKFILE="/tmp/${SCRIPT_NAME}.lock"
acquire_lock() { if ! mkdir "$LOCKFILE" 2>/dev/null; then die "Another instance is running (lock: $LOCKFILE)" fi trap 'rm -rf "$LOCKFILE"' EXIT }
acquire_lock # ... safe to proceed, only one instance runs ... ```
### Stdin or file argument
```bash # Read from file argument or stdin input="${1:--}" # Default to "-" (stdin) if [[ "$input" == "-" ]]; then cat else cat "$input" fi | while IFS= read -r line; do process "$line" done ```
## Tips
- Always start with `set -euo pipefail`. It catches 80% of silent bugs. - Always use `trap cleanup EXIT` for temp files. Never rely on reaching the cleanup code at the end. - Quote all variable expansions: `"$var"` not `$var`. Unquoted variables break on spaces and globs. - Use `[[ ]]` instead of `[ ]` in bash. It handles empty strings, spaces, and pattern matching better. - `shellcheck` is the best linter for shell scripts. Run it: `shellcheck myscript.sh`. Install it if available. - `readonly` for constants prevents accidental overwrite: `readonly DB_HOST="localhost"`. - Write a `usage()` function and call it on `-h`/`--help` and on missing required arguments. Future users (including you) will thank you. - Prefer `printf` over `echo` for anything that might contain special characters or needs formatting. - Test scripts with `bash -n script.sh` (syntax check) before running.