--- title: "Bash Retry Function: Automatically Retry Failing Commands with Exponential Backoff" description: "A reusable Bash function that retries any failing command with configurable attempts and exponential backoff. Ideal for wrapping flaky network calls, AWS CLI commands, or deployment scripts in CI/CD pipelines." author: "Mohammad Abu Mattar" canonical: https://mkabumattar.com/codesnippets/post/bash-retry-function-exponential-backoff --- # Bash Retry Function: Automatically Retry Failing Commands with Exponential Backoff **Quick Tip** Wrap any flaky command in one reusable `retry` function and stop re-running red pipelines by hand. ## The Problem **The Problem** Some commands fail for reasons that have nothing to do with your code. A registry hiccups, an API rate-limits you for a second, a DNS lookup blips, an AWS endpoint returns a `503`. Run the exact same command again and it works. That's a transient failure, and it's everywhere in scripts that touch the network. ### Why blind retries make it worse The naive fix is a quick loop that retries immediately, but that often makes things worse. If a service is already struggling, a tight retry loop hammers it harder. Worse, if a hundred CI jobs all fail at the same moment and all retry at the same moment, they come back in a synchronized wave. That's the thundering herd, and it can keep a recovering service down. ### The real-world impact The everyday cost is a pipeline that goes red for no real reason, so someone re-runs it by hand and loses ten minutes and their focus. The worse cost is a deploy script that gives up on the first blip and leaves a release half-applied. Both come from treating a temporary failure as a permanent one. ## The Solution **The Fix** Retry the command a few times, but wait longer between each attempt, and add a little randomness so retries spread out instead of stacking up. That's exponential backoff with jitter, and it fits in one small Bash function you can wrap around anything. **TL;DR** - Retry any command with a configurable number of attempts. - Back off exponentially (1s, 2s, 4s, 8s) so you stop hammering a struggling service. - Add jitter so a fleet of scripts doesn't retry in lockstep. ## Script Implementation ### Setup and defaults Start with a safe shell, then read the knobs from the environment so the same function works in a laptop shell and a CI job without editing the code. ```shell title="retry.sh" showLineNumbers #!/usr/bin/env bash # retry.sh - retry a command with exponential backoff and jitter. set -euo pipefail # Tunables (override via environment): : "${RETRY_MAX_ATTEMPTS:=5}" # how many tries before giving up : "${RETRY_BASE_DELAY:=1}" # first backoff, in seconds : "${RETRY_MAX_DELAY:=60}" # cap so backoff never runs away ``` ### The retry function The function takes the command and its arguments as `"$@"`, so it wraps anything without quoting gymnastics. It returns the command's own exit code on success, and the last exit code if it runs out of attempts. ```shell retry() { local attempt=1 while true; do # Run the command exactly as passed. On success, we're done. if "$@"; then return 0 fi local exit_code=$? # Out of attempts: surface the failure clearly and pass the code up. if ((attempt >= RETRY_MAX_ATTEMPTS)); then echo "retry: '$*' failed after ${attempt} attempts (exit ${exit_code})" >&2 return "$exit_code" fi # Exponential backoff: base * 2^(attempt-1), capped at max delay. local delay=$((RETRY_BASE_DELAY * 2 ** (attempt - 1))) ((delay > RETRY_MAX_DELAY)) && delay=$RETRY_MAX_DELAY # Full jitter: sleep a random amount between 0 and delay. local wait=$((RANDOM % (delay + 1))) echo "retry: attempt ${attempt}/${RETRY_MAX_ATTEMPTS} failed (exit ${exit_code}); waiting ${wait}s" >&2 sleep "$wait" ((attempt++)) done } ``` ### Why the jitter matters The backoff line grows the wait each round, and the cap stops it from turning a fifth attempt into a multi-minute stall. The jitter line is the part people skip, and it's the one that prevents the thundering herd. Instead of every caller waiting exactly 4 seconds and retrying at the same instant, each one waits a random slice of that window, so the load spreads out and the recovering service gets room to breathe. ### Entry point If you run the file directly it retries whatever you pass on the command line. If you source it, you just get the `retry` function. ```shell # Run directly: retry.sh if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then if (($# == 0)); then echo "usage: retry.sh [args...]" >&2 exit 64 fi retry "$@" fi ``` ## Usage and Benefits **Why This Helps** One function covers every flaky command in your scripts, so you stop copy-pasting `sleep` loops and you get consistent logging for free. Because it wraps `"$@"`, it does not care whether the command is `aws`, `curl`, `kubectl`, or a script of your own. ### Real invocations ```shell # Wrap an AWS CLI upload that sometimes hits throttling. retry aws s3 cp ./build.tar.gz s3://artifacts/build.tar.gz # Wrap a health check that races a service starting up. retry curl -fsS https://api.example.com/health # Tune it inline for a slow, important step. RETRY_MAX_ATTEMPTS=8 RETRY_BASE_DELAY=2 retry ./deploy.sh ``` ### Sourcing it in CI/CD Keep `retry.sh` in a shared scripts library and source it at the top of your pipeline steps, so every job gets the same behavior. ```shell source ./scripts/retry.sh retry terraform apply -auto-approve ``` ## Comparison How the function compares to the other common options. | Approach | Backoff | Jitter | Works for any command | | :----------------------- | :------------------ | :------------------------------------ | :-------------------- | | `retry()` function | Yes, exponential | Yes | Yes | | Tight `until` loop | No | No | Yes | | `curl --retry N` | Yes (curl 7.66+) | Only with `--retry-all-errors` extras | No, curl only | | AWS CLI built-in retries | Yes (adaptive mode) | Yes | No, AWS CLI only | Tool-specific retries are great when you're already in that tool, but the function is the one thing that wraps all of them the same way. ## Frequently Asked Questions > **Why use exponential backoff instead of a fixed delay?** A fixed delay treats a one-second blip and a thirty-second outage the same way. Exponential backoff waits a little after the first failure and progressively longer after each one, so you recover fast from a quick hiccup but stop hammering a service that's genuinely down. The cap (`RETRY_MAX_DELAY`) keeps the later waits from becoming a stall. > **What is jitter and why does it prevent a thundering herd?** Jitter is deliberate randomness added to the wait time. Without it, every script that failed at the same moment retries at the same moment, hitting the recovering service in a synchronized wave. With full jitter, each caller sleeps a random amount between zero and the backoff window, so the retries spread out and the load smooths. > **How do I retry only on certain failures, not every non-zero exit?** Wrap the command in a small function that maps the failures you care about to a non-zero exit and everything else to zero. For example, treat a `curl` exit of 22 (HTTP error) or 28 (timeout) as retryable and return success for the rest, then pass that wrapper to `retry`. That keeps the retry loop generic while you decide what "failure" means. > **Does this work in POSIX sh or only Bash?** It uses Bash features: `RANDOM`, `local`, `((...))` arithmetic with `**`, and `BASH_SOURCE`. In a strict POSIX `sh` you'd swap `RANDOM` for something like `awk` or `/dev/urandom`, replace `local`, and compute the power manually. If you can run `#!/usr/bin/env bash`, keep it as is; it's simpler and clearer. > **How is this different from curl --retry or the AWS CLI's own retries?** Those are excellent inside their own tool. `curl --retry` retries HTTP calls; the AWS CLI has adaptive retry modes for API throttling. The Bash function is the layer above: it retries anything, including your own scripts and combinations of commands, with one consistent policy and one consistent log format. > **How do I cap the total time spent retrying?** Either lower `RETRY_MAX_ATTEMPTS` and `RETRY_MAX_DELAY` so the worst case is bounded, or wrap the call in GNU `timeout`, for example `timeout 120 retry ./deploy.sh`. The `timeout` approach gives you a hard ceiling regardless of how the backoff math works out. ## References - [Bash Reference Manual: shell arithmetic](https://www.gnu.org/software/bash/manual/bash.html#Shell-Arithmetic) - [Bash Reference Manual: RANDOM and special parameters](https://www.gnu.org/software/bash/manual/bash.html#Bash-Variables) - [AWS Architecture Blog: exponential backoff and jitter](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/) - [AWS CLI: retries](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-retries.html) - [curl manual: --retry](https://curl.se/docs/manpage.html#--retry) - [GNU coreutils: timeout](https://www.gnu.org/software/coreutils/manual/html_node/timeout-invocation.html)