Mohammad Abu Mattar | Code Snippets

Bash Retry Function: Automatically Retry Failing Commands with Exponential Backoff

Mohammad Abu Mattar — Mon, 20 Jul 2026 12:29:14 GMT

Quick Tip

Wrap any flaky command in one reusable retry function and stop re-running red pipelines by hand.

The Problem

The Problem

Some commands fail for reasons that have nothing to do with your code. A registry hiccups, an API rate-limits you for a second, a DNS lookup blips, an AWS endpoint returns a 503. Run the exact same command again and it works. That's a transient failure, and it's everywhere in scripts that touch the network.

Why blind retries make it worse

The naive fix is a quick loop that retries immediately, but that often makes things worse. If a service is already struggling, a tight retry loop hammers it harder. Worse, if a hundred CI jobs all fail at the same moment and all retry at the same moment, they come back in a synchronized wave. That's the thundering herd, and it can keep a recovering service down.

The real-world impact

The everyday cost is a pipeline that goes red for no real reason, so someone re-runs it by hand and loses ten minutes and their focus. The worse cost is a deploy script that gives up on the first blip and leaves a release half-applied. Both come from treating a temporary failure as a permanent one.

The Solution

The Fix

Retry the command a few times, but wait longer between each attempt, and add a little randomness so retries spread out instead of stacking up. That's exponential backoff with jitter, and it fits in one small Bash function you can wrap around anything.

TL;DR

Retry any command with a configurable number of attempts.
Back off exponentially (1s, 2s, 4s, 8s) so you stop hammering a struggling service.
Add jitter so a fleet of scripts doesn't retry in lockstep.

Script Implementation

Setup and defaults

Start with a safe shell, then read the knobs from the environment so the same function works in a laptop shell and a CI job without editing the code.

#!/usr/bin/env bash
# retry.sh - retry a command with exponential backoff and jitter.
set -euo pipefail

# Tunables (override via environment):
: "${RETRY_MAX_ATTEMPTS:=5}"   # how many tries before giving up
: "${RETRY_BASE_DELAY:=1}"     # first backoff, in seconds
: "${RETRY_MAX_DELAY:=60}"     # cap so backoff never runs away

The retry function

The function takes the command and its arguments as "$@", so it wraps anything without quoting gymnastics. It returns the command's own exit code on success, and the last exit code if it runs out of attempts.

retry() {
  local attempt=1
  while true; do
    # Run the command exactly as passed. On success, we're done.
    if "$@"; then
      return 0
    fi
    local exit_code=$?

    # Out of attempts: surface the failure clearly and pass the code up.
    if ((attempt >= RETRY_MAX_ATTEMPTS)); then
      echo "retry: '$*' failed after ${attempt} attempts (exit ${exit_code})" >&2
      return "$exit_code"
    fi

    # Exponential backoff: base * 2^(attempt-1), capped at max delay.
    local delay=$((RETRY_BASE_DELAY * 2 ** (attempt - 1)))
    ((delay > RETRY_MAX_DELAY)) && delay=$RETRY_MAX_DELAY

    # Full jitter: sleep a random amount between 0 and delay.
    local wait=$((RANDOM % (delay + 1)))
    echo "retry: attempt ${attempt}/${RETRY_MAX_ATTEMPTS} failed (exit ${exit_code}); waiting ${wait}s" >&2
    sleep "$wait"
    ((attempt++))
  done
}

Why the jitter matters

The backoff line grows the wait each round, and the cap stops it from turning a fifth attempt into a multi-minute stall. The jitter line is the part people skip, and it's the one that prevents the thundering herd. Instead of every caller waiting exactly 4 seconds and retrying at the same instant, each one waits a random slice of that window, so the load spreads out and the recovering service gets room to breathe.

Entry point

If you run the file directly it retries whatever you pass on the command line. If you source it, you just get the retry function.

# Run directly: retry.sh <command...>
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
  if (($# == 0)); then
    echo "usage: retry.sh <command> [args...]" >&2
    exit 64
  fi
  retry "$@"
fi

Usage and Benefits

Why This Helps

One function covers every flaky command in your scripts, so you stop copy-pasting sleep loops and you get consistent logging for free. Because it wraps "$@", it does not care whether the command is aws, curl, kubectl, or a script of your own.

Real invocations

# Wrap an AWS CLI upload that sometimes hits throttling.
retry aws s3 cp ./build.tar.gz s3://artifacts/build.tar.gz

# Wrap a health check that races a service starting up.
retry curl -fsS https://api.example.com/health

# Tune it inline for a slow, important step.
RETRY_MAX_ATTEMPTS=8 RETRY_BASE_DELAY=2 retry ./deploy.sh

Sourcing it in CI/CD

Keep retry.sh in a shared scripts library and source it at the top of your pipeline steps, so every job gets the same behavior.

source ./scripts/retry.sh
retry terraform apply -auto-approve

Comparison

How the function compares to the other common options.

Approach	Backoff	Jitter	Works for any command
`retry()` function	Yes, exponential	Yes	Yes
Tight `until` loop	No	No	Yes
`curl --retry N`	Yes (curl 7.66+)	Only with `--retry-all-errors` extras	No, curl only
AWS CLI built-in retries	Yes (adaptive mode)	Yes	No, AWS CLI only

Tool-specific retries are great when you're already in that tool, but the function is the one thing that wraps all of them the same way.

Frequently Asked Questions

<Accordion client:load title="Why use exponential backoff instead of a fixed delay?"

A fixed delay treats a one-second blip and a thirty-second outage the same way. Exponential backoff waits a little after the first failure and progressively longer after each one, so you recover fast from a quick hiccup but stop hammering a service that's genuinely down. The cap (RETRY_MAX_DELAY) keeps the later waits from becoming a stall.

<Accordion client:load title="What is jitter and why does it prevent a thundering herd?"

Jitter is deliberate randomness added to the wait time. Without it, every script that failed at the same moment retries at the same moment, hitting the recovering service in a synchronized wave. With full jitter, each caller sleeps a random amount between zero and the backoff window, so the retries spread out and the load smooths.

<Accordion client:load title="How do I retry only on certain failures, not every non-zero exit?"

Wrap the command in a small function that maps the failures you care about to a non-zero exit and everything else to zero. For example, treat a curl exit of 22 (HTTP error) or 28 (timeout) as retryable and return success for the rest, then pass that wrapper to retry. That keeps the retry loop generic while you decide what "failure" means.

<Accordion client:load title="Does this work in POSIX sh or only Bash?"

It uses Bash features: RANDOM, local, ((...)) arithmetic with **, and BASH_SOURCE. In a strict POSIX sh you'd swap RANDOM for something like awk or /dev/urandom, replace local, and compute the power manually. If you can run #!/usr/bin/env bash, keep it as is; it's simpler and clearer.

<Accordion client:load title="How is this different from curl --retry or the AWS CLI's own retries?"

Those are excellent inside their own tool. curl --retry retries HTTP calls; the AWS CLI has adaptive retry modes for API throttling. The Bash function is the layer above: it retries anything, including your own scripts and combinations of commands, with one consistent policy and one consistent log format.

<Accordion client:load title="How do I cap the total time spent retrying?"

Either lower RETRY_MAX_ATTEMPTS and RETRY_MAX_DELAY so the worst case is bounded, or wrap the call in GNU timeout, for example timeout 120 retry ./deploy.sh. The timeout approach gives you a hard ceiling regardless of how the backoff math works out.

References

Node.js Environment Variable Validation with Zod at Startup

Mohammad Abu Mattar — Wed, 27 May 2026 20:02:59 GMT

Most Node.js apps treat process.env like a trusted friend. You reach into it whenever you need a value, assume the key is there, assume it's spelled right, and assume the string is actually the type you want. Since process.env hands you everything as a plain string, you end up parsing ports into integers and turning "false" into a real boolean all over the codebase.

The fix is small and boring in the best way: validate your environment once, at startup, with a schema. If the config is wrong, the app refuses to boot and tells you exactly what's broken before a single request is served. Here's how to do it with Zod and a little bit of TypeScript.

The Problem

Why does reading straight from process.env keep biting you?

Silent Configuration Drift

The Problem When you build traditional Node.js apps, it's incredibly common to grab values straight from process.env whenever and wherever you need them. You probably load these into your environment from a local file, but that setup never actually checks if your data makes sense. Your code just assumes every setting is present, spelled correctly, and formatted right. And because process.env treats everything as a string, you find yourself parsing variables by hand all over the place. That's where subtle bugs creep in: maybe you forgot to parse a port into an integer, or a string that says "false" gets evaluated as truthy.

The Real-World Impact

When these settings break, your application usually doesn't crash right away. Instead, you get painful runtime errors much later. Miss a database URL or an API key, and your server might start up perfectly fine. It's only hours later, when a user triggers a specific request, that the app suddenly blows up. These delayed failures make debugging a nightmare because the crash happens far from the actual mistake.

Hunting for a one-character typo in your environment config during a live production outage is exactly the kind of 2 AM stress nobody signed up for. The earlier you catch it, the cheaper it is.

The Solution

How do you catch bad config before the app even starts?

The Startup Schema Assertion

To put an end to this, set up a validation schema that runs the second your application boots. With Zod, you write a clear, strict schema describing exactly what your environment should look like. This runs before any of your server logic starts, so an invalid configuration won't let the app launch at all. Zod acts like a smart gateway: it checks the values and automatically converts strings into real numbers, booleans, or objects.

It helps to picture it as a mapping. Your raw environment is just string keys paired with string values pulled from :

Zod's parser maps that loose, unstructured data into a strictly typed configuration space :

where is the valid typed domain for each key, such as for ports, a format-compliant URI for the database connection, or a value from a fixed enum.

**The Fix:** one schema, verified the moment your app turns on. The rest of your system then works with safe, fully typed values instead of messy raw strings.

Quick Takeaways

TL;DR

Crash early and safely: the app won't boot if any required setting is missing or invalid.
Automatic type conversion: raw strings become real numbers, booleans, and validated URLs, fully typed for TypeScript.
Single source of truth: all configuration checks live in one place, so validation logic isn't scattered everywhere.

Where This Fits in Your Project

Before the code, here's where each piece lives.

your-app/
- src/
  - env.ts the schema and the validated env export
  - index.ts imports env at the very top
  - server.ts uses the typed config
- .env local values, gitignored
- .env.example documented keys, committed
- package.json

The whole trick is that env.ts runs its validation on import, so importing it from your entry file is what makes a broken config stop the boot.

Building the Validator

Four small pieces, wired together once.

First, install Zod (and dotenv so you can load a local .env while developing):

npm install zod dotenv

pnpm add zod dotenv

yarn add zod dotenv

bun add zod dotenv

Now build it up step by step.

Load the local environment. Pull .env into process.env before anything else runs, and bring in Zod.

import dotenv from 'dotenv';
import {z} from 'zod';

// Load our local .env file before we validate anything
dotenv.config();

Declare a strict schema. Spell out every variable you need and the type you expect. Zod handles strings, coerced numbers, booleans, and custom enums for you, and z.infer gives you a matching TypeScript type for free.

// Define the validation rules and expected types for our variables
export const envSchema = z.object({
  NODE_ENV: z
    .enum(['development', 'production', 'staging', 'test'])
    .default('development'),
  PORT: z.coerce.number().min(1).max(65535).default(3000),
  DATABASE_URL: z
    .string()
    .url({message: 'DATABASE_URL must be a valid connection URL'}),
  JWT_SECRET: z
    .string()
    .min(32, {message: 'JWT_SECRET must be at least 32 characters long'}),
  ENABLE_LOGS: z.preprocess((val) => val === 'true', z.boolean()).default(true),
});

// Infer the strict TypeScript type directly from our schema
export type EnvConfig = z.infer<typeof envSchema>;

Parse once and fail loudly. Run the schema against process.env. If anything is wrong, print a clean list of issues and exit before the server starts.

// Parse the environment and handle any validation failures gracefully
const validateEnv = (): EnvConfig => {
  const result = envSchema.safeParse(process.env);

  if (!result.success) {
    console.error('Environment validation failed during boot:');

    // Print each issue clearly without leaking any sensitive data
    result.error.issues.forEach((issue) => {
      const propertyPath = issue.path.join('.');
      console.error(`  - [${propertyPath}]: ${issue.message}`);
    });

    // Terminate the process immediately
    process.exit(1);
  }

  return result.data;
};

// Export our validated configuration so the rest of the app can use it
export const env = validateEnv();

Import it at the very top of your entry file. Because validation runs on import, a broken config stops the process before app.listen is ever reached.

import {env} from './env';
import express from 'express';

const app = express();

// The port is now guaranteed to be a valid, parsed number
app.listen(env.PORT, () => {
  console.log(`Server is running in ${env.NODE_ENV} mode on port ${env.PORT}`);
});

Usage and Benefits

What do you actually get out of this?

Architectural Advantages

Why This Helps This pattern saves you from writing repetitive null checks and manual parsing logic throughout your codebase. You don't have to wonder whether a variable is undefined or whether a port was correctly parsed into a number. Some developers prefer to extend the global ProcessEnv type in TypeScript for editor autocomplete, but that only helps at compile time. It won't stop a runtime error when a variable is missing or a boolean string is parsed incorrectly. By validating at startup, you get both editor autocomplete and guaranteed runtime safety.

Diagnostic Console Output

If you miss a variable or format one incorrectly, the app halts immediately and prints a neat, readable error message. This feedback makes it easy to see exactly what went wrong so you can fix it right away.

$ npm run dev

Environment validation failed during boot:
  - [DATABASE_URL]: DATABASE_URL must be a valid connection URL
  - [PORT]: Number must be less than or equal to 65535
  - [JWT_SECRET]: JWT_SECRET must be at least 32 characters long

Notice the output shows the field path and the message, never the value. A bad `JWT_SECRET` or `DATABASE_URL` won't end up in your production logs.

Comparing Validation Approaches

Is Zod really the best fit, or just the popular one?

Here's a quick look at how Zod stacks up against other common ways developers handle configuration checks.

Validation Pattern	Type Safety (TypeScript)	Automatic Coercion	Boilerplate Required	Error Output Detail
Zod Schema	Native type inference	High, built-in	Low definition overhead	Detailed, structured issues
Joi Library	Requires manual types	Moderate	Medium validation block	Detailed schema errors
Custom Script	Manual type assertions	Manual conversion code	High custom coding	Whatever you write yourself

Manual scripts and older libraries get the job done, but Zod gives you full TypeScript inference, automatic coercion, and clean error reporting with almost no extra work, which makes it a great fit for modern Node.js setups.

Frequently Asked Questions

<Accordion client:load title="Should I validate environment variables in every file or just once?"

Just once, at startup. Validate in a single env.ts module, export the typed env object, and import that everywhere else instead of reading process.env directly. Because the validation runs on import, the check happens exactly one time as the app boots, and every other file consumes already-safe values.

<Accordion client:load title="How do I handle optional variables or sensible defaults?"

Use Zod's .optional() for values that may be absent and .default(...) for ones that should fall back to a known value. For example, PORT: z.coerce.number().default(3000) means a missing PORT quietly becomes 3000, while DATABASE_URL: z.string().url() stays required and fails the boot if it's missing or malformed.

<Accordion client:load title="Does this give me autocomplete on my config?"

Yes, that's one of the best parts. z.infer<typeof envSchema> produces a TypeScript type that matches your schema exactly, so the exported env object is fully typed: env.PORT is a number, env.NODE_ENV is the enum, and so on. You get autocomplete and compile-time checks without manually maintaining a separate type.

<Accordion client:load title="Won't printing validation errors leak my secrets?"

No. The error handler only logs each issue's path and message (for example [JWT_SECRET]: must be at least 32 characters long), never the actual value. Your secrets stay out of the logs even when validation fails, which keeps the output safe to surface in CI and production.

<Accordion client:load title="Can I use this in a frontend or browser bundle?"

This pattern is built for server-side Node.js. In the browser, secrets should never ship in the bundle at all, so rely on your framework's public-env mechanism instead (like NEXT_PUBLIC_ variables or import.meta.env). Keep this Zod schema on the server, and only expose the handful of values that are genuinely safe for the client.

<Accordion client:load title="How is this different from envalid or Joi?"

The core idea is the same: assert your config against a schema at startup. Zod's edge is native TypeScript inference (no separate type definitions), built-in coercion, and the fact that many projects already use Zod for request and form validation. Reusing one library for both means less to learn and one consistent way to describe data.

References

AWS EC2 Instance Management with Boto3: Start, Stop, and Query Instances

Mohammad Abu Mattar — Tue, 14 Apr 2026 20:02:59 GMT

If you've ever spent 20 minutes clicking through the AWS Console just to stop a handful of dev instances, you already know the pain. It's tedious, it doesn't scale, and one wrong click can ruin your afternoon.

That's where Boto3 comes in. It's the official AWS SDK for Python, and it lets you talk to AWS services, including EC2, directly from your code. Instead of pointing and clicking, you write a script once and run it whenever you need it. Your code doesn't get tired, doesn't misclick, and works the same at 3 AM as it does at 3 PM.

This guide covers everything you need to get comfortable with EC2 automation: setting up authentication, starting and stopping instances, waiting for state changes, filtering by tags, running bulk operations, and handling errors like a pro.

Why Automate EC2 Management with Boto3?

Why bother writing code when the AWS Console already does the job?

Fair question. The console is fine when you've got two or three instances. But once you're managing a real environment, manual work starts costing you more than time.

Think about it: every action you take through the console depends on a human doing it right, every single time. With Boto3, you write the logic once, and it runs consistently regardless of who's on call or how tired they are.

Here's where automation pays off most:

Scheduled shutdowns: Stop dev and staging instances overnight or on weekends. There's no reason to pay for servers that nobody's using at 2 AM.
Consistent tagging: Every instance your script touches gets tagged correctly. No more missing Environment or Owner tags because someone forgot.
Incident response: When something goes wrong, your runbook can stop a compromised instance or spin up a replacement without waiting for a human to log in.
Audit trails: Log every action with timestamps, so you've got a clear record of what happened, when, and why.
Fewer mistakes: Repetitive manual work breeds errors. Scripts don't forget steps or click the wrong button.

Once your automation is in place, managing 200 instances takes roughly the same effort as managing two.

Setting Up Boto3 and Authenticating with AWS

How do you connect Boto3 to your AWS account securely?

Start with the install:

pip install boto3

Now, authentication. This is where a lot of people make mistakes, so it's worth doing right from the start. The method you use should depend on where your code runs.

Option 1: IAM Roles (Best for EC2 and Lambda)

If your script runs on an EC2 instance or a Lambda function, attach an IAM role to that resource. Boto3 picks up the credentials automatically from the instance metadata service. You don't need to touch a key file or set any environment variables.

import boto3

# No keys needed here Boto3 reads the IAM role attached to this instance
ec2_client = boto3.client('ec2', region_name='us-east-1')

# Quick check to confirm the connection works
response = ec2_client.describe_instances()
print('Connected via IAM role good to go.')

This is the cleanest approach for production. There are no credentials to rotate, leak, or accidentally commit to Git.

Option 2: Named Profiles (Best for Local Development)

When you're working locally, use a named AWS CLI profile. It keeps credentials in ~/.aws/credentials where they belong, not in your source code.

# Run this once in your terminal to set up a profile
aws configure --profile myproject

import boto3

# Load credentials from the 'myproject' profile in ~/.aws/credentials
session = boto3.Session(profile_name='myproject')
ec2_client = session.client('ec2', region_name='us-east-1')

print(f'Using profile: {session.profile_name}')

You can have multiple profiles for different accounts or environments, which makes switching between dev and prod much easier.

A word on hardcoded credentials: Don't do it. Not in scripts, not in config files checked into source control, not anywhere. Use IAM roles in production and named profiles locally. If you're using access keys, rotate them regularly and give them only the permissions they actually need.

Starting and Stopping EC2 Instances

How do you start and stop instances programmatically with Boto3?

This is probably why you're here, so let's get into it. Boto3 uses start_instances and stop_instances for these operations. Both are simple calls, but there are a few details worth knowing.

Stopping an Instance

import boto3
from botocore.exceptions import ClientError

def stop_instance(instance_id: str, region: str = 'us-east-1') -> dict:
    """Stop a running EC2 instance and return the state transition."""
    ec2 = boto3.client('ec2', region_name=region)

    try:
        response = ec2.stop_instances(InstanceIds=[instance_id])

        # AWS tells us both the old state and the new state
        state_info = response['StoppingInstances'][0]
        previous = state_info['PreviousState']['Name']
        current = state_info['CurrentState']['Name']

        print(f'{instance_id}: {previous} -> {current}')
        return response

    except ClientError as e:
        print(f'Could not stop instance: {e.response["Error"]["Code"]} - {e}')
        raise

# Usage
stop_instance('i-0abcd1234ef567890')

Starting an Instance

import boto3
from botocore.exceptions import ClientError

def start_instance(instance_id: str, region: str = 'us-east-1') -> dict:
    """Start a stopped EC2 instance and return the state transition."""
    ec2 = boto3.client('ec2', region_name=region)

    try:
        response = ec2.start_instances(InstanceIds=[instance_id])

        state_info = response['StartingInstances'][0]
        previous = state_info['PreviousState']['Name']
        current = state_info['CurrentState']['Name']

        print(f'{instance_id}: {previous} -> {current}')
        return response

    except ClientError as e:
        code = e.response['Error']['Code']

        # Don't crash if the instance is already running
        if code == 'IncorrectInstanceState':
            print(f'{instance_id} is already in the desired state.')
        else:
            raise

# Usage
start_instance('i-0abcd1234ef567890')

One thing worth noting: both methods accept a list of instance IDs, so you can act on multiple instances in a single API call. If you're managing more than one instance, that's a lot cleaner than looping and calling separately.

# Stop multiple instances at once no loop needed
ec2 = boto3.client('ec2', region_name='us-east-1')
ec2.stop_instances(InstanceIds=[
    'i-0abcd1234ef567890',
    'i-0efgh5678ij901234',
    'i-0klmn9012op345678',
])

The response tells you the previous and current state for each one, which makes logging and auditing easy.

Using Boto3 Waiters to Block Until a State is Reached

How do you know when an instance has actually finished starting or stopping?

Here's a common mistake: you call start_instances, assume it's done, and immediately try to SSH in. But the instance is still booting. Your script fails, and now you're debugging something that wasn't actually broken.

The fix is waiters. A Boto3 waiter is a built-in polling loop that keeps checking the AWS API until your instance reaches the state you want. It handles the timing for you and raises an error if something goes wrong or takes too long.

import boto3
from botocore.exceptions import WaiterError

def start_and_wait(instance_id: str, region: str = 'us-east-1'):
    """Start an instance and block until it's fully running."""
    ec2 = boto3.client('ec2', region_name=region)

    ec2.start_instances(InstanceIds=[instance_id])
    print(f'Starting {instance_id}...')

    waiter = ec2.get_waiter('instance_running')

    try:
        waiter.wait(
            InstanceIds=[instance_id],
            WaiterConfig={
                'Delay': 15,       # Check every 15 seconds
                'MaxAttempts': 40  # Give up after ~10 minutes
            }
        )
        print(f'{instance_id} is running.')

    except WaiterError as e:
        print(f'Waiter timed out: {e}')
        raise


def stop_and_wait(instance_id: str, region: str = 'us-east-1'):
    """Stop an instance and block until it's fully stopped."""
    ec2 = boto3.client('ec2', region_name=region)

    ec2.stop_instances(InstanceIds=[instance_id])
    print(f'Stopping {instance_id}...')

    waiter = ec2.get_waiter('instance_stopped')

    try:
        waiter.wait(
            InstanceIds=[instance_id],
            WaiterConfig={'Delay': 15, 'MaxAttempts': 40}
        )
        print(f'{instance_id} is stopped.')

    except WaiterError as e:
        print(f'Waiter failed: {e}')
        raise

The four most useful EC2 waiters are instance_running, instance_stopped, instance_terminated, and instance_exists. Each one polls describe_instances under the hood and checks the state automatically, so you don't have to.

If you want to wait on multiple instances at once, just pass all the IDs together:

# Wait for several instances to stop one waiter call handles all of them
waiter = ec2.get_waiter('instance_stopped')
waiter.wait(
    InstanceIds=['i-0abc123', 'i-0def456', 'i-0ghi789'],
    WaiterConfig={'Delay': 15, 'MaxAttempts': 40}
)
print('All instances are stopped.')

Tip: Tune Delay and MaxAttempts based on your actual instances. A small instance with a simple AMI might be running in under a minute. A larger one running a heavy bootstrap script could take five or ten. If your waiter times out too often, bump up MaxAttempts before adding more complex logic.

Filtering and Querying Instances

How do you find the right instances without knowing their IDs ahead of time?

Hardcoding instance IDs is a trap. Instances get replaced, IDs change, and suddenly your script is managing the wrong thing, or nothing at all. A better approach is to query by attributes you control, like tags or state.

Querying by State

import boto3

def get_instances_by_state(state: str, region: str = 'us-east-1') -> list:
    """
    Return all instances in a given state.
    Valid states: 'running', 'stopped', 'pending', 'stopping', 'terminated'
    """
    ec2 = boto3.client('ec2', region_name=region)

    response = ec2.describe_instances(
        Filters=[{'Name': 'instance-state-name', 'Values': [state]}]
    )

    # describe_instances groups results into Reservations, so we flatten them
    instances = []
    for reservation in response['Reservations']:
        instances.extend(reservation['Instances'])

    return instances


# Find everything that's currently running
running = get_instances_by_state('running')
print(f'{len(running)} running instance(s) found.')

for inst in running:
    print(f"  {inst['InstanceId']}  {inst['InstanceType']}  {inst['Placement']['AvailabilityZone']}")

Querying by Tags

import boto3

def get_instances_by_tag(tag_key: str, tag_value: str, region: str = 'us-east-1') -> list:
    """Find instances that have a specific tag key/value pair."""
    ec2 = boto3.client('ec2', region_name=region)

    # AWS filter syntax for tags is 'tag:<key>'
    response = ec2.describe_instances(
        Filters=[{'Name': f'tag:{tag_key}', 'Values': [tag_value]}]
    )

    instances = []
    for reservation in response['Reservations']:
        instances.extend(reservation['Instances'])

    return instances


# Find all production instances
prod = get_instances_by_tag('Environment', 'production')

for inst in prod:
    # Pull the Name tag if it exists, otherwise label it 'unnamed'
    name = next(
        (t['Value'] for t in inst.get('Tags', []) if t['Key'] == 'Name'),
        'unnamed'
    )
    print(f"  {inst['InstanceId']}  {name}  {inst['State']['Name']}")

You can stack filters together, too. AWS applies them with AND logic, so you'll only get instances that match all of them:

# Only running instances in the production environment both filters must match
response = ec2.describe_instances(
    Filters=[
        {'Name': 'tag:Environment', 'Values': ['production']},
        {'Name': 'instance-state-name', 'Values': ['running']}
    ]
)

That single call replaces what would otherwise be a manual filter in the console or a clunky post-processing loop in code.

Bulk Operations: Stopping Instances by Tag or VPC

How do you stop an entire group of instances at once without listing each ID manually?

This is where scripting really earns its keep. Instead of hunting down individual instance IDs, you describe what you want, collect the IDs, and stop them all in one shot.

Stop All Instances in a VPC

import boto3

def stop_instances_in_vpc(vpc_id: str, region: str = 'us-east-1') -> list:
    """Stop all running instances in a given VPC."""
    ec2 = boto3.client('ec2', region_name=region)

    # Find every running instance in this VPC
    response = ec2.describe_instances(
        Filters=[
            {'Name': 'vpc-id', 'Values': [vpc_id]},
            {'Name': 'instance-state-name', 'Values': ['running']}
        ]
    )

    instance_ids = [
        inst['InstanceId']
        for r in response['Reservations']
        for inst in r['Instances']
    ]

    if not instance_ids:
        print(f'Nothing running in {vpc_id}.')
        return []

    print(f'Stopping {len(instance_ids)} instance(s) in {vpc_id}...')
    ec2.stop_instances(InstanceIds=instance_ids)

    return instance_ids


stopped = stop_instances_in_vpc('vpc-0abc12345def67890')
print(f'Stopped: {stopped}')

Stop All Instances with a Specific Tag

import boto3

def stop_instances_by_tag(
    tag_key: str,
    tag_value: str,
    region: str = 'us-east-1',
    dry_run: bool = True
) -> list:
    """
    Stop all running instances that match a tag.
    Set dry_run=True to preview what would be stopped without actually doing it.
    """
    ec2 = boto3.client('ec2', region_name=region)

    response = ec2.describe_instances(
        Filters=[
            {'Name': f'tag:{tag_key}', 'Values': [tag_value]},
            {'Name': 'instance-state-name', 'Values': ['running']}
        ]
    )

    instance_ids = [
        inst['InstanceId']
        for r in response['Reservations']
        for inst in r['Instances']
    ]

    if not instance_ids:
        print(f'No running instances tagged {tag_key}={tag_value}.')
        return []

    if dry_run:
        print(f'[DRY RUN] Would stop {len(instance_ids)} instance(s):')
        for iid in instance_ids:
            print(f'  - {iid}')
        return instance_ids

    ec2.stop_instances(InstanceIds=instance_ids)
    print(f'Stopped {len(instance_ids)} instance(s) tagged {tag_key}={tag_value}.')

    return instance_ids


# Always preview before committing
stop_instances_by_tag('Environment', 'dev', dry_run=True)

# When you're confident, flip the flag
# stop_instances_by_tag('Environment', 'dev', dry_run=False)

Always build a dry_run mode into your bulk scripts. It takes two minutes to add and can save you from a very bad day. Preview first, execute second.

Error Handling: Throttling, Permissions, and Invalid States

What happens when your Boto3 script hits an error, and how do you handle it cleanly?

AWS APIs fail. Not often, but enough that your production scripts need to be ready for it. The three most common issues you'll run into are rate limiting, missing permissions, and trying to do something that doesn't make sense for the instance's current state.

Here's a function that handles all of them gracefully:

import boto3
import time
from botocore.exceptions import ClientError

def robust_stop_instance(
    instance_id: str,
    region: str = 'us-east-1',
    max_retries: int = 3
) -> bool:
    """
    Stop an instance with sensible error handling and retry logic.
    Returns True on success, False if the operation can't be completed.
    """
    ec2 = boto3.client('ec2', region_name=region)

    for attempt in range(1, max_retries + 1):
        try:
            response = ec2.stop_instances(InstanceIds=[instance_id])
            state = response['StoppingInstances'][0]['CurrentState']['Name']
            print(f'Stopped {instance_id}. Current state: {state}')
            return True

        except ClientError as e:
            code = e.response['Error']['Code']

            if code in ('RequestLimitExceeded', 'Throttling'):
                # Back off exponentially and try again
                wait_time = 2 ** attempt
                print(f'Rate limited. Waiting {wait_time}s before retry {attempt}/{max_retries}...')
                time.sleep(wait_time)

            elif code == 'UnauthorizedOperation':
                # No point retrying this is a permissions issue
                print(f'Permission denied: {e.response["Error"]["Message"]}')
                print('Check the IAM policy on your role or profile.')
                return False

            elif code == 'IncorrectInstanceState':
                # Instance might already be stopped or terminated
                print(f'{instance_id} is in an unexpected state. Skipping.')
                return False

            elif code == 'InvalidInstanceID.NotFound':
                # Wrong region, or the instance no longer exists
                print(f'{instance_id} not found in {region}.')
                return False

            else:
                # Something unexpected surface it clearly
                print(f'Unexpected error [{code}]: {e}')
                raise

    print(f'Gave up after {max_retries} attempts.')
    return False

The key error codes to know:

RequestLimitExceeded / Throttling: You're hitting the AWS API rate limit. Use exponential backoff and retry.
UnauthorizedOperation: The IAM role or user doesn't have the right permission. No amount of retrying will fix this you need to update the policy.
IncorrectInstanceState: You're trying to do something the instance's current state doesn't allow, like stopping an instance that's already stopped.
InvalidInstanceID.NotFound: The instance ID doesn't exist in that region. Double-check both the ID and the region.

Frequently Asked Questions

<Accordion client:load title="Can Boto3 manage EC2 instances across multiple regions in one script?"

Yes, and it's pretty easy to do. Just create a separate client for each region you need. You can loop over a list of region names and run the same operations in each one. If you need it to go fast, concurrent.futures lets you run those calls in parallel.

import boto3
from concurrent.futures import ThreadPoolExecutor

regions = ['us-east-1', 'eu-west-1', 'ap-southeast-1']

def count_running(region):
    ec2 = boto3.client('ec2', region_name=region)
    resp = ec2.describe_instances(
        Filters=[{'Name': 'instance-state-name', 'Values': ['running']}]
    )
    count = sum(len(r['Instances']) for r in resp['Reservations'])
    print(f'{region}: {count} running instance(s)')

# Query all regions at the same time instead of one by one
with ThreadPoolExecutor() as pool:
    pool.map(count_running, regions)

<Accordion client:load title="What IAM permissions does my role need to start and stop instances?"

At a minimum you'll need ec2:StartInstances, ec2:StopInstances, ec2:DescribeInstances, and ec2:DescribeInstanceStatus. If you're filtering by tags, add ec2:DescribeTags too. Scope permissions as tightly as you can using IAM condition keys, especially for production environments.

<Accordion client:load title="How do Boto3 waiters differ from just sleeping for a fixed amount of time?"

A fixed sleep is a guess. A waiter actually checks. Waiters poll the AWS API and return the moment the state changes, which means they're faster when things go smoothly and more informative when they don't. If the timeout is exceeded, you get a WaiterError you can handle, rather than a script that silently moves on assuming everything is fine.

<Accordion client:load title="Is there a difference between using a Boto3 client and a resource for EC2?"

Yes. boto3.client('ec2') is the low-level interface that maps directly to API calls and returns raw dictionaries. boto3.resource('ec2') gives you a higher-level, object-oriented interface with things like ec2.Instance('i-0abc123'). Both work, but the client is more flexible for filtering and bulk operations, which is why most automation scripts use it.

<Accordion client:load title="Can I use Boto3 to manage spot instances the same way as on-demand instances?"

Mostly yes. start_instances, stop_instances, and describe_instances all work with spot instances. The main difference is that spot instances can be interrupted by AWS when capacity is reclaimed, so your automation should be ready for unexpected state changes. Use describe_spot_instance_requests to check spot-specific status and watch for interruption notices.

<Accordion client:load title="How do I handle pagination when there are many instances to query?"

describe_instances only returns up to 1,000 results per call. For bigger environments, use a paginator instead of calling directly:

import boto3

ec2 = boto3.client('ec2', region_name='us-east-1')
paginator = ec2.get_paginator('describe_instances')

instances = []
# paginate() handles NextToken automatically no manual looping needed
for page in paginator.paginate(Filters=[{'Name': 'instance-state-name', 'Values': ['running']}]):
    for reservation in page['Reservations']:
        instances.extend(reservation['Instances'])

print(f'Total running instances: {len(instances)}')

The paginator handles NextToken automatically, so you'll always get everything without extra logic.

<Accordion client:load title="What's the safest way to test bulk stop scripts before running them in production?"

Use the dry_run pattern shown in the bulk operations section. Run with dry_run=True first to see exactly which instances would be affected before you commit. Beyond that, test in a staging account or a non-production VPC. Some Boto3 calls also support a DryRun=True parameter at the API level, which validates your permissions without making any changes.

References

Redis Caching Patterns: Cache-Aside, Write-Through & Cache Invalidation

Mohammad Abu Mattar — Tue, 07 Apr 2026 20:02:59 GMT

Need to scale your backend without throwing money at servers? Redis caching patterns are your answer.

Most databases can handle hundreds of queries per second, but thousands? Your app slows to a crawl. Adding another database server just moves the problem. The real solution: serve data from memory 99% of the time using Redis.

The Problem

Database Load Under Scale

100 requests/sec for "get product details":
• Database: 200ms per query
• 100 × 200ms = serious bottleneck
• Add 1000 concurrent users = complete collapse
• Throwing more servers doesn't help (they all queue at the database)

The Solution

Redis Caching Patterns

Redis stores data in memory, delivering results in microseconds instead of milliseconds. But which caching pattern you choose determines whether your cache helps or hurts.

TL;DR

Cache-Aside: Check cache first, miss triggers database load + populate cache (most flexible)
Write-Through: Update cache and database together (most consistent)
TTL Strategy: Keys expire automatically, preventing stale data without manual invalidation
Cache Invalidation: Delete cache when data updates, triggering refresh on next request
Stampede Prevention: Prevent thundering herd when cache expires or misses expire

Pattern 1: Cache-Aside (Lazy Loading)

How It Works

1. Request comes in
2. Check Redis cache
3. Cache HIT → return immediately (microseconds)
4. Cache MISS → query database (milliseconds)
5. Populate cache with TTL
6. Next request hits cache

Python Implementation

import redis
import json
from functools import wraps
import time

redis_client = redis.Redis(host='localhost', port=6379, decode_responses=True)

def cache_aside(ttl=3600):
    """Decorator implementing cache-aside pattern"""
    def decorator(func):
        @wraps(func)
        def wrapper(key, *args, **kwargs):
            # Check cache first
            cached = redis_client.get(key)
            if cached:
                # Cache hit
                return json.loads(cached)

            # Cache miss: call function (hits database)
            result = func(*args, **kwargs)

            # Populate cache with TTL
            redis_client.setex(key, ttl, json.dumps(result))

            return result
        return wrapper
    return decorator

# Database query (expensive)
def get_user_from_db(user_id):
    # Simulated database query
    time.sleep(0.2)  # 200ms
    return {"id": user_id, "name": f"User {user_id}", "email": f"user{user_id}@example.com"}

# Decorated function
@cache_aside(ttl=3600)
def get_user(user_id):
    return get_user_from_db(user_id)

# Usage
print(get_user(f"user:1"))  # First call: 200ms (cache miss)
print(get_user(f"user:1"))  # Second call: <1ms (cache hit)

Node.js Implementation

import {createClient} from 'redis';
import {promisify} from 'util';

const redis = createClient();
redis.connect();

const getAsync = promisify(redis.get).bind(redis);
const setexAsync = promisify(redis.setex).bind(redis);

// Database query (expensive)
async function getUserFromDB(userId: string) {
  // Simulated database query
  await new Promise((resolve) => setTimeout(resolve, 200));
  return {
    id: userId,
    name: `User ${userId}`,
    email: `user${userId}@example.com`,
  };
}

// Cache-aside implementation
async function getUser(userId: string, ttl = 3600) {
  const cacheKey = `user:${userId}`;

  // Check cache
  const cached = await getAsync(cacheKey);
  if (cached) {
    return JSON.parse(cached);
  }

  // Cache miss: hit database
  const user = await getUserFromDB(userId);

  // Populate cache
  await setexAsync(cacheKey, ttl, JSON.stringify(user));

  return user;
}

// Usage
(async () => {
  console.time('first');
  await getUser('user:1'); // 200ms (miss)
  console.timeEnd('first');

  console.time('second');
  await getUser('user:1'); // <1ms (hit)
  console.timeEnd('second');
})();

Go Implementation

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"github.com/redis/go-redis/v9"
	"time"
)

type User struct {
	ID    int    `json:"id"`
	Name  string `json:"name"`
	Email string `json:"email"`
}

var rdb = redis.NewClient(&redis.Options{
	Addr: "localhost:6379",
})

func getUserFromDB(ctx context.Context, userID int) (User, error) {
	// Simulated database query
	time.Sleep(200 * time.Millisecond)
	return User{
		ID:    userID,
		Name:  fmt.Sprintf("User %d", userID),
		Email: fmt.Sprintf("user%d@example.com", userID),
	}, nil
}

func getUser(ctx context.Context, userID int) (User, error) {
	cacheKey := fmt.Sprintf("user:%d", userID)

	// Check cache
	val, err := rdb.Get(ctx, cacheKey).Result()
	if err == nil {
		var user User
		json.Unmarshal([]byte(val), &user)
		return user, nil
	}

	// Cache miss: hit database
	user, err := getUserFromDB(ctx, userID)
	if err != nil {
		return user, err
	}

	// Populate cache (1 hour TTL)
	data, _ := json.Marshal(user)
	rdb.SetEx(ctx, cacheKey, string(data), time.Hour)

	return user, nil
}

Pattern 2: Write-Through

How It Works

1. Data update arrives
2. Update cache AND database together
3. Both succeed or both fail (atomic)
4. Next read hits cache (always consistent)

When to Use

Critical data (payments, user accounts)
Data that changes frequently
When consistency is more important than speed

Python Implementation

import redis
import json

redis_client = redis.Redis(host='localhost', port=6379, decode_responses=True)

def update_user_write_through(user_id: int, user_data: dict):
    """Update cache AND database together"""
    cache_key = f"user:{user_id}"

    try:
        # Update cache first (faster)
        redis_client.setex(cache_key, 3600, json.dumps(user_data))

        # Then update database
        # Assuming `db.update_user(user_id, user_data)` exists
        db.update_user(user_id, user_data)

        return True
    except Exception as e:
        # If database fails, invalidate cache
        redis_client.delete(cache_key)
        raise e

# Usage
update_user_write_through(42, {
    "name": "Alice",
    "email": "alice@example.com"
})

# Next read hits cache
user = redis_client.get("user:42")

Node.js Implementation

async function updateUserWriteThrough(userId: string, userData: any) {
  const cacheKey = `user:${userId}`;

  try {
    // Update cache first
    await redis.setex(cacheKey, 3600, JSON.stringify(userData));

    // Then update database
    await db.updateUser(userId, userData);

    return true;
  } catch (error) {
    // Rollback: invalidate cache on database failure
    await redis.del(cacheKey);
    throw error;
  }
}

Pattern 3: TTL Strategy

Automatic Expiration

Redis automatically deletes keys after TTL expires. No manual invalidation needed.

import redis
from datetime import datetime, timedelta

redis_client = redis.Redis()

# Different TTL for different data types
def cache_with_ttl(key: str, value: str, data_type: str):
    ttl_map = {
        "user_profile": 3600,           # 1 hour (changes infrequently)
        "product": 86400,                # 24 hours (stable data)
        "product_price": 300,            # 5 minutes (changes frequently)
        "session": 1800,                 # 30 minutes (security)
        "statistics": 60,                # 1 minute (real-time)
    }

    ttl = ttl_map.get(data_type, 3600)
    redis_client.setex(key, ttl, value)

# Usage
cache_with_ttl("product:123", "iPhone 15", "product")        # 24 hours
cache_with_ttl("price:apple", "$999", "product_price")       # 5 minutes
cache_with_ttl("stats:daily", "1000 users", "statistics")    # 1 minute

Sliding Window TTL

Reset TTL on each access (useful for sessions):

def get_with_sliding_ttl(key: str, ttl: int):
    """Get value and reset TTL"""
    value = redis_client.get(key)
    if value:
        # Reset TTL on access
        redis_client.expire(key, ttl)
    return value

# Usage: Session stays alive as long as user is active
session = get_with_sliding_ttl("session:abc123", 1800)  # 30 min

Pattern 4: Cache Invalidation

Event-Driven Invalidation

Delete cache when data changes:

def update_product(product_id: int, new_data: dict):
    # Update database
    db.update_product(product_id, new_data)

    # Invalidate cache immediately
    redis_client.delete(f"product:{product_id}")

    # Also invalidate related caches (cascade invalidation)
    redis_client.delete(f"category:{new_data['category']}")
    redis_client.delete("products:all")

# When product price changes
update_product(123, {"price": "$899", "category": "electronics"})
# → Deletes "product:123", "category:electronics", "products:all"

Tag-Based Invalidation

Invalidate multiple keys with one operation:

import json

def cache_with_tags(key: str, value: str, tags: list):
    """Cache value and tag it for bulk invalidation"""
    # Store value
    redis_client.setex(key, 3600, value)

    # Store tag reference (set of keys with this tag)
    for tag in tags:
        redis_client.sadd(f"tag:{tag}", key)

def invalidate_by_tag(tag: str):
    """Invalidate all keys with a tag"""
    # Get all keys with this tag
    keys = redis_client.smembers(f"tag:{tag}")

    # Delete all tagged keys
    if keys:
        redis_client.delete(*keys)

    # Clean up tag
    redis_client.delete(f"tag:{tag}")

# Usage
cache_with_tags("product:1", "iPhone", ["electronics", "apple"])
cache_with_tags("product:2", "MacBook", ["electronics", "apple"])
cache_with_tags("product:3", "Apple Watch", ["wearables", "apple"])

# Invalidate all Apple products at once
invalidate_by_tag("apple")

Preventing Cache Stampedes

The Problem

Thousands of requests for a popular key
↓
Key expires
↓
ALL requests hit database simultaneously (thundering herd)
↓
Database overload, service degradation

Solution: Probabilistic Regeneration

import redis
import random
import time

redis_client = redis.Redis()

def get_with_low_ttl_regen(key: str, expensive_calculation, ttl=3600):
    """
    Regenerate cache probabilistically near expiration
    Avoids thundering herd when TTL reaches 0
    """
    value = redis_client.get(key)

    if value:
        # Check TTL
        remaining_ttl = redis_client.ttl(key)

        # Near end of life? Regenerate probabilistically
        if remaining_ttl < 0:
            # Key expired
            pass  # Recalculate below
        elif remaining_ttl < ttl * 0.2:  # Last 20% of life
            # Probability of regeneration increases as TTL decreases
            prob = 1 - (remaining_ttl / (ttl * 0.2))
            if random.random() < prob:
                # First request regenerates, others get stale data temporarily
                new_value = expensive_calculation()
                redis_client.setex(key, ttl, new_value)
                return new_value

        return value

    # Cache miss: calculate fresh data
    result = expensive_calculation()
    redis_client.setex(key, ttl, result)
    return result

Distributed Lock Pattern

Prevent parallel cache recalculations:

import time
import uuid

def get_with_lock(key: str, expensive_calculation, lock_timeout=10):
    """Use lock to prevent multiple calculations"""
    lock_key = f"lock:{key}"
    cache_key = key

    # Check cache
    cached = redis_client.get(cache_key)
    if cached:
        return cached

    # Try to acquire lock
    lock_id = str(uuid.uuid4())
    acquired = redis_client.set(lock_key, lock_id, nx=True, ex=lock_timeout)

    if acquired:
        # We got the lock, calculate
        try:
            result = expensive_calculation()
            redis_client.setex(cache_key, 3600, result)
            return result
        finally:
            # Release lock
            if redis_client.get(lock_key) == lock_id:
                redis_client.delete(lock_key)
    else:
        # Someone else got the lock, wait for result
        for _ in range(10):
            time.sleep(0.1)
            cached = redis_client.get(cache_key)
            if cached:
                return cached

        # Timeout: calculate ourselves
        return expensive_calculation()

Serialization Strategies

JSON vs MessagePack vs Protocol Buffers

import json
import msgpack

def store_json(key, obj):
    """Simple but larger"""
    redis_client.set(key, json.dumps(obj))

def store_msgpack(key, obj):
    """Smaller, faster"""
    redis_client.set(key, msgpack.packb(obj))

# Benchmark: 1000 products
products = [{"id": i, "name": f"Product {i}", "price": 99.99} for i in range(1000)]

json_size = len(json.dumps(products))          # ~50KB
msgpack_size = len(msgpack.packb(products))    # ~30KB (40% smaller)

# Choice depends on:
# - JSON: Human readable, broad ecosystem
# - MsgPack: Smaller, faster
# - Protobuf: Strongly typed, enterprise

Monitoring Cache Performance

Track Hit Rate

import redis

redis_client = redis.Redis()

def get_cache_stats():
    """Get cache performance metrics"""
    info = redis_client.info('stats')

    total_commands = info.get('total_commands_processed', 0)
    hits = info.get('keyspace_hits', 0)
    misses = info.get('keyspace_misses', 0)

    hit_rate = (hits / (hits + misses)) * 100 if (hits + misses) > 0 else 0

    return {
        "hit_rate": hit_rate,
        "total_hits": hits,
        "total_misses": misses,
        "evictions": info.get('evicted_keys', 0),
        "used_memory": info.get('used_memory_human', 'N/A'),
    }

# Target: >80% hit rate
# <50%: Cache is ineffective, reconsider keys/TTLs
# >95%: Good fit for caching strategy

Connection Pooling

Reuse connections instead of creating new ones:

import redis
from redis.connection import ConnectionPool

# Without pooling (inefficient)
conn = redis.Redis()

# With pooling (efficient)
pool = ConnectionPool.from_url(
    'redis://localhost:6379',
    max_connections=50,
    decode_responses=True
)
redis_client = redis.Redis(connection_pool=pool)

# Reuses connections automatically
for i in range(1000):
    redis_client.get(f"key:{i}")

Best Practices

1. Consistent Key Naming

# Good: Hierarchical, searchable
"user:123"
"user:123:settings"
"product:456"
"product:456:reviews"
"order:789:items"

# Bad: Ambiguous
"u123"
"prod_456"
"item-789"

2. Set Reasonable TTLs

TTL_STRATEGY = {
    "user_profile": 3600,           # 1 hour
    "product_catalog": 86400,       # 24 hours
    "session": 1800,                # 30 minutes
    "api_response": 300,            # 5 minutes
    "real_time_data": 60,           # 1 minute
}

3. Plan for Failures

def safe_cache_get(key: str, fallback_fn):
    """Gracefully handle Redis failures"""
    try:
        value = redis_client.get(key)
        if value:
            return value
    except redis.ConnectionError:
        # Redis down? Use fallback
        pass

    # No cache: call function
    return fallback_fn()

Conclusion

Redis caching is the difference between a responsive application and a slow one.

Choose the right pattern (cache-aside for flexibility, write-through for consistency), set appropriate TTLs, prevent stampedes, and monitor hit rates. Combined with database optimization, caching forms the foundation of high-performance systems.

Resources

PostgreSQL Query Optimization: Indexes, EXPLAIN ANALYZE & Execution Plans

Mohammad Abu Mattar — Sun, 29 Mar 2026 20:02:59 GMT

Need to optimize slow PostgreSQL queries? Here's how with EXPLAIN ANALYZE and strategic indexing.

Slow database queries kill application performance. But most developers don't know where the actual bottleneck is, so they guess at fixes. EXPLAIN ANALYZE reveals exactly what's expensive and where to optimize.

The Problem

Why Queries Get Slow

In development with small datasets, missing indexes don't matter. Switch to production with millions of rows, and suddenly sequential table scans become devastating. Queries that ran in 50ms take 5+ seconds. You need to know why.

Common Performance Issues

Missing indexes: Forcing full table scans on every query
N+1 queries: Fetching data in loops instead of bulk operations
Bad query plans: Using inefficient joins or sorts
Undersized indices: Index on wrong columns
Connection pool exhaustion: Running out of available connections

The Solution

EXPLAIN ANALYZE: Your Debugging Superpower

EXPLAIN ANALYZE shows exactly how PostgreSQL executes your query, including:

Which operations are most expensive (by cost)
Actual vs. estimated row counts
Time spent in each step
Seq Scans vs. Index Scans
Sort and Hash operations

TL;DR

Run EXPLAIN ANALYZE to identify expensive operations
Add indexes where sequential scans happen on large tables
Use pg_stat_statements to automatically find slow queries
Fix N+1 query patterns before rewriting complex queries
Setup connection pooling for better resource utilization

Understanding EXPLAIN ANALYZE

Basic Query Analysis

-- Compare these two queries
EXPLAIN ANALYZE
SELECT * FROM users WHERE created_at > '2025-01-01';

-- Output example:
-- Seq Scan on users  (cost=0.00..35.50 rows=1000 width=100)
--   Filter: (created_at > '2025-01-01')
--   Planning Time: 0.123 ms
--   Execution Time: 45.234 ms

What this tells you:

Seq Scan: Reading entire table (slow for large tables)
cost=0.00..35.50: PostgreSQL's estimated cost
rows=1000: Expected to return 1000 rows
Execution Time: 45.234 ms: Actual time taken

Reading the Cost

EXPLAIN ANALYZE
SELECT u.id, u.name, o.total
FROM users u
LEFT JOIN orders o ON u.id = o.user_id
WHERE u.status = 'active';

-- Output:
-- Hash Left Join  (cost=2534.50..5892.33 rows=5000 width=50)
--   Hash Cond: (o.user_id = u.id)
--   ->  Seq Scan on orders o  (cost=0.00..1234.50 rows=50000 width=8)
--   ->  Hash  (cost=2500.00..2500.00 rows=5000 width=42)
--         ->  Index Scan using idx_users_status on users u  (cost=10.00..2500.00 rows=5000 width=42)

Cost interpretation:

Lower cost = faster execution
cost=A..B: A = startup cost, B = total cost
Focus on the highest-cost operations first

Creating Effective Indexes

Basic Index Creation

-- Simple B-tree index (most common)
CREATE INDEX idx_users_email ON users(email);

-- Multi-column index
CREATE INDEX idx_orders_user_date ON orders(user_id, created_at);

-- Partial index (only index active users)
CREATE INDEX idx_users_active ON users(id) WHERE status = 'active';

-- Unique index (constraint + performance)
CREATE UNIQUE INDEX idx_users_email_unique ON users(email);

Index Types: When to Use Each

-- B-tree (default, best for most queries)
CREATE INDEX idx_standard ON table_name(column);

-- Hash (equality only, rarely faster than B-tree)
CREATE INDEX idx_hash ON table_name USING hash(column);

-- GiST (geometric, full-text search)
CREATE INDEX idx_fulltext ON articles USING gist(search_vector);

-- GIN (array, JSON, full-text - faster for large result sets)
CREATE INDEX idx_json ON logs USING gin(metadata);

Partial Indexes for Common Cases

-- Don't index inactive records
CREATE INDEX idx_active_orders ON orders(id) WHERE status != 'cancelled';

-- Only index recent data
CREATE INDEX idx_recent_events ON events(id) WHERE created_at > NOW() - INTERVAL '90 days';

-- Saves space and speeds up queries on active data

Multi-Column Index Strategy

-- For WHERE + ORDER BY combinations
-- Query: WHERE user_id = ? ORDER BY created_at DESC
CREATE INDEX idx_user_date ON orders(user_id, created_at DESC);

-- Column order matters: Put filtered columns first
-- Good:   WHERE status = ? AND type = ?
CREATE INDEX idx_good ON events(status, type);

-- Bad:    WHERE status = ? AND type = ?
CREATE INDEX idx_bad ON events(type, status);  -- Wrong order

Detecting and Fixing N+1 Queries

The N+1 Problem

# BAD: N+1 queries (1 + N queries)
users = db.query(User).all()  # Query 1
for user in users:
    orders = db.query(Order).filter(Order.user_id == user.id).all()  # Queries 2 to N+1
    print(f"{user.name}: {len(orders)} orders")

# Result with 1000 users = 1001 queries

The Fix: Eager Loading

# GOOD: Eager load with join
from sqlalchemy import joinedload

users = db.query(User).options(joinedload(User.orders)).all()  # Single query with join

for user in users:
    print(f"{user.name}: {len(user.orders)} orders")

SQL Equivalent

-- BAD (N+1):
SELECT * FROM users;  -- 1000 queries...
SELECT * FROM orders WHERE user_id = ?;

-- GOOD (Single query):
SELECT u.id, u.name, o.id, o.total
FROM users u
LEFT JOIN orders o ON u.id = o.user_id;

Finding Slow Queries Automatically

Enable Query Logging

-- Enable logging of slow queries (500ms+)
ALTER SYSTEM SET log_min_duration_statement = 500;
SELECT pg_reload_conf();

-- View current setting
SHOW log_min_duration_statement;

Use pg_stat_statements

-- Install extension
CREATE EXTENSION IF NOT EXISTS pg_stat_statements;

-- Find slowest queries
SELECT query, calls, mean_time, max_time
FROM pg_stat_statements
ORDER BY mean_time DESC
LIMIT 10;

-- Find queries that were called most
SELECT query, calls, mean_time
FROM pg_stat_statements
ORDER BY calls DESC
LIMIT 10;

-- Clear stats to get fresh baseline
SELECT pg_stat_statements_reset();

Query Optimization Patterns

Pattern 1: Missing Index on WHERE Clause

-- Before: Seq Scan (slow)
EXPLAIN ANALYZE
SELECT * FROM orders WHERE customer_id = 42;

-- Fix: Add index
CREATE INDEX idx_orders_customer ON orders(customer_id);

-- After: Index Scan (fast)
EXPLAIN ANALYZE
SELECT * FROM orders WHERE customer_id = 42;

Pattern 2: Inefficient Subqueries

-- SLOW: Subquery evaluated for each row
SELECT * FROM orders
WHERE customer_id IN (
  SELECT id FROM customers WHERE status = 'premium'
);

-- FAST: Use JOIN instead
SELECT DISTINCT o.*
FROM orders o
JOIN customers c ON o.customer_id = c.id
WHERE c.status = 'premium';

Pattern 3: Using Functions in WHERE Clauses

-- SLOW: Function applied to indexed column
SELECT * FROM users WHERE LOWER(email) = 'test@example.com';

-- FAST: Use expression index
CREATE INDEX idx_users_email_lower ON users(LOWER(email));
SELECT * FROM users WHERE LOWER(email) = 'test@example.com';

-- OR: Normalize incoming data instead
SELECT * FROM users WHERE email = 'test@example.com';

Pattern 4: Missing ORDER BY Index

-- SLOW: Sort step required
EXPLAIN ANALYZE
SELECT * FROM events ORDER BY created_at DESC LIMIT 10;

-- Fix: Add index for sort
CREATE INDEX idx_events_date_desc ON events(created_at DESC);

-- Now uses Index Scan + Limit, no Sort step

Connection Pooling

Why Connection Pooling Matters

-- Without pooling: Each request opens/closes connection (expensive)
-- With pooling: Reuse existing connections (cheap)

PgBouncer Configuration

[databases]
myapp = host=localhost port=5432 dbname=myapp_prod

[pgbouncer]
listen_port = 6432
listen_addr = 127.0.0.1
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt

# Connection pooling settings
pool_mode = transaction
max_client_conn = 1000
default_pool_size = 25
min_pool_size = 10
reserve_pool_size = 5
reserve_pool_timeout = 3

Application Connection

# Before pooling: Direct to PostgreSQL
import psycopg2
conn = psycopg2.connect("dbname=myapp user=postgres host=localhost")

# After pooling: Connect to PgBouncer on port 6432
import psycopg2
conn = psycopg2.connect("dbname=myapp user=postgres host=localhost port=6432")

Real-World Optimization Workflow

Step 1: Identify the Slow Query

# From application logs or pg_stat_statements
# Example: SELECT query is taking 8000ms

Step 2: Run EXPLAIN ANALYZE

EXPLAIN ANALYZE
SELECT o.id, o.total, c.name
FROM orders o
JOIN customers c ON o.customer_id = c.id
WHERE o.created_at > '2025-01-01'
ORDER BY o.total DESC;

Step 3: Look for Expensive Operations

Sort (cost=9234.50..9244.50)  ← Look here
  ->  Hash Join (cost=2500.00..9234.00)  ← And here
        ->  Seq Scan on orders o (cost=0.00..5000.00)  ← Sequential scan on large table
        ->  Hash (cost=100.00..100.00 rows=1000)
              ->  Seq Scan on customers c (cost=0.00..100.00)

Step 4: Add Indexes

-- Index for WHERE clause
CREATE INDEX idx_orders_date ON orders(created_at);

-- Index for JOIN condition
CREATE INDEX idx_orders_customer_id ON orders(customer_id);

-- Index for ORDER BY
CREATE INDEX idx_orders_total ON orders(total DESC);

Step 5: Rerun EXPLAIN ANALYZE

-- Should now use Index Scans instead of Seq Scans
-- No Sort step if you have the right index

Best Practices

1. Index Naming Convention

-- Consistent naming helps find related indexes
CREATE INDEX idx_table_column_type ON table_name(column);

-- Examples:
CREATE INDEX idx_users_email ON users(email);
CREATE INDEX idx_orders_customer_date ON orders(customer_id, created_at);
CREATE INDEX idx_products_active ON products(id) WHERE active = true;

2. Monitor Index Usage

-- Find unused indexes (bloat)
SELECT schemaname, tablename, indexname
FROM pg_indexes
WHERE schemaname NOT IN ('pg_catalog', 'information_schema');

-- Check if index is being used
SELECT
  indexrelname,
  idx_scan,
  idx_tup_read,
  idx_tup_fetch
FROM pg_stat_user_indexes
ORDER BY idx_scan DESC;

3. Regular Maintenance

-- Analyze table stats (for query planner)
ANALYZE users;

-- Vacuum to clean up dead rows
VACUUM FULL users;

-- Reindex if fragmented (heavy write workloads)
REINDEX INDEX idx_users_email;

Resources

Multi-Environment Secret Management with HashiCorp Vault

Mohammad Abu Mattar — Tue, 03 Mar 2026 00:00:00 GMT

Need to manage secrets safely across multiple environments? Here's how with HashiCorp Vault.

Storing secrets in .env files, hardcoding them, or even using separate secret managers per environment creates security risks and operational chaos. Vault provides a unified, audited solution for secret generation, rotation, and access control across all your environments.

The Problem

Multi-Environment Secret Chaos

When managing dev, staging, and production environments, secrets management becomes a headache:

Different secrets per environment with no central tracking
Manual rotation processes prone to human error
No audit trail for who accessed which secrets and when
Secrets leaking through Git history or logs
Difficulty revoking compromised credentials instantly

The Solution

HashiCorp Vault Overview

Vault is a secrets management platform that provides:

Dynamic secrets: Generate credentials on-demand that auto-expire
Secret rotation: Automatically rotate credentials without downtime
Audit logging: Complete trail of all secret access and operations
Environment isolation: Separate secrets per environment with shared policies
Multi-auth methods: Support for AppRole, Kubernetes, IAM, JWT, and more

TL;DR

Use HashiCorp Vault as a centralized secret backend
Generate dynamic credentials that auto-expire
Implement automatic rotation for long-lived secrets
Audit all secret access across environments

Installation and Setup

Start Vault with Docker

services:
  vault:
    image: vault:latest
    container_name: vault
    ports:
      - "8200:8200"
    environment:
      VAULT_DEV_ROOT_TOKEN_ID: "root"
      VAULT_DEV_LISTEN_ADDRESS: "0.0.0.0:8200"
    cap_add:
      - IPC_LOCK
    volumes:
      - vault-data:/vault/data
    command: server -dev

volumes:
  vault-data:

Start the Vault server:

docker-compose up -d
export VAULT_ADDR='http://localhost:8200'
export VAULT_TOKEN='root'

CLI Examples

Basic Secret Storage

# Store a static secret
vault kv put secret/dev/database \
  username="dbuser" \
  password="supersecret" \
  host="db.dev.internal"

# Retrieve secret
vault kv get secret/dev/database

# Get specific field
vault kv get -field=password secret/dev/database

Organize Secrets by Environment

# Development secrets
vault kv put secret/dev/app-api \
  api_key="dev-key-12345" \
  api_secret="dev-secret-67890"

# Staging secrets
vault kv put secret/staging/app-api \
  api_key="staging-key-abcde" \
  api_secret="staging-secret-fghij"

# Production secrets
vault kv put secret/prod/app-api \
  api_key="prod-key-xyz123" \
  api_secret="prod-secret-abc456"

Programming Examples

Python: Fetch Secrets

import hvac
import os
from typing import Dict, Any

class VaultClient:
    def __init__(self, vault_addr: str = "http://localhost:8200", token: str = None):
        self.client = hvac.Client(url=vault_addr, token=token or os.getenv('VAULT_TOKEN'))

    def get_secret(self, path: str, field: str = None) -> Dict[str, Any]:
        """Fetch a secret from Vault"""
        response = self.client.secrets.kv.v2.read_secret_version(path=path)
        data = response['data']['data']

        if field:
            return data.get(field)
        return data

    def get_database_creds(self, environment: str) -> Dict[str, str]:
        """Get database credentials for specific environment"""
        path = f"secret/{environment}/database"
        return self.get_secret(path)

    def get_api_keys(self, environment: str) -> Dict[str, str]:
        """Get API keys for specific environment"""
        path = f"secret/{environment}/app-api"
        return self.get_secret(path)

# Usage
if __name__ == "__main__":
    vault = VaultClient()

    # Get dev database credentials
    db_creds = vault.get_database_creds("dev")
    print(f"Database: {db_creds['host']}")
    print(f"User: {db_creds['username']}")

    # Get specific field
    api_key = vault.get_secret("secret/prod/app-api", field="api_key")
    print(f"API Key: {api_key}")

Node.js: Fetch Secrets

import * as VaultClient from 'node-vault';

class SecretManager {
  private vault: ReturnType<typeof VaultClient>;

  constructor(address: string = 'http://localhost:8200', token: string) {
    this.vault = VaultClient({
      apiVersion: 'v1',
      endpoint: address,
      token: token || process.env.VAULT_TOKEN,
    });
  }

  async getSecret(path: string, field?: string): Promise<any> {
    try {
      const response = await this.vault.read(path);
      const data = response.data.data;

      if (field) {
        return data[field];
      }
      return data;
    } catch (error) {
      console.error(`Failed to retrieve secret from ${path}:`, error);
      throw error;
    }
  }

  async getDatabaseConfig(environment: string): Promise<any> {
    return this.getSecret(`secret/${environment}/database`);
  }

  async getApiCredentials(environment: string): Promise<any> {
    return this.getSecret(`secret/${environment}/app-api`);
  }
}

// Usage
(async () => {
  const manager = new SecretManager('http://localhost:8200', 'root');

  const dbConfig = await manager.getDatabaseConfig('dev');
  console.log(`Connecting to: ${dbConfig.host}`);

  const apiKey = await manager.getApiCredentials('prod');
  console.log(`API Key: ${apiKey.api_key}`);
})();

Dynamic Database Credentials

Configure Database Secret Engine

# Enable database secrets engine
vault secrets enable database

# Configure PostgreSQL connection
vault write database/config/postgresql \
  plugin_name=postgresql-database-plugin \
  allowed_roles="readonly,readwrite" \
  connection_url="postgresql://admin:password@db.prod.internal:5432/appdb" \
  username="vault_admin" \
  password="vault_admin_password"

# Create readonly role
vault write database/roles/readonly \
  db_name=postgresql \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

# Create readwrite role
vault write database/roles/readwrite \
  db_name=postgresql \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="6h" \
  max_ttl="24h"

Retrieve Dynamic Credentials

# Get temporary readonly credentials
vault read database/creds/readonly
# Output:
# Key                Value
# ---                -----
# lease_duration     1h
# lease_id           database/creds/readonly/...
# password           a-temporary-password
# username           v-token-readonly-xxxxxxxx

# Get temporary readwrite credentials
vault read database/creds/readwrite

Python: Use Dynamic Credentials

import hvac
import psycopg2
from typing import Generator
import contextlib

class DynamicDatabaseAccess:
    def __init__(self, vault_addr: str, vault_token: str):
        self.vault = hvac.Client(url=vault_addr, token=vault_token)
        self.db_host = os.getenv('DB_HOST', 'db.prod.internal')
        self.db_port = os.getenv('DB_PORT', '5432')
        self.db_name = os.getenv('DB_NAME', 'appdb')

    def get_dynamic_credentials(self, role: str) -> dict:
        """Fetch temporary database credentials from Vault"""
        response = self.vault.secrets.database.read_dynamic_credentials(role)
        return response['data']

    @contextlib.contextmanager
    def get_connection(self, role: str = 'readonly') -> Generator:
        """Get a database connection with dynamic credentials"""
        creds = self.get_dynamic_credentials(role)

        conn = psycopg2.connect(
            host=self.db_host,
            port=self.db_port,
            database=self.db_name,
            user=creds['username'],
            password=creds['password']
        )

        try:
            yield conn
        finally:
            conn.close()

# Usage
db_access = DynamicDatabaseAccess('http://localhost:8200', 'root')

with db_access.get_connection(role='readonly') as conn:
    cursor = conn.cursor()
    cursor.execute("SELECT COUNT(*) FROM users")
    print(f"Total users: {cursor.fetchone()[0]}")

Policy-Based Access Control

Create Policies for Different Roles

# Read-only access to dev/staging secrets
path "secret/data/dev/*" {
  capabilities = ["read", "list"]
}

path "secret/data/staging/*" {
  capabilities = ["read", "list"]
}

# Deny access to production
path "secret/data/prod/*" {
  capabilities = ["deny"]
}

# Allow token self-renewal
path "auth/token/renew-self" {
  capabilities = ["update"]
}

# Full access to production secrets
path "secret/data/prod/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# Read-only access to staging
path "secret/data/staging/*" {
  capabilities = ["read", "list"]
}

# Database credential access
path "database/creds/prod/*" {
  capabilities = ["read"]
}

# Audit logging
path "sys/audit" {
  capabilities = ["read"]
}

Apply Policies

# Create policies
vault policy write dev-team policies/dev-team.hcl
vault policy write prod-team policies/prod-team.hcl

# Assign to users/apps
vault write auth/userpass/users/jane policies="prod-team"
vault write auth/userpass/users/alice policies="dev-team"

Kubernetes Integration

Enable Kubernetes Auth

# Enable Kubernetes authentication
vault auth enable kubernetes

# Configure Kubernetes auth
vault write auth/kubernetes/config \
  token_reviewer_jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token \
  kubernetes_host="https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

# Create role for pods
vault write auth/kubernetes/role/app-role \
  bound_service_account_names=app \
  bound_service_account_namespaces=default \
  policies=default,app-secrets \
  ttl=24h

Pod Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      serviceAccountName: app
      containers:
        - name: myapp
          image: myapp:latest
          env:
            - name: VAULT_ADDR
              value: 'http://vault.vault.svc.cluster.local:8200'
            - name: VAULT_SKIP_VERIFY
              value: 'false'
          volumeMounts:
            - name: vault-token
              mountPath: /vault/secrets
      volumes:
        - name: vault-token
          projected:
            sources:
              - serviceAccountToken:
                  audience: vault
                  expirationSeconds: 3600
                  path: token

Audit Logging

Enable Audit Logging

# Enable file audit backend
vault audit enable file file_path=/vault/logs/audit.log

# Enable syslog audit backend
vault audit enable syslog tag="vault"

# View audit logs
tail -f /vault/logs/audit.log | jq '.'

Query Audit Logs

# Show all secret reads
cat /vault/logs/audit.log | jq 'select(.type == "response" and .auth.policy_results.granted_policies >= 0)'

# Show secret modifications
cat /vault/logs/audit.log | jq 'select(.type == "request" and .request.operation == "write")'

# Show failed auth attempts
cat /vault/logs/audit.log | jq 'select(.response.auth == null)'

Best Practices

1. Use AppRole for Applications

# Create AppRole
vault auth enable approle

# Generate application role
vault write auth/approle/role/my-app \
  token_ttl=1h \
  token_max_ttl=4h \
  policies="app-secrets"

# Get role ID
vault read auth/approle/role/my-app/role-id

# Generate secret ID
vault write -f auth/approle/role/my-app/secret-id

2. Rotate Secrets Regularly

# Configure auto-rotation every 30 days
vault write database/config/postgresql \
  connection_url="postgresql://admin:password@db.prod.internal:5432/appdb" \
  rotation_statements="ALTER ROLE \"{{name}}\" WITH PASSWORD '{{password}}';" \
  rotation_period=720h

3. Implement Least Privilege

# Only grant necessary capabilities
path "secret/data/myapp/*" {
  capabilities = ["read"]  # Not "update" or "delete"
}

Resources

Top 7 Open Source OCR Models for Document Processing

Mohammad Abu Mattar — Fri, 09 Jan 2026 00:00:00 GMT

AI Tool

Turn your documents into perfect digital copies with these powerful open source OCR models. No more dealing with messy text extraction get clean, accurate markdown from PDFs, images, and scanned documents.

The OCR Revolution

Beyond Basic Text Extraction

Complete document understanding

These modern OCR models do way more than just read text. They understand document structure, tables, diagrams, math equations, and multiple languages, turning everything into well-formatted markdown.

Local Processing Benefits

Privacy and control

Run these models right on your own machine without sending sensitive documents to cloud services. You keep complete control over your data while getting accuracy that rivals enterprise solutions.

olmOCR-2-7B-1025

High-Performance Document OCR

Allen Institute's flagship model

This is fine-tuned from Qwen2.5-VL-7B-Instruct using GRPO reinforcement learning, scoring 82.4 on the olmOCR-bench evaluation.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load the model
model_id = "allenai/olmOCR-2-7B-1025"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")

# Process document
messages = [
    {"role": "user", "content": "Convert this document to markdown"},
    {"role": "user", "content": f"[IMAGE_PLACEHOLDER]"}
]

inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=4096)
result = tokenizer.decode(outputs[0], skip_special_tokens=True)

Key Features

Mathematical equations: Handles complex math expressions perfectly
Table recognition: Keeps table structure and formatting intact
Layout understanding: Manages multi-column documents and complex layouts
Large-scale processing: Built for processing millions of documents
Automated retries: Includes error handling and automatic rotation correction

PaddleOCR VL

Efficient Multilingual OCR

Ultra-compact vision-language model

Integrates NaViT visual encoder with ERNIE language model, supporting 109 languages with minimal resource usage.

import paddle
from paddlenlp import Taskflow

# Initialize OCR pipeline
ocr = Taskflow("document_parsing")

# Process document
result = ocr({"doc": "path/to/document.pdf"})
print(result)

Key Features

109 languages: Chinese, English, Japanese, Arabic, Hindi, Thai
Complex elements: Tables, formulas, charts recognition
High accuracy: State-of-the-art performance on OmniDocBench
Fast inference: Optimized for real-world deployment
Compact size: Efficient resource utilization

OCRFlux-3B

Multimodal Document Conversion

Preview release from ChatDOC

Fine-tuned from Qwen2.5-VL-3B-Instruct for clean markdown output from PDFs and images.

from transformers import Qwen2_5_VLForConditionalGeneration, AutoTokenizer, AutoProcessor
from qwen_vl_utils import process_vision_info

# Load model
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "ChatDOC/OCRFlux-3B",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
processor = AutoProcessor.from_pretrained("ChatDOC/OCRFlux-3B")

# Process image
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "path/to/image.jpg"},
            {"type": "text", "text": "Convert to markdown"}
        ]
    }
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs, videos=video_inputs, return_tensors="pt").to(model.device)

generated_ids = model.generate(**inputs, max_new_tokens=4096)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]

Key Features

Clean markdown: Structured output with proper formatting
Cross-page tables: Merges table data across multiple pages
Consumer hardware: Runs on GTX 3090 and similar GPUs
Scalable deployment: vLLM inference support
State-of-the-art accuracy: Superior parsing quality

MiniCPM-V 4.5

Mobile-Optimized OCR

Latest in the MiniCPM-V series

Built on Qwen3-8B and SigLIP2-400M, delivers exceptional performance for text recognition in images, documents, and videos.

from transformers import AutoModel, AutoTokenizer
import torch

# Load model
model = AutoModel.from_pretrained('openbmb/MiniCPM-V-4.5', trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-4.5', trust_remote_code=True)
model.eval().cuda()

# Process image
image = Image.open('path/to/image.jpg').convert('RGB')
question = 'Convert this document to text'

msgs = [{'role': 'user', 'content': [image, question]}]
result = model.chat(msgs=msgs, tokenizer=tokenizer, sampling=True, temperature=0.7)
print(result)

Key Features

Mobile deployment: Optimized for edge devices
Multi-image processing: Handles multiple images simultaneously
Video OCR: Text recognition in video content
State-of-the-art benchmarks: Leading performance across evaluations
Practical efficiency: Everyday application ready

InternVL2.5-4B

Compact Multimodal Understanding

Efficient vision-language model

Combines InternViT vision encoder with Qwen2.5 language model for comprehensive OCR and understanding.

import torch
from transformers import AutoTokenizer, AutoModel

# Load model
model = AutoModel.from_pretrained(
    'OpenGVLab/InternVL2_5-4B',
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    trust_remote_code=True
).eval().cuda()

tokenizer = AutoTokenizer.from_pretrained(
    'OpenGVLab/InternVL2_5-4B',
    trust_remote_code=True
)

# Process image
image = Image.open('path/to/image.jpg').convert('RGB')
question = 'Extract all text from this image'

response = model.chat(tokenizer, image, question)
print(response)

Key Features

Dynamic resolution: 448x448 pixel tile processing
Resource efficient: Suitable for constrained environments
Text recognition: Strong OCR performance
Reasoning tasks: Advanced multimodal understanding
Compact architecture: 4 billion parameters total

Granite Vision 3.3 2b

Document Understanding Specialist

IBM's vision-language model

Built on Granite 3.1-2b-instruct with SigLIP2 vision encoder for automated content extraction.

from transformers import AutoProcessor, AutoModelForVision2Seq
import torch

# Load model
processor = AutoProcessor.from_pretrained("ibm-granite/granite-vision-3.3-2b")
model = AutoModelForVision2Seq.from_pretrained("ibm-granite/granite-vision-3.3-2b", device_map="auto")

# Process image
image = Image.open("path/to/document.png").convert("RGB")
text_prompt = "<|start_of_text|>"

inputs = processor(text=text_prompt, images=image, return_tensors="pt").to(model.device)
generated_ids = model.generate(**inputs, max_new_tokens=500)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]

Key Features

Table extraction: Automated table content extraction
Chart recognition: Infographics and plot understanding
Multi-page support: Handles multi-page documents
Image segmentation: Advanced visual processing
Enhanced safety: Improved security features

TrOCR Large (SROIE Fine-tuned)

Specialized Text Recognition

Transformer-based OCR

Encoder-decoder architecture combining BEiT image transformer with RoBERTa text transformer.

from transformers import TrOCRProcessor, VisionEncoderDecoderModel
import torch

# Load model
processor = TrOCRProcessor.from_pretrained('microsoft/trocr-large-printed')
model = VisionEncoderDecoderModel.from_pretrained('microsoft/trocr-large-printed')

# Process image
image = Image.open('path/to/image.jpg').convert('RGB')
pixel_values = processor(images=image, return_tensors="pt").pixel_values
generated_ids = model.generate(pixel_values)

generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(generated_text)

Key Features

Single-line text: Optimized for printed text recognition
High accuracy: State-of-the-art performance on benchmarks
Transformer architecture: Modern deep learning approach
Pre-trained models: Leverages large-scale training data
Sequence processing: Handles text as sequential tokens

Choosing the Right Model

Use Case Considerations

Match model to needs

High accuracy: olmOCR-2-7B-1025, OCRFlux-3B
Efficiency: PaddleOCR VL, InternVL2.5-4B
Multilingual: PaddleOCR VL, MiniCPM-V 4.5
Mobile/Edge: MiniCPM-V 4.5, InternVL2.5-4B
Specialized: TrOCR (printed text), Granite Vision (documents)

Performance Optimization

Hardware considerations

GPU memory: Match model size to available VRAM
Batch processing: Use models supporting multiple images
Quantization: Consider quantized versions for efficiency
Local deployment: All models support local inference

Implementation Best Practices

Preprocessing

Optimize input quality

# Image preprocessing
def preprocess_image(image_path):
    image = Image.open(image_path).convert('RGB')
    # Resize if needed
    if max(image.size) > 2240:
        image = image.resize((2240, 2240), Image.Resampling.LANCZOS)
    return image

Error Handling

Robust processing

def safe_ocr_processing(model, image_path):
    try:
        image = preprocess_image(image_path)
        result = model.process(image)
        return result
    except Exception as e:
        logging.error(f"OCR processing failed: {e}")
        return None

Output Formatting

Structured results

def format_ocr_output(raw_text, confidence_scores=None):
    """Format OCR output with metadata"""
    return {
        "text": raw_text,
        "confidence": confidence_scores,
        "timestamp": datetime.now().isoformat(),
        "model_version": "olmOCR-2-7B-1025"
    }

These open source OCR models represent the cutting edge of document processing technology. Choose based on your specific requirements accuracy, speed, language support, or hardware constraints and start converting documents with unprecedented quality.

Why printf Beats echo in Linux Scripts

Mohammad Abu Mattar — Fri, 02 Jan 2026 00:00:00 GMT

Scripting Tip

You know that feeling when a script works perfectly on your machine but fails miserably somewhere else? That's probably because you're using echo for output. Let me show you why printf is a much better choice for your Linux scripts.

The Problem with echo

Unpredictable Behavior

echo's reliability issues

The echo command doesn't behave the same way everywhere. Options like -n (to suppress the newline) or escape sequences like \n and \t might work fine in one shell but act completely different in another.

Real-World Script Failures

You've got a script that runs great on your system, but when you deploy it to production or share it with a colleague, it starts acting up. These inconsistencies create bugs that are really hard to track down and fix.

Why printf is More Reliable

POSIX Standard Compliance

Predictable output everywhere

printf follows the POSIX standard, so it works exactly the same across all shells and systems. You don't have to worry about whether your escape sequences will actually work or not.

Advanced Formatting Capabilities

printf gives you real control over how your output looks. You can align text, format numbers precisely, and combine multiple values in a single command.

# Simple alignment
printf "%-10s %s\n" "User:" "$USER"

# Clean numeric formatting
printf "Usage: %.2f%%\n" 85.6789

# Multiple values at once
printf "%s logged in at %s\n" "$USER" "$(date)"

Replacing echo with printf

Basic Text Output

Simple substitution

Instead of echo "Hello, world", just write printf "Hello, world\n". The newline is right there in the command, so you always know what you're getting.

Suppressing Newlines

No more -n confusion

Forget about echo -n "Processing..." and its inconsistent behavior. With printf, you just omit the newline: printf "Processing...". Simple and reliable.

Variable Output Safety

Proper variable handling

Don't use echo $VARIABLE - it can cause all sorts of issues with spaces or special characters. Go with printf "%s\n" "$VARIABLE" instead. It's safe and predictable.

# Safe variable printing
name="John Doe"
printf "User: %s\n" "$name"

# Multiple variables work great
printf "User: %s | UID: %d\n" "$USER" "$UID"

Escape Sequences

Guaranteed processing

printf always handles escape sequences correctly. echo might just print them as literal text, depending on your shell.

# Reliable line breaks
printf "Line 1\nLine 2\n"

# Clean tabbed output
printf "Name:\t%s\nAge:\t%d\n" "$name" "$age"

Structured Output with printf

Table Formatting

Clean data presentation

You can create nicely aligned tables and structured output that other programs can easily read.

# CPU usage table
printf "%-8s %s\n" "CPU" "Usage"
printf "%-8s %d%%\n" "core0" 42
printf "%-8s %d%%\n" "core1" 37

CSV Generation

Consistent data export

Generate CSV files that format properly no matter where your script runs.

printf "%s,%s,%s\n" "Name" "Age" "City"
printf "%s,%s,%s\n" "$name" "$age" "$city"

When to Use Which Command

echo for Quick Tasks

Interactive use cases

echo is still fine for quick checks in the terminal:

Testing something quickly
Simple debugging output
Interactive shell sessions
One-off commands

printf for Scripts

Production code

Use printf when it matters:

Production scripts
Automated tools
Log file generation
Any output that other programs will read
Scripts that need to work across different systems

Locale Considerations

Numeric Formatting

Decimal separator issues

printf respects your system's locale settings. In some countries, decimal points become commas, which can break your scripts if you're not careful.

# Force standard decimal format
LC_NUMERIC=C printf "%.2f\n" 3.14159

Safe Numeric Handling

Prevent locale surprises

For scripts that need to work everywhere, set LC_NUMERIC=C or handle numbers explicitly to avoid any locale-related issues.

Making the Switch

Gradual Migration

Update existing scripts

You can replace your echo statements one by one:

echo "text" becomes printf "text\n"
echo -n "text" becomes printf "text"
echo $var becomes printf "%s\n" "$var"

Testing Output

Verify behavior

Always test your scripts on different systems and shells to make sure the output looks right everywhere.

The printf Advantage

printf gives you the reliability and control you need for serious scripting. echo works for quick tasks, but printf makes sure your scripts behave predictably no matter where they run. I recommend making printf your default choice for output in production code.

Essential Bash Variables for Every Script

Mohammad Abu Mattar — Fri, 26 Dec 2025 00:00:00 GMT

Overview

Quick Tip

You know what's worse than writing scripts? Writing scripts that break every time you move them to a different machine. Let's fix that with some built-in Bash variables that'll make your life way easier.

The Problem with Hard-coded Paths

We've all been there. You hard-code /home/john/.config in your script, and it works great on your laptop. Then you run it on the server, and boom everything breaks because the server uses a different username or directory structure. When you've got dozens (or hundreds) of scripts, tracking down all these hard-coded values becomes a real headache.

The Solution: Built-in Bash Variables

Bash already knows about your system. It's got built-in variables for script paths, user IDs, home directories, and more. Instead of guessing or hard-coding, just ask Bash. Your scripts will work everywhere without modification.

Key Bash Variables Summary

TL;DR

Use $BASH_SOURCE (or $0) to get the script path reliably
Check exit codes with $? to handle errors properly
Access arguments with $1, $2, $@, and $*
Get user IDs with $UID and $EUID for permission checks
Use XDG variables for standard user directories instead of hard-coding paths

Special Parameters

Get the Script Path

Sometimes you need to know where your script lives. Maybe you're building a help menu and want to show the script name, or you need to reference files relative to the script's location. Here's how you grab that info.

#!/usr/bin/env bash

# $BASH_SOURCE gives you the full path to this script
SCRIPT_PATH="$BASH_SOURCE"
echo "Full path: $SCRIPT_PATH"
# Output: /home/user/scripts/my_script.sh

# Use basename to strip the directory and keep just the filename
SCRIPT_NAME="$(basename "$BASH_SOURCE")"
echo "Script name: $SCRIPT_NAME"
# Output: my_script.sh

# Here's a practical use case: building a help menu
cat <<EOF
Usage: $SCRIPT_NAME [OPTIONS]

Options:
  -h, --help     Show this help message
  -v, --version  Show version information
EOF

There's also $0, which works similarly but has a quirk. If someone sources your script (like source my_script.sh), $0 returns "bash" instead of the script name. $BASH_SOURCE doesn't have this issue, so use that if you're writing Bash-specific code.

Check Exit Status

When a command finishes, it leaves behind an exit code kind of like a status report. If everything went fine, you get 0. If something went wrong, you get a number between 1 and 255. The variable $? holds this exit code, and you can use it to figure out what happened and respond accordingly.

#!/usr/bin/env bash

# Method 1: Explicit check using $?
ls /nonexistent_directory
if [[ $? -ne 0 ]]; then
  echo "Error: Directory does not exist"
  exit 1
fi

# Method 2: Cleaner approach test the command directly
# If ls succeeds, run the 'then' block. If it fails, run 'else'
if ls /home; then
  echo "Directory exists"
else
  echo "Directory does not exist"
fi

# Method 3: One-liner with logical operators
# && means "run this if the previous command succeeded"
# || means "run this if the previous command failed"
ls /home && echo "Success!" || echo "Failed!"

# Method 4: Handle specific error codes
# Some commands return different numbers for different errors
grep "pattern" file.txt
case $? in
  0) echo "Pattern found";;
  1) echo "Pattern not found";;
  2) echo "File error or syntax problem";;
  *) echo "Unknown error";;
esac

Here's the catch $? only remembers the last command. If you run another command, the old exit code is gone. So if you need it later, save it right away:

some_command
exit_code=$?  # Save it now!
echo "Did some other stuff"
# Now we can still check the original exit code
if [[ $exit_code -ne 0 ]]; then
  echo "Original command failed"
fi

Access Script Arguments

You'll want your scripts to accept arguments things like filenames, options, or configuration values. Bash makes this pretty straightforward, though the syntax looks a bit weird at first. Let's break it down.

#!/usr/bin/env bash

# Individual arguments are numbered: $1, $2, $3, etc.
echo "First argument: $1"
echo "Second argument: $2"
echo "Third argument: $3"
# Run like: ./script.sh hello world test
# Output: hello, world, test

# Count how many arguments were passed
echo "Total arguments: ${#@}"

# Loop through all arguments
# $@ treats each argument separately, which is usually what you want
echo -e "\nProcessing all arguments:"
for arg in "$@"; do
  echo "  - $arg"
done

# The difference between $@ and $*
# This matters when your arguments contain spaces
function demo_at {
  # Each argument stays separate
  printf '@: [%s]\n' "$@"
}

function demo_star {
  # All arguments merge into one string
  printf '*: [%s]\n' "$*"
}

echo -e "\nDifference between \$@ and \$*:"
demo_at "arg one" "arg two"    # Prints: @: [arg one] and @: [arg two]
demo_star "arg one" "arg two"  # Prints: *: [arg one arg two] (all together)

# Real-world example: A function that processes files
function process_files() {
  # First, make sure we got at least one file
  if [[ ${#@} -eq 0 ]]; then
    echo "Error: No files specified"
    echo "Usage: process_files file1 file2 ..."
    return 1
  fi

  # Process each file
  for file in "$@"; do
    if [[ -f "$file" ]]; then
      echo "Processing: $file"
      # Your actual file processing logic goes here
      # wc -l "$file"  # Example: count lines
    else
      echo "Warning: File not found: $file"
    fi
  done
}

# You'd call it like this:
# process_files report.txt data.csv notes.md

Always use quotes around "$@". If you don't, filenames with spaces will break. For example, without quotes, "my file.txt" becomes two separate arguments: "my" and "file.txt". With quotes, it stays as one argument.

Environment Variables

Get User ID

Sometimes you need to know who's running your script. Maybe you're creating user-specific temp files, or you need to check if someone's running as root. That's where $UID and $EUID come in.

#!/usr/bin/env bash

# Root check: A common pattern for admin scripts
# Root user always has UID 0
if [[ $EUID -eq 0 ]]; then
  echo "Running with root privileges"
  # Safe to do system-level stuff here
else
  echo "Running as regular user (UID: $UID)"
  echo "Some operations may require sudo"
  # Maybe prompt for sudo, or just warn and continue
fi

# Build paths specific to the current user
# This is useful for multi-user systems
USER_CACHE_DIR="/run/user/$UID/cache"
echo "User cache directory: $USER_CACHE_DIR"
# Example: /run/user/1000/cache

# Practical example: Create a temp directory that's unique per user
# This prevents conflicts when multiple users run your script
function create_user_temp() {
  local temp_dir="/tmp/myapp-$UID"

  if [[ ! -d "$temp_dir" ]]; then
    mkdir -p "$temp_dir"
    echo "Created temp directory: $temp_dir"
  fi

  echo "$temp_dir"
}

# Now each user gets their own isolated temp space
TEMP_DIR=$(create_user_temp)
echo "Using temp directory: $TEMP_DIR"
# User 1000: /tmp/myapp-1000
# User 1001: /tmp/myapp-1001

You might be wondering about the difference between $UID and $EUID. Honestly, for most scripts, they're the same. They only differ in weird edge cases involving sudo or special setuid programs. When in doubt, just use $EUID for permission checks it's what matters.

Use XDG Directory Specification

Here's a question: where should your script save config files? Or cache data? Or store downloaded files? You might think "just use ~/.config" but what if a user wants to organize things differently? That's where XDG variables come in they're the standard way Linux systems handle user directories.

#!/usr/bin/env bash

# Set XDG variables if they're not already set
# The syntax ${VAR:=default} means "use VAR if it exists, otherwise use default"
export "${XDG_CONFIG_HOME:=$HOME/.config}"        # Config files
export "${XDG_CACHE_HOME:=$HOME/.cache}"          # Temporary cache data
export "${XDG_DATA_HOME:=$HOME/.local/share}"    # Application data
export "${XDG_STATE_HOME:=$HOME/.local/state}"   # State files (logs, history, etc.)

# Now build your app's specific directories
APP_NAME="myapp"
APP_CONFIG_DIR="$XDG_CONFIG_HOME/$APP_NAME"  # ~/.config/myapp
APP_CACHE_DIR="$XDG_CACHE_HOME/$APP_NAME"    # ~/.cache/myapp
APP_DATA_DIR="$XDG_DATA_HOME/$APP_NAME"      # ~/.local/share/myapp

# Create these directories if they don't exist yet
function initialize_app_directories() {
  mkdir -p "$APP_CONFIG_DIR"
  mkdir -p "$APP_CACHE_DIR"
  mkdir -p "$APP_DATA_DIR"

  echo "Initialized application directories:"
  echo "  Config: $APP_CONFIG_DIR"
  echo "  Cache:  $APP_CACHE_DIR"
  echo "  Data:   $APP_DATA_DIR"
}

initialize_app_directories

# Example: Save settings to the config directory
function save_config() {
  local config_file="$APP_CONFIG_DIR/settings.conf"

  cat > "$config_file" <<EOF
# Application settings
debug_mode=false
log_level=info
max_connections=100
EOF

  echo "Saved config to: $config_file"
}

# Example: Load settings from the config directory
function load_config() {
  local config_file="$APP_CONFIG_DIR/settings.conf"

  if [[ -f "$config_file" ]]; then
    # Source the config file to load variables
    source "$config_file"
    echo "Loaded config from: $config_file"
    echo "Debug mode: $debug_mode"
    echo "Log level: $log_level"
  else
    echo "Config file not found: $config_file"
    return 1
  fi
}

save_config
load_config

This isn't just about being "correct" it's about respecting how users organize their systems. Some people put config files on a separate partition, or use custom paths for backups. By using XDG variables, your script just works in all these scenarios.

Complete Example: Production-Ready Script

Alright, let's put it all together. Here's what a professional script looks like when you use all these variables properly. It's got error handling, logging, argument processing, and follows XDG standards. Feel free to use this as a template for your own scripts.

#!/usr/bin/env bash

# Strict mode: exit on errors, undefined variables, and pipe failures
set -euo pipefail

# Set up XDG directories with sensible defaults
export "${XDG_CONFIG_HOME:=$HOME/.config}"
export "${XDG_CACHE_HOME:=$HOME/.cache}"
export "${XDG_DATA_HOME:=$HOME/.local/share}"

# Grab script info using the variables we learned about
SCRIPT_NAME="$(basename "$BASH_SOURCE")"
SCRIPT_DIR="$(cd "$(dirname "$BASH_SOURCE")" && pwd)"
APP_NAME="myapp"

# Build our application's directory structure
APP_CONFIG_DIR="$XDG_CONFIG_HOME/$APP_NAME"
APP_CACHE_DIR="$XDG_CACHE_HOME/$APP_NAME"
APP_LOG_FILE="$APP_CACHE_DIR/app.log"

# ANSI color codes for pretty output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'  # Reset to no color

# Simple logging function with timestamps
# Usage: log "INFO" "Something happened"
function log() {
  local level="$1"
  shift
  local message="$*"
  local timestamp=$(date '+%Y-%m-%d %H:%M:%S')

  # Print to screen AND save to log file
  echo "[$timestamp] [$level] $message" | tee -a "$APP_LOG_FILE"
}

# Centralized error handling
# This logs the error and exits with a non-zero status
function error_exit() {
  log "ERROR" "$*" >&2
  exit 1
}

# Check if we have everything we need to run
function check_prerequisites() {
  # Example: Some scripts need root privileges
  if [[ $EUID -ne 0 ]]; then
    error_exit "This script must be run as root (current UID: $UID)"
  fi

  # Verify required commands are installed
  local required_commands=("curl" "jq" "aws")
  for cmd in "${required_commands[@]}"; do
    if ! command -v "$cmd" &> /dev/null; then
      error_exit "Required command not found: $cmd. Please install it first."
    fi
  done

  log "INFO" "All prerequisites satisfied"
}

# Set up directories and initialize the app
function initialize() {
  # Create necessary directories
  mkdir -p "$APP_CONFIG_DIR" "$APP_CACHE_DIR"
  touch "$APP_LOG_FILE"

  # Log some useful info for debugging
  log "INFO" "Initialized $APP_NAME"
  log "INFO" "Script: $SCRIPT_NAME (located at $SCRIPT_DIR)"
  log "INFO" "User: $USER (UID: $UID)"
}

# Parse command-line arguments
function process_arguments() {
  # If no arguments, show help and exit
  if [[ ${#@} -eq 0 ]]; then
    cat <<EOF
Usage: $SCRIPT_NAME [OPTIONS] [FILES...]

Options:
  -h, --help     Show this help message
  -v, --verbose  Enable verbose logging
  -d, --debug    Enable debug mode

Examples:
  $SCRIPT_NAME file1.txt file2.txt
  $SCRIPT_NAME --verbose *.log
EOF
    exit 0
  fi

  # Set up some flags
  local verbose=false
  local debug=false
  local files=()

  # Loop through all arguments
  while [[ $# -gt 0 ]]; do
    case "$1" in
      -h|--help)
        process_arguments  # Recursively call to show help
        ;;
      -v|--verbose)
        verbose=true
        log "INFO" "Verbose mode enabled"
        shift
        ;;
      -d|--debug)
        debug=true
        set -x  # This makes Bash print every command before running it
        shift
        ;;
      -*)
        error_exit "Unknown option: $1"
        ;;
      *)
        # Anything that doesn't start with - is treated as a file
        files+=("$1")
        shift
        ;;
    esac
  done

  log "INFO" "Processing ${#files[@]} file(s)"

  # Process each file
  for file in "${files[@]}"; do
    if [[ -f "$file" ]]; then
      log "INFO" "Processing file: $file"
      # Your actual file processing logic would go here
      # Example: wc -l "$file"
    else
      log "WARN" "File not found: $file"
    fi
  done
}

# Main entry point
function main() {
  initialize
  check_prerequisites || exit $?
  process_arguments "$@" || exit $?

  log "INFO" "Script completed successfully"
}

# This is where everything starts
# We pass all command-line arguments to main using "$@"
main "$@"

Quick Reference

Variable	Description	Example
`$0` or `$BASH_SOURCE`	Script path	`/home/user/script.sh`
`$(basename $0)`	Script name only	`script.sh`
`$?`	Exit status of last command	`0` (success) or `1-255` (error)
`$1`, `$2`, `$3`...	Positional parameters	First, second, third argument
`$@`	All arguments as array	Each argument separate
`$*`	All arguments as string	All arguments in one string
`${#@}`	Number of arguments	`3`
`$UID`	Real user ID	`1000`
`$EUID`	Effective user ID	`0` (when using sudo)
`$USER`	Username	`john`
`$HOME`	Home directory	`/home/john`
`$XDG_CONFIG_HOME`	Config directory	`~/.config`
`$XDG_CACHE_HOME`	Cache directory	`~/.cache`
`$XDG_DATA_HOME`	Data directory	`~/.local/share`

Best Practices

Do these things:

Always quote your variables: Use "$var" instead of $var. This prevents weird issues when variables contain spaces or special characters.
Set defaults for XDG variables: Use ${XDG_CONFIG_HOME:=$HOME/.config} so your script works even if the variable isn't set.
Prefer $BASH_SOURCE over $0: It's more reliable, especially when your script might be sourced instead of executed.
Check exit codes for important operations: Don't just assume things worked. Check $? or use if statements.
Quote "$@" when passing arguments: This preserves spaces in filenames and arguments.

Avoid these mistakes:

Don't hard-code paths: Never write /home/john/.config directly. Use variables so your script works for everyone.
Don't use $* by default: It merges all arguments into one string, which usually isn't what you want. Stick with "$@".
Don't assume $? sticks around: It changes with every command. Save it immediately if you need it later.
Don't forget to quote path variables: Unquoted variables break when paths have spaces.
Don't use $UID for permission checks: Use $EUID instead it's what actually matters for access control.

References

Per-App Shell History for Bash

Mohammad Abu Mattar — Thu, 18 Dec 2025 00:00:00 GMT

Terminal Chaos? Organize Your Bash History!

Ever jumped between iTerm2, Ghostty, and VS Code's terminal only to have your command history get all mixed up? This Bash snippet keeps things clean by creating separate history files for each terminal app you use.

The Problem

Mixed Command History

When you use multiple terminal emulators (iTerm, Alacritty, Ghostty, VS Code, etc.), they all share the same shell history file by default. This means commands you ran in one app show up when you press the up arrow in another. It gets messy fast, especially when you use different terminals for different projects or contexts.

Context Switching Issues

Different terminals used for different purposes (work, personal, testing) end up with mixed command histories, making it difficult to maintain context.

The Solution

Per-App History Files

This Bash function detects which terminal application you're using and automatically assigns a dedicated history file for it. Apple Terminal and Ghostty on Linux get the default history, while everything else gets its own isolated file. Clean, simple, and automatic.

Automatic Organization

TL;DR

Automatically creates separate history files for each terminal app.
Keeps Apple Terminal and Ghostty (Linux) on the default history.
Works seamlessly with iTerm2, Alacritty, Warp, Kitty, and more.
Just add it to your ~/.bashrc or ~/.bash_profile and forget about it.

Implementation

Bash Function Setup

function set_app_history() {

  # Get terminal app name (fallback: Apple_Terminal)
  local app="${TERM_PROGRAM:-Apple_Terminal}"

  # Default history file
  HISTFILE="$HOME/.bash/.bash_history"

  # Ghostty on Linux / WSL → use default history
  if [[ "$app" == "Ghostty" && "$(uname -s)" == "Linux" ]]; then
    export HISTFILE
    return
  fi

  # Apple Terminal → use default history
  if [[ "$app" == "Apple_Terminal" ]]; then
    export HISTFILE
    return
  fi

  # Everything else → per-app history
  app="${app//[^A-Za-z0-9]/_}"
  HISTFILE="$HOME/.bash/.bash_history_${app}"
  export HISTFILE
}

set_app_history

Benefits and Usage

Organization Advantages

Why This Helps

This approach keeps your shell history organized without any manual intervention. You can switch between terminals without worrying about command pollution or losing context. Plus, it's especially useful if you use different terminals for work, personal projects, or testing each gets its own clean slate.

Managing History Files

Bonus Tip

Want to see which history files you have? Run this:

ls -lh ~/.bash/.bash_history*

You'll see all your per-app history files. Each one is tied to a specific terminal emulator, so you can even back them up or clean them individually.

Community Discussion

Your History Management

Your Turn

How do you organize your shell history? Do you use per-app separation or something else? Drop your setup below!

Alternative Approaches

Share your favorite methods for keeping shell history organized across different terminal applications.

Per-App Shell History for Zsh

Mohammad Abu Mattar — Thu, 18 Dec 2025 00:00:00 GMT

Terminal Chaos? Organize Your Shell History!

Ever jumped between iTerm2, Ghostty, and VS Code's terminal only to have your command history get all mixed up? This Zsh snippet keeps things clean by creating separate history files for each terminal app you use.

The Problem

Mixed Command History

Context Switching Issues

Different terminals used for different purposes (work, personal, testing) end up with mixed command histories, making it difficult to maintain context.

The Solution

Per-App History Files

This Zsh function detects which terminal application you're using and automatically assigns a dedicated history file for it. Apple Terminal and Ghostty on Linux get the default history, while everything else gets its own isolated file. Clean, simple, and automatic.

Automatic Organization

TL;DR

Automatically creates separate history files for each terminal app.
Keeps Apple Terminal and Ghostty (Linux) on the default history.
Works seamlessly with iTerm2, Alacritty, Warp, Kitty, and more.
Just add it to your ~/.zshrc and forget about it.

Implementation

Zsh Function Setup

function set_app_history() {

  # Get terminal app name (fallback: Apple_Terminal)
  local app=${TERM_PROGRAM:-Apple_Terminal}

  # Default history file
  HISTFILE="$HOME/.zsh/.zhistory"

  # Ghostty on Linux / WSL → use default history
  if [[ "$app" == "Ghostty" && "$(uname -s)" == "Linux" ]]; then
    export HISTFILE
    return
  fi

  # Apple Terminal → use default history
  if [[ "$app" == "Apple_Terminal" ]]; then
    export HISTFILE
    return
  fi

  # Everything else → per-app history
  app=${app//[^A-Za-z0-9]/_}
  HISTFILE="$HOME/.zsh/.zhistory_${app}"
  export HISTFILE
}

set_app_history

Benefits and Usage

Organization Advantages

Why This Helps

Managing History Files

Bonus Tip

Want to see which history files you have? Run this:

ls -lh ~/.zsh/.zhistory*

You'll see all your per-app history files. Each one is tied to a specific terminal emulator, so you can even back them up or clean them individually.

Community Discussion

Your History Management

Your Turn

How do you organize your shell history? Do you use per-app separation or something else? Drop your setup below!

Alternative Approaches

Share your favorite methods for keeping shell history organized across different terminal applications.

Optimizing your python code with slots?

Mohammad Abu Mattar — Wed, 16 Jul 2025 00:00:00 GMT

Memory Optimization with `slots`

Understanding the Problem

Dev Tip: Optimizing Data Models in Big Data Workflows with `slots`

In big data and MLOps workflows, you often work with massive datasets where you create millions of objects to represent data points, features, or model predictions. Traditional Python classes can consume a significant amount of memory due to the hidden __dict__ attribute. This overhead can lead to memory bottlenecks and slow down processing, especially when you're working with large-scale data pipelines and machine learning models.

How `slots` Works

The __slots__ attribute is a simple yet powerful way to optimize your code by drastically reducing the memory footprint of your objects. By defining __slots__, you tell Python to allocate a fixed memory space for a class's attributes, bypassing the need for a dynamic __dict__. This makes your application more memory-efficient and can lead to faster attribute access, which is crucial for high-performance computing tasks in MLOps and big data.

Real-World Use Case: MLOps Data Pipeline

Customer Churn Prediction Example

Real-World Scenario: A Data Pipeline for MLOps

Imagine you're building a data pipeline for a machine learning model that predicts customer churn. Your pipeline processes millions of customer records, and each record is represented by a Python object.

Using a regular class, each customer object would have a memory overhead from its __dict__. When you process millions of these objects, the cumulative memory usage becomes immense, potentially crashing your application or requiring more expensive, higher-memory machines.

By using a class with __slots__, you can create a memory-efficient data model. This approach minimizes memory consumption, allowing you to process more data with the same resources and speeding up your pipeline. This is critical in MLOps where efficient resource utilization is key to managing costs and scaling operations.

Big Data Processing Benefits

Real-World Scenario: MLOps Data Pipeline with `slots`

In an MLOps pipeline, data is often represented as objects for processing, feature engineering, and model training. When dealing with big data, memory efficiency is critical to avoid crashes and keep cloud computing costs low. Using __slots__ can drastically reduce the memory footprint of these data objects, making your pipeline more robust and scalable.

The following full code snippet simulates a big data MLOps workflow where we process a large number of customer records. We will create two versions of a Customer data class: one with the default Python behavior and one optimized with __slots__. The code will then compare the memory usage and time taken to create a million instances of each, demonstrating the practical benefits of __slots__.

Code Implementation

Class Definitions

import sys
import time
import random
import pandas as pd
from typing import List

# --- Part 1: Data Model Definitions ---

class Customer:
    """
    A regular Python class to represent a customer record.
    This class uses a default __dict__ to store attributes.
    """
    def __init__(self, customer_id: int, age: int, monthly_spend: float, churned: bool):
        self.customer_id = customer_id
        self.age = age
        self.monthly_spend = monthly_spend
        self.churned = churned

class OptimizedCustomer:
    """
    An optimized customer class using __slots__ for memory efficiency.
    This class explicitly defines its attributes, eliminating the __dict__ overhead.
    """
    __slots__ = ['customer_id', 'age', 'monthly_spend', 'churned']

    def __init__(self, customer_id: int, age: int, monthly_spend: float, churned: bool):
        self.customer_id = customer_id
        self.age = age
        self.monthly_spend = monthly_spend
        self.churned = churned

Data Generation Functions

# --- Part 2: Data Generation and Object Creation ---

def generate_customer_data(num_records: int) -> List[tuple]:
    """Generates a list of tuples representing raw customer data."""
    data = []
    for i in range(num_records):
        customer_id = i
        age = random.randint(20, 70)
        monthly_spend = round(random.uniform(25.0, 500.0), 2)
        churned = random.choice([True, False])
        data.append((customer_id, age, monthly_spend, churned))
    return data

def create_objects(data: List[tuple], class_type: type) -> List:
    """Creates a list of objects from raw data using the specified class."""
    return [class_type(*record) for record in data]

Performance Testing

# --- Part 3: Performance Comparison ---

def run_performance_test(num_records: int):
    """
    Runs a performance test to compare memory and time for both classes.
    """
    print(f"--- Running performance test with {num_records:,} records ---")
    raw_data = generate_customer_data(num_records)

    # Test the regular class
    start_time_regular = time.time()
    regular_customers = create_objects(raw_data, Customer)
    end_time_regular = time.time()
    memory_regular = sum(sys.getsizeof(c) for c in regular_customers) + sys.getsizeof(regular_customers)

    # Test the optimized class
    start_time_slotted = time.time()
    slotted_customers = create_objects(raw_data, OptimizedCustomer)
    end_time_slotted = time.time()
    memory_slotted = sum(sys.getsizeof(c) for c in slotted_customers) + sys.getsizeof(slotted_customers)

    # Print results
    print("\nRegular Class Performance:")
    print(f"  - Total Memory: {memory_regular / (1024**2):.2f} MB")
    print(f"  - Time Taken: {end_time_regular - start_time_regular:.4f} seconds")

    print("\nSlotted Class Performance:")
    print(f"  - Total Memory: {memory_slotted / (1024**2):.2f} MB")
    print(f"  - Time Taken: {end_time_slotted - start_time_slotted:.4f} seconds")

    # Calculate and print the savings
    memory_saved_mb = (memory_regular - memory_slotted) / (1024**2)
    time_saved_s = (end_time_regular - start_time_slotted)

    print("\n--- Summary of Savings ---")
    print(f"Memory Saved: {memory_saved_mb:.2f} MB ({ (memory_saved_mb / (memory_regular / (1024**2))) * 100:.2f}%)")
    print(f"⏱Slotted Class Creation Time is faster by: {time_saved_s:.4f} seconds")

    # Optional: Demonstrate a simple MLOps task like converting to a DataFrame
    print("\n--- MLOps Task: Converting to a Pandas DataFrame ---")
    # This task is just for demonstration and doesn't show a direct slots benefit here
    df_regular = pd.DataFrame([c.__dict__ for c in regular_customers])
    df_slotted = pd.DataFrame([[c.customer_id, c.age, c.monthly_spend, c.churned] for c in slotted_customers],
                               columns=['customer_id', 'age', 'monthly_spend', 'churned'])
    print("Successfully converted both object lists to Pandas DataFrames.")
    print(f"DataFrame head from Slotted Class:\n{df_slotted.head()}")

# --- Part 4: Execution ---
if __name__ == "__main__":
    NUM_RECORDS = 1_000_000 # 1 million records
    run_performance_test(NUM_RECORDS)

Results and Analysis

Performance Comparison

This code snippet demonstrates how to optimize memory usage in Python by using __slots__ in a data-heavy application, specifically in an MLOps context. It compares the memory and performance of a regular class versus an optimized class with __slots__, showing significant savings in both memory and processing time when handling large datasets.

Practical Benefits

The implementation shows measurable improvements in memory usage and object creation speed, making it ideal for big data processing and MLOps workflows.

List S3 Buckets

Mohammad Abu Mattar — Wed, 25 Jun 2025 00:00:00 GMT

Overview

Multi-Profile S3 Management

Multi-Profile S3 Safari! Ever juggled multiple AWS accounts and needed a quick S3 bucket inventory across all of them? This Python script is your guide!

Use Case

Perfect for organizations managing multiple AWS accounts or developers working with different IAM roles and profiles.

The Problem

Multi-Profile Complexity

When you're working with multiple AWS accounts or IAM roles through different profiles, getting a consolidated view of your S3 buckets can be a hassle. Switching contexts or running commands repeatedly for each profile is inefficient and prone to oversight, making comprehensive inventory or auditing a chore.

Manual Process Inefficiencies

Traditional approaches require manual switching between profiles and running separate commands for each account.

The Solution

Automated Profile Discovery

This Python script, powered by Boto3, automates the discovery of all your configured AWS CLI profiles. It then iterates through each profile, attempting to list its S3 buckets. The script provides a clear, profile-by-profile breakdown and includes robust error handling for issues like missing credentials or access denied for specific profiles, ensuring it runs smoothly even in complex setups.

Comprehensive Error Handling

Gracefully handles various error conditions including missing credentials, access denied, and network issues.

Features

Key Capabilities

TL;DR

Lists S3 buckets across ALL configured AWS CLI profiles in one go.
Automatically discovers available AWS profiles.
Provides a clear, per-profile output of S3 buckets.
Gracefully handles errors for individual profiles (e.g., credential issues, access denied).

Output Format

Organized display showing buckets grouped by AWS profile with clear visual separation.

Code Implementation

Dependencies and Setup

import boto3
from botocore.exceptions import NoCredentialsError, ClientError

Profile Discovery Function

def get_aws_profiles():
    """Get all configured AWS profiles from the AWS CLI configuration."""
    try:
        session = boto3.Session()
        return session.available_profiles
    except Exception as e:
        print(f"Error getting AWS profiles: {str(e)}")
        return []

Bucket Listing Function

def list_s3_buckets_for_profile(profile_name):
    """
    Lists all S3 buckets for a specific AWS profile.
    Returns a list of bucket names or an empty list if an error occurs.
    """
    buckets = []
    try:
        session = boto3.Session(profile_name=profile_name)
        s3_client = session.client('s3')
        response = s3_client.list_buckets()
        if response['Buckets']:
            for bucket in response['Buckets']:
                buckets.append(bucket['Name'])
    except NoCredentialsError:
        print(f"  Warning: No credentials found for profile '{profile_name}'. Skipping.")
    except ClientError as e:
        error_code = e.response.get("Error", {}).get("Code")
        error_message = e.response.get("Error", {}).get("Message")
        print(f"  Warning: AWS Client Error for profile '{profile_name}' ({error_code}): {error_message}. Skipping.")
    except Exception as e:
        print(f"  Warning: An unexpected error occurred for profile '{profile_name}': {e}. Skipping.")
    return buckets

Main Display Function

def display_all_s3_buckets_by_profile():
    """
    Fetches and displays S3 buckets for all configured AWS profiles.
    """
    profiles = get_aws_profiles()

    if not profiles:
        print("No AWS profiles found. Please configure your AWS CLI.")
        return

    print(f"\nChecking S3 Buckets across {len(profiles)} AWS Profiles:")
    print("-" * 40)

    for profile in sorted(profiles):
        print(f"\nProfile: {profile}")
        print("  S3 Buckets:")
        buckets = list_s3_buckets_for_profile(profile)
        if buckets:
            for bucket_name in buckets:
                print(f"    - {bucket_name}")
        else:
            print("    No buckets or inaccessible for this profile.")
        print("-" * 40)

    if not any(list_s3_buckets_for_profile(p) for p in profiles):
        print("\nNo S3 buckets found across any configured profiles or all were inaccessible.")


if __name__ == "__main__":
    display_all_s3_buckets_by_profile()

Benefits and Usage

Operational Advantages

Why This Helps

This script is a game-changer for anyone managing multiple AWS environments. It drastically simplifies S3 bucket auditing, inventory tasks, and compliance checks across your entire AWS footprint. Get a unified view without the manual grind, making your cloud operations more efficient and less error-prone.

Use Cases

Ideal for inventory management, compliance auditing, and multi-account AWS operations.

Community Discussion

Your S3 Management Approaches

Your Turn

How do you manage S3 bucket information in your Python projects? Any favorite Boto3 tricks for S3 you'd like to share?

Alternative Tools

Share your preferred methods for managing S3 resources across multiple AWS accounts.

AWS Secrets Manager

Mohammad Abu Mattar — Wed, 18 Jun 2025 00:00:00 GMT

Need to load secrets in your Node.js app without exposing them? Here's how.

If you're still storing API keys or database credentials in .env files or hardcoding them into your codebase, it's time for a better approach. Secrets should stay secret especially in production.

The Problem

Managing Sensitive Values

Managing sensitive values across environments can get messy fast. Hardcoded secrets are risky, and .env files aren't ideal when you're working with teams or deploying to the cloud. It's easy to lose control over where that information ends up.

The Solution

AWS Secrets Manager Overview

AWS Secrets Manager gives you a safe, centralized place to store secrets. You can fetch them at runtime using the AWS SDK, so you don’t need to keep secrets on disk or in code. This snippet shows how to pull them securely in a Node.js app using SDK v3.

TL;DR

Use AWS Secrets Manager to securely access secrets in Node.js.
Avoid hardcoding secrets or using local .env files in production.
This snippet helps fetch secrets using the AWS SDK v3.

Code Snippet (TypeScript):

import {
  SecretsManagerClient,
  GetSecretValueCommand,
  ResourceNotFoundException,
  SecretsManagerServiceException,
} from '@aws-sdk/client-secrets-manager';
import {config} from 'dotenv';

config();

interface CachedSecret {
  value: string;
  expiry: number;
}

const secretCache = new Map<string, CachedSecret>();

const DEFAULT_CACHE_TTL = 5 * 60 * 1000;

const defaultRegion = process.env.AWS_REGION;

/**
 * Custom error for when a secret is not found in AWS Secrets Manager.
 * This is thrown when the secret does not exist or cannot be accessed.
 */
class SecretNotFoundError extends Error {
  constructor(secretName: string) {
    super(`Secret "${secretName}" not found in AWS Secrets Manager.`);
    this.name = 'SecretNotFoundError';
  }
}

/**
 * Custom error for when a secret's value is invalid.
 * This can happen if the secret is binary, empty, or not a string.
 */
class InvalidSecretValueError extends Error {
  constructor(secretName: string) {
    super(
      `Secret "${secretName}" is binary or empty, or does not contain a string value.`,
    );
    this.name = 'InvalidSecretValueError';
  }
}

let secretsManagerClient: SecretsManagerClient | null = null;

/**
 * Initializes and returns a singleton SecretsManagerClient.
 * This prevents recreating the client on every `fetchSecret` call.
 * @param region - The AWS region to use for the client.
 * @returns An initialized SecretsManagerClient instance.
 */
function getSecretsManagerClient(region: string): SecretsManagerClient {
  if (!region) {
    throw new Error(
      'AWS_REGION is not defined. Please set it in your .env file or pass it as an argument.',
    );
  }
  if (!secretsManagerClient) {
    secretsManagerClient = new SecretsManagerClient({region});
  }
  return secretsManagerClient;
}

/**
 * Fetches a secret's string value from AWS Secrets Manager with optional caching.
 * @param secretName - The name or ARN of the secret.
 * @param options - Optional configuration for fetching the secret.
 * @param options.region - Overrides the default AWS region for this fetch operation.
 * @param options.cacheTTL - Time-to-live for the cached secret in milliseconds. Set to 0 to disable caching for this call.
 * @returns A promise that resolves to the secret string.
 * @throws {SecretNotFoundError} If the secret does not exist.
 * @throws {InvalidSecretValueError} If the secret's value is binary or empty.
 * @throws {Error} For other AWS SDK or network-related errors.
 */
export async function fetchSecret(
  secretName: string,
  options?: {region?: string; cacheTTL?: number},
): Promise<string> {
  const region = options?.region || defaultRegion;
  const cacheTTL =
    options?.cacheTTL !== undefined ? options.cacheTTL : DEFAULT_CACHE_TTL;

  if (!region) {
    throw new Error(
      "AWS_REGION is not defined. Ensure it's in your .env file or passed in options.",
    );
  }

  if (cacheTTL > 0) {
    const cached = secretCache.get(secretName);
    if (cached && Date.now() < cached.expiry) {
      return cached.value;
    }
  }

  const client = getSecretsManagerClient(region);

  try {
    const command = new GetSecretValueCommand({SecretId: secretName});
    const response = await client.send(command);

    if (response.SecretString) {
      const secretValue = response.SecretString;
      if (cacheTTL > 0) {
        secretCache.set(secretName, {
          value: secretValue,
          expiry: Date.now() + cacheTTL,
        });
      }
      return secretValue;
    } else if (response.SecretBinary) {
      throw new InvalidSecretValueError(secretName);
    } else {
      throw new InvalidSecretValueError(secretName);
    }
  } catch (error: any) {
    console.error(`[ERROR] Failed to fetch secret "${secretName}":`, error);
    if (error instanceof ResourceNotFoundException) {
      throw new SecretNotFoundError(secretName);
    } else if (error instanceof SecretsManagerServiceException) {
      throw new Error(
        `AWS Secrets Manager error for "${secretName}": ${error.message} (Code: ${error.name})`,
      );
    } else {
      throw new Error(
        `An unexpected error occurred while fetching secret "${secretName}": ${error.message}`,
      );
    }
  } finally {
    // Add any cleanup or finalization logic here.
    // For example, if you had an active connection or resource to close.
    // In this specific code, there isn't an obvious resource to clean up
    // within the fetchSecret function itself, as the client is a singleton
    // and caching is handled by a Map.
    // However, for demonstration, you could log something:
    console.log(`[INFO] Finished attempting to fetch secret "${secretName}".`);
  }
}

Why This Matters

This setup helps keep your apps more secure and your secrets off disk. It fits perfectly into cloud-native workflows, especially when you’re using IAM roles or CI/CD pipelines. It’s simple, flexible, and secure.

Over to you

How do you handle secrets in your projects? Tried this approach before?

Check S3 Bucket Existence

Mohammad Abu Mattar — Wed, 11 Jun 2025 00:00:00 GMT

Quick Tip

Don’t let your deployment blow up because of a missing S3 bucket. This Bash script lets you check if a bucket exists before anything fails clean and simple.

The Problem

Missing Bucket Failures

The Problem

If your CI/CD pipeline tries to upload files or sync data to an S3 bucket that doesn't exist, things will break fast. You'll waste time, pipeline minutes, and possibly miss something obvious. A quick pre-check could prevent all that.

Impact on Deployments

Failed deployments due to missing S3 buckets can waste valuable CI/CD time and delay releases.

The Solution

Bash Script Overview

The Fix

This script checks if an S3 bucket exists using the AWS CLI. It accepts --bucket or --b flags, gives color-coded logs, and adds a bit of fun to your terminal. Perfect for pre-checks in CI/CD jobs or local scripts.

Key Features

TL;DR

Use this script to check if an S3 bucket exists before your pipeline tries to use it.
Accepts --bucket or --b to pass the bucket name.
Includes clean logs, color output, and fun formatting for better readability.

Script Implementation

Color Definitions and Setup

#!/usr/bin/env bash

# Color definitions
GREEN='\033[0;32m'
RED='\033[0;31m'
YELLOW='\033[1;33m'
CYAN='\033[0;36m'
NC='\033[0m' # No Color

ASCII Banner Function

#######################################
# Prints a fun ASCII banner header.
# Globals:
#   CYAN
#   NC
# Arguments:
#   None
# Outputs:
#   Writes header text to stdout.
#######################################
function print_fun() {
  echo -e "${CYAN}"
  echo "╭──────────────────────────────╮"
  echo "│   S3 Bucket Checker Bash  │"
  echo "╰──────────────────────────────╯"
  echo -e "${NC}"
}

Logging Function

#######################################
# Prints a log message with timestamp and log level.
# Globals:
#   CYAN, GREEN, RED, NC
# Arguments:
#   $1 - Log level (INFO, SUCCESS, ERROR)
#   $2 - Log message
# Outputs:
#   Writes formatted log message to stdout.
#######################################
function log() {
  local type="$1"
  local message="$2"
  local timestamp
  timestamp=$(date "+%Y-%m-%d %H:%M:%S")

  case "$type" in
    INFO)
      echo -e "${CYAN}[${timestamp}] [INFO]:${NC} $message"
      ;;
    SUCCESS)
      echo -e "${GREEN}[${timestamp}] [SUCCESS]:${NC} $message"
      ;;
    ERROR)
      echo -e "${RED}[${timestamp}] [ERROR]:${NC} $message"
      ;;
    *)
      echo -e "[${timestamp}] [LOG]: $message"
      ;;
  esac
}

Initialization and Argument Parsing

#######################################
# Initializes script by parsing CLI arguments.
# Exits with error if bucket name is not provided.
# Globals:
#   BUCKET_NAME, YELLOW, NC
# Arguments:
#   --bucket | --b <bucket-name>
# Outputs:
#   Sets BUCKET_NAME variable.
#######################################
function init() {
  while [[ "$#" -gt 0 ]]; do
    case "$1" in
      --bucket|--b)
        BUCKET_NAME="$2"
        shift 2
        ;;
      *)
        log ERROR "Unknown argument: $1"
        exit 1
        ;;
    esac
  done

  if [[ -z "$BUCKET_NAME" ]]; then
    log ERROR "Bucket name not provided. Use --bucket or --b to specify it."
    echo -e "${YELLOW}Example:${NC} ./check_bucket.sh --bucket my-bucket-name"
    exit 1
  fi
}

Bucket Existence Check

#######################################
# Checks if the specified S3 bucket exists.
# Globals:
#   BUCKET_NAME
# Outputs:
#   Success or error message to stdout.
# Returns:
#   0 if bucket exists, exits with 1 otherwise.
#######################################
function check_bucket_exists() {
  log INFO "Checking if bucket '$BUCKET_NAME' exists..."

  if aws s3api head-bucket --bucket "$BUCKET_NAME" 2>/dev/null; then
    log SUCCESS "Bucket '$BUCKET_NAME' exists and is accessible."
  else
    log ERROR "Bucket '$BUCKET_NAME' does not exist or access is denied."
    exit 1
  fi
}

Main Function

#######################################
# Main function that runs the script.
# Calls init and check_bucket_exists in order.
# Arguments:
#   Passes all script arguments to init.
#######################################
function main() {
  print_fun
  init "$@"
  check_bucket_exists
}

main "$@"

Usage and Benefits

Integration with CI/CD

Why This Helps

Adding this check to your CI/CD steps helps avoid silly failures and gives you faster feedback if something's missing or misconfigured. It's quick, readable, and reusable and you'll thank yourself later.

Command Line Usage

Run the script with: ./check_bucket.sh --bucket my-bucket-name or ./check_bucket.sh --b my-bucket-name

Community Discussion

Your Approaches

Your Turn

How do you handle S3 checks in your automation? Got a trick or tool you use often? Let's swap ideas.

Alternative Solutions

Share your favorite methods for validating S3 bucket existence in different environments.