Python Async HTTP Requests with aiohttp: Fetch Multiple URLs Concurrently

Quick Tip

Reuse one aiohttp session, fan your requests out with asyncio.gather, and cap them with a semaphore to fetch hundreds of URLs in the time one loop would take.

The Problem

The Problem

A script that fetches a list of URLs with requests in a plain for loop spends almost all of its time doing nothing. Each call opens a connection, sends the request, and then blocks until the response comes back before the next one even starts. The network is slow and your CPU is fast, so the program just sits and waits, one round trip at a time.

Why sequential requests waste time

HTTP is I/O bound. The bottleneck isn’t your code, it’s the latency of each request. If a single call takes 200ms and you have 200 URLs, a sequential loop takes 40 seconds, and 39 of those seconds are pure waiting. Nothing about that work needs to happen in order, yet the loop forces it to.

The real-world impact

That waiting shows up as slow batch jobs, sluggish data-collection scripts, and API clients that time out because they can’t keep up. Scale the list up and it gets worse linearly: a report that pulls from a few hundred endpoints turns a quick task into a coffee break, and a service that fans out to several upstreams makes every user wait for the slowest chain of calls.

The Solution

The Fix

Use aiohttp with asyncio so the requests overlap. While one call waits on the network, the event loop starts the next one. Reuse a single ClientSession so connections get pooled, launch everything with asyncio.gather, and put a Semaphore in front to keep the number of in-flight requests sane.

TL;DR

• Create one aiohttp.ClientSession and reuse it for every request.
• Fire all the fetches concurrently with asyncio.gather instead of a blocking loop.
• Cap concurrency with asyncio.Semaphore so you don’t overwhelm the server or get rate-limited.

Script Implementation

Setup and imports

Start with the async pieces you need: asyncio for the event loop and aiohttp for the HTTP calls. A small result type keeps the return values easy to read.

1
#!/usr/bin/env python3
2
"""fetch_urls.py - fetch many URLs concurrently with aiohttp."""
3
import asyncio
4
import time
5
from dataclasses import dataclass
6

7
import aiohttp
8

9
# Cap how many requests run at once. Tune to the server and your rate limits.
10
MAX_CONCURRENCY = 10
11
REQUEST_TIMEOUT = aiohttp.ClientTimeout(total=15)
12

13

14
@dataclass
15
class Result:
16
    url: str
17
    status: int | None  # HTTP status, or None if the request never completed
18
    body: str | None  # response text on success
19
    error: str | None  # error message on failure

Fetching one URL safely

Each fetch takes the shared session and the semaphore. The semaphore blocks once MAX_CONCURRENCY requests are already in flight, so the rest queue up instead of stampeding the server. The try/except keeps a single bad URL from crashing the whole batch.

1
async def fetch_one(
2
    session: aiohttp.ClientSession,
3
    sem: asyncio.Semaphore,
4
    url: str,
5
) -> Result:
6
    # The semaphore is the throttle: only MAX_CONCURRENCY of these run at once.
7
    async with sem:
8
        try:
9
            async with session.get(url) as response:
10
                # Read the body while the connection is still open.
11
                text = await response.text()
12
                return Result(url, response.status, text, None)
13
        except aiohttp.ClientError as exc:
14
            # Network, DNS, or connection errors land here.
15
            return Result(url, None, None, f"client error: {exc}")
16
        except asyncio.TimeoutError:
17
            return Result(url, None, None, "timed out")

Fetching the whole batch

Create one session and one semaphore, then build a task per URL and hand them all to asyncio.gather. Because every fetch_one already catches its own errors, gather returns a clean list of Result objects with no partial failures to unpack.

1
async def fetch_all(urls: list[str]) -> list[Result]:
2
    sem = asyncio.Semaphore(MAX_CONCURRENCY)
3
    # One session for the whole run so connections get pooled and reused.
4
    async with aiohttp.ClientSession(timeout=REQUEST_TIMEOUT) as session:
5
        tasks = [fetch_one(session, sem, url) for url in urls]
6
        # gather runs them concurrently and preserves input order.
7
        return await asyncio.gather(*tasks)

Entry point

Wrap the run in asyncio.run, then print a short summary. Timing the run makes the win obvious the first time you compare it to a sequential loop.

1
def main() -> None:
2
    urls = [f"https://httpbin.org/delay/1?id={i}" for i in range(20)]
3

4
    start = time.perf_counter()
5
    results = asyncio.run(fetch_all(urls))
6
    elapsed = time.perf_counter() - start
7

8
    ok = sum(1 for r in results if r.status == 200)
9
    failed = [r for r in results if r.error]
10
    print(f"fetched {ok}/{len(results)} in {elapsed:.2f}s")
11
    for r in failed:
12
        print(f"  failed: {r.url} ({r.error})")
13

14

15
if __name__ == "__main__":
16
    main()

Usage and Benefits

Why This Helps

One session, one semaphore, and one gather call turn a linear wait into an overlapping one. The requests still take the same time individually, but they no longer take that time one after another, so a batch that scaled with the number of URLs now scales with your concurrency limit instead.

Real invocations

1
# Install the dependency first.
2
pip install aiohttp
3

4
# Run the script. Twenty 1-second requests finish in about 2-3 seconds,
5
# not 20, because up to MAX_CONCURRENCY of them run at the same time.
6
python fetch_urls.py

Tuning concurrency for the target

There’s no universal number for MAX_CONCURRENCY. A generous internal API might handle 50 in-flight requests; a rate-limited public one might block you above 5. Start low, watch for 429 responses, and raise it until you hit the server’s comfort zone.

1
# Lower the limit for a strict public API to avoid 429s.
2
MAX_CONCURRENCY = 5
3

4
# Raise it for a fast internal service that can take the load.
5
MAX_CONCURRENCY = 50

Comparison

How the concurrent approach stacks up against the other common ways to fetch a list of URLs.

The threaded pool is a fine middle ground when the rest of your code is synchronous, but an event loop scales to far more concurrent connections without spinning up a thread per request.

Frequently Asked Questions

References

• aiohttp documentation: client quickstart
• aiohttp documentation: client reference and ClientSession
• Python docs: asyncio.gather
• Python docs: asyncio.Semaphore
• Python docs: asyncio.as_completed
• httpx documentation: async client