Requests vs HTTPX vs aiohttp: Timeouts, Proxies, HTTP/2 and Async Compared

Python has three HTTP clients that people actually use in production, and the differences between them are not about speed. They are about defaults that bite you at 3am, how each one takes a proxy, and whether the client can look like a browser. Versions here are the ones we run: requests 2.34, HTTPX 0.28 and aiohttp 3.14.

The comparison

The defaults that cause outages

The single most expensive difference is the timeout. requests has no default timeout at all. A call without one waits as long as the server keeps the socket open, which is how a scraper quietly stops making progress overnight while looking healthy. Always pass it:

import requests

r = requests.get("https://example.com", timeout=(10, 30))  # (connect, read)

HTTPX applies 5 seconds to connect, read, write and pool by default, which is safer but often too short for slow targets, so set it deliberately. aiohttp is the middle ground: 300 seconds in total and 30 seconds to establish the connection, overridable per request with ClientTimeout. In every case, write the number down rather than inheriting it. Our post on requests timeouts covers the exception types worth catching.

Proxies in each client

This is where people copy the wrong snippet, because the HTTPX argument changed. In 0.28 the plural proxies argument is gone.

PROXY = "http://USERNAME:PASSWORD@HOST:PORT"   # from your dashboard

# requests: a dict, per scheme
import requests
r = requests.get("https://httpbin.org/ip",
                 proxies={"http": PROXY, "https": PROXY}, timeout=30)

# httpx 0.28: one string on the client
import httpx
with httpx.Client(proxy=PROXY, timeout=30.0) as c:
    r = c.get("https://httpbin.org/ip")

# aiohttp: per request, or on the session
import aiohttp, asyncio
async def main():
    async with aiohttp.ClientSession() as s:
        async with s.get("https://httpbin.org/ip", proxy=PROXY) as r:
            print(await r.json())
asyncio.run(main())

Two things trip people up here. aiohttp ignores HTTP_PROXY and HTTPS_PROXY from the environment unless you construct the session with trust_env=True. And aiohttp does not speak SOCKS natively, so a socks5 URL in that proxy argument fails until you install aiohttp-socks and use its connector.

Concurrency, which is the real speed question

requests does one request at a time per thread. If you need a hundred pages, the honest comparison is not requests against httpx but blocking against async. Both async clients handle that with a semaphore to keep the target from being hammered:

import asyncio, httpx

async def fetch(client, url, sem):
    async with sem:
        r = await client.get(url, timeout=30.0)
        return url, r.status_code

async def main(urls):
    sem = asyncio.Semaphore(10)          # ten in flight, not a hundred
    async with httpx.AsyncClient(proxy=PROXY) as client:
        return await asyncio.gather(*(fetch(client, u, sem) for u in urls))

asyncio.run(main(urls))

Pick the concurrency number from the target's tolerance, not from your CPU count. Ten parallel requests through a rotating pool is usually more productive than fifty that trigger rate limiting, and every retry you avoid is bandwidth you do not pay for.

What none of them fix

All three are recognisable as themselves. Their TLS handshakes differ from any browser, and HTTPX is the only one that can offer HTTP/2 at all, so a site with bot protection can often classify the client before reading a single header. If your requests fail on the first hit from a clean residential IP, the client is the tell, not the address. The fix is a client that reproduces a browser handshake, which we cover in TLS fingerprinting and curl_cffi.

Which one to use

• A script, a cron job, an API call. requests, with an explicit timeout. It is everywhere and the code is the shortest.
• A new project that may need async or HTTP/2. HTTPX. The sync API is close enough to requests that migration is mostly mechanical, and you get the async client for free. See our HTTPX guide.
• A crawler pulling thousands of pages. aiohttp or httpx.AsyncClient, with a semaphore, retries and a rotating proxy pool. Our rotating proxies in Python guide covers the rotation side.

Whichever you choose, the proxy layer is separate from the client. Residential IPs start at $1.75 per GB, and the same three lines above work with any of them.

Frequently asked questions

Should I switch from requests to httpx?

Switch when you need async, HTTP/2, or sane defaults. HTTPX takes almost the same code, adds an async client, and applies a 5 second default timeout where requests applies none at all. Stay on requests when the script is small and synchronous, because it is installed everywhere and the ecosystem around it is larger.

Which Python HTTP client is fastest?

For one request at a time the difference is noise, because the network dominates. For many requests at once, aiohttp and httpx.AsyncClient are far ahead of requests, which blocks on each call. The real speed decision is concurrency, not library choice.

How do I set a proxy in each client?

requests takes a dictionary: proxies={'http': url, 'https': url}. HTTPX 0.28 takes a single string on the client: httpx.Client(proxy=url), and the old proxies argument has been removed. aiohttp takes it per request or on the session: session.get(url, proxy=url), and it reads HTTP_PROXY and HTTPS_PROXY from the environment when you pass trust_env=True.

Do any of them support SOCKS5 proxies?

All three, through an extra package. Install requests[socks] for requests, httpx[socks] for HTTPX, and aiohttp-socks for aiohttp. Then use a socks5 or socks5h URL. The h in socks5h means the proxy resolves DNS, which is what you usually want so your local resolver does not leak the hostnames you visit.

Does HTTP/2 matter for scraping?

It matters for how your client looks, not how fast it is. Browsers speak HTTP/2, and the settings they send in that handshake are part of what bot-protection vendors fingerprint. A client that offers only HTTP/1.1 stands out from the browser it claims to be. HTTPX with the h2 package installed is the only one of these three that speaks HTTP/2.