Scrapy Rotating Proxies: Middleware and Retries

Scrapy is the framework most people reach for once a scraping project outgrows a script, and proxies are the first thing that goes wrong at scale. The default settings send 16 concurrent requests per domain from one address, which is a burst no real visitor produces, and the default retry policy gives up on exactly the status codes a proxy is meant to get you past. This guide covers the two ways to wire rotating proxies into Scrapy, the middleware that actually matters, and the sticky-session pattern for anything with a login.

Two ways to rotate, and which one you need

A rotating gateway. Your provider gives you one hostname and port, and hands you a different exit IP on every new connection. You configure a single proxy URL and rotation is the provider's problem. This is how our residential and mobile products work, and it is the right shape for almost every Scrapy project.

A list of proxies. You hold many individual proxy servers, usually datacenter IPs, and your code chooses one per request, tracks which are dead, and retires them. This is what packages like scrapy-rotating-proxies manage. It is more work and it is only necessary when the proxies themselves do not rotate.

The rest of this guide assumes a gateway, because that is the common case, and notes where a list differs.

The minimal setup: one proxy for the whole spider

Scrapy ships with HttpProxyMiddleware enabled. It reads request.meta["proxy"] and, if the URL contains credentials, sets the Proxy-Authorization header. The least code that works is a spider that sets the meta on every request:

import scrapy

PROXY = "http://USERNAME:PASSWORD@HOST:PORT"   # gateway URL from your dashboard

class ProductsSpider(scrapy.Spider):
    name = "products"
    start_urls = ["https://example.com/category/1"]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url, meta={"proxy": PROXY})

    def parse(self, response):
        for href in response.css("a.product::attr(href)").getall():
            yield response.follow(href, callback=self.parse_product, meta={"proxy": PROXY})

    def parse_product(self, response):
        yield {"url": response.url, "title": response.css("h1::text").get()}

Because the gateway rotates per connection, every request already exits from a different IP. You do not need rotation logic; you need the settings below.

Settings that decide whether rotation helps

# settings.py
CONCURRENT_REQUESTS = 32
CONCURRENT_REQUESTS_PER_DOMAIN = 8        # default 16 is a burst; residential targets prefer 4-8
DOWNLOAD_DELAY = 0.5
RANDOMIZE_DOWNLOAD_DELAY = True

AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 4.0

RETRY_ENABLED = True
RETRY_TIMES = 4
RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 524, 408, 429, 403]   # 403 and 429 added

DOWNLOAD_TIMEOUT = 30
COOKIES_ENABLED = False                    # unless you are holding a session on purpose

Two of these do most of the work. Adding 403 and 429 to the retry codes means a blocked or rate-limited request is retried, and with a rotating gateway the retry leaves from a new IP, which is the whole point. Lowering concurrency per domain stops the target from seeing a machine-gun pattern even when each request has a different address; anti-bot systems correlate on timing and paths, not only on IP.

A middleware that swaps the IP on failure

Scrapy's retry middleware already re-sends the request. With a gateway the new connection gets a new IP automatically. If you hold a proxy list instead, or you want to move a sticky session to a fresh identity after a block, write a small downloader middleware:

# middlewares.py
import random, uuid

class ProxyMiddleware:
    def __init__(self, base):
        self.base = base        # "http://USERNAME:PASSWORD@HOST:PORT"

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.get("PROXY_BASE"))

    def process_request(self, request, spider):
        if "proxy" not in request.meta:
            request.meta["proxy"] = self.base

    def process_response(self, request, response, spider):
        if response.status in (403, 429):
            # new sticky session id -> new exit IP on the retry
            request.meta["proxy"] = self.base.replace("USERNAME", "USERNAME-session-" + uuid.uuid4().hex[:8])
            request.dont_filter = True
            return request
        return response

# settings.py
PROXY_BASE = "http://USERNAME:PASSWORD@HOST:PORT"
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ProxyMiddleware": 350,
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 400,
}

The order matters: your middleware runs before HttpProxyMiddleware so the built-in one sees the final proxy URL and sets the authorization header. The session-id trick in process_response depends on your provider's username syntax for sticky sessions; copy the exact format from your dashboard rather than from this example.

Sticky sessions for logins and carts

Rotating on every request is wrong for any flow that carries state. A login followed by a page fetch from a different country is how account checks get triggered. Hold one IP for the flow by using the same sticky proxy URL for every request in it, and turn cookies back on for that spider:

class AccountSpider(scrapy.Spider):
    name = "account"
    custom_settings = {"COOKIES_ENABLED": True, "CONCURRENT_REQUESTS": 1}

    def start_requests(self):
        sid = uuid.uuid4().hex[:8]
        self.proxy = "http://USERNAME-session-%s:PASSWORD@HOST:PORT" % sid
        yield scrapy.FormRequest("https://example.com/login",
                                 formdata={"user": "me", "pass": "secret"},
                                 meta={"proxy": self.proxy}, callback=self.after_login)

    def after_login(self, response):
        yield scrapy.Request("https://example.com/account/orders",
                             meta={"proxy": self.proxy}, callback=self.parse_orders)

Sticky durations differ by product; when the provider's window expires, the next connection gets a new IP, so keep flows shorter than the window or refresh the session id deliberately between flows.

Debugging: what to check when it still fails

407 Proxy Authentication Required means the proxy rejected you, not the target: wrong credentials, wrong port, or an IP-whitelisted plan being used from an unlisted address. Nothing about the target is involved yet.
403 on the first request through a fresh IP is rarely the IP. It is usually the TLS fingerprint or headers: Scrapy's default client is identifiable before it sends a header. See TLS fingerprinting and curl_cffi; for Scrapy specifically, set a consistent browser user agent and consider a fingerprint-aware download handler.
Every request succeeds but the data is wrong often means geo-targeting is not what you think. Log response.meta["proxy"] and check a few exits with the IP lookup.
Throughput collapses when retries pile up. Watch the retry/count stat; if it approaches your request count, lower concurrency before adding IPs.

For the surrounding framework, selectors, pipelines and exports, see the Scrapy tutorial. This post is the part that tutorial only sketched: the proxy layer that decides whether a real crawl finishes.

Frequently asked questions

Do I need a rotation middleware if my provider rotates automatically?

No. A rotating gateway hands you a new exit IP on every connection, so a single proxy URL in settings or in request meta is enough. What you still need is a retry policy that treats 403 and 429 as retryable and, for multi-step flows, a way to hold one IP across several requests.

How do I set a proxy per request in Scrapy?

Set request.meta['proxy'] to the full proxy URL, including username and password, before the request leaves the spider or inside a downloader middleware. Scrapy's built-in HttpProxyMiddleware reads that key and sets the Proxy-Authorization header for you.

Why does Scrapy keep getting 403 or 429 through my proxies?

Usually concurrency. Scrapy's default of 16 concurrent requests per domain, all through one exit, looks like a burst from one visitor. Lower CONCURRENT_REQUESTS_PER_DOMAIN, enable AutoThrottle, and add 403 and 429 to RETRY_HTTP_CODES so a blocked request is retried through a fresh IP instead of being dropped.

How do I keep the same IP for a login flow in Scrapy?

Use your provider's sticky session syntax, which usually means adding a session identifier to the proxy username, and reuse that exact proxy URL for every request in the flow. Generate a new identifier when you want a new identity.

Should I use scrapy-rotating-proxies?

Only if you are managing your own list of individual proxy servers. It tracks which proxies are dead and rotates among the live ones. With a provider gateway that rotates for you, the package adds moving parts without adding rotation.

Rotating Proxies in Scrapy: Middleware, Per-Request Proxies, Retries and Sticky Sessions