How to Reduce Proxy Bandwidth Costs (Without Scraping Less)

Residential proxies are billed per gigabyte, so your bill is decided by how many bytes you pull, not how many pages you visit. Most teams never look at that number — and end up paying for images, fonts, video and analytics scripts they immediately throw away. The gap between a careless scraper and a careful one on the identical job is routinely 10–50x. Here is how to close it.

Where the Bytes Actually Go

Rough figures for a typical e-commerce product page:

The data you actually wanted — a price, a title, a rating — is a few hundred bytes. Everything else is overhead you are paying for by the gigabyte.

1. Block Resources You Never Wanted (Biggest Win)

If you use a headless browser, everything loads by default. Images, fonts, stylesheets, video, analytics. Abort them at the network layer and the page still parses fine:

from playwright.sync_api import sync_playwright

BLOCK = {"image", "media", "font", "stylesheet"}

def fetch(url, proxy):
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True, proxy=proxy)
        page = browser.new_page()
        # abort anything we will never read
        page.route("**/*", lambda route: route.abort()
                   if route.request.resource_type in BLOCK else route.continue_())
        page.goto(url, wait_until="domcontentloaded", timeout=60000)
        html = page.content()
        browser.close()
        return html

This alone commonly cuts 70–90% of bytes. Two notes: keep stylesheet loaded if the site hides content via CSS you depend on, and prefer domcontentloaded over networkidle so you are not waiting on trackers you just blocked.

2. Find the JSON Endpoint

The single largest saving available. Most modern sites render from an internal API the browser already calls — open DevTools, filter the Network tab to Fetch/XHR, and interact with the page. If you find it, you replace a 3 MB render with a 5 KB JSON response, and you drop the browser entirely.

That is a ~600x reduction on that request, plus far lower CPU. Details in advanced web scraping in Python.

3. Turn On Compression

HTML compresses extremely well — often 70–80%. Most HTTP clients request it by default, but check, because a missing header silently multiplies your bill:

import httpx

ENDPOINT = "geo.spyderproxy.com:12321"
PROXY = f"http://USERNAME:PASSWORD@{ENDPOINT}"

headers = {"Accept-Encoding": "gzip, deflate, br"}   # br = brotli, smallest
with httpx.Client(proxy=PROXY, headers=headers, http2=True) as c:
    r = c.get("https://example.com")
    print(len(r.content), "bytes decoded")

Billing counts bytes over the wire, so compressed transfer is what you pay for.

4. Never Fetch the Same Page Twice

Persist what you have fetched and skip it on the next run. Where the server supports it, use conditional requests — a 304 Not Modified costs a few hundred bytes instead of the whole page:

headers = {}
if stored_etag:
    headers["If-None-Match"] = stored_etag

r = client.get(url, headers=headers)
if r.status_code == 304:
    pass                     # unchanged, near-zero bandwidth
else:
    save(url, r.text, r.headers.get("etag"))

Also crawl incrementally: re-check pages that plausibly changed (active listings, live prices) far more often than ones that do not.

5. Take the Narrowest Slice

• Filter server-side. One filtered search request beats fetching a whole category and discarding 90% of it.
• Use HEAD for existence checks — headers only, no body.
• Stop paginating once you have what you need, rather than walking to the last page by habit.
• Skip detail pages when the listing page already carries the fields you want.

6. Only Use a Browser When You Must

A headless browser is the most expensive way to fetch a page — in bandwidth and time. Treat it as a fallback, not a default: try plain HTTP first, and escalate to a browser only for the pages that genuinely require rendering. Many crawls need it for less than 10% of URLs. See best web crawlers for tools that switch between the two automatically.

7. Measure It, Or None of This Sticks

Log bytes per request and per record extracted. Cost per useful record is the number that matters — not price per GB.

total = 0
for url in urls:
    r = client.get(url)
    total += len(r.content)
print(f"{total/1024/1024:.1f} MB for {len(urls)} pages "
      f"= {total/len(urls)/1024:.0f} KB/page")

Once you can see KB per page, the wins above become obvious — and you will notice immediately when a site redesign doubles your costs.

What This Is Worth

Take a 10,000-page crawl at $2.75/GB:

• Full browser render, nothing blocked: ~30 GB → ~$82
• Browser with images/fonts/CSS blocked: ~4 GB → ~$11
• Plain HTTP + compression: ~1 GB → ~$2.75
• Underlying JSON endpoint: ~0.1 GB → ~$0.28

Same data, same 10,000 pages. The difference is entirely in what you chose to download.

Frequently Asked Questions

How much bandwidth does web scraping use?

An HTML-only request is typically 50–200 KB. The same page fully rendered in a headless browser with images, fonts and trackers is usually 2–5 MB. So 10,000 pages can be anywhere from about 1 GB to over 50 GB depending purely on what you let load.

How do I reduce proxy bandwidth costs?

Block images, fonts, media and stylesheets in your headless browser; use the site's underlying JSON endpoint instead of rendering HTML where one exists; enable gzip or brotli compression; cache and use conditional requests so unchanged pages return a 304; and only use a browser for the pages that genuinely need it.

Does blocking images break scraping?

Rarely. Text, prices and structured data live in the HTML and JSON, not the images. Block image, media and font requests by resource type. Keep stylesheets if the site uses CSS to show or hide content you depend on, otherwise block those too.

Is cheaper per GB always better?

No. A cheap pool with a low success rate can cost more per successful request than a more expensive one that works first time. Measure cost per useful record, which accounts for retries and failures, rather than headline price per gigabyte.

Conclusion

Proxy bills are decided by bytes, and most bytes in a typical scrape are things you never wanted. Block them, prefer the JSON the page already fetches, compress, cache, scope narrowly, and reserve the browser for pages that truly need it. Then measure KB per page so the savings hold as sites change. Done properly this is a 10–50x difference — far more than you will ever save by shopping around on price per GB.

Then make the per-GB rate work for you too: SpyderProxy residential proxies from $2.75/GB, or budget residential from $1.75/GB — no minimum commitment.