Most scraping tutorials stop at requests plus BeautifulSoup on a single page. That works until you need ten thousand pages, or the target starts fighting back. This guide covers the patterns that separate a script from a system: concurrency with back-pressure, retries that do not make things worse, finding the JSON the page is already using, parsing that survives redesigns, and crawls you can stop and resume. If you are starting from zero, read how to build a web scraper in Python first.
1. Concurrency With Back-Pressure
Sequential requests waste almost all their time waiting on the network. Async fixes that — but unbounded async is just a faster way to get banned. The pattern you want is a semaphore-bounded worker pool: many requests in flight, but a hard ceiling.
import asyncio, httpx
ENDPOINT = "geo.spyderproxy.com:12321"
PROXY = f"http://USERNAME:PASSWORD@{ENDPOINT}"
CONCURRENCY = 20
async def fetch(client, url, sem):
async with sem: # never more than N in flight
r = await client.get(url, timeout=20)
r.raise_for_status()
return url, r.text
async def crawl(urls):
sem = asyncio.Semaphore(CONCURRENCY)
limits = httpx.Limits(max_connections=CONCURRENCY)
async with httpx.AsyncClient(proxy=PROXY, limits=limits, http2=True) as client:
tasks = [fetch(client, u, sem) for u in urls]
return await asyncio.gather(*tasks, return_exceptions=True)Two details matter. return_exceptions=True stops one failure killing the whole batch. And reusing a single AsyncClient reuses connections — creating a client per request throws away TLS handshakes and is dramatically slower. See the asyncio tutorial and concurrency vs parallelism for the fundamentals.
2. Retries That Do Not Make It Worse
Retrying immediately on failure turns a struggling server into a dead one. Use exponential backoff with jitter, and only retry the errors that are actually transient.
import asyncio, random, httpx
RETRYABLE = {429, 500, 502, 503, 504}
async def fetch_retry(client, url, attempts=4):
for attempt in range(attempts):
try:
r = await client.get(url, timeout=20)
if r.status_code in RETRYABLE:
raise httpx.HTTPStatusError("retryable", request=r.request, response=r)
r.raise_for_status()
return r
except (httpx.TransportError, httpx.HTTPStatusError) as exc:
if attempt == attempts - 1:
raise
wait = (2 ** attempt) + random.uniform(0, 1) # jitter avoids thundering herd
await asyncio.sleep(wait)Do not retry a 404 or a 403 — those will not fix themselves. A 429 means slow down: honour the Retry-After header if it is present. Rotate to a fresh IP on repeated blocks rather than hammering the same one.
3. Find the Hidden API Before You Parse HTML
This is the single highest-leverage habit in scraping. Most modern sites render from a JSON endpoint the browser already calls. Open DevTools, go to the Network tab, filter to Fetch/XHR, and interact with the page.
If you find that endpoint you get clean structured data, no HTML parsing, no headless browser, and far fewer bytes. Three things to check before relying on it: whether it needs auth headers or a token, whether pagination is a cursor or an offset, and whether it enforces its own rate limits. Compare with API vs web scraping.
4. Parsing That Survives a Redesign
Brittle selectors are why scrapers rot. Class names like css-1x7f2k are generated and will change without notice. Anchor on things that carry meaning:
- Structured data first. Many pages embed JSON-LD in a
scripttag — parse that instead of the DOM whenever it exists. - Stable attributes —
data-*,id, ARIA roles — beat generated class names. - Relative anchoring — find a label, then take its sibling, rather than walking a fixed path from the root.
- Validate what you extract. A price that parses to
Noneshould raise, not silently write a null row.
from decimal import Decimal, InvalidOperation
def parse_price(raw):
if not raw:
raise ValueError("missing price")
cleaned = raw.replace("$", "").replace(",", "").strip()
try:
return Decimal(cleaned)
except InvalidOperation:
raise ValueError(f"unparseable price: {raw!r}")Add a canary: if the share of rows failing validation jumps, the site changed and you want to know today, not next month. XPath is often more expressive than CSS for relative anchoring.
5. Make Crawls Resumable
Any long crawl will be interrupted. Design for it from the start: keep a persistent record of what is queued, done, and failed, and check it before each fetch. SQLite is enough — you do not need a queue broker.
import sqlite3
db = sqlite3.connect("crawl.db")
db.execute("CREATE TABLE IF NOT EXISTS seen (url TEXT PRIMARY KEY, status TEXT)")
def claim(url):
try:
db.execute("INSERT INTO seen (url, status) VALUES (?, 'pending')", (url,))
db.commit()
return True # ours to process
except sqlite3.IntegrityError:
return False # already seen, skipNow a restart resumes instead of starting over, and duplicate URLs cost nothing. This pairs directly with the two-phase pattern in list crawling in Python.
6. Look Like a Real Client
Advanced targets fingerprint more than your user agent. Headers must be internally consistent — a Chrome user agent alongside Python-default Accept headers is an obvious mismatch. Some sites also fingerprint the TLS handshake itself, which no header change can disguise; that is when you move to a real browser via Playwright.
The part no fingerprint work can replace is the IP. One address making thousands of requests is the clearest signal there is, regardless of how convincing your headers are. Rotate through residential proxies so requests come from many real ISP addresses. See user agents and avoiding detection.
7. Instrument It
At scale you need to know how things fail, not just that they did. Track status-code distribution over time, p95 latency, retry and validation-failure rates, and throughput per proxy pool. A slow creep in 403s is an early warning that you are being profiled — catching it lets you back off before you get blocked outright.
Frequently Asked Questions
What makes web scraping advanced rather than basic?
Basic scraping fetches and parses a page. Advanced scraping deals with scale and adversarial conditions: bounded concurrency, retry and backoff policies, resumable state, resilient parsing that survives redesigns, client fingerprinting, and IP rotation. The parsing is rarely the hard part — reliability is.
How many concurrent requests should a Python scraper use?
There is no universal number. Start low, around 10 to 20 in flight, and watch your error rate. If 429s or 403s climb, reduce it. Rotating residential proxies let you raise total throughput safely because the load is spread across many IPs rather than concentrated on one.
Should I scrape the HTML or find the site's API?
Always look for the API first. Open DevTools, filter the Network tab to Fetch/XHR, and interact with the page. If the site loads data from a JSON endpoint you get clean structured data with no HTML parsing and no headless browser. Fall back to HTML only when there is no usable endpoint.
How do I stop my scraper breaking when a site redesigns?
Avoid generated class names. Prefer embedded JSON-LD, then stable attributes such as data-* and ARIA roles, then relative anchoring from a text label. Validate every extracted field and alert when the validation failure rate jumps, so you find out the day a site changes rather than weeks later.
Conclusion
Advanced scraping is mostly engineering discipline: bound your concurrency, back off politely, prefer the API the page already calls, parse against things that carry meaning, persist your state so crawls resume, present a consistent client, and measure everything. Do those and the remaining variable is your IP reputation — which is exactly what rotating residential proxies are for.
Run advanced crawls without the blocks: SpyderProxy residential proxies from $2.75/GB — rotating IPs across 195+ countries, HTTP(S) and SOCKS5, unlimited concurrent sessions.