Bot detection used to be a list. Known datacenter ranges, a few bad user agents, a request-per-minute threshold — break one rule, get blocked. That world is gone. Modern anti-bot systems collect dozens of weak signals and feed them to a model trained on labelled human and automated traffic. The output is not a rule violation; it is a score. Understanding how that score is built is the difference between a scraper that quietly works and one that gets blocked for reasons you cannot see.
Why the Shift to Machine Learning Changed Everything
Rule-based detection has an obvious weakness: every rule is a single point of failure that a scraper can fix. Change the user agent, the rule stops firing. Machine learning removes that. A model weighs many signals at once, so no individual signal is decisive — and none of them alone is fixable.
Two practical consequences follow. First, you cannot debug your way out by changing one thing: you fix the header, and the score barely moves. Second, detection is probabilistic, which is why you see graded responses — a CAPTCHA, then throttling, then a 403 — rather than a clean yes or no.
The Six Layers You Are Scored On
1. Network and IP reputation
The heaviest-weighted layer, and the cheapest to evaluate. Systems check the ASN behind your IP (hosting provider or consumer ISP?), the address's historical behaviour, how many distinct sessions have come from it recently, and whether its geography is consistent with your other signals. This is why datacenter ranges get blocked on sight — the ASN alone identifies them, before anything else is examined.
2. TLS and HTTP fingerprinting
Before a single byte of your request is read, your TLS handshake has already described you: cipher suite order, extensions, and supported curves form a fingerprint (the JA3/JA4 family). HTTP/2 adds frame ordering and settings. Header order and casing matter too — real browsers send them in a characteristic sequence that most HTTP libraries do not reproduce.
This is why a Python script claiming to be Chrome is trivially caught: the user agent says Chrome, the TLS fingerprint says Python. No amount of header editing fixes this; it is below the layer you are editing.
3. Browser fingerprinting
If JavaScript runs, the page can interrogate the environment: canvas and WebGL rendering, installed fonts, screen and viewport dimensions, timezone, language, hardware concurrency, and dozens of navigator properties. It also looks for headless tells — missing plugins, automation flags, mismatched dimensions. Consistency is what matters: a Windows user agent reporting Linux fonts and a UTC timezone is a contradiction no real user produces.
4. Behavioural signals
The hardest layer to fake, and increasingly the decisive one. Systems watch mouse movement paths and acceleration, scroll rhythm, dwell time before clicking, typing cadence, and how you move between pages. Humans are irregular: they overshoot, pause, backtrack, re-read. Automation is regular, and regularity is the signal.
5. Session and sequence patterns
Individually plausible requests can be collectively absurd. Hitting 400 product pages without ever loading a category page, never requesting CSS or images, perfect uniform intervals, no cache behaviour, no referrer chain — the navigation graph gives you away even when each request looks clean.
6. Aggregate and cross-site models
Large anti-bot vendors sit in front of many sites, so they see patterns no single site could. If an IP or fingerprint cluster is scraping aggressively somewhere else, you can arrive at a new target with a reputation already attached.
What This Means in Practice
The uncomfortable conclusion: there is no single trick. Rotating user agents alone does nothing. A residential IP paired with a Python TLS fingerprint and robotic timing still scores badly. The signals are evaluated together, so consistency across all of them is what counts — and inconsistency is exactly what a model is good at spotting.
It also means blocking is rarely about one request. It accumulates. You are usually blocked for a pattern established over minutes or hours, which is why the failure often appears sudden and inexplicable. See why is my IP blocked.
What Genuinely Still Matters
- Clean IP reputation. Because the network layer is weighted so heavily and evaluated first, it is the highest-leverage thing you control. Real residential IPs from consumer ISPs do not carry the ASN penalty datacenter ranges do. This is not a cloak — it removes one large negative signal, it does not make the others disappear.
- A real browser when the target fingerprints deeply. Playwright produces a genuine TLS handshake and a real JS environment, which no HTTP library can imitate.
- Internal consistency. IP country, timezone, locale, language headers and user agent should all tell the same story. Mismatches are cheap to detect and very damaging.
- Human-ish pacing. Randomised delays, variable ordering, occasional pauses. Perfectly uniform timing is a fingerprint of its own.
- Low volume per identity. Spreading requests thinly across many IPs beats hammering a few, regardless of how good those few look.
- Loading the page properly. Fetching only the HTML and never the assets is a sequence-level tell.
Our guide to avoiding detection covers the tactics; this is the model behind why they work.
The Honest Limits
Some targets are genuinely not worth fighting. When a site runs full behavioural analysis plus cross-site reputation, the effort required climbs faster than the value of the data — and every escalation makes your traffic more obviously adversarial, not less.
Three honest positions worth holding:
- Use the official API when one exists. It is stable, sanctioned, and immune to all of the above.
- Treat blocks as information. A rising challenge rate means slow down and narrow scope. Escalating instead is how you turn a soft block into a permanent one.
- Some data is not for you. If access requires defeating protections designed to stop exactly what you are doing, that is a signal about the site's intent, and it has legal weight — see is web scraping legal.
The scrapers that survive long term are not the most aggressive. They are the ones that collect modestly, from clean IPs, at a believable pace, and stop when told.
Frequently Asked Questions
How do websites detect web scrapers?
Modern sites score six layers of signals with machine learning: IP and ASN reputation, TLS and HTTP fingerprinting, browser fingerprinting through JavaScript, behavioural signals such as mouse and scroll patterns, session and navigation sequence, and cross-site reputation from anti-bot vendors. No single signal decides it — the model combines them into a score.
Can AI bot detection be bypassed?
There is no single trick that defeats it, because the signals are evaluated together. What reduces your score is consistency: clean residential IP reputation, a real browser so the TLS and JS environment are genuine, matching country, timezone and locale, human-like pacing, and low volume per identity. Some heavily protected sites remain impractical to scrape at scale, and the right response there is an official API.
Why does changing my user agent no longer work?
Because the user agent is one weak signal among dozens. If your TLS fingerprint identifies a Python HTTP library while your user agent claims Chrome, the contradiction is itself a strong bot signal. Header changes cannot alter the TLS handshake, which is produced below the layer you are editing.
Do residential proxies still work against AI detection?
They remain the highest-leverage single factor, because IP and ASN reputation is weighted heavily and evaluated first — a consumer ISP address does not carry the automatic penalty a datacenter range does. But they are not invisibility. Pair them with a real browser, consistent locale and timezone, and human-like pacing, or the other layers will still flag you.
Conclusion
Bot detection became a scoring problem, and that reframes everything. You are not looking for the one rule to avoid; you are trying not to look statistically unusual across six layers at once. Clean IP reputation is the biggest single lever, a real browser closes the fingerprinting gap, and consistency plus restraint handle the rest. Where a site has made it genuinely clear it does not want automated access, the sustainable answer is an API or a different data source — not a bigger hammer.
Start with the layer that matters most: SpyderProxy residential proxies from $2.75/GB — real consumer ISP IPs, ethically sourced, across 195+ countries.