Collect public web data at any scale without getting blocked.
Rotating residential IPs make every request look like a real visitor, so scrapers running Scrapy, Playwright, Puppeteer or plain requests keep a high success rate on protected targets. Pair per-request rotation for breadth with sticky sessions when a flow needs continuity.
Why this needs proxies
Sites vary what they return based on where the request appears to come from, and they rate-limit or block addresses that ask too often. Both problems have the same fix: exit from real consumer IPs in the right places, and spread the load so no single address looks unusual. That is the whole job a proxy does here.
The reason residential IPs succeed where datacenter ones fail is classification by ASN. Hosting networks can be challenged wholesale with a single rule; consumer ISP networks cannot, because doing so would block real customers.
Recommended products
| Product | Price |
|---|---|
| Budget Residential | $1.75/GB |
| Premium Residential | $2.75/GB |
| Rotating Datacenter | $1.00/GB |
Start with the cheapest option your target tolerates and move up only when it blocks you. Concurrent sessions are unlimited on every plan and country targeting is included in the price. Full tiers are on our pricing page.
How this works in practice
- Pick the cheapest product that works. Test the target on rotating datacenter first; move to residential only when you are actually refused.
- Choose the exit country deliberately. The data you get back is the data that country sees, so record which exit produced every record.
- Decide rotation vs sticky. Per-request rotation suits stateless collection; anything with a login or a multi-step flow needs one IP held for the whole session.
- Pace against the target, not your budget. Back off on 429 responses instead of rotating past them, and randomise intervals rather than hitting on the hour.
Common mistakes
- Rotating inside a session. The single most common cause of empty results and flagged accounts.
- Mismatched signals. An exit in one country with an Accept-Language header for another creates a visitor that does not exist, and some sites respond to the contradiction.
- Paying for residential too early. Plenty of targets never check, and the gap between $1.00/GB and $2.75/GB is substantial at volume.
- Not recording the exit. Data without its point of observation cannot be compared across markets.
Sizing your bandwidth
Work from page weight rather than guesswork. A page averaging 250 KB across 100,000 requests is roughly 25 GB. JSON APIs are far lighter, often a few kilobytes per call, while anything rendered in a headless browser pulls images and scripts too and can be several times heavier. Start at 1 GB, measure your actual average, then buy the package that matches.
Getting set up
Every product speaks HTTP(S) and SOCKS5, so this is a one-line change in most stacks. Credentials are assembled from a variable below rather than pasted inline, because a literal user:pass@host string in page copy can be rewritten by email-obfuscation filters.
import requests
endpoint = "geo.spyderproxy.com:12321"
creds = "USERNAME:PASSWORD"
proxy = f"http://{creds}@{endpoint}"
r = requests.get(
"https://example.com",
proxies={"http": proxy, "https": proxy},
timeout=30,
)
print(r.status_code)Confirm your exit is landing where you expect with the IP lookup tool before starting a long run. Bandwidth products begin at 1 GB, so you can validate for under $2.
Frequently Asked Questions
Which proxy type is best for web scraping?
Rotating residential is the default for protected targets: every request exits from a different real home IP, so rate limits and IP bans are much harder to trigger. If your target has weak or no bot protection, rotating datacenter does the same job at $1.00/GB instead of $2.75/GB. Use mobile only for the hardest targets, where the carrier-grade NAT behind a mobile IP makes blocking expensive for the site.
How many proxies do I need for scraping?
With rotating residential you do not buy individual proxies, you buy bandwidth and the pool rotates for you. Size on data volume instead: a page averaging 250 KB across 100,000 pages is roughly 25 GB. Start with 1 GB to validate your parser, then scale.
Will rotating proxies break my session-based scraper?
Per-request rotation will, because each request looks like a different user. For anything that needs a login or a multi-step flow, use sticky sessions instead: Premium Residential holds the same IP for up to 8 hours, Budget Residential for 24. Rotate between sessions, not within them.
Do I need to change my code to use proxies?
No. Every product speaks HTTP(S) and SOCKS5, so it is one line in most stacks: proxies={'http': 'http://user:pass@gateway:port'} in Python requests, or a --proxy-server flag in Playwright and Puppeteer. Concurrent sessions are unlimited and free on every plan.
Is web scraping legal?
Collecting publicly accessible data is generally lawful in the US and EU, but the details matter: personal data brings GDPR into scope, and a site's terms of service may bind you contractually even where copyright does not apply. We cover the current position in our guide on whether web scraping is legal. This is not legal advice.