Feed models with clean, diverse, permissioned web data.
Large-scale training and RAG pipelines need broad, unblocked access to the public web. High-trust residential and mobile IPs keep crawlers running while respecting rate limits and robots directives.
Why this needs proxies
Sites vary what they return based on where the request appears to come from, and they rate-limit or block addresses that ask too often. Both problems have the same fix: exit from real consumer IPs in the right places, and spread the load so no single address looks unusual. That is the whole job a proxy does here.
The reason residential IPs succeed where datacenter ones fail is classification by ASN. Hosting networks can be challenged wholesale with a single rule; consumer ISP networks cannot, because doing so would block real customers.
Recommended products
| Product | Price |
|---|---|
| Rotating Datacenter | $1.00/GB |
| Budget Residential | $1.75/GB |
| Premium Residential | $2.75/GB |
Start with the cheapest option your target tolerates and move up only when it blocks you. Concurrent sessions are unlimited on every plan and country targeting is included in the price. Full tiers are on our pricing page.
How this works in practice
- Pick the cheapest product that works. Test the target on rotating datacenter first; move to residential only when you are actually refused.
- Choose the exit country deliberately. The data you get back is the data that country sees, so record which exit produced every record.
- Decide rotation vs sticky. Per-request rotation suits stateless collection; anything with a login or a multi-step flow needs one IP held for the whole session.
- Pace against the target, not your budget. Back off on 429 responses instead of rotating past them, and randomise intervals rather than hitting on the hour.
Common mistakes
- Rotating inside a session. The single most common cause of empty results and flagged accounts.
- Mismatched signals. An exit in one country with an Accept-Language header for another creates a visitor that does not exist, and some sites respond to the contradiction.
- Paying for residential too early. Plenty of targets never check, and the gap between $1.00/GB and $2.75/GB is substantial at volume.
- Not recording the exit. Data without its point of observation cannot be compared across markets.
Sizing your bandwidth
Work from page weight rather than guesswork. A page averaging 250 KB across 100,000 requests is roughly 25 GB. JSON APIs are far lighter, often a few kilobytes per call, while anything rendered in a headless browser pulls images and scripts too and can be several times heavier. Start at 1 GB, measure your actual average, then buy the package that matches.
Getting set up
Every product speaks HTTP(S) and SOCKS5, so this is a one-line change in most stacks. Credentials are assembled from a variable below rather than pasted inline, because a literal user:pass@host string in page copy can be rewritten by email-obfuscation filters.
import requests
endpoint = "geo.spyderproxy.com:12321"
creds = "USERNAME:PASSWORD"
proxy = f"http://{creds}@{endpoint}"
r = requests.get(
"https://example.com",
proxies={"http": proxy, "https": proxy},
timeout=30,
)
print(r.status_code)Confirm your exit is landing where you expect with the IP lookup tool before starting a long run. Bandwidth products begin at 1 GB, so you can validate for under $2.
Frequently Asked Questions
Why do AI training pipelines need proxies?
Corpus building means crawling millions of pages across thousands of domains, and any single IP gets rate-limited within the first few thousand requests. Rotating residential IPs spread the load so a crawl finishes in days rather than months.
Which proxy type suits large-scale crawling?
Rotating Datacenter at $0.82/GB on the 200 GB package for open, unprotected sources, which is most of a general web corpus. Switch to residential only for the protected domains that reject datacenter ranges.
Does this respect robots.txt and rate limits?
That is your crawler's job, not the proxy's, and we expect you to do it. Our network gives you the IP diversity to crawl at scale; honouring robots directives, crawl-delay and copyright remains your responsibility.
Can I use this for RAG pipelines?
Yes, and it is a common use. RAG retrieval needs fresh, unblocked access to live sources rather than a one-time dump. Rotating residential keeps scheduled re-crawls working as targets tighten their defences.
Are your IPs ethically sourced?
Yes. Every residential IP comes from a consenting peer who opted in and can leave at any time, and we publish how to verify that claim. Provenance matters more here than in most use cases, because training data inherits the ethics of its collection.