Most scrapers do not fail because the parsing was wrong. They fail because they were impolite, brittle, or legally careless — and got blocked, broke silently, or created a problem nobody wanted. These are the practices that keep a scraper running for years instead of days, roughly in the order you should apply them.
1. Look for an API Before You Scrape Anything
Always check whether the data is available through an official API, a partner feed, or a public dataset first. An API is stable, sanctioned, structured, and cannot be broken by a redesign. Even when it does not cover everything you need, it often covers most of it — leaving far less to scrape.
Also check the page's own network traffic: many sites render from an internal JSON endpoint you can call directly, which is cleaner and lighter than parsing HTML. See API vs web scraping.
2. Read robots.txt and the Terms
robots.txt tells you which paths a site asks automated clients to avoid. It is not a legal instrument, but ignoring it is the clearest possible signal of bad faith and is the first thing cited when a site takes issue with your traffic. Check the Terms of Service too — many restrict automated access, and if you accepted them by creating an account they can bind you contractually. Our guide to whether web scraping is legal covers the detail.
3. Throttle. Then Throttle More
The single most important technical practice: send fewer requests, more slowly, than you think you need. Concretely:
- Add a delay between requests and randomise it — perfectly regular timing is itself a bot signal.
- Cap concurrency deliberately rather than firing everything at once.
- Prefer off-peak hours for large jobs.
- Honour 429 responses and any
Retry-Afterheader instead of retrying immediately. - Back off on repeated errors rather than escalating.
A scraper that finishes in six hours without ever straining the target beats one that finishes in twenty minutes and gets the IP range banned.
4. Never Fetch the Same Page Twice
Cache every response and make crawls incremental. Persist which URLs are done, use conditional requests (ETag / If-Modified-Since) where supported, and only re-fetch pages that plausibly changed. This cuts your load on the target, makes re-runs cheap, and means an interrupted job resumes instead of restarting — the pattern described in advanced web scraping in Python.
5. Be Honest About What You Are
Send a real, current user agent with internally consistent headers — a Chrome user agent alongside library-default Accept headers is an obvious mismatch. For research or non-adversarial collection, consider identifying your bot and providing a contact URL; sites are far more likely to raise an issue with you than to silently ban you. Never impersonate Googlebot: it is trivially verifiable by reverse DNS and it destroys any good-faith argument you might have had.
6. Parse Defensively
Sites redesign. Assume yours will:
- Prefer embedded structured data (JSON-LD) over the DOM when it exists.
- Anchor on stable attributes —
data-*,id, ARIA roles — not generated class names likecss-1x7f2k. - Validate every field. A price that fails to parse should raise, not write a silent null.
- Alert when the validation-failure rate jumps. That is how you learn about a redesign the same day rather than a month later.
7. Distribute Your Requests Across IPs
Rate limits are enforced per IP, so a single address is a single point of failure no matter how polite your pacing. Rotating residential proxies spread requests across many real ISP addresses, which is what lets a large crawl complete at a reasonable pace. Two caveats worth stating plainly: residential IPs are far less likely to be challenged than datacenter ranges, and rotation is not a licence to hammer — it removes the bottleneck, it does not remove the obligation.
Use ethically sourced proxies. If you cannot explain where your IPs come from, that is a compliance problem waiting to surface.
8. Treat Personal Data as a Liability
The moment your dataset contains names, emails, profiles or anything identifying a person, you are inside the GDPR and CCPA. Practical rules: collect only public data, never log in to scrape, take the minimum fields you actually need, avoid contact details unless you have a lawful basis, set a retention limit, and secure what you store. Aggregate market data rarely needs anything personal — so do not collect it by default just because it is there.
9. Take Less Than You Can
Scrape the pages you need, not the whole site. Narrow by category, region or price band. Skip images and assets unless they are the point. Stop when you have enough. A smaller footprint is cheaper for you, lighter on the target, and much less likely to attract attention in the first place.
10. Assume Silent Failure
The worst scraper outcome is not crashing — it is quietly returning wrong data for weeks. Monitor row counts per run, status-code distribution, validation-failure rate, and time since last successful run. A crawl that suddenly returns 40% fewer records has not got more efficient; something broke. Store results in a format that keeps structure (see JSON vs CSV) and keep raw responses for a while so you can re-parse without re-fetching.
The Short Checklist
- Is there an API or a hidden JSON endpoint? Use it.
- Read
robots.txtand the Terms. - Throttle, randomise, cap concurrency, honour 429.
- Cache and crawl incrementally.
- Real user agent, consistent headers, no impersonating Googlebot.
- Structured data first, stable attributes second, validate everything.
- Rotate ethically sourced residential IPs.
- Minimise personal data; never scrape behind a login.
- Take the narrowest slice that answers the question.
- Monitor for silent failure.
Frequently Asked Questions
What are the best practices for web scraping?
Check for an official API first, respect robots.txt and the site's terms, throttle requests and honour 429 responses, cache so you never fetch the same page twice, send honest and consistent headers, parse against stable attributes rather than generated class names, rotate residential IPs, minimise personal data, and monitor for silent failure.
How fast can I scrape without getting blocked?
There is no universal rate. Start slow with randomised delays and low concurrency, then watch your error rate: rising 429s or 403s mean back off. Rotating residential proxies raise your total safe throughput because requests are spread across many IPs, but each individual target should still be crawled politely.
Do I have to follow robots.txt?
It is not generally a binding legal document, but it states what the site asks automated clients to do, and ignoring it is the clearest evidence of bad faith if there is ever a dispute. Following it is standard practice and costs you very little.
Is it OK to scrape personal data?
Only with care. Once your dataset identifies people, the GDPR and CCPA apply: you need a lawful basis, data minimisation, a retention limit and appropriate security. Most commercial use cases such as pricing or market analysis do not need personal data at all, so the safest practice is not to collect it.
Conclusion
Good scraping is mostly restraint. Prefer the sanctioned route, go slower than you need to, take less than you could, be honest about who you are, build parsing that expects change, and watch for the failure that does not announce itself. The technical layer that makes all of it sustainable is distributing your requests across clean, ethically sourced IPs — so that being polite does not also mean being slow.
Scrape responsibly and at scale: SpyderProxy residential proxies from $2.75/GB — ethically sourced, rotating, 195+ countries.