WHOIS data — who registered a domain, when, through which registrar, and on which nameservers — powers brand protection, threat intelligence, lead generation, and market research. Collecting it for one domain is trivial; collecting it for thousands runs straight into rate limits and privacy redaction. This guide covers how WHOIS works in 2026, the WHOIS-versus-RDAP shift, and how to scrape domain data reliably at scale with Python and proxies.
What Is WHOIS Data?
WHOIS is the public record of a domain registration. A typical lookup returns the registrar, creation and expiry dates, last-updated date, domain status codes, and nameservers — and, when not redacted, registrant contact details. It is maintained by registries and registrars and queried over the WHOIS protocol (port 43) or, increasingly, its modern replacement RDAP.
WHOIS vs RDAP
The industry is moving from the old plain-text WHOIS protocol to RDAP (Registration Data Access Protocol), which returns structured JSON instead of free-form text. RDAP is easier to parse, supports standardized queries, and is now the preferred method for most generic TLDs. For scraping, prefer RDAP where available — you skip the brittle text-parsing that plagues classic WHOIS — and fall back to port-43 WHOIS for TLDs that have not fully migrated.
Why Scrape WHOIS Data?
- Brand protection — detect newly registered lookalike or typosquat domains targeting your brand.
- Threat intelligence — cluster malicious domains by registrar, nameserver, or registration date.
- Lead generation — find domains registered in a niche or timeframe (where contact data is public).
- Market and portfolio research — track registration trends and monitor expiring domains.
The Challenges of WHOIS at Scale
- Rate limits. Registries and RDAP servers aggressively throttle by IP. Query a few hundred domains from one IP and you will be temporarily blocked.
- Privacy redaction. Since GDPR, most registrant contact fields are redacted or replaced with privacy-service placeholders. Registrar, dates, status, and nameservers remain available.
- Inconsistent formats. Classic WHOIS text varies by registrar, so parsers must handle many layouts (another reason to prefer RDAP's JSON).
- Referral chains. A thin registry response often points to the registrar's WHOIS server, requiring a second lookup.
Why Proxies Are Essential
The single biggest blocker to bulk WHOIS collection is per-IP rate limiting. Route your lookups through a pool of residential proxies and each query can exit from a different IP, so you spread load across many addresses and avoid the throttles and temporary bans that stop a single-IP scraper cold. For high-volume, lightly defended RDAP endpoints, cheaper rotating datacenter proxies also work well.
Scraping WHOIS Data With Python
The cleanest modern approach is to query RDAP over HTTPS and parse JSON, routing requests through a proxy so you can scale. A minimal example:
import requests
proxy = "http://USERNAME:[email protected]:12321"
proxies = {"http": proxy, "https": proxy}
def rdap_lookup(domain):
# rdap.org redirects to the authoritative RDAP server for the TLD
r = requests.get(f"https://rdap.org/domain/{domain}",
proxies=proxies, timeout=20)
if r.status_code != 200:
return {"domain": domain, "error": r.status_code}
d = r.json()
events = {e["eventAction"]: e["eventDate"] for e in d.get("events", [])}
return {
"domain": domain,
"registrar": next((e["vcardArray"][1][1][3]
for e in d.get("entities", [])
if "registrar" in e.get("roles", [])), None),
"created": events.get("registration"),
"expires": events.get("expiration"),
"status": d.get("status", []),
"nameservers": [ns.get("ldhName") for ns in d.get("nameservers", [])],
}
for domain in ["example.com", "openai.com"]:
print(rdap_lookup(domain))Add retries, a small delay, and proxy rotation to run this across thousands of domains. For a refresher on building resilient collectors, see how to build a web scraper in Python, and for structuring the output, JSON vs CSV.
Legal and Ethical Notes
WHOIS and RDAP data is published for legitimate operational purposes, but rules apply. Respect each server's rate limits and terms, do not attempt to defeat privacy redaction, and comply with data-protection laws when any personal contact data is involved — collect only what you need for a lawful purpose. Bulk contact harvesting for spam is both against registry policy and, in many places, illegal.
Frequently Asked Questions
Is scraping WHOIS data legal?
Querying public WHOIS and RDAP records is generally permissible, but you must respect server rate limits and terms of service and comply with privacy laws for any personal data. Bulk-harvesting registrant contact details for spam violates registry policy and often the law.
What is the difference between WHOIS and RDAP?
WHOIS is the older plain-text protocol (port 43) with inconsistent formats. RDAP is its modern replacement, returning structured JSON with standardized fields. For scraping, RDAP is easier to parse and is now preferred for most generic TLDs, with WHOIS as a fallback.
Why do I need proxies to scrape WHOIS at scale?
Registries and RDAP servers rate-limit aggressively by IP, so a single-IP scraper gets blocked after a few hundred queries. Residential proxies let each lookup exit from a different IP, spreading load across many addresses to avoid throttles and bans.
Why is the registrant's contact info missing?
Since GDPR, most registrars redact registrant contact fields or replace them with a privacy service. Registrar name, creation and expiry dates, status codes, and nameservers usually remain publicly available.
Conclusion
Scraping WHOIS data at scale is less about clever parsing and more about not getting rate-limited. Prefer RDAP's structured JSON, handle referral chains and redaction gracefully, stay within each server's rules, and route your lookups through residential proxies so per-IP throttling never stops the job.
Collect domain data reliably with SpyderProxy residential proxies from $2.75/GB — ethically sourced, 195+ countries, built to beat rate limits.