How to Read HTTP Response Headers

Response headers are the most under-read diagnostic in web scraping. When a request fails, the body usually tells you nothing useful and the headers tell you almost everything: whether you were rate-limited, how long to wait, whether you were served a cached response, and whether the far end can tell you are using a proxy.

Headers worth reading every time

Header	What it tells you
`Retry-After`	Exactly how long to wait after a 429 or 503
`Cache-Control`, `Age`	Whether you got a cached copy and how old it is
`Location`	Redirect destination, and whether it changes host
`Set-Cookie`	Session state you probably need to keep
`Content-Encoding`	Compression in use, which affects your byte accounting
`Server`, `CF-Ray`	What is in front of the origin

Rate limit headers

Many APIs publish their limits as you consume them, commonly as X-RateLimit-Limit, X-RateLimit-Remaining and X-RateLimit-Reset. Reading them lets you pace against the actual budget instead of guessing and getting blocked. When you do get a 429, Retry-After is an instruction, not a suggestion — see handling 429 responses.

Reading them in code

Credentials are built from a separate variable below rather than pasted inline — a literal user:pass@host string in a page body can be rewritten by email-obfuscation filters.

import requests

endpoint = "geo.spyderproxy.com:12321"
creds = "USERNAME:PASSWORD"
proxy = f"http://{creds}@{endpoint}"

r = requests.get("https://example.com",
                 proxies={"http": proxy, "https": proxy}, timeout=30)

for name in ("Retry-After", "Cache-Control", "Age", "Server",
             "X-RateLimit-Remaining", "Content-Encoding"):
    if name in r.headers:
        print(f"{name}: {r.headers[name]}")

for hop in r.history:
    print("redirect:", hop.status_code, hop.headers.get("Location"))

Headers that expose a proxy

These are request headers, added by the proxy and seen by the destination: X-Forwarded-For, Via, X-Real-IP and Forwarded. If any of them carries your real address, the proxy is transparent and is disclosing exactly what you were trying to keep private. Our HTTP header checker shows precisely what arrives at the far end, which is the fastest way to catch this.

Security headers you will encounter

Strict-Transport-Security forces HTTPS for future requests. Content-Security-Policy restricts what a page may load, which matters when you are rendering in a headless browser and resources are silently blocked. Set-Cookie flags such as SameSite and Secure change whether your session survives a redirect.

A debugging habit

When something breaks, log the full response headers before anything else. A surprising share of "the site blocked me" reports turn out to be a redirect to a login page, a cached 403 with an Age of several hours, or a 429 with a Retry-After nobody read.

Frequently Asked Questions

Which response header tells me I am rate limited?

A 429 status with Retry-After, which gives the wait in seconds or as a date. Many APIs also publish X-RateLimit-Remaining and X-RateLimit-Reset as you consume the budget.

How can I tell if a proxy is leaking my IP?

Check the request headers arriving at the destination for X-Forwarded-For, Via or X-Real-IP containing your real address. Our HTTP header checker shows exactly what the far end receives.

What does the Age header mean?

How long, in seconds, a cached response has been held by an intermediary. A high Age on an error response often means you are being served a stale failure rather than being blocked now.

Why does my scraper see different headers than my browser?

Servers vary responses by User-Agent, Accept-Encoding and Accept-Language. If your client sends a different set than a browser, you often get a different response entirely.