Quick verdict: Three ways to handle cookies in Python requests. For one-off requests, pass a dict to cookies=. For sessions where the server sets cookies you need to send back, use requests.Session() — it manages cookies automatically. For fine-grained control (specific cookies, domain restrictions, expiration), use http.cookiejar.RequestsCookieJar.
This guide covers all three patterns with working code, the login-then-scrape flow that's the most common production use case, cookie inspection and debugging, and how cookies work with proxies.
import requests
cookies = {"session_id": "abc123", "csrf_token": "xyz789"}
r = requests.get("https://example.com/profile", cookies=cookies)
Simplest pattern. Use when you have specific cookies to send and don't need to track server-set cookies.
session = requests.Session()
# Login — server sets session cookie automatically
session.post("https://example.com/login", data={
"username": "alice",
"password": "secret",
})
# Subsequent calls include the cookie automatically
r = session.get("https://example.com/profile")
print(r.text)
This is the standard pattern for scraping behind a login. Full login-scrape walkthrough here.
from requests.cookies import RequestsCookieJar
jar = RequestsCookieJar()
jar.set("session_id", "abc123", domain="example.com", path="/")
jar.set("preference", "dark", domain="example.com", path="/", secure=True)
r = requests.get("https://example.com/profile", cookies=jar)
Use when you need cookies scoped to specific domains/paths or with expiration.
import requests
from bs4 import BeautifulSoup
session = requests.Session()
session.headers.update({
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/130.0.0.0",
})
# Step 1: GET login page (some sites set CSRF cookie here)
r = session.get("https://example.com/login")
soup = BeautifulSoup(r.text, "lxml")
csrf = soup.select_one('input[name="csrf_token"]')["value"]
# Step 2: POST credentials with CSRF
r = session.post("https://example.com/login", data={
"username": "alice",
"password": "secret",
"csrf_token": csrf,
})
assert r.status_code == 200
# Step 3: Now scrape — cookies are automatic
r = session.get("https://example.com/dashboard")
soup = BeautifulSoup(r.text, "lxml")
data = soup.select(".dashboard-item")
# All cookies in the session
for cookie in session.cookies:
print(f"{cookie.name}={cookie.value} (domain={cookie.domain}, expires={cookie.expires})")
# Specific cookie
session_id = session.cookies.get("session_id", domain="example.com")
# Convert to dict (loses domain/expiration info)
cookie_dict = session.cookies.get_dict()
# Cookies set by THIS specific response
print(r.cookies)
import pickle
# Save cookies after login
with open("cookies.pkl", "wb") as f:
pickle.dump(session.cookies, f)
# Later: load cookies, no re-login needed
session = requests.Session()
with open("cookies.pkl", "rb") as f:
session.cookies.update(pickle.load(f))
# Use the loaded session
r = session.get("https://example.com/dashboard")
Useful for long-running scrapers that should avoid re-authentication on every restart. Cookies still expire server-side; rotate the persisted file when the session goes stale.
session = requests.Session()
session.proxies = {
"http": "http://USER:[email protected]:8080",
"https": "http://USER:[email protected]:8080",
}
# Cookies and proxies work independently
session.post("https://example.com/login", data={...})
r = session.get("https://example.com/dashboard") # Through proxy, with login cookies
For scraping at scale through a rotating residential proxy, keep the session alive across requests so cookies persist while the IP rotates between requests. Full rotating proxies pattern here.
pip install browser-cookie3
import browser_cookie3
import requests
# Pulls cookies directly from Chrome's local storage
cj = browser_cookie3.chrome(domain_name="example.com")
r = requests.get("https://example.com/profile", cookies=cj)
Useful when you've already authenticated in your browser and want to skip the login flow in your script. Works with Chrome, Firefox, Edge, Brave.