How to Scrape Airline and Hotel Fares

Travel is the most aggressively personalised pricing on the consumer web. The same flight, queried from two countries within the same minute, routinely returns two different fares in two different currencies with two different availability sets. That is not a bug you are working around — it is the dataset. Collecting it accurately means controlling where the request appears to come from.

Why fares differ by point of sale

Airlines file fares by point of sale, meaning the market in which the ticket is sold. Hotel and OTA platforms layer on currency, tax display rules, and loyalty or device-based personalisation. The practical consequence is that a fare scraped from a single datacenter IP in one country tells you about exactly one market, and tells you nothing reliable about any other.

What you need

Residential exits in each target market. Travel sites treat datacenter ranges with suspicion because scraping them is so common. Budget Residential at $1.75/GB covers most targets; Premium Residential at $2.75/GB adds city targeting.
Sticky sessions. Search flows are multi-step: a search creates state, and results are fetched against it. Rotating mid-flow breaks the session and returns errors or empty results.
A real browser for the hard targets. Many fare pages assemble results client-side from XHR calls after the initial load.

The session trap

This is where most travel scrapers produce quietly wrong data. Per-request rotation makes every call look like a new user, so a multi-step search either fails or silently returns a default result set. Use a sticky session for the whole search and rotate only between searches. Premium Residential holds an IP for up to 8 hours and Budget Residential for 24, which is far longer than any single query needs.

Credentials are built from a separate variable below rather than pasted inline — a literal user:pass@host string in a page body can be rewritten by email-obfuscation filters.

import requests

endpoint = "geo.spyderproxy.com:12321"
creds = "USERNAME:PASSWORD"          # session id goes in the username per your dashboard
proxy = f"http://{creds}@{endpoint}"

with requests.Session() as sess:
    sess.proxies = {"http": proxy, "https": proxy}
    sess.headers["Accept-Language"] = "en-GB,en;q=0.9"
    search = sess.post("https://example-ota.com/search", json={
        "origin": "LHR", "destination": "JFK", "date": "2026-11-04",
    }, timeout=30)
    results = sess.get(
        f"https://example-ota.com/results/{search.json()['id']}", timeout=30
    )
    print(results.json())

Currency and locale hygiene

An IP in the right country is necessary but not sufficient. If your Accept-Language header says en-US while your exit is in Italy, you have created a visitor that does not exist, and some sites will respond to the mismatch rather than to either signal. Align the exit country, the language header and the requested currency, and record all three alongside every fare you store. A fare without its point of sale is not a data point.

Storage and comparison

Store the query timestamp, exit country, language, currency and the raw response. Fares move continuously, so two prices collected an hour apart are not comparable evidence of a market difference. Sample the same routes on the same cadence from each market if you want the comparison to mean anything.

Frequently Asked Questions

Why do I get different prices from different countries?

Airlines file fares by point of sale and hotel platforms localise currency, tax display and promotions. The difference is genuine market behaviour, not an artefact of scraping.

Which proxy type is best for travel fares?

Residential with sticky sessions. Budget Residential at $1.75/GB handles most targets; Premium Residential at $2.75/GB adds city-level targeting where fares vary within a country.

Why does my scraper return empty results?

Almost always per-request rotation breaking a multi-step search. Hold one IP for the whole search flow and rotate only between searches.

Is scraping fare data legal?

Collecting publicly displayed prices is generally lawful, but site terms may restrict automated access and personal data brings privacy law into scope. See our guide on whether web scraping is legal. This is not legal advice.