Property has always run on information asymmetry. What changed is that the information stopped being scarce and started being messy, which is a different problem and in some ways a harder one.
The interesting question is no longer whether data can inform a property decision. It is which data, collected how, and what breaks along the way.
Where the data actually comes from
Public records. Land registries, deed filings, planning applications, tax assessments. Authoritative, cheap and slow - often weeks or months behind reality. Excellent for verified sale prices, poor for anything current.
Listing portals. Zillow, Redfin, Rightmove, Idealista, Leboncoin and their regional equivalents. This is where current asking prices, inventory levels and time-on-market actually live. It is also where collection is hardest, because portals treat their listings as the asset they are.
Rental and transaction platforms. Airbnb, Booking and long-let sites reveal yields and occupancy that sale listings never show. For investment analysis this is frequently more informative than the sale price.
Location signals. Transit access, school catchments, crime statistics, planning pipelines. Slow-moving, publicly available, and the source of most of the difference between two otherwise identical properties.
Demographic and economic data. Census output, income distribution, employment. The layer that separates a temporary price movement from a structural one.
Alternative data. Footfall and mobility patterns, satellite imagery of construction progress, utility connections, even web traffic to local listings. Expensive and inconsistent, but genuinely predictive when it works.
What it gets used for
Valuation. The obvious one. Automated valuation models compare a property against recent verified sales adjusted for size, condition and location. They are only as good as the comparable set, which is why coverage matters more than model sophistication.
Market forecasting. Inventory levels, time-on-market and asking-to-sale ratios tend to move before headline prices do. Watching them is how you see a turn while it is happening rather than afterwards.
Investment screening. Filtering thousands of listings against yield, price-per-square-metre and rental demand to produce a shortlist a human can actually examine. This is where volume genuinely helps.
Site selection. For developers and retailers, combining footfall, demographics, competitor locations and planning data to answer where rather than whether.
Rental optimisation. Short-let operators adjusting nightly rates against local supply, events and seasonality. The most operationally mature use of property data, largely because the feedback loop is days rather than years.
The collection problems nobody warns you about
Portals are geo-targeted. The same listing page returns different prices, different currency and sometimes different inventory depending on where you appear to be. Collecting Spanish property data from a German IP gets you the version shown to foreign buyers, which is not the market. This is the single most common source of quietly wrong data.
Detection is heavy. Major portals run commercial bot detection and treat listings as proprietary. Datacenter IP ranges are filtered almost immediately, which is why residential proxies are the default here rather than a nicety.
Layouts change constantly. Portals redesign, and a scraper written against one region frequently fails on another operated by the same company. Budget for maintenance, not just for building.
Duplicates are everywhere. The same property appears through multiple agents with different photographs, descriptions and sometimes prices. Deduplication is a substantial piece of work and skipping it inflates every inventory count you produce.
Stale data fails silently. A valuation model does not crash when its comparables are six months old. It just becomes wrong, and stays wrong until somebody checks.
Legal ground worth standing on
Collecting publicly visible listing data is broadly permissible in most jurisdictions, but three things complicate it. Portal terms of service usually prohibit automated collection. Copyright can attach to photographs and written descriptions, which are not facts. And agent names, phone numbers and emails are personal data under GDPR, with everything that implies about lawful basis and retention.
The practical line most teams settle on: collect facts about properties freely, treat images and prose as somebody else's copyright, and avoid collecting personal contact details unless you have a reason you would be comfortable explaining.
Getting the collection layer right
For most property work, budget residential at $1.75/GB handles the volume, and premium residential at $2.75/GB adds the city-level targeting that matters when regional pricing differs. Appearing local is not an optimisation here - it is the difference between the real market and the tourist one.
If you are starting from a specific portal rather than a general strategy, our guides to scraping Zillow, Redfin and Idealista cover the per-site specifics.
Frequently asked questions
What counts as big data in real estate?
Anything beyond the listing itself: public records, transaction histories, rental yields, footfall and mobility data, demographic and income statistics, planning applications, satellite imagery and building sensor output. The listing tells you the asking price. Everything else tells you whether it is a good one.
Where does most property data actually come from?
Three places in practice. Public records held by land registries and municipalities, listing portals such as Zillow, Rightmove or Idealista, and third-party aggregators who have already done the collection. The first is authoritative but slow, the second is current but hostile to automation, and the third is convenient but expensive.
Is scraping property listings legal?
Collecting publicly visible listing data is generally permissible in most jurisdictions, but that is not the whole question. Portal terms of service usually prohibit it, copyright can attach to photographs and descriptions, and agent contact details are personal data under GDPR. Public and permitted are different things - get advice before building on the assumption.
Why do property scrapers break so often?
Portals redesign frequently, run aggressive bot detection, and serve different content by location. A scraper written against one region's layout often fails on another, and a valuation model trained on stale or partial data fails silently rather than loudly, which is worse.
What proxies suit property data collection?
Residential, for the same reason the sites are hard: portals geo-target their content and block datacenter ranges. If you need listings as a local buyer sees them, you need to appear local. Budget residential at $1.75/GB covers most collection; premium at $2.75/GB adds city-level targeting where regional pricing matters.
How much data do you actually need?
Less than people assume, and cleaner than they expect. A valuation model with 5,000 verified comparable sales beats one with 200,000 unverified listings. Sample quality and geographic coverage matter far more than raw volume.