How to Monitor Reviews Across Markets

Review data looks easy to collect and is easy to collect badly. The two failure modes are geographic — you only ever see the reviews your IP is shown — and statistical: you sample the loudest pages and conclude something about the whole catalogue. Both are fixable.

Why reviews are geo-filtered

Most large marketplaces show a locale-specific review set by default: reviews from the visitor's country or in the visitor's language, sometimes with a translated subset appended. A US exit and a German exit therefore produce genuinely different review corpora for the same product. If you are tracking sentiment by market, that filtering is the signal, and you need an exit in each market to see it.

Set up per-market collection

Run one collection pass per target market with a Budget Residential exit in that country, and tag every record with the exit country you used. Without that tag, merged data is unanalysable: you cannot tell a genuine market difference from an artefact of which exit happened to serve the request.

Need	Product	Price
Per-country review sets	Budget Residential	$1.75/GB
City-level or sticky sessions	Premium Residential	$2.75/GB
Paginating lightly protected sites	Rotating Datacenter	$1.00/GB

Deduplication is the hard part

The same review can appear under several URLs: product variants, regional domains, and paginated views that shift as new reviews arrive. Deduplicate on a stable composite key rather than on the review text, which is often truncated differently per view.

import hashlib

def review_key(r):
    # stable across variants, pagination and truncated bodies
    raw = "|".join([
        r["product_id"], r["author_id"], r["posted_date"], str(r["rating"]),
    ])
    return hashlib.sha1(raw.encode()).hexdigest()

seen, unique = set(), []
for r in scraped_reviews:
    k = review_key(r)
    if k in seen:
        continue
    seen.add(k)
    unique.append(r)

Avoiding sampling bias

Pagination on review pages is usually sorted by helpfulness or recency, not randomly. Reading the first two pages of every product gives you a corpus skewed toward whatever the platform promotes, which is generally the most extreme reviews. If you cannot collect everything, sample deliberately: fixed depth per product, the same depth across products, and record the sort order you used.

Cadence

Reviews accumulate slowly compared with prices. Daily collection is enough for most catalogues and weekly is enough for the long tail. Randomise the time of day rather than hitting exactly on the hour, and back off on 429 responses instead of rotating past them.

A note on personal data

Reviews are written by identifiable people, and reviewer names, profiles and histories are personal data under GDPR and comparable regimes. Aggregate sentiment is a very different proposition from storing reviewer identities. Collect the minimum you need and be deliberate about retention. This is not legal advice.

Frequently Asked Questions

Why do reviews differ between countries?

Marketplaces filter reviews by locale and language by default, so each market shows a different subset. Collecting from an exit in each market is the only way to see what local customers see.

How do I avoid duplicate reviews?

Deduplicate on a composite key of product, author, date and rating rather than on review text, which is truncated differently across views.

Which proxies suit review monitoring?

Budget Residential at $1.75/GB is the practical default because you need a real consumer IP in each market. Rotating datacenter at $1.00/GB works on lightly protected sites.

How often should I collect?

Daily for active catalogues and weekly for the long tail. Reviews accumulate far more slowly than prices, so higher frequency mostly buys you rate-limit problems.