Scraping E-Commerce Prices at Scale Without Getting Blocked

Price monitoring projects rarely fail because the crawler broke. They fail because six months in, somebody asks whether a competitor really raised prices in March, and nobody can answer — because the March numbers were collected from a different country, at a different hour, with a promotion running that nobody recorded.

Getting the data is the easy half. Getting data that is still meaningful a quarter later is the part worth designing for.

A price is a measurement, not a fact

The same product page will show different numbers depending on where the request came from, what device it claims to be, whether the session looks like a returning visitor, what promotions are live, and sometimes simply what time it is.

That means a scraped price is a measurement taken under conditions. If you do not record the conditions, you cannot compare two measurements. Most pipelines store (sku, price, timestamp) and discover eighteen months later that the series is uninterpretable.

The minimum record is closer to this:

{
  "sku": "B08N5WRWNW",
  "retailer": "example-marketplace",
  "storefront": "DE",
  "observed_at": "2026-04-09T06:00:00Z",
  "exit_country": "DE",
  "exit_city": "Munich",
  "device_class": "desktop",
  "session_state": "logged_out",
  "currency": "EUR",
  "price": 249.99,
  "price_includes_tax": true,
  "shipping_estimate": 0.0,
  "promotion_label": "Spring deal",
  "seller": "Example Retail GmbH",
  "in_stock": true,
  "raw_html_key": "s3://price-raw/2026/04/09/de/B08N5WRWNW.html.gz"
}

Every field there has cost somebody an analysis at some point.

Model the storefront, not the product

The unit of collection is not a SKU. It is a (retailer, country) pair — a storefront. The same marketplace in Germany and Austria is two different datasets with different sellers, different shipping promises and sometimes a different assortment.

This has a direct consequence for proxy configuration: the German storefront must be collected from German exits. Collecting it from a US datacenter will either redirect you to the US storefront or serve a defaulted version, and in both cases the parser will happily record the wrong numbers.

Pin the exit country per storefront using residential proxies with country targeting, and for categories where local pricing varies within a country, pin the city too.

Fix the observation point

Before writing any code, write down the observation point for each storefront and treat it as a constant:

• Country and city the request originates from
• Device class — desktop or mobile, because they frequently differ
• Session state — logged out is the only reproducible option; a logged-in session accumulates personalisation
• Currency and locale headers

Then hold all of it steady. If you change the observation point, treat it as a new series rather than a continuation of the old one. Silently switching from Frankfurt to Berlin exits in the middle of a year is the kind of thing that produces a "price change" that never happened.

Cadence beats frequency

Teams often ask how often they should collect. The more useful question is how regularly.

An hourly series with gaps is worse for trend analysis than a clean daily one. Pick an interval your infrastructure can hold indefinitely, and hold it:

• Hourly for volatile categories: electronics during promotional periods, travel, anything with an active repricing algorithm on the other side.
• Every six hours for general retail.
• Daily for stable categories: furniture, industrial supplies, books.

Collect at the same wall-clock times each day, in a fixed timezone, and store timestamps in UTC. Retailers run batch price updates on schedules; sampling at a drifting time will alias against those batches and produce artefacts that look like volatility.

Tier the exits to control cost

At scale, price monitoring moves a lot of bytes. A product page is commonly 150–400 KB, and 50,000 SKUs sampled six times a day is somewhere between 45 and 120 GB per day.

Routing all of that through residential is the default mistake. The pattern that works:

1. Discovery on datacenter. Category pages, sitemaps and search result pages are frequently less protected than product detail pages. Enumerate cheaply through datacenter proxies.
2. Detail pages on residential, pinned to the storefront country.
3. Escalate only on classified failure. A challenge page returns HTTP 200; check the body. There is a working classifier in our Python rotating proxy guide.
4. Switch to unlimited above roughly 120 GB per day per gateway. Past that point, unlimited residential gateways with flat daily billing cost less than metered traffic, and the bill stops tracking the workload.

The trade-offs between the first two tiers are covered in more detail in our residential versus datacenter comparison.

Keep the raw payload

Store the compressed HTML or JSON response alongside the parsed record. It roughly triples storage cost and it is worth it every time.

Retailers change page structure without notice. When a selector breaks — and it will — you have two options. With raw payloads you re-parse the archive and recover the history in an afternoon. Without them, the only option is re-collecting data that no longer exists, because yesterday's price is not available at any price today.

Compressed HTML is around 30–50 KB per page. At 300,000 pages a day that is roughly 12 GB, which costs a few dollars a month in object storage. Compare that to losing a quarter of price history.

Parse defensively

A few habits that prevent silent corruption:

Never trust a bare number. Parse the currency from the page, not from a configuration file. Marketplaces do serve unexpected currencies.

Validate against the previous observation. A price that moved more than 60% overnight is more likely a parse error than a real change. Flag it for review instead of writing it through.

Distinguish absent from zero. An out-of-stock product has no price; it does not have a price of zero. Conflating them will destroy any average you compute later.

Record the seller. On marketplaces the buy-box winner rotates, and a price change frequently means a different seller won rather than the same seller repricing.

Check for truncation. A listing page that normally returns 48 results and returns 12 is a partial response, not a shrinking catalogue.

Respect the target

Beyond ethics, restraint is the cheapest form of block avoidance:

• Read and honour robots.txt for the paths you collect.
• Limit concurrency per hostname. Start at four requests per second and watch the target's own latency; if it climbs as you push harder, you are past the comfortable point.
• Collect only publicly accessible pages. Anything behind a login belongs to someone else's account.
• Back off immediately on 429 and 503 rather than retrying into the wall.
• Identify your crawler honestly if the site offers a contact channel. Some retailers will simply give you a feed, which is faster and cheaper than scraping.

Our price monitoring and e-commerce intelligence pages cover the product configuration for these workloads.

Alert on deltas, not levels

The last piece is what you do with the series. Alerting on absolute price levels generates noise; a competitor that is simply cheaper than you is not news.

Alert on change: a move beyond a threshold within a window, a buy-box seller change, a stock transition, a promotion appearing or disappearing. Those are the events that trigger a decision. Everything else belongs in a dashboard nobody needs to watch.

And whatever the alert is, include the observation point in it. "Price dropped 14%" is an incomplete sentence. "Price dropped 14% on the German storefront, desktop, logged out, at 06:00 UTC, with a new promotion label" is something a pricing manager can act on.

How often to recrawl

Not every SKU deserves the same cadence. Fast-moving categories (consumer electronics, fashion drops, grocery promotions) need several observations a day; slow ones (white goods, spare parts) are fine at a daily or twice-weekly pass. A single global interval either wastes traffic on pages that never change or misses the hour in which a competitor actually moved.

A practical split:

• Hot set. The 5–10% of SKUs that generated an alert in the last seven days, plus anything you sell against directly. Recrawl every one to four hours.
• Warm set. The rest of the catalogue on the storefronts you care about. Recrawl daily, staggered so you are not hitting every host at midnight UTC.
• Cold set. Long-tail listings kept for completeness. Recrawl two or three times a week.

Promote a SKU from warm to hot when it alerts; demote it after a quiet week. That one rule usually cuts traffic 30–40% without losing the events a pricing team actually acts on.

When you recrawl, fetch the same observation point you used last time: same country, same device class, same login state. Mixing a mobile-logged-in German price into a desktop-logged-out series will look like a 12% swing that never happened.

Cookies, locale and the first request

The first request to a marketplace often sets a locale cookie from the exit IP. If you then follow a URL that was generated for a different country, the cookie and the path disagree and you get a bounce, a redirect loop, or a page in the wrong language. Two ways out:

Clear cookies at the start of every observation, and let the exit country pick the locale. This is the right default for public prices.

Or pin both: keep the cookie jar and the sticky session together for the length of one storefront visit, then drop both. That is closer to a real shopper and is required once you need a logged-in view (member pricing, lists, cart). Do not reuse that jar on a later visit from a different IP.

Either way, record the locale you actually received, not the one you intended. The observation is only comparable if the next run lands in the same place.