How to Set Up Rotating Proxies in Python: A Complete Guide

Most Python scrapers that "stop working" were never really working. They were getting 200 responses that happened to contain a challenge page, and nobody noticed until the parsed dataset came back empty three weeks later. Rotating proxies fix a real problem, but only if the surrounding code knows the difference between a page and a rejection.

This guide covers the mechanics — how to wire a rotating proxy into requests, httpx and Scrapy — and then the part most tutorials skip: deciding when to rotate, how to classify a response, and how to keep the traffic bill proportional to the data you actually collect.

What rotation actually buys you

A target site that wants to limit automated access has a handful of cheap signals available. IP address is the first one it reaches for, because it is the only identifier present before your client sends a single byte of application data. Rate limits, reputation lookups and geographic rules all hang off it.

Rotating the exit address defeats the naive version of that: counting requests per IP. It does nothing at all about TLS fingerprints, header ordering, cookie state or behavioural timing. If your crawler is being blocked because it announces itself as python-requests/2.31.0 and fetches 40 pages a second with no referrer, adding residential proxies will make the problem more expensive without making it go away.

Rotation is one layer. Treat it as such.

The credential format

Every provider exposes rotation slightly differently, but the common pattern is to encode the session and targeting parameters into the proxy username. A UUIProxy residential endpoint looks like this:

http://USERNAME-country-us-session-abc123:[email protected]:7000

Three things are happening in that string:

• country-us pins the exit to the United States. Drop it and you get the whole pool.
• session-abc123 keeps the same exit IP for every request that carries this identifier, for up to 120 minutes. Drop it and you get a new IP on every single request.
• The host and port are the gateway; they never change.

That is the whole API surface for rotation. Everything below is about deciding what to put in that string and when.

Per-request rotation with requests

The simplest useful setup omits the session identifier entirely, so the gateway hands out a new exit for each call:

import os
import requests
 
PROXY = (
    f"http://{os.environ['PROXY_USER']}-country-us:"
    f"{os.environ['PROXY_PASS']}@gate.uuipproxy.com:7000"
)
 
proxies = {"http": PROXY, "https": PROXY}
 
response = requests.get(
    "https://example.com/catalogue",
    proxies=proxies,
    timeout=30,
    headers={"Accept-Language": "en-US,en;q=0.9"},
)
print(response.status_code, len(response.content))

Two details matter more than they look.

Always set a timeout. Without one, requests waits indefinitely, and a single stalled residential exit will park a worker thread forever. Thirty seconds is generous for a proxy; ten is usually enough.

Set Accept-Language to match the exit country. An IP in Frankfurt asking for en-US content is a mismatch that costs nothing to fix and is trivially detectable.

Sticky sessions with requests.Session

When a workflow spans several pages — a search, then a listing, then a detail page — you want all of it to come from one address. Generate an identifier per logical session and reuse it:

import os
import uuid
import requests
 
def sticky_session(country: str = "us") -> requests.Session:
    token = uuid.uuid4().hex[:12]
    proxy = (
        f"http://{os.environ['PROXY_USER']}-country-{country}-session-{token}:"
        f"{os.environ['PROXY_PASS']}@gate.uuipproxy.com:7000"
    )
    session = requests.Session()
    session.proxies = {"http": proxy, "https": proxy}
    session.headers.update({
        "Accept-Language": "en-US,en;q=0.9",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    })
    return session
 
with sticky_session() as session:
    session.get("https://example.com/search?q=widgets", timeout=30)
    detail = session.get("https://example.com/product/1234", timeout=30)

requests.Session also persists cookies across those calls, which is the other half of looking like one visitor rather than three unrelated ones.

Async rotation with httpx

For anything above a few hundred requests a minute, the synchronous version stops being the bottleneck you care about. httpx gives you the same proxy semantics with asyncio:

import asyncio
import os
import uuid
import httpx
 
BASE = "gate.uuipproxy.com:7000"
 
def proxy_url(country: str = "us") -> str:
    token = uuid.uuid4().hex[:12]
    user = f"{os.environ['PROXY_USER']}-country-{country}-session-{token}"
    return f"http://{user}:{os.environ['PROXY_PASS']}@{BASE}"
 
async def fetch(url: str, country: str = "us") -> httpx.Response:
    async with httpx.AsyncClient(
        proxy=proxy_url(country),
        timeout=httpx.Timeout(30.0, connect=10.0),
        follow_redirects=True,
    ) as client:
        return await client.get(url)
 
async def main(urls: list[str]) -> None:
    semaphore = asyncio.Semaphore(20)
 
    async def bounded(url: str):
        async with semaphore:
            return await fetch(url)
 
    results = await asyncio.gather(*(bounded(url) for url in urls))
    for url, response in zip(urls, results):
        print(url, response.status_code)
 
asyncio.run(main(["https://example.com/a", "https://example.com/b"]))

Note the semaphore. Concurrency is unmetered on the proxy side, but the target site still has a comfortable ceiling, and exceeding it is the fastest way to turn a working crawler into a blocked one.

Scrapy integration

Scrapy's built-in HttpProxyMiddleware reads request.meta["proxy"], so rotation is a five-line middleware:

# middlewares.py
import os
import uuid
 
class RotatingProxyMiddleware:
    def __init__(self):
        self.user = os.environ["PROXY_USER"]
        self.password = os.environ["PROXY_PASS"]
 
    def process_request(self, request, spider):
        country = request.meta.get("proxy_country", "us")
        token = request.meta.get("proxy_session") or uuid.uuid4().hex[:12]
        user = f"{self.user}-country-{country}-session-{token}"
        request.meta["proxy"] = f"http://{user}:{self.password}@gate.uuipproxy.com:7000"

Register it ahead of the built-in middleware:

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.RotatingProxyMiddleware": 340,
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 750,
}
RETRY_TIMES = 3
RETRY_HTTP_CODES = [403, 407, 408, 429, 500, 502, 503, 504]
DOWNLOAD_TIMEOUT = 30
CONCURRENT_REQUESTS_PER_DOMAIN = 8

Setting proxy_session in request.meta lets an individual spider pin a multi-step flow to one exit while everything else keeps rotating.

Classify responses, not status codes

This is the part that separates a crawler that works from one that appears to work.

A challenge page, an empty result set and a truncated listing all arrive with HTTP 200. If your retry logic only inspects response.status_code, every one of those is recorded as a success and written to your dataset as missing data.

BLOCK_MARKERS = (
    "captcha",
    "are you a human",
    "access denied",
    "unusual traffic",
)
 
def classify(response) -> str:
    if response.status_code in (403, 407, 429):
        return "blocked"
    if response.status_code >= 500:
        return "server_error"
    if response.status_code != 200:
        return "unexpected"
 
    body = response.text.lower()
    if any(marker in body for marker in BLOCK_MARKERS):
        return "challenged"
    if len(response.content) < 2048:
        return "suspicious_short"
    return "ok"

The length check catches the most common silent failure: a shell page that renders fine in a browser but contains none of the data your parser expects. Calibrate the threshold against a known-good response from your actual target.

Escalate instead of hammering

Once responses are classified, retries become a routing decision rather than a loop. The pattern that consistently produces the lowest cost per row looks like this:

1. Send the request through datacenter proxies, which are the cheapest exit you have.
2. Classify the response. If it is ok, you are done — and most requests will be.
3. If it is blocked or challenged, re-queue onto residential.
4. If residential also fails, escalate once more to rotating mobile proxies and cap the attempts there.

TIERS = ["datacenter", "residential", "mobile"]
 
def fetch_with_escalation(url: str, tier_index: int = 0):
    if tier_index >= len(TIERS):
        return None
    response = fetch_via(url, tier=TIERS[tier_index])
    if classify(response) == "ok":
        return response
    return fetch_with_escalation(url, tier_index + 1)

On a typical e-commerce target, fewer than one request in five needs to leave the first tier. The difference against routing everything through residential is usually a factor of three or four on the monthly bill. There is more on choosing between the tiers in our comparison of residential and datacenter proxies.

Make the exit match the request

A rotating pool only helps if each exit is internally consistent with the request it carries. Three mismatches account for most avoidable failures:

Geography versus language. An Italian residential IP requesting Accept-Language: en-US is an obvious anomaly. Derive the header from the country you pinned.

Geography versus content. Requesting the German storefront of a marketplace from a Brazilian exit will often return the Brazilian storefront instead, and your parser will silently record the wrong prices.

Session versus cookies. If you rotate the exit IP but keep the cookie jar, you have told the target that one logged-in identity just moved continents mid-session. Rotate both together or neither.

For a deeper treatment of the signals beyond the IP, see our guide to request fingerprints.

Rate limiting per host

Politeness is not only an ethical position; it is the cheapest form of block avoidance available. A token bucket per hostname keeps a burst from a single worker from dragging the whole crawl into rate-limit territory:

import asyncio
import time
from collections import defaultdict
 
class HostLimiter:
    def __init__(self, per_second: float = 4.0):
        self.interval = 1.0 / per_second
        self.next_slot = defaultdict(float)
        self.lock = asyncio.Lock()
 
    async def acquire(self, host: str) -> None:
        async with self.lock:
            now = time.monotonic()
            wait = max(0.0, self.next_slot[host] - now)
            self.next_slot[host] = max(now, self.next_slot[host]) + self.interval
        if wait:
            await asyncio.sleep(wait)

Start at four requests per second per host and adjust based on the target's own response times. If latency climbs as you increase concurrency, you are past the point where the site is comfortable.

Checklist before you scale up

Run through this before pointing a rotating proxy setup at a production workload:

• Every request has an explicit timeout, and the connect timeout is shorter than the read timeout.
• Responses are classified on content, not just on status code.
• Retries escalate through exit tiers instead of repeating on the same tier.
• Sticky sessions are used wherever a workflow spans more than one page.
• Accept-Language and timezone-dependent parameters match the exit country.
• Concurrency is capped per hostname, not just globally.
• Traffic usage is tracked per target domain so an expensive site is visible before the invoice arrives.
• robots.txt directives for the target are respected, and collection is limited to publicly accessible pages.

The first two items alone will fix the majority of "the proxies stopped working" reports we see in support. Rotation is easy. Knowing whether it worked is the actual engineering.

Debugging a silent 200

The most expensive failure mode is a 200 that is not the page you asked for. Challenge HTML, a regional interstitial, a "enable JavaScript" shell and a truncated listing all share one status code. Log the first 512 bytes of every response body alongside the URL, the exit country, the session id and the elapsed time. When a parser starts returning empty fields, grep that log before you buy more traffic.

A working pattern is to keep a rolling sample of classified failures rather than every body. Store 50 examples of each class (ok, challenge, empty, wrong_locale, truncated) and rotate the sample as new failures arrive. That archive is usually enough to see whether a change in block rate is a new challenge template or just a noisier hour.

If you cannot tell the classes apart by eye, you do not have a classifier yet. Write the rules against those samples, not against a theory of what the target "probably" returns.

Logging that you can actually query

Print statements disappear in a 40-worker crawl. Emit one JSON line per request with a fixed schema:

• url, host, status, bytes, elapsed_ms
• exit_country, session_id, tier (datacenter / residential / mobile)
• class from your classifier
• retry_count and final_tier if the request escalated

Ship those lines to whatever you already run — stdout into a collector is enough. The questions you will ask a week later are always the same: which hosts consume the most gigabytes, which classes rose after a deploy, and whether the residential share is drifting up. None of those are answerable from a stack trace.

Do not log credentials, full cookie jars or request bodies that contain account data. The gateway username already encodes country and session; that is the identifier you need.

Processes, not threads

CPython's GIL makes a thread pool a poor fit for a mixed workload of TLS handshakes and HTML parsing. Prefer one process per core, each with its own httpx.AsyncClient (or equivalent) and its own HostLimiter. Share nothing except a queue of URLs and a sink for classified results.

If you must share a session cookie across processes — a logged-in crawl — pin that workflow to a single process and a static exit. Spreading one login across workers is how accounts get challenged. Everything else can fan out.

A 16-core box running 16 async workers at four requests per second per host will saturate most polite targets long before it saturates the proxy gateway. The limiter, not the hardware, should be the bottleneck you watch.