Avoiding IP Bans: A Practical Guide to Request Fingerprints
Why swapping IPs stops working, and what else your crawler is broadcasting: TLS fingerprints, header order, HTTP/2 settings, timing patterns and the consistency rules that keep sessions alive.
The most common support ticket we get reads roughly: "we rotated to a bigger pool and we are still getting blocked." Nearly every time, the IP was never the reason. It was the thing being blamed because it is the thing that is easy to change.
A modern bot-detection stack scores maybe a dozen signals before the first byte of HTML goes out. The IP is one of them, and often not the most discriminating. This is a tour of the others, in rough order of how often they are what is actually giving you away.
TLS fingerprints
Before any HTTP happens, your client sends a TLS ClientHello. That message contains the cipher suites it supports, in order; the extensions it advertises, in order; the elliptic curves and signature algorithms it accepts; and the ALPN protocols it offers.
Every HTTP library produces a distinctive combination. Python's requests on OpenSSL produces one fingerprint. Go's net/http produces another. Chrome produces a third, and Chrome's changes with each major release. Hashing these fields into a JA3 or JA4 string gives a detector a stable identifier that is completely independent of your IP address.
This is why a crawler can rotate through ten thousand residential IPs and get blocked on every one. From the detector's point of view it is not ten thousand visitors; it is one client library with ten thousand addresses.
The fix is to use a client that reproduces a real browser's ClientHello. In Python, curl_cffi impersonates specific Chrome and Safari builds:
from curl_cffi import requests
response = requests.get(
"https://example.com",
impersonate="chrome124",
proxies={"https": PROXY},
timeout=30,
)In Node, undici with a custom TLS configuration or a headless browser gets you the same result. Whatever you choose, keep it current: impersonating a Chrome build from two years ago is its own anomaly.
HTTP/2 settings and header order
If the TLS handshake negotiates HTTP/2 — and against any large site it will — the connection opens with a SETTINGS frame. The parameters in that frame, their order, the initial window size, and the priority tree the client builds are all library-specific. They form a second fingerprint that is as stable as the TLS one.
Header ordering matters just as much. A real browser sends headers in a consistent sequence, and that sequence differs between Chrome, Firefox and Safari. Most HTTP libraries either sort headers alphabetically or emit them in dictionary insertion order. Neither matches any browser.
You cannot fix this by adding a User-Agent string. A request claiming to be Chrome 124 while sending headers in Python's insertion order is a clearer signal than no User-Agent at all, because it proves intent to deceive rather than mere automation.
Either use a client that controls header order properly, or do not claim to be a browser.
Header content consistency
Assuming the transport layer is handled, the headers themselves need to agree with each other and with the exit IP. The mismatches we see most often:
Accept-Language versus IP geography. A residential IP in Warsaw sending Accept-Language: en-US,en;q=0.9 is a mismatch. Derive the header from the country you pinned on the proxy.
User-Agent versus Sec-CH-UA. Chrome sends client hints alongside the user agent. If you spoof one and omit the other, or the version numbers disagree, that is a direct contradiction.
Missing Sec-Fetch-* headers. Modern browsers send Sec-Fetch-Site, Sec-Fetch-Mode, Sec-Fetch-Dest and Sec-Fetch-User. Their absence on a request claiming to come from Chrome is conspicuous.
No Referer on a deep page. Real users arrive at a product page from a listing. A request that lands directly on a deep URL with no referrer, repeatedly, is a crawl pattern.
Accept-Encoding that the client cannot handle. Advertising br and then failing to decode Brotli is an easy tell.
Timing and behavioural patterns
Once the per-request signals check out, the pattern across requests takes over.
Human browsing is bursty and irregular. A person loads a page, reads for eleven seconds, clicks, reads for four, opens a tab, comes back ninety seconds later. A crawler fetching every 500 ms, forever, with a standard deviation near zero, does not resemble that at any pool size.
Jitter helps, but only if it is realistic. Uniform random between 400 and 600 ms is still obviously machine-generated — the distribution is wrong. Human inter-action intervals are roughly log-normal: mostly short, with a long tail. Sampling from that shape is cheap and much more convincing.
Request ordering matters too. Real sessions follow links. A crawler that fetches /product/1, /product/2, /product/3 in ascending numeric order at a constant rate is describing itself accurately.
Finally, real browsers fetch subresources. A client that requests only the HTML document and never the CSS, fonts or images referenced in it is producing a traffic shape no browser produces.
Cookie and session state
A session that changes its exit IP mid-flight while holding the same cookie has told the target that one identity teleported. This is a strong signal and an entirely self-inflicted one.
The rule is simple: the cookie jar and the exit IP have the same lifetime. Rotate both together, or neither. Our Python rotating proxy guide has a sticky-session pattern that keeps these aligned.
For anything involving a login, do not rotate at all. Use a static ISP proxy or a dedicated static mobile proxy so the account keeps one address indefinitely. An account that logs in from a different city every day will accumulate verification challenges regardless of how clean each individual IP is.
Where the IP still matters
None of this means the exit address is irrelevant. It matters in three specific ways.
Class. Hosting-classified addresses are rejected outright by reputation-checking targets. This is the boundary where residential proxies become necessary, discussed in more depth in our residential versus datacenter comparison.
Geography. Localized pricing, inventory and search results depend on where the request appears to come from. Getting this wrong produces data that is confidently incorrect.
History. An address that has been hammering the target for six hours carries that history. Rotation resets it; that is the actual value rotation provides.
What the IP cannot do is compensate for a fingerprint that identifies your client library.
A diagnostic order of operations
When requests start failing, work through this in order rather than reaching for a bigger pool:
- Capture a failing response in full. Headers, body, status. Read it. Frequently the page states the reason.
- Retry the same URL from your own laptop browser, unproxied. If it also fails, the target changed, not your setup.
- Retry through the proxy with a real browser. If the browser succeeds where your client fails, the problem is the client fingerprint, not the IP.
- Compare your TLS fingerprint to Chrome's. Public JA3/JA4 test endpoints make this a two-minute check.
- Diff your headers against a real browser request to the same URL, including order.
- Only then consider the exit type. If a browser through the same proxy also fails, the IP class is genuinely the issue.
Steps three and four resolve the majority of cases. The rest usually come down to timing.
A consistency checklist
Everything above reduces to one principle: every layer of the request should agree about who is making it.
- TLS fingerprint matches a current browser build
- HTTP/2 settings and header order match that same browser
User-Agent, client hints andSec-Fetch-*headers are internally consistentAccept-Languagematches the exit country- Timezone-dependent parameters match the exit country
- Cookie lifetime matches exit IP lifetime
- Inter-request intervals follow a realistic distribution
- Navigation follows links rather than enumerating identifiers
- Logged-in work uses a static exit, never a rotating one
A crawler that satisfies all nine will survive on a much smaller IP pool than one that satisfies only the last item. That is usually the cheaper investment, too.
What a challenge page actually looks like
Most teams only notice a block when the parser returns null. By then you have already paid for the bytes and polluted the dataset. Challenge pages have a small set of tells that a classifier can catch in a few milliseconds:
- A title or heading that mentions "Access denied", "Verify you are human", "Enable JavaScript", or a CDN brand.
- A body under a few kilobytes that is mostly script tags and a noscript warning.
- A
Locationhop to a/cdn-cgi/,/challenge/, or/sorry/path. - A cookie being set whose name you never see on a successful page (
cf_clearance,ak_bmsc, and cousins).
None of these require rendering. Match them on the raw bytes, class the response as challenge, and escalate or skip. Rendering a headless browser "just to be sure" is how a cheap miss turns into a slow, expensive miss.
If the same URL returns a challenge from datacenter and a real page from residential, the IP class is the signal. If both return a challenge, look at TLS and headers before you buy a more expensive exit. The diagnostic order above exists so you do not skip that fork.
About the author
Lead Infrastructure Engineer, UUIProxy
Elena has spent eleven years on distributed networking, the last six of them on proxy gateway architecture. She designed UUIProxy's session-affinity layer and the response-classification pipeline that decides when a request should be retried on a higher-trust exit. She writes about the parts of scraping infrastructure that only become visible at scale.
