429 pacing workflow
Slow the workflow deliberately before you scale the same request harder
A 429 is not the same kind of failure as a malformed URL, missing API key, or target 403. It usually says the current request rhythm is too aggressive for an account limit, plan limit, concurrency setting, retry policy, or the public target's tolerance.
ScraperFly helps teams treat 429 as an operating signal. Check remaining credits, plan allowances, route concurrency, response evidence, and retry windows, then apply backoff or reduce pressure before adding heavier request controls.
What does a 429 mean in scraping?
A 429 response usually means too many requests were sent for the current context. That context can be your account, your plan, a concurrency rule, the target website, a route class, or a narrow window of repeated retries. The right fix depends on which limit was hit.
That is why 429 handling should begin with classification. If every target starts failing at the same time, check account state, credits, monthly allowances, and global concurrency. If one target or route group returns 429 while other targets remain healthy, inspect target-side tolerance, retry rhythm, request spacing, and whether the route is heavier than expected.
Practical takeaway: a 429 is a pacing signal first. Reduce pressure and inspect limits before making every request more expensive.
How do you separate account limits from target-side rate limits?
Account-side limits usually show up across workflows or after a predictable usage threshold. They point to remaining credits, monthly plan allowances, concurrency ceilings, unpaid status, or an operating rule designed to keep the account stable. Target-side rate limits usually show up around a specific public website, route family, country, time window, or request profile.
Both cases deserve different action. Account limits need visibility and planning. Target-side limits need pacing, route grouping, backoff, and sometimes a narrower capture profile. Treating every 429 as a proxy problem can waste credits because the target may simply need fewer concurrent requests or a longer retry window.
| 429 signal | Likely meaning | Best next action |
|---|---|---|
| 429 appears across many targets | Account, plan, or global concurrency pressure | Check credits, plan limits, unpaid status, and worker settings. |
| 429 appears on one target | The public website is limiting the current request rhythm | Reduce concurrency and add route-specific backoff. |
| 429 appears after retries | The retry policy may be amplifying the limit | Use bounded exponential backoff and retry caps. |
| 429 appears only on rendered requests | The request profile may be too heavy for volume | Classify which routes truly need rendering. |
| 429 appears by country or time window | Target tolerance may vary by region or schedule | Group routes and pace requests by target context. |
What should a safe backoff policy do?
A safe backoff policy slows down repeated attempts without hiding the underlying problem. It should retry only when the failure is likely temporary, increase delay after repeated limits, cap total attempts, and log the reason each retry happened. If the target keeps returning 429 after backoff, continuing to retry is usually less useful than reducing concurrency or changing the route strategy.
Backoff should also be scoped. One target's limit should not pause every unrelated workflow unless the evidence points to account-wide pressure. Route groups, domains, countries, and request profiles can have different tolerance. A production scraper becomes more reliable when it slows the right lane instead of stopping or accelerating the whole system.
For broader failure decisions, compare the scraping API 403 errors page and the empty HTML response scraping API guide. They cover cases where the response is blocked, incomplete, or misleading rather than simply rate-limited.
How should teams tune concurrency?
Concurrency should be based on evidence, not ambition. Start with conservative request volume, measure successful responses, latency, response size, and 429 frequency, then increase only when the target remains stable. A route that needs browser rendering, visual files, premium routing, or session continuity should usually scale more carefully than a simple HTML route.
Request weight matters because heavier profiles consume more resources and can increase pressure on both sides of the workflow. A basic HTML request is not the same as a rendered browser session with waits, screenshots, and stronger routing. The safest strategy is to classify route patterns first, then assign concurrency by target and profile.
| Control | Use it when | 429-safe behavior |
|---|---|---|
| Backoff | The limit may be temporary or window-based | Increase delay, cap attempts, and stop noisy retry loops. |
| Concurrency limits | Too many workers hit the same target at once | Throttle by domain, route group, and request profile. |
| Account checks | Limits appear across workflows | Review credits, monthly allowance, unpaid status, and plan fit. |
| Route classification | Some pages are heavier than others | Keep simple pages light and isolate rendered or visual jobs. |
| Request controls | Target behavior depends on context | Add headers, sessions, country routing, or rendering only when evidence supports it. |
When should a 429 change the request profile?
A 429 should change the request profile only after pacing has been checked. Sometimes the answer is simply less concurrency. Sometimes the route needs a different workflow: source HTML instead of rendered output, text or Markdown instead of screenshot and PDF, fewer waits, narrower target selection, or better grouping by domain and route family.
There are cases where request controls matter. Headers can help when a public target expects legitimate context. Sessions can help when rate behavior depends on continuity. Country routing can help when limits differ by geography. Premium routing can help when delivery route quality is the limiting factor. But those are targeted decisions, not default fixes for every 429.
Use the API Builder to test a smaller profile before scaling. Compare source HTML, rendered output, response formats, final URL, status, response size, and latency. The web scraping API response formats page can help choose the lightest output your downstream workflow can use.
How do credits and billing fit into 429 recovery?
Credits and billing do not replace pacing, but they do affect operating decisions. If a project is still discovering target tolerance, pay-as-you-go credits keep the buying path flexible. If recurring volume is predictable and route profiles are proven, monthly credits can make planning easier. Either way, request pressure still needs to match account limits and target tolerance.
ScraperFly's pricing model is useful because it lets teams start small, test the target, and scale after the workload is better understood. A team can keep uneven experiments on flexible credits, move recurring jobs to monthly plans, and avoid turning a short-lived test into a large fixed subscription. Review pricing and the web scraping API credit cost calculator when estimating how request weight and volume interact.
How does ScraperFly make 429 handling practical?
ScraperFly gives teams one API layer for request formats, rendering, routing, sessions, headers, visual output, and diagnostics. That makes rate-limit handling easier because the response evidence, request controls, and cost model stay in the same workflow. You can see whether the problem is account pressure, target pressure, route weight, or retry behavior.
The goal is not to push harder by default. The goal is to make a cleaner scaling decision: start with the lightest useful request, slow down when the target asks for space, reserve heavier profiles for routes that prove they need them, and grow volume once the retry rhythm is stable.
Practical takeaway: good 429 handling turns failed volume into better pacing, clearer limits, and a request profile that can scale without constantly fighting the target.
