proxies

Proxy routing hub for web scraping API delivery

Use proxy routing when a public target proves that network path, geography, reputation, rate pressure, or session continuity changes the response. ScraperFly helps teams start light and escalate delivery only when evidence says it matters.

Proxy routing should solve a proven delivery problem, not become the default answer to every failed scrape. A strong workflow starts with a simple request, inspects the response, and then decides whether rotating routes, premium delivery, country targeting, or session continuity can improve the result.

ScraperFly is built around that practical escalation path. Keep straightforward pages on a lighter request profile, then add stronger routing when the target response shows blocked access, regional behavior, unstable delivery, or a need for continuity across related requests.

Choose routing by the target behavior you can observe

A target may fail for many reasons: malformed input, missing headers, JavaScript-rendered content, rate pressure, a regional redirect, a blocked network path, or a page that returns a valid status but useless content. Proxy routing helps when the issue is delivery-related, but it should be selected from evidence.

That evidence can include status codes, final URL, response size, content type, sample variance across routes, visible challenge pages, location-specific content, and whether related requests depend on cookies or state. The more specific the evidence, the easier it is to choose the right control without wasting request weight.

Routing choice	Use it when	What it helps prove	Cost-control habit
Standard routing	The target returns useful content on a light request	The workflow does not need heavier delivery	Keep this as the default profile
Rotating proxies	Repeated requests need route distribution or delivery diversity	Whether route rotation improves stability	Use after comparing samples, not before testing
Premium routing	Standard delivery returns blocks, challenge pages, or unstable responses	Whether stronger reputation changes the captured result	Escalate only for targets that prove they need it
Country targeting	Public content changes by location, currency, language, availability, or regulation	Whether geography changes the page evidence	Choose the country required by the workflow
Sessions	Related requests need continuity, cookies, preferences, or shared state	Whether persistence changes the response	Use sessions inside related flows, not isolated captures

Keep proxy escalation connected to diagnostics

Proxy features are most valuable when the team can explain why they are being used. A 403 may suggest access or reputation signals. A 429 points to rate pressure. A regional redirect points to country targeting. A login wall or consent flow may need headers, cookies, or sessions. An empty JavaScript shell may need rendering rather than stronger routing.

The practical path is to classify the response first, then choose the next control. That makes proxy routing a precise part of the workflow instead of an expensive blanket setting.

Practical takeaway: use the lightest delivery path that returns usable public page evidence, then escalate routing only when the response proves the simple path is not enough.

DELIVERY ESCALATION

Escalate routing from response evidence, not guesswork

A target 403, challenge page, regional mismatch, unstable response, or repeated route failure should guide the next delivery choice. ScraperFly helps teams start with a light profile and move to stronger routing only when the response proves the need.

A simple HTTP request returning a clean HTML response

Plain HTML request

Use this first for public pages that already include the required data in the initial HTML.

A page changing from an empty HTML shell to a rendered DOM after JavaScript runs

JavaScript rendering

Enable rendering when the server returns a shell, but the useful page content appears only after JavaScript executes.

Headers, cookies and session state preserved across two requests

Headers, cookies & sessions

Use visitor-state controls when the target expects continuity between requests or returns different content after consent, locale, or navigation state is set.

A request routed through a selected country and higher reputation proxy path

Premium proxy & geo

Escalate routing when the response depends on request origin, IP reputation, or target geography.

A browser automation flow with search, filters, scrolling and data capture steps

Browser flow

Use a full browser flow when the data is behind interaction rather than a single request: search boxes, filters, scrolling, tabs, or multi-step navigation.

Escalate when:

Plain HTML request

the raw response contains the fields you need, status codes are stable, and page size/content do not collapse after repeated checks.

Product details, article text, documentation, or listings are visible in the HTML response.
The final URL, status, and content length stay consistent across a small sample.
No consent wall, visitor state, or client-side API call is required for the target fields.

JavaScript rendering

the raw HTML is mostly placeholders, the rendered DOM contains the target fields, or key data arrives through client-side requests.

Raw HTML has app containers, loading skeletons, or empty product grids.
Browser inspection shows the fields after scripts finish.
Screenshots look correct while the HTML response alone is incomplete.

Headers, cookies & sessions

pagination, reviews, filters, or consent flows depend on the same visitor context across multiple requests.

The first request sets cookies that change later responses.
Locale, consent, currency, cart, or filter state must be preserved.
The page redirects or empties out when requests look stateless.

Premium proxy & geo

the same URL behaves differently by region, returns 403/429 from one route, or shows location-specific pricing, availability, SERP, ads, or catalog data.

Datacenter-like traffic receives blocks while a higher-reputation route works.
Country, language, currency, inventory, or search results change by geo.
Repeated requests trigger rate limits or challenge pages on specific routes.

Browser flow

the target data appears only after user-like navigation steps, or protected areas such as reviews, profiles, search result refinements, or special data panels behave differently from simple detail pages.

A click, scroll, search, or filter action is required before the fields appear.
Reviews, profiles, or dynamic result pages load in separate steps.
Simpler fetch/render attempts return partial content, challenge pages, or unstable states.

Stay lower & Credit impact:

Plain HTML request

Stay lower: the page already returns complete data. Adding rendering or premium routing here only adds latency and credit usage without improving the result.

Credit impact: lowest request profile because it avoids browser work and special routing.

JavaScript rendering

Stay lower: the fields are already present in raw HTML. A modern-looking site does not automatically mean JavaScript rendering is needed.

Credit impact: higher than plain HTML because browser work, wait time, and page resources are involved.

Headers, cookies & sessions

Stay lower: public content is stable without state. Do not invent sessions for pages that can be fetched cleanly as anonymous HTML.

Credit impact: usually moderate; the bigger cost is operational complexity and retry discipline.

Premium proxy & geo

Stay lower: the target is globally consistent and accepts normal routing. Premium and geo routing should be tied to evidence, not used as a default blanket setting.

Credit impact: can increase credit usage and latency, so apply it to the URL groups that actually need it.

Browser flow

Stay lower: a single HTML or rendered request already returns the target. Browser flows are the strongest tool in this ladder, not the default starting point.

Credit impact: highest request profile because it combines browser runtime, waits, navigation, and often heavier routing.

Escalate by evidence, not by habit

For most public websites, a plain HTML request or light JavaScript rendering is enough. Start there, inspect the response, and only then decide whether the next step is justified.
Anti-bot signals are usually visible in the response: unexpected 403 or 429 statuses, sudden redirects, tiny or empty HTML, challenge pages, CAPTCHA screens, login or consent loops, and content that disappears after repeated requests.
Large high-traffic targets such as Amazon, Google, LinkedIn, Instagram, TikTok, and major marketplaces often protect some surfaces aggressively. They may require rendering, sessions, careful routing, or a browser flow, and each target needs its own test profile.
Protection is not uniform even inside one site. Product/detail pages may open with a lighter profile, while reviews, profiles, search pages, price-sensitive views, availability panels, or account-adjacent data can require stronger escalation.
Geo and premium proxy settings affect both what you receive and how expensive the request profile becomes. Use them when country-specific content, IP reputation, or repeated blocks prove they matter.
The practical goal is not to turn on every ScraperFly control. The goal is to find the sufficient level that returns complete, repeatable data while keeping credit usage, latency, and failure modes under control.

ROUTING FIT

Map geography, reputation, and session needs before scaling

Proxy strategy depends on what the target actually changes. Check whether content varies by country, whether repeated requests need stable state, whether headers or cookies affect the response, and whether standard delivery returns usable data.

Raw HTML response mapped to visible page data

Raw HTML availability

Check whether the fields you need are already present in the first HTML response.

Best first test for simple pages, articles, docs, listings, and product details.

Before and after JavaScript rendering comparison

JavaScript dependency

Compare raw HTML with the rendered DOM or screenshot before turning on browser rendering.

Use rendering when raw HTML is only a shell and scripts create the useful content.

Cookies, consent, session and access context for scraping

State & access context

Identify whether cookies, consent, login boundaries, pagination, or the same visitor state affect the response.

Use sessions and headers when continuity changes the data you receive.

Regional content variations across countries and routes

Geo/content variation

Test whether country, language, currency, inventory, SERP, ads, or route reputation changes the result.

Use geo or premium routing only when the target proves the response varies by route.

Questions to answer before scraping

Where is the target field?

Find whether the field exists in raw HTML, rendered DOM, an API call, structured data, a file, or a later interaction state.

What changes the response?

Sample a few URLs and compare status, final URL, content length, redirects, empty states, and visible result differences.

What context is required?

Mark any dependency on consent, locale, cookies, headers, authentication boundary, pagination, filters, or previous requests.

What does failure look like?

Define the signals for blocked, partial, stale, redirected, empty, timed-out, or region-mismatched responses before scaling.

What this changes in request setup

Fetch mode

Raw HTML availability decides whether the first request can stay simple or needs rendering, waits, screenshots, or extraction.

Request context

State requirements decide whether to add headers, cookies, sessions, consent handling, or a stable visitor profile.

Routing choice

Geo and reputation findings decide whether standard routing is enough or a country/premium route is justified.

Validation rules

Known failure states become checks for retries, escalation, QA review, and deciding when a lighter profile is still complete.

REQUEST SETUP

Test route, country, session, and output choices in one request profile

A reliable proxy workflow should make delivery choices explicit: standard routing, premium routing, country targeting, sessions, headers, rendering, waits, and output format. Test the target before embedding the profile in production jobs.

HTML

Raw response

Raw HTML for parsers and crawlers

Use the fastest path for static pages, listings, docs, and product pages. Fetch the original source HTML first, then switch on browser rendering when JavaScript creates the markup your scraper needs.

format=htmlrender=falserender=true

TXT

Clean data

Text, Markdown, and JSON

Skip the cleanup layer after page access. Return readable text for agents, Markdown for prompts and RAG, or structured JSON with metadata, links, headings, clean text, and Markdown ready for your pipeline.

format=textformat=markdownformat=json

PDF

Visual files

Screenshots and PDFs

Capture what a real browser sees when you need QA evidence, archived proof, visual regression checks, downloadable reports, or shareable page records for customers and internal teams.

format=screenshotformat=pdfdownload

Controls that keep requests predictable

url

The target page to fetch, render, clean, or capture.

format

Select the response your product needs: html, text, markdown, json, screenshot, or pdf.

premium

Escalate to stronger routing for blocks, unstable targets, or geo-sensitive content.

render

Run Chromium for SPAs, JS-injected markup, screenshots, PDFs, and browser-only pages.

country

Request a country-specific route when content changes by location.

keep_headers

Forward selected request context such as user-agent, cookie, language, and referer.

render_wait

Give dynamic content a controlled extra wait before extraction or capture.

download

Return visual artifacts as downloadable files when the output is screenshot or PDF.

cURL example

Generate Markdown with the same request shape you can move into production

curl "https://scraperfly.com/api/v1?url=https%3A%2F%2Fexample.com&format=markdown" \
  -H "apikey: YOUR_API_KEY"

ROUTING COST

Keep stronger delivery reserved for targets that prove they need it

Premium routing, browser rendering, visual files, and long waits can all add weight. ScraperFly's flexible credit model lets teams validate the target first, then choose pay-as-you-go or monthly credits once routing needs and volume are clear.

A lightweight HTML request returning a simple response

Basic fetch

Use a basic fetch when public HTML already contains the fields you need.

Lightest request profile; keep it when the response is complete.

Rendering

Use rendering when raw HTML is a shell and JavaScript completes the page.

Heavier profile; use it only when browser output changes the result.

Several requests preserving cookies and visitor state

Stateful session

Use sessions when cookies, consent, locale, or pagination change the response.

Heavier state profile; preserve it only when continuity matters.

Request routing through geo and higher quality routes

Premium routing & geo

Use premium routing or geo only when origin, country, reputation, or regional content changes the result.

Heaviest routing profile; apply it only to targets that prove they need it.

What makes a request heavier

More infrastructure work

Browser rendering, routing choices, and longer waits mean the platform has to do more than return a simple response.

More target uncertainty

Redirects, empty states, rate limits, and challenge pages often require stronger delivery options or retries.

More state to preserve

Cookies, sessions, consent, and multi-page flows add moving parts that a plain stateless fetch does not have.

More evidence to return

Screenshots, PDFs, rendered output, and richer response artifacts are useful, but they are heavier than plain HTML.

How to keep request weight predictable

Sample before scale

Test a small URL set first and group targets by the lightest profile that returns complete data.

Do not render by default

Inspect raw HTML before turning on rendering. Many public pages already contain the data you need.

Escalate by evidence

Move up only when the response shows missing content, state dependency, geo variance, or routing problems.

Measure useful results

Optimize around valid records delivered, not raw request volume. A lighter profile is best only when it stays complete.

PROXY FAQ

Common questions about proxy routing for web scraping

Practical answers for teams deciding when rotating proxies, premium delivery, country targeting, and sessions belong in a public web data workflow.

Routing answers

Requests

Making API requests

For developers wiring ScraperFly into agents, jobs, backend services, or quick tests.

What do I send to ScraperFly?

Send a target URL plus the output format you need. The API can return HTML, clean text, Markdown, JSON, screenshots, or PDFs depending on the request options.

Should I use GET or POST?

Both can produce the same scraping result. GET is convenient for simple query parameters; POST is better when you want a clear JSON body or longer request options.

Can I test a request before coding?

Yes. The API Builder generates the endpoint, JSON body, query parameters, and code samples, then lets you send a live request from the dashboard.

Output

HTML, text, Markdown, and JSON

Choose the smallest output that your product actually needs.

When should I use clean text?

Use clean text when you need readable page content for summaries, classification, search indexing, or LLM prompts without layout markup.

What is the JSON output for?

JSON is useful when you want structured metadata, headings, links, text, and Markdown in one response that is easier to inspect programmatically.

Is Markdown different from clean text?

Markdown keeps more content structure, such as headings and lists. Clean text is flatter and better when formatting is less important.

Rendering

JavaScript pages

Use browser rendering only when the target page needs it.

When do I enable render=true?

Enable rendering for SPAs, JavaScript-injected product pages, lazy-loaded content, or pages where the source HTML does not contain the data you need.

Does rendering always cost more?

Yes, browser work is heavier than a basic HTML fetch. Start without rendering, then enable it only for targets that genuinely require JavaScript execution.

Can I control viewport and wait time?

Yes. Screenshot and PDF requests can use viewport dimensions and render wait options so the captured page has time to settle.

Routing

Proxies and location

Route requests through standard or premium pools when target behavior changes.

When do I need premium proxy routing?

Use premium routing for stricter targets, higher block rates, unstable responses, or pages that behave differently under normal datacenter traffic.

Can I request a country?

Yes. Add a country option when the target content depends on location, currency, language, availability, or regional redirects.

Are sessions supported?

Session-style workflows can reuse context across related requests when continuity matters for pagination, locale changes, or multi-step scraping.

Files

Screenshots and PDFs

Capture rendered pages when a visual artifact is the required output.

Can ScraperFly return a screenshot?

Yes. Screenshot output captures the rendered page as an image file, including JavaScript-rendered content when rendering is enabled.

Can ScraperFly generate a PDF?

Yes. PDF output is useful for receipts, reports, archives, compliance snapshots, and pages that need to be saved as a document.

Where do generated files appear?

Registered dashboard users can save and download generated artifacts from request logs when the request produces a downloadable file.

Credits

Costs and credits

Credits track delivery work rather than idle infrastructure.

How many credits does a request use?

A simple HTML, text, Markdown, or JSON request usually uses fewer credits. Rendering, premium routing, screenshots, and PDFs use more because they require heavier delivery work.

Do pay-as-you-go credits expire?

Pay-as-you-go credits stay available until you use them. They are a good fit for uneven workloads or projects that do not need a monthly plan.

Do subscription credits roll over?

Yes. Monthly plan credits refresh every month and unused credits roll over. If payment fails after Stripe retries, the account falls back to Pay as you go.

Security

Keys, headers, and safety

Keep credentials on your backend and forward only the headers you intend to use.

Where should I store my API key?

Store the API key server-side or in a trusted environment. Avoid exposing production keys in public frontend bundles or static pages.

Can I forward request headers?

Yes, selected headers such as user-agent, accept-language, cookie, and referer can be forwarded when a target expects realistic request context.

Does ScraperFly bypass private access controls?

No. It is designed for fetching public web pages and pages you are allowed to access, not for bypassing authentication or private systems.

Billing

Plans, logs, and invoices

Use the dashboard to understand what happened and why credits changed.

Where can I inspect usage?

The dashboard records request logs, credit balance changes, output formats, proxy routing, durations, and billing events so you can audit usage.

Can I download invoices?

Paid transactions include invoice data when Stripe provides it, so invoice PDFs can be opened from the financial log.

How do I manage a subscription?

Subscribed users can open the customer portal to change payment details, review invoices, or cancel. Free and Pay as you go users can go to Pricing instead.