rendering

Rendering hub for JavaScript-heavy web scraping workflows

Use browser rendering when a public page needs JavaScript execution, rendered HTML, screenshots, PDFs, or wait controls. Start with source HTML first, then add rendering only when the target proves it changes the captured result.

Rendering is a decision, not a default. A practical scraping workflow starts by checking the source HTML, then enables browser execution only when the useful public content is created after JavaScript runs or when the team needs visual proof through a screenshot or PDF.

ScraperFly gives teams a controlled path for that decision. Test the target, compare source and rendered output, choose the smallest useful response format, and add waits, screenshots, PDFs, sessions, headers, country routing, or premium delivery only when they improve the captured result.

Use rendering when JavaScript changes the response

Many modern public pages send a thin HTML shell first and build the useful DOM in the browser. Product grids, search results, maps, dashboards, reviews, and SPA routes may appear only after frontend JavaScript loads data and updates the page. In those cases, browser rendering can return a response that source HTML alone cannot provide.

Other pages do not need rendering at all. If the source response already contains the content, a basic HTML request is usually faster, cheaper, and easier to debug. The strongest rendering workflow is evidence-based: inspect the first response, prove what is missing, then enable browser execution for the targets that need it.

Rendering decision	Use it when	Start with	Next control to test
Raw HTML is enough	The useful content is already in the initial response	HTML output without rendering	Parser logic, metadata, or response format
Rendered HTML is needed	The DOM is completed after browser-side JavaScript runs	Rendering with rendered HTML output	Wait timing if the content loads asynchronously
SPA capture is needed	React, Vue, Angular, Next.js, or another frontend route changes visible content	Rendering plus final URL and status inspection	Sessions, headers, or route-specific waits
Visual proof is needed	The workflow must show what the rendered page looked like	Screenshot or PDF output	Viewport, wait timing, download, or archive settings

Rendered output still needs the right format

Rendering solves page execution, but it does not decide what the application should consume. A parser may need rendered HTML. An AI or RAG workflow may prefer clean text or Markdown. A monitoring system may need JSON metadata for status and diagnostics. A QA or compliance workflow may need screenshots or PDFs as evidence.

The practical path is to choose the output format before scaling the request. That keeps the pipeline focused on the asset the product actually needs instead of generating heavy files or raw markup that downstream systems must clean later.

Practical takeaway: render only when the page behavior proves it. Then return the rendered result in the format your application can use immediately.

RENDERING FIT

Check whether JavaScript actually changes the captured page

Rendering should follow evidence. Compare source HTML with the rendered result, inspect final URL, response size, status, and screenshot state, then enable browser execution only for targets where it changes the useful output.

Raw HTML response mapped to visible page data

Raw HTML availability

Check whether the fields you need are already present in the first HTML response.

Best first test for simple pages, articles, docs, listings, and product details.

Before and after JavaScript rendering comparison

JavaScript dependency

Compare raw HTML with the rendered DOM or screenshot before turning on browser rendering.

Use rendering when raw HTML is only a shell and scripts create the useful content.

Cookies, consent, session and access context for scraping

State & access context

Identify whether cookies, consent, login boundaries, pagination, or the same visitor state affect the response.

Use sessions and headers when continuity changes the data you receive.

Regional content variations across countries and routes

Geo/content variation

Test whether country, language, currency, inventory, SERP, ads, or route reputation changes the result.

Use geo or premium routing only when the target proves the response varies by route.

Questions to answer before scraping

Where is the target field?

Find whether the field exists in raw HTML, rendered DOM, an API call, structured data, a file, or a later interaction state.

What changes the response?

Sample a few URLs and compare status, final URL, content length, redirects, empty states, and visible result differences.

What context is required?

Mark any dependency on consent, locale, cookies, headers, authentication boundary, pagination, filters, or previous requests.

What does failure look like?

Define the signals for blocked, partial, stale, redirected, empty, timed-out, or region-mismatched responses before scaling.

What this changes in request setup

Fetch mode

Raw HTML availability decides whether the first request can stay simple or needs rendering, waits, screenshots, or extraction.

Request context

State requirements decide whether to add headers, cookies, sessions, consent handling, or a stable visitor profile.

Routing choice

Geo and reputation findings decide whether standard routing is enough or a country/premium route is justified.

Validation rules

Known failure states become checks for retries, escalation, QA review, and deciding when a lighter profile is still complete.

REQUEST CONTROLS

Build the rendered request profile before it becomes production code

A rendering workflow should make each control explicit: target URL, render flag, wait timing, output format, screenshots or PDFs, sessions, headers, country routing, and downloads. Testing that shape first prevents heavy defaults from spreading across every scrape.

HTML

Raw response

Raw HTML for parsers and crawlers

Use the fastest path for static pages, listings, docs, and product pages. Fetch the original source HTML first, then switch on browser rendering when JavaScript creates the markup your scraper needs.

format=htmlrender=falserender=true

TXT

Clean data

Text, Markdown, and JSON

Skip the cleanup layer after page access. Return readable text for agents, Markdown for prompts and RAG, or structured JSON with metadata, links, headings, clean text, and Markdown ready for your pipeline.

format=textformat=markdownformat=json

PDF

Visual files

Screenshots and PDFs

Capture what a real browser sees when you need QA evidence, archived proof, visual regression checks, downloadable reports, or shareable page records for customers and internal teams.

format=screenshotformat=pdfdownload

Controls that keep requests predictable

url

The target page to fetch, render, clean, or capture.

format

Select the response your product needs: html, text, markdown, json, screenshot, or pdf.

premium

Escalate to stronger routing for blocks, unstable targets, or geo-sensitive content.

render

Run Chromium for SPAs, JS-injected markup, screenshots, PDFs, and browser-only pages.

country

Request a country-specific route when content changes by location.

keep_headers

Forward selected request context such as user-agent, cookie, language, and referer.

render_wait

Give dynamic content a controlled extra wait before extraction or capture.

download

Return visual artifacts as downloadable files when the output is screenshot or PDF.

cURL example

Generate Markdown with the same request shape you can move into production

curl "https://scraperfly.com/api/v1?url=https%3A%2F%2Fexample.com&format=markdown" \
  -H "apikey: YOUR_API_KEY"

Error handling

Turn scraping failures into fixes

Every web data workflow eventually meets a bad response: a missing URL, an invalid key, an empty HTML shell, a target 403, a rate limit, a render timeout, a broken PDF capture, or a JSON payload that does not contain the fields your product expected. ScraperFly is built to make those moments diagnosable. You inspect the HTTP status, the stable error code, the request profile, and the returned artifact, then choose the cheapest useful next move: fix the request, add credits, slow down, retry with backoff, enable rendering, change country, use premium routing, preserve headers, or validate the output before scaling.

Stable error shapeRetry decisionsCost-aware escalationOutput validation

Developer reviewing code and API request parameters on a laptop

01 / Request setup and access

Fix request mistakes first

Some failures are not target problems at all. They happen before ScraperFly can make a meaningful delivery attempt: a missing or malformed URL, an unsupported output format, an incompatible visual file type, a missing API key, an invalid key, or a key that has been paused or disabled. These errors are configuration signals, not retry signals: stop the workflow, point the developer to the exact field that needs correction, and keep credits focused on real target tests instead of repeated bad inputs.

missing_urlinvalid_urlunsupported_formatinvalid_api_key

Validate the URL and encoding before sending a batch.
Use supported outputs: HTML, text, Markdown, JSON, screenshot, or PDF.
Match screenshot file types to image output and PDF file type to PDF output.
Treat missing or invalid API keys as configuration work, not retry work.

Business dashboard with charts, documents and usage planning on a desk

02 / Credits, limits and concurrency

Separate limits from failures

A workload can fail because the account has no usable credits, the monthly plan limit has been reached, or concurrency is too aggressive for the current plan and target mix. ScraperFly makes those limits visible as operating signals, not mysterious scrape failures. That keeps the buying decision practical: start free, test the target, buy pay-as-you-go credits for uneven jobs, move to monthly credits when volume becomes predictable, and use backoff when request pressure is the real issue.

usage_limit_reachedrate_limitedcreditsbackoff

Use account status to check remaining credits before large runs.
Retry rate limits with backoff instead of immediately increasing pressure.
Keep one-off projects on pay-as-you-go when volume is uneven.
Move predictable recurring work to monthly credits when the pattern is proven.

Server room representing delivery paths, routing and network infrastructure

03 / Target delivery and network

Find delivery failures first

Many scraping platforms collapse delivery issues into a vague request failure. In production, that is not enough. The team needs to know whether the request timed out, the upstream target returned a temporary server error, the connection failed, the response was too large, redirects went somewhere unexpected, or the page came back as tiny HTML with no useful content. ScraperFly turns those failures into delivery evidence: try the light profile first, compare status and response size, then decide whether the next request needs backoff, another route, premium proxy, country targeting, or a narrower capture.

timeout5xxredirectsresponse_too_large

Retry temporary network and upstream failures with a controlled backoff policy.
Compare final URL, status, content type and response size across samples.
Escalate routing only when the response proves the simple path is unstable.
Treat oversized responses as a scope problem before scaling the same profile.

Digital security screen representing anti-bot and protected access signals

04 / Anti-bot, geo and access blocks

Read blocks before escalation

A target 403, 401, 429, CAPTCHA screen, WAF challenge, login wall, consent loop, or region-specific mismatch is not just an error; it is a clue about what the site expects from the visitor. Competitors often push teams toward a heavier default profile, but that can burn budget without proving the fix. ScraperFly's better path is evidence-based escalation: preserve realistic headers when context matters, use sessions when continuity changes the response, enable rendering when the useful content is browser-built, and choose premium or country routing when reputation or geography changes the result.

403429CAPTCHAcountry mismatch

Use premium routing when IP reputation is the proven bottleneck.
Use country targeting when language, currency, inventory or search results change by market.
Use headers, cookies and sessions when visitor state changes the response.
Reduce concurrency and add backoff when repeated requests trigger target throttling.

Developer preparing code and output artifacts on a laptop

05 / Rendering and output quality

Ship usable artifacts

A 200 status is not the same thing as a usable scrape. The raw HTML can be an empty app shell, Markdown can miss the main article, JSON can lack expected metadata, text can include noise, screenshots can be blank, and PDF or image output can fail if the browser step times out. ScraperFly is valuable because the same API request can move from raw HTML to rendered DOM, clean text, Markdown, JSON, screenshot or PDF, then be checked against the artifact your product actually needs. That reduces glue code, failed imports, and expensive batch runs that only discover bad output after the fact.

render_timeoutbrowser_render_failedJSON fieldsPDF/image output

Enable rendering when raw HTML is only a shell and the useful content appears after JavaScript.
Validate text, Markdown and JSON against expected fields before batching.
Use screenshot or PDF output when visual proof or audit evidence matters.
Treat blank or partial artifacts as quality failures even when HTTP status is successful.

ScraperFly fit

Cleaner error recovery

ScraperFly helps teams decide what to fix, what to retry, and what to escalate without turning every scrape into the most expensive request profile.

Stable error shape

Use HTTP status and error code together so application logic can choose between fixing input, refreshing credentials, slowing down, retrying, or escalating delivery.

Retry only when it helps

Network failures, rate limits, and temporary server errors can use backoff. Invalid URLs, unsupported formats, and bad keys should be corrected once.

Cost-aware escalation

Start with the light request profile, then add rendering, premium routing, country, sessions, headers, waits, or visual capture only when the failure proves it.

Validate output before scale

Confirm the real artifact your product needs: HTML, text, Markdown, JSON, screenshot, or PDF, before copying the same profile across a large URL set.

REQUEST WEIGHT

Keep browser work reserved for targets that prove they need it

Browser rendering, screenshots, PDFs, premium routing, and long waits are useful controls, but they add weight. ScraperFly helps teams keep source HTML as the default path and add heavier controls only when the response evidence justifies them.

A lightweight HTML request returning a simple response

Basic fetch

Use a basic fetch when public HTML already contains the fields you need.

Lightest request profile; keep it when the response is complete.

Rendering

Use rendering when raw HTML is a shell and JavaScript completes the page.

Heavier profile; use it only when browser output changes the result.

Several requests preserving cookies and visitor state

Stateful session

Use sessions when cookies, consent, locale, or pagination change the response.

Heavier state profile; preserve it only when continuity matters.

Request routing through geo and higher quality routes

Premium routing & geo

Use premium routing or geo only when origin, country, reputation, or regional content changes the result.

Heaviest routing profile; apply it only to targets that prove they need it.

What makes a request heavier

More infrastructure work

Browser rendering, routing choices, and longer waits mean the platform has to do more than return a simple response.

More target uncertainty

Redirects, empty states, rate limits, and challenge pages often require stronger delivery options or retries.

More state to preserve

Cookies, sessions, consent, and multi-page flows add moving parts that a plain stateless fetch does not have.

More evidence to return

Screenshots, PDFs, rendered output, and richer response artifacts are useful, but they are heavier than plain HTML.

How to keep request weight predictable

Sample before scale

Test a small URL set first and group targets by the lightest profile that returns complete data.

Do not render by default

Inspect raw HTML before turning on rendering. Many public pages already contain the data you need.

Escalate by evidence

Move up only when the response shows missing content, state dependency, geo variance, or routing problems.

Measure useful results

Optimize around valid records delivered, not raw request volume. A lighter profile is best only when it stays complete.

RENDERING FAQ

Common questions about JavaScript rendering and headless browser scraping

Practical answers for teams deciding when browser execution is worth using, how to choose rendered outputs, and how to avoid turning every public page scrape into a heavy request.

Rendering answers

Requests

Making API requests

For developers wiring ScraperFly into agents, jobs, backend services, or quick tests.

What do I send to ScraperFly?

Send a target URL plus the output format you need. The API can return HTML, clean text, Markdown, JSON, screenshots, or PDFs depending on the request options.

Should I use GET or POST?

Both can produce the same scraping result. GET is convenient for simple query parameters; POST is better when you want a clear JSON body or longer request options.

Can I test a request before coding?

Yes. The API Builder generates the endpoint, JSON body, query parameters, and code samples, then lets you send a live request from the dashboard.

Output

HTML, text, Markdown, and JSON

Choose the smallest output that your product actually needs.

When should I use clean text?

Use clean text when you need readable page content for summaries, classification, search indexing, or LLM prompts without layout markup.

What is the JSON output for?

JSON is useful when you want structured metadata, headings, links, text, and Markdown in one response that is easier to inspect programmatically.

Is Markdown different from clean text?

Markdown keeps more content structure, such as headings and lists. Clean text is flatter and better when formatting is less important.

Rendering

JavaScript pages

Use browser rendering only when the target page needs it.

When do I enable render=true?

Enable rendering for SPAs, JavaScript-injected product pages, lazy-loaded content, or pages where the source HTML does not contain the data you need.

Does rendering always cost more?

Yes, browser work is heavier than a basic HTML fetch. Start without rendering, then enable it only for targets that genuinely require JavaScript execution.

Can I control viewport and wait time?

Yes. Screenshot and PDF requests can use viewport dimensions and render wait options so the captured page has time to settle.

Routing

Proxies and location

Route requests through standard or premium pools when target behavior changes.

When do I need premium proxy routing?

Use premium routing for stricter targets, higher block rates, unstable responses, or pages that behave differently under normal datacenter traffic.

Can I request a country?

Yes. Add a country option when the target content depends on location, currency, language, availability, or regional redirects.

Are sessions supported?

Session-style workflows can reuse context across related requests when continuity matters for pagination, locale changes, or multi-step scraping.

Files

Screenshots and PDFs

Capture rendered pages when a visual artifact is the required output.

Can ScraperFly return a screenshot?

Yes. Screenshot output captures the rendered page as an image file, including JavaScript-rendered content when rendering is enabled.

Can ScraperFly generate a PDF?

Yes. PDF output is useful for receipts, reports, archives, compliance snapshots, and pages that need to be saved as a document.

Where do generated files appear?

Registered dashboard users can save and download generated artifacts from request logs when the request produces a downloadable file.

Credits

Costs and credits

Credits track delivery work rather than idle infrastructure.

How many credits does a request use?

A simple HTML, text, Markdown, or JSON request usually uses fewer credits. Rendering, premium routing, screenshots, and PDFs use more because they require heavier delivery work.

Do pay-as-you-go credits expire?

Pay-as-you-go credits stay available until you use them. They are a good fit for uneven workloads or projects that do not need a monthly plan.

Do subscription credits roll over?

Yes. Monthly plan credits refresh every month and unused credits roll over. If payment fails after Stripe retries, the account falls back to Pay as you go.

Security

Keys, headers, and safety

Keep credentials on your backend and forward only the headers you intend to use.

Where should I store my API key?

Store the API key server-side or in a trusted environment. Avoid exposing production keys in public frontend bundles or static pages.

Can I forward request headers?

Yes, selected headers such as user-agent, accept-language, cookie, and referer can be forwarded when a target expects realistic request context.

Does ScraperFly bypass private access controls?

No. It is designed for fetching public web pages and pages you are allowed to access, not for bypassing authentication or private systems.

Billing

Plans, logs, and invoices

Use the dashboard to understand what happened and why credits changed.

Where can I inspect usage?

The dashboard records request logs, credit balance changes, output formats, proxy routing, durations, and billing events so you can audit usage.

Can I download invoices?

Paid transactions include invoice data when Stripe provides it, so invoice PDFs can be opened from the financial log.

How do I manage a subscription?

Subscribed users can open the customer portal to change payment details, review invoices, or cancel. Free and Pay as you go users can go to Pricing instead.