Hosted browser capture
Get the rendered page state without owning browser infrastructure
ScraperFly gives teams a hosted browser layer for public pages that need real rendering. Capture the page after JavaScript has run, inspect the evidence, and choose the output your product can actually use.
Keep simple targets on lightweight HTML requests, then reserve browser execution, waits, screenshots, PDFs, sessions, or stronger delivery for pages where they make the result better.
When does scraping need a headless browser?
Scraping needs a headless browser when the useful public page content is created after the first HTML response. Many modern pages return a thin shell with script tags, then build products, articles, dashboards, prices, comments, route views, or search results in the browser. A basic HTTP fetch can return a successful status code and still miss the data your workflow expects.
The first job is not to launch a browser by habit. The first job is to prove where the content lives. If source HTML already contains the required fields, links, text, or tables, a raw HTML request is usually the better default. If the browser changes the DOM in a meaningful way, rendering becomes useful. If the browser displays the wrong state, a fallback page, or missing lazy content, waits, sessions, headers, or routing may need to be tested next.
Practical takeaway: use headless browser scraping for public pages where browser execution reveals useful content that a simple HTML request cannot capture.
What does ScraperFly replace in a browser scraping stack?
Running headless browsers in production is not just a Playwright or Puppeteer script. Teams have to schedule browser workers, control concurrency, handle timeouts, rotate delivery paths, manage memory, preserve cookies when needed, wait for late content, capture screenshots or PDFs, store logs, and decide when a failure deserves a retry. That infrastructure can become more complicated than the data workflow it was meant to support.
ScraperFly gives teams a hosted API layer for this capture work. Your application sends the public URL and a request profile. ScraperFly can fetch source HTML, render the page in a browser when needed, wait for content, return rendered HTML, convert the response to text or Markdown, expose JSON metadata, or produce screenshot and PDF outputs for review. Your product still owns parsing, validation, storage, alerts, and business logic, but it no longer has to operate browser infrastructure just to test whether a target page can be captured.
| Production concern | Own browser stack | ScraperFly API path | Decision value |
|---|---|---|---|
| Browser workers | Provision, update, restart, and monitor browser processes | Enable rendering as a request option | Use browser work without turning it into a platform project. |
| Late content | Write custom waits and timing rules per target | Test render and wait controls against real samples | Keep timing decisions tied to evidence from the page. |
| Output conversion | Build converters for HTML, text, screenshots, and PDFs | Choose the response format the workflow needs | Reduce glue code before downstream parsing or QA. |
| Failure recovery | Diagnose timeouts, blocks, redirects, shells, and blank captures manually | Use response evidence, status, size, final URL, and artifacts | Retry or escalate based on a clear failure signal. |
How should teams decide between raw HTML and rendered output?
The safer workflow is to compare before you commit. Start with source HTML because it is lighter, easier to inspect, and often enough for server-rendered pages, static pages, metadata, links, or tables. Then run a rendered request for representative targets where raw HTML looks empty, thin, or inconsistent with the page in a browser. Compare the body size, meaningful text, expected selectors, status code, final URL, content type, and any visual artifact.
This comparison prevents two common mistakes. The first mistake is rendering everything and paying browser cost even when a simple request already works. The second is refusing to render a JavaScript page and then building fragile downstream fixes around incomplete data. ScraperFly is useful because both paths stay in one API workflow: raw HTML for simple targets, browser rendering for pages that prove they need it.
| Page evidence | Best first action | Escalate when | Practical takeaway |
|---|---|---|---|
| Source HTML contains the data | Keep render disabled | The target starts hiding content or changing output | Do not pay browser cost when markup is already usable. |
| Source HTML is only an app shell | Enable browser rendering | Rendered output still misses expected content | Use the browser when JavaScript builds the useful state. |
| Rendered page loads content late | Add a measured wait | More samples prove timing is unstable | Wait for evidence, not for a guessed number of seconds. |
| Rendered HTML and screenshot disagree | Inspect final URL, status, and page state | The target returns a fallback or blocked view | Visual proof helps separate missing data from wrong page state. |
Which output should a headless browser request return?
Rendered HTML is the natural output when a parser needs DOM structure after JavaScript execution. It preserves elements, links, attributes, tables, and page relationships. Text is better when the downstream workflow needs compact readable content. Markdown is useful for AI, RAG, documentation, and content review because headings, lists, and links remain easier to consume. JSON metadata helps logging and debugging by exposing request evidence such as final URL, status, content type, and response details.
Screenshots and PDFs are not replacements for parsable content. They are evidence formats. Use them when a human needs to verify what the browser saw: a rendered product page, visual regression, compliance record, approval state, monitoring proof, or archived page view. They are heavier than text and HTML, so they should be requested when visual proof matters, not as the default for machine-only workflows.
Practical takeaway: choose rendered HTML for parsers, text or Markdown for content pipelines, metadata for diagnostics, and screenshots or PDFs when visual proof has real value.
What makes headless browser scraping fail?
Browser scraping can fail even when the API request itself is valid. The target can load slowly, redirect to a different page, return an empty shell, serve a challenge or fallback view, block automation, send a response that is too large, or never expose the expected content for that request context. Retrying blindly can make the same bad request more expensive without moving the workflow closer to a usable result.
The better path is diagnosis. Check whether the URL is correct, whether source HTML is thin, whether rendered output changed, whether the final URL is expected, whether the response size looks plausible, whether the screenshot proves the right page state, and whether headers, sessions, waits, country routing, or premium delivery are relevant. The error-handling block below expands that recovery model, but the main rule is simple: fix configuration errors, retry temporary failures with limits, and escalate delivery only when the evidence points there.
How does ScraperFly keep browser scraping cost under control?
Headless browser work is heavier than a simple HTML fetch because it runs a browser environment and may involve waits, rendered state, visual output, or stronger delivery. That extra work can be worth it, but only when it improves the captured result. ScraperFly keeps the cost model practical by letting teams start light, compare responses, and reserve rendering for public pages that prove a browser is necessary.
This matters most before scaling. A team can test representative pages, identify which targets need rendering, decide which ones need screenshots or PDFs, and keep simple pages on cheaper request profiles. That turns browser rendering from an all-or-nothing architecture choice into a controlled request decision.
Where should a developer start?
Start with the API Builder. Test a public URL with rendering off, inspect the response, then enable rendering if raw HTML does not contain the useful content. If the page needs browser execution, compare the output formats before writing converters. Use JavaScript rendering API guidance for broader rendering decisions, response formats for output selection, and pricing when estimating request weight before a larger run.
Once the request profile is proven, copy it into cURL, Postman, Python, Node.js, a backend worker, an agent, a monitoring job, or a parser pipeline. The goal is not to use a browser for every scrape. The goal is to use a browser exactly where it turns an incomplete public page response into usable evidence.
