Render with evidence
Use browser execution when it makes the response complete
ScraperFly helps teams compare source HTML with the rendered page state before committing a workflow to browser execution. That makes JavaScript rendering a precise capture option, not an expensive default for every URL.
Use it when the useful public content appears after client-side code runs, then return the result as rendered HTML, text, Markdown, metadata, screenshot, or PDF.
What is a JavaScript rendering API?
A JavaScript rendering API is a hosted request layer that can execute page JavaScript before returning the response. Instead of maintaining browser workers yourself, your application sends a public target URL and rendering options to the API. The API can open the page in a browser environment, wait for client-side content, and return rendered HTML, readable text, Markdown, JSON metadata, screenshots, or PDFs depending on the workflow.
This matters because many public websites no longer place every important field in the first HTML response. The source response may contain scripts, placeholders, hydration data, route containers, or empty elements that become useful only after the browser runs. Rendering closes that gap when the final page state is materially different from the source.
Practical takeaway: use JavaScript rendering when browser execution changes the data, page state, or visual evidence your workflow receives.
How can a team tell whether rendering is necessary?
The best first step is comparison. Request the same public URL without rendering, then request it with rendering enabled. Look at status code, final URL, response size, source HTML, rendered HTML, visible text, metadata, and screenshot state. If raw HTML already contains the required content, rendering may add cost and latency without adding value. If rendered output contains the fields your parser or review workflow needs, rendering has a clear purpose.
Teams get into trouble when they treat JavaScript rendering as a default instead of an evidence-based option. Browser execution is powerful, but it can mask simpler solutions. Some pages expose useful data in server-rendered markup, structured script payloads, canonical metadata, or static links. Others truly require rendering because the application builds the useful DOM after network calls, route hydration, user-visible state changes, or lazy loading.
| Signal | What it means | Best next action |
|---|---|---|
| Raw HTML contains the target data | The page is usable without browser execution | Keep the request simple and parse the source response. |
| Raw HTML is a thin shell | The page likely depends on client-side rendering | Test rendering and compare the rendered DOM. |
| Rendered output is still empty | The page may need wait timing, session state, headers, or a different route | Inspect evidence before retrying or escalating. |
| Screenshot shows useful content but text output does not | The visual state and machine-readable output disagree | Compare output format and extraction assumptions. |
| Only some targets need rendering | The workload has mixed request profiles | Classify targets instead of rendering the whole batch. |
Which JavaScript page patterns usually need rendering?
Rendering is most useful for public pages where the browser builds the content that users actually see. This can include single-page applications, product grids that hydrate after load, search result pages that update after frontend requests, infinite-scroll lists, pages with lazy sections, route-based content, or visual states that matter for QA. The page might return a valid response while the useful data arrives later through browser-side execution.
Not every JavaScript asset makes rendering necessary. A page can use JavaScript for analytics, tracking, navigation, layout effects, consent banners, or minor interactions while still delivering the important content in source HTML. The decision should be based on the captured result, not on whether the page has scripts.
| Page pattern | Why rendering may help | What to verify | Common mistake |
|---|---|---|---|
| SPA route | The visible page can be assembled after route hydration | Rendered title, body content, links, and state | Parsing the empty app shell. |
| Product or listing grid | Items may appear after frontend network activity | Card count, prices, availability, pagination signals | Stopping before async data is visible. |
| Lazy content | Sections can appear after time, viewport changes, or deferred scripts | Whether the required section appears in rendered output | Adding random waits without evidence. |
| Visual QA page | Humans need proof of how the rendered page looked | Screenshot or PDF state | Using visual files as the main parser input. |
| Mixed static and dynamic pages | Some URLs need the browser and others do not | Representative samples from each template | Rendering every URL in the batch. |
How should rendering fit into a production scraping workflow?
A production workflow should treat rendering as one request profile, not as the whole architecture. Start with raw HTML for a representative public URL. If it is incomplete, enable rendering. If rendering is useful, decide whether the workflow needs rendered HTML for DOM parsing, text or Markdown for content pipelines, JSON metadata for diagnostics, screenshots for visual monitoring, or PDFs for archive-style review.
From there, keep the request profile explicit. A page may need rendering but not screenshots. Another may need a screenshot but not Markdown. A third may need wait timing because content arrives late. Some targets may need session continuity, headers, country routing, or premium delivery. ScraperFly lets teams add those controls when the response proves they matter, which keeps the integration easier to reason about and cheaper to operate.
What should the API return after JavaScript rendering?
The output should match the downstream job. Rendered HTML is useful when a parser needs DOM structure, attributes, links, tables, or selector-based extraction. Clean text can support search, classification, deduplication, lightweight monitoring, or simple content analysis. Markdown is often useful for AI and RAG workflows because it preserves readable headings, lists, and links without carrying the full page layout. JSON metadata helps logging and diagnostics by exposing request context such as status, content type, final URL, and response details.
Screenshots and PDFs solve a different problem. They provide visual evidence for QA, monitoring, compliance review, archive workflows, or human approval. They are rarely the best source for machine parsing, but they are valuable when the question is not only “what text did we receive?” but also “what did the rendered public page look like at capture time?”
Practical takeaway: render the page when needed, then choose the smallest output format that your product can use directly.
What causes blank or partial rendered output?
Blank rendered output does not always mean the target cannot be scraped. It can mean the request reached the wrong final URL, content loaded after the wait window, a script failed, a consent or location state changed the view, the target returned a fallback, the response was too large, or the requested output format did not match the actual page state. Retrying blindly can repeat the same bad request and spend credits without learning anything new.
Good diagnostics compare evidence. Check the source response, rendered response, final URL, status, response size, content type, timing, screenshot state, and any routing or header assumptions. If the rendered screenshot shows the content but the text output does not, the issue may be output selection. If neither source nor rendered output shows the content, the page may require a different timing, session, route, or a narrower target decision.
How can teams control JavaScript rendering cost?
Rendering is heavier than a basic fetch because the request asks the platform to run browser work. That extra work is justified when JavaScript execution unlocks content, creates visual proof, or makes the response usable. It is wasteful when the same data is already available through a lighter request. Cost control begins with classification: simple pages stay simple, JavaScript-heavy pages render, and visual outputs are generated only for workflows that need them.
ScraperFly supports that buying path. Developers can test free or small runs, compare request profiles in API Builder, understand output choices through response formats, and use pricing to estimate request weight before scaling. For pages that clearly require browser execution, the headless browser scraping API path provides the broader browser-capture context.
Where should a developer start?
Start with one representative public page. Send the URL without rendering, inspect the result, then enable JavaScript rendering only if the source response is incomplete. Add wait timing only if the rendered page still misses late content. Choose rendered HTML, text, Markdown, JSON metadata, screenshot, or PDF based on what the downstream workflow needs. When the request profile is proven, copy it into cURL, Postman, Python, Node.js, your backend, an agent, or a monitoring job.
The goal is not to make every scrape a browser scrape. The goal is to make JavaScript rendering available exactly where it turns an incomplete public page response into useful data, cleaner content, or trustworthy visual evidence.
