Timing with proof
Wait only for content that changes the response
ScraperFly render_wait gives a browser-rendered request a short, controlled pause before capture. It is useful when the page starts correctly, but the useful content appears a moment later after JavaScript, route hydration, lazy loading, or a frontend data request finishes.
The goal is not to slow every scrape. The goal is to prove which pages benefit from a wait, keep easy targets on lighter requests, and return the response in the format your product can use immediately.
What is render_wait in a scraping API?
render_wait is a timing control for rendered scraping requests. After ScraperFly opens a public page with browser rendering, the wait gives the page additional time before the response is captured. That extra time can help when useful content arrives after the first browser pass rather than in the initial source HTML.
This is different from treating every timeout, empty response, or partial page as the same problem. A wait helps with timing. It does not fix a missing URL, invalid request, blocked target, exhausted credits, bad output choice, or a page that never exposes the expected content. The best use of render_wait starts with evidence: compare source HTML, rendered HTML, screenshot state, response size, final URL, and the fields your workflow actually needs.
Practical takeaway: use render_wait when a rendered request works, but captures the page before the useful public content has settled.
When should teams add a render wait?
Add a wait when the first rendered capture is close, but incomplete. Common examples include product cards that appear after frontend requests, search results that hydrate after route changes, comments or reviews that load late, charts that render after data fetches, pages that lazy-load important sections, or screenshots that need a stable visual state before capture.
The strongest signal is a before-and-after comparison. If raw HTML is incomplete and rendered output improves the page, rendering is justified. If a short wait makes the rendered output complete, render_wait is justified. If longer waits do not change the captured response, the problem is probably not timing. It may be routing, access state, request context, response size, output format, or target behavior.
| Observed result | Likely meaning | Best next action |
|---|---|---|
| Raw HTML already contains the data | The page does not need browser timing | Keep the request on the light HTML path. |
| Rendered output contains some data, but misses late sections | The page may need a short wait before capture | Test small wait values and compare completeness. |
| Screenshot shows loading placeholders | The capture happened before the visual state settled | Add a modest wait and verify the screenshot again. |
| Rendered output stays empty after waiting | The issue may not be timing | Inspect final URL, status, route, headers, session, and target response. |
| Only a few templates need wait time | The workload has mixed timing profiles | Apply wait selectively instead of to every URL. |
How is render_wait different from render timeout?
render_wait and render timeout solve different parts of the timing problem. render_wait is a deliberate pause before capture. It gives late content time to appear when the browser is already running successfully. A render timeout is the limit that stops a rendered request from running forever when the page is too slow, stalled, blocked, oversized, or never reaches a useful state.
Teams should not use a bigger timeout as the first answer to every slow page. A timeout asks the system to tolerate a longer request. A wait asks the system to capture after a specific delay. The practical workflow is to test whether a short wait improves the response, then keep the shortest value that reliably captures the useful page state. If the page still fails, move into timeout diagnosis rather than repeating the same heavy request.
For deeper timeout recovery, use the render timeout diagnosis page. For broader JavaScript execution decisions, start with the JavaScript rendering API guide.
How should teams choose the shortest useful wait?
The right value is the smallest delay that changes the result in a meaningful way. Test a representative page with no extra wait, then with small increments. Compare the fields that matter to the workflow: rendered HTML, clean text, Markdown, JSON metadata, screenshot, PDF, response size, and final URL. If a small wait completes the target content, keep it. If each larger value produces the same response, stop increasing it.
This keeps the decision practical. A crawler, agent, monitoring job, or data pipeline should not inherit a long wait because one page was slow during testing. Segment targets by template or behavior. Static pages stay light. JavaScript pages render. Pages with late public content get a short wait. Hard failures go through request evidence before routing or retry escalation.
| Wait choice | Good fit | Risk | Decision rule |
|---|---|---|---|
| No extra wait | Content appears immediately after rendering | Late sections may be missed | Use as the baseline for every target sample. |
| Short wait | Lazy lists, route hydration, small async updates | Slightly higher latency | Use when it visibly improves output completeness. |
| Longer wait | Known slow public pages with proven late content | Higher request weight and slower batches | Use only for templates that consistently need it. |
| Retry without diagnosis | Rarely a good production default | Repeats the same incomplete capture | Inspect timing and response evidence first. |
| Routing escalation | Delivery or access evidence points beyond timing | Can add cost without fixing late content | Escalate only after timing evidence is clear. |
Which output formats benefit from waiting?
Waiting can improve any output that depends on the final rendered page state. Rendered HTML can include nodes that did not exist at first capture. Text and Markdown can become cleaner when headings, cards, reviews, or article sections arrive after JavaScript runs. JSON metadata can show whether the request finished with the expected status, final URL, response type, and useful response size. Screenshots and PDFs can prove whether the visible page had settled before capture.
The output still needs to match the downstream job. A screenshot is helpful for QA, monitoring, and visual proof, but it is not usually the best parser input. Rendered HTML is useful for DOM structure. Text is useful for lightweight search or classification. Markdown is useful for AI and RAG workflows when readable headings and links matter. The response formats page helps choose the smallest useful result.
What mistakes make render_wait expensive?
The most common mistake is adding a generous wait to every rendered request because one sample arrived late. That makes fast pages slower, increases request weight, and hides the real differences between page templates. Another mistake is using wait time to mask unrelated problems. A blocked page, invalid URL, unsupported output, exhausted credits, response-too-large issue, or wrong final route will not become healthy just because the browser waited longer.
A better pattern is to classify request profiles. Use no wait for pages that complete quickly. Use a short wait for templates that consistently load useful public content late. Use timeout diagnosis when rendering never finishes. Use sessions, headers, country routing, or stronger delivery only when the evidence points there. That keeps scraping recovery specific instead of turning every difficult page into the most expensive request.
How does ScraperFly keep render waits practical?
ScraperFly gives teams an API-first way to test timing before production code depends on it. Start with a target URL, compare raw and rendered responses, add render_wait only when it changes the result, then choose HTML, text, Markdown, JSON, screenshot, or PDF based on the workflow. The same request profile can move into cURL, Postman, Python, Node.js, a backend service, an agent, or a monitoring pipeline.
This is useful for teams that want reliable public web data without maintaining their own browser fleet, custom retry logic, screenshot workers, PDF capture, proxy routing, and response converters for every edge case. The buying path stays simple too: test the page, prove the timing, estimate request weight with pricing and the credit cost calculator, then scale only the profile that has evidence behind it.
