Next.js capture workflow
Choose raw HTML or rendered capture based on how the route is built
Next.js can serve a complete page before the browser runs, or it can send a light shell that becomes useful only after hydration and client-side data loading. ScraperFly gives teams a practical way to tell the difference before parser logic, retries, and browser capacity become part of the production workflow.
Start with the public source response, inspect whether the target content is already there, then compare rendered output when the page proves it needs JavaScript execution. That keeps static and server-rendered routes fast while still supporting client-built pages that need a browser to become complete.
Why do Next.js pages need a capture decision first?
Next.js is not one rendering pattern. A public route may be server-rendered for every request, statically generated at build time, updated through incremental regeneration, or assembled in the browser from client components and data calls. Two pages on the same site can behave differently, so a single default scraping profile often wastes time or misses content.
The useful question is not whether the site uses Next.js. The useful question is where the data appears. If the source response already contains product cards, article text, prices, links, metadata, pagination, or listings, a lightweight HTML request is often enough. If the first response contains a route shell, placeholders, scripts, or partial state, rendered capture may be the cleanest way to collect the page your users actually see.
Practical takeaway: treat Next.js scraping as a response inspection problem first, then choose the smallest request profile that returns the needed content.
When is source HTML enough for a Next.js scrape?
Source HTML is enough when the initial response includes the fields your application will parse. That is common on server-rendered pages, static marketing pages, documentation pages, article routes, category pages, and public listings where content is placed into the HTML for speed, SEO, or reliability. In those cases, adding a browser can make the request slower and heavier without improving the result.
A source-first workflow also makes debugging easier. Your team can inspect the final URL, status, content type, response size, and markup before adding waits or browser execution. If the required selectors, links, headings, tables, or embedded metadata are already present, keep the route on raw HTML and reserve heavier controls for the targets that prove they need them.
| Next.js route signal | What it usually means | Best ScraperFly profile |
|---|---|---|
| Useful content appears in source HTML | The page is likely server-rendered or static enough for direct parsing | Use raw HTML and keep the request lightweight. |
| Source HTML has a shell, scripts, or placeholders | The visible content may be created after hydration | Compare rendered HTML before building parser logic. |
| Rendered output contains the missing fields | Browser execution changes the scrape result | Use rendering for this route pattern and log why it is needed. |
| Content changes by locale, cookie, or previous state | The route may depend on request context | Add headers, cookies, or sessions only where that context matters. |
| Rendered output is still empty or blocked | The issue may be routing, timing, access rules, or target behavior | Inspect evidence before retrying or escalating delivery. |
When should Next.js scraping use browser rendering?
Use rendering when the useful public content appears only after browser-side code runs. That can happen on hydrated application routes, interactive product grids, search results, infinite lists, review widgets, dynamic pricing blocks, maps, dashboards with public data, or pages where the initial HTML mostly points the browser toward JavaScript bundles.
Rendering should be a targeted control, not a reflex. Compare the source response with the rendered state and look for a real content difference. If rendering adds the missing fields, it is worth using for that route class. If rendering does not improve the output, the next step is not blind retries; inspect timing, redirects, status, resource behavior, response size, and whether the public page actually exposes the data.
For deeper browser decisions, compare this page with the JavaScript rendering API, the headless browser scraping API, and the rendered HTML API guides.
Which output format should a Next.js workflow request?
The right format depends on what the downstream system will do with the page. HTML is best when selectors, DOM relationships, links, attributes, tables, or embedded state matter. Text and Markdown are better when the output feeds search, summaries, classification, content review, or AI pipelines. JSON metadata helps with diagnostics because it preserves request evidence such as status, final URL, response size, and content type. Screenshots and PDFs are useful when a human needs visual proof of what the rendered page showed.
| Output format | Use it for | Next.js advantage |
|---|---|---|
| Raw HTML | Server-rendered or static routes | Fast, parser-friendly, and usually the lowest-weight capture. |
| Rendered HTML | Hydrated routes and browser-built content | Captures the DOM after JavaScript changes the page. |
| Text | Search, monitoring, lightweight analysis | Removes layout noise when structure is not required. |
| Markdown | AI, RAG, documentation, content review | Keeps readable headings, links, and sections for language-model workflows. |
| JSON metadata | Logging and routing decisions | Makes request outcomes easier to compare across route samples. |
| Screenshot or PDF | QA, compliance review, visual monitoring, archives | Shows what the public page looked like after capture. |
How should teams test Next.js routes before production?
Test a small set of representative routes first. Include a static route, a server-rendered route, a dynamic route, and a route that appears client-heavy. Send each target through the API Builder, compare source HTML with rendered output, and choose the smallest format that your parser or application can use directly.
That test set becomes your request playbook. Simple routes can stay on raw HTML. Client-built routes can use rendered HTML or a cleaner text and Markdown output. Routes that need continuity can add sessions. Routes that depend on language, region, or routing context can add headers, cookies, country targeting, or stronger delivery only when the evidence supports it.
Practical takeaway: build the Next.js scraping profile from route evidence, not from a site-wide assumption about the framework.
How does ScraperFly keep Next.js scraping affordable?
Cost control starts by avoiding unnecessary browser work. A page that returns complete server-rendered HTML should not carry the same request weight as a hydrated route with waits, visual output, session context, and stronger routing. ScraperFly makes that decision easier because the same API path can start light and add controls only when the target proves they change the result.
Use pricing as an engineering feedback loop. Classify route patterns, keep raw HTML as the default for complete responses, reserve rendering for client-side content, and generate screenshots or PDFs only when visual proof is part of the workflow. The pricing page and web scraping API credit cost calculator help teams estimate how those choices affect volume before they scale.
How does Next.js scraping compare with React and SPA scraping?
Next.js often sits between classic HTML scraping and full SPA scraping. It can behave like a server-rendered site, a static site, a hydrated React app, or a client-heavy single-page application depending on the route. That makes it important to test the actual public URL rather than treating every Next.js page as a browser-only target.
If the site behaves like a React application, the React scraping API guide is a useful comparison. If navigation and content depend heavily on browser-side routing, review the SPA scraping API workflow. If the target gives you complete source markup, keep the capture path closer to the HTML scraping API approach and save the browser for pages that truly need it.
