Source-first capture
Capture the markup first, then decide what the page really needs
ScraperFly gives teams a clean way to capture public HTML without turning every target into a browser job. Fetch the source response, see whether the useful markup is already there, and keep the request lightweight when the page does not need extra controls.
When a target does need more help, the same API path can add rendering, waits, headers, sessions, country routing, or stronger delivery with a clear reason behind each choice.
What is an HTML scraping API?
An HTML scraping API is a hosted endpoint that retrieves public page markup and returns it to your application through a stable API response. Your product can then parse links, headings, tables, metadata, product cards, article content, pagination, embedded data, or any other fields that are visible in the captured HTML. The API handles the request layer so your team can focus on the extraction logic and the downstream workflow.
The important distinction is that HTML scraping is not automatically the same as browser scraping. Many public pages send useful markup in the initial response. In those cases, a simple HTML request is faster, easier to debug, and usually cheaper than launching a browser. Other pages send a thin shell and rely on JavaScript to build the final DOM. Those pages may still be captured as HTML, but they need rendered HTML rather than raw source HTML.
Practical takeaway: use raw HTML first when the markup already contains the content, and move to rendered HTML only when JavaScript changes the captured page.
When is raw HTML enough?
Raw HTML is enough when the first response includes the information your parser needs. This is common for documentation pages, many articles, simple directories, static landing pages, server-rendered product pages, category listings, public profile pages, and pages that expose useful metadata or structured markup without browser execution. The request stays small, the output is easy to inspect, and the parsing rules can be built against a repeatable source response.
The best way to confirm this is to sample real target URLs. Check whether the title, main content, links, prices, dates, breadcrumbs, pagination, canonical URL, schema snippets, or embedded data are present in source HTML. If they are, adding a browser step may only increase latency and request weight. If they are missing, compare rendered output before assuming the page is impossible or jumping to heavier routing.
| HTML signal | What it tells you | Recommended request profile |
|---|---|---|
| Main content is present in source HTML | The page can likely be parsed without a browser | Use format=html with rendering off. |
| Source contains only an app shell | The visible content may be built by JavaScript | Test rendered HTML before writing selectors. |
| Links and pagination exist in markup | Crawling and discovery can start from raw HTML | Keep the discovery request light. |
| Markup changes by region or session | The target may need request context | Test headers, cookies, country routing, or sessions deliberately. |
| HTML is complete but hard to read | The parser may not need the full markup | Consider text, Markdown, or structured metadata for downstream jobs. |
How does ScraperFly make HTML capture easier to operate?
Teams can fetch HTML with a script, but production scraping becomes harder once targets behave differently. One page may be static. Another may need JavaScript. A third may require a language header, cookie state, country-specific routing, or stronger delivery. A fourth may return the correct status code but the wrong content. ScraperFly gives developers one request surface where those controls can be added only when the target proves they matter.
This keeps HTML scraping from turning into a patchwork of custom fetchers, proxy experiments, browser workers, retry loops, and output converters. The workflow stays understandable: send a public URL, choose HTML output, inspect the evidence, then decide whether raw source, rendered HTML, or a different response format is the right fit. When the request profile is proven, copy it into cURL, Postman, Python, Node.js, a backend job, a parser pipeline, or an agent workflow.
When should HTML scraping use JavaScript rendering?
Use JavaScript rendering when the useful HTML appears only after browser execution. That can happen on single-page applications, client-rendered product grids, search pages with frontend data loading, lazy sections, route-based interfaces, or pages where the initial response contains placeholders instead of the final content. In those cases, rendered HTML gives your parser the DOM after the browser has done the work.
Rendering should still be a measured choice. Start with raw HTML, compare it to rendered HTML, and enable rendering only when the difference matters. If rendered HTML gives you the same content as source HTML, keep the cheaper and simpler request. If rendered HTML reveals the target data, keep rendering for that page type and document why it is needed.
For deeper browser-specific decisions, use the JavaScript rendering API guide or the broader headless browser scraping API page. For this page, the core principle stays the same: HTML capture should be as simple as the target allows.
What should developers parse from HTML?
HTML is useful because it preserves structure. A parser can use selectors, attributes, links, tables, headings, microdata, embedded JSON, canonical tags, Open Graph metadata, pagination controls, and surrounding context. This is valuable when the downstream system needs more than plain readable text. For example, a price monitoring workflow may need product title, current price, currency, availability, and variant links. A content workflow may need headings, author, date, body sections, and internal links.
The cleanest parser is built after looking at real samples. Inspect stable elements, avoid brittle layout-only selectors, watch for repeated boilerplate, and separate page discovery from field extraction. If the target has multiple templates, sample each template before scaling. If the source HTML includes too much layout noise for an AI pipeline, consider whether text, Markdown, JSON metadata, screenshot, or PDF output is a better fit for that downstream step.
| Workflow | Useful HTML evidence | Why HTML helps | When to switch format |
|---|---|---|---|
| Link discovery | Anchors, canonical links, pagination, navigation sections | HTML keeps URLs and page relationships visible | Use text only after discovery is complete. |
| Product monitoring | Product cards, price nodes, availability, variants | Selectors can target repeatable fields | Use screenshots for human QA or visual proof. |
| Article extraction | Headings, body sections, author, date, metadata | Structure helps separate content from boilerplate | Use Markdown when readable structure is enough. |
| Monitoring | Status, final URL, response size, DOM changes | HTML can be compared over time | Use screenshots when visual regressions matter. |
| AI ingestion | Source sections, links, headings, raw context | HTML preserves maximum context before cleanup | Use clean text or Markdown when the model does not need DOM details. |
What mistakes make HTML scraping more expensive than it needs to be?
The most common mistake is treating every page as a hard target before the evidence says so. Rendering every URL, routing every request through premium delivery, adding long waits by default, or generating visual files for machine-only workflows can make a simple HTML job heavier than necessary. Another mistake is assuming that a successful HTTP status means the useful content is present. The response needs to be inspected, not just counted.
Good HTML scraping is deliberate. Start with the lightest request. Confirm whether source HTML contains the target data. Upgrade only the control that solves the observed problem. Use rendering for JavaScript-built content, sessions for shared state, headers for required request context, country routing for regional output, and premium delivery for targets that repeatedly fail through standard paths. That approach keeps request cost aligned with target behavior instead of guesswork.
Practical takeaway: the cheapest reliable HTML scrape is the one that returns complete markup with the fewest enabled controls.
How should teams start with ScraperFly?
Start in the API Builder. Send one public URL as HTML with rendering off. Inspect the source response and verify whether the fields are present. If the page is incomplete, enable rendering and compare the rendered HTML. If geography, cookies, headers, waits, or delivery strength change the output, add those controls one at a time so the request remains understandable.
Once the request profile is proven, use it as the template for production. Keep static pages on raw HTML, classify JavaScript-heavy templates separately, and reserve heavier options for the pages that truly need them. When planning larger runs, review pricing so request profiles match the workload before scaling.
