Product evidence
Capture the product page before your catalog logic maps fields
Product data projects do not fail only because a parser missed a selector. They fail when the capture layer returns a partial page, a regional variant, a blocked response, an empty JavaScript shell, or a visual state that nobody checked before scaling. ScraperFly gives product, ecommerce, and data teams a cleaner first step: capture the public product page in the format your downstream system can actually inspect.
Use source HTML when the page already contains the useful fields, rendered HTML when client-side code builds the final product state, text or Markdown when humans and AI workflows need readable content, JSON metadata when request evidence matters, and screenshots or PDFs when QA needs visual proof. Your catalog logic still owns field mapping, normalization, SKU matching, deduplication, and taxonomy decisions.
What is a product data extraction API?
A product data extraction API is a request layer for capturing public product pages so another system can turn page evidence into structured product records. The capture step may return raw HTML, rendered HTML, clean text, Markdown, response metadata, screenshots, or PDFs. The downstream extraction step then maps that evidence into titles, brands, SKUs, prices, availability, variants, images, descriptions, specifications, seller details, reviews, breadcrumbs, and category signals.
This distinction matters. ScraperFly is not positioned here as a magic product catalog normalizer. It gives your pipeline the page-level evidence needed to build reliable extraction rules and review failures. That is often the missing layer between a brittle scraper and a maintainable product data workflow.
Practical takeaway: treat ScraperFly as the capture and evidence layer, then let your product data system handle field mapping, normalization, and catalog rules.
Which product fields should teams capture?
Product pages rarely expose every field in one neat place. A title may be visible in markup, a price may be rendered after JavaScript, variants may sit inside embedded data, stock may depend on region, and image galleries may behave differently across templates. Before writing extraction logic, define which fields matter and which output gives the best evidence for each field group.
| Field group | Examples | Why it matters | Output to consider |
|---|---|---|---|
| Product identity | Title, brand, SKU, model, GTIN, canonical URL | These fields anchor matching, deduplication, enrichment, and catalog updates. | HTML, rendered HTML, JSON metadata |
| Commercial state | Price, currency, discount, stock, shipping, seller | These fields often change by region, session, variant, or page template. | Rendered HTML, screenshot, metadata |
| Variants | Size, color, bundle, pack count, selected option | Variant logic decides whether one URL represents one product or many records. | HTML, rendered HTML, screenshot |
| Content | Description, bullets, specifications, ingredients, dimensions | Readable content feeds search, enrichment, comparison, and QA workflows. | Markdown, text, HTML |
| Visual proof | Image gallery, page layout, badges, visible price, availability label | Human review needs proof when fields are disputed or visually presented. | Screenshot, PDF |
Why do product pages need source validation?
Product extraction becomes expensive when teams write parser rules before they understand the page source. A public product page can change by category, marketplace seller, availability, cookie state, device layout, language, region, or frontend framework. It can also return a 200 status with a partial page, a consent screen, a temporarily empty product shell, or a template that looks similar but stores the important fields somewhere else.
Source validation keeps that risk visible. Capture a representative set of product URLs before scaling. Compare raw and rendered output. Check final URL, content type, response size, and whether the target fields are present. If a field is missing, decide whether the capture is incomplete, the target variant is unavailable, the region changed, or the parser is looking in the wrong place.
That makes the next step practical instead of expensive. You can add JavaScript rendering only for product templates that need browser execution, keep simple pages on HTML scraping, and reserve visual files for QA rather than generating screenshots for every machine-only request.
How should teams choose the output format?
The right format depends on who consumes the product evidence. A selector-based parser needs structure. A content enrichment workflow may prefer Markdown. A QA analyst may need screenshots. A monitoring system may need metadata, final URLs, response size, and status evidence. Using one format for every product source usually creates extra cleanup work somewhere else in the pipeline.
| Downstream need | Best capture format | Why it works | Watch out for |
|---|---|---|---|
| Field extraction with selectors | Raw or rendered HTML | Preserves DOM structure, attributes, links, tables, and embedded data. | Do not render if source HTML already has the fields. |
| Product descriptions and bullets | Text or Markdown | Reduces layout noise and gives readable content for review or AI workflows. | May remove DOM relationships needed for precise extraction. |
| Request diagnostics | JSON metadata | Shows operational evidence such as status, final URL, content type, and response context. | Metadata does not replace product-specific field extraction. |
| QA and visual confirmation | Screenshot or PDF | Preserves how the product page appeared for human review or audit records. | Visual files are heavier and not ideal as parser inputs. |
| Catalog enrichment | HTML plus Markdown or text | Combines structured page evidence with readable copy for downstream processing. | Keep the workflow clear so each format has a defined consumer. |
When should product extraction use rendering, sessions, or routing?
Start with the lightest request that returns complete product evidence. Add rendering when product fields appear only after browser execution. Add waits when the rendered page needs time for important sections to load. Add headers when language, content type, user agent, or other allowed request context changes the response. Add sessions when related requests depend on shared state, such as variant selection or continuity across a product flow.
Use country targeting when product details vary by location, currency, language, availability, regional redirects, or local seller behavior. Use stronger routing when standard delivery repeatedly returns blocked, incomplete, or unstable responses. Each control should answer a specific observation from the target, not become a global default.
For related request-control decisions, review custom headers, sessions, and country targeting. The goal is to keep straightforward product pages affordable while giving harder sources a deliberate upgrade path.
What should stay downstream from ScraperFly?
Product extraction does not end at capture. Your downstream system still needs to decide how to normalize names, match SKUs, merge variants, resolve duplicate products, map categories, validate currencies, handle units, rank confidence, store media, and reconcile conflicting seller data. Those are business rules, not just scraping controls.
Keeping that boundary clear makes the page easier to operate. ScraperFly retrieves public product pages and returns evidence in useful formats. Your parser, catalog enrichment layer, PIM, monitoring system, or data warehouse then turns that evidence into the product model your business needs. That separation avoids overpromising and makes failures easier to debug: was the page captured incorrectly, or did the downstream extraction rule need adjustment?
Where should a product team start?
Start with a small sample set: one simple product page, one variant-heavy page, one out-of-stock page, one category-specific template, and one source that may require rendering or location context. Test each URL in the API Builder, compare output formats, and decide which request profile produces complete evidence with the least overhead.
Once the capture profile is proven, scale in layers. Keep raw HTML for pages that expose fields in source markup, add rendered HTML only where JavaScript changes the result, and use screenshots or PDFs only where human review needs visual proof. If the workflow includes price or availability changes over time, connect this page with price monitoring and review pricing before turning small tests into recurring production jobs.
