ScraperFlyAI-ready Web Data API

use cases

Web Scraping API use cases for real business workflows

Explore the public web data workflows ScraperFly helps teams ship: monitor prices, track search results, collect product pages, support AI/RAG, follow market signals, and capture listings with the right request profile.

ScraperFly use cases start with a practical question: what decision, product feature, report, or workflow needs public web data? Once that is clear, the request can stay lean. Fetch the target page, inspect the response, choose the smallest useful output format, and add rendering, sessions, headers, country routing, premium delivery, screenshots, or PDFs only when the target proves they improve the result.

This keeps early validation fast and affordable. A price monitoring job, a SERP tracking workflow, an AI content pipeline, a real estate listing feed, and a market research dataset may all use the same API surface, but they should not all use the same request weight.

Choose the use case by the decision it supports

A strong scraping workflow is not just a URL fetch. It has a business reason, a target source, a response format, and a repeatable way to decide when the request needs extra controls. That is why use-case pages should be grouped by workflow intent instead of by isolated features.

Ecommerce teams usually care about product pages, prices, availability, reviews, and category changes. Search teams care about SERP position, visibility, local packs, and competitor movement. AI and RAG teams need readable content with enough source context to trust the answer. Research teams need clean samples, consistent metadata, and a way to compare sources without rebuilding infrastructure for every site.

Use case family Typical source Best starting output Controls to add only when needed
Ecommerce and pricing Product pages, category pages, marketplace listings HTML or structured metadata for parsing product facts Rendering, sessions, country routing, screenshots for proof
SERP and visibility tracking Search result pages and local result views HTML or screenshot depending on the evidence required Country routing, headers, waits, careful retry rules
AI, RAG, and content workflows Articles, documentation, help pages, public resources Text, Markdown, or HTML depending on parser needs Rendering when content is browser-built, metadata for trust
Listings and market research Directories, real estate pages, job boards, local business pages HTML for extraction, text or Markdown for review Sessions, language headers, country routing, PDF or screenshot capture

Use one API path, but do not use one request profile everywhere

The advantage of a web scraping API is not that every workflow becomes identical. The advantage is that every workflow can be tested through the same controlled path. A team can begin with the lightest request, review status, final URL, response size, content type, and captured output, then decide whether the next request needs browser rendering, stronger routing, persistent state, or a different format.

That matters commercially. A small exploratory job should not be forced into the same setup as a production monitoring system. A one-off research sprint should not require a long subscription decision before the target has even been validated. ScraperFly is built for that middle ground: start free, buy credits when project work appears, and move toward monthly credits when the workflow becomes predictable.

Practical takeaway: pick the use case first, then pick the request profile. The right workflow is the one that returns usable evidence with the least extra infrastructure and the fewest unnecessary controls.

USE CASE ATLAS

Match the workflow to the public data your team actually needs

Use cases become easier to plan when the team names the source, the fields, the output format, and the decision the data will support. ScraperFly gives each workflow the same API foundation while leaving room to keep simple targets light and escalate only when the page behavior proves it.

Commerce

Turn public commerce pages into pricing and inventory intelligence

Track competitor prices, stock changes, sellers, marketplace offers, travel rates, and catalog shifts before they affect revenue.

Laptop with analytics charts for price monitoring
Commerce

E-commerce price monitoring

Watch competitor prices, discounts, stock status, shipping terms, and seller movement so pricing teams can react before market changes become invisible.

Useful fields: price, discount, stock, seller, shipping

Person browsing products on a smartphone
Commerce

Product catalog enrichment

Fill missing titles, descriptions, specifications, images, variants, and identifiers from public product pages and manufacturer catalogs.

Useful fields: title, description, specs, images, SKU

Large warehouse aisle representing marketplace operations
Commerce

Marketplace seller intelligence

Compare seller presence, fulfillment models, assortment depth, ratings, and offer changes across marketplaces.

Useful fields: seller, rating, offer, fulfillment, assortment

Warehouse worker organizing inventory shelves
Commerce

Inventory and availability tracking

Detect stockouts, pickup availability, regional inventory, delivery promises, and product availability gaps before customers do.

Useful fields: stock, region, pickup, delivery, timestamp

Person with shopping bags checking a smartphone
Commerce

Brand protection

Find unauthorized sellers, suspicious pricing, counterfeit signals, incorrect product claims, and risky listings in public marketplaces.

Useful fields: seller, price, claims, images, marketplace

Busy outdoor restaurant and hospitality scene
Commerce

Travel and hospitality monitoring

Collect room availability, rates, amenities, cancellation terms, local demand signals, and review context across booking surfaces.

Useful fields: rate, availability, amenities, policy, rating

Phone showing an online shopping interface
Commerce

Classifieds and secondhand tracking

Monitor new listings, asking prices, seller activity, item condition, location, and posting age across resale markets.

Useful fields: listing, price, condition, seller, location

Laptop showing financial graphs and analytics
Commerce

Price comparison and repricing

Turn competitor price snapshots into repricing inputs, assortment gaps, margin alerts, and market-position reports.

Useful fields: competitor, price, rank, availability, change

Growth

Find demand signals before they show up in your own analytics

Monitor search results, ads, competitor pages, business directories, hiring activity, and lead signals to see where attention is moving.

Laptop displaying analytics dashboard for SEO monitoring
Growth

SERP and SEO monitoring

Track ranks, snippets, ads, local packs, language, device, and geo differences to see how visibility changes in the real search surface.

Useful fields: rank, URL, snippet, ads, geo

Laptop open on a news or content page
Growth

Competitor content monitoring

Watch page changes, headings, CTAs, schema, internal links, and content inventory without waiting for manual audits.

Useful fields: heading, CTA, schema, links, updated text

Hands holding a phone with social media content
Growth

Ad intelligence

Collect public ad copy, creative patterns, landing pages, regions, timing, and advertiser activity for campaign research.

Useful fields: copy, creative, landing page, region, date

Business people exchanging documents in an office
Growth

Lead generation and company enrichment

Build account lists and enrich company records with domains, locations, industries, public contacts, hiring signals, and technology hints.

Useful fields: company, domain, industry, location, signals

Small local coffee shop storefront
Growth

Local business directory data

Collect names, categories, addresses, hours, websites, ratings, and review counts from public directory pages.

Useful fields: name, category, address, hours, rating

Office workers reviewing professional documents
Growth

Job market intelligence

Track job titles, skills, salary ranges, seniority, remote policies, and hiring velocity across career pages and job boards.

Useful fields: title, salary, skills, location, seniority

Voice

Turn public conversations into product and reputation signals

Collect reviews, media mentions, community discussions, public posts, and creator signals to understand what customers and markets are saying.

Person reading a newspaper while working on a laptop
Voice

News and media monitoring

Collect articles, authors, dates, topics, entities, links, and source metadata for reputation, research, and alerting workflows.

Useful fields: title, author, date, source, entities

People shopping and checking a smartphone together
Voice

Review and sentiment tracking

Capture rating trends, complaint themes, praise patterns, business responses, and product-location context from public reviews.

Useful fields: rating, review, theme, response, date

Person browsing a social media app on a smartphone
Voice

Social and public web monitoring

Monitor public posts, engagement, hashtags, timestamps, and profile context where policy and source access allow it.

Useful fields: post, engagement, hashtag, profile, time

Person reading news articles in a room
Voice

Community and forum intelligence

Find recurring questions, complaints, feature requests, and problem language from public communities and support forums.

Useful fields: question, topic, complaint, reply count, date

Workspace with laptop, coffee and newspaper
Voice

Creator and influencer discovery

Build discovery lists from public profile pages, content topics, audience hints, posting cadence, and collaboration signals.

Useful fields: profile, topic, audience signal, cadence, link

Records

Make public records and listings easier to monitor at scale

Extract tenders, legal notices, filings, financial data, and real estate listings into sourced datasets that teams can search, compare, and review.

Professionals exchanging business documents
Records

Public procurement and tenders

Track buyers, budgets, deadlines, requirements, attachments, awards, and changes across tender portals and public notices.

Useful fields: buyer, budget, deadline, requirements, award

Professionals discussing legal documents in an office
Records

Regulatory and legal monitoring

Monitor notices, public registers, effective dates, document versions, affected entities, and compliance-relevant changes.

Useful fields: notice, entity, date, document, status

Professionals reviewing financial documents
Records

Financial and public market research

Collect filings, disclosures, tables, news, instrument metadata, and public financial context for research pipelines.

Useful fields: filing, table, issuer, date, metric

House keys and cash for real estate market data
Records

Real estate listings and market data

Track listing prices, addresses, property attributes, agent details, status, history, and neighborhood context.

Useful fields: price, address, beds, agent, status

Data products

Build data products from scraped pages, not page dumps

Clean, normalize, validate, enrich, compare, and deliver public web data into databases, dashboards, alerts, and LLM pipelines.

Laptop screen with programming code
Data products

LLM-ready web data pipelines

Convert public pages into clean text, Markdown, chunks, metadata, citations, and freshness signals for retrieval or model workflows.

Useful fields: clean text, Markdown, chunk, source URL, timestamp

Data center server racks for database workflows
Data products

Structured extraction to database

Turn HTML, PDFs, tables, and page text into typed records that can be stored, queried, joined, and audited later.

Useful fields: JSON, type, source, schema, record ID

Laptop with coding and data analysis interface
Data products

Change detection and alerting

Compare snapshots and alert teams when prices, stock, rank, content, status, or regulatory language changes.

Useful fields: previous value, new value, delta, threshold, alert

Computer screen with colorful code and data
Data products

Entity enrichment and normalization

Normalize product IDs, company names, locations, people, categories, units, currencies, and taxonomy mappings.

Useful fields: entity, canonical ID, category, unit, confidence

Close-up of server rack with blue indicators
Data products

Data quality validation

Check required fields, data types, ranges, duplicates, source evidence, and suspicious records before downstream use.

Useful fields: required field, type, range, evidence, confidence

Laptop with code editor for data reporting feed
Data products

Reporting and dashboard feeds

Feed BI dashboards, market reports, operating scorecards, and recurring analyses with web data that refreshes on schedule.

Useful fields: metric, dimension, period, source, refresh

Why scraping

Why use web scraping at all?

Public web data is valuable because it reflects what customers, competitors, markets, and institutions actually publish. Scraping turns those pages into a repeatable feed when a public API is missing, incomplete, delayed, or too narrow for the workflow.

  • Fill gaps left by APIsMany sources expose useful public pages but no complete API for the exact fields, geos, filters, or histories a team needs.
  • Monitor change over timeRecurring scrapes create a history of prices, content, availability, ranks, reviews, notices, and market movement.
  • Compare across sourcesScraping makes cross-site comparison possible when marketplaces, directories, portals, and publishers structure data differently.
  • Create usable recordsThe useful output is not a page dump; it is a sourced, timestamped, normalized record that downstream systems can trust.
ScraperFly fit

Why ScraperFly fits

One API layer for delivery, rendering, routing, evidence, and output choices across varied public web sources.

One API for varied sources

Use the same integration pattern across stores, search pages, directories, documents, public listings, and data-product pipelines.

Delivery controls when needed

Rendering, sessions, headers, geo, and proxy routing can be added for targets that need them without rebuilding the workflow.

Output that supports post-processing

HTML, screenshots, clean text, Markdown, and structured extraction paths help teams move from page retrieval to usable records.

Less scraping infrastructure to own

Teams can focus on fields, validation, freshness, and business logic instead of browser fleets, routing edge cases, and retry plumbing.

TARGET FIT

Map page behavior before choosing scraping controls

A use case is only reliable when the target requirements are clear. Check whether the data is in source HTML, whether JavaScript changes the result, whether location or language affects the response, and whether the workflow needs visual proof or machine-readable output.

Raw HTML response mapped to visible page data
01

Raw HTML availability

Check whether the fields you need are already present in the first HTML response.

Best first test for simple pages, articles, docs, listings, and product details.

Before and after JavaScript rendering comparison
02

JavaScript dependency

Compare raw HTML with the rendered DOM or screenshot before turning on browser rendering.

Use rendering when raw HTML is only a shell and scripts create the useful content.

Cookies, consent, session and access context for scraping
03

State & access context

Identify whether cookies, consent, login boundaries, pagination, or the same visitor state affect the response.

Use sessions and headers when continuity changes the data you receive.

Regional content variations across countries and routes
04

Geo/content variation

Test whether country, language, currency, inventory, SERP, ads, or route reputation changes the result.

Use geo or premium routing only when the target proves the response varies by route.

Questions to answer before scraping

Where is the target field?

Find whether the field exists in raw HTML, rendered DOM, an API call, structured data, a file, or a later interaction state.

What changes the response?

Sample a few URLs and compare status, final URL, content length, redirects, empty states, and visible result differences.

What context is required?

Mark any dependency on consent, locale, cookies, headers, authentication boundary, pagination, filters, or previous requests.

What does failure look like?

Define the signals for blocked, partial, stale, redirected, empty, timed-out, or region-mismatched responses before scaling.

What this changes in request setup

Fetch mode

Raw HTML availability decides whether the first request can stay simple or needs rendering, waits, screenshots, or extraction.

Request context

State requirements decide whether to add headers, cookies, sessions, consent handling, or a stable visitor profile.

Routing choice

Geo and reputation findings decide whether standard routing is enough or a country/premium route is justified.

Validation rules

Known failure states become checks for retries, escalation, QA review, and deciding when a lighter profile is still complete.

REQUEST SETUP

Build a test request before turning the use case into a pipeline

The fastest way to validate a public web data workflow is to test the exact request shape: target URL, response format, rendering choice, headers, sessions, routing, downloads, and diagnostics. Once the response is proven, the same shape can move into code, Postman, or a scheduled job.

HTML
Raw response

Raw HTML for parsers and crawlers

Use the fastest path for static pages, listings, docs, and product pages. Fetch the original source HTML first, then switch on browser rendering when JavaScript creates the markup your scraper needs.

format=htmlrender=falserender=true
TXT
Clean data

Text, Markdown, and JSON

Skip the cleanup layer after page access. Return readable text for agents, Markdown for prompts and RAG, or structured JSON with metadata, links, headings, clean text, and Markdown ready for your pipeline.

format=textformat=markdownformat=json
PDF
Visual files

Screenshots and PDFs

Capture what a real browser sees when you need QA evidence, archived proof, visual regression checks, downloadable reports, or shareable page records for customers and internal teams.

format=screenshotformat=pdfdownload

Controls that keep requests predictable

url

The target page to fetch, render, clean, or capture.

format

Select the response your product needs: html, text, markdown, json, screenshot, or pdf.

premium

Escalate to stronger routing for blocks, unstable targets, or geo-sensitive content.

render

Run Chromium for SPAs, JS-injected markup, screenshots, PDFs, and browser-only pages.

country

Request a country-specific route when content changes by location.

keep_headers

Forward selected request context such as user-agent, cookie, language, and referer.

render_wait

Give dynamic content a controlled extra wait before extraction or capture.

download

Return visual artifacts as downloadable files when the output is screenshot or PDF.

cURL example

Generate Markdown with the same request shape you can move into production

curl "https://scraperfly.com/api/v1?url=https%3A%2F%2Fexample.com&format=markdown" \
  -H "apikey: YOUR_API_KEY"

USE CASE ECONOMICS

Keep validation flexible until the workflow proves volume

Different use cases carry different request weight. A raw HTML check should stay lightweight, while browser rendering, premium routing, screenshots, and PDFs should be reserved for workflows that need them. That makes free testing, pay-as-you-go credits, and monthly credits fit different stages of the same project.

A lightweight HTML request returning a simple response
01

Basic fetch

Use a basic fetch when public HTML already contains the fields you need.

Lightest request profile; keep it when the response is complete.

JavaScript rendering producing a complete DOM
02

Rendering

Use rendering when raw HTML is a shell and JavaScript completes the page.

Heavier profile; use it only when browser output changes the result.

Several requests preserving cookies and visitor state
03

Stateful session

Use sessions when cookies, consent, locale, or pagination change the response.

Heavier state profile; preserve it only when continuity matters.

Request routing through geo and higher quality routes
04

Premium routing & geo

Use premium routing or geo only when origin, country, reputation, or regional content changes the result.

Heaviest routing profile; apply it only to targets that prove they need it.

What makes a request heavier

More infrastructure work

Browser rendering, routing choices, and longer waits mean the platform has to do more than return a simple response.

More target uncertainty

Redirects, empty states, rate limits, and challenge pages often require stronger delivery options or retries.

More state to preserve

Cookies, sessions, consent, and multi-page flows add moving parts that a plain stateless fetch does not have.

More evidence to return

Screenshots, PDFs, rendered output, and richer response artifacts are useful, but they are heavier than plain HTML.

How to keep request weight predictable

Sample before scale

Test a small URL set first and group targets by the lightest profile that returns complete data.

Do not render by default

Inspect raw HTML before turning on rendering. Many public pages already contain the data you need.

Escalate by evidence

Move up only when the response shows missing content, state dependency, geo variance, or routing problems.

Measure useful results

Optimize around valid records delivered, not raw request volume. A lighter profile is best only when it stays complete.

USE CASE FAQ

Common questions about web scraping use cases

Practical answers for teams choosing what to scrape, how to test a source, when to add heavier controls, and how to keep public web data workflows useful without overbuilding the request.

Use case answers
Requests

Making API requests

For developers wiring ScraperFly into agents, jobs, backend services, or quick tests.

What do I send to ScraperFly?

Send a target URL plus the output format you need. The API can return HTML, clean text, Markdown, JSON, screenshots, or PDFs depending on the request options.

Should I use GET or POST?

Both can produce the same scraping result. GET is convenient for simple query parameters; POST is better when you want a clear JSON body or longer request options.

Can I test a request before coding?

Yes. The API Builder generates the endpoint, JSON body, query parameters, and code samples, then lets you send a live request from the dashboard.

Output

HTML, text, Markdown, and JSON

Choose the smallest output that your product actually needs.

When should I use clean text?

Use clean text when you need readable page content for summaries, classification, search indexing, or LLM prompts without layout markup.

What is the JSON output for?

JSON is useful when you want structured metadata, headings, links, text, and Markdown in one response that is easier to inspect programmatically.

Is Markdown different from clean text?

Markdown keeps more content structure, such as headings and lists. Clean text is flatter and better when formatting is less important.

Rendering

JavaScript pages

Use browser rendering only when the target page needs it.

When do I enable render=true?

Enable rendering for SPAs, JavaScript-injected product pages, lazy-loaded content, or pages where the source HTML does not contain the data you need.

Does rendering always cost more?

Yes, browser work is heavier than a basic HTML fetch. Start without rendering, then enable it only for targets that genuinely require JavaScript execution.

Can I control viewport and wait time?

Yes. Screenshot and PDF requests can use viewport dimensions and render wait options so the captured page has time to settle.

Routing

Proxies and location

Route requests through standard or premium pools when target behavior changes.

When do I need premium proxy routing?

Use premium routing for stricter targets, higher block rates, unstable responses, or pages that behave differently under normal datacenter traffic.

Can I request a country?

Yes. Add a country option when the target content depends on location, currency, language, availability, or regional redirects.

Are sessions supported?

Session-style workflows can reuse context across related requests when continuity matters for pagination, locale changes, or multi-step scraping.

Files

Screenshots and PDFs

Capture rendered pages when a visual artifact is the required output.

Can ScraperFly return a screenshot?

Yes. Screenshot output captures the rendered page as an image file, including JavaScript-rendered content when rendering is enabled.

Can ScraperFly generate a PDF?

Yes. PDF output is useful for receipts, reports, archives, compliance snapshots, and pages that need to be saved as a document.

Where do generated files appear?

Registered dashboard users can save and download generated artifacts from request logs when the request produces a downloadable file.

Credits

Costs and credits

Credits track delivery work rather than idle infrastructure.

How many credits does a request use?

A simple HTML, text, Markdown, or JSON request usually uses fewer credits. Rendering, premium routing, screenshots, and PDFs use more because they require heavier delivery work.

Do pay-as-you-go credits expire?

Pay-as-you-go credits stay available until you use them. They are a good fit for uneven workloads or projects that do not need a monthly plan.

Do subscription credits roll over?

Yes. Monthly plan credits refresh every month and unused credits roll over. If payment fails after Stripe retries, the account falls back to Pay as you go.

Security

Keys, headers, and safety

Keep credentials on your backend and forward only the headers you intend to use.

Where should I store my API key?

Store the API key server-side or in a trusted environment. Avoid exposing production keys in public frontend bundles or static pages.

Can I forward request headers?

Yes, selected headers such as user-agent, accept-language, cookie, and referer can be forwarded when a target expects realistic request context.

Does ScraperFly bypass private access controls?

No. It is designed for fetching public web pages and pages you are allowed to access, not for bypassing authentication or private systems.

Billing

Plans, logs, and invoices

Use the dashboard to understand what happened and why credits changed.

Where can I inspect usage?

The dashboard records request logs, credit balance changes, output formats, proxy routing, durations, and billing events so you can audit usage.

Can I download invoices?

Paid transactions include invoice data when Stripe provides it, so invoice PDFs can be opened from the financial log.

How do I manage a subscription?

Subscribed users can open the customer portal to change payment details, review invoices, or cancel. Free and Pay as you go users can go to Pricing instead.