Request context
Send only the headers that make the public response clearer
ScraperFly helps teams add request context without turning every scrape into a copied browser session. Start from a clean baseline, then forward the headers or cookies that genuinely change the public response: language, accepted content type, consent state, referer path, device behavior, or authenticated-by-the-site preferences that are already available to your workflow.
That keeps custom header scraping understandable. Developers can compare before and after responses, see what changed, and pair headers with rendering, sessions, country targeting, or stronger routing only when the evidence supports the added control.
What is a custom headers scraping API?
A custom headers scraping API is a hosted request layer that can fetch a public page while sending selected HTTP headers or cookies with the request. Instead of building a separate fetch stack for every target, your application sends the target URL, chooses output format and delivery controls, and adds request context only when the target response proves it matters.
Headers are not a magic bypass. They are context signals. A public page may respond differently to language preferences, accepted content types, user-agent families, referer context, cache directives, or cookies. The goal is to make that context explicit and testable, so the scraper can reproduce the useful response without carrying unnecessary browser clutter.
Practical takeaway: use custom headers to explain a required request context, not to disguise an unclear scrape profile.
When should a scraping request forward headers?
Forward headers when they change the response your product can use. A language header may return translated copy. A content negotiation header may affect whether the target sends HTML, JSON-like data, or compressed output. A referer can matter for public flows that depend on a previous page. Cookies can preserve consent, currency, region, or lightweight state. User-agent style behavior can change the layout or markup family a site returns.
The right workflow is simple: send a baseline request, inspect the response, add one kind of context, then compare. If the useful fields appear, the header profile earned its place. If nothing important changes, remove it and keep the request smaller.
| Header or context | What it can affect | Use it when |
|---|---|---|
Accept-Language | Language, regional copy, currency hints, localized labels | The target returns a different useful page by locale or language. |
| User-agent context | Markup family, responsive layout, bot-sensitive behavior | The same URL returns different HTML for different client profiles. |
Accept and content headers | Returned content type, compression, response shape | The target negotiates output based on accepted formats. |
Referer | Public navigation context, redirect path, campaign or listing flow | The target response depends on the public page path that led to it. |
| Cookies | Consent state, preferences, currency, region, session continuity | The workflow needs state that is stable, legitimate, and reproducible. |
Content-Type | POST body handling and API-style target behavior | You send a body and the target expects a specific request format. |
Why should teams avoid copying every browser header?
Copying every browser header feels thorough, but it usually makes scraping harder to operate. A real browser request can include volatile values, device hints, cache state, experiments, tracking context, and headers that are irrelevant to the data you need. When all of that is forwarded by default, failures become harder to explain because the request no longer has a clean cause-and-effect relationship.
A smaller header profile is easier to test, reproduce, monitor, and move into code. It also makes cost decisions clearer. If a page works with source HTML plus one language header, there is no reason to pay the complexity cost of rendering, session continuity, and premium routing. If a page still fails after the right context is present, the team has stronger evidence for the next escalation step.
Use the API Builder to compare a baseline request with a header-aware request before writing parser code around the result.
How are headers different from cookies and sessions?
Headers describe the request. Cookies carry state. Sessions preserve continuity across related requests. They often work together, but they solve different problems. A language header can ask for a regional version of a page. A cookie can keep a consent choice or currency preference. A session can keep that state consistent while the workflow moves across related public pages.
That difference matters because each control should have a clear job. If a single public page needs one language signal, a header may be enough. If a workflow must keep state across a search page, listing page, and detail page, sessions are more appropriate. If useful content appears only after frontend code runs, headers alone will not replace JavaScript rendering.
| Control | Keeps or changes | Best use | Common mistake |
|---|---|---|---|
| Headers | Request context for one request | Language, accepted output, user-agent family, referer context | Forwarding noisy browser headers with no evidence. |
| Cookies | Target-provided state | Consent, preferences, regional state, lightweight continuity | Using stale or unrelated cookies across different targets. |
| Sessions | Continuity across related requests | Multi-step public flows that depend on shared state | Using sessions for isolated pages with no dependency. |
| Rendering | Browser-built page state | SPAs, frontend-loaded content, waits, screenshots, PDFs | Rendering pages where source HTML already contains the data. |
| Country targeting | Geographic request perspective | Pages that vary by region, currency, availability, or redirect | Using geography when a simple language or cookie signal was enough. |
What response evidence proves a header profile works?
The useful proof is in the response, not in the header list. Compare final URL, status code, content type, response size, language, visible fields, extracted links, redirect behavior, and parser results across samples. If the only change is cosmetic, the header may not be worth keeping. If the target fields become available or stabilize across pages, the profile is useful.
Choose the output that makes the comparison easy. HTML is useful when selectors and DOM relationships matter. Text or Markdown can make content differences easier to read. JSON metadata helps with diagnostics such as final URL, status, response size, and content type. Screenshots and PDFs help when a human needs visual proof. The response formats guide explains when each output shape is worth using.
Practical takeaway: keep a header only when it changes the response in a way your product can use or your team can explain.
How do headers work with rendering and routing?
Headers can be part of a broader request profile. A target may need a language header and raw HTML. Another may need cookies and a persistent session. A JavaScript-heavy page may need rendering plus a wait. A region-sensitive page may need country targeting. A stricter target may need stronger delivery through a rotating proxy scraping API or premium routing.
The mistake is enabling every control at once. That hides which control solved the problem and makes production behavior difficult to tune. Start with headers when request context is the suspected gap, then escalate only when response evidence points to rendering, sessions, geography, or routing.
How can custom header scraping stay predictable?
Keep each request profile small, named, and repeatable. Document which headers are forwarded and why. Test representative URLs before scaling. Separate page templates that need different context. Track status, final URL, response size, content type, and extracted field quality so bad responses do not silently enter the pipeline.
Cost control follows the same logic. Headers and cookies are often lighter than browser rendering or premium routing, but a noisy profile can still create debugging overhead. Keep simple pages simple, reserve heavier controls for pages that prove they need them, and review pricing before scaling a new target class.
Where should a team start?
Start with a baseline ScraperFly request for the public target URL. Capture HTML or the response format your parser needs. Add the smallest header or cookie context that could explain the missing or unstable result. Compare outputs. If the response improves, keep the profile. If it does not, remove the header and investigate rendering, sessions, routing, or target-specific errors instead.
For focused header pages, use the dedicated guides for user-agent behavior, Accept-Language, and Referer context. For practical testing, open the API Builder and compare the request profiles before writing production parser code.
