ScraperFlyAI-ready Web Data API

docs

Documentation for your first ScraperFly request

The API is intentionally direct: send a URL, choose the controls you need, and handle the returned HTML or extracted response in your application.

Documentation

ScraperFly Documentation

Quickstart: authenticate with your API key, fetch HTML, enable JS rendering, choose output formats, and use premium proxies. ScraperFly is intentionally direct: send a URL, choose the controls you need, and handle the returned HTML, clean text, Markdown, JSON, screenshot, or PDF in your application.

Endpoint

/api/v1

Use GET query params or POST JSON. Both request shapes return the same result.

Auth

apikey header

Keep keys server-side and rotate them from the dashboard whenever needed.

Credits

1 to 25 credits

Costs depend on premium routing, browser rendering, and visual file output.

Credit Costs & Failed Requests

Credits depend on the request configuration (premium proxies, JS rendering, etc.) and the final result. In practice, each request costs between 1 and 25 credits.

PremiumRenderCost in credits
falsefalse1
falsetrue5
truefalse10
truetrue25
Tips to control costs
  • Start with premium=false, render=false for easy targets.
  • Enable render=true only for JS-heavy pages (React/Vue/Next.js).
  • Use country when the target changes by location.
  • If you hit challenge pages, try premium=true + render=true.

Basic Usage

Fetch the HTML of a target page. For basic usage you only need your apikey and a url. This typically uses standard proxies and costs 1 credit. Response is the raw HTML of the page. If you need structured data, parse HTML on your side (Cheerio/BeautifulSoup/etc.).

8 examples
curl "https://scraperfly.com/api/v1?url=https://example.com" \
  -H "apikey: YOUR_API_KEY_HERE"
// Node 18+ has fetch built-in
const res = await fetch("https://scraperfly.com/api/v1?url=https://example.com", {
  headers: { apikey: "YOUR_API_KEY_HERE" }
});
const html = await res.text();
console.log(html);
import requests

res = requests.get(
  "https://scraperfly.com/api/v1",
  params={"url": "https://example.com"},
  headers={"apikey": "YOUR_API_KEY_HERE"}
)

print(res.text)
<?php
$endpoint = "https://scraperfly.com/api/v1";
$url = "https://example.com";
$ch = curl_init($endpoint . "?url=" . urlencode($url));
curl_setopt($ch, CURLOPT_HTTPHEADER, ["apikey: YOUR_API_KEY_HERE"]);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$html = curl_exec($ch);
curl_close($ch);
echo $html;
package main

import (
  "fmt"
  "io"
  "net/http"
  "net/url"
)

func main() {
  endpoint := "https://scraperfly.com/api/v1?url=" + url.QueryEscape("https://example.com")
  req, _ := http.NewRequest("GET", endpoint, nil)
  req.Header.Set("apikey", "YOUR_API_KEY_HERE")

  resp, err := http.DefaultClient.Do(req)
  if err != nil { panic(err) }
  defer resp.Body.Close()

  b, _ := io.ReadAll(resp.Body)
  fmt.Println(string(b))
}
using System.Net.Http;

var http = new HttpClient();
http.DefaultRequestHeaders.Add("apikey", "YOUR_API_KEY_HERE");

var endpoint = "https://scraperfly.com/api/v1?url=https://example.com";
var html = await http.GetStringAsync(endpoint);

Console.WriteLine(html);
import java.net.http.*;
import java.net.URI;

HttpClient client = HttpClient.newHttpClient();

HttpRequest req = HttpRequest.newBuilder()
  .uri(URI.create("https://scraperfly.com/api/v1?url=https://example.com"))
  .header("apikey", "YOUR_API_KEY_HERE")
  .GET()
  .build();

HttpResponse<String> res = client.send(req, HttpResponse.BodyHandlers.ofString());
System.out.println(res.body());
require "net/http"
require "uri"

endpoint = URI("https://scraperfly.com/api/v1?url=https://example.com")
req = Net::HTTP::Get.new(endpoint)
req["apikey"] = "YOUR_API_KEY_HERE"

res = Net::HTTP.start(endpoint.host, endpoint.port, use_ssl: endpoint.scheme == "https") do |http|
  http.request(req)
end

puts res.body

Request Parameters

Use the canonical endpoint GET /api/v1 with query params, or POST /api/v1 with the same fields in a JSON body. Start with url + format, then add proxy, render, viewport, or file options only when the target needs them.

Core request

ParameterTypeDescription
urlrequiredTarget website URL you want to scrape, e.g. url=https://example.com.
premiumoptional, booleanUses higher-quality proxy pools for harder targets. Costs more credits.
countryoptionalRequested proxy country, for example US. Usage logs store the actual proxy country and city.
renderoptional, booleanHeadless browser rendering for pages that require JavaScript. Auto-enabled for screenshot and pdf.
keep_headersoptional, booleanPass through your own headers (cookies, user-agent, accept-language, etc.).
formatoptionalOutput format: html, text, markdown, json, screenshot, or pdf.

Output formats

formatResponseWhen to use
htmlHTMLDefault. Use when you want raw source or rendered HTML for your own parser.
textplain textUse when you need readable clean text without navigation/layout noise.
markdownMarkdownUse for LLM prompts, RAG, docs, summaries, and indexing.
jsonJSONUse for metadata, title, headings, links, clean text, and Markdown in one response.
screenshotimage fileAlways uses Chromium. Use for visual proof, monitoring, QA, or customer review.
pdfPDF fileAlways uses Chromium. Use for durable page records, audit trails, reports, or offline review.

Browser render and viewport

These options matter only when render=true, format=screenshot, or format=pdf. If render is off and the output is text/html/markdown/json, viewport options are ignored.

ParameterType / defaultDescription
render_waitms, default 0Extra wait after initial page load before capture/extraction. Range: 0..15000. It does not include Chromium startup or navigation time.
viewport_widthnumber, default 1365Browser viewport width. CSS mode accepts 320..3840; Windows mode accepts physical display pixels up to 7680.
viewport_heightnumber, default 768Browser viewport height. CSS mode accepts 320..4000; Windows mode accepts physical display pixels up to 8000.
pixelscss or windowscss means viewport values are browser CSS pixels. windows means you pass Windows Display Settings pixels and ScraperFly converts them using display_scale.
display_scalenumber, default 150Used only with pixels=windows. Accepts 125, 150, 200 or 1.25, 1.5, 2.

Visual file options

These options apply only to format=screenshot and format=pdf. Successful responses are also saved as downloadable artifacts in the request log.

ParameterType / defaultDescription
file_typejpg for screenshot, pdf for PDFScreenshot supports jpg, jpeg, png, webp. PDF supports only pdf. BMP is not supported.
downloadbooleantrue returns Content-Disposition: attachment. false returns inline preview when the client supports it.
filenamestringOptional file name. The extension is normalized from file_type.

Postman examples

Paste these directly into Postman as URL examples or request-body references. GET query params and POST JSON body use the same option names.

Postman ready

Markdown from a simple page

Use a plain GET URL when you want a fast Postman sanity check for clean Markdown output.

https://scraperfly.com/api/v1?url=https://example.com&format=markdown

PDF with Windows-size viewport

Use viewport controls when a visual file must match a real desktop display size.

https://scraperfly.com/api/v1?url=https://example.com&format=pdf&premium=true&render_wait=10000&viewport_width=2560&viewport_height=1600&pixels=windows&display_scale=150&file_type=pdf&download=true

POST JSON body with the same options

Switch to POST when the request has many options and a JSON body is easier to review or store.

POST https://scraperfly.com/api/v1
apikey: YOUR_API_KEY_HERE
content-type: application/json

{
  "url": "https://example.com",
  "format": "screenshot",
  "premium": true,
  "render_wait": 10000,
  "viewport_width": 1280,
  "viewport_height": 720,
  "pixels": "css",
  "file_type": "jpg",
  "download": true
}

Advanced Example (Premium + Render + Headers + Country)

Use this setup when a site blocks standard traffic. It enables premium proxy routing, runs a headless browser, keeps selected headers, and requests a country-specific proxy. If a page still fails, retry with exponential backoff, reduce concurrency, or test another country.

8 examples
curl "https://scraperfly.com/api/v1?url=https://example.com&premium=true&render=true&keep_headers=true&country=US" \
  -H "apikey: YOUR_API_KEY_HERE" \
  -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120 Safari/537.36" \
  -H "Cookie: SESSIONID=27382738"
const params = new URLSearchParams({
  url: "https://example.com",
  premium: "true",
  render: "true",
  keep_headers: "true",
  country: "US"
});

const res = await fetch("https://scraperfly.com/api/v1?" + params.toString(), {
  headers: {
    apikey: "YOUR_API_KEY_HERE",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120 Safari/537.36",
    "Cookie": "SESSIONID=27382738"
  }
});

const html = await res.text();
console.log(html);
import requests

params = {
  "url": "https://example.com",
  "premium": "true",
  "render": "true",
  "keep_headers": "true",
  "country": "US"
}

headers = {
  "apikey": "YOUR_API_KEY_HERE",
  "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120 Safari/537.36",
  "Cookie": "SESSIONID=27382738"
}

res = requests.get("https://scraperfly.com/api/v1", params=params, headers=headers, timeout=30)
print(res.text)
<?php
$endpoint = "https://scraperfly.com/api/v1";
$params = http_build_query([
  "url" => "https://example.com",
  "premium" => "true",
  "render" => "true",
  "keep_headers" => "true",
  "country" => "US",
]);

$ch = curl_init($endpoint . "?" . $params);
curl_setopt($ch, CURLOPT_HTTPHEADER, [
  "apikey: YOUR_API_KEY_HERE",
  "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120 Safari/537.36",
  "Cookie: SESSIONID=27382738",
]);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 30);

$html = curl_exec($ch);
curl_close($ch);
echo $html;
package main

import (
  "fmt"
  "io"
  "net/http"
  "net/url"
  "time"
)

func main() {
  q := url.Values{}
  q.Set("url", "https://example.com")
  q.Set("premium", "true")
  q.Set("render", "true")
  q.Set("keep_headers", "true")
  q.Set("country", "US")

  endpoint := "https://scraperfly.com/api/v1?" + q.Encode()
  req, _ := http.NewRequest("GET", endpoint, nil)
  req.Header.Set("apikey", "YOUR_API_KEY_HERE")
  req.Header.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120 Safari/537.36")
  req.Header.Set("Cookie", "SESSIONID=27382738")

  client := &http.Client{ Timeout: 30 * time.Second }
  resp, err := client.Do(req)
  if err != nil { panic(err) }
  defer resp.Body.Close()

  b, _ := io.ReadAll(resp.Body)
  fmt.Println(string(b))
}
using System.Net.Http;
using System.Net.Http.Headers;

var http = new HttpClient();
http.DefaultRequestHeaders.Add("apikey", "YOUR_API_KEY_HERE");
http.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120 Safari/537.36");
http.DefaultRequestHeaders.Add("Cookie", "SESSIONID=27382738");

var endpoint = "https://scraperfly.com/api/v1?url=https://example.com&premium=true&render=true&keep_headers=true&country=US";
var html = await http.GetStringAsync(endpoint);

Console.WriteLine(html);
import java.net.http.*;
import java.net.URI;

HttpClient client = HttpClient.newBuilder()
  .followRedirects(HttpClient.Redirect.NORMAL)
  .build();

HttpRequest req = HttpRequest.newBuilder()
  .uri(URI.create("https://scraperfly.com/api/v1?url=https://example.com&premium=true&render=true&keep_headers=true&country=US"))
  .header("apikey", "YOUR_API_KEY_HERE")
  .header("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120 Safari/537.36")
  .header("Cookie", "SESSIONID=27382738")
  .GET()
  .build();

HttpResponse<String> res = client.send(req, HttpResponse.BodyHandlers.ofString());
System.out.println(res.body());
require "net/http"
require "uri"

endpoint = URI("https://scraperfly.com/api/v1?url=https://example.com&premium=true&render=true&keep_headers=true&country=US")
req = Net::HTTP::Get.new(endpoint)
req["apikey"] = "YOUR_API_KEY_HERE"
req["User-Agent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120 Safari/537.36"
req["Cookie"] = "SESSIONID=27382738"

res = Net::HTTP.start(endpoint.host, endpoint.port, use_ssl: endpoint.scheme == "https") do |http|
  http.read_timeout = 30
  http.request(req)
end

puts res.body

JS Rendering (Quick Example)

For JavaScript-heavy pages, set render=true. Use this when the source HTML does not contain the data and the page needs browser execution first.

curl "https://scraperfly.com/api/v1?url=https://quotes.toscrape.com/js&render=true" \
  -H "apikey: YOUR_API_KEY_HERE"

Authentication & Remaining Credits

ScraperFly uses API keys for authorization. Recommended: send your key via the apikey header. The /status endpoint returns your remaining credits and account info.

8 examples
curl "https://scraperfly.com/api/v1/status" \
  -H "apikey: YOUR_API_KEY_HERE"
const res = await fetch("https://scraperfly.com/api/v1/status", {
  headers: { apikey: "YOUR_API_KEY_HERE" }
});
const data = await res.json();
console.log(data);
import requests

res = requests.get("https://scraperfly.com/api/v1/status", headers={"apikey": "YOUR_API_KEY_HERE"})
print(res.json())
<?php
$ch = curl_init("https://scraperfly.com/api/v1/status");
curl_setopt($ch, CURLOPT_HTTPHEADER, ["apikey: YOUR_API_KEY_HERE"]);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$out = curl_exec($ch);
curl_close($ch);
echo $out;
package main

import (
  "fmt"
  "io"
  "net/http"
)

func main() {
  req, _ := http.NewRequest("GET", "https://scraperfly.com/api/v1/status", nil)
  req.Header.Set("apikey", "YOUR_API_KEY_HERE")
  resp, err := http.DefaultClient.Do(req)
  if err != nil { panic(err) }
  defer resp.Body.Close()

  b, _ := io.ReadAll(resp.Body)
  fmt.Println(string(b))
}
using System.Net.Http;

var http = new HttpClient();
http.DefaultRequestHeaders.Add("apikey", "YOUR_API_KEY_HERE");

var json = await http.GetStringAsync("https://scraperfly.com/api/v1/status");
Console.WriteLine(json);
import java.net.http.*;
import java.net.URI;

HttpClient client = HttpClient.newHttpClient();

HttpRequest req = HttpRequest.newBuilder()
  .uri(URI.create("https://scraperfly.com/api/v1/status"))
  .header("apikey", "YOUR_API_KEY_HERE")
  .GET()
  .build();

HttpResponse<String> res = client.send(req, HttpResponse.BodyHandlers.ofString());
System.out.println(res.body());
require "net/http"
require "uri"

endpoint = URI("https://scraperfly.com/api/v1/status")
req = Net::HTTP::Get.new(endpoint)
req["apikey"] = "YOUR_API_KEY_HERE"

res = Net::HTTP.start(endpoint.host, endpoint.port, use_ssl: endpoint.scheme == "https") do |http|
  http.request(req)
end

puts res.body

Error Handling

Errors are returned as JSON with a stable shape. Always inspect HTTP status + error.code to decide whether to retry.

{
  "error": {
    "code": "validation_error",
    "status": 422,
    "context": [
      {
        "url": {
          "type": "invalid_url",
          "message": "missing url"
        }
      }
    ]
  }
}
Retry rules

Retry on network/429/temporary 5xx. Do not retry on invalid URL or invalid API key.

Common Error Codes

HTTPTypeMeaning
400missing_urlNo valid url parameter provided.
400invalid_urlInvalid value in url.
401missing_api_keyNo API key supplied.
403invalid_api_keyAPI key is invalid or disabled.
403forbiddenNot enough credits or no access to feature.
429rate_limitedToo many concurrent requests. Slow down or upgrade.
429usage_limit_reachedPlan/monthly limit reached.
500internal_errorServer error. Retry with backoff.

Common Use Cases

Keep the common production patterns in one tabbed block: start simple, then add routing, rendering, headers, or backoff only when the target proves it needs them.

4 patterns

1) Premium Proxies

Enable premium=true for tougher targets. Add country when the response changes by market or location.

curl "https://scraperfly.com/api/v1?url=https://example.com&premium=true&country=US" \
  -H "apikey: YOUR_API_KEY_HERE"

2) Custom Headers (User-Agent / Cookies)

Use keep_headers=true when target context matters and you need to pass selected request headers through.

curl "https://scraperfly.com/api/v1?url=https://example.com&keep_headers=true" \
  -H "apikey: YOUR_API_KEY_HERE" \
  -H "User-Agent: Mozilla/5.0" \
  -H "Accept-Language: en-US,en;q=0.9" \
  -H "Cookie: SESSIONID=27382738"

3) JS Rendering

For SPA, Next.js, and browser-built pages, set render=true so ScraperFly waits for JavaScript output before extracting the response.

curl "https://scraperfly.com/api/v1?url=https://quotes.toscrape.com/js&render=true" \
  -H "apikey: YOUR_API_KEY_HERE"

4) Anti-bot & Challenge Pages

If you receive challenge or blocked content, combine premium routing, browser rendering, realistic allowed headers, lower concurrency, and retry backoff.

curl "https://scraperfly.com/api/v1?url=https://example.com&premium=true&render=true&keep_headers=true&country=US" \
  -H "apikey: YOUR_API_KEY_HERE" \
  -H "User-Agent: Mozilla/5.0" \
  -H "Accept-Language: en-US,en;q=0.9"