ScraperFly Documentation
Quickstart: authenticate with your API key, fetch HTML, enable JS rendering, choose output formats, and use premium proxies. ScraperFly is intentionally direct: send a URL, choose the controls you need, and handle the returned HTML, clean text, Markdown, JSON, screenshot, or PDF in your application.
/api/v1
Use GET query params or POST JSON. Both request shapes return the same result.
apikey header
Keep keys server-side and rotate them from the dashboard whenever needed.
1 to 25 credits
Costs depend on premium routing, browser rendering, and visual file output.
Credit Costs & Failed Requests
Credits depend on the request configuration (premium proxies, JS rendering, etc.) and the final result. In practice, each request costs between 1 and 25 credits.
| Premium | Render | Cost in credits |
|---|---|---|
| false | false | 1 |
| false | true | 5 |
| true | false | 10 |
| true | true | 25 |
- Start with
premium=false,render=falsefor easy targets. - Enable
render=trueonly for JS-heavy pages (React/Vue/Next.js). - Use
countrywhen the target changes by location. - If you hit challenge pages, try
premium=true+render=true.
Basic Usage
Fetch the HTML of a target page. For basic usage you only need your apikey and a url. This typically uses standard proxies and costs 1 credit. Response is the raw HTML of the page. If you need structured data, parse HTML on your side (Cheerio/BeautifulSoup/etc.).
curl "https://scraperfly.com/api/v1?url=https://example.com" \
-H "apikey: YOUR_API_KEY_HERE"
// Node 18+ has fetch built-in
const res = await fetch("https://scraperfly.com/api/v1?url=https://example.com", {
headers: { apikey: "YOUR_API_KEY_HERE" }
});
const html = await res.text();
console.log(html);
import requests
res = requests.get(
"https://scraperfly.com/api/v1",
params={"url": "https://example.com"},
headers={"apikey": "YOUR_API_KEY_HERE"}
)
print(res.text)
<?php
$endpoint = "https://scraperfly.com/api/v1";
$url = "https://example.com";
$ch = curl_init($endpoint . "?url=" . urlencode($url));
curl_setopt($ch, CURLOPT_HTTPHEADER, ["apikey: YOUR_API_KEY_HERE"]);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$html = curl_exec($ch);
curl_close($ch);
echo $html;
package main
import (
"fmt"
"io"
"net/http"
"net/url"
)
func main() {
endpoint := "https://scraperfly.com/api/v1?url=" + url.QueryEscape("https://example.com")
req, _ := http.NewRequest("GET", endpoint, nil)
req.Header.Set("apikey", "YOUR_API_KEY_HERE")
resp, err := http.DefaultClient.Do(req)
if err != nil { panic(err) }
defer resp.Body.Close()
b, _ := io.ReadAll(resp.Body)
fmt.Println(string(b))
}
using System.Net.Http;
var http = new HttpClient();
http.DefaultRequestHeaders.Add("apikey", "YOUR_API_KEY_HERE");
var endpoint = "https://scraperfly.com/api/v1?url=https://example.com";
var html = await http.GetStringAsync(endpoint);
Console.WriteLine(html);
import java.net.http.*;
import java.net.URI;
HttpClient client = HttpClient.newHttpClient();
HttpRequest req = HttpRequest.newBuilder()
.uri(URI.create("https://scraperfly.com/api/v1?url=https://example.com"))
.header("apikey", "YOUR_API_KEY_HERE")
.GET()
.build();
HttpResponse<String> res = client.send(req, HttpResponse.BodyHandlers.ofString());
System.out.println(res.body());
require "net/http"
require "uri"
endpoint = URI("https://scraperfly.com/api/v1?url=https://example.com")
req = Net::HTTP::Get.new(endpoint)
req["apikey"] = "YOUR_API_KEY_HERE"
res = Net::HTTP.start(endpoint.host, endpoint.port, use_ssl: endpoint.scheme == "https") do |http|
http.request(req)
end
puts res.body
Request Parameters
Use the canonical endpoint GET /api/v1 with query params, or POST /api/v1 with the same fields in a JSON body. Start with url + format, then add proxy, render, viewport, or file options only when the target needs them.
Core request
| Parameter | Type | Description |
|---|---|---|
url | required | Target website URL you want to scrape, e.g. url=https://example.com. |
premium | optional, boolean | Uses higher-quality proxy pools for harder targets. Costs more credits. |
country | optional | Requested proxy country, for example US. Usage logs store the actual proxy country and city. |
render | optional, boolean | Headless browser rendering for pages that require JavaScript. Auto-enabled for screenshot and pdf. |
keep_headers | optional, boolean | Pass through your own headers (cookies, user-agent, accept-language, etc.). |
format | optional | Output format: html, text, markdown, json, screenshot, or pdf. |
Output formats
| format | Response | When to use |
|---|---|---|
html | HTML | Default. Use when you want raw source or rendered HTML for your own parser. |
text | plain text | Use when you need readable clean text without navigation/layout noise. |
markdown | Markdown | Use for LLM prompts, RAG, docs, summaries, and indexing. |
json | JSON | Use for metadata, title, headings, links, clean text, and Markdown in one response. |
screenshot | image file | Always uses Chromium. Use for visual proof, monitoring, QA, or customer review. |
pdf | PDF file | Always uses Chromium. Use for durable page records, audit trails, reports, or offline review. |
Browser render and viewport
These options matter only when render=true, format=screenshot, or format=pdf. If render is off and the output is text/html/markdown/json, viewport options are ignored.
| Parameter | Type / default | Description |
|---|---|---|
render_wait | ms, default 0 | Extra wait after initial page load before capture/extraction. Range: 0..15000. It does not include Chromium startup or navigation time. |
viewport_width | number, default 1365 | Browser viewport width. CSS mode accepts 320..3840; Windows mode accepts physical display pixels up to 7680. |
viewport_height | number, default 768 | Browser viewport height. CSS mode accepts 320..4000; Windows mode accepts physical display pixels up to 8000. |
pixels | css or windows | css means viewport values are browser CSS pixels. windows means you pass Windows Display Settings pixels and ScraperFly converts them using display_scale. |
display_scale | number, default 150 | Used only with pixels=windows. Accepts 125, 150, 200 or 1.25, 1.5, 2. |
Visual file options
These options apply only to format=screenshot and format=pdf. Successful responses are also saved as downloadable artifacts in the request log.
| Parameter | Type / default | Description |
|---|---|---|
file_type | jpg for screenshot, pdf for PDF | Screenshot supports jpg, jpeg, png, webp. PDF supports only pdf. BMP is not supported. |
download | boolean | true returns Content-Disposition: attachment. false returns inline preview when the client supports it. |
filename | string | Optional file name. The extension is normalized from file_type. |
Postman examples
Paste these directly into Postman as URL examples or request-body references. GET query params and POST JSON body use the same option names.
Markdown from a simple page
Use a plain GET URL when you want a fast Postman sanity check for clean Markdown output.
https://scraperfly.com/api/v1?url=https://example.com&format=markdown
PDF with Windows-size viewport
Use viewport controls when a visual file must match a real desktop display size.
https://scraperfly.com/api/v1?url=https://example.com&format=pdf&premium=true&render_wait=10000&viewport_width=2560&viewport_height=1600&pixels=windows&display_scale=150&file_type=pdf&download=true
POST JSON body with the same options
Switch to POST when the request has many options and a JSON body is easier to review or store.
POST https://scraperfly.com/api/v1
apikey: YOUR_API_KEY_HERE
content-type: application/json
{
"url": "https://example.com",
"format": "screenshot",
"premium": true,
"render_wait": 10000,
"viewport_width": 1280,
"viewport_height": 720,
"pixels": "css",
"file_type": "jpg",
"download": true
}
Advanced Example (Premium + Render + Headers + Country)
Use this setup when a site blocks standard traffic. It enables premium proxy routing, runs a headless browser, keeps selected headers, and requests a country-specific proxy. If a page still fails, retry with exponential backoff, reduce concurrency, or test another country.
curl "https://scraperfly.com/api/v1?url=https://example.com&premium=true&render=true&keep_headers=true&country=US" \
-H "apikey: YOUR_API_KEY_HERE" \
-H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120 Safari/537.36" \
-H "Cookie: SESSIONID=27382738"
const params = new URLSearchParams({
url: "https://example.com",
premium: "true",
render: "true",
keep_headers: "true",
country: "US"
});
const res = await fetch("https://scraperfly.com/api/v1?" + params.toString(), {
headers: {
apikey: "YOUR_API_KEY_HERE",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120 Safari/537.36",
"Cookie": "SESSIONID=27382738"
}
});
const html = await res.text();
console.log(html);
import requests
params = {
"url": "https://example.com",
"premium": "true",
"render": "true",
"keep_headers": "true",
"country": "US"
}
headers = {
"apikey": "YOUR_API_KEY_HERE",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120 Safari/537.36",
"Cookie": "SESSIONID=27382738"
}
res = requests.get("https://scraperfly.com/api/v1", params=params, headers=headers, timeout=30)
print(res.text)
<?php
$endpoint = "https://scraperfly.com/api/v1";
$params = http_build_query([
"url" => "https://example.com",
"premium" => "true",
"render" => "true",
"keep_headers" => "true",
"country" => "US",
]);
$ch = curl_init($endpoint . "?" . $params);
curl_setopt($ch, CURLOPT_HTTPHEADER, [
"apikey: YOUR_API_KEY_HERE",
"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120 Safari/537.36",
"Cookie: SESSIONID=27382738",
]);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 30);
$html = curl_exec($ch);
curl_close($ch);
echo $html;
package main
import (
"fmt"
"io"
"net/http"
"net/url"
"time"
)
func main() {
q := url.Values{}
q.Set("url", "https://example.com")
q.Set("premium", "true")
q.Set("render", "true")
q.Set("keep_headers", "true")
q.Set("country", "US")
endpoint := "https://scraperfly.com/api/v1?" + q.Encode()
req, _ := http.NewRequest("GET", endpoint, nil)
req.Header.Set("apikey", "YOUR_API_KEY_HERE")
req.Header.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120 Safari/537.36")
req.Header.Set("Cookie", "SESSIONID=27382738")
client := &http.Client{ Timeout: 30 * time.Second }
resp, err := client.Do(req)
if err != nil { panic(err) }
defer resp.Body.Close()
b, _ := io.ReadAll(resp.Body)
fmt.Println(string(b))
}
using System.Net.Http;
using System.Net.Http.Headers;
var http = new HttpClient();
http.DefaultRequestHeaders.Add("apikey", "YOUR_API_KEY_HERE");
http.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120 Safari/537.36");
http.DefaultRequestHeaders.Add("Cookie", "SESSIONID=27382738");
var endpoint = "https://scraperfly.com/api/v1?url=https://example.com&premium=true&render=true&keep_headers=true&country=US";
var html = await http.GetStringAsync(endpoint);
Console.WriteLine(html);
import java.net.http.*;
import java.net.URI;
HttpClient client = HttpClient.newBuilder()
.followRedirects(HttpClient.Redirect.NORMAL)
.build();
HttpRequest req = HttpRequest.newBuilder()
.uri(URI.create("https://scraperfly.com/api/v1?url=https://example.com&premium=true&render=true&keep_headers=true&country=US"))
.header("apikey", "YOUR_API_KEY_HERE")
.header("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120 Safari/537.36")
.header("Cookie", "SESSIONID=27382738")
.GET()
.build();
HttpResponse<String> res = client.send(req, HttpResponse.BodyHandlers.ofString());
System.out.println(res.body());
require "net/http"
require "uri"
endpoint = URI("https://scraperfly.com/api/v1?url=https://example.com&premium=true&render=true&keep_headers=true&country=US")
req = Net::HTTP::Get.new(endpoint)
req["apikey"] = "YOUR_API_KEY_HERE"
req["User-Agent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120 Safari/537.36"
req["Cookie"] = "SESSIONID=27382738"
res = Net::HTTP.start(endpoint.host, endpoint.port, use_ssl: endpoint.scheme == "https") do |http|
http.read_timeout = 30
http.request(req)
end
puts res.body
JS Rendering (Quick Example)
For JavaScript-heavy pages, set render=true. Use this when the source HTML does not contain the data and the page needs browser execution first.
curl "https://scraperfly.com/api/v1?url=https://quotes.toscrape.com/js&render=true" \
-H "apikey: YOUR_API_KEY_HERE"
Authentication & Remaining Credits
ScraperFly uses API keys for authorization. Recommended: send your key via the apikey header. The /status endpoint returns your remaining credits and account info.
curl "https://scraperfly.com/api/v1/status" \
-H "apikey: YOUR_API_KEY_HERE"
const res = await fetch("https://scraperfly.com/api/v1/status", {
headers: { apikey: "YOUR_API_KEY_HERE" }
});
const data = await res.json();
console.log(data);
import requests
res = requests.get("https://scraperfly.com/api/v1/status", headers={"apikey": "YOUR_API_KEY_HERE"})
print(res.json())
<?php
$ch = curl_init("https://scraperfly.com/api/v1/status");
curl_setopt($ch, CURLOPT_HTTPHEADER, ["apikey: YOUR_API_KEY_HERE"]);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$out = curl_exec($ch);
curl_close($ch);
echo $out;
package main
import (
"fmt"
"io"
"net/http"
)
func main() {
req, _ := http.NewRequest("GET", "https://scraperfly.com/api/v1/status", nil)
req.Header.Set("apikey", "YOUR_API_KEY_HERE")
resp, err := http.DefaultClient.Do(req)
if err != nil { panic(err) }
defer resp.Body.Close()
b, _ := io.ReadAll(resp.Body)
fmt.Println(string(b))
}
using System.Net.Http;
var http = new HttpClient();
http.DefaultRequestHeaders.Add("apikey", "YOUR_API_KEY_HERE");
var json = await http.GetStringAsync("https://scraperfly.com/api/v1/status");
Console.WriteLine(json);
import java.net.http.*;
import java.net.URI;
HttpClient client = HttpClient.newHttpClient();
HttpRequest req = HttpRequest.newBuilder()
.uri(URI.create("https://scraperfly.com/api/v1/status"))
.header("apikey", "YOUR_API_KEY_HERE")
.GET()
.build();
HttpResponse<String> res = client.send(req, HttpResponse.BodyHandlers.ofString());
System.out.println(res.body());
require "net/http"
require "uri"
endpoint = URI("https://scraperfly.com/api/v1/status")
req = Net::HTTP::Get.new(endpoint)
req["apikey"] = "YOUR_API_KEY_HERE"
res = Net::HTTP.start(endpoint.host, endpoint.port, use_ssl: endpoint.scheme == "https") do |http|
http.request(req)
end
puts res.body
Error Handling
Errors are returned as JSON with a stable shape. Always inspect HTTP status + error.code to decide whether to retry.
{
"error": {
"code": "validation_error",
"status": 422,
"context": [
{
"url": {
"type": "invalid_url",
"message": "missing url"
}
}
]
}
}
Retry on network/429/temporary 5xx. Do not retry on invalid URL or invalid API key.
Common Error Codes
| HTTP | Type | Meaning |
|---|---|---|
| 400 | missing_url | No valid url parameter provided. |
| 400 | invalid_url | Invalid value in url. |
| 401 | missing_api_key | No API key supplied. |
| 403 | invalid_api_key | API key is invalid or disabled. |
| 403 | forbidden | Not enough credits or no access to feature. |
| 429 | rate_limited | Too many concurrent requests. Slow down or upgrade. |
| 429 | usage_limit_reached | Plan/monthly limit reached. |
| 500 | internal_error | Server error. Retry with backoff. |
Common Use Cases
Keep the common production patterns in one tabbed block: start simple, then add routing, rendering, headers, or backoff only when the target proves it needs them.
1) Premium Proxies
Enable premium=true for tougher targets. Add country when the response changes by market or location.
curl "https://scraperfly.com/api/v1?url=https://example.com&premium=true&country=US" \
-H "apikey: YOUR_API_KEY_HERE"
2) Custom Headers (User-Agent / Cookies)
Use keep_headers=true when target context matters and you need to pass selected request headers through.
curl "https://scraperfly.com/api/v1?url=https://example.com&keep_headers=true" \
-H "apikey: YOUR_API_KEY_HERE" \
-H "User-Agent: Mozilla/5.0" \
-H "Accept-Language: en-US,en;q=0.9" \
-H "Cookie: SESSIONID=27382738"
3) JS Rendering
For SPA, Next.js, and browser-built pages, set render=true so ScraperFly waits for JavaScript output before extracting the response.
curl "https://scraperfly.com/api/v1?url=https://quotes.toscrape.com/js&render=true" \
-H "apikey: YOUR_API_KEY_HERE"
4) Anti-bot & Challenge Pages
If you receive challenge or blocked content, combine premium routing, browser rendering, realistic allowed headers, lower concurrency, and retry backoff.
curl "https://scraperfly.com/api/v1?url=https://example.com&premium=true&render=true&keep_headers=true&country=US" \
-H "apikey: YOUR_API_KEY_HERE" \
-H "User-Agent: Mozilla/5.0" \
-H "Accept-Language: en-US,en;q=0.9"