Getting Started

This documentation for Scrape.do, a simple and very scalable fast web scraper api gateway.
Scrape.do exposes very simple and customizable very fastest proxy web scraper api in an endpoint.
Features
The data center, residential and mobile IPs from a large IP pool are used to crawl a target site with 99.9% success, with using different IPs for every request. 🎉 
Each request returns raw html along with headers and cookies.
You will only be charged for the 200 and 404 status codes. For other occasions, you will not be charged.
If you exceed the rate limit, you will receive an error code 429. You may be able to resolve this issue when you set your request limit to meet the same conditions as your subscription plan.
Please note that there is a 2 MB response size limit for each request! If you exceed the limit, the transaction is considered successful, and 2 MB of the full response will be returned to you.
If the monthly request limit is exceeded or you have an unpaid bill, you will get a 401 error.
You can use the features specified on other pages by sending more than one parameter at the same time..
 Basic Usage
First you need to sign up.
You can see the token on the dashboard page.
API_TOKEN : It is your personal API key.
Url : It is the target website which you will crawl.
Free plan includes 1000 monthly api call with 5 concurrent request. You can test it as you wish.
Bash
Python
Go
C#
Java
NodeJs
Javascript
Ruby
PHP
Swift
Bash
$ curl "http://api.scrape.do?token=API_TOKEN&url=https://www.example.com"
Python
import requests
​
url = "http://api.scrape.do?token=API_TOKEN&url=http://example.com"
​
payload = {}
headers= {}
​
response = requests.request("GET", url, headers=headers, data = payload)
​
print(response.text.encode('utf8'))
​
Go
package main
​
import (
  "fmt"
  "net/http"
  "io/ioutil"
)
​
func main() {
​
  url := "http://api.scrape.do?token=API_TOKEN&url=http://example.com"
  method := "GET"
​
  client := &http.Client {
  }
  req, err := http.NewRequest(method, url, nil)
​
  if err != nil {
    fmt.Println(err)
  }
  res, err := client.Do(req)
  defer res.Body.Close()
  body, err := ioutil.ReadAll(res.Body)
​
  fmt.Println(string(body))
}
C#
var client = new RestClient("http://api.scrape.do?token=API_TOKEN&url=http://example.com");
client.Timeout = -1;
var request = new RestRequest(Method.GET);
IRestResponse response = client.Execute(request);
Console.WriteLine(response.Content);
Java
OkHttpClient client = new OkHttpClient().newBuilder()
  .build();
Request request = new Request.Builder()
  .url("http://api.scrape.do?token=API_TOKEN&url=http://example.com")
  .method("GET", null)
  .build();
Response response = client.newCall(request).execute();
NodeJs
var axios = require('axios');
​
var config = {
  method: 'get',
  url: 'http://api.scrape.do?token=API_TOKEN&url=http://example.com',
  headers: { }
};
​
axios(config)
.then(function (response) {
  console.log(JSON.stringify(response.data));
})
.catch(function (error) {
  console.log(error);
});
Javascript
var xhr = new XMLHttpRequest();
xhr.withCredentials = true;
​
xhr.addEventListener("readystatechange", function() {
  if(this.readyState === 4) {
    console.log(this.responseText);
  }
});
​
xhr.open("GET", "http://api.scrape.do?token=API_TOKEN&url=http://example.com");
​
xhr.send();
Ruby
require "uri"
require "net/http"
​
url = URI("http://api.scrape.do?token=API_TOKEN&url=http://example.com")
​
http = Net::HTTP.new(url.host, url.port);
request = Net::HTTP::Get.new(url)
​
response = http.request(request)
puts response.read_body
PHP
<?php
​
$curl = curl_init();
​
curl_setopt_array($curl, array(
  CURLOPT_URL => "http://api.scrape.do?token=API_TOKEN&url=http://example.com",
  CURLOPT_RETURNTRANSFER => true,
  CURLOPT_ENCODING => "",
  CURLOPT_MAXREDIRS => 10,
  CURLOPT_TIMEOUT => 0,
  CURLOPT_FOLLOWLOCATION => true,
  CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
  CURLOPT_CUSTOMREQUEST => "GET",
));
​
$response = curl_exec($curl);
​
curl_close($curl);
echo $response;
Swift
import Foundation
​
var semaphore = DispatchSemaphore (value: 0)
​
var request = URLRequest(url: URL(string: "http://api.scrape.do?token=API_TOKEN&url=http://example.com")!,timeoutInterval: Double.infinity)
request.httpMethod = "GET"
​
let task = URLSession.shared.dataTask(with: request) { data, response, error in 
  guard let data = data else {
    print(String(describing: error))
    return
  }
  print(String(data: data, encoding: .utf8)!)
  semaphore.signal()
}
​
task.resume()
semaphore.wait()
Result
<!doctype html>
<html>
<head>
    <title>Example Domain</title>
​
    <meta charset="utf-8" />
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
    
    ..............
    ..............
</head>
</html>
 That is all 🎉
If you have any question please feel free to contact us or email us at support@scrape.do​
Next - Client Guides
Headless Browsers (JS Render)
Last updated 5 months ago