The World's most Reliable and Cost-effective Web Scraping API
Get clean data for your AI from any website in 1-5 seconds, at an unbeatable price. The API avoids all bot detection and can parallelize up to 100K requests in minutes
Diagonal Sections
Using the rotation transform is how you might think to do it but I think skew is the way to go!
"The way we’re going to ratchet up our species is to take the best and spread it around to everybody. So that everybody grows up with better things and starts to understand the subtlety of better things"
{
"id": "scrape_56is5c9gyw",
"object": "scrape",
"created": 1317322740,
"result": {
"markdown_content": "# Example quote...."
}
}
Try in the Playground
From seed-stage to Series-A startups and scaleups
Gumloop (W24), Openmart (W24), Khoj (S23) are backed by Y-Combinator
Diagonal Sections
Using the rotation transform is how you might think to do it but I think skew is the way to go!
Get clean data from any URL
Crawl all the subpages on a website
Up to 100K URLs in 5-7 minutes
1import requests
2
3API_URL = 'https://api.olostep.com/v1/scrapes'
4API_KEY = '<your_token>'
5
6headers = {'Authorization': f'Bearer {API_KEY}'}
7data = {"url_to_scrape": "https://github.com"}
8
9response = requests.post(API_URL, headers=headers, json=data)
10result = response.json()
11
12print(f"Scrape ID: {result['id']}")
13print(f"URL: {result['url_to_scrape']}")1// Using native fetch API (Node.js v18+)
2const API_URL = 'https://api.olostep.com/v1/scrapes';
3const API_KEY = '<your_token>';
4
5fetch(API_URL, {
6 method: 'POST',
7 headers: {
8 'Authorization': `Bearer ${API_KEY}`,
9 'Content-Type': 'application/json'
10 },
11 body: JSON.stringify({
12 "url_to_scrape": "https://github.com"
13 })
14})
15.then(response => response.json())
16.then(result => {
17 console.log(`Scrape ID: ${result.id}`);
18 console.log(`URL: ${result.url_to_scrape}`);
19})
20.catch(error => console.error('Error:', error));1import requests
2
3API_URL = 'https://api.olostep.com/v1/crawls'
4API_KEY = '<token>'
5
6headers = {'Authorization': f'Bearer {API_KEY}'}
7data = {
8 "start_url": "https://docs.stripe.com/api",
9 "include_urls": ["/**"],
10 "max_pages": 10
11}
12
13response = requests.post(API_URL, headers=headers, json=data)
14result = response.json()
15
16print(f"Crawl ID: {result['id']}")
17print(f"URL: {result['start_url']}")1// Using native fetch API (Node.js v18+)
2const API_URL = 'https://api.olostep.com/v1/crawls';
3const API_KEY = '<token>';
4
5fetch(API_URL, {
6 method: 'POST',
7 headers: {
8 'Authorization': `Bearer ${API_KEY}`,
9 'Content-Type': 'application/json'
10 },
11 body: JSON.stringify({
12 "start_url": "https://docs.stripe.com/api",
13 "include_urls": ["/**"],
14 "max_pages": 10
15 })
16})
17.then(response => response.json())
18.then(result => {
19 console.log(`Crawl ID: ${result.id}`);
20 console.log(`URL: ${result.start_url}`);
21})
22.catch(error => console.error('Error:', error));1import requests
2
3API_URL = 'https://api.olostep.com/v1/batches'
4API_KEY = '<token>'
5
6headers = {'Authorization': f'Bearer {API_KEY}', 'Content-Type': 'application/json'}
7data = {
8 "items": [
9 {
10 "custom_id": "url_1kdjq3",
11 "url": "https://www.amazon.com/Dream-Machine-M-Mitchell-Waldrop/dp/1732265119"
12 },
13 # up to 10-100k URLs per batch
14 ]
15}
16
17response = requests.post(API_URL, headers=headers, json=data)
18result = response.json()
19
20print(f"Batch ID: {result['id']}")
21print(f"Status: {result['status']}")1// Using native fetch API (Node.js v18+)
2const API_URL = 'https://api.olostep.com/v1/batches';
3const API_KEY = '<token>';
4
5fetch(API_URL, {
6 method: 'POST',
7 headers: {
8 'Authorization': `Bearer ${API_KEY}`,
9 'Content-Type': 'application/json'
10 },
11 body: JSON.stringify({
12 "items": [
13 {
14 "custom_id": "url_1kdjq3",
15 "url": "https://www.amazon.com/Dream-Machine-M-Mitchell-Waldrop/dp/1732265119"
16 }
17 // up to 10-100k URLs per batch
18 ]
19 })
20})
21.then(response => response.json())
22.then(result => {
23 console.log(`Batch ID: ${result.id}`);
24 console.log(`Status: ${result.status}`);
25})
26.catch(error => console.error('Error:', error));Get the data in the format you want
Get Markdown, HTML, PDF or Structured JSON
Pass the URL to the API and retrieve the HTML, Markdown, PDF, or plain text of the website. You can also specify the schema to only get the structured, clean JSON data you want
JS execution + residential ipS
Web-pages rendered in a browser
Full JS support is the norm for every request, as well as premium residential IP addresses and proxies rotation to avoid all bot detection
Crawl
Get all the data from a single URL
Multi-depth crawling enables you to get clean markdown from all the subpages of a website. Works also without a sitemap (e.g. useful for doc websites).
batch executions
Scale at another level
With Batch Executions, you can scrape 100k pages in around 5-7 minutes. You can then run up to 5 threads of Batch Executions and get 500k scrapes in the same amount of time. That's 1 million requests in around 15 minutes.
We handle the heavy lifting
Rotating proxies, solving CAPTCHAs, rate limits and js-rendered content
Crawling
Get the data from all subpages of a website. No sitemap required. This is useful if you are building an AI agent that need to get a specific context from a documentation website
Batches
You can submit from 100 to 100k URLs in a batch and have the content (markdown, html, raw pdfs or structured JSON) back in 5-7 mins. Useful for price tracking services, monitoring social media, and for aggregating data at scale
Reliable
Get the content you want when you want it. All requests are done with a premium proxy
PDF parsing
Olostep can parse and output content from web hosted pdfs, docx, and more.
Actions
Click, type, scroll, wait and more before extracting content.
Parsers: extract JSON
The Web is your API. Use pre-built parsers
for the most common websites or pass your own to the API
Extract search results, knowledge panels, and featured snippets from Google search results
Extract search results, suggested queries, and web page summaries from Brave results
Extract posts, comments, upvote counts, and user information from discussions
Extract user details, follower counts, post metrics, and content from Instagram profiles
Extract product details, pricing, specifications, and reviews from listings
Extract reviews, business information and location information from Google Maps
Request any parser you want on our Slack →
Transparent Pricing
Start for free. Scale with no worries.
We want you to be able to build an actual business on top of Olostep
Plans get cheaper at scale (5M+ requests). To learn more, contact us: info@olostep.com
Frequently asked questions
Have other questions? Get in touch via info@olostep.com
We only charge for successful requests. We are building and improving fallback systems to retry failed requests internally and return the results
Yes, the API can scale up to tens of millions of requests per day
You can pay using the Stripe Payment Links.
You can scrape data from any website that is publicly accessible
Because it's fast, reliable and cost-effective (up to 90% cheaper than existing solutions). You can request the features you need and we will try to build them for you. Plus you can test it for free to see if it fits your need. Get your free API keys from here.
Yes, plans are pro-rated, meaning if you've already paid for a previous plan, the remaining credits will be transferred to your next plan. You won't have to pay again for what you've already covered.
Yes. Just click on the "Get free test API keys". We'll provide you with some test API keys and help you get up and running so you can test the product until you love it.
We’re fully committed to building products that you love. If for whatever reason you’re unsatisfied with the Olostep API, please email us at info@olostep.com to receive a full refund within a few hours. We'll also refund you if it doesn't turn out being useful. If you decide to use it but only after a certain period of time, we'll refund the time you don't use it.
The API returns the id of the request (for future retrievals), the Markdown and the HTML of the page. You can also retrieve JSON with specific parsers or structured data with LLM extraction.
Olostep can be used by anyone who needs to interact and retrieve data from the web in a fast, scalable and cost-effective way. It's especially useful for startups building in the AI space like AI agents, LLMs fine-tuned on a specific dataset or companies that need to enrich data or do price/event tracking.
