Crawlee

Crawlee is a web scraping library for JavaScript and Python that handles blocking, crawling, proxies, and browsers.

Crawlee is a web scraping and browser automation library designed for JavaScript and Python developers. It helps build reliable web scrapers by managing proxies, crawling queues, blocking issues, and headless browsers automatically. Developers use it to extract data, download files, and process web pages using tools like Playwright. The library saves results as JSON or exports datasets directly to files like CSV. It is built by Apify and is provided as a free and open-source tool.

Crawlee at a glance

Best for: Software engineer
Runs on: Web, CLI, self-hosted, open source
Pricing: Forever free and open-source
Released: August 2016
Category: AI search engine
Website: crawlee.dev

Key features

Manage headless browsers automatically
Handle proxies and bypass blocking
Queue URLs and enqueue links
Push and store extracted data
Export datasets to CSV format
Support JavaScript and Python

Use it for

Extract data from websites for LLMs and AI models
Build reliable web scrapers and crawlers
Automate browser tasks and navigate pages
Download files and capture web content

Alternatives to Crawlee

Scrapling — It uses adaptive element tracking to maintain scrapers when websites change.
Crawl4AI — Turns any URL into clean Markdown or typed JSON for AI models.
Spider — It streams clean markdown or JSON from any website in realtime with a single API key.
Firecrawl — Provides direct web data access for AI agents with open source flexibility.
AnyCrawl — Provides clean, structured, and LLM-ready data from any website.
Scrape Anything by Context.dev — Extracts clean Markdown, HTML, and structured data with built-in proxies and JS rendering.
ScrapeGraphAI — Extracts structured data via natural language prompts with no selectors or proxies.
LLM Scraper — Extracts structured data from any webpage using Zod or JSON Schema.
Google Browser Scraper — It uses a real browser and sticky proxy to prevent false empty results.
Anakin — Zero-touch keyless start lets agents make unauthenticated HTTP calls immediately.
XERJ — It answers agent questions from an index instead of wasting tokens on context windows.
jurl — Extracts exact page content without rewriting or hallucinating answers.

All Crawlee alternatives