Trafilatura

Trafilatura is a Python package and command-line tool to gather text and metadata from the web.

Trafilatura is a Python package and command-line tool designed to gather text from the web and turn raw HTML into structured data. It performs web crawling, downloads, scraping, and extraction of main texts, metadata, and comments without requiring a database. The extractor balances precision and recall to focus on actual content while avoiding noise from headers and footers. It supports sitemaps, feeds, live URLs, and offline HTML files, with outputs in TXT, Markdown, CSV, JSON, HTML, XML, and XML-TEI. It is widely used in academic and research projects for data acquisition and text extraction.

Trafilatura at a glance

Best for: Software engineer, Researcher
Runs on: Python, CLI, open source
Pricing: Free
Released: April 2019
Category: AI search engine
Website: github.com/adbar/trafilatura

Key features

Web crawling and text discovery
Support for sitemaps and feeds
Parallel processing of input
Extraction of main text and metadata
Optional comments and tables extraction
Multiple output formats
Language detection add-on

Use it for

Gathering text and metadata from the web
Converting raw HTML into structured data
Processing online and offline input files
Building text databases for research

Alternatives to Trafilatura

LangExtract — It maps every extraction to its exact location in the source text.
Pagesift — Extracts web data with high memory efficiency and speed for AI agents.
Firecrawl — Provides direct web data access for AI agents with open source flexibility.
Markdownee — Extracts main web content cleanly using Trafilatura Core and Crawlee.
Jina — It provides reliable reader and search APIs optimized for LLMs and RAG systems.
Tavily — It delivers fast, secure web data retrieval designed specifically for AI agents.
jurl — Extracts exact page content without rewriting or hallucinating answers.
ScrapeGraphAI — Extracts structured data via natural language prompts with no selectors or proxies.
LLM Scraper — Extracts structured data from any webpage using Zod or JSON Schema.
Crawl4AI — Turns any URL into clean Markdown or typed JSON for AI models.
AnyCrawl — Provides clean, structured, and LLM-ready data from any website.
Maxun — Extracts web data using a no-code recorder or AI prompts with built-in stealth mode.

All Trafilatura alternatives