Apache Hamilton

Apache Hamilton is a general-purpose framework that uses regular Python functions to define, execute, and visualize dataflows as directed acyclic graphs.

Apache Hamilton is a general-purpose framework for writing dataflows using regular Python functions where parameters indicate dependencies. The system automatically connects these functions into a Directed Acyclic Graph that users can execute, visualize, and monitor. It separates transformation logic from execution, enabling deployment on specialized computation engines like Spark, Ray, and Dask. Developers use the provided UI to track lineage, catalog artifacts, and inspect execution plans without running the code. The library integrates with tools like Pandas, Polars, MLFlow, and various cloud platforms to support tasks such as feature engineering, model training, and LLM workflows. It helps teams organize codebases, facilitate collaboration, and maintain unit testing standards for data-intensive projects.

Apache Hamilton at a glance

Best for: Software engineer, Researcher
Runs on: Linux, macOS, Windows, VS Code, CLI, open source
Pricing: Free
Released: February 2023
Category: AI coding tool
Website: hamilton.apache.org

Key features

Automatically constructs Directed Acyclic Graphs from Python functions
Supports unit testing for data transformation code
Includes a visualization UI to inspect dataflows and lineage
Separates transformation logic from execution infrastructure
Integrates with distributed computing engines like Ray, Dask, and Spark
Enables tracking and monitoring of data artifacts
Provides CLI tools for management and interaction

Use it for

Building feature engineering pipelines for machine learning
Orchestrating asynchronous LLM workflows
Developing time-series forecasting models
Processing IoT and telemetry data
Constructing financial decision engines

Alternatives to Apache Hamilton

Metaflow — Scales projects easily from laptops to the cloud with infrastructure integration.
DataFlow — Automates and scales data preparation through reusable, operator-based pipelines.
DeerFlow — Executes long tasks using sandboxes, memories, tools, and subagents.
DataChain — Processes files in S3, GCS, and Azure at warehouse speed.
Genkit — Provides unified APIs and local developer tools for AI application workflows.
Burr — It offers observability and state management for complex AI agents.
Marvin — Provides simple interfaces, smart defaults, and structured data extraction.
Pydantic AI — Provides type safety, structured outputs, validation, and durable execution for reliable agents.
Flyte — Resumes failed tasks automatically from the exact point of interruption.
daidocs — It keeps code graphs as plain text for lower token costs and better accuracy.
SwanLab — Tracks training metrics and system performance with minimal code integration.
MLflow — Complete observability and evaluation for production AI apps.

All Apache Hamilton alternatives