whereismyai
Esp32 Iphone Agent
Esp32 Iphone Agent is an open source agent that lets any OpenAI-compatible vision model control an iPhone using an ESP32 board and AirPlay.
Esp32 Iphone Agent is an open source system that enables AI models to operate a real iPhone using a cheap ESP32 board and AirPlay screen mirroring. It uses UxPlay and GStreamer to stream the phone screen to a server as JPEG frames, while a small ESP32-S3 board plugs into the phone's USB port to act as a USB mouse and keyboard via iOS AssistiveTouch. A planner model creates a plan based on the goal and screen, and an actor model executes actions like tapping, swiping, typing, and drawing. The server tracks pointer positions, calibrates before taps, and detects missed actions to retry. It is designed for developers and users who want to automate iPhone tasks without Xcode or paid developer accounts. Supported inputs and outputs include any OpenAI-compatible vision model, running either in the cloud or locally.
Esp32 Iphone Agent at a glance
Key features
- AirPlay screen mirroring to capture phone display
- USB HID bridge for mouse and keyboard control
- OpenAI-compatible vision model integration
- Automated pointer calibration and error correction
- Web UI for manual control, live logs, and step walls
- Emergency stop and action history clearing
Use it for
- Automating tasks inside mobile applications
- Drawing shapes and lines on digital whiteboards
- Configuring device settings like alarms and Wi-Fi
- Testing user interfaces across various apps
Alternatives to Esp32 Iphone Agent
- iphone-use — Operates real iPhones via HTTP and MCP without touching the host screen.
- Iphone Use App — Runs directly on the iPhone with no computer needed at run time.
- agent-device — Token-efficient snapshots and semantic references for native mobile and TV apps.
- Esp32 Thinks — Runs your own AI stack on low-cost hardware without vendor lock-in.
- CallMeMCP — It connects running artificial intelligence agents directly to phone calls.
- PhoneClaw — It executes AI agent tasks and system tool calls entirely on-device.
- Anan Iphone Use — Enables physical iPhone control via USB and Bluetooth.
- Composio — Equips AI agents with tools, managed user auth, and sandboxes across 1,500+ apps
- Phone Harness — Connects Claude Code or Codex to real iOS and Android devices.
- Mac Computer Use — Enables local desktop automation through native Mac windows and Accessibility controls.
- sim-use — Provides token-efficient screen outlines and alias-cached taps for fast agent loops.
- OpenMinis — Runs an on-device Linux shell, browser automation, and deep system integrations.
All Esp32 Iphone Agent alternatives