ESP32 iPhone Agent: A Roughly $7 ESP32 as the "Hands" and AirPlay Mirroring as the "Eyes" Let a Vision Model Operate a Real iPhone

onehumanbeing/esp32-iphone-agent is an open-source project written in Python under the MIT license. It was created on October 11, 2026, last pushed on October 13, and has about 1,517 stars and 194 forks at the time of writing. The pitch is one line: let an AI model use a real iPhone with a roughly $7 ESP32, with no Mac, no developer account and no jailbreak. Demo videos and updates are on the author's X account and on YouTube. The README gives two examples. In the first, the task is "Open Whiteboard and draw a smiley face", and the agent searches for the app, opens it and draws the face with continuous strokes. In the second, the task is "Set an alarm for 7:00 AM that repeats every day, then save it", and the agent opens Clock, types 0700 into the time wheel, ticks the days under Repeat several taps at a time, and saves. Every step's screenshot, with its tap or stroke drawn on top, stays on a "step wall".

Why take this route? The README explains that most iPhone automation drives the phone through XCTest or WebDriverAgent. That needs a Mac, Xcode, a signed helper app and an Apple developer account, and a free account's signature expires every 7 days. This project uses the same paths a person uses, the screen and a pointer, so the phone needs nothing installed. A comparison table in the README says this approach needs no developer account and no Xcode, runs on a Mac or Linux, installs no app on the phone, and works in any app, including games, because it only sees pixels. XCTest-type tools need a developer account and Xcode, run only on a Mac, need a test runner app, and mostly depend on what the accessibility tree shows.

Technically it has three parts. The eyes are AirPlay mirroring: the iPhone mirrors its screen to UxPlay on your computer, and a GStreamer pipeline turns the stream into JPEG frames for the agent and the web page. The hands are a USB HID bridge: an ESP32-S3 plugged into the phone shows up as a standard USB mouse and keyboard. With iOS AssistiveTouch on, the mouse moves a round on-screen pointer, so a click is a tap and a drag is a swipe. The board connects to the computer over Wi-Fi and runs simple line commands such as move, click and type, answering ok or err, and all logic stays on the computer. The brain is a planner and an actor. The planner writes a short plan from the goal and the current screen. The actor looks at each new screenshot and picks tap, swipe, draw a path, type, press a key, go home or finish. Both roles can be the same model, and any OpenAI-compatible model that accepts images works, from a cloud API or one you run yourself.

What makes this path workable is the closed loop. The pointer is relative, so the server tracks where it is. Before each tap it wiggles the pointer, finds it on the mirrored screen and corrects any miss, refining the calibration as it goes. After each action it checks whether the screen really changed and tells the model, so a missed tap is noticed and retried instead of built upon. Repeated actions trigger a re-plan, and a dropped link or a bad reply can be recovered from. The web UI shows the live phone, where a click taps and a drag swipes, a live log, the step wall with each screenshot and the model's reasoning, and an emergency stop. Esc also stops the agent and releases the pointer. With HIDAGENT_DEBUG=1 each agent run also saves a folder with steps.jsonl, which holds every model call and action, and the screenshot each call saw.

The hardware needs are small. You need an ESP32-S3 board. The author tested the plain Seeed Studio XIAO ESP32S3, without pre-soldered headers, not the Sense camera version and not the Plus, at about $7 from Seeed, and any ESP32-S3 board whose USB-C port is wired to the S3's native USB works. USB-C iPhones plug in directly, and Lightning models need a Lightning to USB Camera Adapter. The computer runs macOS or Linux on the same network as the phone and runs UxPlay and the server. For a fully local setup, the author's development configuration uses two NVIDIA DGX Sparks with tensor parallelism (TP=2) to serve nvidia/Qwen3.8-Flash-Next-NVFP4, at about 1.3 seconds per action and 2 seconds per plan with thinking mode off.

A few points for readers. First, all of this is the project's own description and demos. The README gives no task success rate, failure rate or evaluation over many tasks, so do not read a demo as proof of reliability. Second, the agent controls a whole phone, which means it might reach your email, bank and messaging apps, and with a cloud model every screenshot goes to the model provider. Use a test phone, do not sign in to important accounts, and keep the emergency stop within reach. Third, it lowers the bar for iPhone automation, and it also lowers the bar for controlling a phone through physical access, which is a convenience for testers and a risk that security people must consider. Overall it is a clever small project worth trying, but it is very early.