Mobile Tunnel: Giving AI agents control of Android devices
By Ilia Manolov and Eito Miyamura
Overview
We got an AI agent to pilot a drone autonomously from just a phone, from a single high-level prompt, with minimal tuning. A few weeks ago we introduced stdio tunnels, which let cloud AI agents securely reach MCP servers running on your own computers. As a week-long experiment we wanted to take that concept to the extreme. So we gave GPT-5.6 Sol control of a £100 DJI Neo drone and asked it to find and land on a table, and it did.
Today we'd like to introduce Mobile Tunnel, an open-source Android companion for SealGate that lets an AI agent observe and interact with an Android device remotely - and through it, any hardware the phone can reach.
Once we had agents talking to computers through the SealGate Gateway, the obvious next question was: what about everything else?
Phones can already talk to an enormous range of hardware over Wi-Fi, Bluetooth, USB, NFC, and vendor-specific apps. If we could make an Android phone another location for a SealGate stdio tunnel, an agent could interact with any device the phone can reach.
So we spent about a week experimenting with exactly that.
From controlling computers to controlling devices
Our gateway has long been able to accommodate remote MCP servers, and we recently created stdiod - a daemon that gives SealGate a secure route to MCP servers running on a computer. That already enables quite a lot: cloud agents can interact with local files, desktop applications, Beeper, development environments, or anything else exposed through a local MCP server.
But computers are only one category of device.
Around the office we had a Flipper Zero, a 3D printer, various embedded devices, and a DJI Neo. The drone was particularly interesting because it combines several problems we wanted to experiment with:
- It is controlled through another device rather than exposing a useful public API.
- Successful control requires continual visual feedback.
- Actions have to be composed over time rather than issued as a single API call.
- A human normally has to learn a new control interface before they can use it effectively.
That last point is the interesting one. Modern multimodal models can already look at a UI, reason about what they see, and interact with it. In theory, instead of teaching every user how to pilot a particular device, we could teach an agent once and let the user describe the outcome they want:
"Go do a reconnaissance, identify a table and land on it"
"Identify a black bin and fly towards it"
"Fly under the bridge and land on the balcony"
The question was whether the existing pieces were good enough to actually do it.
How do you connect an agent to a DJI Neo?
The DJI Neo gives us two practical control paths: a dedicated remote controller, or the DJI Fly Android app. We initially wanted to use the remote.
Option 1: the DJI RC-N3 (the road not taken)
Using an RC-N3 would have had several major advantages. The remote gives the Neo kilometres of potential transmission range rather than the roughly 50 metre range of direct smartphone control. It also gives deterministic physical control inputs and a direct video feed. The ideal architecture would have been:
We could run a custom MCP server next to stdiod, translate MCP calls into controller inputs, and feed the drone's camera back to a multimodal agent.
Unfortunately, there was one problem: we needed to speak to the controller, and the USB protocol between the DJI RC-N3 and its host is proprietary. We couldn't find a sufficiently complete open implementation to build on.
We briefly investigated reverse engineering it ourselves. We built a small Android app to inspect the USB traffic between the DJI app and controller, but the communication was sufficiently obfuscated that reproducing the protocol would have turned our one-week experiment into a substantially larger reverse-engineering project. We also considered letting the agent manipulate the Android UI while leaving the RC-N3 connected, but that didn't help much either: the important navigation inputs are handled directly by the physical controller rather than being exposed as usable on-screen controls.
So we moved on.
Option 2: just use the Android app
The DJI Neo can also connect directly to an Android phone over Wi-Fi and expose virtual joysticks through DJI Fly. That architecture is much simpler: there is no DJI protocol to reverse engineer, and the agent just needs to use the phone the same way a human would. It observes the screen, decides what it wants to do, taps or swipes the virtual controls, observes the result, and repeats.
There are some tradeoffs. DJI specifies an effective range of up to around 50 metres for direct smartphone control, versus kilometres with a dedicated controller. The phone also has to remain awake and on the DJI Fly piloting screen while the agent is using it. For this experiment neither was particularly problematic - current multimodal agents are slow enough that we were unlikely to send the drone very far during a single battery anyway.
There was only one missing component: we needed stdiod for Android. So we built it.
Mobile Tunnel: stdio tunnels for Android
Mobile Tunnel is an MVP Android implementation of the device-side functionality behind SealGate stdio tunnels. Once configured, the phone becomes another device an agent can reach through the SealGate Gateway.
Depending on the MCP server and Android hardware involved, this opens up several kinds of interaction:
- Observe the current state of the phone screen.
- Perform taps and swipes.
- Type using the virtual keyboard.
- Communicate with external hardware over Bluetooth.
- Communicate with devices attached over USB.
- Run device-local workflows without exposing the phone directly to the internet.
For the DJI experiment, we expose tools letting the model inspect the current screen and perform touchscreen actions. DJI Fly itself remains completely unmodified. From the application's perspective, it is still being operated by somebody looking at the screen and moving the virtual joysticks. That "somebody" just happens to be an LLM running on another computer.
The Mobile Tunnel app is open source, with reproducible Android builds available from our GitHub.
Teaching an LLM to fly
Giving the model access to the controls did not immediately make it a competent pilot. A human looking at two virtual joysticks already has a useful mental model of how a drone works. An LLM doesn't necessarily start with the same mapping between a particular swipe and the resulting movement of the aircraft.
We therefore built a small set of skills describing the primitive actions available to the model and iteratively calibrated them against the real drone. Those included:
- 3D movement: forward, backward, left, right, up, and down.
- Left/right yaw.
- Camera movement.
- Observing the current scene.
- Performing a systematic reconnaissance pass over an area.
The important part was making these primitives composable. The model doesn't need a special land_on_wooden_table function. It needs to be able to look at the image, identify the table, reason about its current position relative to it, and repeatedly combine basic movement and observation actions until it is in the correct place. In other words, the MCP server provides the motor primitives and the skill documentation; the model provides the planning.
There is plenty more calibration work that could be done here, but after a few rounds the resulting harness was good enough for the experiment we actually cared about.
"Land on the wooden table"
For the main test we connected a DJI Neo to an Android phone running Mobile Tunnel. On the agent side we ran Codex CLI with GPT-5.6 Sol at high reasoning effort, connected through SealGate to the phone. We then gave it a high-level instruction: land the drone on the wooden table.
There was no prerecorded trajectory for the task. Sol had to observe the camera feed, reason about where the table was, manipulate the virtual flight controls, check the result, correct its position, and eventually land.
The interesting bit for us is not specifically that an LLM can fly a DJI Neo. Drone autonomy is a very large field, and specialists will build far more sophisticated control systems than our week-long experiment. The interesting bit is that we didn't need a DJI integration to do it. We gave an agent secure access to a general-purpose Android device, taught it how to manipulate an existing application, and the phone provided the bridge to the physical hardware. The same pattern applies to many other devices which already have an Android application but will probably never ship an MCP server.
Which models could actually do it?
We tried the same setup with a collection of current frontier models, in both text and voice mode where available: GPT-5.6 Luna, GPT-5.6 Sol, Claude Opus 4.6, Claude Opus 5, Claude Haiku 4.5, Grok 4.6, and Muse Spark.
We observed two primary failure modes: models either struggled to maintain spatial consistency across observation-action cycles, or were overly conservative and refused to perform the task despite the low-risk, tightly supervised environment.
Getting a model to actually attempt the flight took careful phrasing to avoid false-positive AI safety responses. We refer to it as a sim drone rather than a drone, we assure the agent that it is in a safe space, and we make it clear that it is taking part in an experiment rather than acting in a live environment. We found it essential to explicitly communicate that a human pilot was present and able to intervene. Even so, we could not get any of the Claude models to pass these AI safety checks - they consistently assumed they were flying a dangerous drone and refused - and Muse Spark would sometimes refuse as well.
Voice mode narrowed the field further. Gemini was out of scope due to a lack of official MCP support. ChatGPT's voice mode does not support MCP tool calls, so that was out of scope too. Claude's voice mode does support MCP tool calls, but its transcription was poor enough that the models failed to understand our instructions about half the time. Grok's voice mode appeared to use a more streamlined model that lacked the multimodal intelligence to operate the drone correctly.
Only Sol in text mode passed our tests consistently. The others either refused or struggled with the same basic tasks.
| Model | Outcome | Notes |
|---|---|---|
| GPT-5.6 Sol | Success | Reliably completed the flight tasks in text mode. |
| GPT-5.6 Luna | Struggled | Could not maintain spatial consistency reliably. |
| Claude (Opus 4.6, Opus 5, Haiku 4.5) | Refused | Refused the flight task despite the safety context. |
| Muse Spark | Inconsistent | Passed some runs, refused others. |
| Grok 4.6 | Struggled | Could not maintain spatial consistency. |
Please note that this is not a general-purpose benchmark: the sample size is small, and performance could likely be improved with further prompt and skill tuning. It is nevertheless a useful reminder that giving an agent access to a tool is only half of the problem. The model still needs to be capable of reasoning reliably over the resulting action loop.
Limitations / work required
Like the original stdio tunnels, Mobile Tunnel is still an early implementation of what we want device connectivity through SealGate to become. There are several obvious limitations:
- Android support is currently experimental.
- UI-control workflows depend on the layout and behaviour of the application being controlled.
- Multimodal control loops are substantially slower than native control software.
- The quality of physical-device control depends heavily on the skills and calibration given to the model.
- The phone has to remain available to the application being controlled; our DJI experiment kept DJI Fly in the foreground throughout the flight.
- Direct DJI Neo smartphone control is limited to roughly 50 metres.
- The DJI Neo does not provide obstacle avoidance, so autonomous experiments require a deliberately controlled environment and active human supervision.
- Different models behave very differently when asked to control physical devices.
Substantially better drone-specific skills could of course be built, and we'll leave that part to drone specialists and prompt engineers. Our goal on the SealGate side is to provide the secure infrastructure underneath it.
Why we're excited about this
Mobile Tunnel changes the scope of what a "local MCP" can represent. A local MCP no longer has to mean software running on your laptop. It can mean a Bluetooth device connected to your phone, a USB peripheral, a piece of equipment with a vendor Android app, an embedded device reachable over the local network - or, apparently, a drone.
There are millions of physical devices which will never expose an MCP interface themselves but can already be operated from Android. Rather than requiring every manufacturer to build an agent integration, Mobile Tunnel gives us a generic bridge between those existing interfaces and MCP-capable agents. And because that bridge still passes through SealGate, organisations can retain a security and policy layer between the agent and the device it is controlling.
Takeaway
stdio tunnels started as a way to let cloud agents reach software running on your computer. Mobile Tunnel extends the same idea to Android, letting agents securely interact with phones and, through them, physical devices over Wi-Fi, Bluetooth, USB, or existing mobile applications.
As a slightly excessive demonstration, we connected GPT-5.6 Sol to a DJI Neo, taught it the basic flight controls, and asked it to land on a table. But the drone isn't really the point. The point is that once an agent can securely reach a general-purpose device, it can start interacting with the much larger ecosystem of hardware already attached to it.
If you want to experiment with similar workflows, you can try the SealGate MCP Gateway at sealgate.ai and find our open-source components on GitHub. Or, if you're less interested in autonomous drones and more interested in making sure the agents already appearing across your company can access data and tools safely, that's what we're building SealGate for too.
References
- stdio tunnels: Enabling Cloud Agents to reach local MCPs
- SealGate open-source components
- DJI Neo
- UK CAA Drone and Model Aircraft Code
All flight testing was performed with a DJI Neo under active human supervision, within visual line of sight, in an area where the flight was permitted, and in accordance with the applicable UK drone rules.