Autonomous Mobile Agents: ukkin's Architecture for On-Device AI

This article was revised in October 2026 to describe ukkin as its repository documents it, and to remove an architecture, permission model and timings it does not implement.

What Does an Agent Look Like on a Phone?

The word “agent” in AI has come to mean many things. In the context of ukkin, it means something specific: a piece of software on a phone that runs a workflow in the background, reads what other apps show on screen, and acts on them the way a user would — tapping, typing, scrolling, switching apps. The environment is the phone itself. The actions are the things a user might do.

This is a different proposition from cloud-based AI agents. A cloud agent has access to powerful models and essentially unlimited compute. It also requires the user’s data — screen contents, personal messages, browsing history — to reach a remote server. A phone-side agent operates under real resource constraints but can keep that data on the device. The trade-off is between capability and privacy, and ukkin is an exploration of where it lands in practice.

What ukkin Is

ukkin is an open-source, MIT-licensed Flutter app for building personal agents on a phone. You describe what you want automated in plain English — “watch for price drops on items in my wishlist and notify me”, “every morning, summarise my unread emails and flag anything urgent” — and a conversational agent builder extracts the workflow, shows you the steps, and creates an agent that runs in the background.

The project is explicit that it is experimental, and explicit about platform support. Android is where it works today. Screen automation runs through the Android Accessibility Service, which the user must enable in system settings before any agent can read screens or interact with apps. iOS automation is not implemented; it sits on the project’s roadmap. Anyone evaluating ukkin on an iPhone should know that before they start.

Architecture

ukkin is built from two layers and a set of agents on top of them.

AgentLib: The Agent Primitives

AgentLib is a standalone Dart SDK for mobile agents. It defines the core abstractions:

• Agent, TaskAgent and RepetitiveTaskAgent — the agent types, with RepetitiveTaskAgent covering background work that runs on a schedule.
• TaskScheduler — coordinates agents according to device state.
• AppPluginSystem — modular, per-app integrations, so knowledge of how a particular app is laid out lives in one place.

Scheduling is device-aware: agents can be set to run only on Wi-Fi, only while charging, at particular times, or when the app they need is available. On a phone this is not a nicety. An agent that drains the battery or burns mobile data is one the user will uninstall, however clever it is.

RealAutomation: The Accessibility Bridge

RealAutomation is the bridge to the Android Accessibility Service. It does the concrete work: reading the screen’s accessibility tree, finding elements, launching apps and navigating between them, and performing taps, typing, scrolls and swipes.

The accessibility tree is the right foundation for screen understanding because it is semantic. It names buttons, labels text fields and exposes list items, so an agent does not have to infer from pixels that a rectangle is a Send button. Its weakness is that it is only as good as the app’s accessibility support, and the repository also contains visual-language-model code for screen understanding where structure alone is not enough.

The Agents

On top of those layers sit the agents themselves. The repository ships specialised ones — an Instagram watcher for posts and followers, an email triage agent that classifies and organises mail, and a price watcher that extracts and tracks prices with history — and new agents can be created from a conversation or written as code extending RepetitiveTaskAgent.

Where the Model Runs

This is the part to be precise about. ukkin’s design intent is local-first: its README states that AI runs on the device and data does not leave the phone. Its roadmap, however, lists an on-device LLM for natural-language understanding as a phase still to come, and the project’s own overview notes that some build configurations use cloud LLMs for the conversational agent builder. The honest summary is that local-first is the goal and the privacy story, and that you should verify the configuration you are running rather than assume it.

On-device inference in a Flutter app is exactly the problem llamafu addresses, which is why the two projects sit next to each other. ukkin’s documentation does not name a specific inference runtime, so we do not claim an integration.

The Autonomy-Safety Spectrum

The central design question for any agent is: how much should it be allowed to do without asking?

At one extreme, the agent does nothing without explicit approval for every action. This is safe but not useful — it reduces the agent to a suggestion engine that still requires the user to perform every tap.

At the other extreme, the agent acts fully autonomously, making purchases, sending messages and changing settings without confirmation. This is useful in theory and dangerous in practice. A misunderstood instruction or a reasoning error could send a message to the wrong person, make an unintended purchase or delete data.

The workloads ukkin targets today sit deliberately at the cautious end. Watching prices, monitoring mentions, classifying email and drafting responses for review are observation and preparation tasks; the human still owns the irreversible step. That is the shape we think phone-side agents should keep until their reasoning is reliable, and it suggests a principle worth stating plainly: classify actions by reversibility, let agents act freely on what can be undone, and require confirmation for anything that sends, buys, deletes or publishes. ukkin does not yet implement a formal permission tier system along those lines; it is the design direction we would argue for, not a feature to rely on.

Comparison with Cloud-Based Agents

Cloud-based agents have clear advantages: larger models, longer context, more sophisticated reasoning, and easy access to web APIs and external tools. For complex tasks that need deep understanding, long-horizon planning or information beyond the device, they are the better choice.

A phone-side agent’s advantages lie elsewhere. When the model runs locally, screen contents and personal data need not leave the device. The agent can act on the apps the user already has, with the user’s own sessions, without asking them to log in to a hosted browser. It can run offline for the parts of a workflow that do not need the network. And it costs nothing per inference.

The practical upshot is that the two serve different use cases. Phone-side agents suit routine, repetitive, privacy-sensitive automation inside apps the user already uses. Cloud agents suit open-ended tasks that need world knowledge, complex reasoning or web services.

Current Limitations

Platform. Android only, today. iOS restricts background operation and accessibility-driven control of other apps, and ukkin’s iOS automation is on the roadmap rather than in the app.

App compatibility. Not every app exposes useful accessibility information. Custom-rendered interfaces, canvas-based UIs and content inside web views the accessibility tree cannot read cleanly are hard for any accessibility-driven agent.

Presence. Many UI flows need the device unlocked, so “background” automation is less invisible than the word suggests. Captchas stop it outright, as they are designed to.

Brittleness. An agent that navigates by finding named elements breaks when an app’s layout changes. Per-app plugins localise that breakage, but they do not prevent it.

No published measurements. We have not published timing, reliability or battery figures for ukkin agents. How long a workflow takes, how often it completes and what it costs in battery are questions to measure on a named device with a named set of apps, not to estimate.

Where This Is Going

ukkin is an early-stage exploration, and its roadmap is candid about the order of work: on-device language understanding and agent composition next, then cross-app workflows, calendar integration and voice or notification triggers, and only then iOS automation and a desktop companion.

The architecture and the trade-offs it embodies are worth sharing now, because the design decisions being made for phone-side agents — what runs locally, what may be done without asking, and how honestly the limits are described — will shape what these agents look like when they mature. We believe local-first inference is the right default for personal devices, that reversibility is the right axis for deciding what an agent may do unsupervised, and that saying exactly which platform works today is worth more than a demo that implies more.