Qwen-UI-Agent Explained: How Alibaba Trains an AI to Use Real Screens

Alibaba’s Qwen-UI-Agent is a GUI-agent foundation model built to operate software the way a person does: by looking at screens and issuing clicks, keystrokes, text input, and swipes. It covers mobile devices, desktop computers, web browsing, and DeepSearch environments rather than depending exclusively on application programming interfaces (APIs) or internal app instrumentation. Alibaba released it publicly on August 20, 2026, after publishing its technical report on arXiv on July 30. 3 33

The central idea is practical: an agent that can use an ordinary interface may be able to automate tasks even when a service has no suitable API. But the benchmark record is more nuanced than a simple claim that Qwen-UI-Agent is best everywhere. Its reported lead is clearest on mobile, especially when the evaluation runs on physical phones.

What Qwen-UI-Agent does

Traditional software agents often interact with structured APIs, browser automation hooks, or app-specific tools. Qwen-UI-Agent instead treats the interface itself as the point of interaction. It reads what is visible, identifies relevant controls, and chooses actions such as tapping, clicking, typing, or swiping. The same model is designed to work across phones, desktops, browsers, and DeepSearch workflows. 33

Its action space also combines graphical interaction with command-line execution. That allows a long task to move between a browser, a desktop application, and a terminal rather than forcing every step through the GUI. The report describes batched actions within a model turn, which is intended to make multi-step workflows more efficient. 1

Why training on real phones matters

GUI agents can perform well in emulators yet fail on physical devices because real hardware introduces differences in screen layouts, timing, touch behavior, device state, and app behavior. Qwen-UI-Agent’s training and evaluation setup was designed to reduce that simulation-to-reality gap.

Alibaba describes a real-device mobile environment covering more than 100 physical phones and more than 150 apps. The hardware was used for task construction, trajectory collection, training, and evaluation—not merely for a final demonstration. 3 5

The project also introduced MobileWorld-Real, a benchmark containing more than 400 real-device tasks across mobile apps. Because the benchmark was created by the same project, its 92.2% result is meaningful evidence of performance on physical hardware but should not be interpreted as a fully independent audit. 1 7

How the training system supports long tasks

The technical report describes online reinforcement learning over trajectories longer than 100 turns and more than 10,000 concurrent rollout environments. It also presents an automated research loop in which agents help construct tasks and environments, diagnose failures, and plan subsequent training iterations. 1

That setup targets a problem beyond isolated button clicks: maintaining state over a long sequence of actions. A useful agent must remember what it has already done, recover from mistakes, and select the right interface or tool at each stage.

The report further describes a lightweight harness for stateful, cross-device workflows and proactive service initiation. These capabilities point toward assistants that could continue a task when a user moves from a phone to a computer, or initiate a helpful intervention instead of waiting for every instruction. Such examples demonstrate the intended direction of the system, not proof that it can safely handle every open-world task without supervision. 1

Reported benchmark results

The published technical report gives Qwen-UI-Agent its strongest results on mobile-use benchmarks. Its headline scores include: 33

• MobileWorld: 82.1%
• MobileWorld-Real: 92.2%
• AndroidDaily: 97.5%
• OSWorld-Verified: 79.5% for desktop computer use
• OSWorld-v2: 40.0% on a partial-progress metric
• WebArena: 73.6%
• ScreenSpot-Pro: 81.5% for GUI grounding

On MobileWorld, the supplied comparison reports 70.1% for GPT-5.6 Sol and 67.5% for Claude Opus 4.8, placing Qwen-UI-Agent 12.0 and 14.6 percentage points ahead, respectively. 7

The desktop and browser results require more careful interpretation. The 79.5% OSWorld-Verified score is strong, but comparisons supplied with the report place Claude Opus 4.8 higher on that benchmark. Qwen-UI-Agent’s 40.0% OSWorld-v2 figure is also a partial-progress score, not a claim that 40% of complete tasks were finished. 33 34 36

The model’s 73.6% WebArena score is supported by the supplied sources, but the evidence does not establish an uncontested first-place ranking across every comparison set. Rankings can change with the models, variants, evaluation procedures, and benchmark splits included. 1 8

What a mixed GUI-and-CLI workflow looks like

The model’s significance is not just that it can press buttons. The report describes workflows in which an agent researches financial information through browser and desktop interfaces, uses command-line tools for analysis or file operations, and then creates and saves a document in a GUI application. 1

That combination gives the agent more than one way to complete a step. It can use a visible interface when that is necessary, then switch to a terminal when structured file manipulation or analysis is more efficient. The challenge is coordinating those tools while preserving the task’s state across applications.

The same design supports the broader concept of cross-device assistance. A workflow could begin on a phone, continue on a desktop, and involve multiple applications without requiring each service to expose a specialized API. In practice, however, reliability would depend on authentication handling, recovery from unexpected screens, and clear boundaries around actions with real-world consequences.

Safety remains the hard part

An agent that can control a screen can also reach sensitive data, delete files, submit forms, or initiate transactions. The appropriate safety pattern is to refuse illegal or high-risk requests, ask for missing information, and obtain confirmation before sensitive actions such as payments, deletion, or privacy authorization.

The supplied primary technical-report evidence does not independently verify each specific refusal and confirmation demonstration described in the original request. Those behaviors should therefore be treated as reported or intended controls rather than established proof of safe deployment.

Benchmark scores also do not show that the system is ready to operate unattended in ordinary users’ accounts. Real deployments would need permission boundaries, confirmation gates, audit logs, interruption controls, error recovery, and independent testing on diverse devices and applications.

What Qwen-UI-Agent means for computer-use AI

Qwen-UI-Agent’s clearest contribution is its emphasis on the physical interface. Training with more than 100 phones and evaluating on real hardware makes the mobile results more relevant to device control than emulator-only testing. Its unified GUI-and-CLI action space also suggests a practical architecture for longer workflows that cross apps, browsers, desktops, and terminals. 1 3

The evidence supports a strong conclusion on mobile: Qwen-UI-Agent is a notable real-device GUI agent with leading reported results on several mobile benchmarks. It supports a more cautious conclusion elsewhere: desktop and browser performance is competitive, but not uniformly superior to every frontier model or benchmark result.

The next test is not whether an agent can complete a scripted benchmark. It is whether it can perform useful everyday work while remaining predictable, interruptible, transparent, and safe when the screen changes or the consequences become irreversible.