You Only Look at Screens: Multimodal Chain-of-Action Agents Article Swipe

MAKE IMPACT

Zhuosheng Zhang , Aston Zhang · YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2309.11436

Autonomous graphical user interface (GUI) agents aim to facilitate task automation by interacting with the user interface without manual intervention. Recent studies have investigated eliciting the capabilities of large language models (LLMs) for effective engagement in diverse environments. To align with the input-output requirement of LLMs, most existing approaches are developed under a sandbox setting where they rely on external tools and application-specific APIs to parse the environment into textual elements and interpret the predicted actions. Consequently, those approaches often grapple with inference inefficiency and error propagation risks. To mitigate the challenges, we introduce Auto-GUI, a multimodal solution that directly interacts with the interface, bypassing the need for environment parsing or reliance on application-dependent APIs. Moreover, we propose a chain-of-action technique -- leveraging a series of intermediate previous action histories and future action plans -- to help the agent decide what action to execute. We evaluate our approach on a new device-control benchmark AITW with 30$K$ unique instructions, spanning multi-step tasks such as application operation, web searching, and web shopping. Experimental results show that Auto-GUI achieves state-of-the-art performance with an action type prediction accuracy of 90\% and an overall action success rate of 74\%. Code is publicly available at https://github.com/cooelf/Auto-GUI.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2309.11436

PDF

https://arxiv.org/pdf/2309.11436

OA Status

green

Cited By

3

Related Works

10

OpenAlex ID

https://openalex.org/W4386976823

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4386976823

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2309.11436

Digital Object Identifier

Title

You Only Look at Screens: Multimodal Chain-of-Action Agents

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2023

Year of publication

Publication date

2023-09-20

Full publication date if available

Authors

Zhuosheng Zhang, Aston Zhang

List of authors in order

Landing page

https://arxiv.org/abs/2309.11436

Publisher landing page

PDF URL

https://arxiv.org/pdf/2309.11436

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2309.11436

Direct OA link when available

Concepts

Computer science, Action (physics), Parsing, Benchmark (surveying), Task (project management), Sandbox (software development), Interface (matter), Human–computer interaction, Inference, Inefficiency, User interface, Code (set theory), Artificial intelligence, Machine learning, Software engineering, Programming language, Bubble, Economics, Maximum bubble pressure method, Set (abstract data type), Management, Parallel computing, Geodesy, Quantum mechanics, Microeconomics, Physics, Geography

Top concepts (fields/topics) attached by OpenAlex

Cited by

3

Total citation count in OpenAlex

Citations by year (recent)

2025: 1, 2024: 2

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload