TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agents Article Swipe
Bofei Zhang
,
Z. J. Shang
,
Gao Zhi
,
Wang Zhang
,
Rui Xie
,
Xiaojian Ma
,
Tao Yuan
,
Xingxing Wu
,
Song‐Chun Zhu
,
Qing Li
·
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2504.12679
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2504.12679
Building Graphical User Interface (GUI) agents is a promising research direction, which simulates human interaction with computers or mobile phones to perform diverse GUI tasks. However, a major challenge in developing generalized GUI agents is the lack of sufficient trajectory data across various operating systems and applications, mainly due to the high cost of manual annotations. In this paper, we propose the TongUI framework that builds generalized GUI agents by learning from rich multimodal web tutorials. Concretely, we crawl and process online GUI tutorials (such as videos and articles) into GUI agent trajectory data, through which we produce the GUI-Net dataset containing 143K trajectory data across five operating systems and more than 200 applications. We develop the TongUI agent by fine-tuning Qwen2.5-VL-3B/7B models on GUI-Net, which show remarkable performance improvements on commonly used grounding and navigation benchmarks, outperforming baseline agents about 10\% on multiple benchmarks, showing the effectiveness of the GUI-Net dataset and underscoring the significance of our TongUI framework. We will fully open-source the code, the GUI-Net dataset, and the trained models soon.
Related Topics
Concepts
No concepts available.
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2504.12679
- https://arxiv.org/pdf/2504.12679
- OA Status
- green
- OpenAlex ID
- https://openalex.org/W4415320065
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4415320065Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2504.12679Digital Object Identifier
- Title
-
TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI AgentsWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2025Year of publication
- Publication date
-
2025-04-17Full publication date if available
- Authors
-
Bofei Zhang, Z. J. Shang, Gao Zhi, Wang Zhang, Rui Xie, Xiaojian Ma, Tao Yuan, Xingxing Wu, Song‐Chun Zhu, Qing LiList of authors in order
- Landing page
-
https://arxiv.org/abs/2504.12679Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2504.12679Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2504.12679Direct OA link when available
- Cited by
-
0Total citation count in OpenAlex
Full payload
| id | https://openalex.org/W4415320065 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2504.12679 |
| ids.doi | https://doi.org/10.48550/arxiv.2504.12679 |
| ids.openalex | https://openalex.org/W4415320065 |
| fwci | |
| type | preprint |
| title | TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agents |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T12031 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9911999702453613 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Speech and dialogue systems |
| topics[1].id | https://openalex.org/T10181 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9847999811172485 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Natural Language Processing Techniques |
| topics[2].id | https://openalex.org/T12128 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9422000050544739 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1702 |
| topics[2].subfield.display_name | Artificial Intelligence |
| topics[2].display_name | AI in Service Interactions |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2504.12679 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2504.12679 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2504.12679 |
| locations[1].id | doi:10.48550/arxiv.2504.12679 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2504.12679 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5073897552 |
| authorships[0].author.orcid | |
| authorships[0].author.display_name | Bofei Zhang |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Zhang, Bofei |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5077936254 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-4884-8129 |
| authorships[1].author.display_name | Z. J. Shang |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Shang, Zirui |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5053519162 |
| authorships[2].author.orcid | https://orcid.org/0000-0003-4101-9923 |
| authorships[2].author.display_name | Gao Zhi |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Gao, Zhi |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5100329323 |
| authorships[3].author.orcid | https://orcid.org/0000-0003-2603-6522 |
| authorships[3].author.display_name | Wang Zhang |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Zhang, Wang |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5021985671 |
| authorships[4].author.orcid | https://orcid.org/0000-0001-6897-1581 |
| authorships[4].author.display_name | Rui Xie |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Xie, Rui |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5100689683 |
| authorships[5].author.orcid | https://orcid.org/0000-0002-2611-6568 |
| authorships[5].author.display_name | Xiaojian Ma |
| authorships[5].author_position | middle |
| authorships[5].raw_author_name | Ma, Xiaojian |
| authorships[5].is_corresponding | False |
| authorships[6].author.id | https://openalex.org/A5051718265 |
| authorships[6].author.orcid | https://orcid.org/0009-0006-9530-207X |
| authorships[6].author.display_name | Tao Yuan |
| authorships[6].author_position | middle |
| authorships[6].raw_author_name | Yuan, Tao |
| authorships[6].is_corresponding | False |
| authorships[7].author.id | https://openalex.org/A5051482208 |
| authorships[7].author.orcid | https://orcid.org/0000-0002-6281-243X |
| authorships[7].author.display_name | Xingxing Wu |
| authorships[7].author_position | middle |
| authorships[7].raw_author_name | Wu, Xinxiao |
| authorships[7].is_corresponding | False |
| authorships[8].author.id | https://openalex.org/A5034228010 |
| authorships[8].author.orcid | |
| authorships[8].author.display_name | Song‐Chun Zhu |
| authorships[8].author_position | middle |
| authorships[8].raw_author_name | Zhu, Song-Chun |
| authorships[8].is_corresponding | False |
| authorships[9].author.id | https://openalex.org/A5100404186 |
| authorships[9].author.orcid | https://orcid.org/0000-0003-4807-030X |
| authorships[9].author.display_name | Qing Li |
| authorships[9].author_position | last |
| authorships[9].raw_author_name | Li, Qing |
| authorships[9].is_corresponding | False |
| has_content.pdf | True |
| has_content.grobid_xml | True |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2504.12679 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-18T00:00:00 |
| display_name | TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agents |
| has_fulltext | True |
| is_retracted | False |
| updated_date | 2025-12-06T23:10:59.065948 |
| primary_topic.id | https://openalex.org/T12031 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9911999702453613 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Speech and dialogue systems |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2504.12679 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2504.12679 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2504.12679 |
| primary_location.id | pmh:oai:arXiv.org:2504.12679 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2504.12679 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2504.12679 |
| publication_date | 2025-04-17 |
| publication_year | 2025 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 7, 26 |
| abstract_inverted_index.In | 56 |
| abstract_inverted_index.We | 114, 160 |
| abstract_inverted_index.as | 85 |
| abstract_inverted_index.by | 69, 119 |
| abstract_inverted_index.in | 29 |
| abstract_inverted_index.is | 6, 34 |
| abstract_inverted_index.of | 37, 53, 148, 156 |
| abstract_inverted_index.on | 123, 130, 142 |
| abstract_inverted_index.or | 17 |
| abstract_inverted_index.to | 20, 49 |
| abstract_inverted_index.we | 59, 77, 96 |
| abstract_inverted_index.200 | 112 |
| abstract_inverted_index.GUI | 23, 32, 67, 82, 90 |
| abstract_inverted_index.and | 45, 79, 87, 109, 134, 152, 169 |
| abstract_inverted_index.due | 48 |
| abstract_inverted_index.our | 157 |
| abstract_inverted_index.the | 35, 50, 61, 98, 116, 146, 149, 154, 164, 166, 170 |
| abstract_inverted_index.web | 74 |
| abstract_inverted_index.10\% | 141 |
| abstract_inverted_index.143K | 102 |
| abstract_inverted_index.User | 2 |
| abstract_inverted_index.cost | 52 |
| abstract_inverted_index.data | 40, 104 |
| abstract_inverted_index.five | 106 |
| abstract_inverted_index.from | 71 |
| abstract_inverted_index.high | 51 |
| abstract_inverted_index.into | 89 |
| abstract_inverted_index.lack | 36 |
| abstract_inverted_index.more | 110 |
| abstract_inverted_index.rich | 72 |
| abstract_inverted_index.show | 126 |
| abstract_inverted_index.than | 111 |
| abstract_inverted_index.that | 64 |
| abstract_inverted_index.this | 57 |
| abstract_inverted_index.used | 132 |
| abstract_inverted_index.will | 161 |
| abstract_inverted_index.with | 15 |
| abstract_inverted_index.(GUI) | 4 |
| abstract_inverted_index.(such | 84 |
| abstract_inverted_index.about | 140 |
| abstract_inverted_index.agent | 91, 118 |
| abstract_inverted_index.code, | 165 |
| abstract_inverted_index.crawl | 78 |
| abstract_inverted_index.data, | 93 |
| abstract_inverted_index.fully | 162 |
| abstract_inverted_index.human | 13 |
| abstract_inverted_index.major | 27 |
| abstract_inverted_index.soon. | 173 |
| abstract_inverted_index.which | 11, 95, 125 |
| abstract_inverted_index.TongUI | 62, 117, 158 |
| abstract_inverted_index.across | 41, 105 |
| abstract_inverted_index.agents | 5, 33, 68, 139 |
| abstract_inverted_index.builds | 65 |
| abstract_inverted_index.mainly | 47 |
| abstract_inverted_index.manual | 54 |
| abstract_inverted_index.mobile | 18 |
| abstract_inverted_index.models | 122, 172 |
| abstract_inverted_index.online | 81 |
| abstract_inverted_index.paper, | 58 |
| abstract_inverted_index.phones | 19 |
| abstract_inverted_index.tasks. | 24 |
| abstract_inverted_index.videos | 86 |
| abstract_inverted_index.GUI-Net | 99, 150, 167 |
| abstract_inverted_index.dataset | 100, 151 |
| abstract_inverted_index.develop | 115 |
| abstract_inverted_index.diverse | 22 |
| abstract_inverted_index.perform | 21 |
| abstract_inverted_index.process | 80 |
| abstract_inverted_index.produce | 97 |
| abstract_inverted_index.propose | 60 |
| abstract_inverted_index.showing | 145 |
| abstract_inverted_index.systems | 44, 108 |
| abstract_inverted_index.through | 94 |
| abstract_inverted_index.trained | 171 |
| abstract_inverted_index.various | 42 |
| abstract_inverted_index.Building | 0 |
| abstract_inverted_index.GUI-Net, | 124 |
| abstract_inverted_index.However, | 25 |
| abstract_inverted_index.baseline | 138 |
| abstract_inverted_index.commonly | 131 |
| abstract_inverted_index.dataset, | 168 |
| abstract_inverted_index.learning | 70 |
| abstract_inverted_index.multiple | 143 |
| abstract_inverted_index.research | 9 |
| abstract_inverted_index.Graphical | 1 |
| abstract_inverted_index.Interface | 3 |
| abstract_inverted_index.articles) | 88 |
| abstract_inverted_index.challenge | 28 |
| abstract_inverted_index.computers | 16 |
| abstract_inverted_index.framework | 63 |
| abstract_inverted_index.grounding | 133 |
| abstract_inverted_index.operating | 43, 107 |
| abstract_inverted_index.promising | 8 |
| abstract_inverted_index.simulates | 12 |
| abstract_inverted_index.tutorials | 83 |
| abstract_inverted_index.containing | 101 |
| abstract_inverted_index.developing | 30 |
| abstract_inverted_index.direction, | 10 |
| abstract_inverted_index.framework. | 159 |
| abstract_inverted_index.multimodal | 73 |
| abstract_inverted_index.navigation | 135 |
| abstract_inverted_index.remarkable | 127 |
| abstract_inverted_index.sufficient | 38 |
| abstract_inverted_index.trajectory | 39, 92, 103 |
| abstract_inverted_index.tutorials. | 75 |
| abstract_inverted_index.Concretely, | 76 |
| abstract_inverted_index.benchmarks, | 136, 144 |
| abstract_inverted_index.fine-tuning | 120 |
| abstract_inverted_index.generalized | 31, 66 |
| abstract_inverted_index.interaction | 14 |
| abstract_inverted_index.open-source | 163 |
| abstract_inverted_index.performance | 128 |
| abstract_inverted_index.annotations. | 55 |
| abstract_inverted_index.improvements | 129 |
| abstract_inverted_index.significance | 155 |
| abstract_inverted_index.underscoring | 153 |
| abstract_inverted_index.applications, | 46 |
| abstract_inverted_index.applications. | 113 |
| abstract_inverted_index.effectiveness | 147 |
| abstract_inverted_index.outperforming | 137 |
| abstract_inverted_index.Qwen2.5-VL-3B/7B | 121 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 10 |
| citation_normalized_percentile |