OpenApps: Simulating Environment Variations to Measure UI-Agent Reliability Article Swipe
Karen Ullrich
,
Arjun Subramonian
,
Amir Bar
,
Ivan Evtimov
,
Nikolaos Tsilivis
,
Randall Balestriero
,
Julia Kempe
·
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2511.20766
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2511.20766
Reliability is key to realizing the promise of autonomous UI-Agents, multimodal agents that directly interact with apps in the same manner as humans, as users must be able to trust an agent to complete a given task. Current evaluations rely on fixed environments, often clones of existing apps, which are limited in that they can only shed light on whether or how often an agent can complete a task within a specific environment. When deployed however, agents are likely to encounter variations in app design and content that can affect an agent's ability to complete a task. To address this blind spot of measuring agent reliability across app variations, we develop OpenApps, a light-weight open-source ecosystem with six apps (messenger, calendar, maps, etc.) that are configurable in appearance and content. OpenApps requires just a single CPU to run, enabling easy generation and deployment of thousands of versions of each app. Specifically, we run more than 10,000 independent evaluations to study reliability across seven leading multimodal agents. We find that while standard reliability within a fixed app is relatively stable, reliability can vary drastically when measured across app variations. Task success rates for many agents can fluctuate by more than $50\%$ across app variations. For example, Kimi-VL-3B's average success across all tasks fluctuates from $63\%$ to just $4\%$ across app versions. We also find agent behaviors such as looping or hallucinating actions can differ drastically depending on the environment configuration. These initial findings highlight the importance of measuring reliability along this new dimension of app variations. OpenApps is available at https://facebookresearch.github.io/OpenApps/
Related Topics
Concepts
No concepts available.
Metadata
- Type
- preprint
- Landing Page
- http://arxiv.org/abs/2511.20766
- https://arxiv.org/pdf/2511.20766
- OA Status
- green
- OpenAlex ID
- https://openalex.org/W4417095248
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4417095248Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2511.20766Digital Object Identifier
- Title
-
OpenApps: Simulating Environment Variations to Measure UI-Agent ReliabilityWork title
- Type
-
preprintOpenAlex work type
- Publication year
-
2025Year of publication
- Publication date
-
2025-11-25Full publication date if available
- Authors
-
Karen Ullrich, Arjun Subramonian, Amir Bar, Ivan Evtimov, Nikolaos Tsilivis, Randall Balestriero, Julia KempeList of authors in order
- Landing page
-
https://arxiv.org/abs/2511.20766Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2511.20766Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2511.20766Direct OA link when available
- Cited by
-
0Total citation count in OpenAlex
Full payload
| id | https://openalex.org/W4417095248 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2511.20766 |
| ids.doi | https://doi.org/10.48550/arxiv.2511.20766 |
| ids.openalex | https://openalex.org/W4417095248 |
| fwci | |
| type | preprint |
| title | OpenApps: Simulating Environment Variations to Measure UI-Agent Reliability |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| language | |
| locations[0].id | pmh:oai:arXiv.org:2511.20766 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2511.20766 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2511.20766 |
| locations[1].id | doi:10.48550/arxiv.2511.20766 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2511.20766 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5058031547 |
| authorships[0].author.orcid | |
| authorships[0].author.display_name | Karen Ullrich |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Ullrich, Karen |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5114482375 |
| authorships[1].author.orcid | |
| authorships[1].author.display_name | Arjun Subramonian |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Subramonian, Arjun |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5040676689 |
| authorships[2].author.orcid | |
| authorships[2].author.display_name | Amir Bar |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Bar, Amir |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5021932110 |
| authorships[3].author.orcid | https://orcid.org/0009-0003-7763-4263 |
| authorships[3].author.display_name | Ivan Evtimov |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Evtimov, Ivan |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5027137668 |
| authorships[4].author.orcid | |
| authorships[4].author.display_name | Nikolaos Tsilivis |
| authorships[4].author_position | last |
| authorships[4].raw_author_name | Tsilivis, Nikolaos |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5047293370 |
| authorships[5].author.orcid | https://orcid.org/0000-0002-5692-4187 |
| authorships[5].author.display_name | Randall Balestriero |
| authorships[5].author_position | middle |
| authorships[5].raw_author_name | Balestriero, Randall |
| authorships[5].is_corresponding | False |
| authorships[6].author.id | https://openalex.org/A5051933322 |
| authorships[6].author.orcid | https://orcid.org/0009-0006-1984-0060 |
| authorships[6].author.display_name | Julia Kempe |
| authorships[6].author_position | middle |
| authorships[6].raw_author_name | Kempe, Julia |
| authorships[6].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2511.20766 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-11-28T00:00:00 |
| display_name | OpenApps: Simulating Environment Variations to Measure UI-Agent Reliability |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-12-07T18:30:23.540059 |
| primary_topic | |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2511.20766 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2511.20766 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2511.20766 |
| primary_location.id | pmh:oai:arXiv.org:2511.20766 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2511.20766 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2511.20766 |
| publication_date | 2025-11-25 |
| publication_year | 2025 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 34, 67, 70, 95, 112, 133, 173 |
| abstract_inverted_index.To | 97 |
| abstract_inverted_index.We | 166, 220 |
| abstract_inverted_index.an | 30, 63, 90 |
| abstract_inverted_index.as | 21, 23, 226 |
| abstract_inverted_index.at | 258 |
| abstract_inverted_index.be | 26 |
| abstract_inverted_index.by | 196 |
| abstract_inverted_index.in | 17, 51, 82, 126 |
| abstract_inverted_index.is | 1, 176, 256 |
| abstract_inverted_index.of | 7, 45, 102, 143, 145, 147, 245, 252 |
| abstract_inverted_index.on | 40, 58, 235 |
| abstract_inverted_index.or | 60, 228 |
| abstract_inverted_index.to | 3, 28, 32, 79, 93, 136, 158, 214 |
| abstract_inverted_index.we | 109, 151 |
| abstract_inverted_index.CPU | 135 |
| abstract_inverted_index.For | 203 |
| abstract_inverted_index.all | 209 |
| abstract_inverted_index.and | 85, 128, 141 |
| abstract_inverted_index.app | 83, 107, 175, 186, 201, 218, 253 |
| abstract_inverted_index.are | 49, 77, 124 |
| abstract_inverted_index.can | 54, 65, 88, 180, 194, 231 |
| abstract_inverted_index.for | 191 |
| abstract_inverted_index.how | 61 |
| abstract_inverted_index.key | 2 |
| abstract_inverted_index.new | 250 |
| abstract_inverted_index.run | 152 |
| abstract_inverted_index.six | 117 |
| abstract_inverted_index.the | 5, 18, 236, 243 |
| abstract_inverted_index.Task | 188 |
| abstract_inverted_index.When | 73 |
| abstract_inverted_index.able | 27 |
| abstract_inverted_index.also | 221 |
| abstract_inverted_index.app. | 149 |
| abstract_inverted_index.apps | 16, 118 |
| abstract_inverted_index.each | 148 |
| abstract_inverted_index.easy | 139 |
| abstract_inverted_index.find | 167, 222 |
| abstract_inverted_index.from | 212 |
| abstract_inverted_index.just | 132, 215 |
| abstract_inverted_index.many | 192 |
| abstract_inverted_index.more | 153, 197 |
| abstract_inverted_index.must | 25 |
| abstract_inverted_index.only | 55 |
| abstract_inverted_index.rely | 39 |
| abstract_inverted_index.run, | 137 |
| abstract_inverted_index.same | 19 |
| abstract_inverted_index.shed | 56 |
| abstract_inverted_index.spot | 101 |
| abstract_inverted_index.such | 225 |
| abstract_inverted_index.task | 68 |
| abstract_inverted_index.than | 154, 198 |
| abstract_inverted_index.that | 12, 52, 87, 123, 168 |
| abstract_inverted_index.they | 53 |
| abstract_inverted_index.this | 99, 249 |
| abstract_inverted_index.vary | 181 |
| abstract_inverted_index.when | 183 |
| abstract_inverted_index.with | 15, 116 |
| abstract_inverted_index.$4\%$ | 216 |
| abstract_inverted_index.These | 239 |
| abstract_inverted_index.agent | 31, 64, 104, 223 |
| abstract_inverted_index.along | 248 |
| abstract_inverted_index.apps, | 47 |
| abstract_inverted_index.blind | 100 |
| abstract_inverted_index.etc.) | 122 |
| abstract_inverted_index.fixed | 41, 174 |
| abstract_inverted_index.given | 35 |
| abstract_inverted_index.light | 57 |
| abstract_inverted_index.maps, | 121 |
| abstract_inverted_index.often | 43, 62 |
| abstract_inverted_index.rates | 190 |
| abstract_inverted_index.seven | 162 |
| abstract_inverted_index.study | 159 |
| abstract_inverted_index.task. | 36, 96 |
| abstract_inverted_index.tasks | 210 |
| abstract_inverted_index.trust | 29 |
| abstract_inverted_index.users | 24 |
| abstract_inverted_index.which | 48 |
| abstract_inverted_index.while | 169 |
| abstract_inverted_index.$50\%$ | 199 |
| abstract_inverted_index.$63\%$ | 213 |
| abstract_inverted_index.10,000 | 155 |
| abstract_inverted_index.across | 106, 161, 185, 200, 208, 217 |
| abstract_inverted_index.affect | 89 |
| abstract_inverted_index.agents | 11, 76, 193 |
| abstract_inverted_index.clones | 44 |
| abstract_inverted_index.design | 84 |
| abstract_inverted_index.differ | 232 |
| abstract_inverted_index.likely | 78 |
| abstract_inverted_index.manner | 20 |
| abstract_inverted_index.single | 134 |
| abstract_inverted_index.within | 69, 172 |
| abstract_inverted_index.Current | 37 |
| abstract_inverted_index.ability | 92 |
| abstract_inverted_index.actions | 230 |
| abstract_inverted_index.address | 98 |
| abstract_inverted_index.agent's | 91 |
| abstract_inverted_index.agents. | 165 |
| abstract_inverted_index.average | 206 |
| abstract_inverted_index.content | 86 |
| abstract_inverted_index.develop | 110 |
| abstract_inverted_index.humans, | 22 |
| abstract_inverted_index.initial | 240 |
| abstract_inverted_index.leading | 163 |
| abstract_inverted_index.limited | 50 |
| abstract_inverted_index.looping | 227 |
| abstract_inverted_index.promise | 6 |
| abstract_inverted_index.stable, | 178 |
| abstract_inverted_index.success | 189, 207 |
| abstract_inverted_index.whether | 59 |
| abstract_inverted_index.OpenApps | 130, 255 |
| abstract_inverted_index.complete | 33, 66, 94 |
| abstract_inverted_index.content. | 129 |
| abstract_inverted_index.deployed | 74 |
| abstract_inverted_index.directly | 13 |
| abstract_inverted_index.enabling | 138 |
| abstract_inverted_index.example, | 204 |
| abstract_inverted_index.existing | 46 |
| abstract_inverted_index.findings | 241 |
| abstract_inverted_index.however, | 75 |
| abstract_inverted_index.interact | 14 |
| abstract_inverted_index.measured | 184 |
| abstract_inverted_index.requires | 131 |
| abstract_inverted_index.specific | 71 |
| abstract_inverted_index.standard | 170 |
| abstract_inverted_index.versions | 146 |
| abstract_inverted_index.OpenApps, | 111 |
| abstract_inverted_index.available | 257 |
| abstract_inverted_index.behaviors | 224 |
| abstract_inverted_index.calendar, | 120 |
| abstract_inverted_index.depending | 234 |
| abstract_inverted_index.dimension | 251 |
| abstract_inverted_index.ecosystem | 115 |
| abstract_inverted_index.encounter | 80 |
| abstract_inverted_index.fluctuate | 195 |
| abstract_inverted_index.highlight | 242 |
| abstract_inverted_index.measuring | 103, 246 |
| abstract_inverted_index.realizing | 4 |
| abstract_inverted_index.thousands | 144 |
| abstract_inverted_index.versions. | 219 |
| abstract_inverted_index.UI-Agents, | 9 |
| abstract_inverted_index.appearance | 127 |
| abstract_inverted_index.autonomous | 8 |
| abstract_inverted_index.deployment | 142 |
| abstract_inverted_index.fluctuates | 211 |
| abstract_inverted_index.generation | 140 |
| abstract_inverted_index.importance | 244 |
| abstract_inverted_index.multimodal | 10, 164 |
| abstract_inverted_index.relatively | 177 |
| abstract_inverted_index.variations | 81 |
| abstract_inverted_index.(messenger, | 119 |
| abstract_inverted_index.Reliability | 0 |
| abstract_inverted_index.drastically | 182, 233 |
| abstract_inverted_index.environment | 237 |
| abstract_inverted_index.evaluations | 38, 157 |
| abstract_inverted_index.independent | 156 |
| abstract_inverted_index.open-source | 114 |
| abstract_inverted_index.reliability | 105, 160, 171, 179, 247 |
| abstract_inverted_index.variations, | 108 |
| abstract_inverted_index.variations. | 187, 202, 254 |
| abstract_inverted_index.Kimi-VL-3B's | 205 |
| abstract_inverted_index.configurable | 125 |
| abstract_inverted_index.environment. | 72 |
| abstract_inverted_index.light-weight | 113 |
| abstract_inverted_index.Specifically, | 150 |
| abstract_inverted_index.environments, | 42 |
| abstract_inverted_index.hallucinating | 229 |
| abstract_inverted_index.configuration. | 238 |
| abstract_inverted_index.https://facebookresearch.github.io/OpenApps/ | 259 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 7 |
| citation_normalized_percentile |