Automating eHMI Action Design with LLMs for Automated Vehicle Communication Article Swipe
Xia Ding
,
Xinyue Gui
,
Fan Gao
,
Dongyuan Li
,
Mark Colley
,
Takeo Igarashi
·
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2505.20711
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2505.20711
The absence of explicit communication channels between automated vehicles (AVs) and other road users requires the use of external Human-Machine Interfaces (eHMIs) to convey messages effectively in uncertain scenarios. Currently, most eHMI studies employ predefined text messages and manually designed actions to perform these messages, which limits the real-world deployment of eHMIs, where adaptability in dynamic scenarios is essential. Given the generalizability and versatility of large language models (LLMs), they could potentially serve as automated action designers for the message-action design task. To validate this idea, we make three contributions: (1) We propose a pipeline that integrates LLMs and 3D renderers, using LLMs as action designers to generate executable actions for controlling eHMIs and rendering action clips. (2) We collect a user-rated Action-Design Scoring dataset comprising a total of 320 action sequences for eight intended messages and four representative eHMI modalities. The dataset validates that LLMs can translate intended messages into actions close to a human level, particularly for reasoning-enabled LLMs. (3) We introduce two automated raters, Action Reference Score (ARS) and Vision-Language Models (VLMs), to benchmark 18 LLMs, finding that the VLM aligns with human preferences yet varies across eHMI modalities.
Related Topics
Concepts
No concepts available.
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2505.20711
- https://arxiv.org/pdf/2505.20711
- OA Status
- green
- OpenAlex ID
- https://openalex.org/W4415036213
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4415036213Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2505.20711Digital Object Identifier
- Title
-
Automating eHMI Action Design with LLMs for Automated Vehicle CommunicationWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2025Year of publication
- Publication date
-
2025-05-27Full publication date if available
- Authors
-
Xia Ding, Xinyue Gui, Fan Gao, Dongyuan Li, Mark Colley, Takeo IgarashiList of authors in order
- Landing page
-
https://arxiv.org/abs/2505.20711Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2505.20711Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2505.20711Direct OA link when available
- Cited by
-
0Total citation count in OpenAlex
Full payload
| id | https://openalex.org/W4415036213 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2505.20711 |
| ids.doi | https://doi.org/10.48550/arxiv.2505.20711 |
| ids.openalex | https://openalex.org/W4415036213 |
| fwci | |
| type | preprint |
| title | Automating eHMI Action Design with LLMs for Automated Vehicle Communication |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10703 |
| topics[0].field.id | https://openalex.org/fields/14 |
| topics[0].field.display_name | Business, Management and Accounting |
| topics[0].score | 0.9524999856948853 |
| topics[0].domain.id | https://openalex.org/domains/2 |
| topics[0].domain.display_name | Social Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1404 |
| topics[0].subfield.display_name | Management Information Systems |
| topics[0].display_name | Business Process Modeling and Analysis |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2505.20711 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | cc-by |
| locations[0].pdf_url | https://arxiv.org/pdf/2505.20711 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | https://openalex.org/licenses/cc-by |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2505.20711 |
| locations[1].id | doi:10.48550/arxiv.2505.20711 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2505.20711 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5036112550 |
| authorships[0].author.orcid | https://orcid.org/0000-0001-8562-7141 |
| authorships[0].author.display_name | Xia Ding |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Xia, Ding |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5062839476 |
| authorships[1].author.orcid | https://orcid.org/0000-0001-6541-224X |
| authorships[1].author.display_name | Xinyue Gui |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Gui, Xinyue |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5100696238 |
| authorships[2].author.orcid | |
| authorships[2].author.display_name | Fan Gao |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Gao, Fan |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5101720373 |
| authorships[3].author.orcid | https://orcid.org/0009-0001-9302-8154 |
| authorships[3].author.display_name | Dongyuan Li |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Li, Dongyuan |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5026729075 |
| authorships[4].author.orcid | https://orcid.org/0000-0001-5207-5029 |
| authorships[4].author.display_name | Mark Colley |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Colley, Mark |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5102743150 |
| authorships[5].author.orcid | https://orcid.org/0000-0002-5495-6441 |
| authorships[5].author.display_name | Takeo Igarashi |
| authorships[5].author_position | last |
| authorships[5].raw_author_name | Igarashi, Takeo |
| authorships[5].is_corresponding | False |
| has_content.pdf | True |
| has_content.grobid_xml | True |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2505.20711 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Automating eHMI Action Design with LLMs for Automated Vehicle Communication |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10703 |
| primary_topic.field.id | https://openalex.org/fields/14 |
| primary_topic.field.display_name | Business, Management and Accounting |
| primary_topic.score | 0.9524999856948853 |
| primary_topic.domain.id | https://openalex.org/domains/2 |
| primary_topic.domain.display_name | Social Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1404 |
| primary_topic.subfield.display_name | Management Information Systems |
| primary_topic.display_name | Business Process Modeling and Analysis |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2505.20711 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | cc-by |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2505.20711 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | https://openalex.org/licenses/cc-by |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2505.20711 |
| primary_location.id | pmh:oai:arXiv.org:2505.20711 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | cc-by |
| primary_location.pdf_url | https://arxiv.org/pdf/2505.20711 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | https://openalex.org/licenses/cc-by |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2505.20711 |
| publication_date | 2025-05-27 |
| publication_year | 2025 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 93, 120, 126, 154 |
| abstract_inverted_index.18 | 177 |
| abstract_inverted_index.3D | 99 |
| abstract_inverted_index.To | 82 |
| abstract_inverted_index.We | 91, 118, 162 |
| abstract_inverted_index.as | 73, 103 |
| abstract_inverted_index.in | 26, 54 |
| abstract_inverted_index.is | 57 |
| abstract_inverted_index.of | 2, 17, 50, 64, 128 |
| abstract_inverted_index.to | 22, 41, 106, 153, 175 |
| abstract_inverted_index.we | 86 |
| abstract_inverted_index.(1) | 90 |
| abstract_inverted_index.(2) | 117 |
| abstract_inverted_index.(3) | 161 |
| abstract_inverted_index.320 | 129 |
| abstract_inverted_index.The | 0, 141 |
| abstract_inverted_index.VLM | 182 |
| abstract_inverted_index.and | 10, 37, 62, 98, 113, 136, 171 |
| abstract_inverted_index.can | 146 |
| abstract_inverted_index.for | 77, 110, 132, 158 |
| abstract_inverted_index.the | 15, 47, 60, 78, 181 |
| abstract_inverted_index.two | 164 |
| abstract_inverted_index.use | 16 |
| abstract_inverted_index.yet | 187 |
| abstract_inverted_index.LLMs | 97, 102, 145 |
| abstract_inverted_index.eHMI | 31, 139, 190 |
| abstract_inverted_index.four | 137 |
| abstract_inverted_index.into | 150 |
| abstract_inverted_index.make | 87 |
| abstract_inverted_index.most | 30 |
| abstract_inverted_index.road | 12 |
| abstract_inverted_index.text | 35 |
| abstract_inverted_index.that | 95, 144, 180 |
| abstract_inverted_index.they | 69 |
| abstract_inverted_index.this | 84 |
| abstract_inverted_index.with | 184 |
| abstract_inverted_index.(ARS) | 170 |
| abstract_inverted_index.(AVs) | 9 |
| abstract_inverted_index.Given | 59 |
| abstract_inverted_index.LLMs, | 178 |
| abstract_inverted_index.LLMs. | 160 |
| abstract_inverted_index.Score | 169 |
| abstract_inverted_index.close | 152 |
| abstract_inverted_index.could | 70 |
| abstract_inverted_index.eHMIs | 112 |
| abstract_inverted_index.eight | 133 |
| abstract_inverted_index.human | 155, 185 |
| abstract_inverted_index.idea, | 85 |
| abstract_inverted_index.large | 65 |
| abstract_inverted_index.other | 11 |
| abstract_inverted_index.serve | 72 |
| abstract_inverted_index.task. | 81 |
| abstract_inverted_index.these | 43 |
| abstract_inverted_index.three | 88 |
| abstract_inverted_index.total | 127 |
| abstract_inverted_index.users | 13 |
| abstract_inverted_index.using | 101 |
| abstract_inverted_index.where | 52 |
| abstract_inverted_index.which | 45 |
| abstract_inverted_index.Action | 167 |
| abstract_inverted_index.Models | 173 |
| abstract_inverted_index.across | 189 |
| abstract_inverted_index.action | 75, 104, 115, 130 |
| abstract_inverted_index.aligns | 183 |
| abstract_inverted_index.clips. | 116 |
| abstract_inverted_index.convey | 23 |
| abstract_inverted_index.design | 80 |
| abstract_inverted_index.eHMIs, | 51 |
| abstract_inverted_index.employ | 33 |
| abstract_inverted_index.level, | 156 |
| abstract_inverted_index.limits | 46 |
| abstract_inverted_index.models | 67 |
| abstract_inverted_index.varies | 188 |
| abstract_inverted_index.(LLMs), | 68 |
| abstract_inverted_index.(VLMs), | 174 |
| abstract_inverted_index.(eHMIs) | 21 |
| abstract_inverted_index.Scoring | 123 |
| abstract_inverted_index.absence | 1 |
| abstract_inverted_index.actions | 40, 109, 151 |
| abstract_inverted_index.between | 6 |
| abstract_inverted_index.collect | 119 |
| abstract_inverted_index.dataset | 124, 142 |
| abstract_inverted_index.dynamic | 55 |
| abstract_inverted_index.finding | 179 |
| abstract_inverted_index.perform | 42 |
| abstract_inverted_index.propose | 92 |
| abstract_inverted_index.raters, | 166 |
| abstract_inverted_index.studies | 32 |
| abstract_inverted_index.channels | 5 |
| abstract_inverted_index.designed | 39 |
| abstract_inverted_index.explicit | 3 |
| abstract_inverted_index.external | 18 |
| abstract_inverted_index.generate | 107 |
| abstract_inverted_index.intended | 134, 148 |
| abstract_inverted_index.language | 66 |
| abstract_inverted_index.manually | 38 |
| abstract_inverted_index.messages | 24, 36, 135, 149 |
| abstract_inverted_index.pipeline | 94 |
| abstract_inverted_index.requires | 14 |
| abstract_inverted_index.validate | 83 |
| abstract_inverted_index.vehicles | 8 |
| abstract_inverted_index.Reference | 168 |
| abstract_inverted_index.automated | 7, 74, 165 |
| abstract_inverted_index.benchmark | 176 |
| abstract_inverted_index.designers | 76, 105 |
| abstract_inverted_index.introduce | 163 |
| abstract_inverted_index.messages, | 44 |
| abstract_inverted_index.rendering | 114 |
| abstract_inverted_index.scenarios | 56 |
| abstract_inverted_index.sequences | 131 |
| abstract_inverted_index.translate | 147 |
| abstract_inverted_index.uncertain | 27 |
| abstract_inverted_index.validates | 143 |
| abstract_inverted_index.Currently, | 29 |
| abstract_inverted_index.Interfaces | 20 |
| abstract_inverted_index.comprising | 125 |
| abstract_inverted_index.deployment | 49 |
| abstract_inverted_index.essential. | 58 |
| abstract_inverted_index.executable | 108 |
| abstract_inverted_index.integrates | 96 |
| abstract_inverted_index.predefined | 34 |
| abstract_inverted_index.real-world | 48 |
| abstract_inverted_index.renderers, | 100 |
| abstract_inverted_index.scenarios. | 28 |
| abstract_inverted_index.user-rated | 121 |
| abstract_inverted_index.controlling | 111 |
| abstract_inverted_index.effectively | 25 |
| abstract_inverted_index.modalities. | 140, 191 |
| abstract_inverted_index.potentially | 71 |
| abstract_inverted_index.preferences | 186 |
| abstract_inverted_index.versatility | 63 |
| abstract_inverted_index.adaptability | 53 |
| abstract_inverted_index.particularly | 157 |
| abstract_inverted_index.Action-Design | 122 |
| abstract_inverted_index.Human-Machine | 19 |
| abstract_inverted_index.communication | 4 |
| abstract_inverted_index.contributions: | 89 |
| abstract_inverted_index.message-action | 79 |
| abstract_inverted_index.representative | 138 |
| abstract_inverted_index.Vision-Language | 172 |
| abstract_inverted_index.generalizability | 61 |
| abstract_inverted_index.reasoning-enabled | 159 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 6 |
| citation_normalized_percentile |