LLM3:Large Language Model-based Task and Motion Planning with Motion Failure Reasoning Article Swipe
Shu Wang
,
Muzhi Han
,
Ziyuan Jiao
,
Zeyu Zhang
,
Ying Wu
,
Song‐Chun Zhu
,
Hangxin Liu
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2403.11552
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2403.11552
Conventional Task and Motion Planning (TAMP) approaches rely on manually crafted interfaces connecting symbolic task planning with continuous motion generation. These domain-specific and labor-intensive modules are limited in addressing emerging tasks in real-world settings. Here, we present LLM^3, a novel Large Language Model (LLM)-based TAMP framework featuring a domain-independent interface. Specifically, we leverage the powerful reasoning and planning capabilities of pre-trained LLMs to propose symbolic action sequences and select continuous action parameters for motion planning. Crucially, LLM^3 incorporates motion planning feedback through prompting, allowing the LLM to iteratively refine its proposals by reasoning about motion failure. Consequently, LLM^3 interfaces between task planning and motion planning, alleviating the intricate design process of handling domain-specific messages between them. Through a series of simulations in a box-packing domain, we quantitatively demonstrate the effectiveness of LLM^3 in solving TAMP problems and the efficiency in selecting action parameters. Ablation studies underscore the significant contribution of motion failure reasoning to the success of LLM^3. Furthermore, we conduct qualitative experiments on a physical manipulator, demonstrating the practical applicability of our approach in real-world settings.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2403.11552
- https://arxiv.org/pdf/2403.11552
- OA Status
- green
- Cited By
- 7
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4392974098
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4392974098Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2403.11552Digital Object Identifier
- Title
-
LLM3:Large Language Model-based Task and Motion Planning with Motion Failure ReasoningWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-03-18Full publication date if available
- Authors
-
Shu Wang, Muzhi Han, Ziyuan Jiao, Zeyu Zhang, Ying Wu, Song‐Chun Zhu, Hangxin LiuList of authors in order
- Landing page
-
https://arxiv.org/abs/2403.11552Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2403.11552Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2403.11552Direct OA link when available
- Concepts
-
Motion (physics), Task (project management), Computer science, Artificial intelligence, Engineering, Systems engineeringTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
7Total citation count in OpenAlex
- Citations by year (recent)
-
2025: 3, 2024: 4Per-year citation counts (last 5 years)
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4392974098 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2403.11552 |
| ids.doi | https://doi.org/10.48550/arxiv.2403.11552 |
| ids.openalex | https://openalex.org/W4392974098 |
| fwci | |
| type | preprint |
| title | LLM3:Large Language Model-based Task and Motion Planning with Motion Failure Reasoning |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10181 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9940999746322632 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Natural Language Processing Techniques |
| topics[1].id | https://openalex.org/T11714 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9783999919891357 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1707 |
| topics[1].subfield.display_name | Computer Vision and Pattern Recognition |
| topics[1].display_name | Multimodal Machine Learning Applications |
| topics[2].id | https://openalex.org/T12031 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9616000056266785 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1702 |
| topics[2].subfield.display_name | Artificial Intelligence |
| topics[2].display_name | Speech and dialogue systems |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C104114177 |
| concepts[0].level | 2 |
| concepts[0].score | 0.7221735715866089 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q79782 |
| concepts[0].display_name | Motion (physics) |
| concepts[1].id | https://openalex.org/C2780451532 |
| concepts[1].level | 2 |
| concepts[1].score | 0.6647769212722778 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q759676 |
| concepts[1].display_name | Task (project management) |
| concepts[2].id | https://openalex.org/C41008148 |
| concepts[2].level | 0 |
| concepts[2].score | 0.6242836713790894 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[2].display_name | Computer science |
| concepts[3].id | https://openalex.org/C154945302 |
| concepts[3].level | 1 |
| concepts[3].score | 0.39982330799102783 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[3].display_name | Artificial intelligence |
| concepts[4].id | https://openalex.org/C127413603 |
| concepts[4].level | 0 |
| concepts[4].score | 0.12124931812286377 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q11023 |
| concepts[4].display_name | Engineering |
| concepts[5].id | https://openalex.org/C201995342 |
| concepts[5].level | 1 |
| concepts[5].score | 0.10980024933815002 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q682496 |
| concepts[5].display_name | Systems engineering |
| keywords[0].id | https://openalex.org/keywords/motion |
| keywords[0].score | 0.7221735715866089 |
| keywords[0].display_name | Motion (physics) |
| keywords[1].id | https://openalex.org/keywords/task |
| keywords[1].score | 0.6647769212722778 |
| keywords[1].display_name | Task (project management) |
| keywords[2].id | https://openalex.org/keywords/computer-science |
| keywords[2].score | 0.6242836713790894 |
| keywords[2].display_name | Computer science |
| keywords[3].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[3].score | 0.39982330799102783 |
| keywords[3].display_name | Artificial intelligence |
| keywords[4].id | https://openalex.org/keywords/engineering |
| keywords[4].score | 0.12124931812286377 |
| keywords[4].display_name | Engineering |
| keywords[5].id | https://openalex.org/keywords/systems-engineering |
| keywords[5].score | 0.10980024933815002 |
| keywords[5].display_name | Systems engineering |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2403.11552 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2403.11552 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2403.11552 |
| locations[1].id | doi:10.48550/arxiv.2403.11552 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2403.11552 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5070346269 |
| authorships[0].author.orcid | https://orcid.org/0000-0002-7920-7025 |
| authorships[0].author.display_name | Shu Wang |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Wang, Shu |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5070386263 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-2649-4577 |
| authorships[1].author.display_name | Muzhi Han |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Han, Muzhi |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5084328887 |
| authorships[2].author.orcid | https://orcid.org/0000-0003-3404-3810 |
| authorships[2].author.display_name | Ziyuan Jiao |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Jiao, Ziyuan |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5100358738 |
| authorships[3].author.orcid | https://orcid.org/0000-0002-7583-1867 |
| authorships[3].author.display_name | Zeyu Zhang |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Zhang, Zeyu |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5101780958 |
| authorships[4].author.orcid | https://orcid.org/0009-0001-6768-5118 |
| authorships[4].author.display_name | Ying Wu |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Wu, Ying Nian |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5034228010 |
| authorships[5].author.orcid | |
| authorships[5].author.display_name | Song‐Chun Zhu |
| authorships[5].author_position | middle |
| authorships[5].raw_author_name | Zhu, Song-Chun |
| authorships[5].is_corresponding | False |
| authorships[6].author.id | https://openalex.org/A5043423420 |
| authorships[6].author.orcid | https://orcid.org/0000-0002-3003-8611 |
| authorships[6].author.display_name | Hangxin Liu |
| authorships[6].author_position | last |
| authorships[6].raw_author_name | Liu, Hangxin |
| authorships[6].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2403.11552 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | LLM3:Large Language Model-based Task and Motion Planning with Motion Failure Reasoning |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10181 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9940999746322632 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Natural Language Processing Techniques |
| related_works | https://openalex.org/W2748952813, https://openalex.org/W2390279801, https://openalex.org/W2358668433, https://openalex.org/W2376932109, https://openalex.org/W2001405890, https://openalex.org/W2382290278, https://openalex.org/W2478288626, https://openalex.org/W4391913857, https://openalex.org/W2350741829, https://openalex.org/W2530322880 |
| cited_by_count | 7 |
| counts_by_year[0].year | 2025 |
| counts_by_year[0].cited_by_count | 3 |
| counts_by_year[1].year | 2024 |
| counts_by_year[1].cited_by_count | 4 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2403.11552 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2403.11552 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2403.11552 |
| primary_location.id | pmh:oai:arXiv.org:2403.11552 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2403.11552 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2403.11552 |
| publication_date | 2024-03-18 |
| publication_year | 2024 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 38, 47, 117, 122, 164 |
| abstract_inverted_index.by | 91 |
| abstract_inverted_index.in | 27, 31, 121, 132, 139, 174 |
| abstract_inverted_index.of | 59, 110, 119, 130, 149, 156, 171 |
| abstract_inverted_index.on | 8, 163 |
| abstract_inverted_index.to | 62, 86, 153 |
| abstract_inverted_index.we | 35, 51, 125, 159 |
| abstract_inverted_index.LLM | 85 |
| abstract_inverted_index.and | 2, 22, 56, 67, 102, 136 |
| abstract_inverted_index.are | 25 |
| abstract_inverted_index.for | 72 |
| abstract_inverted_index.its | 89 |
| abstract_inverted_index.our | 172 |
| abstract_inverted_index.the | 53, 84, 106, 128, 137, 146, 154, 168 |
| abstract_inverted_index.LLMs | 61 |
| abstract_inverted_index.TAMP | 44, 134 |
| abstract_inverted_index.Task | 1 |
| abstract_inverted_index.rely | 7 |
| abstract_inverted_index.task | 14, 100 |
| abstract_inverted_index.with | 16 |
| abstract_inverted_index.Here, | 34 |
| abstract_inverted_index.LLM^3 | 76, 97, 131 |
| abstract_inverted_index.Large | 40 |
| abstract_inverted_index.Model | 42 |
| abstract_inverted_index.These | 20 |
| abstract_inverted_index.about | 93 |
| abstract_inverted_index.novel | 39 |
| abstract_inverted_index.tasks | 30 |
| abstract_inverted_index.them. | 115 |
| abstract_inverted_index.(TAMP) | 5 |
| abstract_inverted_index.LLM^3, | 37 |
| abstract_inverted_index.LLM^3. | 157 |
| abstract_inverted_index.Motion | 3 |
| abstract_inverted_index.action | 65, 70, 141 |
| abstract_inverted_index.design | 108 |
| abstract_inverted_index.motion | 18, 73, 78, 94, 103, 150 |
| abstract_inverted_index.refine | 88 |
| abstract_inverted_index.select | 68 |
| abstract_inverted_index.series | 118 |
| abstract_inverted_index.Through | 116 |
| abstract_inverted_index.between | 99, 114 |
| abstract_inverted_index.conduct | 160 |
| abstract_inverted_index.crafted | 10 |
| abstract_inverted_index.domain, | 124 |
| abstract_inverted_index.failure | 151 |
| abstract_inverted_index.limited | 26 |
| abstract_inverted_index.modules | 24 |
| abstract_inverted_index.present | 36 |
| abstract_inverted_index.process | 109 |
| abstract_inverted_index.propose | 63 |
| abstract_inverted_index.solving | 133 |
| abstract_inverted_index.studies | 144 |
| abstract_inverted_index.success | 155 |
| abstract_inverted_index.through | 81 |
| abstract_inverted_index.Ablation | 143 |
| abstract_inverted_index.Language | 41 |
| abstract_inverted_index.Planning | 4 |
| abstract_inverted_index.allowing | 83 |
| abstract_inverted_index.approach | 173 |
| abstract_inverted_index.emerging | 29 |
| abstract_inverted_index.failure. | 95 |
| abstract_inverted_index.feedback | 80 |
| abstract_inverted_index.handling | 111 |
| abstract_inverted_index.leverage | 52 |
| abstract_inverted_index.manually | 9 |
| abstract_inverted_index.messages | 113 |
| abstract_inverted_index.physical | 165 |
| abstract_inverted_index.planning | 15, 57, 79, 101 |
| abstract_inverted_index.powerful | 54 |
| abstract_inverted_index.problems | 135 |
| abstract_inverted_index.symbolic | 13, 64 |
| abstract_inverted_index.featuring | 46 |
| abstract_inverted_index.framework | 45 |
| abstract_inverted_index.intricate | 107 |
| abstract_inverted_index.planning, | 104 |
| abstract_inverted_index.planning. | 74 |
| abstract_inverted_index.practical | 169 |
| abstract_inverted_index.proposals | 90 |
| abstract_inverted_index.reasoning | 55, 92, 152 |
| abstract_inverted_index.selecting | 140 |
| abstract_inverted_index.sequences | 66 |
| abstract_inverted_index.settings. | 33, 176 |
| abstract_inverted_index.Crucially, | 75 |
| abstract_inverted_index.addressing | 28 |
| abstract_inverted_index.approaches | 6 |
| abstract_inverted_index.connecting | 12 |
| abstract_inverted_index.continuous | 17, 69 |
| abstract_inverted_index.efficiency | 138 |
| abstract_inverted_index.interface. | 49 |
| abstract_inverted_index.interfaces | 11, 98 |
| abstract_inverted_index.parameters | 71 |
| abstract_inverted_index.prompting, | 82 |
| abstract_inverted_index.real-world | 32, 175 |
| abstract_inverted_index.underscore | 145 |
| abstract_inverted_index.(LLM)-based | 43 |
| abstract_inverted_index.alleviating | 105 |
| abstract_inverted_index.box-packing | 123 |
| abstract_inverted_index.demonstrate | 127 |
| abstract_inverted_index.experiments | 162 |
| abstract_inverted_index.generation. | 19 |
| abstract_inverted_index.iteratively | 87 |
| abstract_inverted_index.parameters. | 142 |
| abstract_inverted_index.pre-trained | 60 |
| abstract_inverted_index.qualitative | 161 |
| abstract_inverted_index.significant | 147 |
| abstract_inverted_index.simulations | 120 |
| abstract_inverted_index.Conventional | 0 |
| abstract_inverted_index.Furthermore, | 158 |
| abstract_inverted_index.capabilities | 58 |
| abstract_inverted_index.contribution | 148 |
| abstract_inverted_index.incorporates | 77 |
| abstract_inverted_index.manipulator, | 166 |
| abstract_inverted_index.Consequently, | 96 |
| abstract_inverted_index.Specifically, | 50 |
| abstract_inverted_index.applicability | 170 |
| abstract_inverted_index.demonstrating | 167 |
| abstract_inverted_index.effectiveness | 129 |
| abstract_inverted_index.quantitatively | 126 |
| abstract_inverted_index.domain-specific | 21, 112 |
| abstract_inverted_index.labor-intensive | 23 |
| abstract_inverted_index.domain-independent | 48 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 7 |
| citation_normalized_percentile |