LatBot: Distilling Universal Latent Actions for Vision-Language-Action Models Article Swipe

MAKE IMPACT

LI Zuolei , Gao Xingyu , Wang Xiao-fan , Fu, Jianlong · YOU? · · 2025 · Open Access ·

Learning transferable latent actions from large-scale object manipulation videos can significantly enhance generalization in downstream robotics tasks, as such representations are agnostic to different robot embodiments. Existing approaches primarily rely on visual reconstruction objectives while neglecting physical priors, leading to sub-optimal performance in learning universal representations. To address these challenges, we propose a Universal Latent Action Learning framework that takes task instructions and multiple frames as inputs, and optimizes both future frame reconstruction and action sequence prediction. Unlike prior works, incorporating action predictions (e.g., gripper or hand trajectories and orientations) allows the model to capture richer physical priors such as real-world distances and orientations, thereby enabling seamless transferability to downstream tasks. We further decompose the latent actions into learnable motion and scene tokens to distinguish the robot's active movements from environmental changes, thus filtering out irrelevant dynamics. By distilling the learned latent actions into the latest VLA models, we achieve strong performance across both simulated (SIMPLER and LIBERO) and real-world robot settings. Notably, with only 10 real-world trajectories per task collected on a Franka robot, our approach successfully completes all five challenging tasks, demonstrating strong few-shot transferability in robotic manipulation.

Type

article

Landing Page

http://arxiv.org/abs/2511.23034

PDF

https://arxiv.org/pdf/2511.23034

OA Status

green

OpenAlex ID

https://openalex.org/W7108247020

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W7108247020

Canonical identifier for this work in OpenAlex

Title

LatBot: Distilling Universal Latent Actions for Vision-Language-Action Models

Work title

Type

article

OpenAlex work type

Publication year

2025

Year of publication

Publication date

2025-11-28

Full publication date if available

Authors

LI Zuolei, Gao Xingyu, Wang Xiao-fan, Fu, Jianlong

List of authors in order

Landing page

https://arxiv.org/abs/2511.23034

Publisher landing page

PDF URL

https://arxiv.org/pdf/2511.23034

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2511.23034

Direct OA link when available

Concepts

Artificial intelligence, Computer science, Generalization, Task (project management), Action (physics), Object (grammar), Machine learning, Transferability, Frame (networking), Prior probability, Robot, Robotics, Sequence (biology), Motion (physics), Downstream (manufacturing), Humanoid robot, Programming by demonstration, Cognitive neuroscience of visual object recognition, Task analysis, Generality, GRASP, Sequence learning, Categorization, Trajectory, Adaptability, Dependency (UML), Transfer of learning, Human–computer interaction

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Full payload