Robbyant, the embodied AI unit inside Ant Group, has released the LingBot-VA 2.0.The first embodied-native foundation model. It describes a video-action foundation model for generalist robot manipulation. The research team pretrains the whole stack for embodiment instead of fine-tuning a video generator.
What is LingBot-VA 2.0?
Most video-action models reuse two components built for digital content creation. One is a reconstruction-oriented VAE. The other is a bidirectional video-diffusion backbone, with an action module attached.
This creates three limitations. Pixel-reconstruction latents preserve appearance but carry limited physical structure. Iterative denoising over video tokens is too slow for closed-loop control. Generic video objectives never teach how actions reshape the world.
A fourth mismatch is structural. Backbones use bidirectional attention, while control unfolds strictly forward in time. LingBot VA Version 1.0 finetuned that stack into a causal model. Version 2.0 pretrains a causal DiT natively.
Version 1: The Semantic Visual-Action Tokenizer
Building on that motivation, stage one replaces the compression-only VAE. Following RepWAM, the tokenizer adds two objectives to reconstruction.
Semantic alignment pulls visual latents toward a frozen Perception Encoder teacher. A latent-action objective extracts compact transition variables between consecutive latents. An inverse dynamics model predicts each latent action. A forward dynamics model decodes it into a transport map plus residual.
World states and actions now share one latent space. Unlabeled web video therefore carries action-relevant supervision.
Version 2: A Causal DiT With a Sparse MoE Video Stream
On top of that space, version 2 pretrains a causal DiT. It keeps the Mixture-of-Transformers layout of version 1.0. A video expert and an action expert share one causal self-attention. Each owns a separate feed-forward pathway.
The two streams scale asymmetrically. The video expert replaces its dense FFN with a sparse MoE routed layer. That layer holds 128 routed SwiGLU experts, top-8 routing, one shared expert. Load balancing follows the auxiliary-loss-free Loss-Free Balancing strategy. The action expert keeps a dense FFN at hidden dimension 768.
The video backbone is roughly 13.0B parameters, about 1.9B active. With the action expert and MCP heads, training covers about 15.3B parameters. Roughly 2.5B activate per token at inference. Training uses a rectified-flow objective with a hybrid Muon plus AdamW optimizer.
Where the Training Signal Comes From
Beyond architecture, two objectives shape what the model learns.
Multi-chunk prediction (MCP) fixes myopic supervision. Teacher forcing supervises only the next chunk, so the model can cut loss by copying appearance. MCP attaches three lightweight modules predicting the next three chunks. In ablation it matched the baseline’s 45k-step accuracy in 20k steps, a 2.3x training speedup.
Meanwhile, five objectives are co-trained rather than staged: T2I, T2V, TI2VA, ICL, and human-robot co-training. Sampling follows a coarse-to-fine schedule, from appearance grounding to video-action control. Keeping every objective alive avoids forgetting the earlier priors.
Hierarchical Planning
Chunk-level control cannot sequence long-horizon goals. Above the policy therefore sits a VLM planner, LoRA-finetuned with a frozen vision tower. It emits structured JSON: done, instruction, generation_instruction, local_scene_description. It runs at about 2 Hz behind an asynchronous shared buffer. The policy reads it at each chunk boundary, so planner latency never blocks execution.
Foresight Reasoning
Even with a sparse backbone, deployment hits a serial bottleneck. If the robot waits, model latency becomes control latency.
Foresight Reasoning therefore runs prediction and execution as asynchronous streams. While the robot executes chunk a_t, the video expert imagines its outcome. The action expert decodes a_{t+1} from that.
Running ahead risks drift. So each returning observation is encoded into the true latent z_{t+1}, overwriting the imagined one. A forward-dynamics grounding loss trains the video expert for this role.
# Not runnable: policy, executor and encode() are placeholders.
C = init_kv_cache(encode(obs_0)) # feedback-grounded cache C_t
a = policy.action_expert(C) # cold start: first action chunk a_0
while not done:
executor.start(a) # execution stream, non-blocking
C_tmp = C + [a] # prediction stream: C_t u {a_t}
z_hat = policy.video_expert(C_tmp) # forward dynamics -> imagined z_{t+1}
a_next = policy.action_expert(C_tmp + [z_hat])
obs = executor.wait() # real observation of a_t returns
C = overwrite(C_tmp, z_hat, encode(obs)) # re-ground: z_hat <- true z_{t+1}
a = a_next
Performance
Consequently, evaluation covers simulation and real hardware. On RoboTwin 2.0, every model trains on 2,500 clean plus 25,000 randomized demonstrations, across 50 tasks.
Distillation cuts the video sampler from 5 steps to 2, and the action sampler from 10 to 2. FP8 TensorRT engines, a paged/ragged KV cache with FlashInfer attention, and host-side overhead removal supply the rest.
K = 32 # low-level control steps inside one generated chunk
stack = [(“BF16 PyTorch async rollout baseline”, 927),
(“+ Consistency distillation”, 466),
(“+ Low-precision compiled execution”, 369),
(“+ Long-horizon attention optimization”, 272),
(“+ Runtime overhead reduction”, 142)]
for name, ms in stack:
print(f”{name:40s} {ms:4d} ms {round(1000 / ms * K):4d} Hz”)
print(“end-to-end speedup:”, round(927 / 142, 1), “x”)
Version 1.0 vs Version 2.0
The tokenizer ablation isolates row one. Swapping the WAN2.2 VAE for the semantic tokenizer lifts a 1.3B model from 78.0 to 86.6.
Use Cases and Examples
Beyond benchmarks, four deployment shapes stand out.
- Few-shot onboarding: The report states the model adapts from 10 to 15 demonstrations. Real-world evaluation uses 20 teleoperated demos per task. One multi-task checkpoint covers all four evaluated tasks.
- Demonstration-conditioned control: In-context learning lets a human demonstration video replace the text instruction. After finetuning on four seen tasks, the policy executed unseen compositions. One example: “put the calabash into the green plate.”
- Cheap data scaling: Human-robot co-training retargets hand poses into the robot action space. Each hand becomes a virtual parallel gripper. The egocentric corpus spans 65.4k episodes.
- Reactive control: Demonstrations include Air Hockey and a conveyor belt, where the policy anticipates moving objects.
Key Takeaways
- Pretrains a causal video-action DiT from scratch instead of adapting a video generator.
- A semantic tokenizer puts world states and latent actions in one aligned space.
- Sparse MoE video stream: ~2.5B of ~15.3B parameters activate per token.
- Foresight Reasoning overlaps prediction with execution, re-grounded on every real observation.
- Chunk latency 927 ms to 142 ms; async control 35 Hz to 225 Hz.
Interactive Dynamic Explainer
Check out the Paper and Project Page. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
Note:Thanks to the Ant Research team for the thought leadership/ Resources for this article. Ant Research team has supported this content/article for promotion.
