"Make an omelet" is the easy part. A language model can write that recipe: crack the eggs, whisk, heat the pan. Cracking the egg is where robots stall. Jitendra Malik (Arthur J. Chick Professor at UC Berkeley and VP and Distinguished Scientist at Amazon's Frontier AI and Robotics lab) used that example at ETH Zurich to show where the hard problem sits: below the plan, in the trajectory the hand follows and the force it applies. A recipe specifies neither. Three other researchers spoke that same afternoon, and each pointed at a different missing piece.
On 29 May 2026, ETH Zurich held four distinguished lectures under the title Frontiers of Embodied AI, and the recordings are now on YouTube. This post walks through them in order: Malik on why vision and robotics still lack hands, Vladlen Koltun on spatial cognition in frontier models, Yann LeCun on world models, and Shuran Song on making manipulation data easier to collect.
What is embodied AI, and why these four?
Embodied AI is the study of systems that perceive, reason, and act in the physical world. ETH's own description of the event frames it as machines moving "beyond passive perception" to actively understand and interact with their surroundings. Language models learned from the written record of human knowledge. Robots have no equivalent record. Malik spends part of his lecture on why teleoperation, human video, and simulation each fall short as sources of training data, and Song's lecture is built around the same shortage.
I watched all four recordings, about 3 hours 17 minutes in total, and they read as answers to four different questions. Malik asks what a robot's body needs: trajectories, contact, force, and hands with more than two fingers. Koltun asks how well frontier models understand space. LeCun's question is how a system can learn a world model that predicts what an action will do. Song's is how to collect manipulation data that carries contact information and transfers across robots. The four don't agree on which gap matters most, and I'll say where I land at the end.
1 - Vision And Robotics For Embodied AI
Malik splits robotics into three problems: navigation, locomotion, and manipulation. By his account navigation is close to solved, locomotion is moving fast, and dexterous manipulation is still wide open. The reason is a hierarchy of three levels, and he names each one after a thinker:
Goals and plans (Aristotle): "I want to make an omelet." Language models and vision-language models (VLMs) belong here, because a recipe is semantic and procedural knowledge.
Trajectories (Euclid): "pick up" can mean a sugar cube or a heavy tray, and the two motions look nothing alike. The verb carries the meaning, not the movement.
Forces and torques (Newton): grip force, friction so the object doesn't slip, compensation for gravity, reaction to contact. To change things in the world, you apply forces.
Most of the action sits in the bottom two levels, and he calls language the easy part. The hard part is cracking the egg successfully 99.99% of the time, across locations, objects, and clutter. He goes as far as calling the field's focus on VLMs and vision-language-action models (VLAs) a distraction for the robotics community. He doesn't say they're useless; he says they're aimed at the wrong half of the problem.
Navigation shows what works. GOAT ("Go to Any Thing") passes what Malik calls the Airbnb test: drop a Spot robot in an unfamiliar rental, give it a goal as a category, an image, or a sentence, and it explores, builds a partial semantic map, and remembers what it finds. In the paper, it reached 83% success across 9 unseen homes, climbing from 60% on the first goal to 90% by goals 5–10 as it learned each house. The design is modular on purpose: strong vision components, with learning reserved for deciding where to explore next. Malik says end-to-end policies did worse.
Manipulation breaks every shortcut. His credo is that human-like physical intelligence needs multifingered hands; his test case is a $250 bottle of wine, where he'd want a power grasp, not two points of contact. The usual data sources each fail in a different way. Teleoperation turns the human into a slow puppeteer with no sense of touch. Human video shows the wrong body and carries no forces. Simulation supplies forces but needs rewards: pouring takes a term for every stage, roughly 10 for a typical task and as many as 150 for some, each with coefficients to tune, and the robot may hack them. He calls the sim-to-real gap "just more engineering." Reward design is what he treats as the real bottleneck.
His answer borrows from Alan Turing, who suggested building a machine that learns like a child. A child watching a parent wash dishes doesn't copy motor commands. She picks up the structure of the task, then plays at it in her own body. Malik's robot version runs real to sim to real: lift the human and the objects out of RGB video into 3D, retarget the motion to the robot's body, then practice. VideoMimic ran that loop for humanoid locomotion from roughly 100 recorded sequences.
Then comes touch. Vision can't measure contact forces, so Malik adds tactile signals and argues for "next-touch prediction," the tactile cousin of next-token prediction. ViTacFormer (Heng, Geng, Zhang, Abbeel, and Malik) builds it: a prediction head anticipates future contact signals, and the paper reports roughly 50% higher success than prior state-of-the-art systems, plus an 11-stage task sustained for 2.5 minutes. In the lecture he says touch cuts the demonstrations needed from a few thousand trials to a few hundred, and lifts the ceiling from around 85% to 99%. He's candid about the distance left: a policy that learned to peel with a knife from human preference is, in his words, not ready for a fancy restaurant.
Takeaway: Language is the easy interface; physical interaction is the real intelligence problem.
2 - On Spatial Cognition In Frontier Models
"This thing cannot find the bathtub to save its life." That was Vladlen Koltun (Distinguished Scientist at Apple) describing GPT-4o dropped into a simulated apartment and asked to find the bathtub. It wandered, thrashed about, and never worked out where it was. A few months earlier, GPT-4V had passed his personal Turing test for understanding jokes in images. Koltun calls the gap spatial cognition: knowing where things are, where you are, and how to get somewhere.
He tells the lecture as a story in five acts, and he's a convert. He was a lifelong member of "Team Embodiment," helped popularize the term "embodied AI," and in 2023 argued the opposition at a CVPR debate on whether embodiment is essential. Moravec's paradox is the setup: perception and motor control are hard for computers, while chess and mathematics are easy. The bottom-up reading says follow evolution and build perception and action first. The top-down counter asks, "Why not do the easy things first?" The idea is to use language and the digital record of civilization as a lever. Koltun calls it the "insane, bonkers top-down program," and says it worked better than he and other embodiment researchers expected: "Being wrong is more interesting than being right."
It hasn't worked for space. His group built a benchmark from classic cognitive-psychology tests: mental rotation, perspective taking, maze completion, and Tolman's shortcut test for rats. The answer in Does Spatial Cognition Emerge in Frontier Models? was "Not yet." Models landed near chance on a number of classic tests of animal cognition. Newer results are uneven:
BabyVision (January 2026): the best models sit at roughly the level of a human 3-year-old. Koltun gives both readings: nowhere near 6-, 10-, or 12-year-olds, or only now at a 3-year-old's level. In the paper, the top model scores 49.7% against 94.1% for adults.
MentisOculi (February 2026): the best models match humans who get 5–10 seconds with an image, and fall behind humans given longer.
Perspective taking: superhuman, but by a geometer's route: estimate centroids, draw lines, write code, measure angles.
Maze completion: from about 10% of human level to above 30%.
Mental rotation: little progress.
He calls the pattern "the sparks of spatial cognition."
The technical centerpiece is Do Multimodal Models Imagine Electric Sheep?. The group takes an open model, Qwen3.5 9B, and trains it by behavior cloning on puzzles such as Sokoban, Tangram, and Rush Hour. An expert solver writes the demonstrations, and the model must output the whole solution in one pass with no feedback from the puzzle. Success on held-out puzzles rises from 13% to 83%. Koltun flags that these are new instances from the same domains, not out-of-distribution tasks. Then the probes. Visual decoders, trained separately on the frozen model's activations, reconstruct the intermediate board state, though nobody trained the model to produce images. Add visual supervision and the pictures sharpen. Give the model visual tokens as a scratchpad and it reaches about 89%.
His predictions are plain. Spatial cognition is "on the critical path to what people call superintelligence." It will probably extend the top-down program rather than replace it. Embodiment still matters, but he defines it as "learning from first-hand experience," which can mean a simulated apartment instead of a humanoid. If it works, he calls it a Copernican revolution: a route to intelligence that nature never had access to.
Takeaway: Be empirical, not ideological: being wrong about the path is how you find a better one.
3 - World Models: Enabling The Next AI Revolution
A language model trains on about 20 trillion words. Yann LeCun (Executive Chairman of AMI Labs and Professor at NYU) converts that to roughly 30 trillion tokens at about 3 bytes each: 10¹⁴ bytes of text, which would take a human around 400,000 years to read. Then he runs the same sum on a four-year-old. About 2 million optic nerve fibers, each carrying roughly 1 byte per second, over 16 waking hours a day, lands at the same 10¹⁴ bytes. Same volume, different input. "We're not going to get to anything like human-like intelligence by just training on text."
The child's data is continuous video, and redundancy is what self-supervised learning feeds on. LeCun also rejects the idea that more tokens means more thinking: "The way you coerce an LLM to do reasoning is that you trick it into generating more tokens." His replacement is an action-conditioned world model, a system that predicts the next state of the world given the current state and an action. The loop runs like this:
Perceive the world and build a state, filling gaps from memory.
Propose a sequence of actions.
Let the world model predict where they lead.
Score the outcome with a cost: zero means the task is done, higher means further away.
Optimize the action sequence to minimize that cost.
He points out that this is model predictive control, "very classical stuff in optimal control going back to the 1960s." Safety constraints enter as extra cost terms checked at every step. He contrasts that with LLM safety, which depends on fine-tuning and falls to jailbreaks.
One timescale isn't enough. LeCun's example: he's in his office at NYU and wants to be in Paris tomorrow. Nobody plans that in muscle actions, 10 milliseconds at a time. You plan to reach the airport, then to hail a taxi, then to leave the building, and only at the bottom do familiar motor policies take over. The top-level plan stays rough ("maybe roughly an hour, an hour and a half" to the airport). He calls hierarchical planning unsolved: "Nobody knows how to do this."
The prediction target matters just as much. Rotate a camera around an auditorium and a model can predict that it's an auditorium. It can't predict what each person looks like or which seats are filled. Forced to output one pixel-level future, it averages incompatible possibilities and returns a blur. Video generators make "cute videos," but "do they actually understand the world? The evidence is no." So he predicts in representation space. A Joint Embedding Predictive Architecture (JEPA) encodes both the input and the outcome, then predicts the outcome's representation from the input's and the action. The encoder is free to drop whatever can't be predicted.
That freedom has a catch. The encoder can output the same constant for everything, and the prediction problem becomes trivial. This is collapse, and there are two families of fixes. Contrastive methods push the energy up on bad pairs. Regularized methods limit how much of the representation space can have low energy; LeCun prefers these. His conceptual favorite is SIGReg (Sketched Isotropic Gaussian Regularization) from LeJEPA (Balestriero and LeCun). It keeps a batch of embeddings close to an isotropic Gaussian by projecting them onto random 1D directions and testing each marginal. The paper says it needs no stop-gradient and no teacher-student network, and fits in about 50 lines of code. He concedes that the methods scaled furthest so far are the distillation ones, I-JEPA and V-JEPA, where the target encoder is a slow-moving average.
The evidence is on video. V-JEPA masks a large chunk of a clip and predicts the representation of the whole. Show it a ball that vanishes mid-flight and its prediction error jumps. LeCun says it's the first time he has seen a fully self-supervised system pick up some level of common sense. V-JEPA 2.1, probed for depth from a single image, beat DINOv3 (0.307 RMSE on NYUv2, ahead of a 7-billion-parameter DINOv3 backbone, per the paper).
Robots are where he says the stakes sit. A robot in empty space is easy to simulate. Contact, friction, deformation, and unmodeled objects are not. His answer isn't a better simulator but a learned model of "the system you control and its interaction with the environment," accurate enough to plan with and silent on microscopic detail. A world model, in his view, is not a digital twin and not a video generator. He shows a DINO-based world model planning simulated pushing and pendulum tasks in fewer than 25 steps.
His advice to researchers is blunt:
Skip LLMs and generative models: "If you want physical AI: don't work on LLMs." "Don't work on generative models either."
Prefer JEPAs: energy-based, regularized, learning from observation.
Treat reinforcement learning as the last step: "what you do when you're desperate." He doesn't mean never; he means most learning should come from watching, with RL on top of good representations.
Take hierarchical planning: "completely open," and a great PhD topic.
Takeaway: Predict what can be predicted and throw away the rest.
4 - Making Manipulation As Simple As Possible But Not Simpler
A robot hand reaches for a pinch of salt. The hand blocks the camera's view of its own fingers, so the policy can't tell whether it grabbed anything. Shuran Song (Assistant Professor at Stanford University) says that step stayed surprisingly hard until the hand could sense contact. It's a small failure, and it's her lecture in miniature: simplify a data pipeline too far and you remove the signal the robot needs. Her title is the design rule. Make collection simple enough for non-experts, but don't simplify away contact, force, or the robot's body.
She sets up two foils, and they're the same two Malik raised in Section 1. Teleoperation needs specialized hardware, an unnatural interface, and the robot itself, once per embodiment; she calls it "sugar for robotics." Human video is natural but carries no robot actions, no contact forces, and no link between a human body and a robot one.
Her answer is UMI (Universal Manipulation Interface, paper): a handheld gripper with a GoPro mounted on it. A person demonstrates a task directly, say washing a plate. The camera records what the gripper sees, and visual tracking recovers the gripper's 3D motion. That yields paired data: observations in, end-effector actions out. The representation is embodiment-agnostic on purpose. It says where the end effector should go, and a robot-specific controller turns that into joint commands. Swap the controller and the same policy runs on an arm, a quadruped, or a drone. In the lecture, a drone inserts a lightbulb (shown at 8× speed) from a policy trained on data collected around Stanford and deployed at CMU. She also showed UMI-Underwater, where demonstrations are collected on dry land. The shared data website holds over 1 million open-source demonstrations, and she says about 99% came from labs and companies other than hers.
The first UMI recorded vision and motion only. The operator felt contact; the dataset didn't. So she adds one channel at a time:
Sound: A commercial six-axis force/torque sensor runs about $5,000, breaks when dropped, and is hard to sync with cameras. A contact microphone costs a few dollars and syncs through the camera's audio. In ManiWAV, it hears when contact starts, whether a bottle is full, which side of a strip of Velcro it's touching. Random music, speech, and robot noise added in training teach the policy to ignore motor sound; without that augmentation, she says, audio can hurt. The limit is that a microphone hears contact, not force: it can't say the eraser is pressing too hard. Her advice: "If you're working on contact-rich task, you should add the contact microphone to your robot and you won't regret it."
Force: CoinFT, built in Mark Cutkosky's lab at Stanford, is a capacitive six-axis force/torque sensor the size of a US quarter, weighing 2 g, with under $11 in materials. UMI-FT puts one on each finger. Force data alone changes nothing if the policy only outputs low-frequency position targets, so an adaptive compliance policy also predicts how much to give, and a faster controller reacts. The paper tests whiteboard wiping, skewering zucchini, and lightbulb insertion.
Hands: DexUMI extends the idea to multifingered hands. A wearable exoskeleton limits the human's fingers to what the robot hand can do, and joint encoders line up with the robot's joints, so recorded positions go straight to the robot. No hand tracking, no retargeting. Tactile sensing records contact, which is what finally made the salt step work. The paper reports an 86% average success rate across four tasks on two robot hands.
One gap remains between data and robot. UMI's policy can propose a trajectory the robot can't follow: a fixed arm can't reach it, or a drone reaches the point but can't accelerate hard enough. The communication is one-way. Embodiment-Aware Diffusion Policy, from UMI-on-Air with Guanya Shi's group at CMU, lets the controller talk back. At each denoising step the controller scores the noisy trajectory with a tracking cost, and the gradient nudges sampling toward trajectories that body can execute. It needs a policy that produces diverse trajectories, and it needs no retraining and no new demonstrations. She ties it to a manipulation foundation model that doesn't commit to a robot until inference. Steering narrows the gap most for the most constrained robots, though she says some gap remains.
Then comes the claim she cares about most. "The data problem is actually not a quantity problem." A pick-and-place robot running nonstop for a week could log about 600 GB, more than the roughly 570 GB of text behind GPT-3 on her slide. For anyone with a different robot or task, that data is nearly useless. The bottleneck is reuse. She doesn't claim UMI is the only fix: "I really think that UMI could represent a massive shift when we're thinking about the data problem." And her last point is a door left open: "There's no data size too small," if it's portable enough to aggregate.
Takeaway: Collect the smallest thing that still carries contact, and make it portable.
So Which Gap Is The Real One?
Start with what each system is allowed to sense. Koltun's models see an image or a puzzle board. LeCun's V-JEPA sees video. Malik's ViTacFormer and Song's gripper also sense contact. That list is most of my answer: the four lectures describe two different gaps, and they come in sequence, not in competition.
Koltun's gap is representation. A model that can't find the bathtub has the information, since the apartment is on screen, and lacks the structure to use it. That's why training moved the electric-sheep puzzles from 13% to 83%, and I think he's right that this gap keeps closing under the top-down program. Mental rotation barely moved, but the trend is his.
Malik's gap is information. A policy can only use what its sensors record, and a camera doesn't record how hard a hand is pressing. No amount of modeling recovers a signal that was never captured. Song's contact microphone is the cleanest evidence in the four lectures: a few dollars of hardware let a policy tell the two sides of a strip of Velcro apart. Space is on the screen. Contact isn't.
Here I disagree with Koltun. His third prediction says embodiment can mean a simulated apartment instead of a humanoid. For space, fine. For contact, a simulator supplies forces, but only the physics someone wrote down, and Malik says the cost moves to reward design: roughly 10 terms for a typical pour, as many as 150 for some, each tuned by hand and each open to hacking. Malik also calls the sim-to-real gap "just more engineering," and that bill is still unpaid. Song's sensors record physics nobody wrote down. The robot that cracks the egg will have felt eggs.
The argument stops short of one question. LeCun's world model predicts in representation space, and he says to learn a model of "the system you control and its interaction with the environment." His evidence is visual: a ball that vanishes, a depth probe. The V-JEPA 2.1 paper reports a 20% gain in real-robot grasping success over V-JEPA 2-AC, which measures whether the grasp worked, not whether the model predicted the force. Malik's next-touch prediction is the touch version of the same idea. Whether a world model trained on video learns anything about contact, or needs touch in the loop first, none of the four settled.
The full playlist is on YouTube, and each lecture stands on its own. Thanks to ETH Zurich's Computer Vision and Geometry Group for organizing the event.