The First Thing We Feel

On the character decisions hiding inside robot locomotion

A character tells you who they are before they open their mouth. Before you know their name, what they want, or what they’re afraid of. All by how they move through a room. In Batman Returns, Danny DeVito’s lurching, side-to-side gait is menacing and pathetic. It signals physical wrongness, social exclusion, rage. It communicates Penguin’s full psychology. The same can be said about Terminator’s unstoppable, zero-waste march. Gollum’s broken crawl. Jack Sparrow’s unpredictable, drunken stagger.

Movement is the first thing we feel about a character.

The same is true about a robot. But the people tackling the problem are not actors, writers, or directors. They are locomotion engineers, working in real time to solve one of the most difficult problems in a field jam-packed with difficult problems — how to make a legged robot walk. Yes, it’s been done in controlled environments. You’ve seen the parkour videos and dance battles. But that’s a lab or a stage. Navigating the enormous pin ball machine that is the world? Different story.

Meanwhile, the arrival of LLMs has promised a robotics revolution! Which means engineers are themselves racing to make robots’ bodies worthy of the magical brains inside of them. The problem is that this technical mountain is so steep that anything not absolutely critical gets deferred. For example, thinking about artsy-fartsy things like what a robot’s walk says about its character… tends to slide to the bottom of the to-do list. Which means the most consequential character decision in robotics — the first thing we feel — is often a byproduct of more immediate concerns.

Prof. Dimitrios Kanoulas (left) and Dr. Xue Bin “Jason” Peng (right)

To get a better understanding of how deep the “make it walk” challenge goes, and where character choices might fit in, I spoke with two world experts in the field:

Prof. Dimitrios Kanoulas leads the Robot Perception and Learning Lab at University College London. His UKRI Future Leaders Fellowship, called RoboHike, focuses on teaching a robot to navigate farmland, disaster zones, and rocky terrain autonomously.

Dr. Xue Bin “Jason” Peng is currently an Associate Professor at Simon Fraser University and a Research Scientist at NVIDIA. He authored the foundational paper on using deep reinforcement learning to imitate footage of humans and animals.

Dimitrios and Jason explained that it all starts with striving to replicate how animals and people move. Because the environments robots must navigate — homes, offices, sidewalks, factories — were built for us. Knobs at hand height. Stairs spaced for human legs. Doorways wide enough for human shoulders. And because living things, even as infants, are effortlessly good at something robots currently suck at: reading changing terrain and adjusting in real time.

So, how does one teach a robot to move like an animal or person? It has to be trained on the data, which, historically, has been collected via two methods. The first is motion capture: actors wear tracking suits in a high-tech volume. The second is video reconstruction: extracting movement from normal footage that was shot for entirely different reasons. Mocap currently wins on quality. Video wins on abundance. But, even as I type these words, the terrain is shifting and a new, third option is emerging.

Below is a closer look at how each method works (or, if you want to skip this part like I skip the math in an Andy Weir novel, scroll down to the “character decisions” heading).

---

MOTION CAPTURE

Motion capture places reflective markers on an actor’s body at key anatomical points — shoulders, elbows, knees, spine — and tracks them with a ring of infrared cameras. As the actor moves, the system records the position of every marker in 3D, building a precise skeletal map of the performance. The data that comes out is clean and precise down to 3mm. The tradeoff is cost. You need a volume, you need a suit, and you need to bring the movement to the stage with actors rather than capturing it in the world.

Mocap is best for training humanoid robots. The actor is human, the robot is built like a human, and the anatomy maps most cleanly from one to the other. For quadruped robots, or any other morphology, it’s a different story. Yes, you can put a human in a suit and have them move on all fours — like Andy Serkis did it for Gollum and Caesar in Planet of the Apes — but a human imitating an animal is already a translation away from the thing you want. That translation costs something. Animals wearing motion capture suits is possible, but even more costly and difficult to pull off.

VIDEO RECONSTRUCTION

The alternative to mocap is training the robot on footage of something that already moves how it moves. For humanoid robots, the source footage is people. For quadrupeds, it’s animals. The appeal is obvious. The footage is abundant, cheap, and captures real animals moving through real environments rather than humans approximating the conditions. The catch is that it’s incredibly hard to make work inexpensively, at scale. Pose estimation software must interpret the animal’s 3D skeleton from the 2D footage, and the quality of that reconstruction depends on how the footage was shot.

Dimitrios explained that if all four of an animal’s feet aren’t visible in the frame simultaneously, there is no reconstruction algorithm that can fill in the missing data. His team has worked extensively on capturing cheetah movement with very high-tech equipment: multiple synchronized cameras, bait on a string to provoke a sprint, a controlled environment built to maximize visibility. And yet, “still super hard,” he says. Even with all that infrastructure in place, synchronizing the cameras to the millisecond (which fast animals require) remains a persistent problem.

IMITATION LEARNING

Whether the sourcing method is mocap, video reconstruction, or a combo of the two, that data is then translated into something a robot can learn to walk from. This pipeline — called imitation learning — boils down to four main steps: (1) retargeting maps the source skeleton’s motion onto the robot’s mismatched skeleton; (2) simulation training teaches a digital twin to move like the retargeted data; (3) sim-to-real transfer plugs those learnings onto the physical robot; and then, (4) fingers crossed, it walks. And yes, “fingers crossed” is an official step, because within each of these are countless sub-steps and complications I’m skipping over entirely.

LEARNING WITHOUT IMITATION

The above methods depend on capturing movement from living things. But a third approach asks a different question entirely: what if the robot figured it out on its own? Engineers drop a digital twin of a robot into a virtual physics environment and let it discover movement through trial and error. What makes this compelling is scale. A simulation can run thousands of iterations overnight that would take years to capture on a mocap stage or track down through licensing. As legal pressure on scraped data mounts, that advantage only grows. The tradeoff is that movement discovered this way can be both perfectly stable and creepily unnatural. Optimized for physics that was modeled, not physics that was lived. Or to be lived with.

This is why, if fully simulated learning models win out, they will be built on the backs of real people and animals — using mocap and video reconstruction as a foundation — right up until those methods get put out to pasture. A lot like TV writers making “temporary” side money by teaching AI how to write TV shows. Sorry. Personal gripe.

---

CHARACTER DECISIONS

The question motivating engineers’ decision-making throughout the stack described above is this: “What is the minimal possible input I can capture from a living thing — how fast, how cheap, how scalable — to replicate the motion?” The character of how it moves is a champagne problem. And yet, it strikes me that character decisions are being made throughout. Take video reconstruction. A four-legged robot trained to walk on footage of wolves is not the same character as one trained on French poodles. That response isn’t arbitrary. It’s evolutionary. Humans come hard-wired with our own instincts about the difference between wolves and poodles. Now, in a very bizarre case of upside-down and inside-out Darwinism, robots inherit how people perceive and respond to them, scaled across millions of interactions.

Along the way, someone is deciding which clips, of which animals or people, showing which behaviors, and labeling which movements express which emotional states. Right now the character of movement is a byproduct, but I believe with time this “annotation layer” will be one of the most consequential decisions made in robotics.

But footage selection is only the first step. Once a robot has a stable gait, the question becomes what it expresses while it moves. Curious. Cautious. Excited. These are not decorative states. A robot that communicates its internal condition through movement is one people can read — and a robot people can read is one they can trust. Jason confirmed that humanoid robotics companies are already putting actors in mocap volumes and having them “play out all these character-based scenarios.” Somewhere right now, an actor wearing tracking dots is performing hesitation or curiosity or delight so a robot can learn what it looks like. But who is directing them?

The same question applies when the robot starts struggling. The most endearing moment in any story is not when a character succeeds. It’s when they don’t. Good news, robots fail. A lot. They stumble on thresholds, lose their footing, hit the edge of a capability and lag. Right now, these moments are designed to be minimized. The failure is treated as a problem to be hidden rather than a moment to be performed.

But a failure should be a scene. And like any scene, it can be written well or written badly. A robot that physically expresses its struggle does something remarkable — invites people to put their hands on it and help. That moment of physical contact is not a UX footnote. It’s one of the most potent relationship-building events in Human-Robot Interaction. The user has crossed from observer to participant. They have done something for the robot. And what we do for something, we become attached to it.

Just like owning a pet. Or building anything from IKEA. Or parenthood.

A wheeled-legged robot for doorstep delivery made by RIVR (now Amazon).

As we were wrapping up our conversation, Dimitrios raised something that complicates the whole picture above. He believes that two or four-legged robots will start giving way to wheeled-leg hybrids. A “best of both worlds” mashup that rolls when terrain allows and walks when it demands. For example, the robots made by RIVR (recently acquired by Amazon). If he’s right, it complicates the animal-imitation pipeline even further. There is no animal that walks and rolls simultaneously. No reference footage for a gait that has no biological precedent. This is where Dimitrios and I pondered with straight faces: footage of dogs wearing roller skates? The character question doesn’t go away. It gets harder.

In Batman Returns, Penguin’s walk came from a script. From a character. Burton and DeVito started with who this person was and built the movement outward from that center. Robotics is doing this in reverse. The movement comes first — shaped entirely by what’s technically achievable — and character, if it comes at all, gets retrofitted downstream. But that inversion is not permanent. It’s a bug of this particular moment, when getting the damn robot to just walk is still one of the biggest problems.

That moment will pass. The gaits will stabilize. The sim-to-real gap will narrow. And character people will be in the mix deciding what all that movement actually says. Jason Peng’s closing thought: “It is something I hope robotics will start focusing on more, once we figure out some of the basics.”