so101-ros-physical-ai: The ROS 2 Stack That Lets a Cheap Robot Arm Borrow a Bigger Brain
A full imitation-learning pipeline for the SO-101 arm, from teleoperation and rosbag recording to LeRobot training and remote policy inference over gRPC or ZMQ.
- The repo’s real breakthrough is remote policy inference, which lets a lightweight robot arm borrow GPU-class intelligence without turning the hardware stack into a computer science project.
- Its strongest contribution is the full loop from teleoperation to dataset conversion to training to live execution, not any single ROS package in isolation.
- The `policy_server` is the center of gravity because it favors fresh observations over backlog, which is the difference between a networked demo and a controllable robot.
- The project matters as integration plumbing: it connects ROS 2 actuation with LeRobot policies in a way that is reproducible enough to use, not just admire.
The first thing to notice is not the arm. It is the fact that the arm does not need to think on the arm. In this stack, the cheap SO-101 can stay lean on edge hardware while a heavier policy runs on a nearby GPU box and streams back actions over the network.
I’d like to share SO-101 ROS Physical AI, an open-source ROS 2 stack for the SO-101 robot arm focused on an end-to-end imitation learning workflow on real hardware.
Why this robot can think somewhere else
That split is the project’s sharpest move. Most low-cost arm stacks stop at drivers, controllers, and motion planning. Most learning repos stop at datasets and policies. This repo makes them meet in the middle, which is where physical AI becomes useful instead of theatrical.
Dmitri Manajev built it as a ROS 2-native full pipeline for a real arm, and that matters. The project is not trying to replace ROS. It is using ROS 2 for what it is good at: hardware bringup, state transport, controller switching, and orchestration. The learned policy handles the thing ROS does not: deciding what to do next from demonstrations.
From teleop to trained policy, without leaving ROS 2
The workflow is unusually complete. You can teleoperate a leader arm, record synchronized joint state and camera data, convert rosbag2 or MCAP recordings into the LeRobot format, train policies such as ACT or SmolVLA, and then deploy the result back onto the robot. That is a lot of surface area, but it is also the point.
The repo’s real center of gravity: policy_server
The most interesting technical choice is the one-slot observation queue. It is a small detail with big consequences. Instead of piling up stale frames when latency spikes, the server favors the freshest observation and drops the rest. For robot control, that is not a convenience. It is the difference between a responsive loop and a robot chasing its own past.
That logic sits inside a policy server that can speak both gRPC and ZMQ. The server handles remote inference, the client on the robot side bundles observations, and the policy returns action chunks rather than single-step guesses. That chunking matters because it smooths the loop and reduces the penalty of network jitter.
| Layer | Standard ROS 2 arm stack | Standalone LeRobot project | so101-ros-physical-ai |
|---|---|---|---|
| Hardware bringup | Yes | No | Yes |
| Teleop recording | Sometimes | No | Yes |
| Dataset conversion | No | Yes | Yes |
| Policy inference | No | Yes | Yes |
| Remote GPU support | Rare | Not native | Yes |
| ROS 2 integration | Yes | No | Yes |
| Reproducibility | Mixed | Good | Strong |
Why gRPC and ZMQ both exist here
This is a practical systems trade-off, not a philosophical stance. gRPC gives structure and a familiar service boundary. ZMQ trims overhead for high-bandwidth robot data, especially when images are flowing constantly. The repo does not pretend one transport is perfect. It keeps both because the job changes depending on latency and payload size.
The idea behind this project was to build a ROS 2-native full pipeline around a low-cost, accessible robot arm that people can actually learn on in practice.
What ROS 2 is doing, and what it is not doing
ROS 2 is the hardware and coordination layer. It handles `ros2_control`, joint controllers, Feetech servo integration, MoveIt 2, and the mechanics of switching between direct command mirroring and planned motion. It does not pretend to be the policy brain. That separation keeps the stack legible.
The result is a clean division of labor. ROS 2 keeps the robot physically honest. LeRobot keeps the learning loop statistically useful. The bridge between them is the actual innovation.
Why Pixi and Rerun make this feel unusually modern
A lot of robotics repos feel like a dependency hostage situation. This one is trying to reduce that pain. Pixi helps unify Python and C++ environment management, while Rerun gives a more time-aware debugging experience than the usual static visualization workflow. That combination signals a repo designed for repeated use, not a one-off demo.
The tooling choice matters because end-to-end physical AI only works when the developer loop is tolerable. If setup is brittle, the whole pipeline becomes harder to trust. If visualization is stale, the data never quite feels real.
How this differs from the obvious alternatives
| Question | Classic ROS 2 arm stack | Pure LeRobot training repo | This repo |
|---|---|---|---|
| Can it move the arm? | Yes | No | Yes |
| Can it collect demonstrations? | Sometimes | No | Yes |
| Can it train policies? | No | Yes | Yes |
| Can it run remote inference? | Rarely | Not natively | Yes |
| Does it connect training to hardware execution? | Only indirectly | Only indirectly | Directly |
| Is the learning loop part of the product? | No | Yes | Yes |
That is why the project stands out. It does not compete with ROS 2 drivers or with LeRobot. It makes them compatible, then wraps the result in a workflow that a serious builder can actually follow.
Wow, this is awesome. I think you should reach out to our Physical AI working group to present this project.
Awesome work @dmitri_manajev! This really looks like an amazing project to get started with Imitation Learning and Physical AI with ROS 2.
The lesson: physical AI needs plumbing, not just models
The cleanest read of this repo is also the simplest. It treats physical AI as a systems problem. Hardware, data capture, dataset conversion, remote inference, and robot execution are all first-class parts of the product. That is what turns a promising model into a usable robot workflow.
The deeper lesson is that small robots do not need small ambition. They need infrastructure that lets them borrow better compute without losing control of time, latency, or state. This repo is a strong example of how to do that.
