Robotics Engineering Technical Skills

Exciting updates on Project GR00T! We discover a systematic way to scale up robot data, tackling the most painful pain point in robotics. The idea is simple: human collects demonstration on a real robot, and we multiply that data 1000x or more in simulation. Let’s break it down: 1. We use Apple Vision Pro (yes!!) to give the human operator first person control of the humanoid. Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real time. From the human’s point of view, they are immersed in another body like the Avatar. Teleoperation is slow and time-consuming, but we can afford to collect a small amount of data.  2. We use RoboCasa, a generative simulation framework, to multiply the demonstration data by varying the visual appearance and layout of the environment. In Jensen’s keynote video below, the humanoid is now placing the cup in hundreds of kitchens with a huge diversity of textures, furniture, and object placement. We only have 1 physical kitchen at the GEAR Lab in NVIDIA HQ, but we can conjure up infinite ones in simulation. 3. Finally, we apply MimicGen, a technique to multiply the above data even more by varying the *motion* of the robot. MimicGen generates vast number of new action trajectories based on the original human data, and filters out failed ones (e.g. those that drop the cup) to form a much larger dataset. To sum up, given 1 human trajectory with Vision Pro  -> RoboCasa produces N (varying visuals)  -> MimicGen further augments to NxM (varying motions). This is the way to trade compute for expensive human data by GPU-accelerated simulation. A while ago, I mentioned that teleoperation is fundamentally not scalable, because we are always limited by 24 hrs/robot/day in the world of atoms. Our new GR00T synthetic data pipeline breaks this barrier in the world of bits. Scaling has been so much fun for LLMs, and it's finally our turn to have fun in robotics! We are creating tools to enable everyone in the ecosystem to scale up with us: - RoboCasa: our generative simulation framework (Yuke Zhu). It's fully open-source! Here you go: http://robocasa.ai - MimicGen: our generative action framework (Ajay Mandlekar). The code is open-source for robot arms, but we will have another version for humanoid and 5-finger hands: https://lnkd.in/gsRArQXy - We are building a state-of-the-art Apple Vision Pro -> humanoid robot "Avatar" stack. Xiaolong Wang group’s open-source libraries laid the foundation: https://lnkd.in/gUYye7yt - Watch Jensen's keynote yesterday. He cannot hide his excitement about Project GR00T and robot foundation models! https://lnkd.in/g3hZteCG Finally, GEAR lab is hiring! We want the best roboticists in the world to join us on this moon-landing mission to solve physical AGI: https://lnkd.in/gTancpNK

Agentic AI systems are moving beyond digital environments and into the physical world. We can now see this technology in motion through robotics, autonomous vehicles, and smart infrastructure. How do agents work alongside us in real environments? Our latest AWS Open Source blog explains how teams can build intelligent physical AI systems that bridge edge and cloud computing. By combining Strands Agents SDK, Amazon Bedrock AgentCore, Claude 4.5, NVIDIA GR00T, and Hugging Face LeRobot, customers can create agentic systems that leverage cloud-scale reasoning while maintaining millisecond responsiveness for real-time physical interaction. The architecture enables edge devices to handle fast, instinctual responses while the cloud provides deliberate reasoning and fleet-wide learning. We're seeing remarkable results—from robotic arms performing complex manipulation tasks to autonomous systems that continuously improve through shared experience. Learn about building intelligent physical AI with agentic systems in this deep dive from our team: https://lnkd.in/gEJVuF5F

The timeline for humanoid robots to work completely independently without human assistance depends on advancements in AI, robotics, and sensory technologies. When do you think that would be possible? Here are the key considerations: 1. Current Capabilities Humanoid robots like Boston Dynamics' Atlas, Tesla's Optimus, and Hanson Robotics' Sophia can perform tasks such as walking, object manipulation, and basic communication. However, these tasks often rely on pre-programmed behaviors or limited autonomy. Atlas excels in dynamic movement but lacks decision-making for complex, real-world tasks. Optimus is designed for simple repetitive tasks in controlled environments. Sophia can hold conversations but lacks physical versatility and decision-making independence. 2. Challenges to Autonomy AI Complexity: Generalized intelligence capable of independent reasoning and decision-making remains a significant challenge. Current AI excels in narrow tasks but struggles with adaptability and creativity. Robust Sensing and Perception: While robots can use sensors like cameras and LiDAR, understanding dynamic, cluttered environments with human-level precision is difficult. Energy Efficiency: Robots need better battery technology to function independently for extended periods. Social and Ethical Barriers: Society must address ethical concerns, liability, and regulations for fully autonomous robots in public or professional spaces. 3. Predictions for the Future Experts estimate different timelines depending on the level of autonomy: 2025–2035: Humanoids might perform repetitive or structured tasks (e.g., manufacturing, logistics) with limited supervision. 2040–2050: Robots may handle unstructured, complex environments like caregiving, construction, or public service without significant human intervention. Beyond 2050: Full autonomy across diverse tasks and environments could be possible, potentially rivaling or exceeding human abilities. Current Research and Developments Google DeepMind and OpenAI are advancing general AI capabilities. Companies like Boston Dynamics are improving robot agility and adaptability. Researchers focus on integrating AI with physical robots for real-world applications, such as robotic exoskeletons and disaster recovery #Ai #Innovation #Technology

A few years ago, I learned the hard way that jumping straight into hardware, sensors, motors, and wiring can lead to costly mistakes and late-night headaches. That’s when I discovered the true importance of #simulation in robotics and engineering. During the early phase of my final-year thesis, I spent weeks recreating our school cafeteria with Iman Tokosi in Blender, exporting it as an SDF model and loading it into Gazebo using #ROS2. Suddenly, I could drive a virtual robot through aisles and around tables without the fear of damaging anything real. It was challenging and eye-opening, and it saved me countless hours and resources. Then came the moment that changed everything: integrating #SLAM so the robot could build its own map while moving, and setting up #Nav2 to let it plan and follow paths autonomously. Watching it navigate the environment with precision and independence was a powerful confirmation that the system worked. Now, imagine a world where every structure, product, and system is simulated down to the smallest detail. The result? Reduced costs, faster development, increased reliability, enhanced safety, and stronger adherence to standards. Some may still view simulation as “just for show,” but I’ve experienced firsthand that it’s the foundation of true innovation. Are you leveraging simulation in your next robotics or engineering project? Let’s connect and exchange ideas!

VLA models are systems that combine three capabilities into one framework: seeing the world through cameras, understanding natural language instructions like "pick up the red apple," and generating the actual motor commands to make a robot do it. Before these unified models existed, robots had separate modules for vision, language, and movement that were stitched together with manual engineering, which made them brittle and unable to handle new situations. This review paper covers over 80 VLA models published in the past three years, organizing them into a taxonomy based on their architectures—some use a single end-to-end network, others separate high-level planning from low-level control, some use diffusion models for smoother action sequences. The paper walks through how these models are trained using both internet data and robot demonstration datasets, then maps out where they're being applied. The later sections lay out the concrete technical problems that remain unsolved. Read online with an AI tutor: https://lnkd.in/eZdzYfdu PDF: https://lnkd.in/ezzncewE

A neural network cannot execute a complex task if the end-effector lacks the mechanical capability to express it. Paris-based GenesisAI just highlighted exactly what it takes to close this hardware-software gap. In their latest rollout, they showcased a dual-arm manipulator running a single foundation model—GENE-26.5—to autonomously execute complex, multi-step physical tasks. To achieve this, they bypassed traditional parallel grippers and integrated a tailored set of 22-DoF Wuji Hands. The engineering architecture relies on a unified system rather than fragmented, task-specific code. The AI is trained on a massive triad of multimodal data: - First-person human demonstration videos. - High-fidelity telemetry from motion, force, and tactile feedback gloves. - Large-scale, closed-loop simulation. By standardizing around highly dexterous, human-scale hands, Genesis is ensuring their foundation models can physically execute the workflows, tool operations, and tactile nuances originally designed for people. #PhysicalAI #Robotics #GenesisAI #WujiTech #Engineering #Automation #FoundationModels

A humanoid robot costs $90K to break once. AI lets you break thousands... and learn from every fall. My background is mechanical engineering, robotics, and integration & test. But this field is moving so fast with AI that reading articles wasn't cutting it anymore. I felt out of the loop, so... I recently upgraded my personal setup to support AI training workloads and ran my first experiment: Teaching a bipedal (two-legged) humanoid robot to navigate a custom parkour course using reinforcement learning in NVIDIA Isaac Lab 5.1. But before I share what I learned, let me explain what's actually happening under the hood. A GPU-accelerated AI agent runs thousands of virtual robots in parallel. Each one learns from its own falls and successes simultaneously. The AI develops a "control policy," which is the brain that tells a robot how to move through the physical world. Why does this matter? Because what once required million-dollar labs and months of physical testing can now run on a single AI-capable GPU in hours. Robotics R&D is becoming software-first. Here's what that looked like for this experiment: 76 minutes of CUDA-accelerated training time. 393 million training steps. 4,096 robots learning in parallel on my RTX 5080. So what did I learn so far? Three things stood out to me: 》The setup before you can hit "Run" is a challenge. It took me seven hours to troubleshoot versioning, packages, and dependencies before I could run anything. I forced myself to do it manually because I wanted to understand what's under the hood. YouTube tutorials hit their limit quickly, but thankfully the NVIDIA developer forums saved me. 》The cost case is undeniable. A Unitree H1 costs around $90K. I *virtually* crashed thousands of them. My damage bill? $0. Simulation lets you fail-forward at scale. This gets you to a solid starting point for physical testing, but... 》The Sim-to-Real gap is real. This policy works well in simulation, but I couldn't get a feel for stress points, sensor behavior, or true stability. Failure is not predictable and happens at the edges. The next step would be to transfer this policy to a physical robot, gather real-world data, and continuously aligning the simulation to close that gap. The key thing here is: Testing real hardware is expensive. Simulation in software is cheap. How can you leverage both, intelligently? The benefit isn't limited to cost savings. This workflow also compresses developmental cycles and allows you to field systems faster. Do you think virtual simulation is a game-changer that is here to stay, or a fad? How would you build confidence in a robotic control policy that is trained in a virtual world? #robotics #ai #nvidia #omniverse #isaaclab ~~~~~~~~ Citations: NVIDIA IsaacLab -> https://lnkd.in/ekVMDnDc RSL-RL -> https://lnkd.in/eJye3XTW Unitree H1-> unitree.com/h1/ Note: this is an educational personal project. Opinions are my own, no affiliation or endorsement.

Watch this B-1B Lancer touchdown closely, as the wheels hit hard, the airframe flexes and oscillates and the rudder reacts immediately (this is not pilot input). The first lateral bending elastic mode is excited by the landing loads, and the #FlightControlSystem senses it and responds. For a brief moment, structure, aerodynamics, sensors, and actuators are tightly coupled in a very visible example of #AeroServoElastic coupling. The B-1 was one of the first aircraft to deliberately address elastic dynamics with #ActiveControl, incorporating the #ILAF concept (Instantaneous Location of Acceleration and Force). By colocating accelerometers and control forces (small canards), the system actively alleviated longitudinal elastic modes, improving ride quality and reducing structural loads. It was an early recognition that #StructuralDynamics were not a side effect to be ignored, but a behavior to be managed. One way to manage aeroservoelastic coupling is to restraint. Classical #NotchFilters are designed to remove control sensitivity around specific modal frequencies so the control laws do not chase structural vibration measured by the IMUs. In many cases, the safest response is for the #FlightControlLaws to step aside, preserving handling qualities while preventing energy from being fed back into the structure. But modern #FlightControlSystems can go further than filtering! Aircraft like the A380 actively command surfaces to damp flexible modes, treating #FlexibleModes as states to be controlled rather than avoided. At the cutting edge, #SpatialFiltering techniques, as pioneered on the B-2, distinguish rigid body motion from elastic deformation by shape, not just frequency. 📹 This video is a reminder that airplanes are living, flexible machines, and the most mature control laws are those that know when to listen, when to stay quiet, and when to actively alleviate the structural loads and oscillations! 💡

Build your first robot in simulation! 👾 📌 If you’re self-learning robotics, this is genuinely one of the better repos to save for later. NVIDIA Robotics released a "Getting Started with Isaac Sim" tutorial series covering everything from building your first robot to hardware-in-the-loop deployment. What's inside? → Building Your First Robot Explore the Isaac Sim interface, construct a simple robot model (chassis, wheels, joints), configure physics properties, implement control mechanisms using OmniGraph and ROS 2, integrate sensors (RGB cameras, 2D lidar), and stream sensor data to ROS 2 for real-time visualization in RViz. → Ingesting Robot Assets Import URDF files, prepare simulation environments, add sensors to existing robot models, and access pre-built robots to accelerate development. → Synthetic Data Generation Learn perception models for dynamic robotic tasks, understand synthetic data generation, apply domain randomization with Replicator, generate synthetic datasets, and fine-tune AI perception models with validation. → Software-in-the-Loop (SIL) Build intelligent robots, implement SIL workflows, use OmniGraph for robot control, master Isaac Sim Python scripting, deploy image segmentation with ROS 2 and Isaac ROS, and test with and without simulation. → Hardware-in-the-Loop (HIL) Understand HIL fundamentals, learn NVIDIA Jetson platform, set up the Jetson environment, and deploy Isaac ROS on Jetson hardware. The progression makes sense: start with basics (build a robot), add perception (sensors and data), generate training data (synthetic generation), develop software (SIL), then deploy to hardware (HIL). Each module builds on the previous one. For robotics teams, this is the path to faster iteration. Simulate first, validate in software-in-the-loop, generate synthetic training data at scale, then deploy to hardware with confidence. 🎓 If this helps at least one engineer to become more fluent in the world of robotics, means a lot to me! 🫶🏼 Here's the course (it's free): https://lnkd.in/dRYdkmdi ~~ ♻️ Join the weekly robotics newsletter, and never miss any news → ziegler.substack.com

🚀𝐖𝐞 𝐚𝐫𝐞 𝐢𝐧𝐭𝐫𝐨𝐝𝐮𝐜𝐢𝐧𝐠 𝐒𝐦𝐨𝐥𝐕𝐋𝐀-𝟒𝟓𝟎𝐌, 𝐚𝐧 𝐨𝐩𝐞𝐧-𝐬𝐨𝐮𝐫𝐜𝐞 𝐕𝐢𝐬𝐢𝐨𝐧-𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞-𝐀𝐜𝐭𝐢𝐨𝐧 𝐦𝐨𝐝𝐞𝐥 𝐟𝐨𝐫 𝐫𝐨𝐛𝐨𝐭𝐢𝐜𝐬! SmolVLA achieves best-in-class performance and inference speed, and the best part? It’s trained entirely on open-source datasets from the 🤖 LeRobot project hosted on the Hugging Face Hub. 🔍 Why is SmolVLA so good? Turns out that pretraining on a large, diverse and noisy collection of real-world community robotics data leads to better generalization and control. We saw a 26% boost in task success rate simply from adding community dataset pretraining! ⚡ Why is SmolVLA so fast? 1. We halved the size of SmolVLM and extract intermediate representations 2. Introduced interleaved cross- and self-attention layers in the action expert 3. Enabled asynchronous inference so the robot acts and reacts simultaneously 💡 Unlike most academic datasets, these community-contributed datasets are naturally diverse: ✅ Multiple robots, camera angles, and manipulation tasks ✅ Real-world messiness and complexity ✅ Crowd-sourced and community-cleaned using Qwen2.5-VL for high-quality task descriptions 🌍 SmolVLA is a step toward making robotics research more affordable, reproducible, and collaborative. 📖 Want to dive deeper? Check out our blog post & start using it today: https://lnkd.in/e3Gmy8gT Huge thanks to the team who made this possible: @Mustafa Shukor Francesco Capuano Remi Cadene, and the entire Lerobot team, amazing HF team Andrés Marafioti Merve Noyan Aritra Roy Gosthipaty Pedro Cuenca Loubna Ben Allal, Thomas Wolf  and to the amazing contributors to the LeRobot community: Ville Kuosmanen, Alexandre Chapin, Marina Barannikov, and more!