Semantics in Robotics
Traditional SLAM (Simultaneous Localization and Mapping) algorithms are fantastic at building 3D point-cloud maps, but they have zero semantic understanding. A SLAM algorithm treats a brick wall and a fluffy couch as identical geometric obstacles. It doesn't understand the difference between "This is a puddle, avoid it" and "This is a shadow, ignore it."
Integrating VLMs
We integrated a lightweight Vision-Language Model (VLM) into our drone navigation stack. The VLM acts as a high-level semantic planner.
It takes the front-facing camera feed and processes natural language commands. It outputs text-based waypoints (e.g., "Move towards the red car"), which are then translated into continuous control actions (motor thrust vectors) by a lower-level Reinforcement Learning policy.
This hierarchical architecture completely removes the need for prior mapping. The drone can now follow zero-shot instructions like "Fly through the open window and land on the wooden table," navigating purely by semantic visual reasoning.