12Ac5 Y8xeVP185JApCkkl RQ
In the modern battlefield, GPS cannot be relied upon. This was starkly demonstrated when, as reported by The Wall Street Journal, “Ukrainian officials have found U.S.-made drones fragile and unable to overcome Russian jamming and GPS blackout technology.” GPS signals are susceptible to jamming, spoofing, or complete denial in certain combat scenarios, making them unreliable. Adversaries with advanced electronic warfare capabilities can easily disrupt GPS signals, causing drones to lose their way or even fall into enemy hands. The use of radio control in these situations is equally hazardous, as it is highly detectable and can inadvertently reveal the operator’s position, exposing them to potential attacks. Additionally, long-distance operations in hostile environments often lead to weak or broken signals, limiting the operational range and effectiveness of the drone. This all to say, the need to remove an unmanned system’s dependencies on properly functioning radio frequency is critical in modern warfare.
To combat this challenge, Palantir created Visual Navigation (VNav), a new solution that brings Palantir intelligence and software onboard to enable autonomous drone missions in GPS-compromised areas, providing accurate navigation while operating entirely independent of GPS or radio control signals. Through the use of a simple camera and onboard compute — something many deployed drones now come with by default — it compares the drone’s position against onboard satellite imagery, allowing it to navigate without long-range drift. In essence, VNav does the same thing humans used to do before ubiquitous GPS usage: it lets the drone navigate by reading a map.
Palantir has partnered with Flyby Robotics to help demonstrate the power of VNav. Flyby Robotics is an American UAS developer building modular, ML-enabled drones for industry and defense. Their F-11 drone model integrates seamlessly with VNav. An added advantage is its NDAA-compliance, with a resilient supply chain sourcing exclusively from U.S. or U.S.-allied component manufacturers.
Palantir has spent several years developing products at the intersection of computer vision and geospatial algorithms, especially in the areas of image and video georegistration (accurately fixing the real-world location of visual data). Palantir’s VNav is the evolution of this experience. Its navigational algorithms combine three distinct data sources to create a comprehensive picture of the drone’s position and motion in relation to its mission and seamlessly relay that picture back to the drone to help maintain an accurate flight path. It is built from the ground up for the compute and sensors available on small drones.
Modern drones have a wide selection of onboard sensors, including accelerometers, gyroscopes, magnetometers (often referred to collectively as an “inertial measurement unit” or IMU), barometers, and more. Together these sensors provide information about what direction the drone is facing, how it’s accelerating, and the ways it’s turning. With sufficiently accurate IMU sensors, you can navigate long distances with reasonable accuracy on IMU data alone by “dead reckoning.” Such accurate sensors, however, are both large and costly (i.e., hundreds of thousands of dollars on sensors alone). On a smaller, more affordable device, the IMUs have low accuracy and can typically only achieve a few seconds of dead reckoning navigation before it becomes completely unreliable.
VNav is able to combine this sensor data, even from inexpensive sensors, with computer vision techniques to create a comprehensive solution for autonomous navigation. It uses the sensor data to improve stability and responsiveness while relying on computer vision to solve the problems of long-range navigation.
In addition to inertial sensors, VNav also leverages a technique known as optical flow. Optical flow consists of tracking the motion of pixels from frame to frame in a video feed. Using the motion of the pixels and what we know from the inertial sensors, it is possible to compute an estimate of the drone’s velocity. This velocity computation, however, requires an accurate value for the distance between the drone and the ground. Imagine a distant surface moving rapidly and a nearby surface moving slowly — they will exhibit the same apparent motion on camera and cause dramatic miscalculations of distance. Even when using a barometer to get an estimate of altitude, the calculations quickly become unreliable, making accurate long-distance navigation almost impossible.
The common solution to make optical flow (and its fancier cousin Simultaneous Localization and Mapping, or SLAM) more reliable is to add a depth sensor such as a lidar rangefinder. Unfortunately, these sensors add cost and weight to the platform and often have a limited effective range (at least for sUAS-grade sensors), restricting the maximum altitude at which the drone can fly. VNav does not require any such sensors, using its other sources to accurately calculate altitude.
The integration of inertial measurements and optical flow data facilitates an advanced form of dead reckoning suitable for manual control or short-distance navigation. This technique, often referred to as Visual Inertial Odometry, is a common feature of other “Visual Navigation” solutions on the market and can provide reasonable navigation over several tens to hundreds of meters. This approach also has unique flaws. Because it fundamentally tracks “relative” motion, it creates inevitable, uncorrectable errors and “drift” over time. This is akin to traditional wilderness navigation using a compass and counting paces: estimations of distance and direction can lead to gradual inaccuracies or “drift” in your position, and before you know it, you’re lost in the woods.
The final and most unique data source used by VNav is what we call “Reference Matching.” With computer vision, we can automatically compare the drone’s camera feed with satellite data pre-loaded in the onboard compute and find corresponding points between images. Based on those points, VNav can mathematically and continuously determine the true position of the drone, automatically correcting any accumulated drift that may have occurred.
If this technique is so powerful, why is it not in broader use? While it essentially is a type of classical image matching, it confronts most of the classic challenges in computer vision matching, including natural imagery lacking crisp man-made features, blurry reference photos, huge seasonal changes, terrain destruction, varied lighting, visual versus infrared differences, and many more. Even the heaviest deep learning-based image matching techniques often fail here, not to mention they’re typically designed to use compute that’s more than 1000% larger than what is available on a small drone.
To mitigate these issues, VNav uses Palantir’s new proprietary image matching kernel which has been developed across a wide range of terrains, drone sensors, and operational profiles. This is the last piece of the puzzle in enabling VNav to successfully read its map, correct any drifting navigation errors accumulated by other technologies, and remain on track, keeping the drone locked on its target. Our matching kernel has been tested reliably over urban areas and natural domains, using visual, infrared, and even multispectral imagery.
VNav’s revolutionary approach to drone navigation is a product of its unique data fusion from three distinct sources, all processed through a mathematical construct known as a Kalman Filter. This statistical framework is designed to accurately interpret a system based on incomplete or error-prone measurements. Combined with Palantir’s deep defense expertise and proprietary enhancements, VNav adeptly mitigates limitations inherent in each data source.
Short-term errors in optical flow or reference matching are counterbalanced by reliable inertial telemetry. Drift in optical flow is rectified through reference matching, while any gaps in reference matching data are effectively filled by reckoning via optical flow.
This strategic fusion results in a precise, low-latency model of the drone’s state. VNav leverages this model to provide position and velocity measurements to the drone’s flight controller via standard communication protocols. This not only empowers the drone to navigate effectively but also ensures successful mission execution.
By overcoming traditional limitations and harnessing the power of multiple data sources, VNav offers a robust, reliable solution that redefines the future of drone navigation. Its unique approach and innovative application of technology underscore Palantir’s commitment to pushing boundaries and driving progress in the field of autonomous navigation. With the ever-growing concern around GPS interference on the battlefield, this solution helps drive safer and more effective mission outcomes.
Interested in learning more? Contact us to continue the conversation: visual-nav@palantir.com.
The Future of Drone Navigation was originally published in Palantir Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
Are you looking for Writesonic alternatives? Here are five of the best tools that can…
While server-side Large Language Models (LLMs) demonstrate proficiency in tool integration and complex reasoning, deploying…
Conducting assessments on application portfolios that need to be migrated to the cloud can be…
We recently hosted our annual Google Cloud Startup Summit, and we were thrilled to showcase…
Katanemo's new Arch-Function LLMs promise 12x faster function-calling capabilities, empowering enterprises to build ultra-fast, cost-effective…
The new frontier in large language models is the ability to “reason” their way through…