Comparative Analysis of Small and Large Language Models for Natural Language-based Destination Selection on Real-World Geospatial Data

Abstract

Natural language interfaces serve as a revolutionary alternative to traditional waypoint entry systems seen in robotics, especially in scenarios where user accessibility and hands-free, rapid path input are critical. Although there have been numerous recent, substantial advancements in language models, there is little research regarding how these models can be used for navigation in real-life geospatial contexts.

In this work, we present an end-to-end working implementation that converts a voice prompt into input for a language model to derive an intended destination and create a route to accurately navigate a rover. We evaluate two different models — a fine-tuned Large Language Model (LLM), Gpt-4o-mini-2024-07-18, and a Small Language Model (SLM), all-MiniLM-L6-v2. While the models each produce their own perceived destination, the route-finding backend is unified under the use of Dijkstra’s algorithm to output a latitude-longitude shortest path from a specified start point to the determined endpoint. We use a real-world OpenStreetMap (OSM) data snapshot of the George Mason University (GMU) Fairfax campus for the scope of our experimentation.

Our evaluation methodology used 50 curated test prompts to establish a baseline for comparison. We surveyed the GMU community with these same prompts, as their response would be representative of the highest likelihood of an intended destination on campus. We used efficiency (execution time) and accuracy (intended destination compared to the baseline) as metrics for evaluating the two models. Our studies showed that the LLM was 4% more accurate and 1.55 seconds slower per prompt on average compared to the SLM.