Natural Language Autonomous Drone Control

Project Overview

Developed as a university research project, the system implements a multi-agent pipeline — natural language → intent agent → planner agent → code generation → drone execution — that turns a written mission into autonomous flight inside a ROS 2 / Gazebo simulation (sjtu_drone). An intent agent parses commands into structured JSON, a planner agent backed by LLMs (Gemini, with automatic key rotation and fallback providers) generates executable trajectory code, and a YOLOv8 + OpenCV HSV perception layer detects objects and projects their 3D position to steer search strategies such as lawnmower sweeps and orbital inspection. Real-time collision and stuck detection trigger autonomous LLM-based re-planning, and a Gradio-based web cockpit exposes live telemetry, dual camera feeds, agent logs, and emergency controls. The system was benchmarked across multiple LLM providers for task success rate.

Scope & Capabilities

1. 01

   Natural language mission input

   chat-based task specification parsed into structured intent

2. 02

   LLM-based trajectory planning

   Gemini-backed planner agent generates executable flight code from structured commands

3. 03

   Computer vision perception

   YOLOv8 object detection and HSV color segmentation with 3D world-position projection

4. 04

   Autonomous search strategies

   yaw-based scanning, lawnmower sweeps, and orbital inspection patterns

5. 05

   Collision & fault recovery

   real-time collision/stuck detection triggers autonomous LLM re-planning

6. 06

   ROS 2 / Gazebo simulation

   full flight simulation environment for safe testing before real deployment

7. 07

   Web cockpit dashboard

   Gradio interface with live telemetry, dual camera feeds, agent logs, and emergency abort controls

Technology Stack