Long-Horizon Planning with Large Language Models for Indoor Assistive Navigation of the Visually Impaired

Anonymous Author
Affiliation Hidden

Abstract

For visually impaired individuals, assistive navigation systems play a crucial role in enabling independent mobility. However, long-horizon planning based on natural language (NL) instructions in complex indoor environments remains a significant challenge. Recent studies show the strong potential of Large Language Models (LLMs) in NL understanding and task-level planning. Yet, the inherent limitations of LLMs in mathematical reasoning and their susceptibility to hallucination hinder their reliability in low-level path planning. In this paper, we introduce an LLM-based indoor assistive navigation system that interprets NL instructions from visually impaired users for autonomous navigation. At its core is a novel planning agent that grounds instructions to the environment's topological map and generates optimal route plans. To avoid hallucination in geometric reasoning, the LLM handles only high-level semantic planning, while precise node-level paths are delegated to a classical graph search algorithm. We further implement a wearable assistive device that provides voice and vibrotactile feedback to deliver hands-free navigation. Offline evaluations and real-world experiments demonstrate that our system can reliably plan grounded routes and enable visually impaired users to autonomously complete long-horizon navigation tasks.

System Overview

System Overview

We propose an LLM-based indoor assistive navigation system for the visually impaired, which consists of two layers: high-level planning and low-level execution. The system centers on a proposed novel LLM-based planning agent that interprets users' NL instructions into semantically grounded long-horizon navigation plans. In the execution layer, a wearable assistive device, along with vision-based localization and obstacle avoidance modules, guides visually impaired users in carrying out the generated plan.

High-level Planning Visualization*

High-level Planning Visualization*

The high-level planning first infers location-level candidate plans that satisfy the user’s intent. It then generates node-level candidates via an external planner. Finally, the optimal navigation plan is selected via evaluation and filtering. Should any anomaly be detected during evaluation, the system returns to the first stage for refinement.

Low-level Execution Visualization

System structure

We implement a wearable assistive device to reliably enable independent navigation based on the generated plan. Indoor localization is achieved via signboard detection and recognition, and direction guidance is provided through clear voice prompts based on the user’s heading. During navigation, image-based obstacle detection is employed, and obstacle avoidance is ensured through vibrotactile feedback from the wristbands to ensure the user’s safety.

Comparison with LLM-based Agents for Planning*

LLM-based Planning Agents Comparison

Schematic diagrams of different LLM-based agents for planning. Our agent integrates the strengths of others. An external planner is employed to complement the LLM's limited mathematical reasoning. Diverse candidate plans are generated first, and the optimal one is then selected by an auxiliary LLM, which also performs feedback-driven refinement when necessary.

Comparison with Assistive Systems for the Visually Impaired

System Form Navigation Indoor
Localization
Long-horizon
Planning
Core
Sensors
Input
Mode
Feedback
Mode
NL
Interaction
Hands
Free
DRAGON Wheeled RGB-D Camera,
LiDAR
Audio Audio,
Kinesthetic
Robotic Guide Dog Quadruped RGB-D Camera,
LiDAR,
Force Sensor
- Kinesthetic
Augmented Cane Smart Cane RGB Camera,
LiDAR,
IMU,
GPS
Button Kinesthetic,
Vibrotactile
Intelligent Aid Wearable IMU,
Infrared Sensor,
GPS
- Cloud Alert,
Vibrotactile
Haptic Cuff Wearable RGB-D,
Camera,
CUFF Force Sensor
- Audio,
Force
Ours Wearable RGB-D Camera,
IMU,
Magneto
-meter
Audio Audio,
Vibrotactile

Our system is the first wearable assistive solution that can plan personalized and optimal long-horizon routes based on NL instructions and enable fully hands-free autonomous navigation for visually impaired users.

Device Specifications

Glasses

Cross-Sectional View and Design Concept of Binocular Camera (Glasses).

Component Parameter Specification
Binocular Camera (Glasses) Dimensions 166mm × 78mm × 66mm
Weight 198g
Camera Module Kangcheng Technology Dual-frame Camera
Minimum Resolution 800 × 600
Frame Rate 30FPS
Field of View 120°
Depth Range 0.5m-25m
Monocular Camera Camera Module USB Camera WX172HD
Minimum Resolution 640 × 480
Frame Rate 30FPS
Field of View 120°
Vibration Wristband Microcontroller Unit LILYGO T-Micro32 plus
Wireless Protocol Bluetooth v4.2 BR/EDR and BLE
Pulse Width 150ms
Pulse Interval 100ms
Electronic Compass Angular Resolution 0.005°/LSB
Magnetic Field Resolution 0.0667mG/LSB
Acceleration Resolution 0.5mg/LSB
Output Frequency 0.2Hz-200Hz

Videos

(All faces and sensitive identifiers in the videos have been blurred.)

Real-World Demo1

User Instruction: "Please take me to the skin doctor after visiting the eye clinic, with a stop at the bathroom on the way, and finally bring me to pay the bill."

Environment: Hospital A (Map A)

Real-World Demo2

User Instruction: "Please take me to the eye-doctor."

Environment: Hospital B (Map B)

Maps

mapA

Map A

{
  "nodes": [
        {
            "id": "n0",
            "name": "ENT&HN Surgery Dept.",
            "type": "location-level",
            "coord": [0.06, 0.22]
        },
        {
            "id": "n1",
            "type": "intersection-level",
            "coord": [0.11, 0.14]
        },
        {
            "id": "n2",
            "type": "intersection-level",
            "coord": [0.11, 0.61]
        },
        {
            "id": "n3",
            "name": "Restroom",
            "type": "location-level",
            "coord": [0.15, 0.10]
        },
        {
            "id": "n4",
            "type": "intersection-level",
            "coord": [0.19, 0.14]
        },
        {
            "id": "n5",
            "type": "intersection-level",
            "coord": [0.19, 0.61]
        },
        {
            "id": "n6",
            "name": "Respiratory Dept.",
            "type": "location-level",
            "coord": [0.19, 0.69]
        },
        {
            "id": "n7",
            "name": "Registration",
            "type": "location-level",
            "coord": [0.26, 0.18]
        },
        {
            "id": "n8",
            "name": "Cashier",
            "type": "location-level",
            "coord": [0.28, 0.33]
        },
        {
            "id": "n9",
            "type": "intersection-level",
            "coord": [0.33, 0.14]
        },
        {
            "id": "n10",
            "type": "intersection-level",
            "coord": [0.33, 0.33]
        },
        {
            "id": "n11",
            "type": "intersection-level",
            "coord": [0.33, 0.61]
        },
        {
            "id": "n12",
            "type": "intersection-level",
            "coord": [0.33, 0.69]
        },
        {
            "id": "n13",
            "name": "Thoracic Surgery Dept.",
            "type": "location-level",
            "coord": [0.33, 0.90]
        },
        {
            "id": "n14",
            "type": "intersection-level",
            "coord": [0.42, 0.69]
        },
        {
            "id": "n15",
            "type": "intersection-level",
            "coord": [0.42, 0.93]
        },
        {
            "id": "n16",
            "name": "TCM Physiotherapy",
            "type": "location-level",
            "coord": [0.46, 0.21]
        },
        {
            "id": "n17",
            "type": "intersection-level",
            "coord": [0.50, 0.33]
        },
        {
            "id": "n18",
            "type": "start",
            "coord": [0.50, 0.69]
        },
        {
            "id": "n19",
            "name": "Restroom",
            "type": "location-level",
            "coord": [0.61, 0.27]
        },
        {
            "id": "n20",
            "type": "intersection-level",
            "coord": [0.67, 0.69]
        },
        {
            "id": "n21",
            "name": "Dermatology Dept.",
            "type": "location-level",
            "coord": [0.67, 0.93]
        },
        {
            "id": "n22",
            "name": "Nephrology Dept.",
            "type": "location-level",
            "coord": [0.76, 0.05]
        },
        {
            "id": "n23",
            "type": "intersection-level",
            "coord": [0.76, 0.13]
        },
        {
            "id": "n24",
            "type": "intersection-level",
            "coord": [0.76, 0.33]
        },
        {
            "id": "n25",
            "type": "intersection-level",
            "coord": [0.76, 0.42]
        },
        {
            "id": "n26",
            "type": "intersection-level",
            "coord": [0.76, 0.69]
        },
        {
            "id": "n27",
            "type": "intersection-level",
            "coord": [0.76, 0.90]
        },
        {
            "id": "n28",
            "name": "Clinical Lab",
            "type": "location-level",
            "coord": [0.81, 0.22]
        },
        {
            "id": "n29",
            "name": "Ophthalmology Dept.",
            "type": "location-level",
            "coord": [0.81, 0.51]
        },
        {
            "id": "n30",
            "type": "intersection-level",
            "coord": [0.91, 0.13]
        },
        {
            "id": "n31",
            "type": "intersection-level",
            "coord": [0.91, 0.42]
        },
        {
            "id": "n32",
            "name": "Ophthalmology Operating Room",
            "type": "location-level",
            "coord": [0.91, 0.61]
        },
        {
            "id": "n33",
            "name": "Treatment Center",
            "type": "location-level",
            "coord": [0.95, 0.33]
        }

    ],
    "edges": [
        ["n0", "n1"],
        ["n1", "n2"],
        ["n1", "n3"],
        ["n1", "n4"],
        ["n2", "n5"],
        ["n3", "n4"],
        ["n4", "n5"],
        ["n4", "n7"],
        ["n4", "n9"],
        ["n5", "n6"],
        ["n5", "n11"],
        ["n7", "n9"],
        ["n8", "n10"],
        ["n9", "n10"],
        ["n10", "n11"],
        ["n10", "n17"],
        ["n11", "n12"],
        ["n12", "n13"],
        ["n12", "n14"],
        ["n13", "n15"],
        ["n14", "n15"],
        ["n14", "n18"],
        ["n16", "n17"],
        ["n17", "n18"],
        ["n17", "n19"],
        ["n17", "n24"],
        ["n18", "n20"],
        ["n19", "n24"],
        ["n20", "n21"],
        ["n20", "n26"],
        ["n21", "n27"],
        ["n22", "n23"],
        ["n23", "n24"],
        ["n23", "n28"],
        ["n23", "n30"],
        ["n24", "n25"],
        ["n25", "n26"],
        ["n25", "n29"],
        ["n25", "n31"],
        ["n26", "n27"],
        ["n30", "n31"],
        ["n30", "n33"],
        ["n31", "n33"],
        ["n31", "n32"]
    ]
}

JSON-format Map A

mapB

Map B

{
    "nodes": [
        {
            "id": "n0",
            "name": "Ultrasonography",
            "type": "location-level",
            "coord": [0.09, 0.54]
        },
        {
            "id": "n1",
            "name": "Endocrinology",
            "type": "location-level",
            "coord": [0.09, 0.89]
        },
        {
            "id": "n2",
            "name": "Ophthalmology",
            "type": "location-level",
            "coord": [0.14, 0.07]
        },
        {
            "id": "n3",
            "name": "Ophthalmic Examination",
            "type": "location-level",
            "coord": [0.14, 0.23]
        },
        {
            "id": "n4",
            "name": "Toilet",
            "type": "location-level",
            "coord": [0.22, 0.42]
        },
        {
            "id": "n5",
            "type": "intersection-level",
            "coord": [0.27, 0.20]
        },
        {
            "id": "n6",
            "type": "intersection-level",
            "coord": [0.27, 0.54]
        },
        {
            "id": "n7",
            "type": "intersection-level",
            "coord": [0.27, 0.89]
        },
        {
            "id": "n8",
            "name": "Registration&Cashier",
            "type": "location-level",
            "coord": [0.47, 0.82]
        },
        {
            "id": "n9",
            "name": "Block Therapy Room",
            "type": "location-level",
            "coord": [0.47, 0.12]
        },
        {
            "id": "n10",
            "name": "VIP/Exclusive Clinic",
            "type": "location-level",
            "coord": [0.52, 0.40]
        },
        {
            "id": "n11",
            "type": "intersection-level",
            "coord": [0.52, 0.54]
        },
        {
            "id": "n12",
            "name": "Pharmacy",
            "type": "location-level",
            "coord": [0.52, 0.67]
        },
        {
            "id": "n13",
            "type": "start",
            "coord": [0.56, 0.94]
        },
        {
            "id": "n14",
            "type": "intersection-level",
            "coord": [0.76, 0.20]
        },
        {
            "id": "n15",
            "type": "intersection-level",
            "coord": [0.76, 0.54]
        },
        {
            "id": "n16",
            "type": "intersection-level",
            "coord": [0.76, 0.89]
        },
        {
            "id": "n17",
            "name": "Toilet",
            "type": "location-level",
            "coord": [0.81, 0.42]
        },
        {
            "id": "n18",
            "name": "Hematology",
            "type": "location-level",
            "coord": [0.85, 0.07]
        },
        {
            "id": "n19",
            "name": "General Surgery",
            "type": "location-level",
            "coord": [0.89, 0.44]
        },
        {
            "id": "n20",
            "name": "Otolaryngology",
            "type": "location-level",
            "coord": [0.90, 0.64]
        },
        {
            "id": "n21",
            "name": "Rheumatology&Immunology",
            "type": "location-level",
            "coord": [0.89, 0.80]
        },

    ],
    "edges": [
        ["n0", "n6"],
        ["n1", "n7"],
        ["n2", "n5"],
        ["n3", "n5"],
        ["n4", "n6"],
        ["n5", "n6"],
        ["n5", "n9"],
        ["n5", "n14"],
        ["n6", "n11"],
        ["n6", "n7"],
        ["n7", "n8"],
        ["n7", "n13"],
        ["n7", "n16"],
        ["n8", "n16"],
        ["n9", "n14"],
        ["n10", "n11"],
        ["n11", "n12"],
        ["n11", "n15"],
        ["n13", "n16"],
        ["n14", "n15"],
        ["n14", "n18"],
        ["n15", "n16"],
        ["n15", "n17"],
        ["n15", "n19"],
        ["n15", "n20"],
        ["n16", "n21"]
    ]
}

JSON-format Map B

LLM Prompts

Prompt to LLM-α

Role:
You are the conversational mediator between the user and an indoor navigation system for the visually impaired, called LLM-alpha.
Your task is to process user natural language requests (Q_NL), along with dialogue history (H) and any optional current plan (P), in order to determine whether the request requires navigation assistance.
If navigation-related, structure the request for LLM-beta; otherwise, reply directly to the user.

Output Format:
You must always output a valid JSON object with the following structure:
    {
        "agent_reply": "string (short, user-friendly, ≤2 sentences)",
        "is_navigation": boolean,
        "forward": {
                       "Q_NL": "string (original or clarified user request)",
                       "confirm_required": boolean,
                       "dialogue_context": "string (brief context summary)"
                   } | null
    }

Rules:
1. If the request is not navigation-related (e.g., greetings, status queries, general questions):
   Set is_navigation: false.
   Set forward: null.
   Reply directly in agent_reply.

2. If the request is navigation-related:
   Set is_navigation: true.
   Fill the forward object with:
        Q_NL: the original or clarified navigation request;
        confirm_required: true if you need clarification (e.g., ambiguous destination), otherwise false;
        dialogue_context: brief summary of relevant context from H or current_plan.

3. If confirm_required: true, phrase agent_reply as a concise yes/no or multiple-choice question.

4. If current_plan exists and the user asks about navigation status, incorporate it into your response.


Few-shot Examples:
Examples 1 (Non-navigation request):
    Q_NL: "Where are we in the navigation right now?"
    H: ["Earlier, the system started a route towards the registration."]
    current_plan: {"current_location": "cashier", "next_location": "registration"}

    Output:
    {
      "agent_reply": "We are currently on the way to the registration and just passed by the cashier.",
      "is_navigation": false,
      "forward": null
    }

Examples 2 (Clear navigation request):
    Q_NL: "Please guide me to the reception desk and then to the nearest restroom."
    H: null

    Output:
    {
      "agent_reply": "Understood. I will plan a route to the reception desk, then continue to the nearest restroom.",
      "is_navigation": true,
      "forward": {
        "Q_NL": "Please guide me to the reception desk and then to the nearest restroom.",
        "confirm_required": false,
        "dialogue_context": "Initial navigation request"
      }
    }

Examples 3 (Ambiguous navigation request):
    Q_NL: "I'm feeling tired, maybe I need to go to a place to rest for a while."
    H: null

    Output:
    {
      "agent_reply": "Do you want me to guide you to the nearest rest area?",
      "is_navigation": true,
      "forward": {
        "Q_NL": "I'm feeling tired, maybe I need to go to a place to rest for a while.",
        "confirm_required": true,
        "dialogue_context": "User expressed need for rest; confirmation needed for navigation"
      }
    }

Prompt to LLM-β

Role:
You are the plan generator for an indoor navigation system for the visually impaired, called LLM-beta.
Your task is to generate all location-level candidate plans, based on a confirmed natural-language navigation request (Q_NL) that has already been verified by LLM-alpha.
You should interpret the user navigation request (Q_NL) and identify all relevant locations. Then, ground these locations to nodes in the JSON-format topological map (G).
Finally, generate a Directed Acyclic Graph (DAG) representing candidate plans, including both strict order and flexible constraints.
If feedback (R) is provided, you must regenerate based on the feedback information.

Input Format:
User natural language navigation request (Q_NL).
JSON-format topological map of the environment (G):
  - `nodes`: a list of nodes, each with fields:
      - `id`: unique identifier
      - `name` (optional): real-world location label
      - `type`: `"location-level"` or `"intersection-level"`
      - `coord`: [x,y] coordinates (float)
  - `edges`: undirected connectivity between node IDs
Optional feedback information for replanning (R) (may be empty):
  - {"what": "...", "how": "..."}, evaluator suggests how to adjust candidates;
  - {"where": "..."}, evaluator specifies a new current start location for replanning.

Output Format:
You must always output a valid JSON object with no extra text outside, for example:
-----------------------------------------------
    {
      "nodes": [
        {"id": "node_id1"},
        {"id": "node_id2"},
        {"id": "node_id3"},
        {"id": "node_id4"},
          ...
      ],
      "edges": [
        ["node_id1", "node_id2"],
        ["node_id2", "node_id3"],
          ...
      ],
      "flexible": ["node_id4"],
          ...
      }
-----------------------------------------------
where:
      - `nodes`: list of all location-level nodes IDs required in these plans, in the form[{"id": "..."}, ...];
      - `edges`: list of directed edges encoding strict precedence constraints, in the form[["idA", "idB"], ...];
      - `flexible`: list of location-level node IDs that are unconstrained and may appear at any position.

Rules:
1. Generate DAG-represented location-level plans based on the user navigation requests (Q_NL) and the topological map (G).
2. Ensure all location nodes exist in the topological map G.
3. Respect explicit order constraints in user requests (e.g., "after", "before", "then", etc.).
4. The location named "start" by default serves as the starting node. And it must be the first node unless R is provided.
5. Each plan should be represented as a subgraph of the generated directed acyclic graph (DAG).
6. "flexible" nodes cannot be scheduled before "start". Flexible nodes may be inserted at any position between constrained nodes.
7. Use step-by-step reasoning before generating the final output.
8. Regenerate when feedback R is provided.

Few-shot Examples:
Examples 1 (Type I - Single-Goal Instruction):
    Q_NL: "Please take me to get a fundus examination."
    G: [{mapB}]
    R: null

    Reasoning:
        - User requests navigation to "fundus examination";
        - Ground "fundus examination" to node "n3: Ophthalmic Examination" in the topological map (mapB);
        - only single navigation goal.

    Output:
    {
        "nodes": [
            {"id": "n13"},
            {"id": "n3"}
        ],
        "edges": [
            ["n13", "n3"]
        ],
        "flexible": []
    }

Examples 2 (Type II - Multi-Goal Instruction without Order Constraints):
    Q_NL: "Take me to the registration desk, the respiratory department, the eye clinic, and the Chinese therapy section. Please take me by the quickest route."
    G: [{mapA}]
    R: null

    Reasoning:
        - User requests navigation to "registration desk", "respiratory department", "eye clinic", and "Chinese therapy section".
        - Ground "registration desk", "respiratory department", "eye clinic", and "Chinese therapy section" to node "n7: Registration", "n6: Respiratory Dept.", "n29: Ophthalmology Dept.", and "n16: TCM Physiotherapy" in the topological map (mapA), respectively;
        - No order is specified; all nodes are flexible.

    Output:
    {
        "nodes": [
            {"id": "n18"},
            {"id": "n7"},
            {"id": "n6"},
            {"id": "n29"},
            {"id": "n16"}
        ],
        "edges": [],
        "flexible": ["n7", "n6", "n29", "n16"]
    }

Examples 3 (Type III - Multi-Goal Instruction with Fully or Partially Ordered Goals):
    Q_NL: "Please guide me to get chest surgery consultation first, then later do some physical therapy, and also stop by the eye doctor on the route."
    G: [{mapA}]
    R: null

    Reasoning:
        - User requests navigation to "chest surgery consultation", "physical therapy", and "eye doctor";
        - Ground "chest surgery consultation", "physical therapy", and "eye doctor" to node "n13: Thoracic Surgery Dept.", "n33: Treatment Center", and "n29: Ophthalmology Dept." in the topological map (mapA), respectively;
        - first "n13: Thoracic Surgery Dept.", then "n33: Treatment Center"; "n29: Ophthalmology Dept." is flexible.

    Output:
    {
        "nodes": [
            {"id": "n18"},
            {"id": "n13"},
            {"id": "n33"},
            {"id": "n29"}
        ],
        "edges": [
            ["n18", "n13"],
            ["n13", "n33"]
        ],
        "flexible": ["n29"]
    }

Examples 4 (Replanning Scenario 1):
    Q_NL: "Take me for a chest surgery consultation, then for physical therapy, and stop by the eye doctor on the route."
    G: [{mapA}]
    R: [{"what": "the plan incorrectly arrives at the 'n33: Treatment Center' first", "how": "'n13: Thoracic Surgery Dept.' should be the first"}]

    Reasoning:
        - User requests navigation to "chest surgery consultation", "physical therapy", and "eye doctor";
        - Ground "chest surgery consultation", "physical therapy", and "eye doctor" to node "n13: Thoracic Surgery Dept.", "n33: Treatment Center", and "n29: Ophthalmology Dept." in the topological map (mapA), respectively;
        - first "n33: Treatment Center", then "n13: Thoracic Surgery Dept."; "n29: Ophthalmology Dept." is flexible;
        - Regenerate based on the rule that "'n13: Thoracic Surgery Dept.' should be the first".

    Output:
    {
        "nodes": [
            {"id": "n18"},
            {"id": "n13"},
            {"id": "n33"},
            {"id": "n29"}
        ],
        "edges": [
            ["n18", "n13"],
            ["n13", "n33"]
        ],
        "flexible": ["n29"]
    }

Examples 5 (Replanning Scenario 2):
    Q_NL: "Please guide me to get chest surgery consultation first, then later do some physical therapy, and also stop by the eye doctor on the route."
    G: [{mapA}]
    R: [{"where": "Treatment Center", "Clinical Lab"}]

    Reasoning:
        - User requests navigation to "chest surgery consultation", "physical therapy", and "eye doctor";
        - Ground "chest surgery consultation", "physical therapy", and "eye doctor" to node "n13: Thoracic Surgery Dept.", "n33: Treatment Center", and "n29: Ophthalmology Dept." in the topological map (mapA), respectively;
        - first "n13: Thoracic Surgery Dept.", then "n33: Treatment Center"; "n29: Ophthalmology Dept." is flexible;
        - Regenerate from "n33: Treatment Center", start with "n28: Clinical Lab"; The flexible "n29: Ophthalmology Dept." has not yet arrived.

    Output:
    {
        "nodes": [
            {"id": "n28"},
            {"id": "n33"},
            {"id": "n29"}
        ],
        "edges": [
            ["n28", "n33"]
        ],
        "flexible": ["n29"]
    }


Prompt to LLM-γ

Role:.
You are the plan evaluator and selector for an indoor navigation system for the visually impaired, called LLM-gamma.
Your task is to evaluate node-level candidate plans (P_nod) and select the optimal one, and you should also provide feedback for regeneration when needed.
Ensure that all node-level candidate plans are accurately grounded in the topological map(G) and evaluate them based on hard constraints and path optimality.
Then, select the plan that best satisfies the intent of the user instruction (Q_NL) and minimizes total path length.
If no suitable plan is found, you should provide a structured feedback to LLM-beta.

Input Format:
Original user natural language instruction (Q_NL).
P_nod: JSON-format node-level candidate plans, for example:
-----------------------------------------------
    {
      "plan1": {
        "nodes": ["n18", "n17", "n19", "n24", "n25", "n29", "n26", "n20", "n21", "n10", "n8"],
        "edges": [
          ["n18", "n17"],
          ["n17", "n19"],
          ["n19", "n24"],
          ["n24", "n25"],
          ["n25", "n29"],
          ["n29", "n26"],
          ["n26", "n20"],
          ["n20", "n21"],
          ["n21", "n20"],
          ["n20", "n18"],
          ["n18", "n17"],
          ["n17", "n10"],
          ["n10", "n8"]
        ]
      },
      "plan2":{...},
               ...
    }
-----------------------------------------------
D: List of total path lengths for each candidate plan (same order as P_nod).
JSON-format topological map of the environment (G):
  - `nodes`: a list of nodes, each with fields:
      - `id`: unique identifier
      - `name` (optional): real-world location label
      - `type`: `"location-level"` or `"intersection-level"`
      - `coord`: [x,y] coordinates (float)
  - `edges`: undirected connectivity between node IDs

Output Format:
You must always output a valid JSON object with no extra text outside, for example:
-----------------------------------------------
{
  "selected_plan_index": 3,
  "replanning_required": false,
  "feedback": null
}
-----------------------------------------------
or
-----------------------------------------------
{
  "selected_plan_index": 3,
  "replanning_required": True,
  "feedback": {
    "what": "description of issue if replanning needed",
    "how": "suggestion for improvement"
  }
}
-----------------------------------------------
where:
  - `selected_plan_index`: index of the selected plan in P_nod (-1 if no plan selected);
  - `replanning_required`: true if feedback is provided to LLM-beta;
  - `feedback`: structured feedback for LLM-beta (only present if replanning_required=true).

Rules:
1. Apply two-stage evaluation to all candidate plans based on the user instruction (Q_NL):
   - Stage 1 (Hard Constraints): Check if each plan satisfies:
     a) All location-level nodes from user instruction appear in correct order;
     b) Every consecutive node pair has a valid edge in topological graph G.
   - Stage 2 (Optimality): Among valid plans, select the one with shortest total path length in the List of total path lengths (D).
2. If no candidate passes Stage 1, generate feedback for LLM-beta with:
   - "what": specific constraint violation description
   - "how": concrete suggestion for correction
3. Use step-by-step reasoning before generating the final output.
4. Feedback should be actionable and specific, guiding LLM-beta to generate better candidates.
5. Prioritize plans that fully satisfy user intent over marginally shorter paths.

Few-shot Examples:
Example 1 (Successful Plan Selection):
    P_nod: [
    ["n18", "n29", "n21", "n8", "n19"],
    ["n18", "n29", "n8", "n21", "n19"],
    ["n18", "n21", "n29", "n8", "n19"]
    ]
    D: [85, 95, 110]
    G: [{mapA}]

    Reasoning:
        - Stage 1: All plans contain required locations in correct order? Yes
        - Stage 1: All consecutive edges valid in G? Yes
        - Stage 2: Plan 0 has shortest path length (85)
        - Select plan 0 as optimal

    Output:
    {
      "selected_plan_index": 0,
      "replanning_required": false,
      "feedback": null
    }

Example 2 (Order Intent Violation):
    P_nod: [
      ["n18", "n29", "n21", "n8", "n19"],
      ["n18", "n8", "n29", "n21", "n19"],
      ["n18", "n21", "n8", "n29", "n19"]
    ]
    D: [80, 75, 90]
    G: [{mapA}]

    Reasoning:
        - User instruction requires based on the user instruction (Q_NL) : n29 before n21
        - Plan 0: n29->n21 True (valid order)
        - Plan 1: n8->n29->n21 False (n8 should not come before n29)
        - Plan 2: n21->n8->n29 False (n21 should not come before n29)
        - Only plan 0 satisfies hard constraints
        - Select plan 0 despite longer path

    Output:
    {
      "selected_plan_index": 0,
      "replanning_required": false,
      "feedback": null
    }

Example 3 (Invalid Edge Detection):
    P_nod: [
      ["n18", "n29", "n21", "n8"],
      ["n18", "n29", "n8", "n21"],
      ["n18", "n21", "n29", "n8"]
    ]
    D: [70, 65, 80]
    G: [{mapA}]

    Reasoning:
        - Plan 0: n18->n29->n21->n8 True (all edges valid)
        - Plan 1: n18->n29->n8 False (edge n29->n8 doesn't exist in G)
        - Plan 2: n18->n21 False (edge n18->n21 doesn't exist in G)
        - Only plan 0 satisfies hard constraints

    Output:
    {
      "selected_plan_index": 0,
      "replanning_required": false,
      "feedback": null
    }

Example 4 (Replanning Required - No Valid Plans):
    P_nod: [
      ["n18", "n8", "n29", "n21"],
      ["n18", "n21", "n8", "n29"],
      ["n18", "n8", "n21", "n29"]
    ]
    D: [60, 70, 65]
    G: [{mapA}]

    Reasoning:
        - User instruction (Q_NL) requires: n29 before n21
        - Plan 0: n8->n29->n21 False (n8 should not be first)
        - Plan 1: n21->n8->n29 False (n21 should not be before n29)
        - Plan 2: n8->n21->n29 False (n21 should not be before n29)
        - No plan satisfies hard constraints
        - Generate feedback for LLM-beta

    Output:
    {
      "selected_plan_index": -1,
      "replanning_required": true,
      "feedback": {
        "what": "All plans violate the required order constraint: n29 should come before n21",
        "how": "'n29' should before 'n21' in the sequence"
      }
    }

Example 5 (Missing Location Node):
    P_nod: [
      ["n18", "n29", "n21"],
      ["n18", "n29", "n8", "n21"],
      ["n18", "n21", "n29"]
    ]
    D: [50, 70, 55]
    G: [{mapA}]

    Reasoning:
        - User instruction (Q_NL) requires: visiting n8, n21, n29
        - Plan 0: Missing n8
        - Plan 1: Contains all locations, but check order later
        - Plan 2: Missing n8

    Plan 1 order: n29->n8->n21 (need to verify if this satisfies user constraints)
    Order is acceptable based on the user instruction (Q_NL), select plan 1

    Output:
    {
      "selected_plan_index": 1,
      "replanning_required": false,
      "feedback": null
    }

Example 6 (Optimality Tie-breaker):
    P_nod: [
      ["n18", "n29", "n21", "n8"],
      ["n18", "n29", "n8", "n21"],
      ["n18", "n21", "n29", "n8"]
    ]
    D: [85, 85, 90]
    G: [{mapA}]

    Reasoning:
        - All plans satisfy hard constraints
        - Plans 0 and 1 have equal shortest path length (85)
        - Select the first occurring optimal plan (plan 0)

    Output:
    {
          "selected_plan_index": 0,
          "replanning_required": false,
          "feedback": null
        }
  

Instruction Dataset (260)