Skip to content

SCOUT++ Toolkit

2025 · Lead researcher · Active

Toolkit for multimodal HRI experiments and dataset benchmarking of instruction grounding.

PythonPyTorchVision-LanguageEvaluation
Source code ↗Related paper (PDF)
SCOUT++ example task: a commander instruction "move forward and send an image of the hallway on the left", the robot's navigator-view photo of a corridor, and the model's interpreted command alongside the reference

Highlights

  • Reconstructed and extended the SCOUT corpus into a standardized set of 11,000+ timestamp-aligned images and 2,474 dialogue pairs.
  • Benchmarks three instruction-classification approaches: a neural baseline, GPT-4 text-only, and GPT-4 with synchronized vision.
  • Found that the multimodal variant *underperformed* the text-only one, producing vague or hallucinated outputs — a result worth reporting precisely because it cuts against expectations.
  • Reproducible pipeline for preprocessing, feature engineering, and experiment orchestration.

Background

SCOUT++ supports research into how well vision-language models actually ground natural-language instructions in what a robot can see.

The headline finding is a negative one. Adding synchronized visual input to GPT-4 *degraded* instruction-classification accuracy relative to text alone, with the multimodal variant more prone to vague or hallucinated outputs. For safety-critical settings like urban disaster response, that is exactly the kind of result that needs publishing rather than tuning away.

Stack

  • Python
  • PyTorch
  • TensorFlow
  • OpenCV
  • ROS
  • GPT-4

Gallery

  • SCOUT++ public release overview: a Language/Vision/Action Venn diagram meeting at multimodal instruction data, with counts of 12,000 model input rows, 12,000 multimodal output rows, 11,980 text-only GPT outputs, and 12,003 aligned navigator-view images

    Public release overview — an unofficial derivative of the ARL SCOUT corpus, for research use.

  • SCOUT++ processing pipeline from SCOUT-derived inputs through cleaning and normalization to text-only and multimodal GPT outputs, ending in checksum validation

    From raw corpus rows to model-ready multimodal artifacts, with checksums for reproducibility.

  • The four core SCOUT++ release artifacts: gpt_input.csv, output_results_full.csv, gpt_text_output_results.csv, and the aligned navigator-view images, plus sample data, schema, checksums, and a validation script

    What ships in the public release.