From SCOUT to SCOUT++: Making Data You Can Actually Train On

One thing I’ve learned working with real-world multimodal datasets is that “having data” is very different from having data you can actually train models on.

The original SCOUT dataset contains rich human–robot interaction data, but like many research datasets, much of that richness lives in implicit structure. Useful for analysis, but difficult to use directly for modern machine learning without heavy feature engineering.

My work on SCOUT++ focused on enhancing that foundation.
Specifically, I led the transformation of SCOUT into SCOUT++ by:

• Improving structural consistency across modalities
• Aligning language, perception, and action signals
• Making the data usable for large-scale multimodal learning without extensive manual feature engineering

This kind of dataset work is rarely glamorous, but it’s essential. Model architectures change quickly, well-structured data on the other hand, compounds.
SCOUT++ enabled us to study grounded instruction understanding at scale and ultimately motivated a broader question I’m now working on:

How should intelligent systems decide when to act versus when to ask for clarification under uncertainty?

That question shows up everywhere. In robotics, autonomous agents, copilots and it starts with data that actually reflects how humans and systems interact in the real world.

More to come as this line of work evolves.

---

Originally posted on LinkedIn.