TensorStream

Structuring Multimodal Data for Causal Transformers September 23, 2025 TensorStream Structuring Multimodal Data for Causal Transformers September 23, 2025 TensorStream Structuring Multimodal Data for Causal Transformers Problem Generative causal transformers are 1D computers.

All instructions need to be serialized into a 1D token stream.

For text-only data it's simple because humans process it the same way.

All text-only data is stored/processed as 1D token streams.

The problem gets challenging with multistream/multimodal data e.g. videos.

Not only do videos have separate time-aligned audio and vision streams but also can have multiple metadata streams like transcriptions, narration, or even actions for robotics/video game data.

This presents a challenge - we need some protocol for forming this token stream.

The problem of preparing this data is 2-stage.

The first is about organizing the raw data into a causal chain.

The second is about storing that data in a way it can flexibly be used and ultimately converted into an input to a transformer model.

Design We solve this problem with a data structure called TensorStream and a library built around it.

…