Ulysses Sequence Parallelism: Training with Million-Token Contexts
Ulysses Sequence Parallelism, a technique for training large language models on long sequences by distributing attention computation across multiple GPUs via attention head parallelism. It covers the methodology, communication complexity, and integration with Hugging Face's Accelerate library.
