From the source
From the source
Hugging Face introduces Storage Buckets, a mutable, S3-like object storage on the Hub for ML intermediate files, backed by Xet for chunk-based deduplication, with CLI, Python, and JavaScript support, and pre-warming for data locality.
From the source
Storage Buckets are built exactly for this: mutable, S3-like object storage you can browse on the Hub, script from Python, or manage with the hf CLI. And because they are backed by Xet, they are especially efficient for ML artifacts that share content across files.
huggingface.co
