From the source
Databricks contributed a new 'from_spark' function to Hugging Face's Datasets library, allowing users to directly convert Spark dataframes into Hugging Face datasets, reducing processing time by up to 40%.
Databricks ❤️ Hugging Face: up to 40% faster training and tuning of Large Language Models
Developer tool
From the source
our first of many official commits to the Hugging Face codebase that allows users to easily create a Hugging Face Dataset from an Apache Spark™ dataframe.
huggingface.co
