Accelerate GPU training on Unity Catalog tables with UCDataset
Databricks released UCDataset in Beta on AI Runtime, a native PyTorch data loader that streams Unity Catalog tables directly to every GPU in a training job without exports, copies, or custom loader code. On a simulated production workload streaming a 300 GB feature table across 8 NVIDIA H100 GPUs to train a recommendation model, UCDataset cut training time by 10x on average compared with loading data directly from Parquet. The Beta reads Delta tables in Unity Catalog, applies filters and deletion vectors as data is read, and enforces Unity Catalog permissions, though it cannot read tables with row filters or column masks.