File Format Renaissance: Parquet, Lance, Vortex, Nimble, and the New Physics of Columnar Storage

A wave of new file formats like Lance, Vortex, and Nimble is challenging Apache Parquet's decade-long dominance in analytical storage. These formats are designed for AI workloads, leveraging modern hardware such as NVMe and GPUs. The article breaks down the architectural trade-offs and provides guidance on which format to use for specific use cases.