Dev Tools · 2h ago
DuckDB Performance: file_row_number Beats OFFSET for Parquet Paging
A developer benchmarks two methods for paging through Parquet files in DuckDB: using file_row_number vs. OFFSET. The file_row_number approach is significantly faster, especially on large datasets, as it avoids scanning skipped rows. This optimization matters for data analysts and engineers building efficient data pipelines.
Meridian48 take
While OFFSET is simpler, the performance gap highlights how understanding DuckDB's internal row numbering can yield major speed gains in production queries.
Read the full reporting
Paging Through a Parquet File in DuckDB: File_row_number or Offset? →
Hacker News
duckdbparquet