Dev Tools · 1h ago
Why Two Queries on the Same Iceberg Table Can Return Different Row Counts
Two successful queries on the same Iceberg table returned different row counts: 12,480,032 vs 12,691,404, a difference of 211,372 rows. The discrepancy arises because one query read the table metadata while the other scanned the storage folder directly, which may include obsolete or uncommitted files. Developers must compare inputs—catalog reference vs. folder scan—before debugging query engines.
Meridian48 take
This is a sharp reminder that lakehouse tables are defined by metadata, not directory listings—a subtle but critical distinction for data engineers.
data-engineeringlakehouse