Apache Iceberg V4 proposes a complete structural redesign of table metadata to improve scalability and support subsecond writes.
Daniel Weeks, co-creator of Apache Iceberg and member of the technical staff at Databricks, presented the fourth version of the open table format specification at QCon San Francisco, according to InfoQ.
Originally developed at Netflix in 2017, the open table format became an Apache top-level project in 2020. Iceberg V4 addresses limitations that emerge when managing very large analytical datasets across different data processing engines, InfoQ reported.
“V4 is a major reimagining of a table format,” Weeks said. “We didn’t get everything right in V3 and prior, so now we’re taking the opportunity to fix some of the biggest things that are challenges as we scale up to even larger data sets and we bring on new use cases.”
V4 Cuts Metadata Writes to a Single Operation
Iceberg V3 required at least four I/O operations to add a data file, including manifest creation and rewriting. By letting the root of the metadata tree link straight to data files, V4 streamlines this procedure and cuts the work down to just one metadata write.

“This gets us into use cases where we can stream data much faster,” Weeks said. “We can do commits at the 2nd level or subsecond level, where right now we largely recommend that you don’t do commits under 2 minutes.”
According to InfoQ, the V4 proposal covers three main areas: metadata and scalability, new table features, and new data types. The metadata improvements feature the Adaptive Metadata Tree, single-file commits, improved column statistics, relative paths, and column-level updates.
New Features and Workloads
Iceberg V3 introduced deletion vectors, row lineage, new data types, encryption, and default values. V4 replaces binary column statistics with structured, typed statistics to improve projection and schema evolution.
The new version also introduces expression-based default values, check, unique, and primary key constraints. It adds statistics-only fields that support partitioning, sorting, and filtering without storing additional data, while tracking constraint validation status for engines.
First-class file and vector types open the open table format to workloads such as artificial intelligence and search. On the developer mailing list, Steven Wu proposed the inclusion of monotonic snapshot and row-level update timestamps, whereas Manu Zhang conversed about eliminating equality deletes and streamlining column updates.
Weeks co-created Iceberg during its original development at Netflix in 2017 and later joined Tabular, a company founded to commercialize the project. He has been a central figure in the format’s evolution from its inception through its graduation to an Apache top-level project in 2020 and the recent general availability of V3 on Databricks.
Backward Compatibility Questions
During the Q&A session in San Francisco, Weeks confirmed that V4 metadata will not be backward compatible with older readers. Upgrades allow current V3 metadata to stay put, though legacy readers fail to process the updated format.