A cloud “table” used to be just a pile of files.
In the Hadoop/Hive era, a table was defined by whatever files happened to sit in a directory. Listing the directory was the query plan. That means no atomic changes: while a job rewrites the table, readers see it half-demolished. Run the same overwrite against both designs and watch the reader on each side.
Hive-style directory = table
Iceberg snapshot = table
The Hive reader counts rows from whatever files exist right now — mid-job, that's garbage. The Iceberg reader planned its query against a snapshot, and the new files only became the table when a single pointer flipped. That pointer is the whole trick, and it's where we dive next.
A table format — an open standard for what counts as a table.
Iceberg is a specification (plus libraries that implement it) for tracking exactly which files make up a table, what schema they follow, and how the table has changed over time. It was created at Netflix in 2017 by Ryan Blue and Dan Weeks and donated to the Apache Software Foundation in 2018. The easiest way to pin it down is to say what it is not:
Something else does the computing. Spark, Trino, Flink, Snowflake, DuckDB — any engine that speaks the spec can read and write the very same table.
Your data stays in Parquet (or ORC / Avro). Iceberg is the layer of metadata about those files — it never invents its own way to store rows.
S3, GCS, ADLS, or HDFS holds the bytes. Iceberg just decides what those bytes mean — which files are in the table, and which are dead weight.
A contract between engines and storage: a tree of metadata files that answers, for any moment in the table's history, “exactly which data files are in this table, with what schema, partitioned how?” — so that many engines can share one table with ACID guarantees, without a database server in the middle.
Follow the pointer down.
Every Iceberg table is a tree, rooted at one tiny pointer in a catalog. Follow it down and you reach every byte of data — with statistics at every level so engines can skip what they don't need. Click any node to see what actually lives inside it.
catalog
Every commit is a snapshot. Nothing is edited in place.
Writers never modify existing files — they write new ones and commit a new
snapshot that references a different set. Old snapshots stick around, which is
why time travel is free. Run some operations against this tiny
orders table, then click any
snapshot in the timeline to travel back.
Write side
Each button = one commit.
Reader view catalog → v1.metadata.json
| order_id | item | qty | ts |
|---|
Snapshot timeline
Click a snapshot to time-travel the reader.
Files on disk
This demo uses copy-on-write: an UPDATE rewrites the whole file that holds the row. Iceberg's format v2 also supports merge-on-read — writing small delete files instead and reconciling at read time — which is cheaper for writes, at some read cost. Old files are only physically removed when you expire snapshots.
Query planning that skips almost everything.
This table is partitioned by day(ts) —
a transform of a real column, not a separate magic column you have to remember.
Narrow the date filter and watch planning prune in two stages: the manifest list's
partition ranges eliminate whole manifests, then per-file stats eliminate files.
You filtered ts. Nobody filtered a partition column. Iceberg stores the
relationship order_day = day(ts) in the table metadata and applies it for you —
that's hidden partitioning. In Hive, partitioning by date meant a separate
dt column: filter ts instead and you scan the entire table, silently.
Iceberg can even change the partition scheme later without rewriting old data.
Schema changes without fear — or rewrites.
Iceberg tracks every column by a permanent field ID, not by its name or position. Names are for humans; IDs are the truth. That makes rename, add, and drop pure metadata operations — the data files are never touched. Try it:
Table schema schema-id 0
Schema changes are metadata-only commits.
Data files on disk never rewritten
00012-….parquet · Jan
00048-….parquet · Jun
How a read resolves
schema column → column in an old file, matched by ID
Two writers, one table, no locks.
There's no server coordinating writes. Instead, committing is an atomic compare-and-swap on the catalog pointer: “if the table still points at v5, point it at my v6.” Losers don't corrupt anything — they re-check for real conflicts and retry. Step through a race:
A Streaming ingest (appends rows)
B Compaction job (rewrites small files)
Press play — or step through the race one move at a time.
This is optimistic concurrency. Because snapshots are immutable and commits swap one pointer, readers get serializable isolation for free — a query planned against v5 keeps reading v5's files even while v6 and v7 land. If B's rewrite had touched the same files as A, validation would fail and B would redo its work instead of committing a corrupt table.
From Netflix's problem to everyone's standard.
Born at Netflix
Ryan Blue and Dan Weeks design a table format to fix Hive's correctness and scale problems on S3.
Donated to Apache
Iceberg enters the Apache Incubator as an open specification with a reference implementation in Java.
Top-level project
Graduates to a top-level Apache project; Spark, Trino, and Flink support matures rapidly.
Format v2
Row-level deletes via delete files (merge-on-read), making streaming upserts and CDC practical on a data lake.
The format wars end
Snowflake and Databricks both go all-in (Databricks acquires Tabular, the company founded by Iceberg's creators). Iceberg becomes the de facto standard.
Format v3
Deletion vectors, row lineage, variant and geospatial types — atop the REST catalog protocol (standard since 2022) as the interoperability layer.
Who speaks it
One table on S3, readable and writable from all of these — that's the point.
Keep diving
The canonical sources, in reading order:
Sound your depth.
Four questions. Answer honestly — the explanations are the point.