What is Apache Iceberg? The open table format, explained
Apache Iceberg is an open table format that turns files in object storage into tables with snapshots, ACID commits and time travel. How it works, in Python.

Apache Iceberg is an open table format: a specification for a layer of metadata files that sits on top of ordinary data files (Parquet, Avro or ORC) in object storage and makes a folder of files behave like a database table, with atomic commits, consistent snapshots, schema and partition evolution, and time travel. It is not a database or a query engine. Engines such as Spark and DuckDB read and write the same Iceberg table, which is what makes it the core of most “lakehouse” designs.
Where did Apache Iceberg come from?
Iceberg entered the Apache Incubator on November 16, 2018 and graduated to a top-level Apache project on May 20, 2020. The incubator page describes it in one line: “a table format for large, slow-moving tabular data.”
The problem it solved was the Hive-style data lake, where a “table” was a directory and its partitions were subdirectories. That worked until two jobs wrote at once, or a reader listed a directory halfway through a write, or someone needed to change how the table was partitioned. Iceberg replaced “the table is whatever files are in this folder” with “the table is whatever the current metadata file says.”
How does Apache Iceberg work?
The spec is the clearest explanation. Its goals include:
- Serializable isolation: “Reads will be isolated from concurrent writes and always use a committed snapshot of a table’s data.”
- Speed: “Operations will use O(1) remote calls to plan the files for a scan.”
- Scale: “Job planning will be handled primarily by clients and not bottleneck on a central metadata store.”
- Evolution: “Tables will support full schema and partition spec evolution.”
The structure is a tree of immutable files:
- Data files: Parquet, Avro or ORC, never modified after they are written.
- Manifest files: lists of data files with per-file statistics such as column ranges, so a query can skip files without opening them. The spec: “Data files in snapshots are tracked by one or more manifest files.”
- Manifest list: one per snapshot, naming the manifests that make it up.
- Table metadata file: the schema, partition spec and list of snapshots. “All changes to table state create a new metadata file.”
- A catalog pointer: which metadata file is current.
A commit writes new data files, new manifests and a new metadata file, then swaps the pointer. The spec: “An atomic swap of one table metadata file for another provides the basis for serializable isolation.” If another writer swapped first, the swap fails and the writer retries on top of the new state. That is optimistic concurrency, and it is the whole trick.
What does an Iceberg commit look like?
Here is that mechanism as a toy, in Python’s standard library with JSON files instead of Parquet. Each commit writes a data file, a manifest and a new metadata file, then swaps one pointer. It also shows time travel and a lost race between two writers.
# toy_iceberg.py: the shape of an Iceberg commit, with JSON files in a temp dir. Stdlib only.
# Data files are immutable; each commit writes a data file, a manifest and a new metadata
# file, then swaps one pointer. Readers only ever see a whole snapshot.
import itertools, json, os, tempfile
root, seq = tempfile.mkdtemp(), itertools.count(1)
path = lambda name: os.path.join(root, name)
def write(name, obj):
with open(path(name), "w") as f: json.dump(obj, f)
return name
def current(): # the catalog: one pointer to the current metadata file
with open(path("POINTER")) as f: return json.load(open(path(f.read())))
def commit(base, rows):
n = next(seq)
files = (base["snapshots"][-1]["files"] if base["snapshots"] else []) + [write(f"data-{n}.json", rows)]
snap = {"id": len(base["snapshots"]) + 1, "manifest": write(f"manifest-{n}.json", files), "files": files}
meta = {"version": base["version"] + 1, "snapshots": base["snapshots"] + [snap]}
name = write(f"v{meta['version']}-{n}.metadata.json", meta)
# optimistic concurrency: swap only if nobody committed since we read `base`.
# (A real catalog does this check-and-swap atomically; this toy is single-threaded.)
if current()["version"] != base["version"]:
raise RuntimeError(f"conflict: table moved past v{base['version']}; {name} is now an orphan")
with open(path("POINTER.tmp"), "w") as f: f.write(name)
os.replace(path("POINTER.tmp"), path("POINTER"))
return meta
def scan(meta, snapshot_id=None): # time travel = read an older snapshot's file list
snap = meta["snapshots"][-1] if snapshot_id is None else meta["snapshots"][snapshot_id - 1]
return [row for f in snap["files"] for row in json.load(open(path(f)))]
with open(path("POINTER"), "w") as f: f.write(write("v0.metadata.json", {"version": 0, "snapshots": []}))
m = commit(current(), [{"order": 1, "amt": 40}, {"order": 2, "amt": 15}])
m = commit(m, [{"order": 3, "amt": 99}])
print(f"v{m['version']} scan:", scan(m))
print("time travel to snapshot 1:", scan(m, 1))
stale = m # writer A reads v2 ...
m = commit(m, [{"order": 4, "amt": 7}]) # ... writer B commits v3 first
try: commit(stale, [{"order": 5, "amt": 1}])
except RuntimeError as e: print("writer A:", e)
print("table now:", len(scan(current())), "rows at v%d" % current()["version"])
print("files:", sorted(os.listdir(root)))
Output (Python 3.13.11):
v2 scan: [{'order': 1, 'amt': 40}, {'order': 2, 'amt': 15}, {'order': 3, 'amt': 99}]
time travel to snapshot 1: [{'order': 1, 'amt': 40}, {'order': 2, 'amt': 15}]
writer A: conflict: table moved past v2; v3-4.metadata.json is now an orphan
table now: 4 rows at v3
files: ['POINTER', 'data-1.json', 'data-2.json', 'data-3.json', 'data-4.json', 'manifest-1.json', 'manifest-2.json', 'manifest-3.json', 'manifest-4.json', 'v0.metadata.json', 'v1-1.metadata.json', 'v2-2.metadata.json', 'v3-3.metadata.json', 'v3-4.metadata.json']
Three things in that output map straight to real Iceberg. Time travel is free, because the old snapshot’s file list is still in the metadata and its data files were never touched. The losing writer did not corrupt anything: its swap was refused and the table stayed at 4 rows. And the losing writer left files behind (data-4.json, manifest-4.json, v3-4.metadata.json) that no snapshot references. Real Iceberg has the same problem, and the maintenance docs say so: “task or job failures can leave files that are not referenced by table metadata.”
Real Iceberg adds what the toy skips: manifests with column statistics for file pruning, a manifest list per snapshot so unchanged manifests are reused, and a catalog whose swap is truly atomic.
What are hidden partitioning and schema evolution?
In Hive-style tables, users had to know the partition layout and filter on a derived column, such as event_date, in addition to the real timestamp. In Iceberg, a table has a partition spec “that defines how to produce a tuple of partition values from a record,” using transforms like year, month, day, hour, bucket[N] and truncate[W]. Queries filter on the real column, and the spec says this “simplifies queries because users don’t have to supply both logical predicates and partition predicates.” Because partitioning is metadata, you can change it later without rewriting old data.
Schema evolution works the same way. “Each field in the tuple is named and has an integer id that is unique in the table schema,” and “Columns in Iceberg data files are selected by field id.” So a rename changes the name, “but not the field ID,” and adds, drops, reorders and renames are metadata changes, not rewrites.
What changed across Iceberg format versions?
The spec defines three format versions in use:
- Version 1 “defines how to manage large analytic tables using immutable file formats: Parquet, Avro, and ORC.”
- Version 2 “adds row-level updates and deletes for analytic tables with immutable files.”
- Version 3 adds nanosecond timestamps,
variant,geometryandgeographytypes, column default values, multi-argument partition transforms, row lineage tracking, binary deletion vectors and table encryption keys.
Version 2 is the one that made Iceberg practical for workloads with updates. The spec also notes that “Version 4 is under active development and has not been formally adopted.”
Iceberg vs Delta Lake vs Hudi: what’s the difference?
All three are open table formats that add transactional metadata over Parquet in object storage, and the core idea (immutable data files plus a log or tree of metadata) is shared. The practical differences are in engine support, governance and which catalog your platform standardizes on. I would not pick between them from a feature matrix; I would pick the one every engine you already run can write to safely, which is the question that decides most of these choices.
Where does Apache Iceberg break?
Small files. Streaming and frequent commits create many small data files, and the docs warn that “small data files causes an unnecessary amount of metadata and less efficient queries from file open costs.” Compaction (rewriteDataFiles) has to be scheduled, not hoped for.
Snapshots keep deleted data alive. This is the one I would circle from my years on data platforms with privacy compliance work. “Data files are not deleted until they are no longer referenced by a snapshot that may be used for time travel or rollback.” A row you deleted to honor an erasure request still exists in older snapshots until you expire them. Regular snapshot expiration is a compliance control as much as storage hygiene.
Orphan cleanup is sharp. Removing unreferenced files is necessary, but the docs say “It is dangerous to remove orphan files with a retention interval shorter than the time expected for any write to complete.” Delete too eagerly and you remove files an in-flight commit is about to reference.
Contention. Optimistic concurrency is cheap when writers rarely collide. Many writers committing to the same table every few seconds will spend their time retrying.
The catalog is a dependency. The atomic pointer swap lives in the catalog. Its availability and access control become your table’s availability and access control.
What I would do
For a new analytical platform on object storage, I would default to Iceberg tables, one catalog that every engine uses, and format version 2 or later so updates and deletes are possible. On day one I would schedule three jobs per table: compaction, snapshot expiration with a retention window agreed with legal and compliance, and orphan file cleanup with a conservative retention window.
For small data, I would not start here at all. A single DuckDB process over Parquet files covers a lot of analytics, and DuckDB’s iceberg extension can read Iceberg tables, and write to them through a REST catalog, when you grow into it. If the data is user-edited state that needs to sync across devices instead of analytical tables, the merge model you want is a CRDT, not snapshots. And if the plan is to point an LLM at the lakehouse, read text-to-SQL first, because clean table metadata helps a model as much as it helps a person.
FAQ
Is Apache Iceberg a database?
No. It is a table format specification plus libraries. Data lives in files in object storage; engines such as Spark and DuckDB do the querying and use Iceberg’s metadata to know which files make up the table.
What is a lakehouse?
A design where analytical tables live as open-format files in cheap object storage, with a table format like Iceberg adding transactions, schema and snapshots, so several engines can query the same data without copying it into a warehouse.
How does Iceberg time travel work?
Every commit creates a new snapshot and keeps the old ones in metadata. A query can name an older snapshot ID or timestamp and read exactly the files that snapshot listed, until that snapshot is expired.
Does Iceberg support updates and deletes?
Yes, from format version 2, which “adds row-level updates and deletes.” Version 3 adds binary deletion vectors.
Does Iceberg replace Parquet?
No. Parquet is the file format for the data; Iceberg is the metadata layer that turns many Parquet files into one table.
Sources
- Apache Iceberg, Table spec, accessed October 31, 2026
- Apache Iceberg, Maintenance (docs 1.12.0), accessed October 31, 2026
- DuckDB, Iceberg extension, accessed October 31, 2026
- Apache Incubator, Iceberg project status: entered incubation November 16, 2018, graduated May 20, 2020