Skip to content

Snapshots ​

A snapshot is a table at one moment: for each partition, the offsets the table covers. Each snapshot is an Iceberg snapshot of the same id. A writer commits one every snapshot_interval, 10 seconds by default, and a table load adds the newest records on top.

load, at each last stable offsetpartition 0snapshot Ntailpartition 1snapshot Ntailpartition 2snapshot Ntail

Commits ​

Every snapshot_interval the table's writer reads the indexed ranges from Postgres and commits the next snapshot when anything changed. It writes the Iceberg metadata to the bucket first, then commits the snapshot in Postgres. It never reads Kafka records.

Postgresindexed ranges, snapshotswriterone per tablebucketIceberg metadata1. read the indexed ranges3. commit the snapshot2. write the metadata
PartHolds
Snapshot id1, 2, 3, … in commit order. The Iceberg snapshot has the same id.
OffsetsFor each partition, [start_offset, end_offset)
FilesThe data files inside those offsets
Summarymink.partition.{p}.start-offset and mink.partition.{p}.end-offset for each partition p

Loads ​

A table load returns the latest snapshot. When a partition's last stable offset is past the snapshot's end, the load adds a tail: the offsets in between, as data files built in memory to take their statistics, like an indexer's. The tail is never committed. Two loads that see the same offsets share one tail.

A load covers every record committed in Kafka before it read the last stable offsets.

Committed records only ​

Mink reads Kafka as a read_committed consumer would.

RecordIn the table
Committed, or written without a transactionYes
In an aborted transactionNo
In an open transactionNot until it commits
Control record, such as a transaction markerNo

Time travel ​

Engines read older snapshots by id or by time. See Retention.