Snapshots
A snapshot is a table at one moment: for each partition, the offsets the table covers. Each snapshot is an Iceberg snapshot of the same id. A writer commits one every snapshot_interval, 10 seconds by default, and a table load adds the newest records on top.
Commits
Every snapshot_interval the table's writer reads the indexed ranges from Postgres and commits the next snapshot when anything changed. It writes the Iceberg metadata to the bucket first, then commits the snapshot in Postgres. It never reads Kafka records.
| Part | Holds |
|---|---|
| Snapshot id | 1, 2, 3, … in commit order. The Iceberg snapshot has the same id. |
| Offsets | For each partition, [start_offset, end_offset) |
| Files | The data files inside those offsets |
| Summary | mink.partition.{p}.start-offset and mink.partition.{p}.end-offset for each partition p |
Loads
A table load returns the latest snapshot. When a partition's last stable offset is past the snapshot's end, the load adds a tail: the offsets in between, as data files built in memory to take their statistics, like an indexer's. The tail is never committed. Two loads that see the same offsets share one tail.
A load covers every record committed in Kafka before it read the last stable offsets.
Committed records only
Mink reads Kafka as a read_committed consumer would.
| Record | In the table |
|---|---|
| Committed, or written without a transaction | Yes |
| In an aborted transaction | No |
| In an open transaction | Not until it commits |
| Control record, such as a transaction marker | No |
Time travel
Engines read older snapshots by id or by time. See Retention.