Roadmap
Planned: Mink writes ended time buckets to Iceberg and serves one catalog over that history and Kafka.
History in Iceberg
Planned. Not built.
| Built | Planned | |
|---|---|---|
| Table | The records Kafka holds | The Iceberg history and the newer offsets in Kafka, with no gap or overlap between them |
| Data files | Built from Kafka on read, and cached | Ended, assembled time buckets written to the bucket as Parquet. Newer buckets still built from Kafka. |
| File sizes | Assembled up to assembled_bytes | Written slowly, after assembly, as large files only. No small files and no compaction jobs. |
| Bucket | Iceberg metadata only | Iceberg metadata, and the history's data files |
| History | As far back as Kafka's retention | Kept after Kafka's retention deletes the records |
| Freshness | Every record committed when the table loads | Unchanged |
| Catalog | Read-only | Read-only. Mink is the only writer. |