Why not connectors
A connector is a stream processor that copies a topic into an Iceberg table: Flink, the Iceberg Kafka Connect sink, Spark Structured Streaming, RisingWave or Amazon Data Firehose. Some run as one binary, some as a cluster. The table is a second copy, as fresh as the last commit. Mink serves the topic itself as the table.
Freshness
A connector's table changes only when it commits. Records written since the last commit are not in it.
| Connector | Commits | Default |
|---|---|---|
| Flink | After each checkpoint | Checkpointing is off by default, and without it nothing commits |
| Iceberg Kafka Connect sink | Every iceberg.control.commit.interval-ms | 5 minutes |
| Spark Structured Streaming | Every trigger | Iceberg recommends at least 1 minute |
| RisingWave | Every commit_checkpoint_interval checkpoints | 60, about 60 seconds |
| Amazon Data Firehose | When its buffer fills or its interval ends | 300 seconds or 5 MiB |
A Mink table load covers every record committed in Kafka at that moment. Snapshots set nothing but how much of a load is tail, see Snapshots.
Upkeep
Frequent commits write small files and many snapshots. A connector's table needs maintenance jobs.
| Task | Connector | Mink |
|---|---|---|
| Small files | rewrite_data_files | Assembly joins an ended time bucket's files, without reading Kafka |
| Old snapshots | expire_snapshots | Expired after snapshot_retention |
| Unused files | remove_orphan_files | Collected after collection_grace |
| Manifests | rewrite_manifests | Grouped by time bucket, one per bucket once assembled |
Expiring snapshots or deleting orphan files under a running Flink job can break its state. Mink's writer runs all of these itself, see Retention.
Differences
| Connector | Mink | |
|---|---|---|
| Copies | Kafka, and Parquet in the bucket | Kafka only. The bucket holds metadata. |
| Freshness | The last commit | Every record committed when the table loads |
| Moving parts | A Flink, Connect, Spark or RisingWave deployment, a catalog, maintenance jobs | Mink nodes and Postgres |
| Schema changes | Off by default in the Connect sink. Firehose drops fields the table lacks. | New Schema Registry versions evolve the table |
| History | Kept in the table after Kafka deletes it | As far back as Kafka's retention |
| Transforms | Any, in the job | None. One row per record. |
When a connector fits
- Tables kept longer than the topic's retention.
- Records transformed, joined or filtered before they land.
- Engines that read only from cloud storage.