Writes

A write is an Arrow record batch addressed to a bucket or to a table. Bucket-addressed writes go straight to the leader. Table-addressed writes are split by the receiving node. Either way the row ends up in one bucket's log, acknowledged only once the s3stream WAL upload on object storage has completed.

Routing

Step	Rule
Partition	The partition key columns of each row name the partition. Missing partitions are created on demand
Bucket, with bucket keys	Hash of the encoded bucket-key columns with the table's bucketing rule, modulo bucket_count. Null in a key column is rejected
Bucket, without keys	The whole batch goes to one bucket. The next batch goes to the next bucket, round-robin
Leader	The bucket's leader from the metadata view. A stale view gets a redirect and retries

The Rust client routes on the client side and keeps one idempotent sequence per bucket. The mink write command, the Kafka listener and any client that sends AppendTable or PutTable let the receiving node route instead: it splits the batch, appends buckets it leads locally and forwards the rest to their leaders, then returns a Routed list with the offsets per bucket.

Requests

DoPut descriptors, one per Flight stream:

Descriptor	Fields	Table type
Append	bucket, schema_id, writer_id	Log table
Put	bucket, schema_id, writer_id, target_columns	Primary-key table
AppendTable	path, schema_id	Log table, routed by the node
PutTable	path, schema_id, target_columns	Primary-key table, routed by the node

Each Arrow batch on the stream carries batch_sequence for idempotence and, for Put, an optional changes vector marking rows as upserts or deletes. target_columns turns the batch into a partial update. The batch schema is then a subset of the table schema. schema_id names the table version the rows were encoded with. It is written into the batch header and travels with the rows.

Response per batch: Written { first_offset, last_offset, duplicated } for bucket writes, Routed { buckets[] } for table writes.

On the leader

1. Ownership. The node must lead the bucket in its current view, or it answers NotLeader with a redirect. The tablet's stream is open under the bucket's leader epoch. S3stream rejects appends under an older epoch.
2. Sequence. With a writer_id, the batch sequence is checked against the writer window. Duplicates return the original offsets with duplicated = true.
3. Primary-key tables. The KV tablet reads old rows, applies the merge engine and partial-update targets, and encodes the changelog batch. See KV tablets.
4. Offsets. The log tablet assigns base_offset and commit_timestamp.
5. Durability. The batch is submitted to the stream. s3stream groups concurrent batches into one WAL object PUT. The append resolves when that upload and every earlier one have completed. The high watermark advances and tailing readers are woken.
6. Flush. For primary-key tables, staged rows move from the prewrite buffer into the KV store.

Latency is one object-store PUT on an idle bucket. Under load, group commit spreads that PUT across every batch that arrived while the previous upload was in flight.

Kafka produce

A Kafka Produce request is a batch of records per topic partition. The listener maps the topic to a log table in the Kafka database, the partition to a bucket, and each record to a row of key, value, headers, timestamp. Records are appended as one Arrow batch through the same path as Append. Idempotent producers use the writer window: producer_id is a writer id from the shared allocator and base_sequence is the batch sequence. Transactional produce requests are rejected.

Failure cases

Failure	Outcome
Object store PUT fails or times out	Append fails. Nothing is acknowledged. The client retries with the same sequence
Leader changes mid-request	The old leader's append is rejected by the stream epoch. The client gets NotLeader with the new leader and retries
Node crashes after ack, before KV flush	The changelog is durable. Recovery replays it into the store
Duplicate after retry	Recognized by the writer window, original offsets returned
Partition missing	Created on demand, then written