Metadata

Cluster state is a replicated state machine whose log lives in a SQL database. Nodes propose commands, the sink appends them in order, every node tails the log and applies each command to an in-memory State, and the result is published as a View. There is no consensus protocol in the nodes. The database's single-writer transaction is the ordering point.

The command log

Table	Contents
meta_log	(idx, payload), append-only, one encoded command per row
meta_snapshot	(id, applied_idx, payload), a full State at an index
meta_lease	One row. Whoever holds it runs the coordinator, the tiering worker and the lifecycle loops
meta_node	One row per node with its last heartbeat

The same schema is created on Postgres and on SQLite. SQLite, including sqlite::memory:, is for a single node. Postgres is for a cluster.

Proposals and views

• A proposal is a Command. The sink batches concurrent proposals from the local node into one transaction, tails the log to learn its own index, applies the command and resolves the proposer with the applied index.
• A View is (applied_index, State). Readers take a snapshot of the current view or wait_applied(index) for a specific index.
• Every Flight response carries mink-applied-index, the index the serving node had applied. A client sends it back as mink-min-applied-index on the next request, and the receiving node waits until its own view has caught up. This gives read-your-writes across nodes without pinning a client to one node.

Commands

Group	Commands
s3stream	RegisterNode, CreateStream(s), OpenStream, CloseStream, TrimStream, DeleteStream, PlaceStream, TransferStream, CompleteTransfer, PrepareObject, CommitStreamSetObject, CompactStreamObject, ExpirePreparedObjects, CleanDestroyedObjects, AllocateProducerIds
KV	PutKv, PutKvIfAbsent, DeleteKv, DeleteKvIfMatches
Catalog	CreateDatabase, DropDatabase, CreateTable, DropTable, AlterTable, CreatePartition, DropPartition, LeadBucket, CommitKvSnapshot, DropKvSnapshot, CommitLakeSnapshot, Allocate, RegisterCoordinator
Offsets	RegisterProducerOffsets, DeleteProducerOffsets, ExpireProducerOffsets, CommitGroupOffsets, DeleteGroupOffsets, ExpireGroupOffsets

The s3stream group is the stream engine's own metadata: which node has a stream open and at which epoch, which objects hold which stream ranges, which objects are prepared but not yet committed. The catalog group is Mink's. Both apply through the same state machine, so a table create and the creation of its streams are one ordered sequence. Allocate hands out ids and auto-increment ranges.

Snapshots and truncation

The lease holder encodes the whole State into meta_snapshot periodically and deletes meta_log rows at or below the snapshot index. A starting node loads the latest snapshot and replays the log from there. Snapshot encoding is versioned with a leading version byte.

Lease and heartbeats

	Detail
Lease	One row updated with compare-and-set. Acquired or renewed every lease_ttl / 4. Lost when a renew fails. Leadership is published as a watch that the coordinator, tiering worker and lifecycle loops follow
Heartbeat	Each node updates its meta_node row every lease_ttl / 4. A node whose heartbeat is older than the TTL is dead for assignment and failover
Shutdown	A node releases its lease and expires its own heartbeat row on the way out

Default lease_ttl is 30 seconds. See Ownership and failover for what happens when a node stops heartbeating.