Cluster

Mink nodes share one Postgres and nothing else. Work moves between them through leases in Postgres. Nodes never talk to each other, except to forward a data file read to its owner and an admin request to the node it names.

Members

A node registers in Postgres when it starts and renews its row every lease_ttl / 3, 10 seconds by default. A node is live while its row has not expired. A node that stops cleanly releases its leases and leaves at once. A node that crashes drops out when its row expires, after lease_ttl, 30 seconds by default.

Leases

Lease	Scope	Holder runs
Partition lease	One partition of one table	The partition's indexer
Writer lease	One table	The table's writer: commits, assembly, statistics, expiry, collection

Each lease has an epoch that grows every time it changes hands. Postgres rejects every write under an old epoch, so a node that lost a lease cannot change anything.

Sharing

Every lease_ttl / 3, each node with the indexer role applies the same rules.

Rule	Effect
A table's partition share is its partition count divided by the indexing nodes, rounded up	Partitions spread evenly
A node below its share takes free leases, the ones it prefers most first	New and freed partitions get an indexer
A node above its share releases one lease of that table per round	A joining node takes work gradually
Each table's writer lease goes to the node that prefers it most	One writer per table
max_partitions caps the partition leases one node holds	Limits one node's indexers

A node's preference for a lease is a hash of its id and the lease's scope. Every node computes the same preferences.

Failures

Failure	Effect
A node crashes	Its leases expire after lease_ttl. Other nodes take them, and indexing resumes from the indexed ranges.
A node joins	Every share drops. Other nodes hand back one partition per table per round.
Postgres is unreachable	Table loads, and reads of data files not in the owner's cache, answer 503. Cached files are still served. Indexers and writers stop. Once Postgres answers, every node takes its share again without a restart.
Kafka is unreachable	Table loads, builds and indexing fail until Kafka answers. Cached data files are still served.