Indexing

Indexing turns a partition's committed records into data files with exact statistics. Each partition has one indexer at a time. Records are read once and not kept: Postgres gets each file's offsets, statistics and Parquet footer, and nothing is written to S3.

Where an indexer starts

Partition	Start
Of a newly added table	Its last stable offset. Backfill reads what came before.
Already indexed	The end of its indexed range
Added to the topic later	Its last stable offset

Data files

An indexer groups records into offset ranges in offset order. A range ends at the first of these:

Trigger	Setting	Default
Key, value and header bytes reach	file_bytes	128 MiB
Record timestamps span	file_window	60s
The range's first record was read this long ago	file_window	60s
A record falls in a later time bucket	granularity	hour

A range becomes one data file per time bucket among its records. Offsets with no committed record, such as aborted records and transaction markers, belong to the file before them.

Each file is built once in memory to take its statistics: row count, size, and per-column min/max, null and NaN counts. The bytes are then dropped. Engines get the same bytes later, built again from Kafka on read.

Backfill

Backfill indexes history backwards from the range's start, backfill_offsets at a time, 100,000 by default. Live indexing never waits for it.

history	Backfill goes down to
all, the default	Kafka's earliest offset
{ since = <RFC 3339 time> }	The first offset at or after that time
none	Nothing. The table starts at the last stable offset.

Schemas

Before writing files, the indexer merges every new schema its records name into the table schema. While the Schema Registry cannot be reached, the indexer waits instead of writing files.

Kafka retention

Every retention_check the indexer reads Kafka's earliest offset. See Retention.

Failures

Event	Effect
Indexer stops	Records read but not written are only in memory. The next indexer reads again from the end of the indexed range.
Node loses the lease	The indexer stops at once. Another node takes the partition when the lease expires, after lease_ttl, 30 seconds by default.
Lost lease, late write	Rejected by Postgres. An indexer without its lease adds no file.