Background Indexing

Fluree maintains query-optimized indexes through a background indexing process. This document covers the indexing architecture, configuration, and monitoring.

Index Architecture

Fluree maintains four index permutations for efficient query execution:

SPOT (Subject-Predicate-Object-Time)

Organized by subject first:

ex:alice → schema:name → "Alice" → [t=1, t=5]
ex:alice → schema:age → 30 → [t=1]
ex:alice → schema:age → 31 → [t=10]

Optimized for: “Give me all properties of this subject”

POST (Predicate-Object-Subject-Time)

Organized by predicate first:

schema:name → "Alice" → ex:alice → [t=1, t=5]
schema:age → 30 → ex:alice → [t=1]
schema:age → 31 → ex:alice → [t=10]

Optimized for: “Find all subjects with this property/value”

OPST (Object-Predicate-Subject-Time)

Organized by object first:

"Alice" → schema:name → ex:alice → [t=1, t=5]
30 → schema:age → ex:alice → [t=1]
31 → schema:age → ex:alice → [t=10]

Optimized for: “Find subjects with this object value”

PSOT (Predicate-Subject-Object-Time)

Organized by predicate, then subject:

schema:name → ex:alice → "Alice" → [t=1, t=5]
schema:age → ex:alice → 30 → [t=1]
schema:age → ex:alice → 31 → [t=10]

Optimized for: “Get all values for this predicate”

Indexing Process

1. Transaction Commit

t=42: Transaction committed
  - Flakes written to append-only log
  - Commit metadata created
  - Commit published to nameservice (commit_t=42)

2. Indexer Detection

Background indexing is triggered when the ledger’s novelty exceeds the configured threshold (see Configuration below):

Indexer checks: commit_t=42, index_t=40
Indexer: Need to index t=41, t=42

Background indexing builds a new index snapshot up to a specific to_t (typically the current commit_t when the job starts). During the job, new commits may arrive; those remain in novelty for the next cycle.

Incremental indexing (default path):
  - Load the existing index root (CAS CID) from nameservice
  - Resolve only commits with t in (index_t, to_t]
  - Merge resolved novelty into only the affected leaf blobs (Copy-on-Write)
  - Update dictionaries (forward packs + reverse trees)
  - Assemble a new root referencing mostly-unchanged CAS artifacts

Fallback:
  - If incremental indexing cannot safely proceed, fall back to a full rebuild

4. Index Publishing

When complete:

  - Upload new CAS blobs (leaves, branches, dict blobs) as needed
  - Upload the new index root (CAS CID)
  - Publish index_head_id to nameservice (atomic “commit point”)
  - Update index_t to to_t

Novelty Layer

The novelty layer consists of transactions committed but not yet indexed:

Current State:
  commit_t = 150
  index_t = 145
  novelty = [t=146, t=147, t=148, t=149, t=150]

Query Execution with Novelty

Queries combine indexed data with novelty:

Query for ex:alice's properties:

1. Check SPOT index (up to t=145)
2. Apply novelty layer (t=146 to t=150)
3. Combine results

Impact of Large Novelty

Small novelty (< 10 transactions):

Minimal query overhead
Fast query execution

Large novelty (> 100 transactions):

Significant query overhead
Slower query execution
Higher memory usage

Configuration

Background indexing is on by default. Indexing is triggered based on novelty size thresholds:

Enable/disable background indexing: --indexing-enabled / FLUREE_INDEXING_ENABLED (default true; disable only when a peer/indexer process owns this storage)
Trigger threshold (soft): --reindex-min-bytes / FLUREE_REINDEX_MIN_BYTES
Backpressure threshold (hard): --reindex-max-bytes / FLUREE_REINDEX_MAX_BYTES

See Operations: Configuration for the canonical flag/env/config-file reference.

Incremental parallelism (per ledger)

Within a single incremental indexing job, Fluree can update multiple (graph, index-order) branches concurrently. This is bounded by:

IndexerConfig.incremental_max_concurrency (default: 4)

This setting is part of the Rust IndexerConfig used by the indexer pipeline; it is not a server CLI flag. Increasing it can improve throughput on multi-graph ledgers and can run the four main index orders (SPOT/PSOT/POST/OPST) in parallel, at the cost of higher peak memory.

Monitoring

Check Index Status

curl http://localhost:8090/v1/fluree/info/mydb:main

Response:

{
  "ledger_id": "mydb:main",
  "branch": "main",
  "commit_t": 150,
  "index_t": 145,
  "commit_id": "bafy...headCommit",
  "index_id": "bafy...indexRoot"
}

Key Metrics:

index lag (txns): commit_t - index_t

For byte-level novelty size and indexing trigger decisions, see the indexing block returned by transaction and replication endpoints (e.g. POST /push/<ledger>), documented in API Endpoints.

Key Log Messages

At INFO, background indexing now emits coarse-grained progress logs that make it easier to distinguish:

request queued vs. worker started
current wait status while trigger_index() is blocked
incremental vs. rebuild path selection
commit-chain walking progress
commit resolution progress and phase completion

When background indexing is queued by an HTTP transaction request, the worker logs also include copied request_id and trace_id fields from the triggering request. This provides log-level correlation between the foreground request and the later background build without making the index build part of the original request trace.

At DEBUG, the same wait and commit-walk paths emit more frequent progress updates for incident debugging without changing behavior.

When you call indexing through the Rust API with trigger_index(), wait timeout is optional and should generally be chosen by the caller. Leave TriggerIndexOptions.timeout_ms unset to wait until completion, or set it explicitly for bounded environments such as Lambda jobs, HTTP gateways, or other workers with a fixed maximum runtime.

TriggerIndexResult carries fuel: Option<f64> — the total fuel charged for the build that satisfied the trigger. The background indexer always meters each build, so callers see Some(N) for a real build, Some(0.0) when the requested t was already indexed (no work was needed), and the same value for all coalesced trigger callers satisfied by a single underlying build. See Tracking and Fuel for the cost-ladder and rate semantics.

Health Indicators

Healthy:

index_lag: 0-10 transactions
index_rate > transaction_rate

Warning:

index_lag: 10-50 transactions
index_rate ≈ transaction_rate

Critical:

index_lag: > 50 transactions
index_rate < transaction_rate

Performance Tuning

Optimize for Write-Heavy Loads

fluree server run -- \
  --indexing-enabled \
  --reindex-min-bytes 200000 \
  --reindex-max-bytes 2000000

Larger thresholds reduce indexing frequency (more novelty accumulation), trading some query-time overlay cost for reduced background indexing activity.

Optimize for Read-Heavy Loads

fluree server run -- \
  --indexing-enabled \
  --reindex-min-bytes 50000

Smaller reindex-min-bytes keeps novelty smaller (better query performance) at the cost of more frequent background indexing cycles.

Index Storage

Index Snapshots

Indexes are stored as immutable, content-addressed snapshots:

  - Leaf blobs (FLI3) and branch manifests (FBR3)
  - Dictionary blobs (forward packs, reverse tree leaves/branches)
  - An index root blob (FIR6) that references everything needed for queries

The nameservice stores the current index root CID (index_head_id) and its watermark (index_t). Peers fetch only the CAS objects they need on demand.

Index Retention

Old index snapshots are retained for time-travel safety and concurrent query safety. Cleanup is performed by the binary index garbage collector, governed by:

IndexerConfig.gc_max_old_indexes (--gc-max-old-indexes / FLUREE_GC_MAX_OLD_INDEXES, default 5): old versions to retain.
IndexerConfig.gc_min_time_mins (--gc-min-time-mins / FLUREE_GC_MIN_TIME_MINS, default 30): minimum age before a version can be collected.

Both must be satisfied, so the slower of the two wins. Under a sustained publish rate that is the age guard: a ledger publishing every few seconds holds far more than gc_max_old_indexes versions inside the 30-minute window, and the count bounds nothing. Retention becomes “however many versions fit in the window”, which grows with publish rate and per-version size. Every one of those versions is still reachable and still on disk.

IndexerConfig.gc_hard_max_old_indexes (--gc-hard-max-old-indexes / FLUREE_GC_HARD_MAX_OLD_INDEXES, unset by default): a ceiling past which versions are collected regardless of age.

The ceiling is opt-in because the age guard is what protects a query that started against an older version: until the guard expires, that version’s leaves are still in storage. Past the ceiling they are released regardless, and a query still reading them fails or reads a torn version. Set it well above the number of versions the ledger publishes during your longest query. Each GC pass reports how many versions it collected past the ceiling (age_guard_overridden in the completion log line), so an override that is firing is visible at the default log level.

It is a bound on versions, not bytes. What a retained version costs varies by orders of magnitude between ledgers — one deployment measured ~7.7 GiB per version on one ledger and ~3.3 GiB on another — so size the ceiling from the per-version disk use you observe, and expect objects/history to hold roughly versions × per-version bytes at the ceiling.

The collector runs after every index publish, and also on the worker’s periodic tick (IndexerConfig.catchup_interval, default 300 s), including once at start-up. The periodic pass is what reaches a ledger that stops publishing: a load-once dataset whose final burst left hundreds of versions inside the age guard would otherwise keep them until its next write, which may never come. Passes are serialised per ledger name, the tick runs them one at a time, and releases go out in batches — one DeleteObjects request per thousand artifacts on S3, a few in flight — so a large pass does not take the storage request cap away from readers.

The collector reclaims artifacts by name. Each index root carries a garbage manifest listing what the previous version replaced, and the collector walks the prev-index chain releasing exactly those. It therefore reaches only artifacts that some manifest records.

Dictionary blobs live in the ledger-wide @shared/dicts/ namespace, shared by every branch: a branch created from another starts out reading the source’s dictionaries rather than a copy, so the source’s manifest can name a blob the fork still reads. The collector releases a dictionary blob only when no other branch of the ledger — retracted branches included — still reaches it through its own index chain. A blob a sibling still reaches is deferred; when that sibling replaces it, its own manifest names it and the sibling’s pass releases it. Whichever branch drops a blob last deletes it, and dropping a branch releases the blobs only that branch referenced. A ledger with a single branch releases every blob its manifests name.

A manifest’s word is not final. Content addressing gives byte-identical output the same name, so a later build can recreate a blob an earlier manifest recorded as replaced — reverse-dictionary leaves do this routinely under monotonically increasing keys (ULIDs, UUIDv7, sequential ids). The collector therefore never releases anything a retained root still references, whatever a manifest says. And because a build can recreate such a blob at any moment, a pass plans against a snapshot and then releases inside a short window during which no branch of the ledger builds: it waits for a build in flight, defers new ones, re-reads the current index head of the branch and of each of its siblings, and only then releases. Builds resume when the window closes. A pass that has nothing to release never opens one, and a pass that finds a branch held by a reindex or sweep releases nothing and leaves the work to a later pass. Which branches are siblings comes from the worker’s branch listing, refreshed every catchup_interval, plus any branch the worker has built since; with gc_hard_max_old_indexes set, or an age guard under ten minutes, the listing is taken fresh for every pass. All of this behaves the same on every storage and nameservice backend: it needs only a consistent single-record lookup, which the file, S3 and DynamoDB nameservices all provide. It covers the indexer in one process; two processes indexing the same storage are not coordinated.

Reclaiming orphaned artifacts

Artifacts orphaned outside that chain are invisible to the collector. The main source is a reindex published by Fluree 4.1.4 or earlier: it severed the prev-index chain, leaving every earlier root — and the blobs only those roots referenced — unreachable from any walk. Retention could never truncate past it, so a ledger that had been reindexed accumulated index artifacts without bound.

POST /v1/fluree/sweep reclaims them, and fluree sweep <ledger> does the same from the CLI. A sweep enumerates what storage holds, subtracts everything reachable from a live index chain, and releases the remainder. Use --dry-run (or POST /v1/fluree/sweep/plan) to see what would be released first.

A sweep is also what reclaims what the collector cannot reach any more: dictionary blobs whose manifests were consumed on Fluree 4.2.0 or earlier, which left every dictionary blob to the sweep, blobs a branch drop could not attribute, and anything orphaned off a chain. A ledger that accumulated dictionaries under one of those versions can hold many times its live index in @shared/dicts/; one sweep clears the backlog, and afterwards the collector keeps up. Running one when nothing is reclaimable is a safe no-op.

A sweep is ledger-wide rather than per-branch, because dictionary blobs are shared across a ledger’s branches. It touches only index artifacts — commits, transactions, and config blobs are reachable through the commit chain rather than the index chain, so a sweep cannot establish that they are unreferenced and never considers them.

Single-process deployments only. The hold that keeps index builds from writing during a sweep excludes the server’s own indexer. An external or second-process indexer writing to the same storage is not excluded, and its in-flight artifacts would be indistinguishable from orphans.

POST /v1/fluree/reindex rebuilds the index but does not reclaim anything by itself. A reindex now participates in the GC chain like any incremental build, so ordinary retention applies to the roots it supersedes; a sweep is only needed for artifacts orphaned by an earlier version.

Troubleshooting

High indexing lag

Symptom: commit_t - index_t grows continuously

Causes:

Transaction rate exceeds indexing capacity
Large transactions
Insufficient resources

Solutions:

Reduce reindex-min-bytes so indexing triggers sooner
Increase resources for the indexer (CPU/memory and storage throughput)
Consider running a dedicated indexer process (separate from the transactor)
For incremental indexing, consider increasing IndexerConfig.incremental_max_concurrency

Slow Indexing

Symptom: index_t advances slowly (or stops advancing)