BM25 Full-Text Search

Fluree provides integrated full-text search using the BM25 (Best Matching 25) ranking algorithm. BM25 indexes are implemented as graph sources: they index text content from a source ledger and expose search results that can be joined with structured graph queries.

What is BM25?

BM25 is a probabilistic ranking function that scores documents based on query term frequency and document length normalization. It’s widely used in search engines and information retrieval systems.

Key features:

• Term frequency with saturation (controlled by k1)
• Inverse document frequency weighting
• Document length normalization (controlled by b)
• English stemming and stopword filtering (default analyzer)
• Block-Max WAND for efficient top-k queries (early termination)
• Incremental index updates
• Time-travel: query the index as of any past transaction

Creating a BM25 Index

BM25 indexes can be managed three ways: over HTTP against a running server, via the fluree bm25 CLI, or through the Rust API (Bm25CreateConfig).

Over HTTP, POST /v1/fluree/bm25/create builds an index and POST /v1/fluree/bm25/sync brings one up to date; both are admin-protected. Listing, inspection, and drop are not BM25-specific — an index is a graph source, so GET /ledgers, GET /info/<alias>, and POST /drop already cover it. See Graph Source Endpoints.

An index does not track its source ledger on its own. Either sync it explicitly (from a maintenance job, or on demand), or start the server with --bm25-auto-sync to have it sync each index whenever its source ledger commits. Auto-sync is off by default because it adds background storage writes; under Raft it runs on the leader only, and it does not run on peer-mode servers, which forward their writes elsewhere.

The CLI (fluree bm25 create/list/sync/drop) drives a server when one is reachable, and runs in-process against local storage otherwise — so it also works under docker exec against a running server’s data directory; see CLI Management below.

Basic Index

#![allow(unused)]
fn main() {
use fluree_db_api::{Bm25CreateConfig, FlureeBuilder};
use serde_json::json;

let fluree = FlureeBuilder::file("/path/to/data").build()?;

// Create a ledger and insert some data
let ledger = fluree.create_ledger("docs:main").await?;
let tx = json!({
    "@context": { "ex": "http://example.org/" },
    "@graph": [
        { "@id": "ex:doc1", "@type": "ex:Article", "ex:title": "Rust programming guide" },
        { "@id": "ex:doc2", "@type": "ex:Article", "ex:title": "Python for beginners" },
        { "@id": "ex:doc3", "@type": "ex:Article", "ex:title": "Systems programming in Rust" }
    ]
});
let ledger = fluree.insert(ledger, &tx).await?.ledger;

// Define the indexing query
let query = json!({
    "@context": { "ex": "http://example.org/" },
    "where": [{ "@id": "?x", "@type": "ex:Article", "ex:title": "?title" }],
    "select": { "?x": ["@id", "ex:title"] }
});

// Create the BM25 index
let config = Bm25CreateConfig::new("article-search", "docs:main", query);
let result = fluree.create_full_text_index(config).await?;

println!("Indexed {} documents", result.doc_count);
println!("Graph source: {}", result.graph_source_id); // "article-search:main"
}

The graph source ID is {name}:{branch} — for example, article-search:main.

Indexing Query

The indexing query defines what to index. It’s a standard Fluree JSON-LD query with these requirements:

• Must include @id in the select (to identify documents)
• Must use select with a map form: {"?x": ["@id", "ex:prop1", "ex:prop2"]}
• All selected text properties are extracted and tokenized for search

The query can filter by type, filter by property values, or use any valid Fluree where clause:

{
    "@context": { "ex": "http://example.org/" },
    "where": [
        { "@id": "?x", "@type": "ex:Article", "ex:title": "?title" },
        { "@id": "?x", "ex:status": "published" }
    ],
    "select": { "?x": ["@id", "ex:title", "ex:content", "ex:tags"] }
}

Configuration Options

#![allow(unused)]
fn main() {
let config = Bm25CreateConfig::new("search", "docs:main", query)
    .with_branch("dev")
    .with_k1(1.5)
    .with_b(0.5);
}

Text Analysis

Fluree uses a default English analyzer that applies:

1. Tokenization: Unicode-aware word boundary splitting
2. Lowercasing: All tokens converted to lowercase
3. Stopword filtering: Common English words removed (the, a, an, is, etc.)
4. Stemming: Snowball English stemmer reduces words to root forms (e.g., “programming” -> “program”)

The analyzer is not configurable — it always uses the English pipeline for consistency.

Querying BM25 Indexes

BM25 search is expressed with the f: namespace predicates in a query’s where clause. It is a JSON-LD (FQL) feature and executes over the standard query endpoint — POST /v1/fluree/query — so search hits can be joined with ledger data in a single query. It is not available via SPARQL: the SPARQL parser does not recognize the f: search block, so those predicates would be matched as ordinary triples and return nothing.

Policy-enforced search must be scoped to a single ledger. A search whose from names more than one ledger conservatively returns no hits when any of those ledgers enforces a policy — not filtered hits. Per-flake readability cannot be resolved across a multi-graph scope, so the check hides everything rather than risk leaking. This is fail-closed and therefore safe, but silent: the same query returns rows with policies off and zero rows with them on, and no error explains the difference. Scope policy-enforced searches to one ledger.

JSON-LD Query Syntax

BM25 search is integrated into Fluree’s query system via the f: namespace predicates:

{
    "@context": {
        "ex": "http://example.org/",
        "f": "https://ns.flur.ee/db#"
    },
    "from": "docs:main",
    "where": [
        {
            "f:graphSource": "article-search:main",
            "f:searchText": "rust programming",
            "f:searchLimit": 10,
            "f:searchResult": {
                "f:resultId": "?doc",
                "f:resultScore": "?score"
            }
        },
        { "@id": "?doc", "ex:author": "?author" }
    ],
    "select": ["?doc", "?score", "?author"]
}

Pattern fields:

Combining Search with Structured Queries

The search pattern produces ?doc and ?score bindings. These can be joined with ledger data using normal where clauses:

{
    "@context": {
        "ex": "http://example.org/",
        "f": "https://ns.flur.ee/db#"
    },
    "from": "docs:main",
    "where": [
        {
            "f:graphSource": "article-search:main",
            "f:searchText": "rust",
            "f:searchLimit": 20,
            "f:searchResult": { "f:resultId": "?doc", "f:resultScore": "?score" }
        },
        { "@id": "?doc", "ex:title": "?title" },
        { "@id": "?doc", "ex:author": "?author" }
    ],
    "select": ["?doc", "?title", "?author", "?score"]
}

The BM25 search runs first and produces candidate bindings. The subsequent where clauses join those candidates with the source ledger to retrieve additional properties.

Rust API: Direct Search

You can also use the Rust API directly for programmatic search without the query engine:

#![allow(unused)]
fn main() {
use fluree_db_query::bm25::{Analyzer, Bm25Scorer};

// Load the index
let index = fluree.load_bm25_index("article-search:main").await?;

// Analyze query terms (same pipeline as indexing)
let analyzer = Analyzer::english_default();
let terms = analyzer.analyze_to_strings("rust programming");
let term_refs: Vec<&str> = terms.iter().map(|s| s.as_str()).collect();

// Score and rank
let scorer = Bm25Scorer::new(&index, &term_refs);
let results = scorer.top_k(10);

for (doc_key, score) in &results {
    println!("{}: {:.2}", doc_key.subject_iri, score);
}
}

Rust API: Query with BM25

Use query_connection_with_bm25 for integrated queries:

#![allow(unused)]
fn main() {
let query = json!({
    "@context": { "ex": "http://example.org/", "f": "https://ns.flur.ee/db#" },
    "from": "docs:main",
    "where": [
        {
            "f:graphSource": "article-search:main",
            "f:searchText": "rust",
            "f:searchLimit": 10,
            "f:searchResult": { "f:resultId": "?doc", "f:resultScore": "?score" }
        },
        { "@id": "?doc", "ex:author": "?author" }
    ],
    "select": ["?doc", "?score", "?author"]
});

let result = fluree.query_connection_with_bm25(&query).await?;
}

Index Maintenance

Syncing

BM25 indexes are not automatically updated when the source ledger changes. You must explicitly sync them:

#![allow(unused)]
fn main() {
// Incremental sync (detects changes since last watermark)
let sync_result = fluree.sync_bm25_index("article-search:main").await?;
println!("Upserted: {}, Removed: {}", sync_result.upserted, sync_result.removed);

// Force full resync (rebuilds the entire index)
let sync_result = fluree.resync_bm25_index("article-search:main").await?;
}

Incremental sync uses property dependency tracking to identify which subjects changed since the last indexed commit, then re-runs the indexing query and keeps the rows belonging to those subjects. If no affected subjects are detected, it falls back to a full resync.

Where the query shape allows it, the affected subjects are bound into the query itself as a values clause, so the scan seeks to those subjects instead of walking the corpus. That makes the indexed portion of a sync flat in corpus size rather than linear. Novelty — rows committed since the last index build — has no subject index and is still walked, so the win over that portion is small; reindex_min_bytes is what bounds how much novelty a sync has to walk, and the two are complementary.

Narrowing is skipped, and the whole-ledger scan used, whenever the indexing query is not one that can be narrowed safely: a list-form select (which does not say which variable is the document), a select binding more than one variable, a query that already carries its own top-level values, or any clause that shapes the result set rather than the scan — limit, offset, groupBy, having, selectOne. Those are declined because narrowing would change which rows come back, not just how they are found, and the sync treats a subject the query did not return as one to remove. Falling back is always correct, only slower. Each decline logs at debug with the condition that caused it.

Upgrading an existing index. Indexes built before document identifiers were canonicalised can hold two keys for one document — the full IRI written at build time and a prefixed form written by a later sync. Run one resync_bm25_index after upgrading. An incremental sync will not repair it: the removal pass only visits subjects the commit affected, so it never reaches the orphaned key, and document counts stay inflated until a full resync enumerates the existing keys and rewrites them.

Background Maintenance Worker

For production use, the Bm25MaintenanceWorker can be configured to automatically sync indexes when source ledgers change:

• Watches for commit events on source ledgers
• Debounces rapid commits (configurable interval)
• Bounded concurrency for concurrent sync operations, and at most one sync in flight per index
• Registers/unregisters graph sources dynamically

Budget for it before turning it on over a large corpus. Each sync re-runs the indexing query (see Syncing). Where that query can be narrowed to the affected subjects the indexed portion no longer scales with the corpus, which is what makes a write-heavy source ledger affordable per debounce window. Where it cannot — any of the declined shapes listed above — each sync is still a full indexing-query execution, and debouncing bounds how often that happens rather than what each one costs. Check the debug log for the decline reason if syncs stay expensive; at a few million documents on a query that cannot be narrowed, explicit syncs from a scheduled maintenance job keep that work on a cadence you choose.

Staleness Checking

Check whether an index is behind its source ledger:

#![allow(unused)]
fn main() {
let check = fluree.check_bm25_staleness("article-search:main").await?;
println!("Index at t={}, ledger at t={}, stale: {}, lag: {}",
    check.index_t, check.ledger_t, check.is_stale, check.lag);
}

Time-Travel

Load an index at a specific historical transaction time:

#![allow(unused)]
fn main() {
// Load index as of transaction t=5
let (index, actual_t) = fluree.load_bm25_index_at("article-search:main", 5).await?;
println!("Loaded snapshot at t={}, docs: {}", actual_t, index.num_docs());
}

BM25 maintains a manifest of historical snapshots. The manifest is stored in content-addressed storage and tracks all snapshot versions. load_bm25_index_at selects the snapshot with the largest index_t <= as_of_t.

Dropping an Index

#![allow(unused)]
fn main() {
let drop_result = fluree.drop_full_text_index("article-search:main").await?;
println!("Deleted {} snapshots", drop_result.deleted_snapshots);

// Drop is idempotent
let drop_again = fluree.drop_full_text_index("article-search:main").await?;
assert!(drop_again.was_already_retracted);
}

Dropping marks the graph source as retracted in the nameservice and deletes all snapshot blobs from storage. The index can be recreated with the same name afterward.

Scoring and Top-K Optimization

For top-k queries (the typical case via f:searchLimit), BM25 uses Block-Max WAND (Weak AND) to avoid scoring every matching document. Posting lists are divided into fixed-size blocks (128 postings each) with per-block metadata (maximum term frequency). WAND uses these to compute score upper bounds, skipping entire blocks that cannot contribute to the current top-k results.

This makes top_k(10) on a 100K-document index significantly faster than scoring all matches — the algorithm terminates early once it can prove no remaining document can displace the current top results.

When block metadata is unavailable (e.g., during index building before the first snapshot), scoring falls back to dense accumulation over all postings.

Storage Format

V4 Chunked Format

Large BM25 indexes use a chunked storage format (v4) that splits the index into:

• Root blob: Terms dictionary, document metadata, BM25 statistics, routing table
• Posting leaflet blobs: Compressed posting lists (~2MB each), stored as separate content-addressed objects. Each posting list includes block metadata (128 postings per block with max_doc_id and max_tf) used for WAND score upper bounds and block-level navigation.

This enables selective loading: queries only fetch the leaflets containing terms that match the search query, rather than loading the entire index.

Leaflet Caching

Posting leaflets are cached in the global LeafletCache (shared with core index leaflets). Cache entries are keyed by content ID hash and are immutable (content-addressed data never changes). The cache uses moka’s TinyLFU eviction and is governed by the global cache budget (--cache-max-mb / FLUREE_CACHE_MAX_MB, default tier: <4GB 30%, 4-8GB 40%, >=8GB 35%).

Parallel I/O

Both reads and writes use bounded-concurrency parallel I/O (buffer_unordered(32)) for leaflet operations. This caps socket pressure when working with object stores like S3 while still providing significant throughput improvement over sequential access.

Format Selection

The storage format is selected automatically based on the storage backend:

• File storage: V3 single-blob format (optimized for local filesystem)
• Memory / S3 / object store: V4 chunked format (enables selective loading and caching)

CLI Management

The fluree bm25 command group manages indexes without writing Rust. Each command runs against a server when one is reachable — --remote <name> for a configured remote, or a locally-running server automatically — and in-process against local storage otherwise. Pass --direct to force the in-process route, which works under docker exec against a running server’s data directory (native storage uses per-file locks, so this is safe alongside a running server). See fluree bm25 for the full command reference.

A maintenance loop syncs every stale index:

for i in $(fluree bm25 list --stale); do fluree bm25 sync --index "$i"; done

Deployment Modes

Embedded Mode (Default)

In embedded mode, the BM25 index is loaded and searched within the same process as Fluree. This is the default behavior, and it is what serves an f:searchText query over the main server’s POST /v1/fluree/query route — the query execution context wires a local index provider, so no separate search service is required.

Remote Mode

In remote mode, search queries are delegated to a dedicated search service (fluree-search-httpd):

fluree-search-httpd \
  --storage-root file:///var/fluree/data \
  --nameservice-path file:///var/fluree/ns \
  --listen 0.0.0.0:9090

Both modes use identical analyzer configuration, BM25 scoring algorithm, and time-travel semantics — queries return identical results regardless of deployment mode.

See BM25 Graph Source for details on the remote search protocol.

Related Documentation

• BM25 Graph Source - Graph source integration and remote search protocol
• Background Indexing - Core index architecture
• Vector Search - Similarity search
• Graph Sources Overview - Graph source concepts