fluree bm25

Manage BM25 full-text search indexes.

A BM25 index is a Fluree graph source built over a source ledger: an indexing query selects the documents and text properties to index, and the resulting index is queryable through an FQL f:searchText clause or the standalone fluree-search-httpd service.

Choosing where a command runs

Every subcommand takes one of two routes:

• Against a server. --remote <name> drives a configured remote; with no flag, a locally-running server (started via fluree server start) is used automatically. This is the route to take against a deployment you don’t share a filesystem with. It uses the graph-source endpoints — POST /v1/fluree/bm25/create, POST /v1/fluree/bm25/sync, the shared POST /v1/fluree/drop, and GET /v1/fluree/ledgers.
• In-process against local storage. --direct forces this, and it is also what happens when no server is reachable. Because native file storage coordinates writers with a per-file advisory lock (not an exclusive whole-store lock), this works under docker exec against a directory a running server is already serving.

--remote is per-subcommand; --direct is global, so it goes anywhere on the command line.

The automatic local-server route sends no credentials. --remote carries the token stored with that remote, but a server detected locally is called unauthenticated — so if it was started with an admin_auth mode, create, sync, and drop come back 401. Use --direct there, or configure the server as a remote and drive it with --remote.

Server routing is new. Earlier versions ran every one of these subcommands in-process, so a script or cron job that called fluree bm25 sync under docker exec against a running server’s data directory now reaches that server over HTTP instead. It says so on stderr — server: routing through local server at <addr> — and the outcome is unchanged on a server with no admin_auth configured. On one that has it, the call now fails; add --direct to pin the previous behavior.

One caveat for list over a server: staleness is computed by pairing each index against its source ledger, which needs the dependencies field on the /ledgers response. A server predating that field leaves the source unknown, and list prints a warning naming the affected indexes rather than silently reporting them as current.

Subcommands

fluree bm25 create

Build a BM25 full-text search index over a ledger.

Usage

fluree bm25 create --name <NAME> --ledger <LEDGER> [-e <QUERY> | -f <FILE>] [OPTIONS]

Options

--k1 and --b are omitted from the request when unset, so a server-side create applies the same defaults the local path would.

The indexing query selects the documents and the text properties to index. It must select @id.

Example

fluree bm25 create --name silver-search --ledger silver:main -f index-query.json

With the indexing query inline:

fluree bm25 create --name docsearch --ledger docs:main -e '{
  "@context": {"ex": "http://example.org/"},
  "where": [{"@id": "?x", "@type": "ex:Doc", "ex:title": "?t"}],
  "select": {"?x": ["@id", "ex:title"]}
}'

Output:

Created full-text index docsearch:main (docs=2, terms=37, index_t=3).

fluree bm25 list

List BM25 indexes with their source ledger and staleness.

Usage

fluree bm25 list [--stale]

Options

For each index the table shows its name and branch, the source ledger it covers, the index watermark (INDEX_T), the source ledger’s current t (LEDGER_T), and whether the index is STALE — meaning the source has advanced past the index. Unlike fluree list, this shows the source ledger and staleness, which is what a maintenance job needs to decide what to sync.

Example

$ fluree bm25 list
+------------+--------+---------------+---------+----------+-------+
| NAME       | BRANCH | SOURCE LEDGER | INDEX_T | LEDGER_T | STALE |
+------------+--------+---------------+---------+----------+-------+
| docsearch  | main   | docs:main     | 3       | 5        | YES   |
+------------+--------+---------------+---------+----------+-------+

--stale emits just the aliases, so a maintenance loop can feed them straight to sync:

for i in $(fluree bm25 list --stale); do fluree bm25 sync --index "$i"; done

fluree bm25 sync

Bring an index up to date with its source ledger.

Incremental when possible — only re-indexing the subjects changed since the index’s stored watermark — falling back to a full rebuild when it must. A no-op when the index is already current.

By default nothing syncs on commit, so run this from a maintenance job to keep search fresh. Alternatively, start the server with --bm25-auto-sync (env FLUREE_BM25_AUTO_SYNC, or indexing.bm25_auto_sync in the config file) and it will sync each index whenever its source ledger commits — off by default because it adds background storage writes. Under Raft that runs on the leader only, and it does not run on peer-mode servers.

Usage

fluree bm25 sync --index <INDEX>

Options

Example

$ fluree bm25 sync --index docsearch:main
Synced full-text index docsearch:main (1 upserted, 0 removed, 1 subject; watermark 3 -> 5).

An already-current index reports no work:

Full-text index docsearch:main already up to date (watermark 5).

--t pins how far to sync, which is useful when you want the index to match a known point in the source’s history rather than chasing its head:

fluree bm25 sync --index docsearch:main --t 4

The index is synced through that t. Asking for a t beyond the source ledger’s head is refused.

fluree bm25 drop

Retract an index and delete its snapshot blobs.

Usage

fluree bm25 drop --index <INDEX> --force

Options

Drop is destructive — it retracts the nameservice record and deletes the snapshot blobs — so --force is required, as with fluree drop and fluree iceberg drop. Rebuilding an index over a large corpus is expensive enough to be worth the confirmation.

Example

$ fluree bm25 drop --index docsearch:main --force
Dropped full-text index docsearch:main (deleted 2 snapshots).

Driving a remote server

Point any subcommand at a configured remote with --remote:

fluree bm25 create --name docsearch --ledger docs:main -f index-query.json --remote origin
fluree bm25 list --remote origin
fluree bm25 sync --index docsearch:main --remote origin
fluree bm25 drop --index docsearch:main --force --remote origin

A maintenance loop against a remote reads the same as a local one:

for i in $(fluree bm25 list --stale --remote origin); do
  fluree bm25 sync --index "$i" --remote origin
done

Requests are admin-authenticated the same way as the rest of the CLI’s remote traffic — see remote for configuring one and auth for logging in.

See also

• BM25 Full-Text Search — indexing queries, tuning, and querying an index
• BM25 graph source — deployment topologies, including fluree-search-httpd
• Graph Source Endpoints — the HTTP equivalents of create and sync
• fluree iceberg — the adjacent graph-source family