fluree doc

Turn a folder of documents into a searchable graph, and search it.

fluree doc ingest reads PDF, Markdown, HTML, DOCX, PPTX and image files and writes, per document, one commit holding:

• a DoCO structure graph from the fluree-doc-parse engine: sections, paragraphs, lists, captions and table cells, each with character offsets into the text projection and, for PDFs, page index and bounding box;
• retrieval chunks cut along that structure, each citing the elements it was built from and carrying its section path;
• an embedding per chunk, when an embedding endpoint is configured;
• a document node recording the file, its content hash, the parser revision and the embedding model;
• with --entities and --model, mentions of known entities under their own IRIs, relations the text states (reified with evidence and verdict, and as edges when the ontology admits them), and any new entities the language model found.

It then creates or syncs a BM25 full-text index over the chunks. fluree doc search joins each hit back to its chunk text, section path and source file.

There is no vector index: --mode vector scores every chunk’s doc:embedding exactly with cosineSimilarity, so no ANN library is linked into the CLI and nothing has to be built, synced or rebuilt. An approximate HNSW index for corpora past the point where a scan is cheap is a fluree server capability.

Everything runs in-process against local storage: the ledger written is always a local one, and Fluree AI, when connected, supplies models only. Parsing is deterministic and makes no network connection unless a model endpoint is configured.

This page is the CLI command reference. For the walkthrough — the tiers, connecting a Fluree AI account, what gets built, which model calls are made, and publishing the result — see the Fluree Unstructured section of the docs.

Subcommands

fluree doc ingest

Usage

fluree doc ingest <PATH>... [-l <LEDGER>] [OPTIONS]

Arguments

Options

What a run does

For each document, in path order:

1. Skip if unchanged. The ledger’s document node records the file’s SHA-256, the parser revision, the embedding model and the extraction fingerprint (ontology, entity sources, model, guidance, relation mode). When all match, the document is reported as unchanged and skipped. --force overrides this.
2. Parse, served from the parse cache when the same bytes were parsed with the same settings before. PDF structure is inferred from glyph geometry; the other formats declare their structure. Pages the deterministic pass cannot read (scans, pixel-only regions, tables whose structure it doubts) are sent to the configured vision model as crops, and the readings are spliced back in under the engine’s arbitration rules.
3. Chunk along the structure graph. A chunk closes once it reaches --min-chars, or at a heading once it is at least half full. Table cells carry their row and column headers.
4. Embed each chunk’s text, prefixed with its section path, against [doc.embedding].
5. Extract, when --entities or --model is given. Each chunk is scanned for every label of every known entity (longest whole-word match, case-folded, plus Snowball stems), and each match becomes a mention under the entity’s IRI. With --model, the language model is then asked about the chunk with the ontology as its system prompt and the entities found as known names. Its entities are resolved to known IRIs where a label matches and minted otherwise; an entity whose excerpt is not in the chunk is dropped; a new entity typed outside the ontology is kept flagged doc:offModel (or dropped with --drop-off-model); each relation’s predicate is judged against the ontology as valid, repaired (an unambiguous label, local name or class-to-property fix) or rejected, and written reified with that verdict, a relation whose object names no entity carrying it as a literal. In direct mode an admitted relation with an entity object is also written as an edge. Answers are cached per chunk and asked for several chunks at once; a chunk whose call fails keeps its gazetteer mentions, and the document is not stamped as extracted so the next run retries it.
6. Retract the previous extraction of the same document IRI, if any, then insert the structure graph, the chunks, the document node and the extraction as one commit. The earlier extraction remains queryable at its commit. Entity nodes are shared across documents and not retracted; an edge the earlier extraction asserted is dropped only when no remaining relation supports it.

After the documents, the ledger’s binary index is brought up to the new head, so later invocations read it rather than replaying the commits just written (a CLI process never runs the background indexer that a server would). Then the full-text index <ledger>-text is created or synced. Embeddings need no index step — they are committed with the chunks and scored at search time.

Examples

# Deterministic, offline: structure graph + chunks + full-text index
fluree doc ingest ./contracts -l contracts

# Configure an embedding endpoint once, then ingest with vectors
fluree config set doc.embedding.url http://localhost:11434/v1
fluree config set doc.embedding.model nomic-embed-text
fluree doc ingest ./contracts -l contracts

# See what a folder would produce without writing anything
fluree doc ingest ./drafts -l contracts --dry-run --out-dir ./inspect

# Find the counterparties you already track, under their own IRIs; no model needed
fluree doc ingest ./contracts -l contracts --entities counterparties.ttl

# Add an ontology and a language model: new entities and relations, gated by the ontology
fluree config set doc.llm.url http://localhost:11434/v1
fluree config set doc.llm.model qwen3
fluree doc ingest ./contracts -l contracts --entities counterparties.ttl --model ./contracts-ontology.ttl

Output:

ingest 3 document(s) → contracts
  parser     fluree-doc-parse 8765b40
  escalation none — deterministic tier only; set [doc.vlm] to read scanned pages
  embedding  nomic-embed-text
→ created ledger contracts
  ✓ msa-2024.pdf  parsed: 41p, 512 elements, 138 chunks, embedded  t=1
  ✓ nda.docx  parsed: 19 elements, 4 chunks, embedded  t=2
  = sow/q3.pdf  unchanged
  ⟳ ledger index contracts: t=2
  + full-text index contracts-text:main: 142 chunk(s), 1631 terms

done: 2 ingested, 1 unchanged, 0 failed — 142 chunks, 41 pages, 0 crop(s) read, 0 parse(s) from cache, 9.4s

fluree doc search

Usage

fluree doc search <QUERY> [-l <LEDGER>] [-n <N>] [--mode auto|hybrid|vector|text] [--json]

Options

Example

fluree doc search "termination notice period" -l contracts -n 3

 1. 0.812  msa-2024.pdf  Master Services Agreement / 12. Term and Termination
    Either party may terminate this Agreement for convenience on ninety (90) days' written notice…
    urn:fluree:doc:msa-2024.pdf/chunk/97
 2. 0.774  nda.docx  Term
    This Agreement terminates two years after the Effective Date unless…
    urn:fluree:doc:nda.docx/chunk/2
(2 result(s), vector, 41 ms)

A fused hit shows each method’s own score and rank, which is also the diagnostic for when one method fails: a query that names the answer’s terms but not its meaning can rank tenth by vector and first by text, and lands first fused. The scales differ, a cosine of 0.9 being very high while BM25 reaches 35 and 3 is weak, which is why fusion calibrates before it averages.

 1. 0.527  fin_fed_stmts.pdf  … / l. Federal Reserve Notes  vector 0.53 #10 · text 11.1 #1

The chunk IRI is a real node: follow doc:sourceElement to the paragraphs and cells it was built from, and from those nif:beginIndex, doc:pageIndex and doc:bbox locate the passage on the page.

Configuration

Model endpoints live in the [doc] table of the project’s config.toml, set with fluree config set:

[doc.embedding]
url = "http://localhost:11434/v1"      # any OpenAI-compatible base URL
model = "nomic-embed-text"
# api_key = "$OPENAI_API_KEY"           # a `$NAME` value reads that environment variable
# dimensions = 768                      # for models that accept a `dimensions` parameter

[doc.vlm]                               # reads document crops; falls back to [doc.llm]
url = "https://api.openai.com/v1"
model = "gpt-5-mini"
api_key = "$OPENAI_API_KEY"
# api = "chat"                          # `chat` (default) or `responses`

[doc.llm]                               # entity and relation extraction (--model)
url = "https://api.openai.com/v1"
model = "gpt-5-mini"
api_key = "$OPENAI_API_KEY"

[doc.extraction]                        # how extraction is asked, project-wide
guidance = "prompts/guidance.md"        # priorities placed in the prompt
# system_prompt = "prompts/system.txt"  # replaces the system prompt; keep {model} and {guidance}
# user_prompt = "prompts/user.txt"      # replaces the user prompt; keep {existing} and {document}
# concurrency = 4
# drop_off_model = false

All three slots speak the OpenAI wire shape, so OpenAI, Ollama, vLLM and LM Studio are configured the same way. Each slot can be overridden by environment variables FLUREE_DOC_{EMBEDDING,LLM,VLM}_{URL,MODEL,API_KEY,DIMENSIONS,API}.

Using a Fluree AI account

A Fluree AI stack serves the same routes and holds the model keys, so with an account nothing else needs configuring locally. Register the stack as a remote, log in once, and name it:

fluree remote add acct https://<your-stack>/v1/fluree     # OIDC is auto-discovered
fluree auth login --remote acct                          # browser / device login
fluree config set doc.remote acct

doc.remote fills every slot you have not set explicitly: embeddings through the gateway’s /v1/embeddings (default model text-embedding-3-small), and crop reading and extraction through /v1/responses with model = "auto", which lets the gateway route each intent to the account’s vision or language provider. The remote is looked up in the project config first and the global config second, so one login from your home directory serves every project. An expired login is refreshed before the run.

Any slot set explicitly wins, so a local embedding model can be combined with the account’s vision model:

[doc]
remote = "acct"

[doc.embedding]
url = "http://localhost:11434/v1"
model = "nomic-embed-text"

Without doc.remote, the same wiring can be spelled out by hand with an flr_… API key from Settings → API keys, using api = "responses" and model = "auto" on the vlm and llm slots.

The gateway’s embeddings route forwards to the account’s OpenAI-type provider, so vector search through the stack needs one configured there.

With no slot configured the command never reaches the network: it parses deterministically, writes structure and chunks, and builds the full-text index. Raster images are the one input that cannot be read without a vision model.

Caches

Three caches under .fluree/cache/doc/ make re-runs cheap:

• Parse cache — keyed on the file’s content hash plus the parser revision, the document IRI and the escalation model. A re-run over an unchanged folder parses nothing.
• Reading cache — keyed on the crop’s pixels, the prompt and the model. A parser upgrade re-routes pages, but a crop whose pixels did not change is answered without a model call.
• Extraction cache — keyed on the model, the ontology and guidance, the known entities in the chunk, and the chunk’s text.

--no-cache bypasses all of them. Deleting the directory is always safe.

The graph

Structure elements use the DoCO, NIF and pattern ontologies under doc: (https://ns.flur.ee/doc#) as documented by fluree-doc-parse. Chunks and document nodes extend the same namespace:

Document IRIs are <base-iri><relative path>, elements <document>/element/<n>, chunks <document>/chunk/<n>, mentions <chunk>/mention/<n>, relations <document>/relation/<n>, minted entities <base-iri>entity/<hash of the name>.

The full walkthrough, including a cross-ledger query over the entities graph and the documents graph, is in Entities and relations.