---

# README.md

# Fluree DB

A semantic graph database with time travel, branching, and verifiable data — built on W3C standards.

Fluree DB is a single binary that stores your data as an RDF knowledge graph, queryable with SPARQL or JSON-LD Query, with every commit immutably recorded so you can travel back to any prior state. It supports git-style branching and merging, signed and policy-gated transactions, SHACL validation, OWL/RDFS reasoning, and full-text and vector search — over local files, S3, or IPFS — without bolting on external services.

## What you get

- **Semantic by default.** Your data is RDF. IRIs, JSON-LD `@context`, named graphs, and typed values are first-class. Queries are SPARQL 1.1 or the equivalent JSON-LD Query, both compiling to the same execution engine.
- **Property graphs and statement-level metadata.** Attach properties to a *relationship*, not just a node — `role` and `since` on a `worksFor` edge, `source` and `confidence` on a claim — with RDF 1.2 / SPARQL 1.2 edge annotations. Labeled-property-graph edges, parallel relationships between the same two nodes, and RDF-star provenance all live on one surface, and plain triple queries are left undisturbed. See [Edge annotations](concepts/edge-annotations.md).
- **Time travel.** Every transaction is a commit on an immutable chain. Query the state of the graph at any past moment with a single `t` parameter — no snapshots to restore, no separate audit log to consult.
- **Branching and merging.** Create a branch off any commit, transact against it in isolation, then merge it back. Useful for staging changes, running what-if analyses, or maintaining environment-specific overlays.
- **Verifiable data.** Transactions and commits can be signed (JWS / W3C Verifiable Credentials). The commit chain is content-addressed, so any tampering is detectable. Pair it with policy enforcement to prove *who* changed *what* and *when* they were allowed to.
- **Policy-based access control.** Policies are written as graph data, evaluated per query and per transaction, and travel with the ledger — not bolted on at the API layer.
- **Storage your way.** Local filesystem for development, S3 + DynamoDB for production, IPFS for content-addressed distribution. The same ledger format works across all of them.
- **Search built in.** BM25 full-text indexing and HNSW vector search live alongside SPARQL — no separate search service to operate.
- **Reasoning.** OWL/RDFS inference and Datalog rules run inside the query engine, so derived facts are queryable without a materialization step.
- **Embeddable.** The same engine that powers the server runs as a Rust library, generic over storage and nameservice. Use it directly in your application or run it standalone over HTTP.

## Start here

- **New to Fluree?** → [Getting started](getting-started/)
- **Run the server** → [Quickstart: run the server](getting-started/quickstart-server.md)
- **Create a ledger and write data** → [Quickstart: create a ledger](getting-started/quickstart-ledger.md) → [Quickstart: write data](getting-started/quickstart-write.md)
- **Query data** → [Quickstart: query (JSON-LD + SPARQL)](getting-started/quickstart-query.md)
- **End-to-end walkthrough** → [Tutorial: search, time travel, branching, policies](getting-started/tutorial-end-to-end.md)
- **Coming from SQL?** → [Fluree for SQL developers](getting-started/fluree-for-sql-developers.md)
- **Embedding in Rust?** → [Using Fluree as a Rust library](getting-started/rust-api.md)

## Explore the docs

- [Concepts](concepts/) — ledgers, graph sources, IRIs, datatypes, edge annotations, time travel, policy, verifiable data, reasoning
- [Guides (cookbooks)](guides/) — query patterns, search, time travel, branching, policies, SHACL, edge annotations — task-oriented recipes
- [CLI reference](cli/) — every `fluree` command, flag by flag
- [HTTP API](api/) — endpoints, headers, signed requests, error model
- [Query](query/) — JSON-LD Query, SPARQL, output formats, CONSTRUCT, explain plans, reasoning
- [Transactions](transactions/) — insert, upsert, update, conditional updates, signed transactions
- [Security and policy](security/) — authentication, encryption, commit signing, policy model
- [Indexing and search](indexing-and-search/) — background indexing, BM25, vector search, geospatial
- [Graph sources and integrations](graph-sources/) — Iceberg/Parquet, R2RML, BM25 graph source
- [Operations](operations/) — configuration, Docker, storage modes, telemetry, archive/restore
- [Design](design/) — internals: query execution, storage traits, index format, nameservice
- [Reference](reference/) — glossary, vocabulary, OWL/RDFS support, crate map
- [Troubleshooting](troubleshooting/) — common errors, debugging queries, performance tracing
- [Contributing](contributing/) — dev setup, tests, SPARQL compliance, releasing

The full table of contents is in the sidebar.

## Building with AI and agents

[Fluree for AI and agents](ai/) features for the AI use case — it ties together Agent JSON (token-efficient query output), the MCP server (Fluree as an agent tool), Fluree Memory, vector and full-text search as a RAG substrate, and reasoning.

## Fluree Memory

[Fluree Memory](memory/) is persistent, searchable memory for AI coding assistants — built on Fluree DB and shipped in the same `fluree` binary. If you're here for the memory tooling, jump straight to the [Memory docs](memory/).


---

# cli/README.md

# Fluree CLI

The `fluree` command-line interface provides a convenient way to manage ledgers, run queries, and perform transactions without running a server.

## Installation

Build from source:

```bash
cargo build --release -p fluree-db-cli
```

The binary will be at `target/release/fluree`.

## Quick Start

```bash
# Initialize a project directory
fluree init

# Create a ledger
fluree create myledger

# Insert data
fluree insert '@prefix ex: <http://example.org/> .
ex:alice a ex:Person ; ex:name "Alice" .'

# Query
fluree query 'SELECT ?name WHERE { ?s <http://example.org/name> ?name }'
```

## Global Options

| Option | Description |
|--------|-------------|
| `-v, --verbose` | Enable verbose output |
| `-q, --quiet` | Suppress non-essential output |
| `--no-color` | Disable colored output (also respects `NO_COLOR` env var) |
| `--config <PATH>` | Path to config file |
| `--memory-budget-mb <MB>` | Memory budget in MB for bulk import (0 = auto: 60% of system RAM). Affects chunk size, concurrency, and run budget when creating a ledger with `--from`. Set this to cap memory use; auto-detected thread count shrinks to fit it. |
| `--parallelism <N>` | Number of parallel parse threads for bulk import (0 = auto: most logical cores, capped to fit the memory budget; explicit values honored as-is, floored at 1). Used when creating a ledger with `--from`. |
| `-h, --help` | Print help |
| `-V, --version` | Print version |

## Commands

### Core Commands

| Command | Description |
|---------|-------------|
| [`init`](init.md) | Initialize a new Fluree project directory |
| [`create`](create.md) | Create a new ledger |
| [`use`](use.md) | Set the active ledger |
| [`list`](list.md) | List all ledgers |
| [`info`](info.md) | Show detailed information about a ledger |
| [`drop`](drop.md) | Drop (delete) a ledger |
| [`graph`](graph.md) | Manage named graphs within a ledger (list, drop) |
| [`insert`](insert.md) | Insert data into a ledger |
| [`upsert`](upsert.md) | Upsert data (insert or update existing) |
| [`update`](update.md) | Update with WHERE/DELETE/INSERT patterns |
| [`query`](query.md) | Query a ledger |
| [`history`](history.md) | Show change history for an entity |
| [`export`](export.md) | Export ledger data |
| [`log`](log.md) | Show commit log |
| [`show`](show.md) | Show decoded commit contents (flakes with resolved IRIs) |
| [`index`](index.md) | Build or update the binary index (incremental) |
| [`reindex`](reindex.md) | Full reindex from commit history |

### Remote Sync

| Command | Description |
|---------|-------------|
| [`remote`](remote.md) | Manage remote servers |
| [`upstream`](upstream.md) | Manage upstream tracking configuration |
| [`fetch`](fetch.md) | Fetch refs from a remote |
| [`clone`](clone.md) | Clone a ledger from a remote (full commit download) |
| [`pull`](pull.md) | Pull commits from upstream |
| [`push`](push.md) | Push to upstream remote |
| [`track`](track.md) | Track remote-only ledgers (no local data) |

**Clone and pull** transfer commits and, by default, **binary index data** from the remote (pack protocol), so the local ledger is query-ready without a separate reindex. Use `--no-indexes` to skip index transfer and reduce download size; run `fluree reindex` afterward if you need the index. Large transfers may prompt for confirmation before streaming.

### Server Management

| Command | Description |
|---------|-------------|
| [`server`](server.md) | Manage the Fluree HTTP server (run, start, stop, status, restart, logs) |

Start a server directly from a project directory — it inherits the same `.fluree/` context (config, storage) as the CLI. See [`server`](server.md) for details.

### Implementers

If you're building a custom server that must support the CLI end-to-end (for example, integrating into another app), see:

- [`server-integration`](server-integration.md) - endpoints and auth contract required by the CLI

### Authentication

| Command | Description |
|---------|-------------|
| [`token`](token.md) | Create, inspect, and manage JWS tokens |
| [`auth`](auth.md) | Manage bearer tokens stored on remotes (login/logout/status) |

### Configuration

| Command | Description |
|---------|-------------|
| [`config`](config.md) | Manage configuration |
| [`prefix`](prefix.md) | Manage IRI prefix mappings |
| [`completions`](completions.md) | Generate shell completions |

### Developer Memory

| Command | Description |
|---------|-------------|
| [`memory`](memory.md) | Store and recall facts, decisions, constraints, preferences, and artifact references |
| [`mcp`](mcp.md) | MCP server for IDE agent integration |

For background, IDE setup, team workflows, and the `mem:` schema, see the [Memory section](../memory/README.md) of the docs.

## Project Structure

When you run `fluree init`, a `.fluree/` directory is created with:

```
.fluree/
├── active          # Currently active ledger name
├── config.toml     # Configuration settings
├── prefixes.json   # IRI prefix mappings
└── storage/        # Ledger data storage
```

## Input Resolution

Commands that accept data input (`insert`, `upsert`, `update`, `query`) use flexible argument resolution:

| Arguments | Behavior |
|-----------|----------|
| (none) | Active ledger; provide input via `-e`, `-f`, or stdin |
| `<arg>` | Auto-detected: if it looks like a query/data, uses it inline; if it's an existing file, reads from it; otherwise treats it as a ledger name |
| `<ledger> <input>` | Specified ledger + inline input |

Input is resolved in this priority order: `-e` flag > positional inline > `-f` flag > positional file > stdin.

## Data Format Detection

The CLI auto-detects data format based on content:
- Lines starting with `@prefix` or `@base` → Turtle
- Content starting with `{` or `[` → JSON-LD
- Files with `.ttl` extension → Turtle
- Files with `.json` or `.jsonld` extension → JSON-LD

You can override with `--format turtle` or `--format jsonld`.


---

# cli/init.md

# fluree init

Initialize a new Fluree project directory.

## Usage

```bash
fluree init [OPTIONS]
```

## Options

| Option     | Description                                                                       |
| ---------- | --------------------------------------------------------------------------------- |
| `--global` | Create global config and data directories instead of a local `.fluree/` directory |

## Description

Creates a `.fluree/` directory in the current working directory (or global directories with `--global`). This directory stores:

- `active` - The currently active ledger name
- `config.toml` - Configuration settings
- `prefixes.json` - IRI prefix mappings for compact IRIs
- `storage/` - Ledger data

Running `init` is idempotent - it won't overwrite existing configuration.

## Examples

```bash
# Initialize in current directory
fluree init

# Initialize global config
fluree init --global
```

## Global Directory

With `--global`, the directories are determined by:

1. `$FLUREE_HOME` environment variable (if set) — both config and data go in this single directory.
2. Platform directories (when `$FLUREE_HOME` is not set):

| Content                                      | Linux                                                      | macOS                                  | Windows                 |
| -------------------------------------------- | ---------------------------------------------------------- | -------------------------------------- | ----------------------- |
| Config (`config.toml`)                       | `$XDG_CONFIG_HOME/fluree` (default: `~/.config/fluree`)    | `~/Library/Application Support/fluree` | `%LOCALAPPDATA%\fluree` |
| Data (`storage/`, `active`, `prefixes.json`) | `$XDG_DATA_HOME/fluree` (default: `~/.local/share/fluree`) | `~/Library/Application Support/fluree` | `%LOCALAPPDATA%\fluree` |

On macOS and Windows both resolve to the same directory (unified); on Linux config and data are separated per XDG conventions.

The generated `config.toml` will contain an absolute `storage_path` pointing to the data directory when the directories are split.

## See Also

- [create](create.md) - Create a new ledger after initialization


---

# cli/create.md

# fluree create

Create a new ledger.

## Usage

```bash
fluree create <LEDGER> [OPTIONS]
```

## Arguments

| Argument | Description |
|----------|-------------|
| `<LEDGER>` | Name for the new ledger |

## Options

| Option | Description |
|--------|-------------|
| `--from <PATH>` | Import data from a file (Turtle, N-Triples, N-Quads, TriG, or JSON-LD), optionally `.gz`- or `.zst`-compressed. N-Triples (`.nt`) parses as Turtle; N-Quads (`.nq`) converts to TriG (named graphs supported). A `.flpack` archive (see [export](export.md)) is restored wholesale instead — full ledger including its prebuilt index. |
| `--remote <NAME>` | Create on a remote server instead of locally. With no `--from`, creates an empty ledger. With `--from <archive>.flpack`, streams the archive to the server's import endpoint to restore the ledger remotely. Other `--from` formats are not supported remotely — export to `.flpack` first, or create locally then [publish](publish.md). |
| `--memory [PATH]` | Import memory history from a git-tracked `.fluree-memory/` directory. Defaults to the current repo if no path is given. Mutually exclusive with `--from`. |
| `--no-user` | Exclude user-scoped memories (`.local/user.ttl`) from `--memory` import |
| `--chunk-size-mb <MB>` | Chunk size in MB for splitting large Turtle files (0 = derive from memory budget). Only used when `--from` points to a `.ttl` or `.nt` file. |
| `--leaflet-rows <N>` | Rows per leaflet in the binary index (default: 25000). Larger values produce fewer, bigger leaflets — less I/O per scan, more memory per read. |
| `--leaflets-per-leaf <N>` | Leaflets per leaf file (default: 10). Larger values produce fewer leaf files — shallower tree, bigger reads. |

**Global flags** that affect bulk import when using `--from` (see [CLI README](README.md#global-options)):

- `--memory-budget-mb <MB>` — Memory budget in MB (0 = auto: 60% of system RAM). Drives chunk size, concurrency, and indexer run budget. Set this to cap how much memory the import uses; auto-detected thread count shrinks to fit it.
- `--parallelism <N>` — Number of parallel parse threads (0 = auto: most logical cores, capped to fit the memory budget; explicit values honored as-is, floored at 1).

## Description

Creates a new empty ledger with the given name and sets it as the active ledger. The ledger is stored in `.fluree/storage/`.

Use `--from` to create a ledger pre-populated with data from a Turtle, N-Triples, N-Quads, TriG, or JSON-LD file (or a directory of same-format files). Any input may be gzip- or zstd-compressed and is decoded transparently (`data.ttl.gz`, `dump.nq.zst`, mixed directories — the underlying RDF extension classifies the file). N-Triples (`.nt`) is a strict subset of Turtle and is parsed by the same parser. N-Quads (`.nq`) and TriG (`.trig`) support named graphs — queryable after import via the `#<graph-iri>` fragment. For large Turtle/N-Triples files (including `.ttl.gz`/`.nt.gz`), the CLI splits work into chunks and runs parallel parse threads — though compressed inputs decode single-threaded; TriG/N-Quads/JSON-LD use a serial path. Tune with `--memory-budget-mb` and `--parallelism` if needed.

**Directory imports (`.ttl`/`.nt`)** are *rechunked by bytes* rather than one-chunk-per-file: large files are sub-split at statement boundaries and many small files are coalesced into `~chunk_size` work items. This keeps the import fully parallel and bounds the number of commits and sorted index runs regardless of how the data is packaged — so a directory of one big file, or of hundreds of tiny shards, both import at full speed. Coalescing engages automatically once a directory holds more than 64 sub-`chunk_size` files; a file containing a labeled blank node (`_:`) or an `@base` directive is never coalesced (it would change RDF document scope) and is imported as its own chunk. Set `FLUREE_IMPORT_COALESCE_THRESHOLD=<n>` to change the gate (`0` disables coalescing — every file becomes its own commit, the legacy behavior). Directories containing any `.trig`/`.nq`/`.jsonld` continue to use the per-file serial path.

Use `--memory` to import your project's developer memory history into a time-travel-capable Fluree ledger. Each git commit that touched `.fluree-memory/repo.ttl` (and `.local/user.ttl` unless `--no-user` is set) becomes a Fluree transaction. The git commit message, SHA, and author date are stored as transaction metadata, so you can correlate Fluree `t` values with git history.

### Restoring from a `.flpack` archive

When `--from` points at a `.flpack` file (produced by `fluree export <ledger> --format ledger`), the ledger is restored *wholesale* rather than bulk-imported: every commit, transaction blob, and prebuilt index artifact is streamed straight into storage and the heads are set from the archive — the restored ledger is byte-for-byte identical and immediately queryable, under whatever name you choose.

Add `--remote <name>` to restore onto a server instead of locally; the archive streams to the server's import endpoint, so no local staging instance is needed. This makes `.flpack` the universal way to move any data onto a server — build a ledger locally in any format, export it, then import it remotely. See [pack archive & restore](../operations/pack-archive-restore.md) for the full workflow.

If the remote is size-capped (e.g. an app behind a payload-limited gateway) and advertises it in discovery, the CLI automatically switches to a negotiated upload: it uploads the archive out-of-band to object storage and the server restores from it asynchronously. No extra flags — the path is chosen from the server's capabilities and the archive size.

## Examples

```bash
# Create an empty ledger
fluree create mydb

# Create with initial data
fluree create mydb --from seed-data.ttl

# Create from JSON-LD
fluree create mydb --from initial.jsonld

# Create with explicit memory and parallelism for a large Turtle file
fluree create mydb --from large.ttl --memory-budget-mb 4096 --parallelism 8

# Restore a .flpack archive into a new local ledger (any name)
fluree create restored-db --from mydb.flpack

# Restore a .flpack archive onto a remote server
fluree create restored-db --remote origin --from mydb.flpack

# Import memory history from the current repo
fluree create memories --memory

# Import memory history from another repo, excluding user memories
fluree create memories --memory /path/to/other/repo --no-user
```

## Output

```
Created ledger 'mydb'
Set 'mydb' as active ledger
```

With `--from`:
```
Created ledger 'mydb'
Committed t=1 (42 flakes)
Set 'mydb' as active ledger
```

With `--memory`:
```
Created ledger 'memories' with 42 commits (t=1..43)
  Earliest: bf803255 — initial memory store
  Latest:   9865e5cd — prevent overrides of fluree txn-meta

Query with time travel:
  fluree query memories 'SELECT ?id ?content WHERE { ?id a mem:Fact ; mem:content ?content } LIMIT 5'
  fluree query memories --at-t 2 'SELECT ...'   # state at first commit
```

## See Also

- [list](list.md) - List all ledgers
- [use](use.md) - Switch active ledger
- [drop](drop.md) - Delete a ledger


---

# cli/use.md

# fluree use

Set the active ledger.

## Usage

```bash
fluree use <LEDGER>
```

## Arguments

| Argument | Description |
|----------|-------------|
| `<LEDGER>` | Ledger name to set as active |

## Description

Sets the specified ledger as the active ledger. Subsequent commands that don't specify a ledger will use this one.

## Examples

```bash
# Switch to a different ledger
fluree use production

# Verify with info
fluree info
```

## Output

```
Active ledger set to 'production'
```

## Errors

If the ledger doesn't exist:
```
error: ledger 'nonexistent' not found
```

## See Also

- [list](list.md) - List all ledgers
- [create](create.md) - Create a new ledger


---

# cli/list.md

# fluree list

List all ledgers and graph sources.

## Usage

```bash
fluree list
```

## Description

Lists all ledgers and graph sources (Iceberg, R2RML, BM25, Vector, etc.) in the current Fluree directory. The active ledger is marked with an asterisk (`*`).

When graph sources are present, a TYPE column is shown to distinguish ledgers from graph sources.

## Examples

```bash
fluree list
```

## Output

When only ledgers exist:
```
   LEDGER      BRANCH  T
 * mydb        main    5
   production  main    12
```

When graph sources are also present:
```
   NAME              BRANCH  TYPE     T
 * mydb              main    Ledger   5
   production        main    Ledger   12
   warehouse-orders  main    Iceberg  -
   my-search         main    BM25     5
```

If nothing exists:
```
No ledgers found. Run 'fluree create <name>' to create one.
```

## See Also

- [create](create.md) - Create a new ledger
- [iceberg](iceberg.md) - Map Iceberg tables as graph sources
- [info](info.md) - Show detailed ledger or graph source information
- [use](use.md) - Switch active ledger


---

# cli/info.md

# fluree info

Show detailed information about a ledger or graph source.

## Usage

```bash
fluree info [NAME] [--remote <name>] [--graph <name|IRI>]
```

## Arguments

| Argument | Description |
|----------|-------------|
| `[NAME]` | Ledger or graph source name (defaults to active ledger) |

## Options

| Option | Description |
|--------|-------------|
| `--remote <name>` | Query a remote server (e.g., `origin`) instead of the local installation. |
| `--graph <name\|IRI>` | Scope the `stats` block to a single named graph within the ledger. Accepts a well-known name (`default`, `txn-meta`) or a graph IRI. Not applicable to graph sources. |

## Description

Displays detailed information about a ledger or graph source. The command first checks for a matching ledger; if none is found, it checks for a graph source with the same name.

For ledgers, displays:
- Ledger ID, branch, and type
- Current transaction number (t)
- Commit and index details

For graph sources (Iceberg, R2RML, BM25, etc.), displays:
- Name, branch, and type
- Graph source ID
- Index status
- Dependencies
- Configuration (catalog URI, table, mapping, etc.)

## Examples

```bash
# Info for active ledger
fluree info

# Info for specific ledger
fluree info production

# Info for a graph source
fluree info warehouse-orders

# Query a remote server
fluree info production --remote origin

# Scope stats to the default graph
fluree info mydb --graph default

# Scope stats to the transaction-metadata graph
fluree info mydb --graph txn-meta

# Scope stats to a specific named graph by IRI
fluree info mydb --graph https://example.org/graphs/inventory
```

When `--graph` is set, the command prints the full `ledger-info` JSON response
with the `stats` block scoped to the selected graph (properties, classes,
flakes, size).

## Output

Ledger:
```
Ledger:         mydb
Branch:         main
Type:           Ledger
Ledger ID:      mydb:main
Commit t:       5
Commit ID:      bafybeig...
Index t:        5
Index ID:       bafybeig...
```

Graph source (Iceberg):
```
Name:           warehouse-orders
Branch:         main
Type:           Iceberg
ID:             warehouse-orders:main
Retracted:      false
Index t:        0
Index ID:       (none)

Configuration:
{
  "catalog": {
    "type": "rest",
    "uri": "https://polaris.example.com/api/catalog"
  },
  "table": "sales.orders",
  ...
}
```

## See Also

- [list](list.md) - List all ledgers and graph sources
- [iceberg](iceberg.md) - Map Iceberg tables as graph sources
- [log](log.md) - Show commit history


---

# cli/branch.md

# fluree branch

Manage branches for a ledger.

## Subcommands

### fluree branch create

Create a new branch.

**Usage:**

```bash
fluree branch create <NAME> [OPTIONS]
```

**Arguments:**

| Argument | Description |
|----------|-------------|
| `<NAME>` | Name for the new branch (e.g., "dev", "feature-x") |

**Options:**

| Option | Description |
|--------|-------------|
| `--ledger <LEDGER>` | Ledger name (defaults to active ledger) |
| `--from <BRANCH>` | Source branch to create from (defaults to "main") |
| `--at <COMMIT-REF>` | Commit to branch at (defaults to source branch HEAD). Accepts `t:N` for a transaction number or a hex digest / full CID. |
| `--remote <REMOTE>` | Execute against a remote server |

**Description:**

Creates a new branch for a ledger. By default the branch starts at the source branch's current HEAD, and is fully isolated — subsequent transactions on either branch are invisible to the other.

Pass `--at` to branch from a historical commit on the source branch instead of its HEAD. The commit must be reachable from the source HEAD; the new branch starts with no index and replays from genesis on first query. `t:N` and hex-prefix resolution require the source branch to be indexed (full CIDs work unconditionally).

Branches can be nested: you can create a branch from any existing branch, not just "main".

**Examples:**

```bash
# Create a branch from main (default)
fluree branch create dev

# Create a branch for a specific ledger
fluree branch create dev --ledger mydb

# Create a branch from another branch
fluree branch create feature-x --from dev

# Branch at a historical point on main (transaction number)
fluree branch create rewind --at t:5

# Branch at a historical commit by hex-digest prefix
fluree branch create rewind --at 3dd028a7

# Create a branch on a remote server
fluree branch create staging --ledger mydb --remote origin
```

**Output:**

```
Created branch 'dev' from 'main' at t=5
Ledger ID: mydb:dev
```

### fluree branch list

List all branches for a ledger.

**Usage:**

```bash
fluree branch list [LEDGER] [OPTIONS]
```

**Arguments:**

| Argument | Description |
|----------|-------------|
| `[LEDGER]` | Ledger name (defaults to active ledger) |

**Options:**

| Option | Description |
|--------|-------------|
| `--remote <REMOTE>` | List branches on a remote server |

**Examples:**

```bash
# List branches for the active ledger
fluree branch list

# List branches for a specific ledger
fluree branch list mydb

# List branches on a remote server
fluree branch list mydb --remote origin
```

**Output:**

```
 BRANCH     T   SOURCE
 main       5   -
 dev        7   main
 feature-x  8   dev
```

### fluree branch drop

Drop a branch from a ledger.

**Usage:**

```bash
fluree branch drop <NAME> [OPTIONS]
```

**Arguments:**

| Argument | Description |
|----------|-------------|
| `<NAME>` | Branch name to drop (e.g., "dev", "feature-x") |

**Options:**

| Option | Description |
|--------|-------------|
| `--ledger <LEDGER>` | Ledger name (defaults to active ledger) |
| `--remote <REMOTE>` | Execute against a remote server |

**Description:**

Drops a branch from a ledger. The `main` branch cannot be dropped.

- **Leaf branches** (no children) are fully deleted — storage artifacts are removed and the NsRecord is purged.
- **Branches with children** are retracted (hidden from listings, reject new transactions) but storage is preserved so that child branches continue to work. When the last child is eventually dropped, the retracted parent is automatically cascade-purged.

**Examples:**

```bash
# Drop a branch
fluree branch drop dev

# Drop a branch for a specific ledger
fluree branch drop dev --ledger mydb

# Drop a branch on a remote server
fluree branch drop staging --ledger mydb --remote origin
```

**Output (leaf branch):**

```
Dropped branch 'dev'.
  Artifacts deleted: 5
```

**Output (branch with children):**

```
Branch 'dev' retracted (has children, storage preserved).
```

**Output (cascade):**

```
Dropped branch 'feature'.
  Artifacts deleted: 3
  Cascaded drops: mydb:dev
```

### fluree branch rebase

Rebase a branch onto its source branch's current HEAD.

**Usage:**

```bash
fluree branch rebase <NAME> [OPTIONS]
```

**Arguments:**

| Argument | Description |
|----------|-------------|
| `<NAME>` | Branch name to rebase (e.g., "dev", "feature-x") |

**Options:**

| Option | Description |
|--------|-------------|
| `--ledger <LEDGER>` | Ledger name (defaults to active ledger) |
| `--strategy <STRATEGY>` | Conflict resolution strategy (default: "take-both"). Options: `take-both`, `abort`, `take-source`, `take-branch`, `skip` |
| `--remote <REMOTE>` | Execute against a remote server |

**Description:**

Replays a branch's unique commits on top of the source branch's current HEAD. This brings the branch up to date with upstream changes. The `main` branch cannot be rebased.

If the branch has no unique commits, a fast-forward rebase is performed — the branch point is simply updated to the source's current HEAD.

Conflicts occur when both the branch and source have modified the same (subject, predicate, graph) tuples. See [conflict strategies](../concepts/ledgers-and-nameservice.md#rebasing-a-branch) for details.

**Examples:**

```bash
# Rebase with default strategy
fluree branch rebase dev

# Rebase with abort-on-conflict strategy
fluree branch rebase dev --strategy abort

# Rebase for a specific ledger
fluree branch rebase feature-x --ledger mydb --strategy take-source

# Rebase on a remote server
fluree branch rebase dev --ledger mydb --remote origin
```

**Output (fast-forward):**

```
Fast-forward rebase of 'dev' to t=5.
```

**Output (with replay):**

```
Rebased 'dev': 3 commits replayed, 0 skipped, 1 conflicts, 0 failures.
  New branch point: t=8
```

### fluree branch diff

Show a read-only merge preview between two branches.

**Usage:**

```bash
fluree branch diff <SOURCE> [OPTIONS]
```

**Arguments:**

| Argument | Description |
|----------|-------------|
| `<SOURCE>` | Source branch name to preview merging from (e.g., "dev", "feature-x") |

**Options:**

| Option | Description |
|--------|-------------|
| `--target <BRANCH>` | Target branch to preview merging into (defaults to source's parent branch) |
| `--max-commits <N>` | Cap on per-side commit summaries shown (default: 50; pass 0 for unbounded in local mode) |
| `--max-conflict-keys <N>` | Cap on conflict keys shown (default: 50; pass 0 for unbounded in local mode) |
| `--no-conflicts` | Skip conflict computation for a cheaper preview |
| `--conflict-details` | Include source/target flake values for returned conflict keys |
| `--strategy <STRATEGY>` | Strategy used for conflict detail labels (default: `take-both`). Options: `take-both`, `abort`, `take-source`, `take-branch` |
| `--json` | Emit the raw JSON preview |
| `--ledger <LEDGER>` | Ledger name (defaults to active ledger) |
| `--remote <REMOTE>` | Execute against a remote server |

**Description:**

`branch diff` reports ahead/behind commits, fast-forward eligibility, and conflicting `(subject, predicate, graph)` keys without mutating state. With `--conflict-details`, the preview also shows the source and target values for the returned conflict keys and annotates what the selected strategy would do.

**Examples:**

```bash
# Preview merging dev into its parent
fluree branch diff dev

# Preview a specific target
fluree branch diff dev --target main

# Show value details and source-winning labels
fluree branch diff dev --target main --conflict-details --strategy take-source

# Emit raw JSON for UI tooling
fluree branch diff dev --conflict-details --json
```

### fluree branch merge

Merge a source branch into a target branch.

**Usage:**

```bash
fluree branch merge <SOURCE> [OPTIONS]
```

**Arguments:**

| Argument | Description |
|----------|-------------|
| `<SOURCE>` | Source branch name to merge from (e.g., "dev", "feature-x") |

**Options:**

| Option | Description |
|--------|-------------|
| `--ledger <LEDGER>` | Ledger name (defaults to active ledger) |
| `--target <BRANCH>` | Target branch to merge into (defaults to source's parent branch) |
| `--strategy <STRATEGY>` | Conflict resolution strategy (default: `take-both`). Options: `take-both`, `abort`, `take-source`, `take-branch`. |
| `--remote <REMOTE>` | Execute against a remote server |

**Description:**

Merges a source branch into a target branch. When the target hasn't advanced since the source branched, this is a fast-forward; otherwise `--strategy` controls how conflicting edits are resolved (mirroring `branch rebase`).

When `--target` is omitted, the merge target is inferred from the source branch's parent (the branch it was created from).

After a successful merge, the source branch remains intact and can continue to receive new transactions and be merged again. Only the new commits since the last merge (or branch creation) are copied.

**Examples:**

```bash
# Merge dev into main (inferred from branch point)
fluree branch merge dev

# Merge feature-x into dev (explicit target)
fluree branch merge feature-x --target dev

# Merge for a specific ledger
fluree branch merge dev --ledger mydb

# Merge with source-winning conflict resolution
fluree branch merge dev --target main --strategy take-source

# Merge on a remote server
fluree branch merge dev --ledger mydb --remote origin
```

**Output:**

```
Merged 'dev' into 'main' (fast-forward to t=8, 3 commits copied).
```

**Output (non-fast-forward):**

```
Merged 'dev' into 'main' (t=9, 3 commits copied, 1 conflicts).
```

## See Also

- [create](create.md) - Create a new ledger
- [list](list.md) - List all ledgers
- [info](info.md) - Show ledger details
- [use](use.md) - Switch active ledger


---

# cli/cluster.md

# fluree cluster

Bootstrap and manage a Raft cluster (replicated transaction servers).

Wraps the Fluree server's private `/cluster/*` admin endpoints over HTTP. Run this from your operator workstation against each node's `--raft-listen-addr` (the VPC-internal admin port — not the client-facing port). The admin endpoints carry no built-in authentication; the CLI assumes reachability over a private network or SSH tunnel.

For the full deployment recipe and architectural context, see [Raft clusters (replicated writes)](../operations/raft-clusters.md).

## Subcommands

### fluree cluster init

Bootstrap a fresh single-node cluster on the seed node. Run **once**, against one node. That node becomes a single-voter cluster and auto-elects itself leader on the next election tick. Subsequent peers are added with `cluster add` and `cluster promote`.

**Usage:**

```bash
fluree cluster init [OPTIONS]
```

**Options:**

| Option                  | Description                                                                 |
|-------------------------|-----------------------------------------------------------------------------|
| `--addr <URL>`          | Admin URL of the seed node (e.g. `http://node-1-private:9090`).             |
| `--node-id <U64>`       | This node's id. Must be unique in the cluster and stable across restarts.   |
| `--raft-url <URL>`      | This node's inter-node Raft RPC URL (e.g. `http://node-1-private:9090/raft`). |
| `--client-url <URL>`    | This node's client-facing URL — used by peers when forwarding writes (e.g. `http://node-1:8080`). |

**Example:**

```bash
fluree cluster init \
  --addr        http://node-1-private:9090 \
  --node-id     1 \
  --raft-url    http://node-1-private:9090/raft \
  --client-url  http://node-1:8080
```

Returns immediately on success. Fails with `409 Conflict` if the cluster has already been initialized.

### fluree cluster add

Add a non-voting peer (learner) to an existing cluster. The new node replicates the log from the leader; once caught up it can be promoted to a voter with `cluster promote`. Issue against the current **leader**.

**Usage:**

```bash
fluree cluster add [OPTIONS]
```

**Options:**

| Option                  | Description                                                                 |
|-------------------------|-----------------------------------------------------------------------------|
| `--leader <URL>`        | Admin URL of the cluster leader.                                            |
| `--node-id <U64>`       | Id for the new peer.                                                        |
| `--raft-url <URL>`      | New peer's inter-node Raft RPC URL.                                         |
| `--client-url <URL>`    | New peer's client-facing URL.                                               |
| `--blocking <BOOL>`     | Wait for the learner to catch up before returning. Default: `true`.         |

**Example:**

```bash
fluree cluster add \
  --leader      http://node-1-private:9090 \
  --node-id     2 \
  --raft-url    http://node-2-private:9090/raft \
  --client-url  http://node-2:8080
```

`--blocking true` (default) is the right choice for orchestration scripts that immediately follow up with `cluster promote` — it removes the race between replication and the promote call.

### fluree cluster promote

Change the cluster's voting membership. Promotes learners to voters or demotes / removes existing voters. Issue against the **leader**.

**Usage:**

```bash
fluree cluster promote [OPTIONS]
```

**Options:**

| Option                  | Description                                                                  |
|-------------------------|------------------------------------------------------------------------------|
| `--leader <URL>`        | Admin URL of the cluster leader.                                             |
| `--members <U64>,...`   | New voter set, comma-separated (e.g. `1,2,3`).                               |
| `--retain`              | Keep voters dropped from `--members` as learners. Default: removed entirely. |

**Examples:**

```bash
# Promote learners 2 and 3 into the voting set.
fluree cluster promote \
  --leader  http://node-1-private:9090 \
  --members 1,2,3

# Drain node 3 — drop it from the voter set but keep it replicating
# as a learner so reads stay current while you debug.
fluree cluster promote \
  --leader  http://node-1-private:9090 \
  --members 1,2 \
  --retain
```

### fluree cluster status

Snapshot cluster state (current leader, term, voters, learners, last-applied log index). Any node's admin URL works — followers serve their last-known view, which is good enough for health checks and operator scripts.

**Usage:**

```bash
fluree cluster status --addr <URL>
```

**Example:**

```bash
fluree cluster status --addr http://node-1-private:9090
```

Output:

```
Cluster status
  leader:        1
  term:          4
  last_applied:  1283
  voters:        1, 2, 3
  learners:      (none)
```

## See also

- [Raft clusters (replicated writes)](../operations/raft-clusters.md) — deployment recipe, architecture, security notes.


---

# cli/drop.md

# fluree drop

Hard-drop an entire ledger (every branch under the name) or a graph source.

## Usage

```bash
fluree drop <NAME> --force
```

## Arguments

| Argument | Description |
|----------|-------------|
| `<NAME>` | Ledger name (bare, e.g. `mydb`) or graph source name. Branch-qualified ledger ids (including `mydb:main`) are rejected — use `fluree branch drop <branch> --ledger mydb` to drop a single branch. |

## Options

| Option | Description |
|--------|-------------|
| `--force` | Required flag to confirm deletion |

## Description

Hard-drops a **whole ledger** — every branch under the name, including any retracted-but-not-purged branches, plus the cross-branch `@shared/dicts/` namespace. Branches are dropped leaf-first so partial failure leaves orphan parents rather than dangling children. Equivalent to `POST /drop` with `"hard": true`. Deleted artifacts are irreversible.

The command first tries to drop the name as a ledger. If no nameservice record exists for the name, it tries to drop it as a graph source. This means `fluree drop` works uniformly for both ledgers and graph sources like Iceberg mappings.

The `--force` flag is required to prevent accidental deletion. There is no CLI soft-drop flag; use the HTTP or Rust API if you need to retract a ledger while preserving artifacts. To remove a single branch (not the whole ledger), use `fluree branch drop`.

Graph source cleanup is implementation-specific. The command retracts the graph source record and performs any available hard-drop cleanup for that graph source type; warnings are printed when cleanup is partial.

## Examples

```bash
# Drop the whole "oldledger" ledger (all branches + @shared/dicts/)
fluree drop oldledger --force

# Drop a graph source (Iceberg mapping)
fluree drop warehouse-orders --force
```

## Output

Ledger:
```
Dropped ledger 'oldledger'
```

Ledger with artifact cleanup:
```
Dropped ledger 'oldledger' (deleted 73 artifacts across 3 branches)
```

Graph source:
```
Dropped graph source 'warehouse-orders:main'
```

## Errors

Without `--force`:
```
error: use --force to confirm deletion of 'oldledger'
```

Branch-qualified input with a non-default suffix:
```
error: drop_ledger drops the whole ledger and does not accept a non-default
       branch suffix 'dev'. Use drop_branch("mydb", "dev") to drop a single
       branch, or pass "mydb" to drop the whole ledger.
```

## Dropping a single named graph

To drop just one **named graph** inside a ledger (without removing the
ledger, the branch, or any other graph), use `fluree graph drop`:

```bash
# Drop one named graph (active ledger, default branch)
fluree graph drop urn:example:org/payroll

# Drop on a specific ledger / branch
fluree graph drop urn:example:org/payroll --ledger mydb
fluree graph drop http://example.org/graphs/scratch --ledger mydb:feature-x

# Drop via a tracked remote server
fluree graph drop urn:example:org/payroll --ledger mydb --remote origin
```

Unlike `fluree drop`, `fluree graph drop` is **transactional and history-
preserving**: it produces a normal commit at `t = current + 1` whose
flakes retract every triple currently asserted in the graph, leaves the
graph IRI registered (so subsequent inserts land in the same graph slot),
and lets queries `as-of` an earlier `t` still see the dropped data.

The graph IRI must be an absolute IRI (e.g. `urn:...`, `http://...`).
The default graph and the system graphs (`urn:fluree:{ledger_id}#txn-meta`
and `urn:fluree:{ledger_id}#config`) cannot be dropped.

To see what graphs exist on a ledger, use `fluree graph list` (or look
at the `named-graphs` section of `fluree info <ledger>`).

## See Also

- [create](create.md) - Create a new ledger
- [iceberg](iceberg.md) - Map Iceberg tables as graph sources
- [list](list.md) - List all ledgers and graph sources


---

# cli/graph.md

# fluree graph

Manage **named graphs** within a single branch of a ledger.

Named graphs are created implicitly: the first time you transact a triple under a new graph IRI (via TriG `GRAPH <iri> { ... }`, JSON-LD `@graph` with an `@id`, or SPARQL `INSERT DATA { GRAPH <iri> { ... } }`), the graph is registered and assigned a deterministic `g_id`. There is no separate "create graph" command — `fluree graph` only exposes the operations that don't fall out of `insert` / `upsert` / `update` for free.

## Subcommands

### fluree graph list

List the user-defined named graphs registered on a branch.

**Usage:**

```bash
fluree graph list [OPTIONS]
```

**Options:**

| Option | Description |
|--------|-------------|
| `--ledger <LEDGER>` | Ledger identifier (e.g. `mydb` or `mydb:feature-x`). Defaults to the active ledger. |
| `--remote <REMOTE>` | List graphs on a remote server by remote name (e.g. `origin`). |
| `--json` | Emit the filtered `named-graphs` JSON array instead of a table. |
| `--include-system` | Include the default graph and the system `txn-meta` / `config` graphs. Off by default. |

**Description:**

Reads the `named-graphs` section of the standard `info` payload for the targeted branch — no new endpoint is required. By default, the output is restricted to user-defined graphs (`g_id >= 3`); pass `--include-system` to also surface the default graph (`g_id 0`), `txn-meta` (`g_id 1`), and `config` (`g_id 2`).

Each row reports the graph IRI, kind (`user`, `default`, `system:txn-meta`, `system:config`), `g_id`, current flake count, and on-disk size.

**Auto-routing:** like other read commands, `fluree graph list` auto-routes through a running local `fluree server` if one is up, or through the tracked remote when the active ledger has a `track` config. Pass `--direct` to bypass auto-routing and read from the local store, or `--remote <name>` to force a specific remote.

**Examples:**

```bash
# Active ledger, user graphs only
fluree graph list

# Specific ledger / branch
fluree graph list --ledger mydb:feature-x

# Remote
fluree graph list --ledger mydb --remote origin

# JSON output including system graphs
fluree graph list --ledger mydb --include-system --json
```

### fluree graph drop

Drop a single named graph from one branch by transactionally retracting every triple currently asserted in it.

**Usage:**

```bash
fluree graph drop <IRI> [OPTIONS]
```

**Arguments:**

| Argument | Description |
|----------|-------------|
| `<IRI>` | Full absolute IRI of the named graph to drop (e.g. `urn:example:org/payroll`). |

**Options:**

| Option | Description |
|--------|-------------|
| `--ledger <LEDGER>` | Ledger identifier. Defaults to the active ledger. |
| `--remote <REMOTE>` | Execute against a remote server by remote name. |

**Description:**

`fluree graph drop` is **transactional and history-preserving**:

- The drop produces one new commit at `t = current + 1` whose flakes are retractions of every triple currently asserted under the target graph IRI.
- A query `as-of` an earlier `t` (e.g. via `--at t:<N>`, `--at-time`, or a dataset `TimeSpec`) still sees the graph populated.
- The graph IRI keeps its `g_id`; a subsequent insert into the same IRI lands in the same logical graph rather than a new slot.
- Drops are per-branch — sibling branches that share the same graph IRI are not touched.
- Idempotent: re-running the drop on an already-empty graph reports `committed: false`, `retracted: 0` and produces no new commit.

The following are rejected with a clear error (no commit is produced):

- The **default graph** — refuses an empty IRI, since the default graph cannot be dropped.
- The **system graphs** — `urn:fluree:{ledger_id}#txn-meta` and `urn:fluree:{ledger_id}#config`.
- **Relative references** — `<IRI>` must be an absolute IRI with a valid `<scheme>:<rest>` head (e.g. `urn:...`, `http://...`).
- **Whitespace / control characters / RFC 3987-excluded characters** — leading/trailing whitespace is rejected, not silently trimmed.
- **Unknown graph IRIs** — return a 404-shaped "not registered" error; the registry is never silently extended by a drop.

**Examples:**

```bash
# Drop on the active ledger
fluree graph drop urn:example:org/payroll

# Specific ledger or branch
fluree graph drop urn:example:org/payroll --ledger mydb
fluree graph drop http://example.org/graphs/scratch --ledger mydb:feature-x

# Via a tracked remote server
fluree graph drop urn:example:org/payroll --ledger mydb --remote origin
```

**Output (committed):**

```
Dropped graph <urn:example:org/payroll> from 'mydb:main' — retracted 42 flakes (t=18).
```

**Output (no-op):**

```
Graph <urn:example:org/payroll> in 'mydb:main' was already empty — no commit produced (t=17).
```

## See Also

- [drop](drop.md) — drop a whole ledger or graph source
- [branch](branch.md) — branch management (drop, list, rebase, merge, diff)
- [info](info.md) — full ledger info, including the underlying `named-graphs` payload
- [Datasets and named graphs](../concepts/datasets-and-named-graphs.md) — concept doc
- [server-integration](server-integration.md) — wire contract for custom servers (`POST /drop-graph`, `GET /info`)


---

# cli/insert.md

# fluree insert

Insert data into a ledger.

## Usage

```bash
fluree insert [LEDGER] [DATA] [OPTIONS]
```

## Arguments

| Arguments | Behavior |
|-----------|----------|
| (none) | Active ledger; provide data via `-e`, `-f`, or stdin |
| `<arg>` | Auto-detected: if it looks like data (JSON, Turtle), uses it inline with the active ledger; if it's an existing file, reads from it; otherwise treats it as a ledger name |
| `<ledger> <data>` | Specified ledger + inline data |

## Options

| Option | Description |
|--------|-------------|
| `-e, --expr <EXPR>` | Inline data expression (alternative to positional) |
| `-f, --file <FILE>` | Read data from a file |
| `-m, --message <MSG>` | Commit message |
| `--format <FORMAT>` | Data format: `turtle` or `jsonld` (auto-detected if omitted) |
| `--remote <NAME>` | Execute against a remote server (by remote name, e.g., `origin`) |

## Description

Inserts RDF data into a ledger. Supports both Turtle and JSON-LD formats. Data can come from:
- A positional argument (inline data)
- `-e` flag (inline expression)
- `-f` flag (file)
- Standard input (pipe)

## Examples

```bash
# Insert inline Turtle
fluree insert '@prefix ex: <http://example.org/> .
ex:alice a ex:Person ; ex:name "Alice" .'

# Insert inline JSON-LD
fluree insert '{"@id": "ex:bob", "ex:name": "Bob"}'

# Insert from file
fluree insert -f data.ttl

# Insert with commit message
fluree insert -f data.ttl -m "Added initial users"

# Insert into specific ledger
fluree insert production '<http://example.org/x> a <http://example.org/Thing> .'

# Pipe from stdin
cat data.ttl | fluree insert
```

## Output

```
Committed t=1 (42 flakes)
```

With verbose mode:
```
Committed t=1 (42 flakes)
Commit ID: bafybeig...
```

## Data Format Detection

The format is auto-detected:
- `@prefix` or `@base` at line start → Turtle
- Starts with `{` or `[` → JSON-LD
- `.ttl` or `.nt` file extension → Turtle (N-Triples is parsed as Turtle)
- `.json` or `.jsonld` extension → JSON-LD

Override with `--format turtle` or `--format jsonld`.

## See Also

- [upsert](upsert.md) - Insert or update existing data
- [update](update.md) - Full WHERE/DELETE/INSERT updates
- [query](query.md) - Query the inserted data
- [export](export.md) - Export all data


---

# cli/upsert.md

# fluree upsert

Upsert data into a ledger (insert or update existing).

## Usage

```bash
fluree upsert [LEDGER] [DATA] [OPTIONS]
```

## Arguments

| Arguments | Behavior |
|-----------|----------|
| (none) | Active ledger; provide data via `-e`, `-f`, or stdin |
| `<arg>` | Auto-detected: if it looks like data (JSON, Turtle), uses it inline with the active ledger; if it's an existing file, reads from it; otherwise treats it as a ledger name |
| `<ledger> <data>` | Specified ledger + inline data |

## Options

| Option | Description |
|--------|-------------|
| `-e, --expr <EXPR>` | Inline data expression (alternative to positional) |
| `-f, --file <FILE>` | Read data from a file |
| `-m, --message <MSG>` | Commit message |
| `--format <FORMAT>` | Data format: `turtle` or `jsonld` (auto-detected if omitted) |
| `--remote <NAME>` | Execute against a remote server (by remote name, e.g., `origin`) |

## Description

Upserts RDF data into a ledger. Unlike `insert`, upsert will:
- Insert new entities
- Replace existing values for entities that already exist (matched by `@id`)

This is useful for updating data without needing to know whether it exists.

## Examples

```bash
# Update or insert a user
fluree upsert '@prefix ex: <http://example.org/> .
ex:alice ex:name "Alice Smith" ; ex:age 31 .'

# Upsert from file
fluree upsert -f updates.ttl

# Upsert with commit message
fluree upsert '{"@id": "ex:alice", "ex:status": "active"}' -m "Updated Alice status"
```

## Output

```
Committed t=2 (3 flakes)
```

## Difference from Insert

| Operation | Existing Entity | New Entity |
|-----------|-----------------|------------|
| `insert` | Adds new triples (may create duplicates) | Creates entity |
| `upsert` | Replaces values for given predicates | Creates entity |

## See Also

- [insert](insert.md) - Insert without replacement
- [update](update.md) - Full WHERE/DELETE/INSERT updates
- [query](query.md) - Query data
- [history](history.md) - View change history


---

# cli/update.md

# fluree update

Update data with full WHERE/DELETE/INSERT semantics.

## Usage

```bash
fluree update [LEDGER] [DATA] [OPTIONS]
```

## Arguments

| Arguments | Behavior |
|-----------|----------|
| (none) | Active ledger; provide data via `-e`, `-f`, or stdin |
| `<arg>` | Auto-detected: if it looks like data (JSON or SPARQL UPDATE), uses it inline with the active ledger; if it's an existing file, reads from it; otherwise treats it as a ledger name |
| `<ledger> <data>` | Specified ledger + inline data |

## Options

| Option | Description |
|--------|-------------|
| `-e, --expr <EXPR>` | Inline data expression (alternative to positional) |
| `-f, --file <FILE>` | Read data from a file |
| `-m, --message <MSG>` | Commit message |
| `--format <FORMAT>` | Data format: `jsonld` or `sparql` (auto-detected if omitted) |
| `--remote <NAME>` | Execute against a remote server (by remote name) |
| `--direct` | Bypass auto-routing through a local server (global flag; see note on SPARQL UPDATE below) |

## Description

Executes a WHERE/DELETE/INSERT transaction against a ledger. Unlike `insert` (which only adds data) and `upsert` (which replaces by subject+predicate), `update` supports the full WHERE/DELETE/INSERT pattern, enabling:

- **Conditional deletes**: delete triples matching a WHERE pattern
- **Conditional updates**: delete old values and insert new ones based on WHERE matches
- **Computed updates**: use variables from WHERE to derive new values via `bind`
- **Delete-only operations**: WHERE + DELETE without INSERT
- **Insert-only operations**: equivalent to `insert` but using the update command

### Supported Formats

- **JSON-LD** (default): transaction body with `where`, `delete`, and/or `insert` keys
- **SPARQL UPDATE**: standard `INSERT DATA`, `DELETE DATA`, `DELETE/INSERT WHERE` syntax

### SPARQL UPDATE Note

SPARQL UPDATE requires the server's parsing pipeline. It works automatically when:
- A local server is running (the CLI auto-routes through it by default)
- Using `--remote` to target a remote server

For direct local mode (`--direct`), use JSON-LD format instead.

## Examples

### Conditional Property Update (JSON-LD)

```bash
# Update Alice's age: find old value, delete it, insert new one
fluree update '{
  "@context": {"ex": "http://example.org/"},
  "where": [{"@id": "ex:alice", "ex:age": "?oldAge"}],
  "delete": [{"@id": "ex:alice", "ex:age": "?oldAge"}],
  "insert": [{"@id": "ex:alice", "ex:age": 31}]
}'
```

### Delete-Only

```bash
# Remove all email addresses for alice
fluree update '{
  "@context": {"ex": "http://example.org/"},
  "where": [{"@id": "ex:alice", "ex:email": "?email"}],
  "delete": [{"@id": "ex:alice", "ex:email": "?email"}]
}'
```

### Bulk Conditional Update

```bash
# Set all "pending" users to "active"
fluree update '{
  "@context": {"ex": "http://example.org/"},
  "where": [{"@id": "?person", "ex:status": "pending"}],
  "delete": [{"@id": "?person", "ex:status": "pending"}],
  "insert": [{"@id": "?person", "ex:status": "active"}]
}'
```

### From File

```bash
fluree update -f update.json
fluree update -f update.json -m "Updated user statuses"
```

### SPARQL UPDATE (via server)

```bash
# Requires a running server (fluree server start)
fluree update -e 'PREFIX ex: <http://example.org/>
DELETE { ex:alice ex:age ?oldAge }
INSERT { ex:alice ex:age 31 }
WHERE { ex:alice ex:age ?oldAge }'
```

### Pipe from stdin

```bash
cat update.json | fluree update
```

### Target a specific ledger

```bash
fluree update production -f migration.json
```

## Output

```
Committed t=3, 4 flakes
```

With remote mode, the full server response is printed as JSON.

## Format Detection

The format is auto-detected using this priority:

1. **Explicit flag** (`--format`) — always wins
2. **File extension** (when using `-f` or a positional file path):
   - `.json`, `.jsonld` → JSON-LD
   - `.rq`, `.ru`, `.sparql` → SPARQL UPDATE
3. **Content sniffing**:
   - Valid JSON (full parse, not just first character) → JSON-LD
   - Starts with `INSERT`, `DELETE`, `PREFIX`, or `BASE` → SPARQL UPDATE

Override with `--format jsonld` or `--format sparql`.

## Comparison with Insert and Upsert

| Operation | WHERE clause | DELETE | Conditional | Use case |
|-----------|-------------|--------|-------------|----------|
| `insert` | No | No | No | Add new data |
| `upsert` | No | Auto (per subject+predicate) | No | Replace values for known entities |
| `update` | Yes | Explicit | Yes | Targeted updates, deletes, complex transformations |

## See Also

- [insert](insert.md) - Insert new data
- [upsert](upsert.md) - Insert or replace existing data
- [Update (WHERE/DELETE/INSERT)](../transactions/update-where-delete-insert.md) - Full transaction syntax guide
- [query](query.md) - Query data


---

# cli/query.md

# fluree query

Query a ledger.

## Usage

```bash
fluree query [LEDGER] [QUERY] [OPTIONS]
```

## Arguments

| Arguments | Behavior |
|-----------|----------|
| (none) | Active ledger; provide query via `-e`, `-f`, or stdin |
| `<arg>` | Auto-detected: if it looks like a query, uses it inline with the active ledger; if it's an existing file, reads from it; otherwise treats it as a ledger name |
| `<ledger> <query>` | Specified ledger + inline query |

## Options

| Option | Description |
|--------|-------------|
| `-e, --expr <EXPR>` | Inline query expression (alternative to positional) |
| `-f, --file <FILE>` | Read query from a file |
| `--format <FORMAT>` | Output format: `json`, `typed-json`, `table`, `csv`, `tsv`, or `ndjson` (default: `table`) |
| `--envelope` | With `--format ndjson`, emit the full streaming record protocol verbatim instead of bare binding objects |
| `--sparql` | Force SPARQL query format |
| `--jsonld` | Force JSON-LD query format |
| `--at <TIME>` | Query at a specific point in time |
| `--normalize-arrays` | Always wrap multi-value properties in arrays (graph-crawl JSON-LD queries only) |
| `--bench` | Benchmark mode: time execution only and print the first 5 rows as a table (no full-result JSON formatting) |
| `--explain` | Print the query plan without executing it |
| `--remote <NAME>` | Execute against a remote server (by remote name, e.g., `origin`) |

## Description

Executes a query against a ledger. Supports both SPARQL and JSON-LD query formats.

## Query Formats

### SPARQL

```bash
fluree query 'SELECT ?name WHERE { ?s <http://example.org/name> ?name }'
```

### JSON-LD Query

```bash
fluree query '{"select": ["?name"], "where": {"http://example.org/name": "?name"}}'
```

Format is auto-detected if not specified:
- Contains `SELECT`, `CONSTRUCT`, `ASK`, or `DESCRIBE` → SPARQL
- Otherwise → JSON-LD

## Output Formats

### JSON (default)

```bash
fluree query 'SELECT ?name WHERE { ?s <http://example.org/name> ?name }'
```
```json
{
  "head": {"vars": ["name"]},
  "results": {"bindings": [{"name": {"type": "literal", "value": "Alice"}}]}
}
```

### Table

```bash
fluree query --format table 'SELECT ?name WHERE { ?s <http://example.org/name> ?name }'
```
```
┌───────┐
│ name  │
├───────┤
│ Alice │
│ Bob   │
└───────┘
```

### CSV

```bash
fluree query --format csv 'SELECT ?name WHERE { ?s <http://example.org/name> ?name }'
```
```
name
Alice
Bob
```

Note: `--format csv` (and `--format tsv`) are only supported for **local** ledgers. Tracked/remote ledgers support `json` and `table` output.

### NDJSON (streaming)

`--format ndjson` streams SELECT results **incrementally** as newline-delimited
JSON instead of buffering the whole result set — use it for large result sets
and unix pipelines (`| jq`, `| while read`, etc.). Rows are flushed as the query
produces them, so the CLI never holds the full result in memory.

By default each line is a bare SPARQL-JSON binding object (the same term shapes
as the `json` format's `results.bindings` entries), one per result row:

```bash
fluree query --format ndjson 'SELECT ?name WHERE { ?s <http://example.org/name> ?name }'
```
```
{"name":{"type":"literal","value":"Alice"}}
{"name":{"type":"literal","value":"Bob"}}
```

Add `--envelope` to emit the full streaming record protocol verbatim — the same
bytes the server's [streaming query endpoint](../api/streaming-query.md) produces
(`head` / `row` / `heartbeat` / `end` / `error`). This is useful for debugging
and for detecting truncation (a stream that ends without a terminal record):

```bash
fluree query --format ndjson --envelope 'SELECT ?name WHERE { ?s <http://example.org/name> ?name }'
```
```
{"type":"head","vars":["name"]}
{"type":"row","row":{"name":{"type":"literal","value":"Alice"}}}
{"type":"row","row":{"name":{"type":"literal","value":"Bob"}}}
{"type":"end","rows":2}
```

In bare mode the CLI consumes the terminal record internally and exits **non-zero**
if the stream carried an `error` or ended without a terminal (truncated /
dropped connection). A closed downstream pipe (e.g. `| head`) ends the stream
cleanly with exit 0.

Scope (mirrors the server [streaming endpoint](../api/streaming-query.md)):
NDJSON streaming applies to **SELECT** queries only. `ASK`, `CONSTRUCT`/`DESCRIBE`,
`selectOne`, hydration, and history-range queries are rejected — use a buffered
`--format`. On **local** ledgers, `--at` (time travel) and per-request `--policy`
are not supported with `--format ndjson`; use `--remote` (where they route
through the server's dataset path) or a buffered format instead. `--bench` and
`--explain` are not compatible with `--format ndjson`.

## Time Travel

Query historical states with `--at`:

```bash
# Query at transaction 5
fluree query --at 5 'SELECT * WHERE { ?s ?p ?o }'

# Query at specific commit
fluree query --at abc123def 'SELECT * WHERE { ?s ?p ?o }'

# Query at ISO-8601 timestamp
fluree query --at 2024-01-15T10:30:00Z 'SELECT * WHERE { ?s ?p ?o }'
```

Tracked/remote ledgers also support `--at`. The CLI will translate `--at` into the appropriate dataset/time-travel form when forwarding the query to the remote server.

SPARQL note (remote): if your SPARQL already includes `FROM` / `FROM NAMED`, the CLI will **not** rewrite it for `--at`. In that case, encode time travel directly in the `FROM` IRI (e.g., `FROM <myledger:main@t:5>`).

## Examples

```bash
# Inline SPARQL query (most common)
fluree query 'SELECT ?name WHERE { ?s <http://example.org/name> ?name }'

# JSON-LD query inline
fluree query '{"select": {"?s": ["*"]}, "where": {"@id": "?s"}}'

# Query specific ledger with CSV output
fluree query production --format csv 'SELECT ?s ?p ?o WHERE { ?s ?p ?o } LIMIT 10'

# SPARQL query from file
fluree query -f query.rq

# Time travel query
fluree query --at 3 'SELECT * WHERE { ?s ?p ?o }'

# Pipe from stdin
cat query.rq | fluree query
```

## See Also

- [history](history.md) - View entity change history
- [export](export.md) - Export all data


---

# cli/multi-query.md

# fluree multi-query

Execute a multi-query envelope — bundle multiple JSON-LD and/or SPARQL
queries into a single request that runs them in parallel against one
shared snapshot moment.

## Usage

```bash
fluree multi-query [FILE] [OPTIONS]
```

## Arguments

| Arguments | Behavior |
|-----------|----------|
| (none) | Read envelope JSON from `-e`, `-f`, or stdin |
| `<file>` | Path to envelope JSON file |

## Options

| Option | Description |
|--------|-------------|
| `-e, --expr <JSON>` | Inline envelope JSON (alternative to positional file or `-f`) |
| `-f, --file <FILE>` | Read envelope from a file |
| `--format <FORMAT>` | Per-alias result format: `json` (default — per-language defaults) or `typed-json` (always-typed `@value` / `@type` literal shape). Matches the same flag on [`fluree query`](query.md). |
| `--normalize-arrays` | Always wrap multi-value JSON-LD properties in arrays. Matches the same flag on [`fluree query`](query.md); applies to JSON-LD aliases. |
| `--output <VIEW>` | Envelope display: `json` (compact, default), `pretty` (indented), or `aliases` (per-alias sections). Controls how the response envelope is printed to the terminal — does **not** affect alias result formatting (use `--format` for that). |
| `--remote <NAME>` | Execute against a named remote server |
| `--as <IDENTITY>` | Bearer identity to assume (subject to the impersonation gate — see [Policy Contract](server-integration.md#policy-enforcement-contract)) |
| `--policy-class <IRI>` | Policy class IRI(s); repeatable |
| `--policy <JSON>` | Inline policy JSON (`fluree-policy`) |
| `--policy-file <FILE>` | Read inline policy JSON from a file |
| `--policy-values <JSON>` | Variable bindings for parameterized policies |
| `--policy-values-file <FILE>` | Read policy-values JSON from a file |
| `--default-allow` | Permit access when no matching policy rules exist |

Policy flags ride on the underlying HTTP request as `fluree-policy-*`
headers; the server folds them into the envelope's top-level `opts`
before validation, and the standard envelope → sub-query opts merge carries them into every alias. **They take effect on JSON-LD sub-queries** via the same code path single-query `/query` uses. **For SPARQL sub-queries** the headers are accepted and bearer ledger-scope still applies, but identity / policy threading via `QueryConnectionOptions` is not consumed — the same gap that exists for connection-scoped SPARQL on `/query`. See [Limitations](#limitations) for the canonical list.

## Description

Bundles N independent queries against a shared per-ledger snapshot.
Each sub-query carries its own `from` and its own language (JSON-LD or
SPARQL); the server runs them in parallel under bounded concurrency
and returns a per-alias response map plus an aggregate `status`.

See [Multi-query envelope](../api/multi-query.md) for the full envelope wire format, response shape, snapshot semantics, merge rules, bounds, and current limitations.

## Transport

`fluree multi-query` runs the envelope through whichever of three transports applies:

1. **`--remote <name>`** — explicit; routes through the named remote from `remotes.toml`. OIDC token refresh is persisted back to `config.toml` after the round-trip.
2. **Auto-route to a locally running `fluree server`** — used when `--remote` is omitted and `server.meta.json` reports a live pid. Suppressed by `--direct`.
3. **In-process local** — `fluree multi-query` calls `Fluree::multi_query()` directly against the storage tree configured for this `.fluree/` directory. Used when neither `--remote` nor a running local server is available, or when `--direct` is set. This is the natural counterpart to `fluree query` running locally.

In-process mode reads the same storage path, indexing thresholds, and prefix table as every other local CLI command. No HTTP, no server, no auth — the caller already has direct access to the storage tree.

## Examples

### Inline envelope, locally running server

```bash
fluree server start &
fluree multi-query -e '{
  "queries": {
    "people": {
      "language": "jsonld",
      "query": {
        "@context": {"ex": "http://example.org/"},
        "from": "mydb",
        "select": ["?name"],
        "where": {"@id": "?p", "ex:name": "?name"}
      }
    },
    "orders": {
      "language": "sparql",
      "query": "PREFIX ex: <http://example.org/> SELECT ?id FROM <mydb> WHERE { ?o ex:orderId ?id }"
    }
  }
}'
```

### Envelope file, named remote

```bash
fluree multi-query envelope.json --remote origin
```

### Stdin, pretty-printed response

```bash
cat envelope.json | fluree multi-query --remote origin --output pretty
```

### Typed JSON alias results + pretty envelope view

Pick the per-alias result shape with `--format` and the envelope display
with `--output`. They're independent: `--format` controls what the
server formats into each `results` entry, `--output` controls how the
CLI prints the whole response on your terminal.

```bash
fluree multi-query envelope.json --format typed-json --normalize-arrays --output pretty
```

### Per-alias section view

The `--output aliases` view prints a section per alias, with successful
results and per-alias errors clearly separated. Useful for shell-piping
results when you only want to inspect one alias and the response is
large.

```bash
fluree multi-query envelope.json --remote origin --output aliases
```

Output (abbreviated):

```
status: ok
asOf:   2024-01-01T12:00:00.123Z
  mydb @ t:42

# people (ok)
[
  { "name": "Alice" },
  ...
]

# orders (ok)
{ "head": ..., "results": ... }
```

## Per-alias result format (`--format`)

| Value | JSON-LD aliases | SPARQL aliases |
|-------|-----------------|----------------|
| `json` (default) | JSON-LD shape | SPARQL Results JSON |
| `typed-json` | typed `@value`/`@type` shape | typed `@value`/`@type` shape (cross-language) |

`--normalize-arrays` wraps single-valued JSON-LD properties in arrays.
On JSON-LD aliases it composes with whichever `--format` is in effect.
On SPARQL aliases it's a no-op — SPARQL Results JSON already has its own
binding shape and isn't affected. This matches the same flags on
[`fluree query`](query.md).

`--format typed-json` is cross-language by design — it gives every alias
a unified typed shape. `--format json` (with or without
`--normalize-arrays`) keeps SPARQL aliases on their SPARQL Results JSON
default, so you can `--normalize-arrays` a mixed envelope without
silently changing the SPARQL alias's wire shape.

Format selection rides on the wire as `Fluree-Output-Format` /
`Fluree-Normalize-Arrays` headers when using `--remote` or auto-routing
to a local server; in-process mode wires them straight into the api
crate's `MultiQueryBuilder::format(...)`.

Only JSON-producing shapes are valid inside a multi-query envelope —
TSV / CSV / SPARQL XML / RDF XML are rejected upstream because the
envelope's `results` map can't embed byte/string payloads. Use single
queries against `fluree query` when you need those.

## Envelope display (`--output`)

| Value | Shape | Use case |
|-------|-------|----------|
| `json` (default) | Compact JSON, one line | Machine processing; piping into `jq` |
| `pretty` | Indented JSON | Human reading |
| `aliases` | Per-alias section header + indented result block | Quick visual inspection; per-alias debugging |

The response shape itself is documented in
[Multi-query envelope → Response shape](../api/multi-query.md#response-shape).

## Status mapping

`fluree multi-query` exits **0** when the server returned HTTP 200,
regardless of the body's `status` field. Clients that need to branch
on aggregate outcome (`ok` / `partial` / `all_failed`) should inspect
the response JSON. A non-zero exit code indicates an envelope-level
failure (validation 4xx, snapshot resolution 5xx, transport error, or
envelope JSON malformed).

## Limitations

The full list lives in the [envelope reference](../api/multi-query.md#limitations). Highlights:

- History queries (`to` field / `FROM <…> TO <…>`) are rejected with 400.
- Envelope-level `opts.max-fuel` is rejected with 400 (per-sub-query `opts.max-fuel` still works).
- Response size cap is enforced at assembly, not throughout dispatch; peak memory during dispatch can exceed the envelope cap.
- `opts.t` at any level inside the envelope is rejected — use `from` or envelope `asOf`.
- SPARQL sub-queries do not consume merged policy opts — same gap as single-query connection-scoped SPARQL.

## See also

- [Multi-query envelope (HTTP reference)](../api/multi-query.md)
- [fluree query](query.md) — single-query CLI
- [Server integration: `fluree multi-query`](server-integration.md#fluree-multi-query)


---

# cli/history.md

# fluree history

Show change history for an entity.

## Usage

```bash
fluree history <ENTITY> [OPTIONS]
```

## Arguments

| Argument | Description |
|----------|-------------|
| `<ENTITY>` | Entity IRI (compact or full) |

## Options

| Option | Description |
|--------|-------------|
| `--ledger <LEDGER>` | Ledger name (defaults to active ledger) |
| `--from <TIME>` | Start of time range (default: `1`) |
| `--to <TIME>` | End of time range (default: `latest`) |
| `-p, --predicate <PRED>` | Filter to specific predicate |
| `--format <FORMAT>` | Output format: `json`, `table`, or `csv` (default: `table`) |

## Description

Shows the change history for a specific entity across transactions. Each change shows:
- `t` - Transaction number
- `op` - Operation: `+` (assert) or `-` (retract)
- `predicate` - The property that changed (if not filtered)
- `value` - The value asserted or retracted

## Prefix Expansion

Entity IRIs can use stored prefixes:

```bash
# First, add a prefix
fluree prefix add ex http://example.org/

# Then use compact IRI
fluree history ex:alice
```

Or use the full IRI:
```bash
fluree history http://example.org/alice
```

## Examples

```bash
# Show all changes to an entity
fluree history ex:alice

# Show changes in JSON format
fluree history ex:alice --format json

# Filter to specific predicate
fluree history ex:alice -p ex:name

# Show changes in a time range
fluree history ex:alice --from 1 --to 5

# Query specific ledger
fluree history ex:alice --ledger production
```

## Output

### Table (default)

```
┌───┬────┬─────────────────────────────────┬─────────────┐
│ t │ op │ predicate                       │ value       │
├───┼────┼─────────────────────────────────┼─────────────┤
│ 1 │ +  │ http://example.org/name         │ Alice       │
│ 1 │ +  │ http://example.org/age          │ 30          │
│ 2 │ -  │ http://example.org/name         │ Alice       │
│ 2 │ +  │ http://example.org/name         │ Alice Smith │
└───┴────┴─────────────────────────────────┴─────────────┘
```

### JSON

```json
[
  {"?t": 1, "?op": true, "?p": "http://example.org/name", "?v": "Alice"},
  {"?t": 1, "?op": true, "?p": "http://example.org/age", "?v": 30},
  {"?t": 2, "?op": false, "?p": "http://example.org/name", "?v": "Alice"},
  {"?t": 2, "?op": true, "?p": "http://example.org/name", "?v": "Alice Smith"}
]
```

### CSV

```
t,op,predicate,value
1,+,http://example.org/name,Alice
1,+,http://example.org/age,30
2,-,http://example.org/name,Alice
2,+,http://example.org/name,Alice Smith
```

## See Also

- [prefix](prefix.md) - Manage prefix mappings
- [log](log.md) - Show commit history
- [query](query.md) - Run custom queries


---

# cli/export.md

# fluree export

Export ledger data as Turtle, N-Triples, N-Quads, TriG, or JSON-LD.

## Usage

```bash
fluree export [LEDGER] [OPTIONS]
```

## Arguments

| Argument | Description |
|----------|-------------|
| `[LEDGER]` | Ledger name (defaults to active ledger) |

## Options

| Option | Description |
|--------|-------------|
| `--format <FORMAT>` | Output format: `turtle` (or `ttl`), `ntriples` (or `nt`), `jsonld`, `trig`, or `nquads` (default: `turtle`) |
| `--all-graphs` | Export default + all named graphs including system graphs (dataset export). Requires `--format trig` or `--format nquads`. |
| `--graph <IRI>` | Export a specific named graph by IRI. Mutually exclusive with `--all-graphs`. |
| `--context <JSON>` | JSON-LD context for prefix declarations. Overrides the ledger's default context. |
| `--context-file <FILE>` | Read context from a JSON file. Overrides the ledger's default context. |
| `--at <TIME>` | Export data as of a specific point in time. Accepts a transaction number (`5`), ISO-8601 datetime (`2024-01-15T10:30:00Z`), or commit CID prefix (`abc123def456`). If omitted, exports at the latest committed time (including data committed but not yet persisted to index). |

## Formats

### turtle / jsonld (data snapshot)

Exports a point-in-time snapshot of all triples in the ledger. Output goes to stdout.

### ledger (native pack)

Exports the full native ledger — all commits, transaction blobs, indexes, and dictionaries — as a `.flpack` file. This format preserves the complete history and can be imported into a new Fluree instance via `fluree create <name> --from <file>.flpack`.

The `.flpack` format uses the `fluree-pack-v1` binary wire protocol (the same format used by `fluree clone` and `fluree pull` for network transfers).

All formats (Turtle, N-Triples, N-Quads, TriG, JSON-LD) read directly from the binary SPOT index with a novelty overlay, so export always includes the latest committed transactions — even those not yet persisted to index. Memory usage stays constant regardless of dataset size. JSON-LD streams one subject at a time, so memory is O(largest subject), not O(dataset).

### Prefixes / Context

Turtle, TriG, and JSON-LD output use prefix compaction to produce compact, readable output. The prefix map is resolved in this order:

1. `--context` or `--context-file` (explicit override)
2. The ledger's default context (set via `fluree context set`)
3. No prefixes (falls back to full IRIs)

The context format is a JSON object mapping prefixes to namespace IRIs:

```json
{"ex": "http://example.org/", "schema": "http://schema.org/"}
```

### Prerequisites

All export formats require a binary index. Ledgers that have only been created and inserted into (without an index build) cannot be exported. Run the server to trigger index building first.

## Examples

```bash
# Export as Turtle (default) — uses ledger's default context for prefixes
fluree export > backup.ttl

# Export as Turtle with custom prefixes
fluree export --context '{"ex": "http://example.org/"}' > backup.ttl

# Export as Turtle with prefixes from a file
fluree export --context-file prefixes.json > backup.ttl

# Export as N-Triples (no prefixes, one triple per line)
fluree export --format ntriples > backup.nt

# Export as JSON-LD
fluree export --format jsonld > backup.jsonld

# Export all graphs as TriG
fluree export --all-graphs --format trig > backup.trig

# Export all graphs as N-Quads
fluree export --all-graphs --format nquads > backup.nq

# Export a specific named graph
fluree export --graph "http://example.org/g1" --format turtle > g1.ttl

# Export data as of a specific transaction number
fluree export --at 5 > snapshot-at-t5.ttl

# Export data as of an ISO-8601 datetime
fluree export --at "2024-06-15T12:00:00Z" > snapshot.ttl

# Export data as of a specific commit
fluree export --at abc123def456 > at-commit.ttl

# Export specific ledger
fluree export production > prod-backup.ttl

# Pipe to other tools
fluree export | grep "example.org"
```

## Output

### Turtle (default)

```turtle
@prefix ex: <http://example.org/> .

ex:alice
    a ex:Person ;
    ex:name "Alice" .
ex:bob
    a ex:Person ;
    ex:name "Bob" .
```

### N-Triples

```nt
<http://example.org/alice> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://example.org/Person> .
<http://example.org/alice> <http://example.org/name> "Alice" .
```

### TriG (all graphs)

```trig
@prefix ex: <http://example.org/> .

ex:alice
    ex:name "Alice" .

GRAPH ex:g1 {
ex:bob
    ex:name "Bob" .
}
```

### N-Quads (all graphs)

```nq
<http://example.org/alice> <http://example.org/name> "Alice" .
<http://example.org/bob> <http://example.org/name> "Bob" <http://example.org/g1> .
```

### JSON-LD

```json
{
  "@context": {
    "ex": "http://example.org/"
  },
  "@graph": [
    {"@id": "ex:alice", "@type": "ex:Person", "ex:name": "Alice"},
    {"@id": "ex:bob", "@type": "ex:Person", "ex:name": "Bob", "ex:age": {"@value": 25, "@type": "http://www.w3.org/2001/XMLSchema#long"}}
  ]
}
```

JSON-LD output uses prefix compaction from the context. Value encoding rules:

- Plain strings (`xsd:string`) → JSON string (no `@type`)
- Booleans → native JSON `true`/`false`
- Integers/longs → `{"@value": 42, "@type": "xsd:long"}` (explicit datatype)
- Decimals → `{"@value": "3.14", "@type": "xsd:decimal"}`
- Doubles → `{"@value": 3.14, "@type": "xsd:double"}`
- Language-tagged strings → `{"@value": "Bonjour", "@language": "fr"}`
- References → `{"@id": "ex:other"}`
- Single-cardinality properties are unwrapped (not in `[]`)
- Multi-cardinality properties use arrays

## API Usage

The export feature is available at the API level for upstream applications:

```rust
use fluree_db_api::export::ExportFormat;

// Turtle with default context
let stats = fluree.export("mydb")
    .format(ExportFormat::Turtle)
    .write_to(&mut writer)
    .await?;

// N-Quads with all graphs
let stats = fluree.export("mydb")
    .format(ExportFormat::NQuads)
    .all_graphs()
    .write_to(&mut writer)
    .await?;

// Turtle with custom prefixes
let stats = fluree.export("mydb")
    .format(ExportFormat::Turtle)
    .context(&json!({"ex": "http://example.org/"}))
    .write_to(&mut writer)
    .await?;

// JSON-LD with prefix compaction
let stats = fluree.export("mydb")
    .format(ExportFormat::JsonLd)
    .context(&json!({"ex": "http://example.org/"}))
    .write_to(&mut writer)
    .await?;

// Export a specific named graph
let stats = fluree.export("mydb")
    .format(ExportFormat::Turtle)
    .graph("http://example.org/g1")
    .write_to(&mut writer)
    .await?;

// Time-travel: export as of transaction t=5
let stats = fluree.export("mydb")
    .format(ExportFormat::Turtle)
    .as_of(TimeSpec::at_t(5))
    .write_to(&mut writer)
    .await?;

// Time-travel: export as of an ISO-8601 datetime
let stats = fluree.export("mydb")
    .format(ExportFormat::Turtle)
    .as_of(TimeSpec::at_time("2024-06-15T12:00:00Z"))
    .write_to(&mut writer)
    .await?;

// Convenience: write directly to stdout
let stats = fluree.export("mydb")
    .format(ExportFormat::Turtle)
    .to_stdout()
    .await?;
```

## See Also

- [context](context.md) - Manage default JSON-LD context (prefix map)
- [query](query.md) - Run custom queries


---

# cli/context.md

# fluree context

Manage the default JSON-LD context for a ledger.

## Usage

```bash
fluree context <COMMAND>
```

## Subcommands

| Command | Description |
|---------|-------------|
| `get [LEDGER]` | Show the default JSON-LD context |
| `set [LEDGER]` | Replace the default JSON-LD context |

## Description

Each ledger can have a **default context** — a JSON object mapping prefixes to IRIs (e.g., `{"ex": "http://example.org/"}`). When a JSON-LD query is sent via the **CLI** and omits its own `@context`, the ledger's default context is injected automatically. The HTTP API requires `?default-context=true` to opt in per request, and `fluree-db-api` requires explicit opt-in via its default-context view builders.

Default context is populated automatically during bulk import (from Turtle `@prefix` declarations). This command allows reading or replacing it after the fact.

The context is stored in content-addressed storage (CAS) and referenced from the nameservice config. Updates use compare-and-set semantics, so concurrent writers are safely handled.

## context get

Show the current default context.

```bash
fluree context get [LEDGER]
```

| Argument | Description |
|----------|-------------|
| `[LEDGER]` | Ledger name (defaults to active ledger) |

### Examples

```bash
# Show context for active ledger
fluree context get

# Show context for a specific ledger
fluree context get mydb
```

Output (pretty-printed JSON):

```json
{
  "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
  "rdfs": "http://www.w3.org/2000/01/rdf-schema#",
  "xsd": "http://www.w3.org/2001/XMLSchema#",
  "owl": "http://www.w3.org/2002/07/owl#",
  "ex": "http://example.org/"
}
```

If no default context has been set, a message is printed to stderr.

## context set

Replace the default context with a new JSON object.

```bash
fluree context set [LEDGER] [OPTIONS]
```

| Argument | Description |
|----------|-------------|
| `[LEDGER]` | Ledger name (defaults to active ledger) |

| Option | Description |
|--------|-------------|
| `-e, --expr <JSON>` | Inline JSON context |
| `-f, --file <PATH>` | Read context from a JSON file |

If neither `-e` nor `-f` is provided, context is read from stdin.

The body can be either a bare JSON object or wrapped in `{"@context": {...}}` — both forms are accepted.

### Examples

```bash
# Set inline
fluree context set mydb -e '{"ex": "http://example.org/", "foaf": "http://xmlns.com/foaf/0.1/"}'

# Set from file
fluree context set mydb -f context.json

# Pipe from stdin
cat context.json | fluree context set mydb

# Wrapped form also accepted
fluree context set mydb -e '{"@context": {"ex": "http://example.org/"}}'
```

## See Also

- [prefix](prefix.md) — Manage CLI-local prefix mappings (stored in project config, not the ledger)
- [export](export.md) — Export ledger data (the default context drives prefix output)
- [IRIs, namespaces, and JSON-LD @context](../concepts/iri-and-context.md) — Conceptual overview


---

# cli/log.md

# fluree log

Show commit log for a ledger.

## Usage

```bash
fluree log [LEDGER] [OPTIONS]
```

## Arguments

| Argument | Description |
|----------|-------------|
| `[LEDGER]` | Ledger name (defaults to active ledger) |

## Options

| Option | Description |
|--------|-------------|
| `--oneline` | Show one-line summary per commit |
| `-n, --count <N>` | Maximum number of commits to show |

## Description

Displays the commit history for a ledger, similar to `git log`. Shows transaction numbers, timestamps, and commit details.

## Examples

```bash
# Show full commit log
fluree log

# Show last 5 commits
fluree log -n 5

# One-line format
fluree log --oneline

# Specific ledger
fluree log production --oneline -n 10
```

## Output

### Full Format (default)

```
commit bafybeig2k5...
t: 3
Date: 2024-01-15T10:30:00Z

    Added new users

commit bafybeig7x3...
t: 2
Date: 2024-01-14T09:15:00Z

commit bafybeig9m1...
t: 1
Date: 2024-01-13T08:00:00Z

    Initial data load
```

### One-line Format

```
bafybeig2k5 t=3 Added new users
bafybeig7x3 t=2
bafybeig9m1 t=1 Initial data load
```

## See Also

- [show](show.md) - Show decoded contents of a specific commit
- [info](info.md) - Show ledger details
- [history](history.md) - Show entity change history


---

# cli/show.md

# fluree show

Show the decoded contents of a commit — assertions and retractions with resolved IRIs.

## Usage

```bash
fluree show <COMMIT> [OPTIONS]
```

## Arguments

| Argument | Description |
|----------|-------------|
| `<COMMIT>` | Commit identifier: `t:<N>` transaction number, hex-digest prefix (min 6 chars), or full CID |

## Options

| Option | Description |
|--------|-------------|
| `--ledger <NAME>` | Ledger name (defaults to active ledger) |
| `--remote <NAME>` | Execute against a remote server (by remote name, e.g., "origin") |

## Description

Displays the full decoded contents of a single commit, similar to `git show`. Each flake (assertion or retraction) is rendered with IRIs compacted using the ledger's namespace prefix table.

The commit identifier can be:
- A **transaction number** prefixed with `t:` (e.g., `t:5`) as shown in `fluree log` output
- An **abbreviated hex digest** (minimum 6 characters) as shown in the storage directory or obtained from the txn-meta graph
- A **full CID string** (e.g., `bagaybqabciq...`)

## Policy Filtering

When executed against a remote server (`--remote`), the returned flakes are filtered by the server's data-auth policy. The identity is derived from the Bearer token and the policy class from the server's `default_policy_class` configuration. Flakes the caller is not permitted to read are silently omitted, and the `asserts`/`retracts` counts reflect only the visible flakes.

Unlike the query endpoints, show does not support per-request policy overrides via headers or request body — it uses only the Bearer token identity and server-configured default policy class.

When executed locally (no `--remote`, or with `--direct`), `fluree show` operates with full local-admin access and no policy filtering is applied. This is consistent with other local CLI operations that read directly from storage.

## Output Format

The output is a JSON object containing:

| Field | Description |
|-------|-------------|
| `id` | Full CID of the commit |
| `t` | Transaction number |
| `time` | ISO 8601 timestamp |
| `size` | Commit blob size in bytes |
| `previous` | Previous commit CID |
| `signer` | Transaction signer (if signed) |
| `asserts` | Number of assertion flakes |
| `retracts` | Number of retraction flakes |
| `@context` | Namespace prefix table (prefix → IRI) |
| `flakes` | Array of flake tuples in SPOT order |

Each flake is a tuple: `[subject, predicate, object, datatype, operation]`

- `operation`: `true` = assert (added), `false` = retract (removed)
- Ref objects use `"@id"` as the datatype
- When metadata is present (language tag, list index, or named graph), a 6th element is appended: `{"lang": "en", "i": 0, "graph": "ex:myGraph"}`

## Examples

```bash
# Show a commit by transaction number
fluree show t:5

# Show a commit by hex prefix
fluree show 3dd028

# Show a commit from a specific ledger
fluree show 0303b7 --ledger _system

# Show a commit on a remote server
fluree show t:5 --remote origin

# Show by hex prefix on remote with explicit ledger
fluree show 3dd028 --remote origin --ledger mydb

# Pipe to jq for filtering
fluree show 3dd028 | jq '.flakes[] | select(.[4] == true)'
```

### Example Output

```json
{
  "id": "bagaybqabciqd3ubikmk2zh6gjxngpgjja3vi5myleidf46htiybpswyy2665zra",
  "t": 40,
  "time": "2026-03-12T16:58:18.395474217+00:00",
  "size": 327,
  "previous": "bagaybqabciqc64dbbv46vrueddgqfrafgmo27u4fibkrvwdmr2g6ze4cbaeg23a",
  "asserts": 1,
  "retracts": 1,
  "@context": {
    "xsd": "http://www.w3.org/2001/XMLSchema#",
    "schema": "http://schema.org/",
    "f": "https://ns.flur.ee/db#"
  },
  "flakes": [
    ["urn:fsys:dataset:zoho3", "schema:dateModified", "2026-03-12T14:15:30Z", "xsd:string", false],
    ["urn:fsys:dataset:zoho3", "schema:dateModified", "2026-03-12T16:58:16Z", "xsd:string", true]
  ]
}
```

In this example, one property (`dateModified`) was updated: the old value was retracted (`false`) and the new value asserted (`true`).

## See Also

- [log](log.md) - Show commit log (list of commits)
- [history](history.md) - Show change history for a specific entity
- [info](info.md) - Show ledger details


---

# cli/index.md

# fluree index

Build or update the binary index for a ledger.

## Usage

```bash
fluree index [LEDGER]
```

## Arguments

| Argument | Description |
|----------|-------------|
| `[LEDGER]` | Ledger name (defaults to active ledger) |

## Description

Performs incremental indexing when possible — merges only new commits into the existing index. Falls back to a full rebuild if incremental indexing isn't possible (e.g., no prior index exists).

Run this after transactions to clear the novelty layer and speed up queries. For routine use this is preferred over `reindex`, which always rebuilds from scratch.

## Examples

```bash
# Index the active ledger
fluree index

# Index a specific ledger
fluree index mydb
```

## Output

```
Indexed mydb to t=15 (root: bafyreig...)
```

## When to Use

- **After bulk transactions** — clears accumulated novelty so queries hit the optimized binary index instead of scanning in-memory flakes.
- **Routine maintenance** — keeps query performance consistent as data grows.
- **After `clone --no-indexes` or `pull --no-indexes`** — builds the local index that was skipped during transfer.

For a clean rebuild from commit history (e.g., suspected corruption), use [`reindex`](reindex.md) instead.

## See Also

- [reindex](reindex.md) - Full rebuild from commit history
- [Background indexing](../indexing-and-search/background-indexing.md) - Automatic indexing in the server
- [Reindex API](../indexing-and-search/reindex.md) - Rust API reference


---

# cli/reindex.md

# fluree reindex

Full reindex from commit history.

## Usage

```bash
fluree reindex [LEDGER]
```

## Arguments

| Argument | Description |
|----------|-------------|
| `[LEDGER]` | Ledger name (defaults to active ledger) |

## Description

Rebuilds the binary index from scratch by replaying all commits in order. This is a heavier operation than `index` — use it when the index is corrupted, missing, or you want a guaranteed clean rebuild.

For routine indexing after transactions, prefer [`index`](index.md).

## Examples

```bash
# Reindex the active ledger
fluree reindex

# Reindex a specific ledger
fluree reindex mydb
```

## Output

```
Reindexed mydb to t=15 (root: bafyreig...)
```

## When to Use

- **Suspected index corruption** — query results seem wrong or incomplete.
- **After schema or configuration changes** that affect index structure.
- **Clean slate** — you want to guarantee the index matches the commit history exactly.

For incremental indexing (faster, merges only new commits), use [`index`](index.md) instead.

## See Also

- [index](index.md) - Incremental index build
- [Background indexing](../indexing-and-search/background-indexing.md) - Automatic indexing in the server
- [Reindex API](../indexing-and-search/reindex.md) - Rust API reference


---

# cli/config.md

# fluree config

Manage configuration settings.

## Usage

```bash
fluree config <COMMAND>
```

## Subcommands

| Command | Description |
|---------|-------------|
| `get <KEY>` | Get a configuration value |
| `set <KEY> <VALUE>` | Set a configuration value |
| `list` | List all configuration values |
| `set-origins <LEDGER> --file <PATH>` | Set CID fetch origins for a ledger (writes a `LedgerConfig` to CAS and updates `config_id`) |

## Description

Manages configuration stored in `.fluree/config.toml`. Configuration uses dotted keys for nested values (e.g., `storage.path`).

## Examples

### Get a value

```bash
fluree config get storage.path
```

Output:
```
/custom/storage/path
```

### Set a value

```bash
fluree config set storage.path /custom/storage/path
```

Output:
```
Set 'storage.path' = "/custom/storage/path"
```

### List all values

```bash
fluree config list
```

Output:
```
storage.path = "/custom/storage/path"
storage.encryption = "aes256"
```

If no configuration is set:
```
(no configuration set)
```

## Configuration File

Configuration is stored in `.fluree/config.toml`:

```toml
[storage]
path = "/custom/storage/path"
encryption = "aes256"
```

## Errors

Getting a key that doesn't exist:
```
error: configuration key 'nonexistent' is not set
```

## See Also

- [init](init.md) - Initialize project directory
- [prefix](prefix.md) - Manage IRI prefixes

---

## fluree config set-origins

Store a `LedgerConfig` blob in local CAS and update the ledger's nameservice record to point to it via `config_id`.

This enables **origin-based** `fluree pull` (when no upstream remote is configured) and improves `fluree clone --origin` by allowing the remote to advertise multiple fallback origins.

### Usage

```bash
fluree config set-origins <LEDGER> --file <PATH>
```

### Arguments

| Argument | Description |
|----------|-------------|
| `<LEDGER>` | Ledger ID (e.g., `mydb` or `mydb:main`) |

### Options

| Option | Description |
|--------|-------------|
| `--file <PATH>` | Path to a JSON file containing a `LedgerConfig` |

### LedgerConfig File Format

The file is canonical JSON using compact `f:` keys (not JSON-LD):

```json
{
  "f:origins": [
    { "f:priority": 10, "f:enabled": true, "f:transport": "http://localhost:8090", "f:auth": { "f:mode": "none" } }
  ],
  "f:replication": { "f:preferPack": true, "f:maxPackMiB": 64 }
}
```

Notes:

- `f:transport` is an origin base URL. The CLI normalizes it the same way as remotes: it will append `/fluree` if missing and will use `GET /.well-known/fluree.json` discovery when available.
- Auth requirements are declarative. Credentials are not stored in the `LedgerConfig`.

### Current Limitations

- `fluree pull` via origins currently does not attach a Bearer token from any credential store, so only origins with `f:auth.f:mode = "none"` are usable for pull today.
- `fluree clone --origin ... --token ...` can use a Bearer token for origin fetch.


---

# cli/prefix.md

# fluree prefix

Manage IRI prefix mappings.

## Usage

```bash
fluree prefix <COMMAND>
```

## Subcommands

| Command | Description |
|---------|-------------|
| `add <PREFIX> <IRI>` | Add a prefix mapping |
| `remove <PREFIX>` | Remove a prefix mapping |
| `list` | List all prefix mappings |

## Description

Manages IRI prefix mappings stored in `.fluree/prefixes.json`. These prefixes are used to expand compact IRIs in commands like `history`.

## Examples

### Add a prefix

```bash
fluree prefix add ex http://example.org/
fluree prefix add foaf http://xmlns.com/foaf/0.1/
fluree prefix add schema https://schema.org/
```

Output:
```
Added prefix: ex = <http://example.org/>
```

### List prefixes

```bash
fluree prefix list
```

Output:
```
ex: <http://example.org/>
foaf: <http://xmlns.com/foaf/0.1/>
schema: <https://schema.org/>
```

If no prefixes are defined:
```
(no prefixes defined)

Add prefixes with: fluree prefix add <prefix> <iri>
Example: fluree prefix add ex http://example.org/
```

### Remove a prefix

```bash
fluree prefix remove foaf
```

Output:
```
Removed prefix: foaf
```

## Usage with History

Once prefixes are defined, you can use compact IRIs:

```bash
# Instead of:
fluree history http://example.org/alice

# Use:
fluree history ex:alice
```

## IRI Best Practices

IRI namespaces should end with `/` or `#`:

```bash
# Good
fluree prefix add ex http://example.org/
fluree prefix add foaf http://xmlns.com/foaf/0.1/

# Warning (will still work but may cause issues)
fluree prefix add bad http://example.org
```

## Storage

Prefixes are stored in `.fluree/prefixes.json`:

```json
{
  "ex": "http://example.org/",
  "foaf": "http://xmlns.com/foaf/0.1/"
}
```

## See Also

- [history](history.md) - Uses prefix expansion
- [config](config.md) - Manage other configuration


---

# cli/token.md

# fluree token

Manage JWS tokens for authentication with Fluree servers.

## Subcommands

| Subcommand | Description |
|------------|-------------|
| `create` | Create a new JWS token |
| `keygen` | Generate a new Ed25519 keypair |
| `inspect` | Decode and verify a JWS token |

---

## fluree token create

Create a new JWS token for authenticating with Fluree servers.

### Usage

```bash
fluree token create --private-key <KEY> [OPTIONS]
```

### Options

| Option | Description |
|--------|-------------|
| `--private-key <KEY>` | **Required.** Ed25519 private key (hex, base58, `@filepath`, or `@-` for stdin) |
| `--expires-in <DUR>` | Token lifetime (default: `1h`). Supports `s`, `m`, `h`, `d`, `w` suffixes |
| `--subject <SUB>` | Subject claim (`sub`) - identity of the token holder |
| `--audience <AUD>` | Audience claim (`aud`) - repeatable for multiple audiences |
| `--identity <ID>` | Fluree identity claim (`fluree.identity`) - takes precedence over `sub` for policy |
| `--all` | Grant full access to all ledgers (events, storage, read, and write) |
| `--events-ledger <ALIAS>` | Grant events access to specific ledger (repeatable) |
| `--storage-ledger <ALIAS>` | Grant storage access to specific ledger (repeatable) |
| `--read-all` | Grant data API read access to all ledgers (`fluree.ledger.read.all=true`) |
| `--read-ledger <ALIAS>` | Grant data API read access to specific ledger (repeatable) |
| `--write-all` | Grant data API write access to all ledgers (`fluree.ledger.write.all=true`) |
| `--write-ledger <ALIAS>` | Grant data API write access to specific ledger (repeatable) |
| `--graph-source <ALIAS>` | Grant access to specific graph source (repeatable) |
| `--output <FMT>` | Output format: `token`, `json`, or `curl` (default: `token`) |
| `--print-claims` | Print decoded claims to stderr |

### Private Key Formats

| Format | Example |
|--------|---------|
| Hex | `0x<64 hex chars>` or `<64 hex chars>` |
| Base58 | `z<base58 string>` (multibase) or raw base58 |
| File | `@/path/to/keyfile` or `@~/.fluree/key` (tilde expansion) |
| Stdin | `@-` (read from stdin to avoid shell history) |

### Examples

```bash
# Create a token with full access
fluree token create --private-key 0x1234...abcd --all

# Create a token for specific ledgers (events/storage)
fluree token create --private-key @~/.fluree/key \
  --events-ledger mydb --storage-ledger mydb

# Create a token with data API read+write for specific ledgers
fluree token create --private-key @~/.fluree/key \
  --read-ledger mydb:main --write-ledger mydb:main

# Create a token with identity and audience
fluree token create --private-key @- \
  --identity did:example:alice \
  --audience https://api.example.com \
  --expires-in 7d

# Output as curl command
fluree token create --private-key 0x... --all --output curl

# View claims while creating
fluree token create --private-key 0x... --all --print-claims
```

---

## fluree token keygen

Generate a new Ed25519 keypair for signing tokens.

### Usage

```bash
fluree token keygen [OPTIONS]
```

### Options

| Option | Description |
|--------|-------------|
| `--format <FMT>` | Output format: `hex`, `base58`, or `json` (default: `hex`) |
| `-o, --output <PATH>` | Write private key to file (otherwise prints to stdout) |

### Examples

```bash
# Generate keypair in hex format
fluree token keygen

# Generate in JSON format with all representations
fluree token keygen --format json

# Save private key to file
fluree token keygen --output ~/.fluree/key

# Generate base58 format
fluree token keygen --format base58
```

### Output

Hex format:
```
Private key: 0x1234567890abcdef...
Public key:  0xabcdef1234567890...
DID:         did:key:z6MkhaXgBZDvotDkL5257faiztiGiC2QtKLGpbnnEGta2doK
```

JSON format:
```json
{
  "private_key": {
    "hex": "0x1234...",
    "base58": "z..."
  },
  "public_key": {
    "hex": "0xabcd...",
    "base58": "z..."
  },
  "did": "did:key:z6Mk..."
}
```

---

## fluree token inspect

Decode and optionally verify a JWS token.

### Usage

```bash
fluree token inspect <TOKEN> [OPTIONS]
```

### Arguments

| Argument | Description |
|----------|-------------|
| `<TOKEN>` | JWS token string or `@filepath` |

### Options

| Option | Description |
|--------|-------------|
| `--no-verify` | Skip signature verification (default: verify) |
| `--output <FMT>` | Output format: `pretty`, `json`, or `table` (default: `pretty`) |

### Examples

```bash
# Inspect and verify a token
fluree token inspect eyJhbGciOiJFZERTQSI...

# Inspect without verification
fluree token inspect eyJ... --no-verify

# Output as JSON
fluree token inspect eyJ... --output json

# Read token from file
fluree token inspect @token.txt
```

### Output

Pretty format:
```
Token Information
─────────────────────────────────────────────────────
Issuer:   did:key:z6MkhaXgBZDvotDkL5257faiztiGiC2QtKLGpbnnEGta2doK
Subject:  test@example.com
Issued:   2024-01-15 10:30:00 UTC
Expires:  2024-01-15 11:30:00 UTC

Permissions:
  Events:  all ledgers
  Storage: all ledgers

Signature: ✓ Valid
```

## Token Scopes

Tokens can carry different permission scopes that control access to different server features:

| Scope | Claim | Controls |
|-------|-------|----------|
| Events (all) | `fluree.events.all` | SSE event stream for all ledgers |
| Events (specific) | `fluree.events.ledgers` | SSE event stream for listed ledgers |
| Storage (all) | `fluree.storage.all` | Storage proxy read access (all); also implies data API read |
| Storage (specific) | `fluree.storage.ledgers` | Storage proxy read access (listed); also implies data API read |
| Read (all) | `fluree.ledger.read.all` | Data API query access to all ledgers |
| Read (specific) | `fluree.ledger.read.ledgers` | Data API query access to listed ledgers |
| Write (all) | `fluree.ledger.write.all` | Data API write access to all ledgers |
| Write (specific) | `fluree.ledger.write.ledgers` | Data API write access to listed ledgers |

The `--all` flag sets events, storage, read, and write access for all ledgers.

**Back-compat:** `fluree.storage.*` claims also grant data API read access for the same ledgers.

## See Also

- [auth](auth.md) - Store/manage tokens on remotes
- [remote](remote.md) - Configure remote servers
- [Authentication](../security/authentication.md) - Auth model, modes, and token claims
- [fetch](fetch.md) - Fetch from remotes (requires auth token)
- [push](push.md) - Push to remotes (requires auth token)


---

# cli/auth.md

# fluree auth

Manage authentication tokens for remote servers. Tokens are stored in `.fluree/config.toml` as part of the remote configuration.

Token values are never printed to stdout. The `status` command shows token presence, expiry, and identity only.

## Subcommands

| Subcommand | Description |
|------------|-------------|
| `status` | Show authentication status for a remote |
| `login` | Store a bearer token for a remote |
| `logout` | Clear the stored token for a remote |

---

## fluree auth status

Show the current authentication state for a remote, including token presence, expiry time, identity, and issuer.

### Usage

```bash
fluree auth status [OPTIONS]
```

### Options

| Option | Description |
|--------|-------------|
| `--remote <NAME>` | Remote name (defaults to the only configured remote) |

### Examples

```bash
# Show auth status (single remote)
fluree auth status

# Show auth status for a specific remote
fluree auth status --remote origin
```

### Output

When a token is configured:
```
Auth Status:
  Remote: origin
  Token:  configured
  Expiry: 2026-02-15 12:00 UTC
  Identity: did:example:alice
  Issuer: did:key:z6Mk...
  Subject: alice@example.com
```

When no token is configured:
```
Auth Status:
  Remote: origin
  Token:  not configured
  hint: fluree auth login --remote origin
```

---

## fluree auth login

Store a bearer token for a remote. The token is saved in `.fluree/config.toml` and will be sent as a `Authorization: Bearer` header on subsequent remote operations (`fetch`, `pull`, `push`, `query --remote`, etc.).

> **Which path do I use?** If the remote is backed by an identity provider (`auth.type = "oidc_device"`, auto-discovered), `fluree auth login` runs an interactive browser/device login for you — see [OIDC login flow](#oidc-login-flow) below; you do **not** pass `--token`. The `--token` options documented here are for manually supplying a token you already have, such as a locally-created Ed25519 JWS (`fluree token create`) or a service-account token for automation.

### Usage

```bash
fluree auth login [OPTIONS]
```

### Options

| Option | Description |
|--------|-------------|
| `--remote <NAME>` | Remote name (defaults to the only configured remote) |
| `--token <VALUE>` | Token value, `@filepath` to read from file, or `@-` for stdin |

If `--token` is omitted, you will be prompted to paste the token interactively.

### Token Input Methods

| Method | Example |
|--------|---------|
| Inline value | `--token eyJhbG...` |
| File | `--token @/path/to/token.jwt` |
| File (tilde) | `--token @~/.fluree/token.jwt` |
| Stdin | `--token @-` (pipe or redirect) |
| Interactive | Omit `--token` to be prompted |

### Examples

```bash
# Store a token (prompted interactively)
fluree auth login

# Store a token from a value
fluree auth login --token eyJhbGciOiJFZERTQSI...

# Store a token from a file
fluree auth login --token @~/.fluree/my-token.jwt

# Pipe a token from another command
fluree token create --private-key @~/.fluree/key --all | fluree auth login --token @-

# Login to a specific remote
fluree auth login --remote staging --token @token.jwt
```

### Output

```
Token stored for remote 'origin'
  Expiry: 2026-02-15 12:00 UTC
  Identity: did:example:alice
```

---

## fluree auth logout

Clear the stored token for a remote.

### Usage

```bash
fluree auth logout [OPTIONS]
```

### Options

| Option | Description |
|--------|-------------|
| `--remote <NAME>` | Remote name (defaults to the only configured remote) |

### Examples

```bash
# Clear token for the default remote
fluree auth logout

# Clear token for a specific remote
fluree auth logout --remote staging
```

### Output

```
Token cleared for remote 'origin'
```

---

## Token Types

The `auth` command stores bearer tokens that are sent in the `Authorization` header. Fluree supports two types of bearer tokens:

### Ed25519 JWS Tokens (did:key)

Created locally with `fluree token create`. These contain an embedded JWK (JSON Web Key) in the token header and are verified against the embedded public key. The issuer is a `did:key` identifier derived from the signing key.

```bash
# Create and store a token in one step
fluree token create --private-key @~/.fluree/key --all | fluree auth login --token @-
```

### OIDC/JWKS Tokens (RS256)

Issued by external identity providers (OIDC). These contain a `kid` (Key ID) in the token header and are verified by the server against the provider's JWKS (JSON Web Key Set) endpoint. The issuer is the provider's URL.

The server must be configured with `--jwks-issuer` to trust these tokens. See [Configuration](../operations/configuration.md#oidc--jwks-token-verification).

## Remote Resolution

When `--remote` is omitted:
- If exactly one remote is configured, it is used automatically.
- If no remotes are configured, an error is shown with a hint to use `fluree remote add`.
- If multiple remotes are configured, an error asks you to specify `--remote <name>`.

## Security Notes

- Tokens are stored in plaintext in `.fluree/config.toml`. On Unix, the CLI restricts the config file to `0600` and the `.fluree` directory to `0700` (owner-only) whenever it writes token material, so the file is not group/world-readable. On other platforms, protect the file with appropriate filesystem permissions yourself.
- The `status` command never displays the raw token value.
- On 401 errors from remote operations, the CLI checks token expiry and suggests `fluree auth login` if the token appears expired.

## OIDC login flow

When a remote is configured with `auth.type = "oidc_device"` (auto-discovered from the server's `/.well-known/fluree.json`), `fluree auth login` runs an OIDC interactive login flow and then exchanges the IdP token for a Fluree-scoped Bearer token:

1. Discovers OIDC endpoints from the configured issuer
2. Chooses the flow based on IdP support:
   - If the IdP discovery document includes `device_authorization_endpoint`: use OAuth device-code (prints a URL + code and polls).
   - Otherwise, if it includes `authorization_endpoint`: use OAuth authorization-code + PKCE (opens a browser and receives a localhost callback).
3. Exchanges the IdP token for a Fluree-scoped Bearer token via the server's `exchange_url`
4. Stores the token (and optional refresh token) in the remote config

If the IdP grants fewer scopes than were requested, the CLI prints a warning at login (the downgraded token still works, but a missing scope would otherwise surface later as an unrelated authorization failure). If the exchange returns no refresh token, the CLI notes that you will need to re-run `fluree auth login` when the token expires.

### End-to-end walkthrough

```bash
# 1. Add the remote (auth type is auto-discovered from /.well-known/fluree.json)
fluree remote add origin https://db.example.com

# 2. Log in — opens a browser (PKCE) or prints a device code, then exchanges for a Fluree token
fluree auth login --remote origin
#   Discovering OIDC endpoints...
#   >> Open this URL and enter the code below:  (device flow)
#   ...or a browser window opens automatically  (PKCE flow)
#   IdP authentication successful
#   Exchanging for Fluree token...
#   Token stored for remote 'origin'

# 3. Use the remote
fluree pull --remote origin
```

When the stored token later expires, data-plane commands (`query`, `insert`, …) auto-refresh silently if a refresh token is present; otherwise, or for replication commands, re-run `fluree auth login --remote origin`.

### Automation / headless environments

The browser (PKCE) flow needs a loopback callback port and, for most IdPs, an interactive browser — neither is available in CI or containers. For automation, **do not use the browser flow**; instead provision a service-account token and supply it directly:

```bash
fluree token create --private-key @~/.fluree/key --all | fluree auth login --remote origin --token @-
```

If you must use PKCE non-interactively, note that the callback listener only binds ports `8400`–`8405` (override with `redirect_port` in `/.well-known/fluree.json` or the `FLUREE_AUTH_PORT` env var). The port cannot be OS-assigned because the callback URL must be pre-allowlisted at the IdP (see the Cognito note below).

### Cognito note (Authorization Code + PKCE)

AWS Cognito does not publish `device_authorization_endpoint`, so the CLI will use authorization-code + PKCE.

Cognito requires the callback URL to be pre-allowlisted (no wildcard ports). Allowlist:

- `http://127.0.0.1:8400/callback`
- `http://127.0.0.1:8401/callback`
- `http://127.0.0.1:8402/callback`
- `http://127.0.0.1:8403/callback`
- `http://127.0.0.1:8404/callback`
- `http://127.0.0.1:8405/callback`

If your app only allowlists one callback URL, configure a fixed port with `redirect_port` in `/.well-known/fluree.json` (or set `FLUREE_AUTH_PORT` locally) and allowlist that single callback URL.

On subsequent 401 errors, the CLI automatically attempts a silent refresh using the stored refresh token before prompting for re-login.

See [Auth contract (CLI ↔ Server)](../design/auth-contract.md) for the full protocol specification.

## See Also

- [token](token.md) - Create and inspect JWS tokens
- [remote](remote.md) - Manage remote servers
- [Authentication](../security/authentication.md) - Auth model, modes, and token claims
- [Auth contract (CLI ↔ Server)](../design/auth-contract.md) - Discovery, exchange, and refresh protocol
- [Configuration](../operations/configuration.md) - Server authentication configuration


---

# cli/remote.md

# fluree remote

Manage remote servers for syncing ledgers.

## Subcommands

| Subcommand | Description |
|------------|-------------|
| `add` | Add a remote server |
| `remove` | Remove a remote |
| `list` | List all configured remotes |
| `show` | Show details for a remote |

---

## fluree remote add

Add a remote server configuration.

### Usage

```bash
fluree remote add <NAME> <URL> [OPTIONS]
```

### Arguments

| Argument | Description |
|----------|-------------|
| `<NAME>` | Remote name (e.g., `origin`) |
| `<URL>` | Server URL (e.g., `http://localhost:8090`) |

### Options

| Option | Description |
|--------|-------------|
| `--token <TOKEN>` | Authentication token (or `@filepath` to read from file) |

### Examples

```bash
# Add a remote without authentication
fluree remote add origin http://localhost:8090

# Add a remote with inline token
fluree remote add prod https://api.example.com --token eyJ...

# Add a remote with token from file
fluree remote add staging https://staging.example.com --token @~/.fluree/staging-token
```

---

## fluree remote remove

Remove a remote configuration.

### Usage

```bash
fluree remote remove <NAME>
```

### Arguments

| Argument | Description |
|----------|-------------|
| `<NAME>` | Remote name to remove |

### Examples

```bash
fluree remote remove origin
```

---

## fluree remote list

List all configured remotes.

### Usage

```bash
fluree remote list
```

### Output

```
┌─────────┬─────────────────────────────┬───────┐
│ Name    │ URL                         │ Auth  │
├─────────┼─────────────────────────────┼───────┤
│ origin  │ http://localhost:8090       │ none  │
│ prod    │ https://api.example.com     │ token │
└─────────┴─────────────────────────────┴───────┘
```

---

## fluree remote show

Show detailed information about a remote.

### Usage

```bash
fluree remote show <NAME>
```

### Arguments

| Argument | Description |
|----------|-------------|
| `<NAME>` | Remote name |

### Output

```
Remote:
  Name: origin
  Type: HTTP
  URL:  http://localhost:8090
  Auth: token configured
```

## See Also

- [upstream](upstream.md) - Configure upstream tracking
- [clone](clone.md) - Clone a ledger from a remote
- [fetch](fetch.md) - Fetch refs from a remote
- [token](token.md) - Create authentication tokens


---

# cli/upstream.md

# fluree upstream

Manage upstream tracking configuration for ledgers.

Upstream configuration links a local ledger to a remote ledger, enabling `pull` and `push` operations.

## Subcommands

| Subcommand | Description |
|------------|-------------|
| `set` | Set upstream tracking for a ledger |
| `remove` | Remove upstream tracking |
| `list` | List all upstream configurations |

---

## fluree upstream set

Configure a local ledger to track a remote ledger.

### Usage

```bash
fluree upstream set <LOCAL> <REMOTE> [OPTIONS]
```

### Arguments

| Argument | Description |
|----------|-------------|
| `<LOCAL>` | Local ledger ID (e.g., `mydb` or `mydb:main`) |
| `<REMOTE>` | Remote name (e.g., `origin`) |

### Options

| Option | Description |
|--------|-------------|
| `--remote-alias <ALIAS>` | Remote ledger ID (defaults to local ledger ID) |
| `--auto-pull` | Automatically pull on fetch |

### Examples

```bash
# Track remote ledger with same name
fluree upstream set mydb origin

# Track a differently-named remote ledger
fluree upstream set mydb origin --remote-alias production-db

# Enable auto-pull on fetch
fluree upstream set mydb origin --auto-pull
```

---

## fluree upstream remove

Remove upstream tracking for a ledger.

### Usage

```bash
fluree upstream remove <LOCAL>
```

### Arguments

| Argument | Description |
|----------|-------------|
| `<LOCAL>` | Local ledger ID |

### Examples

```bash
fluree upstream remove mydb
```

---

## fluree upstream list

List all configured upstream tracking relationships.

### Usage

```bash
fluree upstream list
```

### Output

```
┌────────────┬─────────┬────────────────┬───────────┐
│ Local      │ Remote  │ Remote Alias   │ Auto-Pull │
├────────────┼─────────┼────────────────┼───────────┤
│ mydb:main  │ origin  │ mydb           │ no        │
│ test:main  │ staging │ test-ledger    │ yes       │
└────────────┴─────────┴────────────────┴───────────┘
```

## See Also

- [remote](remote.md) - Configure remote servers
- [clone](clone.md) - Clone a ledger from a remote
- [pull](pull.md) - Pull from upstream
- [push](push.md) - Push to upstream


---

# cli/fetch.md

# fluree fetch

Fetch refs from a remote server (similar to `git fetch`).

## Usage

```bash
fluree fetch <REMOTE>
```

## Arguments

| Argument | Description |
|----------|-------------|
| `<REMOTE>` | Remote name (e.g., `origin`) |

## Description

Fetches ledger references from a remote server and updates local tracking data. This does **not** modify your local ledgers - it only updates what the CLI knows about the remote's state.

This is a **replication** operation. It requires a Bearer token with **root / storage-proxy** permissions (`fluree.storage.*`). If you only have permissioned/query access to a ledger, you should use `fluree track` (or `--remote`) and run queries/transactions against the remote instead.

After fetching, you can use `pull` to download and apply new commits to your local ledger.

## Examples

```bash
# Fetch from origin
fluree fetch origin

# Typical workflow
fluree fetch origin
fluree pull mydb
```

## Output

```
Fetching from 'origin'...
Updated:
  mydb -> t=42
  testdb -> t=15
Already up to date: 2 ledger(s) unchanged
```

If no ledgers are found:
```
Fetching from 'origin'...
No ledgers found on remote.
```

## See Also

- [remote](remote.md) - Configure remote servers
- [clone](clone.md) - Clone a ledger from a remote
- [pull](pull.md) - Pull commits from upstream
- [push](push.md) - Push to upstream


---

# cli/pull.md

# fluree pull

Pull commits from upstream and apply them to the local ledger, similar to `git pull`.

## Usage

```bash
fluree pull [OPTIONS] [LEDGER]
```

## Arguments

| Argument | Description |
|----------|-------------|
| `[LEDGER]` | Ledger name (defaults to active ledger) |
| `--no-indexes` | Skip pulling binary index data; only transfer new commits and txn blobs (local index may lag until you run `fluree reindex`) |

## Description

Downloads new commits from the configured upstream and applies them to the local ledger:

1. Queries the remote for its current head (`t` and commit ContentId)
2. Compares with the local head; exits early if already up to date
3. Attempts bulk download of missing commits (and by default **index artifacts**) via the **pack protocol** (single streaming request)
4. Falls back to paginated JSON export if the server does not support pack
5. Stores all commit and transaction blobs to local CAS
6. When index data is requested and transferred, advances the local index head to match the remote
7. Advances the local commit head to the remote head

### Index transfer

As with [clone](clone.md#index-transfer), pull uses the pack protocol to request **index artifacts** by default when the remote has an index. Use **`--no-indexes`** to transfer only new commits and txn blobs. For large estimated transfers (~1 GiB or more), the CLI prompts for confirmation before streaming.

### Transport

Pull uses the same pack protocol as clone -- see [clone: Transport](clone.md#transport) for details.

### Origin-based pull

When no upstream remote is configured, pull falls back to **origin-based fetching** if a LedgerConfig with origins is set on the ledger (see `fluree config set-origins`). This uses the same pack-first / CID-walk-fallback transport as `fluree clone --origin`.

This is a **replication** operation. It requires a Bearer token with **root / storage-proxy** permissions (`fluree.storage.*`). If you only have permissioned/query access to a ledger, you should use `fluree track` (or `--remote`) and run queries/transactions against the remote instead.

The ledger must have an upstream configured (see `fluree upstream set`), **or** a LedgerConfig with origins (see `fluree config set-origins`).

**Restart safety:** If interrupted, the local head reflects the last successful import. The next pull resumes from the local head automatically.

## Examples

```bash
# Pull changes for active ledger
fluree pull

# Pull changes for specific ledger
fluree pull mydb

# Pull commits only (skip index transfer)
fluree pull --no-indexes mydb
```

## Output

Successful pull (with index data when remote has an index):
```
Pulling 'mydb:main' from 'origin' (local t=10, remote t=42)...
✓ 'mydb:main' pulled 32 commit(s) via pack (new head t=42)
```

With `--no-indexes`, only commits (and referenced txn blobs) are transferred; the message does not include index artifact counts.

Already up to date:
```
✓ 'mydb:main' is already up to date
```

No upstream configured:
```
error: no upstream configured for 'mydb:main'
  hint: fluree upstream set mydb:main <remote>
```

## Errors

| Error | Description |
|-------|-------------|
| No upstream configured | Run `fluree upstream set <ledger> <remote>` first, or configure origins via `fluree config set-origins` |
| Ancestry mismatch | Remote chain does not descend from local head (histories diverged) |
| Import validation failure | Commit chain or retraction invariant violation |

## Limitations

- **Index head vs commit head:** When you use `--no-indexes`, the local index head is not updated. Queries still work but may replay more novelty; run `fluree reindex` to bring the index up to the current commit head.
- **Graph source indexes not replicated:** Graph source snapshots (BM25/vector/geo, etc.) are not replicated by `fluree pull` yet. Rebuild graph source indexes in the target environment as needed.

## See Also

- [clone](clone.md) - Clone a ledger from a remote server
- [upstream](upstream.md) - Configure upstream tracking
- [fetch](fetch.md) - Fetch refs without modifying local ledger
- [push](push.md) - Push local changes to upstream


---

# cli/push.md

# fluree push

Push local ledger changes to upstream remote, similar to `git push`.

## Usage

```bash
fluree push [LEDGER]
```

## Arguments

| Argument | Description |
|----------|-------------|
| `[LEDGER]` | Ledger name (defaults to active ledger) |

## Description

Pushes **local commits** to the configured upstream remote by uploading the commit v2 bytes to the server.

The ledger must have an upstream configured (see `fluree upstream set`).

The push uses strict sequencing + CAS semantics:

- The server rejects the push if the remote head is not in your local history (diverged) or if the remote is ahead.
- The server also rejects the push if the first commit’s `t` does not match the server’s next-t.

Unlike `fetch`/`pull`, this is **not** a storage-proxy replication operation. It requires **write** permissions for the ledger (Bearer token with `fluree.ledger.write.*` claims) and the server validates the pushed commits like normal transactions.

If a pushed commit contains **retractions**, the server enforces a strict invariant: each retraction must target a fact that is currently asserted at that point in the push batch. (List retractions require exact list-index metadata match.)

## Examples

```bash
# Push active ledger
fluree push

# Push specific ledger
fluree push mydb
```

## Output

Successful push:
```
Pushing 'mydb:main' to 'origin'...
✓ 'mydb:main' pushed 3 commit(s) (new head t=42)
```

Push rejected (remote is ahead):
```
Pushing 'mydb:main' to 'origin'...
error: push rejected; remote is ahead (local t=10, remote t=42). Pull first.
```

No upstream configured:
```
error: no upstream configured for 'mydb:main'
  hint: fluree upstream set mydb:main <remote>
```

## Errors

| Error | Description |
|-------|-------------|
| No upstream configured | Run `fluree upstream set <ledger> <remote>` first |
| Push rejected (409) | Remote head changed, histories diverged, or first commit `t` does not match next-t |
| Push rejected (422) | Invalid commit bytes, missing required referenced blob, or retraction invariant violation |

## Workflow

Typical sync workflow:

```bash
# Configure remote and upstream (one time)
fluree remote add origin https://api.example.com --token @~/.fluree/token
fluree upstream set mydb origin

# Daily workflow
fluree pull mydb        # Get latest changes
# ... make local changes ...
fluree push mydb        # Push your changes
```

## See Also

- [clone](clone.md) - Clone a ledger from a remote
- [upstream](upstream.md) - Configure upstream tracking
- [pull](pull.md) - Pull changes from upstream
- [fetch](fetch.md) - Fetch refs without modifying local ledger


---

# cli/publish.md

# fluree publish

Publish a local ledger to a remote server. Creates the ledger on the remote if it doesn't exist, pushes all local commits, and configures upstream tracking for subsequent `push`/`pull`.

## Usage

```bash
fluree publish <REMOTE> [LEDGER] [OPTIONS]
```

## Arguments

| Argument | Description |
|----------|-------------|
| `<REMOTE>` | Remote name (e.g., "origin") |
| `[LEDGER]` | Ledger name (defaults to active ledger) |

## Options

| Option | Description |
|--------|-------------|
| `--remote-name <NAME>` | Remote ledger name (defaults to local ledger name) |

## Description

`fluree publish` is the reverse of `fluree clone`. It takes a locally-created ledger and pushes it to a remote server in a single operation:

1. Checks if the ledger exists on the remote (`GET /exists`)
2. Creates it if not (`POST /create`)
3. Pushes all local commits (`POST /push`)
4. Configures upstream tracking so subsequent `fluree push` and `fluree pull` work

This is intended for the "create locally, deploy to server" workflow. If the remote ledger already has data (t > 0), the command will fail — use `fluree push` instead for incremental updates.

## Examples

```bash
# Publish active ledger to origin
fluree publish origin

# Publish a specific ledger
fluree publish origin mydb

# Publish with a different name on the remote
fluree publish origin mydb --remote-name production-db

# Typical workflow: create locally, develop, then publish
fluree create mydb
fluree insert mydb -e '{"@id": "ex:test", "ex:name": "Test"}'
fluree publish origin mydb
```

## Prerequisites

- A remote must be configured: `fluree remote add origin <url>`
- The remote must support the Fluree HTTP API (see [Server implementation guide](../design/server-implementation.md))
- A valid auth token if the remote requires authentication: `fluree auth login --remote origin`

## After Publishing

Once published, the ledger has upstream tracking configured. Use standard sync commands:

```bash
# Push new local commits to remote
fluree push

# Pull remote changes
fluree pull
```

## See Also

- [push](push.md) - Push incremental commits to upstream
- [pull](pull.md) - Pull changes from upstream
- [clone](clone.md) - Clone a remote ledger locally (reverse of publish)
- [remote](remote.md) - Manage remote server configuration
- [upstream](upstream.md) - Manage upstream tracking
- [export](export.md) - Export ledger as `.flpack` for file-based transfer


---

# cli/clone.md

# fluree clone

Clone a ledger from a remote server, similar to `git clone`.

## Usage

```bash
# Named-remote clone
fluree clone [OPTIONS] <REMOTE> <LEDGER>

# Origin-based clone (no pre-configured remote)
fluree clone --origin <URI> [--token <TOKEN>] [OPTIONS] <LEDGER>
```

## Arguments

| Argument | Description |
|----------|-------------|
| `<REMOTE>` | Remote name (configured via `fluree remote add`) |
| `<LEDGER>` | Ledger name on the remote server |
| `--origin <URI>` | Bootstrap URI for CID-based clone (replaces `<REMOTE>`) |
| `--token <TOKEN>` | Auth token for origin server (with `--origin` only) |
| `--no-indexes` | Skip pulling binary index data; only transfer commits and txn blobs (queries will replay from commits until you run `fluree reindex`) |
| `--no-txns` | Skip pulling original transaction payloads. Commits still transfer (chain remains valid and verifiable), but the raw JSON-LD / SPARQL requests that produced each commit are not downloaded. Use for read-only clones of large ledgers. See [Transaction transfer](#transaction-transfer). |

## Description

Downloads all commits from a remote ledger and creates a local copy:

1. Verifies the remote ledger exists and has commits
2. Creates a local ledger with the same name as on the remote
3. Attempts bulk download via the **pack protocol** (single streaming request)
4. Falls back to paginated JSON export if the server does not support pack
5. Stores all commit and transaction blobs to local CAS
6. By default, also transfers **binary index** artifacts when the remote has an index (see [Index transfer](#index-transfer))
7. Sets the local commit head (and index head when index data was transferred) to match the remote
8. Configures the remote as upstream for future `pull`/`push` (named-remote only)

### Index transfer

When using the pack protocol, the CLI requests **index artifacts** by default so the local ledger is query-ready without a full reindex. The server sends missing commit blobs, txn blobs, and binary index artifacts (dictionaries, branches, leaves) in one stream.

- Use **`--no-indexes`** to transfer only commits and txn blobs. This reduces transfer size and time; afterward, run `fluree reindex` to build the index locally if needed.
- For large transfers (estimated size above ~1 GiB), the CLI prompts: *"Estimated transfer size: ~X. This may take several minutes. Continue? [Y/n]"*. Answer `n` to abort or to re-request without index data (commits-only).
- If the remote has no index yet (e.g. a fresh ledger), only commits and txns are transferred regardless of the flag.

### Transaction transfer

Every commit references an **original transaction blob** — the raw request (JSON-LD insert/update or SPARQL Update) that produced the commit. By default, `fluree clone` downloads these so the local ledger has a complete audit trail of the original payloads.

- Use **`--no-txns`** to skip transaction blobs entirely. The commit chain is still cloned and remains valid and verifiable; only the original request payloads are missing.
- The materialized ledger state (what queries return) is reconstructable from commits + indexes alone — transactions are not needed for query answering.
- With `--no-txns`, operations that need the original request payload (e.g., `fluree show --flakes` for transaction-level inspection, or re-running a transaction against a branch) will fail locally for those transactions. Anything that only reads materialized state is unaffected.
- Combine with `--no-indexes` for the smallest possible clone (`fluree clone --no-indexes --no-txns origin mydb`), useful for minimal verification / auditing of the commit chain only.

### Transport

The CLI uses the **pack protocol** (`fluree-pack-v1`) as the primary transport for clone and pull. Pack transfers all missing CAS objects (commits + txn blobs, and by default index artifacts) in a single streaming HTTP request, avoiding per-object round-trips.

If the remote server does not support the pack endpoint (returns 404, 405, 406, or 501), the CLI automatically falls back to:
- **Named-remote mode**: paginated JSON export via `GET /commits/{ledger}` (500 commits per page)
- **Origin mode**: CID chain walk via `GET /storage/objects/{cid}` (one round-trip per commit)

This fallback is transparent -- no user action is required.

### Origin-based clone

The `--origin` flag enables CID-based clone from a server URL without pre-configuring a named remote:

```bash
fluree clone --origin http://localhost:8090 mydb
fluree clone --origin https://api.example.com --token @~/.fluree/token mydb
```

This mode:
1. Fetches the NsRecord from the origin to discover the head commit CID
2. Optionally upgrades to a multi-origin fetcher if a LedgerConfig is advertised
3. Downloads commits via pack (or CID chain walk as fallback)
4. Stores the LedgerConfig locally for future origin-based `pull`
5. Does **not** configure upstream tracking (use `fluree upstream set` manually)

This is a **replication** operation. It requires a Bearer token with **root / storage-proxy** permissions (`fluree.storage.*`). If you only have permissioned/query access to a ledger, you should use `fluree track` (or `--remote`) and run queries/transactions against the remote instead.

**Idempotent CAS writes:** If interrupted mid-clone, CAS blob writes are idempotent. Re-running the clone command will re-fetch all pages (duplicate writes are harmless). The local head is only set after all data is downloaded.

## Examples

```bash
# Clone a ledger from a configured remote
fluree clone origin mydb

# Full workflow: add remote, then clone
fluree remote add production https://api.example.com --token @~/.fluree/token
fluree clone production customers

# Origin-based clone (no remote setup needed)
fluree clone --origin http://localhost:8090 mydb

# Origin-based clone with auth
fluree clone --origin https://api.example.com --token @~/.fluree/token mydb

# Clone without index data (faster; run fluree reindex afterward if needed)
fluree clone --no-indexes origin mydb

# Clone commits + indexes but skip original transaction payloads
fluree clone --no-txns origin mydb

# Smallest possible clone — commits only (no indexes, no transactions)
fluree clone --no-indexes --no-txns origin mydb
```

## Output

Successful clone (via pack, with index data):
```
Cloning 'mydb:main' from 'origin' (remote t=1042)...
  fetched 2084 object(s) via pack
✓ Cloned 'mydb:main' (1042 commits, head t=1042)
  → upstream set to 'origin/mydb:main'
```

With `--no-indexes` (commits and txns only), the object count will be lower and the local index head is not set until you run `fluree reindex`.

Successful clone (fallback to paginated export):
```
Cloning 'mydb:main' from 'origin' (remote t=1042)...
  fetched 500 commits...
  fetched 1000 commits...
  fetched 1042 commits...
✓ Cloned 'mydb:main' (1042 commits, head t=1042)
  → upstream set to 'origin/mydb:main'
```

Origin-based clone:
```
Cloning 'mydb:main' from 'http://localhost:8090' (remote t=50)...
  fetched 100 object(s) via pack
✓ Cloned 'mydb:main' (50 commit(s), head t=50)
```

Remote ledger has no commits:
```
Remote ledger 'mydb:main' has no commits (t=0), nothing to clone.
```

## Errors

| Error | Description |
|-------|-------------|
| Remote not configured | Run `fluree remote add <name> <url>` first |
| Ledger not found on remote | Verify the ledger name matches the remote server |
| Auth failure | Token missing or lacks `fluree.storage.*` permissions |
| Local ledger already exists | Drop the existing ledger first |

## Limitations

- **Post-clone indexing:** If you used `--no-indexes`, run `fluree reindex` to build a binary index locally. Without an index, queries replay from commits and can be slow for large ledgers. When index data is transferred by default (no `--no-indexes`), the local index head is set and no reindex is needed for the core ledger.
- **Missing transactions:** If you used `--no-txns`, the original transaction payloads for historical commits are permanently unavailable on the local clone (re-pull will not fetch them unless you explicitly re-clone without the flag). The ledger state remains queryable; only transaction-level inspection and replay are affected.
- **Graph source indexes not replicated:** Graph source snapshots (BM25/vector/geo, etc.) are not replicated by `fluree clone` yet. After cloning, rebuild graph source indexes in the target environment as needed.

## See Also

- [pull](pull.md) - Pull new commits from upstream
- [push](push.md) - Push local commits to upstream
- [remote](remote.md) - Configure remote servers
- [upstream](upstream.md) - Configure upstream tracking


---

# cli/track.md

# fluree track

Track a remote ledger without storing local data. Tracked ledgers route reads and writes to the configured remote server while keeping a lightweight record locally so you can use short aliases and the active-ledger shortcut.

## Usage

```bash
fluree track <SUBCOMMAND>
```

## Subcommands

### fluree track add

Start tracking a remote ledger under a local alias.

**Usage:**

```bash
fluree track add <LEDGER> [--remote <NAME>] [--remote-alias <NAME>]
```

**Arguments:**

| Argument | Description |
|----------|-------------|
| `<LEDGER>` | Local alias for the tracked ledger |

**Options:**

| Option | Description |
|--------|-------------|
| `--remote <NAME>` | Remote name (e.g., `origin`). Defaults to the only configured remote if unambiguous. |
| `--remote-alias <NAME>` | Alias on the remote (defaults to the local alias) |

**Examples:**

```bash
# Track a remote ledger using the same name locally
fluree track add production --remote origin

# Use a different local alias
fluree track add prod --remote origin --remote-alias production
```

### fluree track remove

Stop tracking a remote ledger. Local data is not affected (tracked ledgers have none).

**Usage:**

```bash
fluree track remove <LEDGER>
```

| Argument | Description |
|----------|-------------|
| `<LEDGER>` | Local alias to stop tracking |

### fluree track list

List all currently tracked ledgers and the remote each resolves to.

**Usage:**

```bash
fluree track list
```

### fluree track status

Show status of tracked ledger(s) by querying the configured remote for each — commit t, index t, and head IDs.

**Usage:**

```bash
fluree track status [LEDGER]
```

| Argument | Description |
|----------|-------------|
| `[LEDGER]` | Local alias (shows all tracked ledgers if omitted) |

**Examples:**

```bash
# Status of all tracked ledgers
fluree track status

# Status for a single tracked ledger
fluree track status production
```

## Description

A tracked ledger is a local pointer to a remote ledger. Queries, transactions, and most administrative commands against a tracked alias are transparently forwarded to the remote. This lets you work against a hosted ledger using the same CLI flow as a local ledger — including the active-ledger shortcut (`fluree use`), without syncing commit/index data to disk.

Use `fluree clone` instead when you need a full local copy of a remote ledger's data.

## See Also

- [remote](remote.md) - Manage named remote servers
- [clone](clone.md) - Clone a remote ledger locally (with data)
- [use](use.md) - Switch active ledger
- [list](list.md) - List local and tracked ledgers


---

# cli/server.md

# server

Manage the Fluree HTTP server from the CLI. The server inherits the same `.fluree/` context (config file, storage path) as the CLI — one directory, two modes of interaction.

## Subcommands

| Subcommand | Description |
|------------|-------------|
| `run`      | Run the server in the foreground (Ctrl-C to stop) |
| `start`    | Start the server as a background process |
| `stop`     | Stop a backgrounded server |
| `status`   | Show server status (PID, address, health) |
| `restart`  | Stop and restart a backgrounded server |
| `logs`     | View server logs |

## Common Options

These options are available on `run`, `start`, and `restart`:

| Option | Description |
|--------|-------------|
| `--listen-addr <ADDR>` | Listen address (e.g., `0.0.0.0:8090`) |
| `--storage-path <PATH>` | Storage path override (local file storage) |
| `--connection-config <FILE>` | JSON-LD connection config for S3, DynamoDB, etc. |
| `--log-level <LEVEL>` | Log level (`trace`, `debug`, `info`, `warn`, `error`) |
| `--profile <NAME>` | Configuration profile to activate |
| `-- <ARGS>...` | Additional server flags (passed through to server config) |

`--storage-path` and `--connection-config` are mutually exclusive. Use `--storage-path` for local file storage or `--connection-config` for remote backends (S3, DynamoDB, split storage). See [Configuration](../operations/configuration.md#connection-configuration-s3-dynamodb-etc) for details.

When no flags are provided, the server discovers its configuration using the same search as the CLI: it walks up from the current working directory looking for a `.fluree/config.toml` (or `config.jsonld`), then falls back to the global Fluree config directory (`$FLUREE_HOME`, or the platform config directory — see [Configuration](../operations/configuration.md)). Server settings live under the `[server]` section. The CLI's `--config` flag is also honored.

## run

Run the server in the foreground. Logs go to stderr. Press Ctrl-C for graceful shutdown.

```bash
# Start with defaults from config.toml
fluree server run

# Override listen address
fluree server run --listen-addr 127.0.0.1:9090

# S3 + DynamoDB backend
fluree server run --connection-config /etc/fluree/connection.jsonld

# Pass through advanced server flags
fluree server run -- --cors-enabled --indexing-enabled
```

## start

Start the server as a background daemon. Writes PID and metadata to `.fluree/` and redirects output to `.fluree/server.log`.

```bash
# Start in background
fluree server start

# Preview resolved config without starting
fluree server start --dry-run

# Start with overrides
fluree server start --listen-addr 0.0.0.0:8090 --log-level debug
```

The `--dry-run` flag prints the fully resolved configuration (config file + env + flag overrides merged) without actually starting the server. Useful for debugging "why is it using port X?".

## stop

Stop a backgrounded server by sending SIGTERM and waiting for graceful shutdown (up to 10 seconds).

```bash
fluree server stop

# Force kill after timeout
fluree server stop --force
```

## status

Check whether the server is running. Shows PID, listen address, uptime, storage path, and performs an HTTP health check.

```bash
fluree server status
```

Example output:

```
ok: Server is running
  pid:          12345
  listen_addr:  0.0.0.0:8090
  storage_path: /path/to/.fluree/storage
  started_at:   2026-02-16T10:30:00Z
  uptime:       2h 15m 30s
  health:       ok
  log:          /path/to/.fluree/server.log
```

When using `--connection-config`, the status shows the connection config path instead of the storage path:

```
ok: Server is running
  pid:          12345
  listen_addr:  0.0.0.0:8090
  connection:   /etc/fluree/connection.jsonld
  started_at:   2026-02-16T10:30:00Z
  uptime:       2h 15m 30s
  health:       ok
```

## restart

Stop and restart a backgrounded server. Recovers the original arguments from `.fluree/server.meta.json`. New flag overrides can be applied on restart.

```bash
fluree server restart

# Restart with a different log level
fluree server restart --log-level debug
```

## logs

View server log output from `.fluree/server.log`.

```bash
# Last 50 lines (default)
fluree server logs

# Last 100 lines
fluree server logs -n 100

# Follow (like tail -f)
fluree server logs -f
```

## Auto-Routing

When a local server is running (started via `fluree server start`), CLI commands that support remote execution are **automatically routed through the server's HTTP API**. This applies to:

- `fluree query`
- `fluree insert`
- `fluree upsert`
- `fluree list`
- `fluree info`

The CLI detects the running server by checking `.fluree/server.meta.json` and verifying the PID is alive. When auto-routing is active, you'll see a hint on stderr:

```
  server: routing through local server at 0.0.0.0:8090 (use --direct to bypass)
```

### Opting out

Use the `--direct` global flag to bypass auto-routing and execute directly via the CLI's file-based path:

```bash
# Route through server (default when server is running)
fluree query 'SELECT * WHERE { ?s ?p ?o } LIMIT 10'

# Bypass server, execute directly
fluree query --direct 'SELECT * WHERE { ?s ?p ?o } LIMIT 10'
```

### Crash detection

If the server has crashed or been killed, the CLI detects the stale PID and falls back to direct execution with a notice:

```
  notice: local server (pid 12345) is no longer running; executing directly
```

Use `fluree server status` to check server health, or `fluree server logs` to view crash output.

## Runtime Files

When a background server is running, these files are created in the `.fluree/` data directory:

| File | Description |
|------|-------------|
| `server.pid` | PID of the background server process |
| `server.log` | stdout + stderr from the background server |
| `server.meta.json` | Metadata for `restart` and `status` (PID, address, args, start time) |

These files are cleaned up automatically by `fluree server stop`.

## Configuration

The server uses the same config file as the CLI (discovered via walk-up or global fallback — see above). Server-specific settings live under the `[server]` section:

```toml
[server]
listen_addr = "0.0.0.0:8090"
storage_path = "/var/lib/fluree"
log_level = "info"
cors_enabled = true
# cache_max_mb = 4096  # global cache budget (MB); default: tiered by RAM (<4GB: 30%, 4-8GB: 40%, >=8GB: 35%)

[server.indexing]
enabled = true
reindex_min_bytes = 100_000
# reindex_max_bytes defaults to 20% of system RAM; uncomment to override
# reindex_max_bytes = 536_870_912  # 512 MB
```

For S3/DynamoDB backends, use `connection_config` instead of `storage_path`:

```toml
[server]
connection_config = "/etc/fluree/connection.jsonld"
cache_max_mb = 4096

[server.indexing]
enabled = true
```

Indexing settings live under the `[server.indexing]` subsection, not directly on `[server]`. Authentication settings similarly use `[server.auth.events]`, `[server.auth.data]`, etc.

See [Configuration](../operations/configuration.md) for the full list of server options.

## Feature Flags

The `server` subcommand requires the `server` Cargo feature (enabled by default). If compiled without it:

```bash
fluree server run
# error: server support not compiled. Rebuild with `--features server`.
```

S3/DynamoDB support via `--connection-config` requires the `aws` feature, which
is **on by default** for the CLI (and the published Docker image). It is dormant
unless an AWS backend is configured. To build a CLI *without* it:

```bash
cargo build -p fluree-db-cli --no-default-features --features server,iceberg,shacl
```

Without this feature, S3 storage configs in the connection config will produce a clear error at startup.


---

# cli/memory.md

# fluree memory

Developer memory — store and recall facts, decisions, and constraints.

> This page is the CLI command reference. For conceptual background, IDE setup, team workflows, and the full schema, see the [Memory section](../memory/README.md) of the docs.

## Usage

```bash
fluree memory <COMMAND>
```

## Subcommands

| Command | Description |
|---------|-------------|
| `init` | Initialize the memory store (creates `__memory` ledger) |
| `add` | Store a new memory |
| `recall` | Search and rank relevant memories |
| `update <ID>` | Update a memory in place |
| `forget <ID>` | Delete a memory |
| `status` | Show memory store status |
| `export` | Export all current memories as JSON |
| `import <FILE>` | Import memories from a JSON file |
| `mcp-install` | Install MCP configuration for an IDE |

## Description

The memory system stores project knowledge as RDF triples in a dedicated `__memory` Fluree ledger. Memories persist across sessions and are searchable by keyword-scored recall.

Run `fluree memory init` before using other memory commands. The MCP server auto-initializes on first tool call.

## fluree memory init

Initialize the memory store and optionally configure MCP for detected AI coding tools. Idempotent — safe to run multiple times.

```bash
fluree memory init [OPTIONS]
```

### Options

| Option | Description |
|--------|-------------|
| `--yes, -y` | Auto-confirm all MCP installations (non-interactive) |
| `--no-mcp` | Skip AI tool detection and MCP configuration entirely |

### What init does

1. **Creates the `__memory` ledger** and transacts the memory schema.
2. **Creates `.fluree-memory/`** at the project root with `repo.ttl`, `.gitignore`, and `.local/user.ttl`.
3. **Migrates existing memories** — if the ledger already has memories (e.g., from a pre-TTL version), they are exported to the appropriate `.ttl` files.
4. **Detects AI coding tools** (Claude Code, Cursor, VS Code, Windsurf, Zed) and offers to install MCP config for each.

### Example

```bash
$ fluree memory init

Memory store initialized at /path/to/project/.fluree-memory

Repo memories are stored in .fluree-memory/repo.ttl (git-tracked).
Commit this directory to share project knowledge with your team.

Detected AI coding tools:
  - Claude Code (already configured)
  - Cursor
  - VS Code (Copilot) (already configured)

Install MCP config for Cursor? [Y/n] Y
  Installed: .cursor/mcp.json
  Installed: .cursor/rules/fluree_rules.md

Configured 1 tool.
```

With `--yes`: auto-confirms all installations without prompting. In a non-interactive shell (piped stdin) without `--yes`, MCP installation is skipped with a message.

### Detection signals

`init` detects an IDE if **any** of these is true:

- A known binary is on `PATH` (`claude`, `code`, `cursor`, `windsurf`, `zed`).
- The IDE's app bundle is in `/Applications` or `~/Applications` (macOS).
- A home-dir marker exists (`~/.claude`, `~/.cursor`, `~/.vscode`, `~/.codeium/windsurf`, `~/.zed`, `~/.config/zed`).
- The IDE's user-config dir exists (`~/.config/Code`, `~/.config/Cursor`, …; on macOS: `~/Library/Application Support/<App>`; on Windows: `%APPDATA%\<App>`).

If your IDE is installed but missed (commonly: a fresh install that's never been launched and isn't on `PATH`), install MCP for it directly with `fluree memory mcp-install --ide <tool>`. See [memory/reference/ide-matrix.md](../memory/reference/ide-matrix.md) for the full table.

## fluree memory add

Store a new memory.

```bash
fluree memory add [OPTIONS]
```

### Options

| Option | Description |
|--------|-------------|
| `--kind <KIND>` | Memory kind: `fact`, `decision`, `constraint` (default: `fact`) |
| `--text <TEXT>` | Content text (or provide via stdin) |
| `--tags <T1,T2>` | **Required.** Comma-separated tags for categorization — the primary recall signal |
| `--refs <R1,R2>` | Comma-separated file/artifact references |
| `--severity <SEV>` | For constraints: `must`, `should`, `prefer` |
| `--scope <SCOPE>` | Scope: `repo` (default) or `user` |
| `--rationale <TEXT>` | Why this memory exists (available on any kind) |
| `--alternatives <TEXT>` | Alternatives considered (comma-separated) |
| `--format <FMT>` | Output format: `text` (default) or `json` |

### Examples

```bash
# Store a fact
fluree memory add --kind fact --text "Tests use cargo nextest" --tags testing,cargo

# Store a constraint with severity
fluree memory add --kind constraint --text "Never suppress dead code with underscore prefix" \
  --tags code-style --severity must

# Store from stdin
echo "The index format uses postcard encoding" | fluree memory add --kind fact --tags indexer

# Store a decision with rationale and alternatives
fluree memory add --kind decision --text "Use postcard for compact index encoding" \
  --rationale "no_std compatible, smaller output than bincode" \
  --alternatives "bincode, CBOR, MessagePack" --refs fluree-db-indexer/

# Store a fact with rationale
fluree memory add --kind fact --text "PSOT queries return supersets — post-filter required" \
  --rationale "B-tree range scan can't filter on non-key predicates" --tags query,index
```

Output (text):
```
Stored memory: mem:fact-01JDXYZ5A2B3C4D5E6F7G8H9J0
```

### Secret detection

If the content contains secrets (API keys, passwords, tokens, connection strings), they are automatically redacted and a warning is printed:

```
  warning: secrets detected in content — storing redacted version.
  Original content contained sensitive data that was replaced with [REDACTED].
Stored memory: mem:fact-01JDXYZ...
```

## fluree memory recall

Search and retrieve relevant memories ranked by score.

```bash
fluree memory recall <QUERY> [OPTIONS]
```

### Arguments

| Argument | Description |
|----------|-------------|
| `<QUERY>` | Natural language search query |

### Options

| Option | Description |
|--------|-------------|
| `-n, --limit <N>` | Maximum results per page (default: 3) |
| `--offset <N>` | Skip the first N results — use for pagination (default: 0) |
| `--kind <KIND>` | Filter to a specific memory kind |
| `--tags <T1,T2>` | Filter to memories with these tags |
| `--scope <SCOPE>` | Filter by scope: `repo` or `user` |
| `--format <FMT>` | Output: `text` (default), `json`, or `context` (XML for LLM) |

### Examples

```bash
# Basic recall (returns top 3)
fluree memory recall "how to run tests"

# Get the next page
fluree memory recall "how to run tests" --offset 3

# Return up to 10 results
fluree memory recall "error handling" -n 10

# Filter by kind and tags
fluree memory recall "error handling" --kind constraint --tags errors

# Output as XML context (for LLM injection)
fluree memory recall "testing patterns" --format context
```

Output (text):
```
Recall: "how to run tests" (2 matches)

1. [score: 13.0] mem:fact-01JDXYZ...
   Tests use cargo nextest
   Tags: testing, cargo

2. [score: 8.0] mem:fact-01JDABC...
   Integration tests use assert_cmd + predicates
   Tags: testing

  (showing results 1–3; use --offset 3 for more)
```

Output (context):
```xml
<memory-context>
  <memory id="mem:fact-01JDXYZ..." kind="fact" score="13.0">
    <content>Tests use cargo nextest</content>
    <tags>testing, cargo</tags>
  </memory>
  <pagination shown="1" offset="0" total_in_store="13" />
</memory-context>
```

When results are cut off, the pagination element includes a hint:

```xml
  <pagination shown="3" offset="0" limit="3" total_in_store="13">Results 1–3. Use offset=3 to retrieve more.</pagination>
```

## fluree memory update

Update a memory in place. Only the fields you provide are changed — the ID stays the same. History is tracked via git.

```bash
fluree memory update <ID> [OPTIONS]
```

### Options

| Option | Description |
|--------|-------------|
| `--text <TEXT>` | New content text |
| `--tags <T1,T2>` | New tags (replaces all existing) |
| `--refs <R1,R2>` | New artifact refs (replaces all existing) |
| `--format <FMT>` | Output: `text` or `json` |

### Example

```bash
fluree memory update mem:fact-01JDXYZ... --text "Tests use cargo nextest with --no-fail-fast"
```

Output:
```
Updated: mem:fact-01JDXYZ...
```

## fluree memory forget

Delete a memory by retracting all its triples.

```bash
fluree memory forget <ID>
```

Output:
```
Forgotten: mem:fact-01JDXYZ...
```

## fluree memory status

Show a summary of the memory store.

```bash
fluree memory status
```

Output:
```
Memory Store Status
  Total memories: 12
  Total tags:     25
  By kind:
    fact: 7
    decision: 2
    constraint: 3
```

## fluree memory export / import

Export all current (non-superseded) memories as JSON, or import from a file.

```bash
fluree memory export > memories.json
fluree memory import memories.json
```

## fluree memory mcp-install

Install MCP configuration for an IDE so agents can use memory tools.

```bash
fluree memory mcp-install [--ide <IDE>]
```

### Options

| Option | Description |
|--------|-------------|
| `--ide <IDE>` | Target IDE (auto-detected if omitted) |

Supported IDE values:

| Value | Config written | Notes |
|-------|----------------|-------|
| `claude-code` | `claude mcp add` (local scope → `~/.claude.json`) | Also appends to `CLAUDE.md` |
| `vscode` | `.vscode/mcp.json` (key: `servers`) | Also installs `.vscode/fluree_rules.md` |
| `cursor` | `.cursor/mcp.json` (key: `mcpServers`) | Also installs `.cursor/rules/fluree_rules.md` |
| `windsurf` | `~/.codeium/windsurf/mcp_config.json` (global) | — |
| `zed` | `.zed/settings.json` (key: `context_servers`) | Skips if JSONC (comments) detected |

Legacy aliases: `claude-vscode` and `github-copilot` map to `vscode`.

When `--ide` is omitted, the first unconfigured detected tool is used; defaults to `claude-code` if none detected.

### Example

```bash
fluree memory mcp-install --ide cursor
```

Output:
```
  Installed: .cursor/mcp.json
  Installed: .cursor/rules/fluree_rules.md
```

### Cursor notes (recommended config)

Cursor’s MCP configuration supports stdio servers with a `type` field and config interpolation like `${workspaceFolder}`. A portable repo-scoped setup looks like:

```json
{
  "mcpServers": {
    "fluree-memory": {
      "type": "stdio",
      "command": "fluree",
      "args": ["mcp", "serve", "--transport", "stdio"],
      "env": {
        "FLUREE_HOME": "${workspaceFolder}/.fluree"
      }
    }
  }
}
```

Setting `FLUREE_HOME` ensures the MCP server uses the current workspace’s `.fluree/` directory even if Cursor spawns the process from a different working directory. That keeps repo memory/logs under `<repo>/.fluree-memory/` instead of a global location.

### Troubleshooting: repo vs global memory

- **Repo-scoped expected**:
  - Memories: `<repo>/.fluree-memory/repo.ttl`
  - MCP log: `<repo>/.fluree-memory/.local/mcp.log` (should show `client initialized` after a full Cursor restart)
- **If it’s using global dirs on macOS**:
  - Memories/log: `~/Library/Application Support/.fluree-memory/...`
  - Fix: ensure your Cursor config sets `env.FLUREE_HOME = "${workspaceFolder}/.fluree"` and restart Cursor fully.

## See Also

- [Memory overview](../memory/README.md) — what it is, when to use it, how it fits into your workflow
- [Memory getting started](../memory/getting-started/README.md) — install, quickstart, and per-IDE setup guides
- [Memory concepts](../memory/concepts/README.md) — repo vs user memory, supersession, recall ranking, secrets
- [Memory guides](../memory/guides/README.md) — team workflows, rules-file customization, migrating from plain markdown
- [Memory reference](../memory/reference/README.md) — IDE support matrix, `mem:` schema, TTL file format
- [mcp](mcp.md) — MCP server for IDE agent integration


---

# cli/mcp.md

# fluree mcp

Model Context Protocol (MCP) server for IDE agent integration.

## Usage

```bash
fluree mcp <COMMAND>
```

## Subcommands

| Command | Description |
|---------|-------------|
| `serve` | Start the MCP server |

## fluree mcp serve

Start an MCP server that exposes developer memory tools to IDE agents.

```bash
fluree mcp serve [--transport <TRANSPORT>]
```

### Options

| Option | Description |
|--------|-------------|
| `--transport <TRANSPORT>` | Transport protocol: `stdio` (default) |

The `stdio` transport reads JSON-RPC requests from stdin and writes responses to stdout. This is the standard transport for IDE integration — the IDE spawns the process and communicates over pipes.

### Available tools

The MCP server exposes 6 tools:

| Tool | Description |
|------|-------------|
| `memory_add` | Store a new memory (fact, decision, constraint, preference, artifact) |
| `memory_recall` | Search and retrieve relevant memories as XML context. Accepts `query`, `limit` (default: 3), `offset` (default: 0), `kind`, `tags`, `scope`. Returns a `<pagination>` element indicating whether more results are available. |
| `memory_update` | Update (supersede) an existing memory |
| `memory_forget` | Delete a memory |
| `memory_status` | Show memory store summary |
| `kg_query` | Execute raw SPARQL against the memory graph |

The server auto-initializes the memory store on first tool call. No separate `fluree memory init` is needed.

### IDE configuration

The easiest way to configure your IDE is with `fluree memory mcp-install`:

```bash
fluree memory mcp-install --ide cursor
```

Or manually add to your IDE's MCP config:

```json
{
  "mcpServers": {
    "fluree-memory": {
      "type": "stdio",
      "command": "/path/to/fluree",
      "args": ["mcp", "serve", "--transport", "stdio"],
      "env": {
        "FLUREE_HOME": "${workspaceFolder}/.fluree"
      }
    }
  }
}
```

### Testing with JSON-RPC

To test the server directly, pipe JSON-RPC to stdin:

```bash
printf '%s\n' \
  '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"smoke","version":"0.0"}}}' \
  '{"jsonrpc":"2.0","method":"notifications/initialized","params":{}}' \
  '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}' \
  | fluree mcp serve --transport stdio
```

### Tracing

CLI tracing is disabled when running `fluree mcp serve` to avoid any log output on stderr that could interfere with the JSON-RPC protocol.

## See Also

- [memory](memory.md) — CLI commands for memory management
- [Memory: MCP server](../memory/concepts/mcp.md) — what the MCP server exposes and how agents use it
- [Memory getting started](../memory/getting-started/README.md) — per-IDE setup (Claude Code, Cursor, VS Code, Windsurf, Zed)
- [Memory IDE support matrix](../memory/reference/ide-matrix.md) — config paths and supported features per IDE


---

# cli/iceberg.md

# fluree iceberg

Manage Apache Iceberg table connections.

## Subcommands

| Subcommand | Description |
|------------|-------------|
| `map` | Map an Iceberg table as a graph source |
| `list` | List Iceberg-family graph sources (`Iceberg` and `R2RML`) |
| `info` | Show details for an Iceberg-family graph source |
| `drop` | Drop an Iceberg-family graph source |

## fluree iceberg map

Map an Iceberg table as a queryable graph source.

### Usage

```bash
fluree iceberg map <NAME> [OPTIONS]
```

### Arguments

| Argument | Description |
|----------|-------------|
| `<NAME>` | Graph source name (e.g., "warehouse-orders") |

### Options

**Catalog mode:**

| Option | Description |
|--------|-------------|
| `--mode <MODE>` | Catalog mode: `rest` (default) or `direct` |

**REST catalog mode options:**

| Option | Description |
|--------|-------------|
| `--catalog-uri <URI>` | REST catalog URI (required for rest mode) |
| `--table <ID>` | Table identifier in `namespace.table` format (required if not specified in R2RML mapping) |
| `--warehouse <NAME>` | Warehouse identifier |
| `--no-vended-credentials` | Disable vended credentials (enabled by default) |

**Direct S3 mode options:**

| Option | Description |
|--------|-------------|
| `--table-location <URI>` | S3 table location (required for direct mode, e.g., `s3://bucket/warehouse/ns/table`) |

**R2RML mapping:**

| Option | Description |
|--------|-------------|
| `--r2rml <PATH>` | R2RML mapping file (Turtle format, required). Defines how table rows become RDF triples. Table references come from the mapping's `rr:tableName` entries. |
| `--r2rml-type <TYPE>` | Mapping media type (e.g., `text/turtle`); inferred from extension if omitted |

**Authentication:**

| Option | Description |
|--------|-------------|
| `--auth-bearer <TOKEN>` | Bearer token for REST catalog authentication |
| `--oauth2-token-url <URL>` | OAuth2 token URL for client credentials auth |
| `--oauth2-client-id <ID>` | OAuth2 client ID |
| `--oauth2-client-secret <SECRET>` | OAuth2 client secret |

**S3 overrides:**

| Option | Description |
|--------|-------------|
| `--s3-region <REGION>` | S3 region override |
| `--s3-endpoint <URL>` | S3 endpoint override (for MinIO, LocalStack) |
| `--s3-path-style` | Use path-style S3 URLs |

**Other:**

| Option | Description |
|--------|-------------|
| `--remote <NAME>` | Execute against a remote server (by remote name) |
| `--branch <NAME>` | Branch name (defaults to "main") |

### Description

Maps an Apache Iceberg table as a graph source that can be queried using SPARQL or JSON-LD queries. The table is accessed read-only; Fluree does not modify the Iceberg table.

An R2RML mapping (`--r2rml`) is required to define how Iceberg table rows are transformed into RDF triples.

Two catalog modes are supported:

- **REST mode** (default): Connects to an Iceberg REST catalog (e.g., Apache Polaris) to discover table metadata. Supports vended credentials and warehouse selection.
- **Direct S3 mode**: Reads table metadata directly from S3 by resolving `version-hint.text` in the table's `metadata/` directory. No catalog server required.

### Examples

```bash
# REST catalog with R2RML mapping
fluree iceberg map airlines \
  --catalog-uri https://polaris.example.com/api/catalog \
  --r2rml mappings/airlines.ttl \
  --auth-bearer $POLARIS_TOKEN

# REST catalog with explicit table and warehouse
fluree iceberg map warehouse-orders \
  --catalog-uri https://polaris.example.com/api/catalog \
  --table sales.orders \
  --r2rml mappings/orders.ttl \
  --auth-bearer $POLARIS_TOKEN \
  --warehouse my-warehouse

# Direct S3 (no catalog server)
fluree iceberg map execution-log \
  --mode direct \
  --table-location s3://my-bucket/warehouse/logs/execution_log \
  --r2rml mappings/execution_log.ttl \
  --s3-region us-east-1

# OAuth2 authentication
fluree iceberg map orders \
  --catalog-uri https://polaris.example.com/api/catalog \
  --table sales.orders \
  --r2rml mappings/orders.ttl \
  --oauth2-token-url https://auth.example.com/token \
  --oauth2-client-id my-client \
  --oauth2-client-secret $CLIENT_SECRET

# Create the graph source on a remote Fluree server
fluree iceberg map warehouse-orders \
  --remote origin \
  --catalog-uri https://polaris.example.com/api/catalog \
  --table sales.orders \
  --r2rml mappings/orders.ttl
```

### Output

```
Mapped Iceberg table as R2RML graph source 'airlines:main'
  Table:       openflights.airlines
  Catalog:     https://polaris.example.com/api/catalog
  R2RML:       mappings/airlines.ttl
  TriplesMaps: 3
  Connection:  verified
  Mapping:     validated
```

### After Mapping

Once mapped, the graph source appears in standard commands:

```bash
# Listed alongside ledgers
fluree list

# Inspect configuration
fluree info warehouse-orders

# Query via SPARQL GRAPH pattern
fluree query mydb 'SELECT ?id ?total FROM <mydb:main> WHERE { GRAPH <warehouse-orders:main> { ?o ex:id ?id ; ex:total ?total } }'

# Remove the mapping
fluree drop warehouse-orders --force
```

### Feature Flag

Requires the `iceberg` feature flag. Without it, the command returns:
```
error: Iceberg support not compiled. Rebuild with `--features iceberg`.
```

## See Also

- [Iceberg / Parquet](../graph-sources/iceberg.md) - Iceberg integration details
- [R2RML](../graph-sources/r2rml.md) - R2RML mapping reference
- [list](list.md) - List ledgers and graph sources
- [info](info.md) - Show graph source details
- [drop](drop.md) - Remove a graph source

## fluree iceberg list

List Iceberg-family graph sources (`Iceberg` and `R2RML` types).

### Usage

```bash
fluree iceberg list [--remote <NAME>]
```

### Examples

```bash
# Local
fluree iceberg list

# Remote
fluree iceberg list --remote origin
```

## fluree iceberg info

Show details for an Iceberg-family graph source.

### Usage

```bash
fluree iceberg info <NAME> [--remote <NAME>]
```

### Examples

```bash
# Local
fluree iceberg info warehouse-orders

# Remote
fluree iceberg info warehouse-orders --remote origin
```

## fluree iceberg drop

Drop an Iceberg-family graph source. This command only targets Iceberg/R2RML graph sources; it does not fall back to dropping ledgers of the same name.

### Usage

```bash
fluree iceberg drop <NAME> --force [--remote <NAME>]
```

### Examples

```bash
# Local
fluree iceberg drop warehouse-orders --force

# Remote
fluree iceberg drop warehouse-orders --force --remote origin
```


---

# cli/completions.md

# fluree completions

Generate shell completions.

## Usage

```bash
fluree completions <SHELL>
```

## Arguments

| Argument | Description |
|----------|-------------|
| `<SHELL>` | Shell to generate completions for |

## Supported Shells

- `bash`
- `zsh`
- `fish`
- `powershell`
- `elvish`

## Description

Generates shell completion scripts that enable tab-completion for `fluree` commands, options, and arguments.

## Installation

### Bash

```bash
# Add to ~/.bashrc
eval "$(fluree completions bash)"

# Or save to a file
fluree completions bash > /etc/bash_completion.d/fluree
```

### Zsh

```bash
# Add to ~/.zshrc
eval "$(fluree completions zsh)"

# Or save to completions directory
fluree completions zsh > ~/.zfunc/_fluree
# Then add to ~/.zshrc: fpath=(~/.zfunc $fpath)
```

### Fish

```bash
fluree completions fish > ~/.config/fish/completions/fluree.fish
```

### PowerShell

```powershell
# Add to your PowerShell profile
fluree completions powershell | Out-String | Invoke-Expression
```

## Examples

```bash
# Generate bash completions
fluree completions bash

# Generate zsh completions and save
fluree completions zsh > ~/.zfunc/_fluree
```

## Usage After Installation

After installing completions, you can use tab to complete:

```bash
fluree <TAB>        # Shows all commands
fluree que<TAB>     # Completes to "query"
fluree query --<TAB> # Shows available options
```


---

# getting-started/README.md

# Getting Started

Welcome to Fluree! This section will guide you through the essential steps to start using Fluree for your graph database needs.

## Quick Navigation

### [Fluree for SQL Developers](fluree-for-sql-developers.md)

Coming from PostgreSQL, MySQL, or SQL Server? This guide maps SQL concepts to Fluree equivalents, shows the same operations in both languages, and highlights where Fluree gives you capabilities that relational databases don't have.

### [Quickstart: Run the Server](quickstart-server.md)

Get Fluree up and running in minutes. Learn how to:
- Install and run the Fluree server
- Configure basic settings
- Verify the server is running
- Access the HTTP API

### [Quickstart: Create a Ledger](quickstart-ledger.md)

Create your first ledger to store data. Learn how to:
- Create a new ledger using the API
- Understand ledger IDs and branching
- Set up initial configuration
- Verify ledger creation

### [Quickstart: Write Data](quickstart-write.md)

Start writing data to your ledger. Learn how to:
- Insert new entities (basic inserts)
- Upsert data (idempotent transactions; predicate-level replacement for supplied predicates)
- Update existing data (WHERE/DELETE/INSERT pattern)
- Understand JSON-LD transaction format

### [Quickstart: Query Data](quickstart-query.md)

Query your data using Fluree's powerful query languages. Learn how to:
- Write basic JSON-LD queries
- Write basic SPARQL queries
- Filter and select data
- Understand query results

### [Tutorial: End-to-End](tutorial-end-to-end.md)

Build a knowledge base that combines Fluree's differentiating features in one workflow:
- Full-text search with BM25 relevance ranking
- Time travel to compare current and historical state
- Branching to experiment safely
- Policies for role-based access control

### [Using Fluree as a Rust Library](rust-api.md)

Embed Fluree directly in your Rust applications. Learn how to:
- Add Fluree as a dependency in Cargo.toml
- Use the Rust API programmatically
- Implement common patterns (insert, query, update)
- Integrate BM25 and vector search
- Handle errors and configuration
- Write tests with Fluree

## What is Fluree?

Fluree is a temporal graph database that stores data as RDF triples with built-in support for:

- **Time Travel**: Query data as it existed at any point in time
- **Full-Text Search**: Integrated BM25 indexing for powerful text search
- **Vector Search**: Approximate nearest neighbor (ANN) queries
- **Policy Enforcement**: Fine-grained, data-level access control
- **Verifiable Data**: Cryptographically signed transactions
- **Graph Sources**: Integration with external data sources (Iceberg, R2RML)

## Learning Path

**For HTTP API users (server-based):**

1. **Bridge the gap**: [Fluree for SQL Developers](fluree-for-sql-developers.md) if coming from relational databases
2. **Start with the Server**: [Run the Server](quickstart-server.md) to get Fluree running
3. **Create Your First Ledger**: [Create a Ledger](quickstart-ledger.md) to set up your database
4. **Add Data**: [Write Data](quickstart-write.md) to insert your first entities
5. **Query Your Data**: [Query Data](quickstart-query.md) to retrieve and explore
6. **See it all together**: [End-to-End Tutorial](tutorial-end-to-end.md) — search, time travel, branching, and policies in one workflow
7. **Core Concepts**: Read [Concepts](../concepts/README.md) to understand Fluree's architecture
8. **Practical Guides**: Explore [Cookbooks](../guides/README.md) for search, time travel, branching, policies, and SHACL validation
9. **Deep Dive**: Explore [Query](../query/README.md), [Transactions](../transactions/README.md), and [Security](../security/README.md)
10. **Production Ready**: Review [Operations](../operations/README.md) for deployment guidance

**For Rust developers (embedded library):**

1. **Rust API Guide**: [Using Fluree as a Rust Library](rust-api.md) for embedding Fluree in your application
2. **Core Concepts**: [Concepts](../concepts/README.md) to understand how Fluree works
3. **Practical Guides**: [Cookbooks](../guides/README.md) for search, time travel, branching, policies, and validation
4. **Advanced Queries**: [Query](../query/README.md) for complex query patterns
5. **Transactions**: [Transactions](../transactions/README.md) for data modification patterns
6. **Production Ready**: [Operations](../operations/README.md) and [Dev Setup](../contributing/dev-setup.md)

## Prerequisites

- Familiarity with JSON format
- HTTP client (curl, Postman, or your programming language's HTTP library)
- No graph database or RDF experience required — [Fluree for SQL Developers](fluree-for-sql-developers.md) bridges the gap from relational databases

## Support and Resources

- **Documentation**: This documentation provides comprehensive coverage
- **API Reference**: See [HTTP API](../api/README.md) for endpoint details
- **Troubleshooting**: Check [Troubleshooting](../troubleshooting/README.md) for common issues

Let's get started!


---

# getting-started/fluree-for-sql-developers.md

# Fluree for SQL Developers

If you've spent years with PostgreSQL, MySQL, or SQL Server and are encountering a graph database for the first time, this guide bridges the gap. It maps SQL concepts you already know to their Fluree equivalents, shows you the same operations in both languages, and highlights where Fluree gives you capabilities that relational databases simply don't have.

## The mental model shift

In SQL, you design tables with fixed columns, then insert rows. In Fluree, you make statements about things — and those statements can describe anything, with any properties, at any time.

| SQL Concept | Fluree Equivalent | Key Difference |
|---|---|---|
| Database | Ledger | Immutable — every change is preserved |
| Table | Type (via `rdf:type`) | No fixed schema required; types are just labels |
| Row | Entity (identified by IRI) | An entity can have any properties, not just those in a "table" |
| Column | Predicate (property) | Not tied to a single type; any entity can use any property |
| Foreign key | Reference (IRI link) | Relationships are first-class, bidirectional, and traversable |
| Value | Object (literal or reference) | Typed values (string, integer, date, etc.) |
| Row (one fact) | Flake | A triple + provenance (graph, transaction time, assert/retract) |
| `NULL` | Absence | Properties simply don't exist if not set — no nulls |

### The flake: Fluree's atomic unit

Every fact in Fluree is stored as a **flake** — an extended triple that adds provenance. At its core, a flake is a statement: `subject → predicate → object`, plus metadata about *when* it was asserted, *which graph* it belongs to, and *whether it's an assertion or retraction*.

```
ex:alice  schema:name  "Alice"       (graph: default, t: 1, op: assert)
ex:alice  schema:age   30            (graph: default, t: 1, op: assert)
ex:alice  schema:knows ex:bob        (graph: default, t: 1, op: assert)
```

Think of it as: "Alice's name is Alice (added in transaction 1)." The provenance is what makes time travel and immutability possible — every change is a new flake, and retractions are recorded alongside assertions.

In SQL terms, imagine a universal table with columns `entity_id`, `attribute`, `value`, `graph`, `transaction`, `operation` — that can represent any data structure without DDL and preserves complete history.

> **Terminology note:** In RDF standards, the core unit is called a "triple" (subject-predicate-object). Fluree's "flake" extends the triple with temporal and provenance metadata. You'll see both terms in the documentation — "triple" when discussing the RDF data model, "flake" when discussing Fluree's storage and history.

## Side by side: common operations

### Creating structure

**SQL — Define a table:**
```sql
CREATE TABLE employees (
  id SERIAL PRIMARY KEY,
  name VARCHAR(255) NOT NULL,
  email VARCHAR(255) UNIQUE,
  department VARCHAR(100),
  salary DECIMAL(10,2),
  manager_id INTEGER REFERENCES employees(id)
);
```

**Fluree — Just insert data:**
```bash
fluree insert '
@prefix schema: <http://schema.org/> .
@prefix ex:     <http://example.org/> .

ex:alice  a schema:Person ;
  schema:name        "Alice Smith" ;
  schema:email       "alice@example.com" ;
  ex:department      "Engineering" ;
  ex:salary          125000 ;
  ex:reportsTo       ex:bob .

ex:bob  a schema:Person ;
  schema:name        "Bob Jones" ;
  schema:email       "bob@example.com" ;
  ex:department      "Engineering" .
'
```

There's no `CREATE TABLE`. Types and properties emerge from the data itself. You can add new properties to any entity at any time without migrations.

### Inserting data

**SQL:**
```sql
INSERT INTO employees (name, email, department, salary)
VALUES ('Carol Davis', 'carol@example.com', 'Marketing', 95000);
```

**Fluree (CLI):**
```bash
fluree insert '
@prefix schema: <http://schema.org/> .
@prefix ex:     <http://example.org/> .

ex:carol  a schema:Person ;
  schema:name        "Carol Davis" ;
  schema:email       "carol@example.com" ;
  ex:department      "Marketing" ;
  ex:salary          95000 .
'
```

**Fluree (HTTP API):**
```bash
curl -X POST http://localhost:8090/v1/fluree/insert?ledger=mydb:main \
  -H "Content-Type: application/ld+json" \
  -d '{
    "@context": {
      "schema": "http://schema.org/",
      "ex": "http://example.org/"
    },
    "@id": "ex:carol",
    "@type": "schema:Person",
    "schema:name": "Carol Davis",
    "schema:email": "carol@example.com",
    "ex:department": "Marketing",
    "ex:salary": 95000
  }'
```

### Basic queries

**SQL:**
```sql
SELECT name, email FROM employees WHERE department = 'Engineering';
```

**Fluree (SPARQL):**
```sparql
PREFIX schema: <http://schema.org/>
PREFIX ex: <http://example.org/>

SELECT ?name ?email
WHERE {
  ?person a schema:Person ;
          schema:name ?name ;
          schema:email ?email ;
          ex:department "Engineering" .
}
```

**Fluree (JSON-LD Query):**
```json
{
  "@context": {"schema": "http://schema.org/", "ex": "http://example.org/"},
  "select": ["?name", "?email"],
  "where": [
    {
      "@id": "?person", "@type": "schema:Person",
      "schema:name": "?name",
      "schema:email": "?email",
      "ex:department": "Engineering"
    }
  ]
}
```

### Joins

In SQL, joins are explicit operations. In Fluree, relationships are just triples — "joining" is following a link.

**SQL — Find employees and their managers:**
```sql
SELECT e.name AS employee, m.name AS manager
FROM employees e
JOIN employees m ON e.manager_id = m.id;
```

**Fluree (SPARQL):**
```sparql
PREFIX schema: <http://schema.org/>
PREFIX ex: <http://example.org/>

SELECT ?employee ?manager
WHERE {
  ?e schema:name ?employee ;
     ex:reportsTo ?m .
  ?m schema:name ?manager .
}
```

No `JOIN` keyword — you just follow the `ex:reportsTo` link from one entity to another. The database traverses relationships natively.

### Multi-hop relationships

This is where graphs shine. "Find everyone in Alice's reporting chain" requires recursive CTEs in SQL but is natural in a graph.

**SQL (recursive CTE):**
```sql
WITH RECURSIVE chain AS (
  SELECT id, name, manager_id FROM employees WHERE name = 'Alice Smith'
  UNION ALL
  SELECT e.id, e.name, e.manager_id
  FROM employees e JOIN chain c ON e.id = c.manager_id
)
SELECT name FROM chain;
```

**Fluree (SPARQL — property path):**
```sparql
PREFIX schema: <http://schema.org/>
PREFIX ex: <http://example.org/>

SELECT ?name
WHERE {
  ex:alice ex:reportsTo+ ?manager .
  ?manager schema:name ?name .
}
```

The `+` after `ex:reportsTo` means "follow this relationship one or more times." No recursion needed.

### Aggregation

**SQL:**
```sql
SELECT department, COUNT(*) as count, AVG(salary) as avg_salary
FROM employees
GROUP BY department
ORDER BY avg_salary DESC;
```

**Fluree (SPARQL):**
```sparql
PREFIX ex: <http://example.org/>

SELECT ?dept (COUNT(?person) AS ?count) (AVG(?salary) AS ?avg_salary)
WHERE {
  ?person ex:department ?dept ;
          ex:salary ?salary .
}
GROUP BY ?dept
ORDER BY DESC(?avg_salary)
```

### Updates

**SQL:**
```sql
UPDATE employees SET salary = 130000 WHERE name = 'Alice Smith';
```

**Fluree (SPARQL UPDATE):**
```sparql
PREFIX schema: <http://schema.org/>
PREFIX ex: <http://example.org/>

DELETE { ?person ex:salary ?oldSalary }
INSERT { ?person ex:salary 130000 }
WHERE  { ?person schema:name "Alice Smith" ; ex:salary ?oldSalary }
```

The `WHERE` finds Alice, `DELETE` removes the old salary, and `INSERT` adds the new one. This is atomic.

**Fluree (CLI — upsert for simpler cases):**
```bash
fluree upsert '{
  "@context": {"schema": "http://schema.org/", "ex": "http://example.org/"},
  "@id": "ex:alice",
  "ex:salary": 130000
}'
```

Upsert replaces the salary value if Alice already exists, or creates the entity if she doesn't.

### Deletes

**SQL:**
```sql
DELETE FROM employees WHERE name = 'Carol Davis';
```

**Fluree (SPARQL UPDATE):**
```sparql
PREFIX schema: <http://schema.org/>

DELETE { ?person ?p ?o }
WHERE  { ?person schema:name "Carol Davis" ; ?p ?o }
```

But here's the key difference: in SQL, the row is gone. In Fluree, the retraction is recorded — you can still query Carol's data at any previous point in time.

## What SQL can't do

These features have no relational equivalent:

### Time travel

Query data as it existed at any point in the past:

```bash
# What was Alice's salary before the raise?
fluree query --at 1 'PREFIX schema: <http://schema.org/>
PREFIX ex: <http://example.org/>

SELECT ?salary WHERE {
  ?person schema:name "Alice Smith" ; ex:salary ?salary .
}'
```

```bash
# Show the full history of salary changes
fluree history 'PREFIX schema: <http://schema.org/>
PREFIX ex: <http://example.org/>

SELECT ?salary ?t ?op WHERE {
  ?person schema:name "Alice Smith" ; ex:salary ?salary .
}'
```

In SQL, you'd need audit tables, temporal extensions, or trigger-based logging. In Fluree, every change is automatically preserved.

### Schema flexibility

Add new properties to any entity without `ALTER TABLE`:

```bash
# Alice now has a phone number — no migration needed
fluree insert '
@prefix schema: <http://schema.org/> .
@prefix ex:     <http://example.org/> .

ex:alice schema:telephone "+1-555-0100" .
'
```

Different entities of the same "type" can have different properties. There's no fixed set of columns.

### Branching

Fork your data to experiment without affecting production:

```bash
fluree branch create experiment
fluree use mydb:experiment

# Try risky changes on the branch
fluree update 'PREFIX ex: <http://example.org/>
DELETE { ?p ex:salary ?s }
INSERT { ?p ex:salary 200000 }
WHERE  { ?p ex:salary ?s }'

# Main branch is untouched
fluree query --ledger mydb:main 'SELECT ?name ?salary WHERE {
  ?p <http://schema.org/name> ?name ; <http://example.org/salary> ?salary
}'
```

### Triple-level access control

SQL databases give you table-level or row-level security. Fluree policies control access to individual facts:

```json
{
  "@id": "ex:hide-salary",
  "f:action": "query",
  "f:resource": { "f:predicate": "ex:salary" },
  "f:allow": false
}
```

This hides salary data from everyone unless another policy explicitly grants access. The same query returns different results for different users, automatically.

### Integrated full-text search

No need for Elasticsearch or Solr alongside your database:

```bash
fluree insert '{
  "@context": {"ex": "http://example.org/"},
  "@id": "ex:doc1",
  "ex:content": {
    "@value": "Fluree is a graph database with time travel and integrated search",
    "@type": "@fulltext"
  }
}'

fluree query '{
  "@context": {"ex": "http://example.org/"},
  "select": ["?id", "?score"],
  "where": [
    {"@id": "?id", "ex:content": "?text"},
    ["bind", "?score", "(fulltext ?text \"graph database search\")"],
    ["filter", "(> ?score 0)"]
  ],
  "orderBy": [["desc", "?score"]]
}'
```

## Common "but in SQL I would..." questions

**"How do I enforce NOT NULL?"**
Use [SHACL shapes](../guides/cookbook-shacl.md) to define constraints like required properties, value types, and cardinality.

**"How do I enforce UNIQUE?"**
Fluree supports [unique constraints](../ledger-config/unique-constraints.md) in the ledger configuration.

**"How do I do transactions?"**
Every Fluree transaction is atomic. Multiple operations in a single request either all succeed or all fail.

**"How do I create indexes?"**
Fluree automatically maintains four indexes (SPOT, POST, OPST, PSOT) that cover all query patterns. You don't need to create indexes manually.

**"How do I paginate?"**
Use `LIMIT` and `OFFSET`, just like SQL:
```sparql
SELECT ?name WHERE { ?p schema:name ?name }
ORDER BY ?name LIMIT 20 OFFSET 40
```

**"How do I do subqueries?"**
SPARQL supports subqueries natively:
```sparql
SELECT ?name ?avgSalary WHERE {
  ?person schema:name ?name ; ex:department ?dept .
  { SELECT ?dept (AVG(?s) AS ?avgSalary) WHERE { ?p ex:department ?dept ; ex:salary ?s } GROUP BY ?dept }
}
```

## Next steps

- [Quickstart: Write Data](quickstart-write.md) — Start writing data with the HTTP API
- [SPARQL Reference](../query/sparql.md) — Full SPARQL 1.1 query reference
- [JSON-LD Query](../query/jsonld-query.md) — Fluree's JSON-native query language
- [Concepts](../concepts/README.md) — Deeper understanding of Fluree's architecture
- [Time Travel](../concepts/time-travel.md) — Full guide to temporal queries


---

# getting-started/quickstart-server.md

# Quickstart: Run the Server

This guide will get the Fluree server running on your machine in minutes.

## Installation

### Option 1: Shell Installer (macOS / Linux)

```bash
curl --proto '=https' --tlsv1.2 -LsSf https://github.com/fluree/db/releases/latest/download/fluree-db-cli-installer.sh | sh
```

### Option 2: Homebrew (macOS / Linux)

```bash
brew install fluree/tap/fluree
```

### Option 3: PowerShell (Windows)

Open PowerShell and run:

```powershell
irm https://github.com/fluree/db/releases/latest/download/fluree-db-cli-installer.ps1 | iex
```

Then open a **new** PowerShell session and verify `fluree --version`. The installer adds `%USERPROFILE%\bin` to your `PATH`. The binary is unsigned, so Windows SmartScreen may prompt on first run — click **More info → Run anyway**.

### Option 4: Download Pre-built Binary

Download the latest release for your platform from [GitHub Releases](https://github.com/fluree/db/releases):

```bash
# Linux (x86_64)
curl -L https://github.com/fluree/db/releases/latest/download/fluree-db-cli-x86_64-unknown-linux-gnu.tar.xz | tar xJ
chmod +x fluree-db-cli-x86_64-unknown-linux-gnu/fluree

# macOS (Apple Silicon)
curl -L https://github.com/fluree/db/releases/latest/download/fluree-db-cli-aarch64-apple-darwin.tar.xz | tar xJ
chmod +x fluree-db-cli-aarch64-apple-darwin/fluree
```

### Option 5: Build from Source

If you have Rust installed:

```bash
# Clone the repository
git clone https://github.com/fluree/db.git
cd db

# Build the CLI (includes embedded server)
cargo build --release -p fluree-db-cli

# Binary will be at target/release/fluree
```

### Option 6: Docker

```bash
# Pull the image
docker pull fluree/server:latest

# Run the container
docker run -p 8090:8090 fluree/server:latest
```

For configuration (mounted JSON-LD/TOML config files, env vars, persistent volumes, S3+DynamoDB, query peers, full Compose example), see [Running with Docker](../operations/docker.md).

## Start the Server

### Memory Storage (Development)

Start the server with in-memory storage (data is lost on restart):

```bash
fluree server run
```

You should see output like:

```text
INFO fluree_db_server: Starting Fluree server
INFO fluree_db_server: Storage mode: memory
INFO fluree_db_server: Server listening on 0.0.0.0:8090
```

### File Storage (Persistent)

For persistent storage, specify a storage path:

```bash
fluree server run --storage-path /var/lib/fluree
```

### Custom Port

```bash
fluree server run --listen-addr 0.0.0.0:9090
```

### Debug Logging

```bash
fluree server run --log-level debug
```

## Verify Installation

### Check Server Health

```bash
curl http://localhost:8090/health
```

Expected response:

```json
{
  "status": "ok",
  "version": "4.0.4"
}
```

### Create a Ledger

```bash
curl -X POST http://localhost:8090/v1/fluree/create \
  -H "Content-Type: application/json" \
  -d '{"ledger": "test:main"}'
```

### Insert Data

```bash
curl -X POST "http://localhost:8090/v1/fluree/insert" \
  -H "Content-Type: application/json" \
  -H "fluree-ledger: test:main" \
  -d '{
    "@context": {"ex": "http://example.org/"},
    "@id": "ex:alice",
    "ex:name": "Alice"
  }'
```

### Query Data

```bash
curl -X POST "http://localhost:8090/v1/fluree/query" \
  -H "Content-Type: application/json" \
  -d '{
    "from": "test:main",
    "select": {"?s": ["*"]},
    "where": [["?s", "ex:name", "?name"]]
  }'
```

## Understanding the Server

### Endpoints

Default server endpoints:

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/health` | GET | Health check |
| `/v1/fluree/create` | POST | Create a ledger |
| `/v1/fluree/drop` | POST | Drop a ledger |
| `/v1/fluree/query` | GET/POST | Execute queries |
| `/v1/fluree/insert` | POST | Insert data |
| `/v1/fluree/update` | POST | Update with WHERE/DELETE/INSERT |
| `/v1/fluree/events` | GET | SSE event stream |

See the [API Reference](../api/endpoints.md) for complete endpoint documentation.

### Storage Modes

**Memory** (default):
- Fast, in-process storage
- Data lost on restart
- Best for development and testing

**File** (with `--storage-path`):
- Persistent local file storage
- Data survives restarts
- Best for single-server deployments

### Configuration

All options can be set via CLI flags or environment variables:

```bash
# CLI flag
fluree server run --storage-path /data --log-level debug

# Environment variables
export FLUREE_STORAGE_PATH=/data
export FLUREE_LOG_LEVEL=debug
fluree server run
```

See [Configuration](../operations/configuration.md) for all options.

## Common Configurations

### Development

```bash
fluree server run --log-level debug
```

### Production (Single Server)

```bash
fluree server run \
  --storage-path /var/lib/fluree \
  --indexing-enabled \
  --events-auth-mode required \
  --events-auth-trusted-issuers did:key:z6Mk...
```

### With Background Indexing

```bash
fluree server run \
  --storage-path /var/lib/fluree \
  --indexing-enabled
```

## Docker Deployment

For the full Docker guide — image internals, configuration via env vars vs mounted JSON-LD/TOML config files, persistent volumes, LRU cache and indexing tuning, S3+DynamoDB connection configs, query peers, and a production-ready Compose example — see [Running with Docker](../operations/docker.md).

Minimal persistent run:

```bash
docker run -d --name fluree \
  -p 8090:8090 \
  -v fluree-data:/var/lib/fluree \
  fluree/server:latest
```

## Troubleshooting

### Port Already in Use

```bash
# Use a different port
fluree server run --listen-addr 0.0.0.0:9090
```

### Permission Denied (File Storage)

```bash
sudo chown -R $USER:$USER /var/lib/fluree
chmod -R 755 /var/lib/fluree
```

### Server Won't Start

Check logs with debug level:

```bash
fluree server run --log-level debug
```

### Connection Refused

Verify the server is running and check the listen address:

```bash
# Listen on all interfaces (not just localhost)
fluree server run --listen-addr 0.0.0.0:8090
```

## Next Steps

Now that your server is running:

1. [Create a Ledger](quickstart-ledger.md) - Set up your first database
2. [Write Data](quickstart-write.md) - Insert your first records
3. [Query Data](quickstart-query.md) - Retrieve and explore your data

For production deployments:

- [Configuration](../operations/configuration.md) - All server options
- [Query Peers](../operations/query-peers.md) - Horizontal scaling
- [Admin Authentication](../api/endpoints.md#admin-authentication) - Protect admin endpoints


---

# getting-started/quickstart-ledger.md

# Quickstart: Create a Ledger

Ledgers are Fluree's fundamental unit of data organization—similar to databases in traditional systems. This guide shows you how to create your first ledger.

## Understanding Ledger IDs

Ledgers are identified by **ledger IDs** with the format `ledger-name:branch`:

- `mydb:main` - Primary branch of the "mydb" ledger
- `customers:dev` - Development branch of the "customers" ledger
- `inventory:prod` - Production branch

The default branch is `main`, so `mydb` is equivalent to `mydb:main`.

## Creating a Ledger

### Rust API (Library Usage)

When using Fluree as a Rust library, create ledgers explicitly with `create_ledger`:

```rust
let fluree = FlureeBuilder::memory().build_memory();

// Create a new ledger (returns LedgerState at t=0)
let ledger = fluree.create_ledger("mydb").await?;

// Now insert data
let result = fluree.graph("mydb:main")
    .transact()
    .insert(&data)
    .commit()
    .await?;
```

`create_ledger` registers the ledger in the nameservice and returns a genesis `LedgerState` ready for transactions. It returns `ApiError::LedgerExists` (HTTP 409) if the ledger already exists.

To load an existing ledger, use `ledger`:

```rust
let ledger = fluree.ledger("mydb:main").await?;
```

### HTTP API (Server Usage)

Via the HTTP API, create a ledger explicitly with `POST /v1/fluree/create`, then write data with `POST /v1/fluree/insert`.

#### Step 1: Create the Ledger

```bash
curl -X POST http://localhost:8090/v1/fluree/create \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb:main"}'
```

Response:

```json
{
  "ledger_id": "mydb:main",
  "t": 0,
  "tx-id": "fluree:tx:sha256:...",
  "commit": {"hash": ""}
}
```

#### Step 2: Insert Data

```bash
curl -X POST http://localhost:8090/v1/fluree/insert \
  -H "Content-Type: application/json" \
  -H "fluree-ledger: mydb:main" \
  -d '{
    "@context": {
      "ex": "http://example.org/ns/",
      "schema": "http://schema.org/"
    },
    "@graph": [
      {
        "@id": "ex:alice",
        "@type": "schema:Person",
        "schema:name": "Alice",
        "schema:email": "alice@example.org"
      }
    ]
  }'
```

Response:

```json
{
  "ledger_id": "mydb:main",
  "t": 1,
  "tx-id": "fluree:tx:sha256:...",
  "commit": {"hash": "bagaybqab..."}
}
```

The ledger `mydb:main` now has data!

## Verifying Ledger Creation

### Check Ledger Exists

```bash
curl http://localhost:8090/v1/fluree/exists/mydb:main
```

Response:

```json
{
  "ledger_id": "mydb:main",
  "exists": true
}
```

### Query the Ledger

Verify you can query the new ledger:

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "schema": "http://schema.org/"
    },
    "from": "mydb:main",
    "select": ["?name"],
    "where": [
      { "@id": "?person", "schema:name": "?name" }
    ]
  }'
```

Response:

```json
[
  { "name": "Alice" }
]
```

## Ledger Naming Best Practices

### Descriptive Names

Choose names that clearly indicate purpose:

Good examples:
- `customers:main`
- `inventory:prod`
- `analytics:warehouse`

Bad examples:
- `db1:main`
- `test:main`
- `data:main`

### Hierarchical Organization

Use slashes for logical grouping:

```text
tenant/app:main
tenant/app:dev
department/project:feature-x
```

### Branch Naming

Establish consistent branch naming conventions:

```text
mydb:main              - Production branch
mydb:dev               - Development branch
mydb:staging           - Staging branch
mydb:feature-auth      - Feature branch
mydb:bugfix-login      - Bug fix branch
```

## Working with Branches

### Creating a New Branch

Branches are independent ledgers. First create the branch, then transact data into it:

```bash
# Create the branch
curl -X POST http://localhost:8090/v1/fluree/create \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb:dev"}'

# Insert data into the branch
curl -X POST http://localhost:8090/v1/fluree/insert?ledger=mydb:dev \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "ex": "http://example.org/ns/",
      "schema": "http://schema.org/"
    },
    "@graph": [
      {
        "@id": "ex:bob",
        "@type": "schema:Person",
          "schema:name": "Bob"
      }
    ]
  }'
```

Now you have two independent ledgers:
- `mydb:main` (with Alice)
- `mydb:dev` (with Bob)

### Understanding Branch Independence

Branches are completely independent—changes in one don't affect the other:

```bash
# Query main branch
curl -X POST http://localhost:8090/v1/fluree/query \
  -d '{"from": "mydb:main", "select": ["?name"], "where": [{"@id": "?person", "schema:name": "?name"}]}'
# Returns: [{"name": "Alice"}]

# Query dev branch
curl -X POST http://localhost:8090/v1/fluree/query \
  -d '{"from": "mydb:dev", "select": ["?name"], "where": [{"@id": "?person", "schema:name": "?name"}]}'
# Returns: [{"name": "Bob"}]
```

## Ledger Metadata

Each ledger maintains metadata accessible via the nameservice:

- **commit_t**: Latest transaction time
- **index_t**: Latest indexed transaction time
- **commit_id**: ContentId (CID) of the latest commit
- **index_id**: ContentId (CID) of the latest index
- **default_context**: Default JSON-LD @context for the ledger

### Checking Ledger Status

```bash
curl http://localhost:8090/v1/fluree/info/mydb:main
```

Response:

```json
{
  "ledger_id": "mydb:main",
  "branch": "main",
  "commit_t": 1,
  "index_t": 1,
  "commit_id": "bafybeig...commitT1",
  "index_id": "bafybeig...indexT1",
  "created": "2024-01-22T10:30:00.000Z",
  "last_updated": "2024-01-22T10:30:05.000Z"
}
```

### Understanding Commit vs Index

- **commit_t**: Most recent transaction (always up-to-date)
- **index_t**: Most recent indexed snapshot (may lag behind commits)
- **Gap**: If `commit_t > index_t`, there's a "novelty layer" being indexed

See [Ledgers and Nameservice](../concepts/ledgers-and-nameservice.md) for details.

## Multi-Tenant Scenarios

For multi-tenant applications, use hierarchical naming:

```text
tenant1/app:main
tenant1/app:dev
tenant2/app:main
tenant2/app:dev
```

Or use separate ledgers per tenant:

```text
tenant1-customers:main
tenant1-orders:main
tenant2-customers:main
tenant2-orders:main
```

## Setting Default Context

A ledger may have a stored default JSON-LD `@context` that the **CLI** and **HTTP server** can auto-inject into queries that omit `@context` / `PREFIX`. Two ways to set it:

1. **At import time:** `fluree create --from data.ttl` captures `@prefix` declarations from the Turtle source and stores them as the default.
2. **Explicitly:** `fluree context set <ledger> <ctx.json>`, or `PUT /v1/fluree/context/{ledger...}` over HTTP.

Regular JSON-LD transactions (insert/update) do **not** update the default context — only the two paths above do.

```json
// One-time setup via the CLI:
// fluree context set mydb context.json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "schema": "http://schema.org/",
    "xsd": "http://www.w3.org/2001/XMLSchema#"
  }
}
```

After this, the CLI (`fluree query`) and the HTTP server query endpoint will inject the stored context into queries that don't supply their own `@context` / `PREFIX`. Direct `fluree-db-api` consumers do **not** get auto-injection — they must opt in via `Fluree::db_with_default_context(...)` or include `@context` in each query. See `docs/concepts/iri-and-context.md` for the full opt-in story.

## Common Patterns

### Development Workflow

```text
1. Create main branch: mydb:main
2. Create dev branch: mydb:dev
3. Develop and test in dev
4. Copy desired state to main (application logic)
5. Repeat
```

### Feature Branching

```text
1. Create feature branch: mydb:feature-x
2. Develop feature in isolation
3. Test thoroughly
4. Merge to main (via application logic)
5. Optionally retract feature branch
```

### Environment Separation

```text
mydb:dev      - Development environment
mydb:staging  - Staging environment
mydb:prod     - Production environment
```

## Troubleshooting

### Ledger Already Exists

If you try to query a ledger before it exists:

```text
Error: Ledger not found: mydb:main
```

Solution: Create the ledger with a transaction first.

### Permission Issues (File Storage)

If using file storage, ensure the server has write permissions:

```bash
# Check data directory permissions
ls -la /path/to/data

# Fix permissions if needed
sudo chown -R fluree:fluree /path/to/data
chmod -R 755 /path/to/data
```

### AWS Storage Issues

For AWS storage, verify credentials and bucket access:

```bash
# Test S3 access
aws s3 ls s3://your-fluree-bucket/

# Test DynamoDB access
aws dynamodb describe-table --table-name fluree-nameservice
```

## Next Steps

Now that you have a ledger:

1. [Write Data](quickstart-write.md) - Learn how to insert, upsert, and update data
2. [Query Data](quickstart-query.md) - Explore your data with queries
3. [Concepts: Ledgers](../concepts/ledgers-and-nameservice.md) - Deep dive into ledger architecture

## Related Documentation

- [Ledgers and Nameservice](../concepts/ledgers-and-nameservice.md) - Architectural details
- [Transactions](../transactions/README.md) - Writing data to ledgers
- [Storage Modes](../operations/storage.md) - Storage backend options


---

# getting-started/quickstart-write.md

# Quickstart: Write Data

This guide shows you how to write data to Fluree using three main patterns: insert, upsert, and update.

## Prerequisites

- Fluree server running (see [Run the Server](quickstart-server.md))
- A ledger created (see [Create a Ledger](quickstart-ledger.md))

## Understanding Fluree Transactions

Fluree stores data as RDF triples (subject-predicate-object). Transactions are submitted as JSON-LD documents that get converted to triples internally.

### Basic Transaction Structure

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "schema": "http://schema.org/"
  },
  "@graph": [
    {
      "@id": "ex:alice",
      "@type": "schema:Person",
      "schema:name": "Alice"
    }
  ]
}
```

This creates triples like:
```text
ex:alice  rdf:type        schema:Person
ex:alice  schema:name     "Alice"
```

## Insert: Adding New Data

The simplest operation is inserting new entities.

### Insert a Single Entity

```bash
curl -X POST http://localhost:8090/v1/fluree/insert?ledger=mydb:main \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "ex": "http://example.org/ns/",
      "schema": "http://schema.org/"
    },
    "@graph": [
      {
        "@id": "ex:alice",
        "@type": "schema:Person",
        "schema:name": "Alice",
        "schema:email": "alice@example.org",
        "schema:age": 30
      }
    ]
  }'
```

Response:

```json
{
  "t": 1,
  "timestamp": "2024-01-22T10:30:00.000Z",
  "commit_id": "bafybeig...commitT1",
  "flakes_added": 4,
  "flakes_retracted": 0
}
```

### Insert Multiple Entities

```bash
curl -X POST http://localhost:8090/v1/fluree/insert?ledger=mydb:main \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "ex": "http://example.org/ns/",
      "schema": "http://schema.org/"
    },
    "@graph": [
      {
        "@id": "ex:bob",
        "@type": "schema:Person",
        "schema:name": "Bob",
        "schema:email": "bob@example.org"
      },
      {
        "@id": "ex:carol",
        "@type": "schema:Person",
        "schema:name": "Carol",
        "schema:email": "carol@example.org"
      }
    ]
  }'
```

### Insert with Relationships

```bash
curl -X POST http://localhost:8090/v1/fluree/insert?ledger=mydb:main \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "ex": "http://example.org/ns/",
      "schema": "http://schema.org/"
    },
    "@graph": [
      {
        "@id": "ex:company-a",
        "@type": "schema:Organization",
        "schema:name": "Acme Corp"
      },
      {
        "@id": "ex:alice",
        "@type": "schema:Person",
        "schema:name": "Alice",
        "schema:worksFor": {"@id": "ex:company-a"}
      }
    ]
  }'
```

## Upsert: Idempotent Transactions

Upsert (update/insert) replaces values for the predicates you supply on an entity. If the entity doesn't exist, it's created.

### Basic Upsert

Use the dedicated `/upsert` endpoint:

```bash
curl -X POST "http://localhost:8090/v1/fluree/upsert?ledger=mydb:main" \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "ex": "http://example.org/ns/",
      "schema": "http://schema.org/"
    },
    "@graph": [
      {
        "@id": "ex:alice",
        "@type": "schema:Person",
        "schema:name": "Alice Smith",
        "schema:email": "alice.smith@example.org",
        "schema:age": 31
      }
    ]
  }'
```

This replaces existing values for the predicates included in the payload (for `ex:alice`, those are `@type`, `schema:name`, `schema:email`, `schema:age`).

### Upsert Behavior

**First transaction** (entity doesn't exist):
- Creates the entity with all specified properties

**Subsequent transactions** (entity exists):
- Retracts existing values for the supplied predicates
- Asserts new values for those predicates
- Leaves other predicates unchanged

### Use Cases for Upsert

Good for:
- Idempotent transactions (can retry safely)
- Syncing from external systems
- Replacing values for the predicates you supply
- Avoiding duplicate checks

Not good for:
- Conditional/targeted changes (use UPDATE instead)

## Update: Targeted Changes (WHERE/DELETE/INSERT)

For targeted changes to existing data, use the UPDATE pattern with WHERE/DELETE/INSERT.

### Basic Update

```bash
curl -X POST http://localhost:8090/v1/fluree/update?ledger=mydb:main \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "ex": "http://example.org/ns/",
      "schema": "http://schema.org/"
    },
    "where": [
      { "@id": "ex:alice", "schema:age": "?oldAge" }
    ],
    "delete": [
      { "@id": "ex:alice", "schema:age": "?oldAge" }
    ],
    "insert": [
      { "@id": "ex:alice", "schema:age": 32 }
    ]
  }'
```

This pattern:
1. **WHERE**: Finds matching data
2. **DELETE**: Retracts specific triples
3. **INSERT**: Asserts new triples

### Update Multiple Properties

```bash
curl -X POST http://localhost:8090/v1/fluree/update?ledger=mydb:main \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "ex": "http://example.org/ns/",
      "schema": "http://schema.org/"
    },
    "where": [
      { "@id": "ex:alice", "schema:name": "?name", "schema:email": "?email" }
    ],
    "delete": [
      { "@id": "ex:alice", "schema:name": "?name", "schema:email": "?email" }
    ],
    "insert": [
      { "@id": "ex:alice", "schema:name": "Alice Johnson", "schema:email": "alice.j@example.org" }
    ]
  }'
```

### Conditional Update

Only update if condition is met:

```bash
curl -X POST http://localhost:8090/v1/fluree/update?ledger=mydb:main \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "ex": "http://example.org/ns/",
      "schema": "http://schema.org/"
    },
    "where": [
      { "@id": "ex:alice", "schema:age": "?age" },
      { "@id": "?age", "@type": "xsd:integer" }
    ],
    "delete": [
      { "@id": "ex:alice", "schema:age": "?age" }
    ],
    "insert": [
      { "@id": "ex:alice", "schema:age": { "@value": "32", "@type": "xsd:integer" } }
    ]
  }'
```

### Adding Properties (Not Replacing)

To add a property without removing existing ones, use INSERT only:

```bash
curl -X POST http://localhost:8090/v1/fluree/update?ledger=mydb:main \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "ex": "http://example.org/ns/",
      "schema": "http://schema.org/"
    },
    "insert": [
      { "@id": "ex:alice", "schema:telephone": "+1-555-0100" }
    ]
  }'
```

This adds the telephone property without affecting other properties.

## Data Types

Fluree supports various data types through JSON-LD typing:

### Strings (Default)

```json
{
  "@id": "ex:alice",
  "schema:name": "Alice"
}
```

### Numbers

```json
{
  "@id": "ex:alice",
  "schema:age": 30,
  "schema:height": 1.68
}
```

### Booleans

```json
{
  "@id": "ex:alice",
  "schema:active": true
}
```

### Dates

```json
{
  "@id": "ex:alice",
  "schema:birthDate": {
    "@value": "1994-05-15",
    "@type": "xsd:date"
  }
}
```

### Timestamps

```json
{
  "@id": "ex:alice",
  "schema:lastLogin": {
    "@value": "2024-01-22T10:30:00Z",
    "@type": "xsd:dateTime"
  }
}
```

### References (Links to Other Entities)

```json
{
  "@id": "ex:alice",
  "schema:worksFor": { "@id": "ex:company-a" }
}
```

## Transaction Receipts

Every successful transaction returns a receipt with metadata:

```json
{
  "t": 5,
  "timestamp": "2024-01-22T10:30:00.000Z",
  "commit_id": "bafybeig...commitT5",
  "flakes_added": 3,
  "flakes_retracted": 2,
  "previous_commit_id": "bafybeig...commitT4"
}
```

Key fields:
- **t**: Transaction time (monotonically increasing)
- **timestamp**: ISO 8601 timestamp
- **commit_id**: Content-addressed identifier (CID) for the commit
- **flakes_added**: Number of triples added
- **flakes_retracted**: Number of triples removed
- **previous_commit_id**: ContentId of the previous commit (present when t > 1)

See [Commit Receipts](../transactions/commit-receipts.md) for details.

## Error Handling

### Transaction Errors

If a transaction fails, you'll receive an error response:

```json
{
  "error": "TransactionError",
  "message": "Invalid IRI: not a valid URI",
  "code": "INVALID_IRI"
}
```

Common errors:
- **INVALID_IRI**: Malformed IRIs
- **PARSE_ERROR**: Invalid JSON-LD syntax
- **TYPE_ERROR**: Type mismatch
- **CONSTRAINT_VIOLATION**: Data constraint violated

### Validation

Transactions are validated before being applied:
- JSON-LD syntax must be valid
- IRIs must be well-formed
- Types must be compatible
- References must resolve (optional)

## Best Practices

### 1. Use Appropriate Transaction Pattern

- **Insert**: New entities, no duplication concerns
- **Upsert**: Idempotent transactions, predicate-level replacement for supplied predicates
- **Update**: Targeted changes, preserve other properties

### 2. Choose Meaningful IRIs

Good:
```json
{"@id": "ex:user-12345"}
{"@id": "ex:product-widget-2024"}
```

Bad:
```json
{"@id": "ex:1"}
{"@id": "ex:thing"}
```

### 3. Use Consistent Namespaces

Define a clear namespace strategy:

```json
{
  "@context": {
    "app": "https://myapp.com/ns/",
    "schema": "http://schema.org/",
    "xsd": "http://www.w3.org/2001/XMLSchema#"
  }
}
```

### 4. Batch Related Changes

Include related entities in a single transaction:

```json
{
  "@graph": [
    {"@id": "ex:order-123", "ex:customer": {"@id": "ex:alice"}},
    {"@id": "ex:order-123", "ex:product": {"@id": "ex:widget"}},
    {"@id": "ex:order-123", "ex:quantity": 5}
  ]
}
```

### 5. Use Typed Literals

Be explicit about types for dates, numbers, etc.:

```json
{
  "@id": "ex:alice",
  "ex:birthDate": {
    "@value": "1994-05-15",
    "@type": "xsd:date"
  }
}
```

## Transaction Size Limits

Be aware of transaction size constraints:

- **Recommended**: < 1000 triples per transaction
- **Maximum**: Configurable (default: 10,000 triples)
- **Large imports**: Use batch processing

See [Indexing Side-Effects](../transactions/indexing-side-effects.md) for performance considerations.

## Next Steps

Now that you can write data:

1. [Query Data](quickstart-query.md) - Learn how to retrieve your data
2. [Transactions Overview](../transactions/overview.md) - Detailed transaction documentation
3. [JSON-LD Context](../concepts/iri-and-context.md) - Understanding @context

## Related Documentation

- [Insert](../transactions/insert.md) - Detailed insert documentation
- [Upsert](../transactions/upsert.md) - Detailed upsert documentation
- [Update](../transactions/update-where-delete-insert.md) - Detailed update documentation
- [Data Types](../concepts/datatypes.md) - Comprehensive type system guide


---

# getting-started/quickstart-query.md

# Quickstart: Query Data

This guide introduces you to querying data in Fluree using both JSON-LD Query and SPARQL.

## Prerequisites

- Fluree server running with data (complete previous quickstarts)
- Sample data from [Write Data](quickstart-write.md) guide

## Query Languages

Fluree supports two query languages:

- **JSON-LD Query**: Fluree's native JSON-based query language
- **SPARQL**: W3C standard RDF query language

Both provide access to the same data and features.

## JSON-LD Query

### Basic SELECT Query

Retrieve all person names:

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "schema": "http://schema.org/"
    },
    "from": "mydb:main",
    "select": ["?name"],
    "where": [
      { "@id": "?person", "schema:name": "?name" }
    ]
  }'
```

Response:

```json
[
  { "name": "Alice" },
  { "name": "Bob" },
  { "name": "Carol" }
]
```

### Query Multiple Properties

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "schema": "http://schema.org/"
    },
    "from": "mydb:main",
    "select": ["?name", "?email"],
    "where": [
      { "@id": "?person", "schema:name": "?name" },
      { "@id": "?person", "schema:email": "?email" }
    ]
  }'
```

Response:

```json
[
  { "name": "Alice", "email": "alice@example.org" },
  { "name": "Bob", "email": "bob@example.org" },
  { "name": "Carol", "email": "carol@example.org" }
]
```

### Filter Results

Query with a specific filter:

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "schema": "http://schema.org/"
    },
    "from": "mydb:main",
    "select": ["?name", "?age"],
    "where": [
      { "@id": "?person", "schema:name": "?name" },
      { "@id": "?person", "schema:age": "?age" }
    ],
    "filter": "?age > 25"
  }'
```

### Query Specific Entity

Query a specific entity by IRI:

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "ex": "http://example.org/ns/",
      "schema": "http://schema.org/"
    },
    "from": "mydb:main",
    "select": ["?name", "?email", "?age"],
    "where": [
      { "@id": "ex:alice", "schema:name": "?name" },
      { "@id": "ex:alice", "schema:email": "?email" },
      { "@id": "ex:alice", "schema:age": "?age" }
    ]
  }'
```

### Query with Relationships

Follow links between entities:

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "schema": "http://schema.org/"
    },
    "from": "mydb:main",
    "select": ["?personName", "?companyName"],
    "where": [
      { "@id": "?person", "schema:name": "?personName" },
      { "@id": "?person", "schema:worksFor": "?company" },
      { "@id": "?company", "schema:name": "?companyName" }
    ]
  }'
```

Response:

```json
[
  { "personName": "Alice", "companyName": "Acme Corp" }
]
```

## SPARQL

### Basic SELECT Query

The same queries in SPARQL syntax:

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/sparql-query" \
  -d '
    PREFIX schema: <http://schema.org/>
    
    SELECT ?name
    FROM <mydb:main>
    WHERE {
      ?person schema:name ?name .
    }
  '
```

### Query Multiple Properties

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/sparql-query" \
  -d '
    PREFIX schema: <http://schema.org/>
    
    SELECT ?name ?email
    FROM <mydb:main>
    WHERE {
      ?person schema:name ?name .
      ?person schema:email ?email .
    }
  '
```

### Filter Results

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/sparql-query" \
  -d '
    PREFIX schema: <http://schema.org/>
    
    SELECT ?name ?age
    FROM <mydb:main>
    WHERE {
      ?person schema:name ?name .
      ?person schema:age ?age .
      FILTER (?age > 25)
    }
  '
```

### Query with Relationships

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/sparql-query" \
  -d '
    PREFIX schema: <http://schema.org/>
    
    SELECT ?personName ?companyName
    FROM <mydb:main>
    WHERE {
      ?person schema:name ?personName .
      ?person schema:worksFor ?company .
      ?company schema:name ?companyName .
    }
  '
```

## Time Travel Queries

Query historical data using time specifiers.

### Query at Specific Transaction

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "schema": "http://schema.org/"
    },
    "from": "mydb:main@t:1",
    "select": ["?name"],
    "where": [
      { "@id": "?person", "schema:name": "?name" }
    ]
  }'
```

This shows data as it existed at transaction 1.

### Query at ISO Timestamp

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "schema": "http://schema.org/"
    },
    "from": "mydb:main@iso:2024-01-22T10:00:00Z",
    "select": ["?name"],
    "where": [
      { "@id": "?person", "schema:name": "?name" }
    ]
  }'
```

### Query at Commit ContentId

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "schema": "http://schema.org/"
    },
    "from": "mydb:main@commit:bafybeig...",
    "select": ["?name"],
    "where": [
      { "@id": "?person", "schema:name": "?name" }
    ]
  }'
```

See [Time Travel](../concepts/time-travel.md) for comprehensive details.

## History Queries

Track changes to entities over time by specifying a time range in the `from` clause.

### Entity History

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "ex": "http://example.org/ns/",
      "schema": "http://schema.org/"
    },
    "from": "mydb:main@t:1",
    "to": "mydb:main@t:latest",
    "select": ["?name", "?age", "?t", "?op"],
    "where": [
      { "@id": "ex:alice", "schema:name": { "@value": "?name", "@t": "?t", "@op": "?op" } },
      { "@id": "ex:alice", "schema:age": "?age" }
    ],
    "orderBy": "?t"
  }'
```

The `@t` annotation binds the transaction time, and `@op` binds the operation type as a boolean (`true` = assert, `false` = retract).

Response shows all changes:

```json
[
  ["Alice", 30, 1, true],
  ["Alice", 30, 5, false],
  ["Alicia", 31, 5, true]
]
```

### Property History

Track changes to a specific property:

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "ex": "http://example.org/ns/",
      "schema": "http://schema.org/"
    },
    "from": "mydb:main@t:1",
    "to": "mydb:main@t:latest",
    "select": ["?age", "?t", "?op"],
    "where": [
      { "@id": "ex:alice", "schema:age": { "@value": "?age", "@t": "?t", "@op": "?op" } }
    ],
    "orderBy": "?t"
  }'
```

Response:

```json
[
  [30, 1, true],
  [30, 5, false],
  [31, 5, true]
]
```

## Aggregations

### Count Results

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/sparql-query" \
  -d '
    PREFIX schema: <http://schema.org/>
    
    SELECT (COUNT(?person) AS ?count)
    FROM <mydb:main>
    WHERE {
      ?person schema:name ?name .
    }
  '
```

### Average, Min, Max

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/sparql-query" \
  -d '
    PREFIX schema: <http://schema.org/>
    
    SELECT (AVG(?age) AS ?avgAge) (MIN(?age) AS ?minAge) (MAX(?age) AS ?maxAge)
    FROM <mydb:main>
    WHERE {
      ?person schema:age ?age .
    }
  '
```

## Limiting Results

### JSON-LD Query Limit

```json
{
  "@context": {
    "schema": "http://schema.org/"
  },
  "from": "mydb:main",
  "select": ["?name"],
  "where": [
    { "@id": "?person", "schema:name": "?name" }
  ],
  "limit": 10
}
```

### SPARQL Limit and Offset

```sparql
PREFIX schema: <http://schema.org/>

SELECT ?name
FROM <mydb:main>
WHERE {
  ?person schema:name ?name .
}
ORDER BY ?name
LIMIT 10
OFFSET 20
```

## Ordering Results

### JSON-LD Query Order

```json
{
  "@context": {
    "schema": "http://schema.org/"
  },
  "from": "mydb:main",
  "select": ["?name", "?age"],
  "where": [
    { "@id": "?person", "schema:name": "?name" },
    { "@id": "?person", "schema:age": "?age" }
  ],
  "orderBy": ["?age"]
}
```

### SPARQL Order

```sparql
PREFIX schema: <http://schema.org/>

SELECT ?name ?age
FROM <mydb:main>
WHERE {
  ?person schema:name ?name .
  ?person schema:age ?age .
}
ORDER BY DESC(?age)
```

## Multi-Ledger Queries

Query across multiple ledgers:

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "schema": "http://schema.org/"
    },
    "from": ["customers:main", "orders:main"],
    "select": ["?customerName", "?orderTotal"],
    "where": [
      { "@id": "?customer", "schema:name": "?customerName" },
      { "@id": "?order", "schema:customer": "?customer" },
      { "@id": "?order", "schema:totalPrice": "?orderTotal" }
    ]
  }'
```

See [Datasets](../query/datasets.md) for comprehensive multi-graph query documentation.

## Understanding Query Results

### JSON-LD Query Results

Results are returned as an array of objects:

```json
[
  { "name": "Alice", "age": 30 },
  { "name": "Bob", "age": 25 }
]
```

### SPARQL Results

SPARQL returns results in SPARQL JSON format:

```json
{
  "head": {
    "vars": ["name", "age"]
  },
  "results": {
    "bindings": [
      {
        "name": { "type": "literal", "value": "Alice" },
        "age": { "type": "literal", "value": "30", "datatype": "http://www.w3.org/2001/XMLSchema#integer" }
      },
      {
        "name": { "type": "literal", "value": "Bob" },
        "age": { "type": "literal", "value": "25", "datatype": "http://www.w3.org/2001/XMLSchema#integer" }
      }
    ]
  }
}
```

See [Output Formats](../query/output-formats.md) for format details.

## Query Performance Tips

### 1. Use Specific Patterns

More specific patterns are faster:

Good:
```json
{ "@id": "ex:alice", "schema:name": "?name" }
```

Less efficient:
```json
{ "@id": "?person", "?predicate": "?value" }
```

### 2. Filter Early

Apply filters in WHERE clauses when possible:

```json
"where": [
  { "@id": "?person", "schema:age": "?age" }
],
"filter": "?age > 25"
```

### 3. Limit Results

Always use LIMIT for large result sets:

```json
"limit": 100
```

### 4. Use Indexes

Queries leverage automatic indexes. Structure queries to take advantage:
- Subject-based lookups are fast
- Predicate-based lookups are fast
- Complex graph patterns may be slower

See [Explain Plans](../query/explain.md) for query optimization.

## Common Query Patterns

### Find All Types

```sparql
SELECT DISTINCT ?type
FROM <mydb:main>
WHERE {
  ?entity a ?type .
}
```

### Find All Predicates

```sparql
SELECT DISTINCT ?predicate
FROM <mydb:main>
WHERE {
  ?subject ?predicate ?object .
}
```

### Inverse Relationships

Find what points to an entity:

```sparql
SELECT ?source ?predicate
FROM <mydb:main>
WHERE {
  ?source ?predicate <http://example.org/ns/alice> .
}
```

### Optional Properties

Query with optional values:

```sparql
PREFIX schema: <http://schema.org/>

SELECT ?name ?email ?phone
FROM <mydb:main>
WHERE {
  ?person schema:name ?name .
  ?person schema:email ?email .
  OPTIONAL { ?person schema:telephone ?phone }
}
```

## Error Handling

### Query Errors

Common query errors:

```json
{
  "error": "QueryError",
  "message": "Ledger not found: mydb:main",
  "code": "LEDGER_NOT_FOUND"
}
```

```json
{
  "error": "ParseError",
  "message": "Invalid JSON-LD: unexpected token",
  "code": "PARSE_ERROR"
}
```

### Empty Results

Empty result set (not an error):

```json
[]
```

## Next Steps

Now that you can query data:

1. **Learn Advanced Queries**: Explore [JSON-LD Query](../query/jsonld-query.md) and [SPARQL](../query/sparql.md) documentation
2. **Understand Time Travel**: Deep dive into [Time Travel](../concepts/time-travel.md)
3. **Optimize Queries**: Read about [Explain Plans](../query/explain.md)
4. **Multi-Graph Queries**: Learn about [Datasets](../query/datasets.md)

## Related Documentation

- [JSON-LD Query](../query/jsonld-query.md) - Complete JSON-LD query reference
- [SPARQL](../query/sparql.md) - Complete SPARQL reference
- [Output Formats](../query/output-formats.md) - Result format options
- [Time Travel](../concepts/time-travel.md) - Historical queries
- [Graph Crawl](../query/graph-crawl.md) - Graph traversal


---

# getting-started/tutorial-end-to-end.md

# Tutorial: Building a Knowledge Base with Fluree

This tutorial walks through a realistic scenario — building a team knowledge base — to show how Fluree's differentiating features work together. You'll use time travel, full-text search, branching, and access control in a single workflow.

**Time:** ~20 minutes
**Prerequisites:** Fluree installed and running (`fluree init && fluree server run`)

## Step 1: Create the ledger and add data

```bash
fluree create knowledge-base
fluree use knowledge-base
```

Insert some articles and team members:

```bash
fluree insert '
@prefix schema: <http://schema.org/> .
@prefix ex:     <http://example.org/> .
@prefix f:      <https://ns.flur.ee/db#> .

ex:alice  a schema:Person ;
  schema:name "Alice Chen" ;
  ex:role     "engineer" ;
  ex:team     "platform" .

ex:bob  a schema:Person ;
  schema:name "Bob Martinez" ;
  ex:role     "engineer" ;
  ex:team     "platform" .

ex:carol  a schema:Person ;
  schema:name "Carol White" ;
  ex:role     "manager" ;
  ex:team     "platform" .

ex:doc1  a ex:Article ;
  schema:name    "Deployment Runbook" ;
  schema:author  ex:alice ;
  ex:team        "platform" ;
  ex:visibility  "internal" ;
  ex:content     "Step 1: Check the monitoring dashboard. Step 2: Run the database migration script. Step 3: Deploy the new container image using the CI pipeline."^^f:fullText .

ex:doc2  a ex:Article ;
  schema:name    "Onboarding Guide" ;
  schema:author  ex:bob ;
  ex:team        "platform" ;
  ex:visibility  "public" ;
  ex:content     "Welcome to the platform team. This guide covers setting up your development environment, accessing the database, and deploying your first service."^^f:fullText .

ex:doc3  a ex:Article ;
  schema:name    "Incident Response Playbook" ;
  schema:author  ex:carol ;
  ex:team        "platform" ;
  ex:visibility  "confidential" ;
  ex:content     "During a production incident, the on-call engineer should check database health, review recent deployments, and escalate if the service is not recovering within 15 minutes."^^f:fullText .
'
```

Verify the data is there:

```bash
fluree query --format table 'PREFIX schema: <http://schema.org/>
PREFIX ex: <http://example.org/>

SELECT ?title ?author_name ?visibility
WHERE {
  ?doc a ex:Article ;
       schema:name ?title ;
       schema:author ?author ;
       ex:visibility ?visibility .
  ?author schema:name ?author_name .
}
ORDER BY ?title'
```

```
┌─────────────────────────────┬───────────────┬──────────────┐
│ title                       │ author_name   │ visibility   │
├─────────────────────────────┼───────────────┼──────────────┤
│ Deployment Runbook          │ Alice Chen    │ internal     │
│ Incident Response Playbook  │ Carol White   │ confidential │
│ Onboarding Guide            │ Bob Martinez  │ public       │
└─────────────────────────────┴───────────────┴──────────────┘
```

This is transaction `t=1`. Remember this — we'll come back to it.

## Step 2: Full-text search

The article content was inserted with the `@fulltext` datatype, so it's automatically indexed for BM25 relevance scoring. Search for articles about deployments:

```bash
fluree query '{
  "@context": {"schema": "http://schema.org/", "ex": "http://example.org/"},
  "select": ["?title", "?score"],
  "where": [
    {
      "@id": "?doc", "@type": "ex:Article",
      "ex:content": "?content",
      "schema:name": "?title"
    },
    ["bind", "?score", "(fulltext ?content \"database deployment\")"],
    ["filter", "(> ?score 0)"]
  ],
  "orderBy": [["desc", "?score"]],
  "limit": 10
}'
```

Results are ranked by relevance — the deployment runbook and incident playbook both mention deployments and databases, while the onboarding guide has a weaker match.

You can combine search with graph filters. Find only **public** articles matching the search:

```bash
fluree query '{
  "@context": {"schema": "http://schema.org/", "ex": "http://example.org/"},
  "select": ["?title", "?score"],
  "where": [
    {
      "@id": "?doc", "@type": "ex:Article",
      "ex:content": "?content",
      "schema:name": "?title",
      "ex:visibility": "public"
    },
    ["bind", "?score", "(fulltext ?content \"database deployment\")"],
    ["filter", "(> ?score 0)"]
  ],
  "orderBy": [["desc", "?score"]]
}'
```

Search results participate in standard graph joins and filters — no separate search service needed.

## Step 3: Update data and use time travel

Let's update the deployment runbook with a new version:

```bash
fluree update 'PREFIX schema: <http://schema.org/>
PREFIX ex: <http://example.org/>
PREFIX f: <https://ns.flur.ee/db#>

DELETE { ex:doc1 ex:content ?old }
INSERT { ex:doc1 ex:content "Step 1: Check the monitoring dashboard and verify all health checks pass. Step 2: Run the database migration script with --dry-run first. Step 3: Deploy the new container image. Step 4: Verify the deployment in staging before promoting to production."^^f:fullText }
WHERE  { ex:doc1 ex:content ?old }'
```

Now query the **current** version:

```bash
fluree query 'PREFIX schema: <http://schema.org/>
PREFIX ex: <http://example.org/>

SELECT ?content WHERE { ex:doc1 ex:content ?content }'
```

And query the **original** version using time travel:

```bash
fluree query --at 1 'PREFIX schema: <http://schema.org/>
PREFIX ex: <http://example.org/>

SELECT ?content WHERE { ex:doc1 ex:content ?content }'
```

The `--at 1` flag queries the data as it was after transaction 1 — before the update. Both versions coexist in the same ledger.

You can also see the full change history:

```bash
fluree history 'PREFIX schema: <http://schema.org/>
PREFIX ex: <http://example.org/>

SELECT ?content ?t ?op WHERE { ex:doc1 ex:content ?content }'
```

Each result includes `?t` (the transaction number) and `?op` (whether it was an assertion or retraction). You see the original content retracted and the new content asserted, with exact timestamps.

**Use cases this enables:**
- **Audit trails** — Who changed what, when?
- **Rollback** — See what the data looked like before a bad change
- **Compliance** — Prove what was known at a specific point in time
- **Debugging** — Compare current vs. historical state to find when a problem was introduced

## Step 4: Branch to experiment safely

Suppose you want to reorganize the knowledge base — maybe split articles into categories, or restructure ownership. You don't want to affect the production data while experimenting.

Create a branch:

```bash
fluree branch create reorganize
fluree use knowledge-base:reorganize
```

On the branch, add categories and reorganize:

```bash
fluree insert '
@prefix schema: <http://schema.org/> .
@prefix ex:     <http://example.org/> .

ex:doc1 ex:category "operations" .
ex:doc2 ex:category "onboarding" .
ex:doc3 ex:category "operations" .
'
```

```bash
fluree update 'PREFIX schema: <http://schema.org/>
PREFIX ex: <http://example.org/>

DELETE { ex:doc3 ex:visibility "confidential" }
INSERT { ex:doc3 ex:visibility "internal" }
WHERE  { ex:doc3 ex:visibility "confidential" }'
```

Verify the branch has the changes:

```bash
fluree query --format table 'PREFIX schema: <http://schema.org/>
PREFIX ex: <http://example.org/>

SELECT ?title ?category ?visibility
WHERE {
  ?doc a ex:Article ;
       schema:name ?title ;
       ex:category ?category ;
       ex:visibility ?visibility .
}
ORDER BY ?title'
```

The main branch is untouched:

```bash
fluree query --ledger knowledge-base:main 'PREFIX ex: <http://example.org/>
PREFIX schema: <http://schema.org/>

SELECT ?title ?visibility
WHERE {
  ?doc a ex:Article ; schema:name ?title ; ex:visibility ?visibility .
  OPTIONAL { ?doc ex:category ?cat }
  FILTER(!BOUND(?cat))
}
ORDER BY ?title'
```

No categories on main — the branch is fully isolated.

When you're happy with the changes, merge back:

```bash
fluree branch merge reorganize
fluree use knowledge-base:main
```

Now main has the categories and the visibility change. The branch can continue for future experiments or be dropped:

```bash
fluree branch drop reorganize
```

## Step 5: Add access control

Now let's add policies so that different users see different articles based on their role and team.

Insert policies into the ledger:

```bash
fluree insert '{
  "@context": {
    "f": "https://ns.flur.ee/db#",
    "ex": "http://example.org/",
    "schema": "http://schema.org/"
  },
  "@graph": [
    {
      "@id": "ex:policy-public-read",
      "@type": "f:Policy",
      "f:action": "query",
      "f:resource": { "ex:visibility": "public" },
      "f:allow": true
    },
    {
      "@id": "ex:policy-team-internal",
      "@type": "f:Policy",
      "f:subject": "?user",
      "f:action": "query",
      "f:resource": {
        "ex:visibility": "internal",
        "ex:team": "?team"
      },
      "f:condition": [
        { "@id": "?user", "ex:team": "?team" }
      ],
      "f:allow": true
    },
    {
      "@id": "ex:policy-manager-confidential",
      "@type": "f:Policy",
      "f:subject": "?user",
      "f:action": "query",
      "f:resource": {
        "ex:visibility": "confidential",
        "ex:team": "?team"
      },
      "f:condition": [
        { "@id": "?user", "ex:team": "?team", "ex:role": "manager" }
      ],
      "f:allow": true
    }
  ]
}'
```

These three policies create a layered access model:

1. **Public articles** — visible to everyone
2. **Internal articles** — visible only to members of the same team
3. **Confidential articles** — visible only to managers on the same team

Query as Alice (engineer, platform team):

```bash
fluree query '{
  "@context": {"schema": "http://schema.org/", "ex": "http://example.org/"},
  "select": ["?title", "?visibility"],
  "where": [
    {"@id": "?doc", "@type": "ex:Article", "schema:name": "?title", "ex:visibility": "?visibility"}
  ],
  "opts": {"identity": "ex:alice"}
}'
```

Alice sees the public onboarding guide and the internal deployment runbook, but **not** the confidential incident playbook.

Query as Carol (manager, platform team):

```bash
fluree query '{
  "@context": {"schema": "http://schema.org/", "ex": "http://example.org/"},
  "select": ["?title", "?visibility"],
  "where": [
    {"@id": "?doc", "@type": "ex:Article", "schema:name": "?title", "ex:visibility": "?visibility"}
  ],
  "opts": {"identity": "ex:carol"}
}'
```

Carol sees all three articles, including the confidential one.

The same query, different results, based on who's asking — enforced by the database, not application code.

## Step 6: Combine everything

Now let's use all features together. Carol (manager) searches for articles about "database" in the knowledge base, with policies applied, and compares what she sees now vs. what existed before the reorganization:

**Current state, with policy:**
```bash
fluree query '{
  "@context": {"schema": "http://schema.org/", "ex": "http://example.org/"},
  "select": ["?title", "?visibility", "?score"],
  "where": [
    {
      "@id": "?doc", "@type": "ex:Article",
      "ex:content": "?content",
      "schema:name": "?title",
      "ex:visibility": "?visibility"
    },
    ["bind", "?score", "(fulltext ?content \"database\")"],
    ["filter", "(> ?score 0)"]
  ],
  "orderBy": [["desc", "?score"]],
  "opts": {"identity": "ex:carol"}
}'
```

**Historical state (before runbook was updated):**
```bash
fluree query '{
  "@context": {"schema": "http://schema.org/", "ex": "http://example.org/"},
  "from": "knowledge-base:main@t:1",
  "select": ["?title", "?score"],
  "where": [
    {
      "@id": "?doc", "@type": "ex:Article",
      "ex:content": "?content",
      "schema:name": "?title"
    },
    ["bind", "?score", "(fulltext ?content \"database\")"],
    ["filter", "(> ?score 0)"]
  ],
  "orderBy": [["desc", "?score"]]
}'
```

In a single database, you've combined:
- **Full-text search** — ranked by relevance
- **Access control** — Carol sees confidential articles, others wouldn't
- **Time travel** — compare current vs. historical content
- **Branching** — experimented with reorganization without risk

## What you've learned

| Feature | What it gave you |
|---|---|
| **Ledger** | A single place for all knowledge base data |
| **Full-text search** | BM25-ranked article discovery, integrated in queries |
| **Time travel** | Complete audit trail, historical comparison, rollback capability |
| **Branching** | Safe experimentation without affecting production |
| **Policies** | Automatic access control based on team and role |
| **SPARQL + JSON-LD** | Two query languages accessing the same engine |

## Next steps

- [Search Cookbook](../guides/cookbook-search.md) — Deeper guide to BM25 and vector search
- [Time Travel Cookbook](../guides/cookbook-time-travel.md) — Practical time-travel patterns
- [Branching Cookbook](../guides/cookbook-branching.md) — Branch/merge workflows
- [Policies Cookbook](../guides/cookbook-policies.md) — Access control patterns
- [SPARQL Reference](../query/sparql.md) — Full SPARQL 1.1 reference
- [JSON-LD Query](../query/jsonld-query.md) — Fluree's native query language


---

# getting-started/rust-api.md

# Using Fluree as a Rust Library

This guide shows how to use Fluree programmatically in your Rust applications by depending on the `fluree-db-api` crate.

## Overview

Fluree can be embedded directly in Rust applications, giving you a powerful graph database without requiring a separate server process. This is ideal for:

- Desktop applications
- Edge computing
- Embedded systems
- Library/framework integration
- Testing and development

## Add Dependency

Add Fluree to your `Cargo.toml`:

```toml
[dependencies]
fluree-db-api = { path = "../fluree-db-api" }
tokio = { version = "1", features = ["full"] }
```

Note: Replace `path` with version when published to crates.io:
```toml
[dependencies]
fluree-db-api = "0.1"
```

### Features

Available feature flags:

- `native` (default) - File storage support
- `credential` (default in server/CLI) - DID/JWS/VerifiableCredential support for signed queries and transactions
- `shacl` (default in server/CLI) - SHACL constraint validation
- `iceberg` (default in server/CLI) - Apache Iceberg/R2RML graph source support
- `aws` - AWS-backed storage support (S3, storage-backed nameservice). Enables `FlureeBuilder::s3()` and S3-based JSON-LD configs.
- `ipfs` - IPFS-backed storage via Kubo HTTP RPC
- `vector` - Embedded vector similarity search (HNSW indexes via usearch)
- `search-remote-client` - Remote search service client (HTTP client for remote BM25 and vector search services)
- `aws-testcontainers` - Opt-in LocalStack-backed S3/DynamoDB tests (auto-start via testcontainers)
- `full` - Convenience bundle: `native`, `credential`, `iceberg`, `shacl`, `ipfs`

## Quick Start

### Basic Setup

```rust
use fluree_db_api::{FlureeBuilder, Result};

#[tokio::main]
async fn main() -> Result<()> {
    // Create a memory-backed Fluree instance
    let fluree = FlureeBuilder::memory().build_memory();

    // Create a new ledger
    let ledger = fluree.create_ledger("mydb").await?;

    println!("Ledger created at t={}", ledger.t());

    Ok(())
}
```

### With File Storage

```rust
use fluree_db_api::{FlureeBuilder, Result};

#[tokio::main]
async fn main() -> Result<()> {
    // Use file-backed storage for persistence
    let fluree = FlureeBuilder::file("./data").build()?;

    // Create a new ledger (or load an existing one)
    let ledger = fluree.create_ledger("mydb").await?;

    // Load an existing ledger by ID (`name:branch`)
    let ledger = fluree.ledger("mydb:main").await?;

    Ok(())
}
```

### Bulk import (high throughput)

For initial ledger bootstraps (large Turtle or JSON-LD datasets), Fluree exposes a bulk import
pipeline as a first-class Rust API:

```rust
use fluree_db_api::{FlureeBuilder, Result};

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::file("./data").build()?;

    // `chunks_dir` can be:
    // - a directory containing *.ttl, *.trig, or *.jsonld files (sorted lexicographically), OR
    // - a single .ttl or .jsonld file.
    // Directories must contain a single format (no mixing Turtle and JSON-LD).
    let result = fluree
        .create("dblp:main")
        .import("./chunks_dir")
        .threads(8)          // parallel TTL parsing; commits remain serial
        .build_index(true)   // write an index root and publish it
        .publish_every(50)   // nameservice checkpoints during long imports (0 disables)
        .cleanup(true)       // delete tmp import files on success
        .execute()
        .await?;

    println!(
        "import complete: t={}, flakes={}, root={:?}",
        result.t, result.flake_count, result.root_id
    );

    // Query normally after import (loads the published V2 root from CAS).
    let view = fluree.view("dblp:main").await?;
    let qr = fluree
        .query(&view, "SELECT * WHERE { ?s ?p ?o } LIMIT 10")
        .await?;
    println!("rows={}", qr.batches.iter().map(|b| b.len()).sum::<usize>());

    Ok(())
}
```

**Temporary files:** the bulk import pipeline uses a session-scoped `tmp_import/` directory and
removes it only on full success (unless `.cleanup(false)` is set). On failure, it keeps the
session directory and logs its path for debugging.

### With S3 Storage

Requires `fluree-db-api` feature `aws` and standard AWS credential/region configuration.

```rust
use fluree_db_api::{FlureeBuilder, Result};

#[tokio::main]
async fn main() -> Result<()> {
    // LocalStack/MinIO: endpoint is required
    let fluree = FlureeBuilder::s3("my-bucket", "http://localhost:4566")
        .build_client()
        .await?;

    let ledger = fluree.create_ledger("mydb").await?;
    println!("Ledger created at t={}", ledger.t());
    Ok(())
}
```

**S3 Express One Zone note:** for directory buckets (`--x-s3` suffix), omit `s3Endpoint` in JSON-LD config and let the SDK handle it.

## Connection Configuration (JSON-LD)

For advanced configuration (tiered storage, address identifier routing, DynamoDB nameservice,
environment variable indirection), use `FlureeBuilder::from_json_ld()` to parse a JSON-LD config
and build from it. The typed builder methods (`build()`, `build_memory()`, `build_s3()`) and
the type-erased `build_client()` all share the same underlying construction logic.

See also: [JSON-LD connection configuration reference](../reference/connection-config-jsonld.md).

```rust
use fluree_db_api::{FlureeBuilder, Result};
use serde_json::json;

#[tokio::main]
async fn main() -> Result<()> {
    let cfg = json!({
        "@context": {"@base": "https://ns.flur.ee/config/connection/", "@vocab": "https://ns.flur.ee/system#"},
        "@graph": [
            {"@id": "s3Index", "@type": "Storage", "s3Bucket": {"envVar": "INDEX_BUCKET"}, "s3Endpoint": {"envVar": "S3_ENDPOINT"}},
            {"@id": "conn", "@type": "Connection", "indexStorage": {"@id": "s3Index"}}
        ]
    });
    // from_json_ld parses the config into builder settings; build_client() constructs
    // a type-erased FlureeClient suitable for runtime-determined backends.
    let fluree = FlureeBuilder::from_json_ld(&cfg)?.build_client().await?;
    Ok(())
}
```

### Environment variables (`ConfigurationValue`)

Any string/number config value can be specified directly or via a `ConfigurationValue` object:

```json
{
  "s3Bucket": { "envVar": "FLUREE_S3_BUCKET", "defaultVal": "my-bucket" },
  "cacheMaxMb": { "envVar": "FLUREE_CACHE_MAX_MB", "defaultVal": "1024" }
}
```

### Supported JSON-LD fields (Rust)

Connection node:
- `parallelism`
- `cacheMaxMb`
- `indexStorage`, `commitStorage`
- `primaryPublisher` (publisher node)

Storage node:
- File: `filePath`, `AES256Key`
- S3: `s3Bucket`, `s3Prefix`, `s3Endpoint`, `s3ReadTimeoutMs`, `s3WriteTimeoutMs`, `s3ListTimeoutMs`, `s3MaxRetries`, `s3RetryBaseDelayMs`, `s3RetryMaxDelayMs`, `s3MaxConcurrentRequests`

Publisher node:
- DynamoDB nameservice: `dynamodbTable`, `dynamodbRegion`, `dynamodbEndpoint`, `dynamodbTimeoutMs`
- Storage-backed nameservice: `storage` (reference to a Storage node)

## Core Patterns

### The Graph API

The primary API revolves around `fluree.graph(graph_ref)`, which returns a lazy `Graph` handle.
No I/O occurs until a terminal method (`.execute()`, `.commit()`, `.load()`) is called.

Use `graph(...).query()` when the target may be a mapped graph source as well as a native ledger. If the query body itself carries `"from"` / `FROM`, use `query_from()`. The lower-level `fluree.db(...)` + `fluree.query(&view, ...)` path is for materialized native ledger snapshots, not graph source aliases.

**When I/O happens:**
- `.execute()` / `.execute_formatted()` / `.execute_tracked()` — loads the graph from storage, then runs the query (each call reloads)
- `.commit()` — loads the cached ledger handle, stages, and commits
- `.stage()` — loads the ledger and stages without committing
- `.load()` — loads the graph once, returning a `GraphSnapshot` for repeated queries without reloading

```rust
// Lazy query — loads graph and executes in one step
let result = fluree.graph("mydb:main")
    .query()
    .sparql("SELECT ?name WHERE { ?s <http://schema.org/name> ?name }")
    .execute()
    .await?;

// Lazy transact + commit
let out = fluree.graph("mydb:main")
    .transact()
    .insert(&data)
    .commit()
    .await?;

// Materialize for reuse (avoids reloading on each query)
let db = fluree.graph("mydb:main").load().await?;
let r1 = db.query().sparql("SELECT ...").execute().await?;
let r2 = db.query().jsonld(&q).execute().await?;

// Time travel
let result = fluree.graph_at("mydb:main", TimeSpec::AtT(42))
    .query()
    .jsonld(&q)
    .execute()
    .await?;
```

### Insert Data

```rust
use fluree_db_api::{FlureeBuilder, Result};
use serde_json::json;

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::memory().build_memory();
    let ledger = fluree.create_ledger("mydb").await?;

    // Insert JSON-LD data using the Graph API
    let data = json!({
        "@context": {
            "schema": "http://schema.org/",
            "ex": "http://example.org/ns/"
        },
        "@graph": [
            {
                "@id": "ex:alice",
                "@type": "schema:Person",
                "schema:name": "Alice",
                "schema:email": "alice@example.org",
                "schema:age": 30
            },
            {
                "@id": "ex:bob",
                "@type": "schema:Person",
                "schema:name": "Bob",
                "schema:email": "bob@example.org",
                "schema:age": 25
            }
        ]
    });

    let result = fluree.graph("mydb:main")
        .transact()
        .insert(&data)
        .commit()
        .await?;

    println!("Transaction committed");

    Ok(())
}
```

### Query Data with JSON-LD Query

```rust
use fluree_db_api::{FlureeBuilder, Result};
use serde_json::json;

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::memory().build_memory();
    let ledger = fluree.create_ledger("mydb").await?;

    // Insert test data first (see Insert Data above)
    // ...

    // Query with JSON-LD using the lazy Graph API
    let query = json!({
        "select": ["?name", "?email"],
        "where": [
            { "@id": "?person", "@type": "schema:Person" },
            { "@id": "?person", "schema:name": "?name" },
            { "@id": "?person", "schema:email": "?email" },
            { "@id": "?person", "schema:age": "?age" }
        ],
        "filter": "?age > 25"
    });

    let result = fluree.graph("mydb:main")
        .query()
        .jsonld(&query)
        .execute_formatted()
        .await?;

    println!("Query results: {}",
        serde_json::to_string_pretty(&result)?);

    Ok(())
}
```

### Query Data with SPARQL

```rust
use fluree_db_api::{FlureeBuilder, Result};

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::memory().build_memory();
    let ledger = fluree.create_ledger("mydb").await?;

    // Insert test data first (see Insert Data above)
    // ...

    // Query with SPARQL using the lazy Graph API
    let sparql = r#"
        PREFIX schema: <http://schema.org/>

        SELECT ?name ?email
        WHERE {
            ?person a schema:Person .
            ?person schema:name ?name .
            ?person schema:email ?email .
            ?person schema:age ?age .
            FILTER (?age > 25)
        }
        ORDER BY ?name
    "#;

    let result = fluree.graph("mydb:main")
        .query()
        .sparql(sparql)
        .execute_formatted()
        .await?;

    println!("Results: {}",
        serde_json::to_string_pretty(&result)?);

    Ok(())
}
```

### Streaming Query Results (NDJSON)

The buffered `.query()` paths above collect the whole result set into a
`QueryResult` before returning. For large result sets or long-running queries,
the streaming API instead formats each batch as a newline-delimited JSON
("NDJSON") record and pushes it through an `mpsc` channel, so rows reach the
consumer incrementally and bytes keep flowing while the query runs. This is the
in-process producer behind the server's
[`/stream/query` endpoint](../api/streaming-query.md); see that reference for
the record protocol (`head` → `row`* → terminal `end`/`error`) and the stable
error codes.

The flow is two steps, mirroring the server handler:

1. **`plan_stream_query(&db, &input)`** — parse, validate, and plan. Parse
   errors and shapes the streaming path does not support (ASK,
   CONSTRUCT/DESCRIBE, `selectOne`, JSON-LD hydration, history) surface here,
   *before* any record is emitted — so a caller can map them to a clean error
   up front.
2. **`run_stream_query(ledger, plan, tracker, options, tx)`** — typically
   `tokio::spawn`ed. It sends a `head` record, then one `row` record per result
   row as batches arrive, then exactly one terminal record (`end` on success,
   `error` on failure). It takes an owned `LedgerState` so it can outlive the
   call site. Backpressure is natural: a full channel suspends the producer.

```rust
use fluree_db_api::{
    FlureeBuilder, GraphDb, OwnedStreamQuery, QueryExecutionOptions, Result, Tracker,
    TrackingOptions,
};
use bytes::Bytes;
use std::sync::Arc;
use tokio::sync::mpsc;

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = Arc::new(FlureeBuilder::file("./data").build()?);

    // Plan against a borrowed GraphDb, then move the owned LedgerState into the
    // spawned producer (GraphDb borrows the state, so plan first).
    let ledger_state = fluree
        .ledger_cached("mydb:main")
        .await?
        .snapshot()
        .await
        .to_ledger_state();

    let input = OwnedStreamQuery::Sparql(
        "SELECT ?s ?name WHERE { ?s <http://schema.org/name> ?name }".to_string(),
    );
    // JSON-LD form: OwnedStreamQuery::JsonLd(serde_json::Value)

    let plan = {
        let db = GraphDb::from_ledger_state(&ledger_state);
        fluree.plan_stream_query(&db, &input).await? // 4xx-worthy errors surface here
    };

    let (tx, mut rx) = mpsc::channel::<Bytes>(64);
    let producer = Arc::clone(&fluree);
    tokio::spawn(async move {
        producer
            .run_stream_query(
                ledger_state,
                plan,
                Tracker::new(TrackingOptions::all_enabled()), // time + fuel on records
                QueryExecutionOptions::new(),                 // optional cancellation handle
                tx,
            )
            .await;
    });

    // Each `Bytes` is one (or more) NDJSON line(s). Treat the absence of a
    // terminal `end`/`error` record as a truncated stream, not success.
    while let Some(chunk) = rx.recv().await {
        print!("{}", String::from_utf8_lossy(&chunk));
    }
    Ok(())
}
```

> **Heartbeats** are injected by the *transport* layer (the HTTP server), not by
> `run_stream_query`. An embedder consuming the channel directly gets only
> `head`/`row`/terminal records; add your own keep-alive if you tunnel the
> stream through an idle-timing proxy.

**Multi-ledger / `from` / policy queries** use the dataset variant. Build a
policy-wrapped `DataSetDb` first, then plan and run against it:

```rust
// JSON-LD with `from`/`fromNamed`, multi-ledger, or an identity/policy input:
let dataset = fluree.build_stream_dataset(&query_json).await?;
let input = OwnedStreamQuery::JsonLd(query_json);
let plan = fluree.plan_stream_query_dataset(&dataset, &input).await?;
// fluree.run_stream_query_dataset(dataset, plan, tracker, options, tx).await;
```

`build_stream_dataset` reads the spec from the query body's `from`/`fromNamed`;
`build_stream_dataset_from_spec` / `build_stream_dataset_for_sparql` take an
explicit `DatasetSpec` or extract one from SPARQL `FROM`/`FROM NAMED`. Per-graph
policy rides on the runtime dataset and is enforced identically to the buffered
dataset path.

### Update Data

```rust
use fluree_db_api::{FlureeBuilder, Result};
use serde_json::json;

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::memory().build_memory();
    let ledger = fluree.create_ledger("mydb").await?;

    // Update using WHERE/DELETE/INSERT pattern
    let update = json!({
        "@context": { "schema": "http://schema.org/" },
        "where": [
            { "@id": "?person", "schema:name": "Alice" },
            { "@id": "?person", "schema:age": "?oldAge" }
        ],
        "delete": [
            { "@id": "?person", "schema:age": "?oldAge" }
        ],
        "insert": [
            { "@id": "?person", "schema:age": 31 }
        ]
    });

    let result = fluree.graph("mydb:main")
        .transact()
        .update(&update)
        .commit()
        .await?;

    println!("Updated successfully");

    Ok(())
}
```

### SPARQL UPDATE

Use SPARQL UPDATE syntax for transactions:

```rust
use fluree_db_api::{
    FlureeBuilder, Result,
    parse_sparql, lower_sparql_update, NamespaceRegistry, TxnOpts,
    SparqlQueryBody,
};

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::file("./data").build()?;

    // Get a cached ledger handle
    let handle = fluree.ledger_cached("mydb:main").await?;

    // SPARQL UPDATE string
    let sparql = r#"
        PREFIX ex: <http://example.org/ns/>

        DELETE {
            ?person ex:age ?oldAge .
        }
        INSERT {
            ?person ex:age 31 .
        }
        WHERE {
            ?person ex:name "Alice" .
            ?person ex:age ?oldAge .
        }
    "#;

    // Parse SPARQL
    let parse_output = parse_sparql(sparql);
    if parse_output.has_errors() {
        // Handle parse errors
        for diag in parse_output.diagnostics.iter().filter(|d| d.is_error()) {
            eprintln!("Parse error: {}", diag.message);
        }
        return Err(fluree_db_api::ApiError::Internal("SPARQL parse error".into()));
    }

    let ast = parse_output.ast.unwrap();

    // Extract the UPDATE operation
    let update_op = match &ast.body {
        SparqlQueryBody::Update(op) => op,
        _ => return Err(fluree_db_api::ApiError::Internal("Expected SPARQL UPDATE".into())),
    };

    // Get namespace registry from the ledger
    let snapshot = handle.snapshot().await;
    let mut ns = NamespaceRegistry::from_db(&snapshot.snapshot);

    // Lower SPARQL UPDATE to Txn IR
    let txn = lower_sparql_update(update_op, &ast.prologue, &mut ns, TxnOpts::default())?;

    // Execute the transaction
    let result = fluree.stage(&handle)
        .txn(txn)
        .execute()
        .await?;

    println!("SPARQL UPDATE committed at t={}", result.receipt.t);

    Ok(())
}
```

**Supported SPARQL UPDATE operations:**
- `INSERT DATA` - Insert ground triples
- `DELETE DATA` - Delete specific triples
- `DELETE WHERE` - Delete matching patterns
- `DELETE/INSERT WHERE` - Full update with patterns

See [SPARQL UPDATE](../query/sparql.md#sparql-update) for syntax details.

### Stage and Preview Changes

```rust
use fluree_db_api::{FlureeBuilder, Result};
use serde_json::json;

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::memory().build_memory();
    let ledger = fluree.create_ledger("mydb").await?;

    let data = json!({
        "@context": {"ex": "http://example.org/ns/"},
        "@graph": [{"@id": "ex:alice", "ex:name": "Alice"}]
    });

    // Stage without committing
    let staged = fluree.graph("mydb:main")
        .transact()
        .insert(&data)
        .stage()
        .await?;

    // Query the staged state to preview changes
    let preview_query = json!({
        "select": ["?name"],
        "where": [{"@id": "ex:alice", "ex:name": "?name"}]
    });

    let preview = staged.query()
        .jsonld(&preview_query)
        .execute()
        .await?;

    println!("Preview: {} rows", preview.row_count());

    Ok(())
}
```

**Note:** `StagedGraph` currently supports querying only. Staging on top of a staged transaction and committing from a `StagedGraph` are not yet supported.

### Export Data

Stream ledger data as Turtle, N-Triples, N-Quads, TriG, or JSON-LD using the builder API:

```rust
use fluree_db_api::{FlureeBuilder, Result};
use fluree_db_api::export::ExportFormat;
use std::io::BufWriter;
use std::fs::File;

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::file("./data").build()?;

    // Export as Turtle to a file
    let file = File::create("backup.ttl").unwrap();
    let mut writer = BufWriter::new(file);
    let stats = fluree.export("mydb")
        .format(ExportFormat::Turtle)
        .write_to(&mut writer)
        .await?;
    println!("Exported {} triples", stats.triples_written);

    // Export as JSON-LD with custom prefixes
    let mut buf = Vec::new();
    let stats = fluree.export("mydb")
        .format(ExportFormat::JsonLd)
        .context(&serde_json::json!({"ex": "http://example.org/"}))
        .write_to(&mut buf)
        .await?;

    // Export all graphs as N-Quads (dataset export)
    let stats = fluree.export("mydb")
        .format(ExportFormat::NQuads)
        .all_graphs()
        .to_stdout()
        .await?;

    Ok(())
}
```

All formats stream directly from the binary SPOT index. Memory usage is O(leaflet size) for line-oriented formats and O(largest subject) for JSON-LD, regardless of dataset size.

**Builder methods:**
- `.format(ExportFormat)` — output format (default: Turtle)
- `.all_graphs()` — include all named graphs including system graphs (requires TriG or NQuads)
- `.graph("iri")` — export a specific named graph by IRI
- `.as_of(TimeSpec)` — time-travel export (transaction number, ISO-8601 datetime, or commit CID prefix)
- `.context(&json)` — override prefix map (default: ledger's context from nameservice)
- `.write_to(&mut writer)` — stream to any `Write` sink
- `.to_stdout()` — convenience for stdout output

See also: [CLI export](../cli/export.md) for command-line usage.

### Materialize for Reuse

When you need to run multiple queries against the same snapshot, materialize a `GraphSnapshot` once:

```rust
use fluree_db_api::{FlureeBuilder, Result};
use serde_json::json;

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::memory().build_memory();

    // Load once, query many times
    let db = fluree.graph("mydb:main").load().await?;

    let r1 = db.query()
        .sparql("SELECT ?name WHERE { ?s <http://schema.org/name> ?name }")
        .execute()
        .await?;

    let q2 = json!({
        "select": ["?email"],
        "where": [{"@id": "?s", "schema:email": "?email"}]
    });
    let r2 = db.query()
        .jsonld(&q2)
        .execute()
        .await?;

    // Access the underlying view if needed
    let view = db.view();

    Ok(())
}
```

## Advanced Usage

### Ledger Caching

Ledger caching is enabled by default on all `FlureeBuilder` constructors. When caching is active, `fluree.ledger()` returns a cached handle and subsequent calls avoid reloading from storage:

```rust
use fluree_db_api::{FlureeBuilder, Result};

#[tokio::main]
async fn main() -> Result<()> {
    // Caching is on by default — no extra call needed
    let fluree = FlureeBuilder::file("./data").build()?;

    // First call loads from storage
    let ledger = fluree.ledger("mydb:main").await?;

    // Subsequent calls return cached state (fast)
    let ledger2 = fluree.ledger("mydb:main").await?;

    Ok(())
}
```

To **disable** caching (e.g., for a CLI tool that runs once and exits):

```rust
let fluree = FlureeBuilder::file("./data")
    .without_ledger_caching()
    .build()?;
```

#### Disconnecting Ledgers

Use `disconnect_ledger` to release a ledger from the connection cache. This forces a fresh load on the next access:

```rust
use fluree_db_api::{FlureeBuilder, Result};

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::file("./data").build()?;

    // Load and use ledger
    let ledger = fluree.ledger("mydb:main").await?;
    println!("Ledger at t={}", ledger.t());

    // Release cached state
    fluree.disconnect_ledger("mydb:main").await;

    // Next access will reload from storage
    let ledger = fluree.ledger("mydb:main").await?;

    Ok(())
}
```

**When to use `disconnect_ledger`:**

- **Force fresh load**: After external changes to the ledger (e.g., another process wrote data)
- **Free memory**: Release memory for ledgers you no longer need
- **Clean shutdown**: Release resources before application exit
- **Testing**: Reset state between test cases

**Note:** If caching is disabled (via `without_ledger_caching()` on builder), `disconnect_ledger` is a no-op.

#### Checking Ledger Existence

Use `ledger_exists` to check if a ledger is registered in the nameservice without loading it:

```rust
use fluree_db_api::{FlureeBuilder, Result};

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::file("./data").build()?;

    // Check if ledger exists (lightweight nameservice lookup)
    if fluree.ledger_exists("mydb:main").await? {
        // Ledger exists - load it
        let ledger = fluree.ledger("mydb:main").await?;
        println!("Loaded ledger at t={}", ledger.t());
    } else {
        // Ledger doesn't exist - create it
        let ledger = fluree.create_ledger("mydb").await?;
        println!("Created new ledger");
    }

    Ok(())
}
```

**When to use `ledger_exists`:**

- **Conditional create-or-load**: Check before deciding whether to create or load
- **Validation**: Verify ledger IDs exist before operations
- **Defensive programming**: Avoid `NotFound` errors in application logic

**Performance note:** This is a lightweight check that only queries the nameservice - it does NOT load the ledger data, indexes, or novelty. Much faster than attempting to load and catching `NotFound` errors.

#### Dropping Ledgers

`drop_ledger` operates on the **whole ledger** — every branch under a ledger
name, including retracted-but-not-purged branches and the cross-branch
`@shared/dicts/` namespace. Use `drop_branch` to remove a single branch.

```rust
use fluree_db_api::{FlureeBuilder, DropMode, DropStatus, Result};

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::file("./data").build()?;

    // Soft drop: retract every branch in the nameservice, preserve artifacts
    let report = fluree.drop_ledger("mydb", DropMode::Soft).await?;
    match report.status {
        DropStatus::Dropped => println!("Ledger dropped"),
        DropStatus::AlreadyRetracted => println!("Already dropped"),
        DropStatus::NotFound => println!("Ledger not found"),
    }

    // Hard drop: delete artifacts for every branch + @shared/dicts (IRREVERSIBLE)
    let report = fluree.drop_ledger("mydb", DropMode::Hard).await?;
    println!(
        "Dropped {} branches; deleted {} storage artifacts",
        report.branch_reports.len(),
        report.artifacts_deleted
    );
    for br in &report.branch_reports {
        println!("  - {} ({:?})", br.ledger_id, br.status);
    }

    Ok(())
}
```

**Accepted inputs:**

| Input | Behavior |
|-------|----------|
| `"mydb"` | Whole-ledger drop (canonical form). |
| `"mydb:<branch>"` (including `"mydb:main"`) | **Rejected** with a `400`/`ApiError::Http`. Use `drop_branch("mydb", "<branch>")` to drop a single branch. |

**Drop Modes:**

| Mode | Behavior | Reversible |
|------|----------|------------|
| `DropMode::Soft` (default) | Marks every branch retracted in the nameservice; artifacts remain | Partially; requires administrative recovery |
| `DropMode::Hard` | Deletes managed storage artifacts for every branch, wipes `@shared/dicts/`, and purges nameservice records so the name can be reused | **No** for deleted artifacts |

**Drop Sequence:**

1. Parses input (rejects non-default branch suffixes).
2. Snapshots every NsRecord under the ledger name via `all_records` (so retracted branches are included). Enumeration failures propagate as `Err`.
3. Sorts branches leaf-first via `source_branch` parent pointers — partial failures leave orphan parents, never dangling children.
4. Cancels and waits for pending background indexing on each branch.
5. For each branch: deletes per-branch artifacts (commit/txn/index/config) in hard mode; retracts (soft) or drops the NS record (hard) using the parent-aware path so surviving parents have accurate child counts.
6. Hard mode: wipes `{ledger_name}/@shared/dicts/` after every branch is gone.
7. Disconnects each branch from the ledger cache.

**Report shape:**

`DropReport.ledger_id` is the bare ledger name. `artifacts_deleted` and
`warnings` aggregate across branches plus the shared cleanup;
`branch_reports: Vec<BranchDropReport>` carries per-branch detail in
leaf-first order — useful for surfacing partial failures.

**`drop_branch` for single branches:**

```rust
let report = fluree.drop_branch("mydb", "dev").await?;
```

`drop_branch` refuses the **root** branch (any branch whose
`source_branch.is_none()`). `"main"` carries no special meaning; a ledger
created with a different initial branch (e.g. `mydb:trunk`) has that
branch as its root and is the one that `drop_branch` will refuse.

**Idempotency:**

Safe to call multiple times:
- Returns `DropStatus::AlreadyRetracted` when every branch was already retracted (hard mode still runs cleanup for these).
- Returns `DropStatus::NotFound` without touching storage when no NsRecord exists for the ledger name. Orphaned storage with no nameservice pointer is not swept by `drop_ledger`; that's a separate admin concern.
- On a real per-branch nameservice failure, returns `ApiError::Drop` without touching parents or `@shared/dicts/`. Each step is idempotent under partial progress, so retry is safe.

**Warnings:**

```rust
let report = fluree.drop_ledger("mydb", DropMode::Hard).await?;
for warning in &report.warnings {
    eprintln!("Warning: {warning}");
}
for br in &report.branch_reports {
    for warning in &br.warnings {
        eprintln!("  {}: {warning}", br.ledger_id);
    }
}
```

#### Refreshing Cached Ledgers

Use `refresh` to poll-check whether a cached ledger is stale and update it if needed.
`refresh` returns a `RefreshResult` containing the ledger's `t` after the operation
and what action was taken:

```rust
use fluree_db_api::{FlureeBuilder, NotifyResult, RefreshOpts, Result};

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::file("./data").build()?;

    // Load ledger into cache
    let _ledger = fluree.ledger_cached("mydb:main").await?;

    // Later, check if the cached state is still fresh
    match fluree.refresh("mydb:main", Default::default()).await? {
        Some(r) => {
            println!("Ledger at t={}, action: {:?}", r.t, r.action);
            match r.action {
                NotifyResult::Current => println!("Already up to date"),
                NotifyResult::Reloaded => println!("Reloaded from storage"),
                NotifyResult::IndexUpdated => println!("Index was updated"),
                NotifyResult::CommitsApplied { count } => {
                    println!("{count} commits applied incrementally");
                }
                NotifyResult::NotLoaded => println!("Not in cache"),
            }
        }
        None => println!("Ledger not found in nameservice"),
    }

    Ok(())
}
```

**Key behaviors:**

- **Does NOT cold-load**: If the ledger isn't already cached, returns `NotLoaded` (no-op)
- **Returns `None`**: If the ledger doesn't exist in the nameservice
- **Alias resolution**: Supports short aliases (`mydb` resolves to `mydb:main`)
- **No-op without caching**: If caching is disabled, returns `NotLoaded`
- **Returns `t`**: The `RefreshResult.t` field always tells you the ledger's current transaction time

**When to use `refresh`:**

- **Poll-based freshness**: When you can't use SSE events but need periodic freshness checks
- **Before critical reads**: Ensure you have the latest state before important queries
- **Peer mode**: Check if the local cache is behind the transaction server

**`refresh` vs `disconnect_ledger`:**

| Behavior | `refresh` | `disconnect_ledger` |
|----------|-----------|---------------------|
| Checks freshness | Yes | No |
| Updates in place | Yes | No (forces full reload on next access) |
| Handles not-cached | Returns `NotLoaded` | No-op |
| Use case | Poll-based updates | Force full reload |

#### Read-After-Write Consistency

Fluree's query engine is **eventually consistent**: when one process writes data and
another (or the same process on a warm cache) queries it, the query may not yet see
the latest commit. The `t` value returned from a transaction is the key to bridging
this gap.

Pass `RefreshOpts { min_t: Some(t) }` to `refresh()` to assert that the cached
ledger has reached at least that transaction time. If it hasn't after pulling the
latest state from the nameservice, `refresh` returns `ApiError::AwaitTNotReached`
with both the `requested` and `current` `t` values. Your code owns retry timing
and timeout policy.

**Basic usage:**

```rust
use fluree_db_api::{FlureeBuilder, RefreshOpts, ApiError, Result};
use serde_json::json;

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::file("./data").build()?;
    let handle = fluree.ledger_cached("mydb:main").await?;

    // Transaction returns the commit's t value
    let receipt = fluree.stage(&handle)
        .insert(&json!({"@id": "ex:item", "ex:count": 42}))
        .commit()
        .await?;
    let committed_t = receipt.t;

    // Ensure the cache reflects at least this t before querying
    let opts = RefreshOpts { min_t: Some(committed_t) };
    let result = fluree.refresh("mydb:main", opts).await?;
    // result.unwrap().t >= committed_t is guaranteed here

    Ok(())
}
```

**Serverless / Lambda pattern (retry with backoff):**

In a serverless environment, the transacting process and the querying process may be
different Lambda invocations. The querying invocation receives `t` (e.g., via an
event payload or API parameter) and must wait for that commit to be visible:

```rust
use fluree_db_api::{RefreshOpts, ApiError};
use std::time::{Duration, Instant};

async fn wait_for_t(
    fluree: &Fluree<impl Storage, impl NameService>,
    ledger_id: &str,
    min_t: i64,
    timeout: Duration,
) -> Result<i64, ApiError> {
    let deadline = Instant::now() + timeout;
    let opts = RefreshOpts { min_t: Some(min_t) };

    loop {
        match fluree.refresh(ledger_id, opts.clone()).await {
            Ok(Some(r)) => return Ok(r.t),   // reached min_t
            Ok(None) => return Err(ApiError::NotFound(
                format!("ledger {ledger_id} not in nameservice"),
            )),
            Err(ApiError::AwaitTNotReached { current, .. }) => {
                if Instant::now() >= deadline {
                    return Err(ApiError::AwaitTNotReached {
                        requested: min_t,
                        current,
                    });
                }
                // Back off before retrying
                tokio::time::sleep(Duration::from_millis(50)).await;
            }
            Err(e) => return Err(e),
        }
    }
}
```

**How it works internally:**

1. **Fast path**: If the cached `t` already satisfies `min_t`, returns immediately
   without hitting the nameservice at all.
2. **Pull**: Queries the nameservice for the latest commit/index pointers and applies
   any new commits incrementally (or reloads if the gap is large).
3. **Check**: If `t` is still below `min_t` after the pull, returns
   `ApiError::AwaitTNotReached` so you can retry.

This design keeps retry/timeout policy out of the database layer. Different
deployment contexts (Lambda with 100ms backoff, HTTP handler with 5s deadline,
integration test with immediate assertion) each wrap the same primitive differently.

### Branch Diff (Merge Preview)

`Fluree::merge_preview` returns the rich diff between two branches —
ahead/behind commit summaries, the common ancestor, conflict keys, and
fast-forward eligibility — **without mutating any state**. It uses the
same primitives as `merge_branch` but skips the publish/copy steps,
making it cheap enough to call on every UI render.

```rust
use fluree_db_api::{FlureeBuilder, MergePreviewOpts, Result};

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::memory().build_memory();

    // ... create ledger, branch, transact on dev, etc.

    // Default: previewing dev → main with the spec defaults
    // (cap each commit list at 500, conflict keys at 200, run conflicts).
    let preview = fluree.merge_preview("mydb", "dev", None).await?;

    println!(
        "{} ahead, {} behind, fast-forward: {}",
        preview.ahead.count, preview.behind.count, preview.fast_forward,
    );

    if preview.fast_forward {
        println!("merge would advance {} → {}", preview.source, preview.target);
    } else {
        println!("merge has {} conflict(s)", preview.conflicts.count);
        for k in &preview.conflicts.keys {
            println!("  - s={} p={}", k.s, k.p);
        }
    }
    Ok(())
}
```

#### Tuning the preview

`merge_preview_with` takes a `MergePreviewOpts` for callers that need
control over response size or want to skip the conflict computation:

```rust
use fluree_db_api::{FlureeBuilder, MergePreviewOpts, Result};

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::memory().build_memory();

    // Cheap preview: counts only, no conflict walks.
    let counts = fluree
        .merge_preview_with(
            "mydb",
            "dev",
            Some("main"),
            MergePreviewOpts {
                max_commits: Some(0),       // counts only — no commit summaries
                max_conflict_keys: Some(0),
                include_conflicts: false,
            },
        )
        .await?;

    // Direct Rust callers can opt in to **unbounded** results — useful for
    // tooling that needs the full divergence. The HTTP layer always supplies
    // a bound, so this is a Rust-only escape hatch.
    let full = fluree
        .merge_preview_with(
            "mydb",
            "dev",
            None,
            MergePreviewOpts {
                max_commits: None,
                max_conflict_keys: None,
                include_conflicts: true,
            },
        )
        .await?;

    Ok(())
}
```

#### What the caps do (and don't) control

`max_commits` and `max_conflict_keys` cap the **size of the returned
lists**, not the cost of computing them:

- `BranchDelta::count` on each side reflects the full unbounded
  divergence — computed by walking every commit envelope between HEAD and
  the common ancestor — regardless of `max_commits`.
- When `include_conflicts: true`, both `compute_delta_keys` walks scan
  the full per-side delta regardless of `max_conflict_keys`.
- When `include_conflict_details: true`, value details are collected only
  for the returned `conflicts.keys` after the `max_conflict_keys` cap is
  applied.
- Set `include_conflicts: false` for a cheap preview on heavily diverged
  branches; you still get accurate `ahead.count` / `behind.count`.

#### Response shape

| Type | Notable fields |
|------|----------------|
| `MergePreview` | `source`, `target`, `ancestor: Option<AncestorRef>`, `ahead`, `behind`, `fast_forward`, `conflicts`, `mergeable` |
| `BranchDelta` | `count` (unbounded), `commits: Vec<CommitSummary>` (newest-first, capped), `truncated` |
| `CommitSummary` | `t`, `commit_id`, `time`, `asserts`, `retracts`, `flake_count`, `message: Option<String>` (extracted from the `f:message` `txn_meta` entry when present) |
| `ConflictSummary` | `count` (unbounded), `keys: Vec<ConflictKey>` (sorted, capped), `truncated`, `strategy`, `details` |
| `ConflictDetail` | `key`, `source_values`, `target_values`, `resolution` (values are the current asserted values at each branch HEAD) |
| `ConflictKey` | `s: Sid`, `p: Sid`, `g: Option<Sid>` |

`mergeable` only reflects whether the selected strategy would abort due to
detected conflicts; it is not full validation of every constraint the eventual
merge commit may encounter. `mergeable=true` does not guarantee a subsequent
merge will succeed; it only reflects the conflict/strategy interaction at
preview time.

All types derive `Serialize` so the response is wire-stable; the HTTP
endpoint at `GET /v1/fluree/merge-preview/{ledger...}` returns the same struct.
See `docs/api/endpoints.md` and `docs/cli/server-integration.md` for the
HTTP contract.

#### Reusable primitives in `fluree-db-core`

The per-commit summary types and DAG walker are factored into core for
reuse outside the merge-preview flow (e.g., git-log-style commit history
viewers, indexer integration). Re-exported from `fluree-db-api`:

- `walk_commit_summaries(store, head, stop_at_t, max) -> Result<(Vec<CommitSummary>, usize)>`
  — newest-first walk that returns both the (capped) summary list and the
  unbounded total count.
- `commit_to_summary(commit) -> CommitSummary` — pure function, no I/O.
- `find_common_ancestor(store, head_a, head_b)` — dual-frontier BFS.

### Time Travel Queries

```rust
use fluree_db_api::{FlureeBuilder, TimeSpec, Result};

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::memory().build_memory();

    // Query at a specific point in time
    let result = fluree.graph_at("mydb:main", TimeSpec::AtT(100))
        .query()
        .sparql("SELECT * WHERE { ?s ?p ?o } LIMIT 10")
        .execute()
        .await?;

    println!("Results at t=100: {:?}", result.row_count());

    Ok(())
}
```

### Multi-Ledger Queries

```rust
use fluree_db_api::{FlureeBuilder, DataSetDb, Result};

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::memory().build_memory();

    // Load views from multiple ledgers
    let customers = fluree.view("customers:main").await?;
    let orders = fluree.view("orders:main").await?;

    // Compose a dataset from multiple graphs
    let dataset = DataSetDb::new()
        .with_default(customers)
        .with_named("orders:main", orders);

    // Query across ledgers using the dataset builder
    let query = r#"
        SELECT ?customerName ?orderTotal
        WHERE {
            ?customer schema:name ?customerName .
            ?customer ex:customerId ?cid .

            GRAPH <orders:main> {
                ?order ex:customerId ?cid .
                ?order ex:total ?orderTotal .
            }
        }
    "#;

    let result = dataset.query(&fluree)
        .sparql(query)
        .execute()
        .await?;

    Ok(())
}
```

### Remote Federation

Query ledgers on remote Fluree servers using SPARQL `SERVICE` with the `fluree:remote:` scheme. Register remote connections at build time — each maps a name to a server URL and optional bearer token:

```rust
use fluree_db_api::{FlureeBuilder, Result};

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::file("./data")
        .remote_connection(
            "acme",
            "https://acme-fluree.example.com",
            Some("eyJhbG...".to_string()),
        )
        .build()?;

    let db = fluree.view("local-ledger:main").await?;

    // Join local data with a ledger on the remote server
    let result = fluree.query(&db, r#"
        PREFIX ex: <http://example.org/ns/>
        SELECT ?name ?email
        WHERE {
          ?person ex:name ?name .
          SERVICE <fluree:remote:acme/customers:main> {
            ?person ex:email ?email .
          }
        }
    "#).await?;

    Ok(())
}
```

The connection name (`acme`) maps to the server URL. The ledger path (`customers:main`) is appended to form the request URL: `POST https://acme-fluree.example.com/v1/fluree/query/customers:main`. The bearer token is sent as `Authorization: Bearer <token>` on every request.

Multiple ledgers on the same remote server use the same connection name — you register the server once and can query any ledger your token is authorized for.

See [Configuration: Remote connections](../operations/configuration.md#remote-connections) for details and [SPARQL: Remote Fluree Federation](../query/sparql.md#remote-fluree-federation) for full query syntax.

### FROM-Driven Queries (Connection Queries)

When the query body itself specifies which ledgers to target (via `"from"` in JSON-LD or `FROM` in SPARQL), use `query_from()`:

```rust
use fluree_db_api::{FlureeBuilder, Result};
use serde_json::json;

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::memory().build_memory();

    // Query where the "from" is embedded in the query body
    let query = json!({
        "from": "mydb:main",
        "select": ["?name"],
        "where": { "@id": "?s", "schema:name": "?name" }
    });

    let result = fluree.query_from()
        .jsonld(&query)
        .execute_formatted()
        .await?;

    // SPARQL with FROM clause
    let result = fluree.query_from()
        .sparql("SELECT ?name FROM <mydb:main> WHERE { ?s <http://schema.org/name> ?name }")
        .execute_formatted()
        .await?;

    Ok(())
}
```

### Background Indexing

```rust
use fluree_db_api::{FlureeBuilder, BackgroundIndexerWorker, Result};
use std::sync::Arc;
use tokio::time::{sleep, Duration};

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = Arc::new(FlureeBuilder::file("./data").build()?);

    // Start background indexer
    let indexer = BackgroundIndexerWorker::new(
        fluree.clone(),
        Duration::from_secs(5), // Index interval
    );

    let indexer_handle = indexer.start();

    // Application logic
    let ledger = fluree.create_ledger("mydb").await?;

    // Transactions will be indexed automatically in background
    for i in 0..100 {
        let txn = json!({
            "@context": {"ex": "http://example.org/ns/"},
            "@graph": [{"@id": format!("ex:item{}", i), "ex:value": i}]
        });

        fluree.graph("mydb:main")
            .transact()
            .insert(&txn)
            .commit()
            .await?;
    }

    // Wait for indexing to complete
    sleep(Duration::from_secs(10)).await;

    // Shutdown indexer
    indexer_handle.shutdown().await?;

    Ok(())
}
```

### BM25 Full-Text Search

```rust
use fluree_db_api::{
    FlureeBuilder, Bm25CreateConfig, Bm25FieldConfig, Result
};
use serde_json::json;

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::memory().build_memory();
    let ledger = fluree.create_ledger("mydb").await?;

    // Insert searchable data and create BM25 index
    // ...

    // Query with full-text search using JSON-LD and the f:graphSource pattern
    let search_query = json!({
        "@context": {
            "schema": "http://schema.org/",
            "f": "https://ns.flur.ee/db#"
        },
        "from": "mydb:main",
        "select": ["?product", "?score", "?name"],
        "where": [
            {
                "f:graphSource": "products-search:main",
                "f:searchText": "laptop",
                "f:searchLimit": 10,
                "f:searchResult": { "f:resultId": "?product", "f:resultScore": "?score" }
            },
            { "@id": "?product", "schema:name": "?name" }
        ],
        "orderBy": [["desc", "?score"]],
        "limit": 10
    });

    let result = fluree.query_from()
        .jsonld(&search_query)
        .execute()
        .await?;

    println!("Found {} matching products", result.row_count());

    Ok(())
}
```

## Configuration

### Builder Options

```rust
use fluree_db_api::{FlureeBuilder, ConnectionConfig, IndexConfig, Result};

#[tokio::main]
async fn main() -> Result<()> {
    let config = ConnectionConfig {
        storage_path: "./data".into(),
        index_config: IndexConfig {
            interval_ms: 5000,
            batch_size: 10,
            memory_mb: 2048,
            threads: 4,
        },
        ..Default::default()
    };

    let fluree = FlureeBuilder::with_config(config).build()?;

    Ok(())
}
```

### Custom Storage Backend

```rust
use fluree_db_api::{
    FlureeBuilder, Storage, StorageWrite, Result
};
use async_trait::async_trait;

// Implement custom storage
struct MyStorage;

#[async_trait]
impl Storage for MyStorage {
    async fn read(&self, address: &str) -> Result<Vec<u8>> {
        // Custom implementation
        todo!()
    }
}

#[async_trait]
impl StorageWrite for MyStorage {
    async fn write(&self, address: &str, data: &[u8]) -> Result<()> {
        // Custom implementation
        todo!()
    }
}

#[tokio::main]
async fn main() -> Result<()> {
    let storage = MyStorage;
    let fluree = FlureeBuilder::custom(storage).build()?;

    Ok(())
}
```

If you need full control over both storage and nameservice (e.g., for proxy mode or custom backends), use `build_with()`:

```rust
let storage = MyStorage;
let nameservice = MyNameService;

let fluree = FlureeBuilder::memory()
    .build_with(storage, nameservice);
```

`build_with()` respects the builder's caching configuration — caching is on by default, or call `.without_ledger_caching()` before `build_with()` to disable it.

## Error Handling

```rust
use fluree_db_api::{FlureeBuilder, ApiError, Result};

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::memory().build_memory();

    // Create a ledger — handles duplicates gracefully
    match fluree.create_ledger("mydb").await {
        Ok(ledger) => {
            println!("Ledger created at t={}", ledger.t());
        }
        Err(ApiError::LedgerExists(ledger_id)) => {
            println!("Ledger {} already exists, loading...", ledger_id);
            let ledger = fluree.ledger("mydb:main").await?;
            println!("Loaded at t={}", ledger.t());
        }
        Err(e) => {
            eprintln!("Error: {}", e);
            return Err(e);
        }
    }

    Ok(())
}
```

## Testing

### Unit Tests

```rust
#[cfg(test)]
mod tests {
    use fluree_db_api::{FlureeBuilder, Result};
    use serde_json::json;

    #[tokio::test]
    async fn test_insert_and_query() -> Result<()> {
        // Use memory storage for tests
        let fluree = FlureeBuilder::memory().build_memory();
        let ledger = fluree.create_ledger("test").await?;

        // Insert data
        let data = json!({
            "@context": {"ex": "http://example.org/ns/"},
            "@graph": [{"@id": "ex:alice", "ex:name": "Alice"}]
        });

        fluree.graph("test:main")
            .transact()
            .insert(&data)
            .commit()
            .await?;

        // Query data
        let query = json!({
            "select": ["?name"],
            "where": [{"@id": "ex:alice", "ex:name": "?name"}]
        });

        let result = fluree.graph("test:main")
            .query()
            .jsonld(&query)
            .execute()
            .await?;

        assert_eq!(result.row_count(), 1);

        Ok(())
    }
}
```

### Integration Tests

```rust
// tests/integration_test.rs
use fluree_db_api::{FlureeBuilder, Result};
use serde_json::json;
use tempfile::TempDir;

#[tokio::test]
async fn test_persistence() -> Result<()> {
    let temp_dir = TempDir::new()?;
    let path = temp_dir.path().to_str().unwrap();

    // Create ledger and write data
    {
        let fluree = FlureeBuilder::file(path).build()?;
        let ledger = fluree.create_ledger("test").await?;

        let data = json!({"@context": {}, "@graph": [{"@id": "ex:test"}]});
        fluree.graph("test:main")
            .transact()
            .insert(&data)
            .commit()
            .await?;
    }

    // Verify persistence by reopening
    {
        let fluree = FlureeBuilder::file(path).build()?;
        let ledger = fluree.ledger("test:main").await?;

        assert!(ledger.t() > 0);
    }

    Ok(())
}
```

## Performance Tips

### Batch Transactions

```rust
// Good: Batch related changes
let batch_data = json!({
    "@graph": [
        {"@id": "ex:item1", "ex:value": 1},
        {"@id": "ex:item2", "ex:value": 2},
        {"@id": "ex:item3", "ex:value": 3}
    ]
});
let result = fluree.graph("mydb:main")
    .transact()
    .insert(&batch_data)
    .commit()
    .await?;

// Bad: Individual transactions (more overhead per commit)
for i in 1..=3 {
    let txn = json!({"@graph": [{"@id": format!("ex:item{}", i), "ex:value": i}]});
    fluree.graph("mydb:main")
        .transact()
        .insert(&txn)
        .commit()
        .await?;
}
```

### Use Appropriate Storage

- **Memory**: Fastest, no persistence (tests, temporary data)
- **File**: Good balance (single server, local development)
- **AWS**: Distributed, durable (production, multi-server)

### Query Optimization

```rust
// Good: Specific patterns
let query = json!({
    "select": ["?name"],
    "where": [
        {"@id": "ex:alice", "schema:name": "?name"}
    ]
});

// Bad: Broad patterns
let query = json!({
    "select": ["?s", "?p", "?o"],
    "where": [
        {"@id": "?s", "?p": "?o"}
    ]
});
```

### Enable Query Tracking

```rust
use fluree_db_api::{FlureeBuilder, Result};

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::memory().build_memory();

    // Use execute_tracked() for fuel/time/policy tracking
    let tracked = fluree.graph("mydb:main")
        .query()
        .sparql("SELECT * WHERE { ?s ?p ?o }")
        .execute_tracked()
        .await?;

    println!("Query used {} fuel", tracked.fuel().unwrap_or(0));

    Ok(())
}
```

## Graph API Reference

The Graph API follows a **lazy-handle** pattern: `fluree.graph(graph_ref)` returns a lightweight handle, and all I/O is deferred to terminal methods.

### Getting a Graph Handle

```rust
// Lazy handle to the current (head) state
let graph = fluree.graph("mydb:main");

// Lazy handle at a specific point in time
let graph = fluree.graph_at("mydb:main", TimeSpec::AtT(100));
```

### Querying

```rust
// JSON-LD query (lazy — loads graph at execution time)
let result = fluree.graph("mydb:main")
    .query()
    .jsonld(&query_json)
    .execute().await?;

// SPARQL query
let result = fluree.graph("mydb:main")
    .query()
    .sparql("SELECT ?s WHERE { ?s a <ex:Person> }")
    .execute().await?;

// Formatted output (JSON-LD or SPARQL JSON based on query type)
let json = fluree.graph("mydb:main")
    .query()
    .jsonld(&query_json)
    .execute_formatted().await?;

// Tracked query (fuel/time/policy metrics)
let tracked = fluree.graph("mydb:main")
    .query()
    .sparql("SELECT * WHERE { ?s ?p ?o }")
    .execute_tracked().await?;
```

### Materializing a GraphSnapshot

```rust
// Load once, query many times (avoids reloading)
let db = fluree.graph("mydb:main").load().await?;

let r1 = db.query().sparql("...").execute().await?;
let r2 = db.query().jsonld(&q).execute().await?;

// Access the underlying GraphDb
let view = db.view();
```

### Transacting

```rust
// Insert and commit
let result = fluree.graph("mydb:main")
    .transact()
    .insert(&data)
    .commit().await?;

// Upsert with options. f:identity is system-controlled (signed DID,
// opts.identity, or CommitOpts::identity). f:message and f:author are
// pure user claims — supply them in the transaction body just like any
// other txn-meta property.
let data = serde_json::json!({
    "@context": {
        "ex": "http://example.org/",
        "f": "https://ns.flur.ee/db#"
    },
    "@graph": [{ "@id": "ex:alice", "ex:name": "Alice" }],
    "f:message": "admin update",
    "f:author": "did:admin"
});

let result = fluree.graph("mydb:main")
    .transact()
    .upsert(&data)
    .commit_opts(CommitOpts::default().identity("did:admin"))
    .commit().await?;

// Stage without committing (preview changes)
let staged = fluree.graph("mydb:main")
    .transact()
    .insert(&data)
    .stage().await?;

// Query staged state
let preview = staged.query()
    .jsonld(&validation_query)
    .execute().await?;
```

### Commit Inspection

Decode and display the contents of a commit — assertions and retractions with IRIs resolved to compact form. Similar to `git show` for individual commits.

```rust
// By exact CID
let detail = fluree.graph("mydb:main")
    .commit(&commit_id)
    .execute().await?;

// By transaction number
let detail = fluree.graph("mydb:main")
    .commit_t(5)
    .execute().await?;

// By hex-digest prefix (min 6 chars, like abbreviated git hashes)
let detail = fluree.graph("mydb:main")
    .commit_prefix("3dd028")
    .execute().await?;

// With a custom @context for IRI compaction
let detail = fluree.graph("mydb:main")
    .commit_prefix("3dd028")
    .context(my_parsed_context)
    .execute().await?;

// Access the result
println!("t={}, +{} -{}", detail.t, detail.asserts, detail.retracts);
for flake in &detail.flakes {
    let op = if flake.op { "+" } else { "-" };
    println!("{} {} {} {} [{}]", op, flake.s, flake.p, flake.o, flake.dt);
}
```

The returned `CommitDetail` contains:
- **Metadata**: `id`, `t`, `time`, `size`, `previous`, `signer`, `asserts`, `retracts`
- **`context`**: prefix → IRI map derived from the ledger's namespace codes
- **`flakes`**: flat list in SPOT order, each with resolved compact IRIs

`CommitDetail` implements `Serialize` — flakes serialize as `[s, p, o, dt, op]` tuples (with an optional 6th metadata element for language tags, list indices, or named graphs).

### Terminal Operations

| Method | Returns | Description |
|--------|---------|-------------|
| `.execute()` | `Result<QueryResult>` | Raw query result |
| `.execute_formatted()` | `Result<JsonValue>` | Formatted JSON output (JSON-LD for `.jsonld()`, SPARQL JSON for `.sparql()`) |
| `.execute_tracked()` | `Result<TrackedQueryResponse>` | Result with fuel/time/policy tracking |
| `.commit()` | `Result<TransactResultRef>` | Stage + commit transaction |
| `.stage()` | `Result<StagedGraph>` | Stage without committing |
| `.load()` | `Result<GraphSnapshot>` | Materialize snapshot for reuse |

### Format Override

```rust
use fluree_db_api::FormatterConfig;

// Force JSON-LD format for a SPARQL query
let result = fluree.graph("mydb:main")
    .query()
    .sparql("SELECT ?name WHERE { ?s <schema:name> ?name }")
    .format(FormatterConfig::jsonld())
    .execute_formatted()
    .await?;
```

### Multi-Ledger Queries (Dataset)

For multi-ledger queries, use `GraphDb` directly:

```rust
let customers = fluree.view("customers:main").await?;
let orders = fluree.view("orders:main").await?;

let dataset = DataSetDb::new()
    .with_default(customers)
    .with_named("orders:main", orders);

let result = dataset.query(&fluree)
    .sparql(query)
    .execute().await?;
```

### FROM-Driven Queries (Connection Queries)

```rust
let result = fluree.query_from()
    .jsonld(&query_with_from)
    .execute().await?;
```

## Transaction Builder API Reference

There are two transaction builder patterns, each suited for different use cases:

### `stage(&handle)` — Server/Application Pattern (Recommended)

Use `stage(&handle)` when building servers or applications with ledger caching enabled. The handle is borrowed and updated in-place on successful commit, ensuring concurrent readers see the update.

```rust
use fluree_db_api::{FlureeBuilder, Result};
use serde_json::json;

#[tokio::main]
async fn main() -> Result<()> {
    // Caching is on by default (required for stage)
    let fluree = FlureeBuilder::file("./data").build()?;

    // Get a cached handle
    let handle = fluree.ledger_cached("mydb:main").await?;

    // Transaction via builder — handle updated in-place
    let data = json!({"@graph": [{"@id": "ex:test", "ex:name": "Test"}]});
    let result = fluree.stage(&handle)
        .insert(&data)
        .execute()
        .await?;

    println!("Committed at t={}", result.receipt.t);

    // Handle now reflects the new state
    let snapshot = handle.snapshot().await;
    assert_eq!(snapshot.t, result.receipt.t);

    Ok(())
}
```

**Why use `stage(&handle)`:**
- **Concurrent safety**: Multiple requests share the same handle; updates are atomic
- **No ownership dance**: You don't need to track and pass around `LedgerState` values
- **Server-friendly**: Matches how the HTTP server handles transactions internally

### `stage_owned(ledger)` — CLI/Script/Test Pattern

Use `stage_owned(ledger)` when you manage your own `LedgerState` directly. This is typical for CLI tools, scripts, and tests where you don't need ledger caching.

```rust
use fluree_db_api::{FlureeBuilder, Result};
use serde_json::json;

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::memory().build_memory();

    // You own the ledger state
    let ledger = fluree.create_ledger("mydb").await?;

    // Transaction consumes ledger, returns updated state
    let data = json!({"@graph": [{"@id": "ex:test", "ex:name": "Test"}]});
    let result = fluree.stage_owned(ledger)
        .insert(&data)
        .execute()
        .await?;

    // Get the updated ledger from the result
    let ledger = result.ledger;
    println!("Now at t={}", ledger.t());

    Ok(())
}
```

**Why use `stage_owned(ledger)`:**
- **Simple ownership**: Good for linear workflows (load → transact → done)
- **No caching required**: Works even with `without_ledger_caching()`
- **Test-friendly**: Each test manages its own state

### Choosing Between Them

| Use Case | Pattern | Why |
|----------|---------|-----|
| HTTP server | `stage(&handle)` | Shared handles, atomic updates |
| Long-running app | `stage(&handle)` | Concurrent access to same ledger |
| CLI tool | `stage_owned(ledger)` | Simple, no caching needed |
| Integration test | `stage_owned(ledger)` | Isolated state per test |
| Script/batch job | `stage_owned(ledger)` | Linear workflow |

### Builder Methods (Both Patterns)

Both `stage(&handle)` and `stage_owned(ledger)` return a builder with identical methods:

```rust
let result = fluree.stage(&handle)  // or stage_owned(ledger)
    .insert(&data)                   // or .upsert(&data), .update(&data)
    .commit_opts(CommitOpts::default().identity("did:admin"))
    .execute()
    .await?;
// (Include `f:message` / `f:author` directly in `data` for user-claim provenance.)
```

| Method | Description |
|--------|-------------|
| `.insert(&json)` | Insert JSON-LD data |
| `.upsert(&json)` | Upsert JSON-LD data |
| `.update(&json)` | Update with WHERE/DELETE/INSERT |
| `.insert_turtle(&ttl)` | Insert Turtle data |
| `.upsert_turtle(&ttl)` | Upsert Turtle data |
| `.txn_opts(opts)` | Set transaction options (branch, context) |
| `.commit_opts(opts)` | Set commit options (identity, raw_txn) |
| `.policy(ctx)` | Set policy enforcement |
| `.execute()` | Stage + commit |
| `.stage()` | Stage without committing (returns `Staged`) |
| `.validate()` | Check configuration without executing |

### Graph API Transactions

The Graph API (`fluree.graph(graph_ref).transact()`) is built on top of `stage(&handle)` internally:

```rust
// Graph API (convenient, uses caching internally)
let result = fluree.graph("mydb:main")
    .transact()
    .insert(&data)
    .commit()
    .await?;

// Equivalent to:
let handle = fluree.ledger_cached("mydb:main").await?;
let result = fluree.stage(&handle)
    .insert(&data)
    .execute()
    .await?;
```

## Ledger Info API

Get comprehensive metadata about a ledger using the `ledger_info()` builder:

```rust
use fluree_db_api::{FlureeBuilder, Result};
use serde_json::json;

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::file("./data").build()?;

    // Get ledger info with optional context for IRI compaction
    let context = json!({
        "schema": "http://schema.org/",
        "ex": "http://example.org/ns/"
    });

    let info = fluree
        .ledger_info("mydb:main")
        .with_context(&context)
        // Optional: include datatype breakdowns under stats.properties[*]
        // .with_property_datatypes(true)
        // Optional: make property datatype details novelty-aware (real-time)
        // .with_realtime_property_details(true)
        .execute()
        .await?;

    // Access metadata sections
    println!("Commit: {}", info["commit"]);
    println!("Nameservice: {}", info["nameservice"]);
    println!("Namespace codes: {}", info["namespace-codes"]);
    println!("Stats: {}", info["stats"]);
    println!("Index: {}", info["index"]);

    Ok(())
}
```

### Ledger Info Response

The response includes:

| Section | Description |
|---------|-------------|
| `commit` | Commit info in JSON-LD format |
| `nameservice` | NsRecord in JSON-LD format |
| `namespace-codes` | Inverted mapping (prefix → code) for IRI expansion |
| `stats` | Flake counts, size, property/class statistics with selectivity |
| `index` | Index metadata (`t`, ContentId, index ID) |

#### Stats freshness (real-time vs indexed)

The `stats` section now uses layered runtime stats assembly:

- Default `ledger_info()` uses the full novelty-aware path, including lookup-backed class/ref enrichment.
- `with_realtime_property_details(false)` downgrades to the lighter fast novelty-aware merge (`Indexed` + novelty deltas, no extra lookups).
- HLL / NDV fields remain index-derived, so they are omitted by default and only included via `with_property_estimates(true)`.

That means the payload still mixes **real-time** values (indexed + novelty deltas) with values that are only available **as-of the last index**.

- **Real-time (includes novelty)**:
  - `stats.flakes`, `stats.size`
  - `stats.properties[*].count` (but not NDV)
  - `stats.properties[*].datatypes` by default
  - `stats.classes[*].count`
  - `stats.classes[*].property-list` and `stats.classes[*].properties` (property presence)
  - `stats.classes[*].properties[*].refs` by default

- **As-of last index**:
  - `stats.indexed` (the index \(t\))
  - `stats.properties[*].ndv-values`, `stats.properties[*].ndv-subjects` when explicitly included via `with_property_estimates(true)`
  - Any selectivity derived from NDV values
  - `stats.classes[*].properties[*].refs` only when callers explicitly disable full detail with `with_realtime_property_details(false)`

## Nameservice Query API

Query metadata about all ledgers and graph sources using the `nameservice_query()` builder:

```rust
use fluree_db_api::{FlureeBuilder, Result};
use serde_json::json;

#[tokio::main]
async fn main() -> Result<()> {
    let fluree = FlureeBuilder::file("./data").build()?;

    // Find all ledgers on main branch
    let query = json!({
        "@context": {"f": "https://ns.flur.ee/db#"},
        "select": ["?ledger", "?t"],
        "where": [{"@id": "?ns", "@type": "f:LedgerSource", "f:ledger": "?ledger", "f:branch": "main", "f:t": "?t"}],
        "orderBy": [{"var": "?t", "desc": true}]
    });

    let results = fluree.nameservice_query()
        .jsonld(&query)
        .execute_formatted()
        .await?;

    println!("Ledgers: {}", serde_json::to_string_pretty(&results)?);

    // SPARQL query
    let results = fluree.nameservice_query()
        .sparql("PREFIX f: <https://ns.flur.ee/db#>
                 SELECT ?ledger ?t WHERE { ?ns a f:LedgerSource ; f:ledger ?ledger ; f:t ?t }")
        .execute_formatted()
        .await?;

    println!("SPARQL results: {}", serde_json::to_string_pretty(&results)?);

    // Convenience method (equivalent to builder with defaults)
    let results = fluree.query_nameservice(&query).await?;

    Ok(())
}
```

### Available Properties

**Ledger Records** (`@type: "f:LedgerSource"`):

| Property | Description |
|----------|-------------|
| `f:ledger` | Ledger name (without branch suffix) |
| `f:branch` | Branch name |
| `f:t` | Current transaction number |
| `f:status` | Status: "ready" or "retracted" |
| `f:ledgerCommit` | Reference to latest commit ContentId |
| `f:ledgerIndex` | Index info with `@id` and `f:t` |

**Graph Source Records** (`@type: "f:GraphSourceDatabase"`):

| Property | Description |
|----------|-------------|
| `f:name` | Graph source name |
| `f:branch` | Branch name |
| `f:config` | Configuration JSON |
| `f:dependencies` | Source ledger dependencies |
| `f:indexAddress` | Index ContentId |
| `f:indexT` | Index transaction number |

### Builder Methods

| Method | Description |
|--------|-------------|
| `.jsonld(&query)` | Set JSON-LD query input |
| `.sparql(query)` | Set SPARQL query input |
| `.format(config)` | Override output format |
| `.execute_formatted()` | Execute and return formatted JSON |
| `.execute()` | Execute with default formatting |
| `.validate()` | Validate without executing |

### Example Queries

```rust
// Find ledgers with t > 100
let query = json!({
    "@context": {"f": "https://ns.flur.ee/db#"},
    "select": ["?ledger", "?t"],
    "where": [{"@id": "?ns", "f:ledger": "?ledger", "f:t": "?t"}],
    "filter": ["(> ?t 100)"]
});

// Find all BM25 graph sources
let query = json!({
    "@context": {"f": "https://ns.flur.ee/db#"},
    "select": ["?name", "?deps"],
    "where": [{"@id": "?gs", "@type": "f:Bm25Index", "f:name": "?name", "f:dependencies": "?deps"}]
});
```

## Examples

See complete examples in `fluree-db-api/examples/`:

- `benchmark_aj_query_1.rs` - Basic query patterns
- `benchmark_aj_query_2.rs` - Complex queries
- `benchmark_aj_query_3.rs` - Aggregations
- `benchmark_aj_query_4.rs` - Time travel queries

Run examples:

```bash
cargo run --example benchmark_aj_query_1 --release
```

## API Reference

For detailed API documentation, see:

```bash
cargo doc --open -p fluree-db-api
```

## Related Documentation

- [Getting Started](README.md) - Overview
- [HTTP API](../api/README.md) - Server-based usage
- [Distributed Tracing Integration](../operations/distributed-tracing.md) - Correlating your app's traces with Fluree
- [Query](../query/README.md) - Query documentation
- [Transactions](../transactions/README.md) - Write operations
- [Crate Map](../reference/crate-map.md) - Architecture overview
- [Dev Setup](../contributing/dev-setup.md) - Development guide


---

# concepts/README.md

# Concepts

Fluree is a graph database that stores and queries data using RDF (Resource Description Framework) semantics. This section explains the core concepts that make Fluree unique and powerful, with special emphasis on the features that differentiate Fluree from other graph databases.

## Recommended Reading Order

These concepts build on each other. If you're new to Fluree, read them in this order:

**Foundations** (read these first):

1. [IRIs, Namespaces, and JSON-LD @context](iri-and-context.md) — How Fluree identifies everything
2. [Datatypes and Typed Values](datatypes.md) — Fluree's type system
3. [Ledgers and the Nameservice](ledgers-and-nameservice.md) — The core unit of data storage

**Core capabilities** (read next):

4. [Time Travel](time-travel.md) — Query any point in history
5. [Branching](ledgers-and-nameservice.md#branching) — Git-like branch, merge, and rebase for your data
6. [Datasets and Named Graphs](datasets-and-named-graphs.md) — Partition and query across graphs
7. [Edge Annotations](edge-annotations.md) — Properties on relationships, RDF-star, and property-graph edges

**Differentiating features** (read as needed):

8. [Graph Sources](graph-sources.md) — Integrated search and external data
9. [Policy Enforcement](policy-enforcement.md) — Fine-grained access control
10. [Verifiable Data](verifiable-data.md) — Cryptographic signatures and trust
11. [Reasoning and Inference](reasoning.md) — Derive facts from ontology rules

If you're coming from a SQL/relational background, start with [Fluree for SQL Developers](../getting-started/fluree-for-sql-developers.md) before diving into the concepts above.

## Core Concepts

### [IRIs, Namespaces, and JSON-LD @context](iri-and-context.md)

Understand how Fluree uses Internationalized Resource Identifiers (IRIs) for all data identifiers, how namespaces provide convenient shorthand notation, and how JSON-LD @context enables compact, readable data exchange.

### [Datatypes and Typed Values](datatypes.md)

Explore Fluree's type system, including support for XSD datatypes (strings, numbers, dates, booleans), RDF datatypes, and how all literal values are strongly typed.

### [Ledgers and the Nameservice](ledgers-and-nameservice.md)

Learn about ledgers (Fluree's equivalent of databases), how they're organized with aliases like `mydb:main`, and how the nameservice provides discovery and metadata management across distributed deployments.

### [Time Travel](time-travel.md)

**Differentiator**: Discover Fluree's temporal database capabilities, including transaction-time versioning, historical queries, and the ability to query data "as of" any previous transaction. Every change is preserved, enabling complete audit trails and historical analysis.

### [Datasets and Named Graphs](datasets-and-named-graphs.md)

Learn about SPARQL datasets, named graphs, and how Fluree supports multi-graph queries across different data sources and time periods.

### [Edge Annotations](edge-annotations.md)

Attach properties to a relationship — role, confidence, since-date — without modeling an intermediate node by hand. Edge annotations cover the property-graph shape (relationship properties, parallel edges) and RDF-star / JSON-LD-star asserted-triple annotations on a single ergonomic surface, while leaving plain RDF queries untouched.

### [Graph Sources](graph-sources.md)

**Differentiator**: Fluree's graph source system enables seamless integration of specialized indexes and external data sources. Built-in BM25 full-text search, vector similarity search (ANN), Apache Iceberg integration, and R2RML relational mappings extend Fluree's query capabilities beyond traditional graph queries.

### [Policy Enforcement](policy-enforcement.md)

**Differentiator**: Fluree's policy system provides fine-grained, data-level access control. Policies are enforced at query time, ensuring users only see data they're authorized to access. This enables secure multi-tenant deployments and compliance with data privacy regulations.

### [Verifiable Data](verifiable-data.md)

**Differentiator**: Fluree supports cryptographically signed transactions using JWS (JSON Web Signatures) and Verifiable Credentials. Every transaction can be cryptographically verified, providing tamper-proof audit trails and enabling trustless data exchange.

### [Reasoning and Inference](reasoning.md)

Fluree's built-in reasoning engine derives new facts from ontology declarations (RDFS, OWL) and user-defined Datalog rules. Query for a superclass and get all subclass instances automatically.

## Architecture Overview

Fluree combines several architectural concepts:

- **Triple Store**: All data is stored as RDF triples (subject-predicate-object)
- **Temporal Database**: Every transaction is timestamped, enabling complete historical access
- **Multi-Graph Support**: Data can be partitioned across named graphs
- **JSON-LD Integration**: Native support for JSON-LD with full IRI expansion/compaction
- **SPARQL & JSON-LD Query**: Support for both SPARQL and Fluree's native JSON-LD Query language

## Key Differentiators

What makes Fluree unique:

1. **Built-in Full-Text Search**: BM25 indexing is integrated directly into the database, not a separate system
2. **Vector Similarity Search**: Native support for approximate nearest neighbor (ANN) queries via embedded HNSW indexes or remote search service
3. **Apache Iceberg Integration**: Query data lake formats directly as graph sources
4. **Complete Time Travel**: Every transaction is preserved with full historical query capabilities
5. **Data-Level Policy Enforcement**: Fine-grained access control enforced at query time, not application level
6. **Cryptographically Verifiable**: Transactions can be signed and verified using industry-standard formats (JWS/VC)

These concepts work together to provide a powerful, standards-compliant graph database with temporal capabilities, integrated search, and enterprise-grade security features.


---

# concepts/ledgers-and-nameservice.md

# Ledgers and the Nameservice

Ledgers are Fluree's fundamental unit of data organization—similar to databases in traditional RDBMS systems. The nameservice is the metadata registry that enables ledger discovery, coordination, and management across distributed deployments.

## Ledgers

A **ledger** in Fluree is an independent, versioned graph database containing:

- A complete graph of RDF triples
- Complete transaction history with temporal versioning
- Independent indexing and storage
- Configurable permissions and policies
- Support for multiple branches

### Ledger IDs

Ledgers are identified by **ledger IDs** with the format `ledger-name:branch`.

A ledger ID serves as both a human-readable identifier and the canonical lookup key used across APIs, CLI, and caching.

**Examples:**

- `mydb:main` - Primary branch of the "mydb" ledger
- `customers:dev` - Development branch of the "customers" ledger
- `inventory:prod` - Production branch of the "inventory" ledger
- `tenant/app:feature-x` - Feature branch with hierarchical naming

**Branch Semantics:**

- The `:branch` suffix allows multiple isolated versions of the same logical ledger to coexist
- The default branch name is `main` when not specified (e.g., `mydb` is equivalent to `mydb:main`)
- Branches are independent—changes in one branch don't affect others
- Branch names can include slashes for hierarchical organization

### Ledger Lifecycle

Ledgers are created implicitly through the first transaction and persist until explicitly retracted. Each ledger maintains:

- **Transaction History**: Every change is recorded as a transaction with a unique timestamp (`t`)
- **Current State**: The latest indexed state of all data
- **Novelty Layer**: Uncommitted transactions since the last index
- **Metadata**: Creation time, latest commit, indexing status

**Creation Flow:**

1. First transaction to a ledger ID creates the ledger automatically
2. Transaction is committed and assigned a transaction time (`t`)
3. Commit ID is published to the nameservice
4. Background indexing process creates queryable indexes
5. Index ID is published to the nameservice when complete

**Retraction:**

Ledgers can be marked as retracted (soft delete), which:
- Marks the ledger as inactive in the nameservice
- Preserves storage artifacts
- Prevents normal load/create/write paths from treating the alias as active
- Keeps the alias reserved until an administrator purges or otherwise repairs the nameservice record

## The Nameservice

The **nameservice** is Fluree's metadata registry that enables ledger discovery and coordination. It acts as a directory service, tracking where ledger data is stored and what state each ledger is in.

### Purpose and Role

The nameservice provides:

- **Discovery**: Find ledgers by ledger ID across distributed deployments
- **Coordination**: Track commit and index state for consistency
- **Metadata Management**: Store ledger configuration and status
- **Multi-Process Support**: Enable coordination across multiple Fluree instances

### What the Nameservice Stores

For each ledger, the nameservice maintains a **nameservice record** (`NsRecord`) containing:

#### Core Identifiers

- **`id`**: Canonical ledger ID with branch (e.g., `"mydb:main"`)
- **`name`**: Ledger name without branch suffix (e.g., `"mydb"`)
- **`branch`**: Branch name (e.g., `"main"`)

#### Commit State

- **`commit_id`**: ContentId (CIDv1) of the latest commit
- **`commit_t`**: Transaction time of the latest commit

The commit represents the most recent transaction that has been persisted. Commits are published immediately after each successful transaction. The `commit_id` is a content-addressed identifier derived from the commit's bytes — it is storage-agnostic and does not depend on where the commit is physically stored.

#### Index State

- **`index_id`**: ContentId (CIDv1) of the latest index root
- **`index_t`**: Transaction time of the latest index

The index represents a queryable snapshot of the ledger state. Indexes are created by background processes and may lag behind commits. Like commits, the `index_id` is a content-addressed identifier.

#### Branch Metadata

- **`source_branch`**: For branches created via `create_branch`, records the name of the source branch (e.g., `"main"`). `None` for the initial branch.

The divergence point (common ancestor) between a branch and its source is computed on demand by walking the commit chains rather than being stored. This avoids stale metadata and supports merge scenarios where the relationship between branches changes over time.

#### Additional Metadata

- **`default_context_id`**: ContentId of the default JSON-LD @context for the ledger
- **`retracted`**: Whether the ledger has been marked as inactive

### Commit vs Index: Understanding the Difference

This distinction is crucial for understanding Fluree's architecture:

**Commits (`commit_t`):**
- Created immediately after each transaction
- Represent the transaction log (what changed)
- Small, append-only files
- Published synchronously
- Always up-to-date with latest transactions

**Indexes (`index_t`):**
- Created by background indexing processes
- Represent queryable database snapshots (complete state)
- Large, optimized data structures
- Published asynchronously
- May lag behind commits (this gap is the "novelty layer")

**Example Timeline:**

```text
t=1:  Transaction committed → commit_t=1, index_t=0
t=2:  Transaction committed → commit_t=2, index_t=0
t=3:  Transaction committed → commit_t=3, index_t=0
       [Background indexing completes] → index_t=3
t=4:  Transaction committed → commit_t=4, index_t=3
t=5:  Transaction committed → commit_t=5, index_t=3
       [Novelty layer: t=4, t=5 not yet indexed]
```

Queries combine the indexed state (up to `index_t`) with the novelty layer (transactions between `index_t` and `commit_t`) to provide real-time results.

### Nameservice Operations

The nameservice supports these key operations:

#### Lookup

Find ledger metadata by ledger ID:

```rust
// Pseudo-code
let record = nameservice.lookup("mydb:main").await?;
// Returns: NsRecord with commit_id, index_id, timestamps, etc.
```

#### Publishing

Record new commits and indexes:

- **`RefPublisher::compare_and_set_ref()` / `fast_forward_commit()`**: Advance the commit head with explicit CAS conflict handling
- **`publish_index(ledger_id, index_id, index_t)`**: Update index state (monotonic: only if `new_t > existing_t`)

Commit-head publishing is **CAS-based** so concurrent writers get an explicit conflict result instead of a silent no-op. Index publishing remains monotonic and only accepts updates that advance time forward.

#### Branching

Create and list branches:

- **`create_branch(ledger_name, new_branch, source_branch, at_commit)`**: Create a new branch from the source. When `at_commit` is `None`, the branch starts at the source's current HEAD; when `Some((commit_id, commit_t))`, the branch starts at the supplied historical commit instead (callers are expected to verify reachability from source HEAD before passing it in).
- **`list_branches(ledger_name)`**: List all non-retracted branches for a ledger

#### Discovery

List all available ledgers:

```rust
// Pseudo-code
let all_ledgers = nameservice.all_records().await?;
// Returns: Vec<NsRecord> for all known ledgers
```

### Querying the Nameservice

The nameservice can be queried using standard JSON-LD query or SPARQL syntax. This enables powerful ledger discovery, filtering, and metadata analysis across all managed databases.

#### Rust API (Builder Pattern)

```rust
// Find all ledgers on main branch
let query = json!({
    "@context": {"f": "https://ns.flur.ee/db#"},
    "select": ["?ledger"],
    "where": [{"@id": "?ns", "f:ledger": "?ledger", "f:branch": "main"}]
});

let results = fluree.nameservice_query()
    .jsonld(&query)
    .execute_formatted()
    .await?;

// Query with SPARQL
let results = fluree.nameservice_query()
    .sparql("PREFIX f: <https://ns.flur.ee/db#>
             SELECT ?ledger ?t WHERE {
               ?ns a f:LedgerSource ;
                   f:ledger ?ledger ;
                   f:t ?t
             }")
    .execute_formatted()
    .await?;

// Convenience method (equivalent to builder with defaults)
let results = fluree.query_nameservice(&query).await?;
```

#### HTTP API

```bash
# List ledgers and graph sources from the nameservice
curl http://localhost:8090/v1/fluree/ledgers
```

#### Available Properties

**Ledger Records** (`@type: "f:LedgerSource"`):

| Property | Description |
|----------|-------------|
| `f:ledger` | Ledger name (without branch suffix) |
| `f:branch` | Branch name (e.g., "main", "dev") |
| `f:t` | Current transaction number |
| `f:status` | Status: "ready" or "retracted" |
| `f:ledgerCommit` | Reference to latest commit ContentId |
| `f:ledgerIndex` | Index info object with `@id` (ContentId) and `f:t` |
| `f:sourceBranch` | Source branch name (e.g., `"main"`) if this is a branched ledger |
| `f:defaultContextCid` | Default JSON-LD context ContentId (if set) |

**Graph Source Records** (`@type: "f:GraphSourceDatabase"`):

| Property | Description |
|----------|-------------|
| `f:name` | Graph source name |
| `f:branch` | Branch name |
| `f:status` | Status: "ready" or "retracted" |
| `f:config` | Configuration JSON |
| `f:dependencies` | Array of source ledger dependencies |
| `f:indexId` | Index ContentId |
| `f:indexT` | Index transaction number |

#### Example Queries

**Find all ledgers with t > 100:**
```json
{
  "@context": {"f": "https://ns.flur.ee/db#"},
  "select": ["?ledger", "?t"],
  "where": [
    {"@id": "?ns", "f:ledger": "?ledger", "f:t": "?t"}
  ],
  "filter": ["(> ?t 100)"]
}
```

**Find ledgers by name pattern (hierarchical):**
```json
{
  "@context": {"f": "https://ns.flur.ee/db#"},
  "select": ["?ledger", "?branch"],
  "where": [
    {"@id": "?ns", "f:ledger": "?ledger", "f:branch": "?branch"}
  ],
  "filter": ["(strStarts ?ledger \"tenant1/\")"]
}
```

**Find all BM25 graph sources:**
```json
{
  "@context": {
    "f": "https://ns.flur.ee/db#"
  },
  "select": ["?name", "?deps"],
  "where": [
    {"@id": "?gs", "@type": "f:Bm25Index", "f:name": "?name", "f:dependencies": "?deps"}
  ]
}
```

#### Retraction

Mark ledgers as inactive without deleting storage artifacts:

```rust
// Pseudo-code
nameservice.retract("mydb:old-branch").await?;
// Sets retracted=true; the alias remains reserved.
```

### Storage Backends

The nameservice can be backed by various storage systems, each suited for different deployment scenarios:

#### File System (`FileNameService`)

- **Use Case**: Single-server deployments, development, testing
- **Storage**: Files in `ns@v2/` directory structure
- **Format**: JSON files per ledger (`{ledger}/{branch}.json`)
- **Characteristics**: Simple, local, no external dependencies

#### AWS S3 (`StorageNameService`)

- **Use Case**: Distributed deployments using S3 for both data and metadata
- **Storage**: S3 objects with ETag-based compare-and-swap (CAS)
- **Characteristics**: Scalable, distributed, requires AWS credentials

#### AWS DynamoDB (`DynamoDbNameService`)

- **Use Case**: Distributed deployments needing low-latency metadata coordination
- **Storage**: DynamoDB table with composite-key layout (one item per concern)
- **Format**: Separate items for `meta`, `head`, `index`, `config`, `status` per ledger/graph source
- **Characteristics**: Single-digit millisecond latency, per-concern write independence, conditional expressions for monotonic updates
- See [DynamoDB Nameservice Guide](../operations/dynamodb-guide.md) for setup and schema details

#### Memory (`MemoryNameService`)

- **Use Case**: Testing, in-process applications
- **Storage**: In-memory data structures
- **Format**: No persistence
- **Characteristics**: Fast, ephemeral, process-local

### Graph Sources

The nameservice also tracks **graph sources**—specialized indexes and integrations:

- **BM25**: Full-text search indexes
- **Vector**: Vector similarity search
- **R2RML**: Relational database mappings
- **Iceberg**: Apache Iceberg table integrations

Graph sources have their own nameservice records (`GraphSourceRecord`) with similar metadata but different semantics. See the [Graph Sources](graph-sources.md) documentation for details.

## Example Usage

### Creating a Ledger

Ledgers are created automatically on the first transaction. Specify the ledger ID in your transaction:

```json
POST /insert?ledger=mydb:main
Content-Type: application/json

{
  "@context": {
    "ex": "http://example.org/ns/",
    "foaf": "http://xmlns.com/foaf/0.1/"
  },
  "@graph": [
    {
      "@id": "ex:alice",
      "@type": "foaf:Person",
      "foaf:name": "Alice"
    }
  ]
}
```

**What Happens:**

1. Transaction is processed and committed (assigned `t=1`)
2. Commit is stored and its ContentId published to nameservice
3. Nameservice record created/updated with `commit_t=1`
4. Background indexing begins
5. When indexing completes, `index_t=1` is published

### Querying a Ledger

Specify the ledger ID in your query:

**SPARQL:**

```sparql
PREFIX ex: <http://example.org/ns/>
PREFIX foaf: <http://xmlns.com/foaf/0.1/>

SELECT ?name
FROM <mydb:main>
WHERE {
  ex:alice foaf:name ?name
}
```

The `FROM <mydb:main>` clause specifies which ledger to query. The query engine:
1. Looks up `mydb:main` in the nameservice
2. Retrieves the index ContentId for efficient querying
3. Combines indexed data with novelty layer for current results

**JSON-LD Query:**

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "foaf": "http://xmlns.com/foaf/0.1/"
  },
  "select": ["?name"],
  "from": "mydb:main",
  "where": [
    { "@id": "ex:alice", "foaf:name": "?name" }
  ]
}
```

### Checking Ledger Status

Query the nameservice to check ledger state:

```rust
// Pseudo-code
let record = nameservice.lookup("mydb:main").await?;

if let Some(record) = record {
    println!("Latest commit: t={}", record.commit_t);
    println!("Latest index: t={}", record.index_t);
    
    if record.has_novelty() {
        println!("Novelty layer: {} transactions pending index", 
                 record.commit_t - record.index_t);
    }
    
    if record.retracted {
        println!("Ledger is retracted (inactive)");
    }
}
```

### Branching

Branches let you create isolated copies of a ledger's state for independent development. After branching, transactions on one branch are invisible to the other.

#### Creating a Branch

Branches are created from a source branch (default: `main`). The new branch starts at the same transaction time as the source:

```text
mydb:main (t=5)
  └── create_branch("mydb", "dev")
mydb:dev  (t=5)  # starts with same data as main at t=5
```

Branches can also be nested — you can branch from a branch:

```text
mydb:main (t=5)
  └── mydb:dev (t=7)      # branched from main at t=5, then advanced
        └── mydb:feature (t=8)  # branched from dev at t=7, then advanced
```

#### Data Isolation

After branching, each branch has its own independent transaction history:

```text
mydb:main   → t=5 (shared) → t=6: insert Bob   → t=7: insert Dave
mydb:dev    → t=5 (shared) → t=6: insert Carol
```

Querying `main` returns Alice + Bob + Dave. Querying `dev` returns Alice + Carol. Bob and Dave never appear on `dev`; Carol never appears on `main`.

#### Storage Model

Branches share storage efficiently through a **`BranchedContentStore`** — a recursive content store that reads from the branch's own namespace first, then falls back to parent namespaces for pre-branch-point content.

- **Commits are not copied** — historical commits are read from the source namespace via fallback
- **Index files are copied** — protects the branch from garbage collection on the source after reindexing
- **String dictionaries are globally shared** — stored in a per-ledger `@shared` namespace (e.g., `mydb/@shared/dicts/`) rather than per-branch paths, so all branches read and write to the same location without copying or fallback. The `@` prefix cannot collide with branch names. See [Storage Traits — Global Dictionary Storage](../design/storage-traits.md#global-dictionary-storage-shared-namespace) for details.

Each branch is a fully independent `LedgerState` with its own snapshot, novelty layer, commit chain, storage namespace, and `t` sequence.

#### Nameservice Metadata

When a branch is created, the nameservice records the **source branch name** on the new branch's `NsRecord` (e.g., `source_branch: Some("main")`). The divergence point between the branch and its source is computed on demand by walking the commit chains rather than being stored as a static snapshot.

This metadata enables the system to reconstruct the `BranchedContentStore` tree when loading a branch. For nested branches, the ancestry chain is walked recursively via `source_branch` lookups.

#### API

**Rust:**
```rust
// Create a branch from main (default)
let record = fluree.create_branch("mydb", "dev", None).await?;

// Create a branch from another branch
let record = fluree.create_branch("mydb", "feature", Some("dev")).await?;

// List all branches
let branches = fluree.list_branches("mydb").await?;
```

**HTTP:**
```bash
# Create branch
curl -X POST http://localhost:8090/v1/fluree/branch \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb", "branch": "dev"}'

# List branches
curl http://localhost:8090/v1/fluree/branch/mydb
```

**CLI:**
```bash
# Create branch
fluree branch create dev --ledger mydb

# Create branch from another branch
fluree branch create feature-x --from dev --ledger mydb

# List branches
fluree branch list --ledger mydb
```

#### Dropping a Branch

Branches can be deleted with `drop_branch`. The **root** branch — any branch
whose `source_branch.is_none()` on its `NsRecord` — cannot be dropped via
`drop_branch`; use [`drop_ledger`](#dropping-a-whole-ledger) to remove the
entire ledger including its root. `"main"` is the default branch name and so
is the root on most ledgers, but the refusal is structural: a ledger created
as `mydb:trunk` has `trunk` as its root, and `drop_branch("mydb", "trunk")`
will be refused. Conversely, a non-root branch named `"main"` is droppable.

Branches use **reference counting** (`branches` field on `NsRecord`) to track child branches. This enables safe deletion:

- **Leaf branch** (no children, `branches == 0`): Fully dropped — storage artifacts are deleted, the NsRecord is purged, and the parent's child count is decremented. If the parent was previously retracted and its count reaches 0, it is cascade-dropped.
- **Branch with children** (`branches > 0`): Retracted (hidden from listings, transactions rejected) but storage is preserved so children can still read parent data via `BranchedContentStore` fallback. When the last child is dropped and the count reaches 0, the retracted branch is automatically cascade-purged.

**Rust API:**
```rust
// Drop a leaf branch
let report = fluree.drop_branch("mydb", "dev").await?;

// report.deferred == false for leaf branches
// report.deferred == true for branches with children
// report.cascaded contains any ancestor branches that were cascade-dropped
```

**HTTP API:**
```bash
curl -X POST http://localhost:8090/v1/fluree/drop-branch \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb", "branch": "dev"}'
```

**CLI:**
```bash
fluree branch drop dev --ledger mydb
```

See [POST /branch](../api/endpoints.md#post-branch), [GET /branch/{ledger-name}](../api/endpoints.md#get-branchledger-name), and [POST /drop-branch](../api/endpoints.md#post-drop-branch) for full endpoint details.

### Rebasing a Branch

After a branch diverges from its source, you can **rebase** it to replay its unique commits on top of the source branch's current HEAD. This brings the branch up to date with upstream changes without merging.

Rebase detects conflicts when both the branch and source have modified the same (subject, predicate, graph) tuples. Five conflict resolution strategies are available:

| Strategy | Behavior |
|----------|----------|
| `take-both` (default) | Replay as-is, both values coexist (multi-cardinality) |
| `abort` | Fail on first conflict, no changes applied |
| `take-source` | Drop branch's conflicting flakes (source wins) |
| `take-branch` | Keep branch's flakes, retract source's conflicting values |
| `skip` | Skip entire commit if any flakes conflict |

If the branch has no unique commits, rebase performs a **fast-forward**: it simply updates the branch point to the source's current HEAD without replaying anything.

**Rust API:**
```rust
use fluree_db_api::ConflictStrategy;

let report = fluree.rebase_branch("mydb", "dev", ConflictStrategy::TakeBoth).await?;
// report.replayed — number of commits successfully replayed
// report.conflicts — conflicts detected and resolved
// report.fast_forward — true if no branch commits to replay
```

**HTTP API:**
```bash
curl -X POST http://localhost:8090/v1/fluree/rebase \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb", "branch": "dev", "strategy": "take-both"}'
```

**CLI:**
```bash
fluree branch rebase dev --ledger mydb --strategy take-both
```

See [POST /rebase](../api/endpoints.md#post-rebase) for full endpoint details.

## Architecture Deep Dive

### Ledger State Composition

Each ledger combines two layers for query execution:

#### 1. Indexed Database

- **What**: Persisted, optimized snapshot of ledger state
- **When**: Created by background indexing processes
- **Storage**: Large, read-optimized data structures
- **Query Performance**: Fast, efficient for historical queries
- **Update Frequency**: Asynchronous, may lag behind commits

#### 2. Novelty Overlay

- **What**: In-memory representation of uncommitted transactions
- **When**: Transactions between `index_t` and `commit_t`
- **Storage**: Transaction log entries
- **Query Performance**: Slower, requires transaction replay
- **Update Frequency**: Real-time, always current

**Query Execution Model:**

```text
Query Result = Indexed Database (up to t=index_t) 
             + Novelty Overlay (t=index_t+1 to commit_t)
```

This architecture provides:
- **Fast historical queries**: Use appropriate index snapshot
- **Real-time current queries**: Include latest transactions via novelty
- **Efficient background indexing**: Doesn't block new writes
- **Consistent snapshots**: Each query sees a consistent state

### Concurrency Control

The nameservice ensures consistency through several mechanisms:

#### Ref Publishing

- **Commits**: `RefPublisher` uses compare-and-set semantics on the current head identity plus a monotonic `t` guard
- **Indexes**: `publish_index()` only accepts `new_index_t > existing_index_t`
- **Guarantee**: Writers either advance the head or receive an explicit conflict outcome

#### Optimistic Concurrency

- **CAS Operations**: Storage-backed nameservices use compare-and-swap (ETags)
- **Conflict Handling**: Retry on conflicts (expected under contention)
- **Atomic Updates**: Metadata updates are atomic per ledger

#### Consistency Guarantees

- **Read Consistency**: All readers see the same nameservice state
- **Write Consistency**: Monotonic updates prevent time-travel inconsistencies
- **Eventual Consistency**: In distributed deployments, updates propagate eventually

### Distributed Coordination

The nameservice enables coordination across distributed deployments:

#### Multi-Process Coordination

- **Shared State**: Nameservice provides shared view of ledger state
- **Process Discovery**: Processes can discover ledgers created by other processes
- **State Synchronization**: Commit/index state visible to all processes

#### Geographic Distribution

- **Storage Backends**: S3/DynamoDB enable cross-region coordination
- **Replication**: Storage backends handle replication
- **Consistency**: Eventual consistency with monotonic guarantees

#### Scalability Patterns

- **Horizontal Scaling**: Multiple Fluree instances can share nameservice
- **Load Distribution**: Queries can be distributed across instances
- **Storage Distribution**: Ledger data can be stored across multiple backends

### Nameservice Record Lifecycle

Understanding how records evolve:

```text
1. Initialization
   - init("mydb:main")
   - Creates record with commit_t=0, index_t=0

2. First Transaction
   - Transaction committed at t=1
   - Commit head advanced via `RefPublisher` CAS to `(commit_cid_1, 1)`
   - Record: commit_t=1, index_t=0

3. Indexing Completes
   - Index created for t=1
   - publish_index("mydb:main", index_cid_1, 1)
   - Record: commit_t=1, index_t=1

4. More Transactions
   - Transactions at t=2, t=3, t=4
   - Commit head advanced via CAS for each
   - Record: commit_t=4, index_t=1 (novelty: t=2,3,4)

5. Next Index
   - Index created for t=4
   - publish_index("mydb:main", index_cid_2, 4)
   - Record: commit_t=4, index_t=4 (no novelty)
```

## Best Practices

### Ledger Naming

1. **Use Descriptive Names**: Choose names that clearly indicate purpose
   - Good: `customers:main`, `inventory:prod`, `analytics:warehouse`
   - Bad: `db1:main`, `test:main`, `data:main`

2. **Hierarchical Organization**: Use slashes for logical grouping
   - Good: `tenant/app:main`, `tenant/app:dev`
   - Good: `department/project:branch`

3. **Branch Naming Conventions**: Establish consistent branch naming
   - Good: `feature/authentication`, `bugfix/login-error`
   - Good: `release/v1.2.0`, `hotfix/security-patch`

### Nameservice Configuration

1. **Choose Appropriate Backend**: Match backend to deployment needs
   - Development: File system
   - Single server: File system
   - Distributed/Cloud: S3/DynamoDB

2. **Monitor Novelty Layer**: Track gap between commits and indexes
   - Large gaps indicate indexing lag
   - May need to tune indexing frequency or resources

3. **Handle Retraction Carefully**: Retracted ledgers preserve storage artifacts
   - Use for soft deletes, not hard deletes
   - Do not assume normal query/load APIs will open a retracted ledger; administrative recovery may require nameservice repair or purge

### Performance Considerations

1. **Index Frequency**: Balance indexing frequency with query needs
   - More frequent indexing: Better query performance, more storage
   - Less frequent indexing: Lower overhead, larger novelty layer

2. **Query Patterns**: Understand your query patterns
   - Historical queries: Benefit from frequent indexing
   - Current-only queries: Can tolerate larger novelty layer

3. **Storage Planning**: Plan for index storage growth
   - Each index is a complete snapshot
   - Historical indexes accumulate over time
   - Consider retention policies for old indexes

### Operational Guidelines

1. **Monitor Nameservice Health**: Track nameservice operations
   - Lookup latency
   - Publish success rates
   - Storage backend health

2. **Backup Strategy**: Include nameservice in backup plans
   - File-based: Backup `ns@v2/` directory
   - Storage-based: Use backend backup mechanisms

3. **Error Handling**: Handle nameservice errors gracefully
   - Lookup failures: May indicate ledger doesn't exist
   - Publish failures: May indicate contention (retry)
   - Storage errors: May indicate backend issues

## Troubleshooting

### Ledger Not Found

**Symptom**: Query fails with "ledger not found"

**Possible Causes:**
- Ledger ID misspelled
- Ledger not yet created (no transactions yet)
- Ledger retracted
- Nameservice backend misconfigured

**Solutions:**
- Verify ledger ID spelling and format
- Check if ledger exists: `nameservice.lookup(ledger_id)`
- Verify nameservice backend configuration
- Check ledger status (retracted?)

### Stale Query Results

**Symptom**: Queries don't see latest transactions

**Possible Causes:**
- Novelty layer not being applied
- Index lagging significantly behind commits
- Query caching issues

**Solutions:**
- Check `commit_t` vs `index_t` gap
- Verify indexing process is running
- Check query execution logs
- Consider forcing index update

### Nameservice Contention

**Symptom**: Publish operations failing with conflicts

**Possible Causes:**
- Multiple processes updating same ledger
- High transaction rate
- Storage backend throttling

**Solutions:**
- Implement retry logic with backoff
- Reduce transaction rate if possible
- Scale storage backend (if S3/DynamoDB)
- Check for process coordination issues

This foundation of ledgers and the nameservice enables Fluree's distributed, temporal graph database capabilities, providing the coordination layer needed for scalable, consistent data management.

**Differentiator**: Fluree's nameservice architecture enables true distributed deployments with coordination across multiple processes and machines, unlike single-instance databases. The separation of commits and indexes, combined with the novelty layer, enables real-time queries while maintaining efficient background indexing—a unique architectural advantage.


---

# concepts/graph-sources.md

# Graph Sources

**Differentiator**: Graph sources are one of Fluree's most powerful features, enabling seamless integration of specialized indexes and external data sources directly into graph queries. Unlike traditional databases that require separate systems for full-text search, vector similarity, or data lake access, Fluree makes these capabilities first-class citizens in the query language.

## What Are Graph Sources?

A **graph source** is anything you can address by a graph name/IRI in Fluree query execution. Graph sources may be backed by:
- **Ledger graphs** (default graph and named graphs stored as RDF triples)
- **Index graph sources** (BM25 and vector/HNSW indexes)
- **Mapped graph sources** (R2RML and Iceberg-backed mappings)

### Key Characteristics

- **Query integration**: Graph sources can be queried using the same SPARQL and JSON-LD Query interfaces
- **Transparent access**: Applications don't need to know whether data comes from a ledger graph source or a non-ledger graph source
- **Specialization**: Each graph source type is optimized for specific query patterns
- **Time travel (type-specific)**: Some graph sources support time-travel queries, but support is not uniform across all types. Time-travel is implemented by each graph source type (not by the nameservice).

## Graph Source Types

### BM25 Full-Text Search

**Differentiator**: Fluree includes built-in BM25 full-text search indexing, eliminating the need for separate search systems like Elasticsearch.

**Use Cases:**
- Product search with relevance ranking
- Document search with keyword matching
- Content discovery with fuzzy matching

**Example:**

```json
{
  "@context": {
    "f": "https://ns.flur.ee/db#"
  },
  "from": "products:main",
  "select": ["?product", "?score"],
  "where": [
    {
      "f:graphSource": "products-search:main",
      "f:searchText": "laptop",
      "f:searchLimit": 10,
      "f:searchResult": { "f:resultId": "?product", "f:resultScore": "?score" }
    }
  ],
  "orderBy": [["desc", "?score"]],
  "limit": 10
}
```

**Key Features:**
- Relevance scoring (BM25 algorithm)
- Configurable parameters (k1, b)
- Language-aware search
- Optional time-travel support (BM25-owned manifest; see “Time Travel” below)

See the [BM25 documentation](../indexing-and-search/bm25.md) for details.

### Vector Similarity Search (ANN)

**Differentiator**: Native support for approximate nearest neighbor (ANN) queries via embedded HNSW indexes, enabling semantic search and similarity queries. Can run embedded (in-process) or via a dedicated remote search service.

**Use Cases:**
- Semantic search (find similar documents)
- Recommendation systems
- Image similarity search
- Embedding-based queries

**Key Features:**
- Approximate nearest neighbor search (HNSW algorithm)
- Configurable distance metrics (cosine, euclidean, dot product)
- Embedded indexes (no external service required) or remote mode via `fluree-search-httpd`
- Support for high-dimensional vectors
- Snapshot-based persistence with watermarks (head-only in v1; time-travel not supported)

See the [Vector Search documentation](../indexing-and-search/vector-search.md) for details.

### Apache Iceberg Integration

**Differentiator**: Query Apache Iceberg tables and Parquet files directly as graph sources, enabling seamless integration with data lake architectures.

**Use Cases:**
- Query data lake formats without ETL
- Combine graph data with tabular data
- Analytics queries over large datasets
- Integration with existing data pipelines

**Example:**

```sparql
# Query Iceberg table as graph source
SELECT ?customer ?order ?amount
FROM <iceberg:sales:main>
WHERE {
  ?order ex:customer ?customer .
  ?order ex:amount ?amount .
  FILTER(?amount > 1000)
}
```

**Key Features:**
- Direct querying of Iceberg tables
- Parquet file support
- R2RML mapping for tabular data (Iceberg-backed)
- Time-travel via Iceberg snapshots
- **Direct S3 mode**: bypass REST catalog servers for `iceberg-rust` / self-managed tables — reads `version-hint.text` for automatic version discovery

See the [Iceberg documentation](../graph-sources/iceberg.md) for details.

### R2RML Relational Mapping

**Differentiator**: Map relational databases to RDF using R2RML (R2RML Mapping Language), enabling graph queries over SQL databases.

**Use Cases:**
- Adopt graph queries alongside SQL data sources
- Query SQL databases using SPARQL
- Integrate existing systems
- Unified query interface across data sources

**Example:**

```sparql
# Query relational database via R2RML mapping
SELECT ?customer ?order
FROM <r2rml:orders:main>
WHERE {
  ?customer ex:hasOrder ?order .
  ?order ex:status "pending" .
}
```

**Key Features:**
- R2RML standard compliance
- Automatic RDF mapping from SQL schemas
- Read-only access to source databases
- Support for complex joins and transformations

See the [R2RML documentation](../graph-sources/r2rml.md) for details.

## Graph Source Lifecycle

### Creation

Graph sources are created through administrative operations, specifying:
- **Type**: BM25, Vector, Iceberg, or R2RML
- **Configuration**: Type-specific settings
- **Dependencies**: Source ledgers or data sources
- **Branch**: Graph sources support branching like ledgers

**Example BM25 Graph Source Creation:**

```json
{
  "@type": "f:Bm25Index",
  "f:name": "products-search",
  "f:branch": "main",
  "f:sourceLedger": "products:main",
  "f:config": {
    "k1": 1.2,
    "b": 0.75,
    "fields": ["name", "description"]
  }
}
```

### Indexing

Graph sources maintain their own indexes:
- **BM25**: Full-text indexes are built from source ledger data
- **Vector**: Embeddings stored in HNSW indexes (embedded or remote)
- **Iceberg**: Metadata is cached for efficient querying
- **R2RML**: Mapping rules are applied to generate RDF

### Querying

Graph sources are queried like regular ledgers:

```sparql
# Query any graph source
SELECT ?result
FROM <graph-source-name:branch>
WHERE {
  # Query patterns specific to graph source type
}
```

### Time Travel

Some graph sources support historical queries using the `@t:` syntax in the ledger reference, but the behavior is **graph-source-type specific**:

```json
{
  "@context": { "f": "https://ns.flur.ee/db#" },
  "from": "products:main@t:1000",
  "select": ["?product"],
  "where": [
    {
      "f:graphSource": "products-search:main",
      "f:searchText": "laptop",
      "f:searchLimit": 20,
      "f:searchResult": { "f:resultId": "?product" }
    }
  ]
}
```

#### BM25

BM25 can support time travel by maintaining a **BM25-owned manifest** in storage that maps transaction watermarks (`t`) to index snapshot addresses. The nameservice stores only a **head pointer** (an opaque address to the latest BM25 manifest/root) and does not store snapshot history.

#### Vector

Vector search is **head-only** in v1. If a query requests an `@t:` (or otherwise requests an historical view), vector search rejects the request with a clear “time-travel not supported” error.

#### Iceberg

Iceberg time travel (when used) is handled by **Iceberg’s own snapshot/metadata model**, not by nameservice-managed snapshot history.

## Graph Source Architecture

### Nameservice Integration

Graph sources are tracked in the nameservice alongside ledgers:

- **Discovery**: List all graph sources via nameservice
- **Metadata**: Configuration and status stored in nameservice
- **Coordination**: Index state tracked separately from source ledgers

**Important**: for graph sources, the nameservice stores only **configuration** and a **head pointer** (as a ContentId) to the graph source's latest index root/manifest. Snapshot history (if any) lives in graph-source-owned manifests in the content store.

### Query Execution

When querying a graph source:

1. **Resolution**: Query engine resolves graph source from nameservice
2. **Type Detection**: Determines graph source type (BM25, Vector, etc.)
3. **Specialized Execution**: Routes to type-specific query handler
4. **Result Integration**: Results integrated with regular graph queries

### Performance Characteristics

Each graph source type has different performance characteristics:

- **BM25**: Fast keyword search, relevance scoring
- **Vector**: Approximate similarity search, configurable accuracy/speed tradeoff
- **Iceberg**: Columnar storage, efficient for analytical queries
- **R2RML**: Depends on source database performance

## Use Cases

### Multi-Modal Search

Combine full-text search, vector similarity, and graph queries:

```json
{
  "@context": {
    "ex": "http://example.org/",
    "f": "https://ns.flur.ee/db#"
  },
  "from": "products:main",
  "select": ["?product", "?textScore", "?vectorScore"],
  "values": [
    ["?queryVec"],
    [{"@value": [0.1, 0.2, 0.3], "@type": "https://ns.flur.ee/db#embeddingVector"}]
  ],
  "where": [
    { "@id": "?product", "ex:category": "electronics" },
    {
      "f:graphSource": "products-search:main",
      "f:searchText": "wireless",
      "f:searchLimit": 20,
      "f:searchResult": { "f:resultId": "?product", "f:resultScore": "?textScore" }
    },
    {
      "f:graphSource": "products-vector:main",
      "f:queryVector": "?queryVec",
      "f:searchLimit": 10,
      "f:searchResult": { "f:resultId": "?product", "f:resultScore": "?vectorScore" }
    }
  ],
  "orderBy": [["desc", "(?textScore + ?vectorScore)"]]
}
```

Vector/HNSW graph sources are currently queried via JSON-LD Query using `f:*` patterns (e.g. `f:graphSource`, `f:queryVector`, `f:searchResult`). SPARQL query syntax for HNSW vector indexes is not currently available.

### Data Lake Integration

Query both graph and tabular data:

```sparql
SELECT ?customer ?graphData ?lakeData
FROM <customers:main>           # Graph ledger
FROM <iceberg:sales:main>        # Iceberg graph source
WHERE {
  # Graph data
  ?customer ex:preferences ?graphData .
  
  # Data lake data
  GRAPH <iceberg:sales:main> {
    ?sale ex:customer ?customer .
    ?sale ex:total ?lakeData .
  }
}
```

### Hybrid Search

Combine semantic and keyword search:

```json
{
  "@context": {
    "f": "https://ns.flur.ee/db#"
  },
  "from": "documents:main",
  "select": ["?document"],
  "where": [
    {
      "f:graphSource": "documents-search:main",
      "f:searchText": "machine learning",
      "f:searchLimit": 20,
      "f:searchResult": { "f:resultId": "?document" }
    }
  ]
}
```

Semantic similarity via HNSW vector indexes is also queried via JSON-LD Query using `f:*` patterns. SPARQL syntax for BM25 and vector index search is not currently available.

## Best Practices

### Graph Source Design

1. **Choose Appropriate Type**: Match graph source type to query patterns
   - Keyword search → BM25
   - Semantic search → Vector
   - Analytics → Iceberg
   - SQL integration → R2RML

2. **Configuration Tuning**: Optimize graph source parameters
   - BM25: Tune k1 and b for relevance
   - Vector: Choose appropriate distance metric
   - Iceberg: Optimize partition strategy

3. **Dependency Management**: Understand source data dependencies
   - BM25/Vector: Keep in sync with source ledger
   - Iceberg: Handle schema evolution
   - R2RML: Map schema changes

### Performance Optimization

1. **Index Maintenance**: Keep graph source indexes up-to-date
   - Monitor indexing lag
   - Tune indexing frequency
   - Handle large data volumes

2. **Query Planning**: Optimize queries using graph sources
   - Use graph sources for appropriate query patterns
   - Combine with graph queries efficiently
   - Consider cost of graph source queries

3. **Caching**: Cache frequently accessed graph source results
   - Cache query results when appropriate
   - Consider graph source snapshot caching
   - Balance freshness vs performance

### Operational Considerations

1. **Monitoring**: Track graph source health
   - Index build status
   - Query performance
   - Storage usage

2. **Backup**: Include graph sources in backup strategy
   - BM25 indexes can be rebuilt (or restored from stored snapshots/manifests, depending on configuration)
   - Vector indexes are stored as head snapshots (time-travel not supported in v1)
   - Iceberg metadata in nameservice

3. **Scaling**: Plan for graph source scaling
   - BM25: Scale with source ledger size
   - Vector: Scale with embedding count
   - Iceberg: Leverage Iceberg partitioning

## Comparison with Traditional Approaches

### Traditional Architecture

```
Application
    ├── Graph Database (Neo4j, etc.)
    ├── Search Engine (Elasticsearch)
    ├── Vector DB (Pinecone, etc.)
    └── Data Lake (Spark, Presto)
```

**Challenges:**
- Multiple systems to manage
- Data synchronization complexity
- Different query languages
- Separate authentication/authorization

### Fluree Graph Source Architecture

```
Application
    └── Fluree
        ├── Graph Ledgers
        ├── BM25 Graph Sources (built-in)
        ├── Vector Graph Sources
        └── Iceberg Graph Sources
```

**Benefits:**
- Single query interface (SPARQL/JSON-LD Query)
- Unified access control (policy enforcement)
- Consistent time-travel across all data
- Simplified operations and deployment

Graph sources make Fluree a unified platform for graph, search, vector, and data lake queries, eliminating the complexity of managing multiple specialized systems.


---

# concepts/iri-and-context.md

# IRIs, Namespaces, and JSON-LD @context

## Internationalized Resource Identifiers (IRIs)

In Fluree, all data identifiers use **Internationalized Resource Identifiers (IRIs)** - the internationalized version of URIs. IRIs uniquely identify:

- **Subjects**: Entities in your data (people, products, concepts)
- **Predicates**: Relationships or properties
- **Objects**: Values or other entities
- **Graphs**: Named data partitions

### IRI Examples

```turtle
# Full IRIs
<http://example.org/person/alice> <http://xmlns.com/foaf/0.1/name> "Alice" .
<http://example.org/person/alice> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://xmlns.com/foaf/0.1/Person> .

# IRIs with Unicode characters
<http://例え.org/人物/アリス> <http://xmlns.com/foaf/0.1/name> "アリス" .
```

### IRI Best Practices

- **Use stable domains**: Choose domains you control or well-established standards
- **Hierarchical structure**: Organize IRIs with meaningful paths
- **Avoid query parameters**: IRIs should be clean identifiers, not URLs with parameters
- **Internationalization**: IRIs support Unicode characters for global identifiers

## Namespaces

**Namespaces** provide shorthand notation for IRIs, making data more readable and manageable. A namespace maps a prefix to a base IRI.

### Defining Namespaces

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "foaf": "http://xmlns.com/foaf/0.1/",
    "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
    "xsd": "http://www.w3.org/2001/XMLSchema#"
  }
}
```

### Using Namespaced IRIs

With the above context, you can write compact IRIs:

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "foaf": "http://xmlns.com/foaf/0.1/"
  },
  "@graph": [
    {
      "@id": "ex:alice",
      "@type": "foaf:Person",
      "foaf:name": "Alice Smith"
    }
  ]
}
```

This expands to:

```json
{
  "@graph": [
    {
      "@id": "http://example.org/ns/alice",
      "@type": "http://xmlns.com/foaf/0.1/Person",
      "http://xmlns.com/foaf/0.1/name": "Alice Smith"
    }
  ]
}
```

## JSON-LD @context

The **@context** is a JSON-LD mechanism that defines how to interpret the data. In Fluree, @context serves multiple purposes:

### IRI Expansion/Compaction

```json
{
  "@context": {
    "name": "http://xmlns.com/foaf/0.1/name",
    "Person": "http://xmlns.com/foaf/0.1/Person"
  },
  "@graph": [
    {
      "@id": "http://example.org/alice",
      "@type": "Person",
      "name": "Alice"
    }
  ]
}
```

The @context maps `name` → `http://xmlns.com/foaf/0.1/name` and `Person` → `http://xmlns.com/foaf/0.1/Person`.

### Standard Prefixes

Fluree includes many standard prefixes by default:

```json
{
  "@context": {
    "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
    "rdfs": "http://www.w3.org/2000/01/rdf-schema#",
    "xsd": "http://www.w3.org/2001/XMLSchema#",
    "owl": "http://www.w3.org/2002/07/owl#",
    "foaf": "http://xmlns.com/foaf/0.1/",
    "dc": "http://purl.org/dc/elements/1.1/"
  }
}
```

### @context in Queries

@context is also used in query results for compact output:

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "foaf": "http://xmlns.com/foaf/0.1/"
  },
  "@graph": [
    {
      "@id": "ex:alice",
      "@type": "foaf:Person",
      "foaf:name": "Alice"
    }
  ]
}
```

### How `@vocab` and `@base` compact results

When a query `@context` sets `@vocab` and/or `@base`, Fluree follows JSON-LD 1.1
about *which* IRIs each keyword may shorten:

- **`@vocab`** governs **property names, `@type` values, and `@type: @vocab`
  term values** only. A predicate or class IRI that falls under `@vocab` is
  emitted as a bare term (e.g. `name`, `Person`).
- **`@base`** governs **node identifiers** — `@id` and reference values. An
  `@id` that falls under `@base` is emitted as a base-relative reference.
- **`@id` is never shortened by `@vocab`.** A node identifier whose IRI falls
  under the `@vocab` namespace keeps its full IRI (or an explicit-prefix /
  `@base`-relative form) — it is *not* collapsed to a bare term, because a
  compliant JSON-LD processor would re-resolve a bare `@id` against `@base`,
  not `@vocab`.

Explicit prefix/term mappings apply in both positions. As a special case,
setting `@vocab: ""` maps the vocabulary to `@base`, so bare terms and
base-relative identifiers share the same namespace.

## IRI Resolution Rules

Fluree follows strict IRI resolution rules:

### Absolute IRIs

These are used as-is:
- `http://example.org/person/alice`
- `https://data.example.com/product/123`

### Prefixed IRIs

These expand using @context:
- `ex:alice` → `http://example.org/ns/alice` (if `ex` maps to `http://example.org/ns/`)
- `foaf:name` → `http://xmlns.com/foaf/0.1/name`

### Relative IRIs

These are resolved relative to a base IRI:
- `alice` → `http://example.org/ns/alice` (if base is `http://example.org/ns/`)

## Strict Compact-IRI Guard

JSON-LD parsing in Fluree (queries and transactions) is **strict by default** about compact IRIs. If you write a value that *looks* like a compact IRI — `prefix:suffix` — but the prefix is not defined in `@context`, Fluree rejects the request at parse time with a clear error:

```text
Unresolved compact IRI 'ex:Person': prefix 'ex' is not defined in @context.
If this is intended as an absolute IRI, use a full form (e.g. http://...)
or add the prefix to @context.
```

### Why strict by default

Without the guard, a missing or misspelled prefix passes through silently — `ex:Person` gets stored as the literal string `"ex:Person"` instead of being expanded to a real IRI like `http://example.org/Person`. This produces incorrect data and confusing query results that are very hard to diagnose later.

The guard catches the most common cause of these bugs: forgetting an `@context`.

### What the guard accepts

- IRIs that resolve through `@context` (the normal happy path).
- Hierarchical absolute IRIs whose suffix starts with `//` — `http://...`, `https://...`, `ftp://...`, etc.
- A small allowlist of well-known non-hierarchical schemes — `urn:`, `did:`, `mailto:`, `tel:`, `data:`, `ipfs:`, `ipns:`, `geo:`, `blob:`, `magnet:`, `fluree:`. Scheme names are matched case-insensitively per RFC 3986.
- Variables (`?x`) and blank nodes (`_:b0`) bypass the guard entirely.

### Where the guard applies

The guard runs at every position that semantically expects an IRI in JSON-LD:

- `@id`, `@type`, predicates / property names
- Datatype IRIs in `@type` of `@value` objects
- Graph names and graph-crawl roots
- Selection predicates (forward and reverse)
- VALUES `@id` cells
- `@path` aliases inside `@context`

It does **not** apply to:

- SPARQL queries
- Turtle / TriG transactions
- Literal string values (only IRI positions)
- Other consumers of the underlying JSON-LD expander (e.g. connection-config parsing)

### Opting out per request

If you really need to accept unresolved compact-looking strings — for example, when migrating legacy data that uses bare `prefix:suffix` strings as opaque identifiers — set `opts.strictCompactIri: false` in the JSON-LD payload itself:

```json
{
  "@context": {"ex": "http://example.org/ns/"},
  "opts": {"strictCompactIri": false},
  "@graph": [
    {"@id": "ex:alice", "ex:name": "Alice"},
    {"@id": "legacy:bob", "ex:name": "Bob"}
  ]
}
```

The same key works on both queries and transactions. The default is `true`. Keep it on unless you have a concrete reason to disable it.

For programmatic use from Rust, transactions can also set `TxnOpts.strict_compact_iri` directly; that takes precedence over `opts.strictCompactIri` in the JSON.

## Blank Nodes and Anonymous Entities

**Blank nodes** represent entities without global identifiers:

```json
{
  "@graph": [
    {
      "@id": "_:b1",
      "foaf:name": "Anonymous Person"
    }
  ]
}
```

Blank nodes are:
- Local to a single transaction
- Cannot be referenced across transactions
- Useful for temporary or anonymous data

## Best Practices

### Namespace Organization

1. **Use stable prefixes**: Don't change prefix mappings once data is committed
2. **Standard vocabularies**: Use well-known prefixes (foaf, dc, rdf, etc.)
3. **Custom domains**: Use your own domain for application-specific terms
4. **Versioning**: Consider versioning in namespace IRIs for evolution

### IRI Design

1. **Descriptive paths**: Use meaningful hierarchical paths
2. **Avoid special characters**: Stick to URL-safe characters
3. **Consistent casing**: Use consistent capitalization conventions
4. **Future-proofing**: Design IRIs to accommodate future extensions

### @context Management

1. **Shared contexts**: Reuse @context definitions across transactions
2. **Minimal contexts**: Only define prefixes you actually use
3. **Documentation**: Document custom prefixes and their meanings
4. **Evolution**: Plan for @context changes over time

## Default Context

Each ledger can store a **default context** — a JSON object mapping prefixes to IRIs. This context is available for retrieval and can be injected into queries by compatibility surfaces (the Fluree HTTP server and CLI), but is **not** applied automatically by the core API (`fluree-db-api`).

### How it's populated

- **Bulk import:** When importing Turtle data via `fluree create --from`, all `@prefix` declarations are captured and stored as the ledger's default context, augmented with built-in prefixes (`rdf`, `rdfs`, `xsd`, `owl`, `sh`, `geo`).
- **Manual update:** Use the CLI (`fluree context set`) or HTTP API (`PUT /v1/fluree/context/{ledger...}`) to set or replace the context at any time.

### Core API behavior

When using `fluree-db-api` directly (e.g., embedding Fluree in a Rust application), queries must supply their own `@context` (JSON-LD) or `PREFIX` declarations (SPARQL). If a query omits context, IRIs are not compacted and compact IRIs without a matching prefix will produce an error.

To opt in to default context injection when using the API directly, fetch the stored context and use the `with_default_context` builder:

```rust
let ctx = fluree.get_default_context("mydb").await?;
let ledger = fluree.ledger("mydb").await?;
let view = GraphDb::from_ledger_state(&ledger)
    .with_default_context(ctx);
```

Or use the convenience method:

```rust
let view = fluree.db_with_default_context("mydb").await?;
```

### Server and CLI behavior

The **CLI** automatically injects the ledger's default context into queries that don't provide their own. The HTTP API defaults this behavior off; pass `?default-context=true` on a query request to opt in.

When default context injection is enabled:

1. **Query-level `@context`** (JSON-LD) or **`PREFIX` declarations** (SPARQL) — always win
2. **Ledger default context** — applied only when the query provides no context of its own
3. **Built-in prefixes** — `rdf`, `rdfs`, `xsd`, etc. are always available

### Use with SPARQL (server/CLI)

The default context provides prefix definitions for SPARQL queries, so you don't need to repeat `PREFIX` declarations in every query when injection is enabled. If the ledger's default context includes `{"ex": "http://example.org/"}`, then you can write:

```sparql
SELECT ?name WHERE {
  ex:alice ex:name ?name .
}
```

without an explicit `PREFIX ex: <http://example.org/>` declaration. If you declare any `PREFIX` in the query, the default context is not used at all — you must declare every prefix you need.

### Use with JSON-LD queries (server/CLI)

Similarly, JSON-LD queries sent through an opt-in surface that omit `@context` receive the default context:

```json
{
  "select": ["?name"],
  "where": [["ex:alice", "ex:name", "?name"]]
}
```

### Viewing and updating

```bash
# View the default context
fluree context get mydb

# Replace it
fluree context set mydb -e '{"ex": "http://example.org/", "foaf": "http://xmlns.com/foaf/0.1/"}'
```

Via the HTTP API:

```bash
# Read
curl http://localhost:8090/v1/fluree/context/mydb:main

# Replace
curl -X PUT http://localhost:8090/v1/fluree/context/mydb:main \
  -H "Content-Type: application/json" \
  -d '{"ex": "http://example.org/"}'
```

See [CLI context command](../cli/context.md) and [API endpoints](../api/endpoints.md#get-contextledger) for full details.

### Opting out of the default context

When using a default-context-enabled surface, you may want full, unexpanded IRIs in query results — for debugging, interoperability with other RDF tools, or simply to avoid any prefix assumptions. You can opt out of the default context:

**JSON-LD queries** — pass an empty `@context` object:

```json
{
  "@context": {},
  "select": ["?s", "?p", "?o"],
  "where": [["?s", "?p", "?o"]]
}
```

Results will contain full IRIs (e.g., `http://example.org/ns/alice`) instead of compacted forms (`ex:alice`).

**SPARQL queries** — include any `PREFIX` declaration. When a query declares its own prefixes, the default context is not injected. To opt out without defining any real prefix, use an empty default prefix:

```sparql
PREFIX : <>
SELECT ?s ?p ?o WHERE { ?s ?p ?o }
```

Or simply declare the specific prefixes you need — the default context is only injected when the query has *no* `PREFIX` declarations whatsoever.

### Storage

The default context is stored as a content-addressed blob in CAS, with a pointer (ContentId) in the nameservice config. Updates use compare-and-set semantics, so concurrent writers are safely handled. After an update, the server invalidates the cached ledger state so subsequent operations use the new context.

## Integration with Standards

Fluree's IRI system is fully compatible with:

- **RDF Standards**: Works with RDF/XML, Turtle, N-Triples
- **SPARQL**: IRIs work seamlessly in SPARQL queries
- **Linked Data**: Enables publishing and consuming linked data
- **Semantic Web**: Supports OWL ontologies and RDF Schema

This foundation enables Fluree to participate in the broader semantic web ecosystem while providing the convenience of JSON-LD's compact syntax.

---

# concepts/datatypes.md

# Datatypes and Typed Values

Fluree enforces strong typing for all literal values, ensuring data consistency and enabling efficient indexing and querying. Every literal value has an explicit datatype, following RDF and XSD standards.

## Core Principle: No Untyped Literals

Unlike some databases that allow "plain" strings, Fluree requires every literal to have a datatype. This design provides:

- **Type Safety**: Prevents type confusion in queries and applications
- **Consistent Comparisons**: Typed values compare predictably
- **Standards Compliance**: Follows RDF and SPARQL specifications
- **Query Optimization**: Enables efficient indexing and query planning

## XSD Datatypes

Fluree supports the core XML Schema Definition (XSD) datatypes:

### String Types

```json
{
  "@context": {
    "xsd": "http://www.w3.org/2001/XMLSchema#",
    "ex": "http://example.org/ns/"
  },
  "@graph": [
    {
      "@id": "ex:book1",
      "ex:title": "The Great Gatsby",
      "ex:author": {
        "@value": "F. Scott Fitzgerald",
        "@type": "xsd:string"
      }
    }
  ]
}
```

**xsd:string** is the default for plain string literals when no type is specified.

### Numeric Types

```json
{
  "@graph": [
    {
      "@id": "ex:product1",
      "ex:price": {
        "@value": "29.99",
        "@type": "xsd:decimal"
      },
      "ex:quantity": {
        "@value": "100",
        "@type": "xsd:integer"
      },
      "ex:rating": {
        "@value": "4.5",
        "@type": "xsd:double"
      }
    }
  ]
}
```

Supported numeric types:
- **xsd:integer**: Whole numbers (-∞, ∞)
- **xsd:long**: 64-bit integers
- **xsd:int**: 32-bit integers
- **xsd:short**: 16-bit integers
- **xsd:byte**: 8-bit integers
- **xsd:decimal**: Arbitrary precision decimals
- **xsd:double**: 64-bit floating point
- **xsd:float**: 32-bit floating point

### Boolean Type

```json
{
  "@graph": [
    {
      "@id": "ex:user1",
      "ex:isActive": {
        "@value": "true",
        "@type": "xsd:boolean"
      },
      "ex:hasVerifiedEmail": {
        "@value": "false",
        "@type": "xsd:boolean"
      }
    }
  ]
}
```

**xsd:boolean** accepts: `true`, `false`, `1`, `0`.

### Date and Time Types

```json
{
  "@graph": [
    {
      "@id": "ex:event1",
      "ex:startDate": {
        "@value": "2024-01-15",
        "@type": "xsd:date"
      },
      "ex:startTime": {
        "@value": "14:30:00Z",
        "@type": "xsd:time"
      },
      "ex:createdAt": {
        "@value": "2024-01-15T14:30:00Z",
        "@type": "xsd:dateTime"
      }
    }
  ]
}
```

Temporal types:
- **xsd:date**: Dates without time (e.g., `2024-01-15`)
- **xsd:time**: Times without date (e.g., `14:30:00Z`)
- **xsd:dateTime**: Full timestamps (e.g., `2024-01-15T14:30:00Z`)

### Other XSD Types

```json
{
  "@graph": [
    {
      "@id": "ex:resource1",
      "ex:homepage": {
        "@value": "https://example.com",
        "@type": "xsd:anyURI"
      },
      "ex:duration": {
        "@value": "PT1H30M",
        "@type": "xsd:duration"
      }
    }
  ]
}
```

Additional types include:
- **xsd:anyURI**: Web addresses and identifiers
- **xsd:duration**: Time periods (ISO 8601 format)
- **xsd:gYear**, **xsd:gMonth**, **xsd:gDay**: Partial date components

## RDF Datatypes

Beyond XSD, Fluree supports RDF-specific datatypes:

### Language-Tagged Strings

```json
{
  "@graph": [
    {
      "@id": "ex:book1",
      "ex:title": {
        "@value": "The Great Gatsby",
        "@language": "en"
      },
      "ex:titel": {
        "@value": "Der große Gatsby",
        "@language": "de"
      }
    }
  ]
}
```

**rdf:langString** represents strings with language tags. This is distinct from plain strings and enables language-aware queries.

> **Annotating literal values.** Any literal — plain, typed, or language-tagged — can carry statement-level metadata (source, confidence, timestamp) via an edge annotation. Because a JSON scalar has no room for sibling keys, an annotated literal must be written in value-object form (`@value` plus `@annotation`); language-tagged annotations are language-pinned, so `"chat"@fr` and `"chat"@en` annotate independently. See [Edge annotations → Annotating literal-valued edges](edge-annotations.md#annotating-literal-valued-edges).

### JSON Data

```json
{
  "@graph": [
    {
      "@id": "ex:config1",
      "ex:settings": {
        "@value": "{\"theme\": \"dark\", \"notifications\": true}",
        "@type": "@json"
      }
    }
  ]
}
```

**rdf:JSON** stores JSON data as typed literals. This is useful for storing complex structured data that doesn't fit the RDF model.

### Geographic Data

```json
{
  "@context": {
    "geo": "http://www.opengis.net/ont/geosparql#",
    "ex": "http://example.org/"
  },
  "@graph": [
    {
      "@id": "ex:location1",
      "ex:coordinates": {
        "@value": "POINT(2.3522 48.8566)",
        "@type": "geo:wktLiteral"
      }
    }
  ]
}
```

**geo:wktLiteral** stores geographic data in Well-Known Text (WKT) format. POINT geometries are automatically converted to an optimized binary encoding, while other geometry types (polygons, lines) are stored as strings.

See [Geospatial](../indexing-and-search/geospatial.md) for complete documentation.

### Vector Data

```json
{
  "@context": {
    "ex": "http://example.org/"
  },
  "@graph": [
    {
      "@id": "ex:doc1",
      "ex:embedding": {
        "@value": [0.1, 0.2, 0.3, 0.4],
        "@type": "@vector"
      }
    }
  ]
}
```

**@vector** (full IRI: `https://ns.flur.ee/db#embeddingVector`, prefix form: `f:embeddingVector`) stores numeric arrays as embedding vectors. Values are quantized to IEEE-754 f32 at ingest for compact storage and SIMD-accelerated similarity computation. In Turtle/SPARQL, use `f:embeddingVector` with the `^^` typed-literal syntax.

Without this type annotation, plain JSON arrays are decomposed into individual RDF values where duplicates may be removed and ordering is lost.

See [Vector Search](../indexing-and-search/vector-search.md) for complete documentation.

### Fulltext Data

```json
{
  "@context": {
    "ex": "http://example.org/"
  },
  "@graph": [
    {
      "@id": "ex:article-1",
      "ex:content": {
        "@value": "Rust is a systems programming language focused on safety and performance",
        "@type": "@fulltext"
      }
    }
  ]
}
```

**@fulltext** (full IRI: `https://ns.flur.ee/db#fullText`, prefix form: `f:fullText`) marks a string value for full-text search indexing. Values annotated with `@fulltext` are automatically analyzed (tokenized, stemmed, stopword-filtered) and indexed into per-predicate fulltext arenas during background index builds. This enables BM25-ranked relevance scoring via the `fulltext()` query function.

Without this type annotation, strings are stored as plain `xsd:string` values and support only exact matching and prefix queries -- not relevance-ranked full-text search.

See [Inline Fulltext Search](../indexing-and-search/fulltext.md) for complete documentation.

## Type Coercion and Compatibility

### Automatic Type Promotion

Fluree handles type compatibility intelligently:

```sparql
# This works - integer can be used where decimal is expected
SELECT ?price
WHERE {
  ?product ex:price ?price .
  FILTER(?price > 10.0)  # decimal comparison
}
```

### Comparisons Between Incompatible Types

When a filter compares values of incompatible types (e.g., a number and a string), the behavior depends on the operator:

- **Equality** (`=`) returns `false` — values of different types are never equal
- **Inequality** (`!=`) returns `true` — values of different types are never equal
- **Ordering** (`<`, `<=`, `>`, `>=`) raises an error — ordering between incompatible types is undefined

Numeric types (long, double, bigint, decimal) are mutually comparable via automatic promotion, so cross-numeric comparisons work as expected. Similarly, temporal types can be compared with string representations that parse to the same temporal type.

### Type Casting in Queries

SPARQL provides functions for type conversion:

```sparql
SELECT ?name (xsd:string(?id) AS ?idString)
WHERE {
  ?person ex:name ?name ;
          ex:id ?id .
}
```

## Best Practices

### Choosing Datatypes

1. **Be Specific**: Use the most appropriate type for your data
   - Use `xsd:integer` for whole numbers that will be used in calculations
   - Use `xsd:string` for identifiers and labels
   - Use `xsd:dateTime` for timestamps

2. **Consider Query Patterns**: Choose types that support your intended queries
   - Numeric types enable range queries and aggregations
   - Date types enable temporal queries
   - String types support text search

3. **Standards Alignment**: Use standard datatypes where possible
   - Prefer XSD types over custom types
   - Use established vocabularies with well-defined ranges

### Type Consistency

1. **Consistent Usage**: Use the same datatype for equivalent properties across your data
2. **Change Planning**: Plan for type changes as your data model evolves
3. **Validation**: Validate data types at ingestion time

### Performance Considerations

1. **Index Efficiency**: Different types have different indexing characteristics
   - Numeric types support efficient range queries
   - String types support prefix and substring matching
   - Date types enable temporal range queries

2. **Storage Size**: Some types are more storage-efficient than others
   - `xsd:integer` is more compact than `xsd:string`
   - `xsd:boolean` is more efficient than string representations

## Type System Architecture

### Internal Representation

Fluree stores all typed values with their datatype information:

- **Value Storage**: The literal value as a string
- **Type Metadata**: The datatype IRI
- **Comparison Logic**: Type-aware comparison functions

### Query Processing

The type system affects query processing:

- **Type Checking**: Ensures type compatibility in filters and joins
- **Index Selection**: Chooses appropriate indexes based on types
- **Result Formatting**: Formats results according to datatype rules

### Standards Compliance

Fluree's type system is fully compliant with:

- **RDF 1.1 Concepts**: Literal typing requirements
- **SPARQL 1.1**: Type promotion and compatibility rules
- **XSD 1.1**: Datatype definitions and constraints
- **JSON-LD 1.1**: Typed value syntax

This strong typing foundation ensures data consistency, enables optimization, and maintains interoperability with the broader semantic web ecosystem.

---

# concepts/datasets-and-named-graphs.md

# Datasets and Named Graphs

Fluree supports **SPARQL datasets**, allowing queries to span multiple graphs simultaneously. This enables complex data integration scenarios where data from different sources or time periods needs to be queried together.

## SPARQL Datasets

A **dataset** in SPARQL is a collection of graphs used for query execution:

- **Default Graph**: The primary graph for triple patterns without GRAPH clauses
- **Named Graphs**: Additional graphs identified by IRIs, accessible via GRAPH clauses

### Dataset Structure

```sparql
# Dataset with one default graph and two named graphs
FROM <ledger:main>           # Default graph
FROM NAMED <ledger:archive>  # Named graph
FROM NAMED <ledger:staging>  # Another named graph
```

## Named Graphs

In SPARQL, **named graphs** are additional graphs (identified by IRIs) that participate in query execution and are accessed via `GRAPH <iri> { ... }`.

In Fluree, named graphs are used in several ways:

- **Multi-graph execution (datasets)**: `FROM NAMED <...>` identifies additional **graph sources** (often other ledgers or non-ledger graph sources) that you can reference with `GRAPH <...> { ... }`.
- **System named graphs**: Fluree provides two built-in named graphs:
  - **`txn-meta`** (`#txn-meta`): commit/transaction metadata, queryable via the `#txn-meta` fragment (e.g., `<mydb:main#txn-meta>`)
  - **`config`** (`#config`): ledger-level configuration (policy, SHACL, reasoning, uniqueness constraints). See [Ledger configuration](../ledger-config/README.md).
- **User-defined named graphs**: Fluree supports ingesting data into user-defined named graphs using TriG format. These graphs are identified by their IRI and can be queried using the structured `from` object syntax with a `graph` field.

### HTTP endpoints and default graph behavior

Fluree exposes two query styles over HTTP:

- **Connection-scoped** (`POST /query`): the ledger(s) and graphs are identified by `from` / `fromNamed` (JSON-LD) or `FROM` / `FROM NAMED` (SPARQL). This is the dataset path and supports multi-ledger datasets.
- **Ledger-scoped** (`POST /query/{ledger}`): the ledger is fixed by the URL. The request may still select a **named graph inside that ledger**:
  - JSON-LD: `"from": "default"`, `"from": "txn-meta"`, or `"from": "<graph IRI>"`
  - SPARQL: `FROM <default>`, `FROM <txn-meta>`, `FROM <graph IRI>`, and `FROM NAMED <graph IRI>`
  - `GRAPH <iri> { ... }` and `GRAPH ?g { ... }` resolve the ledger's registered user named graphs **without** an explicit `FROM NAMED` (the reserved `#txn-meta` / `#config` graphs stay private). Supplying `FROM NAMED` still narrows resolution to exactly the graphs listed.

If the request body tries to target a different ledger than the one in the URL, the server rejects it with a "Ledger mismatch" error.

### Txn metadata named graph (`#txn-meta`)

The `txn-meta` graph contains per-commit metadata stored as triples. This is useful for auditing and operational metadata (machine address, internal user id, job id, etc.).

**Querying txn-meta via SPARQL:**

```sparql
PREFIX f: <https://ns.flur.ee/db#>
PREFIX ex: <http://example.org/ns/>

SELECT ?commit ?t ?machine
FROM <mydb:main#txn-meta>
WHERE {
  ?commit f:t ?t .
  OPTIONAL { ?commit ex:machine ?machine }
}
```

Notes:
- Using `FROM <mydb:main#txn-meta>` makes txn-meta the **default graph** for the query.
- You can also use dataset syntax (`FROM NAMED` + `GRAPH`) if you need to mix default graph and txn-meta in one query.

### User-Defined Named Graphs

Fluree supports ingesting data into user-defined named graphs using **TriG format**. TriG extends Turtle by adding `GRAPH` blocks that assign triples to specific named graphs.

**Creating named graphs via TriG:**

```trig
@prefix ex: <http://example.org/ns/> .
@prefix schema: <http://schema.org/> .

# Default graph triples
ex:company a schema:Organization ;
    schema:name "Acme Corp" .

# Named graph for product data
GRAPH <http://example.org/graphs/products> {
    ex:widget a schema:Product ;
        schema:name "Widget" ;
        schema:price "29.99"^^xsd:decimal .
}

# Named graph for inventory
GRAPH <http://example.org/graphs/inventory> {
    ex:widget schema:inventory 42 ;
        schema:warehouse "main" .
}
```

Submit TriG data via HTTP API:

```bash
curl -X POST "http://localhost:8090/v1/fluree/upsert?ledger=mydb:main" \
  -H "Content-Type: application/trig" \
  --data-binary '@data.trig'
```

**Querying user-defined named graphs (JSON-LD):**

Use the structured `from` object with a `graph` field:

```json
{
  "@context": { "schema": "http://schema.org/" },
  "from": {
    "@id": "mydb:main",
    "graph": "http://example.org/graphs/products"
  },
  "select": ["?name", "?price"],
  "where": [
    { "@id": "?product", "schema:name": "?name" },
    { "@id": "?product", "schema:price": "?price" }
  ]
}
```

**System and user graphs:**
- **Default graph** (implicit): User data without GRAPH blocks
- **`urn:fluree:{ledger_id}#txn-meta`**: Commit metadata
- **`urn:fluree:{ledger_id}#config`**: Ledger configuration (see [Ledger configuration](../ledger-config/README.md))
- **User-defined named graphs**: Identified by their IRI, allocated in order of first use

**Notes:**
- Named graph IRIs are stored in the commit's `graph_delta` field for replay
- Queries against named graphs are scoped to the indexed data (post-indexing)
- Maximum 256 named graphs can be introduced per transaction
- Maximum IRI length is 8KB per graph IRI

### Querying Named Graphs

```sparql
# Query specific named graphs
SELECT ?name
FROM NAMED <http://example.org/ns/graph1>
WHERE {
  GRAPH <http://example.org/ns/graph1> {
    ?person ex:name ?name
  }
}

# Query across multiple graphs
SELECT ?graph ?name
FROM NAMED <http://example.org/ns/graph1>
FROM NAMED <http://example.org/ns/graph2>
WHERE {
  GRAPH ?graph {
    ?person ex:name ?name
  }
}
```

## Default Graph Semantics

The **default graph** contains triples that are not in any named graph:

```sparql
# Query only the default graph
SELECT ?name
FROM <ledger:main>
WHERE {
  ?person ex:name ?name
  # This matches triples in the default graph only
}
```

### Union Default Graph

Some SPARQL implementations create a "union default graph" containing triples from all graphs. Fluree keeps them separate by default, but you can achieve union semantics:

```sparql
# Manual union across graphs
SELECT ?name
FROM NAMED <ledger:main>
FROM NAMED <ledger:archive>
WHERE {
  { GRAPH <ledger:main> { ?person ex:name ?name } }
  UNION
  { GRAPH <ledger:archive> { ?person ex:name ?name } }
}
```

## Multi-Ledger Datasets

Datasets can span multiple ledgers:

```sparql
# Dataset across different ledgers
SELECT ?product ?price
FROM <inventory:main>        # Default graph from inventory ledger
FROM NAMED <pricing:main>    # Named graph from pricing ledger
WHERE {
  ?product ex:name "Widget" .
  GRAPH <pricing:main> {
    ?product ex:price ?price
  }
}
```

This enables **federated queries** across different data sources.

## Time-Aware Datasets

Named graphs can represent different time periods:

```sparql
# Query current and historical data
SELECT ?version ?name
FROM NAMED <ledger:main>      # Current data
FROM NAMED <ledger:archive>   # Historical data
WHERE {
  { GRAPH <ledger:main> {
      ?person ex:name ?name .
      BIND("current" AS ?version)
    }
  }
  UNION
  { GRAPH <ledger:archive> {
      ?person ex:name ?name .
      BIND("archive" AS ?version)
    }
  }
}
```

## Graph Management

### Graph Operations

Fluree supports graph-level operations:

```sparql
# Insert into a specific graph
INSERT DATA {
  GRAPH <http://example.org/ns/metadata> {
    <http://example.org/data/doc1> ex:created "2024-01-15T10:00:00Z"^^xsd:dateTime .
  }
}

# Delete from a specific graph
DELETE {
  GRAPH <http://example.org/ns/temp> {
    ?s ?p ?o
  }
}
WHERE {
  GRAPH <http://example.org/ns/temp> {
    ?s ?p ?o
  }
}
```

### Creation is implicit

There is no "create graph" operation. The first transaction that targets a new graph IRI — via TriG, JSON-LD with `@graph`, or SPARQL `INSERT … GRAPH <iri> …` — registers it and assigns a stable `g_id` (3+ for user graphs). Subsequent inserts into the same IRI land in the same registered slot.

### Listing and dropping graphs

Two CLI commands cover the rest of the lifecycle:

- **[`fluree graph list`](../cli/graph.md#fluree-graph-list)** — lists user graphs registered on a branch (with `--include-system` to also show the default and system graphs). Reads the `named-graphs` section of the standard `/info` response.
- **[`fluree graph drop`](../cli/graph.md#fluree-graph-drop)** — transactionally retracts every triple currently asserted under a named graph. Produces one new commit at `t + 1` whose flakes are all retractions; history at older `t` values is preserved, and the graph IRI keeps its `g_id` so future inserts land in the same slot. Drops are per-branch.

The default graph, `urn:fluree:{ledger_id}#txn-meta`, and `urn:fluree:{ledger_id}#config` cannot be dropped. The Rust API entry point is `Fluree::drop_named_graph(ledger_id, graph_iri)`; over HTTP it is `POST /v1/fluree/drop-graph` (admin-protected). See the [server-integration contract](../cli/server-integration.md#drop-named-graph-contract) for the wire details.

### Graph Metadata

For transaction-scoped metadata, Fluree uses the **`txn-meta`** named graph (see above). Transaction metadata is stored as properties on commit subjects in `txn-meta`, and can be queried independently of user data.

## Use Cases

### Data Partitioning

Separate different types of data:

```sparql
FROM NAMED <urn:customers>
FROM NAMED <urn:products>
FROM NAMED <urn:orders>

SELECT ?customer ?product
WHERE {
  GRAPH <urn:customers> { ?customer foaf:name ?name }
  GRAPH <urn:orders> {
    ?order ex:customer ?customer ;
           ex:product ?product .
  }
}
```

### Access Control

Different graphs can have different permissions:

- Public graph: Open access
- Private graph: Restricted access
- Admin graph: Administrative data

### Data Provenance

Track data sources and quality:

```sparql
FROM NAMED <urn:sensor1>
FROM NAMED <urn:sensor2>

SELECT ?sensor ?reading ?quality
WHERE {
  GRAPH ?sensor {
    ?obs ex:reading ?reading ;
         ex:quality ?quality .
  }
  FILTER(?quality > 0.8)  # Only high-quality readings
}
```

### Version Management

Maintain different versions of data:

```sparql
FROM NAMED <urn:v1.0>
FROM NAMED <urn:v2.0>

SELECT ?feature ?version
WHERE {
  GRAPH ?version {
    ?feature ex:status "active"
  }
}
```

## Performance Considerations

### Index Optimization

Named graphs affect indexing strategy:

- **Graph-aware indexes**: Indexes can be partitioned by graph
- **Cross-graph joins**: May require special optimization
- **Graph statistics**: Maintain statistics per graph for query planning

### Query Planning

The query planner considers:

- **Graph selectivity**: Which graphs contain relevant data
- **Join patterns**: How graphs are connected in the query
- **Graph size**: Larger graphs may need different strategies

### Best Practices

1. **Logical Partitioning**: Use graphs for logical data separation
2. **Size Considerations**: Very large graphs may impact query performance
3. **Naming Conventions**: Use consistent IRI patterns for graph names
4. **Documentation**: Document the purpose and schema of each graph

## Standards Compliance

Fluree's dataset implementation follows:

- **SPARQL 1.1 Query**: FROM and FROM NAMED clauses
- **SPARQL 1.1 Update**: GRAPH clauses in updates
- **RDF 1.1 Datasets**: Named graph semantics
- **JSON-LD 1.1**: @graph syntax for named graphs

This enables seamless integration with other RDF tools and SPARQL endpoints while providing Fluree's unique temporal and ledger capabilities.

---

# concepts/edge-annotations.md

# Edge Annotations

Edge annotations let you attach properties to a *relationship* — the connection between two subjects — without modeling an intermediate node by hand. A property graph user calls these "edge properties" or "relationship properties." A SPARQL user calls them "annotations on a quoted triple." A Fluree user gets one ergonomic surface that reads correctly from either side.

```text
ex:alice ──[ ex:worksFor: { role: "Engineer", since: 2024-01-01, confidence: 0.97 } ]──▶ ex:acme
```

Annotations are first-class RDF data: properties on the annotation subject are stored as ordinary triples and participate in policy, history, indexing, and query like everything else. The only thing that's special is the *attachment*: a sidecar relation that records which annotation subject belongs to which edge. The fact indexes (and queries that don't ask for annotations) are unchanged.

For the storage-internals view — how the attachment sidecar is laid out, how the `f:reifies*` bundle encodes an edge, and how the index root carries the annotation arena — see the [Edge annotations design doc](../design/edge-annotations.md).

## When to use edge annotations

Reach for `@annotation` when you need any of these:

- **Property-graph-shaped edges.** A `worksFor` relationship needs `role` and `since`. Modeling that as a separate `Employment` node works but distorts the graph.
- **Provenance / quality on a fact.** "This `ex:hasAuthor` claim has confidence 0.97 from source X." Classic RDF reification with quoted triples.
- **Multiple parallel relationships** between the same two subjects — e.g. Alice was both an Engineer and later a Manager at Acme. Plain RDF can't distinguish two `ex:worksFor` triples; annotations can.
- **Property-graph imports.** Relationship properties round-trip without forcing every edge through an intermediate `:Relationship` node.

If a fact is naturally about a *node* (Alice's birthdate, Acme's industry), put it on the node — not on the edge. Annotations are for facts about the *relationship*.

## Where can I write edge annotations?

| Surface | How | Notes |
|---|---|---|
| **JSON-LD insert / upsert / update** | `@annotation` (or alias `@edge`) on a value object | Most ergonomic. Covers literal-valued edges (with explicit `@type` / `@language`), parallel annotations, named reifiers, body cascades, and **named-graph edges** (an annotation on an edge inside a named graph is written into that same graph, keeping the edge's graph identity). `@reifies` is a query-side construct, **not** an insert form (user-authored `@reifies` on a write is rejected). |
| **SPARQL 1.2 UPDATE** | `INSERT DATA { :s :p :o {\| ... \|} }`, `~ <reifier>`, optional `INSERT { } WHERE { }` templates | Use this when integrating with SPARQL pipelines or when porting from RDF 1.2 / SPARQL-star. **Default graph only:** an annotation tail inside an explicit `GRAPH { }` block, or under a `WITH <g>` template, is rejected — use the JSON-LD surface for named-graph edge annotations. See [SPARQL 1.2 surface](#sparql-12--rdf-12-surface) below for the per-operation rules. |
| **Turtle / TriG / N-Triples / N-Quads file ingest** | Not natively (today) | These ingest paths are RDF 1.1 + Fluree extensions; they do **not** parse RDF 1.2 annotation tails (`{\| ... \|}`), the `~` reifier, or `<<( ... )>>` triple terms — a file containing them **fails to parse** with a lexer error (e.g. `unexpected character '~'`). Two routes work: (a) convert your file to JSON-LD before ingesting, or (b) ingest the plain edges first and then add annotations with a follow-up SPARQL `INSERT DATA { :s :p :o {\| ... \|} }` transaction. Either route ends up with the same on-disk shape. |

Annotations are backed by a set of [reserved system predicates](../reference/vocabulary.md#edge-annotation-predicates-reserved) that application writes can't author by hand — mint annotations through `@annotation` / `@edge` (JSON-LD) or the RDF 1.2 annotation tail (`~`, `{| |}`) in SPARQL. (Bulk import is an administrative bootstrap path that may ingest already-lowered annotation bundles without up-front validation; see the [storage-internals design doc](../design/edge-annotations.md) for how malformed bundles are handled.)

## The surface

### Inserting an annotated edge

The annotation block lives under the value object — `@id` is the edge target, `@annotation` carries the relationship's properties.

```json
{
  "@context": {
    "ex": "http://example.org/",
    "xsd": "http://www.w3.org/2001/XMLSchema#"
  },
  "insert": {
    "@id": "ex:alice",
    "ex:worksFor": {
      "@id": "ex:acme",
      "@annotation": {
        "ex:role": "Engineer",
        "ex:since": { "@value": "2024-01-01", "@type": "xsd:date" },
        "ex:confidence": 0.97
      }
    }
  }
}
```

Internally this commits four things atomically:

1. The base edge `ex:alice ex:worksFor ex:acme`.
2. A fresh annotation subject (a blank node by default).
3. An attachment row recording that the annotation belongs to that edge.
4. The annotation properties (`ex:role`, `ex:since`, `ex:confidence`) as ordinary triples on the annotation subject.

`@edge` is accepted as an alias for `@annotation`; the two are interchangeable.

### Naming the annotation explicitly

You can give the annotation an IRI when you need stable identity — for updates, external references, signatures, or "the contract for Alice's 2024 employment."

```json
{
  "@id": "ex:alice",
  "ex:worksFor": {
    "@id": "ex:acme",
    "@annotation": {
      "@id": "ex:employment/alice-acme-2024",
      "ex:role": "Engineer",
      "ex:since": { "@value": "2024-01-01", "@type": "xsd:date" }
    }
  }
}
```

Two inserts that target the same explicit `@id` reattach to the same annotation subject — idempotent. Two inserts with no explicit `@id` mint two distinct annotations on the same edge (see *Parallel annotations* below).

### Annotating literal-valued edges

RDF 1.2 permits annotations on triples whose object is a literal — `:alice :name "Alice" {| :source :hr |}` in Turtle-star, equivalently:

```json
{
  "@id": "ex:alice",
  "ex:name": {
    "@value": "Alice",
    "@annotation": { "ex:source": "ex:hr-system" }
  }
}
```

Because JSON scalars can't carry sibling metadata, an annotated literal **must** be written as a JSON-LD value object — the expanded form with `@value`. The same applies to typed and language-tagged literals:

```json
{
  "@id": "ex:alice",
  "ex:joinedAt": {
    "@value": "2024-01-01",
    "@type": "xsd:date",
    "@annotation": { "ex:source": "ex:hr-system" }
  },
  "ex:label": {
    "@value": "chat",
    "@language": "fr",
    "@annotation": { "ex:source": "ex:lexicon" }
  }
}
```

A few rules that keep the annotation's identity in sync with the base flake:

- **The value object must carry its `@type` / `@language` explicitly when the predicate's `@context` would otherwise coerce them.** When `@annotation` is present, the lowering pass rejects two coercion paths that the JSON-LD value-object expander applies: a term-level `@type` on the predicate's context entry, and a default `@language` on the active context. (Per-term `@language` overrides are intentionally ignored, mirroring the value-object expander's own behavior — it reads `context.language` directly, not the per-term entry.) The non-annotated form continues to use context coercion normally; this stricter rule applies only to annotated literals so the annotation's stored edge identity cannot silently diverge from the base flake's.
- **Language-tagged literals are language-pinned.** Two annotations on `"chat"@fr` and `"chat"@en` are independent; selector-form retracts and hydration both match on language.
- **Hydration promotes annotated literals to value-object form.** A subject expansion (`select: {"?s": ["*"]}`) renders unannotated `ex:name "Alice"` as the scalar `"Alice"`, but renders the annotated form as `{"@value": "Alice", "@annotation": {...}}` so the annotation has somewhere to attach.

The deferred shapes from "Current limits" below (list occurrences, multi-triple reifiers, triple terms as object values) still apply on the literal path.

### Querying inline: edge first, metadata second

The query shape mirrors the insert shape. Match the base edge, then constrain or project annotation metadata.

```json
{
  "@context": { "ex": "http://example.org/" },
  "select": ["?person", "?org", "?role", "?since"],
  "where": {
    "@id": "?person",
    "ex:worksFor": {
      "@id": "?org",
      "@annotation": {
        "ex:role": "?role",
        "ex:since": "?since"
      }
    }
  }
}
```

This binds one row per `(edge, annotation)` pair currently asserted.

### Querying annotation-rooted: metadata first, edge second

When you start from the metadata — "find every employment with `role = Engineer`" — use `@reifies` to walk back to the edge.

```json
{
  "@context": { "ex": "http://example.org/" },
  "select": ["?person", "?org", "?since"],
  "where": {
    "ex:role": "Engineer",
    "ex:since": "?since",
    "@reifies": {
      "@id": "?person",
      "ex:worksFor": { "@id": "?org" }
    }
  }
}
```

`@reifies` is the same idea as `rdf:reifies` in RDF 1.2 — given an annotation subject, walk to the edge it reifies. Fluree resolves it through the reverse attachment index, so it's cheap regardless of how many annotations exist in the ledger.

### Subject expansion

Graph-crawl projection preserves the annotation block in the output:

```json
{
  "select": {
    "?person": [
      "@id",
      {
        "ex:worksFor": [
          "@id",
          { "@annotation": ["ex:role", "ex:since", "ex:confidence"] }
        ]
      }
    ]
  },
  "where": { "@id": "?person", "ex:worksFor": { "@id": "?org" } }
}
```

Output:

```json
{
  "@id": "ex:alice",
  "ex:worksFor": {
    "@id": "ex:acme",
    "@annotation": {
      "ex:role": "Engineer",
      "ex:since": "2024-01-01",
      "ex:confidence": 0.97
    }
  }
}
```

## Cardinality: the multiplicity contract

This is the rule to internalize:

> **A bare triple pattern returns one row per `(s, p, o)`. Binding an annotation variable returns one row per `(edge, annotation)`.**

Concretely:

- `?s ex:worksFor ?o` returns the same rows whether the edge has zero, one, or twenty annotations attached. RDF set semantics are preserved; existing queries don't change behavior just because a ledger started using annotations.
- `?s ex:worksFor ?o, @annotation { ?ann }` (or any `@annotation` body that binds a variable / matches a property) returns one row per annotation occurrence on each matching edge.

This is what lets a property-graph traversal faithfully return parallel-edge rows while leaving plain RDF queries undisturbed.

`select: "*"` follows the same rule — it does not multiply by occurrence count unless the WHERE binds an annotation variable.

## Parallel annotations on one edge

Two annotation blocks on the same `(s, p, o)` mint two distinct annotation subjects (anonymous case) or attach to the same subject (explicit-`@id` case).

```json
{
  "@graph": [
    {
      "@id": "ex:alice",
      "ex:worksFor": {
        "@id": "ex:acme",
        "@annotation": {
          "@id": "ex:emp/2020",
          "ex:role": "Engineer"
        }
      }
    },
    {
      "@id": "ex:alice",
      "ex:worksFor": {
        "@id": "ex:acme",
        "@annotation": {
          "@id": "ex:emp/2024",
          "ex:role": "Manager"
        }
      }
    }
  ]
}
```

Querying with an `@annotation` binding returns two rows:

```text
?person     ?org     ?role
ex:alice    ex:acme  Engineer
ex:alice    ex:acme  Manager
```

Querying without binding the annotation (`?person ex:worksFor ?org`) returns one row.

## Anonymous vs explicit annotation IDs

The two forms have deliberately different lifecycle behavior. The default is conservative: anonymous annotations behave like LPG edge properties; explicit-IRI annotations behave like ordinary RDF resources.

| | Anonymous (no `@id`) | Explicit `@id` |
|---|---|---|
| Visible in `select: "*"` | No — hidden from wildcard subject expansion | Yes |
| Visible in graph crawl | Only via `@annotation` projection | Yes, like any subject |
| Retract base edge → owned facts cascade | Yes (the annotation is intrinsic to the edge) | No, by default — explicit IRIs are not deleted surprisingly |

The anonymous-hide rule means a user wildcard query against Alice doesn't suddenly start returning a sea of internal annotation SIDs once you adopt edge metadata. Annotations participate in queries that ask for them and stay out of the way otherwise.

The explicit-ID-doesn't-cascade rule protects user-named resources from accidental deletion when an edge gets retracted. Opt out via *LPG mode* (below) when you actually want property-graph "delete the relationship deletes its properties" semantics.

## Retraction semantics

### RDF mode (default)

Retracting a base edge removes the attachment and any owned facts on **anonymous** annotations. Explicit-IRI annotations keep their non-attachment facts — only the attachment row is retracted.

```json
{
  "delete": {
    "@id": "ex:alice",
    "ex:worksFor": { "@id": "ex:acme" }
  }
}
```

After this:
- Anonymous `_:annN` subjects attached to the edge: gone (attachment + body).
- Explicit `ex:employment/alice-acme-2024`: attachment retracted, but `ex:role`, `ex:since`, etc. are still in the graph as ordinary RDF.

History preserves both events — query at the pre-retract `t` and the annotation comes back, unchanged.

### LPG mode (opt-in per transaction)

For property-graph relationship lifecycle — "deleting the relationship deletes the relationship's properties" — set `lpgEdgeLifecycle: true` in transaction options:

```json
{
  "delete": {
    "@id": "ex:alice",
    "ex:worksFor": { "@id": "ex:acme" }
  },
  "opts": { "lpgEdgeLifecycle": true }
}
```

Now explicit-IRI annotations cascade their owned metadata too.

### Updating annotation properties

Updating metadata is normal RDF update against the annotation subject. Once you've bound the occurrence by `@id` or by selector, treat it like any other subject:

```json
{
  "where": {
    "@id": "ex:alice",
    "ex:worksFor": {
      "@id": "ex:acme",
      "@annotation": { "@id": "?edge", "ex:role": "Engineer" }
    }
  },
  "delete": { "@id": "?edge", "ex:confidence": "?old" },
  "insert": { "@id": "?edge", "ex:confidence": 0.99 }
}
```

## Empty annotation blocks

In RDF mode, `"@annotation": {}` is a no-op: no annotation subject is minted, no attachment row is written. Inserts stay idempotent at the `(s, p, o)` level.

In LPG mode, an empty block mints a fresh annotation subject — a property-less relationship still has identity, the way property-graph relationships do.

## SPARQL 1.2 / RDF 1.2 surface

`@annotation` lowers to the same on-disk model as the RDF 1.2 annotation tail. The equivalent **write** forms below produce identical storage; pick whichever is ergonomic for your input. (`@reifies` / `rdf:reifies` are **query-side** constructs — see [Querying annotation-rooted](#querying-annotation-rooted-metadata-first-edge-second) — and are rejected on the write side.)

### Equivalent forms

JSON-LD `@annotation`:

```json
{
  "@id": "ex:alice",
  "ex:worksFor": {
    "@id": "ex:acme",
    "@annotation": { "ex:role": "Engineer" }
  }
}
```

SPARQL 1.2 / RDF 1.2 annotation block (anonymous reifier):

```sparql
PREFIX ex: <http://example.org/>
INSERT DATA {
  ex:alice ex:worksFor ex:acme {| ex:role "Engineer" |} .
}
```

SPARQL 1.2 / RDF 1.2 named reifier (`~`):

```sparql
PREFIX ex: <http://example.org/>
INSERT DATA {
  ex:alice ex:worksFor ex:acme ~ ex:emp1 {| ex:role "Engineer" |} .
}
```

### Grammar reference (subset)

The annotation tail attaches to the triple — not the object — per the RDF 1.2 grammar mirrored by SPARQL 1.2:

```text
annotation       ::= ( reifier | annotationBlock )*
reifier          ::= '~' ( iri | BlankNode | Var )?
annotationBlock  ::= '{|' predicateObjectList '|}'
tripleTerm       ::= '<<(' ttSubject verb ttObject ')>>'
```

Notes:

- An `annotationBlock` without a preceding `~` mints a fresh anonymous reifier.
- A bare `~` (no identifier) is equivalent to `~` + a fresh blank node — useful when you want a reifier variable bound in WHERE but don't care about its IRI.
- `tripleTerm` (the parenthesized `<<( s p o )>>` form) is accepted **only** as the object of `rdf:reifies`. Other uses error at parse time.
- Property-path triples cannot carry an annotation tail. `?s ex:p1/ex:p2 ?o {| ... |}` is rejected — write a simple-predicate triple instead.

### SPARQL UPDATE rules by operation

Different UPDATE operations place different constraints on reifier identity per SPARQL Update §3.1 and §4.1. The contract below is what Fluree enforces.

| Operation | `~ <iri>` (named) | `~ ?var` (variable) | `~ _:label` (blank) | `{\| \|}` with no `~` (anonymous) | `?ann rdf:reifies <<( ... )>>` |
|---|---|---|---|---|---|
| `INSERT DATA` | ✅ resolved via nameservice | ❌ vars not allowed in DATA | ✅ minted as fresh Sid for the operation | ✅ fresh blank reifier minted | ❌ DATA accepts only the `~ {\| \|}` form (semantically equivalent) |
| `DELETE DATA` | ✅ addresses an existing reifier by stable IRI | ❌ vars not allowed in DATA | ❌ rejected per SPARQL §3.1.3 — blank nodes have no addressable identity in `DELETE DATA` | ❌ rejected (same reason) — use `DELETE WHERE` with a binding instead | ❌ same as `INSERT DATA` |
| `INSERT { } WHERE { }` template | ✅ resolved | ✅ var bound by WHERE; resolves per solution | ✅ per-solution fresh blank (same label across the template = same per-solution blank) | ✅ per-solution fresh blank | ❌ INSERT templates accept only `~ {\| \|}` |
| `DELETE { } WHERE { }` template | ✅ resolved | ✅ required — the only addressable identity in a DELETE template | ❌ blank nodes forbidden in DELETE templates per SPARQL §3.1.3 | ❌ rejected — anonymous reifier has no addressable identity. Use a named `~ ?ann` bound by WHERE. | ❌ same |
| `DELETE { } INSERT { } WHERE { }` | DELETE clause follows DELETE rules; INSERT clause follows INSERT rules; WHERE follows query rules | Variable bound by WHERE; usable in both clauses | DELETE: rejected. INSERT: per-solution blank | DELETE: rejected. INSERT: per-solution blank | ❌ |

The reserved-predicate firewall fires across all UPDATE entry points: `INSERT DATA { _:a f:reifiesSubject ex:b }` is rejected at parse time with an error pointing at `@annotation` / the `~ {| |}` syntax. Mint annotations only through the supported surface forms.

### Querying annotations from SPARQL

Three query shapes, each backed by a different sidecar lookup:

**Inline anonymous** — match base edge, match metadata, no reifier identity needed:

```sparql
PREFIX ex: <http://example.org/>
SELECT ?role WHERE {
  ex:alice ex:worksFor ex:acme {| ex:role ?role |} .
}
```

**Inline with bound reifier** — one row per parallel annotation on the edge:

```sparql
PREFIX ex: <http://example.org/>
SELECT ?ann ?role WHERE {
  ?p ex:worksFor ex:acme ~ ?ann {| ex:role ?role |} .
}
```

**Annotation-rooted via `rdf:reifies`** — filter by metadata, return reified-edge endpoints:

```sparql
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX ex:  <http://example.org/>
SELECT ?person ?org WHERE {
  ?ann rdf:reifies <<( ?person ex:worksFor ?org )>> .
  ?ann ex:role "Engineer" .
}
```

Sibling triples about the reifier (here `?ann ex:role "Engineer"`) live in the surrounding scope and join via the standard executor — they do **not** need to live inside the `<<( ... )>>` term.

#### Blank nodes in `WHERE` clauses

Per SPARQL §4.1.4, a blank-node label in a `WHERE` clause is a **non-distinguished variable** — bindable inside the BGP but not exposable via `SELECT`. The same rule applies to reifiers: `?p ex:worksFor ex:acme ~ _:ann { ... }` lets `_:ann` join across the BGP but does not surface in the result.

Anonymous annotation blocks (`{| |}` without `~`) lower to a fresh non-distinguished variable internally (the formatter hides it from `SELECT *` so query output stays clean).

#### One annotation, one edge (single-target invariant)

An annotation subject reifies **exactly one live edge**. A single edge can carry many parallel annotations, but a given annotation `@id` cannot point at two different edges at once. Asserting an attachment that would leave one annotation reifying two edges is rejected at stage time with an `InvariantViolation`. To move an explicit-IRI annotation from one edge to another, retract the old attachment and assert the new one in the same transaction — the retract + assert nets to a single live edge and is accepted. (This keeps the reverse lookup, retract cascade, and by-id cleanup unambiguous. Bulk import does not enforce this up front — see the import caveat above.)

#### Lifecycle coupling

Fluree's annotation is *lifecycle-coupled* to an asserted edge: the annotation describes a triple that's currently in the graph. RDF 1.2 also allows reifiers for unasserted propositions ("X claims Alice works for Acme without us asserting it"). That mode is **not supported in v1** — see *Current limits* below.

### Deferred SPARQL shapes (rejected at parse time)

These produce a clear error with a span pointing at the offending construct:

- **Triple terms outside `rdf:reifies`.** `ex:doc ex:mentions <<( :s :p :o )>>` is rejected. Use a separate annotation subject with `rdf:reifies` if you need to refer to a triple.
- **Nested triple terms.** `<<( :s :p <<( :a :b :c )>> )>>` is rejected.
- **Multi-triple reifiers.** One reifier identifier reifying more than one triple term in the same scope is rejected. A reifier corresponds to one edge occurrence.
- **Annotation on a property-path triple.** `?s ex:p1/ex:p2 ?o {| ... |}` is rejected — the grammar only attaches annotations to simple-predicate triples.
- **Annotation tail inside an explicit `GRAPH { }` block.** `INSERT DATA { GRAPH <g> { :s :p :o {| ... |} } }` is rejected. SPARQL UPDATE annotations are **default-graph only** in v1 — the SPARQL surface doesn't carry the enclosing graph's identity into the stored annotation. Use the JSON-LD `@annotation` surface to annotate an edge inside a named graph.
- **Annotation tail under a `WITH <g>` template.** `WITH <g> INSERT { :s :p :o {| ... |} } WHERE { ... }` is rejected for the same reason: the annotation would land in `<g>` without recording that graph as the edge's identity, yielding a default-graph edge identity in a named graph. Again, use the JSON-LD surface for named-graph edge annotations.
- **SPARQL `CONSTRUCT` template projecting annotation metadata.** Until the Turtle-star vs RDF 1.2 reifier output decision lands, a CONSTRUCT template containing an annotation tail or `rdf:reifies` returns `UnsupportedFeature`. CONSTRUCT *without* annotation in the template still works even when the WHERE pattern uses annotations to filter.

Annotations on literal-valued objects (plain, typed, and language-tagged) are supported on **both** the JSON-LD and SPARQL UPDATE write surfaces — the SPARQL path records the language tag for language-tagged objects so the stored annotation matches the base edge.

### Legacy Fluree-specific `<< s p ?o >>` syntax

Fluree predates RDF 1.2. The bare `<< s p o >>` SPARQL-star quoted-triple form (without parens) remains valid for the **Fluree-specific** `f:t` / `f:op` flake-metadata extraction:

```sparql
PREFIX f:  <https://ns.flur.ee/db#>
PREFIX ex: <http://example.org/>
SELECT ?age ?t ?op WHERE {
  << ex:alice ex:age ?age >> f:t ?t ; f:op ?op .
}
```

This binds `?t` to the transaction time and `?op` to the assert/retract flag of the matched flake. It is **not** edge annotations and is unrelated to the RDF 1.2 reifier surface above. Use `<<( ... )>>` (parenthesized) and `{| ... |}` for edge annotations; use bare `<< ... >>` only for `f:t` / `f:op`.

The bare-quoted-triple form combined with an annotation tail (`<< :s :p :o >> :pred :obj {| ... |}`) is rejected at parse time — the two surfaces don't compose.

## Current limits

Today's surface covers the common LPG / RDF-star use cases. The following are not yet supported and produce a clear validation error rather than silent partial behavior:

- **Annotations on list-occurrence triples.** `@list` membership is in scope as a future extension; the on-disk format already reserves space for it. Today, annotating a list element is rejected at parse time.
- **Reifiers for unasserted triples.** `@reifies` must point at an asserted edge. Pure-proposition reification (claims about triples that are not in the graph) is deferred.
- **Reifiers for multiple triples.** One annotation subject corresponds to one edge. Reifying several unrelated triples from a single annotation isn't allowed.
- **Triple terms as object values.** `ex:doc ex:mentions << ex:s ex:p ex:o >>` is not yet a representable value. Use a separate annotation subject.
- **Non-JSON-LD output.** v1 emits annotations in JSON-LD output. Turtle, TriG, N-Quads, and SPARQL CONSTRUCT need a separate surface-form decision (Turtle-star vs RDF 1.2 reifier vs other) and currently return `UnsupportedFeature` when a CONSTRUCT against those targets projects annotation metadata.
- **SPARQL 1.2 triple-term functions and constructor.** `TRIPLE`, `SUBJECT`, `PREDICATE`, `OBJECT`, `isTRIPLE`, and the `BIND(<<( ?s ?p ?o )>> AS ?t)` triple-term constructor are deferred — they presuppose triple terms as first-class values, which v1's LPG model does not represent. The compact reifier delimiter `<< s p o ~ r >>` is also deferred (it collides with the legacy `f:t`/`f:op` extraction form above); use the supported `s p o ~ r {| ... |}` annotation tail instead.

The mandated SPARQL 1.2 `VERSION "1.2"` prologue declaration is **accepted** (lex-and-skipped): the RDF 1.2 surface runs ungated, so a conformant 1.2 client that emits the declaration parses normally.

## Storage and indexing — the short version

- A ledger with no annotations creates no annotation artifacts. The annotation arena is `Option<...>` on the index root and is omitted entirely when unused.
- Plain triple queries take exactly the same plan they did before annotations existed; the planner only routes through the attachment indexes when the query mentions `@annotation`, `@reifies`, or RDF-star quoted triples.
- Annotation properties are stored as ordinary RDF facts. Time travel, policy, history, export, and reasoning all work on them without special cases.
- Retraction cascade has a fast path: when both the index root and current novelty know the ledger has no annotations, base-edge retracts skip the attachment lookup entirely.
- Branch fork, pack/sync, and ledger drop all walk annotation arena artifacts as part of the index reachability set, so annotated ledgers round-trip cleanly across these operations.

For the index format, sidecar layout, sort orders, and garbage-collection treatment, see the [Edge annotations design doc](../design/edge-annotations.md).

## Property-graph (LPG) model

Edge annotations are the storage primitive for the labeled-property-graph shape: a relationship that carries its own properties, and parallel relationships between the same two nodes. A property-graph relationship with properties maps directly onto an annotated edge —

- the relationship's properties become the `@annotation` body,
- the relationship's identity is the annotation subject (anonymous, or pinned with an explicit `@id`),
- two relationships of the same type between the same endpoints become two parallel annotations (see *Parallel annotations* above),
- *LPG mode* (`lpgEdgeLifecycle: true`) gives the "delete the relationship deletes its properties" lifecycle property graphs expect.

A dedicated property-graph query/write language front-end is a separate surface and is not part of this release; the JSON-LD `@annotation` surface and the SPARQL 1.2 annotation tail are the supported ways to write and read this shape today.

## See also

- [Edge annotations design](../design/edge-annotations.md) — storage internals (EdgeKey, sidecar arena, sticky-bit state machine, GC reachability).
- [Datasets and named graphs](datasets-and-named-graphs.md) — annotations work in named graphs (via the JSON-LD `@annotation` surface) as well as the default graph; the SPARQL UPDATE surface is default-graph only.
- [Time travel](time-travel.md) — annotation events live in history like every other fact.
- [Policy enforcement](policy-enforcement.md) — annotation properties pass through normal policy checks.


---

# concepts/time-travel.md

# Time Travel

**Differentiator**: Fluree is a **temporal database** that preserves the complete history of all changes. Every transaction is timestamped, enabling queries against any previous state of the data. This "time travel" capability is fundamental to Fluree's architecture and provides capabilities that most databases cannot match.

## Query Formats

Time travel is supported in both JSON-LD and SPARQL query formats. Examples in this document primarily use JSON-LD syntax with SPARQL equivalents shown where relevant.

## Transaction Time

Every transaction in Fluree receives a unique **transaction time** (`t`) - a monotonically increasing integer that represents the logical time of the transaction.

### Transaction Ordering

```text
Transaction 1: t=1
Transaction 2: t=2
Transaction 3: t=3
...
```

- **Monotonic**: Each new transaction gets a higher `t` than all previous transactions
- **Unique**: No two transactions share the same `t`
- **Global**: Transaction times are unique across the entire Fluree instance

### Current Time

The **current time** is the highest transaction time that has been committed. Queries without a time specifier automatically query the current state:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "select": ["?name"],
  "where": [
    { "@id": "?person", "ex:name": "?name" }
  ]
}
```

You can also explicitly specify `@t:latest` to query the latest state:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "ledger:main@t:latest",
  "select": ["?name"],
  "where": [
    { "@id": "?person", "ex:name": "?name" }
  ]
}
```

## Historical Queries

Fluree supports querying data as it existed at any point in time using the `@` syntax in ledger references.

### Point-in-Time Queries

Query data as it existed at a specific transaction using the `from` field with `@t:`:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "ledger:main@t:100",
  "select": ["?name"],
  "where": [
    { "@id": "?person", "ex:name": "?name" }
  ]
}
```

### Query at ISO Timestamp

Query using ISO 8601 datetime with `@iso:`:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "ledger:main@iso:2024-01-15T10:30:00Z",
  "select": ["?name"],
  "where": [
    { "@id": "?person", "ex:name": "?name" }
  ]
}
```

### Query at Commit ContentId

Query at a specific commit using `@commit:` with a commit ContentId:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "ledger:main@commit:bafybeig...",
  "select": ["?name"],
  "where": [
    { "@id": "?person", "ex:name": "?name" }
  ]
}
```

## Temporal Data Model

### Immutable Facts

Once committed, data is **immutable**. Changes are represented as new facts that supersede previous ones:

```text
t=1: Alice age 25  (assertion)
t=5: Alice age 26  (retraction of age 25, assertion of age 26)
```

History queries capture both the retraction and assertion with `@op`:

```json
[
  [25, 1, true],
  [25, 5, false],
  [26, 5, true]
]
```

Each row shows `[value, transaction_time, op]` where `op` is `true` for assertions and `false` for retractions.

### Valid Time vs Transaction Time

Fluree primarily uses **transaction time** (when the fact was recorded in the database). For applications needing **valid time** (when the fact was true in the real world), this can be modeled explicitly as properties:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "@graph": [
    {
      "@id": "ex:alice-employment-1",
      "ex:person": "ex:alice",
      "ex:company": "ex:company-a",
      "ex:validFrom": "2020-01-01T00:00:00Z",
      "ex:validTo": "2023-12-31T23:59:59Z"
    }
  ]
}
```

This allows you to query by both:
- **Transaction time**: When was this recorded? (using `@t:`, `@iso:`, `@commit:`)
- **Valid time**: When was this true? (using standard WHERE clause filters on `ex:validFrom`/`ex:validTo`)

## Snapshot and Indexing

### Database Snapshots

Fluree maintains **indexed snapshots** at regular intervals for efficient historical access:

- **Index**: A complete, optimized snapshot of the database at a specific `t`
- **Novelty**: Uncommitted transactions since the last index
- **Background Indexing**: Continuous process that creates new indexes

### Query Execution Model

Queries combine indexed data with novelty:

```text
Query Result = Indexed Database (up to t=index) + Novelty (t=index+1 to current)
```

This provides:
- **Fast historical queries**: Use appropriate index
- **Real-time current queries**: Include latest transactions
- **Consistent snapshots**: Each query sees a consistent state

## Consistency and Read-After-Write

Fluree's query engine is **eventually consistent**. When a transaction commits at `t=N`, queries running against a different process or a warm cache may still see a state older than `t=N` until the cache is refreshed.

### The Problem

```text
Process A: transact → receives t=42
Process B: query    → sees t=40 (stale cache)
```

This is expected in architectures where the query server is a separate peer, or in serverless environments where a warm Lambda invocation holds a cached ledger state from a previous request.

### The Solution: `refresh()` with `min_t`

The `refresh()` API accepts a `min_t` parameter that asserts the cached ledger has reached at least a specific transaction time. If the ledger hasn't reached that `t` after pulling the latest state from the nameservice, the call returns an error so the caller can retry.

**Flow:**

```text
1. Client transacts → receives t=42
2. Client calls refresh(ledger, min_t=42)
3. Fluree checks cached t:
   - If cached t >= 42 → immediate success (no I/O)
   - If cached t < 42  → pull latest from nameservice, apply commits
   - If still t < 42   → return AwaitTNotReached error
4. Client queries at t >= 42 with confidence
```

### Usage Patterns

**Same-process (embedded Fluree):**

In a single process where you transact and query through the same `Fluree` instance, the cache is updated in-place by the transaction. `min_t` is typically not needed, but can serve as a safety assertion.

**Multi-process / Serverless:**

When the transacting process and querying process are separate (e.g., a Lambda that writes and another that reads), pass the `t` from the transaction receipt through your event/message payload and use `min_t` to gate the query:

```text
Writer Lambda:
  receipt = transact(data)
  publish_event({ t: receipt.t, ... })

Reader Lambda:
  event = receive_event()
  refresh(ledger, min_t=event.t, timeout=5s)
  query(ledger)  // guaranteed to see at least t=event.t
```

**HTTP API:**

HTTP query clients can request the same read-after-write guarantee by sending the transaction time in the `fluree-min-t` header. The server refreshes the target ledger(s) until each reaches at least that `t`, or until the min-t wait timeout expires:

```http
fluree-min-t: 42
```

JSON-LD query bodies may also use `opts.min-t` (or `opts.minT`) for the same request-level guarantee. Numeric query snapshot pins like `from: "ledger:main@t:42"` or `from: {"@id": "ledger:main", "t": 42}` also force the server to observe at least that `t` before executing, then read that pinned snapshot. ISO and commit pins still resolve through normal snapshot resolution.

The wait is capped by `query_min_t_timeout_ms` (default 5 seconds), even if the general query execution timeout is disabled.

Long-running HTTP query servers can also enable TTL-gated query-time refresh with `--query-refresh-enabled` and `--query-refresh-ttl-ms`. This makes the server call `refresh()` on demand before current-head queries, after that ledger's TTL window expires, so warm caches observe nameservice advances without checking DynamoDB on every request. The first query in a new TTL window pays the nameservice round-trip; idle ledgers are not refreshed. The TTL bounds ordinary staleness, but callers that must prove they observed a specific transaction should use `fluree-min-t` or `opts.min-t`.

### Rust API

See [Using Fluree as a Rust Library — Read-After-Write Consistency](../getting-started/rust-api.md#read-after-write-consistency) for full code examples including retry-with-backoff patterns.

```rust
use fluree_db_api::RefreshOpts;

// After a transaction returns t=42:
let opts = RefreshOpts { min_t: Some(42) };
let result = fluree.refresh("mydb:main", opts).await?;
// result.t >= 42 is guaranteed if Ok
```

## History Queries for Change Tracking

History queries let you see all changes (assertions and retractions) within a time range. Specify the range using `from` and `to` keys with time-specced endpoints.

### Entity History (JSON-LD)

Track all changes to a specific entity over time by specifying a time range:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "ledger:main@t:1",
  "to": "ledger:main@t:latest",
  "select": ["?name", "?t", "?op"],
  "where": [
    { "@id": "ex:alice", "ex:name": { "@value": "?name", "@t": "?t", "@op": "?op" } }
  ],
  "orderBy": "?t"
}
```

The `@t` and `@op` annotations bind the transaction time and operation type:
- **@t** - Transaction time (integer) when the fact was asserted or retracted.
- **@op** - Boolean: `true` for assertions, `false` for retractions. Mirrors `Flake.op` on disk. Both literal- and IRI-valued objects carry the metadata.

Returns results showing all changes:

```json
[
  ["Alice", 1, true],
  ["Alice", 5, false],
  ["Alicia", 5, true]
]
```

### Entity History (SPARQL)

The same query in SPARQL uses RDF-star syntax with `FROM...TO`:

```sparql
PREFIX ex: <http://example.org/ns/>
PREFIX f: <https://ns.flur.ee/db#>

SELECT ?name ?t ?op
FROM <ledger:main@t:1>
TO <ledger:main@t:latest>
WHERE {
  << ex:alice ex:name ?name >> f:t ?t .
  << ex:alice ex:name ?name >> f:op ?op .
}
ORDER BY ?t
```

### Property-Specific History

Query changes for specific properties:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "ledger:main@t:1",
  "to": "ledger:main@t:100",
  "select": ["?age", "?t", "?op"],
  "where": [
    { "@id": "ex:alice", "ex:age": { "@value": "?age", "@t": "?t", "@op": "?op" } }
  ],
  "orderBy": "?t"
}
```

### All Properties History

Query all property changes for an entity:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "ledger:main@t:1",
  "to": "ledger:main@t:latest",
  "select": ["?p", "?v", "?t", "?op"],
  "where": [
    { "@id": "ex:alice", "?p": { "@value": "?v", "@t": "?t", "@op": "?op" } }
  ],
  "orderBy": "?t"
}
```

### Time Range with Datetime

Query history using ISO 8601 datetime strings:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "ledger:main@iso:2024-01-01T00:00:00Z",
  "to": "ledger:main@iso:2024-12-31T23:59:59Z",
  "select": ["?name", "?t", "?op"],
  "where": [
    { "@id": "ex:alice", "ex:name": { "@value": "?name", "@t": "?t", "@op": "?op" } }
  ]
}
```

### Filter by Operation Type

Filter to show only assertions or only retractions:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "ledger:main@t:1",
  "to": "ledger:main@t:latest",
  "select": ["?name", "?t"],
  "where": [
    { "@id": "ex:alice", "ex:name": { "@value": "?name", "@t": "?t", "@op": "?op" } },
    ["filter", "(= ?op \"retract\")"]
  ]
}
```

### Pattern History Across Subjects

Query changes for a specific property across all subjects:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "ledger:main@t:1",
  "to": "ledger:main@t:latest",
  "select": ["?person", "?status", "?t", "?op"],
  "where": [
    { "@id": "?person", "ex:status": { "@value": "?status", "@t": "?t", "@op": "?op" } }
  ],
  "orderBy": "?t"
}
```

## Performance Characteristics

### Time Resolution Performance

Different time specifiers have different performance characteristics:

- **@t:NNN** (fastest): Direct transaction number, no resolution needed
- **@iso:DATETIME**: O(log n) binary search through commit timestamps using POST index
- **@commit:CID**: Bounded SPOT scan, O(k) where k is commits matching prefix (use longer prefixes for better performance)

### Index Selection

Fluree automatically selects the most appropriate index for historical queries:

- **Recent history**: Uses current index + novelty (uncommitted transactions)
- **Historical snapshots**: Uses closest index snapshot to target time
- **Point queries** (`@t:`): Direct index lookup for specific transaction

### History Query Performance

History queries scan flakes within the specified time range:

- **Entity history** (specific `@id`): SPOT index scan on subject
- **Property history** (specific predicate): Narrower SPOT scan with predicate filter
- **All properties** (variable predicate `?p`): Full SPOT scan for subject
- **Cross-entity** (variable subject `?s`): POST/PSOT index scan (can be slower for common predicates)

### Optimization Strategies

1. **Use Transaction Numbers**: When possible, use `@t:NNN` instead of `@iso:DATETIME`
2. **Narrow History Patterns**: Use `[subject, predicate]` instead of `[subject]` when you only need specific properties
3. **Limit Time Ranges**: Specify realistic `from`/`to` bounds rather than querying all history
4. **ContentId Prefix Length**: Use sufficiently long ContentId prefixes to avoid ambiguity checks
5. **Index Density**: More frequent indexing improves historical query performance for distant past

### Storage Implications

- **Full History**: All transaction history is preserved (immutable append-only)
- **Index Snapshots**: Periodic snapshots enable efficient historical queries without replaying all transactions
- **Commit Metadata**: Stored as queryable flakes (~8-9 flakes per commit)
- **Transaction JSON**: Optionally stored for audit trails (enable with `txn: true`)

## Practical Applications

### Version Control

Treat data like code with version control:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "app:production@t:1000",
  "select": ["?config"],
  "where": [
    { "@id": "?setting", "ex:value": "?config" }
  ]
}
```

### Regulatory Compliance

Maintain complete audit trails - query data as it existed at time of consent:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "users:main@iso:2024-05-25T14:30:00Z",
  "select": ["?predicate", "?data"],
  "where": [
    { "@id": "ex:alice", "?predicate": "?data" }
  ]
}
```

### Change History Analysis

Track how data evolved over time:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "sales:main@iso:2024-01-01T00:00:00Z",
  "to": "sales:main@iso:2024-12-31T23:59:59Z",
  "select": ["?order", "?amount", "?t", "?op"],
  "where": [
    { "@id": "?order", "ex:amount": { "@value": "?amount", "@t": "?t", "@op": "?op" } }
  ],
  "orderBy": "?t"
}
```

### Debugging and Troubleshooting

Investigate system state at time of incident:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "system:config@iso:2024-01-15T09:15:00Z",
  "select": ["?setting", "?config"],
  "where": [
    { "@id": "?setting", "ex:value": "?config" }
  ]
}
```

## Time Travel in Multi-Ledger Scenarios

### Cross-Ledger Temporal Queries

Query across ledgers at consistent time points:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": [
    "customers:main@t:1000",
    "orders:main@t:1000"
  ],
  "select": ["?customer", "?order"],
  "where": [
    { "@id": "?customer", "ex:name": "Alice" },
    { "@id": "?order", "ex:customer": "?customer" }
  ]
}
```

### Ledger Branching

Time travel enables sophisticated branching workflows by querying historical states:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "ledger:main@t:500",
  "select": ["?entity", "?property", "?value"],
  "where": [
    { "@id": "?entity", "?property": "?value" }
  ]
}
```

You can then use this historical state as a basis for creating a new branch or comparing against current state.

## Common Patterns

### Compare Current vs Historical State

Query the same entity at two different points in time:

```json
// Query current state
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "ledger:main",
  "select": ["?price"],
  "where": [
    { "@id": "ex:product-123", "ex:price": "?price" }
  ]
}

// Query historical state
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "ledger:main@t:100",
  "select": ["?price"],
  "where": [
    { "@id": "ex:product-123", "ex:price": "?price" }
  ]
}
```

### Find When a Change Occurred

Use history queries to identify when a specific change happened:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "ledger:main@t:1",
  "to": "ledger:main@t:latest",
  "select": ["?status", "?t", "?op"],
  "where": [
    { "@id": "ex:product-123", "ex:status": { "@value": "?status", "@t": "?t", "@op": "?op" } }
  ],
  "orderBy": "?t"
}
```

The results show when `ex:status` changed, with `?op = false` (retract) for the old value and `?op = true` (assert) for the new value at the same transaction time.

### Audit Trail for Compliance

Generate a complete audit trail for a sensitive entity:

```json
{
  "@context": { "schema": "http://schema.org/" },
  "from": "users:main@iso:2024-01-01T00:00:00Z",
  "to": "users:main@t:latest",
  "select": ["?property", "?value", "?t", "?op"],
  "where": [
    { "@id": "schema:Person/12345", "?property": { "@value": "?value", "@t": "?t", "@op": "?op" } }
  ],
  "orderBy": "?t"
}
```

This returns all changes with transaction times for audit purposes. Each result row shows the property, value, when it was changed, and whether it was an assertion or retraction.

### Rollback Detection

Find what changed after a specific commit:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "config:main@t:50",
  "to": "config:main@t:latest",
  "select": ["?setting", "?value", "?t", "?op"],
  "where": [
    { "@id": "?setting", "ex:config": { "@value": "?value", "@t": "?t", "@op": "?op" } }
  ],
  "orderBy": "?t"
}
```

This shows all configuration changes since transaction 50, useful for identifying what to rollback. You can first query `"from": "config:main@commit:bafybeig..."` to find the transaction number (using point-in-time queries), then use that in the history query.

### Reproduce a Bug at Specific Time

Query the exact state of the system when a bug was reported:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": [
    "products:main@iso:2024-06-15T14:30:00Z",
    "inventory:main@iso:2024-06-15T14:30:00Z"
  ],
  "select": ["?product", "?stock", "?reserved"],
  "where": [
    { "@id": "?product", "ex:stockLevel": "?stock" },
    { "@id": "?product", "ex:reserved": "?reserved" }
  ]
}
```

This recreates the exact state across multiple ledgers at the time the bug occurred, making debugging much easier.

## Best Practices

### Time Travel Guidelines

1. **Explicit Time References**: Always specify clear time references (`@t:`, `@iso:`, or `@commit:`) for reproducible queries
2. **Time Zone Awareness**: Use UTC for ISO timestamps to avoid ambiguity
3. **ContentId Length**: Use sufficiently long ContentId prefixes to avoid collisions
4. **Performance Testing**: Test query performance across different time ranges and ledger sizes

### History Query Patterns

1. **Narrow Your Scope**: Use specific property patterns rather than wildcard `?p` when you only need certain properties
2. **Limit Time Ranges**: Specify realistic time ranges with `from` and `to` rather than `@t:1` to `@t:latest`
3. **Use Filters**: Filter by `@op` to show only assertions or retractions when you don't need both
4. **Order Results**: Use `orderBy: "?t"` to see changes in chronological order

### Data Modeling for Time

1. **Temporal Validity**: Model valid time explicitly when needed (separate from transaction time)
2. **Change Tracking**: Use history queries rather than storing change logs manually
3. **Immutable Design**: Design for immutability from the start - never update in place
4. **Audit Patterns**: Leverage history queries for audit trails instead of separate audit tables

### Operational Considerations

1. **Index Maintenance**: Monitor and tune background indexing for optimal historical query performance
2. **Storage Planning**: Plan storage growth for historical data (all history is preserved)
3. **Query Optimization**: Use time-specific queries (`@t:`) rather than datetime resolution (`@iso:`) when transaction numbers are known
4. **Backup Strategy**: Include temporal aspects in backup/recovery plans - commits and indexes are both critical

## Implementation Architecture

### Transaction Pipeline

1. **Transaction Reception**: Assign new transaction time (`t`)
2. **Validation**: Check against current state
3. **Commitment**: Persist transaction with ISO timestamp
4. **Commit Metadata**: Store commit ContentId, timestamp, and optional transaction JSON
5. **Indexing**: Background process creates new indexes
6. **Publication**: Update nameservice with new transaction time

### Time Travel Resolution

When you query with `@t:`, `@iso:`, or `@commit:`:

1. **@t:NNN** - Direct transaction number (fastest)
2. **@iso:DATETIME** - Binary search through commit timestamps using POST index
3. **@commit:CID** - Bounded SPOT scan to find matching commit

### Query Execution

1. **Time Resolution**: Resolve time specifiers to specific `t` values
2. **Index Selection**: Choose appropriate index for target time
3. **Novelty Application**: Apply intervening transactions if needed
4. **Result Generation**: Return consistent snapshot

### History Query Execution

1. **Time Range Detection**: The `from` and `to` keys with time-specced endpoints activates history mode
2. **Pattern Resolution**: WHERE patterns are executed with history mode enabled
3. **Metadata Capture**: Transaction time (`@t`) and operation (`@op`) are captured for each binding
4. **Result Generation**: Results include both assertions and retractions within the time range

This temporal foundation makes Fluree uniquely powerful for applications requiring complete historical visibility, audit capabilities, and temporal analytics.

---

# concepts/policy-enforcement.md

# Policy Enforcement

Fluree enforces access control inside the database. Individual facts (flakes) are filtered against policy rules during query and transaction execution, so the same query returns different results to different identities — automatically. The application doesn't filter; the database does.

## Why triple-level

Most databases enforce access at the row, table, or schema level. That granularity is awkward for graph data, where a single subject may have facts that are public (`schema:name`), employee-only (`ex:department`), and HR-only (`ex:salary`). Fluree's enforcement happens **per flake** — `?subject ?predicate ?object` — so policies can permit `name`, allow `department` to platform employees, and restrict `salary` to managers in the same department, all from one query.

The consequences:

- **No application-side filtering.** Security can't be bypassed by buggy code paths because the database never returns flakes the requester isn't allowed to see.
- **Auditable.** Policies are themselves data. They live in the ledger, are time-travelable, and can be queried — `SELECT ?p WHERE { ?p a f:AccessPolicy }`.
- **Multi-tenant ready.** A single ledger can serve many tenants, with isolation enforced at flake level.
- **Compliance-friendly.** GDPR / HIPAA-style "minimum necessary" access is the default behavior, not a check the app forgot to do.

## What a policy looks like

Every policy is a JSON-LD node typed `f:AccessPolicy`. A policy has three orthogonal pieces:

- **Targeting** — `f:onProperty`, `f:onClass`, `f:onSubject` (each an array of `@id` references). Omit them all to make a *default policy* that applies to every flake.
- **Action** — `f:action` with values `f:view` (queries) and/or `f:modify` (transactions).
- **Decision** — either:
  - `f:allow: true` — unconditional allow, or
  - `f:allow: false` — unconditional deny, or
  - `f:query: "<JSON-encoded WHERE>"` — allow when the embedded query produces at least one binding for the targeted flake.

Two further knobs:

- `f:required: true` — the policy *must* allow for access to the targeted flake to be granted, even when `default-allow` is true. Use it for hard constraints (PII protection, write barriers).
- `f:exMessage` — a string returned to the caller when this policy denies a transaction.

A worked example:

```json
{
  "@id": "ex:salary-restriction",
  "@type": ["f:AccessPolicy", "ex:CorpPolicy"],
  "f:required": true,
  "f:onProperty": [{"@id": "ex:salary"}],
  "f:action": [{"@id": "f:view"}],
  "f:query": "{\"where\": {\"@id\": \"?$identity\", \"http://example.org/role\": \"manager\"}}"
}
```

Translation: *for every flake whose property is `ex:salary` and that someone is trying to read, this policy must allow. The embedded `f:query` runs with `?$identity` pre-bound to the requester; if it returns a binding (i.e. the identity has role `"manager"`), the flake is permitted.*

## Variables in `f:query`

Inside an `f:query`, two variables are pre-bound:

| Variable | Meaning |
|----------|---------|
| `?$this` | The subject of the targeted flake (the entity being read or written). |
| `?$identity` | The IRI of the requesting identity, supplied via `policy-values`. |

Anything else is bound by the embedded WHERE just like a normal Fluree query.

## How the engine combines policies

When a request hits a flake, the engine collects every policy that targets it:

1. **Required policies** (with `f:required: true`) must all allow. If any required policy denies — including by returning no `f:query` bindings — the flake is denied.
2. If no required policies target the flake, **any** allow is enough. Fluree uses *allow-overrides* across the non-required set.
3. If no policies apply at all, the request falls back to `default-allow`.

`default-allow: false` is fail-closed and the right choice for most production deployments.

## Where policies come from

Two delivery channels, often mixed:

- **Stored** — write policies into the ledger as data. Tag each policy with a class (e.g. `ex:CorpPolicy`), and tag each identity entity with `f:policyClass` linking to that class. At request time, pass `policy-class: ["ex:CorpPolicy"]` and the engine pulls the matching policy set from the ledger automatically. Stored policies are versioned, time-travelable, and consistent across all callers — the right approach for production.
- **Inline** — pass policies in `opts.policy` (an array of policy nodes) or via the `fluree-policy` HTTP header. Useful for ad-hoc queries, automated tests, and admin scripts.

The two can be combined: a query can carry a `policy-class` *and* an additional inline `policy`.

## Identity binding

An identity entity ties a caller (DID, JWT subject, application user) to graph nodes that policies can reason about:

```json
{
  "@id": "ex:aliceIdentity",
  "ex:user": {"@id": "ex:alice"},
  "f:policyClass": [{"@id": "ex:CorpPolicy"}]
}
```

Caller traffic carrying `identity: "ex:aliceIdentity"` causes:

1. Fluree binds `?$identity` to `ex:aliceIdentity` in every `f:query`.
2. Stored policies tagged `ex:CorpPolicy` are loaded.
3. Each policy's `f:query` runs against the snapshot, with `?$identity` and `?$this` pre-bound, deciding flake by flake whether the request is permitted.

The `ex:user` link is a domain-specific convention — your `f:query`s use it to reach from the identity to the human/service the policies should reason about. Any modeling works; nothing about that link is special to Fluree.

## What you control at the request boundary

Each request can supply:

- **`identity`** — IRI of the calling identity entity. Used to pre-bind `?$identity` and to discover the identity's `f:policyClass`.
- **`policy-class`** — one or more class IRIs to pull stored policies by class.
- **`policy-values`** — an object of additional `?$var` bindings injected into every policy's `f:query`.
- **`policy`** — an inline JSON-LD policy array.
- **`default-allow`** — boolean fallback for flakes no policy targets.

Over JSON-LD, these go inside `opts`. Over SPARQL, they're sent as `fluree-*` headers (SPARQL has no `opts` block). When the server is configured with a default policy class, a verified bearer token's identity is auto-applied — see the [policy cookbook](../guides/cookbook-policies.md#invoking-policies-via-http) for the request shapes and the server-side `data_auth_default_policy_class` option in [Configuration](../operations/configuration.md).

## Query enforcement vs transaction enforcement

The same policy model governs both, distinguished by `f:action`:

- **`f:view`** — runs during query execution. Flakes that fail the policy are filtered from the result; the query never sees them.
- **`f:modify`** — runs during transaction staging. The transaction is rejected (with `f:exMessage` if provided) if a write would touch flakes the identity isn't allowed to modify.

A single policy can govern both (`"f:action": [{"@id": "f:view"}, {"@id": "f:modify"}]`). Most realistic policy sets mix view-only restrictions, modify-only restrictions, and a small number of `[f:view, f:modify]` defaults.

## Policies are data

Because policies are flakes:

- **Time travel.** Query at past `t` to see what was in effect.
- **Branchable.** Trial policies on a branch before merging.
- **Versionable.** Edit through normal transactions; full history kept.
- **Self-querying.** Run reports over the policies themselves.

This makes policy management a normal Fluree workflow rather than a sidecar problem.

## Performance shape

Policy evaluation has two phases — load (read the policies relevant to this request once) and apply (filter flakes during plan execution). Cost scales mostly with the apply phase: how many flakes the request touches, and how expensive each policy's `f:query` is.

Two practical implications:

- **Target policies.** A policy with `f:onProperty` or `f:onClass` only runs on flakes whose predicate or rdf:type matches. Default policies (no targeting) run on every flake. Prefer targeting wherever it makes sense. Targeting also lets the engine keep its index-metadata fast paths: a bare `COUNT` over a predicate the policy provably can't restrict is still answered without a per-flake scan (see [Policy in queries](../security/policy-in-queries.md#performance-considerations)).
- **Keep `f:query` cheap.** Lean on identity attributes already loaded (`@type`, `f:policyClass`, role flags) rather than deep traversals.

For deeper architectural detail see [Policy model and inputs](../security/policy-model.md), [Policy in queries](../security/policy-in-queries.md), and [Policy in transactions](../security/policy-in-transactions.md).

## Related documentation

- [Cookbook: Access control policies](../guides/cookbook-policies.md) — worked examples for common patterns
- [Policy model and inputs](../security/policy-model.md) — full reference
- [Policy in queries](../security/policy-in-queries.md) — query-time behavior
- [Policy in transactions](../security/policy-in-transactions.md) — transaction-time behavior
- [Programmatic policy API (Rust)](../security/programmatic-policy.md) — building policy contexts in code
- [Cross-ledger policy](../security/cross-ledger-policy.md) — govern many data ledgers from one model ledger
- [Authentication](../security/authentication.md) — identity, JWTs, and bearer tokens
- [Configuration](../operations/configuration.md) — server-side policy defaults (`data_auth_default_policy_class`, etc.)


---

# concepts/verifiable-data.md

# Verifiable Data

**Differentiator**: Fluree supports cryptographically signed transactions using industry-standard formats (JWS and Verifiable Credentials), enabling tamper-proof audit trails and trustless data exchange. Every transaction can be cryptographically verified, providing cryptographic proof of data provenance and integrity.

**Note:** Requires the `credential` feature flag. See [Compatibility and Feature Flags](../reference/compatibility.md#fluree-db-api-features).

## What Is Verifiable Data?

**Verifiable data** in Fluree refers to transactions that are cryptographically signed, providing proof of:
- **Authenticity**: Who created the transaction
- **Integrity**: That the data hasn't been tampered with
- **Non-repudiation**: The signer cannot deny creating the transaction
- **Provenance**: The origin and history of the data

### Key Characteristics

- **Cryptographic Signatures**: Transactions signed using standard cryptographic algorithms
- **Industry Standards**: Support for JWS (JSON Web Signatures) and Verifiable Credentials (VC)
- **Tamper-Proof**: Any modification to signed data invalidates the signature
- **Verifiable**: Anyone can verify signatures without special access

## Why Verifiable Data Matters

### Traditional Database Limitations

Most databases provide:
- **Authentication**: Who can access the database
- **Authorization**: What they can do
- **Audit Logs**: What happened (but logs can be modified)

**Problems:**
- No cryptographic proof of data origin
- Audit logs can be tampered with
- Difficult to prove data integrity
- No way to verify data across systems

### Fluree's Approach

Fluree provides:
- **Cryptographic Signatures**: Every transaction can be signed
- **Tamper-Proof History**: Signed transactions cannot be modified
- **Verifiable Provenance**: Anyone can verify data origin
- **Trustless Exchange**: Data can be shared without trusting intermediaries

**Benefits:**
- **Audit Compliance**: Cryptographic proof for compliance requirements
- **Data Integrity**: Detect any tampering with data
- **Trustless Systems**: Enable trustless data exchange
- **Provenance Tracking**: Track data origin cryptographically

## Signed Transactions

### JWS (JSON Web Signatures)

Fluree supports JWS for signing transactions:

**Transaction Structure:**

```json
{
  "ledger": "mydb:main",
  "tx": [
    {
      "@id": "ex:alice",
      "ex:name": "Alice"
    }
  ],
  "signature": {
    "protected": {
      "alg": "ES256",
      "kid": "key-1"
    },
    "signature": "base64-encoded-signature"
  }
}
```

**Verification:**
- Extract signature from transaction
- Verify signature using signer's public key
- Confirm transaction hasn't been modified

### Verifiable Credentials

Fluree supports Verifiable Credentials (VC) for credential-based transactions:

**VC Structure:**

```json
{
  "@context": [
    "https://www.w3.org/2018/credentials/v1"
  ],
  "type": ["VerifiableCredential"],
  "credentialSubject": {
    "@id": "ex:alice",
    "ex:name": "Alice"
  },
  "proof": {
    "type": "Ed25519Signature2020",
    "created": "2024-01-15T10:00:00Z",
    "verificationMethod": "did:example:alice#key-1",
    "proofValue": "base64-encoded-signature"
  }
}
```

**Verification:**
- Verify credential proof
- Check credential issuer
- Validate credential structure
- Confirm credential hasn't been revoked

## Transaction Signing

### Signing a Transaction

**Step 1: Prepare Transaction**

```json
{
  "ledger": "mydb:main",
  "tx": [
    {
      "@id": "ex:alice",
      "ex:name": "Alice"
    }
  ]
}
```

**Step 2: Create Signature**

```javascript
// Pseudo-code
const payload = JSON.stringify(tx);
const signature = sign(payload, privateKey);
```

**Step 3: Add Signature**

```json
{
  "ledger": "mydb:main",
  "tx": [...],
  "signature": {
    "protected": {
      "alg": "ES256",
      "kid": "key-1"
    },
    "signature": signature
  }
}
```

### Signature Algorithms

Fluree supports standard signature algorithms:

- **ES256**: ECDSA with P-256 and SHA-256
- **ES384**: ECDSA with P-384 and SHA-384
- **ES512**: ECDSA with P-521 and SHA-512
- **Ed25519**: EdDSA with Ed25519 curve

### Key Management

**Public Key Storage:**

Public keys can be stored:
- In the ledger itself (as data)
- In a separate key registry
- In a DID (Decentralized Identifier) document

**Example Public Key in Ledger:**

```json
{
  "@id": "ex:alice",
  "ex:publicKey": {
    "kty": "EC",
    "crv": "P-256",
    "x": "base64-x",
    "y": "base64-y"
  }
}
```

## Transaction Verification

### Verifying a Signed Transaction

**Step 1: Extract Signature**

```json
{
  "signature": {
    "protected": {...},
    "signature": "base64-signature"
  }
}
```

**Step 2: Get Public Key**

```javascript
// Pseudo-code
const kid = signature.protected.kid;
const publicKey = getPublicKey(kid);
```

**Step 3: Verify Signature**

```javascript
// Pseudo-code
const payload = JSON.stringify(tx);
const isValid = verify(payload, signature.signature, publicKey);
```

### Verification in Fluree

Fluree automatically verifies signed transactions:

1. **Signature Extraction**: Extract signature from transaction
2. **Key Resolution**: Resolve public key from signature
3. **Signature Verification**: Verify cryptographic signature
4. **Transaction Acceptance**: Accept transaction if signature valid

**If verification fails:**
- Transaction is rejected
- Error returned to client
- No data is committed

## Use Cases

### Audit Compliance

**Requirement**: Cryptographic proof of all data changes

**Solution**: Sign all transactions

```json
{
  "ledger": "audit:main",
  "tx": [
    {
      "@id": "ex:change1",
      "ex:action": "update",
      "ex:timestamp": "2024-01-15T10:00:00Z"
    }
  ],
  "signature": {...}
}
```

**Benefits:**
- Cryptographic proof of changes
- Tamper-proof audit trail
- Compliance with regulations

### Trustless Data Exchange

**Requirement**: Share data without trusting intermediaries

**Solution**: Sign data at source

```json
{
  "ledger": "shared:main",
  "tx": [
    {
      "@id": "ex:data1",
      "ex:value": "sensitive-data",
      "ex:source": "ex:system-a"
    }
  ],
  "signature": {
    "protected": {
      "kid": "ex:system-a#key-1"
    },
    "signature": "..."
  }
}
```

**Benefits:**
- Verify data origin
- Detect tampering
- Trustless data sharing

### Multi-Party Systems

**Requirement**: Multiple parties contribute data

**Solution**: Each party signs their transactions

```json
{
  "ledger": "consortium:main",
  "tx": [
    {
      "@id": "ex:contribution1",
      "ex:party": "ex:party-a",
      "ex:data": "..."
    }
  ],
  "signature": {
    "protected": {
      "kid": "ex:party-a#key-1"
    },
    "signature": "..."
  }
}
```

**Benefits:**
- Identify data contributors
- Verify party contributions
- Enable accountability

### Regulatory Compliance

**Requirement**: Prove data integrity for regulations

**Solution**: Sign all regulated data

**Examples:**
- **HIPAA**: Healthcare data integrity
- **GDPR**: Personal data provenance
- **SOX**: Financial data integrity
- **FDA**: Pharmaceutical data integrity

## Verifiable Credentials

### Credential Structure

Verifiable Credentials follow W3C VC standard:

```json
{
  "@context": [
    "https://www.w3.org/2018/credentials/v1"
  ],
  "id": "ex:credential-1",
  "type": ["VerifiableCredential", "ex:IdentityCredential"],
  "issuer": "did:example:issuer",
  "issuanceDate": "2024-01-15T10:00:00Z",
  "credentialSubject": {
    "@id": "ex:alice",
    "ex:name": "Alice",
    "ex:email": "alice@example.com"
  },
  "proof": {
    "type": "Ed25519Signature2020",
    "created": "2024-01-15T10:00:00Z",
    "verificationMethod": "did:example:issuer#key-1",
    "proofValue": "base64-signature"
  }
}
```

### Credential Verification

**Step 1: Verify Proof**

```javascript
// Pseudo-code
const proof = credential.proof;
const publicKey = resolvePublicKey(proof.verificationMethod);
const isValid = verifyProof(credential, proof, publicKey);
```

**Step 2: Check Issuer**

```javascript
// Pseudo-code
const issuer = credential.issuer;
const isTrusted = checkIssuerTrust(issuer);
```

**Step 3: Validate Credential**

```javascript
// Pseudo-code
const isValid = validateCredentialStructure(credential);
```

### Credential Revocation

Credentials can be revoked:

```json
{
  "@id": "ex:revocation-1",
  "@type": "ex:CredentialRevocation",
  "ex:credentialId": "ex:credential-1",
  "ex:revokedAt": "2024-01-20T10:00:00Z"
}
```

Verification should check revocation status.

## Data Provenance

### Tracking Data Origin

Signed transactions enable provenance tracking:

**Query Transaction History:**

```sparql
SELECT ?tx ?signer ?timestamp
WHERE {
  ?tx ex:signature ?sig .
  ?sig ex:signer ?signer .
  ?tx ex:timestamp ?timestamp .
}
ORDER BY DESC(?timestamp)
```

**Verify Data Chain:**

```sparql
SELECT ?data ?origin ?signer
WHERE {
  ?data ex:origin ?origin .
  ?origin ex:signature ?sig .
  ?sig ex:signer ?signer .
}
```

### Provenance Verification

**Step 1: Find Data Origin**

```sparql
SELECT ?tx
WHERE {
  ?tx ex:created ?data .
}
```

**Step 2: Verify Transaction Signature**

```javascript
// Pseudo-code
const tx = getTransaction(txId);
const isValid = verifySignature(tx);
```

**Step 3: Trace Provenance Chain**

```sparql
SELECT ?chain
WHERE {
  ?data ex:provenance ?chain .
  ?chain ex:signature ?sig .
}
```

## Best Practices

### Key Management

1. **Secure Storage**: Store private keys securely
2. **Key Rotation**: Rotate keys regularly
3. **Key Backup**: Backup keys securely
4. **Key Recovery**: Plan for key recovery

### Signature Practices

1. **Always Sign**: Sign all important transactions
2. **Verify Before Trust**: Always verify signatures
3. **Standard Algorithms**: Use standard signature algorithms
4. **Key Identification**: Use clear key identifiers

### Credential Management

1. **Issuer Trust**: Establish issuer trust relationships
2. **Credential Validation**: Validate credential structure
3. **Revocation Checking**: Check revocation status
4. **Credential Storage**: Store credentials securely

### Compliance

1. **Audit Logging**: Log all signature verifications
2. **Provenance Tracking**: Track data provenance
3. **Regulatory Alignment**: Align with regulations
4. **Documentation**: Document verification processes

## Comparison with Traditional Approaches

### Traditional Audit Logs

**Traditional Approach:**
- Logs stored in database
- Can be modified by admins
- No cryptographic proof
- Difficult to verify

**Problems:**
- Logs can be tampered with
- No proof of authenticity
- Difficult to verify
- Not suitable for trustless systems

### Fluree Verifiable Data

**Fluree Approach:**
- Transactions cryptographically signed
- Signatures cannot be forged
- Anyone can verify
- Suitable for trustless systems

**Benefits:**
- Tamper-proof history
- Cryptographic proof
- Easy verification
- Trustless data exchange

## Architecture

### Signature Storage

Signatures are stored with transactions:

- **Transaction Metadata**: Signature stored in transaction metadata
- **Queryable**: Signatures can be queried like any data
- **Versioned**: Signature history tracked over time

### Verification Engine

The verification engine:

- **Automatic Verification**: Verifies signatures automatically
- **Key Resolution**: Resolves public keys from signatures
- **Standard Compliance**: Follows JWS and VC standards

### API Integration

Verification integrated with:

- **Transaction API**: Verifies signatures on transaction submission
- **Query API**: Can query signature information
- **Admin API**: Administrative operations on signatures

Verifiable data makes Fluree uniquely suited for applications requiring cryptographic proof of data integrity, audit compliance, and trustless data exchange. By supporting industry-standard signature formats, Fluree enables integration with existing identity systems and credential ecosystems.


---

# concepts/reasoning.md

# Reasoning and Inference

Fluree includes a built-in reasoning engine that can derive new facts from your
data based on ontology declarations (RDFS and OWL) or user-defined rules
(Datalog). This page introduces the core concepts; see
[Query-time reasoning](../query/reasoning.md) for usage syntax,
[Datalog rules](../query/datalog-rules.md) for custom rules, and the
[OWL & RDFS reference](../reference/owl-rdfs-support.md) for a full list of
supported constructs.

## Why reasoning?

In a plain triple store every fact must be stated explicitly. If you assert that
Alice is a `Student` and that `Student` is a subclass of `Person`, a query for
all `Person` instances will *not* return Alice — unless you also assert
`Alice rdf:type Person`.

With reasoning enabled, Fluree can **infer** the missing fact automatically:

```
Alice  rdf:type  Student       (asserted)
Student  rdfs:subClassOf  Person   (schema)
────────────────────────────────────────────
Alice  rdf:type  Person        (inferred)
```

This keeps your data clean (no redundant assertions) while giving your queries
the full power of schema-aware retrieval.

## Reasoning modes

Fluree supports four reasoning profiles that can be enabled independently or in
combination. They are listed here from lightest to most powerful:

| Mode | What it does | Cost |
|------|-------------|------|
| **RDFS** | Expands `rdfs:subClassOf` and `rdfs:subPropertyOf` hierarchies so that querying for a superclass or superproperty also returns instances of its subclasses/subproperties. | Very low — query rewriting only, no materialization. |
| **OWL 2 QL** | Everything RDFS does, plus `owl:inverseOf` expansion and `rdfs:domain`/`rdfs:range` type inference via query rewriting. Based on the OWL 2 QL profile designed for query answering. | Low — query rewriting only. |
| **OWL 2 RL** | Forward-chaining materialization of a comprehensive rule set (symmetric, transitive, and inverse properties; functional properties; property chains; class restrictions; `owl:sameAs` equivalence; and more). See the [OWL & RDFS reference](../reference/owl-rdfs-support.md) for the full list. | Medium — derives facts before query execution; results are cached. |
| **Datalog** | User-defined if/then rules expressed in a familiar JSON-LD pattern syntax. Rules run in a fixpoint loop and can chain off each other or off OWL-derived facts. See [Datalog rules](../query/datalog-rules.md). | Depends on the rules — can be lightweight or heavy. |

### Combining modes

Modes can be combined freely. For example, `["rdfs", "owl2rl", "datalog"]`
first materializes OWL 2 RL entailments, then runs your Datalog rules over the
combined base + OWL-derived data, and finally applies RDFS query rewriting on
top. This layering lets you start simple (RDFS) and add more powerful inference
only where you need it.

## How it works

Fluree uses two complementary techniques depending on the mode:

### Query rewriting (RDFS, OWL 2 QL)

The query planner rewrites your patterns at compile time. For example, a
`?x rdf:type ex:Person` pattern is expanded into a UNION over `Person` and all
of its subclasses. No extra data is stored; the rewriting is transparent to the
caller.

### Forward-chaining materialization (OWL 2 RL, Datalog)

Before your query runs, the engine:

1. **Loads the ontology** — extracts OWL/RDFS declarations (property types,
   class hierarchies, restrictions) from your data.
2. **Compiles the ontology into ground rules** — each axiom becomes a concrete
   rule indexed by the facts that can fire it (its trigger predicate or
   class), so the fixpoint only ever evaluates rules a new fact can actually
   trigger.
3. **Applies rules in a fixpoint loop** — each iteration dispatches the newly
   derived facts to their triggered rules. The loop stops when no new facts
   are produced (fixpoint) or a budget limit is reached.
4. **Overlays derived facts** — the inferred triples are layered on top of your
   base data as a read-only overlay. Your original data is never modified.
5. **Caches the result** — if the same database state is queried again with the
   same reasoning modes, the cached materialization is reused instantly.

### Budget controls

To guarantee termination, materialization enforces configurable limits:

| Limit | Default | What happens when exceeded |
|-------|---------|--------------------------|
| Time | 30 seconds | Materialization stops; partial results used |
| Derived facts | 1,000,000 | Materialization stops; partial results used |
| Memory | 100 MB | Materialization stops; partial results used |

When a budget is exceeded the query still runs — but over an **incomplete**
closure, so results may be missing entailments. A capped materialization is
surfaced in tracked query responses (the `reasoning` block /
`x-fdb-reasoning` header) and logged as a server WARN. The fact and time
limits are configurable per ledger (`f:reasoningMaxFacts` /
`f:reasoningMaxSeconds`), per query (`"reasoningBudget"` or SPARQL budget
pragmas), and server-wide via environment variables — see
[Reasoning — materialization budget](../query/reasoning.md#materialization-budget).

## Enabling reasoning

There are two levels of control:

### 1. Ledger-wide defaults (configuration graph)

Set reasoning defaults so every query against a ledger uses a particular mode
without having to specify it each time:

```json
{
  "@context": {"f": "https://ns.flur.ee/db#"},
  "insert": {
    "@id": "urn:fluree:mydb:main:config:ledger",
    "@type": "f:LedgerConfig",
    "f:reasoningDefaults": {
      "f:reasoningModes": {"@id": "f:RDFS"},
      "f:overrideControl": {"@id": "f:OverrideAll"}
    }
  }
}
```

See [Setting groups — reasoningDefaults](../ledger-config/setting-groups.md)
for full configuration options.

### 2. Per-query override

Any query can specify or override the reasoning mode:

```json
{
  "select": ["?s"],
  "where": {"@id": "?s", "@type": "ex:Person"},
  "reasoning": "rdfs"
}
```

Use `"reasoning": "none"` to explicitly disable reasoning for a single query,
even if the ledger has defaults configured.

See [Query-time reasoning](../query/reasoning.md) for complete syntax and
examples.

## Key concepts

### Schema as data

Unlike systems with external schema files, Fluree stores ontology declarations
as regular triples in your graph. An `rdfs:subClassOf` assertion is just another
triple — you add it via a normal transaction:

```json
{
  "@context": {
    "rdfs": "http://www.w3.org/2000/01/rdf-schema#",
    "ex": "http://example.org/"
  },
  "insert": {
    "@id": "ex:Student",
    "rdfs:subClassOf": {"@id": "ex:Person"}
  }
}
```

This means your schema evolves with your data, is time-travelable, and is
subject to the same policy controls as any other data.

### Derived facts are virtual

Inferred triples exist only in a query-time overlay — they are never written to
storage. This means:

- **No storage bloat** — you don't pay disk costs for derived facts.
- **Always consistent** — derived facts are recomputed from the current state,
  so they can never go stale.
- **Time-travel safe** — querying a historical point in time materializes based
  on that point's data and schema.

### owl:sameAs and identity

When OWL 2 RL is enabled, the engine tracks `owl:sameAs` equivalences using an
efficient union-find data structure. If two resources are determined to be the
same (via functional properties, inverse functional properties, or `owl:hasKey`),
all their facts are merged under a canonical representative. Queries
transparently resolve through these equivalences.

## What to read next

| Topic | Page |
|-------|------|
| Using reasoning in queries | [Query-time reasoning](../query/reasoning.md) |
| Writing custom inference rules | [Datalog rules](../query/datalog-rules.md) |
| Full list of supported OWL & RDFS constructs | [OWL & RDFS reference](../reference/owl-rdfs-support.md) |
| Configuring ledger-wide defaults | [Setting groups](../ledger-config/setting-groups.md) |


---

# guides/README.md

# Guides

Practical, task-oriented cookbooks for Fluree's key features. Each guide shows working patterns you can adapt to your use case.

If you're new to Fluree, start with the [Getting Started](../getting-started/README.md) section first.

## Cookbooks

### [Query Patterns](cookbook-query-patterns.md)

Recipes for the list-value and path operators: dense / gap-filled series with `unwind` + `range`, collecting values into lists, the collect→unwind round-trip, working with list values, and shortest-path queries.

### [Full-Text and Vector Search](cookbook-search.md)

Set up BM25 full-text search and vector similarity. Insert searchable data, write relevance-ranked queries, combine search with graph patterns, and build hybrid text+vector search.

### [Time Travel](cookbook-time-travel.md)

Practical patterns for temporal queries: audit trails, point-in-time comparison, compliance snapshots, recovering deleted data, and transaction metadata.

### [Branching and Merging](cookbook-branching.md)

Git-like workflows for data: safe experimentation, review-before-merge, multi-environment setups, feature branches, and rebase strategies.

### [Access Control Policies](cookbook-policies.md)

Set up fine-grained access control: department isolation, role-based access, property redaction, multi-tenant isolation, and default-deny patterns.

### [SHACL Validation](cookbook-shacl.md)

Define data quality constraints: required properties, datatype validation, value ranges, string patterns, cardinality, and allowed values.

### [Edge Annotations](cookbook-edge-annotations.md)

Attach properties to a relationship: model property-graph edges, record statement-level provenance, represent parallel relationships, query inline or annotation-rooted, and understand the retract cascade — in JSON-LD and SPARQL 1.2.


---

# guides/cookbook-query-patterns.md

# Cookbook: Query Patterns

Practical recipes built on Fluree's list-value and path operators — `unwind`,
`range`, `collect`, the list functions, and `shortestPath`. These are generic
query constructs (they work on any data, not just RDF 1.2 / property-graph
edges). For the full reference, see [JSON-LD Query](../query/jsonld-query.md).

All examples assume `"@context": { "ex": "http://example.org/" }`.

## Dense / gap-filled series

**Problem.** You want a value for *every* point on an axis — every month, every
bucket, every status — including the points that have **no data**. A plain
`groupBy` can't do this: it only ever produces keys that occur in the data, so
empty periods silently vanish and a chart or report has to guess the gaps.

**Pattern.** Generate the axis with `range` + `unwind`, then LEFT JOIN the data
with `optional`. The driving rows come from the generated axis, so empty buckets
survive as zero.

```json
{
  "@context": { "ex": "http://example.org/" },
  "select": ["?year", "(as (count ?o) ?orders)"],
  "where": [
    ["unwind", "?year", "(range 2019 2023)"],
    ["optional", { "@id": "?o", "@type": "ex:Order", "ex:orderYear": "?year" }]
  ],
  "groupBy": ["?year"],
  "orderBy": ["?year"]
}
```

```text
[[2019, 1], [2020, 2], [2021, 0], [2022, 1], [2023, 0]]
```

`2021` and `2023` appear with `0` even though no order has those years —
something neither `values` (constants only) nor a triple pattern (stored data
only) can produce. If the bounds should come from the data, compute the
min/max in a sub-select and feed them to `range`.

## Collecting values into a list

**Problem.** Instead of one row per (author, paper), you want one row per author
with the list of their papers.

**Pattern.** `collect` folds a group's values into a single list value (a JSON
array).

```json
{
  "@context": { "ex": "http://example.org/" },
  "select": ["?name", "(as (collect ?title) ?papers)"],
  "where": [
    { "@id": "?a", "ex:name": "?name", "ex:authored": "?paper" },
    { "@id": "?paper", "ex:title": "?title" }
  ],
  "groupBy": ["?a", "?name"]
}
```

```text
[["Alice", ["Graphs", "Indexes", "Graphs"]], ["Bob", ["Joins"]]]
```

Use `collect-distinct` to drop duplicates that arrive via a join (e.g. the same
subject reached through two papers):

```json
{ "select": ["?name", "(as (collect-distinct ?subject) ?subjects)"] }
```

> RDF stores identical triples once, so `collect` and `collect-distinct` differ
> only when a value reaches the aggregate on multiple solution rows (typically
> through a join), not when a predicate is merely repeated in the source data.

## Round-trip: collect, transform, re-expand

**Problem.** You need to operate on a group *as a list* — sort it, take the
first few, dedup it — and then go back to one row per element.

**Pattern.** `collect` in a sub-select, then `unwind` the result in the outer
query. `unwind` is the inverse of `collect`.

```json
{
  "@context": { "ex": "http://example.org/" },
  "select": ["?subject"],
  "where": [
    ["query", {
      "@context": { "ex": "http://example.org/" },
      "select": ["(as (collect-distinct ?s) ?subjects)"],
      "where": [
        { "@id": "ex:alice", "ex:authored": "?paper" },
        { "@id": "?paper", "ex:subject": "?s" }
      ]
    }],
    ["unwind", "?subject", "?subjects"]
  ],
  "orderBy": ["?subject"]
}
```

This yields Alice's distinct subjects, one per row — deduped in the list stage,
then re-expanded.

## Working with list values

`range`, `list`, and `collect` produce lists; the list functions inspect and
transform them. They compose, and pair naturally with `unwind`.

```json
{
  "@context": { "ex": "http://example.org/" },
  "select": ["?count", "?first", "?last"],
  "where": [
    ["bind", "?nums", "(range 1 100)"],
    ["bind", "?count", "(size ?nums)"],
    ["bind", "?first", "(head ?nums)"],
    ["bind", "?last", "(nth ?nums -1)"]
  ]
}
```

```text
[[100, 1, 100]]
```

- `(size ?l)`, `(head ?l)`, `(last ?l)`, `(nth ?l ?i)` (0-based; negatives from
  the end), `(tail ?l)`, `(reverse ?l)`.
- `tail` / `reverse` return lists, so compose them: `(head (reverse ?l))` is the
  last element.

The same functions apply to a list produced by `collect` (in a sub-select) or by
`nodes` on a `shortestPath` result.

## Generating a sequence

`range` + `unwind` is a row generator — useful for pagination windows, numeric
buckets, or any fixed sequence the data doesn't contain.

```json
{
  "select": ["?page"],
  "where": [["unwind", "?page", "(range 1 10)"]]
}
```

For a constant *set* of values, prefer [`values`](../query/jsonld-query.md#values-patterns)
— it's clearer and cheaper. Reach for `unwind` when the list is **computed**
(`range`, `collect`, or a bound list).

## Shortest path between two entities

**Problem.** How are two people connected, and how many hops apart are they?

**Pattern.** `shortestPath` binds the path to a variable; the path functions
read it.

```json
{
  "@context": { "ex": "http://example.org/" },
  "select": ["(as (size (path-pairs ?p)) ?hops)"],
  "where": [
    ["shortestPath", {
      "from": "ex:alice", "to": "ex:dan",
      "via": "ex:knows", "direction": "out",
      "maxHops": 6, "bind": "?p"
    }]
  ]
}
```

- `(path-pairs ?p)` is the list of consecutive node pairs, so its `size` is the
  hop count; `(nodes ?p)` is the ordered node list.
- `["allShortestPaths", { … }]` returns one row per minimal-length path when
  several tie.
- List the actual nodes along the path by unwinding `nodes`:

```json
{
  "select": ["?node"],
  "where": [
    ["shortestPath", { "from": "ex:alice", "to": "ex:dan", "via": "ex:knows", "bind": "?p" }],
    ["unwind", "?node", "(nodes ?p)"]
  ]
}
```

## See also

- [JSON-LD Query](../query/jsonld-query.md) — full reference for `unwind`,
  aggregation functions, list/path functions, and `shortestPath`.
- [Property paths](../query/jsonld-query.md#property-paths) — transitive
  traversal that matches across many hops but binds only the endpoint (vs.
  `shortestPath`, which binds the whole path).
- [Edge annotations](../concepts/edge-annotations.md) — attaching metadata to
  relationships (RDF 1.2 / property-graph edges).


---

# guides/cookbook-search.md

# Cookbook: Full-Text and Vector Search

Fluree integrates BM25 full-text search and vector similarity directly into the query engine. Search results participate in joins, filters, and aggregations like any other graph pattern — no external search service needed.

This guide covers practical patterns for both approaches.

## Quick start: full-text search

### 1. Insert searchable data

Annotate string values with `@fulltext` to make them searchable:

```bash
fluree insert '{
  "@context": {"ex": "http://example.org/"},
  "@graph": [
    {
      "@id": "ex:doc1",
      "@type": "ex:Article",
      "ex:title": "Introduction to Graph Databases",
      "ex:body": {
        "@value": "Graph databases model data as nodes and edges, making relationship queries fast and intuitive. Unlike relational databases, graph databases traverse relationships without expensive joins.",
        "@type": "@fulltext"
      }
    },
    {
      "@id": "ex:doc2",
      "@type": "ex:Article",
      "ex:title": "Time Series vs Graph: When to Use Which",
      "ex:body": {
        "@value": "Time series databases excel at ordered, append-only data. Graph databases shine when relationships between entities matter more than temporal ordering.",
        "@type": "@fulltext"
      }
    },
    {
      "@id": "ex:doc3",
      "@type": "ex:Article",
      "ex:title": "Building REST APIs with Rust",
      "ex:body": {
        "@value": "Rust provides memory safety without garbage collection, making it ideal for high-performance API servers. Popular frameworks include Actix and Axum.",
        "@type": "@fulltext"
      }
    }
  ]
}'
```

In Turtle, use `^^f:fullText`:

```bash
fluree insert '
@prefix ex: <http://example.org/> .
@prefix f:  <https://ns.flur.ee/db#> .

ex:doc4 a ex:Article ;
  ex:title "SPARQL Query Optimization" ;
  ex:body  "Optimizing SPARQL queries requires understanding triple patterns, join ordering, and index selection. The query planner reorders patterns based on estimated cardinality."^^f:fullText .
'
```

### 2. Search with relevance scoring

```bash
fluree query '{
  "@context": {"ex": "http://example.org/"},
  "select": ["?title", "?score"],
  "where": [
    {"@id": "?doc", "@type": "ex:Article", "ex:body": "?body", "ex:title": "?title"},
    ["bind", "?score", "(fulltext ?body \"graph database relationships\")"],
    ["filter", "(> ?score 0)"]
  ],
  "orderBy": [["desc", "?score"]],
  "limit": 10
}'
```

The `fulltext()` function returns a BM25 relevance score. Higher scores mean better matches. Documents with none of the search terms score 0.

### 3. Combine search with graph filters

Search only within a specific category:

```bash
fluree query '{
  "@context": {"ex": "http://example.org/"},
  "select": ["?title", "?score"],
  "where": [
    {
      "@id": "?doc", "@type": "ex:Article",
      "ex:body": "?body", "ex:title": "?title",
      "ex:category": "databases"
    },
    ["bind", "?score", "(fulltext ?body \"query optimization\")"],
    ["filter", "(> ?score 0)"]
  ],
  "orderBy": [["desc", "?score"]]
}'
```

Place graph filters **before** the `fulltext()` bind to reduce the number of documents scored.

## Patterns

### Search across multiple properties

If both `title` and `body` are `@fulltext`, score them separately and combine:

```bash
fluree query '{
  "@context": {"ex": "http://example.org/"},
  "select": ["?title", "?combined"],
  "where": [
    {"@id": "?doc", "ex:ftTitle": "?ft", "ex:body": "?body", "ex:title": "?title"},
    ["bind", "?titleScore", "(fulltext ?ft \"graph databases\")"],
    ["bind", "?bodyScore", "(fulltext ?body \"graph databases\")"],
    ["bind", "?combined", "(+ (* ?titleScore 2.0) ?bodyScore)"],
    ["filter", "(> ?combined 0)"]
  ],
  "orderBy": [["desc", "?combined"]]
}'
```

This weights title matches 2x higher than body matches.

### Search with time travel

Search the knowledge base as it existed at a previous point in time:

```bash
fluree query '{
  "@context": {"ex": "http://example.org/"},
  "from": "mydb:main@t:5",
  "select": ["?title", "?score"],
  "where": [
    {"@id": "?doc", "ex:body": "?body", "ex:title": "?title"},
    ["bind", "?score", "(fulltext ?body \"deployment\")"],
    ["filter", "(> ?score 0)"]
  ],
  "orderBy": [["desc", "?score"]]
}'
```

### Search with aggregation

Count matches by category:

```bash
fluree query '{
  "@context": {"ex": "http://example.org/"},
  "select": ["?category", "?count"],
  "where": [
    {"@id": "?doc", "ex:body": "?body", "ex:category": "?category"},
    ["bind", "?score", "(fulltext ?body \"database\")"],
    ["filter", "(> ?score 0)"]
  ],
  "groupBy": "?category",
  "aggregate": {"?count": ["count", "?doc"]}
}'
```

## Quick start: vector search

### 1. Insert vector embeddings

Annotate arrays with `@vector`:

```bash
fluree insert '{
  "@context": {"ex": "http://example.org/"},
  "@graph": [
    {
      "@id": "ex:product1",
      "@type": "ex:Product",
      "ex:name": "Wireless Headphones",
      "ex:embedding": {"@value": [0.82, 0.15, 0.91, 0.23], "@type": "@vector"}
    },
    {
      "@id": "ex:product2",
      "@type": "ex:Product",
      "ex:name": "Bluetooth Speaker",
      "ex:embedding": {"@value": [0.78, 0.12, 0.88, 0.31], "@type": "@vector"}
    },
    {
      "@id": "ex:product3",
      "@type": "ex:Product",
      "ex:name": "Running Shoes",
      "ex:embedding": {"@value": [0.11, 0.95, 0.05, 0.87], "@type": "@vector"}
    }
  ]
}'
```

Vectors are stored as f32. Values are quantized at ingest time.

### 2. Find similar items

Use `cosineSimilarity` (or `dotProduct`, `euclideanDistance`) to rank by similarity:

```bash
fluree query '{
  "@context": {"ex": "http://example.org/"},
  "select": ["?name", "?sim"],
  "where": [
    {"@id": "?product", "@type": "ex:Product", "ex:name": "?name", "ex:embedding": "?vec"},
    ["bind", "?sim", "(cosineSimilarity ?vec [0.80, 0.14, 0.90, 0.25])"],
    ["filter", "(> ?sim 0.9)"]
  ],
  "orderBy": [["desc", "?sim"]],
  "limit": 5
}'
```

### 3. Combine vector search with graph patterns

Find products similar to a query vector, but only in a specific category:

```bash
fluree query '{
  "@context": {"ex": "http://example.org/"},
  "select": ["?name", "?sim"],
  "where": [
    {
      "@id": "?product", "@type": "ex:Product",
      "ex:name": "?name", "ex:embedding": "?vec",
      "ex:category": "electronics"
    },
    ["bind", "?sim", "(cosineSimilarity ?vec [0.80, 0.14, 0.90, 0.25])"]
  ],
  "orderBy": [["desc", "?sim"]],
  "limit": 10
}'
```

## Hybrid search: text + vector

Combine BM25 keyword relevance with vector semantic similarity for the best of both:

```bash
fluree query '{
  "@context": {"ex": "http://example.org/"},
  "select": ["?name", "?hybrid"],
  "where": [
    {
      "@id": "?doc", "ex:name": "?name",
      "ex:description": "?desc", "ex:embedding": "?vec"
    },
    ["bind", "?textScore", "(fulltext ?desc \"wireless audio\")"],
    ["bind", "?vecScore", "(cosineSimilarity ?vec [0.80, 0.14, 0.90, 0.25])"],
    ["bind", "?hybrid", "(+ (* ?textScore 0.4) (* ?vecScore 0.6))"],
    ["filter", "(> ?hybrid 0)"]
  ],
  "orderBy": [["desc", "?hybrid"]],
  "limit": 10
}'
```

Adjust the weights (0.4 text, 0.6 vector) based on your use case. Keyword search is better for exact term matching; vector search is better for semantic similarity.

## When to use which

| Approach | Best for | Scale |
|---|---|---|
| Inline `@fulltext` | Keyword search, document ranking | Up to ~500K documents per property |
| [BM25 graph source](../indexing-and-search/bm25.md) | Large-scale text search with WAND pruning | 1M+ documents |
| Inline `@vector` + similarity | Small-to-medium similarity search | Up to ~100K vectors |
| [HNSW index](../indexing-and-search/vector-search.md) | Large-scale approximate nearest neighbor | 100K+ vectors |

## Performance tips

1. **Place graph filters before search** — Reduce the candidate set before scoring
2. **Use `limit`** — BM25 and similarity scoring are per-document operations
3. **Wait for indexing** — Inline `@fulltext` works without an index (novelty fallback) but is 7x faster with a built index
4. **Choose the right scale** — Inline functions work well up to hundreds of thousands of documents. For millions, use the dedicated graph source pipeline

## Related documentation

- [Inline Fulltext Search](../indexing-and-search/fulltext.md) — `@fulltext` datatype reference
- [BM25 Graph Source](../indexing-and-search/bm25.md) — Large-scale text search pipeline
- [Vector Search](../indexing-and-search/vector-search.md) — `@vector` datatype and HNSW indexes
- [JSON-LD Query](../query/jsonld-query.md) — Full query language reference


---

# guides/cookbook-time-travel.md

# Cookbook: Time Travel

Every transaction in Fluree is immutable. The database preserves complete history automatically — no audit tables, no trigger-based logging, no slowly-changing dimensions. This guide covers practical patterns for using time travel.

## Basics

### Query by transaction number

Every transaction increments a counter (`t`). Query data as it was after any transaction:

```bash
# Current state
fluree query 'SELECT ?name ?salary WHERE { ?p schema:name ?name ; ex:salary ?salary }'

# State after transaction 5
fluree query --at 5 'SELECT ?name ?salary WHERE { ?p schema:name ?name ; ex:salary ?salary }'

# State after the very first transaction
fluree query --at 1 'SELECT ?s ?p ?o WHERE { ?s ?p ?o }'
```

### Query by ISO timestamp

Use a timestamp to query the state at a specific moment:

```bash
fluree query --at 2025-01-15T00:00:00Z \
  'SELECT ?name ?email WHERE { ?p schema:name ?name ; schema:email ?email }'
```

Fluree finds the most recent transaction at or before the given timestamp.

### Query by commit ID

Every commit has a content-addressed ID (CID). Query by exact commit:

```bash
fluree query --at bafyreif... \
  'SELECT ?s ?p ?o WHERE { ?s ?p ?o }'
```

### HTTP API

```bash
# By transaction number
curl -X POST 'http://localhost:8090/v1/fluree/query?ledger=mydb:main&t=5' \
  -H "Content-Type: application/sparql-query" \
  -d 'SELECT ?s ?p ?o WHERE { ?s ?p ?o }'

# By timestamp (URL-encoded)
curl -X POST 'http://localhost:8090/v1/fluree/query?ledger=mydb:main&t=2025-01-15T00%3A00%3A00Z' \
  -H "Content-Type: application/sparql-query" \
  -d 'SELECT ?s ?p ?o WHERE { ?s ?p ?o }'
```

### JSON-LD query with time specifier

```json
{
  "from": "mydb:main@t:5",
  "select": ["?name"],
  "where": [{"@id": "?p", "schema:name": "?name"}]
}
```

## Patterns

### Audit trail: who changed what

View the history of changes to a specific entity:

```bash
fluree history 'PREFIX schema: <http://schema.org/>
PREFIX ex: <http://example.org/>

SELECT ?prop ?value ?t ?op WHERE {
  ex:alice ?prop ?value .
}'
```

Each result includes:
- `?t` — the transaction number
- `?op` — `assert` (added) or `retract` (removed)

### Point-in-time comparison

Compare an entity before and after a change:

```bash
# Before the change (t=5)
fluree query --at 5 'SELECT ?salary WHERE { ex:alice ex:salary ?salary }'

# After the change (t=6)
fluree query --at 6 'SELECT ?salary WHERE { ex:alice ex:salary ?salary }'
```

### Find when a value changed

Track salary history:

```bash
fluree history 'SELECT ?salary ?t ?op WHERE { ex:alice ex:salary ?salary }'
```

Output:
```
?salary  ?t  ?op
85000    1   assert      ← Initial salary
85000    4   retract     ← Old value removed
95000    4   assert      ← New value added
95000    7   retract
110000   7   assert      ← Another raise
```

Each update produces a retract/assert pair at the same `t`.

### Compliance snapshot

Generate a report of all data as it existed on a specific date:

```bash
fluree query --at 2025-06-30T23:59:59Z --format csv \
  'PREFIX schema: <http://schema.org/>
   PREFIX ex: <http://example.org/>

   SELECT ?name ?department ?role
   WHERE {
     ?person a schema:Person ;
             schema:name ?name ;
             ex:department ?department ;
             ex:role ?role .
   }
   ORDER BY ?department ?name' > compliance-report-2025-Q2.csv
```

This is a reproducible snapshot — running the same query with the same timestamp always returns the same results.

### Debugging: find what changed between two points

Compare entity states across a range:

```bash
# What was added or removed between t=10 and t=15?
fluree history 'SELECT ?s ?p ?o ?t ?op WHERE {
  ?s ?p ?o .
  FILTER(?t >= 10 && ?t <= 15)
}'
```

### Recover deleted data

Data that was retracted still exists in history:

```bash
# Carol was deleted at t=8. Recover her data from t=7:
fluree query --at 7 'SELECT ?prop ?value WHERE { ex:carol ?prop ?value }'
```

To restore, simply re-insert the data from the historical query.

### Multi-ledger time travel

Query two ledgers at different points in time:

```json
{
  "from": {
    "products": {"ledger": "catalog:main", "t": 10},
    "orders": {"ledger": "orders:main", "t": 25}
  },
  "select": ["?product", "?price", "?qty"],
  "where": [
    {"@id": "?order", "ex:product": "?p", "ex:quantity": "?qty", "@graph": "orders"},
    {"@id": "?p", "schema:name": "?product", "schema:price": "?price", "@graph": "products"}
  ]
}
```

This joins product data from `t=10` with order data from `t=25` — useful for price-at-time-of-purchase analysis.

### Temporal aggregation

Track how a metric changed over time:

```bash
fluree history 'SELECT ?count ?t ?op WHERE {
  ex:dashboard ex:activeUsers ?count
}'
```

### Transaction metadata

Every commit records metadata. Query it via the `txn-meta` graph:

```sparql
PREFIX f: <https://ns.flur.ee/db#>

SELECT ?t ?timestamp ?author
FROM <urn:fluree:knowledge-base:main#txn-meta>
WHERE {
  ?commit f:t ?t ;
          f:time ?timestamp .
  OPTIONAL { ?commit f:author ?author }
}
ORDER BY DESC(?t)
LIMIT 10
```

## Common questions

**Is time travel expensive?**
No. Querying a historical state uses the same indexes as querying the current state. The cost is O(log n) for index lookups.

**Does old data use extra storage?**
Yes — immutability means retracted values are preserved. Storage grows with the number of changes, not just the current state size. For most workloads this is negligible.

**Can I query "between" two points?**
History queries return all changes with their transaction numbers. Use `FILTER` on `?t` to scope to a range.

**Can I delete history?**
No. Immutability is a core guarantee. If you need to remove data for compliance (e.g., GDPR right to erasure), contact the Fluree team about data compaction options.

## Related documentation

- [Time Travel Concepts](../concepts/time-travel.md) — Architecture and design
- [SPARQL Reference](../query/sparql.md) — History query syntax
- [JSON-LD Query](../query/jsonld-query.md) — Time specifiers in JSON-LD
- [Commit Receipts](../transactions/commit-receipts.md) — Transaction metadata


---

# guides/cookbook-branching.md

# Cookbook: Branching and Merging

Fluree lets you fork a ledger into independent branches, each with its own commit history. Experiment freely, then merge changes back when ready. Think of it like `git branch` for your data.

## Quick start

```bash
# Create a branch from main
fluree branch create experiment

# Switch to the branch
fluree use mydb:experiment

# Make changes (only on the branch)
fluree insert '...'
fluree update '...'

# See both branches
fluree branch list

# Merge back into main
fluree branch merge experiment

# Clean up
fluree branch drop experiment
```

## Core concepts

- **Branches are isolated** — Transactions on one branch are invisible to others
- **Branches are cheap** — Creating a branch doesn't copy data; it creates a new commit pointer
- **Merge is fast-forward** — The target branch must not have diverged. If it has, rebase first
- **Source branch survives merge** — After merging, the branch can continue receiving transactions

## Patterns

### Safe experimentation

Try a risky change without affecting production:

```bash
fluree branch create try-new-schema

fluree use mydb:try-new-schema

# Restructure data
fluree update 'PREFIX ex: <http://example.org/>
DELETE { ?doc ex:category ?cat }
INSERT { ?doc ex:tags ?cat }
WHERE  { ?doc ex:category ?cat }'

# Verify the change looks right
fluree query 'SELECT ?doc ?tag WHERE { ?doc ex:tags ?tag }'

# If it works, merge back
fluree branch merge try-new-schema
fluree branch drop try-new-schema

# If it doesn't work, just drop the branch — main is untouched
fluree branch drop try-new-schema
```

### Review before merge

Use branches as a staging area for data changes:

```bash
# Data engineer creates a branch for the weekly import
fluree branch create weekly-import

fluree use mydb:weekly-import

# Import new data
fluree insert -f new-data.ttl

# Verify: count new entities
fluree query 'SELECT (COUNT(?s) AS ?count) WHERE { ?s a ex:NewRecord }'

# Verify: no duplicates
fluree query 'SELECT ?id (COUNT(?s) AS ?count) WHERE {
  ?s ex:externalId ?id
} GROUP BY ?id HAVING(?count > 1)'

# Looks good — merge into main
fluree branch merge weekly-import
```

### Multi-environment workflow

Use branches to model dev/staging/prod environments within a single ledger:

```bash
# Create environment branches
fluree branch create staging
fluree branch create dev --from staging

# Developers work on dev
fluree use mydb:dev
fluree insert '...'

# Promote to staging via merge
fluree branch merge dev --target staging

# Promote to main (production) after testing
fluree use mydb:staging
# ... run validation queries ...
fluree branch merge staging
```

### Feature branches

Multiple people can work on different features simultaneously:

```bash
# Team member A: add product categories
fluree branch create feature-categories

# Team member B: update pricing
fluree branch create feature-pricing

# Each works independently on their branch
# ...

# Merge sequentially — first one is a fast-forward
fluree branch merge feature-categories

# Second one may need rebase if main advanced
fluree branch rebase feature-pricing
fluree branch merge feature-pricing
```

### Rebase to catch up with upstream

When main has advanced since you branched:

```bash
# Main has new commits that your branch doesn't have
fluree branch rebase my-branch
```

This replays your branch's commits on top of main's current HEAD. Conflict strategies:

| Strategy | Behavior |
|---|---|
| `take-both` (default) | Keep both the source and branch changes |
| `abort` | Stop if any conflicts — let you inspect |
| `take-source` | Source (main) wins on conflict |
| `take-branch` | Branch wins on conflict |
| `skip` | Skip conflicting commits entirely |

```bash
# Rebase with abort on conflict for manual review
fluree branch rebase my-branch --strategy abort

# Rebase where main always wins
fluree branch rebase my-branch --strategy take-source
```

### Compare branches

See what's different between two branches:

```bash
# Query branch for entities not in main
fluree query --ledger mydb:my-branch 'SELECT ?s ?p ?o WHERE {
  ?s ?p ?o .
  FILTER NOT EXISTS {
    SERVICE <fluree:ledger:mydb:main> { ?s ?p ?o }
  }
}'
```

### Time travel across branches

Each branch has its own transaction history. Query any branch at any point in time:

```bash
# Branch state after its 3rd transaction
fluree query --ledger mydb:experiment --at 3 'SELECT ?s ?p ?o WHERE { ?s ?p ?o }'
```

### Branch at a historical point

By default, `branch create` starts the new branch at the source's current HEAD. Pass `--at` to start it at an earlier commit on the source branch instead — useful for recovering to a known-good state, forking off an older release, or experimenting with what-if scenarios from a past point in time.

```bash
# Start a branch at transaction 5 on main
fluree branch create rewind --at t:5

# Or use a hex-digest prefix of the commit
fluree branch create rewind --at 3dd028a7
```

The commit must be reachable from the source branch's HEAD (branching from an unrelated branch's commit is rejected). The new branch starts with no index and replays from genesis on first query — acceptable for small/medium histories; if replay cost matters, transact a small no-op to force an index rebuild.

Full CIDs are also accepted (`--at fluree:commit:sha256:...`) and resolve without requiring the source to be indexed; `t:N` and hex prefixes require an indexed source.

## Branch lifecycle

```
create ──→ transact ──→ rebase (if needed) ──→ merge ──→ drop
              ↑                                  │
              └──────── continue working ←───────┘
```

After merging, the branch is still alive. You can:
- Continue transacting on it (for ongoing work)
- Merge again later (only new commits since last merge are copied)
- Drop it when done

## HTTP API

```bash
# Create a branch
curl -X POST http://localhost:8090/v1/fluree/branch \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb", "branch": "dev", "source": "main"}'

# Branch at a historical commit
curl -X POST http://localhost:8090/v1/fluree/branch \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb", "branch": "rewind", "at": "t:5"}'

# Query a specific branch
curl -X POST 'http://localhost:8090/v1/fluree/query?ledger=mydb:dev' \
  -H "Content-Type: application/sparql-query" \
  -d 'SELECT ?s ?p ?o WHERE { ?s ?p ?o }'

# Merge
curl -X POST http://localhost:8090/v1/fluree/merge \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb", "source": "dev"}'
```

## Best practices

1. **Name branches descriptively** — `weekly-import-2025-04`, `feature-product-tags`, not `test1`
2. **Keep branches short-lived** — Long-lived branches diverge more, making rebase harder
3. **Merge frequently** — Small, frequent merges are easier than large, infrequent ones
4. **Test before merging** — Run validation queries on the branch before promoting
5. **Drop after merging** — Clean up branches you're done with

## Related documentation

- [CLI: branch](../cli/branch.md) — Full command reference
- [Ledgers and Nameservice](../concepts/ledgers-and-nameservice.md) — Branch architecture
- [Time Travel](../concepts/time-travel.md) — Temporal queries on branches


---

# guides/cookbook-policies.md

# Cookbook: Access Control Policies

Fluree policies enforce access control inside the database — individual facts (flakes) are filtered based on the requesting identity. The same query returns different results for different users, automatically. No application-layer filtering needed.

This cookbook walks through the common patterns. For the underlying model see [Policy enforcement](../concepts/policy-enforcement.md); for the full reference see [Policy model and inputs](../security/policy-model.md).

## How a policy is shaped

Every policy is a JSON-LD node typed `f:AccessPolicy`. It has three orthogonal pieces:

| Field | Purpose |
|-------|---------|
| **What it targets** | `f:onProperty`, `f:onClass`, `f:onSubject` (any combination, each an array of `@id` references). Omit all three to make a default policy that applies to every flake. |
| **What it governs** | `f:action` — `f:view` (queries), `f:modify` (transactions), or both. |
| **Whether it permits** | Either `f:allow: true` (unconditional allow), `f:allow: false` (deny), or `f:query: "<JSON-encoded WHERE>"` (allow when the embedded query returns at least one binding for the target). |

Two more knobs:

- `f:required: true` — the policy *must* allow for access to be granted on its targets, even if `default-allow` is true. Use it for hard constraints.
- `f:exMessage` — error message returned to the caller when the policy denies a transaction.

Inside `f:query`, two special variables are pre-bound: `?$this` (the entity being checked) and `?$identity` (the requesting identity, supplied via `policy-values`).

## Quick start

### 1. Insert sample data

```bash
fluree insert '{
  "@context": {
    "schema": "http://schema.org/",
    "ex": "http://example.org/"
  },
  "@graph": [
    {
      "@id": "ex:alice",
      "@type": "schema:Person",
      "schema:name": "Alice Chen",
      "ex:role": "engineer",
      "ex:department": "platform",
      "ex:salary": 130000
    },
    {
      "@id": "ex:bob",
      "@type": "schema:Person",
      "schema:name": "Bob Martinez",
      "ex:role": "manager",
      "ex:department": "platform",
      "ex:salary": 155000
    },
    {
      "@id": "ex:carol",
      "@type": "schema:Person",
      "schema:name": "Carol White",
      "ex:role": "engineer",
      "ex:department": "marketing",
      "ex:salary": 115000
    }
  ]
}'
```

Add identity records that link DIDs / users to the entities they represent:

```bash
fluree insert '{
  "@context": {"ex": "http://example.org/", "f": "https://ns.flur.ee/db#"},
  "@graph": [
    { "@id": "ex:aliceIdentity", "ex:user": {"@id": "ex:alice"},
      "f:policyClass": [{"@id": "ex:CorpPolicy"}] },
    { "@id": "ex:bobIdentity",   "ex:user": {"@id": "ex:bob"},
      "f:policyClass": [{"@id": "ex:CorpPolicy"}] }
  ]
}'
```

`f:policyClass` tags an identity with the set of policy classes that apply to it — every stored policy of that class will be loaded automatically when this identity makes a request.

### 2. Insert policies

Policies are data — they go into the ledger like any other graph:

```bash
fluree insert '{
  "@context": {
    "f": "https://ns.flur.ee/db#",
    "ex": "http://example.org/",
    "schema": "http://schema.org/"
  },
  "@graph": [
    {
      "@id": "ex:salary-restriction",
      "@type": ["f:AccessPolicy", "ex:CorpPolicy"],
      "f:required": true,
      "f:onProperty": [{"@id": "ex:salary"}],
      "f:action": [{"@id": "f:view"}],
      "f:query": "{\"where\": {\"@id\": \"?$identity\", \"http://example.org/user\": {\"@id\": \"?$subject\"}, \"http://example.org/role\": \"manager\", \"http://example.org/department\": \"?dept\"}, \"$where\": {\"@id\": \"?$this\", \"http://example.org/department\": \"?dept\"}}"
    },
    {
      "@id": "ex:default-view",
      "@type": ["f:AccessPolicy", "ex:CorpPolicy"],
      "f:action": [{"@id": "f:view"}],
      "f:allow": true
    }
  ]
}'
```

What this set of two policies says:

1. **`ex:salary-restriction`** is **required** for `ex:salary`: a request can read `ex:salary` only when `f:query` returns a binding. The query says: *given the identity, find the user it represents; if that user is a manager in the same department as the entity being viewed (`?$this`), allow*.
2. **`ex:default-view`** allows reading everything else.

`f:query` is stored as a JSON string inside the policy because RDF can't hold structured JSON natively. When loaded, the engine parses it and runs it as a subquery with `?$this` and `?$identity` pre-bound.

### 3. Query as different identities

**As Alice (engineer in platform — no manager privilege)**:

```bash
fluree query '{
  "@context": {"schema": "http://schema.org/", "ex": "http://example.org/"},
  "from": "mydb:main",
  "select": ["?name", "?salary"],
  "where": [
    {"@id": "?p", "schema:name": "?name"},
    ["optional", {"@id": "?p", "ex:salary": "?salary"}]
  ],
  "opts": {
    "identity": "ex:aliceIdentity",
    "policy-class": ["ex:CorpPolicy"],
    "default-allow": false
  }
}'
```

Alice sees every name but no salaries — the required policy denies `ex:salary` because she isn't a manager.

**As Bob (manager in platform)**:

Same query, but `"identity": "ex:bobIdentity"`. Bob sees salaries for Alice and Bob (same department) but Carol's salary stays hidden — different department.

## Inline policies (no insert needed)

Don't want to commit policies to the ledger yet? Pass them inline via `opts.policy`:

```json
{
  "from": "mydb:main",
  "select": "?name",
  "where": [{"@id": "?p", "schema:name": "?name"}],
  "opts": {
    "policy": [
      {
        "@id": "ex:adhoc-allow",
        "@type": "f:AccessPolicy",
        "f:action": "f:view",
        "f:allow": true
      }
    ],
    "default-allow": false
  }
}
```

Inline policies are useful for one-off queries, automated tests, and admin scripts. Stored policies (with `policy-class`) are the right approach for production access control because they're versioned, time-travelable, and consistent across all requests.

## Patterns

### Public read

```json
{
  "@id": "ex:public-read",
  "@type": "f:AccessPolicy",
  "f:action": [{"@id": "f:view"}],
  "f:allow": true
}
```

A default-allow policy with no targeting applies to every flake.

### Owner-only access

```json
{
  "@id": "ex:owner-only",
  "@type": ["f:AccessPolicy", "ex:CorpPolicy"],
  "f:required": true,
  "f:action": [{"@id": "f:view"}, {"@id": "f:modify"}],
  "f:query": "{\"where\": {\"@id\": \"?$identity\", \"http://example.org/user\": {\"@id\": \"?$user\"}}, \"$where\": {\"@id\": \"?$this\", \"http://example.org/owner\": {\"@id\": \"?$user\"}}}"
}
```

The query resolves `?$identity → user`, then checks that `?$this` (the entity being read or written) has that user as its `ex:owner`.

### Property redaction (hide a property unless permitted)

```json
[
  {
    "@id": "ex:hide-ssn",
    "@type": ["f:AccessPolicy", "ex:CorpPolicy"],
    "f:required": true,
    "f:onProperty": [{"@id": "ex:ssn"}],
    "f:action": [{"@id": "f:view"}],
    "f:query": "{\"where\": {\"@id\": \"?$identity\", \"http://example.org/role\": \"hr\"}}"
  },
  {
    "@id": "ex:default-view",
    "@type": ["f:AccessPolicy", "ex:CorpPolicy"],
    "f:action": [{"@id": "f:view"}],
    "f:allow": true
  }
]
```

`f:onProperty` scopes the restriction to `ex:ssn` only — every other property still falls under `ex:default-view`. `f:required: true` means the SSN policy MUST allow for any SSN flake to be visible (the default allow doesn't override it on this property).

### Class-scoped restriction

```json
{
  "@id": "ex:employee-only",
  "@type": ["f:AccessPolicy", "ex:CorpPolicy"],
  "f:required": true,
  "f:onClass": [{"@id": "ex:Employee"}],
  "f:action": [{"@id": "f:view"}],
  "f:query": "{\"where\": {\"@id\": \"?$identity\", \"@type\": \"http://example.org/Employee\"}}"
}
```

Anyone querying for `ex:Employee` instances must themselves be tagged as an employee.

### Multi-tenant isolation

```json
{
  "@id": "ex:tenant-isolation",
  "@type": ["f:AccessPolicy", "ex:CorpPolicy"],
  "f:required": true,
  "f:action": [{"@id": "f:view"}, {"@id": "f:modify"}],
  "f:query": "{\"where\": {\"@id\": \"?$identity\", \"http://example.org/tenant\": \"?tenant\"}, \"$where\": {\"@id\": \"?$this\", \"http://example.org/tenant\": \"?tenant\"}}"
}
```

Each tenant only sees and writes data tagged with their own `ex:tenant`. Required-no-targeting means it applies to every flake.

### Hierarchical access (manager sees direct reports)

```json
{
  "@id": "ex:manager-sees-reports",
  "@type": ["f:AccessPolicy", "ex:CorpPolicy"],
  "f:onClass": [{"@id": "schema:Person"}],
  "f:action": [{"@id": "f:view"}],
  "f:query": "{\"where\": {\"@id\": \"?$identity\", \"http://example.org/user\": {\"@id\": \"?$mgr\"}}, \"$where\": {\"@id\": \"?$this\", \"http://example.org/reportsTo\": {\"@id\": \"?$mgr\"}}}"
}
```

### Write protection

```json
{
  "@id": "ex:no-direct-writes",
  "@type": ["f:AccessPolicy", "ex:CorpPolicy"],
  "f:required": true,
  "f:onProperty": [{"@id": "ex:approved"}],
  "f:action": [{"@id": "f:modify"}],
  "f:exMessage": "ex:approved is set by the workflow service only.",
  "f:query": "{\"where\": {\"@id\": \"?$identity\", \"@type\": \"http://example.org/WorkflowService\"}}"
}
```

When the policy denies a transaction, `f:exMessage` is returned to the client.

## Combining algorithm

When multiple policies match a flake:

- A **required** policy must allow. If any required policy denies (or returns no `f:query` bindings), access is denied.
- If no required policy applies, **any** allow is enough — Fluree uses *allow-overrides* over the non-required set.
- If no policy applies, the request falls back to `default-allow`. Setting `default-allow: false` is the fail-closed default for production.

See [Policy model and inputs](../security/policy-model.md#policy-combining-algorithm) for the full state diagram.

## Invoking policies via HTTP

Policies are passed via opts on JSON-LD requests, and via headers on SPARQL requests.

### JSON-LD

```bash
curl -X POST 'http://localhost:8090/v1/fluree/query?ledger=mydb:main' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $JWT" \
  -d '{
    "from": "mydb:main",
    "select": "?name",
    "where": [{"@id": "?p", "schema:name": "?name"}],
    "opts": {
      "identity": "ex:aliceIdentity",
      "policy-class": ["ex:CorpPolicy"],
      "default-allow": false
    }
  }'
```

### SPARQL (headers — no `opts` block in SPARQL)

```bash
curl -X POST 'http://localhost:8090/v1/fluree/query?ledger=mydb:main' \
  -H 'Content-Type: application/sparql-query' \
  -H "Authorization: Bearer $JWT" \
  -H 'fluree-identity: ex:aliceIdentity' \
  -H 'fluree-policy-class: ex:CorpPolicy' \
  -H 'fluree-default-allow: false' \
  -d 'SELECT ?name WHERE { ?p <http://schema.org/name> ?name }'
```

| Header | JSON-LD `opts` field | Value |
|--------|----------------------|-------|
| `fluree-identity` | `identity` | IRI of an identity entity |
| `fluree-policy-class` | `policy-class` | Comma-separated or repeated header; matches `f:policyClass` on stored policies |
| `fluree-policy-values` | `policy-values` | JSON object — extra `?$var` bindings for policy queries |
| `fluree-policy` | `policy` | Inline JSON-LD policy array |
| `fluree-default-allow` | `default-allow` | `true` / `false` |

When the bearer token is verified and the server is configured with `data_auth_default_policy_class`, the verified identity is auto-applied to `policy-values` and the configured class to `policy-class`. See [Configuration](../operations/configuration.md) for those server-side settings.

## Policies are data

Because policies live as flakes in the ledger:

- **Time-travel** — query at any past `t` to see the policies in effect then.
- **Audit** — `SELECT ?p ?action WHERE { ?p a f:AccessPolicy ; f:action ?action }`.
- **Versionable** — change policies through normal transactions; full history kept.
- **Branchable** — try new policies on a branch before merging to main.

## Best practices

1. **Start with `default-allow: false` and required policies.** Fail-closed is easier to reason about than fail-open.
2. **Tag every stored policy with a class** (e.g. `ex:CorpPolicy`) and tag every identity with `f:policyClass`. Pass `policy-class` at query time — Fluree pulls in the matching policy set automatically.
3. **Use `f:onProperty` / `f:onClass` / `f:onSubject` aggressively.** A targeted policy is cheaper to evaluate than a default policy, because Fluree can short-circuit during flake filtering.
4. **Keep `f:query` simple.** It runs once per flake-target during evaluation. Lean on tagged identity properties (`@type`, `f:policyClass`, role flags) rather than deep traversals.
5. **Test with multiple identities.** Verify the same query returns the right shape for each role.
6. **Document intent.** Add `rdfs:label` and `rdfs:comment` to your policy nodes so audits are readable.

## Related documentation

- [Policy enforcement (concepts)](../concepts/policy-enforcement.md) — model and architecture
- [Policy model and inputs](../security/policy-model.md) — full reference
- [Policy in queries](../security/policy-in-queries.md) — query-time enforcement details
- [Policy in transactions](../security/policy-in-transactions.md) — transaction-time enforcement
- [Programmatic policy API (Rust)](../security/programmatic-policy.md) — building policy contexts in code
- [Authentication](../security/authentication.md) — identity, JWTs, and bearer tokens


---

# guides/cookbook-shacl.md

# Cookbook: SHACL Validation

SHACL (Shapes Constraint Language) is a W3C standard for defining constraints on graph data. In Fluree, SHACL shapes are evaluated **at transaction time** — invalid data is rejected before it's committed (or logged as a warning, depending on your config).

This guide covers:

- [When SHACL runs](#when-shacl-runs) — with and without a config graph
- [Enabling SHACL via the config graph](#enabling-shacl-via-the-config-graph)
- [Defining shapes](#defining-shapes) — node shapes, property shapes, targets
- [Constraint patterns](#constraint-patterns) — cardinality, datatype, ranges, patterns, values, class, pair, logical
- [Subclass reasoning](#rdfs-subclass-reasoning-for-shclass) for `sh:class`
- [Predicate-target shapes](#predicate-target-shapes) — `sh:targetSubjectsOf` / `sh:targetObjectsOf`
- [Per-graph enable/disable and warn vs reject](#per-graph-configuration) modes
- [Storing shapes in a named graph](#storing-shapes-in-a-named-graph) with `f:shapesSource`
- [What isn't enforced yet](#not-yet-supported)

## When SHACL runs

Fluree decides whether to run SHACL validation on each transaction using this order:

1. **If a config graph exists with `f:shaclDefaults`** — follow the configured settings per graph (enable/disable, mode).
2. **If no config graph section is present** — fall back to the **shapes-exist heuristic**: if any SHACL shapes are present in the database (as regular RDF triples), validation runs in `Reject` mode. If no shapes are present, validation is skipped entirely (zero overhead).

This means you can start using SHACL **without writing any config** — just transact shapes and they're enforced.

The `shacl` feature must be enabled at build time (it's on by default for the server and CLI binaries). See [Standards and feature flags](../reference/compatibility.md).

## Enabling SHACL via the config graph

Writing ledger config is done via transactions into the **config graph**, whose IRI is always `urn:fluree:{ledger_id}#config`. See [Writing config data](../ledger-config/writing-config.md) for the full pattern.

### Minimal config: enable SHACL, shapes in the default graph

```trig
@prefix f: <https://ns.flur.ee/db#> .

GRAPH <urn:fluree:mydb:main#config> {
  <urn:config:main> a f:LedgerConfig ;
    f:shaclDefaults [
      f:shaclEnabled true ;
      f:validationMode f:ValidationReject
    ] .
}
```

Notes:
- `f:shaclEnabled` defaults to `false` when a `f:shaclDefaults` section exists without it — make the enable decision explicit.
- `f:validationMode` defaults to `f:ValidationReject`. Use `f:ValidationWarn` to log violations without failing the transaction.
- With no explicit `f:shapesSource`, shapes are compiled from the **default graph** (`f:defaultGraph`, g_id=0). See [Storing shapes in a named graph](#storing-shapes-in-a-named-graph) to load from elsewhere.

## Defining shapes

Shapes are ordinary RDF — transact them like any other data. They can be written in Turtle, TriG, or JSON-LD.

### Node shape with property constraints

```turtle
@prefix sh:     <http://www.w3.org/ns/shacl#> .
@prefix schema: <http://schema.org/> .
@prefix ex:     <http://example.org/> .
@prefix xsd:    <http://www.w3.org/2001/XMLSchema#> .

ex:PersonShape a sh:NodeShape ;
  sh:targetClass schema:Person ;
  sh:property [
    sh:path schema:name ;
    sh:datatype xsd:string ;
    sh:minCount 1 ;
    sh:maxCount 1 ;
    sh:message "Every person must have exactly one name"
  ] ;
  sh:property [
    sh:path schema:email ;
    sh:datatype xsd:string ;
    sh:pattern "^[^@]+@[^@]+\\.[^@]+$" ;
    sh:message "Email must be a valid email address"
  ] ;
  sh:property [
    sh:path ex:age ;
    sh:datatype xsd:integer ;
    sh:minInclusive 0 ;
    sh:maxInclusive 200
  ] .
```

### Target types

| Target | Effect |
|--------|--------|
| `sh:targetClass <C>` | Every subject with `rdf:type <C>` (including RDFS subclasses of `<C>` when the hierarchy is available) |
| `sh:targetNode <N>` | The specific subject `<N>` |
| `sh:targetSubjectsOf <P>` | Every subject that currently has predicate `<P>` |
| `sh:targetObjectsOf <P>` | Every node that currently appears as the object of `<P>` |

See [Predicate-target shapes](#predicate-target-shapes) for notes on how the staged-path validator discovers focus nodes for `sh:targetSubjectsOf` / `sh:targetObjectsOf`.

## Constraint patterns

### Cardinality — required and multi-valued

```turtle
ex:ArticleShape a sh:NodeShape ;
  sh:targetClass ex:Article ;
  sh:property [ sh:path ex:title ; sh:minCount 1 ; sh:maxCount 1 ] ;
  sh:property [ sh:path ex:tag   ; sh:minCount 1 ] .
```

### Datatype

```turtle
ex:ProductShape a sh:NodeShape ;
  sh:targetClass ex:Product ;
  sh:property [ sh:path ex:price   ; sh:datatype xsd:decimal ] ;
  sh:property [ sh:path ex:inStock ; sh:datatype xsd:boolean ] .
```

### Numeric ranges

```turtle
ex:OrderShape a sh:NodeShape ;
  sh:targetClass ex:Order ;
  sh:property [
    sh:path ex:quantity ;
    sh:datatype xsd:integer ;
    sh:minInclusive 1 ;
    sh:maxInclusive 10000
  ] .
```

Available: `sh:minInclusive`, `sh:maxInclusive`, `sh:minExclusive`, `sh:maxExclusive`.

### String patterns and length

```turtle
ex:UserShape a sh:NodeShape ;
  sh:targetClass ex:User ;
  sh:property [
    sh:path ex:username ;
    sh:datatype xsd:string ;
    sh:minLength 3 ;
    sh:maxLength 32 ;
    sh:pattern "^[a-zA-Z0-9_]+$"
  ] .
```

`sh:pattern` accepts an optional `sh:flags` string (e.g. `"i"` for case-insensitive).

### Node kind

```turtle
ex:RefShape sh:property [
  sh:path ex:owner ;
  sh:nodeKind sh:IRI
] .
```

Values: `sh:IRI`, `sh:BlankNode`, `sh:Literal`, `sh:BlankNodeOrIRI`, `sh:BlankNodeOrLiteral`, `sh:IRIOrLiteral`.

### Enumerated values

```turtle
ex:TaskShape a sh:NodeShape ;
  sh:targetClass ex:Task ;
  sh:property [
    sh:path ex:status ;
    sh:in ( "todo" "in-progress" "review" "done" )
  ] .
```

`sh:hasValue` requires a specific value to be present.

### Class constraint (with RDFS subclass reasoning)

```turtle
ex:OrderShape a sh:NodeShape ;
  sh:targetClass ex:Order ;
  sh:property [
    sh:path ex:customer ;
    sh:class schema:Person ;
    sh:minCount 1
  ] .
```

Each value of `ex:customer` must have `rdf:type schema:Person` — or `rdf:type` of any class that is `rdfs:subClassOf* schema:Person`. See [RDFS subclass reasoning for sh:class](#rdfs-subclass-reasoning-for-shclass).

### Pair constraints — comparing two properties

```turtle
ex:EventShape a sh:NodeShape ;
  sh:targetClass ex:Event ;
  sh:property [
    sh:path ex:startYear ;
    sh:lessThan ex:endYear
  ] ;
  sh:property [
    sh:path ex:primaryEmail ;
    sh:disjoint ex:secondaryEmail
  ] .
```

| Constraint | Semantic |
|-----------|----------|
| `sh:equals <P>` | Value sets for this path and `<P>` must be identical |
| `sh:disjoint <P>` | Value sets must not overlap |
| `sh:lessThan <P>` | Every value on this path must be strictly less than every value of `<P>` |
| `sh:lessThanOrEquals <P>` | Every value on this path must be ≤ every value of `<P>` |

### Logical constraints

```turtle
ex:ContactShape a sh:NodeShape ;
  sh:targetClass ex:Contact ;
  sh:or (
    [ sh:property [ sh:path schema:email     ; sh:minCount 1 ] ]
    [ sh:property [ sh:path schema:telephone ; sh:minCount 1 ] ]
  ) .
```

Available: `sh:not`, `sh:and`, `sh:or`, `sh:xone`.

### Closed shapes

```turtle
ex:StrictPersonShape a sh:NodeShape ;
  sh:targetClass ex:StrictPerson ;
  sh:closed true ;
  sh:ignoredProperties ( rdf:type ) ;
  sh:property [ sh:path schema:name ; sh:minCount 1 ] .
```

A closed shape forbids any property not explicitly declared (or listed in `sh:ignoredProperties`). `rdf:type` is implicitly ignored per the SHACL spec.

## RDFS subclass reasoning for `sh:class`

`sh:class` honors `rdfs:subClassOf`. Example:

```turtle
ex:Novelist rdfs:subClassOf schema:Person .
ex:pratchett rdf:type ex:Novelist .

ex:BookShape sh:property [
  sh:path ex:author ;
  sh:class schema:Person
] .
```

A book whose `ex:author` is `ex:pratchett` conforms — `ex:pratchett` is a `schema:Person` via `rdfs:subClassOf`.

Fluree resolves this in two tiers:

1. **Fast path**: the ledger's indexed schema hierarchy (`SchemaHierarchy`). Expanded at engine build time so same-class and descendant-class matches are O(1) hashmap hits.
2. **Live fallback**: when the subclass relation was asserted in the current transaction (or any earlier unindexed commit), the fast path misses. The engine then walks `rdfs:subClassOf` via a BFS on the database's SPOT index. This walk is **scoped to the default graph** regardless of the subject's own graph — matching how `SchemaHierarchy` is built and preventing cross-graph issues.

## Predicate-target shapes

`sh:targetSubjectsOf(P)` and `sh:targetObjectsOf(P)` depend on the current state of the database — a subject is a focus node iff it actually has (or is referenced by) predicate `P` in the **post-transaction** view.

Fluree does not precompute target hints from staged flakes. Instead, for each focus node being validated, the engine does a bounded existence check against the post-state:

- `sh:targetSubjectsOf(P)` → SPOT range query `(focus, P, _)`. Non-empty → shape applies.
- `sh:targetObjectsOf(P)` → OPST range query `(_, P, focus)`. Non-empty → shape applies.

This means:
- A **base-state** `(alice, ex:ssn, "123")` makes `sh:targetSubjectsOf(ex:ssn)` fire on alice even when this transaction only retracts `ex:name`.
- A **retraction-only** transaction that removes the last matching edge means the shape no longer applies — the post-state check returns empty.
- The check is **bounded** by the number of predicate-targeted shapes in the cache, not the data size.

Ref-objects of asserted flakes are pulled into the focus set for their graph, so newly-introduced inbound edges trigger validation of the referenced node.

## Per-graph configuration

Each named graph can have its own `f:shaclEnabled` and `f:validationMode` via `f:graphOverrides`:

```trig
@prefix f: <https://ns.flur.ee/db#> .
@prefix ex: <http://example.org/> .

GRAPH <urn:fluree:mydb:main#config> {
  <urn:config:main> a f:LedgerConfig ;
    # Ledger-wide: SHACL on, reject on violation.
    f:shaclDefaults [
      f:shaclEnabled true ;
      f:validationMode f:ValidationReject ;
      f:overrideControl f:OverrideAll
    ] ;
    # Per-graph: ex:scratch has SHACL off; ex:audit uses warn mode.
    f:graphOverrides
      [ a f:GraphConfig ;
        f:targetGraph ex:scratch ;
        f:shaclDefaults [ f:shaclEnabled false ]
      ],
      [ a f:GraphConfig ;
        f:targetGraph ex:audit ;
        f:shaclDefaults [ f:validationMode f:ValidationWarn ]
      ] .
}
```

With this config:

- A violating write to the **default graph** is **rejected** (ledger-wide `Reject`).
- A violating write to `ex:scratch` **passes** without validation (graph disabled).
- A violating write to `ex:audit` **passes** but emits a `tracing::warn!` (`Warn` mode).
- A single multi-graph transaction can mix modes: reject-bucket violations fail the txn; warn-bucket violations get logged.

### Monotonicity

Per-graph configs can only **tighten** the ledger-wide posture:

| Ledger-wide | Per-graph | Effective |
|-------------|-----------|-----------|
| `enabled: false`, `OverrideNone` | `enabled: true` | **disabled** (OverrideNone blocks per-graph) |
| `enabled: true`, `OverrideAll` | `enabled: false` | **disabled** for that graph |
| `mode: warn`, `OverrideAll` | `mode: reject` | **reject** for that graph |

See [Override control](../ledger-config/override-control.md) for the full ruleset.

## Storing shapes in a named graph

`f:shapesSource` points the shape compiler at a specific graph. Useful when you want schema / shapes isolated from data — even the config graph itself can be used as a shape source.

```trig
@prefix f: <https://ns.flur.ee/db#> .

GRAPH <urn:fluree:mydb:main#config> {
  <urn:config:main> a f:LedgerConfig ;
    f:shaclDefaults [
      f:shaclEnabled true ;
      f:shapesSource [
        a f:GraphRef ;
        f:graphSource [ f:graphSelector <http://example.org/shapes> ]
      ]
    ] .
}
```

Semantics:

- `f:shapesSource` is **authoritative, not additive**: when set, shapes come exclusively from the configured graph. Shapes in the default graph are ignored.
- `f:shapesSource` is **non-overridable** — it can only be set in the config graph, not via transaction/query-time options.
- Use `f:graphSelector f:defaultGraph` to explicitly point at the default graph (same as omitting `f:shapesSource`).
- `f:shapesSource` also supports **cross-ledger references** — set `f:ledger` on the inner `f:graphSource` to compile shapes from a different ledger at validation time. See [Cross-ledger governance — Cross-ledger SHACL shapes](../security/cross-ledger-policy.md#cross-ledger-shacl-shapes) for the end-to-end pattern.

## Inline shapes per transaction

In addition to shapes stored in a ledger, a transaction can supply
**inline shapes** via the `opts.shapes` field. The shapes are
enforced only for that one transaction and never written into the
ledger.

```json
{
  "@context": {"ex": "http://example.org/ns/"},
  "@id":   "ex:alice",
  "@type": "ex:Person",
  "opts": {
    "shapes": {
      "@context": {
        "ex":  "http://example.org/ns/",
        "sh":  "http://www.w3.org/ns/shacl#",
        "xsd": "http://www.w3.org/2001/XMLSchema#"
      },
      "@graph": [
        {
          "@id":            "ex:PersonShape",
          "@type":          "sh:NodeShape",
          "sh:targetClass": {"@id": "ex:Person"},
          "sh:property":    {"@id": "ex:pshape_name"}
        },
        {
          "@id":         "ex:pshape_name",
          "sh:path":     {"@id": "ex:name"},
          "sh:minCount": 1,
          "sh:datatype": {"@id": "xsd:string"}
        }
      ]
    }
  }
}
```

Semantics:

- **Additive with the configured source.** Inline shapes enforce
  *alongside* whatever `f:shapesSource` resolves to. A subject
  must satisfy every shape from both sources. Note that
  `f:shapesSource` is itself singular — its `f:graphSource` is
  either local (no `f:ledger`) or cross-ledger (with `f:ledger`),
  not both at the same time. Inline shapes don't change that;
  they layer on top of whichever variant is configured.
- **Transient.** The shapes never appear in the ledger's data and
  vanish after the transaction completes. The next transaction
  without `opts.shapes` runs without them.
- **Gated by config.** If `f:shaclEnabled false` (or no graph is
  enabled), inline shapes do not bypass that posture — operator
  config wins. To use inline shapes on a fresh ledger with no
  config, the shapes-exist heuristic enables validation
  automatically.
- **No audit trail.** Because inline shapes don't persist, it
  isn't possible to reconstruct "which shapes validated which
  commit" from ledger history. If auditability matters, store
  shapes in a `f:shapesSource` graph instead.

Use cases that fit well: ad-hoc shape testing before committing
to `f:shapesSource`, per-tenant validation layers in an
application server, request-scoped governance that should not
become part of permanent ledger state.

## Validation modes

- **`f:ValidationReject`** (default): on any violation, the transaction fails with `ShaclViolation(report)`. The formatted report lists each violation's focus node, property path, and message.
- **`f:ValidationWarn`**: violations are logged via `tracing::warn!` and the transaction proceeds. Any **non-violation** error from the SHACL pipeline (compile failure, range-scan failure) still propagates — Warn mode never silently admits a broken validation pipeline.

## Working with shapes across write surfaces

SHACL validation runs consistently on every write surface:

- JSON-LD / SPARQL transactions (`fluree insert`, `fluree upsert`, `fluree update`)
- Turtle / TriG ingest (`fluree insert-turtle`, `stage_turtle_insert`)
- Commit replay (`push_commits_with_handle`, followers applying upstream commits)

All three routes go through the same post-stage helper, so the ledger's configured SHACL posture (enable/disable, mode, per-graph, shapes source) applies uniformly.

## Not yet supported

The following SHACL constructs are parsed/compiled but currently **no-ops** at validation time. Shapes using them load without error but don't constrain data:

- `sh:uniqueLang`, `sh:languageIn` — require language-tag metadata on flakes, which isn't yet threaded through the validation path.
- `sh:qualifiedValueShape` (+ `sh:qualifiedMinCount` / `sh:qualifiedMaxCount`) — requires recursive nested-shape counting.

These are tracked in the SHACL compliance effort. Contributors: see [Contributing / SHACL implementation](../contributing/shacl-implementation.md).

## Shapes are data

Because shapes live as regular RDF in your ledger:

- **Time-travelable** — `@atT` query any shape's history to see what validation was in effect at a given commit.
- **Versionable** — `delete`/`insert` constraints through ordinary transactions.
- **Queryable** — `SELECT ?shape ?target WHERE { ?shape sh:targetClass ?target }`.
- **Branchable** — test new constraints on a branch; merge when verified.

## Best practices

1. **Start with `sh:minCount`** — missing-value bugs are the most common data quality issue.
2. **Incremental rollout** — deploy shapes in `f:ValidationWarn` mode first. Watch the logs for a sprint, then flip to `f:ValidationReject`.
3. **Per-graph scratch zones** — for experimentation, disable SHACL on a named graph so exploratory transactions don't fail your CI.
4. **`sh:message` everywhere** — custom messages are what end users see when a transaction is rejected. Invest in them early.
5. **`f:shapesSource` for schema hygiene** — keep shapes out of user data graphs so deletes / retractions on user data can't accidentally touch your schema.

## Related documentation

- [Setting Groups — SHACL](../ledger-config/setting-groups.md#shacl-defaults) — Configuration reference for `f:shaclDefaults`
- [Override Control](../ledger-config/override-control.md) — Per-graph / query-time override rules
- [Writing Config Data](../ledger-config/writing-config.md) — How to transact into the config graph
- [Contributing / SHACL implementation](../contributing/shacl-implementation.md) — How the pipeline works internally (for contributors)


---

# guides/cookbook-edge-annotations.md

# Cookbook: Edge Annotations

Edge annotations attach properties to a *relationship* — the connection between two subjects — without hand-modeling an intermediate node. They cover three overlapping needs on one surface: property-graph relationship properties, RDF-star / RDF 1.2 statement-level provenance, and parallel relationships between the same two subjects.

This guide is a set of working patterns. For the model, the cardinality contract, and the full boundary list, read the [Edge annotations concept doc](../concepts/edge-annotations.md) first; for the on-disk representation, see the [storage-internals design doc](../design/edge-annotations.md).

Throughout, the running example is employment: a `worksFor` edge that needs a `role`, a `since` date, and a `confidence`.

## Picking a surface

| You have… | Use | Why |
|---|---|---|
| JSON-LD writes, or you need named-graph edges, or literal-valued edges | **JSON-LD `@annotation`** | Most complete surface — covers everything below. |
| A SPARQL 1.1/1.2 pipeline, or you're porting RDF-star data | **SPARQL 1.2 annotation tail** (`{\| \|}`, `~`, `rdf:reifies`) | Standards syntax. Default-graph only today. |
| A Turtle/TriG/N-Triples/N-Quads file with annotations | Convert to JSON-LD, **or** ingest plain edges then add annotations via SPARQL UPDATE | Those ingest paths don't parse RDF 1.2 tails (see [Turtle ingest](../transactions/turtle.md#edge-annotations-rdf-12--turtle-star)). |

## Attach metadata to a relationship

The `@annotation` block sits on the object node-map of the edge it describes. Everything inside is ordinary RDF stored on a fresh annotation subject.

```json
{
  "@context": {
    "ex": "http://example.org/",
    "xsd": "http://www.w3.org/2001/XMLSchema#"
  },
  "insert": {
    "@id": "ex:alice",
    "ex:worksFor": {
      "@id": "ex:acme",
      "@annotation": {
        "ex:role": "Engineer",
        "ex:since": { "@value": "2024-01-01", "@type": "xsd:date" },
        "ex:confidence": 0.97
      }
    }
  }
}
```

This commits the base edge `ex:alice ex:worksFor ex:acme`, mints an (anonymous) annotation subject, attaches it to that edge, and writes `ex:role` / `ex:since` / `ex:confidence` on it. `@edge` is an interchangeable alias for `@annotation`.

## Query inline: edge first, metadata second

The query shape mirrors the write shape — match the edge, then pull or constrain its metadata.

```json
{
  "@context": { "ex": "http://example.org/" },
  "select": ["?person", "?org", "?role", "?since"],
  "where": {
    "@id": "?person",
    "ex:worksFor": {
      "@id": "?org",
      "@annotation": { "ex:role": "?role", "ex:since": "?since" }
    }
  }
}
```

One row per `(edge, annotation)` pair. **The plain edge pattern is undisturbed:** `?person ex:worksFor ?org` with *no* `@annotation` block still returns one row per edge, regardless of how many annotations hang off it. Annotations only multiply cardinality through the `@annotation` / `@reifies` keywords.

## Query annotation-rooted: metadata first, edge second

When you start from the metadata — "find every employment where `role = Engineer`" — walk back to the edge with `@reifies`.

```json
{
  "@context": { "ex": "http://example.org/" },
  "select": ["?person", "?org", "?since"],
  "where": {
    "ex:role": "Engineer",
    "ex:since": "?since",
    "@reifies": {
      "@id": "?person",
      "ex:worksFor": { "@id": "?org" }
    }
  }
}
```

`@reifies` resolves through the reverse attachment index, so it stays cheap no matter how many annotations the ledger holds. It is **query-side only** — using `@reifies` on a write is rejected; write with `@annotation`.

## Parallel relationships between the same two subjects

Plain RDF can't distinguish two `ex:worksFor` triples between Alice and Acme. Annotations can — Alice was an Engineer, then a Manager:

```json
{
  "@context": { "ex": "http://example.org/" },
  "insert": {
    "@graph": [
      { "@id": "ex:alice", "ex:worksFor": {
          "@id": "ex:acme",
          "@annotation": { "@id": "ex:emp/2020", "ex:role": "Engineer" }
      }},
      { "@id": "ex:alice", "ex:worksFor": {
          "@id": "ex:acme",
          "@annotation": { "@id": "ex:emp/2024", "ex:role": "Manager" }
      }}
    ]
  }
}
```

The inline query returns two rows:

```text
?person     ?org      ?role
ex:alice    ex:acme   Engineer
ex:alice    ex:acme   Manager
```

while `?person ex:worksFor ?org` (no annotation binding) still returns one.

## Provenance on a fact (including literal-valued edges)

The classic RDF-star use case: record where a claim came from and how confident you are. Annotations work on literal-valued edges too — write the literal as a JSON-LD value object so the annotation has a sibling key to attach to.

```json
{
  "@context": { "ex": "http://example.org/" },
  "insert": {
    "@id": "ex:alice",
    "ex:name": {
      "@value": "Alice",
      "@annotation": { "ex:source": "ex:hr-system", "ex:confidence": 0.92 }
    }
  }
}
```

A scalar (`"ex:name": "Alice"`) can't carry sibling metadata — the value-object form is required when annotating a literal. Typed and language-tagged literals follow the same rule (`@type` / `@language` plus `@annotation`), and language-tagged annotations are language-pinned: `"chat"@fr` and `"chat"@en` annotate independently.

## Stable annotation identity

Give an annotation an explicit `@id` when you need to reference, sign, or update it later — "the contract for Alice's 2024 employment":

```json
{
  "@id": "ex:alice",
  "ex:worksFor": {
    "@id": "ex:acme",
    "@annotation": {
      "@id": "ex:employment/alice-acme-2024",
      "ex:role": "Engineer"
    }
  }
}
```

Two inserts targeting the same explicit `@id` reattach to the same subject (idempotent). Two with no `@id` mint two distinct annotations. Explicit-IRI annotations are visible in `select: "*"` and graph crawls like any resource; anonymous ones stay hidden from wildcards and surface only through `@annotation`.

## Update annotation metadata

Once you've bound the annotation — by `@id` or by selector — it's an ordinary RDF subject. Bump Alice's confidence:

```json
{
  "@context": { "ex": "http://example.org/" },
  "where": {
    "@id": "ex:alice",
    "ex:worksFor": {
      "@id": "ex:acme",
      "@annotation": { "@id": "?edge", "ex:role": "Engineer" }
    }
  },
  "delete": { "@id": "?edge", "ex:confidence": "?old" },
  "insert": { "@id": "?edge", "ex:confidence": 0.99 }
}
```

## Retract an edge — and understand the cascade

Retracting the base edge cascades to the annotation's attachment. What happens to the annotation's *body* depends on the mode.

```json
{
  "delete": {
    "@id": "ex:alice",
    "ex:worksFor": { "@id": "ex:acme" }
  }
}
```

- **RDF mode (default):** anonymous annotation subjects on the edge are fully removed (attachment + body). Explicit-IRI annotations keep their body facts as ordinary RDF — only the attachment is retracted, so a user-named resource is never deleted by surprise.
- **LPG mode (`opts.lpgEdgeLifecycle: true`):** explicit-IRI annotations cascade their body too — the property-graph "delete the relationship deletes its properties" lifecycle.

```json
{
  "delete": { "@id": "ex:alice", "ex:worksFor": { "@id": "ex:acme" } },
  "opts": { "lpgEdgeLifecycle": true }
}
```

History preserves both events either way — query at the pre-retract `t` and the annotation comes back. See [Retractions](../transactions/retractions.md#edge-annotation-cascade) for the metadata-only-retract and same-transaction-replacement rules.

## The same patterns in SPARQL 1.2

The SPARQL 1.2 annotation tail lowers to the identical on-disk shape. A conformant `VERSION "1.2"` prologue is accepted (lexed and skipped).

**Write** (anonymous, or named with `~`):

```sparql
PREFIX ex: <http://example.org/>
INSERT DATA {
  ex:alice ex:worksFor ex:acme {| ex:role "Engineer" ; ex:since "2024-01-01" |} .
}
```

```sparql
PREFIX ex: <http://example.org/>
INSERT DATA {
  ex:alice ex:worksFor ex:acme ~ ex:emp1 {| ex:role "Engineer" |} .
}
```

**Query inline:**

```sparql
PREFIX ex: <http://example.org/>
SELECT ?role WHERE {
  ex:alice ex:worksFor ex:acme {| ex:role ?role |} .
}
```

**Query annotation-rooted** with `rdf:reifies` and a parenthesized triple term:

```sparql
PREFIX ex:  <http://example.org/>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
SELECT ?person ?org WHERE {
  ?ann rdf:reifies <<( ?person ex:worksFor ?org )>> ;
       ex:role "Engineer" .
}
```

The triple term `<<( s p o )>>` is accepted **only** as the object of `rdf:reifies`. The bare, parenthesis-free `<< s p o >>` form is the separate Fluree `f:t`/`f:op` flake-metadata construct — the two don't compose. Per-operation reifier rules (variables are template-only; blank/anonymous reifiers are rejected in `DELETE DATA`) are tabulated in the [concept doc](../concepts/edge-annotations.md#sparql-update-rules-by-operation).

## Annotate an edge inside a named graph

Edge annotations live in the same graph as the edge they reify. On the JSON-LD surface, name the target graph with a node-level `@graph` selector — the annotation is written into that same graph and carries the graph identity automatically:

```json
{
  "@context": { "ex": "http://example.org/" },
  "insert": {
    "@id": "ex:alice",
    "@graph": "ex:hr-graph",
    "ex:worksFor": {
      "@id": "ex:acme",
      "@annotation": { "ex:role": "Engineer" }
    }
  }
}
```

> **SPARQL UPDATE is default-graph only today.** An annotation tail inside an explicit `GRAPH { }` block or under a `WITH <g>` template is rejected — use the JSON-LD surface above for named-graph edge annotations.

## Add annotations to data you already ingested as Turtle

The Turtle/N-Triples/TriG/N-Quads ingest paths don't parse annotation tails. Ingest the plain edges, then layer annotations on with SPARQL UPDATE:

```bash
# 1. Ingest the plain edges
curl -X POST "http://localhost:8090/v1/fluree/upsert?ledger=mydb:main" \
  -H "Content-Type: text/turtle" \
  --data-binary '@employments.ttl'

# 2. Add annotations
curl -X POST "http://localhost:8090/v1/fluree/update?ledger=mydb:main" \
  -H "Content-Type: application/sparql-update" \
  --data-binary @- <<'SPARQL'
PREFIX ex: <http://example.org/>
INSERT DATA {
  ex:alice ex:worksFor ex:acme {| ex:role "Engineer" ; ex:since "2024-01-01" |} .
}
SPARQL
```

## Gotchas

- **An annotation reifies exactly one live edge.** A single edge carries many parallel annotations, but one annotation `@id` can't point at two edges at once. To re-home an explicit-IRI annotation, retract the old attachment and assert the new one in the same transaction.
- **Don't write `f:reifies*` predicates by hand.** They're reserved and rejected on every write surface; they're also hidden from `?p` scans and `select: "*"`. Use `@annotation` / the annotation tail. (See [Vocabulary](../reference/vocabulary.md#edge-annotation-predicates-reserved).)
- **Empty `@annotation: {}`** is a no-op in RDF mode (no subject minted); in LPG mode it mints a property-less relationship with identity.
- **Not yet supported** (all reject cleanly, no silent partial results): annotations on `@list` elements, reifiers for unasserted triples, triple terms as object values, annotation output in Turtle/CONSTRUCT, and the SPARQL 1.2 triple-term functions (`TRIPLE`, `isTRIPLE`, …). See [Current limits](../concepts/edge-annotations.md#current-limits).

## See also

- [Edge annotations](../concepts/edge-annotations.md) — the full model and contract
- [Edge annotations storage internals](../design/edge-annotations.md) — on-disk representation
- [Insert](../transactions/insert.md#edge-annotations) and [Retractions](../transactions/retractions.md#edge-annotation-cascade) — write/lifecycle reference
- [JSON-LD query](../query/jsonld-query.md#edge-annotations) and [SPARQL](../query/sparql.md#edge-annotations-sparql-12--rdf-12) — query reference


---

# guides/cookbook-owl-imports.md

# Cookbook: `owl:imports` across named graphs

This walkthrough builds a small two-file ontology, links it together with
`owl:imports`, applies it to instance data, and shows OWL 2 QL and OWL 2 RL
inference firing through the import.

In Fluree, an `owl:imports` target must resolve to **another named graph in the
same ledger** (or to a local graph via `f:ontologyImportMap`). Cross-ledger
imports are not supported. This tutorial uses three named graphs in one ledger:

| Graph IRI                                  | Role                                |
|--------------------------------------------|-------------------------------------|
| *(default graph)*                          | Instance data                       |
| `<http://example.org/onto/core>`           | Core ontology — class hierarchy + `owl:imports` hub |
| `<http://example.org/onto/behaviors>`      | Imported ontology — property characteristics |
| `<urn:fluree:demo:main#config>`            | Ledger config — wires up reasoning  |

> See [Reasoning and inference](../concepts/reasoning.md) for background
> and [Setting groups → reasoningDefaults](../ledger-config/setting-groups.md)
> for the full config schema.

---

## 1. Create the ledger

```bash
fluree init
fluree create demo
```

`demo` becomes the active ledger. Its full ID is `demo:main`, which means the
config named graph IRI is `urn:fluree:demo:main#config` (the `#config`
fragment is a Fluree convention).

---

## 2. Insert instance data into the default graph

Save as `01-data.ttl`:

```turtle
@prefix ex: <http://example.org/> .

# People (typed directly, will be classified further by reasoning)
ex:alice  a ex:GradStudent .
ex:bob    a ex:Person .
ex:carol  a ex:Professor .

# Ancestor chain — exercises owl:TransitiveProperty (declared in the import)
ex:alice  ex:hasAncestor ex:eve .
ex:eve    ex:hasAncestor ex:frank .

# Living arrangement — exercises owl:SymmetricProperty
ex:alice  ex:livesWith   ex:bob .

# Parent/child — exercises owl:inverseOf
ex:carol  ex:parentOf    ex:alice .

# Teaching — exercises rdfs:domain / rdfs:range
ex:professor1 ex:teaches ex:cs101 .
```

Insert it:

```bash
fluree upsert -f 01-data.ttl
# → Committed t=1, 8 flakes
```

> Use `upsert` (not `insert`) for any TriG document that contains `GRAPH`
> blocks. The CLI's `insert` path parses Turtle straight to flakes and does
> not extract `GRAPH` blocks; over HTTP, `/v1/fluree/insert` rejects
> `Content-Type: application/trig` outright. `upsert` handles both Turtle
> and TriG.

---

## 3. Stage the ontology and reasoning config (TriG)

Save as `02-ontology.trig`:

```turtle
@prefix f:    <https://ns.flur.ee/db#> .
@prefix owl:  <http://www.w3.org/2002/07/owl#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix rdf:  <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix ex:   <http://example.org/> .

# ---- Core ontology: class hierarchy + owl:imports hub -----------------
GRAPH <http://example.org/onto/core> {
  <http://example.org/onto/core>
      a owl:Ontology ;
      owl:imports <http://example.org/onto/behaviors> .

  ex:Student      rdfs:subClassOf  ex:Person .
  ex:GradStudent  rdfs:subClassOf  ex:Student .
  ex:Professor    rdfs:subClassOf  ex:Person .
}

# ---- Imported ontology: property characteristics + domain/range -------
GRAPH <http://example.org/onto/behaviors> {
  ex:hasAncestor  a              owl:TransitiveProperty .
  ex:livesWith    a              owl:SymmetricProperty .
  ex:parentOf     owl:inverseOf  ex:childOf .
  ex:teaches      rdfs:domain    ex:Professor ;
                  rdfs:range     ex:Course .
}

# ---- Reasoning configuration ------------------------------------------
# schemaSource = <onto/core>, followOwlImports = true
# → reasoner walks the import closure and projects schema triples from
#   BOTH graphs onto the default graph for inference.
GRAPH <urn:fluree:demo:main#config> {
  <urn:demo:cfg>
      a f:LedgerConfig ;
      f:reasoningDefaults <urn:demo:cfg:reasoning> .

  <urn:demo:cfg:reasoning>
      f:schemaSource      <urn:demo:cfg:schemaref> ;
      f:followOwlImports  true .

  <urn:demo:cfg:schemaref>
      a f:GraphRef ;
      f:graphSource <urn:demo:cfg:schemasrc> .

  <urn:demo:cfg:schemasrc>
      f:graphSelector <http://example.org/onto/core> .
}
```

Submit it:

```bash
fluree upsert -f 02-ontology.trig --format turtle
# → Committed t=2, 17 flakes
```

`--format turtle` is needed because the file extension `.trig` is not on the
auto-detect list; the parser treats the contents as Turtle/TriG.

---

## 4. Verify base data

Without reasoning, only asserted facts are returned:

```bash
fluree query --format json '{
  "@context":{"ex":"http://example.org/"},
  "select":"?s",
  "where":{"@id":"?s","@type":"ex:Person"},
  "reasoning":"none"
}'
# → ["ex:bob"]
```

Only `bob` is *directly* typed `Person`. The schema and the rest of the
classifications are still hidden behind reasoning.

---

## 5. RDFS subclass expansion

`rdfs:subClassOf` is declared in `<onto/core>` (the schemaSource).
With RDFS reasoning, querying for `Person` returns every subclass instance:

```bash
fluree query --format json '{
  "@context":{"ex":"http://example.org/"},
  "select":"?s",
  "where":{"@id":"?s","@type":"ex:Person"},
  "reasoning":"rdfs"
}'
# → ["ex:bob", "ex:carol", "ex:alice"]
```

`alice` (GradStudent → Student → Person) and `carol` (Professor → Person)
are now classified through the hierarchy.

---

## 6. OWL 2 RL inference *through* the import

Everything below uses axioms declared **in the imported `<onto/behaviors>`
graph** — they reach the reasoner only because `owl:imports` resolved
correctly.

### 6.1 `owl:TransitiveProperty` — `ex:hasAncestor`

```bash
fluree query --format json '{
  "@context":{"ex":"http://example.org/"},
  "select":"?a",
  "where":{"@id":"ex:alice","ex:hasAncestor":"?a"},
  "reasoning":"owl2rl"
}'
# → ["ex:eve", "ex:frank"]
```

Asserted: `alice → eve`, `eve → frank`. Inferred via the
TransitiveProperty axiom in the imported graph: `alice → frank`.

### 6.2 `owl:SymmetricProperty` — `ex:livesWith`

```bash
fluree query --format json '{
  "@context":{"ex":"http://example.org/"},
  "select":"?p",
  "where":{"@id":"ex:bob","ex:livesWith":"?p"},
  "reasoning":"owl2rl"
}'
# → ["ex:alice"]
```

Only `alice livesWith bob` was asserted; the symmetric pair is inferred.

### 6.3 `owl:inverseOf` — `parentOf` / `childOf`

```bash
fluree query --format json '{
  "@context":{"ex":"http://example.org/"},
  "select":"?p",
  "where":{"@id":"ex:alice","ex:childOf":"?p"},
  "reasoning":"owl2rl"
}'
# → ["ex:carol"]
```

Asserted: `carol parentOf alice`. Inferred: `alice childOf carol`.

### 6.4 `rdfs:domain` / `rdfs:range`

```bash
fluree query --format json '{
  "@context":{"ex":"http://example.org/"},
  "select":"?p",
  "where":{"@id":"?p","@type":"ex:Professor"},
  "reasoning":"owl2rl"
}'
# → ["ex:carol", "ex:professor1"]
```

`professor1` was never typed. The reasoner infers it from
`teaches rdfs:domain Professor` (declared in the imported graph) plus the
asserted `professor1 teaches cs101`.

```bash
fluree query --format json '{
  "@context":{"ex":"http://example.org/"},
  "select":"?c",
  "where":{"@id":"?c","@type":"ex:Course"},
  "reasoning":"owl2rl"
}'
# → ["ex:cs101"]
```

Same idea on the range side: `cs101` is classified as a Course because of
`teaches rdfs:range Course` in the import.

---

## 7. OWL 2 QL — query rewriting only

OWL 2 QL handles the same constructs as RDFS plus `owl:inverseOf` and
`rdfs:domain`/`range`, but at **query rewrite time** rather than via fact
materialisation. For the patterns above where you query the *inferred*
direction directly, OWL 2 RL is the simpler choice. OWL 2 QL is best when
you want zero materialisation and your queries already align with the
rewriting (e.g., asking for any superclass type).

```bash
fluree query --format json '{
  "@context":{"ex":"http://example.org/"},
  "select":"?s",
  "where":{"@id":"?s","@type":"ex:Person"},
  "reasoning":"owl2ql"
}'
# → ["ex:bob", "ex:carol", "ex:alice"]
```

Same answer as RDFS for this pattern, with no materialisation step.

---

## 8. Full chain: combining modes

Combining `rdfs` + `owl2rl` lets schema hierarchy and forward-chained facts
work together. `professor1` appears as a `Person` via:

1. `teaches rdfs:domain Professor`           (imported axiom, OWL 2 RL)
2. `professor1 teaches cs101`                (asserted)
3. ⇒ `professor1 a Professor`                (derived)
4. `Professor rdfs:subClassOf Person`        (core ontology, RDFS)
5. ⇒ `professor1 a Person`                   (derived)

```bash
fluree query --format json '{
  "@context":{"ex":"http://example.org/"},
  "select":"?s",
  "where":{"@id":"?s","@type":"ex:Person"},
  "reasoning":["rdfs","owl2rl"]
}'
# → ["ex:bob", "ex:carol", "ex:professor1", "ex:alice"]
```

---

## Submitting TriG over the HTTP API

The CLI's `upsert` command is one way to load TriG. Against a running
`fluree-db-server`, the same payload goes through the HTTP API. Both
endpoints below accept Turtle/TriG when sent with `Content-Type:
application/trig` (or `text/turtle`):

```bash
# Connection-scoped (specify ledger via query string)
curl -X POST 'http://localhost:8090/v1/fluree/upsert?ledger=demo:main' \
     -H 'Content-Type: application/trig' \
     --data-binary @02-ontology.trig

# Ledger-scoped path form
curl -X POST 'http://localhost:8090/v1/fluree/upsert/demo:main' \
     -H 'Content-Type: application/trig' \
     --data-binary @02-ontology.trig
```

The same TriG `GRAPH` blocks land in the same named graphs as via the CLI;
nothing else changes about the reasoning wiring.

See [HTTP endpoints](../api/endpoints.md) for the full surface area and
[Datasets and named graphs](../concepts/datasets-and-named-graphs.md) for
how named graphs participate in queries.

---

## What was actually proved

Each query above is a load-bearing test that the import closure is being
walked correctly:

| Query                              | Axiom location                | Without `owl:imports` resolution it would… |
|------------------------------------|-------------------------------|--------------------------------------------|
| §6.1 transitive ancestors          | imported graph (`behaviors`)  | …only return `ex:eve` (no transitive closure) |
| §6.2 symmetric `livesWith`         | imported graph                | …return empty (`bob livesWith alice` not asserted) |
| §6.3 `childOf` via inverse         | imported graph                | …return empty (`childOf` is never asserted) |
| §6.4 domain/range classification   | imported graph                | …not classify `professor1` / `cs101` |

If you change `f:followOwlImports` to `false` in the config graph, every
query in §6 except `bob livesWith` collapses back to base data — a useful
toggle for confirming the closure walk is what's doing the work.

## Related references

- [Concepts: Reasoning and inference](../concepts/reasoning.md)
- [Query-time reasoning syntax](../query/reasoning.md)
- [Setting groups → reasoningDefaults](../ledger-config/setting-groups.md)
- [Design: ontology imports](../design/ontology-imports.md)
- [Concepts: Datasets and named graphs](../concepts/datasets-and-named-graphs.md)


---

# design/README.md

# Design

Architecture and design documents for Fluree's internal systems. These documents describe the rationale behind key design decisions, wire formats, and trait architectures.

## Documents

### [Query execution and overlay merge](query-execution.md)

How queries run through a single preparation/execution pipeline, how scan operators select the binary-cursor path vs the range fallback, and where overlay novelty merges with indexed data (including graph scoping boundaries).

### [Auth Contract (CLI ↔ Server)](auth-contract.md)

Wire-level contract between the Fluree CLI and any Fluree-compatible server, covering OIDC device auth, token refresh, and storage proxy authentication.

### [Nameservice Schema v2](nameservice-schema-v2.md)

Design of the nameservice schema: ledger records, graph source records, configuration payloads, and the ref/config/tracking store abstractions.

### [Storage-agnostic Commits and Sync](storage-agnostic-commits-and-sync.md)

How ContentId (CIDv1) values decouple the commit chain from storage backends, enabling replication across filesystem, S3, and IPFS. Includes the pack protocol wire format for efficient bulk transfer.

### [ContentId and ContentStore](content-id-and-contentstore.md)

The content-addressed identity layer: `ContentId` type, `ContentStore` trait, multicodec content kinds, and the bridge between CID-based identity and storage-backend addressing.

### [Index Format](index-format.md)

Binary columnar index format: branch/leaf/leaflet hierarchy, dictionary artifacts, SPOT/PSOT/POST/OPST/TSPO layout, and encoding details.

### [Edge annotations (storage internals)](edge-annotations.md)

On-disk representation of RDF 1.2 edge annotations: the durable `f:reifies*` flake bundle, the derived `EdgeKey ↔ subject` annotation arena, the indexer state machine, and garbage-collection reachability. (User-facing contract lives in [Edge annotations (concept doc)](../concepts/edge-annotations.md).)

### [Spatial Index](spatial-index.md)

Geospatial indexing internals: inline GeoPoint encoding (packed 60-bit lat/lng for POINT geometries with latitude-band scans) and the S2 cell-based index for complex geometries, plus the query pipeline and time-travel semantics.

### [Namespace allocation and fallback modes](namespace-allocation.md)

How Fluree assigns `ns_code` values for IRIs (prefix trie matching, fallback split modes), including bulk-import preflight mitigation and how the “host-only” fallback persists for future transactions.

### [Ontology imports (`f:schemaSource` + `owl:imports`)](ontology-imports.md)

How the reasoner consumes schema from a named `f:schemaSource` graph and transitively resolves `owl:imports`: resolution order, the `SchemaBundleOverlay` projection, schema-triple whitelist, and caching.

### [Cross-ledger model enforcement](cross-ledger-model-enforcement.md)

How a single **model ledger** can hold the ontology, SHACL shapes, policy rules, datalog rules, and uniqueness constraints that govern many **data ledgers** that reference it via `f:GraphRef` (`f:ledger`, `f:atT`). Specifies the shared resolver contract, term-space translation, policy IR identity split, failure variants, caching, and phasing.

### [Storage Traits](storage-traits.md)

Storage trait architecture: `StorageRead`, `StorageWrite`, `ContentAddressedWrite`, `Storage`, and `NameService` trait design with guidance for implementing new backends.

### [Raft command queue and replicated state machine](raft-command-queue.md)

How `fluree-db-consensus` replicates writes across a cluster: the queue → stage → apply flow, log entry types, snapshot model, and the rationale behind splitting "decisions" (in the Raft log) from "bytes" (in the shared content-addressed store). Operator-facing recipe lives in [Raft clusters (replicated writes)](../operations/raft-clusters.md).

## Related Documentation

- [Crate Map](../reference/crate-map.md) - Workspace architecture
- [Contributing](../contributing/README.md) - Development guidelines
- [Graph Identities and Naming](../reference/graph-identities.md) - Naming conventions (user-facing and internal)


---

# design/query-execution.md

---
title: Query execution and overlay merge
---

This document describes the **single query execution pipeline** in Fluree DB and how it combines:

- **Indexed data** (binary columnar indexes)
- **Overlay data** (novelty + staged flakes)

It also calls out where **graph scoping** (`g_id`) is applied so named graphs remain isolated.

## Pipeline overview

```mermaid
flowchart TD
  LedgerState -->|produces| LedgerSnapshot
  LedgerSnapshot -->|shared substrate| GraphDb
  GraphDb -->|single-ledger| QueryRunner
  GraphDb -->|member_of| DataSetDb
  DataSetDb -->|federated| QueryRunner
  QueryRunner -->|scan index + merge overlay| DatasetOperator
  DatasetOperator -->|per-graph| BinaryScanOperator
  BinaryScanOperator -->|fast path| BinaryCursor
  BinaryScanOperator -->|fallback| range_with_overlay
  BinaryCursor -->|graph-scoped decode| BinaryGraphView
  range_with_overlay -->|delegates| RangeProvider
```

## Where this exists in code

- **API entrypoints**
  - `fluree-db-api/src/view/query.rs`: single-ledger `GraphDb` queries (`query`)
  - `fluree-db-api/src/view/dataset_query.rs`: dataset queries (`DataSetDb`)

- **Unified query runner**
  - `fluree-db-query/src/execute/runner.rs`
    - `prepare_execution(db: GraphDbRef<'_>, query: &ExecutableQuery)` builds derived facts/ontology (if enabled), rewrites patterns, and builds the operator tree.
    - `execute_prepared(...)` runs the operator tree using an `ExecutionContext`.

- **Dataset operator**
  - `fluree-db-query/src/dataset_operator.rs`
    - `DatasetOperator` wraps every triple-pattern scan. In single-graph mode (the common case) it passes through to one inner `BinaryScanOperator` with negligible overhead. In multi-graph mode (FROM/FROM NAMED datasets) it fans out one inner operator per active graph, drives their lifecycles, and stamps ledger provenance (`Binding::IriMatch`) on results that span multiple ledgers.
    - `DatasetBuilder` trait (factory pattern): the planner constructs a `ScanDatasetBuilder` at plan time; `DatasetOperator` calls `build()` at execution time during `open()` to produce per-graph `BinaryScanOperator`s.
    - Nested composition: inner operators can themselves be `DatasetOperator`s — provenance stamping passes `IriMatch` through unchanged.

- **Scan operators**
  - `fluree-db-query/src/binary_scan.rs`
    - `BinaryScanOperator` handles single-graph scanning only. Selects between binary cursor (streaming, integer-ID pipeline) and range fallback at `open()` time based on the `ExecutionContext`.

- **Range fallback**
  - `fluree-db-core/src/range.rs`: `range_with_overlay(snapshot, g_id, overlay, ...)`
  - `fluree-db-core/src/range_provider.rs`: `RangeProvider` trait implemented by the binary range provider

## Graph scoping (`g_id`)

Graph scoping is applied at two key boundaries:

- **Binary streaming path**: `BinaryCursor` operates on a `BinaryGraphView` (graph-scoped decode handle), ensuring leaf/leaflet decoding, predicate dictionaries, and specialty arenas are graph-isolated.
- **Range path**: `range_with_overlay(snapshot, g_id, overlay, ...)` passes `g_id` into the `RangeProvider`, which routes the range query to the correct per-graph index segments.

Overlay providers are **graph-scoped** at the trait boundary: the overlay hook receives `g_id` and must only return flakes for that graph. This keeps multi-tenant named graphs isolated even when overlay data is sourced externally.

## Overlay merge semantics (high level)

Both scan paths implement the same logical behavior:

- Read matching flakes from the **indexed base** (binary files)
- Read matching flakes from the **overlay** (novelty/staged)
- Merge them using `(t, op)` semantics so retractions cancel assertions as-of the query time bound

The details differ:

- `BinaryScanOperator` translates overlay flakes into integer-ID space and merges them into the decoded columnar stream.
- `RangeScanOperator` delegates to `range_with_overlay`, which combines `RangeProvider` output with overlay output.

## Planner fast paths

Before building the generic operator tree, `build_operator_tree_inner`
(`fluree-db-query/src/execute/operator_tree.rs`) runs a chain of `detect_*`
shape recognizers. When one matches, it builds a specialized `FastPathOperator`
that captures the slow generic tree as a `fallback`; the operator returns
`Ok(None)` from its `open()`-time closure to defer to that fallback whenever its
runtime preconditions do not hold. The whole chain is disabled in History mode
(`!planning.is_history()`).

Fast paths choose one of two overlay strategies (`fast_path_common.rs`):

- **strategy (a) — bail:** `fast_path_store` / `allow_fast_path` decline when
  there is uncommitted overlay (`epoch != 0`), `to_t < max_t`, multi-ledger, a
  `from_t`, or non-root policy. Used by base-only micro-optimizations.
- **strategy (b) — merge:** `allow_cursor_fast_path` admits overlay and
  time-travel because the operator reads through an overlay-merging
  `BinaryCursor` (`build_psot_cursor_for_predicate` /
  `build_post_cursor_for_predicate`) that folds novelty and honors `to_t`.

### Reverse-POST `ORDER BY DESC(?o) LIMIT k`

`detect_post_order_desc_limit` → `fast_post_order_limit::post_order_desc_limit_operator`.

Recognizes `SELECT ?s ?o WHERE { ?s <p> ?o [ ; ?s a <Class> ] } ORDER BY DESC(?o) LIMIT k`
(optional `OFFSET`/`DISTINCT`; `DISTINCT` requires `?o` projected). The POST
index is sorted `(p_id, o_type, o_key, o_i, s_id)`, so for a predicate whose
objects share one **order-preserving** `o_type` (numeric / temporal / boolean —
`fast_path_common::is_post_desc_orderable`), the physical tail of the predicate's
POST range is exactly the DESC top-k. The operator walks POST leaf entries from
the tail, decodes only the rows it keeps, and stops after `OFFSET + LIMIT`
survivors — avoiding a full-predicate drain into the top-k `SortOperator`.

Correctness is enforced at runtime (the detector is shape-only):

- A directory prepass proves the predicate's objects are a single
  order-preserving `o_type` before collecting. If they are not — dict-backed
  strings/refs (which sort *above* numerics/temporals), a mixed leaflet, or more
  than one `o_type` — the operator bails to the generic top-k rather than emit a
  wrong order. (A full prepass, not a streaming check, is required because the
  tail walk stops after `OFFSET + LIMIT` rows.)
- Profitability: for the `?s a <Class>` shape, `detect_post_order_desc_limit`
  consults `StatsView` and declines (defers to the generic plan) when the class
  is selective — i.e. the estimated tail rows to collect `OFFSET + LIMIT` class
  members (`need / (class_count / ndv_subjects(p))`) exceeds `class_count`, so
  anchoring on the class directly is cheaper. The bare shape (no class) always
  wins; missing stats fall through to a runtime scan budget that bails after a
  bounded number of inspected rows. (Note the detector already rejects any other
  pattern, so an arbitrary selective filter like `?s :employeeId 1234` never
  reaches this path — it goes to the generic planner, which anchors on it.)

The gate is `allow_cursor_fast_path` + `to_t >= index_t` (deep time-travel, which
needs the history sidecar, defers). Two lanes split on whether novelty is present:

- **Base lane** (`epoch == 0`, `to_t == index_t`): a plain reverse leaf-walk over
  the persisted index. Class membership uses `batched_lookup_predicate_refs`
  (persisted is exact here).
- **Overlay lane** (`epoch != 0`): the same reverse leaf-walk is merged, in
  descending value order, with the predicate's resolved novelty ops — a *row-set*
  merge (skip base rows retracted by overlay; add overlay asserts; dedup by the
  full V3 fact identity `(s_id, o_key, o_i)` within the single proven `o_type` —
  `o_i` keeps repeated/list values distinct), which sidesteps the "base + asserts
  − retracts" arithmetic pitfall that only afflicts overlay-aware *counting*.
  `rdf:type` class membership is likewise evaluated overlay-correctly (persisted
  base ± novelty type asserts/retracts), and novelty-only subjects are
  materialized through a novelty-aware graph view (the persisted-dict
  `encoded_sid` form would not resolve them).



---

# design/auth-contract.md

# Auth contract (CLI ↔ Server)

This document defines the wire-level contract between the Fluree CLI and any Fluree-compatible server (a standalone `fluree-server`, an OIDC-capable application embedding Fluree, or future products). Any implementation that exposes these endpoints will get zero-configuration CLI auth.

For the overall authentication model, see [Authentication](../security/authentication.md).

## Implementer checklist (CLI compatibility)

An implementation is considered **CLI-compatible** if the Fluree CLI can:

- discover how to authenticate,
- obtain/store a Bearer token, and
- use that token for data-plane operations (and optionally refresh it).

### Required (for “it works”)

- **Auth discovery**: implement `GET /.well-known/fluree.json`.
  - Return at least `{ "version": 1 }`.
  - If you support automated login, include an `auth` object with `type="oidc_device"` and required fields (`issuer`, `client_id`, `exchange_url`).
  - If you do not support automated login, you may omit `auth` (CLI will use manual token input), or return `auth.type="token"` to be explicit.
- **Token exchange / refresh** (only for `auth.type="oidc_device"`): implement `POST {exchange_url}`:
  - `grant_type="urn:ietf:params:oauth:grant-type:token-exchange"` for exchanging an IdP token into a Fluree-scoped token.
  - `grant_type="refresh_token"` for refreshing without user interaction (optional; CLI will still work without refresh, but requires re-login when tokens expire).
- **Issue Fluree-scoped JWTs**: the `access_token` you return MUST include the standard Fluree claims used by `fluree-server`:
  - identity: `fluree.identity` (recommended) and standard `iss/sub/exp/iat`
  - scopes: `fluree.ledger.read.*`, `fluree.ledger.write.*`, `fluree.events.*` (as applicable)
  - replication scopes (`fluree.storage.*`) MUST be reserved for operator/service principals only.

### Recommended (for good UX and supportability)

- **Stable error messages**: keep `error` strings stable and human-readable. The CLI may pattern-match on substrings (e.g. `"Bearer token required"`, `"Untrusted issuer"`) to provide hints.
- **Anti-leak semantics**: for data endpoints, return `404` for out-of-scope ledgers (do not leak existence).
- **Verified diagnostics**: implement `GET /v1/fluree/whoami` (or an equivalent endpoint) to return `token_present`, `verified`, `auth_method`, identity, and scope summary.

## Auth discovery

### `GET /.well-known/fluree.json`

The CLI fetches this endpoint when a remote is added (`fluree remote add`) to auto-configure auth. The server MAY expose this endpoint. If absent, the CLI falls back to manual token configuration.

**Response** (200 OK, `application/json`):

```json
{
  "version": 1,
  "api_base_url": "https://data.example.com/v1/fluree",
  "auth": {
    "type": "oidc_device",
    "issuer": "https://issuer.example.com",
    "client_id": "fluree-cli",
    "exchange_url": "https://data.example.com/v1/fluree/auth/exchange",
    "scopes": ["openid", "profile"],
    "redirect_port": 8400
  }
}
```

### `api_base_url`

`api_base_url` tells the CLI where the Fluree HTTP API is mounted.
It is specifically intended to support implementations that:

- mount the Fluree API under a non-root prefix (e.g. `/v1/fluree`), and/or
- want discovery served from a different host than the data plane (e.g. `www.example.com` serving discovery that points at `data.example.com`).

**Contract:**

- `api_base_url` MAY be:
  - an **absolute URL**, e.g. `https://data.example.com/v1/fluree`, or
  - an **absolute-path reference** (relative to the discovery origin), e.g. `/v1/fluree`.
- If `api_base_url` is an absolute-path reference, the CLI MUST resolve it against the **origin** (scheme + host + port)
  of the discovery document URL it fetched (i.e., the URL used for `GET /.well-known/fluree.json`).
  - Example: discovery fetched from `https://abc123.cloudfront.net/.well-known/fluree.json` and `api_base_url="/v1/fluree"`
    resolves to `https://abc123.cloudfront.net/v1/fluree`.
- `api_base_url` SHOULD include the full prefix including `fluree` and SHOULD NOT have a trailing slash.
- The CLI MUST use the resolved `api_base_url` as the base for subsequent API calls (query/insert/upsert/update/info/exists).
- If `api_base_url` is absent, the CLI MUST derive it from the configured remote URL:
  - If the remote URL already ends with `/fluree`, use it as-is.
  - Otherwise, append `/fluree`.
  - If you mount a versioned API (for example `/v1/fluree`), you SHOULD include `api_base_url` in discovery to avoid ambiguity.

### `auth.type` values

| Type | Meaning | CLI behavior |
|------|---------|--------------|
| `oidc_device` | OIDC interactive login + token exchange | `fluree auth login` uses device-code if the IdP supports it, otherwise auth-code+PKCE |
| `token` | Manual Bearer token (no automated login flow) | `fluree auth login --token <value>` |

### Field reference (`oidc_device`)

| Field | Required | Description |
|-------|----------|-------------|
| `issuer` | Yes | OIDC issuer URL (used for `/.well-known/openid-configuration` discovery) |
| `client_id` | Yes | OAuth client ID for the CLI (public client; no client secret) |
| `exchange_url` | Yes | Absolute URL for the Fluree token exchange endpoint |
| `scopes` | No | OAuth scopes to request (default: `["openid"]`) |
| `redirect_port` | No | Port for auth-code callback listener (default: first available in `8400..8405`; also overrideable via `FLUREE_AUTH_PORT`) |

### Fallback behavior

- Discovery endpoint absent (404 or connection error) → CLI assumes `token` type, prompts user to provide a token manually
- `version` > 1 → CLI warns but attempts to parse known fields

## Token exchange

### `POST {exchange_url}`

After the CLI completes OIDC login with the IdP, it calls the exchange endpoint to trade the IdP token for a Fluree-scoped Bearer token. This endpoint is hosted by the application that manages authorization (e.g., an app embedding Fluree and maintaining user entitlements).

**Request:**

```http
POST /v1/fluree/auth/exchange HTTP/1.1
Content-Type: application/json

{
  "grant_type": "urn:ietf:params:oauth:grant-type:token-exchange",
  "subject_token": "<idp-access-token-or-id-token>",
  "subject_token_type": "urn:ietf:params:oauth:token-type:access_token"
}
```

**Success response** (200 OK):

```json
{
  "access_token": "<fluree-bearer-token>",
  "token_type": "Bearer",
  "expires_in": 3600,
  "refresh_token": "<optional-refresh-token>"
}
```

**Error response** (401/403):

```json
{
  "error": "invalid_grant",
  "error_description": "IdP token is invalid or user is not authorized for Fluree access"
}
```

### Contract

- The exchange endpoint validates the IdP token (against the IdP's JWKS or userinfo), looks up the user's Fluree entitlements, and mints a Fluree-scoped JWT.
- The returned `access_token` MUST be a JWT that `fluree-server` can verify (via JWKS). It MUST include the standard Fluree claims (`fluree.identity`, `fluree.ledger.*`, and optionally `fluree.storage.*`). See [Bearer token claim set](../security/authentication.md#bearer-token-claim-set).
- `refresh_token` is OPTIONAL. If present, the CLI stores it and uses it for silent refresh.
- `subject_token_type` MAY be `urn:ietf:params:oauth:token-type:id_token` if the CLI sends the ID token instead of the access token.

This loosely follows [RFC 8693 (OAuth 2.0 Token Exchange)](https://datatracker.ietf.org/doc/html/rfc8693).

## Token refresh

### `POST {exchange_url}`

If the CLI holds a `refresh_token`, it can request a new access token without user interaction.

**Request:**

```json
{
  "grant_type": "refresh_token",
  "refresh_token": "<stored-refresh-token>"
}
```

**Success response:** Same shape as token exchange success.

**Failure:** CLI clears stored tokens and prompts `fluree auth login`.

## CLI TOML config format

The CLI stores auth configuration per-remote in `.fluree/config.toml`:

```toml
[[remotes]]
name = "solo-prod"
type = "Http"
base_url = "https://solo.example.com"

[remotes.auth]
type = "oidc_device"
issuer = "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_abc123"
client_id = "fluree-cli"
exchange_url = "https://solo.example.com/v1/fluree/auth/exchange"
scopes = ["openid", "profile"]
redirect_port = 8400
token = "eyJ..."           # cached Fluree Bearer token (written by 'fluree auth login')
refresh_token = "eyJ..."   # refresh token (written by 'fluree auth login')

[[remotes]]
name = "local"
type = "Http"
base_url = "http://localhost:8090"

[remotes.auth]
type = "token"
token = "eyJ..."           # manually provided via 'fluree auth login --token'
```

**Backward compatibility:** If `type` is absent, infer `"token"` if `token` is present, otherwise treat as unauthenticated.

## CLI `fluree auth login` behavior

```
fluree auth login [--remote <name>]
```

1. Resolve the target remote.
2. Check `auth.type`:
   - **`oidc_device`**:
     1. Discover OIDC endpoints from `{issuer}/.well-known/openid-configuration`.
     2. If the discovery document includes `device_authorization_endpoint`, run OAuth device-code:
        - POST to `device_authorization_endpoint` to get `device_code`, `user_code`, `verification_uri`.
        - Print: `Open {verification_uri} and enter code: {user_code}`
        - Poll `token_endpoint` until user completes browser auth.
     3. Otherwise, if the discovery document includes `authorization_endpoint`, run OAuth authorization-code + PKCE:
        - Start a localhost callback listener on `http://127.0.0.1:{port}/callback` (port selection: `redirect_port`/`FLUREE_AUTH_PORT`, else first available in `8400..8405`).
        - Open the system browser to the `authorization_endpoint` URL including `code_challenge` and requested `scopes`.
        - Receive the callback, then exchange the code at `token_endpoint`.
        - Note for Cognito: callback URLs must be pre-allowlisted (no wildcard ports); allowlist `http://127.0.0.1:8400/callback` through `http://127.0.0.1:8405/callback` (or your chosen fixed port).
     4. POST IdP token to `exchange_url` → get Fluree Bearer token.
     5. Store `token` and `refresh_token` in remote config.
   - **`token`**: Prompt for token (or accept `--token <value|@file|@->`). Store in config.
   - **Unset / no discovery**: Attempt discovery at `{base_url}/.well-known/fluree.json`. If found, configure auth type and proceed. If not found, fall back to `token` flow.

See [CLI auth command](../cli/auth.md) for full command reference.

## CLI auto-refresh on 401

Auto-refresh applies to **data-plane commands** (`query`, `insert`, `upsert`, `info`) that use `RemoteLedgerClient` in tracked mode or `--remote` mode.

When a data-plane command receives a 401 from the remote:

1. If `auth.type == "oidc_device"` and `refresh_token` is present:
   - Attempt silent refresh via the exchange endpoint.
   - On success: update stored token and (if rotated) refresh token in `.fluree/config.toml`, retry the original request once.
   - On failure: clear tokens, print `Token expired. Run: fluree auth login --remote <name>`
2. Otherwise: print `Authentication failed. Run: fluree auth login --remote <name>`

### Replication commands (`fetch`, `pull`, `push`)

A replication command uses **two** clients, and only one of them auto-refreshes:

- **Metadata calls** (nameservice lookups, commit pagination) go through `RemoteLedgerClient`, which **does** auto-refresh on 401 exactly as described above. After a successful command the CLI persists any rotated token back to `.fluree/config.toml` (`context::persist_refreshed_tokens`).
- **The bulk pack transfer** goes through `HttpRemoteClient` (from `fluree-db-nameservice-sync`), which holds a static token and does **not** auto-refresh.

This split is intentional:

- Replication requires `fluree.storage.*` scopes, which are reserved for operators and service accounts.
- Operator tokens are typically long-lived or non-expiring. If an operator token expires, the user should run `fluree auth login` to obtain a new one.
- Regular users who only have query-scoped tokens should use `fluree track` + `--remote` mode instead of `fetch`/`pull`/`push`.

If the token expires **during** a pack transfer, the CLI does not resume it. It detects the resulting auth failure and fails fast with the `fluree auth login` guidance rather than silently retrying a doomed paginated fallback. Re-authenticate and re-run the transfer.

## Scope rules

- The exchange endpoint MUST NOT grant `fluree.storage.*` to regular users. Replication scope is for operators and service accounts only. See [Replication vs query boundary](../security/authentication.md#replication-vs-query-access-boundary).
- If a user with only query-scoped tokens attempts `fluree pull` or `fluree fetch`, the CLI MUST fail with a clear message explaining that replication requires `fluree.storage.*` and suggesting `fluree track` instead.

## Token diagnostic endpoint

### `GET /v1/fluree/whoami`

A verified diagnostic endpoint that performs full cryptographic verification of the Bearer token (if present) using the same code path as data endpoints. This is the recommended way for the CLI or an implementing application to validate a token without side effects.

**No token:**

```json
{ "token_present": false }
```

**Valid token (verified):**

```json
{
  "token_present": true,
  "verified": true,
  "auth_method": "embedded_jwk",
  "issuer": "did:key:z6Mk...",
  "subject": "admin@example.com",
  "identity": "did:key:z6Mk...",
  "expires_at": 1739012345,
  "scopes": {
    "ledger_read_all": true,
    "ledger_write_all": true
  }
}
```

**Invalid token (verification failed):**

```json
{
  "token_present": true,
  "verified": false,
  "error": "Token expired",
  "issuer": "did:key:z6Mk...",
  "subject": "admin@example.com",
  "expires_at": 1738900000
}
```

When verification fails, the response includes **unverified** decoded claims (base64-decoded without signature check) for debugging. These fields are explicitly untrustworthy — they help diagnose _why_ verification failed (e.g., wrong issuer, expired token) but must never be used for authorization decisions.

The `auth_method` field is only present on successful verification: `"embedded_jwk"` for Ed25519/JWS tokens, `"oidc"` for JWKS/RS256 tokens.

This endpoint always returns `200` regardless of token validity — it is diagnostic, not a gate.

## Error semantics

### Standard error response shape

`fluree-server` returns errors as JSON with a consistent structure. Implementers
SHOULD follow this shape so the CLI can display meaningful diagnostics.

```json
{
  "error": "<human-readable description>",
  "status": 401,
  "@type": "err:db/Unauthorized",
  "cause": {
    "error": "<nested cause (optional)>",
    "status": 400,
    "@type": "err:db/JsonParse"
  }
}
```

Notes:
- `error` is the primary human-readable message. The CLI may pattern-match on substrings inside this field.
- `@type` is a compact error type IRI used as a stable, machine-readable code.
- `cause` is optional and may be nested.
- Implementers MAY include additional fields, but MUST keep `error` stable and human-readable.

### Status codes

| Code | Meaning | When |
|------|---------|------|
| `200` | Success | Request completed successfully |
| `400` | Bad request | Malformed body, invalid JSON, missing required fields |
| `401` | Unauthorized | Missing Bearer token, expired token, invalid signature, unknown signing key |
| `403` | Forbidden | Valid token but insufficient scope (e.g., query-only token on admin endpoint) |
| `404` | Not found **or** unauthorized | Ledger does not exist, **or** token lacks access to this ledger (anti-leak) |
| `409` | Conflict | Ledger already exists (`/fluree/create`), concurrent transaction conflict |
| `500` | Internal error | Server-side failure |

### Anti-leak pattern: 404 for out-of-scope ledgers

Data endpoints (`/fluree/query`, `/fluree/update`, etc.) return `404` rather than `403` when a valid token lacks access to the requested ledger. This prevents authenticated users from discovering the existence of ledgers they are not authorized to access.

**Implication for CLI and implementers:** A `404` on a data endpoint can mean either:
- The ledger genuinely does not exist, or
- The token does not have scope for that ledger.

The CLI should present both possibilities in error messages. Implementers should not attempt to distinguish these cases client-side.

### Token verification errors (401)

Common `401` error messages and their causes:

| Server message | Cause | CLI hint |
|----------------|-------|----------|
| `Bearer token required` | No `Authorization: Bearer ...` header | `fluree auth login --remote <name>` |
| `Invalid token` | Malformed JWT/JWS, bad signature | Re-issue token; check signing key |
| `Token expired` | `exp` claim is in the past | Refresh or re-login |
| `Untrusted issuer` | `iss` / signing key not in trusted list | Check `--trusted-issuer` / `--jwks-issuer` config |
| `OIDC issuer not configured` | Token has `kid` header but no JWKS configured | Add `--jwks-issuer` to server config |
| `Token lacks storage proxy permissions` | Valid token but missing `fluree.storage.*` | Use operator token or `fluree track` instead |

## Implementor checklist

Any Fluree-compatible server that wants zero-config CLI auth must:

1. Expose `GET /.well-known/fluree.json` with the discovery payload
2. Implement `POST {exchange_url}` for token exchange and refresh
3. Issue Fluree-scoped JWTs with the [standard claim set](../security/authentication.md#bearer-token-claim-set)
4. Publish a JWKS endpoint so `fluree-server` can verify issued tokens (configured via `--jwks-issuer`)

### Conformance checklist (status codes)

Implementors MUST return these status codes consistently so the CLI can provide good diagnostics:

| Endpoint | Success | Missing token | Bad token | Insufficient scope | Not found / no access |
|----------|---------|---------------|-----------|---------------------|-----------------------|
| `GET /.well-known/fluree.json` | `200` | n/a | n/a | n/a | `404` (not implemented) |
| `POST /v1/fluree/create` | `201` | `401` | `401` | `403` | n/a |
| `POST /v1/fluree/drop` | `200` | `401` | `401` | `403` | `404` |
| `POST /v1/fluree/query` | `200` | `401` | `401` | `404` (anti-leak) | `404` (anti-leak) |
| `POST /v1/fluree/update` | `200` | `401` | `401` | `404` (anti-leak) | `404` (anti-leak) |
| `POST /v1/fluree/auth/exchange` | `200` | n/a | `401` | `403` | n/a |
| `GET /v1/fluree/whoami` | `200` | `200` (token_present=false) | `200` (verified=false) | n/a | n/a |

### Conformance checklist (error bodies)

All error responses MUST include a JSON body. The body SHOULD include at least an `error` or `message` field. The CLI pattern-matches on specific substrings (e.g., `"Bearer token required"`, `"Untrusted issuer"`) to provide targeted hints, so error messages should be stable across releases.

## See also

- [Authentication](../security/authentication.md) — Auth model, modes, claim set, and access boundaries
- [Configuration — OIDC](../operations/configuration.md#oidc--jwks-token-verification) — Server `--jwks-issuer` setup
- [CLI auth command](../cli/auth.md) — `auth login`, `auth status`, `auth logout`
- [CLI token command](../cli/token.md) — Ed25519 token minting (Mode 2)


---

# design/nameservice-schema-v2.md

# Nameservice Schema v2 Design

**Schema Version: 2**

## Overview

This document describes the design for a unified nameservice schema that supports:

1. **Ledgers** with named graphs and independent indexing
2. **Non-ledger graph sources** (indexes/mappings like BM25, Iceberg/R2RML, Vector/HNSW, JDBC, etc.) with varying versioning semantics
3. **Four independent atomic concerns** that can be updated without contention
4. **Watermarked updates** for client subscription and push notifications
5. **Pluggable backends** (DynamoDB, S3, filesystem) with consistent semantics

Terminology:
- Prefer **graph source** in docs and user-facing API descriptions.
- Non-ledger data sources (BM25, vector, Iceberg, R2RML) are called **graph sources**.

## Design Goals

- **Stable schema**: Minimize attribute changes as features evolve
- **Flexible payloads**: Use JSON Maps for evolving/variable content
- **Reduced conflict probability**: Logically independent concerns minimize contention
- **Client subscriptions**: Watermarks enable efficient change detection
- **Coordination via status**: Soft locks/leases for distributed process coordination

---

## The Four Concerns Model

Each nameservice record has four independent concerns, each with its own watermark and payload:

| # | Concern | Watermark | Payload | Updated By |
|---|---------|-----------|---------|------------|
| 1 | **Head** | `commit_t` | `commit` | Transactor (on commit) |
| 2 | **Index** | `index_t` | `index` | Indexer (on index publish) |
| 3 | **Status** | `status_v` | `status` | Various (state changes, metrics, locks) |
| 4 | **Config** | `config_v` | `config` | Admin (settings changes) |

Each concern can be pushed independently without affecting or contending with the others.

---

## DynamoDB Schema

### Table Name

`fluree-nameservice` (configurable)

### Physical layout: item-per-concern (PK+SK)

DynamoDB serializes writes per *item*, not per attribute. To achieve true per-concern independence (transactor vs indexer vs admin), represent each concern as a **separate item** under the same address partition:

- `pk` (partition key): record address in the `name:branch` form (e.g., `"mydb:main"`, `"products-search:main"`)
- `sk` (sort key): concern discriminator

Recommended `sk` values:
- `meta`
- `head` (ledgers only)
- `index` (ledgers + graph sources)
- `config` (ledgers + graph sources)
- `status` (ledgers + graph sources)

This layout aligns with the file-backed v2 pattern (`.index.json` separate) while also eliminating DynamoDB physical contention between writers.

### Design Note: Per-Concern Independence

Each concern is logically independent:
- **No shared `updated_at`**: Each concern’s watermark (`commit_t`, `index_t`, etc.) serves as its timestamp/version marker
- **Disjoint items**: Updating one concern does not touch any attributes of another concern
- **Reduced conflict probability**: Independent concerns minimize logical contention

With the item-per-concern layout, DynamoDB contention is limited to writers of the **same concern**.

### Entity kinds and graph source types

The `meta` item carries the record discriminator:
- `kind`: `ledger` | `graph_source`
- `source_type` (graph sources only): a type string (e.g., `f:Bm25Index`, `f:HnswIndex`, `f:IcebergSource`, `f:R2rmlSource`, `f:JdbcSource`)

Use `graph_source` naming consistently in `pk` values and type strings.

---

## Watermark Semantics

Watermarks are **strict monotonic** per concern. This ensures:
1. Clients can detect changes by comparing watermarks.
2. No change is ever "invisible" to subscribers.
3. Simple comparison logic: `if remote_watermark > local_watermark then changed`.

### commit_t (Ledger commit watermark)

- **Value**: Equals the commit `t` (transaction time).
- **Update rule**: Strict monotonic (`new_t > current_t`).
- **Rationale**: Commits are already strictly ordered by `t`, so `t` IS the version

### index_t (Index watermark)

- **Value**: Transaction time `t` that the published index covers.
- **Update rule**: Strict monotonic (`new_t > current_t`).
- **Admin reindex**: allow idempotent overwrite at the same `t` (`new_t >= current_t`) when rebuilding an index to the same watermark with a new address.

### status_v (Status Watermark)

- **Value**: Atomic incrementing integer
- **Update rule**: Strict monotonic (`new_v > current_v`)
- **Rationale**: Status has no `t` relation; version is just a change counter

### config_v (Config Watermark)

- **Value**: Atomic incrementing integer
- **Update rule**: Strict monotonic (`new_v > current_v`)
- **Rationale**: Config has no `t` relation; version is just a change counter

### Unborn State Semantics

When a record is initialized but has no data yet for a concern:

| Concern | Unborn Watermark | Unborn Payload | Meaning |
|---------|------------------|----------------|---------|
| `head` | `commit_t = 0` | `commit = null` | Ledger initialized, no commits yet |
| `index` | `index_t = 0` | `index = null` | No index published yet |
| `status` | `status_v = 1` | `status = {state: "ready"}` | Always has initial status |
| `config` | `config_v = 0` | `config = null` | No config set yet |

**Key distinction**:
- `*_v = 0` with `payload = null`: Initialized but unborn (record exists)
- Record not found (GetItem returns nothing): Unknown/never created

---

## Payload Schemas

### commit (Ledger)

```json
{
  "id": "bafybeigdyr...commitCid",
  "t": 42
}
```

| Field | Type | Description |
|-------|------|-------------|
| `id` | String | ContentId (CIDv1) of the commit |
| `t` | Number | Transaction time (redundant with `commit_t` but explicit) |

> See [ContentId and ContentStore](content-id-and-contentstore.md) for details on the CID format.

### index (Ledger with Named Graphs)

```json
{
  "default": {
    "id": "bafybeig...indexRootDefault",
    "t": 42,
    "rev": 0
  },
  "txn-metadata": {
    "id": "bafybeig...indexRootTxnMeta",
    "t": 42,
    "rev": 1
  },
  "audit-log": null
}
```

| Field | Type | Description |
|-------|------|-------------|
| `{named-graph}` | Object \| null | Index state per named graph |
| `.id` | String | ContentId (CIDv1) of the index root |
| `.t` | Number | Transaction time the index covers |
| `.rev` | Number | Revision at that `t` (0, 1, 2... for reindex operations) |

**Named graph = `null`** means that graph exists but hasn't been indexed yet.

### index (Graph Source)

For graph sources with index state (e.g., BM25, vector, spatial, Iceberg, etc.), the nameservice stores a **head pointer** to the graph source's latest index root/manifest. The payload is intentionally **opaque** to nameservice: the graph source implementation defines what the ContentId points to and how (or whether) it supports time travel.

```json
{
  "id": "bafybeig...graphSourceIndexRoot",
  "index_t": 42
}
```

For graph sources with no index concept (e.g., JDBC mappings): `null`.

**Design note**: Snapshot history (if any) is stored in **graph-source-owned manifests in storage**, not in nameservice. See `docs/design/graph-source-index-manifests.md`.

### status

```json
{
  "state": "ready",
  "queue_depth": 3,
  "last_commit_ms": 45
}
```

| Field | Type | Description |
|-------|------|-------------|
| `state` | String | Current state (see State Values below) |
| `*` | Any | Additional metadata varies by state and entity type |

#### State Values

| State | Description | Typical Metadata |
|-------|-------------|------------------|
| `ready` | Normal operating state (default initial state) | `queue_depth`, `last_commit_ms` |
| `indexing` | Background indexing in progress | `index_lock` |
| `reindexing` | Full reindex in progress | `reindex_lock`, `progress` |
| `syncing` | Graph source syncing from source | `progress`, `source_t`, `synced_t` |
| `maintenance` | Administrative maintenance in progress | `maintenance_lock` |
| `retracted` | Soft-deleted | `retracted_at`, `reason` |
| `error` | Error state | `error`, `error_at` |

### status with Locks (Coordination)

```json
{
  "state": "indexing",
  "index_lock": {
    "holder": "indexer-7f3a",
    "target_t": 45,
    "acquired_at": 1705312200,
    "expires_at": 1705316100
  }
}
```

| Field | Type | Description |
|-------|------|-------------|
| `index_lock` | Object \| null | Soft lock for indexing coordination |
| `.holder` | String | Identifier of the process holding the lock |
| `.target_t` | Number | The `t` being indexed |
| `.acquired_at` | Number | Unix epoch when lock was acquired |
| `.expires_at` | Number | Unix epoch when lock expires (lease timeout) |

### config

```json
{
  "default_context_id": "bafkreih...contextCid",
  "index_threshold": 1000,
  "replication": {
    "factor": 3,
    "regions": ["us-east-1", "us-west-2"]
  }
}
```

Config is fully flexible JSON. Common fields:

| Field | Type | Description |
|-------|------|-------------|
| `default_context_id` | String | ContentId (CIDv1) of default JSON-LD context |
| `index_threshold` | Number | Commits before auto-index |
| `replication` | Object | Replication settings |

For graph sources, config contains type-specific settings:

**BM25:**
```json
{
  "k1": 1.2,
  "b": 0.75,
  "fields": ["title", "body", "description"]
}
```

**JDBC:**
```json
{
  "connection_string": "jdbc:postgresql://host:5432/db",
  "schema": "public",
  "pool_size": 10
}
```

---

## DynamoDB Operations

### CAS Semantics (Git-like Push)

All push operations support **compare-and-set (CAS)** semantics with expected old values. This enables Git-like divergence detection:

- Caller provides `expected` (the last-known state) and `new` (the desired state)
- Backend rejects if current state doesn't match `expected`
- On rejection, backend returns `actual` current state for caller to reconcile

This is stronger than simple watermark monotonicity: it detects divergence, not just staleness.

### Create (Initialize)

```
Operation: PutItem
ConditionExpression: attribute_not_exists(#pk)
Item: {
  pk: "mydb:main",
  sk: "meta",
  schema: 2,
  kind: "ledger",
  name: "mydb",
  branch: "main",
  dependencies: null,
  created_at: <now>,          // optional
  updated_at_ms: <now_ms>,
  retracted: false,
}
```

### push_commit (Publish Commit)

**Option A: Monotonic only** (simpler, allows fast-forward by any newer commit)
```
Operation: UpdateItem
Key: { pk: "mydb:main", sk: "head" }
ConditionExpression: attribute_not_exists(#ct) OR #ct < :new_t
UpdateExpression: SET #ct = :new_t, #c = :commit
ExpressionAttributeNames: {
  "#ct": "commit_t",
  "#c": "commit"
}
ExpressionAttributeValues: {
  ":new_t": 42,
  ":commit": { "id": "bafybeig...commitT42", "t": 42 }
}
```

**Option B: CAS with expected value** (Git-like, detects divergence)

CAS checks both watermark equality AND payload equality. The condition is a single OR'd expression handling both existing and unborn cases:

```
Operation: UpdateItem
Key: { pk: "mydb:main", sk: "head" }

// Single condition: existing case OR unborn case
ConditionExpression:
  (#ct = :expected_t AND #c = :expected_commit AND :new_t > :expected_t)
  OR
  (#ct = :zero AND attribute_type(#c, :null_type) AND :new_t > :zero)

UpdateExpression: SET #ct = :new_t, #c = :commit
ExpressionAttributeNames: {
  "#ct": "commit_t",
  "#c": "commit"
}
ExpressionAttributeValues: {
  ":expected_t": 41,                                              // caller's last-known watermark
  ":expected_commit": { "id": "bafybeig...commitT41", "t": 41 },  // caller's last-known payload
  ":new_t": 42,
  ":commit": { "id": "bafybeig...commitT42", "t": 42 },
  ":zero": 0,
  ":null_type": "NULL"
}
```

**Caller logic**: Set `:expected_v` and `:expected_head` based on last-known state:
- If unborn: `:expected_v = 0`, `:expected_head` can be any value (the unborn clause matches on `#hv = :zero`)
- If existing: `:expected_v = last_v`, `:expected_head = last_payload`

**Note**: DynamoDB *does* support nested paths like `#h.#addr` (with `#h=head`, `#addr=address`). However, comparing the entire map (`#h = :expected_head`) is simpler and avoids partial-match edge cases.

**Recommendation**: Use Option B (CAS) for transactors to detect divergence. Use Option A for distributed sync where fast-forward is acceptable.

### push_index (Publish Index)

**CAS with expected watermark + monotonic enforcement:**
```
Operation: UpdateItem
Key: { pk: "mydb:main", sk: "index" }
ConditionExpression: (attribute_not_exists(#it) OR #it < :new_t)
UpdateExpression: SET #it = :new_t, #i = :index
ExpressionAttributeNames: {
  "#it": "index_t",
  "#i": "index"
}
ExpressionAttributeValues: {
  ":new_t": 42,
  ":index": {
    "default": { "id": "bafybeig...indexDefault", "t": 42, "rev": 0 },
    "txn-metadata": { "id": "bafybeig...indexTxnMeta", "t": 42, "rev": 1 }
  },
}
```

**Note**: For admin rebuilds at the same watermark, allow `#it <= :new_t` as the condition (idempotent overwrite at equal `t`).

### push_status (Update Status)

```
Operation: UpdateItem
Key: { pk: "mydb:main", sk: "status" }
ConditionExpression: (#sv = :expected_v AND :new_v > :expected_v)
                     OR
                     (attribute_not_exists(#sv) AND :expected_v = :zero)
UpdateExpression: SET #sv = :new_v, #s = :status
ExpressionAttributeNames: {
  "#sv": "status_v",
  "#s": "status"
}
ExpressionAttributeValues: {
  ":expected_v": 89,
  ":zero": 0,
  ":new_v": 90,
  ":status": { "state": "ready", "queue_depth": 0 }
}
```

**Note**: `status_v` starts at 1 (not 0) on creation, so `attribute_not_exists(#sv)` handles cases where the attribute is missing (e.g., partially-written or manually-created items). Normal updates use the first clause.

### push_config (Update Config)

```
Operation: UpdateItem
Key: { pk: "mydb:main", sk: "config" }
ConditionExpression: (#cv = :expected_v AND :new_v > :expected_v)
                     OR
                     (#cv = :zero AND attribute_type(#c, :null_type) AND :expected_v = :zero)
UpdateExpression: SET #cv = :new_v, #c = :config
ExpressionAttributeNames: {
  "#cv": "config_v",
  "#c": "config"
}
ExpressionAttributeValues: {
  ":expected_v": 2,
  ":zero": 0,
  ":new_v": 3,
  ":config": { "default_context_id": "bafkreih...", "index_threshold": 500 },
  ":null_type": "NULL"
}
```

**Note**: Unborn clause checks both `#cv = :zero` AND `attribute_type(#c, NULL)` to prevent accepting writes against inconsistent states.

### Retract

```
Operation: UpdateItem
Key: { pk: "mydb:main", sk: "meta" }
UpdateExpression: SET #r = :true, #sv = :new_sv, #s = :status
ExpressionAttributeNames: {
  "#r": "retracted",
  "#sv": "status_v",
  "#s": "status"
}
ExpressionAttributeValues: {
  ":true": true,
  ":new_sv": 91,
  ":status": { "state": "retracted", "retracted_at": 1705315800 }
}
```

### Lookup (Read)

```
Operation: GetItem
Key: { pk: "mydb:main", sk: "meta" }
ConsistentRead: true
```

To read full state, query all items for the record address: `pk = "mydb:main"` and assemble `meta + head + index + status + config` as present.

### List by Kind

```
Operation: Query (requires GSI on kind)
KeyConditionExpression: #kind = :kind
ExpressionAttributeNames: { "#kind": "kind" }
ExpressionAttributeValues: { ":kind": "ledger" }
```

To list graph sources, query `kind = graph_source`.

To list graph sources of a specific type (optional GSI), query `source_type = f:Bm25Index`, etc.

---

## Push Result Handling

Each push operation returns one of:

| Result | Meaning | Action |
|--------|---------|--------|
| `Updated` | Update accepted | Proceed |
| `Conflict` | Expected didn't match current | Reconcile using `actual` |

### Rust Types (aligned with existing RefKind/CasResult vocabulary)

```rust
/// Which concern is being read or updated.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
pub enum ConcernKind {
    /// The commit head pointer (`commit_t` + `commit` payload)
    Head,
    /// The index state (`index_t` + `index` payload)
    Index,
    /// The status state (status_v + status payload)
    Status,
    /// The config state (config_v + config payload)
    Config,
}

/// Value of a concern: watermark + optional payload.
///
/// - `Some(ConcernValue { v: 0, payload: None })` — unborn (initialized, no data)
/// - `Some(ConcernValue { v: N, payload: Some(...) })` — has data
/// - `None` (at Option level) — record doesn't exist
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ConcernValue<T> {
    pub v: i64,
    pub payload: Option<T>,
}

/// Outcome of a compare-and-set push operation.
///
/// Conflicts are NOT errors — they are expected outcomes of concurrent
/// writes and must be handled by the caller (retry, report, etc.).
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum CasResult<T> {
    /// CAS succeeded — the concern was updated to the new value.
    Updated,
    /// CAS failed — `expected` did not match the current value.
    /// `actual` carries the current concern value so the caller can decide
    /// what to do next (retry, diverge, etc.).
    Conflict { actual: Option<ConcernValue<T>> },
}
```

### Conflict Handling

On `Conflict`, the caller receives the actual current state and can:

1. **Fast-forward**: If `actual.v < new.v`, retry with `expected = actual`
2. **Divergence**: If `actual.v >= new.v` or addresses differ unexpectedly, handle merge/error
3. **Retry loop**: For distributed systems, implement bounded retry with backoff

```rust
async fn push_with_retry<T>(
    ns: &impl ConcernPublisher<T>,
    address: &str,
    kind: ConcernKind,
    new: ConcernValue<T>,
    max_retries: usize,
) -> Result<CasResult<T>> {
    let mut expected = ns.get_concern(address, kind).await?;

    for _ in 0..max_retries {
        match ns.push_concern(address, kind, expected.as_ref(), &new).await? {
            CasResult::Updated => return Ok(CasResult::Updated),
            CasResult::Conflict { actual } => {
                // Check if fast-forward is still possible
                if let Some(ref act) = actual {
                    if new.v <= act.v {
                        // Diverged - can't fast-forward
                        return Ok(CasResult::Conflict { actual });
                    }
                }
                // Retry with new expected
                expected = actual;
            }
        }
    }

    // Exhausted retries
    let actual = ns.get_concern(address, kind).await?;
    Ok(CasResult::Conflict { actual })
}
```

---

## Example Records

### DynamoDB (item-per-concern) examples

This section shows the DynamoDB **physical layout** (multiple items per address partition).
Other backends serialize the same logical concerns differently.

#### Ledger (typical items)

Ledger records are represented as multiple items under the same `pk`:

```json
{
  "pk": "mydb:main",
  "sk": "meta",
  "schema": 2,
  "kind": "ledger",
  "name": "mydb",
  "branch": "main",
  "created_at": 1705312200,
  "updated_at_ms": 1705312200123,
  "retracted": false
}
```

```json
{
  "pk": "mydb:main",
  "sk": "head",
  "schema": 2,
  "commit_t": 42,
  "commit": { "id": "bafybeig...commitT42", "t": 42 }
}
```

```json
{
  "pk": "mydb:main",
  "sk": "index",
  "schema": 2,
  "index_t": 42,
  "index": {
    "default": { "id": "bafybeig...indexDefaultT42", "t": 42, "rev": 0 }
  }
}
```

```json
{
  "pk": "mydb:main",
  "sk": "config",
  "schema": 2,
  "config_v": 2,
  "config": { "default_context_id": "bafkreih...contextCid", "index_threshold": 1000 }
}
```

```json
{
  "pk": "mydb:main",
  "sk": "status",
  "schema": 2,
  "status_v": 89,
  "status": { "state": "ready", "queue_depth": 3, "last_commit_ms": 45 }
}
```

#### Ledger (unborn)

An "unborn" ledger has all 5 concern items created atomically at initialization. The `head` and `index` items have watermarks set to `0` with null payloads. The `status` item starts at `status_v=1` with `state="ready"`. The `config` item starts at `config_v=0` (unborn).

### Graph Source (BM25)

```json
{
  "pk": "search:main",
  "sk": "meta",
  "schema": 2,
  "kind": "graph_source",
  "source_type": "f:Bm25Index",
  "name": "search",
  "branch": "main",
  "dependencies": ["mydb:main"],
  "created_at": 1705312200,
  "updated_at_ms": 1705312200123,
  "retracted": false
}
```

Additional concern items for the same `pk` (examples):

```json
{
  "pk": "search:main",
  "sk": "config",
  "schema": 2,
  "config_v": 1,
  "config": { "k1": 1.2, "b": 0.75, "fields": ["title", "body"] }
}
```

```json
{
  "pk": "search:main",
  "sk": "index",
  "schema": 2,
  "index_t": 42,
  "index": { "id": "bafybeig...bm25IndexRoot" }
}
```

### Graph Source (Iceberg)

```json
{
  "pk": "analytics:main",
  "sk": "meta",
  "schema": 2,
  "kind": "graph_source",
  "source_type": "f:IcebergSource",
  "name": "analytics",
  "branch": "main",
  "dependencies": ["mydb:main"],
  "created_at": 1705312200,
  "updated_at_ms": 1705312200123,
  "retracted": false,
  "...": "see config/index items"
}
```

### Graph Source (JDBC - No Index)

```json
{
  "pk": "erp:main",
  "sk": "meta",
  "schema": 2,
  "kind": "graph_source",
  "source_type": "f:JdbcSource",
  "name": "erp",
  "branch": "main",
  "dependencies": null,
  "created_at": 1705312200,
  "updated_at_ms": 1705312200123,
  "retracted": false,
  "...": "see config item; index item may be absent or have index_t=0"
}
```

---

## Git-like Push Model

The nameservice follows a git-like model where:

1. **Local nameservice**: Each node has a local NS for reads and local writes
2. **Upstream nameservice**: The "source of truth" that accepts or rejects pushes
3. **Push operations**: Local changes are pushed upstream
4. **Forward operations**: Requests can be forwarded upstream without local write

```
┌─────────────────┐         push_head         ┌─────────────────────┐
│  Transactor     │ ────────────────────────▶ │                     │
│  (local NS)     │                           │   Upstream NS       │
└─────────────────┘                           │                     │
                                              │  - DynamoDB, or     │
┌─────────────────┐         push_index        │  - S3 + ETags, or   │
│  Indexer        │ ────────────────────────▶ │  - FS + locks, or   │
│  (local NS)     │                           │  - Service          │
└─────────────────┘                           │                     │
        ▲                                     │  Enforces:          │
        │              pull/sync              │  - Watermark rules  │
        └─────────────────────────────────────│  - Serialization    │
                                              └─────────────────────┘
```

### Upstream NS Backend Options

| Backend | How It Enforces Rules |
|---------|----------------------|
| **DynamoDB** | Conditional expressions on watermarks |
| **S3** | ETags for CAS + application logic |
| **Filesystem** | File locks or single-writer process |
| **Service** | Queue + application logic |

The push interface is the same regardless of backend.

---

## Status-based Coordination (Soft Locks)

Status can carry soft locks for coordinating distributed processes:

### Lock Acquisition Flow

```
1. Indexer starts up
2. Read current status
3. If index_lock exists and not expired:
     → Another indexer is working, wait or skip
4. If no lock or lock expired:
     → Push status with our lock claim (status_v + 1)
     → If accepted: we own the lock, proceed
     → If rejected: someone else claimed it, back off
5. Do indexing work (periodically refresh lock by pushing status)
6. Push index update
7. Push status: clear lock, set state to ready
```

### Lock Expiry (Crash Recovery)

If a process crashes while holding a lock:
- The `expires_at` timestamp allows other processes to take over
- No manual intervention needed
- Typical lease duration: 5-15 minutes depending on operation

### Lock Refresh

Long-running operations should periodically refresh their lock:

```json
{
  "state": "indexing",
  "index_lock": {
    "holder": "indexer-7f3a",
    "target_t": 45,
    "acquired_at": 1705312200,
    "expires_at": 1705316100,
    "refreshed_at": 1705314000
  },
  "progress": 0.67
}
```

---

## Client Subscription Model

Clients track watermarks to detect changes:

```json
{
  "subscriptions": {
    "mydb:main": {
      "kind": "ledger",
      "commit_t": 42,
      "index_t": 42,
      "status_v": 89,
      "config_v": 2
    },
    "search:main": {
      "kind": "graph_source",
      "source_type": "f:Bm25Index",
      "index_t": 42,
      "status_v": 12,
      "config_v": 1
    }
  }
}
```

### Change Detection

1. Client polls or receives notification
2. Compare watermarks: `if remote.commit_t > local.commit_t`
3. Fetch only the changed concern(s)
4. Update local cache

### Subscription Granularity

Clients can subscribe to:
- **All concerns** for an address
- **Specific concerns** (e.g., only `commit_t` for a query client)
- **All addresses** of a kind (e.g., all ledgers)

---

## File-backed Nameservice Considerations

The logical concerns (head/index/status/config) can be stored in different **physical layouts** depending on the backend.

The file-backed and storage-backed implementations in this repo use the **`ns@v2` JSON-LD** format (see `fluree-db-nameservice/src/file.rs` and `fluree-db-nameservice/src/storage_ns.rs`):
- Main record: `ns@v2/{name}/{branch}.json` (commit/head + status + config-ish fields)
- Index record: `ns@v2/{name}/{branch}.index.json` (index head pointer only)

Field names differ from the DynamoDB layout, but the **semantics match**:
- logical `commit_t` is stored as `f:t`
- logical `commit.id` is stored as `f:ledgerCommit.@id` (a CID string)
- logical `index_t` is stored as `f:ledgerIndex.f:t` (or `f:indexT` for graph source index files)
- logical `index.id` is stored as `f:ledgerIndex.@id` (a CID string, or `f:indexId` for graph source index files)

### Layout Options

**Option A: Single File (Unified)**
```
ns@v2/{name}/{branch}.json
```
- Contains all four concerns in one file
- Simplest for reads (one fetch)
- Requires single-writer discipline or file-level CAS

**Option B: Separate Head and Index Files (Current Implementation)**
```
ns@v2/{name}/{branch}.json        # head + status + config
ns@v2/{name}/{branch}.index.json  # index only
```
- Matches current implementation
- Allows transactor and indexer to write independently
- 2 files to read per entity for full state
- **Trade-off**: Status and config updates contend with head updates at file-lock level. Acceptable if status updates are low-frequency (state changes only, not high-frequency metrics).

**Option C: Fully Separate Files (Maximum Independence)**
```
ns@v2/{name}/{branch}.head.json
ns@v2/{name}/{branch}.index.json
ns@v2/{name}/{branch}.status.json
ns@v2/{name}/{branch}.config.json
```
- Each concern in its own file
- Maximum write independence
- 4 files to read per entity

### Recommended Approach

Use **Option B** (separate head/index) as the default:
- Proven in current implementation
- Solves the main contention issue (transactor vs indexer)
- Reasonable read overhead (2 files)
- **Constraint**: Status updates should be coarse-grained (state transitions, not per-transaction metrics). If high-frequency status updates are needed, consider Option C.

Use **Option C** (fully separate files) when:
- Status updates are frequent (e.g., real-time queue depth reporting)
- Multiple independent processes update different concerns
- Write independence is more important than read efficiency

For **queryable nameservice** with many entities:
- Read files in parallel
- Consider in-memory caching with file-change notification
- The 2-file layout is acceptable; 4-file layout may add too much I/O

### Atomicity Mechanisms

| Backend | Mechanism | Notes |
|---------|-----------|-------|
| Filesystem | Atomic rename (write to temp, rename) | POSIX guarantees |
| S3 | ETags for CAS | `If-Match` header |
| GCS | Generation numbers | Similar to ETags |

### File Content Format

Each file contains JSON matching the concern's payload plus metadata:

**head file** (`{name}/{branch}.json`):
```json
{
  "@context": { "f": "https://ns.flur.ee/db#" },
  "@id": "mydb:main",
  "@type": ["f:Database", "f:LedgerSource"],
  "f:ledger": { "@id": "mydb" },
  "f:branch": "main",
  "f:ledgerCommit": { "@id": "bafybeig...commitT42" },
  "f:t": 42,
  "f:ledgerIndex": { "@id": "bafybeig...indexRootT42", "f:t": 42 },
  "f:status": "ready"
}
```

**index file** (`{name}/{branch}.index.json`):
```json
{
  "@context": { "f": "https://ns.flur.ee/db#" },
  "f:ledgerIndex": { "@id": "bafybeig...indexRootT42", "f:t": 42 }
}
```

---

## Global Secondary Indexes (GSIs)

### GSI1: `gsi1-kind` (Implemented)

| GSI Name | Partition Key | Sort Key | Use Case |
|----------|---------------|----------|----------|
| `gsi1-kind` | `kind` | `pk` | List all entities of a kind (`ledger`, `graph_source`) |

- Only `meta` items carry the `kind` attribute and project into the GSI
- Projection: `INCLUDE` with `name`, `branch`, `source_type`, `dependencies`, `retracted`
- Used by `all_records()` (kind=`ledger`) and `all_vg_records()` (kind=`graph_source`)
- After GSI query returns meta items, `BatchGetItem` fetches remaining concern items (`config`, `index`) to assemble full records

## Appendix: Attribute Reference

All items share:

| Attribute | Type | Description |
|-----------|------|-------------|
| `pk` | String | Record address (`name:branch`) |
| `sk` | String | Concern discriminator (`meta`, `head`, `index`, `status`, `config`) |
| `schema` | Number | Schema version (always `2`) |

### `meta` item

| Attribute | Type | Description |
|-----------|------|-------------|
| `kind` | String | `ledger` \| `graph_source` |
| `name` | String | Base name |
| `branch` | String | Branch name |
| `retracted` | Boolean | Soft-delete flag |
| `branches` | Number | Child branch reference count (0 for leaf branches, omitted when 0 in JSON-LD) |
| `dependencies` | List\<String\> \| null | Graph-source dependencies (optional) |
| `source_type` | String \| null | Graph-source type (e.g., `f:Bm25Index`) |
| `created_at` | Number | Creation timestamp (epoch seconds, optional) |
| `updated_at_ms` | Number | Last update time (epoch millis, optional) |

### `meta` item: Branch Attributes

For branches created via `create_branch`, the `meta` item carries an additional attribute recording the source branch:

| Attribute | Type | Description |
|-----------|------|-------------|
| `bp_source` | String \| null | Source branch name (e.g., `"main"`) |

This attribute is `null`/absent for the original `main` branch. The JSON-LD format uses `f:sourceBranch`. The divergence point between a branch and its source is computed on demand by walking the commit chains rather than being stored.

### `head` item (ledgers only)

| Attribute | Type | Description |
|-----------|------|-------------|
| `commit_t` | Number | Commit watermark (`t`) |
| `commit` | Map \| null | `{ id, t }` (id is a ContentId CID string) |

### `index` item (ledgers + graph sources)

| Attribute | Type | Description |
|-----------|------|-------------|
| `index_t` | Number | Index watermark (`t`) |
| `index` | Map \| null | Ledger index map or graph-source head pointer payload |

### `status` item (ledgers + graph sources)

| Attribute | Type | Description |
|-----------|------|-------------|
| `status_v` | Number | Status change counter |
| `status` | Map | Status payload |
| `status_state` | String \| null | Optional denormalized `status.state` for a GSI |

### `config` item (ledgers + graph sources)

| Attribute | Type | Description |
|-----------|------|-------------|
| `config_v` | Number | Config change counter |
| `config` | Map \| null | Config payload |

### Watermark Semantics Summary

| Watermark | Semantics | Initial Value | Update Rule |
|-----------|-----------|---------------|-------------|
| `commit_t` | = commit `t` | 0 (unborn) | Strict: `new > current` |
| `index_t` | = index `t` | 0 (unborn) | Strict: `new > current` (admin may allow equal) |
| `status_v` | Counter | 1 (ready) | Strict: `new > current` |
| `config_v` | Counter | 0 (unborn) | Strict: `new > current` |


---

# design/storage-agnostic-commits-and-sync.md

# Storage-agnostic commits and sync

Fluree uses **ContentId** (CIDv1) values as the primary identifiers for commits, index roots, and other immutable artifacts. This decouples the commit chain and nameservice references from any specific storage backend, enabling replication across different storage systems (filesystem, S3, IPFS, etc.) without rewriting commit data.

## Pack protocol (`fluree-pack-v1`)

The pack protocol enables efficient bulk transfer of CAS objects between Fluree instances. Instead of fetching each commit individually (one HTTP round-trip per commit), the pack protocol streams all missing objects in a single binary response.

### How it works

1. **Client** sends a `POST /pack/{ledger}` request with `want` (CIDs the client needs, typically the remote head) and `have` (CIDs the client already has, typically the local head). Optionally includes `include_indexes: true` with `want_index_root_id` / `have_index_root_id` to request binary index artifacts.
2. **Server** walks the commit chain from each `want` backward until it reaches a `have`, collecting all missing commits and their referenced txn blobs. When indexes are requested, computes the diff of index artifact CIDs between the want and have index roots.
3. **Server** streams commit + txn objects as binary data frames (oldest-first topological order), followed by a Manifest frame and index artifact data frames when indexes are included.
4. **Client** decodes frames incrementally via a `BytesMut` buffer, verifies integrity of each object, and writes to local CAS.

The CLI uses a **peek-then-ingest** pattern: it reads the Header frame first (via `peek_pack_header`) to inspect `estimated_total_bytes`, then prompts for confirmation on large transfers (>1 GiB) before consuming the rest of the stream via `ingest_pack_stream_with_header`.

### Wire format

```text
[Preamble: FPK1 + version(1)] [Header frame] [Data frames...] [Manifest frame]? [Data frames...]? [End frame]
```

| Frame    | Type byte | Content |
|----------|-----------|---------|
| Header   | `0x00`    | JSON metadata: protocol, capabilities, commit count, index artifact count, `estimated_total_bytes` |
| Data     | `0x01`    | CID binary + raw object bytes (commit, txn blob, or index artifact) |
| Error    | `0x02`    | UTF-8 error message (terminates stream) |
| Manifest | `0x03`    | JSON metadata for phase transitions (e.g. start of index artifact phase) |
| End      | `0xFF`    | End of stream |

### Client-side verification

Each data frame is verified before writing to CAS:
- **Commit blobs** (`FCV2` magic): SHA-256 of full blob via `verify_commit_blob()`
- **All other blobs** (txn, index artifacts, config): Full-bytes SHA-256 via `ContentId::verify()`

Integrity failure is terminal -- the entire ingest is aborted.

### Fallback

When the server does not support the pack endpoint (returns 404, 405, 406, or 501), CLI commands automatically fall back to:
- **Named-remote**: Paginated JSON export via `GET /commits/{ledger}`
- **Origin-based**: CID chain walk via `GET /storage/objects/{cid}`

### Implementation

| Component | Location |
|-----------|----------|
| Wire format (encode/decode), estimation constants | `fluree-db-core/src/pack.rs` |
| Server-side pack generation + index artifact diff | `fluree-db-api/src/pack.rs` |
| Server HTTP endpoint | `fluree-db-server/src/routes/pack.rs` |
| Client-side streaming ingest (`ingest_pack_stream`, `peek_pack_header`, `ingest_pack_stream_with_header`) | `fluree-db-nameservice-sync/src/pack_client.rs` |
| Origin fetcher pack methods | `fluree-db-nameservice-sync/src/origin.rs` |
| CLI pull/clone with index transfer + size confirmation | `fluree-db-cli/src/commands/sync.rs` |
| `set_index_head()` API method | `fluree-db-api/src/commit_transfer.rs` |

For the full design document including graph source packing and protocol evolution, see `STORAGE_AGNOSTIC_COMMITS_AND_SYNC.md` (repo root).


---

# design/content-id-and-contentstore.md

# ContentId and ContentStore

This document describes the content-addressed identity and storage layer introduced by the storage-agnostic commits design. For the full design rationale, see [Storage-agnostic commits and sync](../design/storage-agnostic-commits-and-sync.md).

## Overview

Fluree's storage-agnostic architecture separates **identity** (what something is) from **location** (where its bytes live). Every immutable artifact—commit, transaction payload, index root, index leaf, dictionary blob—is identified by a **ContentId** (a CIDv1 value) and stored/retrieved via a **ContentStore** trait.

> **Identity is a content ID; location is a local configuration detail.**

## ContentId

`ContentId` is a CIDv1 (multiformats) value that encodes three things:

1. **Version**: CIDv1
2. **Multicodec**: identifies the *kind* of the bytes (e.g., Fluree commit, index root)
3. **Multihash**: identifies the *hash function + digest* (SHA-256)

### Multicodec assignments (private-use range)

Fluree uses the multicodec private-use range for type-tagged CIDs:

| Codec value | ContentKind | Description |
|-------------|-------------|-------------|
| `0x300001` | `Commit` | Commit payload |
| `0x300002` | `Txn` | Original transaction payload |
| `0x300003` | `IndexRoot` | Binary index root descriptor |
| `0x300004` | `IndexBranch` | Index branch manifest |
| `0x300005` | `IndexLeaf` | Index leaf file |
| `0x300006` | `DictBlob` | Dictionary artifact |
| `0x300007` | `DefaultContext` | Default JSON-LD @context |

### String representation

The canonical string form is **base32-lower multibase** (the familiar `bafy…` / `bafk…` prefixes from IPFS/IPLD). This is the form used in JSON APIs, logs, nameservice records, and CLI output.

```text
bafybeigdyr...   (commit CID)
bafkreihdwd...   (index root CID)
```

### Binary representation

The compact binary form (varint version + varint codec + multihash bytes) is used for:
- On-wire pack streams
- Internal caches and indexes
- Embedded references inside commit payloads

### Creating a ContentId

A ContentId is derived by hashing the **canonical bytes** of an artifact with SHA-256, then wrapping the digest as a CIDv1 with the appropriate multicodec:

```rust
use fluree_db_core::content_id::{ContentId, ContentKind};

let bytes: &[u8] = /* canonical commit bytes */;
let cid = ContentId::from_bytes(ContentKind::Commit, bytes);

// String form for JSON/logs
let s = cid.to_string(); // "bafybeig..."

// Parse back
let parsed = ContentId::from_str(&s)?;
assert_eq!(cid, parsed);
```

### ContentId in commit references

Commits reference parents and related artifacts by ContentId only—never by storage addresses:

```json
{
  "t": 42,
  "previous": "bafybeigdyr...commitParent",
  "txn": "bafkreihdwd...txnBlob",
  "index": "bafybeigdyr...indexRoot"
}
```

## ContentKind

`ContentKind` is an enum that maps 1:1 to multicodec values. It serves two purposes:

1. **Embedded in CIDs**: the multicodec tag lets stores, caches, and validators identify what an object is without parsing its bytes.
2. **Routing**: the ContentStore uses `ContentKind` to route objects to the appropriate storage tier (commit store vs index store).

```rust
pub enum ContentKind {
    Commit,
    Txn,
    IndexRoot,
    IndexBranch,
    IndexLeaf,
    DictBlob,
    DefaultContext,
}
```

### Routing by kind (replaces URL parsing)

Previously, storage routing parsed URL path segments (e.g., looking for `"/commit/"` in an address string). With ContentId, routing is explicit:

- `Commit` + `Txn` → commit-tier store(s)
- `IndexRoot` + `IndexBranch` + `IndexLeaf` + `DictBlob` → index-tier store(s)

## ContentStore trait

`ContentStore` provides content-addressed get/put operations keyed by `ContentId`:

```rust
#[async_trait]
pub trait ContentStore: Debug + Send + Sync {
    /// Retrieve bytes by content ID
    async fn get(&self, id: &ContentId) -> Result<Vec<u8>>;

    /// Store bytes, returning the computed ContentId
    async fn put(&self, kind: ContentKind, bytes: &[u8]) -> Result<ContentId>;

    /// Check whether an object exists
    async fn has(&self, id: &ContentId) -> Result<bool>;
}
```

### Relationship to Storage trait

`ContentStore` is the primary abstraction for immutable object access. The `Storage` / `StorageRead` / `ContentAddressedWrite` traits handle address-routed I/O for the underlying storage backends (filesystem, S3, etc.), while `ContentStore` provides the content-addressed layer on top.

### Implementations

- **`MemoryContentStore`**: In-memory `HashMap<ContentId, Vec<u8>>` for testing.
- **`BridgeContentStore`**: Adapter that wraps a `Storage` implementation, mapping ContentIds to physical storage addresses.
- **Filesystem / S3 / IPFS**: Direct implementations that store objects keyed by CID.

### Layered composition

ContentStore implementations can be layered:

```text
Local cache (filesystem)
    ↓ miss
Shared store (S3 / IPFS / shared filesystem)
```

Reads fall through from cache to shared store. Writes go to both (policy-configurable).

## How ContentId flows through the system

### Transaction path

1. Transactor produces commit bytes
2. `ContentId::from_bytes(ContentKind::Commit, &bytes)` computes the CID
3. `content_store.put(Commit, &bytes)` stores the blob
4. Nameservice head is updated: `commit_head_id = cid, commit_t = t`

### Index path

1. Indexer builds binary index, producing root descriptor bytes
2. `ContentId::from_bytes(ContentKind::IndexRoot, &root_bytes)` computes the CID
3. All artifacts (branches, leaves, dicts) are stored via `content_store.put()`
4. Nameservice index head is updated: `index_head_id = cid, index_t = t`

### Query path

1. Query engine reads nameservice to get `index_head_id`
2. `content_store.get(&index_head_id)` fetches the index root
3. Index root references branches/leaves/dicts by their ContentIds
4. Each artifact is fetched via `content_store.get()` (with caching)

### Replication path (clone/pull/push)

1. Client fetches remote nameservice heads (ContentIds + watermarks)
2. Client sends `have[]` / `want[]` roots to server
3. Server walks commit chain and (optionally) index graph to compute missing objects
4. Missing objects streamed as `(ContentId, bytes)` pairs
5. Client stores objects in local ContentStore and advances local nameservice heads

No address rewriting is needed because commits contain no storage addresses.

## Implementation status

- `ContentId` type and `ContentKind` enum: `fluree-db-core/src/content_id.rs`
- `ContentStore` trait + `MemoryContentStore` + bridge adapter: `fluree-db-core/src/storage.rs`
- `Commit` and `CommitRef` use `ContentId` for all references (index pointers are tracked exclusively via nameservice, not embedded in commits)
- Nameservice records use `head_commit_id` / `index_head_id` as ContentId values
- `IndexRoot` (FIR6) references all artifacts by ContentId
- Transact and indexer paths use `ContentStore` for all object I/O

## Related documentation

- [Storage-agnostic commits and sync](../design/storage-agnostic-commits-and-sync.md) — full design rationale
- [Storage traits](storage-traits.md) — existing storage trait hierarchy
- [Index format](index-format.md) — binary index format (IndexRoot / FIR6)
- [Nameservice schema v2](nameservice-schema-v2.md) — nameservice record schema


---

# design/index-format.md

# Binary index format (leaf / leaflet / dictionaries)

This document describes the on-disk / blob-store formats used by Fluree’s binary indexes:
the **branch → leaf → leaflet** hierarchy for fact indexes, and the **dictionary artifacts**
used to translate between IRIs/strings and compact numeric IDs.

The intent is to make the formats easy to reason about (for debugging and tooling) and to
highlight why **leaf files contain multiple leaflets**: it materially improves performance and
cost characteristics on blob/object storage by reducing object counts and request rates while
preserving fine-grained decompression and caching at the leaflet level.

## Overview

A binary index build produces:

- **Per-graph, per-sort-order fact indexes**:
  - a content-addressed **branch manifest** (`FBR3`, file extension `.fbr`)
  - a set of content-addressed **leaf files** (`FLI3`, file extension `.fli`)
  - each leaf contains multiple **leaflets** (compressed blocks with independently compressed regions)
- **Shared dictionary artifacts**:
  - small dictionaries (predicates, graphs, datatypes, languages) embedded in the **index root** (CAS) and/or persisted as flat files in local builds
  - large dictionaries (subjects, strings) stored as **CoW single-level B-tree-like trees**
    (a branch manifest `DTB1` + multiple leaf blobs `DLF1`/`DLR1`)
- **Manifests / roots** that describe how to load the above either from a local directory layout
  or from the content store via `IndexRoot` (FIR6 binary format, CID-based).

Fact indexes exist in up to four sort orders (see `RunSortOrder`):

- **SPOT**: \((g, s, p, o, dt, t, op)\)
- **PSOT**: \((g, p, s, o, dt, t, op)\)
- **POST**: \((g, p, o, dt, s, t, op)\)
- **OPST**: \((g, o, dt, p, s, t, op)\)

## Design goals

- **Blob-store efficiency**: keep object counts low and object sizes in a “healthy” range for
  S3/GCS/Azure-like stores, avoiding “many tiny objects” request overhead.
- **Fast routing**: branch manifest enables binary search routing to the relevant leaf range(s).
- **Cheap decompression**: leaflets are internally structured so query paths can decompress
  *only what they need* (e.g., Region 1 to filter before paying for Region 2).
- **Content-addressed immutability**: leaves/branches/dict leaves can be cached aggressively
  and safely, because their CAS address (or content hash filename) uniquely identifies content.
- **Simple versioning**: each binary artifact begins with a magic + version and can be rejected
  early if incompatible.

## Terminology

- **Leaflet**: a compressed block of rows (default build target: `leaflet_rows = 25_000`).
- **Leaf**: a container of multiple leaflets (default: `leaflets_per_leaf = 10`) plus a directory for
  random access to its leaflets.
- **Branch manifest**: maps key ranges to leaf files; used for routing.
- **Region**: a separately compressed section inside a leaflet.
- **Dictionary tree**: a `DTB1` branch + `DLF1`/`DLR1` leaves for large keyspaces (subjects/strings).
- **ContentId**: a CIDv1 value that uniquely identifies a content-addressed artifact by its hash and type. See [ContentId and ContentStore](content-id-and-contentstore.md).

## Physical layout (local build output)

When built to a filesystem directory (see `IndexBuildConfig`), the output layout is:

```text
index/
  index_manifest_spot.json
  index_manifest_psot.json
  index_manifest_post.json
  index_manifest_opst.json
  graph_<g_id>/
    spot/
      <branch_hash>.fbr
      <leaf_hash_0>.fli
      <leaf_hash_1>.fli
      ...
    psot/
      ...
    post/
      ...
    opst/
      ...
```

The `.fbr` and `.fli` files are content-addressed by **SHA-256 hex** of their bytes (the filename is the hash).
`index_manifest_<order>.json` is a small routing manifest that points to the per-graph directory and branch hash.

### Per-order index manifest (`index_manifest_<order>.json`)

The per-order manifest is JSON and summarizes all graphs for a sort order:

- `total_rows`: total indexed asserted facts for that order
- `max_t`: max transaction `t` in the indexed snapshot
- `graphs[]`: `g_id`, `leaf_count`, `total_rows`, `branch_hash`, and `directory` (relative path)

## Root descriptor (CAS): `IndexRoot` (FIR6)

When publishing an index to nameservice / CAS, the canonical entrypoint is the **FIR6 root**
(`IndexRoot`, binary wire format, magic bytes `FIR6`).

Key properties:

- **CID references** for all artifacts (dicts, branches, leaves).
- Deterministic binary encoding so the root itself is suitable for content hashing to derive its own ContentId.
- Tracks `index_t` (max transaction covered) and `base_t` (earliest time for which Region 3 history is valid).
- Embeds **predicate ID mapping** and **namespace prefix table** inline, so query-time predicate IRI → `p_id` translation does not require fetching a redundant predicate dictionary blob.
- Embeds small dictionaries (**graphs**, **datatypes**, **languages**) inline, so query-time graph/dt/lang resolution does not require fetching tiny dict blobs (important for S3 cold starts).
- **Default graph routing is inline**: leaf entries (first/last key, row count, leaf CID) are embedded directly, avoiding an extra branch fetch for the common single-graph case.
- **Named graph routing uses branch CID pointers**: larger multi-graph setups reference branch manifests by CID.
- Optional binary sections for **stats**, **schema**, **prev_index** (GC chain), **garbage** manifest, and **sketch** (HLL).
- Import-only performance hint: `IndexRoot.lex_sorted_string_ids` indicates whether `StringId` assignment preserves
  lexicographic UTF-8 byte order of strings (true for bulk imports). Query execution can use this to avoid
  materializing simple string values during `ORDER BY` comparisons. This flag must be cleared on the first
  post-import write because incremental dictionary appends break the invariant.
  When the flag is absent (older roots) or false, query execution must assume no lexical ordering.

At a high level the root contains:

- **Inline small dictionaries** (embedded in the binary root):
  - `graph_iris[]` (dict_index → graph IRI; `g_id = dict_index + 1`)
  - `datatype_iris[]` (dt_id → datatype IRI)
  - `language_tags[]` (lang_id-1 → tag string; `lang_id = index + 1`, 0 = "no tag")
- **Dictionary ContentIds** (CAS artifacts):
  - tree blobs: subject/string forward & reverse (`DTB1` branch + `DLF1`/`DLR1` leaves)
  - optional per-predicate numbig arenas
  - optional per-predicate vector arenas (manifest + shards)
- **Default graph routing** (inline leaf entries per sort order)
- **Named graph routing** (branch CIDs per sort order per graph)

## Branch manifest (`FBR3`, `.fbr`)

A branch manifest is a single-level index mapping key ranges to leaf files. It is written per graph
per order and read via binary search to route a lookup/range scan.

### File format

```text
[BranchHeader: 16 bytes]
  magic: "FBR3" (4B)
  version: u8
  _pad: [u8; 3]
  leaf_count: u32
  _reserved: u32
[LeafEntries: leaf_count × 104 bytes]
  first_key: key bytes (44B, little-endian)  [1]
  last_key:  key bytes (44B, little-endian)  [1]
  row_count: u64
  path_offset: u32
  path_len: u16
  _pad: u16
[PathTable]
  Concatenated UTF-8 relative paths (typically "<leaf_hash>.fli")
```

Notes:

- `first_key` and `last_key` use the same 44-byte key wire encoding produced by the index builder (see footnote [1]).
- The path table stores **relative filenames**; on read, paths are resolved against the `.fbr`’s directory.
- In local builds, paths are `<leaf_hash>.fli` to match the content-addressed leaf filenames.

**[1] Key encoding note (internal)**: the 44-byte key is the `RunRecord` wire layout used by the import/index-build
pipeline and stored here only for routing. It is an internal build artifact detail (not a core runtime fact type).

## Leaf file (`FLI3`, `.fli`)

A leaf file groups multiple leaflets into a single blob, and includes a small directory so leaflets can
be accessed without scanning the entire file.

### File format

```text
[LeafHeader: variable size]
  magic: "FLI3" (4B)
  version: u8          (currently 1)
  order: u8
  dt_width: u8         (currently 1; may widen to 2)
  p_width: u8          (2=u16, 4=u32)
  total_rows: u64
  first_key: SortKey (28B)
  last_key:  SortKey (28B)
  [LeafletDirectory: leaflet_count × 40B]    (v2: 28B, lacks first_o_*)
    offset: u64
    compressed_len: u32
    row_count: u32
    first_s_id: u64
    first_p_id: u32
    first_o_kind: u8   (v3+)
    _pad: [u8; 3]      (v3+)
    first_o_key: u64   (v3+)
[LeafletData: concatenated encoded leaflets]
```

The v3 leaflet directory adds `first_o_kind` and `first_o_key` to each entry.
These fields enable **leaflet-boundary skip-decoding**: if two adjacent leaflet
directory entries share the same `(p_id, o_kind, o_key)`, the entire earlier
leaflet is guaranteed to contain only that `(p, o)` combination. Fast-path
COUNT + GROUP BY operators use this property to count rows by `row_count`
without decompressing Region 1, which significantly reduces CPU and I/O for
large predicate scans. v2 leaves (which lack these fields) are still readable
but always require full leaflet decoding.

### `SortKey` (leaf routing key)

`SortKey` is a compact 28-byte key stored in leaf headers:

```text
g_id: u32
s_id: u64
p_id: u32
dt:  u16
o_kind: u8
_pad: u8
o_key: u64
```

`SortKey` exists to reduce leaf header overhead; the branch manifest uses full `RunRecord` boundaries.
It also intentionally omits `t`, `op`, `lang_id`, and `i` — leaf header keys are useful for coarse
metadata and diagnostics, while precise routing is done via the branch’s full `RunRecord` ranges.

### Why “leaf contains leaflets” (blob-store optimization)

If every leaflet were its own object:

- range scans and joins would issue **many more GETs** (request overhead dominates)
- caches would be pressured by **object metadata overhead** and higher churn

By grouping N leaflets into one leaf object:

- we reduce object count and request rate roughly by a factor of N
- we still keep leaflet-sized “micro-partitions” internally for:
  - selective decompression (region-by-region)
  - caching hot leaflets (decoded) independent of unrelated ones
  - future optimizations like ranged reads (leaflet offsets are explicit)

The default build targets (`leaflet_rows = 25_000`, `leaflets_per_leaf = 10`) yield a leaf that is
large enough to amortize object-store overhead but still small enough to cache and move efficiently.

## Leaflet format (compressed block inside a leaf)

A leaflet is a compressed block of rows containing three regions. Each region is independently zstd-compressed.

### Leaflet header (fixed 61 bytes)

```text
row_count: u32
region1_offset: u32
region1_compressed_len: u32
region1_uncompressed_len: u32
region2_offset: u32
region2_compressed_len: u32
region2_uncompressed_len: u32
region3_offset: u32
region3_compressed_len: u32
region3_uncompressed_len: u32
first_s_id: u64
first_p_id: u32
first_o_kind: u8
first_o_key: u64
```

### Regions

- **Region 1 (core columns)**: order-dependent layout optimized for scan/join filtering.
  - includes an RLE-encoded “primary” column (e.g., `s_id` in SPOT)
  - stores the other core columns as dense arrays
  - `p_id` may be stored as `u16` or `u32` depending on dictionary cardinality (`p_width`)
- **Region 2 (metadata columns)**: values needed to reconstruct full flakes (datatype, transaction time, etc.).
  - stored in a layout that supports sparse `lang_id` and `i` without per-row overhead
  - `dt` is stored as `u8` today (`dt_width = 1`) and may widen to `u16`
- **Region 3 (history journal)**: optional operation log to support time-travel semantics from `base_t` onward.
  - stored as a sequence of fixed-size entries in **reverse chronological order** (newest first)

#### Region 1 layouts (uncompressed)

Region 1’s uncompressed bytes vary by sort order:

- **SPOT**: `RLE(s_id:u64)`, `p_id[p_width]`, `o_kind[u8]`, `o_key[u64]`
- **PSOT**: `RLE(p_id:u32)`, `s_id[u64]`, `o_kind[u8]`, `o_key[u64]`
- **POST**: `RLE(p_id:u32)`, `o_kind[u8]`, `o_key[u64]`, `s_id[u64]`
- **OPST**: `RLE(o_key:u64)`, `p_id[p_width]`, `s_id[u64]`
  - OPST leaflets are **type-homogeneous** (segmented by `o_type`), so the per-row object type
    column can be omitted and stored as a constant in the leaflet directory entry. When a leaflet
    contains mixed types in other orders, `o_type` is stored as a per-row column.

RLE encoding is:

```text
run_count: u32
[(key, run_len)] × run_count
```

with `(key=u64, run_len=u32)` or `(key=u32, run_len=u32)` depending on the field.

#### Region 2 layout (uncompressed)

```text
dt: [dt_width bytes] × row_count
t:  [i64] × row_count
lang_bitmap:  u8 × ceil(row_count/8)
lang_values:  u16 × popcount(lang_bitmap)
i_bitmap:     u8 × ceil(row_count/8)
i_values:     i32 × popcount(i_bitmap)
```

- `lang_id` is 0 when absent; otherwise stored in `lang_values` keyed by bitmap position.
- `i` uses `ListIndex::none()` (sentinel) when absent; otherwise stored sparsely.

#### Region 3 layout (uncompressed)

Region 3 is an operation journal stored newest-first:

```text
entry_count: u32
[Region3Entry; entry_count]    // 37 bytes per entry
```

`Region3Entry` wire layout (37 bytes):

```text
s_id: u64
p_id: u32
o_kind: u8
o_key: u64
t_signed: i64      // positive = assert, negative = retract, abs() = t
dt: u16
lang_id: u16
i: i32
```

## Dictionary artifacts

Binary indexes store facts in numeric-ID form. Dictionaries are required to:

- translate query inputs (IRIs, strings) to numeric IDs for scans
- decode numeric IDs back to user-visible values when returning flakes

### Small flat dictionaries (`FRD1`)

Several dictionaries use a simple “count + length-prefixed UTF-8” format:

```text
magic: "FRD1" (4B)
count: u32
for each entry:
  len: u32
  utf8_bytes: [u8; len]
```

This format is used for predicate-like dictionaries. In local builds these are written
as flat files (e.g., `graphs.dict`, `datatypes.dict`, `languages.dict`), but in CAS
publishes (FIR6 root) these small dictionaries are embedded inline in the binary root.

### Legacy forward files + index (`FSI1`) (primarily build-time)

Some build paths still write a forward file (`*.fwd`) plus a separate index (`*.idx`):

`FSI1` index format:

```text
magic: "FSI1" (4B)
count: u32
offsets: [u64] × count
lens:    [u32] × count
```

The forward file itself is a raw concatenation of bytes; access is via `(offset,len)` from the index.

### Large dictionaries as CoW trees (`DTB1` + leaf blobs)

Subjects and strings are large enough that we represent them as single-level CoW trees:

- **Branch**: `DTB1` mapping key ranges to leaf ContentIds
- **Leaves**:
  - forward leaf (`DLF1`): numeric ID → value bytes
  - reverse leaf (`DLR1`): key bytes → numeric ID

#### Dictionary branch (`DTB1`)

```text
[magic: 4B "DTB1"]
[leaf_count: u32]
[offset_table: u32 × leaf_count]  // byte offset of each leaf entry
[leaf entries...]
  entry :=
    [first_key_len: u32] [first_key_bytes]
    [last_key_len: u32]  [last_key_bytes]
    [entry_count: u32]
    [content_id_len: u16]   [content_id_bytes]
```

Keys are treated as raw bytes and compared lexicographically. For forward trees keyed by numeric ID,
the branch uses **8-byte big-endian** keys (so lexical order matches numeric order).

#### Forward dict leaf (`DLF1`)

```text
[magic: 4B "DLF1"]
[entry_count: u32]
[offset_table: u32 × entry_count]
[data section]
  entry := [id: u64 LE] [value_len: u32] [value_bytes]
```

#### Reverse dict leaf (`DLR1`)

```text
[magic: 4B "DLR1"]
[entry_count: u32]
[offset_table: u32 × entry_count]
[data section]
  entry := [key_len: u32] [key_bytes] [id: u64 LE]
```

Subject reverse key format is:

```text
[ns_code: u16 BE][suffix bytes]
```

The `u16` big-endian prefix ensures that lexicographic byte comparisons match logical `(ns_code, suffix)` ordering.

## Endianness and encoding conventions

- Numeric fields in file formats are **little-endian**, unless explicitly stated otherwise.
- Subject reverse keys embed `ns_code` in **big-endian** for byte-sort correctness.
- Compression is currently **zstd** via independent region compression within a leaflet.
- Fact keys are keyed by numeric IDs; ID assignment is provided by dictionary artifacts and/or the root.

## Integrity, caching, and lifecycle

- Leaf and branch filenames (local) are derived from **SHA-256** content hashes; remote references use ContentId (CIDv1).
- Content-addressed artifacts are immutable; caches can key by ContentId.
- `IndexRoot` (FIR6) provides a GC chain (`prev_index`) and an optional garbage manifest pointer to
  support retention-based cleanup of replaced artifacts.

## Versioning notes

- Fact artifacts:
  - branch: magic `FBR3`, version `1`
  - leaf: magic `FLI3`, version `1`
- Dictionary tree artifacts:
  - branch: magic `DTB1`
  - leaves: magic `DLF1` / `DLR1`
- Small dict blobs: magic `FRD1`

When adding new fields, prefer:

- bumping the per-file `version` byte (when present), and
- keeping old readers strict (fail fast on unsupported versions)
  to avoid silent corruption.



---

# design/edge-annotations.md

# Edge annotations — storage internals

User-facing surface and contract live in [Edge annotations (concept doc)](../concepts/edge-annotations.md). This page is for contributors: the on-disk representation, the indexes that back annotation lookup, and the state machine the indexer maintains.

## Two layers, one source of truth

Annotations live in two coordinated places:

1. **Durable `f:reifies*` flakes** — the source of truth. Each annotation lowers to a small fixed bundle of system-controlled flakes about the annotation subject. These flakes ride the same pipeline as every other flake: commits, replay, history, policy, snapshot visibility.
2. **Annotation arena** — a derived secondary index that maps `EdgeKey ↔ annotation subject`. Rebuilt from the durable flakes at index time. Lookups merge the indexed arena with novelty under one visibility pass.

This layering is why a ledger with no annotations creates zero annotation artifacts: the arena is `Option<AnnotationIndexRoot>` on `IndexRoot`, and it's `None` when no annotations have ever been observed.

## `f:reifies*` durable encoding

Seven Fluree-namespaced predicates encode an attachment. The bundle is atomic: every reifies-bundle assert and retract is a complete set of flakes, never split.

```text
_:ann   f:reifiesGraph     <named-graph-iri>      # optional, omitted for the default graph
_:ann   f:reifiesSubject   <s>                    # required, IRI ref
_:ann   f:reifiesPredicate <p>                    # required, IRI ref
_:ann   f:reifiesObject    <o>                    # required, any FlakeValue
_:ann   f:reifiesDatatype  <dt>                   # optional (JSON-LD lowering omits; arena recovers from f:reifiesObject's flake-level dt)
_:ann   f:reifiesLang      "fr"                   # optional, only for langString objects
_:ann   f:reifiesListIndex 3                      # v1: always omitted, deferred
```

Rules:

- **Bundle flakes share a graph.** Every `f:reifies*` flake in one bundle has the same flake-level `g`. The decoder rejects mixed-graph bundles outright (`MixedFlakeGraphs`).
- **`f:reifiesGraph` value must match the bundle's flake-level `g`.** Named-graph edges carry `f:reifiesGraph`; default-graph edges omit it. Disagreement is `GraphMismatch`.
- **Reserved predicates are firewalled at application write surfaces.** `is_reserved_reifies_predicate(sid)` rejects user-authored `f:reifies*` mention in JSON-LD insert/update/upsert/where+delete+insert and SPARQL UPDATE (all clauses). Bulk import is an administrative bootstrap path and may ingest already-lowered `f:reifies*` bundles; import records `has_annotations=true`, and the indexer builds the arena from those durable facts.
- **Replay validation skips malformed bundles.** A partial bundle (missing required slot), a duplicate slot, or a graph-mismatch is skipped at warmup / arena-build with a `tracing::warn!` and counted in `AttachmentNovelty::observed_malformed_bundle_count()`. The annotation's *non*-`f:reifies` metadata facts remain visible as ordinary RDF; only the attachment binding is lost.
- **Cross-commit multi-target is detected at arena build.** A single annotation SID can reify two different edges across *separate commits* (each bundle is individually well-formed, so the per-bundle decode above can't catch it — the bundles are grouped by `(graph, ann_sid, t, op)`). The transaction path rejects this at stage time (see [Stage-time invariants](#stage-time-invariants)), so it only arises from malformed bulk-import data. `build_arenas_from_flakes` / `build_arenas_from_event_pairs` count annotations whose *net live* state resolves to more than one edge and surface the count on `ArenaBuildOutput.multi_target_annotations` with a `tracing::warn!`. The arena is event-sourced, so the affected reverse lookup returns multiple live edges for those annotations; the count is a data-quality signal, not a rejection (import does not validate input).

## EdgeKey

The arena keys on `EdgeKey`, which captures the edge identity from a flake minus the `t/op/m`-bookkeeping that's tracked separately on attachment rows.

```rust
pub struct EdgeKey {
    pub g:      Option<Sid>,   // None = default graph
    pub s:      Sid,
    pub p:      Sid,
    pub o:      FlakeValue,    // refs, literals, langStrings — full canonical value
    pub dt:     Sid,
    pub lang:   Option<String>,
    pub list_i: Option<i32>,   // v1: always None, reserved for list-occurrence annotations
}
```

`EdgeKey::from_reifies_facts(&[Flake]) -> Result<Self, EdgeKeyDecodeError>` decodes a bundle and enforces the rules listed above. `EdgeKey::to_reifies_facts(ann, t, op)` emits the full bundle including `f:reifiesDatatype`; `EdgeKey::to_reifies_facts_jsonld_compatible` omits `f:reifiesDatatype` so the inverse retract bundle matches the assertion shape produced by the JSON-LD lowering (asserting one shape and retracting the other would leave a phantom retract).

## Sidecar arena layout

The arena lives in `fluree-db-binary-index/src/annotation_arena/` and mirrors the dictionary CAS trees:

- **Forward arena** — `EdgeKey -> annotation subjects`. Branches range-route on `EdgeKey`; leaves store sorted `(EdgeKey, ann_sid, t, op)` rows. Used by inline annotation queries and base-edge retract cascade.
- **Reverse arena** — `annotation subject -> EdgeKey`. Branches range-route on `ann_sid`; leaves store sorted `(ann_sid, EdgeKey, t, op)` rows. Used by `@reifies` / `rdf:reifies` annotation-rooted queries and by-id retract.

Both arenas are content-addressed, immutable, range-routable, lazily loaded. Magic numbers `EAFB1`/`EAFL1` and `EARB1`/`EARL1` mark forward/reverse branch/leaf blobs.

Per-edge multiplicity is preserved: the forward arena is a multimap on `EdgeKey`. Two `@annotation` blocks against the same base edge with anonymous subjects produce two distinct rows; two with the same explicit `@id` produce one (idempotent).

## `AnnotationStats` and planner integration

Each arena seal computes per-slot stats consumed by the planner's cardinality estimator:

```text
forward_rows, reverse_rows                  # total event rows across history
distinct_edges, distinct_annotations         # live counts
live_attachment_pairs                        # number of currently-asserted (edge, ann) pairs
distinct_reified_{subjects,predicates,objects}
reifies_graph_rows, distinct_reified_graphs, distinct_graph_anns
reifies_lang_rows, distinct_reified_langs, distinct_lang_anns
```

`stats_view::merge_annotation_stats` overlays these onto the regular `IndexStats.properties` HLL for the seven `f:reifies*` predicates so the planner gets sharp selectivity for `?ann f:reifies* <const>`-shape probes. `f:reifiesDatatype` is intentionally not synthesized from the arena — the arena reconstructs `dt` from the `f:reifiesObject` row's flake-level dt and can't tell whether the on-wire bundle emitted a separate `f:reifiesDatatype` flake. The regular HLL is the source of truth for that slot.

Every field is `#[serde(default)]` so older arena roots that predate any given field deserialize cleanly. A zero NDV means "no information"; the planner falls back to `IndexStats.properties`.

## IndexRoot signals and the sticky bit

`IndexRoot` carries three coordinated signals around annotations:

| Signal | Meaning |
|---|---|
| `annotation_index: Option<AnnotationIndexRoot>` | When `Some`, forward + reverse arena CIDs and stats are loaded lazily. |
| `has_annotations: bool` | Sticky flag: true once any `f:reifies*` SID has appeared in the predicate dictionary. Drives the cascade fast-path's zero-cost gate. |
| `had_annotation_arena: bool` | **Sticky bit, never cleared.** Lives in the FIR6 extended-flags byte (low byte of the historically-zero `pad(2)` header field). |

The truth table for the indexed-arena guarantee:

| `has_annotations` | `annotation_index` | Meaning |
|---|---|---|
| `false` | `None` | Hard guarantee: zero attachments. Cascade and reads short-circuit. |
| `true` | `Some(_)` | Builder ran. Forward/reverse arenas are authoritative for `t ≤ max_t`; novelty supplies the tail. |
| `true` | `None` | Pre-builder or defensive-drop transitional state. Snapshot may carry `f:reifies*` flakes but no arena yet — readers fall back to scan, cascade still runs. |
| `false` | `Some(_)` | **Invariant violation.** The encoder coerces `has_annotations=true` whenever an arena is present, so this state never reaches the wire. The `encode()` `debug_assert!` catches in-memory regressions in dev/CI. |

### Sticky-bit state machine

`had_annotation_arena`'s load-bearing role is **"base-index bootstrap is not allowed"**. The provider's one-time PSOT scan of `f:reifies*` flakes (`ApiAttachmentEventsProvider::scan_base_index_for_attachment_events`) is correct only when the base index is the complete history. Any indexer pass that's already touched the annotation history owns it from then on; the provider must not later reconstruct a live-only `Authoritative` arena from such a root.

The bit is set in three places, all flipping it to `true` and never clearing:

- `IncrementalRootBuilder::build()` — coerces on any incremental root with `has_annotations=true`, regardless of whether this pass sealed an arena (covers no-provider passes that left `annotation_index=None`).
- `encode_and_write_root_v6` — coerces in the full-rebuild root-assembly path, same condition.
- Decoder coercion — `IndexRoot::decode` and `LedgerSnapshot::from_root_bytes` both coerce the bit from `annotation_index.is_some()` when the wire byte is zero, covering legacy pre-extended-flags roots.

Only fresh bulk-import roots leave it `false`. Bulk import constructs `IndexRoot` directly in `fluree-db-api/src/import.rs`, bypassing both root-assembly paths. That makes `has_annotations=true, annotation_index=None, had_annotation_arena=false` the unique bootstrap-eligible state.

### Provider 4-gate eligibility

`ApiAttachmentEventsProvider::attachment_events` boots a one-time base-index scan only when **all four** hold:

1. `try_running_attachment_events` returns an empty event set (the running overlay carries no `f:reifies*` events for this ledger).
2. `snapshot.has_annotations` is true.
3. `snapshot.annotation_index.is_none()` (no arena currently sealed).
4. `!snapshot.had_annotation_arena` (the indexer has never touched this ledger's annotation history).

When eligible, the provider walks the base index via per-predicate PSOT scans (PSOT-direction sidesteps a SPOT quirk where constant blank-node subjects don't return rows reliably across all backends), groups results by annotation SID, decodes each bundle into an `EdgeKey`, and returns `AttachmentEventCoverage::Authoritative`. The indexer then seals an authoritative arena from the events. Any subsequent indexer pass coerces `had_annotation_arena = true`, so the same ledger is no longer bootstrap-eligible — defensive drops carry the bit forward and stay in the M2a scan-fallback state until the next provider-backed reindex re-seals.

PSOT scan errors are logged with `tracing::warn!` and surface as `Option::None` (no coverage); the indexer then skips arena seal this pass instead of silently treating the failure as "no annotations."

## Reads merge arena + novelty under one visibility pass

`AnnotationArenaReader` exposes two merged lookups:

```rust
fn current_annotations_merged(edge: &EdgeKey, novelty_events: &[(Sid, i64, bool)], as_of_t: i64) -> Vec<Sid>
fn current_targets_merged(ann: &Sid, novelty_events: &[(EdgeKey, i64, bool)], as_of_t: i64) -> Vec<EdgeKey>
```

Callers (the hydration injector, the cascade pass, the planner-flattened `f:reifies*` triples) feed the arena's `collect_*_events` output into a single `(t, op)` visibility pass that resolves to currently-live state. Arena and novelty events can interleave arbitrarily — the merge applies one latest-wins pass over the union, so an arena `op = true` followed by a novelty `op = false` (or vice versa) resolves correctly without the caller doing any pre-merging.

The scan fallback (used when no arena is sealed) runs `db.range(POST, f:reifiesSubject, edge.s)` to find candidate annotation SIDs, then walks each candidate's bundle via `db.range(SPOT, s=ann_sid)` for structural decode. Bundle decode at the SPOT step bypasses view policy: the `f:reifies*` flakes are system-controlled discriminators, not user data, and policy-filtering them at decode would let an incidental policy that hides FLUREE_DB-namespace predicates collapse the bundle and drop the annotation entirely. The annotation **body** continues through the policy-filtered `format_subject` path, so user-data visibility is unchanged.

## Stage-time invariants

- **Cascade dedup is graph-aware.** Cascade retracts are deduped against existing retracts in the staged flake set via an explicit key tuple `(Option<Sid>, Sid, Sid, FlakeValue, Sid, Option<FlakeMeta>)` — `Flake`'s `Eq`/`Hash` ignore `g`, so a plain `HashSet<Flake>` would collapse two retracts targeting the same `(s, p, o, dt, m)` in different named graphs.
- **Single-target invariant (Fluree v1 storage contract).** RDF 1.2 allows one reifier to be related to several propositions; this implementation deliberately stores one live edge attachment per annotation SID so retract cascade, by-id cleanup, and the forward/reverse arena stay unambiguous. At stage time, for every annotation SID this transaction asserts a `f:reifies*` flake for, the *net* asserted bundle — current snapshot/novelty state, minus this txn's retracts, plus its asserts, deduped as an RDF set keyed on `(g, s, p, o, dt)` — is decoded via `EdgeKey::from_reifies_facts`; a malformed or multi-target result (e.g. a duplicate subject/predicate/object slot) errors with `InvariantViolation`. Validating the *net* bundle rather than counting this txn's `f:reifiesSubject` flakes catches two cases a count misses: re-pointing an `@id` already attached to a different edge in a *prior* transaction (no retract in this txn), and same-subject/different-slot multiplicity within one txn (the subject slot dedupes while predicate/object diverge). The legitimate re-point pattern — retract the old attachment + assert the new — passes because the net bundle resolves to one edge; an idempotent re-assert passes because the set-keyed net bundle is unchanged.
- **Empty `@annotation: {}` is RDF-mode no-op.** No annotation subject is minted; no attachment row is written. In LPG mode (`opts.lpgEdgeLifecycle: true`), an empty block mints a fresh property-less annotation subject so the relationship retains identity.

## Cascade and lifecycle modes

The cascade gate is mode-independent: when both `snapshot.has_annotations` and `novelty.attachments.has_annotations()` are false, base-edge retracts skip the cascade lookup entirely (zero-cost gate, non-annotation ledgers pay nothing).

When annotations are possible:

- **Plain edge retract.** Look up annotations attached to the retracted edge via merged arena + novelty. Retract the complete `f:reifies*` bundle for each. Retract anonymous annotation body metadata always (RDF default). Retract explicit-IRI body metadata only when `opts.lpgEdgeLifecycle: true`.
- **By-id retract.** A delete-clause pre-pass (`lower_delete_annotation_blocks`) rewrites `@annotation: { @id ex:foo }` into explicit `f:reifies*` retract templates before the standard lowering. Threads named-graph context (`f:reifiesGraph` + node-level `@graph` selector) so retracts cancel the correct asserted flake identity.
- **By-selector retract.** A WHERE-bound retract: the pre-pass mints a fresh internal variable, synthesizes a `f:reifies*` WHERE pattern that constrains the variable to live annotations matching the selector body, and emits a by-variable delete template. The mint counter seeds past any user-visible `?_fluree_del_ann_N` occurrence to avoid collision.

## GC reachability

Annotation forward and reverse branch CIDs are returned by `IndexRoot::all_cas_ids`, so they participate in the same root-chain reachability model as fact indexes and dictionary trees. When a new index root supersedes an old one:

- New annotation CIDs are reachable from the new root.
- Old annotation CIDs become garbage only when no retained root references them.
- Leaf-level CIDs behind a branch are walked during the GC-diff pass; `drop.rs` and `gc/collector.rs` both call into the expanded-CAS expansion helpers so a strict GC pass never deletes a still-reachable leaf.

## Query IR expansion

`Pattern::EdgeAnnotation` and `Pattern::AnnotationTarget` are not executed as dedicated operators. `expand_edge_annotation_patterns` (`fluree-db-query/src/execute/where_plan.rs`) flattens them into a base-edge `Pattern::Triple` plus three required `f:reifies*` triples (`f:reifiesSubject` / `f:reifiesPredicate` / `f:reifiesObject`) plus body patterns before the join planner runs. The standard scan/join/dedup machinery handles the rest, and the planner's `reorder_patterns` picks the cheaper direction (edge-first vs annotation-first) based on the merged `AnnotationStats` selectivity.

The expansion does **not** emit an `f:reifiesGraph` constraint triple. Graph correlation is structural instead: each chain is scoped to one source at a time (the `Pattern::DefaultGraphSource` wrapper below, or an enclosing `Pattern::Graph`), so every `f:reifies*` lookup resolves against the same flake-level `g` per iteration — matching how `EdgeKey` derives graph identity. **Consequence:** the strict bundle-wellformedness rules under [`f:reifies*` durable encoding](#freifies-durable-encoding) above (required-slot, duplicate-slot, and `f:reifiesGraph`-vs-`g` `GraphMismatch` rejection) are guaranteed only on the **decode paths** — warmup / arena-build / hydration / cascade. Query-side annotation matching keys off the subject/predicate/object slots within the scoped graph; a malformed bundle introduced by bulk import (e.g. a named-graph bundle missing `f:reifiesGraph`) that decode would skip can still satisfy an `@annotation` / `rdf:reifies` query. Well-formed Fluree writes never produce such bundles — the JSON-LD writer emits `f:reifiesGraph` for named-graph edges and the SPARQL UPDATE surface is default-graph only — so this boundary matters only for hand-authored or externally-imported `f:reifies*` data.

Multi-source default-graph datasets (`from: [g1, g2]`) wrap each expanded chain in `Pattern::DefaultGraphSource` so the base-edge match correlates per source — otherwise a base-edge from `g1` would cross-join with annotations from `g2`. `collect_var_stats` walks into `DefaultGraphSource` so bridge variables (e.g. `?ann` shared between the wrapper's inner chain and a sibling external triple) are correctly counted as join vars.

## See also

- [Edge annotations (concept doc)](../concepts/edge-annotations.md) — the user-facing surface.
- [Index format](index-format.md) — fact indexes, dictionary trees, FIR6 root layout.
- Rustdoc on `fluree_db_core::edge::EdgeKey`, `fluree_db_core::annotation_index::AnnotationIndexRoot`, `fluree_db_novelty::attachments::AttachmentNovelty`, and `fluree_db_binary_index::annotation_arena::AnnotationArenaReader` for type-level contracts.


---

# design/spatial-index.md

# Spatial Index Design

This document describes the design of Fluree's geospatial indexing system, covering both the inline GeoPoint encoding for POINT geometries and the S2 cell-based spatial index for complex geometries.

## Overview

Fluree provides two complementary approaches to geospatial indexing:

1. **Inline GeoPoint**: POINT geometries are encoded directly in the binary index as packed 60-bit lat/lng values, enabling efficient latitude-band scans for proximity queries.

2. **S2 Spatial Index**: Complex geometries (polygons, linestrings, etc.) are indexed using Google's S2 cell system, which maps spherical geometries to hierarchical cell IDs.

This dual approach optimizes for the most common use cases while supporting the full range of OGC GeoSPARQL operations.

## Architecture

```
                        ┌─────────────────────────────────────────┐
                        │             Query Engine                │
                        └─────────────────────────────────────────┘
                                        │
                    ┌───────────────────┼───────────────────┐
                    ▼                   ▼                   ▼
            ┌───────────────┐   ┌───────────────┐   ┌───────────────┐
            │ GeoSearchOp   │   │ S2SearchOp    │   │ RangeProvider │
            │ (POINT nearby)│   │ (within/etc)  │   │ (standard)    │
            └───────────────┘   └───────────────┘   └───────────────┘
                    │                   │                   │
                    ▼                   ▼                   ▼
            ┌───────────────┐   ┌───────────────┐   ┌───────────────┐
            │ POST Index    │   │ S2 Sidecar    │   │ Binary Index  │
            │ (lat-band)    │   │ (cell index)  │   │ (SPOT/PSOT)   │
            └───────────────┘   └───────────────┘   └───────────────┘
```

## Part 1: Inline GeoPoint

### Encoding

POINT geometries are stored using a 60-bit packed encoding:

```
┌──────────────────────────────────────────────────────────────┐
│  Upper 30 bits: Latitude    │  Lower 30 bits: Longitude     │
│  [-90, 90] → [0, 2³⁰-1]     │  [-180, 180] → [0, 2³⁰-1]     │
└──────────────────────────────────────────────────────────────┘
```

This provides:
- **~0.3mm precision** at the equator (30 bits = 1 billion divisions per range)
- **8 bytes total** vs ~25+ bytes for WKT string storage
- **Ordered by latitude** for efficient latitude-band range scans

### Latitude-Band Queries

For proximity queries, the GeoSearchOperator:

1. Converts the query radius to a latitude delta: `δ_lat = radius / EARTH_RADIUS`
2. Computes latitude bounds: `[center_lat - δ, center_lat + δ]`
3. Scans the POST index for the latitude band
4. Post-filters with exact haversine distance

**False Positive Rate**: 22-70% depending on latitude and query radius (eliminated by post-filter).

### Antimeridian Handling

Queries crossing the antimeridian (±180°) are split into two scans:
- `[min_lng, 180]` in the eastern hemisphere
- `[-180, max_lng]` in the western hemisphere

Results are merged and deduplicated.

## Part 2: S2 Spatial Index

### Why S2?

The S2 cell system provides several properties ideal for spatial indexing:

1. **Hierarchical**: Cells at level N contain 4 children at level N+1
2. **Hilbert-curve ordering**: Nearby cells have nearby IDs (good for range scans)
3. **Equal-area**: Cells at the same level have roughly equal area
4. **No gaps/overlaps**: Cells tile the sphere exactly

### Cell Levels

| Level | Approx. Area | Use Case |
|-------|--------------|----------|
| 0 | 85M km² | Hemisphere |
| 8 | 300k km² | Countries |
| 12 | 20k km² | Cities |
| 16 | 1.3 km² | Neighborhoods |
| 20 | 80 m² | Buildings |
| 30 | 0.7 cm² | Sub-centimeter |

Default configuration: `min_level=4`, `max_level=16`, `max_cells=8`

### Index Structure

The spatial index stores entries sorted by `(cell_id, subject_id, t DESC, op ASC)`:

```rust
struct CellEntry {
    cell_id: u64,       // S2 cell ID (Hilbert order)
    subject_id: u64,    // Subject that owns this geometry
    geo_handle: u32,    // Handle into geometry arena
    t: i64,             // Transaction time
    op: u8,             // 1 = assert, 0 = retract
}
```

This sort order enables:
- Efficient range scans for S2 coverings
- Time-travel via replay (max-t wins at each `(cell_id, subject_id)`)
- Global deduplication across cells (same subject may appear in multiple cells)

### Geometry Arena

WKT strings are stored in a deduplicated geometry arena:

```rust
struct ArenaEntry {
    wkt: Vec<u8>,           // Original WKT bytes
    metadata: GeometryMetadata,
}

struct GeometryMetadata {
    geom_type: GeometryType,
    bbox: Option<BBox>,     // For bbox prefiltering
    centroid: Option<(f64, f64)>,
}
```

The arena:
- Deduplicates identical WKT strings
- Stores precomputed bounding boxes for fast rejection
- Uses WKT as source of truth (no normalization)

### Query Pipeline

```
Query Geometry
     │
     ▼
┌─────────────────┐
│ Generate S2     │  ~10 µs per geometry
│ Covering        │
└─────────────────┘
     │
     ▼
┌─────────────────┐
│ Merge Cell      │  Combine adjacent ranges
│ Ranges          │
└─────────────────┘
     │
     ▼
┌─────────────────┐
│ Scan Index      │  Binary search + sequential scan
│ + Novelty       │  Merge with uncommitted changes
└─────────────────┘
     │
     ▼
┌─────────────────┐
│ Replay at t     │  Time-travel filtering
└─────────────────┘
     │
     ▼
┌─────────────────┐
│ Global Dedup    │  Same subject from multiple cells
└─────────────────┘
     │
     ▼
┌─────────────────┐
│ BBox Prefilter  │  Reject by bounding box
└─────────────────┘
     │
     ▼
┌─────────────────┐
│ Exact Predicate │  within/contains/intersects
└─────────────────┘
     │
     ▼
Results
```

### Time-Travel Semantics

The index supports Fluree's time-travel model:

1. **Replay**: For each `(cell_id, subject_id)` group, take the entry with maximum `t ≤ to_t`. If that entry is a retraction (`op=0`), the geometry is not visible at `to_t`.

2. **Same-t disambiguation**: When multiple operations share the same `(cell_id, subject_id, t)`, `op` is used as a deterministic tie-break in the sort key (`op ASC`).

3. **Novelty overlay**: Uncommitted changes are merged with the snapshot index using a merge-sorted iterator, with novelty entries winning on exact ties by merge contract.

## Benchmarks

This document focuses on the design. Benchmark results depend on dataset shape, S2 covering configuration, hardware, and query selectivity.

For v1, it is recommended to maintain a benchmark suite that measures:
- Index build throughput and artifact sizes (leaflets, arena, manifest)
- Query latency for `within` / `contains` / `intersects` / `nearby` across a range of query sizes
- Overlay-on vs overlay-off query performance (epoch caching effectiveness)

The benchmark suite lives in `fluree-db-spatial/benches/spatial_bench.rs` and is runnable with:

```bash
cargo bench -p fluree-db-spatial
```

## Design Decisions

### 1. WKT as Source of Truth

**Decision**: Store WKT bytes directly without normalization.

**Rationale**:
- Simple deduplication via string hashing
- No risk of altering geometry semantics
- Normalization (winding order, dateline wrapping) can be added later if needed

### 2. Separate Index for Complex Geometries

**Decision**: Use a sidecar S2 index rather than extending the main binary index.

**Rationale**:
- Complex geometries generate multiple cell entries (8+ per geometry)
- Spatial queries have different access patterns than standard SPOT/POST
- Isolation allows independent optimization and evolution

### 3. Global Dedup Across Cells

**Decision**: Deduplicate after scanning all covering ranges, not per-cell.

**Rationale**:
- Polygon coverings often return the same subject from multiple cells
- Per-cell dedup would miss cross-cell duplicates
- HashMap-based dedup is fast enough (O(n) amortized)

### 4. BBox Prefiltering

**Decision**: Store precomputed bounding boxes and filter before exact tests.

**Rationale**:
- BBox intersection is O(1) vs O(n) for polygon intersection
- Eliminates 40-60% of candidates in typical queries
- Minimal storage overhead (4 floats per geometry)

### 5. Time-Travel via Replay

**Decision**: Store all versions and replay at query time.

**Rationale**:
- Consistent with Fluree's immutable ledger model
- No special handling for historical queries
- Novelty overlay uses same replay logic

### 6. Graph-Scoped Indexes

**Decision**: Spatial indexes are keyed by `(graph_id, predicate_iri)`.

**Rationale**:
- Named graphs may contain different geometry schemas
- Allows graph-specific index configuration
- Fallback to default graph (g_id=0) for convenience

**Implementation**: Each named graph has a numeric `g_id` (0 for default, 1+ for named graphs). The `GraphRef` struct carries this `g_id`, and `GraphOperator` updates `ctx.binary_g_id` when entering a GRAPH pattern. The S2 search operator uses `ctx.binary_g_id` together with the predicate IRI to construct the provider map key (format: `"g{g_id}:{predicate_iri}"`).

**Cross-Ledger Datasets**: The `g_id` is only meaningful relative to the active `binary_store`. Each ledger has its own binary index with its own g_id namespace. In cross-ledger datasets, switching to a different ledger's graph also requires switching the `binary_store` context.

## Limitations

### Known Limitations

1. **Time-travel floor**: Queries for `t < index.base_t` are rejected. The index must be rebuilt to cover earlier times.

2. **Antimeridian polygons**: Polygons crossing the antimeridian (±180°) produce large coverings due to bbox spanning nearly 360° of longitude. A test case with a small antimeridian-crossing polygon generated 226 cells (vs typical 8). Query performance degrades proportionally.

3. **Polar regions**: Polar regions are actually handled efficiently by S2 (4 cells for 85°N arctic polygon), but the cell geometry differs significantly from mid-latitudes.

### Future Work

1. **R-tree for spatial joins**: Ephemeral R-tree construction for joining two geometry columns.

2. **Streaming query results**: Currently all candidates are collected before filtering; streaming would reduce memory for large result sets.

3. **Remote spatial service**: The `fluree-db-spatial` crate supports a provider trait for future remote deployment.

## Related Documentation

- [Geospatial](../indexing-and-search/geospatial.md) - User-facing geospatial documentation
- [Index Format](index-format.md) - Binary index format details
- [Background Indexing](../indexing-and-search/background-indexing.md) - Index lifecycle


---

# design/namespace-allocation.md

# Namespace allocation and fallback modes

Fluree encodes IRIs as compact **SIDs**: a `(ns_code, local)` pair where:

- `ns_code` is a `u16` namespace code that identifies an IRI prefix
- `local` is the remaining suffix (bytes) after removing the matched prefix

The database maintains a **namespace table** (`LedgerSnapshot.namespace_codes`: `ns_code -> prefix string`).
That table is embedded in the published index root and is loaded whenever a `LedgerSnapshot` is opened.

This document describes how Fluree chooses a namespace prefix for an IRI, and how it mitigates
datasets that would otherwise allocate an excessive number of distinct namespace prefixes.

## Goals

- **Keep declared namespaces intact**: if a dataset declares `@prefix foo: <...>`, we want IRIs in
  that namespace to use that exact prefix, not a derived/split prefix.
- **Stable behavior across writes**: after importing an “outlier” dataset, subsequent transactions
  should continue using the same fallback rules for *previously unseen* IRIs (e.g. new hosts),
  avoiding regression back to finer-grained splitting.
- **Contain namespace explosion**: avoid allocating one namespace code per highly-specific leaf
  (e.g. splitting on the last `/` for IRIs whose paths are effectively unique).

## Core rule: declared-prefix trie match wins

Namespace resolution is **trie-first**:

1. Load all known prefixes (predefined defaults + DB namespace table) into a byte-level trie.
2. For each IRI, perform a **longest-prefix match**.
3. If a match is found, emit `Sid(ns_code, iri[prefix_len..])` and do **not** run fallback logic.

Only IRIs with **no** matching prefix fall through to the fallback splitter.

Implementation: `fluree-db-transact/src/namespace.rs`

- `NamespaceRegistry::sid_for_iri` (transactions, serial paths)
- `SharedNamespaceAllocator::sid_for_iri` (parallel bulk import)

## Fallback split modes (only for unmatched IRIs)

Fluree uses a small set of fallback “splitters” that derive `(prefix, local)` for IRIs that do not
match any known prefix.

The active fallback behavior is represented by `NsFallbackMode`:

- `LastSlashOrHash` (default): split on the last `/` or `#` (prefix is inclusive)
- `CoarseHeuristic` (outlier mitigation):
  - http(s): usually `scheme://host/<seg1>/`
  - special-case: DBLP-style `.../pid/<digits>/` buckets may keep 2 segments
  - non-http(s) with `:` but no `/` or `#`: split at the **2nd** `:` when present (e.g. `urn:isbn:`),
    else the 1st `:`
- `HostOnly` (“fallback to the fallback”):
  - http(s): `scheme://host/`
  - non-http(s) with `:` but no `/` or `#`: split at the **1st** `:`
  - else: last-slash-or-hash

Implementation: `fluree-db-transact/src/namespace.rs`

## Bulk import: streaming preflight + dynamic mitigation

For large Turtle streaming imports, Fluree attempts to detect “namespace explosion” early without
an extra I/O pass:

1. `StreamingTurtleReader` samples bounded byte windows within the first chunk region and counts
   distinct prefixes under `LastSlashOrHash`.
2. If the sample exceeds a budget (`NS_PREFLIGHT_BUDGET`, currently 255), the reader publishes a
   preflight result recommending mitigation.
3. The import forwarder enables `CoarseHeuristic` on the shared allocator **before parsing begins**
   (so the earliest allocations are already coarse).
4. If allocations under `CoarseHeuristic` still grow beyond the u8-ish threshold (>255), the shared
   allocator switches to `HostOnly` so new, unseen hosts do not allocate deeper-than-host namespaces.

Implementation:

- Preflight detector: `fluree-graph-turtle/src/splitter.rs`
- Policy application: `fluree-db-api/src/import.rs`
- Runtime switch: `SharedNamespaceAllocator::get_or_allocate` in `fluree-db-transact/src/namespace.rs`

## Transactions after import: preventing regression for unseen IRIs

Bulk import can upgrade fallback behavior at runtime (shared allocator). For subsequent **normal
transactions**, we also need “outlier mode” to persist so new IRIs do not regress to `LastSlashOrHash`.

Fluree derives this from the DB’s namespace table at open time:

- When a `LedgerSnapshot` is opened, `NamespaceRegistry::from_db(db)` loads `db.namespace_codes`.
- If the DB has already allocated namespace codes beyond the u8-ish threshold (>255), the registry
  sets its fallback mode to `HostOnly`.

That means a new IRI like:

`http://some-unseen-host/blah/123/456`

will allocate (if needed) at:

`http://some-unseen-host/`

instead of falling back to a finer last-slash split.

Implementation: `NamespaceRegistry::from_db` and `NamespaceRegistry::sid_for_iri` in
`fluree-db-transact/src/namespace.rs`

## Notes and trade-offs

- `HostOnly` can still result in many namespaces if a dataset genuinely contains many distinct hosts
  (one per host), but it prevents deeper fragmentation that is common in path-heavy IRIs.
- The `OVERFLOW` namespace code is a sentinel used when `u16` codes are exhausted; it is not a
  fallback mode. Overflow SIDs store the **full IRI** as the SID name.



---

# design/ontology-imports.md

# Ontology imports (`f:schemaSource` + `owl:imports`)

Reasoning in Fluree needs to see a ledger's **ontology** — class and
property hierarchies, OWL axioms — even when those triples don't live in
the same graph as the instance data being queried. This document describes
how that binding is configured, resolved, and plumbed into the reasoning
pipeline.

Topics:

- Config-layer contract (`f:schemaSource`, `f:followOwlImports`,
  `f:ontologyImportMap`).
- Resolution algorithm for the `owl:imports` closure.
- `SchemaBundleOverlay` — how the resolved closure is presented to the
  reasoner without changing reasoner internals.
- Caching, error semantics, and the schema-triple whitelist.

Related docs:

- [Query execution and overlay merge](query-execution.md)
- [Reasoning and inference](../concepts/reasoning.md)

## Configuration

Reasoning config is declared in the ledger's config graph (`g_id=2`), on the
`f:LedgerConfig` resource's `f:reasoningDefaults`. Three fields drive
ontology resolution:

```turtle
@prefix f:    <https://ns.flur.ee/db#> .
@prefix rdf:  <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix owl:  <http://www.w3.org/2002/07/owl#> .

GRAPH <urn:fluree:myapp:main#config> {
  <urn:myapp:config> a f:LedgerConfig ;
    f:reasoningDefaults <urn:myapp:config:reasoning> .

  <urn:myapp:config:reasoning>
    f:reasoningModes ( "rdfs" "owl2-rl" ) ;
    f:schemaSource <urn:myapp:config:schema-ref> ;
    f:followOwlImports true ;
    f:ontologyImportMap <urn:myapp:config:bfo-binding> .

  <urn:myapp:config:schema-ref> a f:GraphRef ;
    f:graphSource <urn:myapp:config:schema-source> .
  <urn:myapp:config:schema-source>
    f:graphSelector <http://example.org/ontology/core> .

  <urn:myapp:config:bfo-binding>
    f:ontologyIri <http://purl.obolibrary.org/obo/bfo.owl> ;
    f:graphRef   <urn:myapp:config:bfo-ref> .
  <urn:myapp:config:bfo-ref> a f:GraphRef ;
    f:graphSource <urn:myapp:config:bfo-source> .
  <urn:myapp:config:bfo-source>
    f:graphSelector <http://example.org/ontology/local/bfo> .
}
```

Field reference:

| Field                    | Type                            | Meaning |
|--------------------------|---------------------------------|---------|
| `f:schemaSource`         | `f:GraphRef`                    | Starting graph for schema extraction. When absent, reasoning uses the default graph directly. |
| `f:followOwlImports`     | `xsd:boolean`                   | When `true`, resolve the transitive closure of `owl:imports` triples starting from `f:schemaSource`. When absent or `false`, the bundle contains only the starting graph. |
| `f:ontologyImportMap`    | list of `OntologyImportBinding` | Mapping table from external ontology IRIs to local graphs. Consulted when an `owl:imports` IRI doesn't match a named graph in the current ledger. |

An `OntologyImportBinding` has two fields:

- `f:ontologyIri` — the IRI that appears in `owl:imports` statements.
- `f:graphRef` — a nested `f:GraphRef` identifying the local graph.

The `GraphRef` shape supported for `f:schemaSource` and
`f:ontologyImportMap.graphRef` is the same-ledger shape:
`f:graphSelector` naming a local named graph, `f:defaultGraph`, or a
registered graph IRI. References are resolved at the query's effective
`to_t` — every named graph in a Fluree ledger shares the ledger's
monotonic `t`, so the entire closure is consistent at a single point in
time without per-import bookkeeping.

## Resolution algorithm

For each `owl:imports <X>` triple discovered while walking the closure, the
resolver (`fluree_db_api::ontology_imports::resolve_schema_bundle`) applies
this order:

1. **Named-graph match** — if `<X>` is registered as a graph IRI in the
   current ledger's [`GraphRegistry`], resolve to that `GraphId`.
2. **Mapping-table fallback** — if `<X>` appears in `f:ontologyImportMap`,
   resolve via the bound `GraphSourceRef`.
3. **Strict error** — otherwise, fail the query with
   `ApiError::OntologyImport`. There is no silent skip.

The walk is BFS, deduplicated by resolved `GraphId`, and cycle-safe by
construction (we only push unseen IDs onto the queue). The result is a
`ResolvedSchemaBundle { ledger_id, to_t, sources: Vec<GraphId> }`.

### System graphs are off-limits

Imports resolving to `CONFIG_GRAPH_ID` (g_id=2) or `TXN_META_GRAPH_ID`
(g_id=1) are rejected — those graphs are structurally reserved and would
leak framework triples into reasoning. The guard sits in the single
`resolve_local_graph_source` chokepoint, so **every** resolution path
(direct graph-IRI match, `f:ontologyImportMap` entry, `f:schemaSource`
selector) is covered.

### `owl:imports` discovery is subject-wildcarded

Every `?s owl:imports ?o` triple in a schema graph is treated as
authoritative, regardless of whether `?s` is typed `owl:Ontology`. This is
broader than strict OWL 2 (which restricts `owl:imports` to the ontology
header) and matches real-world OWL inputs that rely on file-level
provenance. The resolution layer's strictness still applies: a stray
`owl:imports` triple that doesn't map to a local graph fails the query
rather than silently expanding the closure.

### Reasoning-disabled queries don't trigger resolution

Queries that opt out of reasoning (`"reasoning": "none"`) skip bundle
resolution entirely — a broken ontology import in the ledger's config
shouldn't produce errors for a non-reasoning workload. The short-circuit
lives in `attach_schema_bundle` (both the single-view and dataset paths).

## Projecting the bundle into reasoning

RDFS and OWL extraction code reads schema triples out of the default graph
(`g_id=0`). The resolver feeds that code via a
`SchemaBundleOverlay` that
**projects** whitelisted triples from every bundle source onto `g_id=0`,
so the reasoner sees the full closure without being aware of it.

The projection happens in two phases:

1. **Materialize.** `build_schema_bundle_flakes` runs targeted reads against
   every source graph — one PSOT scan per schema predicate and one OPST
   scan per schema class — and collects the matching flakes into per-index
   sorted arrays (SPOT / PSOT / POST / OPST). Reads go through the normal
   `range_with_overlay` path, so both committed index data and novelty are
   visible.
2. **Overlay.** `SchemaBundleOverlay::new(base_overlay, flakes)` wraps the
   query's base overlay. For `g_id != 0` it delegates straight to the
   base. For `g_id == 0` it emits a linear merge of base flakes and
   bundle flakes in index order.

The reasoner sees: base default-graph flakes ∪ projected schema flakes,
presented as a single ordered stream at `g_id=0`. Reasoner code is
unmodified.

### Schema-triple whitelist

Only the following predicates are eligible for projection:

- **RDFS:** `rdfs:subClassOf`, `rdfs:subPropertyOf`, `rdfs:domain`, `rdfs:range`
- **OWL:** `owl:inverseOf`, `owl:equivalentClass`, `owl:equivalentProperty`,
  `owl:sameAs`, `owl:imports`

And `rdf:type` triples are projected **only when the object is** one of:
`owl:Class`, `owl:ObjectProperty`, `owl:DatatypeProperty`,
`owl:SymmetricProperty`, `owl:TransitiveProperty`, `owl:FunctionalProperty`,
`owl:InverseFunctionalProperty`, `owl:Ontology`, `rdf:Property`.

Anything else in an import graph — in particular, instance data —
**does not surface** in the reasoner's view. See
`fluree_db_core::{is_schema_predicate, is_schema_class}` for the canonical
checks and
`fluree-db-api/tests/it_reasoning_imports.rs::instance_data_in_schema_graph_does_not_leak`
for the regression test.

## Caching

`global_schema_bundle_cache()` is a process-wide `moka::sync::Cache` keyed
by:

- `ledger_id: Arc<str>`
- `to_t: i64`
- `starting_g_id: GraphId` (the resolved `f:schemaSource`)
- `follow_imports: bool`

Because config lives in the same ledger (g_id=2) and any config change
advances `t`, the `to_t` dimension is sufficient to express "config
version" — there is no separate config_epoch key, and no explicit
invalidation logic. Stale entries age out via LRU.

The cache stores the **resolution result** (`Vec<GraphId>`); the projected
flake arrays are rebuilt per query. Materialization is cheap relative to
reasoning itself, and keeping the cached value small lets many entries
coexist for many ledgers without memory pressure.

## Error semantics

`ApiError::OntologyImport` is raised when the configured closure is
invalid. Every message identifies the offending resource and suggests
remediation. Queries fail rather than silently returning reduced results,
so broken ontology references surface early. Sources of this error:

- An `owl:imports <X>` that doesn't match a local named graph and has no
  `f:ontologyImportMap` entry.
- A resolution that would land on a reserved system graph (config or
  txn-meta), whether via direct graph-IRI match, mapping table, or
  `f:schemaSource` selector.
- A `GraphRef` that targets a different ledger, uses `f:atT`, or carries a
  `f:trustPolicy` / `f:rollbackGuard`. The bundle is resolved at the
  query's single `to_t`, same-ledger scope only, and accepting these
  fields silently would create a gap between declared intent and actual
  behavior.

## Wiring at query time

`Fluree::query(&db, ...)` (and the dataset-query counterpart) call
`build_executable_for_view` → `attach_schema_bundle` on every query. The
attach step:

1. Reads `db.resolved_config().reasoning`. If there is no `f:schemaSource`,
   returns immediately — the legacy default-graph path applies unchanged.
2. Calls `resolve_schema_bundle` for the closure, consulting the cache.
3. Materializes `SchemaBundleFlakes` via `build_schema_bundle_flakes`.
4. Sets `executable.options.schema_bundle` so `prepare_execution` wraps
   `db.overlay` in a `SchemaBundleOverlay` for the reasoning_prep block.

Downstream, `schema_hierarchy_with_overlay`, `reason_owl2rl`, and
`Ontology::from_db_with_overlay` all receive the same wrapped overlay and
see the full closure on `g_id=0` reads.

## Testing

The acceptance suite lives in
`fluree-db-api/tests/it_reasoning_imports.rs` and covers:

- Same-ledger auto resolution of a named schema source.
- Transitive `A → B` with a subclass edge in `B`.
- Mapping table fallback for external IRIs.
- Unresolved imports surface as `ApiError::OntologyImport`.
- Cycle `A → B → A` terminates and still yields the correct closure.
- Mapping entries that would target a reserved system graph are rejected.
- `"reasoning": "none"` queries skip resolution entirely (no spurious
  errors from unrelated config).
- `f:atT` on a `GraphRef` is rejected with a clear message.
- Instance data in the schema graph does **not** leak into query results.
- **End-to-end OWL2-RL rule firing through a transitive import:**
  `owl:TransitiveProperty`, `owl:inverseOf`, and `rdfs:domain` axioms
  declared in an imported graph produce the expected entailments against
  instance data in the default graph.

Module-level unit tests cover the cache keys, empty-bundle passthrough,
and non-default-graph delegation.


---

# design/cross-ledger-model-enforcement.md

# Cross-ledger model enforcement

Fluree's `f:GraphRef` shape lets a data ledger reference a graph
containing policy / shapes / schema / rules / constraints. Without
cross-ledger references, every data ledger has to carry its own
copy of those governance artifacts. This document explains the
contracts that make `f:GraphRef` work cross-ledger so a single
**model ledger** — holding the ontology, SHACL shapes, policy rule
set, datalog rules, and uniqueness constraints — can be referenced
by many **data ledgers** it governs.

For the user-facing how-to (TriG examples, configuration steps),
see [Cross-ledger policy](../security/cross-ledger-policy.md). This
document explains the design decisions behind that mechanism: why
the resolver returns term-neutral artifacts, why the cache is
keyed the way it is, what the identity contract is, and how
failures are surfaced.

Topics:

- Glossary and what "cross-ledger" means at the contract level.
- The resolver contract: a single `resolve_graph_ref` helper shared
  by every subsystem, returning term-neutral artifacts.
- Term-space translation — why model-ledger Sids/GraphIds/t values
  cannot leak into data-ledger execution.
- Resolution time, caching, and failure variants.
- Policy IR identity split — definitional vs contextual term
  binding.
- Trust model, reserved-graph guards, cycle detection, drop
  interaction.
- Scope: what the resolver covers today and what's deferred.

Related docs:

- [Ontology imports](ontology-imports.md) — the same-ledger
  schema-source mechanism this generalizes.
- [Policy enforcement](../concepts/policy-enforcement.md) — the
  enforcement model the model ledger contributes rules to.
- [Configuration / setting groups](../ledger-config/setting-groups.md)
  — the `f:GraphRef` shape and where it appears.

## Glossary

| Term                        | Meaning |
|-----------------------------|---------|
| **Data ledger** (D)         | The ledger holding application data and serving the request. |
| **Model ledger** (M)        | A ledger referenced by D's `#config` to provide governance artifacts (policy / shapes / schema / rules / constraints). |
| **Governance artifact**     | The materialized output of resolving a `f:GraphRef` for one subsystem: a policy rule set, a shape set, a schema closure, a datalog rule set, or a constraint set. |
| **Resolved t** (`resolved_t`) | The transaction time of M at which the artifact is materialized for this request. |
| **Canonical ledger id**     | `NsRecord.ledger_id` from `nameservice.lookup()`, never a user-typed alias. |

## What "cross-ledger" means at the contract level

A data ledger D's `#config` declares a `f:GraphRef` whose `f:ledger`
field names a model ledger M. When D is served a request, every
configured `f:*Source` predicate that points at M is resolved to a
governance artifact materialized from M and applied to the request
against D. The data ledger's policy authority, identity, and term
space remain the binding authority for the request; M contributes
*rules*, not *identities*.

This is read-only: cross-ledger writes are out of scope.

## The resolver contract

Every subsystem that needs to read a cross-ledger graph goes
through the same helper:

```rust
pub async fn resolve_graph_ref(
    graph_ref: &GraphSourceRef,
    kind: ArtifactKind,
    ctx: &mut ResolveCtx<'_>,
) -> Result<Arc<ResolvedGraph>, CrossLedgerError>;
```

where `ResolveCtx` carries everything the resolver needs without
each subsystem rebuilding the surrounding state:

```rust
pub struct ResolveCtx<'a> {
    /// Canonical data-ledger id D.
    pub data_ledger_id: &'a str,
    /// The Fluree instance hosting D and (per the same-instance
    /// constraint) the referenced model ledger.
    pub fluree: &'a Fluree,
    /// Governance-context capture: lazily-populated map from
    /// canonical model ledger id → `resolved_t` for this request.
    /// The first reference to a given model populates the entry;
    /// every subsequent reference in the same request reuses it.
    pub resolved_ts: HashMap<String, i64>,
    /// Active resolution stack — the chain of `(kind, ledger,
    /// graph, resolved_t)` tuples currently being resolved. Used
    /// only for cycle detection; an entry is pushed before
    /// recursion and popped after. `ArtifactKind` is part of the
    /// key so a `PolicyRules` resolve of `(M, graph, t)` doesn't
    /// make a `Shapes` resolve of the same `(M, graph, t)` look
    /// like a cycle (or vice versa).
    pub active: Vec<(ArtifactKind, String, String, i64)>,
    /// Per-request memo of fully-resolved artifacts. Hits short-
    /// circuit without storage round-trip and without entering the
    /// cycle stack. Same `ArtifactKind`-extended key as `active`
    /// so different artifact kinds can't return each other's
    /// entries.
    pub memo: HashMap<(ArtifactKind, String, String, i64), Arc<ResolvedGraph>>,
}
```

The helper performs, in order:

1. **Same-instance check.** M must live on the same nameservice and
   storage namespace as D. Cross-instance federation is out of
   scope (see below).
2. **Canonical id resolution.** `f:ledger` is run through
   `nameservice.lookup()`; the returned `NsRecord.ledger_id` is used
   for every subsequent step, never the user-typed alias. (Same
   discipline as the default-context content store.)
3. **Reserved-graph guard.** Selectors that would resolve to
   `#config` or `#txn-meta` on M are rejected *before* any storage
   round-trip.
4. **Governance-context capture (`resolved_t`).** If `f:atT N` is set,
   `resolved_t = N`. Otherwise, look up `ctx.resolved_ts[model_id]`;
   on miss, read M's current head `t` and store it. The capture is
   **lazy and per request**: the head is consulted only on the first
   unpinned reference to a given model, and every later unpinned
   reference in the same request reuses the same `resolved_t` so
   policy and shapes can never disagree about which version of M
   they're enforcing.
5. **Memo / cycle check.** Form the tuple `(canonical_model_ledger_id,
   graph_iri, resolved_t)`. If it appears in `ctx.memo`, return the
   memoized artifact (this is the cross-subsystem de-dup path —
   `policySource` and `shapesSource` pointing at the same model graph
   resolve once). Otherwise check it against `ctx.active` (the
   resolution stack); presence there means a true cycle and is an
   error. Two different `atT` pins of the same `(ledger, graph)` are
   not a cycle; the same triple is. Push the tuple onto `active`
   before recursing into transitive imports.
6. **Translation and materialization.** The graph at `resolved_t` is
   read and projected into term-neutral form (see next section).
   Pop the tuple from `active` and insert the resolved artifact into
   `ctx.memo` and the global cache.
7. **Caching.** On cache hit the materialized artifact is returned
   directly. On miss the artifact is inserted under the key
   `(ArtifactKind, canonical_model_ledger_id, graph_iri, resolved_t)`.

A `ResolvedGraph` is term-neutral and t-fixed:

```rust
pub struct ResolvedGraph {
    pub model_ledger_id: String,       // canonical
    pub graph_iri: String,
    pub resolved_t: i64,
    pub artifact: GovernanceArtifact,  // tagged union per subsystem
}
```

`GovernanceArtifact` is a tagged union with one variant per
subsystem. Only `PolicyRules` is implemented today; the rest are
named in [Scope](#scope) and land as new variants when their
materializers do.

```rust
pub enum GovernanceArtifact {
    PolicyRules(PolicyArtifactWire),   // IRI-form policy rules
}
```

Each variant is **term-neutral**: every subject, predicate, object,
class, and datatype reference is stored as an IRI (or canonical
literal), never as a model-ledger Sid or GraphId. The data-ledger
consumer re-interns IRIs against its own dictionary at use time.

## Term-space translation

This is the load-bearing technical claim and the reason the resolver
contract changes rather than just gaining a new `f:ledger` branch.

Within a ledger, Fluree internally identifies subjects/predicates by
`Sid(namespace_code, local_id)` and graphs by `GraphId(u16)`. Both
are *ledger-local*. The IRI `<http://example.org/Person>` may be
`Sid(ns=7, id=42)` in M and `Sid(ns=13, id=200)` in D. M's
`GraphId(3)` and D's `GraphId(3)` have nothing to do with each
other. M's `t=10` and D's `t=3` are not comparable.

Today's same-ledger schema bundle (`build_schema_bundle_flakes` in
`fluree-db-query/src/schema_bundle.rs`) returns raw flakes and
relies on the data query's `to_t` for further filtering. That
contract makes three implicit assumptions that all break
cross-ledger:

1. Sid namespace codes are shared between source and consumer.
2. Graph ids are shared.
3. The artifact's `t` is comparable to the consumer's `t`.

The cross-ledger contract therefore returns a **term-space-neutral,
model-t-fixed** artifact. Concretely:

- All identifiers in the artifact are IRIs (interned at use site
  against D's term space) or canonical literal values.
- All time filtering against M is applied *at materialization time*
  inside the resolver; the consumer never re-applies its own `to_t`
  to the resolved artifact.
- The data-ledger executor re-interns IRIs as needed against D's
  dictionary. New IRIs that D has not seen are interned on demand
  (the standard intern path); they do not need to pre-exist in D's
  registry.

The cache key is `(canonical_model_ledger_id, graph_iri,
resolved_t)` — *not* parameterized by the data ledger's term space.
This is the property that makes "model edit propagates atomically to
all governed datasets" cheap: one cache entry per `(model, graph,
t)` is reused across every data ledger that references it.

Per-data-ledger interning is a derived view of the cached artifact,
memoized at the use site only if profiling shows interning is a
bottleneck.

## Policy IR identity split

Policy rules typically carry two kinds of term references that the
existing IR conflates:

| Reference kind | Examples                                      | Must bind in |
|----------------|-----------------------------------------------|--------------|
| Definitional  | Rule classes, predicate IRIs, target classes  | M (model ledger) |
| Contextual    | Request identity, tested resource subjects    | D (data ledger) + request context |

A naive "load rules from M, evaluate them as if D were M" build
produces a worse-than-useless result: only model-ledger identities
could ever satisfy the rules.

The IR therefore distinguishes the two binding scopes explicitly.
When the resolver returns a `PolicyRules` artifact, every term
reference is tagged with its scope (`Definitional` resolves in M's
term space at materialization time; `Contextual` is left as an IRI
to bind at evaluation time against D).

Authentication and identity flow only one way: from D and the
request context. The model ledger contributes rules and definitions;
it never contributes identity, authentication keys, or session
state. This keeps trust one-directional.

### Identity-mode resolution under cross-ledger policy

The same-ledger materializer in `policy_builder.rs` supports three
modes, in priority order: identity (request identity → policies via
that identity's `f:policyClass`), policy_class (configured class IRIs
→ all policies of that class), and policy (inline JSON-LD).

Cross-ledger does **not** generalize "identity mode" by querying
M for `<identity> f:policyClass ?class`. The identity binding lives
in D and the request context; querying M for identity records would
either return empty (M has no entry for D's user) or — worse — match a
model-ledger identity that happens to share an IRI with D's user. The
resulting policy attribution is silently wrong.

Concretely, cross-ledger policy resolution always uses
**policy_class mode**: the data ledger's effective policy classes
(from `opts.policyClass` or D's config) are looked up in M to load
the corresponding policy rules. The request identity is bound to
`?$identity` at evaluation time against D and the request context,
exactly as in the same-ledger flow.

If the data ledger's effective config does not specify
`f:policyClass`, the filter defaults to `{f:AccessPolicy}` — the
canonical policy class IRI. Cross-ledger enforcement then pulls
in every rule in M's policy graph that's typed as
`f:AccessPolicy` directly. Custom-typed rules (e.g.,
`ex:OrgPolicy`) require an explicit `f:policyClass` entry in D's
config to be enforced. This default is intentionally a small
allowlist rather than "every structural rule from M": operators
opt into custom-class enforcement by naming it, never by
omission. Inline `opts.policy` JSON-LD continues to merge against
D, never against M.

## Resolution time and `f:atT`

| Case                       | Behavior |
|----------------------------|----------|
| No `f:atT`                 | On first unpinned reference to M in this request, read M's current head `t` and cache it in `ctx.resolved_ts[model_id]`. Subsequent unpinned references to the same M reuse it. |
| `f:atT N`                  | `resolved_t = N`. Per-resolve, not stored in `resolved_ts`. M time-travels to that point. |
| `f:atT N`, N pruned        | Fail closed (see [Failure variants](#failure-variants)). No fallback to nearest-available. |
| Mid-request M advancement  | Not reflected in the current request. The next request re-captures on its first reference. |

Capture is **lazy and per request**: a request that never needs M
never pays for M's head lookup. Once captured, `resolved_t` is
stable across every subsystem in that request — policy and shapes
can never disagree about which version of M they're enforcing. This
is what makes the "model edit propagates atomically to all governed
datasets" property hold within a single request boundary.

## Caching

Resolved artifacts are cached at the **API layer** (in
`fluree-db-api`), not in `fluree-db-binary-index::LeafletCache`. The
binary-index crate sits below `fluree-db-api`, `fluree-db-policy`,
and the cross-ledger module; making it depend upward on typed
governance-artifact representations would be a layering inversion.

The implementation is a Moka TinyLFU cache bounded by entry count
(see `cross_ledger::GovernanceCache`), scoped to a `Fluree`
instance. Single memory-pool unification with `LeafletCache` would
require adding an opaque-blob variant to the binary-index cache
(serializing the artifact to bytes at the cache boundary) and is
deliberately deferred until the artifact representation
stabilizes — keeping the two caches separate while artifact shapes
are still evolving prevents premature coupling.

The key is `(ArtifactKind, canonical_model_ledger_id, graph_iri,
resolved_t)`. `ArtifactKind` is part of the key so a memoized
`PolicyRules` entry never short-circuits a `Shapes` lookup of the
same `(M, graph, t)`. New commits to M produce new keys without
explicit invalidation; unreferenced entries age out under the
cache's eviction policy. There is no "watermark-on-write" channel.

The cache value is the term-neutral `ResolvedGraph` (IRIs, not Sids).
Per-data-ledger interning is not part of the cache key — the cache
is shareable across every data ledger that references the same
`(model, graph, t)`, which is what makes "model edit propagates
atomically to all governed datasets" cheap.

## Failure variants

Cross-ledger failures must be distinguishable for audit; they MUST
NOT collapse into a single generic import error.

```rust
pub enum CrossLedgerError {
    /// `f:ledger` names a ledger that does not exist or has been
    /// dropped on this instance.
    ModelLedgerMissing { ledger_id: String },

    /// `f:ledger` resolves but the named graph IRI has no entry in
    /// the model ledger's graph registry at `resolved_t`.
    GraphMissingAtT { ledger_id: String, graph_iri: String, resolved_t: i64 },

    /// `f:atT N` was requested but M no longer retains state at N
    /// (index pruning, history retention).
    TAtUnavailable { ledger_id: String, requested_t: i64, oldest_available_t: i64 },

    /// The selector targets `#config` or `#txn-meta` on the model
    /// ledger.
    ReservedGraphSelected { graph_iri: String },

    /// The resolver successfully read the graph but could not
    /// translate it to term-neutral form (missing IRI on a Sid that
    /// the model dictionary lost, malformed rule, etc.).
    TranslationFailed { ledger_id: String, graph_iri: String, detail: String },

    /// `f:trustPolicy` verification failed (reserved for when
    /// trust-policy enforcement is implemented; see Scope).
    TrustCheckFailed { ledger_id: String, detail: String },

    /// `f:atT`, `f:trustPolicy`, or `f:rollbackGuard` was set on
    /// a `GraphSourceRef`. Those fields are parsed by the config
    /// layer but their semantics are not yet implemented (see
    /// Scope); the request fails closed rather than silently
    /// ignoring the field.
    UnsupportedFeature { feature: &'static str, phase: &'static str, ledger_id: String },

    /// `f:ledger` targets a ledger on a different instance.
    CrossInstanceUnsupported { ledger_id: String },

    /// Cycle detected through the `(kind, ledger, graph, resolved_t)`
    /// chain.
    CycleDetected {
        chain: Vec<(ArtifactKind, String, String, i64)>,
    },
}
```

Every variant is fail-closed: the request fails. There is no silent
fallback to "no policy" or "no shapes."

## Trust model

A data ledger D's `#config` declaring `f:ledger <M>` is itself the
capability assertion. Writing to D's `#config` already requires
policy authority on D, so "whoever can write D's `#config` asserts
that M is a trusted governance source for D" is the binding
decision; no separate consent is required from M.

Cross-instance federation would require a different trust model
(auth, transport, signing for ledgers hosted on different
nameservices) and is out of scope. See [Scope](#scope).

## Reserved-graph guard

The same-ledger version of this guard lives in
`ontology_imports::resolve_local_graph_source`: selectors resolving
to `g_id=1` (`#txn-meta`) or `g_id=2` (`#config`) are rejected.

The cross-ledger version applies the same check by IRI *before* any
storage round-trip on M. The motivation is doubled: `#txn-meta` on
M could leak commit metadata to D's request handler.

## Cycle detection

Two structures, distinct purposes:

- `ctx.active` is the **active resolution stack** — push the
  `(canonical_ledger_id, graph_iri, resolved_t)` tuple before
  recursing into a transitive import, pop on return. A tuple is a
  cycle only if it is encountered while *already on the stack*.
- `ctx.memo` is the **per-request completed map**. Once a tuple
  resolves successfully, it lands here. Subsequent references to
  the same tuple — from any subsystem in the same request — short-
  circuit on the memo, never enter `active`, and never trip cycle
  detection.

So if `policySource` and `shapesSource` both reference the same
`(ledger, graph, t)`, the second resolve is a memo hit, not a
cycle. Two different `atT` pins of the same `(ledger, graph)` are
not a cycle. Two different graphs on the same ledger are not a
cycle. Only re-entering an in-flight tuple is.

This generalizes the BFS+dedupe pattern the existing same-ledger
`owl:imports` resolver uses to a 3-tuple, with the
active-vs-completed distinction surfaced explicitly.

## Drop interaction

If a data ledger D references model ledger M and M is dropped,
the next request against D that needs governance from M fails
closed with `ModelLedgerMissing`. There is no reverse-reference
index and no rejection of M's drop based on outstanding
references.

This is the smallest contract that's safe — operators get a clear
failure on the next governed request rather than a silent shift in
enforcement. Introducing reverse indexes would require nameservice
schema work and is out of scope (see [Scope](#scope)). Operators
who need stronger guarantees coordinate drops at the application
layer for now.

## Same-instance constraint

Both D and M must:

- Belong to the same nameservice instance.
- Live within the same storage namespace.

A reference that targets a ledger on a different instance surfaces
as `CrossInstanceUnsupported` before any storage round-trip. This
boundary is enforced implicitly by the nameservice lookup — a
ledger not present in this instance's nameservice can't be
canonicalized — and the variant exists so a future cross-instance
mode can be added without rewriting the failure taxonomy.

## Scope

### Implemented

- `f:policySource` cross-ledger via `resolve_graph_ref`. The
  policy IR carries definitional/contextual term references
  separately so the model ledger contributes rules while the
  data ledger contributes identity binding.
- `f:constraintsSource` cross-ledger via the same shared
  resolver. M's `f:enforceUnique true` annotations on
  properties apply to D's transactions; a tx that would
  create a duplicate value on one of those properties is
  rejected with `TransactError::UniqueConstraintViolation`
  even though the annotation never lives on D.
- `f:schemaSource` cross-ledger via the same shared resolver.
  The whitelisted schema axiom triples in M's ontology graph
  (rdfs:subClassOf, rdfs:subPropertyOf, rdfs:domain, rdfs:range,
  owl:equivalentClass / equivalentProperty / inverseOf / sameAs /
  imports, and rdf:type for the schema-class set) are projected
  into a `SchemaBundleFlakes` against D's snapshot and feed D's
  reasoner. Single-graph only today; transitive `owl:imports`
  recursion across multiple model ledgers is reserved.
- Per-request memo + per-instance governance cache, both keyed
  on `(ArtifactKind, canonical_model_ledger_id, graph_iri,
  resolved_t)`.
- Reserved-graph guard (rejects `#config` / `#txn-meta` on M
  before any storage round-trip).
- Reserved-feature rejection: `f:atT`, `f:trustPolicy`, and
  `f:rollbackGuard` are surfaced as `UnsupportedFeature` rather
  than silently ignored.
- Identity-mode + cross-ledger policy combination fails closed
  with a config error — the design's "M contributes rules, D
  contributes identity" boundary is enforced at the request
  surface.

### Reserved

The following subsystems share the resolver's contract but their
materializers aren't implemented. Each lands as a new
`GovernanceArtifact` variant + per-subsystem materializer:

- Transitive `owl:imports` recursion across multiple model
  ledgers (Phase 1b's full scope). The single-graph schema
  materialization above already projects M's `owl:imports`
  triples into the wire so a future reader can see what M
  declared — the recursion through `resolve_graph_ref`
  (which would exercise `ResolveCtx.active` for cycle detection
  across ledgers) lands separately.
- `f:ontologyImportMap` cross-ledger (mapping table entries
  whose `f:graphRef` targets another model ledger).
- `f:shapesSource` cross-ledger (SHACL shapes).
- `f:rulesSource` cross-ledger (datalog rules). The same-ledger
  routing for `f:rulesSource` is also still pending; the
  cross-ledger path will route through the shared resolver
  once same-ledger lands.
- `f:atT` temporal pinning (currently rejected as
  `UnsupportedFeature`).
- `f:trustPolicy` and `f:rollbackGuard` (rejected as
  `UnsupportedFeature` for the same reason).

### Out of scope

- **Cross-instance federation.** Different nameservices,
  transport, cross-org auth/signing.
- **Auto-resolution by IRI namespace.** "Which model governs
  `schema:*`?" — application-layer concern.
- **Writing back to a model ledger from a governed ledger's
  request.** Cross-ledger references are read-only.
- **Reverse-reference indexes for safe drop.** A drop on M
  surfaces on the next governed request against D, not at
  drop time.
- **Subclass entailment in policy_class filtering.** The
  filter is exact-IRI; D must name the same class IRI M
  declared. Mirrors same-ledger `load_policies_by_class`
  semantics.

## Error type and HTTP mapping

`CrossLedgerError` surfaces through the API crate as a dedicated
variant:

```rust
pub enum ApiError {
    // ...
    /// Cross-ledger governance resolution failed. The wrapped
    /// variant carries the specific failure (missing ledger, graph
    /// missing at t, retention pruned, etc.) for audit and
    /// operator diagnostics.
    #[error("Cross-ledger error: {0}")]
    CrossLedger(#[from] CrossLedgerError),
}
```

It is **not** collapsed into `ApiError::Http { status: 502, .. }` —
preserving the structured variant is what makes "the model ledger
this data ledger depends on is broken" distinguishable from "your
data ledger is broken" in logs and audit trails.

HTTP status mapping: **502 Bad Gateway** is the default. A model
ledger dependency that cannot be resolved or used is conceptually
an upstream-dependency failure, not an internal panic. 424 Failed
Dependency is semantically closer but less commonly handled by
client tooling; 502 is the pragmatic choice. The server layer reads
the variant for the response body so callers can branch on the
specific failure even when the status is generic.



---

# design/storage-traits.md

# Storage Traits Design

This document describes the storage trait architecture in Fluree DB, explaining the design rationale and providing guidance for implementing new storage backends.

## Overview

Fluree uses a layered storage abstraction that separates:
- **Content-addressed access** (`fluree-db-core`): The `ContentStore` trait provides get/put/has operations keyed by `ContentId` (CIDv1). This is the primary interface for all immutable artifact access (commits, index roots, leaves, dicts).
- **Physical storage traits** (`fluree-db-core`): Runtime-agnostic storage operations (`StorageRead`, `StorageWrite`, `ContentAddressedWrite`) with standard `Result<T>` error handling. These handle the physical I/O layer beneath ContentStore.
- **Extension traits** (`fluree-db-nameservice`): Nameservice-specific operations with `StorageExtResult<T>` for richer error semantics (CAS operations, pagination, etc.).

See [ContentId and ContentStore](content-id-and-contentstore.md) for the content-addressed identity model.

## Quick Start: The Prelude

For convenient imports, use the storage prelude:

```rust
use fluree_db_core::prelude::*;

// Now you have access to:
// - Storage, StorageRead, StorageWrite, ContentAddressedWrite (traits)
// - MemoryStorage, FileStorage (implementations)
// - ContentKind, ContentWriteResult, ReadHint (types)

async fn example<S: Storage>(storage: &S) -> Result<()> {
    let bytes = storage.read_bytes("some/address").await?;
    storage.write_bytes("other/address", &bytes).await?;
    Ok(())
}
```

For API consumers, `fluree-db-api` re-exports all storage traits:

```rust
use fluree_db_api::{Storage, StorageRead, MemoryStorage};
```

## Trait Hierarchy

```text
              ┌──────────────────────┐
              │    ContentStore      │  get(ContentId), put(ContentKind, bytes), has(ContentId)
              └──────────────────────┘
                    (primary interface for immutable artifacts)

              ┌─────────────────┐
              │   StorageRead   │  read_bytes, exists, list_prefix
              └────────┬────────┘
                       │
              ┌────────┴────────┐
              │  StorageWrite   │  write_bytes, delete
              └────────┬────────┘
                       │
        ┌──────────────┴──────────────┐
        │   ContentAddressedWrite     │  content_write_bytes[_with_hash]
        └──────────────┬──────────────┘
                       │
              ┌────────┴────────┐
              │     Storage     │  (marker trait - blanket impl)
              └─────────────────┘
                    (physical I/O layer)
```

`ContentStore` is the content-addressed layer that sits above the physical storage traits. It maps `ContentId` values to physical storage locations via the underlying `Storage` implementation.

## ContentStore (fluree-db-core)

The `ContentStore` trait is the primary interface for accessing immutable, content-addressed artifacts (commits, index roots, leaves, dictionaries, etc.).

```rust
#[async_trait]
pub trait ContentStore: Debug + Send + Sync {
    /// Retrieve bytes by content ID
    async fn get(&self, id: &ContentId) -> Result<Vec<u8>>;

    /// Store bytes, returning the computed ContentId
    async fn put(&self, kind: ContentKind, bytes: &[u8]) -> Result<ContentId>;

    /// Check whether an object exists
    async fn has(&self, id: &ContentId) -> Result<bool>;
}
```

**Design notes:**
- `ContentId` is a CIDv1 value encoding the hash function, digest, and content kind (multicodec). See [ContentId and ContentStore](content-id-and-contentstore.md).
- `ContentKind` enables routing to different storage tiers (commit store vs index store) without parsing URL paths.
- `put` computes the content hash and returns the derived `ContentId`.
- Implementations include `MemoryContentStore` (for testing) and `BridgeContentStore` (adapts a `Storage` backend).

## Physical Storage Traits (fluree-db-core)

The physical storage traits handle raw byte I/O against storage backends (filesystem, S3, memory). `ContentStore` implementations typically wrap these.

### StorageRead

Read-only storage operations. Implement this for any storage that can retrieve data.

```rust
#[async_trait]
pub trait StorageRead: Debug + Send + Sync {
    /// Read raw bytes from an address
    async fn read_bytes(&self, address: &str) -> Result<Vec<u8>>;

    /// Read with a hint for content type optimization
    /// Default implementation ignores the hint
    async fn read_bytes_hint(&self, address: &str, hint: ReadHint) -> Result<Vec<u8>> {
        self.read_bytes(address).await
    }

    /// Check if an address exists
    async fn exists(&self, address: &str) -> Result<bool>;

    /// Read a `[start, end)` byte range from an address.
    /// Default implementation fetches the full object and slices; backends
    /// that support native range reads (S3, HTTP) should override.
    async fn read_byte_range(&self, address: &str, range: std::ops::Range<u64>)
        -> Result<Vec<u8>>;

    /// List all addresses with a given prefix
    async fn list_prefix(&self, prefix: &str) -> Result<Vec<String>>;

    /// List addresses under a prefix together with byte sizes.
    /// Default implementation returns an error indicating the backend does
    /// not support cheap metadata listing. Backends with native list+size
    /// (S3 `list_objects_v2`, GCS, etc.) should override.
    async fn list_prefix_with_metadata(&self, prefix: &str)
        -> Result<Vec<RemoteObject>>;

    /// Resolve a CAS address to a local filesystem path, if available.
    fn resolve_local_path(&self, address: &str) -> Option<PathBuf> { None }
}

/// `(address, size)` pair returned by `list_prefix_with_metadata`.
pub struct RemoteObject {
    pub address: String,
    pub size_bytes: u64,
}
```

**Design notes:**
- `read_bytes_hint` enables optimizations like returning pre-encoded flakes for leaf nodes
- `read_byte_range` allows partial reads against backends with native HTTP/S3 range support;
  the default impl is correct but does N full-object fetches for N range reads
- `list_prefix` is essential for garbage collection and administrative operations
- `list_prefix_with_metadata` is used by the bulk-import remote-source path so the
  importer can size each chunk before fetching. Backends without cheap size metadata
  return an error; callers can fall back to caller-supplied object lists
- `resolve_local_path` lets callers (e.g., import scratch staging) skip a copy when
  the storage already exposes data on the local filesystem (`FileStorage`)
- All methods return `fluree_db_core::Result<T>` (alias for `std::result::Result<T, Error>`)

### StorageWrite

Mutating storage operations. Implement alongside `StorageRead` for read-write storage.

```rust
#[async_trait]
pub trait StorageWrite: Debug + Send + Sync {
    /// Write raw bytes to an address
    async fn write_bytes(&self, address: &str, bytes: &[u8]) -> Result<()>;

    /// Delete data at an address
    async fn delete(&self, address: &str) -> Result<()>;
}
```

**Design notes:**
- `delete` is part of the core write trait (not separate) because any writable storage should support deletion
- Implementations should be idempotent: deleting a non-existent address succeeds silently

### ContentAddressedWrite

Extension trait for content-addressed (hash-based) writes. Extends `StorageWrite`.

```rust
#[async_trait]
pub trait ContentAddressedWrite: StorageWrite {
    /// Write bytes with a pre-computed content hash
    /// Returns the canonical address and metadata
    async fn content_write_bytes_with_hash(
        &self,
        kind: ContentKind,
        ledger_id: &str,
        content_hash_hex: &str,
        bytes: &[u8],
    ) -> Result<ContentWriteResult>;

    /// Write bytes, computing the hash internally
    /// Default implementation computes SHA-256 and delegates
    async fn content_write_bytes(
        &self,
        kind: ContentKind,
        ledger_id: &str,
        bytes: &[u8],
    ) -> Result<ContentWriteResult> {
        let hash = sha256_hex(bytes);
        self.content_write_bytes_with_hash(kind, ledger_id, &hash, bytes).await
    }
}
```

**Design notes:**
- `ContentKind` indicates whether data is a commit or index, enabling routing to different storage tiers
- The default `content_write_bytes` implementation handles hash computation, so most backends only need to implement `content_write_bytes_with_hash`
- Content-addressed storage enables deduplication and integrity verification

### Storage (Marker Trait)

A convenience marker trait indicating full storage capability.

```rust
/// Full storage capability: read + content-addressed write
pub trait Storage: StorageRead + ContentAddressedWrite {}

/// Blanket implementation for any type implementing both traits
impl<T: StorageRead + ContentAddressedWrite> Storage for T {}
```

**Usage:**
```rust
// Instead of this verbose bound:
fn process<S: StorageRead + StorageWrite + ContentAddressedWrite>(storage: &S)

// Use this:
fn process<S: Storage>(storage: &S)
```

## Extension Traits (fluree-db-nameservice)

The nameservice crate defines additional traits with `StorageExtResult<T>` for richer error handling (e.g., `PreconditionFailed` for CAS operations).

### StorageList

Paginated listing for large-scale storage backends.

```rust
#[async_trait]
pub trait StorageList {
    async fn list_prefix(&self, prefix: &str) -> StorageExtResult<Vec<String>>;

    async fn list_prefix_paginated(
        &self,
        prefix: &str,
        continuation_token: Option<String>,
        max_keys: usize,
    ) -> StorageExtResult<ListResult>;
}
```

### StorageCas

Compare-and-swap operations for consistent distributed updates.

```rust
#[async_trait]
pub trait StorageCas {
    /// Write only if the address doesn't exist
    async fn write_if_absent(&self, address: &str, bytes: &[u8]) -> StorageExtResult<bool>;

    /// Write only if the current version matches expected_etag
    async fn write_if_match(
        &self,
        address: &str,
        bytes: &[u8],
        expected_etag: &str,
    ) -> StorageExtResult<String>;

    /// Read with version/etag for subsequent CAS operations
    async fn read_with_etag(&self, address: &str) -> StorageExtResult<(Vec<u8>, String)>;
}
```

### StorageDelete (nameservice)

Delete with nameservice error semantics.

```rust
#[async_trait]
pub trait StorageDelete {
    async fn delete(&self, address: &str) -> StorageExtResult<()>;
}
```

**Why separate from core `StorageWrite::delete`?**
- Nameservice operations need `StorageExtResult` for errors like `PreconditionFailed`
- Core operations use standard `Result` for simplicity
- Storage backends typically implement both, with the nameservice version delegating to core

## Implementing a Storage Backend

### Minimal Read-Only Backend

For a read-only backend (e.g., `ProxyStorage` that fetches via HTTP):

```rust
#[async_trait]
impl StorageRead for MyReadOnlyStorage {
    async fn read_bytes(&self, address: &str) -> Result<Vec<u8>> {
        // Fetch from remote
    }

    async fn exists(&self, address: &str) -> Result<bool> {
        // Check existence (can implement as try-read)
        match self.read_bytes(address).await {
            Ok(_) => Ok(true),
            Err(Error::NotFound(_)) => Ok(false),
            Err(e) => Err(e),
        }
    }

    async fn list_prefix(&self, _prefix: &str) -> Result<Vec<String>> {
        Err(Error::storage("list_prefix not supported"))
    }
}

// Must also implement StorageWrite (with error stubs) and ContentAddressedWrite
// if you want to satisfy the Storage marker trait
#[async_trait]
impl StorageWrite for MyReadOnlyStorage {
    async fn write_bytes(&self, _: &str, _: &[u8]) -> Result<()> {
        Err(Error::storage("read-only storage"))
    }
    async fn delete(&self, _: &str) -> Result<()> {
        Err(Error::storage("read-only storage"))
    }
}

#[async_trait]
impl ContentAddressedWrite for MyReadOnlyStorage {
    async fn content_write_bytes_with_hash(&self, ...) -> Result<ContentWriteResult> {
        Err(Error::storage("read-only storage"))
    }
}
```

### Full Read-Write Backend

For a complete backend (e.g., S3, filesystem):

```rust
// 1. Implement core traits
#[async_trait]
impl StorageRead for MyStorage {
    async fn read_bytes(&self, address: &str) -> Result<Vec<u8>> { ... }
    async fn exists(&self, address: &str) -> Result<bool> { ... }
    async fn list_prefix(&self, prefix: &str) -> Result<Vec<String>> { ... }
}

#[async_trait]
impl StorageWrite for MyStorage {
    async fn write_bytes(&self, address: &str, bytes: &[u8]) -> Result<()> { ... }
    async fn delete(&self, address: &str) -> Result<()> { ... }
}

#[async_trait]
impl ContentAddressedWrite for MyStorage {
    async fn content_write_bytes_with_hash(
        &self,
        kind: ContentKind,
        ledger_id: &str,
        content_hash_hex: &str,
        bytes: &[u8],
    ) -> Result<ContentWriteResult> {
        // Build address from kind + alias + hash
        let address = build_content_address(kind, ledger_id, content_hash_hex);
        self.write_bytes(&address, bytes).await?;
        Ok(ContentWriteResult {
            address,
            content_hash: content_hash_hex.to_string(),
            size_bytes: bytes.len(),
        })
    }
}

// Storage marker trait is automatically satisfied via blanket impl

// 2. Optionally implement nameservice traits for advanced features
#[async_trait]
impl StorageList for MyStorage {
    async fn list_prefix(&self, prefix: &str) -> StorageExtResult<Vec<String>> {
        // Delegate to core trait, convert error
        StorageRead::list_prefix(self, prefix)
            .await
            .map_err(|e| StorageExtError::Other(e.to_string()))
    }
    // ... paginated version
}
```

## BranchedContentStore (fluree-db-core)

`BranchedContentStore<S>` is a recursive `ContentStore` implementation that provides namespace-scoped fallback reads for branched ledgers. When a branch is created, it gets its own storage namespace for new writes, but needs to read pre-branch-point content (commits, dictionaries) from ancestor namespaces.

### Structure

```rust
pub struct BranchedContentStore<S: Storage> {
    branch_store: StorageContentStore<S>,
    parents: Vec<BranchedContentStore<S>>,
}
```

- **`branch_store`** — the branch's own namespace store; all writes go here
- **`parents`** — ancestor stores to fall back to for reads (recursive tree)

The recursive structure supports arbitrarily deep branch chains (main → dev → feature) and is designed to support future merge scenarios where a branch may have multiple parents (DAG ancestry).

### Constructors

```rust
// Root branch (e.g., main) — no parents
let store = BranchedContentStore::leaf(storage, "mydb:main");

// Branch with parent fallback
let parent = BranchedContentStore::leaf(storage, "mydb:main");
let store = BranchedContentStore::with_parents(storage, "mydb:dev", vec![parent]);
```

### Read Behavior

`get()` tries the branch's own namespace first, then recurses into parents:

1. Try `branch_store.get(id)` — if found, return immediately
2. If `NotFound` and parents exist, try each parent in order
3. If no parent finds it, return the last `NotFound` error
4. **Non-`NotFound` errors propagate immediately** — only `NotFound` triggers fallback

`has()` and `resolve_local_path()` follow the same fallback pattern.

### Write Behavior

`put()` and `put_with_id()` always write to `branch_store` — never to parents. This ensures branch isolation: new content is always scoped to the branch's own namespace.

### What Is and Isn't Copied at Branch Time

| Artifact | Copied? | Reason |
|----------|---------|--------|
| **Commits** | No | Immutable chain, never deleted; read via fallback |
| **Index structure files** (root, leaves, branches, arenas) | Yes | Source may GC old indexes after reindexing |
| **String dictionaries** | No | Stored globally in the `@shared` namespace; all branches read from the same location |

### Global Dictionary Storage (`@shared` Namespace)

String dictionaries (mappings between IRIs/strings and compact integer IDs) are the largest index artifact. Rather than copying them per-branch or relying on `BranchedContentStore` fallback reads, dictionaries are stored in a **global namespace** shared by all branches of a ledger.

The `content_path` function routes all `DictBlob` CIDs to a shared path:

```text
mydb/@shared/dicts/<sha256hex>.subject    # Subject dict
mydb/@shared/dicts/<sha256hex>.string     # String dict
mydb/@shared/dicts/<sha256hex>.predicate  # Predicate dict
...
```

The `@shared` prefix uses the `@` character, which is forbidden in branch names by `validate_branch_name`, so it cannot collide with any branch namespace. The constant is defined as `SHARED_NAMESPACE` in `fluree-db-core::address_path`.

**Legacy fallback:** Existing deployments may have dictionaries stored at the old per-branch path (e.g., `mydb/main/index/objects/dicts/<sha>.dict`). `StorageContentStore` automatically falls back to the legacy path when a dict CID is not found at the new `@shared` location. After the next index build, new writes go to the `@shared` path — no manual migration is needed.

### Building the Store Tree

`LedgerState::build_branched_store()` recursively walks the branch ancestry via nameservice `source_branch` metadata, constructing the `BranchedContentStore` tree. This uses `Box::pin` for the recursive async calls.

The actual ancestry walk lives in **`fluree-db-nameservice`** (`branched_store::build_branched_store`), and `LedgerState::build_branched_store` is a thin wrapper that delegates there. This keeps the helper available to crates that don't depend on `fluree-db-ledger` (notably `fluree-db-indexer`'s background worker).

### When to Use BranchedContentStore

Any code path that walks the commit chain or loads index blobs for a branched ledger MUST use a branch-aware content store. Per-query reads against an already-loaded `LedgerState` are fine — `LedgerState::load` already wires the branched store up.

Use the nameservice helpers, not the flat `StorageBackend::content_store(...)`:

| Helper | When to use |
|---|---|
| `fluree_db_nameservice::branched_content_store_for_record(backend, ns, &record)` | An `NsRecord` is in scope (no extra lookup) |
| `fluree_db_nameservice::branched_content_store_for_id(backend, ns, ledger_id)` | No `NsRecord` available — does one nameservice lookup |
| `Fluree::branched_content_store(&self, ledger_id)` | API / CLI callers — wraps `_for_id` |

Both helpers return the flat namespace store unchanged for non-branched ledgers, so adding them to non-branch code paths costs at most a single nameservice lookup.

A flat `backend.content_store(ledger_id)` on the commit-chain walk path will 404 the moment the walker steps past the fork point and tries to read an ancestor commit from the wrong namespace.

## Type Erasure with AnyStorage

For dynamic dispatch (e.g., runtime-selected storage backends), use `AnyStorage`:

```rust
/// Type-erased storage wrapper
pub struct AnyStorage {
    inner: Arc<dyn Storage>,
}

impl AnyStorage {
    pub fn new<S: Storage + 'static>(storage: S) -> Self {
        Self { inner: Arc::new(storage) }
    }
}
```

**When to use:**
- `FlureeClient` uses `AnyStorage` to support any backend at runtime
- Generic code should prefer concrete types (`S: Storage`) for better optimization
- Use `AnyStorage` when storage type is determined at runtime (e.g., from config)

## Wrapper Storages

Several wrapper types add functionality to underlying storage:

### TieredStorage

Routes commits and indexes to different backends:

```rust
pub struct TieredStorage<S> {
    commit_storage: S,
    index_storage: S,
}
```

### EncryptedStorage

Adds transparent encryption:

```rust
pub struct EncryptedStorage<S, K> {
    inner: S,
    key_provider: K,
}
```

### AddressIdentifierResolverStorage

Routes reads based on address format (e.g., different storage backends by identifier segment):

```rust
pub struct AddressIdentifierResolverStorage {
    default_storage: Arc<dyn Storage>,
    identifier_storages: HashMap<String, Arc<dyn Storage>>,
}
```

## Error Handling

### Core Errors (`fluree_db_core::Error`)

Standard errors for storage operations:
- `NotFound` - Address doesn't exist
- `Storage` - Generic storage failure
- `Io` - Underlying I/O error

### Nameservice Errors (`StorageExtError`)

Extended errors for nameservice operations:
- `NotFound` - Address doesn't exist
- `PreconditionFailed` - CAS condition not met
- `Other` - Generic error with message

## Summary

| Type | Crate | Purpose | Error Type |
|------|-------|---------|------------|
| `ContentStore` (trait) | core | Content-addressed get/put/has by `ContentId` | `Result<T>` |
| `BranchedContentStore` (struct) | core | Recursive `ContentStore` with namespace fallback for branches | `Result<T>` |
| `StorageRead` (trait) | core | Physical read operations | `Result<T>` |
| `StorageWrite` (trait) | core | Physical write + delete | `Result<T>` |
| `ContentAddressedWrite` (trait) | core | Hash-based physical writes | `Result<T>` |
| `Storage` (trait) | core | Marker (full physical capability) | - |
| `StorageList` (trait) | nameservice | Paginated listing | `StorageExtResult<T>` |
| `StorageCas` (trait) | nameservice | Compare-and-swap | `StorageExtResult<T>` |
| `StorageDelete` (trait) | nameservice | Delete with ext errors | `StorageExtResult<T>` |

Application code typically interacts with `ContentStore` for immutable artifact access. Storage backend implementors implement the physical traits (`StorageRead`, `StorageWrite`, `ContentAddressedWrite`) and the `Storage` marker trait is automatically satisfied. For branched ledgers, `BranchedContentStore` wraps the physical storage with recursive namespace fallback — see [BranchedContentStore](#branchedcontentstore-fluree-db-core) above.


---

# design/raft-command-queue.md

# Raft command queue and replicated state machine

How `fluree-db-consensus`'s Raft mode replicates writes across a cluster. The operations-facing recipe (cluster bootstrap, day-2 admin, security boundaries) lives in [Raft clusters (replicated writes)](../operations/raft-clusters.md); this doc covers the design choices behind the implementation.

## Goals and constraints

The transactional path on a Fluree server is non-trivial: a write has to parse JSON-LD / SPARQL Update, evaluate policy, generate flakes, resolve conflicts, write a commit blob to content-addressed storage (CAS), and update the branch head. Replicating this naively across a Raft cluster — by replaying the full work on every node — runs into three problems:

1. **The work isn't deterministic in the inputs the log carries.** Policy evaluation, conflict resolution, indexing side-effects, and CAS writes all depend on state the log doesn't replicate (cache contents, wall-clock ordering, storage backend addresses).
2. **The log would bloat.** Commit envelopes can be megabytes; replicating them through the log instead of CAS doubles the network and storage cost of every write for no benefit — every node already has CAS access.
3. **Stage time is unbounded.** Some transactions take seconds (large updates, complex policy, indexing). Blocking the openraft commit path on that work would extend leader heartbeat latency and degrade liveness.

The design splits the work: the Raft log replicates **decisions** (branch head moved to CID `x` at queue position `n`), and the CAS holds the **bytes** (envelopes, commit blobs, index artifacts). Only the leader stages, but every node observes the result.

## Component map

The Raft consensus crate (`fluree-db-consensus/src/raft/`) is structured as a set of cooperating components:

| Component                  | Lives where                                              | Job                                                                                                         |
| -------------------------- | -------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| `Command`, `Response`      | `state_machine.rs`                                       | The log entry types. ~20 variants spanning transaction flow, ledger lifecycle, and metadata.                |
| `NameServiceState`         | `state_machine.rs`                                       | The replicated in-memory state: branch heads, ledger registry, per-branch queues, idempotency cache.        |
| `StateMachineAdapter`      | `state_machine_adapter.rs`                               | openraft's `RaftStateMachine` impl. Applies entries, takes/installs snapshots, resolves waiters.            |
| `LogStore`, `SnapshotStore`| `log_adapter.rs`, `storage/{fs,memory}.rs`               | openraft's `RaftLog`/`RaftSnapshotBuilder`. Local-disk persistence.                                          |
| `HttpRaftNetworkFactory`   | `network.rs`                                             | Inter-node RPC (`/raft/vote`, `/raft/append-entries`, `/raft/install-snapshot`) over HTTP.                  |
| `RaftAdmin` / `/cluster/*` | `admin.rs`                                               | Operator-facing membership endpoints (`initialize`, `add-learner`, `change-membership`, `status`).         |
| Follower-forward middleware| `forward.rs`                                             | Axum middleware that proxies leader-only client requests to the current leader.                            |
| `QueuedTransactor`         | `queued_transactor.rs`                                   | Client-side proposer. Builds envelopes, writes to CAS, proposes `EnqueueCommand`, awaits the typed receipt. |
| `CommitWorker`             | `commit_worker.rs`                                       | Leader-only. Drains per-branch queues, stages work, proposes `ApplyHead`.                                  |
| `EvictionScheduler`        | `eviction_scheduler.rs`                                  | Leader-only. Periodically proposes `EvictIdempotency` to age out the cache.                                |
| `RaftNameService`          | `nameservice.rs`                                         | The replicated `NameService` impl. Reads observe `NameServiceState`; writes propose log entries.            |
| `WaiterMap`                | `waiter.rs`                                              | Per-process oneshot registry keyed by `queue_id`. Bridges propose and apply.                                |
| `StagedReceiptMap`         | `staged_receipt.rs`                                      | Per-process map carrying typed apply receipts (flake counts, tally, conflict resolution) from worker to transactor on the same node. |

Three of these (`CommitWorker`, `EvictionScheduler`, follower-forward middleware) are gated on leadership: the integration's leader watcher spawns / stops them in response to `current_leader()` changes.

## Submission flow in detail

Stages, traced end-to-end:

```
Client → POST /api/transact
   ↓ (any node)
[follower-forward middleware]
   ├─ this node is leader  → next.run() (continue locally)
   └─ this node is follower → HTTP forward to leader's client_addr
        ↓
[QueuedTransactor on leader]
   1. write QueuedRequest envelope to CAS  → envelope_cid
   2. register oneshot waiter on WaiterMap → rx
   3. propose Command::EnqueueCommand { envelope_cid, body_hash, kind, idempotency_key? }
        ↓
[Raft consensus]
   4. leader appends to log, replicates to quorum
   5. on quorum, state machine applies on every node:
        - state.queues[branch].push_back(QueueEntry { queue_id, envelope_cid, ... })
        - leader records waiter assignment
        ↓
[CommitWorker on leader]
   6. polls state.queues[branch].front()
   7. fetches envelope from CAS, stages via `Fluree` API
   8. writes commit blob to CAS → head_cid
   9. stashes AppliedReceipt in StagedReceiptMap[queue_id]
  10. propose Command::ApplyHead { branch, queue_id, head_cid, ... }
        ↓
[Raft consensus]
  11. leader appends, replicates to quorum
  12. on quorum, state machine applies on every node:
        - state.refs[branch].head = head_cid
        - state.queues[branch].pop_front()
        - leader: take StagedReceiptMap[queue_id] → resolve WaiterMap[queue_id] with receipt
        ↓
[QueuedTransactor on leader]
  13. rx returns receipt → return to client
        ↓
[follower-forward middleware (if forwarded)]
  14. relay response verbatim to client
```

Why two separate log entries (`EnqueueCommand` then `ApplyHead`) instead of one combined "apply this transaction"?

- The state machine must apply deterministically given the log entry alone. The output of staging (head CID, flake count, conflict outcome) depends on the leader's local state and CAS writes — neither is in the log. Putting the *result* on the log lets every node apply the same outcome without re-running the stage.
- An idempotent retry that lands while the original is still queued can be deduplicated at `EnqueueCommand` (the state machine sees the matching idempotency key and short-circuits) — without needing to re-stage.
- The queue is the unit of fairness. Multiple writers to the same branch land in a FIFO; the worker drains in order and `ApplyHead` references the `queue_id` it's draining. A racing admin operation that resets the branch head produces a `BranchHeadReset` poison record on the front entry rather than corrupting the queue.

### Receipts: replicated vs. process-local

Apply receipts come in two flavors:

- **`AppliedReceipt::Detailed { tally, conflict_outcome, ... }`** — the typed result of a successful stage, including flake counts, indexing status, and conflict resolution. Carried out-of-band in `StagedReceiptMap` on the leader and signaled to the local waiter. Not in the log.
- **`AppliedReceipt::Minimal { head, t }`** — head identity only. Synthesized on every node when applying `ApplyHead`, used by remote nodes and as a fallback on the leader when the staged receipt is missing (e.g. after a leader transition that stranded the receipt).

The split keeps the log encoding small (the heavy receipt fields don't replicate) while still giving the proposing client the rich result on the happy path. Followers that forwarded the original request see the leader's full response; followers reading the head independently see the minimal version.

## Log entry types

`Command` variants are grouped by purpose:

**Transaction flow:**

- `EnqueueCommand { branch, envelope_cid, body_hash, kind, idempotency_key? }` — append a queue entry.
- `ApplyHead { branch, queue_id, head_cid, ... }` — strict head advance, pop the queue front.
- `PoisonQueueEntry { branch, queue_id, reason }` — abandon a queued entry with a typed reason (`BranchHeadReset`, `BranchDropped`, `Poisoned { ... }`).
- `EvictIdempotency { cutoff_millis }` — age out idempotency cache entries older than the cutoff. Released CAS envelopes fan out per node.

**Ledger lifecycle:**

- `CreateLedger`, `CreateBranch`, `DropBranch`, `PurgeLedger`, `RetractLedger`, `ResetHead` — admin operations that mutate the registry. Each has companion entries to clear matching idempotency entries / queues.

**Metadata and refs:**

- `AdvanceIndexHead`, `RewriteIndexHead` — index pointer updates. `Advance` is strict-monotonic; `Rewrite` allows equal-or-different for admin rebinds.
- `CompareAndSetRef` — generic CAS over a named ref, gated by an expected-prior CID.
- `PushStatus`, `PushConfig`, `PublishGraphSource*` — peer / graph-source state.

The log can carry any of these; the state machine resolves them deterministically against the in-memory `NameServiceState`.

## Snapshot design

openraft snapshots the state machine every N entries (configured via `RaftConfig::snapshot_policy`). The adapter serializes `NameServiceState` via [postcard](https://github.com/jamesmunns/postcard) — chosen for its compact binary representation and stable schema-by-struct discipline — plus `last_applied` and `last_membership`.

Snapshots are stored as:

```
<raft_storage_path>/raft/snapshots/
   current             # plain-text snapshot id
   <id>.meta           # postcard-encoded SnapshotMeta
   <id>.data           # raw NameServiceState bytes
```

Atomic writes throughout: write to a temp file, fsync, rename, fsync the parent directory. A crash mid-snapshot leaves the previous snapshot intact via `current`.

On restart, `StateMachineAdapter::open` restores the latest snapshot first; openraft then replays the log from `last_applied + 1`. Without the restore step, log replay would have to start at index 0, which fails once the log has been compacted past it (i.e. always after the first snapshot).

Snapshot ids are validated against a path-traversal guard before any disk path is constructed: non-empty, ≤ 128 bytes, alphanumeric + `-_.` only, no `..`. This is enforced on receipt of an `install-snapshot` RPC — a malicious or buggy peer cannot push a snapshot id that would escape the snapshots subtree.

## Network transport

Inter-node RPC is plain HTTP over `reqwest`, with:

- `connect_timeout`: 250 ms (default)
- `rpc_timeout` (vote, append): 500 ms (default)
- `snapshot_timeout`: 30 s (default)
- Redirects disabled (SSRF guard against 302s to internal addresses).
- Per-route body size limits: vote 1 MiB, append-entries 64 MiB, install-snapshot 1 GiB.

These are exposed on `NetworkConfig` and can be overridden at integration time, though the server binary doesn't currently expose tuning knobs for them.

Why plain HTTP rather than gRPC or a custom protocol? Two reasons:

- The same axum router that hosts the inter-node port also hosts `/cluster/*` admin, which fits naturally in HTTP. Operators can curl the admin endpoints; tooling can intercept with standard HTTP proxies.
- openraft's `RaftNetworkFactory` trait abstracts the transport. HTTP is the lowest-overhead option that's portable across operator environments. Switching to gRPC later doesn't disturb the state machine.

## Why this design

A few alternatives we considered and rejected:

**Replicate commit blobs through the log.** Simplest design, but the log inflates with the size of every transaction. With a 64 MiB append-entries limit and large commits, the log churns through snapshots aggressively, and follower catch-up bandwidth scales with the total commit history rather than with the head set.

**Stage on every node (deterministic state machine apply).** Would let a follower verify the leader's work, but staging touches non-deterministic state: cache contents, wall-clock timestamps in indexing decisions, CAS write outcomes that depend on backend behavior. Forcing determinism would require reworking the entire commit path. The current design treats the leader as the source of truth for the *result* and replicates the result.

**Use openraft's built-in linearizable read API.** openraft offers a primitive for serving reads through the consensus log, but our reads have stricter latency requirements than that path provides. Reading directly from the locally-applied `NameServiceState` is much faster and is correct because the state machine applies under the same lock that the read path acquires.

**Single combined "apply commit" log entry.** Discussed under [Submission flow](#submission-flow-in-detail). The two-entry design (`EnqueueCommand` → `ApplyHead`) gives idempotent retries cheap deduplication and lets the staging work happen between the two replication rounds.

## See also

- [Raft clusters (replicated writes)](../operations/raft-clusters.md) — operator-facing recipe.
- [`fluree cluster` CLI](../cli/cluster.md) — admin subcommand reference.
- [Nameservice schema v2](nameservice-schema-v2.md) — the underlying nameservice model that the Raft variant replicates.
- [Storage-agnostic commits and sync](storage-agnostic-commits-and-sync.md) — the CAS layer the Raft state machine sits on top of.


---

# api/README.md

# HTTP API

The Fluree HTTP API provides RESTful endpoints for all database operations. This section documents the complete API surface including request formats, authentication, and error handling.

## Core Endpoints

### [Overview](overview.md)

High-level introduction to the Fluree HTTP API, including:
- API design principles
- Authentication overview
- Rate limiting and quotas
- API versioning

### [Endpoints](endpoints.md)

Complete reference for all HTTP endpoints:
- `POST /update` - Submit update transactions (WHERE/DELETE/INSERT or SPARQL UPDATE)
- `POST /query` - Execute queries
- `POST /multi-query` - Bundle multiple queries against a shared snapshot ([dedicated doc](multi-query.md))
- `GET /v1/fluree/ledgers` - List ledgers
- `GET /health` - Health checks
- `GET /v1/fluree/stats` - Server status
- And more...

### [Multi-query envelope](multi-query.md)

Bundle multiple JSON-LD and/or SPARQL queries into a single request that runs against one shared snapshot moment, with envelope-level `@context` / `opts` defaults and per-alias result assembly.

### [Headers, Content Types, and Request Sizing](headers.md)

HTTP headers and request format details:
- Content-Type negotiation
- Accept headers for response formats
- Request size limits
- Compression support
- Custom headers

### [Signed Requests (JWS/VC)](signed-requests.md)

Cryptographically signed and verifiable requests:
- JSON Web Signature (JWS) format
- Verifiable Credentials (VC) support
- Public key verification
- DID authentication
- Signature validation

### [Errors and Status Codes](errors.md)

HTTP status codes and error responses:
- Standard HTTP status codes
- Fluree-specific error codes
- Error response format
- Troubleshooting common errors

## API Characteristics

### RESTful Design

The Fluree API follows REST principles:
- Resource-oriented URLs
- Standard HTTP methods (GET, POST)
- Stateless requests
- Standard status codes

### Content Negotiation

Fluree supports multiple content types for requests and responses:

**Request Content-Types:**
- `application/json` - JSON-LD transactions and queries
- `application/sparql-query` - SPARQL queries
- `text/turtle` - Turtle RDF format
- `application/ld+json` - Explicit JSON-LD

**Response Content-Types:**
- `application/json` - Default JSON format
- `application/ld+json` - JSON-LD with context
- `application/sparql-results+json` - SPARQL result format

### Authentication

Fluree supports multiple authentication mechanisms:

1. **No Authentication** (development only)
2. **Signed Requests** (JWS/VC for production)
3. **API Keys** (simple token-based auth)
4. **Bearer Tokens** (JWT authentication)

See [Signed Requests](signed-requests.md) for cryptographic authentication details.

## Quick Examples

### Transaction Request

```bash
curl -X POST http://localhost:8090/v1/fluree/insert?ledger=mydb:main \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "ex": "http://example.org/ns/"
    },
    "@graph": [
      { "@id": "ex:alice", "ex:name": "Alice" }
    ]
  }'
```

### Query Request

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/json" \
  -d '{
    "from": "mydb:main",
    "select": ["?name"],
    "where": [
      { "@id": "?person", "ex:name": "?name" }
    ]
  }'
```

### SPARQL Query

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/sparql-query" \
  -d 'SELECT ?name FROM <mydb:main> WHERE { ?person ex:name ?name }'
```

### Health Check

```bash
curl http://localhost:8090/health
```

## API Clients

### Command Line (curl)

All examples in this documentation use `curl` for simplicity. Curl is available on all major platforms.

### Programming Languages

Fluree's HTTP API can be accessed from any language with HTTP client support:

**JavaScript/TypeScript:**
```javascript
const response = await fetch('http://localhost:8090/v1/fluree/query', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    from: 'mydb:main',
    select: ['?name'],
    where: [{ '@id': '?person', 'ex:name': '?name' }]
  })
});
const results = await response.json();
```

**Python:**
```python
import requests

response = requests.post('http://localhost:8090/v1/fluree/query', json={
    'from': 'mydb:main',
    'select': ['?name'],
    'where': [{'@id': '?person', 'ex:name': '?name'}]
})
results = response.json()
```

**Java:**
```java
HttpClient client = HttpClient.newHttpClient();
HttpRequest request = HttpRequest.newBuilder()
    .uri(URI.create("http://localhost:8090/v1/fluree/query"))
    .header("Content-Type", "application/json")
    .POST(HttpRequest.BodyPublishers.ofString(queryJson))
    .build();
HttpResponse<String> response = client.send(request, 
    HttpResponse.BodyHandlers.ofString());
```

## Development vs Production

### Development Setup

For local development, the API typically runs without authentication:

```bash
./fluree-db-server --port 8090 --storage memory
```

Access: `http://localhost:8090`

### Production Setup

For production deployments, enable authentication and use HTTPS:

```bash
./fluree-db-server \
  --port 8090 \
  --storage aws \
  --require-signed-requests \
  --https-cert /path/to/cert.pem \
  --https-key /path/to/key.pem
```

Access: `https://api.yourdomain.com`

Always use:
- HTTPS in production
- Signed requests or API keys
- Rate limiting
- Request size limits

## Performance Considerations

### Request Size Limits

Default limits (configurable):
- Transaction size: 10MB
- Query size: 1MB
- Response size: 100MB

See [Headers and Request Sizing](headers.md) for details.

### Connection Management

- Keep-alive connections supported
- HTTP/2 support available
- WebSocket support for streaming (planned)

### Caching

- Query results can be cached (ETag support)
- Immutable historical queries cache well
- Current queries should not be cached aggressively

## Related Documentation

- [Getting Started](../getting-started/README.md) - Quickstart guides
- [Transactions](../transactions/README.md) - Transaction details
- [Query](../query/README.md) - Query language documentation
- [Security](../security/README.md) - Policy and access control
- [Operations](../operations/README.md) - Configuration and deployment


---

# api/overview.md

# API Overview

The Fluree HTTP API provides a complete RESTful interface for database operations. This document provides a high-level overview of API design principles and capabilities.

## API Design Principles

### Resource-Oriented

The API is organized around resources:
- **Ledgers**: Database instances
- **Transactions**: Write operations
- **Queries**: Read operations
- **Commits**: Transaction history

### Standard HTTP Methods

Operations use standard HTTP methods:
- `GET` - Retrieve information (idempotent, cacheable)
- `POST` - Submit operations (transactions, queries)
- `PUT` - Update resources (planned)
- `DELETE` - Remove resources (planned)

### JSON-First

All request and response bodies use JSON by default:
- Native JSON-LD support
- Clean, readable syntax
- Easy integration with modern applications

### Stateless

All requests are stateless:
- No session management required
- Each request contains complete information
- Enables horizontal scaling

## Core Concepts

### Ledger Identification

Ledgers are identified using aliases with branch names:

```text
ledger-name:branch-name
```

Examples:
- `mydb:main` - Main branch of mydb ledger
- `customers:prod` - Production branch of customers ledger
- `tenant/app:dev` - Development branch with hierarchical naming

### Time Travel in URLs

Historical queries use time specifiers in ledger IDs:

```text
ledger:branch@t:100           # Transaction number
ledger:branch@iso:2024-01-15  # ISO timestamp
ledger:branch@commit:bafybeig...  # Commit ID
```

These work in all query contexts (FROM clauses, dataset specs, etc.).

### Content Type Negotiation

Request format determined by `Content-Type` header:
- `application/json` - JSON-LD (default)
- `application/sparql-query` - SPARQL
- `text/turtle` - Turtle RDF

Response format determined by `Accept` header:
- `application/json` - Compact JSON (default)
- `application/ld+json` - Full JSON-LD with context
- `application/sparql-results+json` - SPARQL result format

## API Endpoints

Except for root diagnostics such as `/health` and `/.well-known/fluree.json`,
HTTP API paths are under the discovered API base URL. The standalone server
defaults to `/v1/fluree`.

### Transaction Endpoints

**POST /update**
- Submit update transactions (WHERE/DELETE/INSERT JSON-LD or SPARQL UPDATE)
- Parameters: `ledger`, `context`
- Returns: Transaction receipt with commit info

**POST /insert** / **POST /upsert**
- Insert or upsert data (JSON-LD and Turtle; TriG on upsert)

### Query Endpoints

**POST /query**
- Execute queries (JSON-LD Query or SPARQL)
- Parameters: None (ledger specified in query body)
- Returns: Query results
- Supports history queries via time range in `from` clause (see [Time Travel](../concepts/time-travel.md))

### Ledger Management

**GET /ledgers**
- List all ledgers
- Parameters: None
- Returns: Array of ledger metadata

**GET /info/:ledger-id**
- Get specific ledger metadata
- Parameters: `ledger-id` (ledger:branch)
- Returns: Ledger details (commit_t, index_t, etc.)

**POST /create**
- Create a new ledger explicitly
- Parameters: `ledger`
- Returns: Ledger metadata

### System Endpoints

**GET /health**
- Health check endpoint
- Parameters: None
- Returns: Server health status

**GET /stats**
- Server status and statistics
- Parameters: None
- Returns: Detailed server state

## Request Format

### URL Structure

```text
https://[host]:[port]/[endpoint]?[parameters]
```

Example:
```text
http://localhost:8090/v1/fluree/update?ledger=mydb:main
```

### Query Parameters

Common parameters:
- `ledger` - Target ledger (format: `name:branch`)
- `context` - Default context URL
- `format` - Response format override

### Request Headers

Essential headers:
```http
Content-Type: application/json
Accept: application/json
Authorization: Bearer [token]
```

See [Headers](headers.md) for complete list.

### Request Body

JSON-LD format for transactions:

```json
{
  "@context": {
    "ex": "http://example.org/ns/"
  },
  "@graph": [
    { "@id": "ex:alice", "ex:name": "Alice" }
  ]
}
```

JSON-LD Query format:

```json
{
  "@context": {
    "ex": "http://example.org/ns/"
  },
  "from": "mydb:main",
  "select": ["?name"],
  "where": [
    { "@id": "?person", "ex:name": "?name" }
  ]
}
```

## Response Format

### Success Response

Successful operations return appropriate status codes with JSON bodies.

**Transaction Response:**
```json
{
  "t": 5,
  "timestamp": "2024-01-22T10:30:00.000Z",
  "commit_id": "bafybeig...commitT5",
  "flakes_added": 3,
  "flakes_retracted": 1
}
```

**Query Response:**
```json
[
  { "name": "Alice" },
  { "name": "Bob" }
]
```

### Error Response

Errors return appropriate HTTP status codes with structured error objects:

```json
{
  "error": "Invalid IRI: not a valid URI",
  "status": 400,
  "@type": "err:db/BadRequest"
}
```

See [Errors and Status Codes](errors.md) for complete error reference.

## Authentication

Fluree supports multiple authentication mechanisms, configured per endpoint group (data, events, admin, storage proxy). Each can be set to `none`, `optional`, or `required`. See [Configuration](../operations/configuration.md) for full details.

### Development Mode

No authentication required (default):

```bash
curl http://localhost:8090/v1/fluree/query/mydb:main \
  -H "Content-Type: application/json" \
  -d '{"select": ["?s"], "where": [{"@id": "?s"}]}'
```

### Bearer Token Authentication

Bearer tokens in the `Authorization` header. Fluree supports two token types with automatic dual-path dispatch:

**Ed25519 JWS (did:key)** - Locally minted tokens with an embedded JWK. Created with `fluree token create`:

```bash
TOKEN=$(fluree token create --private-key @~/.fluree/key --read-all --write-all)

curl http://localhost:8090/v1/fluree/query/mydb:main \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"select": ["?s"], "where": [{"@id": "?s"}]}'
```

**OIDC/JWKS (RS256)** - Tokens from external identity providers, verified against the provider's JWKS endpoint. Requires the `oidc` feature and `--jwks-issuer` server configuration:

```bash
curl http://localhost:8090/v1/fluree/query/mydb:main \
  -H "Authorization: Bearer <oidc-token>" \
  -H "Content-Type: application/json" \
  -d '{"select": ["?s"], "where": [{"@id": "?s"}]}'
```

The server inspects the token header to determine the verification path:
- **Embedded JWK** (Ed25519): Verifies against the embedded public key; issuer is a `did:key`
- **kid header** (RS256): Verifies against the issuer's JWKS endpoint

#### Token Scopes

Bearer tokens carry permission scopes that control access:

- **Read**: `fluree.ledger.read.all=true` or `fluree.ledger.read.ledgers=[...]`
- **Write**: `fluree.ledger.write.all=true` or `fluree.ledger.write.ledgers=[...]`
- **Back-compat**: `fluree.storage.*` claims also imply read access for data endpoints

#### Connection-Scoped SPARQL

When a bearer token is present for connection-scoped SPARQL queries (`/v1/fluree/query` with `Content-Type: application/sparql-query`), FROM/FROM NAMED clauses are checked against the token's read scope (`fluree.ledger.read.all` or `fluree.ledger.read.ledgers`). Out-of-scope ledgers return 404 (no existence leak).

### Signed Requests (JWS/VC)

Cryptographically signed request bodies using Ed25519 JWS or Verifiable Credentials. The signed payload carries the request itself plus the signer's identity for policy evaluation.

```bash
curl http://localhost:8090/v1/fluree/query/mydb:main \
  -H "Content-Type: application/jose" \
  -d '<compact-jws-string>'
```

See [Signed Requests](signed-requests.md) for detailed documentation.

## Rate Limiting

### Default Limits

Production deployments should implement rate limiting:
- Queries: 100 requests per minute
- Transactions: 10 requests per minute
- History: 50 requests per minute

### Rate Limit Headers

Responses include rate limit information:

```http
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 95
X-RateLimit-Reset: 1642857600
```

### Exceeding Limits

When limits are exceeded:
- Status code: `429 Too Many Requests`
- Response body includes retry information
- `Retry-After` header indicates wait time

## API Versioning

### Current Version

The current API is version 1 (v1).

### Version in URL (Future)

Future versions may use URL-based versioning:

```text
https://api.example.com/v2/query
```

## Common Patterns

### Idempotent Transactions

Use the upsert endpoint for idempotent transactions:

```bash
curl -X POST "http://localhost:8090/v1/fluree/upsert?ledger=mydb:main" \
  -H "Content-Type: application/json" \
  -d '{...}'
```

### Batch Operations

Submit multiple entities in a single transaction:

```json
{
  "@graph": [
    { "@id": "ex:alice", "ex:name": "Alice" },
    { "@id": "ex:bob", "ex:name": "Bob" },
    { "@id": "ex:carol", "ex:name": "Carol" }
  ]
}
```

### Conditional Updates

Use WHERE/DELETE/INSERT for conditional changes:

```json
{
  "where": [
    { "@id": "ex:alice", "ex:age": "?oldAge" }
  ],
  "delete": [
    { "@id": "ex:alice", "ex:age": "?oldAge" }
  ],
  "insert": [
    { "@id": "ex:alice", "ex:age": 31 }
  ]
}
```

### Historical Queries

Query past states using time specifiers:

```json
{
  "from": "mydb:main@t:100",
  "select": ["?name"],
  "where": [
    { "@id": "?person", "ex:name": "?name" }
  ]
}
```

## Best Practices

### 1. Use Appropriate HTTP Methods

- GET for read-only operations (health, status)
- POST for write and query operations

### 2. Set Correct Content-Type

Always specify the request format:

```http
Content-Type: application/json
```

### 3. Handle Errors Gracefully

Check status codes and parse error responses:

```javascript
if (response.status !== 200) {
  const error = await response.json();
  console.error(`Error ${error.code}: ${error.message}`);
}
```

### 4. Use Connection Pooling

Reuse HTTP connections for better performance:

```javascript
const agent = new https.Agent({ keepAlive: true });
```

### 5. Implement Retry Logic

Retry failed requests with exponential backoff:

```javascript
async function retryRequest(fn, maxRetries = 3) {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await fn();
    } catch (err) {
      if (i === maxRetries - 1) throw err;
      await sleep(Math.pow(2, i) * 1000);
    }
  }
}
```

### 6. Monitor Rate Limits

Track rate limit headers and back off when approaching limits.

### 7. Use Compression

Enable compression for large payloads:

```http
Accept-Encoding: gzip, deflate
```

## Security Considerations

### HTTPS in Production

Always use HTTPS in production:
- Prevents eavesdropping
- Protects credentials
- Enables trust

### Validate Input

Validate all user input before sending to API:
- Check IRI formats
- Validate JSON structure
- Sanitize user data

### Secure Credentials

Never expose credentials in code or logs:
- Use environment variables
- Rotate keys regularly
- Use signed requests for highest security

### Implement CORS Carefully

If exposing API to web applications, configure CORS appropriately:

```http
Access-Control-Allow-Origin: https://your-app.com
Access-Control-Allow-Methods: POST, GET
Access-Control-Allow-Headers: Content-Type, Authorization
```

## Performance Tips

### 1. Batch Related Operations

Combine related entities in single transactions for better performance.

### 2. Use Appropriate Time Specifiers

- `@t:NNN` is fastest (direct lookup)
- `@iso:DATETIME` requires binary search
- `@commit:CID` requires scan

### 3. Limit Result Sets

Always use LIMIT for potentially large result sets:

```json
{
  "select": ["?name"],
  "where": [...],
  "limit": 100
}
```

### 4. Cache Historical Queries

Historical queries (with time specifiers) are immutable and cache well.

### 5. Use Streaming for Large Results

For very large result sets, consider streaming responses (when supported).

## Related Documentation

- [Endpoints](endpoints.md) - Complete endpoint reference
- [Headers](headers.md) - HTTP headers and content types
- [Signed Requests](signed-requests.md) - Cryptographic authentication
- [Errors](errors.md) - Error codes and troubleshooting


---

# api/endpoints.md

# API Endpoints

Complete reference for all Fluree HTTP API endpoints.

## Base URL / versioning

All endpoints listed below are under the server’s **API base URL** (`api_base_url` from `GET /.well-known/fluree.json`).

- Standalone `fluree-server` default: `api_base_url = "/v1/fluree"`
- All curl examples in this document use the full URL including the base path (e.g., `http://localhost:8090/v1/fluree/query/<ledger...>`)

## Discovery and diagnostics

### GET /.well-known/fluree.json

CLI auth discovery endpoint. Used by `fluree remote add` and `fluree auth login` to auto-configure authentication for a remote.

See [Auth contract (CLI ↔ Server)](../design/auth-contract.md) for the full schema.

Standalone `fluree-server` returns:

- `{"version":1,"api_base_url":"/v1/fluree"}` when no server auth is enabled
- `{"version":1,"api_base_url":"/v1/fluree","auth":{"type":"token"}}` when any server auth mode is enabled (data/events/admin)

OIDC-capable implementations should return `auth.type="oidc_device"` plus `issuer`, `client_id`, and `exchange_url`.
The CLI treats `oidc_device` as "OIDC interactive login": it uses device-code when the IdP supports it, otherwise authorization-code + PKCE.

Implementations MAY also return `api_base_url` to tell the CLI where the Fluree API is mounted (for example,
when the API is hosted under `/v1/fluree` or on a separate `data` subdomain).

Implementations MAY also return an `import` block advertising `.flpack` import capabilities (`modes` incl. `multipart-put`, `direct_max_bytes`, multipart hints) so the CLI can negotiate the upload path — see [Negotiated upload import](#negotiated-upload-import-import-upload).

### GET {api_base_url}/whoami

Diagnostic endpoint for Bearer tokens. Returns a summary of the principal:

- `token_present`: whether a Bearer token was present
- `verified`: whether cryptographic verification succeeded
- `auth_method`: `"embedded_jwk"` (Ed25519) or `"oidc"` (JWKS/RS256)
- identity + scope summary (when verified)

This endpoint is intended for debugging and operator support. See also [Admin, health, and stats](../operations/admin-and-health.md).

## Transaction Endpoints

### POST /update

Submit an **update** transaction (WHERE/DELETE/INSERT JSON-LD or SPARQL UPDATE) to write data to a ledger.

**URL:**
```
POST /update?ledger={ledger-id}
POST /update/{ledger-id}
```

**Query Parameters:**
- `ledger` (required for /update): Target ledger (format: `name:branch`)
- `context` (optional): URL to default JSON-LD context

**Request Headers:**

For JSON-LD transactions:
```http
Content-Type: application/json
Accept: application/json
```

For SPARQL UPDATE:
```http
Content-Type: application/sparql-update
Accept: application/json
```

Note: Turtle/TriG are not accepted on `/update`. Use `/insert` (Turtle) or `/upsert` (Turtle/TriG).

**Request Body (JSON-LD):**

JSON-LD transaction document:
```json
{
  "@context": {
    "ex": "http://example.org/ns/"
  },
  "@graph": [
    { "@id": "ex:alice", "ex:name": "Alice" }
  ]
}
```

Or WHERE/DELETE/INSERT update:
```json
{
  "@context": {
    "ex": "http://example.org/ns/"
  },
  "where": [
    { "@id": "ex:alice", "ex:age": "?oldAge" }
  ],
  "delete": [
    { "@id": "ex:alice", "ex:age": "?oldAge" }
  ],
  "insert": [
    { "@id": "ex:alice", "ex:age": 31 }
  ]
}
```

**Request Body (SPARQL UPDATE):**

```sparql
PREFIX ex: <http://example.org/ns/>

INSERT DATA {
  ex:alice ex:name "Alice" .
  ex:alice ex:age 30 .
}
```

Or with DELETE/INSERT:

```sparql
PREFIX ex: <http://example.org/ns/>

DELETE {
  ?person ex:age ?oldAge .
}
INSERT {
  ?person ex:age 31 .
}
WHERE {
  ?person ex:name "Alice" .
  ?person ex:age ?oldAge .
}
```

`INSERT DATA` and `DELETE DATA` accept named-graph blocks per SPARQL 1.1 Update
§3.1.1 (`QuadData`). A `GRAPH <iri> { ... }` block routes its (ground) triples
into the named graph; this is what RDF4J's `SPARQLConnection.add(stmts, context)`
emits:

```sparql
INSERT DATA {
  GRAPH <https://example.org/g/1> {
    <https://example.org/s/1> <https://example.org/p> "v" .
  }
}
```

The graph name must be a fixed IRI — a variable graph name (`GRAPH ?g { ... }`)
is rejected, since DATA must be ground. Use DELETE/INSERT … WHERE for variable
graph targets.

**Response:**

```json
{
  "t": 5,
  "timestamp": "2024-01-22T10:30:00.000Z",
  "commit_id": "bafybeig...commitT5",
  "flakes_added": 3,
  "flakes_retracted": 1,
  "previous_commit_id": "bafybeig...commitT4"
}
```

**Status Codes:**
- `200 OK` - Transaction successful
- `400 Bad Request` - Invalid transaction syntax
- `401 Unauthorized` - Authentication required
- `403 Forbidden` - Not authorized for this ledger
- `404 Not Found` - Ledger not found
- `409 Conflict` - Optimistic-concurrency conflict that survived the server's
  bounded reconcile-and-retry (rare; safe to retry the request)
- `413 Payload Too Large` - Transaction exceeds size limit
- `500 Internal Server Error` - Server error

**Concurrency:** Writes to a single ledger are serialized by a per-ledger write
lock (concurrent writes to *different* ledgers proceed in parallel). When a
transaction is lowered/sequenced against a snapshot that is no longer the head
by commit time — two writers racing on a first-time namespace code, or a cached
writer state that fell behind the durable head — the server reconciles the
cached state to the current head and re-tries (bounded, up to 16 attempts). A
`409 Conflict` is returned only if the retry budget is exhausted; clients should
treat it as retryable (distinct from a `400` bad request).

**Examples:**

JSON-LD transaction:
```bash
curl -X POST "http://localhost:8090/v1/fluree/update?ledger=mydb:main" \
  -H "Content-Type: application/json" \
  -d '{
    "@context": { "ex": "http://example.org/ns/" },
    "@graph": [{ "@id": "ex:alice", "ex:name": "Alice" }]
  }'
```

SPARQL UPDATE (ledger-scoped endpoint):
```bash
curl -X POST http://localhost:8090/v1/fluree/update/mydb:main \
  -H "Content-Type: application/sparql-update" \
  -d 'PREFIX ex: <http://example.org/ns/>
      INSERT DATA { ex:alice ex:name "Alice" }'
```

SPARQL UPDATE (connection-scoped with header):
```bash
curl -X POST http://localhost:8090/v1/fluree/update \
  -H "Content-Type: application/sparql-update" \
  -H "Fluree-Ledger: mydb:main" \
  -d 'PREFIX ex: <http://example.org/ns/>
      DELETE { ?s ex:age ?old } INSERT { ?s ex:age 31 }
      WHERE { ?s ex:name "Alice" . ?s ex:age ?old }'
```

Note: Turtle and TriG are not accepted on `/update`. Use `/insert` (Turtle) or `/upsert` (Turtle/TriG).

### POST /insert

Insert new data into a ledger. Data must not conflict with existing data.

**URL:**
```
POST /insert?ledger={ledger-id}
POST /insert/{ledger-id}
```

**Supported Content Types:**
- `application/json` - JSON-LD
- `text/turtle` - Turtle (fast direct flake path)
- `application/n-triples` - N-Triples (parsed as Turtle)

**Note:** TriG (`application/trig`) is **not supported** on the insert endpoint. Named graph ingestion via GRAPH blocks requires the upsert path. Use `/upsert` for TriG data.

**Example (JSON-LD):**
```bash
curl -X POST "http://localhost:8090/v1/fluree/insert?ledger=mydb:main" \
  -H "Content-Type: application/json" \
  -d '{
    "@context": { "ex": "http://example.org/ns/" },
    "@graph": [{ "@id": "ex:alice", "ex:name": "Alice" }]
  }'
```

**Example (Turtle):**
```bash
curl -X POST "http://localhost:8090/v1/fluree/insert?ledger=mydb:main" \
  -H "Content-Type: text/turtle" \
  -d '@prefix ex: <http://example.org/ns/> .
      ex:alice ex:name "Alice" ; ex:age 30 .'
```

### POST /upsert

Upsert data into a ledger. For each (subject, predicate) pair, existing values are retracted before new values are asserted.

**URL:**
```
POST /upsert?ledger={ledger-id}
POST /upsert/{ledger-id}
```

**Supported Content Types:**
- `application/json` - JSON-LD
- `text/turtle` - Turtle
- `application/n-triples` - N-Triples (parsed as Turtle)
- `application/trig` - TriG with named graphs

**Example (JSON-LD):**
```bash
curl -X POST "http://localhost:8090/v1/fluree/upsert?ledger=mydb:main" \
  -H "Content-Type: application/json" \
  -d '{
    "@context": { "ex": "http://example.org/ns/" },
    "@id": "ex:alice",
    "ex:age": 31
  }'
```

**Example (TriG with named graphs):**
```bash
curl -X POST "http://localhost:8090/v1/fluree/upsert?ledger=mydb:main" \
  -H "Content-Type: application/trig" \
  -d '@prefix ex: <http://example.org/ns/> .

      # Default graph
      ex:company ex:name "Acme Corp" .

      # Named graph for products
      GRAPH <http://example.org/graphs/products> {
          ex:widget ex:name "Widget" ;
                    ex:price "29.99"^^xsd:decimal .
      }'
```

Both W3C TriG graph-block forms are accepted: the SPARQL-style keyword form
`GRAPH <iri> { ... }` shown above and the compact form `<iri> { ... }` (the
`GRAPH` keyword is optional per the [TriG grammar](https://www.w3.org/TR/trig/#sec-graph)).
The compact form is what stock RDF tooling — rdflib, Apache Jena, RDF4J — emits
by default, so payloads generated by those libraries are ingested as-is.

### POST /push/*ledger

Push precomputed commit v2 blobs to the server.

This endpoint is intended for Git-like workflows (`fluree push`) where a client has written commits locally and wants the server to validate and commit them.

**URL:**

```
POST /push/<ledger...>
```

**Request Headers:**

```http
Content-Type: application/json
Accept: application/json
Authorization: Bearer <token>
Idempotency-Key: <string>   (optional; recommended)
```

If `Idempotency-Key` is provided, servers MAY treat `POST /push/*ledger` as idempotent for that key (same request body + key should yield the same response), returning the prior success response instead of `409` on client retry after timeouts.

**Request Body:**

JSON object:

- `commits`: array of base64-encoded commit v2 blobs (oldest → newest)
- `blobs` (optional): map of `{ cid: base64Bytes }` for referenced blobs (currently: `commit.txn` when present)

**Response Body (200 OK):**

```json
{
  "ledger": "mydb:main",
  "accepted": 3,
  "head": {
    "t": 42,
    "commit_id": "bafy...headCommit"
  },
  "indexing": {
    "enabled": false,
    "needed": true,
    "novelty_size": 524288,
    "index_t": 30,
    "commit_t": 42
  }
}
```

| Field | Description |
|-------|-------------|
| `indexing.enabled` | Whether background indexing is active on this server. |
| `indexing.needed` | Whether novelty has exceeded `reindex_min_bytes` and indexing should be triggered. |
| `indexing.novelty_size` | Current novelty size in bytes after the push. |
| `indexing.index_t` | Transaction time of the last indexed state. |
| `indexing.commit_t` | Transaction time of the latest committed data (after push). |

When `enabled` is `false` (external indexer mode), the caller should use `needed` and related fields to decide whether to trigger indexing through its own mechanism.

**Error Responses:**

- `409 Conflict`: head changed / diverged / first commit `t` did not match next-t
- `422 Unprocessable Entity`: invalid commit bytes, missing referenced blob, or retraction invariant violation

### GET /show/*ledger

Fetch and decode a single commit's contents with resolved IRIs. This is the server-side equivalent of `fluree show` — it returns assertions, retractions, and flake tuples with IRIs compacted using the ledger's namespace prefix table.

**URL:**

```
GET /show/<ledger...>?commit=<ref>
```

**Query Parameters:**

- `commit` (required): Commit identifier — `t:<N>` for transaction number, hex-digest prefix (min 6 chars), or full CID

**Request Headers:**

```http
Authorization: Bearer <token>   (when data auth is enabled)
```

**Response Body (200 OK):**

```json
{
  "id": "bagaybqabciq...",
  "t": 5,
  "time": "2026-03-12T16:58:18.395474217+00:00",
  "size": 327,
  "previous": "bagaybqabciq...",
  "asserts": 1,
  "retracts": 1,
  "@context": {
    "xsd": "http://www.w3.org/2001/XMLSchema#",
    "schema": "http://schema.org/"
  },
  "flakes": [
    ["urn:fsys:dataset:zoho3", "schema:dateModified", "2026-03-12T14:15:30Z", "xsd:string", false],
    ["urn:fsys:dataset:zoho3", "schema:dateModified", "2026-03-12T16:58:16Z", "xsd:string", true]
  ]
}
```

Each flake is a tuple: `[subject, predicate, object, datatype, operation]`. Operation `true` = assert (added), `false` = retract (removed). When metadata is present (language tag, list index, or named graph), a 6th element is appended.

**Policy filtering:** Flakes are filtered by the caller's data-auth identity (extracted from the Bearer token) and the server's configured `default_policy_class`. When neither is present, all flakes are returned (root/admin access). Flakes the caller cannot read are silently omitted — the `asserts` and `retracts` counts reflect only the visible flakes. Unlike the query endpoints, show does not accept per-request policy overrides via headers or request body.

**Responses:**

- `200 OK`: Decoded commit returned
- `400 Bad Request`: Missing or invalid `commit` parameter
- `401 Unauthorized`: Bearer token required but missing
- `404 Not Found`: Ledger or commit not found
- `501 Not Implemented`: Proxy storage mode (no local index available)

**Peer mode:** Forwards to the transactor.

### GET /log/*ledger

Return a paginated list of lightweight commit summaries (newest-first by `t`). Server-side equivalent of `fluree log`. Read-auth — does **not** require storage-replication permissions, unlike `/commits`.

**URL:**

```
GET /log/<ledger...>?limit=<N>
```

**Query Parameters:**

- `limit` (optional, default `100`): Number of summaries to return. Server clamps to a hard maximum (reference: `5000`).

**Request Headers:**

```http
Authorization: Bearer <token>   (when data auth is enabled)
```

**Response Body (200 OK):**

```json
{
  "ledger_id": "mydb:main",
  "commits": [
    {
      "t": 12,
      "commit_id": "bafy...",
      "time": "2026-04-25T12:00:00Z",
      "asserts": 3,
      "retracts": 0,
      "flake_count": 3,
      "message": null
    }
  ],
  "count": 12,
  "truncated": false
}
```

`commits` is strictly newest-first by `t` and capped by `limit`. `count` is the full chain length; `truncated == count > commits.len()`. `message` is extracted from `txn_meta` when an `f:message` entry with a string value is present, otherwise `null`. Each summary mirrors `fluree_db_core::CommitSummary`.

**Branch-aware walk:** The walk loads commit envelopes via a branch-aware content store so it can cross fork points — pre-fork commits live under the source branch's namespace.

**Responses:**

- `200 OK`: Summaries returned (possibly empty array when the ledger has no commits)
- `401 Unauthorized`: Bearer token required but missing
- `404 Not Found`: Ledger does not exist; or the bearer cannot `can_read`
- `5xx`: Storage / nameservice errors during walk

**Peer mode:** Forwards to the transactor.

### GET /commits/*ledger

Export commit blobs from a ledger using stable cursors. Pages walk backward via each commit's `parents` — O(limit) per page regardless of ledger size. Used by `fluree pull` and `fluree clone`.

**Requires replication-grade permissions** (`fluree.storage.*`). The storage proxy must be enabled on the server.

**URL:**

```
GET /commits/<ledger...>?limit=100&cursor_id=<cid>
```

**Query Parameters:**

- `limit` (optional): Max commits per page (default 100, server clamps to max 500)
- `cursor_id` (optional): Commit CID cursor for pagination. Omit for first page (starts from head). Use `next_cursor_id` from the previous response for subsequent pages.

**Request Headers:**

```http
Authorization: Bearer <token>   (requires fluree.storage.* claims)
```

**Response Body (200 OK):**

```json
{
  "ledger": "mydb:main",
  "head_commit_id": "bafy...headCommit",
  "head_t": 42,
  "commits": ["<base64>", "<base64>"],
  "blobs": { "bafy...txnBlob": "<base64>" },
  "newest_t": 42,
  "oldest_t": 41,
  "next_cursor_id": "bafy...prevCommit",
  "count": 2,
  "effective_limit": 100
}
```

- `commits`: Raw commit v2 blobs, newest → oldest within each page.
- `blobs`: Referenced txn blobs keyed by CID string.
- `next_cursor_id`: CID cursor for the next page; `null` when genesis is reached.
- `effective_limit`: Actual limit used (after server clamping).

**Responses:**

- `200 OK`: Page of commits returned
- `401 Unauthorized`: Missing or invalid storage token
- `404 Not Found`: Storage proxy not enabled, ledger not found, or not authorized for this ledger

**Pagination:**

Commit CIDs in the immutable chain are stable cursors. New commits appended to the head do not affect backward pointers, so cursors remain valid across pages even when new commits arrive between requests.

### POST /pack/*ledger

Stream all missing CAS objects for a ledger in a single binary response. This is the primary transport for `fluree clone` and `fluree pull`, replacing multiple paginated `GET /commits` requests or per-object `GET /storage/objects` fetches with a single streaming request.

**Requires replication-grade permissions** (`fluree.storage.*`). The storage proxy must be enabled on the server.

**URL:**

```
POST /pack/<ledger...>
```

**Request Headers:**

```http
Content-Type: application/json
Accept: application/x-fluree-pack
Authorization: Bearer <token>   (requires fluree.storage.* claims)
```

**Request Body:**

```json
{
  "protocol": "fluree-pack-v1",
  "want": ["bafy...remoteHead"],
  "have": ["bafy...localHead"],
  "include_indexes": true,
  "include_txns": true,
  "want_index_root_id": "bafy...indexRoot",
  "have_index_root_id": "bafy...localIndexRoot"
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `protocol` | string | Yes | Must be `"fluree-pack-v1"` |
| `want` | string[] | Yes | ContentId CIDs the client wants (typically the remote commit head) |
| `have` | string[] | No | ContentId CIDs the client already has (typically the local commit head). Server stops walking the commit chain when it reaches a `have` CID. Empty for full clone. |
| `want_index_root_id` | string | No | Index root CID the client wants (typically remote nameservice `index_head_id`). Required when `include_indexes=true`. |
| `have_index_root_id` | string | No | Index root CID the client already has (typically local nameservice `index_head_id`). Used for index artifact diff. |
| `include_indexes` | bool | Yes | Include index artifacts in the stream. When true, the stream contains commit + txn objects plus index root/branch/leaf/dict artifacts. |
| `include_txns` | bool | Yes | Include original transaction blobs referenced by each commit. When false, only commits (and optionally index artifacts) are streamed — commit envelopes still reference their `txn` CIDs, but the client will not have the transaction payloads locally. The ledger state is fully reconstructable from commits + indexes; transactions are the original request payloads (e.g., JSON-LD insert/update requests). |

**Response:**

Binary stream using the `fluree-pack-v1` wire format (`Content-Type: application/x-fluree-pack`):

```
[Preamble: FPK1 + version(1)] [Header frame] [Data frames...] [End frame]
```

| Frame | Type byte | Content |
|-------|-----------|---------|
| Header | `0x00` | JSON metadata: protocol version, capabilities, `commit_count`, `index_artifact_count`, `estimated_total_bytes` |
| Data | `0x01` | CID binary + raw object bytes (commit, txn blob, or index artifact) |
| Error | `0x02` | UTF-8 error message (terminates stream) |
| Manifest | `0x03` | JSON metadata for phase transitions (e.g. start of index phase) |
| End | `0xFF` | End of stream (no payload) |

Data frames are streamed in **oldest-first topological order** (parents before children), so the client can write objects to CAS as they arrive without buffering the entire stream.

The Header frame includes an `estimated_total_bytes` field that the CLI uses to warn users before large transfers (~1 GiB or more). The estimate is ratio-based (derived from commit count) and may differ from actual transfer size. Set to `0` for commits-only requests.

**Status Codes:**

- `200 OK`: Binary pack stream
- `401 Unauthorized`: Missing or invalid storage token
- `404 Not Found`: Storage proxy not enabled, ledger not found, or not authorized for this ledger

**Example:**

```bash
# Download all commits for a ledger (full clone)
curl -X POST "http://localhost:8090/v1/fluree/pack/mydb:main" \
  -H "Content-Type: application/json" \
  -H "Accept: application/x-fluree-pack" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"protocol":"fluree-pack-v1","want":["bafy...head"],"have":[],"include_indexes":false,"include_txns":true}' \
  --output pack.bin

# Download commits without transaction payloads (smaller clone, read-only use)
curl -X POST "http://localhost:8090/v1/fluree/pack/mydb:main" \
  -H "Content-Type: application/json" \
  -H "Accept: application/x-fluree-pack" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"protocol":"fluree-pack-v1","want":["bafy...head"],"have":[],"include_indexes":true,"include_txns":false,"want_index_root_id":"bafy...indexRoot"}' \
  --output pack.bin

# Download only missing commits (incremental pull)
curl -X POST "http://localhost:8090/v1/fluree/pack/mydb:main" \
  -H "Content-Type: application/json" \
  -H "Accept: application/x-fluree-pack" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"protocol":"fluree-pack-v1","want":["bafy...remoteHead"],"have":["bafy...localHead"],"include_indexes":false,"include_txns":true}' \
  --output pack.bin

# Download commits + index artifacts (default for CLI pull/clone)
curl -X POST "http://localhost:8090/v1/fluree/pack/mydb:main" \
  -H "Content-Type: application/json" \
  -H "Accept: application/x-fluree-pack" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"protocol":"fluree-pack-v1","want":["bafy...head"],"have":[],"include_indexes":true,"include_txns":true,"want_index_root_id":"bafy...indexRoot"}' \
  --output pack.bin
```

### POST /import/*ledger

Create a **new** ledger by restoring a `.flpack` archive — the inbound
counterpart of `POST /pack/*ledger`. The request body is the raw `.flpack`
stream (as produced by `fluree export <ledger> --format ledger` or
`Fluree::archive_ledger`). The server streams the archive into storage — commits,
transaction blobs, and any prebuilt index artifacts — then finalizes the commit
and index heads from the archive's embedded nameservice manifest.

The new ledger is named by the URL path and is **independent of the source
ledger's name**, so the same archive can be restored under any name. Unlike
`/push`, the archive is trusted byte-for-byte (every frame is SHA-256 verified)
and not replayed, and the prebuilt index rides along — so the restored ledger is
immediately queryable with no reindex.

**Admin-protected** (same bracket as `/create` and `/drop`): the body carries
prebuilt index artifacts the server did not produce, so this is an admin-grade
operation. The archive is decoded frame-by-frame and never buffered whole, so
multi-gigabyte archives restore without exhausting server memory.

**URL:**

```
POST /import/<ledger...>
```

**Request Headers:**

```http
Content-Type: application/x-fluree-pack
Authorization: Bearer <token>   (admin token when configured)
```

**Request Body:** the raw `.flpack` byte stream.

**Response:** `201 Created` with a JSON summary:

```json
{
  "ledger_id": "restored-db:main",
  "commits": 12,
  "txn_blobs": 12,
  "index_artifacts": 34,
  "commit_t": 12,
  "index_t": 12
}
```

`index_artifacts` is `0` and `index_t` is omitted for a commits-only archive
(exported with `--no-indexes`); such a ledger replays from commits on first load.

**Status codes:**

- `201 Created`: ledger restored
- `400 Bad Request`: malformed archive (bad preamble/frame, missing manifest, or a manifest head CID not present in the archive)
- `409 Conflict`: a ledger with that name already exists
- `401 Unauthorized`: missing or invalid admin token

On any mid-stream failure the partially-created ledger is rolled back, so a
failed import never leaves a live, half-ingested ledger behind.

**Example:**

```bash
# Restore an archive into a brand-new ledger named "restored-db:main"
curl -X POST "http://localhost:8090/v1/fluree/import/restored-db:main" \
  -H "Content-Type: application/x-fluree-pack" \
  -H "Authorization: Bearer $TOKEN" \
  --data-binary @mydb.flpack
```

This is the transport behind `fluree create restored-db --remote origin --from mydb.flpack`.

### Negotiated upload import (`/import-upload`)

For clients that cannot send a large body to `POST /import` (e.g. behind a payload-capped gateway), an optional out-of-band upload handshake lets the client upload the `.flpack` directly to object storage, then have the server restore from it asynchronously. Servers advertise support in discovery (`GET /.well-known/fluree.json`):

```jsonc
"import": {
  "modes": ["direct", "presigned-put", "multipart-put"],  // negotiated modes
  "direct_max_bytes": 6291456,             // archives larger than this negotiate
  "multipart_threshold_bytes": 5368709120, // ≥ this size → multipart (5 GiB PUT cap)
  "multipart_part_size_bytes": 268435456   // target part size hint (≥ 5 MiB for S3)
}
```

| Step | Endpoint | Result |
|------|----------|--------|
| Mint | `POST /import-upload` `{ledger, size?}` | single: `{ import_id, upload: { method, url, headers, expires_at_unix } }` — or multipart (when `size ≥ threshold`): `{ import_id, multipart: { upload_id, part_size_bytes, parts:[{part_number,url,headers}], expires_at_unix } }` |
| Upload | `PUT <upload.url>` (single) **or** `PUT <part.url>` per part (multipart) | bytes staged (direct to object storage; **no bearer auth** — the URL is the capability). Each part PUT returns its `ETag`. |
| Complete | `POST /import-upload/{import_id}/complete` | body: empty (single) or `{ parts:[{part_number, etag}] }` (multipart) → `202` `{ import_id, status:"running" }` — restore begins asynchronously |
| Status | `GET /import-upload/{import_id}` | `{ status, result?, error? }`, `status ∈ {awaiting-upload, running, succeeded, failed}` |

Mint / complete / status are **admin-protected**; the blob/part `PUT`s are token-authorized via the minted URL. The server picks single-PUT vs multipart by the declared `size` (a single S3 PUT caps at 5 GiB). On `succeeded`, `result` is the same summary as `POST /import`. The Fluree server ships a reference backend behind `FLUREE_IMPORT_PRESIGN_ENABLED=true` (stages parts to local disk and concatenates them); production servers mint presigned object-store URLs and drive `CreateMultipartUpload`/`CompleteMultipartUpload`. See the [Negotiated Upload Import Contract](../cli/server-integration.md#negotiated-upload-import-contract) for the full implementer spec.

This is the transport behind `fluree create … --remote … --from big.flpack` when the server is size-capped — the CLI negotiates automatically.

## Storage Proxy Endpoints

These endpoints are intended for peer mode and `fluree clone`/`pull` workflows. They require the storage proxy to be enabled on the server and use replication-grade Bearer tokens (`fluree.storage.*` claims).

### GET /storage/ns/:ledger-id

Fetch the nameservice record for a ledger.

**URL:**

```
GET /storage/ns/{ledger-id}
```

**Request Headers:**

```http
Authorization: Bearer <token>   (requires fluree.storage.* claims)
```

**Response (200 OK):**

```json
{
  "ledger_id": "mydb:main",
  "name": "mydb",
  "branch": "main",
  "commit_head_id": "bafy...commitCid",
  "commit_t": 42,
  "index_head_id": "bafy...indexCid",
  "index_t": 40,
  "default_context": null,
  "retracted": false,
  "config_id": "bafy...configCid"
}
```

| Field | Description |
|-------|-------------|
| `ledger_id` | Canonical ledger ID (e.g., "mydb:main") |
| `name` | Ledger name without branch (e.g., "mydb") |
| `config_id` | CID of the `LedgerConfig` object (origin discovery), if set |

**Status Codes:**

- `200 OK`: Record found
- `404 Not Found`: Storage proxy disabled, ledger not found, or not authorized

### POST /storage/block

Fetch a storage block (index branch or leaf) by CID. The server derives the storage address internally. Leaf blocks are always policy-filtered before return.

Only replication-relevant content kinds are allowed (commits, txns, config, index roots/branches/leaves, dict blobs). Internal metadata kinds (GC records, stats sketches, graph source snapshots) are rejected with 404.

**URL:**

```
POST /storage/block
```

**Request Headers:**

```http
Content-Type: application/json
Authorization: Bearer <token>
Accept: application/octet-stream | application/x-fluree-flakes | application/x-fluree-flakes+json
```

**Request Body:**

Both fields are required:

```json
{
  "cid": "bafy...branchOrLeafCid",
  "ledger": "mydb:main"
}
```

**Responses:**

- `200 OK`: Block bytes (branches) or encoded flakes (leaves)
- `400 Bad Request`: Invalid CID string
- `404 Not Found`: Block not found, disallowed kind, or not authorized

### GET /storage/objects/:cid

Fetch a CAS (content-addressed storage) object by its content identifier. Returns the raw bytes of the stored object after verifying integrity.

This is a **replication-grade** endpoint for `fluree clone`/`pull` workflows. The client knows the CID (from the nameservice record or the commit chain) and wants the raw bytes.

**URL:**

```
GET /storage/objects/{cid}?ledger={ledger-id}
```

**Path Parameters:**

- `cid`: CIDv1 string (base32-lower multibase, e.g., `"bafybeig..."`)

**Query Parameters:**

- `ledger` (required): Ledger ID (e.g., `"mydb:main"`). Required because storage paths are ledger-scoped.

**Request Headers:**

```http
Authorization: Bearer <token>   (requires fluree.storage.* claims)
```

**Kind Allowlist:**

All replication-relevant content kinds are served:

| Kind | Description |
|------|-------------|
| `commit` | Commit chain blobs |
| `txn` | Transaction data blobs |
| `config` | LedgerConfig origin discovery objects |
| `index-root` | Binary index root (FIR6) |
| `index-branch` | Index branch manifests |
| `index-leaf` | Index leaf files |
| `dict` | Dictionary artifacts (predicates, subjects, strings, etc.) |

Only `GarbageRecord` (internal GC metadata) returns 404.

**Response Headers:**

- `Content-Type: application/octet-stream`
- `X-Fluree-Content-Kind`: Content kind label (`commit`, `txn`, `config`, `index-root`, `index-branch`, `index-leaf`, `dict`)

**Response Body:**

Raw bytes of the stored object.

**Integrity Verification:**

The server verifies the hash of the stored bytes against the CID before returning. Commit blobs are format-sniffed:

- **Commit-v2 blobs** (`FCV2` magic): Uses the canonical sub-range hash (SHA-256 over the payload excluding the trailing hash + signature block).
- **All other blobs** (txn, config, future commit formats): Full-bytes SHA-256.

If verification fails, the server returns `500 Internal Server Error` — this indicates storage corruption.

**Status Codes:**

- `200 OK`: Object found and integrity verified
- `400 Bad Request`: Invalid CID string
- `404 Not Found`: Object not found, disallowed kind, not authorized, or storage proxy disabled
- `500 Internal Server Error`: Hash verification failed (storage corruption)

**Example:**

```bash
# Fetch a commit blob by CID
curl -H "Authorization: Bearer $TOKEN" \
  "http://localhost:8090/v1/fluree/storage/objects/bafybeig...commitCid?ledger=mydb:main"

# Fetch a config blob by CID
curl -H "Authorization: Bearer $TOKEN" \
  "http://localhost:8090/v1/fluree/storage/objects/bafybeig...configCid?ledger=mydb:main"

# Fetch an index leaf by CID
curl -H "Authorization: Bearer $TOKEN" \
  "http://localhost:8090/v1/fluree/storage/objects/bafybeig...leafCid?ledger=mydb:main"
```

## Nameservice Sync Endpoints

Used by replication clients and peer instances to push ref updates, initialize
ledgers, and fetch snapshots of all nameservice records. These are the
server-side counterpart to the `fluree-db-nameservice-sync` crate.

**Authorization:** All endpoints require a Bearer token with storage-proxy
permissions. Per-alias endpoints verify the principal is authorized for that
ledger. `/snapshot` filters results to the principal's authorized scope
(`storage_all` returns everything; otherwise results are filtered to
`storage_ledgers` and graph sources are excluded).

**Availability:** These endpoints are only available on transaction servers
(direct storage mode). Proxy-mode instances return `404 Not Found`.

### POST /nameservice/refs/{alias}/commit

Compare-and-set push for a ledger's commit-head ref.

**Request Body:**

```json
{
  "expected": { /* RefValue or null for initial creation */ },
  "new":      { /* RefValue */ }
}
```

**Response (200 OK — updated):**

```json
{ "status": "updated", "ref": { /* new RefValue */ } }
```

**Response (409 Conflict — CAS failed):**

```json
{ "status": "conflict", "actual": { /* current server-side RefValue */ } }
```

### POST /nameservice/refs/{alias}/index

Compare-and-set push for a ledger's index-head ref. Same request/response shape
as `/commit` above.

### POST /nameservice/refs/{alias}/init

Create a ledger entry in the nameservice if it does not already exist.
Idempotent.

**Response:**

```json
{ "created": true }   // new ledger entry was registered
{ "created": false }  // already existed; no change
```

### GET /nameservice/snapshot

Return a full snapshot of all ledger (`NsRecord`) and graph-source
(`GraphSourceRecord`) records visible to the caller.

**Response:**

```json
{
  "ledgers":       [ /* NsRecord, … */ ],
  "graph_sources": [ /* GraphSourceRecord, … */ ]
}
```

**Status Codes:**
- `200 OK` — snapshot returned
- `401 Unauthorized` — missing/invalid storage-proxy token
- `404 Not Found` — endpoint disabled (proxy mode)

## Query Endpoints

### POST /query

Execute a query against one or more ledgers.

**URL:**
```
POST /query
GET  /query?query={urlencoded-sparql}   # SPARQL Protocol GET form
```

The `GET` form is provided for W3C SPARQL Protocol compliance. It accepts SPARQL queries via the `query` query parameter; the body forms below are preferred for larger queries and for JSON-LD. The same form is available on the ledger-scoped `/query/{ledger}` route.

**Optional Query Parameters:**

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `default-context` | boolean | `false` | When `true`, use the ledger's stored default JSON-LD context if the request omits its own `@context` (JSON-LD) or `PREFIX` declarations (ledger-scoped SPARQL). |

**Request Headers:**
```http
Content-Type: application/json
Accept: application/json
Fluree-Min-T: 42  # optional read-after-write guarantee
```

Or for SPARQL:
```http
Content-Type: application/sparql-query
Accept: application/sparql-results+json
Fluree-Min-T: 42  # optional read-after-write guarantee
```

`Fluree-Min-T` makes the server refresh the referenced ledger(s) until they have reached at least that transaction time before executing the query. JSON-LD bodies can also send `opts.min-t`, `opts.min_t`, or `opts.minT`; body opts take precedence over the header.

**Request Body (JSON-LD Query):**

```json
{
  "@context": {
    "ex": "http://example.org/ns/"
  },
  "from": "mydb:main",
  "select": ["?name", "?age"],
  "where": [
    { "@id": "?person", "ex:name": "?name" },
    { "@id": "?person", "ex:age": "?age" }
  ],
  "orderBy": ["?name"],
  "limit": 100
}
```

**Multi-ledger / dataset queries:** The connection-scoped `/query` route (no
`{ledger}` in the path) accepts the full dataset `from` surface, not just a
single ledger string:

- `"from": ["a:main", "b:main"]` — union the listed ledgers as the default graph.
- `"from": { "@id": "a:main@t:5" }` — a single source with time-travel / graph selectors.
- `"fromNamed": { "alias": { "@id": "b:main" } }` — named graphs addressed by a
  GRAPH pattern in the `where` (a query may use `fromNamed` with no `from`).

The query is authorized against the first resolvable source for the bearer
scope; per-ledger policy is then applied to each source during execution.

**Request Body (SPARQL):**

```sparql
PREFIX ex: <http://example.org/ns/>

SELECT ?name ?age
FROM <mydb:main>
WHERE {
  ?person ex:name ?name .
  ?person ex:age ?age .
}
ORDER BY ?name
LIMIT 100
```

**Response (JSON-LD Query):**

```json
[
  { "name": "Alice", "age": 30 },
  { "name": "Bob", "age": 25 }
]
```

**Response (SPARQL):**

```json
{
  "head": {
    "vars": ["name", "age"]
  },
  "results": {
    "bindings": [
      {
        "name": { "type": "literal", "value": "Alice" },
        "age": { "type": "literal", "value": "30", "datatype": "http://www.w3.org/2001/XMLSchema#integer" }
      }
    ]
  }
}
```

**SPARQL output negotiation (`Accept` header):**

The full byte-format negotiation below is available on the **ledger-scoped**
[`POST /query/{ledger}`](#post-queryledger) route. The **connection-scoped**
`POST /query` route (SPARQL with `FROM <ledger>`) returns pre-formatted JSON only
— it supports the JSON family (JSON-LD, SPARQL-results JSON, AgentJson) but not the
byte formats; RDF/XML, SPARQL-results XML, and CSV/TSV require the ledger-scoped
route (see the [connection-scoped note](#connection-scoped-sparql-output) below).

| Query form | Default (no/`*/*`/`application/json`) | `application/ld+json` | `application/rdf+xml` | `application/sparql-results+json` | `text/csv` / `text/tab-separated-values` | `application/sparql-results+xml` | `application/vnd.fluree.agent+json` |
|---|---|---|---|---|---|---|---|
| `SELECT` / `ASK` | SPARQL-results JSON | JSON-LD | **406** | SPARQL-results JSON | CSV / TSV | SPARQL-results XML | AgentJson |
| `CONSTRUCT` / `DESCRIBE` | **JSON-LD** | JSON-LD | RDF/XML | JSON-LD | **406** | **406** | **406** |

A `CONSTRUCT` / `DESCRIBE` produces an RDF graph, which has no solution/binding-table
form, so it is always returned as **JSON-LD** (`Content-Type: application/ld+json`)
unless `application/rdf+xml` is explicitly requested; the solution-table formats
(SPARQL-results XML, CSV/TSV, AgentJson) are rejected with `406`. A `SELECT` / `ASK`
defaults to SPARQL-results JSON and only switches to JSON-LD when `application/ld+json`
is requested explicitly — a bare `application/json` keeps the SPARQL-results-JSON
shape — and RDF/XML (a graph format) is rejected with `406`.

<a name="connection-scoped-sparql-output"></a>
> **Connection-scoped `POST /query` (SPARQL):** this route returns pre-formatted
> JSON only. The JSON-family columns above apply (CONSTRUCT/DESCRIBE → JSON-LD;
> SELECT/ASK → SPARQL-results JSON, or JSON-LD with `Accept: application/ld+json`;
> AgentJson via `application/vnd.fluree.agent+json`, rejected `406` for graph
> queries). CSV/TSV are rejected with `406`, and `application/rdf+xml` /
> `application/sparql-results+xml` are **not** negotiated here — use
> `POST /query/{ledger}` for those byte formats.

**Status Codes:**
- `200 OK` - Query successful
- `400 Bad Request` - Invalid query syntax
- `401 Unauthorized` - Authentication required
- `404 Not Found` - Ledger not found
- `406 Not Acceptable` - Requested output format is not available for this query form (e.g. RDF/XML for a `SELECT`)
- `413 Payload Too Large` - Query exceeds size limit
- `500 Internal Server Error` - Server error
- `503 Service Unavailable` - Query timeout or resource limit

**Example:**

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/json" \
  -d '{
    "from": "mydb:main",
    "select": ["?name"],
    "where": [{ "@id": "?person", "ex:name": "?name" }]
  }'
```

### POST /query/{ledger}

Execute a query against a specific ledger (ledger-scoped).

This endpoint is designed for **single-ledger** queries, but supports selecting **named graphs inside the ledger**.

**URL:**
```
POST /query/{ledger}
```

**Default graph semantics:**
- If the request does not specify a graph selector, the query runs against the ledger's **default graph**.
- The built-in **txn-meta** graph can be selected as either:
  - JSON-LD: `"from": "txn-meta"`, or
  - SPARQL: `FROM <txn-meta>`

**Named graph selection (within the same ledger):**

- **JSON-LD**: you can use `"from"` to pick a graph in this ledger:
  - `"from": "default"` → default graph
  - `"from": "txn-meta"` → txn-meta graph
  - `"from": "<graph IRI>"` → a user-defined named graph IRI within this ledger
  - Structured form: `"from": { "@id": "<ledger>", "graph": "<graph selector>" }`

- **SPARQL**: if the query includes `FROM` / `FROM NAMED`, the server interprets those IRIs as **graphs within this ledger** (not other ledgers):
  - `FROM <default>` / `FROM <txn-meta>` / `FROM <graph IRI>` selects the default graph for triple patterns outside `GRAPH {}`.
  - `FROM NAMED <graph IRI>` makes that named graph available via `GRAPH <graph IRI> { ... }`.

**Ledger mismatch protection:**

If the body includes a ledger reference that targets a different ledger than `{ledger}`, the server returns `400 Bad Request` with a "Ledger mismatch" error.

**Examples:**

JSON-LD (query txn-meta):

```bash
curl -X POST "http://localhost:8090/v1/fluree/query/mydb:main" \
  -H "Content-Type: application/json" \
  -d '{
    "from": "txn-meta",
    "select": ["?commit", "?t"],
    "where": [{ "@id": "?commit", "https://ns.flur.ee/db#t": "?t" }]
  }'
```

JSON-LD (query a user-defined named graph by IRI):

```bash
curl -X POST "http://localhost:8090/v1/fluree/query/mydb:main" \
  -H "Content-Type: application/json" \
  -d '{
    "from": "http://example.org/graphs/products",
    "select": ["?name"],
    "where": [{ "@id": "?p", "http://example.org/ns/name": "?name" }]
  }'
```

SPARQL (select txn-meta as default graph):

```bash
curl -X POST "http://localhost:8090/v1/fluree/query/mydb:main" \
  -H "Content-Type: application/sparql-query" \
  -d 'PREFIX f: <https://ns.flur.ee/db#>
SELECT ?commit ?t
FROM <txn-meta>
WHERE { ?commit f:t ?t }'
```

### History Queries via POST /query

Query the history of entities using the standard `/query` endpoint with `from` and `to` keys specifying the time range.

**Request Body:**

```json
{
  "@context": {
    "ex": "http://example.org/ns/"
  },
  "from": "mydb:main@t:1",
  "to": "mydb:main@t:latest",
  "select": ["?name", "?age", "?t", "?op"],
  "where": [
    { "@id": "ex:alice", "ex:name": { "@value": "?name", "@t": "?t", "@op": "?op" } },
    { "@id": "ex:alice", "ex:age": "?age" }
  ],
  "orderBy": "?t"
}
```

The `@t` and `@op` annotations capture transaction metadata:
- **@t** - Transaction time (integer) when the fact was asserted or retracted.
- **@op** - Operation type as a boolean: `true` for assertions, `false` for retractions. (Mirrors `Flake.op` on disk; constants `"assert"` / `"retract"` are not accepted.)

Both annotations work uniformly for literal-valued and IRI-valued objects.

**Response:**

```json
[
  ["Alice", 30, 1, true],
  ["Alice", 30, 5, false],
  ["Alicia", 31, 5, true]
]
```

**Example:**

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/json" \
  -d '{
    "@context": { "ex": "http://example.org/ns/" },
    "from": "mydb:main@t:1",
    "to": "mydb:main@t:latest",
    "select": ["?name", "?t", "?op"],
    "where": [
      { "@id": "ex:alice", "ex:name": { "@value": "?name", "@t": "?t", "@op": "?op" } }
    ],
    "orderBy": "?t"
  }'
```

**SPARQL History Query:**

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/sparql-query" \
  -d 'PREFIX ex: <http://example.org/ns/>
PREFIX f: <https://ns.flur.ee/db#>

SELECT ?name ?t ?op
FROM <mydb:main@t:1>
TO <mydb:main@t:latest>
WHERE {
  << ex:alice ex:name ?name >> f:t ?t .
  << ex:alice ex:name ?name >> f:op ?op .
}
ORDER BY ?t'
```

### POST /stream/query and /stream/query/{ledger}

Stream SELECT results incrementally as newline-delimited JSON
(`application/x-ndjson`) instead of buffering the whole result into one
response body, with a heartbeat that keeps long-running queries alive past
proxy idle timeouts. Same content-type negotiation as `/query` (JSON-LD or
`application/sparql-query`). Two forms: ledger-scoped (ledger in the greedy
path tail) and connection-scoped (`POST /stream/query`, no path ledger — the
ledger(s) come from JSON-LD `from`/`fromNamed` or SPARQL `FROM`).

```bash
curl -N -X POST http://localhost:8090/v1/fluree/stream/query/my/ledger \
  -H 'Content-Type: application/json' \
  -d '{"@context":{"ex":"http://example.org/"},"select":["?name"],"where":{"@id":"?s","ex:name":"?name"}}'
```

The response is one self-describing JSON record per line (`head` → `row`* with
interleaved `heartbeat`s → a terminal `end` or `error`). SELECT only; ASK,
CONSTRUCT/DESCRIBE, `selectOne`, hydration, and history (JSON-LD `to` / SPARQL
`FROM … TO …`) are rejected with `4xx`. Policy, `from`/`fromNamed`, SPARQL
`FROM`, and multi-ledger queries (JSON-LD and SPARQL) are enforced identically
to `/query`. See **[Streaming query (NDJSON)](streaming-query.md)** for the full
record protocol, the terminal-record (truncation) contract, policy behavior,
and client examples.

### POST /multi-query

Execute a bundle of independent JSON-LD and/or SPARQL queries in parallel against a single shared snapshot moment, with envelope-level `@context` / `opts` defaults that lift into each sub-query.

**URL:**
```
POST /multi-query
```

**Request Headers:**
```http
Content-Type: application/json
```

**Request Body (envelope):**

```json
{
  "@context": { "ex": "http://example.org/" },
  "asOf":     "2024-01-01T12:00:00Z",
  "opts":     { "meta": true, "timeoutMs": 30000, "maxConcurrency": 8 },
  "queries": {
    "alice": {
      "language": "jsonld",
      "query": {
        "from":   "myledger:main",
        "select": ["?name"],
        "where":  { "@id": "?p", "ex:name": "?name" }
      }
    },
    "bob": {
      "language": "sparql",
      "query":    "SELECT ?name FROM <other:main> WHERE { ?p ex:name ?name }"
    }
  }
}
```

**Response:**

```json
{
  "status":  "ok",
  "snapshot": {
    "asOf":    "2024-01-01T12:00:00Z",
    "ledgers": { "myledger:main": 1042, "other:main": 87 }
  },
  "results": { "alice": [...], "bob": {...} }
}
```

(`errors` is omitted when no sub-query failed; `meta` is omitted when `opts.meta` is unset.) Clients branch on `body.status` (`"ok"` | `"partial"` | `"all_failed"`) rather than HTTP code for the aggregate outcome; per-alias errors and timeouts live inside `errors` when present.

**Status Codes:**
- `200 OK` — envelope parsed and executed; body's `status` reports the per-alias aggregate (including `all_failed`).
- `400 Bad Request` — envelope validation failed (bounds, `asOf` collision, missing `from`, history query, envelope `max-fuel`, `maxConcurrency: 0`, malformed body).
- `401 Unauthorized` — authentication required and missing.
- `500 Internal Server Error` — envelope infra failed (snapshot resolution couldn't load a ledger; response exceeds the size cap during assembly).

**Full reference:** see [Multi-query envelope](multi-query.md) for the complete envelope contract, merge rules, snapshot semantics, bounds table, examples, and the explicit list of current limitations (history queries, envelope-level fuel budget, response cap enforcement, SPARQL policy gap).

### GET/POST /explain

Return a query plan without executing the query. Accepts the same body formats and authentication as `/query` (JSON-LD, SPARQL via `application/sparql-query` or `?query=`, and JWS/VC signed requests).

**URL:**
```
GET  /explain[/{ledger...}]
POST /explain[/{ledger...}]
```

**Behavior:**
- JSON-LD body: returns the logical plan for the parsed query.
- SPARQL body: returns the plan for the parsed SPARQL query. The ledger-scoped endpoint (`/explain/{ledger}`) rejects queries containing `FROM` / `FROM NAMED` — strip dataset clauses to explain the core plan.
- SPARQL UPDATE is rejected (HTTP 400) — use `/update` for updates.
- Same ledger-scope enforcement for Bearer tokens as `/query`.

**Response:**

A JSON object describing the logical / physical plan. Shape mirrors the query engine's internal plan representation; treat it as informational and non-stable across releases.

**Status Codes:**
- `200 OK` — plan returned
- `400 Bad Request` — SPARQL UPDATE sent, or `FROM` clauses on the ledger-scoped explain
- `401 Unauthorized` — authentication required and missing
- `404 Not Found` — ledger not found or not authorized

**Examples:**

```bash
# Explain a SPARQL query
curl -X POST http://localhost:8090/v1/fluree/explain/mydb \
  -H "Content-Type: application/sparql-query" \
  --data 'SELECT ?s ?p ?o WHERE { ?s ?p ?o } LIMIT 10'

# Explain a JSON-LD query
curl -X POST http://localhost:8090/v1/fluree/explain/mydb \
  -H "Content-Type: application/json" \
  -d '{"select":["?s"],"where":{"@id":"?s"}}'
```

## Nameservice Metadata

The standalone server does not expose a general-purpose `POST /nameservice/query`
endpoint. Use `GET /ledgers` to list ledgers and graph sources,
`GET /info/{ledger-id}` for metadata about a single ledger or graph source, and
`GET /nameservice/snapshot` for authenticated remote-sync snapshots.

## Ledger Management Endpoints

### GET /ledgers

List all ledgers and graph sources.

**URL:**
```
GET /ledgers
```

**Response:**

```json
{
  "ledgers": [
    {
      "ledger_id": "mydb:main",
      "branch": "main",
      "commit_t": 5,
      "index_t": 5,
      "created": "2024-01-22T10:00:00.000Z",
      "last_updated": "2024-01-22T10:30:00.000Z"
    },
    {
      "ledger_id": "mydb:dev",
      "branch": "dev",
      "commit_t": 3,
      "index_t": 2,
      "created": "2024-01-22T11:00:00.000Z",
      "last_updated": "2024-01-22T11:15:00.000Z"
    }
  ]
}
```

**Example:**

```bash
curl http://localhost:8090/v1/fluree/ledgers
```

For metadata about a specific ledger or graph source, use `GET /info/{ledger-id}`.
To create a ledger, use `POST /create`.

### POST /create

Create a new ledger.

**URL:**
```
POST /create
```

**Authentication:** When admin auth is enabled (`--admin-auth-mode=required`), requires Bearer token from a trusted issuer. See [Admin Authentication](#admin-authentication).

**Request Body:**

```json
{
  "ledger": "mydb:main"
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `ledger` | string | Yes | Ledger ID (e.g., "mydb" or "mydb:main") |

**Response:**

```json
{
  "ledger": "mydb:main",
  "t": 0,
  "commit_id": "bafybeig...commitT0"
}
```

| Field | Description |
|-------|-------------|
| `ledger` | Normalized ledger ID |
| `t` | Transaction time (0 for new ledger) |
| `commit_id` | ContentId of the initial commit |

**Status Codes:**
- `201 Created` - Ledger created successfully
- `400 Bad Request` - Invalid request body
- `401 Unauthorized` - Bearer token required (when admin auth enabled)
- `409 Conflict` - Ledger already exists
- `500 Internal Server Error` - Server error

**Examples:**

```bash
# Create ledger (no auth required in default mode)
curl -X POST http://localhost:8090/v1/fluree/create \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb:main"}'

# Create ledger with auth token (when admin auth enabled)
curl -X POST http://localhost:8090/v1/fluree/create \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer eyJ..." \
  -d '{"ledger": "mydb:main"}'

# Create with short ledger ID (auto-resolves to :main)
curl -X POST http://localhost:8090/v1/fluree/create \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb"}'
```

### POST /drop

Drop a whole ledger (every branch under the supplied name) or, as a
fallback, a graph source with the same name.

**URL:**
```
POST /drop
```

**Authentication:** When admin auth is enabled (`--admin-auth-mode=required`), requires Bearer token from a trusted issuer. See [Admin Authentication](#admin-authentication).

**Request Body:**

```json
{
  "ledger": "mydb",
  "hard": false
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `ledger` | string | Yes | Ledger name (e.g., `"mydb"`). Any branch-qualified form (including `"mydb:main"`) is **rejected** with a `400` — use the [`POST /drop-branch`](#post-drop-branch) endpoint (or call `drop_branch` in the Rust API) to drop a single branch. |
| `hard` | boolean | No | If `true`, delete managed storage artifacts and purge the nameservice records. Default: `false` (soft drop). |

**Scope:**

`/drop` operates on the **whole ledger** — every branch under the ledger name, including any retracted-but-not-purged branches. Branches are dropped leaf-first so that if the operation aborts mid-way the surviving state stays consistent (orphan parents, never dangling children). The cross-branch `@shared/dicts/` namespace is cleaned up at the very end.

**Drop Modes:**

- **Soft drop** (`hard: false`, default): Marks every branch as retracted in the nameservice and preserves storage artifacts. Aliases remain reserved; normal create/load paths treat the ledger as unavailable.
- **Hard drop** (`hard: true`): Deletes managed storage artifacts for every branch and purges the nameservice records so the name can be reused. **This is irreversible for deleted artifacts.**

If no ledger is found by name, the server tries the same name as a graph source on branch `main`. Graph source hard-drop cleanup is best effort; graph-source fallback responses omit `branches_dropped` and `files_deleted`.

**Response:**

```json
{
  "ledger_id": "mydb",
  "status": "dropped",
  "files_deleted": 73,
  "branches_dropped": ["mydb:feature-x", "mydb:dev", "mydb:main"]
}
```

| Field | Type | Description |
|-------|------|-------------|
| `ledger_id` | string | Ledger name (or graph source ID if the graph-source fallback handled the request) |
| `status` | string | Aggregate status across branches. One of: `"dropped"`, `"already_retracted"`, `"not_found"` |
| `files_deleted` | integer | Number of managed storage artifacts deleted (sum across branches + `@shared/dicts/` cleanup); omitted when zero |
| `branches_dropped` | string[] | Per-branch `ledger_id`s that were dropped, in leaf-first order; omitted when empty |
| `warnings` | string[] | Non-fatal cleanup warnings; omitted when empty |

**Status Codes:**
- `200 OK` - Drop successful (or already dropped/not found)
- `400 Bad Request` - Invalid request body, or any branch-qualified ledger id was supplied
- `401 Unauthorized` - Bearer token required (when admin auth enabled)
- `500 Internal Server Error` - Branch enumeration failed, or another unrecoverable error

**Drop Sequence:**

1. Parses input. `"mydb"` is the canonical form; any branch-qualified id (`"mydb:main"`, `"mydb:dev"`, …) returns a `400`.
2. Enumerates every NsRecord under the ledger name (including retracted ones).
3. Sorts branches leaf-first via the `source_branch` parent pointers.
4. Cancels and waits for pending background indexing on each branch.
5. For each branch (leaf-first): deletes managed storage artifacts (hard mode) and retracts (soft) or removes the NS record (hard). Hard mode uses the parent-aware drop path so child counts on surviving parents stay accurate even under partial failure.
6. Hard mode only: wipes the cross-branch `{ledger_name}/@shared/dicts/` namespace.
7. Disconnects every branch from the ledger cache.

**Idempotency:**

Safe to call multiple times:
- Returns `"already_retracted"` when every branch was already retracted (hard mode still proceeds with cleanup for these).
- Returns `"not_found"` without touching storage when no nameservice record exists for the ledger name. Truly orphaned artifacts with no nameservice pointer are **not** swept by `/drop`; that's a separate admin concern.

**Examples:**

```bash
# Soft drop the whole "mydb" ledger
curl -X POST http://localhost:8090/v1/fluree/drop \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb"}'

# Hard drop (delete every branch's artifacts + @shared/dicts - IRREVERSIBLE)
curl -X POST http://localhost:8090/v1/fluree/drop \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb", "hard": true}'

# Drop with auth token (when admin auth enabled)
curl -X POST http://localhost:8090/v1/fluree/drop \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer eyJ..." \
  -d '{"ledger": "mydb", "hard": true}'

# Backwards-compatible form (accepted with a warning; prefer the bare name)
curl -X POST http://localhost:8090/v1/fluree/drop \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb:main"}'
```

### GET /context/{ledger...}

Get the default JSON-LD context for a ledger.

**URL:**
```
GET /context/{ledger-id}
```

**Path Parameters:**
- `ledger-id`: Ledger identifier (e.g., `mydb` or `mydb:main`)

**Response:**

```json
{
  "@context": {
    "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
    "rdfs": "http://www.w3.org/2000/01/rdf-schema#",
    "xsd": "http://www.w3.org/2001/XMLSchema#",
    "owl": "http://www.w3.org/2002/07/owl#",
    "ex": "http://example.org/"
  }
}
```

If no default context has been set, `"@context"` is `null`.

**Status Codes:**
- `200 OK` - Context returned (may be `null`)
- `404 Not Found` - Ledger does not exist

**Example:**

```bash
curl http://localhost:8090/v1/fluree/context/mydb:main
```

### PUT /context/{ledger...}

Replace the default JSON-LD context for a ledger.

**URL:**
```
PUT /context/{ledger-id}
```

**Path Parameters:**
- `ledger-id`: Ledger identifier (e.g., `mydb` or `mydb:main`)

**Request Body:**

A JSON object mapping prefixes to IRIs. Either a bare object or wrapped in `{"@context": {...}}`:

```json
{
  "ex": "http://example.org/",
  "foaf": "http://xmlns.com/foaf/0.1/",
  "schema": "http://schema.org/"
}
```

**Response (success):**

```json
{
  "status": "updated"
}
```

**Status Codes:**
- `200 OK` - Context replaced successfully
- `400 Bad Request` - Body is not a valid JSON object; or peer mode (writes not available)
- `404 Not Found` - Ledger does not exist
- `409 Conflict` - Concurrent update conflict (retry the request)

**Concurrency:** The update uses compare-and-set semantics internally (up to 3 retries). A 409 means all retries were exhausted — this is rare and indicates heavy concurrent updates.

**Cache invalidation:** After a successful update, the server invalidates the cached ledger state. Subsequent queries will use the new context.

**Examples:**

```bash
# Set context
curl -X PUT http://localhost:8090/v1/fluree/context/mydb:main \
  -H "Content-Type: application/json" \
  -d '{"ex": "http://example.org/", "foaf": "http://xmlns.com/foaf/0.1/"}'

# Wrapped form also accepted
curl -X PUT http://localhost:8090/v1/fluree/context/mydb:main \
  -H "Content-Type: application/json" \
  -d '{"@context": {"ex": "http://example.org/"}}'
```

### POST /branch

Create a new branch for a ledger.

**URL:**
```
POST /branch
```

**Authentication:** When admin auth is enabled (`--admin-auth-mode=required`), requires Bearer token from a trusted issuer. See [Admin Authentication](#admin-authentication).

**Request Body:**

```json
{
  "ledger": "mydb",
  "branch": "feature-x",
  "source": "main",
  "at": "t:5"
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `ledger` | string | Yes | Ledger name without branch suffix (e.g., "mydb") |
| `branch` | string | Yes | New branch name to create (e.g., "feature-x") |
| `source` | string | No | Source branch to create from. Default: `"main"` |
| `at` | string | No | Commit on the source branch to start from. `"t:N"` for a transaction number, or a hex digest / full CID for prefix resolution. When omitted, the branch starts at the source's current HEAD. `t:` / prefix resolution requires the source to be indexed. |

**Response:**

```json
{
  "ledger_id": "mydb:feature-x",
  "branch": "feature-x",
  "source": "main",
  "t": 5
}
```

| Field | Description |
|-------|-------------|
| `ledger_id` | Full ledger:branch identifier for the new branch |
| `branch` | Branch name |
| `source` | Source branch this was created from |
| `t` | Transaction time of the commit at the branch point |

**Status Codes:**
- `201 Created` - Branch created successfully
- `400 Bad Request` - Invalid request body (including malformed `at` value)
- `401 Unauthorized` - Bearer token required (when admin auth enabled)
- `404 Not Found` - Source branch does not exist, or `at` commit is not reachable from source HEAD
- `409 Conflict` - Branch already exists
- `500 Internal Server Error` - Server error

**Examples:**

```bash
# Create branch from main (default source)
curl -X POST http://localhost:8090/v1/fluree/branch \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb", "branch": "feature-x"}'

# Create branch from a specific source branch
curl -X POST http://localhost:8090/v1/fluree/branch \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb", "branch": "staging", "source": "dev"}'

# Branch at a historical commit on main
curl -X POST http://localhost:8090/v1/fluree/branch \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb", "branch": "rewind", "at": "t:5"}'
```

### GET /branch/{ledger-name}

List all non-retracted branches for a ledger.

**URL:**
```
GET /branch/{ledger-name}
```

**Response:**

```json
[
  {
    "branch": "main",
    "ledger_id": "mydb:main",
    "t": 5
  },
  {
    "branch": "feature-x",
    "ledger_id": "mydb:feature-x",
    "t": 5,
    "source": "main"
  }
]
```

| Field | Description |
|-------|-------------|
| `branch` | Branch name |
| `ledger_id` | Full ledger:branch identifier |
| `t` | Current transaction time on this branch |
| `source` | Source branch (only present for branches created via `/branch`) |

**Examples:**

```bash
curl http://localhost:8090/v1/fluree/branch/mydb
```

### POST /drop-branch

Drop a branch from a ledger. Admin-protected.

**URL:**
```
POST /drop-branch
```

**Request body:**

```json
{
  "ledger": "mydb",
  "branch": "feature-x"
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `ledger` | string | Yes | Ledger name without branch suffix (e.g., "mydb") |
| `branch` | string | Yes | Branch name to drop (e.g., "feature-x") |

**Response body (200 OK):**

```json
{
  "ledger_id": "mydb:feature-x",
  "status": "dropped",
  "deferred": false,
  "files_deleted": 5,
  "cascaded": [],
  "warnings": []
}
```

| Field | Type | Description |
|-------|------|-------------|
| `ledger_id` | string | Full ledger:branch identifier of the dropped branch |
| `status` | string | Drop status (`"dropped"`, `"already_retracted"`, `"not_found"`) |
| `deferred` | boolean | `true` if the branch has children — retracted but storage preserved |
| `files_deleted` | integer | Number of storage artifacts removed; omitted when zero |
| `cascaded` | string[] | List of ancestor branch ledger_ids that were cascade-dropped; omitted when empty |
| `warnings` | string[] | Any non-fatal warnings during the drop; omitted when empty |

**Behavior:**

- **Cannot drop `main`**: Returns 400 Bad Request.
- **Leaf branch** (no children): Fully drops — deletes storage artifacts, purges NsRecord, decrements parent's child count. If the parent was previously retracted and its child count reaches 0, the parent is cascade-dropped too.
- **Branch with children** (`branches > 0`): Retracted (hidden from listings, rejects new transactions) but storage is preserved for children. When the last child is eventually dropped, the retracted parent is cascade-purged automatically.

**Status codes:**

- `200 OK` - Branch dropped (or deferred) successfully
- `400 Bad Request` - Cannot drop the main branch
- `404 Not Found` - Ledger or branch does not exist
- `500 Internal Server Error` - Server error

**Examples:**

```bash
# Drop a leaf branch
curl -X POST http://localhost:8090/v1/fluree/drop-branch \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb", "branch": "feature-x"}'

# Drop a branch with children (will be deferred)
curl -X POST http://localhost:8090/v1/fluree/drop-branch \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb", "branch": "dev"}'
```

### POST /drop-graph

Drop a single named graph from one branch of a ledger by transactionally
retracting every triple currently asserted under that graph IRI. The drop
runs as a normal commit at `t = current_t + 1` — history at earlier `t`
values is preserved. Admin-protected.

**URL:**
```
POST /drop-graph
```

**Request body:**

```json
{
  "ledger": "mydb:main",
  "graph": "urn:example:org/payroll"
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `ledger` | string | Yes | Full ledger identifier. A bare ledger name (`"mydb"`) is normalized to `"mydb:main"`. |
| `graph` | string | Yes | Full **absolute** IRI of the named graph to drop. Must have a `<scheme>:<rest>` head (relative references like `payroll` are rejected) and contain no whitespace or RFC 3987-excluded characters. Leading/trailing whitespace is rejected, not trimmed. |

**Response body (200 OK):**

```json
{
  "ledger_id": "mydb:main",
  "graph_iri": "urn:example:org/payroll",
  "retracted": 42,
  "committed": true,
  "t": 18
}
```

| Field | Type | Description |
|-------|------|-------------|
| `ledger_id` | string | Normalized `ledger:branch` identifier the drop targeted |
| `graph_iri` | string | Graph IRI that was dropped (echoed) |
| `retracted` | integer | Number of flakes retracted by the drop commit; `0` for a no-op |
| `committed` | boolean | `true` when a new commit was produced; `false` for a no-op drop on an empty graph |
| `t` | integer | Current commit `t` for the branch after the drop |

**Behavior:**

- **Transactional and history-preserving.** A query `as-of` an earlier `t` still sees the graph populated.
- **Per-branch.** Only affects the targeted branch — sibling branches that share the same graph IRI are not touched.
- **Refuses system graphs.** The default graph, `urn:fluree:{ledger_id}#txn-meta`, and `urn:fluree:{ledger_id}#config` cannot be dropped (400 Bad Request).
- **Refuses unknown graphs.** Returns 404 when `graph` is not registered in the ledger's graph registry — the call never auto-registers a new graph slot.
- **Idempotent.** A second call on an already-empty graph returns `committed: false`, `retracted: 0` without producing a commit.

**Status codes:**

- `200 OK` - Drop succeeded (commit produced or no-op)
- `400 Bad Request` - Malformed IRI, empty IRI, or system-graph IRI
- `401`/`403` - Admin token required and absent/invalid
- `404 Not Found` - Ledger or graph IRI not found
- `500 Internal Server Error` - Server error

**Examples:**

```bash
# Drop a named graph on the default branch
curl -X POST http://localhost:8090/v1/fluree/drop-graph \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb", "graph": "urn:example:org/payroll"}'

# Drop on a non-default branch
curl -X POST http://localhost:8090/v1/fluree/drop-graph \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb:feature-x", "graph": "http://example.org/graphs/scratch"}'
```

### POST /rebase

Rebase a branch onto its source branch's current HEAD. Admin-protected.

**URL:**
```
POST /rebase
```

**Request body:**

```json
{
  "ledger": "mydb",
  "branch": "feature-x",
  "strategy": "take-both"
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `ledger` | string | Yes | Ledger name without branch suffix (e.g., "mydb") |
| `branch` | string | Yes | Branch name to rebase (e.g., "feature-x") |
| `strategy` | string | No | Conflict resolution strategy (default: "take-both"). Options: `take-both`, `abort`, `take-source`, `take-branch`, `skip` |

**Response body (200 OK):**

```json
{
  "ledger_id": "mydb:feature-x",
  "branch": "feature-x",
  "fast_forward": false,
  "replayed": 3,
  "skipped": 0,
  "conflicts": 1,
  "failures": 0,
  "total_commits": 3,
  "source_head_t": 8
}
```

| Field | Type | Description |
|-------|------|-------------|
| `ledger_id` | string | Full ledger:branch identifier |
| `branch` | string | Branch name |
| `fast_forward` | bool | `true` if the branch had no unique commits |
| `replayed` | number | Number of commits successfully replayed |
| `skipped` | number | Number of commits skipped (Skip strategy) |
| `conflicts` | number | Number of conflicts detected |
| `failures` | number | Number of commits that failed validation |
| `total_commits` | number | Total branch commits considered |
| `source_head_t` | number | Transaction time of the source branch HEAD |

**Conflict strategies:**

| Strategy | Behavior |
|----------|----------|
| `take-both` | Replay as-is, both values coexist (multi-cardinality) |
| `abort` | Fail on first conflict, no changes applied |
| `take-source` | Drop branch's conflicting flakes (source wins) |
| `take-branch` | Keep branch's flakes, retract source's conflicting values |
| `skip` | Skip entire commit if any flakes conflict |

**Status codes:**

- `200 OK` - Rebase completed successfully
- `400 Bad Request` - Cannot rebase main, invalid strategy, or missing branch point
- `404 Not Found` - Ledger or branch does not exist
- `409 Conflict` - Rebase aborted due to conflict (abort strategy)
- `500 Internal Server Error` - Server error

**Examples:**

```bash
# Rebase with default strategy (take-both)
curl -X POST http://localhost:8090/v1/fluree/rebase \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb", "branch": "feature-x"}'

# Rebase with abort strategy (fail on conflicts)
curl -X POST http://localhost:8090/v1/fluree/rebase \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb", "branch": "feature-x", "strategy": "abort"}'
```

### POST /merge

Merge a source branch into a target branch. Admin-protected.

Fast-forward merges copy the source commit chain into the target namespace and advance the target HEAD. When the target has diverged, Fluree performs a general merge: it computes the source and target deltas since their common ancestor, resolves overlapping `(s, p, g)` conflicts according to the requested strategy, and creates a merge commit on the target branch.

**URL:**
```
POST /merge
```

**Request body:**

```json
{
  "ledger": "mydb",
  "source": "feature-x",
  "target": "dev",
  "strategy": "take-both"
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `ledger` | string | Yes | Ledger name without branch suffix (e.g., "mydb") |
| `source` | string | Yes | Source branch to merge from (e.g., "feature-x") |
| `target` | string | No | Target branch to merge into (defaults to source's parent branch) |
| `strategy` | string | No | Conflict resolution strategy for non-fast-forward merges. Defaults to `take-both`. Options: `take-both`, `abort`, `take-source`, `take-branch` |

**Conflict strategies:**

| Strategy | Behavior |
|----------|----------|
| `take-both` | Keep source flakes as-is, so both source and target values can coexist |
| `abort` | Fail if conflicts are detected; no merge commit is created |
| `take-source` | Source wins: keep source flakes and retract target's conflicting values |
| `take-branch` | Target wins: drop source flakes for conflicting keys |

`skip` is a rebase-only strategy and is not supported for non-fast-forward merges.

**Response body (200 OK):**

```json
{
  "ledger_id": "mydb:dev",
  "target": "dev",
  "source": "feature-x",
  "fast_forward": false,
  "new_head_t": 8,
  "commits_copied": 3,
  "conflict_count": 1,
  "strategy": "take-both"
}
```

| Field | Type | Description |
|-------|------|-------------|
| `ledger_id` | string | Full ledger:branch identifier of the target |
| `target` | string | Target branch name |
| `source` | string | Source branch name |
| `fast_forward` | bool | Whether this merge advanced the target directly to the source HEAD |
| `new_head_t` | number | New commit HEAD transaction time of the target |
| `commits_copied` | number | Number of commit blobs copied to the target namespace |
| `conflict_count` | number | Number of overlapping `(s, p, g)` keys detected during a non-fast-forward merge |
| `strategy` | string | Conflict strategy used for a non-fast-forward merge. Omitted for fast-forward merges |

**Status codes:**

- `200 OK` - Merge completed successfully
- `400 Bad Request` - Source has no branch point (e.g., main), self-merge, unknown strategy, or unsupported merge strategy
- `404 Not Found` - Ledger or branch does not exist
- `409 Conflict` - Merge aborted due to conflicts when using the `abort` strategy, or the target HEAD changed during commit publishing
- `500 Internal Server Error` - Server error

**Examples:**

```bash
# Merge feature-x into its parent (inferred from branch point)
curl -X POST http://localhost:8090/v1/fluree/merge \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb", "source": "feature-x"}'

# Merge dev into main (explicit target)
curl -X POST http://localhost:8090/v1/fluree/merge \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb", "source": "dev", "target": "main"}'

# Non-fast-forward merge with source-winning conflict resolution
curl -X POST http://localhost:8090/v1/fluree/merge \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb", "source": "dev", "target": "main", "strategy": "take-source"}'
```

### GET /merge-preview/{ledger-name}

Read-only preview of merging a source branch into a target branch. Returns the rich diff — ahead/behind commit summaries, conflict keys, and fast-forward eligibility — without mutating any nameservice or content store state.

Bearer token required when `data_auth.mode = required`; reads are gated on `bearer.can_read(ledger)`.

**URL:**
```
GET /merge-preview/{ledger-name}?source={source}&target={target}&max_commits={n}&max_conflict_keys={n}&include_conflicts={bool}&include_conflict_details={bool}&strategy={strategy}
```

**Path / Query Parameters:**

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `ledger` (path) | string | Yes | Ledger name (e.g., "mydb") |
| `source` | string | Yes | Source branch to merge from (e.g., "feature-x") |
| `target` | string | No | Target branch (defaults to the source's parent branch) |
| `max_commits` | number | No | Cap on per-side commit summaries returned (default 500). Server clamps to a hard maximum of 5,000 — values above are silently lowered. Bounds response size, **not** divergence-walk cost (the unbounded `count` is still computed). |
| `max_conflict_keys` | number | No | Cap on conflict keys returned (default 200). Server clamps to a hard maximum of 5,000. Bounds response size, **not** the conflict-delta walks. |
| `include_conflicts` | bool | No | When false, skips the conflict computation (default true). Use this to make the preview cheap on diverged branches. |
| `include_conflict_details` | bool | No | When true, includes source/target flake values for the returned conflict keys. Defaults to false. Details are computed after `max_conflict_keys` is applied. |
| `strategy` | string | No | Strategy used to annotate conflict details. Defaults to `take-both`. Options: `take-both`, `abort`, `take-source`, `take-branch`. |

**Response body (200 OK):**

```json
{
  "source": "feature-x",
  "target": "main",
  "ancestor": { "commit_id": "bafy...", "t": 5 },
  "ahead": {
    "count": 3,
    "commits": [
      { "t": 8, "commit_id": "bafy...", "time": "2026-04-25T12:00:00Z",
        "asserts": 2, "retracts": 0, "flake_count": 2, "message": null }
    ],
    "truncated": false
  },
  "behind": { "count": 1, "commits": [...], "truncated": false },
  "fast_forward": false,
  "mergeable": true,
  "conflicts": {
    "count": 1,
    "keys": [{ "s": [100, "alice"], "p": [100, "status"], "g": null }],
    "truncated": false,
    "strategy": "take-source",
    "details": [
      {
        "key": { "s": [100, "alice"], "p": [100, "status"], "g": null },
        "source_values": [["ex:alice", "ex:status", "active", "xsd:string", true]],
        "target_values": [["ex:alice", "ex:status", "archived", "xsd:string", true]],
        "resolution": {
          "source_action": "kept",
          "target_action": "retracted",
          "outcome": "source-wins"
        }
      }
    ]
  }
}
```

| Field | Type | Description |
|-------|------|-------------|
| `source` | string | Source branch name |
| `target` | string | Target branch name (resolved from default when not supplied) |
| `ancestor` | object \| null | Common ancestor `{commit_id, t}`. `null` when both heads are absent |
| `ahead` | object | Commits on source not on target (`count`, `commits`, `truncated`) |
| `behind` | object | Commits on target not on source |
| `fast_forward` | bool | True when target HEAD == ancestor (or both heads absent) |
| `mergeable` | bool | False only when the selected preview strategy would abort, e.g. `strategy=abort` with conflicts. This is a strategy/conflict signal, not full transaction validation. `mergeable=true` does not guarantee a subsequent `POST /merge` will succeed; it only reflects the conflict/strategy interaction at preview time. |
| `conflicts` | object | Overlapping `(s, p, g)` keys touched on both sides since the ancestor. Empty when `fast_forward` or `include_conflicts=false` |

Per-commit summaries (`ahead.commits[]` / `behind.commits[]`) are newest-first and include assert/retract counts plus an optional `message` extracted from `txn_meta` when an `f:message` string entry is present.

When `include_conflict_details=true`, `conflicts.details[]` contains one entry for each returned conflict key. `source_values` and `target_values` are the current asserted values for that key at each branch HEAD, using the same resolved flake tuple format as `/show`: `[subject, predicate, object, datatype, operation]`, with an optional metadata object as the 6th tuple item. The `resolution` object is an annotation only; preview does not apply the strategy or mutate state.

**Status codes:**

- `200 OK` — Preview computed successfully
- `400 Bad Request` — Source has no branch point (e.g., main), `source == target`, unknown strategy, unsupported preview strategy, `include_conflict_details=true` with `include_conflicts=false`, or `strategy=abort` with `include_conflicts=false`
- `401 Unauthorized` — Bearer token required
- `404 Not Found` — Ledger or branch does not exist (or bearer cannot read it)

**Examples:**

```bash
# Default target (source's parent), defaults for caps and conflict computation
curl "http://localhost:8090/v1/fluree/merge-preview/mydb?source=feature-x"

# Counts only — skip the conflict walks for a faster response
curl "http://localhost:8090/v1/fluree/merge-preview/mydb?source=dev&target=main&include_conflicts=false"

# Cap commit lists at 50 per side
curl "http://localhost:8090/v1/fluree/merge-preview/mydb?source=dev&max_commits=50"

# Include value details and labels for a source-winning merge
curl "http://localhost:8090/v1/fluree/merge-preview/mydb?source=dev&target=main&include_conflict_details=true&strategy=take-source"
```

### GET /info/{ledger-id}

Get ledger metadata. Used by the CLI for `info`, `push`, `pull`, and `clone`.

**URL:**
```
GET /info/{ledger-id}
```

**Path Parameters:**
- `ledger-id`: Ledger ID (e.g., "mydb" or "mydb:main")

**Response (non-proxy mode):**

Returns comprehensive ledger metadata including namespace codes, property stats, and class counts. Always includes:

```json
{
  "ledger_id": "mydb:main",
  "t": 42,
  "commitId": "bafybeig...headCommitCid",
  "indexId": "bafybeig...indexRootCid",
  "namespaces": { ... },
  "properties": { ... },
  "classes": [ ... ]
}
```

**Response (proxy storage mode):**

Returns simplified nameservice-only metadata:

```json
{
  "ledger_id": "mydb:main",
  "t": 42,
  "commit_head_id": "bafybeig...commitCid",
  "index_head_id": "bafybeig...indexCid"
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `ledger_id` | string | Yes | Canonical ledger ID |
| `t` | integer | **Yes** | Current transaction time. Used by push/pull for head comparison. |
| `commitId` | string | No | Head commit CID (non-proxy mode) |
| `commit_head_id` | string | No | Head commit CID (proxy mode) |

> **Important:** The `t` field is required by the CLI for push/pull/clone operations. See [CLI-Server API Contract](../design/cli-server-contract.md) for details.

**Optional query parameters:**

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `realtime_property_details` | boolean | true | When `false`, use the lighter fast novelty-aware stats path instead of the default full lookup-backed path |
| `include_property_datatypes` | boolean | true | Include datatype info for properties |
| `include_property_estimates` | boolean | false | Include index-derived NDV/selectivity estimates for properties |

**Status Codes:**
- `200 OK` - Ledger found
- `401 Unauthorized` - Authentication required
- `404 Not Found` - Ledger not found

**Examples:**

```bash
# Get ledger info
curl "http://localhost:8090/v1/fluree/info/mydb:main"

# With auth token
curl "http://localhost:8090/v1/fluree/info/mydb:main" \
  -H "Authorization: Bearer eyJ..."
```

### GET /exists/{ledger-id}

Check if a ledger exists in the nameservice.

**URL:**
```
GET /exists/{ledger-id}
```

**Path Parameters:**
- `ledger-id`: Ledger ID (e.g., "mydb" or "mydb:main")

**Response:**

```json
{
  "ledger": "mydb:main",
  "exists": true
}
```

| Field | Type | Description |
|-------|------|-------------|
| `ledger` | string | Ledger ID (echoed back) |
| `exists` | boolean | Whether the ledger is registered in the nameservice |

**Status Codes:**
- `200 OK` - Check completed successfully (regardless of whether ledger exists)
- `500 Internal Server Error` - Server error

**Usage Notes:**

This is a lightweight check that only queries the nameservice without loading the ledger data. Use this to:

- Check if a ledger exists before attempting to load it
- Implement conditional create-or-load logic
- Validate ledger IDs in application code

**Examples:**

```bash
# Check a ledger ID
curl "http://localhost:8090/v1/fluree/exists/mydb:main"

# Conditional create-or-load in shell
if curl -s "http://localhost:8090/v1/fluree/exists/mydb" | jq -e '.exists == false' > /dev/null; then
  curl -X POST http://localhost:8090/v1/fluree/create \
    -H "Content-Type: application/json" \
    -d '{"ledger": "mydb"}'
fi
```

## System Endpoints

### GET /health

Health check endpoint for monitoring.

**URL:**
```
GET /health
```

**Response:**

```json
{
  "status": "healthy",
  "version": "0.1.0",
  "storage": "memory",
  "uptime_ms": 123456
}
```

**Status Codes:**
- `200 OK` - System healthy
- `503 Service Unavailable` - System unhealthy

**Example:**

```bash
curl http://localhost:8090/health
```

### GET /stats

Detailed server statistics.

**URL:**
```
GET /stats
```

**Response:**

```json
{
  "version": "0.1.0",
  "uptime_ms": 123456789,
  "storage": {
    "mode": "memory",
    "total_bytes": 12345678,
    "ledgers": 5
  },
  "queries": {
    "total": 1234,
    "active": 3,
    "average_duration_ms": 45
  },
  "transactions": {
    "total": 567,
    "average_duration_ms": 89
  },
  "indexing": {
    "active": true,
    "pending_ledgers": 2
  }
}
```

**Example:**

```bash
curl http://localhost:8090/v1/fluree/stats
```

## Events Endpoint

### GET /events

Server-Sent Events (SSE) stream of nameservice changes for ledgers and graph sources. Available on transaction servers only (not peers).

**Query parameters:**

| Parameter | Description |
|-----------|-------------|
| `all=true` | Subscribe to all ledgers and graph sources |
| `ledger=<id>` | Subscribe to a specific ledger (repeatable) |
| `graph-source=<id>` | Subscribe to a specific graph source (repeatable) |

**Event types:**

| Event | Description |
|-------|-------------|
| `ns-record` | A ledger or graph source was published/updated |
| `ns-retracted` | A ledger or graph source was deleted |

**Authentication:** Configurable via `--events-auth-mode none|optional|required`. See [Query peers and replication](../operations/query-peers.md) for full details including auth configuration, event payloads, and peer subscription setup.

## Graph Source Endpoints

> **Note:** HTTP endpoints for BM25 and vector index lifecycle management (create, sync, drop) are not yet implemented in the server. BM25 and vector indexes are currently managed via the Rust API (`Bm25CreateConfig`, `create_full_text_index`, `sync_bm25_index`, `drop_full_text_index`). See [BM25 Full-Text Search](../indexing-and-search/bm25.md) and [Vector Search](../indexing-and-search/vector-search.md) for API usage.
>
> BM25 search **is** available in queries via the `f:graphSource` / `f:searchText` pattern in where clauses — see the query documentation for details.

Graph source metadata can be discovered via `GET /ledgers` or `GET /info/{graph-source-id}`.

### POST {api_base_url}/iceberg/map

Map an Iceberg table (or R2RML-mapped relational source backed by Iceberg) as a graph source. Admin-protected — requires the admin Bearer token when an admin token is configured. Available only when the server is built with the `iceberg` feature.

**URL:**
```
POST {api_base_url}/iceberg/map
```

For the standalone server and Docker image defaults, this is:

```bash
POST http://localhost:8090/v1/fluree/iceberg/map
```

**Request Body:**

```json
{
  "name": "warehouse-orders",
  "mode": "rest",
  "catalog_uri": "https://polaris.example.com/api/catalog",
  "table": "sales.orders",
  "branch": "main",
  "r2rml": "@prefix rr: <http://www.w3.org/ns/r2rml#> . ...",
  "r2rml_type": "text/turtle",
  "warehouse": "prod",
  "auth_bearer": "…",
  "oauth2_token_url": "https://idp.example.com/token",
  "oauth2_client_id": "…",
  "oauth2_client_secret": "…",
  "no_vended_credentials": false,
  "s3_region": "us-east-1",
  "s3_endpoint": "https://s3.example.com",
  "s3_path_style": false,
  "table_location": "s3://bucket/warehouse/sales/orders"
}
```

| Field | Type | Description |
|-------|------|-------------|
| `name` | string | Graph source name (required) |
| `mode` | string | `rest` (default) or `direct` |
| `catalog_uri` | string | REST catalog URI (required in `rest` mode) |
| `table` | string | Table identifier `namespace.table` (required in `rest` mode) |
| `table_location` | string | S3 table location (required in `direct` mode) |
| `r2rml` | string | Inline R2RML mapping (Turtle/JSON-LD). Omit to auto-generate a direct mapping. |
| `r2rml_type` | string | Media type of `r2rml` (`text/turtle`, `application/ld+json`) |
| `branch` | string | Branch name (default: `main`) |
| `auth_bearer` | string | Bearer token for catalog auth |
| `oauth2_*` | string | OAuth2 client-credentials flow for the catalog |
| `warehouse` | string | Warehouse identifier |
| `no_vended_credentials` | bool | Disable vended credentials |
| `s3_region`, `s3_endpoint`, `s3_path_style` | | S3 overrides for `direct` mode |

**Response:**

```json
{
  "graph_source_id": "warehouse-orders:main",
  "table_identifier": "sales.orders",
  "catalog_uri": "https://polaris.example.com/api/catalog",
  "connection_tested": true,
  "mapping_source": "r2rml-inline",
  "triples_map_count": 3,
  "mapping_validated": true
}
```

**Status Codes:**
- `201 Created` — graph source created
- `400 Bad Request` — missing required fields or invalid R2RML
- `401/403` — admin auth required
- `500 Internal Server Error` — catalog connection or mapping failure

See also the CLI wrapper: [fluree iceberg map](../cli/iceberg.md).

## Admin Endpoints

### POST /reindex

Trigger a full manual reindex for a ledger. Walks the entire commit chain and rebuilds the binary index from scratch using the server's configured indexer settings. Admin-protected — requires the admin Bearer token when admin auth is enabled.

This endpoint runs the reindex synchronously and returns when the new root is committed. For large ledgers it may run for many minutes; configure your HTTP client timeout accordingly. In peer mode, the request is forwarded to the transaction server.

**URL:**
```
POST /reindex
```

**Request Body:**

```json
{
  "ledger": "mydb:main"
}
```

| Field | Type | Description |
|-------|------|-------------|
| `ledger` | string | Ledger alias (`name` or `name:branch`). Required. |
| `opts`   | object | Reserved for future per-request indexer overrides. Currently accepted but ignored. |

**Example:**

```bash
curl -X POST http://localhost:8090/v1/fluree/reindex \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer <admin-token>' \
  -d '{"ledger": "mydb:main"}'
```

**Response:**

```json
{
  "ledger_id": "mydb:main",
  "index_t": 42,
  "root_id": "fluree:cid:bafy…",
  "stats": {
    "flake_count": 184273,
    "leaf_count": 614,
    "branch_count": 23,
    "total_bytes": 47185920
  },
  "fuel": 1734.0
}
```

| Field | Description |
|-------|-------------|
| `ledger_id` | Ledger alias the reindex was run against |
| `index_t` | Transaction time the new index was built at (matches the head commit) |
| `root_id` | ContentId of the newly written index root |
| `stats.flake_count` | Total flakes in the rebuilt index |
| `stats.leaf_count` | Number of leaf nodes written |
| `stats.branch_count` | Number of branch nodes written |
| `stats.total_bytes` | Bytes written to storage during the reindex |
| `fuel` | Total decimal fuel charged for the reindex's CAS writes (1.000 per write + 1.000 per re-encoded leaflet in FLI3 leaves). `0.0` if the index was already current. See [Tracking and Fuel](../query/tracking-and-fuel.md#indexing-fuel) for the full schedule. |

**Status Codes:**
- `200 OK` — reindex complete
- `400 Bad Request` — missing/invalid `ledger`
- `401/403` — admin auth required
- `404 Not Found` — ledger does not exist
- `500 Internal Server Error` — reindex failed

When triggering indexing through the Rust API instead, see `Fluree::reindex` and `ReindexOptions`. For background incremental indexing (which runs automatically as commits are made), see [Background indexing](../indexing-and-search/background-indexing.md).

### POST /export/*ledger

Return ledger data as RDF in the requested format (Turtle, N-Triples, N-Quads, TriG, or JSON-LD). Server-side equivalent of `fluree export`.

**Auth bracket: admin-protected** — same middleware as `/create`, `/drop`, `/reindex`, and the branch admin endpoints. Today's implementation reads from the binary index without per-flake policy filtering, so it does not live in the data-read bracket alongside `/query` and `/show`. Adding policy-filtered streaming export would let it move to read-auth in the future.

**URL:**

```
POST /export/<ledger...>
```

**Request Body:**

```json
{
  "format": "turtle",
  "all_graphs": false,
  "graph": "http://example.org/people",
  "context": { "ex": "http://example.org/" },
  "at": "t:42"
}
```

| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `format` | string | No | `"turtle"` | One of `turtle`/`ttl`, `ntriples`/`nt`, `nquads`/`n-quads`, `trig`, `jsonld`/`json-ld`/`json`. Case-insensitive. |
| `all_graphs` | bool | No | `false` | Export every named graph as a dataset. Requires `format` ∈ `trig` / `nquads`. Mutually exclusive with `graph`. |
| `graph` | string | No | — | IRI of a single named graph to export. Mutually exclusive with `all_graphs`. |
| `context` | object | No | ledger default | Prefix map for Turtle/TriG/JSON-LD output. Either a bare object or `{"@context": {…}}`. |
| `at` | string | No | latest | Time spec — integer (`"42"`), ISO-8601 datetime, or commit CID prefix. |

An empty body is treated as all-default (Turtle export at HEAD).

**Response Headers:**

| Format | Content-Type |
|--------|--------------|
| Turtle | `text/turtle; charset=utf-8` |
| N-Triples | `application/n-triples; charset=utf-8` |
| N-Quads | `application/n-quads; charset=utf-8` |
| TriG | `application/trig; charset=utf-8` |
| JSON-LD | `application/ld+json; charset=utf-8` |

**Response Body (200 OK):**

The raw RDF for the requested format. The reference server today buffers the full export in memory before responding; implementations are free to stream chunked bodies, and clients MUST be prepared to read until EOF.

**Status Codes:**

- `200 OK` — export complete
- `400 Bad Request` — unknown format; conflicting `all_graphs` + `graph`; `all_graphs` with non-dataset format; unknown graph IRI; malformed JSON; ledger not indexed (`ApiError::Config`)
- `401` / `403` — admin token required and absent/invalid
- `404 Not Found` — ledger does not exist
- `5xx` — storage / nameservice / encoding errors

**Peer mode:** Forwards to the transactor.

## Admin Authentication

Administrative endpoints (`/create`, `/drop`, `/reindex`, branch operations, and Iceberg mapping when enabled) can be protected with Bearer token authentication.

### Configuration

Enable admin authentication with CLI flags:

```bash
# Production: require trusted tokens
fluree-server \
  --admin-auth-mode=required \
  --admin-auth-trusted-issuer=did:key:z6MkhaXgBZDvotDkL5257faiztiGiC2QtKLGpbnnEGta2doK

# Development: no authentication (default)
fluree-server --admin-auth-mode=none
```

**Environment Variables:**
- `FLUREE_ADMIN_AUTH_MODE`: `none` (default) or `required`
- `FLUREE_ADMIN_AUTH_TRUSTED_ISSUERS`: Comma-separated list of trusted did:key identifiers

### Token Format

Admin tokens use the same JWS format as other Fluree tokens. Required claims:

```json
{
  "iss": "did:key:z6MkhaXgBZDvotDkL5257faiztiGiC2QtKLGpbnnEGta2doK",
  "exp": 1705932000,
  "sub": "admin@example.com"
}
```

| Claim | Required | Description |
|-------|----------|-------------|
| `iss` | Yes | Issuer did:key (must be in trusted issuers list) |
| `exp` | Yes | Expiration timestamp (Unix seconds) |
| `sub` | No | Subject identifier |
| `fluree.identity` | No | Identity for audit logging |

### Making Authenticated Requests

Include the token in the Authorization header:

```bash
curl -X POST http://localhost:8090/v1/fluree/create \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer eyJhbGciOiJFZERTQSIsImp3ayI6ey..." \
  -d '{"ledger": "mydb:main"}'
```

### Issuer Trust

Tokens must be signed by a trusted issuer. Configure trusted issuers:

```bash
# Single issuer
--admin-auth-trusted-issuer=did:key:z6MkhaXgBZDvotDkL5257faiztiGiC2QtKLGpbnnEGta2doK

# Multiple issuers
--admin-auth-trusted-issuer=did:key:z6Mk... \
--admin-auth-trusted-issuer=did:key:z6Mn...

# Fallback to events auth issuers
--events-auth-trusted-issuer=did:key:z6Mk...
```

If no admin-specific issuers are configured, admin auth falls back to `--events-auth-trusted-issuer`.

### Response Codes

- `401 Unauthorized`: Missing or invalid Bearer token
- `401 Unauthorized`: Token expired
- `401 Unauthorized`: Untrusted issuer

## Error Responses

All endpoints may return error responses in this format (and should return `Content-Type: application/json`):

```json
{
  "error": "Human-readable error message",
  "status": 409,
  "@type": "err:db/Conflict",
  "cause": {
    "error": "Optional nested error detail",
    "status": 409,
    "@type": "err:db/SomeInnerError"
  }
}
```

See [Errors and Status Codes](errors.md) for complete error reference.

## CLI Compatibility Requirements

This section summarizes the contract that third-party server implementations (e.g., Solo) must follow to be compatible with the Fluree CLI (`fluree-db-cli`). The CLI discovers the API base URL via `fluree remote add` and constructs endpoint URLs as `{base_url}/{operation}/{ledger}`.

### Required endpoints

| Endpoint | CLI commands |
|----------|-------------|
| `GET /info/{ledger}` | `info`, `push`, `pull`, `clone` |
| `GET /show/{ledger}?commit=<ref>` | `show --remote` |
| `POST /query/{ledger}` | `query` (JSON-LD and SPARQL) |
| `POST /insert/{ledger}` | `insert` |
| `POST /upsert/{ledger}` | `upsert` |
| `GET /exists/{ledger}` | `clone` (pre-create check) |
| `GET /context/{ledger}` | `context get` |
| `PUT /context/{ledger}` | `context set` |
| `GET /ledgers` | `list --remote` |

For sync workflows (`clone`/`push`/`pull`), these additional endpoints are needed:

| Endpoint | CLI commands | Notes |
|----------|-------------|-------|
| `POST /push/{ledger}` | `push` | Required for push |
| `GET /commits/{ledger}` | `clone`, `pull` | Paginated export fallback |
| `POST /pack/{ledger}` | `clone`, `pull` | Preferred bulk transport; CLI falls back to `/commits` on 404/405/501 |
| `GET /storage/ns/{ledger}` | `clone`, `pull` | Pack preflight (head CID discovery) |

### Critical response field: `t`

The `GET /info/{ledger}` response **must** include a `t` field (integer) representing the current transaction time. This field is used by the CLI for:

- **push**: Comparing `local_t` vs `remote_t` to determine what commits to send and detect divergence
- **pull**: Comparing `remote_t` vs `local_t` to determine if new commits are available
- **clone**: Guarding against cloning empty ledgers (`t == 0`) and displaying progress

Omitting `t` from the info response will cause `push` and `pull` to fail with `"remote ledger-info response missing 't'"`.

### Transaction response format

The `/insert` and `/upsert` endpoints should return a JSON object. The CLI displays the full response as pretty-printed JSON. Common fields include `t`, `tx-id`, and `commit.hash`, but the exact shape is not prescribed — the CLI does not parse individual fields from transaction responses.

### Authentication

All endpoints accept `Authorization: Bearer <token>`. On `401`, the CLI attempts a single token refresh (if OIDC is configured) and retries. See [Auth contract](../design/auth-contract.md) for the full authentication lifecycle.

### Error responses

Error bodies should be JSON with an `error` or `message` field. The CLI extracts the first available string from `message` or `error` for display. Plain-text error bodies are also accepted.

## Related Documentation

- [Overview](overview.md) - API overview and principles
- [Headers](headers.md) - HTTP headers and content types
- [Signed Requests](signed-requests.md) - Authentication
- [Errors](errors.md) - Error codes and troubleshooting


---

# api/multi-query.md

# Multi-query envelope

> **Endpoint:** `POST /v1/fluree/multi-query`
> **Status:** envelope contract stable; some features explicitly out of scope — see [Limitations](#limitations).

Bundle multiple independent queries — JSON-LD, SPARQL, or both — into a single
HTTP request that runs them in parallel against one shared snapshot moment.
The response carries per-alias results, per-alias errors, and an explicit
record of the per-ledger transaction (`t`) every sub-query observed.

## When to use it

- You want N queries to see the **same data** without coordinating two round
  trips ("get `t`, then issue N pinned queries").
- You want **parallel execution** without standing up N concurrent HTTP
  requests (in-process cache sharing wins over N independent connections).
- You want a **single signed-request boundary** when working with JWS/VC.

## When *not* to use it

- The queries don't need to share a snapshot — N concurrent single-query
  requests are simpler and have looser per-query bounds.
- You need **history range queries** (`FROM <a@t:1> TO <a@t:latest>` or
  JSON-LD `to`) — those are rejected in the envelope (see
  [Limitations](#limitations)).
- You need a **shared envelope fuel budget** — currently each sub-query
  carries its own `max-fuel` (see [Limitations](#limitations)).

---

## Request envelope

```json
{
  "@context": { "schema": "http://schema.org/" },
  "asOf":     "2024-01-01T12:00:00Z",
  "opts":     { "meta": true, "timeoutMs": 30000, "maxConcurrency": 8 },
  "queries": {
    "alice": {
      "language": "jsonld",
      "query": {
        "from":   "myledger:main",
        "select": ["?name"],
        "where":  { "@id": "?p", "schema:name": "?name" }
      }
    },
    "bob": {
      "language": "sparql",
      "query":    "SELECT ?name FROM <other:main> WHERE { ?p schema:name ?name }"
    }
  }
}
```

### Top-level fields

| Field | Required | Type | Purpose |
|-------|----------|------|---------|
| `queries` | **yes** | object (alias → sub-query) | Independent sub-queries to run. Alias keys become keys in `results` / `errors`. |
| `@context` | no | object | Envelope-level JSON-LD context. Merged into each sub-query (see [Merge rules](#merge-rules)). |
| `asOf` | no | integer or ISO 8601 string | Shared snapshot pin (see [Snapshot semantics](#snapshot-semantics)). |
| `opts` | no | object | Envelope-default opts. Merged into each sub-query (see [Merge rules](#merge-rules)). |

### Per-sub-query fields

| Field | Required | Type | Purpose |
|-------|----------|------|---------|
| `language` | **yes** | `"jsonld"` or `"sparql"` (alias `"json-ld"` accepted) | Selects the parser. |
| `query` | **yes** | object (jsonld) or string (sparql) | Query body. Carries its own `from` — each sub-query specifies its own dataset. |
| `opts` | no | object | Per-sub-query overrides. Merged onto envelope opts with sub-query winning on conflict. `opts.t` is **not allowed** here — pin time via `from` or envelope `asOf`. |

---

## Merge rules

Two rules cover the entire "what shadows what" question:

1. **Mergeable** (`@context`, `opts`): **shallow merge**, sub-query wins on
   key conflict. Envelope serves as the default.
2. **Temporal pin** (`asOf` vs any inner `@t:` / `t` field / SPARQL
   `FROM <ledger@t:...>`): **collision is an error**, never a silent
   override.

### `@context` inheritance

- Sub-query has no `@context`: inherits the envelope context unchanged.
- Sub-query has an `@context` object: shallow-merged onto the envelope
  context. Sub-query keys win on conflict.
- Sub-query has `@context: null`: explicit reset — that sub-query runs
  with **no** context.

For **SPARQL** sub-queries, the envelope context contributes to the query
in two ways:

- **`PREFIX` injection** — if the SPARQL query has *zero* `PREFIX`
  declarations of its own, every prefix-shaped entry of the merged
  envelope context is injected as a `PREFIX` declaration. The injection
  is per-directive-class and all-or-nothing: declaring even a single
  `PREFIX` in your SPARQL turns off envelope `PREFIX` injection (but not
  `BASE` injection — those are decided independently).
- **`BASE` injection** — same all-or-nothing rule applied to the
  envelope's `@base`.

Only prefix-shaped entries (key is a valid SPARQL `PN_PREFIX`, value is a
namespace IRI ending with `#` or `/` or containing `://`) are eligible.
JSON-LD term aliases like `"name": "schema:name"` are not — they would
produce invalid SPARQL.

### `opts` merge

Opts come from three layers, merged shallowly with **the most specific layer winning** on key conflict:

| Layer | Where it lives | Precedence |
|-------|----------------|------------|
| 1. Envelope opts | top-level `opts` on the envelope | lowest — applies as defaults to every sub-query |
| 2. Sub-query opts | `opts` on each entry of the `queries` map (the wrapper around `language` + `query`) | middle — wins over envelope |
| 3. Query body opts | `opts` *inside* the JSON-LD query body (`sub.query["opts"]`) | **highest — wins over both** |

This precedence is security-relevant. The HTTP server's bearer/identity gate runs against the **pre-merged** opts and writes its decision into the body layer (layer 3), where the dispatcher's merge then makes it the final word. Inverting the order would let an envelope-level or sub-query-level `opts.identity` clobber the gate's decision.

- Per-sub-query overrides recognised at any layer: `meta`, `policy`, `policy-class`, `policy-values`, `identity`, `default-allow`, `timeoutMs`, `max-fuel`, `min-t` / `minT`.
- `opts.min-t` / `opts.minT` requests read-after-write freshness before the envelope snapshot is resolved. Envelope-level values apply to every referenced ledger; sub-query values apply to that sub-query's ledger(s). The `Fluree-Min-T` header provides the same guarantee as an envelope default.
- `opts.t` is **rejected at every layer** inside a multi-query envelope (envelope opts, sub-query opts, AND body opts). Pin time via `from` (per sub-query) or envelope `asOf`.
- Envelope-level `opts.max-fuel` is rejected up-front (see [Limitations](#limitations)); per-sub-query `opts.max-fuel` (in either layer 2 or layer 3) is honoured.

---

## Snapshot semantics

When the server processes an envelope, it resolves the snapshot **once**
at envelope entry, then uses that resolved per-ledger `t` map for every
unpinned sub-query.

| `asOf` value | Behavior |
|--------------|----------|
| omitted | Server captures wall-clock "now" once at envelope entry, then pins each distinct ledger to its current `t`. The captured moment is echoed back in `snapshot.asOf` so the request is reproducible. |
| integer (e.g. `"asOf": 42`) | The envelope must reference **exactly one ledger** — the integer pin is applied directly. Multi-ledger envelopes paired with an integer `asOf` are rejected with `400 Bad Request`. |
| ISO 8601 string (e.g. `"asOf": "2024-01-01T12:00:00Z"`) | Each distinct ledger resolves to its latest commit at or before that moment. Different ledgers may receive different numeric `t` values. |

> **Atomicity caveat.** `asOf` provides **shared time resolution**, not
> distributed atomicity. Different ledgers commit on independent clocks,
> so two ledgers' "as of this moment" `t` values reflect each ledger's
> latest commit independently — they are not synchronized across ledgers.
> The response's `snapshot.ledgers` map exposes exactly what each
> sub-query observed.

### Temporal collision

When `asOf` is set, no sub-query may carry its own temporal pin. The
following are all collision errors and reject the envelope with
`400 Bad Request`:

- JSON-LD `from: "ledger@t:42"`
- JSON-LD `from: { "@id": "ledger", "t": 42 }`
- JSON-LD `from: { "@id": "ledger", "at": "commit:abc123" }`
- JSON-LD body with an inner `t` field
- SPARQL `FROM <ledger@t:42>` / `FROM <ledger@iso:...>` /
  `FROM <ledger@commit:...>`

This rule is intentional: the alternative (silent inner-wins override)
made it too easy to query a different snapshot than you asked for.

When `asOf` is **omitted**, inner temporal pins are allowed and the
envelope is still atomic per the rule above (snapshot resolved once at
envelope entry).

---

## Response shape

```json
{
  "status":  "ok",
  "snapshot": {
    "asOf":    "2024-01-01T12:00:00.000Z",
    "ledgers": { "myledger:main": 1042, "other:main": 87 }
  },
  "results": {
    "alice": [ { "name": "Alice" } ],
    "bob":   { "head": { "vars": ["name"] }, "results": { "bindings": [...] } }
  },
  "tracking": {
    "alice": { "time": "5ms", "fuel": 1024.0 },
    "bob":   { "time": "3ms", "fuel": 210.5 }
  },
  "meta": { "fuel_total": 1234.5, "elapsed_ms": 87 }
}
```

> **Field omission:** `errors` is omitted from the response when no sub-query failed (zero entries → field skipped, not emitted as `{}`). `tracking` is omitted when no sub-query ran with tracking enabled. `meta` is omitted when `opts.meta` isn't set at the envelope level. `snapshot.asOf` is omitted when the envelope used an integer `asOf` (no shared wall-clock interpretation). Examples below show only the fields that would appear in each scenario.

### Fields

| Field | Type | Description |
|-------|------|-------------|
| `status` | `"ok"` \| `"partial"` \| `"all_failed"` | Aggregate over per-alias outcomes. **Clients should branch on this**, not on HTTP status. |
| `snapshot.asOf` | string \| absent | ISO 8601 moment used for resolution. Echoes envelope `asOf` (ISO form) or the server's wall-clock at envelope entry. Absent when envelope used integer `asOf`. |
| `snapshot.ledgers` | object (ledger → integer) | Per-ledger numeric `t` every sub-query observed. **Each value is independent** — see the atomicity caveat above. |
| `results` | object (alias → query result) | Successful sub-queries, keyed by alias. JSON-LD aliases get the JSON-LD query result shape; SPARQL aliases get SPARQL Results JSON. Aliases that errored are absent here. |
| `errors` | object (alias → error entry) | Failed or timed-out sub-queries. Each entry has `code`, `message`, and (for timeouts) `effective_timeout_ms`. Omitted when empty. |
| `tracking` | object (alias → tally) | Per-alias telemetry (`time`, `fuel`, `policy`) for each sub-query that ran with tracking enabled. Mirrors single-query `/query`'s tracked-response shape, one entry per alias. Indexed by alias and ordered to match `results`. Omitted when no sub-query tracked. |
| `meta.fuel_total` | number \| absent | Envelope-level rollup of per-alias fuel. Sum across every sub-query that tracked. Included when envelope `opts.meta` is enabled. |
| `meta.elapsed_ms` | number \| absent | Envelope wall-clock duration (entry → response assembly). Included when envelope `opts.meta` is enabled. Note: this is the envelope wall-clock, not a sum across parallel sub-queries — sub-queries run concurrently, so summing per-alias `time` would over-count. |

#### Tracking detail

Each entry in `tracking` mirrors the single-query [`TrackedQueryResponse`](endpoints.md#post-query) shape with three optional siblings:

- `time` — formatted execution time string (e.g., `"5ms"`).
- `fuel` — decimal fuel consumed.
- `policy` — per-policy `{ executed, allowed }` stats when policy tracking is requested.

Which siblings populate depends on what the sub-query's merged opts requested (`opts.meta: true` enables all three; selective `opts.meta: { time: true, fuel: true }` enables a subset; `opts.max-fuel: N` implicitly enables fuel tracking).

A sub-query whose opts didn't enable tracking will not appear in the `tracking` map at all — making it easy for tracking-unaware clients to ignore the field entirely.

### HTTP status mapping

| HTTP code | Meaning |
|-----------|---------|
| `200` | Envelope parsed, validated, executed. Body's `status` reports the aggregate (`ok` / `partial` / `all_failed`). Per-alias errors and timeouts live inside `errors`. |
| `400` | Envelope validation failed (bounds violation, `asOf` collision, missing `from`, malformed body, history query, envelope `max-fuel`, `maxConcurrency: 0`, etc.). No `results` / `errors` keys — the body is the standard error shape. |
| `401` | Authentication required and missing. |
| `500` | Envelope infrastructure failed: snapshot resolution couldn't load a ledger, response would exceed the configured size cap during assembly, server-side panic. |

---

## Bounds

The server enforces several limits per envelope.

> The "Override surface" column below lists the request-side knobs that already work (`opts.maxConcurrency`, `opts.timeoutMs`). The other limits are compile-time defaults (`MultiQueryBounds::DEFAULT`) and not server-tunable today.

| Bound | Value | Override surface |
|-------|-------|------------------|
| Max sub-queries / envelope | 64 | request cannot override |
| Max distinct ledgers / envelope | 8 | request cannot override |
| Max concurrent sub-queries | 16 | `opts.maxConcurrency` (clamped to 16) |
| Envelope wall deadline | 60_000 ms | `opts.timeoutMs` (clamped to 60_000) |
| Per-sub-query timeout | `min(opts.timeoutMs, remaining envelope budget)` | `opts.timeoutMs` per-sub-query or per-envelope |
| Response size | 64 MiB | request cannot override |

**Per-sub-query effective timeout** is computed when the sub-query
acquires its concurrency permit, not at envelope entry. A sub-query that
waits 30 s in the permit queue on a 60 s envelope gets ≤30 s of execution
regardless of its own `opts.timeoutMs`. The total wall-clock budget is
the envelope's promise.

When the envelope deadline fires, in-flight sub-queries are cancelled
and reported with `code: "timeout"` in the per-alias errors map. Already
completed sub-queries land in `results` normally.

---

## Output formatting

By default, each alias formats with its language's natural shape: JSON-LD
sub-queries emit JSON-LD, SPARQL sub-queries emit SPARQL 1.1 Results JSON.

### In-process (Rust builder)

The builder accepts an envelope-wide `FormatterConfig` via `.format(...)`
matching the single-query `fluree.query_from().format(...)` vocabulary.

```rust
use fluree_db_api::FormatterConfig;

let response = fluree
    .multi_query()
    .envelope(envelope)
    .format(FormatterConfig::typed_json().with_normalize_arrays())
    .execute()
    .await?;
```

The envelope's `results` map is always JSON, so only JSON-producing
formats are accepted. Their treatment differs by design — cross-language
shapes apply to every alias, JSON-LD applies only to JSON-LD aliases,
non-JSON formats are rejected up front:

| Format | Builder method | JSON-LD aliases | SPARQL aliases |
|--------|----------------|-----------------|----------------|
| Typed JSON | `FormatterConfig::typed_json()` | applies | **applies** (cross-language typed shape) |
| SPARQL Results JSON | `FormatterConfig::sparql_json()` | applies | **applies** (cross-language SPARQL Results shape) |
| Agent JSON | `FormatterConfig::agent_json()` | applies | **applies** (cross-language agent envelope; honours `with_max_bytes(...)`) |
| JSON-LD | `FormatterConfig::jsonld()` | applies | **skipped** — SPARQL Results JSON default kept |
| TSV / CSV / SPARQL XML / RDF/XML | (any non-JSON `OutputFormat`) | rejected at `.execute()` with `MultiQueryError::UnsupportedFormat` | rejected |

The "JSON-LD applies only to JSON-LD aliases" rule keeps `--normalize-arrays`
(which builds `FormatterConfig::jsonld().with_normalize_arrays()`) from
silently coercing SPARQL `SELECT` results out of SPARQL Results JSON. If
you want a unified shape across both languages, pick `TypedJson`,
`SparqlJson`, or `AgentJson` instead.

Non-JSON formats (TSV, CSV, SPARQL XML, RDF/XML) are rejected at
`.execute()` time with `MultiQueryError::UnsupportedFormat` — a multi-query
envelope can't embed byte/string payloads inside its JSON `results` map.

### Over HTTP

The HTTP handler picks a format from request headers in this precedence
order (most specific wins):

1. **`Fluree-Output-Format` header** — explicit per-alias selector.
   `json` keeps per-language defaults; `typed-json` selects
   [`FormatterConfig::typed_json`]. Unknown values return `400 Bad
   Request`. This is what the CLI's `--format` flag sends on the wire.
2. **`Fluree-Normalize-Arrays: true`** — layers array normalization on
   the chosen format (or on the default JSON-LD shape when no
   `Fluree-Output-Format` is set). Matches the CLI's
   `--normalize-arrays`.
3. **`Accept` header** — standard content negotiation.
   `application/vnd.fluree.agent+json` selects
   [`FormatterConfig::agent_json`] (honouring `Fluree-Max-Bytes` if set).

`Accept` values that produce byte/string payloads — `text/tab-separated-values`,
`text/csv`, `application/sparql-results+xml`, `application/rdf+xml` —
are rejected with **406 Not Acceptable** since they can't be embedded
inside the envelope's JSON `results` map.

| Header configuration | Effect |
|----------------------|--------|
| (none) | Per-language defaults (JSON-LD aliases → JSON-LD, SPARQL aliases → SPARQL JSON). |
| `Fluree-Output-Format: json` | Same as default. |
| `Fluree-Output-Format: typed-json` | All aliases format as typed JSON. |
| `Fluree-Normalize-Arrays: true` | Default JSON-LD shape with single-value properties wrapped in arrays (applies to JSON-LD aliases). |
| `Fluree-Output-Format: typed-json` + `Fluree-Normalize-Arrays: true` | Typed JSON with array normalization. |
| `Accept: application/vnd.fluree.agent+json` | All aliases format as Agent JSON. `Fluree-Max-Bytes` sets per-alias byte budget. |
| `Accept: text/csv`, `Accept: application/rdf+xml`, etc. | **406 Not Acceptable**. |

---

## Examples

### Minimal — two JSON-LD queries, no envelope settings

```json
{
  "queries": {
    "all_people": {
      "language": "jsonld",
      "query": {
        "@context": { "ex": "http://example.org/" },
        "from":     "myledger",
        "select":   ["?id", "?name"],
        "where":    { "@id": "?id", "ex:type": "ex:Person", "ex:name": "?name" }
      }
    },
    "all_orders": {
      "language": "jsonld",
      "query": {
        "@context": { "ex": "http://example.org/" },
        "from":     "myledger",
        "select":   ["?id", "?total"],
        "where":    { "@id": "?id", "ex:type": "ex:Order",  "ex:total": "?total" }
      }
    }
  }
}
```

### Shared `@context` lifted to envelope

```json
{
  "@context": { "ex": "http://example.org/" },
  "queries": {
    "all_people": {
      "language": "jsonld",
      "query": {
        "from":   "myledger",
        "select": ["?id", "?name"],
        "where":  { "@id": "?id", "ex:type": "ex:Person", "ex:name": "?name" }
      }
    },
    "all_orders": {
      "language": "jsonld",
      "query": {
        "from":   "myledger",
        "select": ["?id", "?total"],
        "where":  { "@id": "?id", "ex:type": "ex:Order", "ex:total": "?total" }
      }
    }
  }
}
```

### Mixed-language with envelope `@context`

The envelope `@context` lifts to **both** JSON-LD sub-queries (as JSON-LD
context inheritance) and SPARQL sub-queries (as `PREFIX` injection — the
SPARQL query below has no `PREFIX` of its own, so envelope prefixes are
injected).

```json
{
  "@context": { "ex": "http://example.org/" },
  "queries": {
    "by_jsonld": {
      "language": "jsonld",
      "query": {
        "from": "myledger",
        "select": ["?name"],
        "where": { "@id": "?p", "ex:name": "?name" }
      }
    },
    "by_sparql": {
      "language": "sparql",
      "query": "SELECT ?name FROM <myledger> WHERE { ?p ex:name ?name }"
    }
  }
}
```

### Per-sub-query opts override

`opts.meta` is enabled at the envelope so every sub-query reports
tracking, but `bob` overrides with a tighter per-sub-query timeout.

```json
{
  "opts": { "meta": true, "timeoutMs": 30000 },
  "queries": {
    "alice": {
      "language": "jsonld",
      "query": { "from": "myledger", "select": ["?name"],
                 "where": { "@id": "?p", "ex:name": "?name" } }
    },
    "bob": {
      "language": "jsonld",
      "query": { "from": "myledger", "select": ["?age"],
                 "where": { "@id": "?p", "ex:age": "?age" } },
      "opts": { "timeoutMs": 5000 }
    }
  }
}
```

### Pinning to a wall-clock moment across ledgers

```json
{
  "asOf": "2024-01-01T12:00:00Z",
  "queries": {
    "users": {
      "language": "jsonld",
      "query": { "from": "users:main",
                 "select": ["?name"],
                 "where": { "@id": "?u", "ex:name": "?name" } }
    },
    "orders": {
      "language": "jsonld",
      "query": { "from": "orders:main",
                 "select": ["?id"],
                 "where": { "@id": "?o", "ex:orderId": "?id" } }
    }
  }
}
```

Response (abbreviated):

```json
{
  "status": "ok",
  "snapshot": {
    "asOf":    "2024-01-01T12:00:00Z",
    "ledgers": { "users:main": 4108, "orders:main": 9421 }
  },
  "results": { "users": [ ... ], "orders": [ ... ] }
}
```

Each ledger's `t` is its latest commit at or before `2024-01-01T12:00:00Z`
— neither value is "the same `t`" because that's not a meaningful
concept across ledgers.

### Partial failure

```json
{
  "queries": {
    "good": {
      "language": "jsonld",
      "query": { "from": "myledger", "select": ["?name"],
                 "where": { "@id": "?p", "ex:name": "?name" } }
    },
    "bad": {
      "language": "sparql",
      "query":    "SELECT ?x FROM <myledger> WHERE { this is not SPARQL }"
    }
  }
}
```

Response (HTTP 200):

```json
{
  "status":   "partial",
  "snapshot": { "asOf": "...", "ledgers": { "myledger": 42 } },
  "results":  { "good": [ ... ] },
  "errors":   {
    "bad": { "code": "api_error", "message": "SPARQL parse error: ..." }
  }
}
```

---

## Limitations

The following are not supported and produce documented behaviour rather than silent partial success.

- **History queries are rejected.** A JSON-LD body with a `to` field or a SPARQL query with `FROM <a@t:1> TO <a@t:latest>` is rejected with `400 Bad Request`. History spans a `t`-range; the envelope's shared-snapshot contract has no meaning over a range. Use single queries against `/query` for history.
- **Envelope-level fuel budget is rejected.** `opts.max-fuel` / `max_fuel` / `maxFuel` at the envelope level is rejected with `400 Bad Request` because the dispatcher does not currently share a fuel budget across parallel sub-queries. Per-sub-query `opts.max-fuel` works unchanged.
- **Cancellation latency is bounded by in-flight storage reads.** When the envelope deadline fires, the dispatcher aborts the `JoinSet`; in-flight blocking storage reads still complete before the dropped future is observed (cache hit: under 5 ms; remote S3 range read: up to a few hundred ms). No new work starts after cancellation.
- **`opts.t` is rejected at every level inside the envelope.** Pin time via `from` (e.g., `from: "ledger@t:42"`) or envelope `asOf`.
- **Response size cap is enforced at assembly, not throughout dispatch.** Each sub-query result is checked against the per-sub-query cap once after it returns, and the assembler enforces the envelope-level cap as it stitches the response. Peak memory during dispatch is bounded by `max_concurrency × max_subquery_response_bytes`, which can exceed the envelope cap while individual sub-queries are running.
- **SPARQL sub-queries do not consume merged policy opts (identity, policy-class, policy, policy-values, default-allow).** The headers ride through the transport, the server folds them into the envelope's top-level `opts`, and the envelope → sub-query opts merge carries them into each sub-query's opts — but the connection-scoped SPARQL dispatch path (`query_from().sparql()`) does not read body opts. JSON-LD sub-queries get full policy threading via `apply_auth_identity_to_opts`; SPARQL sub-queries observe bearer ledger-scope only. This is the same gap that exists for single-query connection-scoped SPARQL (`POST /query` with `Content-Type: application/sparql-query` and an inline `FROM`).
- **Output formats are limited to JSON-producing shapes.** The envelope always assembles a JSON response body, so TSV, CSV, SPARQL Results XML, and RDF/XML are not available per-alias. Use single queries against `/query` when you need a byte/string payload.

---

## See also

- [Query endpoint reference](endpoints.md#post-query) — single-query
  `POST /query`
- [`fluree multi-query` CLI](../cli/multi-query.md)
- [Headers and content types](headers.md)
- [Signed requests (JWS/VC)](signed-requests.md)
- [Errors and status codes](errors.md)
- [SPARQL compliance notes](../contributing/sparql-compliance.md)


---

# api/streaming-query.md

# Streaming query (NDJSON)

> **Endpoints:**
> - `POST /v1/fluree/stream/query/<ledger...>` — ledger-scoped (ledger in path)
> - `POST /v1/fluree/stream/query` — connection-scoped (ledger from `from`/`FROM`)
>
> **Content type (response):** `application/x-ndjson`
> **Status:** v1 — SELECT only; some shapes explicitly rejected (see [Unsupported shapes](#unsupported-shapes)).

Stream SELECT results incrementally as newline-delimited JSON ("NDJSON")
instead of buffering the entire result set into a single JSON response body.
Each line is one self-describing record. A wall-clock **heartbeat** keeps the
connection alive past proxy idle timeouts (e.g. CloudFront/ALB ~60s) during
long-running queries.

This is a separate endpoint from [`/query`](endpoints.md#post-query); the
standard buffered query path is unchanged.

## When to use it

- **Large result sets** — start consuming rows before the whole result is
  built, and avoid holding the entire serialized response in memory.
- **Long-running queries behind a proxy** — heartbeats keep the connection from
  being killed by an idle timeout while the query is still executing.
- **Pipelined clients** — process each row as it arrives (ETL, export, UI
  incremental render).

## When *not* to use it

- You need a single JSON document, ASK/CONSTRUCT/DESCRIBE, `selectOne`, JSON-LD
  hydration, or history (`to`) queries — use [`/query`](endpoints.md#post-query).
- Blocking operators dominate the query. `ORDER BY` / `GROUP BY` / aggregates
  buffer internally and emit in a burst at the end, so streaming yields no rows
  until they finish (heartbeats still flow — see [Heartbeats](#heartbeats)).

## Request

Two forms, mirroring `/query`:

- **Ledger-scoped** — `POST /stream/query/<ledger...>`. The ledger is the greedy
  path tail, exactly like [`/query/<ledger...>`](endpoints.md#post-query).
- **Connection-scoped** — `POST /stream/query` (no path ledger). The ledger(s)
  come entirely from the request: JSON-LD `from`/`fromNamed` (or the
  `Fluree-Ledger` header), or SPARQL `FROM`/`FROM NAMED`. This is always the
  connection/dataset path; a request with no ledger spec is rejected `4xx`.

Either form is content-type-negotiated:

- `Content-Type: application/json` — a JSON-LD query document.
- `Content-Type: application/sparql-query` — a raw SPARQL `SELECT` string.

```bash
# Ledger-scoped
curl -N -X POST http://localhost:8090/v1/fluree/stream/query/my/ledger \
  -H 'Content-Type: application/json' \
  -d '{"@context":{"ex":"http://example.org/"},
       "select":["?name"],"where":{"@id":"?s","ex:name":"?name"}}'

# Connection-scoped (ledgers from `from`)
curl -N -X POST http://localhost:8090/v1/fluree/stream/query \
  -H 'Content-Type: application/json' \
  -d '{"@context":{"ex":"http://example.org/"},"from":["a:main","b:main"],
       "select":["?name"],"where":{"@id":"?s","ex:name":"?name"}}'
```

> `curl -N` disables output buffering so records print as they arrive.

> **Note:** connection-scoped **SPARQL** does not apply per-request identity
> policy (it has no single ledger to resolve against), so a `/stream/query`
> SPARQL request carrying policy signals is rejected — use the ledger-scoped
> `/stream/query/<ledger>` (which enforces SPARQL policy) or `/query`.
> Connection-scoped **JSON-LD** policy is enforced normally.

## Response: the NDJSON record protocol

The response is a stream of newline-terminated JSON objects, one per line. The
`type` field discriminates the record kind.

| `type` | Shape | Meaning |
|--------|-------|---------|
| `head` | `{"type":"head","vars":["s","name"]}` | First record. The ordered output column names. Emitted before the first row is pulled, so the client learns the schema immediately. |
| `row` | `{"type":"row","row":{ ... }}` | One result row. The `row` body is a [SPARQL-Results-JSON binding object](https://www.w3.org/TR/sparql11-results-json/) (`{"name":{"type":"literal","value":"Alice"}}`) — byte-identical to the `bindings` entries `/query` would return. |
| `heartbeat` | `{"type":"heartbeat","t_ms":14982,"fuel":84.213}` | Keep-alive emitted during stalls. `fuel` is present only when fuel tracking is active. |
| `end` | `{"type":"end","rows":2,"t":42,"fuel":1.01,"time":"3.4ms"}` | **Success terminator.** Final row count plus `t`/`fuel`/`time` when tracked. |
| `error` | `{"type":"error","error":{"code":"timeout","message":"..."},"rows":1}` | **Failure terminator.** Carries a machine-readable `code` (see below), a human `message`, and rows emitted before the failure. Emitted *instead of* `end`. |

Example stream:

```
{"type":"head","vars":["name"]}
{"type":"row","row":{"name":{"type":"literal","value":"Alice"}}}
{"type":"row","row":{"name":{"type":"literal","value":"Bob"}}}
{"type":"end","rows":2,"t":42,"fuel":1.02,"time":"2.1ms"}
```

### Terminal record requirement (read this)

Every successful stream ends with exactly one `end` record; every failed stream
ends with exactly one `error` record. **A client MUST treat the absence of a
terminal record as a failure** (a truncated/dropped stream), not as success.

This matters because the HTTP status is committed to `200 OK` as soon as the
first byte is flushed. If the connection drops mid-stream (proxy timeout,
network failure, server crash), the bytes received so far are
indistinguishable from a complete result *unless* you require the explicit
terminator. Do not assume "connection closed cleanly" means "all rows
received" — require `end`.

### Error codes

The `error` record's `error.code` is a stable, machine-readable string so
clients can branch on the failure kind without parsing the message:

| `code` | Meaning |
|--------|---------|
| `timeout` | The server query timeout fired (the query ran too long). |
| `fuel_exhausted` | The query exceeded its `max-fuel` budget. |
| `cancelled` | The query was cancelled (e.g. the client disconnected). |
| `invalid_query` | The query was rejected at plan/validation time. |
| `resource_limit` | A non-fuel resource limit was hit. |
| `internal` | An unexpected server-side error. |

A `code` you don't recognize should be treated as a generic failure. Note that
because the `200 OK` is committed before execution, even a request that fails
the fuel floor immediately is reported as a single `error` terminal on a `200`
stream — not as a `4xx`. (`4xx` is reserved for failures detected *before* the
stream starts: parse errors and [unsupported shapes](#unsupported-shapes).)

### Heartbeats

When no record has flowed for the heartbeat interval (default 15s, configurable
via `FLUREE_STREAM_HEARTBEAT_MS` / `--stream-heartbeat-ms`; `0` disables), the
server emits a `heartbeat` record. This is driven by a wall-clock timer in the
transport layer, independent of query execution, so it fires even while a
blocking operator (a large `ORDER BY` / `GROUP BY` drain) is producing no rows.
Set the interval below your fronting proxy's idle timeout. When the query was submitted with fuel
tracking, the heartbeat carries the live running `fuel` total, which climbs as
scans charge — a useful "still making progress" signal during a long stall.

Clients should ignore unknown record types (forward compatibility) and simply
skip `heartbeat` records.

## Unsupported shapes

The streaming endpoint covers SELECT result rows only. The following are
rejected with a `4xx` **before** the stream starts (so you get a normal JSON
error, not a `200` stream), and should use [`/query`](endpoints.md#post-query):

- **ASK** — boolean result, nothing to stream.
- **CONSTRUCT / DESCRIBE** — produce an RDF graph, not solution rows.
- **`selectOne`** — single-object JSON-LD shape.
- **JSON-LD hydration** — needs async per-row database expansion during
  formatting.
- **History (`to` / `FROM … TO …`)** — top-level JSON-LD `to` and the SPARQL
  history range use a distinct history execution path.

## Auth, policy, and dataset behavior

The streaming endpoint routes through the same execution path as `/query` and
enforces policy the same way `/query` does — which differs by query language and
route, exactly as on `/query`:

- **No policy, single ledger** — runs the lean single-ledger path.
- **`from`/`fromNamed` (JSON-LD), SPARQL `FROM`/`FROM NAMED`, multi-ledger, or a
  policy input** — **routes to the connection/dataset path**, which builds a
  policy-wrapped dataset and enforces per-graph policy. A restricted
  identity/policy-class streams strictly fewer rows than an unrestricted one.

**JSON-LD policy** is enforced on both endpoint forms (ledger-scoped and
connection). Inputs: `opts.identity` / `opts.policy-class`, the server
`default_policy_class`, or `Fluree-Policy*` / `Fluree-Identity` headers.

**SPARQL policy** is enforced **only on the ledger-scoped route**
(`/stream/query/<ledger>`). SPARQL has no body `opts`, so policy arrives via the
resolved identity (bearer / `Fluree-Identity`) and the `Fluree-Policy*` /
`Fluree-Default-Allow` headers; `FROM`/`FROM NAMED` select named graphs *within*
the path ledger. The **connection-scoped** SPARQL form has no single ledger to
resolve an identity against, so it **rejects** explicit policy signals (the
`Fluree-Identity` / `Fluree-Policy*` / `Fluree-Default-Allow` headers) rather
than run them unenforced — use the ledger-scoped route or `/query`. This matches
`/query`, where connection SPARQL is likewise not identity-policy-scoped.

> **`default_policy_class`** is a JSON-LD-path setting: it is applied to JSON-LD
> queries (on both `/query` and streaming) but **not** to SPARQL on either
> endpoint. (Making it global across query languages would be a separate change
> affecting `/query` too.)

- **Bearer scope** — a token must be authorized for the path ledger and every
  ledger referenced via `from`/`fromNamed` / SPARQL `FROM`; out-of-scope
  requests return `404` (no existence leak), same as `/query`.
- **`fluree-min-t`** freshness barriers and the stored default-context
  injection are applied before planning, matching `/query`.

The only SPARQL dataset feature still rejected outright is the **history range**
(`FROM <…> TO <…>`) — use [`/query`](endpoints.md#post-query) for that.

## Fuel and tracking

The endpoint tracks fuel and time by default. `max-fuel` is honored from JSON-LD
`opts.max-fuel` and, for SPARQL (which has no body `opts`), from the
`Fluree-Max-Fuel` header. The running fuel total rides on `heartbeat` records
and the final total on the `end` record; a `max-fuel` overrun surfaces as a
`{"type":"error","error":{"code":"fuel_exhausted"}}` terminal.

## Compression

NDJSON streaming relies on records being flushed promptly. Do **not** place a
buffering response-compression layer in front of `/stream/*`; gzip middleware
that coalesces small writes defeats both incremental delivery and the
heartbeat. The server sends `Cache-Control: no-transform` to discourage
intermediaries from re-encoding the body.

## Consuming the stream

### JavaScript / TypeScript

```ts
const res = await fetch(`/v1/fluree/stream/query/${ledger}`, {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify(query),
});

const reader = res.body!.getReader();
const decoder = new TextDecoder();
let buf = "";
let sawTerminal = false;

while (true) {
  const { value, done } = await reader.read();
  if (done) break;
  buf += decoder.decode(value, { stream: true });
  let nl: number;
  while ((nl = buf.indexOf("\n")) >= 0) {
    const line = buf.slice(0, nl).trim();
    buf = buf.slice(nl + 1);
    if (!line) continue;
    const rec = JSON.parse(line);
    switch (rec.type) {
      case "head": /* rec.vars */ break;
      case "row": handleRow(rec.row); break;
      case "heartbeat": /* progress: rec.fuel */ break;
      case "end": sawTerminal = true; break;
      case "error": sawTerminal = true; throw new Error(rec.error.message);
    }
  }
}
// REQUIRED: a stream that ends without `end`/`error` was truncated.
if (!sawTerminal) throw new Error("stream truncated before terminal record");
```

### Rust (reqwest)

This is an HTTP *client* consuming the endpoint over the wire. To produce the
same NDJSON stream **in-process** from the `fluree-db-api` library (without the
HTTP server), use `Fluree::plan_stream_query` + `run_stream_query` — see
[Streaming query results (NDJSON)](../getting-started/rust-api.md#streaming-query-results-ndjson)
in the Rust library guide.

```rust
let resp = client
    .post(format!("{base}/v1/fluree/stream/query/{ledger}"))
    .json(&query)
    .send()
    .await?;

let mut lines = tokio::io::BufReader::new(
    tokio_util::io::StreamReader::new(
        resp.bytes_stream().map_err(std::io::Error::other),
    ),
)
.lines();

let mut saw_terminal = false;
while let Some(line) = lines.next_line().await? {
    if line.is_empty() { continue; }
    let rec: serde_json::Value = serde_json::from_str(&line)?;
    match rec["type"].as_str() {
        Some("row") => handle_row(&rec["row"]),
        Some("end") => { saw_terminal = true; }
        Some("error") => { saw_terminal = true; anyhow::bail!("{}", rec["error"]["message"]); }
        _ => {} // head, heartbeat, unknown
    }
}
anyhow::ensure!(saw_terminal, "stream truncated before terminal record");
```

### CLI (`fluree query --format ndjson`)

The [`fluree query`](../cli/query.md) command consumes this stream for you.
`--format ndjson` prints one **bare** binding object per line (the inner `row`
body, with `head`/`heartbeat`/terminal consumed internally); add `--envelope`
to print the full record protocol verbatim. The CLI exits non-zero on an `error`
terminal or a truncated stream, and exits cleanly on a closed downstream pipe.

```bash
# bare rows, jq-friendly
fluree query --format ndjson 'SELECT ?name WHERE { ?s <http://example.org/name> ?name }'

# verbatim protocol (head/row/heartbeat/end/error)
fluree query --remote origin --format ndjson --envelope -f big-select.rq
```

For a **local** ledger the CLI drives the in-process producer
(`run_stream_query`) directly; with `--remote` it POSTs to this endpoint and
streams the response. Time travel (`--at`) and per-request policy on the
streaming path are supported only via `--remote` (they route through the
server's dataset path). See [`fluree query`](../cli/query.md#ndjson-streaming)
for the full scope.

## Relationship to `/query`

`/stream/query/<ledger>` is purpose-built for incremental delivery; it does not
replace [`/query`](endpoints.md#post-query). The standard endpoint remains the
path for ASK/CONSTRUCT/DESCRIBE, hydration, history (JSON-LD `to` or SPARQL
`FROM … TO …`), and any client that wants a single buffered JSON (or
TSV/CSV/XML) document.


---

# api/headers.md

# Headers, Content Types, and Request Sizing

This document covers HTTP headers, content type negotiation, request size limits, and related considerations for the Fluree HTTP API.

## Request Headers

### Content-Type

Specifies the format of the request body.

**Supported Values:**

**JSON-LD Transactions and Queries:**
```http
Content-Type: application/json
```
Default for JSON-LD transactions and JSON-LD queries.

```http
Content-Type: application/ld+json
```
Explicit JSON-LD content type.

**SPARQL Queries:**
```http
Content-Type: application/sparql-query
```
For SPARQL SELECT, ASK, CONSTRUCT queries.

```http
Content-Type: application/sparql-update
```
For SPARQL UPDATE operations. See [SPARQL Transactions](../transactions/sparql.md) for supported operations.

**RDF Formats:**
```http
Content-Type: text/turtle
```
For Turtle RDF format transactions. Supported on `/insert` (fast direct path) and `/upsert`.

```http
Content-Type: application/trig
```
For TriG format transactions with named graphs (GRAPH blocks). **Only supported on `/upsert`** - returns 400 error on `/insert` because named graph ingestion requires the upsert path.

```http
Content-Type: application/n-triples
```
For N-Triples format (future support).

```http
Content-Type: application/rdf+xml
```
For RDF/XML format (future support).

### Accept

Specifies the desired response format.

**Supported Values:**

```http
Accept: application/json
```
Compact JSON format (default).

```http
Accept: application/ld+json
```
Full JSON-LD with @context.

```http
Accept: application/sparql-results+json
```
SPARQL JSON Results format (for SPARQL queries).

```http
Accept: application/sparql-results+xml
```
SPARQL XML Results format (for SPARQL SELECT/ASK queries).

```http
Accept: text/turtle
```
Turtle RDF format (for CONSTRUCT queries).

```http
Accept: application/rdf+xml
```
RDF/XML graph format (for CONSTRUCT/DESCRIBE queries).

```http
Accept: application/vnd.fluree.agent+json
```
Agent JSON format — optimized for LLM/agent consumption. Returns a self-describing envelope with schema, compact rows, and pagination support. See [Output Formats](../query/output-formats.md#agent-json-format) for details.

Use the `Fluree-Max-Bytes` header to set a byte budget for response truncation:
```http
Fluree-Max-Bytes: 32768
```

```http
Accept: application/n-triples
```
N-Triples format (future support).

**Multiple Accept Values:**

You can specify multiple formats with quality values:

```http
Accept: application/ld+json; q=1.0, application/json; q=0.8
```

The server will choose the best match based on quality values and support.

### Authorization

Authentication credentials. Only required when the server has authentication enabled for the relevant endpoint group (see [Configuration](../operations/configuration.md)).

**Bearer Token (Ed25519 JWS or OIDC):**
```http
Authorization: Bearer eyJhbGciOiJFZERTQSIsImp3ayI6eyJrdHkiOiJPS1AiLCJjcnYiOiJFZDI1NTE5IiwieCI6Ii4uLiJ9fQ...
```

The server automatically dispatches to the correct verification path based on the token header:
- Tokens with an embedded `jwk` field use the Ed25519 verification path
- Tokens with a `kid` field use the OIDC/JWKS verification path (requires `oidc` feature)

**Signed Requests:**

For JWS/VC signed request bodies, set Content-Type to `application/jose`:
```http
Content-Type: application/jose
```

See [Signed Requests](signed-requests.md) for details.

### Content-Length

The server requires Content-Length for all POST requests:

```http
Content-Length: 1234
```

Most HTTP clients set this automatically.

### Accept-Encoding

Request compressed responses:

```http
Accept-Encoding: gzip, deflate
```

The server will compress responses when appropriate, reducing bandwidth usage.

**Response Header:**
```http
Content-Encoding: gzip
```

### User-Agent

Identify your client application:

```http
User-Agent: MyApp/1.0.0 (https://example.com)
```

Helpful for server logs and troubleshooting.

### X-Request-ID

Client-supplied request ID for tracing:

```http
X-Request-ID: abc-123-def-456
```

The server will include this in logs and response headers for correlation. When a request queues background indexing work, the copied `X-Request-ID` also appears on the background indexer worker logs so you can connect the foreground request and later indexing activity in plain log search.

### Fluree-Min-T

Request a hard read-after-write guarantee for query and explain endpoints:

```http
Fluree-Min-T: 42
```

Before executing the request, the server refreshes the referenced ledger(s) until each has reached at least the requested transaction time. If the target `t` is not visible before `query_min_t_timeout_ms`, the server returns a read-after-write timeout error. JSON-LD query bodies may also specify the same requirement as `opts.min-t`, `opts.min_t`, or `opts.minT`; body opts take precedence over the header for that query body.

Numeric time-travel snapshots such as `from: "ledger:main@t:42"` also wait until that `t` is visible, then query that pinned snapshot. `Fluree-Min-T` is useful when the query itself reads current HEAD but must not run until a known transaction has arrived.

The header must resolve to at least one target ledger. On a ledger-scoped endpoint the endpoint's ledger is used; on the connection-scoped query endpoint the target comes from the query's `FROM` clause. Sending `Fluree-Min-T` to the connection-scoped endpoint with no `FROM` (and no other resolvable ledger) is rejected with `400 Bad Request` rather than silently ignored.

## Response Headers

### Content-Type

Indicates the format of the response body:

```http
Content-Type: application/json; charset=utf-8
```

### Content-Length

Size of the response body in bytes:

```http
Content-Length: 5678
```

### X-Fluree-T

The transaction time of the data returned (for queries):

```http
X-Fluree-T: 42
```

Useful for tracking which version of data was queried.

### X-Fluree-Commit

The commit ContentId of the data returned:

```http
X-Fluree-Commit: abc123def456789...
```

### ETag

Entity tag for caching:

```http
ETag: "abc123def456"
```

Can be used with `If-None-Match` for conditional requests.

### Cache-Control

Caching directives:

**For current queries:**
```http
Cache-Control: no-cache
```

**For historical queries:**
```http
Cache-Control: public, max-age=31536000, immutable
```

Historical queries are immutable and cache indefinitely.

### X-RateLimit Headers

Rate limit information (if enabled):

```http
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 95
X-RateLimit-Reset: 1642857600
```

### X-Request-ID

Echo of client-supplied request ID or server-generated ID:

```http
X-Request-ID: abc-123-def-456
```

### X-Response-Time

Server processing time in milliseconds:

```http
X-Response-Time: 45
```

## Content Type Details

### JSON-LD (application/json, application/ld+json)

**Request Example:**

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "schema": "http://schema.org/"
  },
  "@graph": [
    {
      "@id": "ex:alice",
      "@type": "schema:Person",
      "schema:name": "Alice"
    }
  ]
}
```

**Compact vs Expanded:**

`application/json` returns compact JSON:
```json
[
  { "name": "Alice" }
]
```

`application/ld+json` returns with full context:
```json
{
  "@context": {
    "name": "http://schema.org/name"
  },
  "@graph": [
    { "name": "Alice" }
  ]
}
```

### SPARQL Query (application/sparql-query)

**Request Example:**

```sparql
PREFIX ex: <http://example.org/ns/>
PREFIX schema: <http://schema.org/>

SELECT ?name
FROM <mydb:main>
WHERE {
  ?person a schema:Person .
  ?person schema:name ?name .
}
```

Plain text SPARQL query in the request body.

### SPARQL Results JSON (application/sparql-results+json)

**Response Example:**

```json
{
  "head": {
    "vars": ["name"]
  },
  "results": {
    "bindings": [
      {
        "name": {
          "type": "literal",
          "value": "Alice",
          "datatype": "http://www.w3.org/2001/XMLSchema#string"
        }
      }
    ]
  }
}
```

Follows W3C SPARQL 1.1 Query Results JSON Format specification.

### Turtle (text/turtle)

**Transaction Request:**

```turtle
@prefix ex: <http://example.org/ns/> .
@prefix schema: <http://schema.org/> .

ex:alice a schema:Person ;
  schema:name "Alice" ;
  schema:age 30 .
```

**CONSTRUCT Response:**

```turtle
@prefix ex: <http://example.org/ns/> .
@prefix schema: <http://schema.org/> .

ex:alice a schema:Person .
ex:alice schema:name "Alice" .
```

## Request Size Limits

### Default Limits

The server enforces size limits to prevent resource exhaustion:

**Transaction Requests:**
- Default limit: 10 MB
- Configurable: `--max-transaction-size`

**Query Requests:**
- Default limit: 1 MB
- Configurable: `--max-query-size`

**History Requests:**
- Default limit: 1 MB
- Configurable: `--max-history-size`

### Exceeding Limits

If a request exceeds size limits:

**Status Code:** `413 Payload Too Large`

**Response:**
```json
{
  "error": "Request body exceeds maximum size of 10485760 bytes",
  "status": 413,
  "@type": "err:http/PayloadTooLarge"
}
```

### Configuration

Set custom limits when starting the server:

```bash
./fluree-db-server \
  --max-transaction-size 20971520 \    # 20 MB
  --max-query-size 2097152 \           # 2 MB
  --max-response-size 104857600        # 100 MB
```

### Response Size Limits

The server also limits response sizes:

**Default limit:** 100 MB

If a query result exceeds the limit:

**Status Code:** `413 Payload Too Large`

**Response:**
```json
{
  "error": "Query result exceeds maximum response size",
  "status": 413,
  "@type": "err:http/ResponseTooLarge"
}
```

**Solution:** Use LIMIT and pagination:

```json
{
  "select": ["?name"],
  "where": [...],
  "limit": 1000,
  "offset": 0
}
```

## Compression

### Request Compression

Send compressed requests (for large transactions):

```http
Content-Encoding: gzip
Content-Type: application/json
```

The request body should be gzip-compressed JSON.

### Response Compression

Request compressed responses:

```http
Accept-Encoding: gzip, deflate
```

The server will compress responses when:
- Client accepts compression
- Response is larger than threshold (typically 1 KB)
- Content-Type is compressible

**Response Headers:**
```http
Content-Encoding: gzip
Vary: Accept-Encoding
```

**Compression Benefits:**
- Reduced bandwidth usage (typically 70-90% for JSON)
- Faster response times on slower connections
- Lower costs for cloud deployments

## Character Encoding

All text content uses UTF-8 encoding.

**Request:**
```http
Content-Type: application/json; charset=utf-8
```

**Response:**
```http
Content-Type: application/json; charset=utf-8
```

Unicode characters are supported in:
- IRIs
- Literal values
- Property names
- Comments

## CORS Headers

For web browser access, the server supports Cross-Origin Resource Sharing (CORS).

### CORS Request Headers

**Preflight Request:**
```http
OPTIONS /query HTTP/1.1
Origin: https://example.com
Access-Control-Request-Method: POST
Access-Control-Request-Headers: Content-Type
```

### CORS Response Headers

**Preflight Response:**
```http
Access-Control-Allow-Origin: https://example.com
Access-Control-Allow-Methods: GET, POST, OPTIONS
Access-Control-Allow-Headers: Content-Type, Authorization
Access-Control-Max-Age: 86400
```

**Actual Response:**
```http
Access-Control-Allow-Origin: https://example.com
Access-Control-Allow-Credentials: true
```

### CORS Configuration

Configure CORS when starting the server:

```bash
./fluree-db-server \
  --cors-origin "https://example.com" \
  --cors-methods "GET,POST,OPTIONS" \
  --cors-headers "Content-Type,Authorization"
```

**Allow all origins (development only):**
```bash
./fluree-db-server --cors-origin "*"
```

Never use `--cors-origin "*"` in production with credentials.

## Caching Headers

### ETag and Conditional Requests

The server supports ETags for efficient caching.

**Initial Request:**
```http
GET /ledgers/mydb:main HTTP/1.1
```

**Response:**
```http
HTTP/1.1 200 OK
ETag: "abc123def456"
Cache-Control: no-cache
```

**Conditional Request:**
```http
GET /ledgers/mydb:main HTTP/1.1
If-None-Match: "abc123def456"
```

**Not Modified Response:**
```http
HTTP/1.1 304 Not Modified
ETag: "abc123def456"
```

### Immutable Historical Data

Historical queries with time specifiers are immutable:

**Query:**
```http
POST /query HTTP/1.1
{"from": "mydb:main@t:100", ...}
```

**Response:**
```http
HTTP/1.1 200 OK
Cache-Control: public, max-age=31536000, immutable
ETag: "mydb:main@t:100:query-hash"
```

Clients can cache these responses indefinitely.

## Custom Headers

### X-Fluree-Fuel-Limit

Set query fuel limit to prevent runaway queries:

```http
X-Fluree-Fuel-Limit: 1000000
```

See [Tracking and Fuel Limits](../query/tracking-and-fuel.md) for details.

### X-Fluree-Timeout

Set query timeout in milliseconds:

```http
X-Fluree-Timeout: 30000
```

### X-Fluree-Policy

Specify a policy to apply (if authorized):

```http
X-Fluree-Policy: ex:restrictive-policy
```

## Best Practices

### 1. Always Set Content-Type

Explicitly set Content-Type for all requests:

```http
Content-Type: application/json
```

### 2. Accept Compression

Always request compression for better performance:

```http
Accept-Encoding: gzip, deflate
```

### 3. Use Appropriate Accept Headers

Request the format you need:

```http
Accept: application/json
```

### 4. Include User-Agent

Identify your application:

```http
User-Agent: MyApp/1.0.0
```

### 5. Handle ETags

Implement ETag caching for frequently accessed resources:

```javascript
const etag = localStorage.getItem('ledger-etag');
if (etag) {
  headers['If-None-Match'] = etag;
}
```

### 6. Monitor Rate Limits

Check rate limit headers and back off when needed:

```javascript
const remaining = response.headers.get('X-RateLimit-Remaining');
if (remaining < 10) {
  // Slow down requests
}
```

### 7. Use Request IDs

Include request IDs for tracing:

```http
X-Request-ID: uuid-v4-here
```

## Related Documentation

- [Overview](overview.md) - API overview
- [Endpoints](endpoints.md) - Endpoint reference
- [Signed Requests](signed-requests.md) - Authentication
- [Errors](errors.md) - Error handling


---

# api/signed-requests.md

# Signed Requests (JWS/VC)

Fluree supports cryptographically signed requests using **JSON Web Signatures (JWS)** and **Verifiable Credentials (VC)**. This provides tamper-proof authentication and enables trustless data exchange.

**Note:** Requires the `credential` feature flag. See [Compatibility and Feature Flags](../reference/compatibility.md#fluree-db-api-features).

## Why Sign Requests?

Signed requests provide:

- **Authentication**: Prove the identity of the request sender
- **Integrity**: Ensure the request hasn't been tampered with
- **Non-repudiation**: Sender cannot deny sending the request
- **Authorization**: Cryptographically link requests to specific identities
- **Auditability**: Complete audit trail of who did what

## JSON Web Signatures (JWS)

JWS is an IETF standard (RFC 7515) for representing digitally signed content as JSON.

### JWS Structure

A JWS consists of three parts:

1. **Protected Header**: Metadata about the signature (base64url-encoded)
2. **Payload**: The actual content being signed (base64url-encoded)
3. **Signature**: Cryptographic signature (base64url-encoded)

**Compact Serialization:**
```
eyJhbGciOiJFZDI1NTE5In0.eyJmcm9tIjoibXlkYjptYWluIn0.c2lnbmF0dXJl
|_______header_______|.|______payload______|.|_signature_|
```

**JSON Serialization:**
```json
{
  "payload": "eyJmcm9tIjoibXlkYjptYWluIn0",
  "signatures": [
    {
      "protected": "eyJhbGciOiJFZDI1NTE5In0",
      "signature": "c2lnbmF0dXJl"
    }
  ]
}
```

### Supported Algorithm

Fluree uses **EdDSA (Ed25519)** for JWS verification. All signed requests must use `"alg": "EdDSA"` in the protected header.

## Creating Signed Requests

### Step 1: Prepare the Payload

Create your query or transaction as usual:

```json
{
  "@context": {
    "ex": "http://example.org/ns/"
  },
  "from": "mydb:main",
  "select": ["?name"],
  "where": [
    { "@id": "?person", "ex:name": "?name" }
  ]
}
```

### Step 2: Encode the Payload

Base64url-encode the JSON payload:

```javascript
const payload = JSON.stringify(query);
const encodedPayload = base64url.encode(payload);
```

### Step 3: Create the Protected Header

Create a header specifying the algorithm and key ID:

```json
{
  "alg": "EdDSA",
  "kid": "did:key:z6MkhaXgBZDvotDkL5257faiztiGiC2QtKLGpbnnEGta2doK"
}
```

Base64url-encode the header:

```javascript
const header = JSON.stringify({ alg: "EdDSA", kid: keyId });
const encodedHeader = base64url.encode(header);
```

### Step 4: Sign

Create the signing input and sign it:

```javascript
const signingInput = encodedHeader + "." + encodedPayload;
const signature = sign(signingInput, privateKey);
const encodedSignature = base64url.encode(signature);
```

### Step 5: Construct the JWS

Create the complete JWS:

**Compact Format:**
```javascript
const jws = encodedHeader + "." + encodedPayload + "." + encodedSignature;
```

**JSON Format:**
```json
{
  "payload": "<encodedPayload>",
  "signatures": [
    {
      "protected": "<encodedHeader>",
      "signature": "<encodedSignature>"
    }
  ]
}
```

### Step 6: Send the Request

Send the JWS to Fluree:

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/jose" \
  -d '{
    "payload": "eyJmcm9tIjoibXlkYjptYWluIn0...",
    "signatures": [{
      "protected": "eyJhbGciOiJFZDI1NTE5In0...",
      "signature": "c2lnbmF0dXJl..."
    }]
  }'
```

## Verifiable Credentials (VC)

Verifiable Credentials are a W3C standard for cryptographically verifiable digital credentials.

### VC Structure

A Verifiable Credential includes:

```json
{
  "@context": [
    "https://www.w3.org/2018/credentials/v1"
  ],
  "type": ["VerifiableCredential"],
  "issuer": "did:key:z6Mkh...",
  "issuanceDate": "2024-01-22T10:00:00Z",
  "credentialSubject": {
    "id": "did:key:z6Mkh...",
    "flureeAction": {
      "query": {
        "from": "mydb:main",
        "select": ["?name"],
        "where": [...]
      }
    }
  },
  "proof": {
    "type": "Ed25519Signature2020",
    "created": "2024-01-22T10:00:00Z",
    "verificationMethod": "did:key:z6Mkh...#z6Mkh...",
    "proofPurpose": "authentication",
    "proofValue": "z58DAdFfa9SkqZMVP..."
  }
}
```

### Creating a Verifiable Credential

Use a VC library to create signed credentials:

```javascript
import { issue } from '@digitalbazaar/vc';

const credential = {
  '@context': ['https://www.w3.org/2018/credentials/v1'],
  type: ['VerifiableCredential'],
  issuer: didKey,
  issuanceDate: new Date().toISOString(),
  credentialSubject: {
    id: didKey,
    flureeAction: {
      query: queryObject
    }
  }
};

const verifiableCredential = await issue({
  credential,
  suite: new Ed25519Signature2020({ key: keyPair }),
  documentLoader
});
```

### Sending a VC

Send the VC to Fluree:

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/vc+ld+json" \
  -d '{
    "@context": ["https://www.w3.org/2018/credentials/v1"],
    "type": ["VerifiableCredential"],
    ...
  }'
```

## Decentralized Identifiers (DIDs)

Fluree uses DIDs to identify public keys.

### Supported DID Methods

**did:key** - Public key embedded in the DID (recommended):
```
did:key:z6MkhaXgBZDvotDkL5257faiztiGiC2QtKLGpbnnEGta2doK
```

**did:web** - Web-based DID resolution:
```
did:web:example.com:users:alice
```

**did:ion** - ION network DIDs (future support):
```
did:ion:EiClkZMDxPKqC9c-umQfTkR8vvZ9JPhl_xLDI9Nfk38w5w
```

### DID Resolution

Fluree resolves DIDs to public keys:

1. **did:key**: Public key extracted directly from DID
2. **did:web**: Fetched from `https://example.com/.well-known/did.json`
3. **did:ion**: Resolved via ION network

### Public Key Resolution

Standalone server signed requests verify Ed25519 JWS material from the request
itself (for example embedded JWK / `did:key`) or configured OIDC/JWKS issuers.
There is no `/admin/keys` registration endpoint.

## Request Verification

### Verification Process

When Fluree receives a signed request:

1. **Extract the signature and header**
2. **Resolve the key ID (kid) to a public key**
3. **Verify the signature** using the public key
4. **Check expiration** (if `exp` claim present)
5. **Validate issuer** (if required)
6. **Apply authorization policies** based on DID

### Verification Failure

If verification fails:

**Status Code:** `401 Unauthorized`

**Response:**
```json
{
  "error": "Invalid signature",
  "status": 401,
  "@type": "err:auth/InvalidSignature"
}
```

## Key Management

### Generating Keys

**Ed25519 (EdDSA):**

```javascript
import { generateKeyPair } from '@stablelib/ed25519';

const keyPair = generateKeyPair();
// keyPair.publicKey - 32 bytes
// keyPair.secretKey - 64 bytes
```

### Storing Keys

**Secure Storage:**
- Hardware Security Modules (HSM)
- Key Management Services (AWS KMS, Azure Key Vault)
- Encrypted files with strong passphrases
- Hardware wallets for blockchain-based DIDs

**Never:**
- Store private keys in code
- Commit keys to version control
- Send keys over insecure channels
- Share keys between applications

### Key Rotation

Rotate keys regularly:

1. Generate new key pair
2. Register new public key with Fluree
3. Update client to use new key
4. Revoke old key after transition period
5. Remove old key from Fluree

## Authorization with Signed Requests

### Identity-Based Policies

Fluree policies can use the signer's DID for authorization:

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "f": "https://ns.flur.ee/db#"
  },
  "@id": "ex:admin-policy",
  "f:policy": [
    {
      "f:subject": "did:key:z6Mkh...",
      "f:action": ["query", "transact"],
      "f:allow": true
    }
  ]
}
```

### Role-Based Access

Link DIDs to roles:

```json
{
  "@id": "did:key:z6Mkh...",
  "@type": "ex:User",
  "ex:role": "ex:Administrator"
}
```

Policy checks the role:

```json
{
  "f:policy": [
    {
      "f:subject": { "ex:role": "ex:Administrator" },
      "f:action": "*",
      "f:allow": true
    }
  ]
}
```

## Code Examples

### JavaScript/TypeScript

```typescript
import jose from 'jose';

async function signQuery(query: object, privateKey: Uint8Array) {
  const payload = JSON.stringify(query);
  
  const jws = await new jose.SignJWT(query)
    .setProtectedHeader({ alg: 'EdDSA', kid: 'did:key:z6Mkh...' })
    .setIssuedAt()
    .setExpirationTime('5m')
    .sign(privateKey);
  
  return jws;
}

// Send signed request
const signedQuery = await signQuery(query, privateKey);
const response = await fetch('http://localhost:8090/v1/fluree/query', {
  method: 'POST',
  headers: { 'Content-Type': 'application/jose' },
  body: signedQuery
});
```

### Python

```python
from jwcrypto import jwk, jws
import json

def sign_query(query, private_key):
    # Create JWK from private key
    key = jwk.JWK.from_json(private_key)
    
    # Create JWS
    payload = json.dumps(query).encode('utf-8')
    jws_token = jws.JWS(payload)
    jws_token.add_signature(key, alg='EdDSA', 
                           protected=json.dumps({"kid": "did:key:z6Mkh..."}))
    
    return jws_token.serialize()

# Send signed request
signed_query = sign_query(query, private_key)
response = requests.post('http://localhost:8090/v1/fluree/query',
                        headers={'Content-Type': 'application/jose'},
                        data=signed_query)
```

## Best Practices

### 1. Use EdDSA (Ed25519)

EdDSA provides:
- Excellent security (128-bit security level)
- Fast signing and verification
- Small signatures (64 bytes)
- Deterministic (no random number generation needed)

### 2. Include Expiration

Always set an expiration time:

```json
{
  "alg": "EdDSA",
  "exp": 1642857600
}
```

### 3. Use Short Expiration Times

For interactive requests: 5-15 minutes
For batch processes: 1-24 hours
Never: No expiration

### 4. Rotate Keys Regularly

Rotate signing keys every 90-180 days.

### 5. Secure Key Storage

Use proper key management:
- Development: Encrypted local storage
- Production: HSM or KMS

### 6. Validate on Server

Never trust client-side validation alone. Fluree always validates signatures server-side.

### 7. Use HTTPS

Always use HTTPS with signed requests to prevent replay attacks.

### 8. Implement Nonce/JTI

Include a unique identifier to prevent replay:

```json
{
  "alg": "EdDSA",
  "jti": "unique-request-id-12345"
}
```

## Troubleshooting

### "Invalid Signature" Error

**Causes:**
- Wrong private key used
- Payload modified after signing
- Incorrect base64url encoding
- Algorithm mismatch

**Solution:** Verify the signing process end-to-end.

### "Key Not Found" Error

**Causes:**
- DID not registered with Fluree
- Incorrect key ID (kid) in header
- DID resolution failed

**Solution:** Register public key or check DID format.

### "Signature Expired" Error

**Causes:**
- Request sent after expiration time
- Clock skew between client and server

**Solution:** Use NTP to sync clocks, increase expiration time.

## Related Documentation

- [Overview](overview.md) - API overview
- [Endpoints](endpoints.md) - API endpoints
- [Headers](headers.md) - HTTP headers
- [Security](../security/README.md) - Policy and access control
- [Verifiable Data](../concepts/verifiable-data.md) - Verifiable credentials concepts


---

# api/errors.md

# Errors and Status Codes

This document provides a complete reference for HTTP status codes and error responses in the Fluree API.

## Error Response Format

`fluree-server` errors return a consistent JSON structure:

```json
{
  "error": "Human-readable error description",
  "status": 400,
  "@type": "err:db/BadRequest",
  "cause": {
    "error": "Optional nested cause",
    "status": 400,
    "@type": "err:db/JsonParse"
  }
}
```

**Fields:**
- `error`: Human-readable error message (primary diagnostic text)
- `status`: HTTP status code (numeric)
- `@type`: Compact error type IRI (stable, machine-readable category)
- `cause`: Optional nested cause chain (only present for select errors)

**Stability note:** clients (including the Fluree CLI) may pattern-match on substrings within the `error` field for targeted hints, so error messages should be stable across releases.

## HTTP Status Codes

### Success Codes (2xx)

#### 200 OK

The request succeeded.

**Used for:**
- Successful queries
- Successful transactions
- Successful GET requests

**Example:**
```json
{
  "t": 5,
  "timestamp": "2024-01-22T10:30:00.000Z",
  "commit_id": "bafybeig...commitT5"
}
```

**Multi-query envelope (`POST /multi-query`) returns 200 even when individual sub-queries fail.** The aggregate outcome is reported in the response body's `status` field (`"ok"` | `"partial"` | `"all_failed"`), with per-alias failures in `errors`. Clients should branch on `body.status` for the aggregate; HTTP 4xx / 5xx are reserved for envelope-level failures (validation, snapshot resolution, response-size cap). See [Multi-query envelope](multi-query.md#http-status-mapping) for the full mapping.

#### 201 Created

A new resource was created.

**Used for:**
- Ledger creation
- Index creation

**Example:**
```json
{
  "ledger_id": "mydb:main",
  "created": "2024-01-22T10:00:00.000Z"
}
```

#### 204 No Content

Request succeeded with no response body.

**Used for:**
- DELETE operations
- Administrative commands

### Client Error Codes (4xx)

#### 400 Bad Request

The request is malformed or contains invalid data.

**Common Causes:**
- Invalid JSON syntax
- Invalid JSON-LD structure
- Invalid SPARQL syntax
- Invalid IRI format
- Type mismatch

**Error typing:**

The server includes a compact error type IRI in the `@type` field. This is the
preferred stable, machine-readable category for programmatic handling.

**Example:**
```json
{
  "error": "Invalid JSON: expected value at line 5, column 12",
  "status": 400,
  "@type": "err:db/JsonParse"
}
```

**How to Fix:**
- Validate JSON syntax
- Check IRI formats
- Verify JSON-LD structure
- Review the `error` message and optional `cause`

#### 401 Unauthorized

Authentication is required but not provided or invalid.

**Common Causes:**
- Missing authentication credentials
- Invalid API key
- Expired JWT token
- Invalid signature (for signed requests)

**Example:**
```json
{
  "error": "Bearer token required",
  "status": 401,
  "@type": "err:db/Unauthorized"
}
```

**How to Fix:**
- Provide valid authentication credentials
- Check API key or token
- Renew expired tokens
- Verify signature process for signed requests

#### 403 Forbidden

Authentication succeeded but authorization failed.

**Common Causes:**
- Insufficient permissions for operation
- Policy denies access
- Ledger access restricted

**Example:**
```json
{
  "error": "access denied (403)",
  "status": 403,
  "@type": "err:db/Forbidden"
}
```

**How to Fix:**
- Verify user has required permissions
- Check policy configuration
- Contact administrator for access

#### 404 Not Found

The requested resource doesn't exist.

**Common Causes:**
- Ledger doesn't exist
- Entity not found
- Endpoint doesn't exist

**Example:**
```json
{
  "error": "Ledger not found: mydb:main",
  "status": 404,
  "@type": "err:db/LedgerNotFound"
}
```

**How to Fix:**
- Verify ledger name spelling
- Check if ledger was created
- Verify entity IRI

#### 408 Request Timeout

The request took too long to process.

**Common Causes:**
- Query timeout exceeded
- Complex query taking too long
- Database under heavy load

**Example:**
```json
{
  "error": "Query execution exceeded timeout",
  "status": 408,
  "@type": "err:db/Timeout"
}
```

**How to Fix:**
- Simplify query
- Add more specific filters
- Use LIMIT clause
- Increase timeout setting
- Check server load

#### 409 Conflict

The request conflicts with current server state.

**Common Causes:**
- Concurrent modification conflict
- Ledger already exists
- Resource state conflict

**Example:**
```json
{
  "error": "Ledger already exists: mydb:main",
  "status": 409,
  "@type": "err:db/LedgerExists"
}
```

**How to Fix:**
- Use different ledger name
- Handle concurrent modifications with retry logic
- Check resource state before modifying

#### 413 Payload Too Large

The request or response exceeds size limits.

**Common Causes:**
- Transaction too large
- Query result too large
- Request body exceeds limit

**Example:**
```json
{
  "error": "request body exceeds configured limit",
  "status": 413,
  "@type": "err:db/PayloadTooLarge"
}
```

**How to Fix:**
- Split large transactions into batches
- Use LIMIT clause for queries
- Use pagination for large result sets
- Increase size limits (if appropriate)

#### 415 Unsupported Media Type

The Content-Type is not supported.

**Common Causes:**
- Wrong Content-Type header
- Unsupported format
- Missing Content-Type header

**Example:**
```json
{
  "error": "Content-Type not supported: text/plain",
  "status": 415,
  "@type": "err:db/UnsupportedMediaType"
}
```

**How to Fix:**
- Set correct Content-Type header
- Use supported format
- Check API documentation for supported types

#### 422 Unprocessable Entity

The request is well-formed but semantically invalid.

**Common Causes:**
- Invalid data values
- Business rule violation
- Semantic constraint violation

**Example:**
```json
{
  "error": "semantic constraint violation",
  "status": 422,
  "@type": "err:db/ConstraintViolation"
}
```

**How to Fix:**
- Validate data before submitting
- Check business rules
- Review constraint requirements

#### 429 Too Many Requests

Rate limit exceeded.

**Common Causes:**
- Too many requests in time window
- Exceeded quota

**Example:**
```json
{
  "error": "rate limit exceeded",
  "status": 429,
  "@type": "err:db/RateLimited"
}
```

**Response Headers:**
```http
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1642857645
Retry-After: 45
```

**How to Fix:**
- Wait before retrying (check Retry-After header)
- Implement exponential backoff
- Reduce request rate
- Request higher rate limit

### Server Error Codes (5xx)

#### 500 Internal Server Error

An unexpected error occurred on the server.

**Common Causes:**
- Unhandled exception
- Database error
- Internal logic error

**Example:**
```json
{
  "error": "internal error",
  "status": 500,
  "@type": "err:db/Internal"
}
```

**How to Fix:**
- Check server logs
- Report to system administrator
- Retry request
- Contact support if persists

#### 502 Bad Gateway

Error communicating with upstream service.

**Common Causes:**
- Storage backend unavailable
- Nameservice unavailable
- Network error

**Example:**
```json
{
  "error": "upstream service error",
  "status": 502,
  "@type": "err:db/BadGateway"
}
```

**How to Fix:**
- Check storage backend status
- Verify network connectivity
- Check AWS/cloud service status
- Retry with backoff

#### 503 Service Unavailable

The server is temporarily unavailable.

**Common Causes:**
- Server overloaded
- Maintenance mode
- Resource exhaustion

**Example:**
```json
{
  "error": "service unavailable",
  "status": 503,
  "@type": "err:db/ServiceUnavailable"
}
```

**Response Headers:**
```http
Retry-After: 300
```

**How to Fix:**
- Wait and retry (check Retry-After header)
- Implement retry logic with exponential backoff
- Check service status page

#### 504 Gateway Timeout

Upstream service didn't respond in time.

**Common Causes:**
- Storage backend timeout
- Long-running query
- Network latency

**Example:**
```json
{
  "error": "gateway timeout",
  "status": 504,
  "@type": "err:db/GatewayTimeout"
}
```

**How to Fix:**
- Retry request
- Check storage backend performance
- Simplify query
- Increase timeout settings

## Error Handling Best Practices

### 1. Always Check Status Codes

Check HTTP status before parsing response:

```javascript
const response = await fetch(url, options);
if (!response.ok) {
  const err = await response.json();
  // err.error is the primary human-readable message, err["@type"] is the stable category.
  throw new Error(`${err["@type"] || "err:unknown"}: ${err.error}`);
}
```

### 2. Implement Retry Logic

Retry transient errors with exponential backoff:

```javascript
async function retryRequest(fn, maxRetries = 3) {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await fn();
    } catch (err) {
      if (!isRetryable(err) || i === maxRetries - 1) {
        throw err;
      }
      await sleep(Math.pow(2, i) * 1000);
    }
  }
}

function isRetryable(err) {
  return [408, 429, 502, 503, 504].includes(err.status);
}
```

### 3. Handle Rate Limits

Respect rate limit headers:

```javascript
if (response.status === 429) {
  const retryAfter = response.headers.get('Retry-After');
  await sleep(retryAfter * 1000);
  return retryRequest(fn);
}
```

### 4. Log Error Details

Log complete error context for debugging:

```javascript
console.error({
  status: response.status,
  error: errorData.error,
  error_type: errorData["@type"],
  cause: errorData.cause,
  requestId: response.headers.get('X-Request-ID')
});
```

### 5. User-Friendly Messages

Show appropriate messages to users:

```javascript
function getUserMessage(error) {
  switch (error["@type"]) {
    case 'err:db/LedgerNotFound':
      return 'Database not found. Please check the name.';
    case 'err:db/Timeout':
      return 'Query took too long. Please try a simpler query.';
    case 'err:db/RateLimited':
      return 'Too many requests. Please wait a moment.';
    default:
      return 'An error occurred. Please try again.';
  }
}
```

### 6. Graceful Degradation

Handle errors gracefully:

```javascript
try {
  const data = await query(ledger);
  return data;
} catch (err) {
  if (err["@type"] === 'err:db/LedgerNotFound') {
    // Create ledger and retry
    await createLedger(ledger);
    return await query(ledger);
  }
  throw err;
}
```

### 7. Circuit Breaker Pattern

Prevent cascading failures:

```javascript
class CircuitBreaker {
  constructor(threshold = 5, timeout = 60000) {
    this.failures = 0;
    this.threshold = threshold;
    this.timeout = timeout;
    this.state = 'CLOSED';
  }
  
  async execute(fn) {
    if (this.state === 'OPEN') {
      throw new Error('Circuit breaker is OPEN');
    }
    
    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (err) {
      this.onFailure();
      throw err;
    }
  }
  
  onSuccess() {
    this.failures = 0;
    this.state = 'CLOSED';
  }
  
  onFailure() {
    this.failures++;
    if (this.failures >= this.threshold) {
      this.state = 'OPEN';
      setTimeout(() => {
        this.state = 'HALF_OPEN';
        this.failures = 0;
      }, this.timeout);
    }
  }
}
```

## Related Documentation

- [Overview](overview.md) - API overview
- [Endpoints](endpoints.md) - API endpoints
- [Signed Requests](signed-requests.md) - Authentication
- [Troubleshooting](../troubleshooting/README.md) - General troubleshooting
- [Common Errors](../troubleshooting/common-errors.md) - Common error solutions


---

# query/README.md

# Query

Fluree supports two powerful query languages for querying graph data: **JSON-LD Query** (Fluree's native query language) and **SPARQL** (the W3C standard). Both languages provide access to Fluree's unique features including time travel, graph sources, and policy enforcement.

## Query Languages

### [JSON-LD Query](jsonld-query.md)

Fluree's native query language that uses JSON-LD syntax. JSON-LD Query provides a natural, JSON-based interface for querying graph data, making it easy to integrate with modern applications.

**Key Features:**
- JSON-based syntax (no string parsing)
- Full support for time travel (`@t:`, `@iso:`, `@commit:`)
- Graph source integration
- Policy enforcement
- History queries

### [SPARQL](sparql.md)

Industry-standard SPARQL 1.1 query language. Fluree provides full SPARQL support, enabling compatibility with existing RDF tools and knowledge graphs.

**Key Features:**
- W3C SPARQL 1.1 compliant
- FROM and FROM NAMED clauses
- CONSTRUCT queries
- Time travel support (planned)
- Standard SPARQL functions

## Query Features

### [Output Formats](output-formats.md)

Fluree supports multiple output formats for query results:
- **JSON-LD**: Compact, context-aware JSON with IRI expansion/compaction
- **SPARQL JSON**: Standard SPARQL result format
- **Typed JSON**: Type-preserving JSON with datatype information

### [Datasets and Multi-Graph Execution](datasets.md)

Query across multiple graphs and ledgers:
- **FROM clauses**: Specify default graphs
- **FROM NAMED**: Query named graphs
- **Multi-ledger queries**: Query across different ledgers
- **Time-aware datasets**: Query graphs at different time points

### [CONSTRUCT Queries](construct.md)

Generate RDF graphs from query results:
- Transform query results into RDF
- Create new graph structures
- Extract subgraphs

### [Graph Crawl](graph-crawl.md)

Traverse graph relationships:
- Follow links between entities
- Recursive graph traversal
- Depth-limited crawling

### [Explain Plans](explain.md)

Understand query execution:
- View query plans
- Analyze index usage
- Optimize query performance

### [Tracking and Fuel Limits](tracking-and-fuel.md)

Monitor and control query execution:
- Query tracking and debugging
- Fuel limits for resource control
- Performance monitoring

### Nameservice Queries

Query metadata about all ledgers and graph sources in the system. The nameservice stores information about every database including commit state, index state, and configuration.

**JSON-LD Query:**
```json
{
  "@context": {"f": "https://ns.flur.ee/db#"},
  "select": ["?ledger", "?t"],
  "where": [
    { "@id": "?ns", "@type": "f:LedgerSource", "f:ledger": "?ledger", "f:t": "?t" }
  ]
}
```

**SPARQL:**
```sparql
PREFIX f: <https://ns.flur.ee/db#>
SELECT ?ledger ?t WHERE { ?ns a f:LedgerSource ; f:ledger ?ledger ; f:t ?t }
```

See the [Ledgers and Nameservice](../concepts/ledgers-and-nameservice.md) concept documentation for details.

## Time Travel in Queries

Fluree supports querying historical data using time specifiers in ledger references:

**Transaction Number:**
```
ledger:main@t:100
```

**ISO 8601 Timestamp:**
```
ledger:main@iso:2024-01-15T10:30:00Z
```

**Commit ContentId:**
```
ledger:main@commit:bafybeig...
```

See the [Time Travel](../concepts/time-travel.md) concept documentation for details.

## Graph Source Queries

Query graph sources (BM25, Vector, Iceberg, R2RML) using the same syntax as regular ledgers:

```json
{
  "@context": {
    "f": "https://ns.flur.ee/db#"
  },
  "from": "products:main",
  "select": ["?product"],
  "where": [
    {
      "f:graphSource": "products-search:main",
      "f:searchText": "laptop",
      "f:searchLimit": 20,
      "f:searchResult": { "f:resultId": "?product" }
    }
  ]
}
```

See the [Graph Sources](../concepts/graph-sources.md) concept documentation for details.

## Policy Enforcement

Policies are automatically enforced during query execution, ensuring users only see data they're authorized to access. No special syntax is required—policies are applied transparently.

See the [Policy Enforcement](../concepts/policy-enforcement.md) concept documentation for details.

## Getting Started

### Basic JSON-LD Query

```json
{
  "@context": {
    "ex": "http://example.org/ns/"
  },
  "select": ["?name"],
  "where": [
    { "@id": "?person", "ex:name": "?name" }
  ]
}
```

### Basic SPARQL Query

```sparql
PREFIX ex: <http://example.org/ns/>

SELECT ?name
WHERE {
  ?person ex:name ?name .
}
```

### Query with Time Travel

```json
{
  "@context": {
    "ex": "http://example.org/ns/"
  },
  "from": "ledger:main@t:100",
  "select": ["?name"],
  "where": [
    { "@id": "?person", "ex:name": "?name" }
  ]
}
```

## Query Performance

Fluree's query engine is optimized for:
- **Automatic Join Ordering**: The planner reorders all WHERE-clause patterns
  (triples, UNION, OPTIONAL, MINUS, search patterns, and more) using
  statistics-driven cardinality estimates. When database statistics are
  available, it uses HLL-derived property counts; otherwise it falls back to
  heuristic constants. Estimates are context-aware — the planner tracks which
  variables are already bound and adjusts costs accordingly, so a triple
  whose subject is bound from an earlier pattern is scored as a cheap
  per-subject lookup rather than a full scan.
- **Index Selection**: Automatically chooses optimal indexes (SPOT, POST,
  OPST, PSOT) based on which triple components are bound.
- **Filter Optimization**: Filters are automatically applied as soon as their
  required variables are bound, regardless of where they appear in the query.
  Range-safe filters are pushed down to index scans, and filters are
  evaluated inline during joins when possible.
- **Streaming Execution**: Results stream as they're computed
- **Parallel Processing**: Parallel execution where possible

## Best Practices

1. **Use Appropriate Indexes**: Structure queries to leverage indexes
2. **Limit Result Sets**: Use LIMIT clauses for large result sets
3. **Time Travel Efficiency**: Use `@t:` when transaction numbers are known
4. **Graph Source Selection**: Choose appropriate graph sources for query patterns
5. **Policy Awareness**: Understand how policies affect query results

## Bundling queries

When you have multiple independent queries that should all see the same data — or that you'd otherwise issue as N concurrent HTTP requests — use the [Multi-query envelope](../api/multi-query.md). It runs N JSON-LD and/or SPARQL queries in parallel against a single resolved snapshot, with shared `@context` / `opts` defaults and per-alias result assembly.

## Related Documentation

- [Concepts](../concepts/README.md): Core concepts including time travel, graph sources, and policy
- [Transactions](../transactions/README.md): Writing data to Fluree
- [Security and Policy](../security/README.md): Policy configuration and management
- [Multi-query envelope](../api/multi-query.md): Bundle multiple queries against a shared snapshot


---

# query/jsonld-query.md

# JSON-LD Query

JSON-LD Query is Fluree's native query language, providing a JSON-based interface for querying graph data. It combines the expressiveness of SPARQL with the convenience of JSON, making it easy to integrate with modern applications.

## Overview

JSON-LD Query uses JSON-LD syntax to express queries, leveraging `@context` for IRI expansion and compaction. Queries are structured as JSON objects with familiar clauses like `select`, `where`, `from`, etc.

### Basic Query Structure

```json
{
  "@context": {
    "ex": "http://example.org/ns/"
  },
  "select": ["?name", "?age"],
  "where": [
    { "@id": "?person", "ex:name": "?name", "ex:age": "?age" }
  ]
}
```

## Query Clauses

### @context

The `@context` defines namespace mappings for IRI expansion/compaction:

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "schema": "http://schema.org/",
    "foaf": "http://xmlns.com/foaf/0.1/"
  }
}
```

When querying via the **CLI**, omitting `@context` causes the ledger's [default context](../concepts/iri-and-context.md#default-context) to be injected automatically. The HTTP API defaults this behavior off; pass `?default-context=true` to opt in for a request. To opt out explicitly, pass an empty object: `"@context": {}`. See [opting out of the default context](../concepts/iri-and-context.md#opting-out-of-the-default-context).

> **Note:** When using `fluree-db-api` directly (embedded), `@context` is not injected automatically. Queries must supply their own context or use full IRIs. Use `db_with_default_context()` or `GraphDb::with_default_context()` to opt in.

### select

Specifies what to return in results. The shape of `select` determines the shape of each output row.

**Bare variable** — one column, each row is the bound value (not wrapped in an array):

```json
{
  "select": "?name"
}
```

**Variable list** — each row is `[v1, v2, ...]`:

```json
{
  "select": ["?name", "?age"]
}
```

**Wildcard** — every variable bound in the WHERE clause:

```json
{
  "select": "*"
}
```

**Subject expansion** — return a nested JSON-LD object instead of a flat row. The key is either a variable (the WHERE clause binds it to subjects) or an IRI constant (the named subject is expanded directly, no WHERE needed):

```json
{
  "select": { "?person": ["*", { "schema:knows": ["@id", "schema:name"] }] },
  "where": { "@id": "?person", "@type": "schema:Person" }
}
```

```json
{
  "select": { "ex:alice": ["*"] }
}
```

The array value is the selection spec — `"*"` for all forward properties, individual property names (`"schema:name"`), or nested object forms for sub-selections. Add `"depth": N` at the query top level to bound auto-expansion of unselected references.

**Mixed array** — combine flat variables and subject expansions in one row, in any order. Each object is an independent expansion with its own root and selection spec:

```json
{
  "select": [
    "?age",
    { "?person": ["@id", "schema:name"] },
    { "?org": ["@id", "schema:name"] }
  ],
  "where": {
    "@id": "?person",
    "ex:age": "?age",
    "ex:worksFor": "?org"
  }
}
```

Each row is `[age, expanded_person, expanded_org]`. When every column is an IRI-constant expansion (no variable dependency anywhere in `select`), the output is independent of the WHERE solution count: the formatter emits one row regardless of how many solutions the WHERE produced.

**S-expression columns** — a select item that is a string starting with `(` is an S-expression, in two flavors:

*Aggregates*. Auto-aliased (`?count`, `?sum`, etc.) or with an explicit alias via `(as ...)`:

```json
{
  "select": ["?category", "(count ?product)"],
  "groupBy": ["?category"]
}
```

```json
{
  "select": ["?category", "(as (count ?product) ?total)"],
  "groupBy": ["?category"]
}
```

*Scalar expressions* (COALESCE, IF, arithmetic, string/hash/date functions, …). Always require an explicit alias via `(as <expr> ?alias)`. Mirrors SPARQL `SELECT (expr AS ?alias)`:

```json
{
  "select": ["?p", "(as (coalesce ?titleFr ?titleEn \"untitled\") ?title)"]
}
```

```json
{
  "select": [
    "?name",
    "(as (coalesce ?email \"no-email\") ?contact)",
    "(as (count ?favNums) ?count)"
  ],
  "groupBy": ["?name", "?contact"]
}
```

Scalar select expressions desugar to a `bind` in the WHERE pattern list. If the expression references an aggregate's output variable (e.g. `(as (+ ?count 1) ?adjusted)`) the bind runs after aggregation; otherwise it runs before, so the alias is also a valid `groupBy` key.

The same expression language is shared with `bind` and `filter`. The one exception is `in` / `not-in`, which require the bracketed-list form and are not accepted in select expressions — rewrite as `(or (= ?x 1) (= ?x 2) …)` instead.

### ask

Tests whether a set of patterns has any solution, returning `true` or `false`. No variables are projected. Equivalent to SPARQL `ASK`. The value of `ask` is the where clause itself — an array or object of the same patterns accepted by `where`:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "ask": [
    { "@id": "?person", "ex:name": "Alice" }
  ]
}
```

Single-pattern shorthand (object instead of array):

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "ask": { "@id": "?person", "ex:name": "Alice" }
}
```

Returns `true` if at least one solution exists, `false` otherwise. Internally, `LIMIT 1` is applied for efficiency.

### from

Specifies which ledger(s) to query:

**Single Ledger:**

```json
{
  "from": "mydb:main"
}
```

**Multiple Ledgers:**

```json
{
  "from": ["mydb:main", "otherdb:main"]
}
```

**Time Travel:**

```json
{
  "from": "mydb:main@t:100"
}
```

```json
{
  "from": "mydb:main@iso:2024-01-15T10:30:00Z"
}
```

```json
{
  "from": "mydb:main@commit:bafybeig..."
}
```

### where

The `where` clause contains query patterns:

**Basic Pattern:**

```json
{
  "where": [
    { "@id": "?person", "ex:name": "?name" }
  ]
}
```

**Multiple Patterns:**

```json
{
  "where": [
    { "@id": "?person", "ex:name": "?name" },
    { "@id": "?person", "ex:age": "?age" }
  ]
}
```

**Type Pattern:**

```json
{
  "where": [
    { "@id": "?person", "@type": "ex:User", "ex:name": "?name" }
  ]
}
```

## Pattern Types

### Object Patterns

Match triples where subject, predicate, and object are specified:

```json
{
  "@id": "ex:alice",
  "ex:name": "Alice"
}
```

### Variable Patterns

Use variables (starting with `?`) to match unknown values:

```json
{
  "@id": "?person",
  "ex:name": "?name"
}
```

### Type Patterns

Match entities by type:

```json
{
  "@id": "?person",
  "@type": "ex:User",
  "ex:name": "?name"
}
```

### Property Join Patterns

Match multiple properties of the same subject:

```json
{
  "@id": "?person",
  "ex:name": "?name",
  "ex:age": "?age",
  "ex:email": "?email"
}
```

## Advanced Patterns

### Optional Patterns

Match optional data that may not exist:

```json
{
  "where": [
    { "@id": "?person", "ex:name": "?name" },
    ["optional", { "@id": "?person", "ex:email": "?email" }]
  ]
}
```

#### Sibling vs. grouped OPTIONAL — semantics

The two forms below are **not** equivalent. Each `["optional", ...]` array is a
single OPTIONAL block in SPARQL terms — every item inside is part of the same
conjunctive group, and a row is null-extended only when the group as a whole
fails to match. To express two independent left joins, write two sibling
arrays.

**Sibling OPTIONALs — two independent left joins:**

```json
{
  "where": [
    { "@id": "?person", "ex:name": "?name" },
    ["optional", { "@id": "?person", "ex:email": "?email" }],
    ["optional", { "@id": "?person", "ex:phone": "?phone" }]
  ]
}
```

Equivalent SPARQL:

```sparql
?person ex:name ?name .
OPTIONAL { ?person ex:email ?email }
OPTIONAL { ?person ex:phone ?phone }
```

`?email` and `?phone` are independent — a person with only an email keeps
`?email` bound and gets `null` for `?phone`, and vice versa.

**Grouped OPTIONAL — one conjunctive left join:**

```json
{
  "where": [
    { "@id": "?person", "ex:name": "?name" },
    ["optional",
     { "@id": "?person", "ex:email": "?email" },
     { "@id": "?person", "ex:phone": "?phone" }
    ]
  ]
}
```

Equivalent SPARQL:

```sparql
?person ex:name ?name .
OPTIONAL { ?person ex:email ?email . ?person ex:phone ?phone }
```

`?email` and `?phone` are bound together — a person who has an email but no
phone is null-extended on **both** variables, because the inner conjunctive
group did not match as a whole.

#### Filters and binds inside OPTIONAL

`filter` and `bind` constrain or compute from existing bindings, so they need
something to anchor to inside the OPTIONAL block. Any binding-producing
pattern qualifies as an anchor — a node-map, `values`, an earlier `bind`, a
nested `optional`, or a sub-`query`. A `filter` or `bind` as the very first
item in an OPTIONAL array is rejected.

```json
["optional",
  { "@id": "?person", "ex:age": "?age" },
  ["filter", "(> ?age 18)"]
]
```

```json
["optional",
  ["values", ["?x", [1, 2, 3]]],
  ["filter", "(> ?x 0)"]
]
```

### Union Patterns

Match data from multiple alternative patterns:

```json
{
  "where": [
    ["union",
     { "@id": "?person", "ex:name": "?name" },
     { "@id": "?person", "ex:alias": "?name" }
    ]
  ]
}
```

### Negation Patterns

Restrict results to subjects that *do not* match a given pattern. Two forms
are available; **prefer `not-exists` for the common "subjects without
property `p`" case.**

#### `not-exists` (canonical)

`["not-exists", { ... }]` keeps rows for which the inner pattern has **no**
solution under the current bindings:

```json
{
  "where": [
    { "@id": "?file", "@type": "ex:File" },
    ["not-exists", { "@id": "?file", "ex:parent": "?p" }]
  ],
  "select": "?file"
}
```

When the inner pattern is correlated to the outer **only via vars the inner
itself produces** (here `?file`, produced by both the outer triple and the
inner triple), and the inner does not reference any outer-only variable
(e.g. via a `["filter", ...]` mentioning an outer var the inner doesn't
produce), the planner builds the inner solution set **once** and probes it
per outer row — the performant path on large outer streams. If either
condition fails, the inner is rebuilt and re-run per outer row (still
correct, but linear in the outer cardinality).

`["exists", { ... }]` is the affirmative dual and uses the same dispatch.

#### `OPTIONAL` + `(not (bound ?v))`

The SPARQL-muscle-memory form `OPTIONAL { ... } FILTER (!bound(?v))` is
spelled in JSON-LD as an `optional` followed by a filter using the supported
`bound` function:

```json
{
  "where": [
    { "@id": "?file", "@type": "ex:File" },
    ["optional", { "@id": "?file", "ex:parent": "?p" }],
    ["filter", "(not (bound ?p))"]
  ],
  "select": "?file"
}
```

The data filter form is equivalent: `["filter", ["not", ["bound", "?p"]]]`.

When the `optional` block binds `?v`, the filter immediately tests
`!bound(?v)`, and `?v` is not used after the filter, the planner rewrites
this shape to a NOT EXISTS and dispatches it through the same strategy
chooser as `not-exists`. The two forms therefore have the same plan and the
same performance.

#### `minus`

`["minus", { ... }]` removes solutions whose shared variables match the
inner pattern. Use when you want set-difference semantics rather than
negation-as-failure (see SPARQL 1.1 §8.3 for the difference).

#### Filter expression — supported "is-unbound" idioms

The s-expression parser recognizes `not`, `and`, `or` as AST operators and
`bound` as a function. The following are **not** parser tokens:

| Attempt | Result |
|---|---|
| `(not (bound ?p))` | ✅ supported |
| `["not", ["bound", "?p"]]` | ✅ supported (data filter form) |
| `(!bound ?p)`, `(! (bound ?p))` | ❌ parse error — use `(not (bound ?p))` |
| `(nil? ?p)` | ❌ parse error — use `(not (bound ?p))` |

The "Unknown function" parse error includes a hint pointing at
`(not (bound ?v))` or `["not-exists", ...]` for these common attempts.

#### SPARQL → JSON-LD translation

| SPARQL | JSON-LD |
|---|---|
| `FILTER NOT EXISTS { ?s :p ?v }` | `["not-exists", {"@id":"?s","ex:p":"?v"}]` |
| `FILTER EXISTS { ?s :p ?v }` | `["exists", {"@id":"?s","ex:p":"?v"}]` |
| `OPTIONAL { ?s :p ?v } FILTER(!bound(?v))` | `["optional", {"@id":"?s","ex:p":"?v"}]` then `["filter", "(not (bound ?v))"]` |
| `?a :p ?b MINUS { ?a :q ?c }` | `{"@id":"?a","ex:p":"?b"}` then `["minus", {"@id":"?a","ex:q":"?c"}]` |

### Graph Patterns

Scope patterns to a named graph:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "mydb:main",
  "fromNamed": {
    "products": {
      "@id": "mydb:main",
      "@graph": "http://example.org/graphs/products"
    }
  },
  "select": ["?product", "?name"],
  "where": [
    ["graph", "products", { "@id": "?product", "ex:name": "?name" }]
  ]
}
```

Notes:
- `fromNamed` is an object whose keys are dataset-local aliases. Each value is an object with `@id` (ledger reference) and optional `@graph` (graph selector IRI).
- The second element of `["graph", ...]` can be a dataset-local alias (recommended) or a graph IRI.
- The legacy `"from-named": [...]` array format is still accepted for backward compatibility.
- For dataset and named-graph configuration details, see `docs/query/datasets.md`.

### Filter Patterns

Apply conditions to filter results:

**Single Filter:**

```json
{
  "where": [
    { "@id": "?person", "ex:age": "?age" },
    ["filter", "(> ?age 18)"]
  ]
}
```

**Multiple Filters:**

```json
{
  "where": [
    { "@id": "?person", "ex:age": "?age", "ex:name": "?name" },
    ["filter", "(> ?age 18)", "(strStarts ?name \"A\")"]
  ]
}
```

**Complex Filters:**

```json
{
  "where": [
    { "@id": "?person", "ex:age": "?age", "ex:last": "?last" },
    ["filter", "(and (> ?age 45) (strEnds ?last \"ith\"))"]
  ]
}
```

### Bind Patterns

Compute values and bind to variables:

```json
{
  "where": [
    { "@id": "?person", "ex:age": "?age" },
    ["bind", "?nextAge", "(+ ?age 1)"]
  ]
}
```

### Values Patterns

Provide initial bindings:

```json
{
  "where": [
    ["values", "?name", ["Alice", "Bob", "Carol"]],
    { "@id": "?person", "ex:name": "?name" }
  ]
}
```

### Unwind Patterns

`unwind` expands a **list-valued expression** into one row per element, binding
each element to a variable. It is the inverse of an aggregate like `collect`:
where aggregation folds many rows into one list, `unwind` fans one list back out
into many rows.

```json
["unwind", "?n", "(range 1 5)"]
```

The list source is any expression that evaluates to a list:

- **`(range start end)`** / **`(range start end step)`** — an inclusive integer
  sequence (`(range 1 5)` → `1,2,3,4,5`). `start > end` yields an empty list.
- **`(list a b c …)`** — a list literal built from its arguments.
- **a bound list variable** — e.g. a list produced earlier by `bind`, or the
  result of an aggregate such as `collect`.

Per Cypher/GQL semantics, an **empty list, an unbound value, or a non-list
value produces zero rows** for that input (it does not error and does not emit a
null row).

```json
{
  "@context": { "ex": "http://example.org/" },
  "select": ["?n"],
  "where": [["unwind", "?n", "(range 1 3)"]]
}
// → ?n = 1, then 2, then 3
```

#### When to use `unwind` (and when not to)

`unwind` earns its place in exactly one situation: **generating rows that do not
exist in the stored data**, from a *computed* list.

- **Do not** use it for a constant set of values — `values` is clearer and
  cheaper (`["values", "?s", ["a", "b"]]`).
- **Do not** use it to read a multi-valued predicate — in RDF that is already
  multiple triples, so a plain pattern `{ "@id": "?s", "ex:tag": "?tag" }`
  yields one row per value natively.
- **Do** use it when the driving rows must come from a `range` (a numeric axis,
  a calendar, buckets, a pagination window) or from a list you computed — none
  of which `values` (constants only) or triple patterns (stored data only) can
  produce.

#### Canonical use case: a dense / gap-filled series

A `groupBy` only ever produces keys that occur in the data. To report a value
for **every** point on an axis — including the empty ones — generate the axis
with `range` and `unwind`, then LEFT JOIN the data with `optional`:

```json
{
  "@context": { "ex": "http://example.org/" },
  "select": ["?year", "(as (count ?o) ?orders)"],
  "where": [
    ["unwind", "?year", "(range 2019 2023)"],
    ["optional", { "@id": "?o", "@type": "ex:Order", "ex:orderYear": "?year" }]
  ],
  "groupBy": ["?year"],
  "orderBy": ["?year"]
}
```

Every year `2019`–`2023` appears in the result, with `0` for years that have no
orders — because the driving rows came from the generated range and the
`optional` contributes a count only where matching data exists. The same shape
fills gaps for any axis the data does not itself contain. Where the bounds are
themselves derived from the data (e.g. a sub-select computing the min/max year),
this produces a dynamically-sized dense axis that `values` cannot express.

#### Ordering and correlation

`unwind` is placed after whatever produces the variables its list expression
reads, so a list computed by an earlier `bind` is expanded correctly:

```json
{
  "where": [
    ["bind", "?nums", "(range 1 3)"],
    ["unwind", "?n", "?nums"]
  ]
}
```

### Property Paths

Property paths enable transitive traversal of predicates, following chains of relationships across multiple hops. Define a path alias in `@context` using `@path`, then use the alias as a key in WHERE node-maps.

**Defining a Path Alias:**

Add a term definition with `@path` to your `@context`. The value of `@path` can be a string (SPARQL property path syntax) or an array (S-expression form).

**String Form (SPARQL syntax):**

```json
{
  "@context": {
    "ex": "http://example.org/",
    "knowsPlus": { "@path": "ex:knows+" }
  },
  "select": ["?who"],
  "where": [
    { "@id": "ex:alice", "knowsPlus": "?who" }
  ]
}
```

This returns all entities reachable from `ex:alice` by following one or more `ex:knows` edges transitively.

**Array Form (S-expression):**

```json
{
  "@context": {
    "ex": "http://example.org/",
    "knowsPlus": { "@path": ["+", "ex:knows"] }
  },
  "select": ["?who"],
  "where": [
    { "@id": "ex:alice", "knowsPlus": "?who" }
  ]
}
```

The array form uses the operator as the first element followed by its operands.

**Supported Operators:**

| Operator | String syntax | Array syntax | Description |
|----------|--------------|--------------|-------------|
| One or more | `ex:p+` | `["+", "ex:p"]` | Transitive closure (1+ hops) |
| Zero or more | `ex:p*` | `["*", "ex:p"]` | Reflexive transitive closure (0+ hops) |
| Inverse | `^ex:p` | `["^", "ex:p"]` | Traverse predicate in reverse direction |
| Alternative | <code>ex:a&#124;ex:b</code> | <code>["&#124;", "ex:a", "ex:b"]</code> | Match any of several predicates |
| Sequence | `ex:a/ex:b` | `["/", "ex:a", "ex:b"]` | Follow a chain of predicates (property chain) |
| Alternation-transitive | <code>(ex:a&#124;ex:b)+</code> | <code>["+", ["&#124;", "ex:a", "ex:b"]]</code> | Transitive closure following an edge of **any** listed predicate per hop |

Zero-or-more (`*`) includes the starting node itself in the results (zero hops).

Sequence (`/`) compiles into a chain of triple patterns joined by internal
intermediate variables. Each step must be a simple predicate or an inverse simple
predicate (`^ex:p`). For example, `"ex:friend/ex:name"` matches paths where
subject has a `ex:friend` whose `ex:name` is the result.

**Parsed but Not Yet Supported:**

The following operators are recognized by the parser but currently rejected (not yet supported for execution):

| Operator | String syntax | Array syntax |
|----------|--------------|--------------|
| Zero or one | `ex:p?` | `["?", "ex:p"]` |

**Subject and Object Variables:**

Path aliases work with variables on either side:

```json
{
  "@context": {
    "ex": "http://example.org/",
    "knowsPlus": { "@path": "ex:knows+" }
  },
  "select": ["?x", "?y"],
  "where": [
    { "@id": "?x", "knowsPlus": "?y" }
  ]
}
```

This returns all pairs `(?x, ?y)` where `?y` is transitively reachable from `?x` via `ex:knows`.

**Fixed Subject or Object:**

You can also fix one end to an IRI:

```json
{
  "@context": {
    "ex": "http://example.org/",
    "knowsPlus": { "@path": "ex:knows+" }
  },
  "select": ["?who"],
  "where": [
    { "@id": "?who", "knowsPlus": { "@id": "ex:bob" } }
  ]
}
```

This finds all entities that can reach `ex:bob` through one or more `ex:knows` hops.

**Inverse Example:**

Find entities that know `ex:bob` (traverse `ex:knows` in reverse):

```json
{
  "@context": {
    "ex": "http://example.org/",
    "knownBy": { "@path": "^ex:knows" }
  },
  "select": ["?who"],
  "where": [
    { "@id": "ex:bob", "knownBy": "?who" }
  ]
}
```

**Alternative Example:**

Match entities connected by either `ex:knows` or `ex:likes`:

```json
{
  "@context": {
    "ex": "http://example.org/",
    "connected": { "@path": "ex:knows|ex:likes" }
  },
  "select": ["?who"],
  "where": [
    { "@id": "ex:alice", "connected": "?who" }
  ]
}
```

**Alternation-Transitive Example:**

Apply a transitive modifier (`+` or `*`) to an alternative of predicates to follow an edge of *any* listed predicate at each hop. For example, `(ex:knows|ex:likes)+` reaches every node connected through a chain of `ex:knows` and/or `ex:likes` edges, in any combination:

```json
{
  "@context": {
    "ex": "http://example.org/",
    "reaches": { "@path": "(ex:knows|ex:likes)+" }
  },
  "select": ["?who"],
  "where": [
    { "@id": "ex:alice", "reaches": "?who" }
  ]
}
```

Array form:

```json
{ "@path": ["+", ["|", "ex:knows", "ex:likes"]] }
```

This differs from running `ex:knows+` and `ex:likes+` separately: a single closure mixes both predicates within one path (e.g. `alice -knows-> bob -likes-> carol`). The branches under the transitive modifier must be **simple forward predicate IRIs** — inverse (`^ex:p`), sequence, or nested-transitive branches are not supported in this position. Like other arbitrary-length paths, the closure is duplicate-free and cycle-safe.

Inverse can also be applied to complex paths (sequences and alternatives):

- `^(ex:friend/ex:name)` — inverse of a sequence: reverses the step order and inverts each step, producing `(^ex:name)/(^ex:friend)`
- `^(ex:name|ex:nick)` — inverse of an alternative: distributes the inverse into each branch, producing `(^ex:name)|(^ex:nick)`
- Double inverse cancels: `^(^ex:p)` simplifies to `ex:p`

Array form examples:

```json
{ "@path": ["^", ["/", "ex:friend", "ex:name"]] }
{ "@path": ["^", ["|", "ex:name", "ex:nick"]] }
```

Inverse is supported inside alternative branches (e.g. `ex:knows|^ex:knows` matches both directions of the `ex:knows` predicate).

Alternative branches can also be sequence chains. For example, `ex:friend/ex:name|ex:colleague/ex:name` returns the name of a friend OR the name of a colleague:

```json
{
  "@context": {
    "ex": "http://example.org/",
    "contactName": { "@path": "ex:friend/ex:name|ex:colleague/ex:name" }
  },
  "select": ["?name"],
  "where": [
    { "@id": "ex:alice", "contactName": "?name" }
  ]
}
```

Branches can freely mix simple predicates, inverse predicates, and sequence chains (e.g. `ex:name|ex:friend/ex:name|^ex:colleague`).

Alternative uses UNION semantics (bag, not set): when multiple branches match the same `(subject, object)` pair, duplicate solutions are produced. Use `selectDistinct` if set semantics are needed.

**Sequence (Property Chain) Example:**

Follow a chain of predicates. The string form uses `/` to separate steps:

```json
{
  "@context": {
    "ex": "http://example.org/",
    "friendName": { "@path": "ex:friend/ex:name" }
  },
  "select": ["?person", "?name"],
  "where": [
    { "@id": "?person", "friendName": "?name" }
  ]
}
```

The array form uses `"/"` as the operator:

```json
{ "@path": ["/", "ex:friend", "ex:name"] }
```

Sequence steps can include inverse predicates. For example, `"^ex:parent/ex:name"` traverses the `ex:parent` link backwards, then follows `ex:name`:

```json
{ "@path": "^ex:parent/ex:name" }
```

Longer chains are supported: `"ex:friend/ex:address/ex:city"` follows three hops.

Sequence steps can also be alternatives. For example, `"ex:friend/(ex:name|ex:nick)"` distributes the alternative into a union of chains (`ex:friend/ex:name` and `ex:friend/ex:nick`):

```json
{ "@path": "ex:friend/(ex:name|ex:nick)" }
```

Array form:

```json
{ "@path": ["/", "ex:friend", ["|", "ex:name", "ex:nick"]] }
```

Multiple alternative steps are supported: `"(ex:a|ex:b)/(ex:c|ex:d)"` expands to 4 chains. A safety limit of 64 expanded chains is enforced to prevent combinatorial explosion.

Each step must be a simple predicate (`ex:p`), inverse simple predicate (`^ex:p`), or an alternative of simple predicates (`(ex:a|ex:b)`). Transitive (`+`/`*`) and nested sequence modifiers are not allowed inside sequence steps.

**Rules:**

- `@path` and `@reverse` are mutually exclusive on the same term definition (produces an error).
- `@path` and `@id` may coexist on the same term definition; when the alias key appears in a WHERE node-map, the `@path` definition is used.
- Cycle detection is built in: transitive traversal terminates when it encounters a node already visited.
- Variable names starting with `?__` are reserved for internal use (e.g., intermediate join variables generated by sequence paths). These variables will not appear in wildcard (`select: "*"`) output.

### Shortest Path

`shortestPath` / `allShortestPaths` find the shortest path(s) between two nodes over a single predicate and **bind the path to a variable** (unlike a property path, which matches transitively but binds only the endpoint). It is a `where`-clause object form:

```json
["shortestPath", {
  "from": "?a", "to": "?b",
  "via": "ex:knows",
  "direction": "out",
  "minHops": 1, "maxHops": 6,
  "bind": "?p"
}]
```

| Key | Meaning |
|-----|---------|
| `from` / `to` | Endpoints — a variable (`?x`) or a constant IRI. Usually bound by prior patterns. |
| `via` | The predicate IRI to traverse. A single predicate; unknown IRI → no rows. |
| `bind` | Variable bound to the resulting **path value**. |
| `direction` | `out` (default), `in`, or `both` (undirected). |
| `minHops` / `maxHops` | Optional hop bounds. Omit `maxHops` for unbounded (subject to safety caps). |

- `["shortestPath", …]` binds **one** shortest path (or no row if none exists); `["allShortestPaths", …]` binds **every** path of the minimal length (one row each).
- The path value is consumed by the path functions below; it is not itself a list of bindings.

```json
{
  "@context": { "ex": "http://example.org/" },
  "select": ["?hops"],
  "where": [
    ["shortestPath", { "from": "ex:alice", "to": "ex:dan", "via": "ex:knows", "bind": "?p" }],
    ["bind", "?hops", "(size (path-pairs ?p))"]
  ]
}
```

#### Path value functions

Consume a path value bound by `shortestPath` / `allShortestPaths`:

- `(nodes ?p)` — the list of nodes along the path (in order). A list value, so it can feed [`unwind`](#unwind-patterns) or the [list functions](#list-value-functions).
- `(path-pairs ?p)` — the consecutive node pairs `[[a,b],[b,c],…]`; its `size` is the hop count.

### Edge Annotations

Edge annotations attach metadata to a specific `(subject, predicate, object)` edge. Two query forms are supported:

- **Inline form** with `@annotation` — match an edge and pull metadata about it.
- **Annotation-rooted form** with `@reifies` — match metadata first, find the edges it reifies.

`@edge` is an alias for `@annotation`; the two are interchangeable. For how to *write* annotations (`@annotation` on insert), the storage model, the cardinality contract, and worked output, see the [Edge annotations](../concepts/edge-annotations.md) concept doc. Note `@reifies` is a **query-side** construct only — user-authored `@reifies` on an insert/update is rejected; write with `@annotation` instead.

**Inline form (`@annotation`):**

The `@annotation` block sits inside the object node-map and binds the annotation subject. Each annotation occurrence on the matched edge produces one row.

```json
{
  "@context": { "ex": "http://example.org/" },
  "select": ["?person", "?org", "?role"],
  "where": {
    "@id": "?person",
    "ex:worksFor": {
      "@id": "?org",
      "@annotation": { "ex:role": "?role" }
    }
  }
}
```

When two parallel annotations reify the same edge, the inline form returns two rows (property-graph fidelity). A bare triple pattern (`?person ex:worksFor ?org` with no `@annotation` block) returns one row regardless of how many annotations exist — annotations only multiply cardinality through the `@annotation` / `@reifies` keywords.

The annotation subject can be bound to a variable or filtered by an explicit IRI:

```json
"@annotation": { "@id": "?ann", "ex:role": "?role" }
```

```json
"@annotation": { "@id": "ex:emp/alice-acme" }
```

**Annotation-rooted form (`@reifies`):**

Filter by annotation metadata first, then surface the reified edge.

```json
{
  "@context": { "ex": "http://example.org/" },
  "select": ["?person", "?org"],
  "where": {
    "ex:role": "Engineer",
    "@reifies": {
      "@id": "?person",
      "ex:worksFor": { "@id": "?org" }
    }
  }
}
```

The base edge identified by `@reifies` is also matched as an ordinary triple, so the visibility check is automatic — if the edge is currently retracted or hidden by policy, the row drops.

**Subject expansion output:**

Wildcard hydration (`select: {"?s": ["*"]}`) on an annotated subject's base edge surfaces each annotation under an `@annotation` key on the expanded value. Anonymous (blank-node) annotation subjects render their body without `@id`; explicit-IRI annotations keep theirs. Multiple parallel annotations render as an array under `@annotation`.

```json
{
  "@id": "ex:alice",
  "ex:worksFor": {
    "@id": "ex:acme",
    "@annotation": { "ex:role": "Engineer" }
  }
}
```

**Reserved predicates:**

Annotations are backed by reserved system predicates you can't query directly — use `@annotation` / `@reifies` instead. Naming them in a query is rejected at parse time, and they're hidden from variable-predicate scans (`?p`) and wildcard hydration so they never surface as ordinary RDF. See the [vocabulary reference](../reference/vocabulary.md#edge-annotation-predicates-reserved) for the list.

## Filter Functions

### Comparison Functions

Comparison operators accept two or more arguments. With multiple arguments, they chain pairwise: `(< ?a ?b ?c)` means `?a < ?b AND ?b < ?c`.

- `(= ?x ?y ...)` - Equality
- `(!= ?x ?y ...)` - Inequality
- `(> ?x ?y ...)` - Greater than
- `(>= ?x ?y ...)` - Greater than or equal
- `(< ?x ?y ...)` - Less than
- `(<= ?x ?y ...)` - Less than or equal

When comparing incomparable types (e.g., a number and a string):

- `=` yields `false` — values of different types are not equal
- `!=` yields `true` — values of different types are not equal
- `<`, `<=`, `>`, `>=` raise an error — ordering between incompatible types is undefined

### Logical Functions

- `(and ...)` - Logical AND
- `(or ...)` - Logical OR
- `(not ...)` - Logical NOT

### String Functions

- `(strStarts ?str ?prefix)` - String starts with
- `(strEnds ?str ?suffix)` - String ends with
- `(contains ?str ?substr)` - String contains
- `(regex ?str ?pattern)` - Regular expression match

### Numeric Functions

Arithmetic operators accept two or more arguments. With multiple arguments, they fold left: `(+ ?x ?y ?z)` evaluates as `(?x + ?y) + ?z`. A single argument returns the value unchanged.

- `(+ ?x ?y ...)` - Addition
- `(- ?x ?y ...)` - Subtraction
- `(* ?x ?y ...)` - Multiplication
- `(/ ?x ?y ...)` - Division
- `(- ?x)` - Unary negation (single argument)
- `(abs ?x)` - Absolute value

### List Value Functions

Operate on list values — those produced by `range`, `list`, or [`collect`](#aggregation-functions). Usable anywhere expressions are (`bind`, `filter`, `select`), and the list-producing ones drive [`unwind`](#unwind-patterns).

Producers:

- `(range ?start ?end)` / `(range ?start ?end ?step)` - Inclusive integer sequence; `?start > ?end` yields an empty list
- `(list ?a ?b ...)` - List literal from the given arguments

Accessors / transforms:

- `(size ?list)` - Element count (also the length of a string)
- `(head ?list)` - First element (null if empty)
- `(last ?list)` - Last element (null if empty)
- `(nth ?list ?i)` - Element at 0-based index `?i`; negative indexes count from the end; out-of-range/non-integer/non-list → null
- `(tail ?list)` - The list without its first element
- `(reverse ?list)` - The list reversed (also reverses a string)

`head`, `last`, and `nth` return a single element; `tail` and `reverse` return a list (compose them, e.g. `(head (reverse ?xs))`).

### Vector Similarity Functions

Used with `bind` to compute similarity scores between `@vector` values:

- `(dotProduct ?vec1 ?vec2)` - Dot product (inner product)
- `(cosineSimilarity ?vec1 ?vec2)` - Cosine similarity (-1 to 1)
- `(euclideanDistance ?vec1 ?vec2)` - Euclidean (L2) distance

Function names are case-insensitive. See [Vector Search](../indexing-and-search/vector-search.md) for usage examples.

### Type Functions

- `(bound ?var)` - Variable is bound. To test for **absence**, see
  [Negation Patterns](#negation-patterns) — `(not (bound ?v))` after
  `optional` is supported, or use `["not-exists", ...]` directly.
- `(isIRI ?x)` - Is an IRI
- `(isBlank ?x)` - Is a blank node
- `(isLiteral ?x)` - Is a literal

## Query Modifiers

### orderBy

Sort results:

```json
{
  "orderBy": ["?name"]
}
```

**Descending Order:**

```json
{
  "orderBy": [["desc", "?age"]]
}
```

**Multiple Sort Keys:**

```json
{
  "orderBy": ["?last", ["desc", "?age"]]
}
```

### limit

Limit number of results:

```json
{
  "limit": 10
}
```

### offset

Skip results:

```json
{
  "offset": 20,
  "limit": 10
}
```

### groupBy

Group results:

```json
{
  "select": ["?category", "(count ?product)"],
  "groupBy": ["?category"],
  "where": [
    { "@id": "?product", "ex:category": "?category" }
  ]
}
```

### having

Filter grouped results:

```json
{
  "select": ["?category", "(count ?product)"],
  "groupBy": ["?category"],
  "having": [["filter", "(> (count ?product) 10)"]],
  "where": [
    { "@id": "?product", "ex:category": "?category" }
  ]
}
```

## Aggregation Functions

- `(count ?var)` / `(count *)` — count non-null values; `*` counts solution rows
- `(count-distinct ?var)` — count distinct non-null values
- `(sum ?var)` — sum numeric values
- `(avg ?var)` — average numeric values
- `(min ?var)` / `(max ?var)` — extremum
- `(median ?var)` — median
- `(variance ?var)` / `(stddev ?var)` — population variance / standard deviation
- `(sample ?var)` — implementation-defined sample value
- `(groupconcat ?var)` / `(groupconcat ?var ", ")` — concatenate string values, optional separator (defaults to a single space)
- `(collect ?var)` — gather every non-null value in the group into a **list value** (a JSON array), keeping row-level duplicates
- `(collect-distinct ?var)` — like `collect`, but deduplicates within the group

Each aggregate auto-aliases to `?<fn-name>` (`?count`, `?sum`, …). Use `(as (<fn> ?var) ?alias)` to choose an explicit alias.

`collect` is the inverse of [`unwind`](#unwind-patterns): it folds many rows into one list, and `unwind` fans a list back out into rows. A `collect` result is a first-class list value, so it can be re-expanded — e.g. a sub-select that `collect`s values, with an outer `unwind` over the result.

> Note: in RDF a repeated *identical* value is a single triple, so `collect` and `collect-distinct` differ only when the same value reaches the aggregate on multiple solution rows (typically via a join).

## Time Travel Queries

Query historical data using time specifiers in `from`:

**Transaction Number:**

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "ledger:main@t:100",
  "select": ["?name"],
  "where": [
    { "@id": "?person", "ex:name": "?name" }
  ]
}
```

**ISO Timestamp:**

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "ledger:main@iso:2024-01-15T10:30:00Z",
  "select": ["?name"],
  "where": [
    { "@id": "?person", "ex:name": "?name" }
  ]
}
```

**Commit ContentId:**

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "ledger:main@commit:bafybeig...",
  "select": ["?name"],
  "where": [
    { "@id": "?person", "ex:name": "?name" }
  ]
}
```

**Multiple Ledgers at Different Times:**

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": ["ledger1:main@t:100", "ledger2:main@t:200"],
  "select": ["?data"],
  "where": [
    { "@id": "?entity", "ex:data": "?data" }
  ]
}
```

## History Queries

History queries let you see all changes (assertions and retractions) within a time range. Specify the range using `from` and `to` keys with time-specced endpoints:

### Time Range Syntax

```json
{
  "from": "ledger:main@t:1",
  "to": "ledger:main@t:latest"
}
```

### Binding Transaction Metadata

Use `@t` and `@op` annotations on value objects to capture metadata:

- **@t** - Binds the transaction time (integer) when the fact was asserted/retracted.
- **@op** - Binds the operation type as a boolean: `true` for assertions, `false` for retractions. (Mirrors `Flake.op` on disk; constants `"assert"` / `"retract"` are *not* accepted — use `true` / `false`.)

Both annotations work uniformly for literal-valued and IRI-valued objects.

**Entity History:**

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "ledger:main@t:1",
  "to": "ledger:main@t:latest",
  "select": ["?name", "?age", "?t", "?op"],
  "where": [
    { "@id": "ex:alice", "ex:name": { "@value": "?name", "@t": "?t", "@op": "?op" } },
    { "@id": "ex:alice", "ex:age": "?age" }
  ],
  "orderBy": "?t"
}
```

**Property-Specific History:**

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "ledger:main@t:1",
  "to": "ledger:main@t:100",
  "select": ["?age", "?t", "?op"],
  "where": [
    { "@id": "ex:alice", "ex:age": { "@value": "?age", "@t": "?t", "@op": "?op" } }
  ],
  "orderBy": "?t"
}
```

**Time Range with ISO:**

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "ledger:main@iso:2024-01-01T00:00:00Z",
  "to": "ledger:main@iso:2024-12-31T23:59:59Z",
  "select": ["?name", "?t", "?op"],
  "where": [
    { "@id": "ex:alice", "ex:name": { "@value": "?name", "@t": "?t", "@op": "?op" } }
  ]
}
```

**Filter by Operation:**

You can either use a constant `@op` shorthand (preferred) or filter on the bound variable:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "ledger:main@t:1",
  "to": "ledger:main@t:latest",
  "select": ["?name", "?t"],
  "where": [
    { "@id": "ex:alice", "ex:name": { "@value": "?name", "@t": "?t", "@op": false } }
  ]
}
```

The shorthand `"@op": false` lowers to `FILTER(op(?name) = false)`. Equivalent long form using a bound variable: `"@op": "?op"` plus `["filter", "(= ?op false)"]`.

**All Properties History:**

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "ledger:main@t:1",
  "to": "ledger:main@t:latest",
  "select": ["?property", "?value", "?t", "?op"],
  "where": [
    { "@id": "ex:alice", "?property": { "@value": "?value", "@t": "?t", "@op": "?op" } }
  ],
  "orderBy": "?t"
}
```

## Graph Source Queries

Query graph sources using the same syntax:

**BM25 Search:**

```json
{
  "@context": {
    "f": "https://ns.flur.ee/db#"
  },
  "from": "products:main@t:1000",
  "select": ["?product", "?score"],
  "where": [
    {
      "f:graphSource": "products-search:main",
      "f:searchText": "laptop",
      "f:searchLimit": 10,
      "f:searchResult": { "f:resultId": "?product", "f:resultScore": "?score" }
    }
  ],
  "orderBy": [["desc", "?score"]],
  "limit": 10
}
```

**Vector Similarity:**

```json
{
  "@context": {
    "ex": "http://example.org/",
    "f": "https://ns.flur.ee/db#"
  },
  "from": "documents:main",
  "select": ["?document", "?similarity"],
  "values": [
    ["?queryVec"],
    [{"@value": [0.1, 0.2, 0.3], "@type": "https://ns.flur.ee/db#embeddingVector"}]
  ],
  "where": [
    {
      "f:graphSource": "documents-vector:main",
      "f:queryVector": "?queryVec",
      "f:searchLimit": 5,
      "f:searchResult": { "f:resultId": "?document", "f:resultScore": "?similarity" }
    }
  ],
  "orderBy": [["desc", "?similarity"]],
  "limit": 5
}
```

Note: `f:*` keys used for graph source queries should be defined in your `@context` for clarity.

## Complete Examples

### Simple Select Query

```json
{
  "@context": {
    "ex": "http://example.org/ns/"
  },
  "select": ["?name", "?age"],
  "where": [
    {
      "@id": "?person",
      "@type": "ex:User",
      "ex:name": "?name",
      "ex:age": "?age"
    },
    ["filter", "(> ?age 18)"]
  ],
  "orderBy": ["?name"],
  "limit": 10
}
```

### Complex Query with Joins

```json
{
  "@context": {
    "ex": "http://example.org/ns/"
  },
  "select": ["?person", "?friend", "?friendName"],
  "where": [
    { "@id": "?person", "ex:name": "?name" },
    { "@id": "?person", "ex:friend": "?friend" },
    { "@id": "?friend", "ex:name": "?friendName" },
    ["filter", "(= ?name \"Alice\")"]
  ]
}
```

### Aggregation Query

```json
{
  "@context": {
    "ex": "http://example.org/ns/"
  },
  "select": [
    "?category",
    "(as (count ?product) ?count)",
    "(as (avg ?price) ?avgPrice)"
  ],
  "groupBy": ["?category"],
  "having": [["filter", "(> (count ?product) 5)"]],
  "where": [
    { "@id": "?product", "ex:category": "?category", "ex:price": "?price" }
  ],
  "orderBy": [["desc", "?count"]]
}
```

## Parse Options

JSON-LD queries accept parse-time options under a top-level `opts` object. These control how the query is parsed (not what it returns).

### `strictCompactIri`

By default, JSON-LD queries reject unresolved compact-looking IRIs (`prefix:suffix` where the prefix is not in `@context`) at parse time. To opt out:

```json
{
  "@context": {"ex": "http://example.org/ns/"},
  "opts": {"strictCompactIri": false},
  "select": ["?id", "?name"],
  "where": {"@id": "?id", "ex:name": "?name"}
}
```

The default is `true`. Disable only when you are intentionally working with bare `prefix:suffix` strings as opaque identifiers. See [IRIs and @context — Strict Compact-IRI Guard](../concepts/iri-and-context.md#strict-compact-iri-guard) for the full policy.

## Best Practices

1. **Always Provide @context**: Makes queries readable and maintainable
2. **Use Specific Patterns**: More specific patterns are more efficient
3. **Limit Result Sets**: Use `limit` for large result sets
4. **Flexible Filter Placement**: Filters can be placed anywhere in `where` clauses - the query engine automatically applies each filter as soon as all its required variables are bound
5. **Use Time Specifiers**: Use `@t:` when transaction numbers are known (fastest)
6. **Graph Source Selection**: Choose appropriate graph sources for query patterns

## Related Documentation

- [SPARQL](sparql.md): SPARQL query language
- [Time Travel](../concepts/time-travel.md): Historical queries
- [Graph Sources](../concepts/graph-sources.md): Graph source queries
- [Output Formats](output-formats.md): Query result formats
- [IRIs and @context](../concepts/iri-and-context.md): IRI resolution and the strict compact-IRI guard


---

# query/sparql.md

# SPARQL

Fluree provides full support for SPARQL 1.1, the W3C standard query language for RDF, plus the RDF 1.2 / SPARQL 1.2 edge-annotation subset (see [Edge annotations](#edge-annotations-sparql-12--rdf-12) below). SPARQL enables compatibility with existing RDF tools, knowledge graphs, and semantic web applications.

## Overview

SPARQL (SPARQL Protocol and RDF Query Language) is the industry standard for querying RDF data. Fluree implements SPARQL 1.1, providing full compatibility with SPARQL endpoints and tools.

### Basic SPARQL Query

```sparql
PREFIX ex: <http://example.org/ns/>

SELECT ?name ?age
WHERE {
  ?person ex:name ?name .
  ?person ex:age ?age .
}
```

### Default Prefixes

When querying via the **CLI**, a ledger's [default context](../concepts/iri-and-context.md#default-context) prefix mappings are injected into SPARQL queries that have no explicit `PREFIX` declarations. The HTTP API defaults this behavior off; pass `?default-context=true` on ledger-scoped query requests to opt in. For example, if the default context includes `{"ex": "http://example.org/ns/"}`, this query works without a `PREFIX` line when default-context injection is enabled:

```sparql
SELECT ?name ?age
WHERE {
  ?person ex:name ?name .
  ?person ex:age ?age .
}
```

If a query includes any `PREFIX` declarations, the default context is not used — you must declare every prefix you need. To explicitly opt out of the default context without defining any real prefix, use `PREFIX : <>`. See [opting out of the default context](../concepts/iri-and-context.md#opting-out-of-the-default-context) for details.

> **Note:** When using `fluree-db-api` directly (embedded), queries must declare their own `PREFIX` declarations. The default context is not injected automatically by the core API. Use `db_with_default_context()` or `GraphDb::with_default_context()` to opt in. See [Default Context](../concepts/iri-and-context.md#default-context) for details.

You can view and manage the default context with `fluree context get/set` or `GET/PUT /v1/fluree/context/{ledger...}`.

## Query Forms

### SELECT Queries

Return variable bindings:

```sparql
PREFIX ex: <http://example.org/ns/>

SELECT ?name ?email
WHERE {
  ?person ex:name ?name .
  ?person ex:email ?email .
}
```

**DISTINCT Results:**

```sparql
SELECT DISTINCT ?name
WHERE {
  ?person ex:name ?name .
}
```

**Reduced Results:**

```sparql
SELECT REDUCED ?name
WHERE {
  ?person ex:name ?name .
}
```

### CONSTRUCT Queries

Generate RDF graphs from query results:

```sparql
PREFIX ex: <http://example.org/ns/>

CONSTRUCT {
  ?person ex:displayName ?name .
}
WHERE {
  ?person ex:name ?name .
}
```

See [CONSTRUCT Queries](construct.md) for details.

### ASK Queries

Return boolean indicating if query matches:

```sparql
PREFIX ex: <http://example.org/ns/>

ASK {
  ?person ex:name "Alice" .
}
```

### DESCRIBE Queries

Return RDF description of resources:

```sparql
PREFIX ex: <http://example.org/ns/>

DESCRIBE ex:alice
```

Fluree’s DESCRIBE returns **outgoing triples** for each described resource (equivalent to `CONSTRUCT { ?r ?p ?o } WHERE { VALUES ?r { ... } . ?r ?p ?o }`).

## Basic Graph Patterns

### Triple Patterns

Match RDF triples:

```sparql
?person ex:name ?name .
```

### Multiple Patterns

Combine patterns with AND semantics:

```sparql
?person ex:name ?name .
?person ex:age ?age .
?person ex:email ?email .
```

### Property Paths

SPARQL property paths allow complex traversal patterns in the predicate position of a triple pattern.

#### Supported Operators

| Syntax | Name | Description |
|--------|------|-------------|
| `p+` | One or more | Transitive closure (follows `p` one or more hops) |
| `p*` | Zero or more | Reflexive transitive closure (includes self) |
| `^p` | Inverse | Traverses `p` in reverse direction |
| `p\|q` | Alternative | Matches either `p` or `q` (UNION semantics) |
| `p/q` | Sequence | Follows `p` then `q` (property chain) |

**One or More (`+`):**

```sparql
?person ex:parent+ ?ancestor .
```

**Zero or More (`*`):**

```sparql
?person ex:parent* ?ancestorOrSelf .
```

**Inverse (`^`):**

```sparql
?child ^ex:parent ?parent .
```

This is equivalent to `?parent ex:parent ?child` — it reverses the traversal direction.

Inverse can also be applied to complex paths (sequences and alternatives):

```sparql
?s ^(ex:friend/ex:name) ?o .   -- inverse of a sequence
?s ^(ex:name|ex:nick) ?o .     -- inverse of an alternative
```

- `^(ex:friend/ex:name)` reverses the step order and inverts each step: `(^ex:name)/(^ex:friend)`
- `^(ex:name|ex:nick)` distributes inverse into each branch: `(^ex:name)|(^ex:nick)`
- Double inverse cancels: `^(^ex:p)` simplifies to `ex:p`

**Alternative (`|`):**

```sparql
?person ex:friend|ex:colleague ?related .
```

This produces UNION semantics: results from both `ex:friend` and `ex:colleague` are combined (bag semantics, so duplicates are preserved).

Three-way and inverse alternatives are supported:

```sparql
?s ex:a|ex:b|ex:c ?o .
?s ex:friend|^ex:colleague ?related .
```

Alternative branches can also be sequence chains. For example, to get the name via either the friend or colleague path:

```sparql
?s (ex:friend/ex:name)|(ex:colleague/ex:name) ?name .
```

Branches can freely mix simple predicates, inverse predicates, and sequence chains:

```sparql
?s ex:name|(ex:friend/ex:name)|^ex:colleague ?val .
```

**Sequence (`/`) — Property Chains:**

```sparql
?person ex:friend/ex:name ?friendName .
```

This follows `ex:friend` then `ex:name`, expanding into a chain of triple patterns joined by internal variables. Multi-step chains are supported:

```sparql
?person ex:friend/ex:friend/ex:name ?fofName .
```

Sequence steps can include inverse predicates:

```sparql
?person ^ex:friend/ex:name ?name .
```

This traverses `ex:friend` backwards (finding who links to `?person`), then follows `ex:name` forward.

Sequence steps can also be alternatives. For example, `ex:friend/(ex:name|ex:nick)` distributes the alternative into a union of chains (`ex:friend/ex:name` and `ex:friend/ex:nick`):

```sparql
?person ex:friend/(ex:name|ex:nick) ?label .
```

Multiple alternative steps are supported: `(ex:a|ex:b)/(ex:c|ex:d)` expands to 4 chains. A safety limit of 64 expanded chains is enforced to prevent combinatorial explosion.

**Rules:**

- Transitive paths (`+`, `*`) require at least one variable (both subject and object cannot be constants).
- Sequence (`/`) steps must be simple predicates (`ex:p`), inverse simple predicates (`^ex:p`), or alternatives of simple predicates (`(ex:a|ex:b)`). Transitive (`+`/`*`) and nested sequence modifiers are not allowed inside sequence steps.
- Variable names starting with `?__` are reserved for internal use and will not appear in `SELECT *` (wildcard) output.

#### Not Yet Supported

The following operators are parsed but not yet supported for execution:

| Syntax | Name |
|--------|------|
| `p?` | Zero or one (optional step) |
| `!p` or `!(p\|q)` | Negated property set |

## Query Modifiers

### FILTER

Filter results with conditions:

```sparql
SELECT ?name ?age
WHERE {
  ?person ex:name ?name .
  ?person ex:age ?age .
  FILTER (?age > 18)
}
```

**Multiple Filters:**

```sparql
FILTER (?age > 18 && ?age < 65)
FILTER (regex(?name, "^A"))
```

### OPTIONAL

Match optional patterns:

```sparql
SELECT ?name ?email
WHERE {
  ?person ex:name ?name .
  OPTIONAL { ?person ex:email ?email . }
}
```

**Multiple Optionals:**

```sparql
SELECT ?name ?email ?phone
WHERE {
  ?person ex:name ?name .
  OPTIONAL { ?person ex:email ?email . }
  OPTIONAL { ?person ex:phone ?phone . }
}
```

### UNION

Match alternative patterns:

```sparql
SELECT ?name
WHERE {
  { ?person ex:name ?name . }
  UNION
  { ?person ex:alias ?name . }
}
```

### MINUS

Exclude matching patterns:

```sparql
SELECT ?person
WHERE {
  ?person ex:type ex:User .
  MINUS { ?person ex:status ex:Inactive . }
}
```

### BIND

Compute values:

```sparql
SELECT ?name ?nextAge
WHERE {
  ?person ex:name ?name .
  ?person ex:age ?age .
  BIND (?age + 1 AS ?nextAge)
}
```

### VALUES

Provide initial bindings:

```sparql
SELECT ?person ?name
WHERE {
  VALUES ?name { "Alice" "Bob" "Carol" }
  ?person ex:name ?name .
}
```

## Aggregation

### GROUP BY

Group results by variable:

```sparql
SELECT ?category (COUNT(?product) AS ?count)
WHERE {
  ?product ex:category ?category .
}
GROUP BY ?category
```

**Expression-based GROUP BY:**

Group by a computed expression using `(expr AS ?alias)` syntax:

```sparql
SELECT ?initial (COUNT(?name) AS ?count)
WHERE {
  ?person ex:name ?name .
}
GROUP BY (SUBSTR(?name, 1, 1) AS ?initial)
```

The expression is evaluated per row and bound to the alias variable before grouping. Any SPARQL expression is supported, including function calls, arithmetic, and type casts.

### HAVING

Filter grouped results:

```sparql
SELECT ?category (COUNT(?product) AS ?count)
WHERE {
  ?product ex:category ?category .
}
GROUP BY ?category
HAVING (COUNT(?product) > 10)
```

### Aggregation Functions

- `COUNT(?var)` - Count non-null values
- `SUM(?var)` - Sum numeric values
- `AVG(?var)` - Average numeric values
- `MIN(?var)` - Minimum value
- `MAX(?var)` - Maximum value
- `SAMPLE(?var)` - Arbitrary value from group
- `GROUP_CONCAT(?var; separator=",")` - Concatenate values

All aggregate functions support the `DISTINCT` modifier, which eliminates duplicate values before aggregation:

```sparql
SELECT ?category (COUNT(DISTINCT ?customer) AS ?unique_buyers)
                 (SUM(DISTINCT ?price) AS ?unique_price_total)
WHERE {
  ?order ex:category ?category .
  ?order ex:customer ?customer .
  ?order ex:price ?price .
}
GROUP BY ?category
```

**Aggregate result types:** COUNT and SUM of integers return `xsd:integer`. SUM of mixed numeric types and AVG return `xsd:double`.

## Sorting and Limiting

### ORDER BY

Sort results:

```sparql
SELECT ?name ?age
WHERE {
  ?person ex:name ?name .
  ?person ex:age ?age .
}
ORDER BY ?name
```

**Descending:**

```sparql
ORDER BY DESC(?age)
```

**Multiple Sort Keys:**

```sparql
ORDER BY ?last ASC(?first) DESC(?age)
```

### LIMIT

Limit number of results:

```sparql
SELECT ?name
WHERE {
  ?person ex:name ?name .
}
LIMIT 10
```

### OFFSET

Skip results:

```sparql
SELECT ?name
WHERE {
  ?person ex:name ?name .
}
OFFSET 20
LIMIT 10
```

## Datasets

### FROM

Specify default graph:

```sparql
PREFIX ex: <http://example.org/ns/>

SELECT ?name
FROM <mydb:main>
WHERE {
  ?person ex:name ?name .
}
```

**Multiple Default Graphs:**

```sparql
SELECT ?name
FROM <mydb:main>
FROM <otherdb:main>
WHERE {
  ?person ex:name ?name .
}
```

### FROM NAMED

Specify named graphs:

```sparql
PREFIX ex: <http://example.org/ns/>

SELECT ?graph ?name
FROM NAMED <mydb:main>
FROM NAMED <otherdb:main>
WHERE {
  GRAPH ?graph {
    ?person ex:name ?name .
  }
}
```

Fluree also exposes a built-in named graph inside each ledger for transaction / commit metadata:
- `FROM <mydb:main#txn-meta>` (txn-meta as the default graph), or
- `FROM NAMED <mydb:main#txn-meta>` and `GRAPH <mydb:main#txn-meta> { ... }`

See [Datasets](datasets.md) for details.

### GRAPH patterns without `FROM NAMED`

On the ledger-scoped query endpoint (`POST /query/{ledger}`), `GRAPH` patterns
resolve the ledger's registered named graphs by default — no `FROM NAMED` is
required:

```sparql
# Returns the named graph's triples even with no FROM NAMED
SELECT ?s ?p ?o WHERE { GRAPH <urn:probegraph> { ?s ?p ?o } }

# Discovers every user-registered named graph (plus the ledger alias,
# which addresses the default graph)
SELECT DISTINCT ?g WHERE { GRAPH ?g { ?s ?p ?o } }
```

Only user-registered named graphs are exposed this way; the reserved system
graphs (`#txn-meta`, `#config`) remain addressable only via an explicit
`FROM NAMED`. Supplying `FROM NAMED` still narrows `GRAPH` resolution to exactly
the graphs listed.

## SPARQL Functions

### String Functions

- `STR(?x)` - String value
- `LANG(?x)` - Language tag
- `LANGMATCHES(?lang, ?pattern)` - Language match
- `REGEX(?str, ?pattern)` - Regular expression
- `REPLACE(?str, ?pattern, ?replacement)` - Replace
- `SUBSTR(?str, ?start, ?length)` - Substring
- `STRLEN(?str)` - String length
- `UCASE(?str)` - Uppercase
- `LCASE(?str)` - Lowercase
- `ENCODE_FOR_URI(?str)` - URI encode
- `CONCAT(?str1, ?str2, ...)` - Concatenate

### Numeric Functions

- `ABS(?x)` - Absolute value
- `ROUND(?x)` - Round
- `CEIL(?x)` - Ceiling
- `FLOOR(?x)` - Floor
- `RAND()` - Random number

### Date/Time Functions

- `NOW()` - Current timestamp
- `YEAR(?date)` - Year
- `MONTH(?date)` - Month
- `DAY(?date)` - Day
- `HOURS(?time)` - Hours
- `MINUTES(?time)` - Minutes
- `SECONDS(?time)` - Seconds

### Type Conversion

- `STRDT(?str, ?datatype)` - String to typed literal
- `STRLANG(?str, ?lang)` - String with language
- `DATATYPE(?literal)` - Datatype IRI of a literal (per W3C SPARQL 1.1 §17.4.2.3, an IRI term — e.g. the `xsd:decimal` IRI, **not** the string `"xsd:decimal"`). Compare it against a datatype IRI directly, e.g. `FILTER(DATATYPE(?v) = xsd:decimal)`. SPARQL results render it as the full IRI; JSON-LD results compact it against the active `@context`.
- `IRI(?str)` - IRI from string
- `URI(?str)` - URI from string
- `BNODE(?str)` - Blank node

### XSD Datatype Constructors (Casts)

Per W3C SPARQL 1.1 §17.5, XSD constructor functions cast values between datatypes. Invalid casts produce unbound (no binding), not errors.

- `xsd:boolean(?x)` - Cast to boolean (`"true"`, `"1"` → true; `"false"`, `"0"` → false; numeric 0 → false, non-zero → true)
- `xsd:integer(?x)` - Cast to integer (truncates doubles, parses strings)
- `xsd:float(?x)` - Cast to single-precision float
- `xsd:double(?x)` - Cast to double-precision float
- `xsd:decimal(?x)` - Cast to decimal (rejects scientific notation strings)
- `xsd:string(?x)` - Cast to string (canonical form for decimals)

### Logical Functions

- `BOUND(?var)` - Variable is bound
- `IF(?condition, ?then, ?else)` - Conditional
- `COALESCE(?x, ?y, ...)` - First non-null value
- `ISIRI(?x)` - Is IRI
- `ISURI(?x)` - Is URI
- `ISBLANK(?x)` - Is blank node
- `ISLITERAL(?x)` - Is literal
- `ISNUMERIC(?x)` - Is numeric

## Subqueries

Nest queries:

```sparql
SELECT ?person ?name
WHERE {
  ?person ex:name ?name .
  {
    SELECT ?person
    WHERE {
      ?person ex:age ?age .
      FILTER (?age > 18)
    }
  }
}
```

## Service Queries

SERVICE enables cross-ledger queries within Fluree. You can execute patterns against different ledgers within the same query using the `fluree:ledger:` URI scheme.

### Basic Cross-Ledger Query

Query data from another ledger in your dataset:

```sparql
PREFIX ex: <http://example.org/ns/>

SELECT ?customer ?name ?total
FROM <customers:main>
FROM NAMED <orders:main>
WHERE {
  ?customer ex:name ?name .
  SERVICE <fluree:ledger:orders:main> {
    ?order ex:customer ?customer ;
           ex:total ?total .
  }
}
```

### Endpoint URI Format

For local Fluree ledger queries, use the `fluree:ledger:` scheme:

| Format | Description | Matches dataset ledger ID |
|--------|-------------|----------------------|
| `fluree:ledger:<name>` | Query ledger with default branch (main) | `<name>:main` |
| `fluree:ledger:<name>:<branch>` | Query specific branch | `<name>:<branch>` |

Where:
- `<name>` is the ledger name **without** the branch (e.g., `orders`, `acme/people`)
- `<branch>` is the branch name (e.g., `main`, `dev`)
- The full dataset ledger ID is always `<name>:<branch>` (e.g., `orders:main`, `acme/people:dev`)

The endpoint is resolved by matching against the full `ledger_id` in the dataset.

**Examples:**

```sparql
SERVICE <fluree:ledger:orders> { ... }         -- matches orders:main
SERVICE <fluree:ledger:orders:main> { ... }    -- matches orders:main (explicit)
SERVICE <fluree:ledger:orders:dev> { ... }     -- matches orders:dev
```

### SERVICE SILENT

Use `SERVICE SILENT` to return empty results instead of failing if the service errors or is unavailable:

```sparql
PREFIX ex: <http://example.org/ns/>

SELECT ?name ?order
WHERE {
  ?person ex:name ?name .
  SERVICE SILENT <fluree:ledger:orders:main> {
    ?order ex:customer ?person .
  }
}
```

If the `orders` ledger is not in the dataset or encounters an error, the query returns results with unbound `?order` values instead of failing.

### Variable Endpoints

SERVICE supports variable endpoints that iterate over available ledgers:

```sparql
PREFIX ex: <http://example.org/ns/>

SELECT ?ledger ?person ?name
FROM NAMED <db1:main>
FROM NAMED <db2:main>
WHERE {
  SERVICE ?ledger {
    ?person ex:name ?name .
  }
}
```

This queries all named ledgers in the dataset.

### Cross-Ledger Join Example

Join customer data from one ledger with their orders from another:

```sparql
PREFIX ex: <http://example.org/ns/>

SELECT ?customerName ?productName ?quantity
FROM <customers:main>
FROM NAMED <orders:main>
FROM NAMED <products:main>
WHERE {
  # Get customer from default graph (customers ledger)
  ?customer ex:name ?customerName .

  # Get orders for this customer from orders ledger
  SERVICE <fluree:ledger:orders:main> {
    ?order ex:customer ?customer ;
           ex:product ?product ;
           ex:quantity ?quantity .
  }

  # Get product details from products ledger
  SERVICE <fluree:ledger:products:main> {
    ?product ex:name ?productName .
  }
}
```

### Requirements

- The target ledger must be included in the dataset (via `FROM` or `FROM NAMED` clauses)
- Results are joined with the outer query on shared variables
- SERVICE patterns are executed as correlated subqueries (like EXISTS)

### Remote Fluree Federation

SERVICE supports querying ledgers on **remote Fluree instances** using the `fluree:remote:` scheme. This enables cross-server federation — a single SPARQL query can join data from local ledgers with data from ledgers on other Fluree servers.

#### Remote Endpoint Format

| Format | Description |
|--------|-------------|
| `fluree:remote:<connection>/<ledger>` | Query a ledger on a registered remote server |

Where:
- `<connection>` is a named remote connection registered at build time (maps to a server URL + bearer token)
- `<ledger>` is the ledger ID on the remote server (e.g., `customers:main`, `acme/people:main`)

#### Example: Cross-Server Join

```sparql
PREFIX ex: <http://example.org/ns/>

SELECT ?localName ?remoteEmail
WHERE {
  ?person ex:name ?localName .
  SERVICE <fluree:remote:acme/customers:main> {
    ?person ex:email ?remoteEmail .
  }
}
```

This queries `?person ex:name` from the local ledger and joins with `?person ex:email` from the `customers:main` ledger on the remote server named `acme`.

#### Multiple Ledgers on the Same Remote Server

A single remote connection gives access to any ledger the bearer token is authorized for:

```sparql
PREFIX ex: <http://example.org/ns/>

SELECT ?customer ?orderId ?productName
WHERE {
  SERVICE <fluree:remote:acme/customers:main> {
    ?customer ex:name ?name .
    ?customer ex:id ?customerId .
  }
  SERVICE <fluree:remote:acme/orders:main> {
    ?order ex:customerId ?customerId .
    ?order ex:orderId ?orderId .
    ?order ex:product ?product .
  }
  SERVICE <fluree:remote:acme/products:main> {
    ?product ex:name ?productName .
  }
}
```

#### SILENT with Remote Endpoints

`SERVICE SILENT` works with remote endpoints. If the remote server is unreachable, the connection is not registered, or the bearer token is rejected, the SERVICE block returns empty results instead of failing the query:

```sparql
SERVICE SILENT <fluree:remote:partner/inventory:main> {
  ?item ex:sku ?sku .
}
```

#### Registering Remote Connections

Remote connections are registered at connection build time via the Rust API or server configuration. See [Configuration: Remote connections](../operations/configuration.md#remote-connections) and [Rust API: Remote federation](../getting-started/rust-api.md#remote-federation) for setup details.

#### Datatype Handling

Remote query results preserve their original datatypes. Values returned from a remote server are parsed into the same rich type system used for local data — `xsd:dateTime`, `xsd:date`, `xsd:decimal`, `xsd:integer`, etc. are all stored with their proper typed representations. Custom datatypes (e.g., `http://example.org/myType`) are also preserved: the value is kept as a string with the original datatype IRI retained, so round-tripping and downstream FILTER comparisons on shared custom types work correctly.

#### Limitations (v1)

- **Uncorrelated execution only.** The SERVICE body is sent to the remote server as a standalone query. Parent-row bindings are not injected as VALUES (bound-join). This means a SERVICE block that references variables bound in the outer query will not push those constraints to the remote server — the remote returns all matching rows, and the join happens locally.
- **SPARQL queries only.** Remote SERVICE is available in SPARQL queries. JSON-LD queries do not currently support the `fluree:remote:` scheme.
- **No query cancellation propagation.** If the local query is cancelled, in-flight remote HTTP requests are not aborted.
- **Policy is local only.** The remote server enforces its own policy based on the bearer token. The local server's policy engine does not filter rows returned from a remote SERVICE.

### External SPARQL Endpoints

Federated queries to non-Fluree SPARQL endpoints (e.g., Wikidata, DBpedia) are not yet supported. Only the `fluree:ledger:` (local) and `fluree:remote:` (remote Fluree) schemes are currently available.

## Time Travel

### Point-in-Time Queries

Query data as it existed at a specific time using time specifiers in the `FROM` clause:

```sparql
PREFIX ex: <http://example.org/ns/>

SELECT ?name ?age
FROM <ledger:main@t:100>
WHERE {
  ?person ex:name ?name ;
          ex:age ?age .
}
```

Time specifiers:
- `@t:100` - Transaction number
- `@iso:2024-01-15T10:30:00Z` - ISO 8601 datetime
- `@commit:bafybeig...` - Commit ContentId
- `@t:latest` - Current/latest state

### History Queries

Query all changes (assertions and retractions) within a time range using `FROM...TO` with RDF-star syntax:

```sparql
PREFIX ex: <http://example.org/ns/>
PREFIX f: <https://ns.flur.ee/db#>

SELECT ?age ?t ?op
FROM <ledger:main@t:1>
TO <ledger:main@t:latest>
WHERE {
  << ex:alice ex:age ?age >> f:t ?t .
  << ex:alice ex:age ?age >> f:op ?op .
}
ORDER BY ?t
```

The `<< subject predicate object >>` syntax (RDF-star) treats the triple as an entity that can have metadata:
- `f:t` - Transaction time (integer) when the fact was asserted or retracted.
- `f:op` - Operation type as a boolean: `true` for assertions, `false` for retractions. Mirrors `Flake.op` on disk.

**Filter by operation type:**

```sparql
PREFIX ex: <http://example.org/ns/>
PREFIX f: <https://ns.flur.ee/db#>

SELECT ?age ?t
FROM <ledger:main@t:1>
TO <ledger:main@t:latest>
WHERE {
  << ex:alice ex:age ?age >> f:t ?t .
  << ex:alice ex:age ?age >> f:op ?op .
  FILTER(?op = false)
}
```

**History with ISO datetime range:**

```sparql
PREFIX ex: <http://example.org/ns/>
PREFIX f: <https://ns.flur.ee/db#>

SELECT ?name ?t ?op
FROM <ledger:main@iso:2024-01-01T00:00:00Z>
TO <ledger:main@iso:2024-12-31T23:59:59Z>
WHERE {
  << ex:alice ex:name ?name >> f:t ?t .
  << ex:alice ex:name ?name >> f:op ?op .
}
```

## Edge annotations (SPARQL 1.2 / RDF 1.2)

In addition to SPARQL 1.1, Fluree supports the RDF 1.2 / SPARQL 1.2 **edge-annotation** subset — the annotation tail `{| ... |}`, the `~` reifier, and `rdf:reifies` with `<<( ... )>>` triple terms — on both the query and UPDATE surfaces. The mandated `VERSION "1.2"` prologue declaration is accepted (lexed and skipped), so conformant 1.2 clients parse normally. This is the standards-based way to attach metadata to a specific edge `(subject, predicate, object)`. For the full model, output shape, and lifecycle, see **[Edge annotations](../concepts/edge-annotations.md)**.

### Querying annotations

Match an edge's annotation metadata with an annotation tail in `WHERE`:

```sparql
PREFIX ex: <http://example.org/>
SELECT ?role WHERE {
  ex:alice ex:worksFor ex:acme {| ex:role ?role |} .
}
```

Walk from an annotation subject back to the edge it reifies with `rdf:reifies` and a triple term:

```sparql
PREFIX ex:  <http://example.org/>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
SELECT ?ann ?role WHERE {
  ?ann rdf:reifies <<( ex:alice ex:worksFor ex:acme )>> ;
       ex:role ?role .
}
```

A triple term `<<( s p o )>>` is accepted **only** as the object of `rdf:reifies`. The bare (parenthesis-free) `<< s p o >>` form is a separate, Fluree-specific construct for `f:t` / `f:op` flake-metadata extraction (see [Time Travel](#history-queries) above) — the two do not compose.

### Updating with annotations

The reifier can be named (`~ <iri>`), a variable (`~ ?v`, templates only), blank (`~ _:b`), or anonymous (a bare `{| ... |}`):

```sparql
PREFIX ex: <http://example.org/>
INSERT DATA {
  ex:alice ex:worksFor ex:acme ~ ex:emp1 {| ex:role "Engineer" |} .
}
```

Annotation tails are supported in `INSERT DATA`, `DELETE DATA`, and `INSERT { } WHERE { }` / `DELETE { } WHERE { }` templates. Per-operation reifier rules (e.g. variables are template-only; blank/anonymous reifiers are rejected in `DELETE DATA`) are tabulated in the [concept doc](../concepts/edge-annotations.md#sparql-update-rules-by-operation).

### Boundaries (rejected at parse / lowering time)

- **Default graph only.** An annotation tail inside an explicit `GRAPH { }` block, or under a `WITH <g>` template, is rejected — SPARQL UPDATE annotations target the default graph. Use the JSON-LD `@annotation` surface to annotate an edge inside a named graph.
- **Simple-predicate triples only.** `?s ex:p1/ex:p2 ?o {| ... |}` (property-path) is rejected.
- **Triple terms only as `rdf:reifies` objects**; any other use errors at parse time.
- **No reserved predicates by hand.** The [system predicates](../reference/vocabulary.md#edge-annotation-predicates-reserved) that back annotations are rejected on every UPDATE clause; mint annotations only through the `~` / `{| |}` surface.
- **No annotations in `CONSTRUCT` templates** (the template output form is deferred); a `CONSTRUCT` whose `WHERE` uses annotations to filter still works.
- **SPARQL 1.2 triple-term functions** (`TRIPLE`, `SUBJECT`, `PREDICATE`, `OBJECT`, `isTRIPLE`, and the `BIND(<<( ?s ?p ?o )>> AS ?t)` constructor) are deferred.

## SPARQL UPDATE

Fluree supports SPARQL 1.1 Update for modifying data using standard SPARQL syntax, plus the RDF 1.2 annotation tail described in [Edge annotations](#edge-annotations-sparql-12--rdf-12) above. SPARQL UPDATE requests use the `application/sparql-update` content type and are sent to the update endpoints.

### INSERT DATA

Insert ground triples (no variables):

```sparql
PREFIX ex: <http://example.org/ns/>

INSERT DATA {
  ex:alice ex:name "Alice" .
  ex:alice ex:age 30 .
  ex:alice ex:email "alice@example.org" .
}
```

**HTTP Request:**

```bash
curl -X POST http://localhost:8090/v1/fluree/update/mydb:main \
  -H "Content-Type: application/sparql-update" \
  -d 'PREFIX ex: <http://example.org/ns/>
      INSERT DATA { ex:alice ex:name "Alice" }'
```

### DELETE DATA

Delete specific ground triples:

```sparql
PREFIX ex: <http://example.org/ns/>

DELETE DATA {
  ex:alice ex:email "alice@example.org" .
}
```

### DELETE WHERE

Delete triples matching a pattern:

```sparql
PREFIX ex: <http://example.org/ns/>

DELETE WHERE {
  ex:alice ex:age ?age .
}
```

This finds all `ex:age` values for `ex:alice` and deletes them.

### DELETE/INSERT (Modify)

The most powerful form combines WHERE, DELETE, and INSERT clauses:

```sparql
PREFIX ex: <http://example.org/ns/>

DELETE {
  ?person ex:age ?oldAge .
}
INSERT {
  ?person ex:age ?newAge .
}
WHERE {
  ?person ex:name "Alice" .
  ?person ex:age ?oldAge .
  BIND(?oldAge + 1 AS ?newAge)
}
```

**Update multiple properties:**

```sparql
PREFIX ex: <http://example.org/ns/>

DELETE {
  ?person ex:name ?oldName .
  ?person ex:status ?oldStatus .
}
INSERT {
  ?person ex:name "Alicia" .
  ?person ex:status ex:Active .
}
WHERE {
  ?person ex:name "Alice" .
  OPTIONAL { ?person ex:name ?oldName }
  OPTIONAL { ?person ex:status ?oldStatus }
}
```

### Dataset scoping for MODIFY (`WITH` / `USING` / `USING NAMED`)

SPARQL UPDATE `MODIFY` supports dataset scoping for named graphs:

- **`WITH <iri>`**: sets the default graph for INSERT/DELETE templates that don’t use an explicit `GRAPH <iri> { ... }` block.
- **`USING <iri>`**: scopes the default graph(s) for `WHERE` evaluation. Repeated `USING` clauses are evaluated as a **merged default graph**.
- **`USING NAMED <iri>`**: scopes which named graphs are visible to `WHERE` `GRAPH <iri> { ... }` patterns. Repeated `USING NAMED` clauses allow multiple named graphs.

### Blank Nodes in INSERT

Blank nodes can be used in INSERT templates to create new entities:

```sparql
PREFIX ex: <http://example.org/ns/>

INSERT DATA {
  _:newPerson ex:name "Bob" .
  _:newPerson ex:age 25 .
}
```

### Typed Literals

Specify datatypes explicitly:

```sparql
PREFIX ex: <http://example.org/ns/>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>

INSERT DATA {
  ex:alice ex:birthDate "1990-05-15"^^xsd:date .
  ex:alice ex:salary "75000.00"^^xsd:decimal .
  ex:alice ex:active "true"^^xsd:boolean .
}
```

### Language-Tagged Strings

Insert strings with language tags:

```sparql
PREFIX ex: <http://example.org/ns/>

INSERT DATA {
  ex:alice ex:name "Alice"@en .
  ex:alice ex:name "Alicia"@es .
  ex:alice ex:name "アリス"@ja .
}
```

### SPARQL UPDATE Restrictions

Current restrictions / boundaries:

- **Graph management operations**: `LOAD`, `CLEAR`, `DROP`, `CREATE`, `ADD`, `MOVE`, `COPY` are not yet supported.
- **Template graph variables**: INSERT/DELETE templates support `GRAPH <iri> { ... }` blocks, but `GRAPH ?g { ... }` is not yet supported.
- **DELETE WHERE + GRAPH blocks**: `GRAPH <iri> { ... }` blocks are not yet supported inside `DELETE WHERE { ... }`.
- **SERVICE**: Only local-ledger endpoints of the form `fluree:ledger:<name>[:<branch>]` are supported; arbitrary remote HTTP `SERVICE` endpoints are not supported.
- **Property paths**: Supported in `WHERE` (subject to Fluree capability settings).
- **Edge annotations are default-graph only**: an annotation tail (`{| ... |}`) inside an explicit `GRAPH { }` block or under a `WITH <g>` template is rejected; a blank or anonymous reifier is rejected in `DELETE DATA`. See [Edge annotations](#edge-annotations-sparql-12--rdf-12) for the full boundary list.

### Endpoint Usage

SPARQL UPDATE uses the update endpoints with `Content-Type: application/sparql-update`:

| Endpoint | Description |
|----------|-------------|
| `POST /v1/fluree/update` | Connection-scoped, requires `Fluree-Ledger` header |
| `POST /v1/fluree/update/<ledger...>` | Ledger-scoped, ledger from URL path |

**Examples:**

```bash
# Ledger-scoped (recommended)
curl -X POST http://localhost:8090/v1/fluree/update/mydb:main \
  -H "Content-Type: application/sparql-update" \
  -d 'PREFIX ex: <http://example.org/ns/>
      INSERT DATA { ex:alice ex:name "Alice" }'

# Connection-scoped with header
curl -X POST http://localhost:8090/v1/fluree/update \
  -H "Content-Type: application/sparql-update" \
  -H "Fluree-Ledger: mydb:main" \
  -d 'PREFIX ex: <http://example.org/ns/>
      INSERT DATA { ex:alice ex:name "Alice" }'
```

## Best Practices

1. **Use PREFIX Declarations**: Makes queries readable
2. **Automatic Pattern Optimization**: The query planner automatically reorders patterns for efficient execution using statistics-driven cardinality estimates
3. **Flexible FILTER Placement**: Filters can be placed anywhere in the WHERE clause — the query engine automatically applies each filter as soon as all its required variables are bound
4. **Limit Results**: Use LIMIT for large result sets
5. **Avoid Cartesian Products**: Structure queries to avoid large joins

## Related Documentation

- [JSON-LD Query](jsonld-query.md): Fluree's native query language
- [Edge annotations](../concepts/edge-annotations.md): RDF 1.2 / SPARQL 1.2 edge metadata (annotation tail, `~` reifier, `rdf:reifies`)
- [CONSTRUCT Queries](construct.md): Generating RDF graphs
- [Datasets](datasets.md): Multi-graph queries
- [Output Formats](output-formats.md): Query result formats
- [Transactions](../transactions/overview.md): JSON-LD transaction format


---

# query/output-formats.md

# Output Formats

Fluree supports multiple output formats for query results, each optimized for different use cases. You can choose the format that best fits your application's needs.

## Supported Formats

### JSON-LD Format

**Default format** for JSON-LD Query. Provides compact, context-aware JSON with IRI expansion/compaction.

**Characteristics:**
- Uses `@context` for IRI compaction
- Compact IRIs (e.g., `ex:alice` instead of full IRIs)
- Inferable datatypes (string, long, double, boolean) rendered as bare values
- Language tags preserved

**Example (graph crawl):**

```json
[
  {
    "@id": "ex:alice",
    "schema:name": "Alice",
    "schema:age": 30,
    "schema:knows": {"@id": "ex:bob"}
  }
]
```

**Example (tabular SELECT):**

```json
[
  ["Alice", 30],
  ["Bob", 25]
]
```

### SPARQL JSON Format

Standard SPARQL 1.1 result format for SPARQL queries.

**Characteristics:**
- W3C SPARQL 1.1 compliant
- Standard `results` and `bindings` structure
- Datatype information included
- Language tags included

**Example:**

```json
{
  "head": {
    "vars": ["name", "age"]
  },
  "results": {
    "bindings": [
      {
        "name": {
          "type": "literal",
          "value": "Alice"
        },
        "age": {
          "type": "literal",
          "value": "30",
          "datatype": "http://www.w3.org/2001/XMLSchema#integer"
        }
      }
    ]
  }
}
```

### Typed JSON Format

Type-preserving JSON format with explicit datatype information on every value. Works with both tabular SELECT queries and graph crawl (entity-centric) queries.

**Characteristics:**
- Every literal includes `{"@value": ..., "@type": "..."}` — even inferable types
- References use `{"@id": "..."}`
- Language-tagged strings use `{"@value": ..., "@language": "..."}`
- `@json` values use `{"@value": <parsed>, "@type": "@json"}`
- Nested entities in graph crawl results are also fully typed
- IRIs compacted via `@context`

**Example (tabular SELECT):**

```json
[
  {
    "?name": {"@value": "Alice", "@type": "xsd:string"},
    "?age": {"@value": 30, "@type": "xsd:long"}
  }
]
```

**Example (graph crawl):**

```json
[
  {
    "@id": "ex:alice",
    "@type": ["schema:Person"],
    "schema:name": {"@value": "Alice", "@type": "xsd:string"},
    "schema:age": {"@value": 30, "@type": "xsd:long"},
    "schema:knows": {
      "@id": "ex:bob",
      "schema:name": {"@value": "Bob", "@type": "xsd:string"}
    },
    "ex:data": {"@value": {"key": "val"}, "@type": "@json"}
  }
]
```

### Agent JSON Format

**Optimized for LLM/agent consumption.** Returns a self-describing envelope with a schema header, compact object rows using native JSON types, and built-in pagination support.

**Request via HTTP:**
```http
Accept: application/vnd.fluree.agent+json
Fluree-Max-Bytes: 32768
```

**Characteristics:**
- Schema-once header: datatypes declared per variable, not repeated per value
- Native JSON types for values (strings, numbers, booleans — no wrappers for inferable types)
- Non-inferable datatypes annotated inline only where needed (`{"@value": ..., "@type": "..."}`)
- Byte-budget truncation with `hasMore` flag and resume query
- Time-pinning metadata (`t` for single-ledger, `iso` wallclock timestamp for cross-ledger)

**Example (single-ledger, no truncation):**

```json
{
  "schema": {
    "?name": "xsd:string",
    "?age": "xsd:integer",
    "?s": "uri"
  },
  "rows": [
    {"?name": "Alice", "?age": 30, "?s": "ex:alice"},
    {"?name": "Bob", "?age": 25, "?s": "ex:bob"}
  ],
  "rowCount": 2,
  "t": 5,
  "iso": "2026-03-26T14:30:00Z",
  "hasMore": false
}
```

**Example (truncated, with resume query):**

```json
{
  "schema": {
    "?name": "xsd:string",
    "?age": "xsd:integer"
  },
  "rows": [
    {"?name": "Alice", "?age": 30},
    {"?name": "Bob", "?age": 25}
  ],
  "rowCount": 2,
  "t": 5,
  "iso": "2026-03-26T14:30:00Z",
  "hasMore": true,
  "message": "Response truncated due to size limit of 32768 bytes. Use the query below to retrieve the next batch.",
  "resume": "SELECT ?name ?age FROM <mydb:main@t:5> WHERE { ?s ex:name ?name ; ex:age ?age } OFFSET 2 LIMIT 100"
}
```

**Schema types:**
- Single type → string: `"?name": "xsd:string"`
- Mixed types → array: `"?value": ["xsd:string", "xsd:integer"]`
- IRI references → `"uri"`

**Envelope fields:**

| Field | Present | Description |
|-------|---------|-------------|
| `schema` | Always | Per-variable datatype map |
| `rows` | Always | Array of `{variable: value}` objects |
| `rowCount` | Always | Number of rows included |
| `t` | Single-ledger only | Transaction number used for the query |
| `iso` | Always | ISO-8601 wallclock timestamp at query time |
| `hasMore` | Always | Whether more rows exist beyond the byte budget |
| `message` | When truncated | Human-readable truncation explanation |
| `resume` | When truncated, single-FROM only | Ready-to-execute SPARQL with `@t:` pinning and OFFSET |

**Multi-ledger queries:** The `t` field is omitted (each ledger has its own timeline). The `resume` field is also omitted; instead, the `message` instructs the caller to use `@iso:` on each FROM clause for time-pinning.

**Byte budget:** Set via the `Fluree-Max-Bytes` header. When the cumulative serialized size of rows exceeds this limit, the formatter stops adding rows and sets `hasMore: true`. The budget applies to row data only (schema and envelope overhead are excluded from the count).

## Array Normalization

By default, graph crawl results return single-valued properties as bare scalars and multi-valued properties as arrays:

```json
{"schema:name": "Alice", "ex:tags": ["rust", "wasm"]}
```

This can be problematic for typed struct deserialization (e.g., a `Vec<String>` field that receives a bare string when only one value exists).

**`normalize_arrays`** forces all property values into arrays regardless of cardinality:

```json
{"schema:name": ["Alice"], "ex:tags": ["rust", "wasm"]}
```

This is orthogonal to typed JSON and can be combined with any format:

```rust
// Typed + normalized — most predictable for struct deserialization
let config = FormatterConfig::typed_json().with_normalize_arrays();

// JSON-LD + normalized — compact values but predictable shapes
let config = FormatterConfig::jsonld().with_normalize_arrays();
```

The `@container: @set` context annotation still forces arrays per-property and works regardless of the `normalize_arrays` setting.

## Format Selection

### JSON-LD Query

JSON-LD Query defaults to JSON-LD format. You can specify the format explicitly:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "select": ["?name", "?age"],
  "where": [
    { "@id": "?person", "ex:name": "?name", "ex:age": "?age" }
  ],
  "format": "jsonld"
}
```

### SPARQL

SPARQL queries return SPARQL JSON format by default:

```sparql
PREFIX ex: <http://example.org/ns/>

SELECT ?name ?age
WHERE {
  ?person ex:name ?name .
  ?person ex:age ?age .
}
```

## Datatype Handling

### String Types

**JSON-LD:**

```json
"Hello"
```

**Typed JSON:**

```json
{"@value": "Hello", "@type": "xsd:string"}
```

**SPARQL JSON:**

```json
{"type": "literal", "value": "Hello"}
```

### Numeric Types

**JSON-LD:**

```json
42
```

**Typed JSON:**

```json
{"@value": 42, "@type": "xsd:long"}
```

**SPARQL JSON:**

```json
{"type": "literal", "value": "42", "datatype": "http://www.w3.org/2001/XMLSchema#integer"}
```

### Language-Tagged Strings

All formats use the same representation:

```json
{"@value": "Hello", "@language": "en"}
```

### IRIs

**JSON-LD / Typed JSON:**

```json
{"@id": "ex:alice"}
```

**SPARQL JSON:**

```json
{"type": "uri", "value": "http://example.org/ns/alice"}
```

## Rust API

Use `FormatterConfig` to control output format via the query builder API:

```rust
use fluree_db_api::FormatterConfig;

// Single-ledger query with explicit format
let db = fluree.db("mydb:main").await?;
let result = db.query(&fluree)
    .sparql("SELECT ?name WHERE { ?s <schema:name> ?name }")
    .format(FormatterConfig::typed_json())
    .execute_formatted()
    .await?;

// Dataset query with format
let result = dataset.query(&fluree)
    .sparql("SELECT * WHERE { ?s ?p ?o }")
    .format(FormatterConfig::sparql_json())
    .execute_formatted()
    .await?;

// Connection-level query with format
let result = fluree.query_from()
    .jsonld(&query_with_from)
    .format(FormatterConfig::jsonld())
    .execute_formatted()
    .await?;

// AgentJson with byte budget and resume support
use fluree_db_api::AgentJsonContext;

let config = FormatterConfig::agent_json()
    .with_max_bytes(32768)
    .with_agent_json_context(AgentJsonContext {
        sparql_text: Some(sparql.to_string()),
        from_count: 1,
        iso_timestamp: Some(chrono::Utc::now().to_rfc3339()),
    });
let result = db.query(&fluree)
    .sparql("SELECT ?name ?age WHERE { ?s ex:name ?name ; ex:age ?age }")
    .format(config)
    .execute_formatted()
    .await?;

// Or directly on QueryResult:
let json = result.to_agent_json(&snapshot)?;                       // no budget
let json = result.to_agent_json_with_config(&snapshot, &config)?;  // with budget
```

Available format constructors:
- `FormatterConfig::jsonld()` — JSON-LD (default for JSON-LD queries)
- `FormatterConfig::sparql_json()` — SPARQL 1.1 JSON Results (default for SPARQL queries)
- `FormatterConfig::typed_json()` — Typed JSON with explicit datatypes on every value
- `FormatterConfig::agent_json()` — Agent JSON envelope for LLM/agent consumers

Builder methods:
- `.with_normalize_arrays()` — Force array wrapping for all graph crawl properties
- `.with_pretty()` — Pretty-print JSON output
- `.with_max_bytes(n)` — Set byte budget for AgentJson truncation
- `.with_agent_json_context(ctx)` — Set SPARQL text, FROM count, and ISO timestamp for AgentJson resume queries

All three query paths (`db.query()`, `dataset.query()`, `fluree.query_from()`) support `.format()`.

### Direct formatting on QueryResult

For graph crawl queries (which require async DB access):

```rust
// Typed JSON with graph crawl support
let json = result.to_typed_json_async(db.as_graph_db_ref()).await?;

// Custom config (e.g., typed + normalize_arrays)
let config = FormatterConfig::typed_json().with_normalize_arrays();
let json = result.format_async(db.as_graph_db_ref(), &config).await?;
```

When no `.format()` is set:
- JSON-LD queries default to JSON-LD format
- SPARQL queries default to SPARQL JSON format

## CLI Usage

The `fluree query` command supports format selection via `--format`:

```bash
# Default table output
fluree query "SELECT ?s ?p ?o WHERE { ?s ?p ?o } LIMIT 5"

# JSON output
fluree query --format json '{"select": {"ex:alice": ["*"]}, "from": "mydb:main"}'

# Typed JSON output (explicit types on every value)
fluree query --format typed-json '{"select": {"ex:alice": ["*"]}, "from": "mydb:main"}'

# Normalize arrays (force all properties to arrays)
fluree query --format json --normalize-arrays '{"select": {"ex:alice": ["*"]}, "from": "mydb:main"}'

# Typed JSON + normalize arrays (most predictable for programmatic use)
fluree query --format typed-json --normalize-arrays '{"select": {"ex:alice": ["*"]}, "from": "mydb:main"}'
```

## Performance Considerations

- **JSON string output is streamed** — when emitting a JSON **string** (the `format_results_string*` entry points used by the HTTP server), JSON-LD, SPARQL JSON, Typed JSON, and Agent JSON serialize directly into the output buffer, skipping the intermediate `serde_json::Value` DOM and its second serialization pass. Output is byte-identical to the DOM path (enforced by parity tests). The DOM path (`serde_json::Value`-returning `format_results*`) is still used for the programmatic `Value` API, for `pretty` output, and as the parity reference; `selectOne`, ASK, CONSTRUCT/DESCRIBE, and hydration also fall back to it.
- **JSON-LD** is the most efficient format — inferable types skip the `@value`/`@type` wrapper
- **Typed JSON** adds a constant-factor overhead per literal value (the `@value`/`@type` wrapper). Query execution is unaffected — only the formatting phase is slower.
- **normalize_arrays** adds zero overhead when disabled (default). When enabled, it skips the `len() == 1` check — no additional allocations beyond the array wrapper.
- **TSV/CSV** bypass JSON construction entirely for maximum throughput

## Best Practices

1. **Use JSON-LD for human-facing apps**: Compact and readable
2. **Use Typed JSON for struct deserialization**: Unambiguous types prevent parsing surprises
3. **Use `normalize_arrays` for typed consumers**: Ensures `Vec<T>` fields always get arrays
4. **Use SPARQL JSON for standard tooling**: Interoperable with SPARQL clients
5. **Use TSV/CSV for bulk export**: Highest throughput, smallest memory footprint
6. **Use Agent JSON for LLM/agent integrations**: Schema-once + pagination prevents context window overflow

## Related Documentation

- [JSON-LD Query](jsonld-query.md): JSON-LD Query language
- [SPARQL](sparql.md): SPARQL query language
- [Datatypes](../concepts/datatypes.md): Type system details


---

# query/datasets.md

# Datasets and Multi-Graph Execution

Fluree supports SPARQL datasets, enabling queries across multiple graphs and ledgers simultaneously. This provides powerful data integration capabilities for complex applications.

## SPARQL Datasets

A **dataset** in SPARQL is a collection of graphs used for query execution:

- **Default Graph**: The primary graph for triple patterns without GRAPH clauses
- **Named Graphs**: Additional graphs identified by IRIs, accessible via GRAPH clauses

## FROM Clauses

### Single Default Graph

Specify a single default graph:

**JSON-LD Query:**

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "mydb:main",
  "select": ["?name"],
  "where": [
    { "@id": "?person", "ex:name": "?name" }
  ]
}
```

**SPARQL:**

```sparql
PREFIX ex: <http://example.org/ns/>

SELECT ?name
FROM <mydb:main>
WHERE {
  ?person ex:name ?name .
}
```

### Multiple Default Graphs

Specify multiple default graphs (union semantics):

**JSON-LD Query:**

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": ["mydb:main", "otherdb:main"],
  "select": ["?name"],
  "where": [
    { "@id": "?person", "ex:name": "?name" }
  ]
}
```

**SPARQL:**

```sparql
PREFIX ex: <http://example.org/ns/>

SELECT ?name
FROM <mydb:main>
FROM <otherdb:main>
WHERE {
  ?person ex:name ?name .
}
```

## FROM NAMED Clauses

### Named graph sources (datasets)

In SPARQL, `FROM NAMED` identifies **named graphs in the dataset**. In Fluree, these are often *graph sources* such as:
- another ledger (federation / multi-ledger queries), or
- a graph source (search, tabular mapping, etc.).

Note: On the **ledger-scoped HTTP query endpoint** (`POST /query/{ledger}`), `FROM` / `FROM NAMED` is also supported, but is interpreted as selecting **named graphs inside that same ledger**. Use the connection-scoped endpoint (`POST /query`) when you want a dataset that spans multiple ledgers.

Query across multiple named graph sources:

**JSON-LD Query:**

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "fromNamed": {
    "mydb": { "@id": "mydb:main" },
    "otherdb": { "@id": "otherdb:main" }
  },
  "select": ["?graph", "?name"],
  "where": [
    ["graph", "?graph", { "@id": "?person", "ex:name": "?name" }]
  ]
}
```

**SPARQL:**

```sparql
PREFIX ex: <http://example.org/ns/>

SELECT ?graph ?name
FROM NAMED <mydb:main>
FROM NAMED <otherdb:main>
WHERE {
  GRAPH ?graph {
    ?person ex:name ?name .
  }
}
```

### Specific Named Graph

Query a specific named graph:

**SPARQL:**

```sparql
PREFIX ex: <http://example.org/ns/>

SELECT ?name
FROM NAMED <mydb:main>
WHERE {
  GRAPH <mydb:main> {
    ?person ex:name ?name .
  }
}
```

### Ledger named graph: `txn-meta`

Fluree provides a built-in named graph inside each ledger for transactional / commit metadata: **`txn-meta`**.

Use the `#txn-meta` fragment on a ledger reference:
- `mydb:main#txn-meta`
- `mydb:main@t:100#txn-meta` (time pinned)

**JSON-LD Query (txn-meta as the default graph):**

```json
{
  "@context": {
    "f": "https://ns.flur.ee/db#",
    "ex": "http://example.org/ns/"
  },
  "from": "mydb:main#txn-meta",
  "select": ["?commit", "?t", "?machine"],
  "where": [
    { "@id": "?commit", "f:t": "?t" },
    { "@id": "?commit", "ex:machine": "?machine" }
  ]
}
```

**SPARQL Query:**

```sparql
PREFIX f: <https://ns.flur.ee/db#>
PREFIX ex: <http://example.org/ns/>

SELECT ?commit ?t ?machine
FROM <mydb:main#txn-meta>
WHERE {
  ?commit f:t ?t .
  OPTIONAL { ?commit ex:machine ?machine }
}
```

### User-Defined Named Graphs

Fluree supports user-defined named graphs ingested via TriG format. These graphs are queryable using the structured `from` object syntax with a `graph` field.

For the ledger-scoped HTTP endpoint (`POST /query/{ledger}`), the server also accepts a convenient shorthand:
- `"from": "txn-meta"` / `"from": "default"` / `"from": "<graph IRI>"`
to select a graph **within** the ledger in the URL.

**Ingesting data with named graphs (TriG):**

```bash
curl -X POST "http://localhost:8090/v1/fluree/upsert?ledger=mydb:main" \
  -H "Content-Type: application/trig" \
  -d '@prefix ex: <http://example.org/ns/> .

      GRAPH <http://example.org/graphs/products> {
          ex:widget ex:name "Widget" ;
                    ex:price "29.99"^^xsd:decimal .
      }'
```

**Querying the named graph (JSON-LD):**

Use the structured `from` object with a `graph` field specifying the graph IRI:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": {
    "@id": "mydb:main",
    "graph": "http://example.org/graphs/products"
  },
  "select": ["?name", "?price"],
  "where": [
    { "@id": "?product", "ex:name": "?name" },
    { "@id": "?product", "ex:price": "?price" }
  ]
}
```

**With time-travel:**

```json
{
  "from": {
    "@id": "mydb:main",
    "t": 100,
    "graph": "http://example.org/graphs/products"
  },
  "select": ["?name", "?price"],
  "where": [...]
}
```

**Combining multiple graphs (JSON-LD):**

Query across the default graph and user-defined named graphs:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": "mydb:main",
  "fromNamed": {
    "products": {
      "@id": "mydb:main",
      "@graph": "http://example.org/graphs/products"
    }
  },
  "select": ["?company", "?product", "?price"],
  "where": [
    { "@id": "?company", "@type": "ex:Company" },
    ["graph", "products", { "@id": "?product", "ex:name": "?productName", "ex:price": "?price" }]
  ]
}
```

**Notes:**
- Named graphs are queryable after indexing completes
- The `@graph` field accepts the full graph IRI (no URL-encoding required)
- Time-travel is specified via the `t`, `iso`, or `sha` field in the object form
- Object keys in `fromNamed` serve as dataset-local aliases for use in GRAPH patterns

## Graph Source Object Schema

### `fromNamed` (named graphs) — preferred format

`fromNamed` is an object whose keys are dataset-local aliases. Each value has:

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `@id` | string | Yes | Ledger reference (e.g., `mydb:main`, `mydb:main@t:100`) |
| `@graph` | string | No | Graph selector: `"default"`, `"txn-meta"`, or full IRI |
| `t` | integer | No | Time-travel: specific transaction number |
| `at` | string | No | Time-travel: ISO-8601 timestamp or `commit:<hash>` |
| `policy` | object | No | Per-source policy override (see below) |

### `from` (default graphs) — object syntax

When using object syntax for `from`, the following fields are available:

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `@id` | string | Yes | Ledger reference (e.g., `mydb:main`, `mydb:main@t:100`) |
| `alias` | string | No | Dataset-local alias for GRAPH pattern reference |
| `graph` | string | No | Graph selector: `"default"`, `"txn-meta"`, or full IRI |
| `t` | integer | No | Time-travel: specific transaction number |
| `iso` | string | No | Time-travel: ISO-8601 timestamp |
| `commit_id` | string | No | Time-travel: commit ContentId |
| `policy` | object | No | Per-source policy override (see below) |

> **Legacy format:** The array format `"from-named": [...]` with `"alias"` and `"graph"` fields is still accepted for backward compatibility. The `"fromNamed"` object format is preferred.

### Dataset-Local Aliases

Aliases provide short names for referencing graphs in query patterns. They are especially useful when:

1. **Same graph IRI exists in multiple ledgers** - Use distinct aliases to disambiguate
2. **Complex IRIs** - Use short aliases instead of repeating long IRIs

**Example: Disambiguating same graph IRI across ledgers**

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "fromNamed": {
    "salesProducts": {
      "@id": "sales:main",
      "@graph": "http://example.org/vocab#products"
    },
    "inventoryProducts": {
      "@id": "inventory:main",
      "@graph": "http://example.org/vocab#products"
    }
  },
  "select": ["?g", "?sku", "?data"],
  "where": [
    ["graph", "?g", { "@id": "?sku", "ex:data": "?data" }]
  ]
}
```

In this example, both ledgers have a graph with the same IRI (`http://example.org/vocab#products`). The aliases `salesProducts` and `inventoryProducts` (the object keys) allow you to reference them distinctly.

**Validation Rules:**
- Aliases must be unique across the entire dataset (both `from` and `fromNamed`)
- Aliases cannot collide with identifiers (the `@id` values)
- Duplicate aliases will cause an error

### Graph Selector Values

The `graph` field accepts three types of values:

| Value | Meaning |
|-------|---------|
| `"default"` | Explicitly select the ledger's default graph |
| `"txn-meta"` | Select the built-in transaction metadata graph (`urn:fluree:{ledger_id}#txn-meta`) |
| `"<full-iri>"` | Select a user-defined named graph by its full IRI |

**Note:** If using `#txn-meta` fragment syntax in `@id`, do not also specify `graph: "txn-meta"`. This is considered ambiguous and will return an error.

### Per-Source Policy Override

Each graph source can have its own policy, enabling fine-grained access control where different graphs in the same query use different policies.

**Policy object fields:**
- `identity`: Identity IRI string
- `policy-class`: Policy class IRI or array of IRIs
- `policy`: Inline policy JSON
- `policy-values`: Policy parameter values
- `default-allow`: Boolean (default: false). Governs access when no policies match. Ignored (forced false) if `identity` is specified but has no subject node in the ledger.

**Example:**

```json
{
  "from": [
    {
      "@id": "public:main",
      "policy": {
        "default-allow": true
      }
    },
    {
      "@id": "sensitive:main",
      "policy": {
        "identity": "did:fluree:alice",
        "policy-class": ["ex:EmployeePolicy"],
        "default-allow": false
      }
    }
  ],
  "select": ["?data"],
  "where": [{ "@id": "?s", "ex:data": "?data" }]
}
```

**Policy Precedence:**
- Per-source `policy` takes precedence over global `opts` policy
- If a source has no `policy` field, the global policy (if any) applies

## Multi-Ledger Queries

Query across different ledgers:

**JSON-LD Query:**

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": ["customers:main", "orders:main"],
  "select": ["?customer", "?order"],
  "where": [
    { "@id": "?customer", "ex:name": "Alice" },
    { "@id": "?order", "ex:customer": "?customer" }
  ]
}
```

**SPARQL:**

```sparql
PREFIX ex: <http://example.org/ns/>

SELECT ?customer ?order
FROM <customers:main>
FROM <orders:main>
WHERE {
  ?customer ex:name "Alice" .
  ?order ex:customer ?customer .
}
```

## Time-Aware Datasets

Query graphs at different time points:

**JSON-LD Query:**

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": ["ledger1:main@t:100", "ledger2:main@t:200"],
  "select": ["?data"],
  "where": [
    { "@id": "?entity", "ex:data": "?data" }
  ]
}
```

**SPARQL:**

```sparql
PREFIX ex: <http://example.org/ns/>

SELECT ?data
FROM <ledger1:main@t:100>
FROM <ledger2:main@t:200>
WHERE {
  ?entity ex:data ?data .
}
```

## Graph Patterns

### Default Graph Only

Query only the default graph:

**SPARQL:**

```sparql
SELECT ?name
FROM <mydb:main>
WHERE {
  ?person ex:name ?name .
  # Matches triples in default graph only
}
```

### Named Graph Only

Query only named graphs:

**SPARQL:**

```sparql
SELECT ?name
FROM NAMED <mydb:main>
WHERE {
  GRAPH <mydb:main> {
    ?person ex:name ?name .
  }
}
```

### Mixed Patterns

Combine default and named graph patterns:

**SPARQL:**

```sparql
PREFIX f: <https://ns.flur.ee/db#>
PREFIX ex: <http://example.org/ns/>

SELECT ?name ?commit ?t
FROM <mydb:main>
FROM NAMED <mydb:main#txn-meta>
WHERE {
  ?person ex:name ?name .
  GRAPH <mydb:main#txn-meta> {
    ?commit f:t ?t .
  }
}
```

## Use Cases

### Data Integration

Combine data from multiple sources:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": ["customers:main", "products:main", "orders:main"],
  "select": ["?customer", "?product", "?order"],
  "where": [
    { "@id": "?customer", "ex:name": "Alice" },
    { "@id": "?order", "ex:customer": "?customer" },
    { "@id": "?order", "ex:product": "?product" }
  ]
}
```

### Cross-Ledger Joins

Join data across different ledgers:

```sparql
PREFIX ex: <http://example.org/ns/>

SELECT ?customer ?order ?product
FROM <customers:main>
FROM <orders:main>
FROM <products:main>
WHERE {
  ?customer ex:name "Alice" .
  ?order ex:customer ?customer .
  ?order ex:product ?product .
}
```

### SERVICE for Cross-Ledger Queries

Use SPARQL SERVICE to explicitly target specific ledgers within a query:

```sparql
PREFIX ex: <http://example.org/ns/>

SELECT ?customerName ?productName ?quantity
FROM <customers:main>
FROM NAMED <orders:main>
FROM NAMED <products:main>
WHERE {
  # Get customer from default graph
  ?customer ex:name ?customerName .

  # Get orders from orders ledger
  SERVICE <fluree:ledger:orders:main> {
    ?order ex:customer ?customer ;
           ex:product ?product ;
           ex:quantity ?quantity .
  }

  # Get product details from products ledger
  SERVICE <fluree:ledger:products:main> {
    ?product ex:name ?productName .
  }
}
```

SERVICE provides explicit control over which ledger each pattern executes against, enabling complex cross-ledger joins with clear data provenance.

See [SPARQL Service Queries](sparql.md#service-queries) for full documentation.

### Time-Consistent Queries

Query multiple ledgers at the same point in time:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "from": [
    "products:main@t:1000",
    "inventory:main@t:1000",
    "pricing:main@t:1000"
  ],
  "select": ["?product", "?stock", "?price"],
  "where": [
    { "@id": "?product", "ex:stockLevel": "?stock" },
    { "@id": "?product", "ex:price": "?price" }
  ]
}
```

## Error Handling

### Common Dataset Errors

| Error | Cause | Resolution |
|-------|-------|------------|
| Duplicate alias | Same `alias` used twice in dataset spec | Use unique aliases for each source |
| Alias collision | `alias` matches an existing `@id` | Choose a different alias name |
| Ambiguous graph selector | Both `#txn-meta` fragment AND `graph` field specified | Use only one method |
| Unknown ledger | Ledger reference not found | Verify ledger exists and is accessible |
| Unknown graph IRI | Graph IRI not found in ledger | Verify graph was ingested and indexed |
| Binary index required | Named graph query requires binary index | Ensure ledger has been indexed |

### Example Error Messages

**Duplicate alias:**
```json
{
  "error": "Duplicate dataset-local alias: 'products' appears multiple times"
}
```

**Ambiguous graph selector:**
```json
{
  "error": "Ambiguous graph selector: cannot specify both #txn-meta fragment and graph field"
}
```

## SPARQL Execution Modes

Fluree supports two SPARQL execution modes:

### Ledger-Bound Mode

When a query targets a single ledger (via endpoint or single FROM clause), GRAPH patterns reference **named graphs within that ledger**:

```sparql
-- Ledger-bound: GRAPH references graphs inside mydb:main
SELECT ?name ?price
FROM <mydb:main>
WHERE {
  GRAPH <http://example.org/graphs/products> {
    ?product ex:name ?name ;
             ex:price ?price .
  }
}
```

### Connection-Bound Mode

When querying across multiple ledgers, use SERVICE to select which ledger each pattern executes against:

```sparql
-- Connection-bound: SERVICE selects the target ledger
SELECT ?name ?stock
WHERE {
  SERVICE <fluree:ledger:sales:main> {
    GRAPH <http://example.org/graphs/products> {
      ?product ex:name ?name .
    }
  }
  SERVICE <fluree:ledger:inventory:main> {
    ?product ex:stock ?stock .
  }
}
```

**When to use each mode:**
- **Ledger-bound**: Single ledger queries, standard SPARQL datasets within one ledger
- **Connection-bound**: Multi-ledger queries, explicit control over data provenance

## Best Practices

1. **Consistent Time Points**: Use the same time specifier for all graphs in a query
2. **Graph Selection**: Use FROM NAMED when you need to identify the source graph
3. **Use Aliases**: Create meaningful aliases for complex graph IRIs or disambiguation
4. **Performance**: Queries across multiple ledgers may be slower
5. **Data Locality**: Consider data locality when designing multi-ledger queries
6. **Policy Granularity**: Use per-source policy when different graphs need different access control

## Related Documentation

- [JSON-LD Query](jsonld-query.md): JSON-LD Query syntax
- [SPARQL](sparql.md): SPARQL syntax
- [Time Travel](../concepts/time-travel.md): Historical queries
- [Datasets and Named Graphs](../concepts/datasets-and-named-graphs.md): Concept documentation


---

# query/construct.md

# CONSTRUCT Queries

CONSTRUCT queries generate RDF graphs from query results, enabling you to transform and reshape data into new graph structures.

## Overview

CONSTRUCT queries return RDF graphs instead of variable bindings. They're useful for:
- Extracting subgraphs
- Transforming data structures
- Creating new graph views
- Generating RDF for export

## Basic CONSTRUCT

### SPARQL CONSTRUCT

```sparql
PREFIX ex: <http://example.org/ns/>

CONSTRUCT {
  ?person ex:displayName ?name .
}
WHERE {
  ?person ex:name ?name .
}
```

This generates a new graph with `ex:displayName` properties from `ex:name` values.

### Multiple Triples

Construct multiple triples per solution:

```sparql
PREFIX ex: <http://example.org/ns/>

CONSTRUCT {
  ?person ex:displayName ?name .
  ?person ex:hasAge ?age .
}
WHERE {
  ?person ex:name ?name .
  ?person ex:age ?age .
}
```

## Complex Patterns

### Conditional Construction

Use filters to conditionally construct triples:

```sparql
PREFIX ex: <http://example.org/ns/>

CONSTRUCT {
  ?person ex:status ex:Adult .
}
WHERE {
  ?person ex:age ?age .
  FILTER (?age >= 18)
}
```

### Transitive Relationships

Construct inferred relationships:

```sparql
PREFIX ex: <http://example.org/ns/>

CONSTRUCT {
  ?person ex:knows ?friendOfFriend .
}
WHERE {
  ?person ex:friend ?friend .
  ?friend ex:friend ?friendOfFriend .
}
```

## CONSTRUCT with Aggregation

Construct triples from aggregated data:

```sparql
PREFIX ex: <http://example.org/ns/>

CONSTRUCT {
  ?category ex:productCount ?count .
}
WHERE {
  {
    SELECT ?category (COUNT(?product) AS ?count)
    WHERE {
      ?product ex:category ?category .
    }
    GROUP BY ?category
  }
}
```

## Use Cases

### Extract Subgraph

Extract a subgraph for a specific entity:

```sparql
PREFIX ex: <http://example.org/ns/>

CONSTRUCT {
  ?s ?p ?o .
}
WHERE {
  ex:alice ?p ?o .
  BIND (ex:alice AS ?s)
}
```

### Transform Data Structure

Transform data into a different structure:

```sparql
PREFIX ex: <http://example.org/ns/>

CONSTRUCT {
  ?order ex:hasItem [
    ex:product ?product ;
    ex:quantity ?quantity
  ] .
}
WHERE {
  ?order ex:item ?item .
  ?item ex:product ?product .
  ?item ex:quantity ?quantity .
}
```

### Generate Inferred Facts

Generate inferred relationships:

```sparql
PREFIX ex: <http://example.org/ns/>

CONSTRUCT {
  ?person ex:ancestor ?ancestor .
}
WHERE {
  ?person ex:parent+ ?ancestor .
}
```

## Best Practices

1. **Specific Patterns**: Construct specific patterns rather than wildcards
2. **Filter Early**: Apply filters in WHERE clause, not CONSTRUCT
3. **Avoid Duplicates**: Use DISTINCT if needed
4. **Performance**: CONSTRUCT can be expensive for large result sets

## Related Documentation

- [SPARQL](sparql.md): SPARQL query language
- [JSON-LD Query](jsonld-query.md): JSON-LD Query language
- [Output Formats](output-formats.md): Result formats


---

# query/graph-crawl.md

# Graph Crawl

Graph crawl enables recursive traversal of relationships — following links between entities to discover connected data. This is built on **property paths**, which provide operators for transitive, inverse, and multi-predicate traversal.

## Overview

Graph crawl queries traverse relationships in the graph, following links from one entity to another. Common use cases:

- **Social networks** — Find friends-of-friends, influence chains
- **Organizational hierarchies** — Traverse reporting structures
- **Knowledge graphs** — Follow related concepts across multiple hops
- **Dependency graphs** — Trace transitive dependencies
- **Bill of materials** — Recursive part containment

## Property path operators

Property paths are the foundation of graph crawl. They let you follow relationships beyond a single hop.

| Operator | Syntax | Description | Example |
|---|---|---|---|
| One or more (`+`) | `ex:knows+` | Follow 1+ times (transitive closure) | Friends of friends |
| Zero or more (`*`) | `ex:knows*` | Follow 0+ times (includes self) | Self and all reachable |
| Inverse (`^`) | `^ex:reportsTo` | Follow in reverse direction | Who reports to me? |
| Alternative (`\|`) | `ex:knows\|ex:colleague` | Match any of several predicates | Social or professional connections |
| Sequence (`/`) | `ex:knows/ex:name` | Chain of predicates | Names of friends |

### JSON-LD Query syntax

Property paths are defined using `@path` in the `@context`:

```json
{
  "@context": {
    "ex": "http://example.org/",
    "allReports": { "@path": "^ex:reportsTo+" }
  },
  "select": ["?name"],
  "where": [
    { "@id": "ex:ceo", "allReports": "?person" },
    { "@id": "?person", "ex:name": "?name" }
  ]
}
```

Two syntax forms are available:

**String form** (SPARQL-style):
```json
"knowsTransitive": { "@path": "ex:knows+" }
```

**Array form** (S-expression):
```json
"knowsTransitive": { "@path": ["+", "ex:knows"] }
```

### SPARQL syntax

Property paths are native SPARQL syntax:

```sparql
PREFIX ex: <http://example.org/>

# All people reachable through ex:knows (1+ hops)
SELECT ?person WHERE {
  ex:alice ex:knows+ ?person .
}
```

## Patterns

### Friend-of-friend network

Find everyone Alice knows, directly or transitively:

**SPARQL:**
```sparql
PREFIX ex: <http://example.org/>
PREFIX schema: <http://schema.org/>

SELECT ?name WHERE {
  ex:alice ex:knows+ ?person .
  ?person schema:name ?name .
}
```

**JSON-LD:**
```json
{
  "@context": {
    "ex": "http://example.org/",
    "schema": "http://schema.org/",
    "knowsTransitive": { "@path": "ex:knows+" }
  },
  "select": ["?name"],
  "where": [
    { "@id": "ex:alice", "knowsTransitive": "?person" },
    { "@id": "?person", "schema:name": "?name" }
  ]
}
```

### Organizational hierarchy

Find all people who report to a manager (at any level):

```sparql
PREFIX ex: <http://example.org/>
PREFIX schema: <http://schema.org/>

SELECT ?name WHERE {
  ?person ex:reportsTo+ ex:vp-engineering .
  ?person schema:name ?name .
}
```

Or use inverse path to start from the top:

```sparql
SELECT ?name WHERE {
  ex:vp-engineering ^ex:reportsTo+ ?person .
  ?person schema:name ?name .
}
```

### Class hierarchy (RDFS)

Find all subclasses of a class:

```sparql
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>

SELECT ?subclass WHERE {
  ?subclass rdfs:subClassOf+ ex:Vehicle .
}
```

### Path chaining (sequence)

Follow a chain of different predicates:

```sparql
PREFIX ex: <http://example.org/>
PREFIX schema: <http://schema.org/>

# Names of Alice's friends' managers
SELECT ?managerName WHERE {
  ex:alice ex:knows/ex:reportsTo ?manager .
  ?manager schema:name ?managerName .
}
```

In JSON-LD:
```json
{
  "@context": {
    "ex": "http://example.org/",
    "schema": "http://schema.org/",
    "friendManager": { "@path": "ex:knows/ex:reportsTo" }
  },
  "select": ["?managerName"],
  "where": [
    { "@id": "ex:alice", "friendManager": "?manager" },
    { "@id": "?manager", "schema:name": "?managerName" }
  ]
}
```

### Multi-relationship traversal

Follow any of several relationship types:

```sparql
PREFIX ex: <http://example.org/>
PREFIX schema: <http://schema.org/>

# People connected by friendship OR professional relationship
SELECT ?name WHERE {
  ex:alice (ex:knows|ex:colleague)+ ?person .
  ?person schema:name ?name .
}
```

### Self-inclusive traversal (zero or more)

Use `*` to include the starting node:

```sparql
PREFIX ex: <http://example.org/>
PREFIX schema: <http://schema.org/>

# Alice and everyone she knows (transitively)
SELECT ?name WHERE {
  ex:alice ex:knows* ?person .
  ?person schema:name ?name .
}
```

With `*`, Alice herself is included in results (zero hops). With `+`, only her connections are returned.

### Inverse relationships

Find who links to a given entity:

```sparql
PREFIX ex: <http://example.org/>
PREFIX schema: <http://schema.org/>

# Who has Alice as a friend?
SELECT ?name WHERE {
  ?person ex:knows ex:alice .
  ?person schema:name ?name .
}

# Same thing using inverse path syntax
SELECT ?name WHERE {
  ex:alice ^ex:knows ?person .
  ?person schema:name ?name .
}
```

Inverse paths are especially useful in transitive queries:

```sparql
# All ancestors in a taxonomy
SELECT ?ancestor WHERE {
  ex:goldRetriever rdfs:subClassOf+ ?ancestor .
}

# All descendants (inverse)
SELECT ?descendant WHERE {
  ex:animal ^rdfs:subClassOf+ ?descendant .
}
```

## Performance considerations

### Property path cost

| Operator | Cost | Notes |
|---|---|---|
| Simple predicate | O(log n) | Single index lookup |
| Sequence (`/`) | O(k * log n) | k joins, each indexed |
| One-or-more (`+`) | O(reachable * log n) | Breadth-first expansion |
| Zero-or-more (`*`) | O(reachable * log n) | Same as `+` plus start node |
| Alternative (`\|`) | O(sum of alternatives) | Each alternative evaluated |
| Inverse (`^`) | O(log n) | Uses OPST index |

Transitive operators (`+`, `*`) expand breadth-first and track visited nodes to detect cycles. The cost is proportional to the number of reachable nodes, not the total graph size.

### Optimizing traversals

1. **Start from the specific side** — If you know one endpoint, start there. `ex:alice ex:knows+ ?person` is faster than `?person ex:knows+ ex:alice` because it anchors the traversal.

2. **Add filters after traversal** — Filter the results of a traversal rather than trying to filter during:
   ```sparql
   SELECT ?name WHERE {
     ex:alice ex:knows+ ?person .
     ?person schema:name ?name .
     ?person ex:department "Engineering" .
   }
   ```

3. **Use `+` over `*` when possible** — `*` includes the start node and typically has one more step to evaluate.

4. **Prefer sequence over transitive for known depth** — If you know the relationship is exactly 2 hops, use a sequence (`ex:a/ex:b`) or two explicit patterns instead of `ex:a+`.

5. **Combine with LIMIT** — For exploration, limit results to avoid materializing the full reachable set:
   ```sparql
   SELECT ?person WHERE {
     ex:alice ex:knows+ ?person .
   } LIMIT 100
   ```

### Cycle handling

Fluree's property path engine tracks visited nodes during transitive expansion. If a cycle is encountered (e.g., A knows B knows C knows A), the traversal stops at the already-visited node. This prevents infinite loops without requiring user intervention.

## Property paths vs. explicit patterns

For fixed-depth queries, explicit patterns are equivalent and sometimes clearer:

**Property path (2 hops):**
```sparql
SELECT ?fof WHERE {
  ex:alice ex:knows/ex:knows ?fof .
}
```

**Explicit patterns (2 hops):**
```sparql
SELECT ?fof WHERE {
  ex:alice ex:knows ?friend .
  ?friend ex:knows ?fof .
}
```

Both produce the same results. Use property paths when:
- The depth is variable or unknown (transitive closure)
- You want compact syntax for chains
- You need alternative or inverse traversal

Use explicit patterns when:
- The depth is fixed and small
- You need to bind intermediate variables (e.g., `?friend` above)
- You want maximum clarity

## Related documentation

- [JSON-LD Query — Property Paths](jsonld-query.md#property-paths) — Full JSON-LD property path syntax
- [SPARQL — Property Paths](sparql.md) — SPARQL property path reference
- [Datasets](datasets.md) — Multi-graph traversal
- [Explain Plans](explain.md) — Understand query execution


---

# query/explain.md

# Explain Plans

Explain plans provide insight into how the query planner reorders WHERE-clause
patterns, helping you understand optimization decisions and diagnose
performance issues.

## Overview

Explain plans show:
- Whether patterns were reordered and why
- Whether database statistics were available for optimization
- The cardinality category and cost estimate assigned to each pattern
- The original vs. optimized pattern order
- Execution strategy hints for special fast paths such as fused property-join stars

## Requesting Explain Plans

### JSON-LD Query

Use the `/fluree/explain` endpoint (or the CLI `fluree query --explain ...`) to get a plan without executing.
For JSON-LD, the explain request body is the same as a normal JSON-LD query body.

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "select": ["?name"],
  "where": [
    { "@id": "?person", "ex:name": "?name" }
  ],
  "from": "mydb:main"
}
```

### SPARQL

Use the explain endpoint with SPARQL content type:

```sparql
PREFIX ex: <http://example.org/ns/>

SELECT ?name
WHERE {
  ?person ex:name ?name .
}
```

## How the Query Planner Works

The query planner reorders WHERE-clause patterns to minimize the number of
intermediate rows flowing through the execution pipeline. It uses a greedy
algorithm that places patterns one at a time, choosing the cheapest eligible
pattern at each step.

### Pattern Categories

Every pattern is classified into one of four cardinality categories:

| Category     | Meaning                                | Patterns                                                                          |
| ------------ | -------------------------------------- | --------------------------------------------------------------------------------- |
| **Source**   | Produces rows (estimated row count)    | Triple, VALUES, UNION, Subquery, IndexSearch, VectorSearch, GeoSearch, S2Search, Graph, PropertyPath, R2rml, Service |
| **Reducer**  | Shrinks the stream (multiplier < 1.0)  | MINUS, EXISTS, NOT EXISTS                                                         |
| **Expander** | Grows the stream (multiplier >= 1.0)   | OPTIONAL                                                                          |
| **Deferred** | No cardinality effect                  | FILTER, BIND                                                                      |

### Placement Priority

The greedy loop places patterns in this priority order:

1. **Eligible reducers** (lowest multiplier first) — shrink the stream as
   early as possible.
2. **Sources** (lowest row count first, preferring patterns that join on
   already-bound variables) — most selective first.
3. **Eligible expanders** (lowest multiplier first) — defer row expansion
   until prerequisite variables are bound.

A reducer or expander is "eligible" when at least one of its variables is
already bound by a previously placed pattern.

FILTER and BIND patterns are integrated into the greedy loop: after each
source, reducer, or expander is placed, any deferred patterns whose input
variables are now satisfied are drained in original-position order. For BIND
patterns, only the expression's input variables must be bound — the target
variable is an output that feeds back into `bound_vars`, potentially enabling
further deferred patterns to be placed immediately (cascading placement).

### Compound Pattern Nesting

When a deferred pattern (FILTER or BIND) becomes ready and the last placed
pattern is a compound pattern (UNION, Graph, or Service), the planner nests
the deferred pattern *into* the compound pattern's inner lists instead of
appending it after. This enables the deferred pattern to participate in the
compound pattern's inner `reorder_patterns` pipeline, unlocking:

- Optimal placement after the specific triple that binds its variable
- Range-safe filter pushdown to index scans
- Inline evaluation during joins

Nesting occurs only when the compound pattern **guarantees** the deferred
pattern's required variable is bound:

| Compound     | Nest? | Guarantee                                              |
| ------------ | ----- | ------------------------------------------------------ |
| **UNION**    | Yes   | Variable must appear in the intersection of all branches |
| **Graph**    | Yes   | Variable is in inner patterns or is the graph name variable |
| **Service**  | Yes   | Variable is in inner patterns or is the endpoint variable |
| OPTIONAL     | No    | Left-join: inner vars may be Unbound                   |
| MINUS        | No    | Anti-join: inner vars not exported to outer scope      |
| EXISTS       | No    | Filter-only: inner vars not exported                   |
| NOT EXISTS   | No    | Filter-only: inner vars not exported                   |

For UNION, the deferred pattern is cloned into every branch. For Graph and
Service, it is appended to the inner pattern list. Recursion is handled
naturally: when a nested filter lands inside a branch containing another
compound pattern, the branch's `reorder_patterns` call applies the same logic.

### Bound-Variable-Aware Estimation

The planner tracks which variables become bound as each pattern is placed.
This significantly affects estimates for subsequent patterns:

- A triple `?s :name ?name` with `?s` **unbound** is a property scan —
  estimated at the full property count (or a 1000-row fallback).
- The same triple with `?s` **already bound** from an earlier pattern is a
  per-subject lookup — estimated at `count / ndv_subjects` (typically ~10
  rows).

This context-aware scoring also applies inside compound patterns: UNION
branches and subqueries receive database statistics and use the same
selectivity model for their inner patterns.

### Statistics-Based vs. Fallback Scoring

When a `StatsView` is available (after at least one indexing cycle), the
planner uses HLL-derived property statistics:

- **count**: total number of triples for this predicate
- **ndv_subjects**: number of distinct subjects
- **ndv_values**: number of distinct objects

Without statistics, the planner falls back to heuristic constants:

| Pattern Type   | Fallback Estimate |
| -------------- | ----------------: |
| ExactMatch     |                 1 |
| BoundSubject   |                10 |
| BoundObject    |             1,000 |
| PropertyScan   |             1,000 |
| FullScan       |            1e12   |

### Search and Graph Source Estimates

Search patterns (IndexSearch, VectorSearch, GeoSearch, S2Search) use their
`limit` field when present. Without an explicit limit, the planner assumes a
default of 100 rows. Graph patterns recursively estimate their inner
patterns. Service patterns use a very high estimate (1e12) so they are placed
last among sources, minimizing data sent to the remote endpoint.

## Reading Explain Output

Explain returns a JSON object `{ "query": <echo>, "plan": { ... } }`. The
`plan` object contains:

| Field                  | Meaning |
| ---------------------- | ------- |
| `optimization`         | `"reordered"`, `"unchanged"`, or `"none"` (no statistics) |
| `statistics-available` | whether HLL statistics were available for cost estimation |
| `statistics`           | summary stats (e.g. `total-flakes`) |
| `logical`              | the **compound-aware** join order the planner produces (see below) |
| `physical`             | the **planned physical operator tree** the executor will build (see below) |
| `original`             | the query's triple patterns in original order, with selectivity inputs |
| `optimized`            | the same triples in the planner's chosen order |
| `execution-hints`      | specialized execution strategies the executor will use (see [Execution Hints](#execution-hints)) |

The `original` and `optimized` arrays are a flattened, triple-level view. The
`optimized` order is produced by the **same** `reorder_patterns` routine the
executor uses, so the order shown is the order that runs — there is no separate
explain-only ordering algorithm.

### The `logical` plan

`logical` is the recommended view. Unlike `original`/`optimized` (which flatten
all triples into one list), it preserves compound structure — `OPTIONAL`,
`UNION`, `MINUS`, `EXISTS`, subqueries — and shows each node in the planner's
chosen execution order. It is present even when statistics are unavailable
(the planner falls back to heuristic estimates). Each node carries:

- `kind`: `triple`, `optional`, `union`, `minus`, `exists`, `not-exists`,
  `subquery`, `filter`, `bind`, `values`, `property-path`, `graph`, `service`,
  or a search kind.
- `category`: `source` (produces rows), `reducer` (shrinks), `expander`
  (grows — e.g. `OPTIONAL`), or `deferred` (`FILTER`/`BIND`).
- `estimate`: `{ "row-count": N }` for sources, `{ "multiplier": M }` for
  reducers/expanders.
- For triples, a `pattern` object (`subject`/`property`/`object`); for compound
  nodes, a `patterns` array (or `branches` for `UNION`) of child nodes.

Example (`?person :name ?name . OPTIONAL { ?person :email ?email }`):

```jsonc
"logical": [
  { "kind": "triple",   "category": "source",
    "estimate": { "row-count": 50 },
    "pattern": { "subject": "?person", "property": "ex:name", "object": "?name" } },
  { "kind": "optional", "category": "expander",
    "estimate": { "multiplier": 1.0 },
    "patterns": [
      { "kind": "triple", "category": "source",
        "estimate": { "row-count": 50 },
        "pattern": { "subject": "?person", "property": "ex:email", "object": "?email" } }
    ] }
]
```

Key things to look for:

- **Source `row-count`**: Lower values are placed first. A high row count early
  in the plan may indicate missing statistics or an inherently broad pattern.
- **Reducer `multiplier`**: Values below 1.0 indicate the fraction of rows that
  survive. A MINUS with multiplier 0.90 removes ~10% of rows.
- **Deferred placement**: FILTERs and BINDs appear immediately after all of
  their input variables become bound. BIND outputs cascade — a BIND placed
  early can enable subsequent FILTERs or BINDs that depend on its target
  variable. If a FILTER appears late, check whether its variables could be
  bound sooner.
- **`optimization: "none"`**: No statistics were available, so cost-based
  reordering is skipped for the `original`/`optimized` arrays. The `logical`
  view still shows the planner's heuristic order. Run at least one indexing
  cycle to enable statistics-based optimization.

### The `physical` plan

`physical` is the operator tree the executor will actually build — the
"join plan". It is produced by building the **real** operator tree (the same
`build_operator_tree` execution uses) and walking it; the query is **not
executed** (no scans, no joins run). Because the fast-path / count-planner /
fold selection happens at build time, `physical` shows what `logical` cannot:
the chosen physical operators.

Each node has:

- `op`: the operator (e.g. `ProjectOperator`, `HashJoinOperator`,
  `PropertyJoinOperator`, `CyclicBgpOperator`, `NestedLoopJoinOperator`,
  `DatasetOperator`, a count or other fast-path operator).
- `est-rows`: build-time cardinality estimate, when the operator exposes one.
- `details`: operator-specific attributes (e.g. a scan's `pattern` and planned
  `index-hint`; a `PropertyJoinOperator`'s fused `predicates`).
- `children`: child edges, each `{ "rel": ..., "node": { ... } }`.

For an object→subject join, the node carries the planner's hash-join decision,
so you can see **whether** a hash join was chosen and **why**:

- `hash-join-chosen`: `true` on a `HashJoinOperator`, `false` on a
  `NestedLoopJoinOperator` that was a hash-join candidate but lost.
- `hash-join-reason`: `forced-on` / `forced-off` (the `FLUREE_HASH_JOIN` env),
  `cost-wins`, `probe-too-small`, `scan-ratio-too-high`, `no-probe-stats`, or
  `subject-driven-forward-join`.
- `probe-count`, `driving-est`, `scan-ratio`: the cost inputs the planner
  weighed (present when statistics are available).

For eligible fixed-predicate triangle/square BGPs, the physical plan may show
`CyclicBgpOperator`. This is a conservative fast path for cyclic joins that
would otherwise be planned as left-deep nested-loop joins. Its `details` include:

- `strategy`: `cyclic_bgp_join`
- `shape`: `triangle` or `square`
- `predicates`: predicates in the detected cyclic block
- `predicate-row-estimates`: per-predicate row-count estimates, aligned with
  `predicates`; `null` when statistics are unavailable
- `enabled`: whether `FLUREE_CYCLIC_BGP` allows the fast path
- `max-predicate-rows`: the per-predicate row cap used by the fast path
- `object-only-values`: `iri-ref` for the raw SID-only cycle path, or `encoded`
  when object-only cycle variables can join on late-materialized encoded object
  values, not just IRI references
- `pruning`: `semi_join` when the runtime fast path prunes each edge to values
  supported by the other edge incident to the same cycle variable
- `driver-selection`: how the runtime driver edge is selected after pruning
- `square-strategy`: `wedge_pair_hash` when an encoded square uses exact wedge
  pair sizing, hashes the smaller opposite-vertex wedge, and streams the other
  wedge by center
- `square-build-pairs` / `square-probe-pairs`: exact wedge pair counts for the
  chosen square decomposition, present after the fast path has opened
- `square-wedge-pair-cap`: the hard cap for materializing the build wedge
- `bounded-probe-strategy`: `cascading_subject_probe` when at least one edge
  was loaded by per-subject PSOT probes instead of a full predicate scan. The
  cascade scans the cheapest edge first, then repeatedly probes any remaining
  edge whose subject variable is bounded to a small frontier by the relations
  already loaded (each new relation tightens the frontiers), falling back to a
  full scan per edge when the probe-vs-scan gate fails
- `bounded-probe-edges`: how many edges were loaded by probing
- `bounded-probe-subject-cap`: the maximum frontier size eligible for probing
- `bounded-probe-scan-ratio`: minimum estimated scan rows per probe required
  to probe instead of scanning
- `raw-relation-rows` / `pruned-relation-rows`: runtime row totals, present when
  a plan is described after the fast path has opened

Set `FLUREE_CYCLIC_BGP=0` (or `false`) to disable this operator for A/B
testing. `FLUREE_CYCLIC_BGP_MAX_ROWS` can lower or raise the per-predicate row
cap. `FLUREE_CYCLIC_BGP_MAX_WEDGE_PAIRS` controls the encoded-square build-side
wedge cap (default 5,000,000 pairs); when both exact wedge decompositions exceed
the cap, the operator falls back to the regular cyclic enumerator.
`FLUREE_CYCLIC_BGP_MAX_BOUNDED_SUBJECTS` controls the per-edge frontier cap for
bounded subject probing (default 65,536 subjects), and
`FLUREE_CYCLIC_BGP_PROBE_SCAN_RATIO` controls the probe-vs-scan gate (default
64: probe only when the edge's estimated row count is at least 64× the frontier
size). Probing requires HEAD execution (no overlay, `to_t == max_t`); otherwise
every edge is loaded by a full overlay-correct scan as before. Setting
`FLUREE_CYCLIC_BGP_MAX_BOUNDED_SUBJECTS=0` disables probing without disabling
the operator. When statistics prove no probe can ever pass the subject cap
(every edge's subject variable is only exposed by edges whose distinct counts
exceed the cap with margin), the edge scans and relation-index builds run
concurrently on the shared worker pool instead of the sequential cascade. The node exposes the old nested-loop
plan as a `fallback` child; the fallback runs when the runtime mode is
unsupported by the fast path.

The edge `rel` distinguishes a real input from an alternative:

| `rel`         | Meaning |
| ------------- | ------- |
| `child`       | a real input the operator consumes |
| `fallback`    | a correctness fallback run *instead* of the fast path when it bails at open (overlay/history/policy/multi-graph). Only one of the two executes. |
| `conditional` | a path chosen at runtime |

Example — a same-subject star collapses to a single fused operator:

```jsonc
"physical": {
  "op": "ProjectOperator",
  "children": [
    { "rel": "child", "node": { "op": "PropertyJoinOperator" } }
  ]
}
```

**Planned vs. actual.** `physical` is the *planned* tree. A few decisions are
finalized only at execution `open()` and are **not** reflected here: the
multi-graph hash-join → nested-loop downgrade, whether a fast path takes its
fallback, and the exact index permutation (SPOT/POST/OPST/PSOT) a scan chooses.
Surfacing those requires actually running the query (a future `EXPLAIN ANALYZE`
mode), which `EXPLAIN` deliberately does not do.

### Execution Hints

Explain responses may also include an `execution-hints` array. These are not
generic cardinality estimates; they describe when the executor expects to use a
specialized path after planning.

For the star-join work, look for:

- `property_join`: the planner chose the same-subject property-join path
- `property_join_fused_star`: the planner chose property join and also fused
  trailing same-subject single-triple `OPTIONAL`s plus eligible trailing
  `FILTER`/`BIND` patterns into the same star operator

Typical fields include:

- `required_triples`: number of required star predicates
- `fused_optional_triples`: number of fused trailing `OPTIONAL` triples
- `fused_filters`: number of trailing filters evaluated inside the star path
- `fused_binds`: number of trailing binds evaluated inside the star path
- `width_score`: weighted star width used by the property-join gate
- `optional_bonus`: how much of the width score came from trailing optionals

This is the clearest signal that a query like:

```sparql
?deal a crm:Deal ;
      crm:name ?name ;
      crm:amount ?amount ;
      crm:stage ?stage .
OPTIONAL { ?deal crm:probability ?probability }
OPTIONAL { ?deal crm:closedAt ?closedAt }
FILTER (!STRSTARTS(STR(?stage), "Closed"))
```

is using the fused two-pass star path rather than falling back to separate
OPTIONAL and FILTER operators.

## Indexes

Scan operations use one of four index permutations depending on which
components of the triple pattern are bound:

- **SPOT**: Subject-Predicate-Object-Time — used when the subject is bound
- **POST**: Predicate-Object-Subject-Time — used for predicate+object lookups
- **OPST**: Object-Predicate-Subject-Time — used for object-based lookups
- **PSOT**: Predicate-Subject-Object-Time — used for full predicate scans

## Filter Optimization

Filters are automatically optimized by the query engine in three ways:

- **Dependency-based placement**: Filters and BINDs are placed as soon as all
  their input variables are bound, as part of the greedy reordering loop.
  BIND target variables feed back into the bound set, enabling cascading
  placement of dependent patterns.
- **Index pushdown**: Range-safe filters (comparisons like `>`, `<`, `>=`,
  `<=` on indexed properties) are pushed down to the index scan, reducing the
  number of rows read.
- **Inline evaluation**: Filters whose variables are all bound by a join are
  evaluated inside the join operator itself, avoiding the overhead of a
  separate filter pass.
- **BIND filter fusion**: When a FILTER's last required variable is the output
  of a BIND, the filter is fused into the BindOperator and evaluated inline
  after computing each row's BIND value. Failing rows are dropped before
  materialization, eliminating a separate FilterOperator pass.

## Best Practices

1. **Review plans for new queries**: Use explain to verify that the planner
   chose a reasonable order, especially for queries with many patterns.
2. **Ensure statistics are available**: Statistics enable much better estimates.
   If explain shows "Statistics available: no", check that at least one
   indexing cycle has completed.
3. **Check for high row counts early in the plan**: A source with a very high
   row count placed first can indicate a missing join variable or an overly
   broad pattern.
4. **Use LIMIT on search patterns**: IndexSearch, VectorSearch, GeoSearch, and
   S2Search patterns use their `limit` field for cost estimation. Providing an
   explicit limit helps the planner place them more accurately.

## Related Documentation

- [JSON-LD Query](jsonld-query.md): JSON-LD Query syntax
- [SPARQL](sparql.md): SPARQL syntax
- [Indexing and Search](../indexing-and-search/README.md): Index details
- [Debugging Queries](../troubleshooting/debugging-queries.md): Troubleshooting guide


---

# query/tracking-and-fuel.md

# Tracking and Fuel Limits

Fluree provides query tracking and fuel limits to monitor and control query execution, ensuring system stability and performance.

## Query Tracking

Query tracking provides visibility into query execution, helping you understand query behavior and performance.

### Enable Tracking

Enable tracking via the `opts` object. Use `"meta": true` to enable all tracking, or selectively enable specific metrics:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "select": ["?name"],
  "where": [
    { "@id": "?person", "ex:name": "?name" }
  ],
  "opts": { "meta": true }
}
```

Or enable specific metrics:

```json
{
  "opts": {
    "meta": {
      "time": true,
      "fuel": true,
      "policy": true
    }
  }
}
```

### Tracked Information

Tracking provides:
- **time**: Query execution duration (formatted as "12.34ms")
- **fuel**: Total cost as a decimal value (rounded to 3 places)
- **policy**: Policy evaluation statistics (`{policy-id: {executed: N, allowed: M}}`)

## Fuel Limits

Fuel limits control resource consumption, preventing runaway queries from consuming excessive resources.

### What Is Fuel?

Fuel is a decimal measure of query/transaction cost. Internally it is accumulated as **micro-fuel** (1 fuel = 1000 micro-fuel) and reported back rounded to 3 decimal places. Costs reflect actual work — primarily I/O — rather than output cardinality.

Cost ladder (per event):

| Event | Cost (fuel) |
|---|---|
| Query floor (once per query, charged at entry before parsing) | 1.000 |
| Index leaflet touched (per scan batch, regardless of cache state) | 0.010 |
| Forward-dict touch (per dict-backed value resolved during result materialization) | 0.010 |
| History-scan leaflet base (per leaflet; per-row costs below add on top) | 0.010 |
| Flake returned from a `db.range` call (e.g. SHACL graph reads, graph crawl) | 0.001 |
| Overlay/novelty row materialized | 0.001 |
| History row scanned (base + in-range sidecar rows) | 0.001 |
| R2RML row emitted (Iceberg/Parquet) | 0.001 |
| Transaction commit baseline (once per commit, including each bulk-import chunk) | 10.000 |
| Staged flake (per flake in a transaction or bulk-import chunk) | 0.001 |
| Indexer CAS write (per successful `ContentStore::put` / `put_with_id` made by an index build) | 1.000 |
| Re-encoded leaflet inside an FLI3 leaf write (passthrough leaflets are not charged) | 1.000 |
| `REGEX` / `REPLACE` evaluation | 0.001 |
| Hash function (`MD5`, `SHA1`, `SHA256`, `SHA384`, `SHA512`) | 0.001 |
| `UUID` / `STRUUID` | 0.001 |
| `geof:distance` | 0.001 |
| Vector similarity (`DotProduct`, `CosineSimilarity`, `EuclideanDistance`) | 0.002 |
| `Fulltext` (per-row BM25 scoring) | 0.005 |

Cheap operations (comparisons, arithmetic, type checks, simple string ops, datetime extraction, etc.) cost zero — instrumentation overhead would dwarf the actual cost.

The **query floor** guarantees every fuel-tracked query reports at least `1.000` fuel: a query touching no persisted data still costs the floor, and a query that errors during parsing/planning still reports it. I/O "touches" cost `0.010` each, so a scan-dominated query reports roughly `1.000 + 0.010 × (leaflet/dict touches)`. The fuel schedule above is defined in one place — `fluree-db-core/src/tracking.rs` (`tracking::schedule`).

#### Graph-crawl projection: materialization fuel scales with selected predicates

When a JSON-LD query's projection lists explicit forward predicates (e.g. `select: {"?x": ["@id", "ex:a", "ex:b"]}`), the range provider receives a predicate allow-list via `RangeOptions::predicate_filter` and drops base flakes for non-listed predicates **before** decoding the object value, resolving the subject Sid, or charging dict-touch fuel.

What this changes:
- **Per-flake (`0.001`), subject-resolve, object-decode, and dict-touch (`0.010`) fuel** all scale with the number of selected predicates rather than the subject's total predicate count.

What this does **not** change:
- **Index leaflet touches (`0.010` each)** still scale with the cursor's scanned subject range. For K ≥ 2 the cursor opens `SPOT(s,*,*)` and pays one `INDEX_TOUCH` per leaflet batch returned, the same as a full crawl over the subject — the filter narrows which rows get *materialized*, not how many leaflets get *read*. For K = 1 the cursor opens `SPOT(s,p,*)` directly via `RangeMatch::subject_predicate`, which narrows the leaflet key-range to the `(s, p)` prefix — same `INDEX_TOUCH` count for small subjects (one batch either way), strictly fewer touches for subjects whose flakes span multiple batches.

Wildcard projections (`select: {"?x": ["*"]}`) take the unchanged decode-everything path — every flake on the subject is materialized and charged. Refinements on a wildcard level (e.g. `{"ex:friend": ["*"]}` inside a `["*", ...]` selection) keep the parent level wildcard; only the explicit-list form opts into the filter.

Explicit projections with no forward predicates (e.g. `["@id"]` or reverse-only) skip the forward SPOT scan entirely — no cursor, no `INDEX_TOUCH`. `@id` is derived from the subject Sid and reverse properties go through a dedicated POST scan.

The overlay path also honors the filter, so per-subject hydration over a ledger with uncommitted novelty pays only for retracts / asserts on predicates the projection cares about. Novelty-only predicates (no persisted p_id yet) are honored too: the row-loop allow-set is extended with ephemeral p_ids the overlay translator assigned for any selected predicate Sid.

### Indexing Fuel

Indexer CAS writes are billed through a `MeteredContentStore` wrapper that the build entry points install around the caller-supplied content store. Every successful `ContentStore::put` / `put_with_id` charges the base **1.000 fuel** rate — including index leaves, branch manifests, root manifests, dict packs / reverse-tree nodes, history sidecars, garbage records, stats sketches, and (incremental) spatial / fulltext arenas. The lower-level `Storage::content_write_bytes` is **not** currently wrapped; the only indexer code path that uses it is the dead-code spatial rebuild helper. If that path is wired up, add a storage-level wrapper first (or migrate it to `ContentStore::put`).

FLI3 leaf writes carry an additional per-leaflet charge of **1.000 fuel per re-encoded leaflet**. Passthrough leaflets (byte-copies carried forward from a prior leaf during an incremental update) are **not** charged because no zstd encoding work was performed — so a 100-leaflet leaf where only two leaflets were touched by novelty bills `1 + 2 = 3` fuel, not `1 + 100`. The two FLI3 leaf upload sites (`build::upload::upload_indexes_to_cas` for full rebuild, `build::incremental::upload_leaf_blobs` for incremental) compute the count from `LeafInfo::re_encoded_leaflet_count`.

Indexing fuel is **measurement only**: indexer trackers are no-limit. A partial index is worse than a slow one, so the indexer never aborts mid-build on a fuel limit. The plain public entry points (`build_index_for_record`, `rebuild_index_from_commits`) pass a disabled tracker and report `fuel: None`. The `*_with_tracker` variants wrap the store, propagate a fuel-enabled tracker, and stamp the final tally on `IndexResult::fuel` — `Some(0.0)` for an already-current build, `Some(N)` for one that did real work.

Where fuel surfaces depends on who initiated the build:

- **`/reindex` (standalone, user-triggered):** the API creates a per-request fuel tracker, returns `ReindexResponse.fuel`. CLI prints `Fuel: N.NNN`.
- **`trigger_index` (standalone, waits for background completion):** the orchestrator creates a per-build tracker; `TriggerIndexResult.fuel` carries the result. **Coalesced trigger callers** all receive the fuel of the single build that satisfied them.
- **Background indexer (no caller waiting):** the orchestrator still creates a per-build tracker and logs the tally on the completion `info!` line (`fuel = ...`), but does not return it anywhere.
- **Combined transactor + post-commit indexing (`maybe_refresh_after_commit` / `require_refresh_before_commit`):** measured with a per-build tracker and logged on the completion line; intentionally **not** attributed to the transaction's tracking response.

`IndexOutcome::Completed` carries `fuel: Option<f64>` so `wait()` callers and waiter handles see the same value. Already-satisfied early-returns (waiter satisfied by a previously-published index) report `Some(0.0)` so callers can distinguish "no work" from "not tracked".

### Setting Fuel Limits

Set fuel limits via `opts.max-fuel` (decimal allowed). Setting a fuel limit implicitly enables fuel tracking:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "select": ["?name"],
  "where": [
    { "@id": "?person", "ex:name": "?name" }
  ],
  "opts": { "max-fuel": 10000 }
}
```

You can also use `"maxFuel"` or `"max_fuel"` as alternative key names. The HTTP equivalent is the `fluree-max-fuel` header.

Because the `1.000` query floor is charged before execution and counts toward the limit, `max-fuel` must leave room for it:

- `max-fuel: 1` permits exactly the floor — a query that needs even one persisted touch is rejected.
- `max-fuel: 1.01` permits the floor plus one persisted touch.
- `max-fuel` below `1.0` (e.g. `0.5`) is rejected up front, before parsing.

### Fuel Limit Behavior

When fuel limit is exceeded:
- Query execution stops
- Error returned to client
- Partial results not returned

## Response Format

When tracking is enabled, the response includes tracking information as top-level siblings:

```json
{
  "status": 200,
  "result": [...],
  "time": "12.34ms",
  "fuel": 42.317,
  "policy": {
    "http://example.org/myPolicy": {
      "executed": 10,
      "allowed": 8
    }
  }
}
```

The `fuel` value is decimal with up to 3 places of precision. The HTTP `x-fdb-fuel` response header carries the same value.

When a reasoning mode ran (e.g. `"reasoning": "owl2rl"`), tracked responses
also include a top-level `reasoning` block describing the OWL2-RL
materialization — most importantly whether it was **capped** (hit its budget
before reaching fixpoint, meaning results may be missing entailments):

```json
{
  "status": 200,
  "result": [...],
  "reasoning": {
    "capped": false,
    "derived_facts": 47213,
    "iterations": 4,
    "duration_ms": 380
  }
}
```

The same JSON rides the `x-fdb-reasoning` response header. See
[Reasoning — materialization budget](reasoning.md#materialization-budget)
for budget configuration.

Tracked transaction responses (`/insert`, `/upsert`, `/update`, including Turtle/TriG and SPARQL UPDATE when tracking headers are used) expose the same top-level `time`, `fuel`, and `policy` fields when present, alongside the transaction receipt fields.

## Best Practices

### Tracking

1. **Enable for Debugging**: Use `"opts": {"meta": true}` to debug slow queries
2. **Monitor Performance**: Track query performance over time
3. **Identify Bottlenecks**: Use tracking to identify performance bottlenecks

### Fuel Limits

1. **Set Appropriate Limits**: Set fuel limits based on expected query complexity
2. **Monitor Fuel Usage**: Track fuel usage to optimize queries
3. **Prevent Runaway Queries**: Use fuel limits to prevent resource exhaustion

## Related Documentation

- [JSON-LD Query](jsonld-query.md): JSON-LD Query syntax
- [SPARQL](sparql.md): SPARQL syntax
- [Explain Plans](explain.md): Query execution plans


---

# query/reasoning.md

# Query-Time Reasoning

This page covers how to enable and use reasoning in your queries. For
background concepts see [Reasoning and inference](../concepts/reasoning.md); for
the full list of supported OWL/RDFS constructs see the
[OWL & RDFS reference](../reference/owl-rdfs-support.md).

## The `reasoning` parameter

Add a `"reasoning"` key to any JSON-LD query to control which inference modes
are active:

### Single mode

```json
{
  "@context": {"ex": "http://example.org/"},
  "select": ["?s"],
  "where": {"@id": "?s", "@type": "ex:Person"},
  "reasoning": "rdfs"
}
```

### Multiple modes

```json
{
  "select": ["?s"],
  "where": {"@id": "?s", "@type": "ex:Person"},
  "reasoning": ["rdfs", "owl2rl"]
}
```

### Disable reasoning

```json
{
  "select": ["?s"],
  "where": {"@id": "?s", "@type": "ex:Person"},
  "reasoning": "none"
}
```

Use `"none"` to override any view- or ledger-wide reasoning defaults for
this query (with no defaults configured, it is a no-op affirmation).

### Valid mode strings

| String | Aliases | Mode |
|--------|---------|------|
| `"rdfs"` | — | RDFS subclass/subproperty expansion |
| `"owl2ql"` | `"owl-ql"`, `"owlql"` | OWL 2 QL query rewriting (includes RDFS) |
| `"owl2rl"` | `"owl-rl"`, `"owlrl"` | OWL 2 RL forward-chaining materialization |
| `"datalog"` | — | Datalog rule execution |
| `"none"` | — | Disable all reasoning |

## Default behavior

Reasoning is **opt-in**. When the `reasoning` key is absent from a query, no
reasoning runs — even if the data contains `rdfs:subClassOf` /
`rdfs:subPropertyOf` hierarchies. Plain queries match asserted triples only,
pay no reasoning-prep cost, and behave like other SPARQL engines under simple
entailment.

To enable reasoning without setting it per query, configure a default at the
view level (`GraphDb::with_reasoning(...)` in the Rust API) or in the ledger
configuration graph (`reasoningModes`). To override such a default for a
single query, use `"reasoning": "none"`.

> **Behavior change**: versions prior to this release auto-enabled RDFS when a
> schema hierarchy existed in the data (though for imported ledgers the
> auto-detection often silently failed). If you relied on automatic subclass /
> subproperty expansion, add `"reasoning": "rdfs"` to your queries or set a
> ledger default.

## Examples

The examples below assume this schema and data have been transacted:

```json
{
  "@context": {
    "ex": "http://example.org/",
    "rdfs": "http://www.w3.org/2000/01/rdf-schema#",
    "owl": "http://www.w3.org/2002/07/owl#"
  },
  "insert": [
    {"@id": "ex:Student", "rdfs:subClassOf": {"@id": "ex:Person"}},
    {"@id": "ex:GradStudent", "rdfs:subClassOf": {"@id": "ex:Student"}},
    {"@id": "ex:alice", "@type": "ex:GradStudent", "ex:name": "Alice"},
    {"@id": "ex:bob", "@type": "ex:Person", "ex:name": "Bob"},

    {"@id": "ex:livesWith", "@type": "owl:SymmetricProperty"},
    {"@id": "ex:alice", "ex:livesWith": {"@id": "ex:bob"}},

    {"@id": "ex:hasAncestor", "@type": "owl:TransitiveProperty"},
    {"@id": "ex:carol", "ex:hasAncestor": {"@id": "ex:dave"}},
    {"@id": "ex:dave", "ex:hasAncestor": {"@id": "ex:eve"}},

    {"@id": "ex:hasMother", "owl:inverseOf": {"@id": "ex:childOf"}},
    {"@id": "ex:frank", "ex:hasMother": {"@id": "ex:grace"}}
  ]
}
```

### RDFS: subclass expansion

Query for all `ex:Person` instances — Alice is returned even though she was
only typed as `ex:GradStudent`:

```json
{
  "@context": {"ex": "http://example.org/"},
  "select": ["?name"],
  "where": {
    "@id": "?s", "@type": "ex:Person",
    "ex:name": "?name"
  },
  "reasoning": "rdfs"
}
```

**Result:** `["Alice", "Bob"]`

Without reasoning (or with `"reasoning": "none"`), only `"Bob"` is returned
because Alice's explicit type is `GradStudent`, not `Person`.

### OWL 2 RL: symmetric properties

Query who lives with Bob — Alice is inferred even though only
`alice livesWith bob` was asserted:

```json
{
  "@context": {"ex": "http://example.org/"},
  "select": ["?who"],
  "where": {"@id": "ex:bob", "ex:livesWith": "?who"},
  "reasoning": "owl2rl"
}
```

**Result:** `["ex:alice"]`

### OWL 2 RL: transitive properties

Query for all ancestors of Carol — Eve is inferred through transitivity:

```json
{
  "@context": {"ex": "http://example.org/"},
  "select": ["?ancestor"],
  "where": {"@id": "ex:carol", "ex:hasAncestor": "?ancestor"},
  "reasoning": "owl2rl"
}
```

**Result:** `["ex:dave", "ex:eve"]`

### OWL 2 QL: inverse properties

Query `childOf` — inferred from the `hasMother` / `inverseOf` declaration:

```json
{
  "@context": {"ex": "http://example.org/"},
  "select": ["?child"],
  "where": {"@id": "ex:grace", "ex:childOf": "?child"},
  "reasoning": "owl2ql"
}
```

**Result:** `["ex:frank"]`

### OWL 2 RL: domain and range inference

If your schema declares `rdfs:domain` and `rdfs:range`:

```json
{
  "insert": [
    {"@id": "ex:teaches", "rdfs:domain": {"@id": "ex:Professor"},
                          "rdfs:range": {"@id": "ex:Course"}},
    {"@id": "ex:alice", "ex:teaches": {"@id": "ex:cs101"}}
  ]
}
```

Then with `"reasoning": "owl2rl"`:
- `ex:alice rdf:type ex:Professor` is inferred (from domain)
- `ex:cs101 rdf:type ex:Course` is inferred (from range)

### Combined modes

Enable RDFS + OWL 2 RL + Datalog together:

```json
{
  "select": ["?s"],
  "where": {"@id": "?s", "@type": "ex:Person"},
  "reasoning": ["rdfs", "owl2rl", "datalog"],
  "rules": [
    {
      "@context": {"ex": "http://example.org/"},
      "where": {"@id": "?p", "ex:parent": {"ex:parent": "?gp"}},
      "insert": {"@id": "?p", "ex:grandparent": {"@id": "?gp"}}
    }
  ]
}
```

OWL 2 RL facts are materialized first, then Datalog rules run over the
combined base + OWL data, and finally RDFS query rewriting is applied.

## SPARQL

In SPARQL queries, reasoning is controlled via the Fluree-specific
`PRAGMA reasoning` directive. Property paths (`+`, `*`, `^`) provide a
complementary mechanism for navigating transitive and inverse relationships
directly in the query pattern — see [SPARQL](sparql.md) for details.

## Inline ontology per query

In addition to ontology axioms stored in the ledger (via
`f:schemaSource`), a query can supply **inline ontology axioms**
via the top-level `ontology` field. The axioms are used only for
this query's reasoning pass and never persist.

```json
{
  "@context": {"ex": "http://example.org/ns/"},
  "select":    "?name",
  "where":     {"@id": "?p", "@type": "ex:Person", "ex:name": "?name"},
  "reasoning": "rdfs",
  "ontology": {
    "@context": {
      "ex":   "http://example.org/ns/",
      "rdfs": "http://www.w3.org/2000/01/rdf-schema#"
    },
    "@id":             "ex:Employee",
    "rdfs:subClassOf": {"@id": "ex:Person"}
  }
}
```

Semantics:

- **Additive, not replacing.** Inline axioms layer on top of
  whatever `f:schemaSource` configured for the ledger (same- or
  cross-ledger). Both contribute to the bundle the reasoner sees.
- **Transient.** Axioms never persist. The next query without
  `ontology` runs against only the configured bundle.
- **Reasoning mode still required.** Inline axioms don't enable
  reasoning on their own — set `reasoning` so the engine actually
  uses them.
- **Namespace-scoped.** IRIs the snapshot already knows reuse
  their codes; previously-unseen IRIs allocate request-scoped
  codes that are discarded with the response — the on-disk
  dictionary is untouched.
- **No audit trail.** Without persistence, "which axioms drove
  which result" can't be reconstructed from history. Store
  long-lived ontologies in a graph and reference via
  `f:schemaSource` if auditability matters.

Use cases that fit well: testing a candidate ontology before
committing it to the ledger, per-tenant axiom layers, exploratory
analytics with hypothetical sub-class chains.

## Interaction with ledger configuration

If `f:reasoningDefaults` is set in the ledger configuration graph (see
[Setting groups](../ledger-config/setting-groups.md)), those modes are the
baseline for every query. The per-query `reasoning` parameter can:

- **Add modes** — the query modes are merged with the defaults.
- **Disable all** — `"reasoning": "none"` overrides the defaults entirely.

The `f:overrideControl` setting on the ledger config determines whether
query-time overrides are allowed. See
[Override control](../ledger-config/override-control.md) for details.

## Materialization budget

OWL 2 RL materialization runs under a budget (default: 1,000,000 derived
facts / 30 seconds). When the closure exceeds the budget it is **capped**:
the query still answers, but over an incomplete closure — results may be
missing entailments. A capped run is therefore surfaced, not just logged:

- Tracked responses (`"opts": {"meta": true}` or the `fluree-track-*`
  headers) carry a top-level `reasoning` block:

  ```json
  {
    "status": 200,
    "result": [...],
    "reasoning": {
      "capped": true,
      "capped_reason": "facts",
      "derived_facts": 1000000,
      "iterations": 3,
      "duration_ms": 12450
    }
  }
  ```

- The same JSON rides the `x-fdb-reasoning` response header.
- The server logs a WARN per capped materialization.

The budget is configurable at three levels (highest precedence first):

1. **Per query** — JSON-LD `"reasoningBudget": {"maxFacts": 20000000,
   "maxSeconds": 300}`, or SPARQL `# PRAGMA reasoning-max-facts: 20000000` /
   `# PRAGMA reasoning-max-seconds: 300`. Subject to the ledger's
   `f:overrideControl` on `f:reasoningDefaults`.
2. **Per ledger** — `f:reasoningMaxFacts` / `f:reasoningMaxSeconds` in
   `f:reasoningDefaults` (see
   [Setting groups](../ledger-config/setting-groups.md)).
3. **Server-wide** — `FLUREE_REASONING_MAX_FACTS` /
   `FLUREE_REASONING_MAX_SECONDS` environment variables.

## Performance considerations

| Mode | Overhead | Caching |
|------|----------|---------|
| RDFS | Negligible — query rewriting only | N/A |
| OWL 2 QL | Negligible — query rewriting only | N/A |
| OWL 2 RL | First query materializes derived facts; subsequent queries use cache | LRU cache (16 entries), keyed on database state + reasoning modes |
| Datalog | Each unique rule set + database state combination is cached | Same LRU cache as OWL 2 RL |

**Tips:**
- Start with **RDFS** if you only need class/property hierarchies — it has
  virtually zero overhead.
- Use **OWL 2 QL** when you also need inverse properties and domain/range
  inference but want to stay in the query-rewriting approach.
- Use **OWL 2 RL** when you need the full rule set (transitive, symmetric,
  functional properties, `owl:sameAs`, restrictions, property chains).
- The materialization cache is invalidated when the underlying data changes
  (new transactions), so the first query after a write will re-materialize.

## Reasoning under access policy

Reasoning composes with view policy, but mind the contract when both are on:

- **OWL 2 QL** and **RDFS** rewrite the query and execute under your identity, so they are filtered like any normal query.
- **OWL 2 RL** and **datalog** materialize derived facts into the query overlay; those facts are filtered by the **same per-flake view policy as base data**. A derived flake you may not view is dropped.
- The engine filters a derived fact by its own `(subject, predicate, object)` — **not** by the base facts it was derived from. A rule or ontology axiom can therefore re-express hidden data under a viewable predicate (e.g. `ex:ssn rdfs:subPropertyOf ex:identifier`, or a rule deriving `ex:isHighEarner` from a hidden `ex:salary`) and the derived value will surface.

**If you run reasoning under a non-root policy, your policy must cover the derived properties and classes** — deny them, or use `default-allow: false` so anything not explicitly allowed (including reasoning-introduced predicates) stays hidden.

**Query-time rules are admin-only.** Under a non-root view policy, caller-supplied `rules` are stripped before execution — a restricted caller cannot inject inference rules (a rule with a viewable head could launder hidden data). Database-stored `f:rule` definitions and OWL/RDFS reasoning are administrator-controlled and still apply. See [Policy in queries → Reasoning](../security/policy-in-queries.md#reasoning-rdfs--owl--datalog).

## Related pages

| Topic | Page |
|-------|------|
| Conceptual introduction | [Reasoning and inference](../concepts/reasoning.md) |
| Custom inference rules | [Datalog rules](datalog-rules.md) |
| Supported OWL & RDFS constructs | [OWL & RDFS reference](../reference/owl-rdfs-support.md) |
| Ledger-wide reasoning config | [Setting groups](../ledger-config/setting-groups.md) |
| Reasoning under access policy | [Policy in queries](../security/policy-in-queries.md#reasoning-rdfs--owl--datalog) |


---

# query/datalog-rules.md

# Datalog Rules

Datalog rules let you define custom inference logic that goes beyond what
OWL and RDFS provide. Rules are expressed in a familiar JSON-LD pattern syntax
with `where` (conditions) and `insert` (conclusions) clauses, and execute in a
fixpoint loop that can chain rules together.

For background concepts see [Reasoning and inference](../concepts/reasoning.md);
for enabling reasoning in queries see
[Query-time reasoning](reasoning.md).

## Quick example

Infer a `grandparent` relationship from two `parent` hops:

```json
{
  "@context": {"ex": "http://example.org/"},
  "select": ["?gp"],
  "where": {"@id": "ex:alice", "ex:grandparent": "?gp"},
  "reasoning": "datalog",
  "rules": [
    {
      "@context": {"ex": "http://example.org/"},
      "where": {"@id": "?person", "ex:parent": {"ex:parent": "?gp"}},
      "insert": {"@id": "?person", "ex:grandparent": {"@id": "?gp"}}
    }
  ]
}
```

The rule says: *"For any `?person` whose `parent` has a `parent` `?gp`, insert
that `?person` has a `grandparent` `?gp`."* The query then finds Alice's
grandparents using the inferred facts.

## Rule format

Each rule is a JSON object with three parts:

| Key | Required | Description |
|-----|----------|-------------|
| `@context` | Yes | JSON-LD context for expanding compact IRIs |
| `where` | Yes | Pattern(s) that must match for the rule to fire |
| `insert` | Yes | Pattern(s) of new facts to derive when the rule fires |
| `@id` | No | Optional name/IRI for the rule (for documentation/debugging) |

### Where clause

The `where` clause defines the conditions under which the rule fires. It
follows the same pattern syntax as JSON-LD queries.

**Single pattern:**
```json
"where": {"@id": "?person", "ex:parent": "?parent"}
```

**Multiple patterns (implicit join on shared variables):**
```json
"where": [
  {"@id": "?person", "ex:parent": "?parent"},
  {"@id": "?parent", "ex:name": "?parentName"}
]
```

**Nested patterns (shorthand for multi-hop traversal):**
```json
"where": {"@id": "?person", "ex:parent": {"ex:parent": "?gp"}}
```
This is equivalent to two patterns joined on an intermediate variable.

**With filters:**
```json
"where": [
  {"@id": "?person", "ex:age": "?age"},
  ["filter", "(>= ?age 65)"]
]
```

### Insert clause

The `insert` clause defines what facts to produce for each set of matching
variable bindings.

```json
"insert": {"@id": "?person", "ex:grandparent": {"@id": "?gp"}}
```

- Variables (`?person`, `?gp`) are replaced with the bound values from `where`.
- Use `{"@id": "?var"}` for IRI/entity values; use `"?var"` directly for
  literal values.
- Multiple triples can be generated from a single insert pattern.

## Providing rules

Rules can be provided in two ways:

### 1. Query-time rules

Pass rules directly in the query via the `rules` array. This is the simplest
approach and doesn't require any prior setup:

```json
{
  "select": ["?result"],
  "where": {"@id": "?s", "ex:derived": "?result"},
  "reasoning": "datalog",
  "rules": [ ... ]
}
```

> **Note:** Providing a `rules` array automatically enables datalog reasoning —
> you don't strictly need `"reasoning": "datalog"`, though including it is
> recommended for clarity.

### 2. Database-stored rules

Rules can be stored in the database as `f:rule` assertions and referenced via
ledger configuration. This is useful for rules that should apply consistently
across all queries.

**Store a rule:**
```json
{
  "@context": {
    "f": "https://ns.flur.ee/db#",
    "ex": "http://example.org/"
  },
  "insert": {
    "@id": "ex:grandparentRule",
    "f:rule": {
      "@context": {"ex": "http://example.org/"},
      "where": {"@id": "?person", "ex:parent": {"ex:parent": "?gp"}},
      "insert": {"@id": "?person", "ex:grandparent": {"@id": "?gp"}}
    }
  }
}
```

**Configure the ledger to use stored rules:**
```json
{
  "insert": {
    "@id": "urn:fluree:mydb:main:config:ledger",
    "@type": "f:LedgerConfig",
    "f:datalogDefaults": {
      "f:datalogEnabled": true,
      "f:rulesSource": {
        "@type": "f:GraphRef",
        "f:graphSource": {"f:graphSelector": {"@id": "f:defaultGraph"}}
      },
      "f:allowQueryTimeRules": true
    }
  }
}
```

See [Setting groups — datalogDefaults](../ledger-config/setting-groups.md) for
full configuration options.

`f:rulesSource` also supports cross-ledger references — set
`f:ledger` on the inner `f:graphSource` to pull `f:rule` JSON
bodies from another ledger at query time. See
[Cross-ledger governance — Cross-ledger datalog rules](../security/cross-ledger-policy.md#cross-ledger-datalog-rules)
for the end-to-end pattern and failure modes.

When stored rules, cross-ledger rules, and query-time rules are
present, they are all **merged** and execute together in the
same fixpoint loop.

## Examples

### Sibling inference

Infer siblings from shared parents:

```json
{
  "@context": {"ex": "http://example.org/"},
  "select": ["?sibling"],
  "where": {"@id": "ex:alice", "ex:sibling": "?sibling"},
  "reasoning": "datalog",
  "rules": [
    {
      "@context": {"ex": "http://example.org/"},
      "where": [
        {"@id": "?person", "ex:parent": "?parent"},
        {"@id": "?sibling", "ex:parent": "?parent"}
      ],
      "insert": {"@id": "?person", "ex:sibling": {"@id": "?sibling"}}
    }
  ]
}
```

> **Note:** This rule will also infer that a person is their own sibling. You
> could add a filter `["filter", "(!= ?person ?sibling)"]` to exclude
> self-references.

### Chained rules (uncle + aunt)

Multiple rules that build on each other:

```json
{
  "@context": {"ex": "http://example.org/"},
  "select": ["?aunt"],
  "where": {"@id": "ex:alice", "ex:aunt": "?aunt"},
  "reasoning": "datalog",
  "rules": [
    {
      "@context": {"ex": "http://example.org/"},
      "where": {"@id": "?person", "ex:parent": {"ex:brother": "?uncle"}},
      "insert": {"@id": "?person", "ex:uncle": {"@id": "?uncle"}}
    },
    {
      "@context": {"ex": "http://example.org/"},
      "where": {
        "@id": "?person",
        "ex:uncle": {
          "ex:spouse": {"@id": "?aunt", "ex:gender": {"@id": "ex:Female"}}
        }
      },
      "insert": {"@id": "?person", "ex:aunt": {"@id": "?aunt"}}
    }
  ]
}
```

The second rule (aunt) depends on facts derived by the first rule (uncle). The
fixpoint loop handles this automatically — it keeps iterating until no new facts
are produced.

### Rules with filters

Classify people by age:

```json
{
  "@context": {"ex": "http://example.org/"},
  "select": ["?person"],
  "where": {"@id": "?person", "ex:status": "senior"},
  "reasoning": "datalog",
  "rules": [
    {
      "@context": {"ex": "http://example.org/"},
      "where": [
        {"@id": "?person", "ex:age": "?age"},
        ["filter", "(>= ?age 65)"]
      ],
      "insert": {"@id": "?person", "ex:status": "senior"}
    }
  ]
}
```

### Combining with OWL reasoning

Datalog rules can build on OWL-derived facts. For example, use OWL 2 RL to
materialize transitive and symmetric properties, then use Datalog for custom
business logic:

```json
{
  "select": ["?recommendation"],
  "where": {"@id": "ex:alice", "ex:recommended": "?recommendation"},
  "reasoning": ["owl2rl", "datalog"],
  "rules": [
    {
      "@context": {"ex": "http://example.org/"},
      "where": [
        {"@id": "?person", "ex:friend": "?friend"},
        {"@id": "?friend", "ex:likes": "?item"},
        {"@id": "?person", "ex:likes": "?item"}
      ],
      "insert": {"@id": "?person", "ex:recommended": {"@id": "?item"}}
    }
  ]
}
```

If `ex:friend` is declared as a `owl:SymmetricProperty`, OWL 2 RL
materializes the reverse friendship links, and then the Datalog rule can
find items liked by mutual friends.

## Execution model

### Fixpoint evaluation

Rules execute in a **fixpoint loop**:

1. All rules are applied against the current data (base + previously derived
   facts).
2. New facts produced in this iteration are collected.
3. If any new facts were produced, go back to step 1 with the expanded fact set.
4. When no new facts are produced (fixpoint reached), the loop terminates.

This means:
- **Recursive rules work.** A rule can produce facts that trigger itself again.
- **Rule chaining works.** Rule A can produce facts that trigger Rule B, and
  vice versa.
- **Termination is guaranteed** by the budget controls (max iterations, max
  facts, max time, max memory).

### Execution order

Rules are topologically sorted by their predicate dependencies: a rule that
generates `ex:uncle` triples runs before a rule that consumes `ex:uncle` in its
`where` clause. This minimizes the number of fixpoint iterations needed.

### Interaction with OWL 2 RL

When both OWL 2 RL and Datalog are enabled:

1. OWL 2 RL materialization runs first.
2. Datalog rules run over the combined base data + OWL-derived facts.
3. Both result sets are merged into a single overlay for query execution.

## Filter expressions

Filters use S-expression syntax within the `where` array:

```json
["filter", "(expression)"]
```

### Available operators

| Category | Operators |
|----------|-----------|
| Comparison | `=`, `!=`, `<`, `>`, `<=`, `>=` |
| Logical | `and`, `or`, `not` |
| Arithmetic | `+`, `-`, `*`, `/` |
| String | `str`, `strlen`, `contains`, `strstarts`, `strends` |
| Type checking | `isIRI`, `isBlank`, `isLiteral`, `bound` |

### Examples

```json
["filter", "(> ?age 21)"]
["filter", "(and (>= ?age 18) (< ?age 65))"]
["filter", "(contains ?name \"Smith\")"]
["filter", "(!= ?person ?other)"]
```

## Performance considerations

- **Keep rules focused.** Broad rules that match many patterns produce more
  derived facts and require more iterations.
- **Budget limits apply.** The same time/fact/memory budgets as OWL 2 RL
  materialization apply to Datalog execution (default: 30s, 1M facts, 100MB).
- **Results are cached.** The same rule set + database state returns instantly
  from cache on subsequent queries.
- **Query-time rules disable caching** across queries with different rule sets,
  since the cache key includes a hash of the rules.

## Related pages

| Topic | Page |
|-------|------|
| Conceptual introduction | [Reasoning and inference](../concepts/reasoning.md) |
| Enabling reasoning in queries | [Query-time reasoning](reasoning.md) |
| OWL & RDFS constructs | [OWL & RDFS reference](../reference/owl-rdfs-support.md) |
| Ledger-wide config | [Setting groups](../ledger-config/setting-groups.md) |


---

# transactions/README.md

# Transactions

Transactions are how you write data to Fluree. This section covers all transaction patterns, formats, and behaviors.

## Transaction Patterns

### [Overview](overview.md)

High-level introduction to Fluree transactions:
- Transaction lifecycle
- Commit process
- Indexing pipeline
- Transaction semantics

### [Insert](insert.md)

Adding new data to the database:
- Basic inserts
- Batch inserts
- Entity creation
- Relationship creation

### [Upsert](upsert.md)

Idempotent transactions that replace values for supplied predicates:
- Upsert semantics
- Use cases for upsert
- Idempotent operations
- Synchronization patterns

### [Update (WHERE/DELETE/INSERT)](update-where-delete-insert.md)

Targeted updates to existing data:
- WHERE clause patterns
- DELETE operations
- INSERT operations
- Conditional updates
- Partial updates

### [Retractions](retractions.md)

Removing data from the database:
- Retract specific triples
- Retract entire entities
- Retraction semantics
- Time travel and retractions

## Transaction Formats

### [Turtle Ingest](turtle.md)

Import RDF data in Turtle format:
- Turtle syntax
- Bulk imports
- File uploads
- Format conversion

### [Signed / Credentialed Transactions](signed-transactions.md)

Cryptographically signed transactions:
- JWS signed transactions
- Verifiable Credentials
- Identity-based transactions
- Audit trails

## Transaction Metadata

### [Commit Receipts and tx-id](commit-receipts.md)

Understanding transaction receipts:
- Receipt structure
- Transaction ID (t)
- Commit ID
- Timestamps
- Flake counts

### [Indexing Side-Effects](indexing-side-effects.md)

How transactions affect indexing:
- Background indexing
- Novelty layer
- Index triggers
- Performance considerations

## Transaction Concepts

### Immutability

Once committed, transactions are immutable:
- Changes are represented as new assertions and retractions
- Historical data is never modified
- Complete audit trail preserved
- Time travel enabled by immutability

### Atomicity

Transactions are atomic:
- All changes succeed or all fail
- No partial commits
- Consistent state guaranteed
- Validation before commit

### Transaction Time

Every transaction receives a unique transaction time:
- Monotonically increasing integer (t)
- Unique across all ledgers in instance
- Used for time travel queries
- Basis for temporal ordering

### Assertions and Retractions

Transactions consist of two operations:
- **Assertions**: Add new triples
- **Retractions**: Remove existing triples

Updates are represented as retraction + assertion pairs.

## Common Transaction Patterns

### Create Entity

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "schema": "http://schema.org/"
  },
  "@graph": [
    {
      "@id": "ex:alice",
      "@type": "schema:Person",
      "schema:name": "Alice",
      "schema:email": "alice@example.org"
    }
  ]
}
```

### Update Property

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "schema": "http://schema.org/"
  },
  "where": [
    { "@id": "ex:alice", "schema:age": "?oldAge" }
  ],
  "delete": [
    { "@id": "ex:alice", "schema:age": "?oldAge" }
  ],
  "insert": [
    { "@id": "ex:alice", "schema:age": 31 }
  ]
}
```

### Add Relationship

```json
{
  "@graph": [
    {
      "@id": "ex:alice",
      "schema:worksFor": { "@id": "ex:company-a" }
    }
  ]
}
```

### Remove Property

```json
{
  "delete": [
    { "@id": "ex:alice", "schema:telephone": "?phone" }
  ],
  "where": [
    { "@id": "ex:alice", "schema:telephone": "?phone" }
  ]
}
```

### Replace Entity (Upsert)

```bash
POST /upsert?ledger=mydb:main
```

```json
{
  "@graph": [
    {
      "@id": "ex:alice",
      "@type": "schema:Person",
      "schema:name": "Alice Smith",
      "schema:email": "alice.smith@example.org"
    }
  ]
}
```

## Transaction Types

- **Insert** (`POST /insert`) — add triples (JSON-LD or Turtle)
- **Update** (`POST /update`) — WHERE/DELETE/INSERT (JSON-LD) or SPARQL UPDATE
- **Upsert** (`POST /upsert`) — replace values for the predicates you supply (JSON-LD, Turtle, TriG)

## Transaction Validation

Before commit, transactions are validated:

**Syntax Validation:**
- Valid JSON/JSON-LD syntax
- Well-formed IRIs
- Correct datatype formats

**Semantic Validation:**
- Type compatibility
- Constraint adherence
- Reference integrity (optional)

**Policy Validation:**
- Authorization checks
- Access control enforcement
- Data-level permissions

Validation failures result in transaction rejection with detailed error messages.

## Transaction Size Limits

**Default Limits:**
- Transaction size: 10 MB
- Triple count: 10,000 triples
- Configurable per deployment

**Large Transactions:**
- Split into batches for large imports
- Use streaming for bulk data
- Monitor indexing lag

See [Indexing Side-Effects](indexing-side-effects.md) for performance considerations.

## Error Handling

### Transaction Errors

Common errors:
- `PARSE_ERROR` - Invalid JSON-LD
- `INVALID_IRI` - Malformed IRI
- `TYPE_ERROR` - Type mismatch
- `CONSTRAINT_VIOLATION` - Constraint violated
- `POLICY_DENIED` - Not authorized

### Retry Logic

Implement retry for transient errors:
- Network errors: Retry with backoff
- Conflicts: Retry with updated data
- Timeouts: Retry after delay
- Server errors: Retry with backoff

### Idempotency

For idempotent transactions:
- Use replace mode
- Include unique identifiers
- Design for retry safety
- Use deterministic IRIs

## Best Practices

### 1. Use Meaningful IRIs

Good:
```json
{"@id": "ex:user-alice-123"}
```

Bad:
```json
{"@id": "ex:1"}
```

### 2. Batch Related Changes

Combine related entities in single transaction:

```json
{
  "@graph": [
    { "@id": "ex:order-123", "ex:customer": { "@id": "ex:alice" } },
    { "@id": "ex:order-123", "ex:product": { "@id": "ex:widget" } },
    { "@id": "ex:order-123", "ex:total": 99.99 }
  ]
}
```

### 3. Use Appropriate Mode

- Default mode: For additive operations
- Replace mode: For complete replacements, sync operations

### 4. Include Types

Always specify entity types:

```json
{
  "@id": "ex:alice",
  "@type": "schema:Person"
}
```

### 5. Use Typed Literals

Be explicit about types:

```json
{
  "schema:birthDate": {
    "@value": "1990-05-15",
    "@type": "xsd:date"
  }
}
```

### 6. Design for History

Consider how data will look in historical queries:
- Use descriptive property names
- Include relevant metadata
- Design for temporal queries

### 7. Monitor Performance

Track transaction metrics:
- Commit time
- Indexing lag
- Error rates
- Transaction size

## Related Documentation

- [Getting Started: Write Data](../getting-started/quickstart-write.md) - Quickstart guide
- [Concepts: Time Travel](../concepts/time-travel.md) - Temporal semantics
- [API: POST /update](../api/endpoints.md#post-update) - HTTP endpoint details
- [Indexing](../indexing-and-search/README.md) - Indexing and search


---

# transactions/overview.md

# Transaction Overview

This document provides a comprehensive overview of how transactions work in Fluree, from submission to final indexing.

## What is a Transaction?

A **transaction** in Fluree is a set of changes to the database, represented as RDF triple assertions and retractions. Each transaction is:

- **Atomic**: All changes succeed or all fail
- **Immutable**: Once committed, never modified
- **Timestamped**: Assigned a unique transaction time (t)
- **Auditable**: Complete metadata preserved

## Transaction Lifecycle

### 1. Submission

Client submits transaction to Fluree using either JSON-LD or SPARQL UPDATE:

**JSON-LD Transaction:**
```bash
POST /update?ledger=mydb:main
Content-Type: application/json

{
  "@context": { "ex": "http://example.org/ns/" },
  "@graph": [{ "@id": "ex:alice", "ex:name": "Alice" }]
}
```

**SPARQL UPDATE:**
```bash
POST /update/mydb:main
Content-Type: application/sparql-update

PREFIX ex: <http://example.org/ns/>
INSERT DATA { ex:alice ex:name "Alice" }
```

### 2. Parsing

Fluree parses the transaction:
- Parse JSON/JSON-LD structure
- Expand compact IRIs using @context
- Convert to internal representation

### 3. Validation

Transaction is validated:
- **Syntax validation**: Well-formed IRIs, valid datatypes
- **Semantic validation**: Type compatibility, constraints
- **Policy validation**: Authorization checks

If validation fails, transaction is rejected with error details.

### 4. Conversion to Flakes

Transaction is converted to **flakes** (Fluree's internal triple format):

```text
Subject    Predicate           Object                    Operation
------------------------------------------------------------------------
ex:alice   rdf:type           schema:Person             assert
ex:alice   schema:name        "Alice"^^xsd:string       assert
```

Each flake is a tuple: (subject, predicate, object, transaction-time, operation, metadata)

### 5. Assignment of Transaction Time

Fluree assigns a unique transaction time (t):
- Monotonically increasing integer
- Unique across all transactions
- Used for temporal queries

Example: `t=42`

### 6. Commit

Transaction is committed to storage:
- Flakes written to transaction log
- Commit metadata created (ContentId, timestamp, etc.)
- Commit ID published to nameservice

**Commit Data:**
```json
{
  "t": 42,
  "timestamp": "2024-01-22T10:30:00.000Z",
  "commit_id": "bafybeig...commitT42",
  "flakes_added": 2,
  "flakes_retracted": 0
}
```

### 7. Nameservice Update

Nameservice is updated with new commit:
- `commit_t` updated to 42
- `commit_id` updated
- Other processes can see new commit

### 8. Indexing (Asynchronous)

Background process indexes the transaction:
- Flakes added to index structures (SPOT, POST, OPST, PSOT)
- Query-optimized data structures built
- Graph sources updated (if applicable)

### 9. Index Publication

When indexing completes:
- `index_t` updated to 42
- `index_id` published
- Novelty layer reduced

## Transaction Components

### @context

Defines namespace mappings:

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "schema": "http://schema.org/",
    "xsd": "http://www.w3.org/2001/XMLSchema#"
  }
}
```

The @context can be:
- Inline (as above)
- External URL: `"@context": "http://example.org/context.jsonld"`
- Array of contexts: `"@context": [url1, {...}]`

### @graph

Contains the entities being asserted:

```json
{
  "@graph": [
    {
      "@id": "ex:alice",
      "@type": "schema:Person",
      "schema:name": "Alice"
    },
    {
      "@id": "ex:bob",
      "@type": "schema:Person",
      "schema:name": "Bob"
    }
  ]
}
```

### opts

Top-level parse-time options. These control how the transaction is parsed (not what it writes).

```json
{
  "@context": {"ex": "http://example.org/ns/"},
  "opts": {"strictCompactIri": false},
  "@graph": [{"@id": "legacy:bob", "ex:name": "Bob"}]
}
```

Currently supported keys:

- **`strictCompactIri`** (bool, default `true`): Reject unresolved compact-looking IRIs (`prefix:suffix` where the prefix is missing from `@context`). Disable only for legacy data where bare `prefix:suffix` strings are intentional. See [IRIs and @context — Strict Compact-IRI Guard](../concepts/iri-and-context.md#strict-compact-iri-guard).

Programmatic Rust callers can override `strictCompactIri` via `TxnOpts.strict_compact_iri`, which takes precedence over the JSON `opts` value.

### WHERE/DELETE/INSERT

For updates, specify what to match, delete, and insert:

```json
{
  "where": [
    { "@id": "ex:alice", "schema:age": "?oldAge" }
  ],
  "delete": [
    { "@id": "ex:alice", "schema:age": "?oldAge" }
  ],
  "insert": [
    { "@id": "ex:alice", "schema:age": 31 }
  ]
}
```

### SPARQL UPDATE

Alternatively, use SPARQL UPDATE syntax with `Content-Type: application/sparql-update`:

```sparql
PREFIX ex: <http://example.org/ns/>

DELETE {
  ?person ex:age ?oldAge .
}
INSERT {
  ?person ex:age 31 .
}
WHERE {
  ?person ex:name "Alice" .
  ?person ex:age ?oldAge .
}
```

SPARQL UPDATE supports:
- `INSERT DATA` - Insert ground triples
- `DELETE DATA` - Delete specific triples
- `DELETE WHERE` - Delete matching patterns
- `DELETE/INSERT WHERE` - Full update with patterns

See [SPARQL UPDATE](../query/sparql.md#sparql-update) for complete documentation.

## Transaction Endpoints

Fluree exposes three transaction endpoints (all under `/v1/fluree/`):

- `POST /insert` — add triples (JSON-LD or Turtle)
- `POST /update` — WHERE/DELETE/INSERT (JSON-LD) and SPARQL UPDATE
- `POST /upsert` — replace values for the predicates you supply (JSON-LD, Turtle, TriG)

See [Insert](insert.md), [Update](update-where-delete-insert.md), and [Upsert](upsert.md) for details.

## Transaction Semantics

### Assertions

**Assertions** add new triples to the database:

```json
{
  "@id": "ex:alice",
  "schema:name": "Alice"
}
```

Creates triple:
```
ex:alice schema:name "Alice"
```

### Retractions

**Retractions** remove existing triples:

```json
{
  "delete": [
    { "@id": "ex:alice", "schema:age": "?age" }
  ],
  "where": [
    { "@id": "ex:alice", "schema:age": "?age" }
  ]
}
```

Removes matching triples.

### Updates

Updates are retraction + assertion:

```text
t=10: ex:alice schema:age 30 (assert)
t=20: ex:alice schema:age 30 (retract), ex:alice schema:age 31 (assert)
```

Historical queries can see both states.

## Commit Metadata

Each commit includes rich metadata:

**Core Fields:**
- `t`: Transaction time
- `timestamp`: ISO 8601 timestamp
- `commit_id`: Content-addressed identifier (CIDv1)

**Counts:**
- `flakes_added`: Number of assertions
- `flakes_retracted`: Number of retractions

**Provenance (in `txn-meta` graph, under the commit subject):**
- `f:identity`: Authenticated identity acting on the transaction. System-controlled — verified DID for signed requests, otherwise from `opts.identity` / `CommitOpts::identity`. Any user-supplied `f:identity` in the body is overridden.
- `f:author`: Optional author claim. Pure user txn-meta — supply `f:author` as a top-level property in the envelope-form transaction body.
- `f:message`: Optional commit message. Pure user txn-meta — supply `f:message` as a top-level property in the envelope-form transaction body.
- `previous_commit_id`: ContentId of previous commit (in the commit envelope).

See [Commit Receipts](commit-receipts.md) for details.

## Indexing Pipeline

### Commit vs Index

**Commit (immediate):**
- Transaction written to log
- Available for time travel queries
- Small, append-only files

**Index (asynchronous):**
- Query-optimized data structures
- Background process
- May lag behind commits

### Novelty Layer

The **novelty layer** is uncommitted data between index and commit:

```text
index_t = 40
commit_t = 45
novelty layer = transactions 41, 42, 43, 44, 45
```

Queries combine:
- Indexed data (up to t=40)
- Novelty layer (t=41 to t=45)

### Index Structures

Fluree maintains four index permutations (SPOT, POST, OPST, PSOT):

**SPOT** (Subject-Predicate-Object-Time):
```
ex:alice → schema:name → "Alice" → t=10
```

**POST** (Predicate-Object-Subject-Time):
```
schema:name → "Alice" → ex:alice → t=10
```

**OPST** (Object-Predicate-Subject-Time):
```
"Alice" → schema:name → ex:alice → t=10
```

**PSOT** (Predicate-Subject-Object-Time):
```
schema:name → ex:alice → "Alice" → t=10
```

Different query patterns use different indexes for optimal performance.

## Transaction Properties

### Atomicity

All-or-nothing execution:
- Validation failure rejects entire transaction
- Parse error rejects entire transaction
- No partial commits

### Consistency

Database remains consistent:
- Constraints enforced
- Types validated
- References checked (optionally)

### Isolation

Transactions are isolated:
- Each sees consistent snapshot
- No dirty reads
- Serializable execution

### Durability

Committed data is durable:
- Written to persistent storage
- Replicated (if configured)
- Immutable

## Error Handling

### Validation Errors

```json
{
  "error": "ValidationError",
  "message": "Invalid IRI format",
  "code": "INVALID_IRI",
  "details": {
    "iri": "not a uri",
    "line": 3
  }
}
```

### Conflict Errors

```json
{
  "error": "ConflictError",
  "message": "Concurrent modification detected",
  "code": "CONCURRENT_MODIFICATION"
}
```

### Policy Errors

```json
{
  "error": "Forbidden",
  "message": "Policy denies transact on mydb:main",
  "code": "POLICY_DENIED"
}
```

## Performance Considerations

### Transaction Size

- Recommended: < 1,000 triples per transaction
- Maximum: Configurable (default 10,000)
- Large transactions increase commit time

### Indexing Lag

- Background indexing may lag behind commits
- Monitor `commit_t - index_t` gap
- Tune indexing frequency if needed

### Batch Operations

For bulk imports:
- Batch into reasonably-sized transactions
- Monitor memory usage
- Allow time for indexing between batches

For initial ledger bootstraps (large Turtle datasets), prefer the Rust bulk import API which
streams commits and builds multi-order binary indexes:

- [Using Fluree as a Rust library → Bulk import Turtle chunks](../getting-started/rust-api.md#bulk-import-turtle-chunks-high-throughput)

See [Indexing Side-Effects](indexing-side-effects.md) for details.

## Best Practices

### 1. Meaningful Transaction Units

Group related changes in single transaction:

Good:
```json
{
  "@graph": [
    { "@id": "ex:order-123", "ex:customer": { "@id": "ex:alice" } },
    { "@id": "ex:order-123", "ex:items": [...] },
    { "@id": "ex:order-123", "ex:total": 99.99 }
  ]
}
```

### 2. Include Metadata

Add provenance information:

```json
{
  "@graph": [
    {
      "@id": "ex:alice",
      "schema:name": "Alice",
      "ex:created": "2024-01-22T10:00:00Z",
      "ex:createdBy": "user-123"
    }
  ]
}
```

### 3. Use Descriptive IRIs

Good: `ex:user-alice-123`
Bad: `ex:1`

### 4. Test Transactions

Test transactions before production:
- Validate JSON-LD syntax
- Check IRI formats
- Verify types and constraints

### 5. Monitor Performance

Track metrics:
- Average commit time
- Indexing lag
- Transaction size
- Error rate

### 6. Handle Errors Gracefully

Implement retry logic for transient errors:
- Network errors
- Timeout errors
- Conflict errors (with updated data)

### 7. Design for Time Travel

Remember data is immutable:
- Changes create new versions
- Historical queries see all versions
- Design with temporal access in mind

## Related Documentation

- [Insert](insert.md) - Adding new data
- [Upsert](upsert.md) - Replace mode
- [Update](update-where-delete-insert.md) - Targeted updates
- [Commit Receipts](commit-receipts.md) - Receipt details
- [Indexing Side-Effects](indexing-side-effects.md) - Indexing behavior


---

# transactions/insert.md

# Insert

Insert operations add new data to Fluree. This is the most common transaction type for creating new entities and relationships.

## Basic Insert

### Single Entity

Insert a single entity with properties:

```bash
curl -X POST "http://localhost:8090/v1/fluree/insert?ledger=mydb:main" \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "ex": "http://example.org/ns/",
      "schema": "http://schema.org/"
    },
    "@graph": [
      {
        "@id": "ex:alice",
        "@type": "schema:Person",
        "schema:name": "Alice",
        "schema:email": "alice@example.org",
        "schema:age": 30
      }
    ]
  }'
```

**Result:**
```json
{
  "t": 1,
  "timestamp": "2024-01-22T10:30:00.000Z",
  "commit_id": "bafybeig...commitT1",
  "flakes_added": 4,
  "flakes_retracted": 0
}
```

This creates 4 triples:
```
ex:alice rdf:type schema:Person
ex:alice schema:name "Alice"
ex:alice schema:email "alice@example.org"
ex:alice schema:age 30
```

## Multiple Entities

Insert multiple entities in one transaction:

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "schema": "http://schema.org/"
  },
  "@graph": [
    {
      "@id": "ex:alice",
      "@type": "schema:Person",
      "schema:name": "Alice"
    },
    {
      "@id": "ex:bob",
      "@type": "schema:Person",
      "schema:name": "Bob"
    },
    {
      "@id": "ex:carol",
      "@type": "schema:Person",
      "schema:name": "Carol"
    }
  ]
}
```

**Benefits:**
- Atomic: All entities created or none
- Efficient: Single commit, single index update
- Consistent: All entities at same transaction time

## Insert with Relationships

Create entities with relationships:

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "schema": "http://schema.org/"
  },
  "@graph": [
    {
      "@id": "ex:company-a",
      "@type": "schema:Organization",
      "schema:name": "Acme Corp"
    },
    {
      "@id": "ex:alice",
      "@type": "schema:Person",
      "schema:name": "Alice",
      "schema:worksFor": { "@id": "ex:company-a" }
    }
  ]
}
```

This creates:
```
ex:company-a rdf:type schema:Organization
ex:company-a schema:name "Acme Corp"
ex:alice rdf:type schema:Person
ex:alice schema:name "Alice"
ex:alice schema:worksFor ex:company-a
```

## Nested Objects

Create nested structures:

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "schema": "http://schema.org/"
  },
  "@graph": [
    {
      "@id": "ex:alice",
      "@type": "schema:Person",
      "schema:name": "Alice",
      "schema:address": {
        "@id": "ex:alice-address",
        "@type": "schema:PostalAddress",
        "schema:streetAddress": "123 Main St",
        "schema:addressLocality": "Springfield",
        "schema:postalCode": "12345"
      }
    }
  ]
}
```

This creates two entities (alice and alice-address) linked by schema:address.

## Multi-Valued Properties

Add multiple values for a property:

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "schema": "http://schema.org/"
  },
  "@graph": [
    {
      "@id": "ex:alice",
      "@type": "schema:Person",
      "schema:name": "Alice",
      "schema:email": ["alice@example.org", "alice@work.com"],
      "schema:telephone": ["+1-555-0100", "+1-555-0101"]
    }
  ]
}
```

Creates separate triples for each value:
```
ex:alice schema:email "alice@example.org"
ex:alice schema:email "alice@work.com"
ex:alice schema:telephone "+1-555-0100"
ex:alice schema:telephone "+1-555-0101"
```

## Typed Literals

### Dates

```json
{
  "@id": "ex:alice",
  "schema:birthDate": {
    "@value": "1994-05-15",
    "@type": "xsd:date"
  }
}
```

### Timestamps

```json
{
  "@id": "ex:event",
  "schema:startDate": {
    "@value": "2024-01-22T10:30:00Z",
    "@type": "xsd:dateTime"
  }
}
```

### Numbers

```json
{
  "@id": "ex:product",
  "schema:price": {
    "@value": "29.99",
    "@type": "xsd:decimal"
  }
}
```

### Booleans

```json
{
  "@id": "ex:alice",
  "schema:active": {
    "@value": "true",
    "@type": "xsd:boolean"
  }
}
```

Or use native JSON boolean:
```json
{
  "@id": "ex:alice",
  "schema:active": true
}
```

## Language Tags

Add language-tagged strings:

```json
{
  "@id": "ex:alice",
  "schema:name": {
    "@value": "Alice",
    "@language": "en"
  },
  "schema:description": [
    { "@value": "Software engineer", "@language": "en" },
    { "@value": "Ingénieure logicielle", "@language": "fr" },
    { "@value": "Softwareingenieurin", "@language": "de" }
  ]
}
```

The language tag is part of a value's identity. Two language-tagged values
that share the same string but differ in their `@language` (for example
`"animal"@en` and `"animal"@fr`) are **distinct** facts and are both retained;
they are not deduplicated against each other.

## Blank Nodes

Create entities without explicit IRIs:

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "schema": "http://schema.org/"
  },
  "@graph": [
    {
      "@id": "ex:alice",
      "schema:address": {
        "@type": "schema:PostalAddress",
        "schema:streetAddress": "123 Main St"
      }
    }
  ]
}
```

Fluree generates a unique IRI for the blank node address.

## Adding to Existing Entities

Add properties to existing entities:

**Initial Insert (t=1):**
```json
{
  "@graph": [
    {
      "@id": "ex:alice",
      "schema:name": "Alice"
    }
  ]
}
```

**Add Email (t=2):**
```json
{
  "@graph": [
    {
      "@id": "ex:alice",
      "schema:email": "alice@example.org"
    }
  ]
}
```

After t=2, ex:alice has both name and email.

## Edge Annotations

Attach metadata to a specific edge via an `@annotation` block on the object node-map. The annotation reifies the `(subject, predicate, object)` triple it sits under:

```json
{
  "@id": "ex:alice",
  "ex:worksFor": {
    "@id": "ex:acme",
    "@annotation": {
      "ex:role": "Engineer",
      "ex:since": { "@value": "2024-01-01", "@type": "xsd:date" }
    }
  }
}
```

`@edge` is an alias for `@annotation`; the two are interchangeable.

The annotation subject can be left anonymous (a blank node is minted) or pinned to an explicit IRI for stable identity:

```json
"@annotation": {
  "@id": "ex:emp/alice-acme-2024",
  "ex:role": "Engineer"
}
```

**Multiple parallel annotations** on the same `(s, p, o)` edge are supported — assert two `@annotation` blocks on separate copies of the edge in the same transaction (each with its own `@id` to keep them distinct):

```json
{
  "@graph": [
    { "@id": "ex:alice", "ex:worksFor": {
        "@id": "ex:acme",
        "@annotation": { "@id": "ex:emp/2020", "ex:role": "Engineer" }
    }},
    { "@id": "ex:alice", "ex:worksFor": {
        "@id": "ex:acme",
        "@annotation": { "@id": "ex:emp/2024", "ex:role": "Manager" }
    }}
  ]
}
```

Inline `@annotation` queries return one row per occurrence.

**Annotated literal-valued edges** are supported — write the object as a JSON-LD value object so the annotation has a sibling key to attach to:

```json
{
  "@id": "ex:alice",
  "ex:name": {
    "@value": "Alice",
    "@annotation": { "ex:source": "ex:hr-system" }
  }
}
```

**Deferred shapes** error with explicit messages:

- Annotations on list-occurrence triples (`@list` membership).
- Reifiers for unasserted triples (`@reifies` must point at an asserted edge).
- Multi-triple `@reifies` (more than one predicate-object pair under `@reifies`).
- Annotation-of-annotation (nested `@annotation` inside an annotation body).
- `@reifies` on the insert side (use the inline `@annotation` form instead).
- Hand-authored mention of the [reserved system predicates](../reference/vocabulary.md#edge-annotation-predicates-reserved) that back annotations (compact or full IRI form).

For the full surface — including SPARQL 1.2 / RDF 1.2 annotation tails (`{| |}`), the named reifier (`~`), the cardinality / multiplicity contract, anonymous vs explicit-IRI lifecycle, and named-graph behavior — see the [Edge annotations concept doc](../concepts/edge-annotations.md). For cascade semantics when a base edge or annotation metadata is removed, see [Retractions](retractions.md).

## Insert Semantics

### Additive by Default

Inserts are additive—they don't remove existing data:

```text
t=1: INSERT { ex:alice schema:name "Alice" }
     Result: ex:alice has name "Alice"

t=2: INSERT { ex:alice schema:age 30 }
     Result: ex:alice has name "Alice" AND age 30
```

### Duplicate Prevention

Inserting the same triple again is a no-op:

```text
t=1: INSERT { ex:alice schema:name "Alice" }
t=2: INSERT { ex:alice schema:name "Alice" }
     (No change—triple already exists)
```

A triple's identity includes its datatype and language tag, so values that
differ only by language tag are not duplicates:

```text
INSERT { ex:alice schema:name "Alice"@en, "Alice"@fr }
     Result: ex:alice has TWO name values (one per language tag)
```

### Multi-Value Handling

Multiple values create multiple triples:

```text
t=1: INSERT { ex:alice schema:email "alice@example.org" }
t=2: INSERT { ex:alice schema:email "alice@work.com" }
     Result: ex:alice has TWO email values
```

## IRI Generation

### Explicit IRIs

Specify IRIs explicitly:

```json
{
  "@id": "ex:user-12345",
  "schema:name": "Alice"
}
```

### UUID-Based IRIs

Generate UUIDs for unique IRIs:

```javascript
const uuid = crypto.randomUUID();
const entity = {
  "@id": `ex:user-${uuid}`,
  "schema:name": "Alice"
};
```

### Content-Addressable IRIs

Use content hashing for deterministic IRIs:

```javascript
const hash = sha256(JSON.stringify(data));
const entity = {
  "@id": `ex:entity-${hash}`,
  ...data
};
```

## Batch Inserts

### Small Batches (Recommended)

```json
{
  "@graph": [
    { "@id": "ex:user-1", "schema:name": "Alice" },
    { "@id": "ex:user-2", "schema:name": "Bob" },
    { "@id": "ex:user-3", "schema:name": "Carol" }
    // ... 100-1000 entities
  ]
}
```

### Large Imports

For very large imports:

```javascript
const batchSize = 1000;
for (let i = 0; i < entities.length; i += batchSize) {
  const batch = entities.slice(i, i + batchSize);
  await transact({ "@graph": batch });
  
  // Optional: wait for indexing
  await sleep(1000);
}
```

## Error Handling

### Common Insert Errors

**Invalid IRI:**
```json
{
  "error": "ValidationError",
  "message": "Invalid IRI format",
  "code": "INVALID_IRI"
}
```

**Type Mismatch:**
```json
{
  "error": "TypeError",
  "message": "Expected number, got string",
  "code": "TYPE_ERROR"
}
```

**Constraint Violation:**
```json
{
  "error": "ConstraintViolation",
  "message": "Unique constraint violated",
  "code": "CONSTRAINT_VIOLATION"
}
```

### Validation Before Insert

Validate data before inserting:

```javascript
function validateEntity(entity) {
  if (!entity['@id']) {
    throw new Error('Entity must have @id');
  }
  if (!isValidIRI(entity['@id'])) {
    throw new Error('Invalid IRI format');
  }
  // Additional validation...
}
```

## Best Practices

### 1. Use Meaningful IRIs

Good:
```json
{ "@id": "ex:user-alice-12345" }
```

Bad:
```json
{ "@id": "ex:1" }
```

### 2. Always Include Type

```json
{
  "@id": "ex:alice",
  "@type": "schema:Person"
}
```

### 3. Use Appropriate Datatypes

```json
{
  "schema:age": 30,
  "schema:price": 29.99,
  "schema:active": true,
  "schema:birthDate": { "@value": "1994-05-15", "@type": "xsd:date" }
}
```

### 4. Batch Related Entities

Insert related entities in same transaction:

```json
{
  "@graph": [
    { "@id": "ex:order-123", ... },
    { "@id": "ex:order-item-1", ... },
    { "@id": "ex:order-item-2", ... }
  ]
}
```

### 5. Use Consistent Namespaces

Define and use consistent namespace prefixes:

```json
{
  "@context": {
    "app": "https://myapp.com/ns/",
    "schema": "http://schema.org/"
  }
}
```

### 6. Include Metadata

Add creation metadata:

```json
{
  "@id": "ex:alice",
  "schema:name": "Alice",
  "app:createdAt": "2024-01-22T10:00:00Z",
  "app:createdBy": "user-admin"
}
```

### 7. Validate Before Insert

Always validate:
- JSON-LD syntax
- IRI formats
- Required fields
- Type constraints

## Performance Tips

### 1. Batch Appropriately

- Recommended: 100-1000 entities per batch
- Too small: Many commits, slow
- Too large: Memory pressure, long commits

### 2. Monitor Indexing

Track indexing lag after large inserts:

```bash
curl http://localhost:8090/v1/fluree/info/mydb:main
# Check: t - index.t
```

### 3. Use Efficient IRIs

Short IRIs are more efficient:

Good: `ex:user-123`
Less efficient: `https://example.org/very/long/path/user-123`

### 4. Minimize Context Size

Use compact contexts:

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "schema": "http://schema.org/"
  }
}
```

## Related Documentation

- [Overview](overview.md) - Transaction overview
- [Upsert](upsert.md) - Replace mode inserts
- [Update](update-where-delete-insert.md) - Updating existing data
- [Data Types](../concepts/datatypes.md) - Supported datatypes
- [API Endpoints](../api/endpoints.md) - HTTP API details


---

# transactions/upsert.md

# Upsert

Upsert operations provide idempotent transactions by **replacing the values of the predicates you supply** for an entity (matched by `@id`).

## What is Upsert?

**Upsert** = Update or Insert:
- If the entity exists: for each predicate present in your payload, retract existing values for that predicate and assert the new value(s)
- If the entity doesn’t exist: create it with the supplied triples

This makes upserts safe to retry: sending the same upsert repeatedly produces the same current-state values for those predicates.

## HTTP Endpoint

Use the dedicated upsert endpoint:

```bash
curl -X POST "http://localhost:8090/v1/fluree/upsert?ledger=mydb:main" \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "ex": "http://example.org/ns/",
      "schema": "http://schema.org/"
    },
    "@graph": [
      {
        "@id": "ex:alice",
        "@type": "schema:Person",
        "schema:name": "Alice Smith",
        "schema:email": "alice.smith@example.org"
      }
    ]
  }'
```

## Upsert Behavior

### First Transaction (Entity Doesn't Exist)

```json
{
  "@graph": [
    {
      "@id": "ex:alice",
      "@type": "schema:Person",
      "schema:name": "Alice",
      "schema:email": "alice@example.org"
    }
  ]
}
```

**Result:** Entity created with specified properties.

**Triples After t=1:**
```
ex:alice rdf:type schema:Person
ex:alice schema:name "Alice"
ex:alice schema:email "alice@example.org"
```

### Second Transaction (Entity Exists)

```json
{
  "@graph": [
    {
      "@id": "ex:alice",
      "@type": "schema:Person",
      "schema:name": "Alice Smith",
      "schema:email": "alice.smith@example.org",
      "schema:age": 30
    }
  ]
}
```

**Operations:**
1. Retract ALL existing properties of ex:alice
2. Assert new properties

**Flakes:**
```
# Retractions (t=2)
ex:alice schema:name "Alice" (retract)
ex:alice schema:email "alice@example.org" (retract)

# Assertions (t=2)
ex:alice rdf:type schema:Person (assert)
ex:alice schema:name "Alice Smith" (assert)
ex:alice schema:email "alice.smith@example.org" (assert)
ex:alice schema:age 30 (assert)
```

**Triples After t=2:**
```
ex:alice rdf:type schema:Person
ex:alice schema:name "Alice Smith"
ex:alice schema:email "alice.smith@example.org"
ex:alice schema:age 30
```

Note: The `@type` is re-asserted (types are always included in replace).

## Idempotency

Replace mode is idempotent—repeated submissions produce the same result:

**First Submission (t=1):**
```json
{"@id": "ex:alice", "schema:name": "Alice", "schema:age": 30}
```
Result: Entity created.

**Second Submission (t=2):**
```json
{"@id": "ex:alice", "schema:name": "Alice", "schema:age": 30}
```
Result: No actual changes (retracts and re-asserts same values).

**Third Submission (t=3):**
```json
{"@id": "ex:alice", "schema:name": "Alice", "schema:age": 30}
```
Result: No actual changes.

This makes upserts safe to retry.

## Comparison: Insert vs Update vs Upsert

### Insert

```bash
POST /insert?ledger=mydb:main
```

**Behavior:**
- Additive: asserts the triples you submit
- Does not retract existing values automatically

**Example:**
```text
t=1: INSERT { ex:alice schema:name "Alice", schema:age 30 }
t=2: INSERT { ex:alice schema:email "alice@example.org" }

Result: ex:alice has name, age, AND email (all three)
```

### Update (WHERE/DELETE/INSERT)

```bash
POST /update?ledger=mydb:main
```

**Behavior:**
- Explicit: you retract exactly what you match in `where`/`delete`, then assert `insert`
- Most flexible (conditional updates, partial updates, computed values)

**Example:**
```text
t=1: INSERT { ex:alice schema:name "Alice", schema:age 30 }
t=2: UPDATE { DELETE { ex:alice schema:age 30 } INSERT { ex:alice schema:age 31 } WHERE { ex:alice schema:age 30 } }

Result: ex:alice has name "Alice", age 31
```

### Upsert

```bash
POST /upsert?ledger=mydb:main
```

**Behavior:**
- Replaces values **for the predicates you supply** (per subject)
- Leaves other predicates unchanged
- Retry-safe/idempotent for the supplied predicates

## Use Cases

### 1. Synchronization from External Systems

Sync data from external database:

```javascript
async function syncUser(externalUser) {
  await fetch('http://localhost:8090/v1/fluree/upsert?ledger=mydb:main', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      "@graph": [{
        "@id": `ex:user-${externalUser.id}`,
        "@type": "schema:Person",
        "schema:name": externalUser.name,
        "schema:email": externalUser.email,
        "schema:telephone": externalUser.phone
      }]
    })
  })
}

// Safe to call repeatedly—always matches external state
await syncUser(fetchUserFromDB(123));
```

### 2. Idempotent API Operations

Make API operations retry-safe:

```javascript
// Safe to retry on failure
async function updateProduct(productId, productData) {
  return await fetch('http://localhost:8090/v1/fluree/upsert?ledger=mydb:main', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      "@graph": [{
        "@id": `ex:product-${productId}`,
        ...productData
      }]
    })
  })
}
```

### 3. Configuration Management

Update configuration atomically:

```json
{
  "@graph": [
    {
      "@id": "ex:config",
      "@type": "ex:Configuration",
      "ex:apiEndpoint": "https://api.example.com",
      "ex:timeout": 30000,
      "ex:retries": 3,
      "ex:enabled": true
    }
  ]
}
```

Each update replaces entire configuration—no orphaned settings.

### 4. State Machine Transitions

Model state machines where entity has well-defined state:

```json
{
  "@graph": [
    {
      "@id": "ex:order-123",
      "@type": "ex:Order",
      "ex:status": "shipped",
      "ex:shippedAt": "2024-01-22T10:30:00Z",
      "ex:carrier": "FedEx",
      "ex:trackingNumber": "123456789"
    }
  ]
}
```

## Batch Upserts

Upsert multiple entities:

```bash
POST /upsert?ledger=mydb:main
```

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "schema": "http://schema.org/"
  },
  "@graph": [
    {
      "@id": "ex:user-1",
      "@type": "schema:Person",
      "schema:name": "Alice"
    },
    {
      "@id": "ex:user-2",
      "@type": "schema:Person",
      "schema:name": "Bob"
    },
    {
      "@id": "ex:user-3",
      "@type": "schema:Person",
      "schema:name": "Carol"
    }
  ]
}
```

Each entity is replaced independently.

## Type Handling

### Types are Preserved

Upsert preserves existing `@type` values unless you explicitly include `@type` in the upsert payload (in which case `rdf:type` is treated like any other predicate and its values are replaced for that subject).

```json
{
  "@graph": [
    {
      "@id": "ex:alice",
      "@type": "schema:Person",
      "schema:name": "Alice"
    }
  ]
}
```

The `@type` is always asserted, even if it existed before.

### Multiple Types

Entities can have multiple types:

```json
{
  "@graph": [
    {
      "@id": "ex:alice",
      "@type": ["schema:Person", "ex:Employee"],
      "schema:name": "Alice"
    }
  ]
}
```

All types are replaced together.

## Edge Cases

### Empty Replacement

Replacing with minimal data removes other properties:

**Before (t=1):**
```json
{
  "@id": "ex:alice",
  "schema:name": "Alice",
  "schema:email": "alice@example.org",
  "schema:age": 30,
  "schema:telephone": "+1-555-0100"
}
```

**Replace (t=2):**
```json
{
  "@id": "ex:alice",
  "@type": "schema:Person",
  "schema:name": "Alice"
}
```

**After t=2:**
```json
{
  "@id": "ex:alice",
  "@type": "schema:Person",
  "schema:name": "Alice"
}
```

Email, age, and telephone are removed.

### Partial Updates Not Possible

Replace mode replaces ALL properties—partial updates not supported.

For partial updates, use [WHERE/DELETE/INSERT](update-where-delete-insert.md).

## Error Handling

### Same Errors as Default Mode

Replace mode has same validation errors:

```json
{
  "error": "ValidationError",
  "message": "Invalid IRI format",
  "code": "INVALID_IRI"
}
```

### No Special Errors

Replace mode doesn't introduce new error types—it's just different semantics for the same operations.

## Performance Considerations

### Retraction Overhead

Replace mode may retract many triples:

```text
Entity with 50 properties:
- 50 retractions
- 50 assertions
= 100 flakes per entity
```

For entities with many properties, this can be expensive.

### Indexing Impact

Each retraction and assertion updates indexes:
- More work for indexing process
- May increase indexing lag
- Consider batch size for large replacements

## Best Practices

### 1. Use for Idempotent Operations

Good use:
```javascript
// Idempotent sync
await upsertUser(userId, userData);
await upsertUser(userId, userData); // Safe to retry
```

### 2. Include All Required Properties

Always include all properties entity should have:

Good:
```json
{
  "@id": "ex:user-123",
  "@type": "schema:Person",
  "schema:name": "Alice",
  "schema:email": "alice@example.org",
  "ex:status": "active"
}
```

Bad (incomplete):
```json
{
  "@id": "ex:user-123",
  "schema:name": "Alice"
}
```

### 3. Use Consistent Schema

Define entity schema and always include all fields:

```javascript
function createUserTransaction(user) {
  return {
    "@id": `ex:user-${user.id}`,
    "@type": "schema:Person",
    "schema:name": user.name || null,
    "schema:email": user.email || null,
    "schema:telephone": user.phone || null,
    "ex:status": user.status || "active"
  };
}
```

### 4. Document Upsert Usage

Comment when using upsert for idempotent sync:

```javascript
// Upsert for idempotent sync with external API
await fetch('http://localhost:8090/v1/fluree/upsert?ledger=users:main', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify(userPayload),
});
```

### 5. Test Idempotency

Verify operations are truly idempotent:

```javascript
const result1 = await upsert(data);
const result2 = await upsert(data);
// Should produce same final state
```

### 6. Monitor Performance

Track metrics for replace operations:
- Flakes retracted
- Flakes asserted
- Commit time
- Indexing lag

### 7. Consider Alternatives

For partial updates, use WHERE/DELETE/INSERT:

```json
{
  "where": [{ "@id": "ex:alice", "schema:age": "?oldAge" }],
  "delete": [{ "@id": "ex:alice", "schema:age": "?oldAge" }],
  "insert": [{ "@id": "ex:alice", "schema:age": 31 }]
}
```

## Comparison Table

| Feature | Default Mode | Replace Mode |
|---------|--------------|--------------|
| **Behavior** | Additive | Replace all |
| **Existing properties** | Preserved | Removed |
| **Idempotent** | No | Yes |
| **Partial updates** | Yes (with WHERE/DELETE/INSERT) | No |
| **Use case** | Adding data | Synchronization |
| **Retry safety** | Requires care | Safe by default |
| **Performance** | Fewer operations | More operations |

## Related Documentation

- [Insert](insert.md) - Adding new data
- [Update](update-where-delete-insert.md) - Partial updates
- [Overview](overview.md) - Transaction overview
- [API Endpoints](../api/endpoints.md) - HTTP API details


---

# transactions/update-where-delete-insert.md

# Update (WHERE/DELETE/INSERT)

The WHERE/DELETE/INSERT pattern enables targeted updates to existing data in Fluree. This is the most flexible update mechanism, allowing conditional modifications, partial updates, and complex transformations.

## Basic Pattern

The WHERE/DELETE/INSERT pattern has three clauses:

1. **WHERE**: Pattern to match existing data
2. **DELETE**: Triples to retract (using variables from WHERE)
3. **INSERT**: Triples to assert (using variables from WHERE)

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "schema": "http://schema.org/"
  },
  "where": [
    { "@id": "ex:alice", "schema:age": "?oldAge" }
  ],
  "delete": [
    { "@id": "ex:alice", "schema:age": "?oldAge" }
  ],
  "insert": [
    { "@id": "ex:alice", "schema:age": 31 }
  ]
}
```

This:
1. Finds the current age of ex:alice
2. Deletes that age value
3. Inserts the new age value

## WHERE clause capabilities

The update transaction `where` clause uses the **same pattern grammar as JSON-LD queries**, so you can use rich patterns like OPTIONAL, UNION, FILTER, VALUES, and subqueries.

Two common forms:

- **Node-map**: a single object (simple triple patterns)
- **Array**: a sequence of node-maps plus special forms (recommended for anything beyond basic matching)

Supported special forms inside the `where` array:

- `["filter", <expr>]`
- `["bind", "?var", <expr>]` (may include multiple var/expr pairs)
- `["optional", <pattern>]`
- `["union", <pattern>, <pattern>, ...]`
- `["minus", <pattern>]`
- `["exists", <pattern>]` / `["not-exists", <pattern>]`
- `["values", <values-clause>]`
- `["query", <subquery>]` (subquery can use `select`, `groupBy`, aggregates like `(max ?x)`, etc.)
- `["graph", <graph-name>, <pattern>]`

Expression format for `filter`/`bind` supports either:

- **Data expressions** like `["+", "?x", 1]`, `["and", [">=", "?age", 18], ["=", "?status", "pending"]]`
- **S-expressions** like `"(+ ?x 1)"`

## Graph scoping (named graphs)

JSON-LD update supports writing into **user-defined named graphs** (ingested via TriG or JSON-LD `@graph`) and scoping the update to a named graph.

### Default graph for WHERE/DELETE/INSERT

Use a top-level `graph` key to scope the update to a named graph **as the default graph**:

```json
{
  "@context": { "ex": "http://example.org/ns/", "schema": "http://schema.org/" },
  "graph": "http://example.org/graphs/audit",
  "where":  { "@id": "ex:event1", "schema:description": "?old" },
  "delete": { "@id": "ex:event1", "schema:description": "?old" },
  "insert": { "@id": "ex:event1", "schema:description": "new" }
}
```

This is the JSON-LD UPDATE analog of SPARQL UPDATE `WITH <iri>`:
- WHERE patterns are evaluated against the named graph
- DELETE/INSERT templates without an explicit graph are written to that named graph

### Writing templates to specific graphs

There are two ways to target graphs in `insert` / `delete` templates:

- **Per-node `@graph`**: attach a graph IRI to a node object (overrides the transaction-level `graph`)

```json
{
  "insert": [
    { "@id": "ex:event1", "@graph": "http://example.org/graphs/audit", "schema:description": "v" }
  ]
}
```

- **Template sugar**: inside `insert` / `delete` arrays, use `["graph", "<graph IRI>", <pattern>]`

```json
{
  "insert": [
    ["graph", "http://example.org/graphs/audit", { "@id": "ex:event1", "schema:description": "v" }]
  ]
}
```

Notes:
- `graph` is a **graph IRI** (a string like `"http://example.org/graphs/audit"`)
- Named-graph reads are available after indexing completes (see `docs/query/datasets.md`)

## Dataset scoping for WHERE (`from` / `fromNamed`)

JSON-LD update reuses the **same dataset keys as JSON-LD query** to control where the `where` clause reads from:

- **`from`**: scopes the default graph used for `where` evaluation (equivalent to SPARQL UPDATE `USING <iri>`)
- **`fromNamed`**: restricts which named graphs are visible to `where` `["graph", ...]` patterns (equivalent to SPARQL UPDATE `USING NAMED <iri>`)

This is why JSON-LD update uses `from` rather than introducing new keywords: it matches the existing JSON-LD query language vocabulary and keeps dataset configuration consistent across read-only queries and updates.

### `from` (WHERE default graph)

When `from` is present, it scopes the `where` clause evaluation without changing where templates write:

- `graph` (if present) controls the default graph for DELETE/INSERT templates (SPARQL UPDATE `WITH`)
- `from` controls the default graph(s) for `where` evaluation (SPARQL UPDATE `USING`)

Notes:
- `from` can be:
  - a string graph IRI (shorthand for `{"graph": "<iri>"}`)
  - an object with `{"graph": "<iri>"}` (or `{"graph": ["<iri1>", "<iri2>"]}`)
  - an array of graph IRIs/selectors (multiple graphs are evaluated as a merged default graph)
- If your `insert` / `delete` templates write into the same graph as the top-level `graph`, you can omit per-template graph selection. The top-level `graph` becomes the default target for templates that don't specify `@graph` (or `["graph", ...]` sugar).
- If you want to write to **multiple** graphs in one update, keep a top-level `graph` as the default (optional) and use per-template `["graph", ...]` for the exceptions.

```json
{
  "@context": { "ex": "http://example.org/ns/", "schema": "http://schema.org/" },
  "graph": "http://example.org/g2",
  "from": { "graph": "http://example.org/g1" },
  "where": { "@id": "ex:s", "schema:description": "?d" },
  "insert": [{ "@id": "ex:s", "schema:copyFromG1": "?d" }]
}
```

Example: read from one graph, write to two graphs

```json
{
  "@context": { "ex": "http://example.org/ns/", "schema": "http://schema.org/" },
  "graph": "http://example.org/g2",
  "from": { "graph": "http://example.org/g1" },
  "where": { "@id": "ex:s", "schema:description": "?d" },
  "insert": [
    { "@id": "ex:s", "schema:copyFromG1": "?d" },
    ["graph", "http://example.org/audit", { "@id": "ex:event1", "schema:description": "copied description" }]
  ]
}
```

### `fromNamed` (WHERE named graphs allowlist)

Use `fromNamed` to allow (and optionally alias) named graphs for `where` `["graph", ...]` patterns:

Notes:
- In `where` GRAPH patterns, you can reference the graph by **alias** (e.g. `"g2"`) or by the **graph IRI** (e.g. `"http://example.org/g2"`). Aliases are just convenience names for matching.
- In `insert` / `delete` templates, graph selection is a **write target**. You can use:
  - the full graph IRI (`"http://example.org/g2"`)
  - a compact IRI/term that expands via `@context` (e.g. `"ex:g2"`)
  - the `fromNamed` **alias** (e.g. `"g2"`) for consistency within the same update transaction

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "fromNamed": [
    { "alias": "g2", "graph": "http://example.org/g2" }
  ],
  "where": [
    ["graph", "g2", { "@id": "ex:s", "ex:p": "?o" }]
  ],
  "insert": [["graph", "g2", { "@id": "ex:s", "ex:q": "touched" }]]
}
```

Same example, but with a compacted graph IRI via `@context`:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "fromNamed": [{ "alias": "g2", "graph": "ex:g2" }],
  "where": [["graph", "g2", { "@id": "ex:s", "ex:p": "?o" }]],
  "insert": [["graph", "ex:g2", { "@id": "ex:s", "ex:q": "touched" }]]
}
```

Same idea without an explicit alias (the `fromNamed` string acts as its own identifier):

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "fromNamed": ["ex:g2"],
  "where": [["graph", "ex:g2", { "@id": "ex:s", "ex:p": "?o" }]],
  "insert": [["graph", "ex:g2", { "@id": "ex:s", "ex:q": "touched" }]]
}
```

## Simple Property Update

Update a single property value:

```bash
curl -X POST "http://localhost:8090/v1/fluree/update?ledger=mydb:main" \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "ex": "http://example.org/ns/",
      "schema": "http://schema.org/"
    },
    "where": [
      { "@id": "ex:alice", "schema:email": "?oldEmail" }
    ],
    "delete": [
      { "@id": "ex:alice", "schema:email": "?oldEmail" }
    ],
    "insert": [
      { "@id": "ex:alice", "schema:email": "alice.new@example.org" }
    ]
  }'
```

## Multiple Property Updates

Update several properties at once:

```json
{
  "where": [
    { "@id": "ex:alice", "schema:name": "?oldName" },
    { "@id": "ex:alice", "schema:email": "?oldEmail" }
  ],
  "delete": [
    { "@id": "ex:alice", "schema:name": "?oldName" },
    { "@id": "ex:alice", "schema:email": "?oldEmail" }
  ],
  "insert": [
    { "@id": "ex:alice", "schema:name": "Alice Johnson" },
    { "@id": "ex:alice", "schema:email": "alice.j@example.org" }
  ]
}
```

## Conditional Updates

Only update if condition is met:

```json
{
  "where": [
    { "@id": "ex:alice", "schema:age": "?age" },
    { "@id": "ex:alice", "ex:status": "?status" },
    ["filter", ["and", [">=", "?age", 18], ["=", "?status", "pending"]]]
  ],
  "delete": [
    { "@id": "ex:alice", "ex:status": "?status" }
  ],
  "insert": [
    { "@id": "ex:alice", "ex:status": "approved" }
  ]
}
```

The update only happens if Alice is 18+ and status is "pending".

## Pattern Matching

### Find and Update

Find entities matching a pattern and update them:

```json
{
  "where": [
    { "@id": "?person", "@type": "schema:Person" },
    { "@id": "?person", "ex:status": "pending" }
  ],
  "delete": [
    { "@id": "?person", "ex:status": "pending" }
  ],
  "insert": [
    { "@id": "?person", "ex:status": "active" }
  ]
}
```

This updates ALL people with status="pending" to status="active".

### Relationship-Based Updates

Update based on relationships:

```json
{
  "where": [
    { "@id": "?employee", "schema:worksFor": "ex:company-a" },
    { "@id": "?employee", "ex:salary": "?oldSalary" },
    ["bind", "?newSalary", ["*", "?oldSalary", 1.1]]
  ],
  "delete": [
    { "@id": "?employee", "ex:salary": "?oldSalary" }
  ],
  "insert": [
    { "@id": "?employee", "ex:salary": "?newSalary" }
  ]
}
```

Gives all company-a employees a 10% raise.

## Variable Transformation

Use variables from WHERE in INSERT with transformations:

```json
{
  "where": [
    { "@id": "ex:product-123", "ex:price": "?currentPrice" },
    ["bind", "?newPrice", ["*", "?currentPrice", 0.9]]
  ],
  "delete": [
    { "@id": "ex:product-123", "ex:price": "?currentPrice" }
  ],
  "insert": [
    { "@id": "ex:product-123", "ex:price": "?newPrice" },
    { "@id": "ex:product-123", "ex:previousPrice": "?currentPrice" }
  ]
}
```

Applies 10% discount and saves previous price.

## Partial Updates

Update only specific properties, leaving others unchanged:

**Current State:**
```text
ex:alice schema:name "Alice"
ex:alice schema:email "alice@example.org"
ex:alice schema:age 30
ex:alice schema:telephone "+1-555-0100"
```

**Update Only Age:**
```json
{
  "where": [
    { "@id": "ex:alice", "schema:age": "?oldAge" }
  ],
  "delete": [
    { "@id": "ex:alice", "schema:age": "?oldAge" }
  ],
  "insert": [
    { "@id": "ex:alice", "schema:age": 31 }
  ]
}
```

**Result:**
```text
ex:alice schema:name "Alice"              (unchanged)
ex:alice schema:email "alice@example.org" (unchanged)
ex:alice schema:age 31                     (updated)
ex:alice schema:telephone "+1-555-0100"   (unchanged)
```

## Adding Properties

Add a property without WHERE (when it might not exist):

```json
{
  "insert": [
    { "@id": "ex:alice", "schema:telephone": "+1-555-0100" }
  ]
}
```

Or conditionally add if missing:

```json
{
  "where": [
    { "@id": "ex:alice", "schema:name": "?name" },
    ["optional", { "@id": "ex:alice", "schema:telephone": "?existingPhone" }],
    ["filter", ["not", ["bound", "?existingPhone"]]]
  ],
  "insert": [
    { "@id": "ex:alice", "schema:telephone": "+1-555-0100" }
  ]
}
```

## Removing Properties

Remove a property entirely:

```json
{
  "where": [
    { "@id": "ex:alice", "schema:telephone": "?phone" }
  ],
  "delete": [
    { "@id": "ex:alice", "schema:telephone": "?phone" }
  ]
}
```

No INSERT clause—just deletes.

## Multi-Value Properties

### Replace One Value

```json
{
  "where": [
    { "@id": "ex:alice", "schema:email": "alice.old@example.org" }
  ],
  "delete": [
    { "@id": "ex:alice", "schema:email": "alice.old@example.org" }
  ],
  "insert": [
    { "@id": "ex:alice", "schema:email": "alice.new@example.org" }
  ]
}
```

### Add Value

```json
{
  "insert": [
    { "@id": "ex:alice", "schema:email": "alice.work@example.org" }
  ]
}
```

### Remove One Value

```json
{
  "where": [
    { "@id": "ex:alice", "schema:email": "alice.old@example.org" }
  ],
  "delete": [
    { "@id": "ex:alice", "schema:email": "alice.old@example.org" }
  ]
}
```

### Remove All Values

```json
{
  "where": [
    { "@id": "ex:alice", "schema:email": "?email" }
  ],
  "delete": [
    { "@id": "ex:alice", "schema:email": "?email" }
  ]
}
```

## Relationship Updates

### Change Relationship

```json
{
  "where": [
    { "@id": "ex:alice", "schema:worksFor": "?oldCompany" }
  ],
  "delete": [
    { "@id": "ex:alice", "schema:worksFor": "?oldCompany" }
  ],
  "insert": [
    { "@id": "ex:alice", "schema:worksFor": "ex:company-b" }
  ]
}
```

### Add Relationship

```json
{
  "insert": [
    { "@id": "ex:alice", "schema:knows": "ex:bob" }
  ]
}
```

### Remove Relationship

```json
{
  "where": [
    { "@id": "ex:alice", "schema:knows": "ex:bob" }
  ],
  "delete": [
    { "@id": "ex:alice", "schema:knows": "ex:bob" }
  ]
}
```

## Complex Updates

### Cascading Updates

Update related entities:

```json
{
  "where": [
    { "@id": "ex:order-123", "ex:status": "?oldStatus" },
    { "@id": "ex:order-123", "ex:items": "?item" },
    { "@id": "?item", "ex:status": "?itemStatus" }
  ],
  "delete": [
    { "@id": "ex:order-123", "ex:status": "?oldStatus" },
    { "@id": "?item", "ex:status": "?itemStatus" }
  ],
  "insert": [
    { "@id": "ex:order-123", "ex:status": "shipped" },
    { "@id": "?item", "ex:status": "shipped" }
  ]
}
```

### Computed Values

Calculate new values based on old:

```json
{
  "where": [
    { "@id": "ex:product-123", "ex:inventory": "?current" },
    { "@id": "ex:product-123", "ex:sold": "?sold" },
    ["bind", "?newInventory", ["-", "?current", "?sold"]]
  ],
  "delete": [
    { "@id": "ex:product-123", "ex:inventory": "?current" }
  ],
  "insert": [
    { "@id": "ex:product-123", "ex:inventory": "?newInventory" }
  ]
}
```

## Error Handling

### No Match

If WHERE doesn't match, nothing happens (not an error):

```json
{
  "where": [
    { "@id": "ex:nonexistent", "schema:name": "?name" }
  ],
  "delete": [...],
  "insert": [...]
}
```

Result: No changes, no error.

### Multiple Matches

If WHERE matches multiple entities, all are updated:

```json
{
  "where": [
    { "@id": "?person", "ex:status": "pending" }
  ],
  "delete": [
    { "@id": "?person", "ex:status": "pending" }
  ],
  "insert": [
    { "@id": "?person", "ex:status": "approved" }
  ]
}
```

Updates ALL entities with status="pending".

## Comparison: WHERE/DELETE/INSERT vs Replace Mode

| Feature | WHERE/DELETE/INSERT | Replace Mode |
|---------|---------------------|--------------|
| **Granularity** | Property-level | Entity-level |
| **Other properties** | Preserved | Removed |
| **Conditional** | Yes (with filters) | No |
| **Pattern matching** | Yes | No |
| **Idempotent** | Depends on logic | Yes |
| **Use case** | Partial updates | Complete replacement |

## Best Practices

### 1. Be Specific in WHERE

Good (specific):
```json
{
  "where": [
    { "@id": "ex:alice", "schema:age": "?oldAge" }
  ]
}
```

Risky (might match many):
```json
{
  "where": [
    { "@id": "?person", "schema:age": "?age" }
  ]
}
```

### 2. Always Use Variables

Use variables from WHERE in DELETE:

Good:
```json
{
  "where": [{ "@id": "ex:alice", "schema:age": "?oldAge" }],
  "delete": [{ "@id": "ex:alice", "schema:age": "?oldAge" }]
}
```

Bad (deletes all ages):
```json
{
  "where": [{ "@id": "ex:alice", "schema:age": "?oldAge" }],
  "delete": [{ "@id": "ex:alice", "schema:age": "?age" }]
}
```

### 3. Test Updates

Test on development data first:

```javascript
// Test update logic
const result = await transact(updateQuery);
console.log(`Updated ${result.flakes_retracted} values`);
```

### 4. Use Filters for Safety

Add filters to prevent unintended updates:

```json
{
  "where": [
    "...",
    ["filter", ["and", [">=", "?age", 0], ["<=", "?age", 150]]]
  ],
  "delete": [...],
  "insert": [...]
}
```

### 5. Handle No Matches

Decide if no matches should be an error in your application:

```javascript
const result = await transact(updateQuery);
if (result.flakes_retracted === 0) {
  console.warn('Update matched no entities');
}
```

### 6. Document Complex Updates

Comment complex update logic:

```javascript
// Update inventory after sale completion
// - Decrement stock by sold quantity
// - Update last-sold timestamp
// - Mark as low-stock if below threshold
const updateInventory = { ... };
```

## Performance Considerations

### Index Usage

WHERE clauses use indexes:
- Subject-based: Fast
- Predicate-based: Fast
- Pattern-based: May be slower

### Batch Updates

For many updates, consider batching:

```javascript
const updates = entities.map(e => createUpdateQuery(e));
for (const update of updates) {
  await transact(update);
}
```

## Related Documentation

- [Conditional updates (atomic / CAS patterns)](conditional-updates.md) - Increment, compare-and-swap, state machines, transfers
- [Insert](insert.md) - Adding new data
- [Upsert](upsert.md) - Replace mode
- [Retractions](retractions.md) - Removing data
- [Overview](overview.md) - Transaction overview
- [Query WHERE Clauses](../query/jsonld-query.md) - WHERE pattern syntax


---

# transactions/conditional-updates.md

# Conditional Updates (Atomic / Compare-and-Swap Patterns)

Fluree's WHERE/DELETE/INSERT transaction model supports powerful conditional update patterns that depend on the current database state. Every operation runs atomically within a single transaction — the WHERE clause reads current state, and the DELETE/INSERT templates modify it, all as one unit.

This guide covers common patterns for state-dependent updates with both **JSON-LD** and **SPARQL UPDATE** syntax.

## Key Concept: How Conditional Updates Work

```
┌──────────────────────────────────────────────────────┐
│  1. WHERE   — query current state, bind variables    │
│  2. FILTER  — guard: eliminate rows that don't pass  │
│  3. BIND    — compute new values from bound vars     │
│  4. DELETE  — retract matched triples                │
│  5. INSERT  — assert new triples                     │
│                                                      │
│  All steps execute atomically in one transaction.    │
│  If WHERE returns zero rows, nothing happens (no-op).│
└──────────────────────────────────────────────────────┘
```

The WHERE clause runs against the **current** database state. If it matches, the bound variables flow into DELETE (to retract old values) and INSERT (to assert new ones). If WHERE returns zero rows — because a FILTER eliminated them or a pattern didn't match — DELETE is skipped entirely (nothing to retract) and INSERT templates with unbound variables produce zero flakes.

### How INSERT interacts with WHERE solutions

INSERT templates are instantiated **once per WHERE solution row**, following SPARQL 1.1 Update §3.1.3:

- **WHERE returns one or more rows** — each INSERT template fires once per row. A template term bound to a WHERE variable takes that row's value; a term whose variable is unbound for that row (e.g., from an unmatched OPTIONAL) is skipped, while all-literal terms still assert.
- **WHERE returns zero rows** — nothing is inserted, *even for all-literal INSERT templates*. A guard (required pattern or `FILTER`) that matches nothing is a true no-op. This applies identically to JSON-LD and SPARQL UPDATE.
- **No WHERE clause at all** — the INSERT fires once (the plain insert path).

This means an "always insert, delete-if-exists" idiom must be written so WHERE still yields a row when the entity is absent — use `OPTIONAL`, which produces one row with the probe variable unbound, alongside the literal INSERT (see [Insert-If-Not-Exists](#6-insert-if-not-exists-conditional-create)). A bare *required* pattern that fails to match yields zero rows and therefore inserts nothing.

---

## 1. Atomic Increment / Decrement

Read the current value, compute a new one, write it back — all in one transaction. Classic use cases: counters, inventory quantities, vote tallies, loyalty points.

### JSON-LD

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "where": [
    { "@id": "ex:counter", "ex:count": "?old" },
    ["bind", "?new", "(+ ?old 1)"]
  ],
  "delete": { "@id": "ex:counter", "ex:count": "?old" },
  "insert": { "@id": "ex:counter", "ex:count": "?new" }
}
```

### SPARQL UPDATE

```sparql
PREFIX ex: <http://example.org/ns/>

DELETE { ex:counter ex:count ?old }
INSERT { ex:counter ex:count ?new }
WHERE {
  ex:counter ex:count ?old .
  BIND (?old + 1 AS ?new)
}
```

### Variations

- **Decrement**: `["bind", "?new", "(- ?old 1)"]`
- **Increment by N**: `["bind", "?new", "(+ ?old 50)"]`
- **Multiply**: `["bind", "?new", "(* ?old 2)"]`

---

## 2. Compare-and-Swap (Optimistic Concurrency)

Only update if the current value matches what the client last read. If another transaction changed the data since the read, the WHERE won't match and the update is a no-op. This is the foundation of **optimistic concurrency control**.

### JSON-LD

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "where":  { "@id": "?s", "ex:version": 1, "ex:price": "?oldPrice" },
  "delete": { "@id": "?s", "ex:version": 1, "ex:price": "?oldPrice" },
  "insert": { "@id": "?s", "ex:version": 2, "ex:price": 24.99 }
}
```

**How it works:**

1. Client reads `ex:item` and sees `version: 1, price: 19.99`
2. Client submits update pinning `version: 1` in WHERE
3. If version is still 1 → match → update succeeds, version bumps to 2
4. If another client already changed version to 2 → no match → no-op

### SPARQL UPDATE

```sparql
PREFIX ex: <http://example.org/ns/>

DELETE { ?s ex:version 1 . ?s ex:price ?oldPrice }
INSERT { ?s ex:version 2 . ?s ex:price 24.99 }
WHERE {
  ?s ex:version 1 ;
     ex:price ?oldPrice .
}
```

### Application-Level Handling

When a CAS update is a no-op (stale read), the client can detect this by checking whether `t` advanced:

```
response.t == request.t_before  →  stale read, retry with fresh data
response.t  > request.t_before  →  update succeeded
```

---

## 3. State Machine Transitions

Only allow transitions from a valid source state. Invalid transitions (e.g., trying `shipped → delivered` when the current state is `pending`) are silently rejected as no-ops.

### JSON-LD

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "where":  { "@id": "?order", "ex:status": "pending" },
  "delete": { "@id": "?order", "ex:status": "pending" },
  "insert": { "@id": "?order", "ex:status": "approved" }
}
```

This only fires when the order's current status is exactly `"pending"`. If the status is anything else, the WHERE returns zero rows and nothing changes.

### Multi-Step Chain

Chain transitions across sequential transactions:

```
pending  →  approved  →  shipped  →  delivered
```

Each step is its own transaction, each guarded by the expected source state. If any step finds the state has already moved (e.g., another process approved it), that step is a no-op.

### SPARQL UPDATE

```sparql
PREFIX ex: <http://example.org/ns/>

DELETE { ?order ex:status "pending" }
INSERT { ?order ex:status "approved" }
WHERE  { ?order ex:status "pending" }
```

---

## 4. Guarded Update (Threshold / Precondition)

Only apply a change when a numeric (or other) precondition is met. Classic use case: prevent overdrafts by checking balance before deducting.

### JSON-LD

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "where": [
    { "@id": "ex:account", "ex:balance": "?bal" },
    ["filter", "(>= ?bal 100)"],
    ["bind", "?newBal", "(- ?bal 100)"]
  ],
  "delete": { "@id": "ex:account", "ex:balance": "?bal" },
  "insert": { "@id": "ex:account", "ex:balance": "?newBal" }
}
```

**How it works:**

- If `balance >= 100` → FILTER passes → deduction applied
- If `balance < 100` → FILTER eliminates the row → no-op (overdraft prevented)

### SPARQL UPDATE

```sparql
PREFIX ex: <http://example.org/ns/>

DELETE { ex:account ex:balance ?bal }
INSERT { ex:account ex:balance ?newBal }
WHERE {
  ex:account ex:balance ?bal .
  FILTER (?bal >= 100)
  BIND (?bal - 100 AS ?newBal)
}
```

---

## 5. Atomic Transfer (Double-Entry)

Move a value between two entities atomically in a single transaction. Both the debit and credit happen together — if the guard fails, neither side is modified. Classic use cases: balance transfers, inventory moves between warehouses.

### JSON-LD

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "where": [
    { "@id": "ex:alice-acct", "ex:balance": "?aliceBal" },
    { "@id": "ex:bob-acct",   "ex:balance": "?bobBal" },
    ["filter", "(>= ?aliceBal 150)"],
    ["bind", "?newAlice", "(- ?aliceBal 150)",
             "?newBob",   "(+ ?bobBal 150)"]
  ],
  "delete": [
    { "@id": "ex:alice-acct", "ex:balance": "?aliceBal" },
    { "@id": "ex:bob-acct",   "ex:balance": "?bobBal" }
  ],
  "insert": [
    { "@id": "ex:alice-acct", "ex:balance": "?newAlice" },
    { "@id": "ex:bob-acct",   "ex:balance": "?newBob" }
  ]
}
```

### SPARQL UPDATE

```sparql
PREFIX ex: <http://example.org/ns/>

DELETE {
  ex:alice-acct ex:balance ?aliceBal .
  ex:bob-acct   ex:balance ?bobBal .
}
INSERT {
  ex:alice-acct ex:balance ?newAlice .
  ex:bob-acct   ex:balance ?newBob .
}
WHERE {
  ex:alice-acct ex:balance ?aliceBal .
  ex:bob-acct   ex:balance ?bobBal .
  FILTER (?aliceBal >= 150)
  BIND (?aliceBal - 150 AS ?newAlice)
  BIND (?bobBal + 150 AS ?newBob)
}
```

---

## 6. Insert-If-Not-Exists (Conditional Create)

Create an entity only if it doesn't already exist. Useful for preventing duplicate records.

This pattern uses OPTIONAL + FILTER to check for absence. For query-only
absence tests, `["not-exists", { ... }]` is the canonical form (see
[Negation Patterns](../query/jsonld-query.md#negation-patterns)). The
OPTIONAL/BOUND form below is shown here because the example also keeps
`?existing` available to subsequent BIND/DELETE templates when bob does
exist — `not-exists` filters rows out and exposes nothing.

### JSON-LD

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "schema": "http://schema.org/"
  },
  "where": [
    ["optional", { "@id": "ex:bob", "schema:name": "?existing" }],
    ["filter", "(not (bound ?existing))"]
  ],
  "insert": {
    "@id": "ex:bob",
    "schema:name": "Bob",
    "schema:age": 25
  }
}
```

**How it works:**

- If `ex:bob` **does not** exist: OPTIONAL leaves `?existing` unbound → `(not (bound ?existing))` is true → INSERT fires
- If `ex:bob` **exists**: OPTIONAL binds `?existing` → `(not (bound ?existing))` is false → FILTER eliminates the row → INSERT is skipped (zero solution rows = zero template instantiations)

### SPARQL UPDATE

```sparql
PREFIX ex: <http://example.org/ns/>
PREFIX schema: <http://schema.org/>

INSERT { ex:bob schema:name "Bob" ; schema:age 25 }
WHERE {
  OPTIONAL { ex:bob schema:name ?existing }
  FILTER (!BOUND(?existing))
}
```

---

## 7. Capped Accumulator (Increment with Ceiling)

Increment a value but never exceed a maximum. Useful for loyalty points, rate limits, or any bounded counter.

### JSON-LD

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "where": [
    { "@id": "ex:user", "ex:points": "?pts" },
    ["filter", "(< ?pts 1000)"],
    ["bind", "?new", "(if (> (+ ?pts 150) 1000) 1000 (+ ?pts 150))"]
  ],
  "delete": { "@id": "ex:user", "ex:points": "?pts" },
  "insert": { "@id": "ex:user", "ex:points": "?new" }
}
```

**How it works:**

- If `pts < 1000` → FILTER passes → BIND computes `min(pts + 150, 1000)` → update applied
- If `pts >= 1000` → FILTER eliminates → no-op (already at cap)

### SPARQL UPDATE

```sparql
PREFIX ex: <http://example.org/ns/>

DELETE { ex:user ex:points ?pts }
INSERT { ex:user ex:points ?new }
WHERE {
  ex:user ex:points ?pts .
  FILTER (?pts < 1000)
  BIND (IF(?pts + 150 > 1000, 1000, ?pts + 150) AS ?new)
}
```

---

## 8. Cascading / Dependent Update (Graph Traversal)

Update one entity based on values from a related entity. The WHERE clause traverses the graph to gather data from multiple nodes.

### JSON-LD

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "where": [
    { "@id": "ex:order1", "ex:customer": "?cust", "ex:total": "?orderTotal" },
    { "@id": "?cust", "ex:lifetimeSpend": "?ls" },
    ["bind", "?newLs", "(+ ?ls ?orderTotal)"]
  ],
  "delete": { "@id": "?cust", "ex:lifetimeSpend": "?ls" },
  "insert": { "@id": "?cust", "ex:lifetimeSpend": "?newLs" }
}
```

This traverses `order → customer` and accumulates the order total into the customer's lifetime spend — all atomically.

### SPARQL UPDATE

```sparql
PREFIX ex: <http://example.org/ns/>

DELETE { ?cust ex:lifetimeSpend ?ls }
INSERT { ?cust ex:lifetimeSpend ?newLs }
WHERE {
  ex:order1 ex:customer ?cust ;
            ex:total ?orderTotal .
  ?cust ex:lifetimeSpend ?ls .
  BIND (?ls + ?orderTotal AS ?newLs)
}
```

---

## 9. Batch Conditional Update (Multi-Entity)

Apply the same transformation to every entity matching a pattern. The WHERE clause acts as a filter across the dataset.

### Give All Engineers a 10% Raise

#### JSON-LD

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "where": [
    { "@id": "?emp", "ex:dept": "engineering", "ex:salary": "?sal" },
    ["bind", "?newSal", "(+ ?sal (/ ?sal 10))"]
  ],
  "delete": { "@id": "?emp", "ex:salary": "?sal" },
  "insert": { "@id": "?emp", "ex:salary": "?newSal" }
}
```

#### SPARQL UPDATE

```sparql
PREFIX ex: <http://example.org/ns/>

DELETE { ?emp ex:salary ?sal }
INSERT { ?emp ex:salary ?newSal }
WHERE {
  ?emp ex:dept "engineering" ;
       ex:salary ?sal .
  BIND (?sal + ?sal / 10 AS ?newSal)
}
```

### Batch Status Change

Approve all pending tasks in one transaction:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "where":  { "@id": "?task", "ex:status": "pending" },
  "delete": { "@id": "?task", "ex:status": "pending" },
  "insert": { "@id": "?task", "ex:status": "approved" }
}
```

Only entities with `status: "pending"` are affected; all others remain untouched.

---

## 10. Update with Audit Trail

Change a value and simultaneously record the old value for auditing — in a single atomic transaction.

### JSON-LD

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "where": [
    { "@id": "ex:product", "ex:price": "?oldPrice" },
    ["bind", "?newPrice", "(- ?oldPrice 10)"]
  ],
  "delete": { "@id": "ex:product", "ex:price": "?oldPrice" },
  "insert": {
    "@id": "ex:product",
    "ex:price": "?newPrice",
    "ex:previousPrice": "?oldPrice"
  }
}
```

After the update, the product has both its new price and a record of the previous price.

> **Note:** Fluree's immutable ledger also preserves full history via [time travel](../concepts/time-travel.md), so you can always query any prior state. This pattern is useful when you want the previous value accessible without time-travel queries.

---

## Pattern Summary

| Pattern | WHERE Matches | FILTER | BIND | Effect on No-Match |
|---------|:---:|:---:|:---:|---|
| Atomic increment | Current value | — | Compute new value | No-op |
| Compare-and-swap | Expected value | — | — | No-op (stale read) |
| State machine | Expected state | — | — | No-op (invalid transition) |
| Guarded update | Current value | Threshold check | Compute new value | No-op (guard failed) |
| Atomic transfer | Both accounts | Sender balance | Both new balances | No-op (insufficient) |
| Insert-if-not-exists | OPTIONAL probe | `not bound` | — | No-op (already exists) |
| Capped accumulator | Current value | Below cap | Min(new, cap) | No-op (at cap) |
| Cascading update | Graph traversal | — | Derived value | No-op (path broken) |
| Batch update | All matching | — | Per-entity transform | Only matching entities |
| Audit trail | Current value | — | New value | No-op |

## Best Practices

1. **Prefer pattern matching over FILTER for equality.** Pinning a value in the WHERE pattern (e.g., `"ex:status": "pending"`) is simpler and more efficient than `["filter", "(= ?st \"pending\")"]`.

2. **Check `t` to detect no-ops.** When your application needs to distinguish between "update succeeded" and "condition not met," compare `t` before and after the transaction.

3. **Use BIND for all computed values.** The `["bind", "?var", "(expression)"]` form keeps computation inside the transaction, ensuring atomicity.

4. **Test for absence with the form that fits the update.** Two idioms are
   supported. `["optional", ...]` followed by `["filter", "(not (bound ?var))"]`
   leaves the optional variables in scope for downstream BIND/DELETE/INSERT
   templates — pick this when the update needs the absent variable. For pure
   "does this exist?" guards where no downstream template needs the variable,
   `["not-exists", { ... }]` is the canonical query form (see
   [Negation Patterns](../query/jsonld-query.md#negation-patterns)). Both
   lower to the same planner strategy chooser.

5. **Leverage Fluree's immutability.** Every transaction creates an immutable commit. Even without explicit audit trail patterns, you can always query previous states using time travel. Use the audit trail pattern when you want the old value readily accessible in the current state.

## Related Documentation

- [Update (WHERE/DELETE/INSERT)](update-where-delete-insert.md) — Core syntax reference
- [Insert](insert.md) — Adding new data
- [Upsert](upsert.md) — Replace mode
- [Time travel](../concepts/time-travel.md) — Querying historical states


---

# transactions/retractions.md

# Retractions

Retractions remove data from Fluree. While data is never truly deleted (it remains in history), retractions mark triples as no longer current.

## What is a Retraction?

A **retraction** removes a triple from the current state:
- The triple existed at some point (was asserted)
- The retraction marks it as no longer true
- Historical queries can still see the triple
- Current queries don't see the triple

## Basic Retraction

Remove a specific triple:

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "schema": "http://schema.org/"
  },
  "where": [
    { "@id": "ex:alice", "schema:age": "?age" }
  ],
  "delete": [
    { "@id": "ex:alice", "schema:age": "?age" }
  ]
}
```

This removes the age property from ex:alice.

## Retract Specific Property

Remove a specific property value:

```bash
curl -X POST "http://localhost:8090/v1/fluree/update?ledger=mydb:main" \
  -H "Content-Type: application/json" \
  -d '{
    "@context": {
      "ex": "http://example.org/ns/",
      "schema": "http://schema.org/"
    },
    "where": [
      { "@id": "ex:alice", "schema:email": "alice.old@example.org" }
    ],
    "delete": [
      { "@id": "ex:alice", "schema:email": "alice.old@example.org" }
    ]
  }'
```

## Retract All Values of a Property

Remove all values for a property:

```json
{
  "where": [
    { "@id": "ex:alice", "schema:telephone": "?phone" }
  ],
  "delete": [
    { "@id": "ex:alice", "schema:telephone": "?phone" }
  ]
}
```

If ex:alice has multiple phone numbers, this removes them all.

## Retract Multiple Properties

Remove several properties at once:

```json
{
  "where": [
    { "@id": "ex:alice", "schema:email": "?email" },
    { "@id": "ex:alice", "schema:telephone": "?phone" },
    { "@id": "ex:alice", "ex:preferences": "?prefs" }
  ],
  "delete": [
    { "@id": "ex:alice", "schema:email": "?email" },
    { "@id": "ex:alice", "schema:telephone": "?phone" },
    { "@id": "ex:alice", "ex:preferences": "?prefs" }
  ]
}
```

## Retract Entire Entity

Remove all triples for an entity:

```json
{
  "where": [
    { "@id": "ex:alice", "?predicate": "?value" }
  ],
  "delete": [
    { "@id": "ex:alice", "?predicate": "?value" }
  ]
}
```

This finds all triples where ex:alice is the subject and retracts them all.

**Result:** Entity is "deleted" from current state (but remains in history).

## Conditional Retractions

Retract only if conditions are met:

```json
{
  "where": [
    { "@id": "?user", "@type": "schema:Person" },
    { "@id": "?user", "ex:lastLogin": "?lastLogin" },
    { "@id": "?user", "ex:status": "?status" }
  ],
  "filter": "?lastLogin < '2023-01-01' && ?status == 'inactive'",
  "delete": [
    { "@id": "?user", "?predicate": "?value" }
  ],
  "where": [
    { "@id": "?user", "?predicate": "?value" }
  ]
}
```

Removes all inactive users who haven't logged in since 2023.

## Retract Relationships

### Remove Single Relationship

```json
{
  "where": [
    { "@id": "ex:alice", "schema:knows": "ex:bob" }
  ],
  "delete": [
    { "@id": "ex:alice", "schema:knows": "ex:bob" }
  ]
}
```

### Remove All Relationships of a Type

```json
{
  "where": [
    { "@id": "ex:alice", "schema:knows": "?person" }
  ],
  "delete": [
    { "@id": "ex:alice", "schema:knows": "?person" }
  ]
}
```

### Bidirectional Relationship Removal

Remove relationship in both directions:

```json
{
  "where": [
    { "@id": "ex:alice", "schema:knows": "ex:bob" },
    { "@id": "ex:bob", "schema:knows": "ex:alice" }
  ],
  "delete": [
    { "@id": "ex:alice", "schema:knows": "ex:bob" },
    { "@id": "ex:bob", "schema:knows": "ex:alice" }
  ]
}
```

## Cascading Retractions

Retract an entity and all related entities:

```json
{
  "where": [
    { "@id": "ex:order-123", "ex:items": "?item" },
    { "@id": "?item", "?itemPred": "?itemVal" },
    { "@id": "ex:order-123", "?orderPred": "?orderVal" }
  ],
  "delete": [
    { "@id": "?item", "?itemPred": "?itemVal" },
    { "@id": "ex:order-123", "?orderPred": "?orderVal" }
  ]
}
```

Deletes order and all its items.

## Edge-Annotation Cascade

When a transaction retracts a base edge that has annotations attached (see [Insert: Edge Annotations](insert.md#edge-annotations)), the transactor automatically retracts the link that attaches each annotation to that edge. Without this cascade, retracted edges would still surface their annotations through `@reifies` queries.

**Base-edge retract** — fires on every annotated retract:

- The attachment linking each currently-asserted annotation to the edge is retracted in the same transaction.
- Anonymous (blank-node) annotation subjects also have their body metadata retracted, since the synthetic SID is unaddressable once the attachment is gone.
- Explicit-IRI annotation subjects keep their body metadata as ordinary RDF on the named subject (default RDF mode). To extend cleanup to explicit-IRI annotations as well, set `opts.lpgEdgeLifecycle: true` on the transaction — this matches the property-graph relationship lifecycle.

**Metadata-only retract** — fires when the user retracts every body fact of an annotation subject without touching the base edge:

- The attachment is also retracted, so the annotation is fully disposed of and inline `@annotation` queries no longer surface it.
- Same-transaction replacements (delete one body fact, insert another on the same annotation in a single update) keep the attachment — the post-transaction metadata set is non-empty, so the cascade reads "the user is updating, not removing."
- Partial retracts (some body facts gone, others still asserted) keep the attachment — the annotation is still meaningful.

The cascade is graph-aware: named-graph annotations are retracted in the same named graph as the edge they reify, never by mismatched-graph retracts.

Manage annotation lifecycle through `@annotation` and the cascade above — the [reserved system predicates](../reference/vocabulary.md#edge-annotation-predicates-reserved) that back annotations can't be written by hand on any surface.

## Soft Delete vs Hard Retraction

### Soft Delete (Recommended)

Mark as deleted without retracting:

```json
{
  "where": [
    { "@id": "ex:alice", "ex:status": "?status" }
  ],
  "delete": [
    { "@id": "ex:alice", "ex:status": "?status" }
  ],
  "insert": [
    { "@id": "ex:alice", "ex:status": "deleted" },
    { "@id": "ex:alice", "ex:deletedAt": "2024-01-22T10:30:00Z" }
  ]
}
```

**Benefits:**
- Easy to "undelete"
- Audit trail of deletion
- Can query deleted entities
- Less impact on indexes

### Hard Retraction

Retract all data:

```json
{
  "where": [
    { "@id": "ex:alice", "?predicate": "?value" }
  ],
  "delete": [
    { "@id": "ex:alice", "?predicate": "?value" }
  ]
}
```

**When to use:**
- Legal requirement to remove data
- Sensitive data that must be removed
- Test data cleanup

**Note:** Data still exists in history. For true deletion, see data purging operations.

## Pattern-Based Retractions

### Retract by Type

Remove all entities of a type:

```json
{
  "where": [
    { "@id": "?entity", "@type": "ex:TempData" },
    { "@id": "?entity", "?predicate": "?value" }
  ],
  "delete": [
    { "@id": "?entity", "?predicate": "?value" }
  ]
}
```

### Retract by Property Value

Remove entities with specific property:

```json
{
  "where": [
    { "@id": "?entity", "ex:expired": true },
    { "@id": "?entity", "?predicate": "?value" }
  ],
  "delete": [
    { "@id": "?entity", "?predicate": "?value" }
  ]
}
```

## Retraction Semantics

### Idempotent

Retracting a non-existent triple is a no-op:

```text
t=1: No triple exists
t=2: DELETE { ex:alice schema:age 30 }
     Result: No change (triple didn't exist)
```

### No Cascading by Default

Retracting an entity doesn't automatically retract references to it:

```text
t=1: ex:alice schema:worksFor ex:company-a
     ex:company-a schema:name "Acme"

t=2: DELETE all triples for ex:company-a

Result:
- ex:company-a properties are gone
- ex:alice schema:worksFor ex:company-a REMAINS
- Reference is now "dangling"
```

To cascade, explicitly match and delete references.

## Time Travel and Retractions

### Historical Queries See Retracted Data

```bash
# Current query (after retraction at t=5)
curl -X POST http://localhost:8090/v1/fluree/query \
  -d '{"from": "mydb:main", "select": ["?name"], ...}'
# Returns: [] (no results)

# Historical query (before retraction)
curl -X POST http://localhost:8090/v1/fluree/query \
  -d '{"from": "mydb:main@t:3", "select": ["?name"], ...}'
# Returns: [{"name": "Alice"}] (data visible)
```

### History Shows Retractions

Query the history to see both assertions and retractions:

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -d '{
    "@context": { "schema": "http://schema.org/" },
    "from": "mydb:main@t:1",
    "to": "mydb:main@t:latest",
    "select": ["?name", "?t", "?op"],
    "where": [
      { "@id": "ex:alice", "schema:name": { "@value": "?name", "@t": "?t", "@op": "?op" } }
    ],
    "orderBy": "?t"
  }'
```

Response:
```json
[
  ["Alice", 1, true],
  ["Alice", 5, false]
]
```

The `@t` annotation captures the transaction time and `@op` binds a boolean — `true` for assertions, `false` for retractions (mirroring `Flake.op` on disk).

## Error Handling

### Common Errors

**No Match (Not an Error):**
```json
{
  "where": [{ "@id": "ex:nonexistent", "schema:name": "?name" }],
  "delete": [{ "@id": "ex:nonexistent", "schema:name": "?name" }]
}
```
Result: No changes, no error.

**Invalid Pattern:**
```json
{
  "error": "QueryError",
  "message": "Invalid WHERE pattern",
  "code": "INVALID_PATTERN"
}
```

## Performance Considerations

### Index Updates

Retractions update all indexes:
- Each retracted triple updates SPOT, POST, OPST, PSOT
- Large retractions can impact performance
- Consider batch size for bulk deletions

### Indexing Lag

Large retractions may increase indexing lag:
- Monitor `commit_t - index_t`
- Allow time for indexing between large retractions
- Consider scheduling during low-traffic periods

### Vacuum/Compaction

Eventually, consider compaction to reclaim space from retracted data (implementation-specific).

## Best Practices

### 1. Use Soft Deletes

Prefer marking as deleted:

Good:
```json
{
  "insert": [{ "@id": "ex:alice", "ex:status": "deleted" }]
}
```

Over:
```json
{
  "delete": [{ "@id": "ex:alice", "?pred": "?val" }]
}
```

### 2. Add Audit Metadata

Include deletion metadata:

```json
{
  "insert": [
    { "@id": "ex:alice", "ex:status": "deleted" },
    { "@id": "ex:alice", "ex:deletedAt": "2024-01-22T10:30:00Z" },
    { "@id": "ex:alice", "ex:deletedBy": "user-admin" },
    { "@id": "ex:alice", "ex:deleteReason": "User request" }
  ]
}
```

### 3. Be Specific in WHERE

Avoid accidentally retracting too much:

Good:
```json
{
  "where": [{ "@id": "ex:alice", "schema:age": "?age" }],
  "delete": [{ "@id": "ex:alice", "schema:age": "?age" }]
}
```

Dangerous:
```json
{
  "where": [{ "@id": "?entity", "schema:age": "?age" }],
  "delete": [{ "@id": "?entity", "?pred": "?val" }]
}
```

### 4. Test Retractions

Test on development data:

```javascript
// Count before
const countBefore = await query('SELECT (COUNT(?e) as ?count) WHERE { ... }');

// Retract
await transact(retractionQuery);

// Count after
const countAfter = await query('SELECT (COUNT(?e) as ?count) WHERE { ... }');

console.log(`Retracted ${countBefore - countAfter} entities`);
```

### 5. Handle Cascading Explicitly

Don't rely on cascading—make it explicit:

```json
{
  "where": [
    { "@id": "ex:order-123", "?pred": "?val" },
    { "@id": "?item", "ex:orderId": "ex:order-123" },
    { "@id": "?item", "?itemPred": "?itemVal" }
  ],
  "delete": [
    { "@id": "ex:order-123", "?pred": "?val" },
    { "@id": "?item", "?itemPred": "?itemVal" }
  ]
}
```

### 6. Document Deletion Logic

Comment deletion logic:

```javascript
// Hard delete expired sessions older than 30 days
// - Finds all sessions with expired=true and oldDate
// - Retracts all properties
// - Logs count of deleted sessions
await retractExpiredSessions();
```

### 7. Monitor Impact

Track retraction metrics:
- Count of retractions
- Entities affected
- Indexing lag after large retractions
- Query performance impact

## Data Privacy Compliance

### GDPR "Right to be Forgotten"

For compliance, consider:

1. **Soft delete first** (marks as deleted)
2. **Schedule purge** (actual removal from history)
3. **Anonymize references** (replace with pseudonymous ID)

Example:
```json
{
  "where": [{ "@id": "ex:user-123", "?pred": "?val" }],
  "delete": [{ "@id": "ex:user-123", "?pred": "?val" }],
  "insert": [{
    "@id": "ex:user-123",
    "ex:anonymized": true,
    "ex:anonymizedAt": "2024-01-22T10:30:00Z"
  }]
}
```

Note: True purging from history requires administrative operations beyond standard retractions.

## Related Documentation

- [Insert](insert.md) - Adding data
- [Update](update-where-delete-insert.md) - Updating data
- [Time Travel](../concepts/time-travel.md) - Historical queries
- [History Queries](../query/jsonld-query.md) - Viewing changes over time


---

# transactions/turtle.md

# Turtle and TriG Ingest

Fluree supports ingesting RDF data in **Turtle** (Terse RDF Triple Language) and **TriG** formats. Turtle is a compact, human-readable format for RDF triples, while TriG extends Turtle to support named graphs.

## What is Turtle?

Turtle is a W3C standard format for writing RDF triples. It's more readable than XML-based formats and commonly used in the Semantic Web community.

**Example Turtle:**
```turtle
@prefix ex: <http://example.org/ns/> .
@prefix schema: <http://schema.org/> .

ex:alice a schema:Person ;
  schema:name "Alice" ;
  schema:email "alice@example.org" ;
  schema:age 30 .

ex:bob a schema:Person ;
  schema:name "Bob" ;
  schema:email "bob@example.org" .
```

## Transaction Endpoints

Fluree supports Turtle and TriG on different endpoints with different semantics:

| Endpoint | Turtle (`text/turtle`) | TriG (`application/trig`) |
|----------|------------------------|---------------------------|
| `/insert` | Supported (fast direct path) | Not supported (400 error) |
| `/upsert` | Supported | Supported |

- **Insert** (`/insert`): Pure insert semantics. Uses fast direct flake parsing. Will fail if subjects already exist with conflicting data. TriG is not supported because named graphs require the upsert path for GRAPH block extraction.
- **Upsert** (`/upsert`): For each (subject, predicate) pair, existing values are retracted before new values are asserted. Supports TriG with GRAPH blocks for named graph ingestion.

## Basic Turtle Transaction

Submit Turtle data via HTTP API:

```bash
# Insert (pure insert, fast path)
curl -X POST "http://localhost:8090/v1/fluree/insert?ledger=mydb:main" \
  -H "Content-Type: text/turtle" \
  --data-binary '@data.ttl'

# Or upsert (replace existing values)
curl -X POST "http://localhost:8090/v1/fluree/upsert?ledger=mydb:main" \
  -H "Content-Type: text/turtle" \
  --data-binary '@data.ttl'
```

**File: data.ttl**
```turtle
@prefix ex: <http://example.org/ns/> .
@prefix schema: <http://schema.org/> .

ex:alice a schema:Person ;
  schema:name "Alice" ;
  schema:email "alice@example.org" .
```

## Turtle Syntax

### Prefixes

Define namespace prefixes:

```turtle
@prefix ex: <http://example.org/ns/> .
@prefix schema: <http://schema.org/> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
```

### Basic Triples

```turtle
ex:alice schema:name "Alice" .
ex:alice schema:age 30 .
ex:alice schema:email "alice@example.org" .
```

### Semicolon Shorthand

Share subject across predicates:

```turtle
ex:alice schema:name "Alice" ;
         schema:age 30 ;
         schema:email "alice@example.org" .
```

Equivalent to three separate triples.

### Comma Shorthand

Share subject and predicate:

```turtle
ex:alice schema:email "alice@example.org" ,
                      "alice@work.com" ,
                      "alice@personal.net" .
```

Creates three triples with same subject and predicate.

### Type Shorthand

```turtle
ex:alice a schema:Person .
```

Equivalent to:
```turtle
ex:alice rdf:type schema:Person .
```

### Literals

**Plain String:**
```turtle
ex:alice schema:name "Alice" .
```

**Typed Literal:**
```turtle
ex:alice schema:age "30"^^xsd:integer .
ex:alice schema:price "29.99"^^xsd:decimal .
ex:alice schema:birthDate "1994-05-15"^^xsd:date .
```

**Language-Tagged:**
```turtle
ex:alice schema:name "Alice"@en .
ex:alice schema:name "アリス"@ja .
```

**Boolean:**
```turtle
ex:alice schema:active true .
```

**Numbers:**
```turtle
ex:alice schema:age 30 .
ex:alice schema:height 1.68 .
```

### IRIs

**Full IRI:**
```turtle
<http://example.org/ns/alice> schema:name "Alice" .
```

**Prefixed IRI:**
```turtle
ex:alice schema:name "Alice" .
```

### Blank Nodes

**Anonymous:**
```turtle
ex:alice schema:address [
  a schema:PostalAddress ;
  schema:streetAddress "123 Main St" ;
  schema:addressLocality "Springfield"
] .
```

**Labeled:**
```turtle
ex:alice schema:address _:addr1 .

_:addr1 a schema:PostalAddress ;
  schema:streetAddress "123 Main St" .
```

### Collections

**RDF Lists:**
```turtle
ex:alice schema:favoriteColors ( "red" "blue" "green" ) .
```

Equivalent to linked list structure in RDF.

## Bulk Import

### From File

```bash
curl -X POST "http://localhost:8090/v1/fluree/upsert?ledger=mydb:main" \
  -H "Content-Type: text/turtle" \
  --data-binary '@large-dataset.ttl'
```

### From URL

```bash
curl -X POST "http://localhost:8090/v1/fluree/upsert?ledger=mydb:main" \
  -H "Content-Type: text/turtle" \
  -d "@https://example.org/data.ttl"
```

### Streaming Large Files

For very large files, split into batches:

```bash
# Split large file
split -l 10000 large-dataset.ttl batch-

# Import batches
for file in batch-*; do
  curl -X POST "http://localhost:8090/v1/fluree/upsert?ledger=mydb:main" \
    -H "Content-Type: text/turtle" \
    --data-binary "@$file"
  sleep 1  # Allow indexing time
done
```

## Complete Example

```turtle
@prefix ex: <http://example.org/ns/> .
@prefix schema: <http://schema.org/> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

# Company
ex:company-a a schema:Organization ;
  schema:name "Acme Corp" ;
  schema:url <https://acme.example.com> ;
  schema:foundingDate "2000-01-15"^^xsd:date .

# People
ex:alice a schema:Person ;
  schema:name "Alice" ;
  schema:email "alice@example.org" , "alice@work.com" ;
  schema:age 30 ;
  schema:worksFor ex:company-a ;
  schema:address [
    a schema:PostalAddress ;
    schema:streetAddress "123 Main St" ;
    schema:addressLocality "Springfield" ;
    schema:postalCode "12345"
  ] .

ex:bob a schema:Person ;
  schema:name "Bob" ;
  schema:email "bob@example.org" ;
  schema:age 25 ;
  schema:worksFor ex:company-a ;
  schema:knows ex:alice .

ex:carol a schema:Person ;
  schema:name "Carol" ;
  schema:email "carol@example.org" ;
  schema:knows ex:alice , ex:bob .
```

## Format Conversion

### From JSON-LD to Turtle

Many tools can convert between formats:

```bash
# Using rapper (from Redland)
rapper -i json-ld -o turtle data.jsonld > data.ttl

# Using riot (from Apache Jena)
riot --output=turtle data.jsonld > data.ttl
```

### From RDF/XML to Turtle

```bash
rapper -i rdfxml -o turtle data.rdf > data.ttl
```

### From N-Triples to Turtle

```bash
rapper -i ntriples -o turtle data.nt > data.ttl
```

## Validation

Validate Turtle syntax before importing:

```bash
# Using rapper
rapper -i turtle -c data.ttl

# Using riot
riot --validate data.ttl
```

## Error Handling

### Syntax Errors

```json
{
  "error": "ParseError",
  "message": "Invalid Turtle syntax at line 5",
  "code": "TURTLE_PARSE_ERROR",
  "details": {
    "line": 5,
    "column": 12,
    "token": "unexpected EOF"
  }
}
```

### Invalid IRIs

```json
{
  "error": "ValidationError",
  "message": "Invalid IRI: not a valid URI",
  "code": "INVALID_IRI",
  "details": {
    "iri": "not a uri",
    "line": 8
  }
}
```

## Performance Tips

### 1. Use Batch Import

Import large datasets in batches of 10,000-100,000 triples.

### 2. Optimize Prefixes

Use short prefixes for efficiency:

Good:
```turtle
@prefix ex: <http://example.org/ns/> .
ex:alice ex:name "Alice" .
```

Less efficient:
```turtle
<http://example.org/ns/alice> <http://example.org/ns/name> "Alice" .
```

### 3. Monitor Memory

Large Turtle files consume memory during parsing. Split very large files.

### 4. Allow Indexing Time

After large imports, wait for indexing:

```bash
# Import
curl -X POST ... --data-binary '@batch.ttl'

# Wait for indexing
sleep 5

# Import next batch
curl -X POST ... --data-binary '@batch2.ttl'
```

## Best Practices

### 1. Use Standard Vocabularies

Prefer well-known vocabularies:

```turtle
@prefix schema: <http://schema.org/> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
@prefix dc: <http://purl.org/dc/terms/> .
```

### 2. Include Types

Always specify entity types:

```turtle
ex:alice a schema:Person ;
  schema:name "Alice" .
```

### 3. Use Typed Literals

Be explicit about datatypes:

```turtle
ex:alice schema:birthDate "1994-05-15"^^xsd:date ;
         schema:age "30"^^xsd:integer ;
         schema:height "1.68"^^xsd:decimal .
```

### 4. Document Namespaces

Comment your prefixes:

```turtle
# Schema.org vocabulary for general entities
@prefix schema: <http://schema.org/> .

# Application-specific namespace
@prefix ex: <http://example.org/ns/> .

# Standard XSD datatypes
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
```

### 5. Validate Before Import

Always validate Turtle syntax:

```bash
rapper -i turtle -c data.ttl
```

### 6. Split Large Files

For files > 100MB, split into smaller batches.

### 7. Include Provenance

Add metadata about the import:

```turtle
ex:dataset-import-2024-01-22 a ex:DatasetImport ;
  schema:dateCreated "2024-01-22T10:00:00Z"^^xsd:dateTime ;
  schema:author <https://example.org/users/admin> ;
  ex:sourceFile "data-2024-01.ttl" ;
  ex:recordCount 1234567 .
```

## Edge annotations (RDF 1.2 / Turtle-star)

The Turtle ingest path — and the related N-Triples (`.nt`), TriG, and N-Quads paths, which share the same lexer — is RDF 1.1 + Fluree extensions. It does **not** parse RDF 1.2 annotation tails (`{| ... |}`), the `~` reifier, or the parenthesized `<<( ... )>>` triple term. A file containing those productions **fails to parse** with a lexer error (e.g. `unexpected character '~'`; a `<<` triple term errors as a malformed IRI) — the data is rejected, not silently ingested without the annotations.

If you want to ingest edge annotations on data that lives in Turtle today, two paths work:

**Path 1: Convert to JSON-LD before ingest.** Re-emit the file as JSON-LD with `@annotation` on the value objects you want annotated. The on-disk shape after ingest is identical to what an RDF 1.2 Turtle ingest would produce.

**Path 2: Ingest plain Turtle first, then add annotations with SPARQL UPDATE.** Useful when the base edges are already in your Turtle export and the annotations come from a separate process.

```bash
# 1. Ingest the plain edges
curl -X POST "http://localhost:8090/v1/fluree/upsert?ledger=mydb:main" \
  -H "Content-Type: text/turtle" \
  --data-binary '@employments.ttl'

# 2. Add annotations via SPARQL UPDATE
curl -X POST "http://localhost:8090/v1/fluree/update?ledger=mydb:main" \
  -H "Content-Type: application/sparql-update" \
  --data-binary @- <<'SPARQL'
PREFIX ex: <http://example.org/>
INSERT DATA {
  ex:alice ex:worksFor ex:acme {| ex:role "Engineer" ; ex:since "2024-01-01" |} .
}
SPARQL
```

See the [Edge annotations concept doc](../concepts/edge-annotations.md) for the full SPARQL 1.2 surface — `~` for named reifiers, `rdf:reifies` for annotation-rooted queries, the per-operation rules for INSERT DATA / DELETE DATA / INSERT WHERE / DELETE WHERE templates, and the deferred shapes that produce parse errors.

A `.ttl`-native annotation ingest would land alongside a Turtle-star vs RDF 1.2 reifier output decision and is tracked as a future extension.

## Comparing Formats

### JSON-LD vs Turtle

**JSON-LD:**
- Native to Fluree
- Easy for JavaScript applications
- Verbose for large datasets

**Turtle:**
- More compact
- Standard in RDF community
- Better for bulk imports
- Requires conversion for JavaScript apps

### When to Use Turtle

Use Turtle for:
- Large bulk imports
- Integration with RDF tools
- Data from Semantic Web sources
- Data exchange with RDF systems

Use JSON-LD for:
- Application integration
- Real-time transactions
- JavaScript/TypeScript apps
- REST API interactions

## TriG Format (Named Graphs)

TriG extends Turtle to support **named graphs**. Each named graph groups triples under a graph IRI.

### What is TriG?

TriG (TriG RDF Triple Graph) is a W3C standard format that adds named graph support to Turtle syntax. It allows you to partition data into logical groups that can be queried independently.

### Basic TriG Syntax

```trig
@prefix ex: <http://example.org/ns/> .
@prefix schema: <http://schema.org/> .

# Default graph triples (no GRAPH block)
ex:company a schema:Organization ;
    schema:name "Acme Corp" .

# Named graph for products
GRAPH <http://example.org/graphs/products> {
    ex:widget a schema:Product ;
        schema:name "Widget" ;
        schema:price "29.99"^^xsd:decimal .

    ex:gadget a schema:Product ;
        schema:name "Gadget" ;
        schema:price "49.99"^^xsd:decimal .
}

# Named graph for inventory
GRAPH <http://example.org/graphs/inventory> {
    ex:widget schema:inventory 42 ;
        schema:warehouse "main" .

    ex:gadget schema:inventory 15 ;
        schema:warehouse "secondary" .
}
```

### Submitting TriG Data

TriG is only supported on the **upsert** endpoint (or transact). Use the `application/trig` content type:

```bash
# TriG requires upsert (for named graph support)
curl -X POST "http://localhost:8090/v1/fluree/upsert?ledger=mydb:main" \
  -H "Content-Type: application/trig" \
  --data-binary '@data.trig'
```

TriG on the `/insert` endpoint will return a 400 error because named graph extraction requires the upsert path.

### Querying Named Graphs

After ingesting TriG data, query specific graphs using JSON-LD with the structured `from` object:

```json
{
  "@context": { "schema": "http://schema.org/" },
  "from": {
    "@id": "mydb:main",
    "graph": "http://example.org/graphs/products"
  },
  "select": ["?name", "?price"],
  "where": [
    { "@id": "?product", "schema:name": "?name" },
    { "@id": "?product", "schema:price": "?price" }
  ]
}
```

For cross-graph queries, use `fromNamed` with aliases:

```json
{
  "@context": { "schema": "http://schema.org/" },
  "from": "mydb:main",
  "fromNamed": [
    { "@id": "mydb:main", "alias": "products", "graph": "http://example.org/graphs/products" },
    { "@id": "mydb:main", "alias": "inventory", "graph": "http://example.org/graphs/inventory" }
  ],
  "select": ["?name", "?inventory", "?warehouse"],
  "where": [
    ["graph", "products", { "@id": "?product", "schema:name": "?name" }],
    ["graph", "inventory", { "@id": "?product", "schema:inventory": "?inventory", "schema:warehouse": "?warehouse" }]
  ]
}
```

### Graph IDs

Fluree assigns internal graph IDs to named graphs:

| Graph ID | Purpose |
|----------|---------|
| 0 | Default graph (triples without GRAPH block) |
| 1 | txn-meta (commit metadata) |
| 2+ | User-defined named graphs |

### TriG with Transaction Metadata

You can combine named graphs with transaction metadata using the special `#txn-meta` graph fragment:

```trig
@prefix ex: <http://example.org/ns/> .
@prefix f: <https://ns.flur.ee/db#> .

# Transaction metadata (stored in txn-meta graph)
GRAPH <#txn-meta> {
    fluree:commit:this ex:jobId "batch-import-001" ;
        ex:source "warehouse-export" ;
        ex:operator "system-admin" .
}

# User data in named graph
GRAPH <http://example.org/graphs/products> {
    ex:widget a ex:Product ;
        ex:name "Widget" .
}
```

### Limits

- Maximum 256 named graphs per transaction
- Maximum 8KB per graph IRI
- Named graphs are queryable after indexing completes

### When to Use TriG

Use TriG when you need to:
- Partition data into logical groups
- Separate data by source, tenant, or domain
- Maintain provenance at the graph level
- Integrate with RDF quad stores

Use plain Turtle when:
- All data belongs in the default graph
- Graph partitioning isn't needed
- Working with simpler data models

## Bulk import (Rust API)

For high-throughput ingest of large Turtle datasets into a **fresh ledger**, prefer the bulk import
pipeline exposed by `fluree-db-api`:

- See: [Using Fluree as a Rust library → Bulk import Turtle chunks](../getting-started/rust-api.md#bulk-import-turtle-chunks-high-throughput)

This pipeline:
- Parses Turtle in parallel, but **writes commits serially** (hash-linked commit chain).
- Streams run generation during import and builds multi-order binary indexes (SPOT/PSOT/POST/OPST).
- Writes an index root to CAS and publishes it to the nameservice so queries can use the normal
  `db()` / `query()` path.
  
Temporary `tmp_import/` session files are cleaned up on success (configurable).

## Tools and Libraries

### Command-Line Tools

**Rapper (Redland):**
```bash
# Install on macOS
brew install redland

# Parse Turtle
rapper -i turtle data.ttl
```

**Riot (Apache Jena):**
```bash
# Install
# Download from https://jena.apache.org/

# Validate
riot --validate data.ttl
```

### Programming Libraries

**JavaScript/TypeScript:**
```javascript
import { Parser } from 'n3';

const parser = new Parser();
const quads = parser.parse(turtleString);
```

**Python:**
```python
from rdflib import Graph

g = Graph()
g.parse('data.ttl', format='turtle')
```

**Java:**
```java
import org.apache.jena.rdf.model.*;

Model model = ModelFactory.createDefaultModel();
model.read("data.ttl", "TURTLE");
```

## Related Documentation

- [Insert](insert.md) - Adding data via JSON-LD
- [Overview](overview.md) - Transaction overview
- [Datasets and Named Graphs](../concepts/datasets-and-named-graphs.md) - Named graph concepts
- [Data Types](../concepts/datatypes.md) - Supported datatypes
- [API Headers](../api/headers.md) - Content-Type specifications


---

# transactions/signed-transactions.md

# Signed / Credentialed Transactions

Fluree supports cryptographically signed transactions using **JSON Web Signatures (JWS)** and **Verifiable Credentials (VC)**. Signed transactions provide authentication, integrity, and non-repudiation for all transaction operations.

## Why Sign Transactions?

Signed transactions provide:

- **Authentication**: Prove who submitted the transaction
- **Integrity**: Ensure transaction hasn't been tampered with
- **Non-repudiation**: Transaction author cannot deny authorship
- **Authorization**: Link transaction to specific identity for policy enforcement
- **Audit Trail**: Complete provenance of all data changes

## Basic Signed Transaction

### Step 1: Create Transaction

Create your transaction as normal:

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "schema": "http://schema.org/"
  },
  "@graph": [
    {
      "@id": "ex:alice",
      "@type": "schema:Person",
      "schema:name": "Alice"
    }
  ]
}
```

### Step 2: Sign with JWS

Sign the transaction using JWS:

```javascript
import jose from 'jose';

const privateKey = ... // Your Ed25519 private key

const jws = await new jose.SignJWT(transaction)
  .setProtectedHeader({
    alg: 'EdDSA',
    kid: 'did:key:z6MkhaXgBZDvotDkL5257faiztiGiC2QtKLGpbnnEGta2doK'
  })
  .setIssuedAt()
  .setExpirationTime('15m')
  .sign(privateKey);
```

### Step 3: Submit

Submit the signed transaction:

```bash
curl -X POST "http://localhost:8090/v1/fluree/upsert?ledger=mydb:main" \
  -H "Content-Type: application/jose" \
  -d "$jws"
```

## JWS Format

### Compact Serialization

```text
eyJhbGciOiJFZDI1NTE5IiwidHlwIjoiSldUIn0.eyJAY29udGV4dCI6eyJleCI6Imh0...
```

Three base64url-encoded parts separated by dots:
1. Header (algorithm, key ID)
2. Payload (transaction)
3. Signature

### JSON Serialization

```json
{
  "payload": "eyJAY29udGV4dCI6eyJleCI6Imh0...",
  "signatures": [
    {
      "protected": "eyJhbGciOiJFZDI1NTE5In0",
      "signature": "c2lnbmF0dXJl..."
    }
  ]
}
```

## Verifiable Credentials

Use W3C Verifiable Credentials for transactions:

```json
{
  "@context": [
    "https://www.w3.org/2018/credentials/v1"
  ],
  "type": ["VerifiableCredential"],
  "issuer": "did:key:z6Mkh...",
  "issuanceDate": "2024-01-22T10:00:00Z",
  "credentialSubject": {
    "id": "did:key:z6Mkh...",
    "flureeTransaction": {
      "@context": {
        "ex": "http://example.org/ns/"
      },
      "@graph": [
        { "@id": "ex:alice", "schema:name": "Alice" }
      ]
    }
  },
  "proof": {
    "type": "Ed25519Signature2020",
    "created": "2024-01-22T10:00:00Z",
    "verificationMethod": "did:key:z6Mkh...#z6Mkh...",
    "proofPurpose": "authentication",
    "proofValue": "z58DAdFfa9SkqZMVP..."
  }
}
```

Submit with:
```bash
curl -X POST "http://localhost:8090/v1/fluree/upsert?ledger=mydb:main" \
  -H "Content-Type: application/vc+ld+json" \
  -d @credential.json
```

## Supported Algorithm

**EdDSA (Ed25519):**
- Fast, secure, deterministic
- 64-byte signatures
- 128-bit security level

## Identity Management

### Decentralized Identifiers (DIDs)

Use DIDs to identify transaction authors:

**did:key** (simplest):
```
did:key:z6MkhaXgBZDvotDkL5257faiztiGiC2QtKLGpbnnEGta2doK
```

**did:web** (organization-managed):
```
did:web:example.com:users:alice
```

**did:ion** (blockchain-based):
```
did:ion:EiClkZMDxPKqC9c-umQfTkR8vvZ9JPhl_xLDI9Nfk38w5w
```

### Key Resolution

Standalone server signed requests verify Ed25519 JWS material from the request
itself (for example embedded JWK / `did:key`) or configured OIDC/JWKS issuers.
There is no `/admin/keys` registration endpoint.

## Transaction Provenance

Signed transactions include author information in commit metadata:

```json
{
  "t": 42,
  "timestamp": "2024-01-22T10:30:00Z",
  "commit_id": "bafybeig...commitT42",
  "author": "did:key:z6Mkh...",
  "signature": "z58DAdFfa9...",
  "flakes_added": 3,
  "flakes_retracted": 0
}
```

Query provenance:

```sparql
PREFIX f: <https://ns.flur.ee/db#>

SELECT ?t ?author ?timestamp
WHERE {
  ?commit f:t ?t ;
          f:author ?author ;
          f:timestamp ?timestamp .
}
ORDER BY DESC(?t)
```

## Policy-Based Authorization

Use signed transaction author for authorization:

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "f": "https://ns.flur.ee/db#"
  },
  "@id": "ex:admin-policy",
  "f:policy": [
    {
      "f:subject": "did:key:z6MkhaXgBZDvotDkL5257faiztiGiC2QtKLGpbnnEGta2doK",
      "f:action": "transact",
      "f:allow": true
    }
  ]
}
```

Only transactions signed by this DID will be accepted.

## Code Examples

### JavaScript/TypeScript

```typescript
import jose from 'jose';
import { Ed25519VerificationKey2020 } from '@digitalbazaar/ed25519-verification-key-2020';

async function signTransaction(transaction: object, privateKey: Uint8Array) {
  const jws = await new jose.SignJWT(transaction)
    .setProtectedHeader({
      alg: 'EdDSA',
      kid: 'did:key:z6Mkh...'
    })
    .setIssuedAt()
    .setExpirationTime('15m')
    .sign(privateKey);
  
  return jws;
}

async function submitSignedTransaction(ledger: string, transaction: object) {
  const signed = await signTransaction(transaction, privateKey);
  
  const response = await fetch(`http://localhost:8090/v1/fluree/upsert?ledger=${ledger}`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/jose' },
    body: signed
  });
  
  return await response.json();
}
```

### Python

```python
from jwcrypto import jwk, jws
import json

def sign_transaction(transaction, private_key):
    # Create JWK from private key
    key = jwk.JWK.from_json(private_key)
    
    # Create JWS
    payload = json.dumps(transaction).encode('utf-8')
    jws_token = jws.JWS(payload)
    jws_token.add_signature(
        key,
        alg='EdDSA',
        protected=json.dumps({"kid": "did:key:z6Mkh..."})
    )
    
    return jws_token.serialize()

def submit_signed_transaction(ledger, transaction, private_key):
    signed = sign_transaction(transaction, private_key)
    
    response = requests.post(
        f'http://localhost:8090/v1/fluree/upsert?ledger={ledger}',
        headers={'Content-Type': 'application/jose'},
        data=signed
    )
    
    return response.json()
```

## Verification Process

When Fluree receives a signed transaction:

1. **Extract signature and header**
2. **Resolve key ID (kid) to public key**
3. **Verify signature** using public key
4. **Check expiration** (if exp claim present)
5. **Validate issuer** (if required by policy)
6. **Apply authorization policies** based on DID
7. **Process transaction** if verification succeeds

## Error Handling

### Invalid Signature

```json
{
  "error": "SignatureVerificationFailed",
  "message": "Invalid signature",
  "code": "INVALID_SIGNATURE",
  "details": {
    "kid": "did:key:z6Mkh...",
    "reason": "Signature does not match"
  }
}
```

### Expired Transaction

```json
{
  "error": "TokenExpired",
  "message": "Transaction signature expired",
  "code": "TOKEN_EXPIRED",
  "details": {
    "exp": 1642857600,
    "now": 1642858000
  }
}
```

### Key Not Found

```json
{
  "error": "KeyNotFound",
  "message": "Public key not registered",
  "code": "KEY_NOT_FOUND",
  "details": {
    "kid": "did:key:z6Mkh..."
  }
}
```

### Unauthorized

```json
{
  "error": "Forbidden",
  "message": "Policy denies transact permission",
  "code": "POLICY_DENIED",
  "details": {
    "subject": "did:key:z6Mkh...",
    "action": "transact",
    "ledger": "mydb:main"
  }
}
```

## Best Practices

### 1. Use EdDSA (Ed25519)

Best security and performance:
```javascript
{
  "alg": "EdDSA",
  "kid": "did:key:z6Mkh..."
}
```

### 2. Set Expiration

Always include expiration:
```javascript
.setExpirationTime('15m')  // 15 minutes
```

### 3. Secure Key Storage

Never hardcode private keys:

Good:
```javascript
const privateKey = await loadKeyFromSecureStorage();
```

Bad:
```javascript
const privateKey = "hardcoded-key-here";
```

### 4. Use did:key for Simplicity

For simple deployments:
```
did:key:z6MkhaXgBZDvotDkL5257faiztiGiC2QtKLGpbnnEGta2doK
```

### 5. Implement Key Rotation

Rotate keys every 90-180 days:

```javascript
async function rotateKey() {
  const newKey = generateKeyPair();
  await registerKey(newKey.publicKey);
  await revokeKey(oldKey.kid);
  updateApplicationKey(newKey);
}
```

### 6. Include Request ID

Add unique ID to prevent replay:

```javascript
.setClaim('jti', crypto.randomUUID())
```

### 7. Use HTTPS

Always use HTTPS with signed transactions to prevent replay attacks.

## Compliance and Auditing

### Complete Audit Trail

Signed transactions provide complete audit trail:

```sparql
SELECT ?t ?author ?timestamp ?action
WHERE {
  ?commit f:t ?t ;
          f:author ?author ;
          f:timestamp ?timestamp .
  ?commit f:assert ?assertion .
  ?assertion ?predicate ?object .
}
ORDER BY DESC(?t)
```

### Regulatory Compliance

Signed transactions support:
- SOC 2 (audit trails)
- HIPAA (data provenance)
- GDPR (data processing records)
- PCI DSS (transaction logs)

### Non-Repudiation

Cryptographic signatures provide non-repudiation:
- Author cannot deny submitting transaction
- Tampering is detectable
- Legal admissibility in disputes

## Related Documentation

- [API: Signed Requests](../api/signed-requests.md) - HTTP API details
- [Commit Signing and Attestation](../security/commit-signing.md) - Infrastructure-level commit signatures
- [Security: Policy Model](../security/policy-model.md) - Authorization policies
- [Verifiable Data](../concepts/verifiable-data.md) - Cryptographic verification concepts
- [Commit Receipts](commit-receipts.md) - Transaction metadata


---

# transactions/commit-receipts.md

# Commit Receipts and tx-id

Every successful transaction returns a **commit receipt** containing metadata about the transaction. This receipt provides important information for tracking, auditing, and referencing transactions.

## Commit Receipt Structure

Basic commit receipt:

```json
{
  "t": 42,
  "timestamp": "2024-01-22T10:30:00.000Z",
  "commit_id": "bafybeig...commitT42",
  "flakes_added": 15,
  "flakes_retracted": 3,
  "previous_commit_id": "bafybeig...commitT41"
}
```

## Receipt Fields

### Transaction Time (t)

The **transaction time** is a monotonically increasing integer uniquely identifying this transaction:

```json
{
  "t": 42
}
```

**Properties:**
- Unique across all ledgers in the Fluree instance
- Monotonically increasing (never decreases)
- Used for time travel queries
- Basis for temporal ordering

**Usage:**

```bash
# Query at specific transaction
curl -X POST http://localhost:8090/v1/fluree/query \
  -d '{"from": "mydb:main@t:42", ...}'
```

**Read-after-write consistency:** The `t` value is the key to ensuring queries
see freshly committed data. Pass it as `min_t` to `refresh()` to gate queries
on a minimum transaction time. See [Time Travel — Consistency and Read-After-Write](../concepts/time-travel.md#consistency-and-read-after-write) for details.

### Timestamp

ISO 8601 formatted timestamp of when the transaction was committed:

```json
{
  "timestamp": "2024-01-22T10:30:00.000Z"
}
```

**Properties:**
- UTC timezone
- Millisecond precision
- Server-assigned (not client-provided)
- Monotonic (within same transaction time ordering)

**Usage:**

```bash
# Query at specific time
curl -X POST http://localhost:8090/v1/fluree/query \
  -d '{"from": "mydb:main@iso:2024-01-22T10:30:00Z", ...}'
```

### Commit ID

Content-addressed identifier for the commit:

```json
{
  "commit_id": "bafybeig...commitT42"
}
```

**Properties:**
- CIDv1 value (base32-lower multibase string)
- Derived from the commit's canonical bytes via SHA-256
- Storage-agnostic -- does not depend on where the commit is stored
- Can be used to fetch the commit from any content store

**Usage:**

```bash
# Query at specific commit
curl -X POST http://localhost:8090/v1/fluree/query \
  -d '{"from": "mydb:main@commit:bafybeig...commitT42", ...}'
```

### Flake Counts

Number of triples added and retracted:

```json
{
  "flakes_added": 15,
  "flakes_retracted": 3
}
```

**flakes_added:** Number of new triples asserted
**flakes_retracted:** Number of existing triples removed

Net change: `flakes_added - flakes_retracted`

### Previous Commit

ContentId of the previous commit (forms a chain):

```json
{
  "previous_commit_id": "bafybeig...commitT41"
}
```

**Properties:**
- Links to parent commit by ContentId
- Forms immutable commit chain
- Enables commit history traversal
- `null` for first transaction (t=1)

## Extended Receipt Fields

### Author (Signed Transactions)

For signed transactions, includes author DID:

```json
{
  "t": 42,
  "author": "did:key:z6MkhaXgBZDvotDkL5257faiztiGiC2QtKLGpbnnEGta2doK",
  "signature": "z58DAdFfa9SkqZMVP...",
  ...
}
```

### Message

Optional commit message (if provided):

```json
{
  "t": 42,
  "message": "Add new customer records for Q1 2024",
  ...
}
```

### Ledger

Ledger ID:

```json
{
  "t": 42,
  "ledger": "mydb:main",
  ...
}
```

### Duration

Transaction processing time in milliseconds:

```json
{
  "t": 42,
  "duration_ms": 45,
  ...
}
```

## Using Transaction IDs

### Referencing Transactions

Store transaction ID for later reference:

```javascript
const receipt = await transact({
  "@graph": [{ "@id": "ex:alice", "schema:name": "Alice" }]
});

// Store for audit trail
await logTransaction({
  entity: "ex:alice",
  operation: "create",
  transactionId: receipt.t,
  timestamp: receipt.timestamp
});
```

### Historical Queries

Query data at specific transaction:

```javascript
// Get data as it was at transaction 42
const historicalData = await query({
  from: `mydb:main@t:${receipt.t}`,
  select: ["?name"],
  where: [{ "@id": "ex:alice", "schema:name": "?name" }]
});
```

### Commit Verification

Verify commit integrity by re-deriving the ContentId from fetched bytes:

```javascript
async function verifyCommit(receipt) {
  const bytes = await contentStore.get(receipt.commit_id);
  const derivedCid = computeContentId("Commit", bytes);

  if (derivedCid !== receipt.commit_id) {
    throw new Error('Commit integrity violation!');
  }
}
```

## Commit Chain

Commits form an immutable chain:

```text
t=1 (cid:aaa) ← t=2 (cid:bbb) ← t=3 (cid:ccc) ← t=4 (cid:ddd)
  ↑                ↑                ↑                ↑
  |                |                |                |
previous=null   previous=aaa    previous=bbb    previous=ccc
```

### Traversing History

Walk the commit chain:

```javascript
async function getCommitHistory(ledger, fromT, toT) {
  const history = [];
  let currentT = fromT;
  
  while (currentT >= toT) {
    const commit = await getCommit(ledger, currentT);
    history.push(commit);
    currentT = commit.previous_t;
  }
  
  return history;
}
```

## Querying Commit Metadata

### SPARQL Query for Commits

```sparql
PREFIX f: <https://ns.flur.ee/db#>

SELECT ?t ?timestamp ?commitId ?author
WHERE {
  ?commit a f:Commit ;
          f:t ?t ;
          f:timestamp ?timestamp ;
          f:commitId ?commitId .
  OPTIONAL { ?commit f:author ?author }
}
ORDER BY DESC(?t)
LIMIT 10
```

### JSON-LD Query for Recent Commits

```json
{
  "@context": {
    "f": "https://ns.flur.ee/db#"
  },
  "select": ["?t", "?timestamp", "?commitId"],
  "where": [
    { "@id": "?commit", "@type": "f:Commit" },
    { "@id": "?commit", "f:t": "?t" },
    { "@id": "?commit", "f:timestamp": "?timestamp" },
    { "@id": "?commit", "f:commitId": "?commitId" }
  ],
  "orderBy": ["-?t"],
  "limit": 10
}
```

## Receipt Storage

### Application Database

Store receipts in your application database:

```sql
CREATE TABLE transaction_receipts (
  id SERIAL PRIMARY KEY,
  ledger VARCHAR(255),
  transaction_t INTEGER,
  commit_id TEXT,
  timestamp TIMESTAMP,
  flakes_added INTEGER,
  flakes_retracted INTEGER,
  author VARCHAR(255),
  created_at TIMESTAMP DEFAULT NOW()
);
```

### Document Store

Store as JSON documents:

```javascript
await mongodb.collection('receipts').insertOne({
  ledger: receipt.ledger,
  t: receipt.t,
  commit_id: receipt.commit_id,
  timestamp: receipt.timestamp,
  flakes: {
    added: receipt.flakes_added,
    retracted: receipt.flakes_retracted
  },
  metadata: {
    author: receipt.author,
    duration_ms: receipt.duration_ms
  }
});
```

### Time-Series Database

For analytics:

```javascript
await influxdb.writePoint({
  measurement: 'transactions',
  tags: { ledger: receipt.ledger },
  fields: {
    t: receipt.t,
    flakes_added: receipt.flakes_added,
    flakes_retracted: receipt.flakes_retracted,
    duration_ms: receipt.duration_ms
  },
  timestamp: new Date(receipt.timestamp)
});
```

## Audit Trail

### Transaction Log

Build complete audit log from receipts:

```javascript
async function buildAuditLog(ledger, startDate, endDate) {
  const receipts = await fetchReceipts(ledger, startDate, endDate);
  
  return receipts.map(r => ({
    time: r.timestamp,
    transactionId: r.t,
    author: r.author || 'anonymous',
    changes: {
      added: r.flakes_added,
      removed: r.flakes_retracted
    },
    commit: r.commit_id,
    verifiable: true
  }));
}
```

### Compliance Reports

Generate compliance reports:

```javascript
async function generateComplianceReport(ledger, period) {
  const receipts = await fetchReceipts(ledger, period.start, period.end);
  
  return {
    period: period,
    totalTransactions: receipts.length,
    totalChanges: receipts.reduce((sum, r) => sum + r.flakes_added, 0),
    authors: [...new Set(receipts.map(r => r.author))],
    verifiedChain: verifyCommitChain(receipts)
  };
}
```

## Performance Monitoring

### Transaction Metrics

Track transaction performance:

```javascript
function analyzeReceipts(receipts) {
  const durations = receipts.map(r => r.duration_ms);
  const sizes = receipts.map(r => r.flakes_added + r.flakes_retracted);
  
  return {
    avgDuration: average(durations),
    maxDuration: Math.max(...durations),
    avgSize: average(sizes),
    maxSize: Math.max(...sizes),
    throughput: receipts.length / (period.hours)
  };
}
```

### Alert on Anomalies

```javascript
function checkForAnomalies(receipt) {
  if (receipt.duration_ms > 1000) {
    alert(`Slow transaction: ${receipt.t} took ${receipt.duration_ms}ms`);
  }
  
  if (receipt.flakes_added > 10000) {
    alert(`Large transaction: ${receipt.t} added ${receipt.flakes_added} flakes`);
  }
}
```

## Best Practices

### 1. Always Store Receipts

Store transaction receipts for audit trail:

```javascript
const receipt = await transact(transaction);
await storeReceipt(receipt);
```

### 2. Verify Commit Chain

Periodically verify commit chain integrity:

```javascript
async function verifyChainIntegrity(ledger) {
  const receipts = await fetchAllReceipts(ledger);
  
  for (let i = 1; i < receipts.length; i++) {
    if (receipts[i].previous_commit_id !== receipts[i-1].commit_id) {
      throw new Error(`Chain broken at t=${receipts[i].t}`);
    }
  }
}
```

### 3. Use Transaction IDs for References

Store transaction IDs rather than timestamps:

Good:
```javascript
{ entity: "ex:alice", createdAt_t: 42 }
```

Less reliable:
```javascript
{ entity: "ex:alice", createdAt: "2024-01-22T10:30:00Z" }
```

### 4. Monitor Performance

Track receipt metadata for performance insights:

```javascript
const avgDuration = receipts.reduce((sum, r) => sum + r.duration_ms, 0) / receipts.length;
```

### 5. Include in Error Handling

Log receipt info on errors:

```javascript
try {
  const receipt = await transact(transaction);
  logger.info(`Transaction successful: t=${receipt.t}`);
} catch (err) {
  logger.error(`Transaction failed`, {
    error: err.message,
    transaction: transaction
  });
}
```

## Related Documentation

- [Overview](overview.md) - Transaction overview
- [Signed Transactions](signed-transactions.md) - Transaction signing
- [Commit Signing and Attestation](../security/commit-signing.md) - Commit-level signatures
- [Time Travel](../concepts/time-travel.md) - Historical queries
- [Indexing Side-Effects](indexing-side-effects.md) - Indexing behavior


---

# transactions/indexing-side-effects.md

# Indexing Side-Effects

Transactions in Fluree trigger background indexing processes that build query-optimized data structures. Understanding these side-effects is crucial for performance tuning and capacity planning.

## What is Indexing?

**Indexing** is the process of building query-optimized data structures from transaction data. Fluree maintains four index permutations (SPOT, POST, OPST, PSOT) that enable efficient query execution.

### Commit vs Index

**Commit (immediate):**
- Transaction written to log
- Small, append-only files
- Published to nameservice immediately
- Available for time travel queries

**Index (asynchronous):**
- Query-optimized structures built
- Background process
- Published to nameservice when complete
- May lag behind commits

## Index Structure

Fluree maintains four index permutations:

### SPOT (Subject-Predicate-Object-Time)

```text
ex:alice → schema:name → "Alice" → [t=1, t=5, t=10]
ex:alice → schema:age → 30 → [t=1]
ex:alice → schema:age → 31 → [t=10]
```

Optimized for: "What are all properties of this subject?"

### POST (Predicate-Object-Subject-Time)

```text
schema:name → "Alice" → ex:alice → [t=1, t=5, t=10]
schema:age → 30 → ex:alice → [t=1]
schema:age → 31 → ex:alice → [t=10]
```

Optimized for: "What subjects have this property/value?"

### OPST (Object-Predicate-Subject-Time)

```text
"Alice" → schema:name → ex:alice → [t=1, t=5, t=10]
30 → schema:age → ex:alice → [t=1]
31 → schema:age → ex:alice → [t=10]
```

Optimized for: "What subjects have this value?"

### PSOT (Predicate-Subject-Object-Time)

```text
schema:name → ex:alice → "Alice" → [t=1, t=5, t=10]
schema:age → ex:alice → 30 → [t=1]
schema:age → ex:alice → 31 → [t=10]
```

Optimized for: "What are all values for this predicate?"

## Indexing Pipeline

### 1. Transaction Commit

```text
t=42: Transaction committed
  - Flakes written to transaction log
  - Commit published to nameservice
  - commit_t updated to 42
```

### 2. Index Trigger

Background indexing process detects new commits:
```text
Indexer: commit_t=42, index_t=40
Indexer: Need to index t=41, t=42
```

### 3. Index Building

Process transactions to build indexes:
```text
For each flake in t=41, t=42:
  - Update SPOT index
  - Update POST index
  - Update OPST index
  - Update PSOT index
```

### 4. Index Publication

When complete, publish new index:
```text
  - Write index snapshot to storage
  - Publish index_id to nameservice
  - Update index_t to 42
```

## Novelty Layer

The **novelty layer** is the gap between indexed and committed data:

```text
commit_t = 45
index_t = 40
novelty layer = [t=41, t=42, t=43, t=44, t=45]
```

### Query Execution with Novelty

Queries combine index + novelty:

```text
Query Result = Indexed Data (t ≤ 40) + Novelty Layer (41 ≤ t ≤ 45)
```

**Performance Impact:**
- Small novelty: Fast queries (mostly indexed)
- Large novelty: Slower queries (more transaction replay)

## Indexing Performance

### Transaction Size Impact

Larger transactions take longer to index:

```text
Transaction with 10 flakes:
  - 10 flakes × 4 indexes = 40 index updates
  - Indexing time: ~1ms

Transaction with 10,000 flakes:
  - 10,000 flakes × 4 indexes = 40,000 index updates
  - Indexing time: ~100ms
```

### Indexing Rate

Typical indexing rates:

```text
Light load:
  - 1,000 flakes/second
  - ~10 moderate transactions/second

Heavy load:
  - 10,000 flakes/second
  - ~100 moderate transactions/second
```

Actual rates depend on:
- Hardware (CPU, disk I/O)
- Storage backend (memory, file, AWS)
- Transaction patterns
- System load

## Monitoring Indexing

### Check Indexing Status

```bash
curl http://localhost:8090/v1/fluree/info/mydb:main
```

Response:
```json
{
  "ledger_id": "mydb:main",
  "commit_t": 150,
  "index_t": 140
}
```

**Indexing lag (txns):** `commit_t - index_t` = number of unindexed transactions

### Healthy vs Unhealthy

**Healthy:**
```text
commit_t = 1000
index_t = 998
novelty = 2 transactions (good!)
```

**Unhealthy:**
```text
commit_t = 1000
index_t = 850
novelty = 150 transactions (indexing lag!)
```

## Indexing Lag

**Indexing lag** occurs when indexing can't keep up with transaction rate.

### Causes

1. **High Transaction Rate**
   - More transactions than indexing can handle
   - Sustained write load

2. **Large Transactions**
   - Individual transactions with many flakes
   - Bulk imports

3. **Resource Constraints**
   - CPU bottleneck
   - Disk I/O bottleneck
   - Memory pressure

4. **Storage Backend Latency**
   - Slow storage (network attached)
   - AWS S3 latency

### Impact

Large indexing lag affects:

**Query Performance:**
- More novelty to replay
- Slower query execution
- Higher CPU usage for queries

**Memory Usage:**
- Novelty layer held in memory
- Larger memory footprint

**Backup/Recovery:**
- Larger gap to replay
- Longer recovery times

## Tuning Indexing

Background indexing is controlled primarily by:

- Enabling/disabling background indexing (`--indexing-enabled` / `FLUREE_INDEXING_ENABLED`)
- Novelty thresholds that trigger indexing / apply backpressure (`--reindex-min-bytes`, `--reindex-max-bytes`)

See [Operations: Configuration](../operations/configuration.md#background-indexing) and [Background Indexing](../indexing-and-search/background-indexing.md) for the canonical settings and tuning guidance.

### 4. Dedicated Indexing Process

For high-load deployments, run dedicated indexer:

```bash
# Main server (transact only; background indexing disabled)
fluree-server --indexing-enabled=false

# Indexing server
./fluree-db-indexer --ledgers mydb:main,mydb:dev
```

## Transaction Patterns and Indexing

### Batch Transactions

Good pattern:
```javascript
// Batch into reasonable sizes
const batchSize = 1000;
for (let i = 0; i < entities.length; i += batchSize) {
  const batch = entities.slice(i, i + batchSize);
  await transact({ "@graph": batch });
  
  // Allow indexing time
  if (i % (batchSize * 10) === 0) {
    await sleep(1000);
  }
}
```

Bad pattern:
```javascript
// Single giant transaction
await transact({ "@graph": allEntities });  // 1 million entities!
```

### Continuous Transactions

For continuous transaction load:

```javascript
async function writeWithBackpressure(data) {
  const status = await checkIndexingStatus();
  
  const lag = status.commit_t - status.index_t;
  if (lag > 100) {
    // Too much lag, slow down
    await sleep(1000);
  }
  
  await transact(data);
}
```

### Bulk Imports

For large imports:

```javascript
async function bulkImport(entities) {
  const batchSize = 1000;
  
  for (let i = 0; i < entities.length; i += batchSize) {
    const batch = entities.slice(i, i + batchSize);
    await transact({ "@graph": batch });
    
    // Wait for indexing to catch up every 10 batches
    if ((i / batchSize) % 10 === 0) {
      await waitForIndexing();
    }
    
    console.log(`Imported ${i + batch.length} / ${entities.length}`);
  }
}

async function waitForIndexing() {
  while (true) {
    const status = await checkIndexingStatus();
    const lag = status.commit_t - status.index_t;
    if (lag < 5) break;
    await sleep(1000);
  }
}
```

## Graph Source Indexing

Graph sources have their own indexing processes:

### BM25 Indexing

Full-text search indexes built asynchronously:

```text
t=100: Transaction with new documents
  - Main index updated
  - BM25 indexer triggered
  - Documents added to BM25 index
```

### Vector Search Indexing

Vector embeddings can be indexed separately for approximate nearest-neighbor (ANN) search via HNSW vector indexes (implemented with `usearch`, feature-gated behind the `vector` feature).

Inline similarity functions (`dotProduct`, `cosineSimilarity`, `euclideanDistance`) do **not** require a separate graph-source index; they compute scores directly during query execution.

```text
t=100: Transaction with embeddings
  - Main index updated
  - Vector indexer triggered
  - Vectors added to vector index
```

See [Vector Search](../indexing-and-search/vector-search.md) for details on HNSW vector indexes and query syntax.

## Best Practices

### 1. Monitor Novelty Layer

Track indexing lag:

```javascript
setInterval(async () => {
  const status = await checkIndexingStatus();
  const lag = status.commit_t - status.index_t;
  metrics.gauge('index_lag_txns', lag);
  
  if (lag > 100) {
    logger.warn(`High indexing lag: ${lag} transactions`);
  }
}, 10000);  // Check every 10 seconds
```

### 2. Batch Appropriately

Keep transactions reasonable size:
- Recommended: 100-1000 entities per transaction
- Maximum: 10,000 entities per transaction

### 3. Rate Limiting

Implement rate limiting for heavy write loads:

```javascript
const rateLimiter = new RateLimiter({
  tokensPerInterval: 100,
  interval: "minute"
});

await rateLimiter.removeTokens(1);
await transact(data);
```

### 4. Scheduled Imports

Run large imports during off-hours:

```javascript
if (isOffPeakHours()) {
  await runBulkImport();
} else {
  logger.info('Deferring bulk import to off-peak hours');
}
```

### 5. Alert on Lag

Set up alerts for indexing lag:

```javascript
const lag = status.commit_t - status.index_t;
if (lag > 200) {
  alert('Critical: Indexing lag > 200 transactions');
}
```

### 6. Capacity Planning

Plan capacity based on write load:

```text
Expected load: 10,000 transactions/day
Average size: 100 flakes/transaction
Total: 1,000,000 flakes/day

Indexing capacity needed: ~12 flakes/second
With 4× safety margin: ~50 flakes/second
```

## Troubleshooting

### High indexing lag

**Symptom:** `commit_t - index_t` growing continuously

**Causes:**
- Transaction rate exceeds indexing capacity
- Large transactions
- Resource constraints

**Solutions:**
- Reduce transaction rate
- Split large transactions
- Increase indexing resources
- Tune indexing parameters

### Slow Queries

**Symptom:** Queries slower than expected

**Possible Cause:** Large novelty layer

**Check:**
```bash
curl http://localhost:8090/v1/fluree/info/mydb:main | jq '.t - .index.t'
```

**Solution:** Wait for indexing or reduce write rate

### Index Memory Usage

**Symptom:** High memory usage

**Cause:** Large indexes or large novelty layer

**Solutions:**
- Increase system memory
- Reduce novelty layer
- Compact indexes (if supported)

## Related Documentation

- [Overview](overview.md) - Transaction overview
- [Insert](insert.md) - Adding data
- [Commit Receipts](commit-receipts.md) - Transaction metadata
- [Background Indexing](../indexing-and-search/background-indexing.md) - Indexing configuration


---

# ledger-config/README.md

# Ledger Configuration (Config Graph)

Fluree stores **ledger-level configuration as data** inside each ledger, in a dedicated system graph called the **config graph**. This is distinct from [server configuration](../operations/configuration.md) (TOML files, environment variables) which controls how the Fluree process runs.

The config graph holds RDF triples that define operational defaults for the ledger: which policy rules apply, whether SHACL validation runs, what reasoning modes are active, which properties enforce uniqueness, and more. Because config lives inside the ledger, it is:

- **Immutable and time-travelable** — config at any historical `t` is recoverable
- **Auditable** — every config change is a signed, committed transaction
- **Replicable** — config travels with the ledger across nodes and forks
- **Replay-safe** — deterministic interpretation without runtime environment state

## Graph layout

Every ledger reserves system named graphs:

| Graph | IRI pattern | Purpose |
|-------|-------------|---------|
| Default graph | (implicit) | Application data |
| Txn-meta | `urn:fluree:{ledger_id}#txn-meta` | Commit metadata |
| Config graph | `urn:fluree:{ledger_id}#config` | Ledger configuration |

User-defined named graphs (created via TriG) are identified by their IRI and allocated after the system graphs.

The config graph IRI is deterministic — derived from the ledger identifier. For a ledger `mydb:main`, the config graph is `urn:fluree:mydb:main#config`.

## Core concepts

### `f:LedgerConfig`

A single `f:LedgerConfig` resource in the config graph defines ledger-wide defaults. If multiple exist, the one with the lexicographically smallest `@id` wins (with a logged warning).

### Setting groups

Configuration is organized into independent **setting groups**, each governing a different subsystem:

| Setting group | Subsystem | Key fields |
|---------------|-----------|------------|
| [`f:policyDefaults`](setting-groups.md#policy-defaults) | Policy enforcement | `f:defaultAllow`, `f:policySource`, `f:policyClass` |
| [`f:shaclDefaults`](setting-groups.md#shacl-defaults) | SHACL validation | `f:shaclEnabled`, `f:shapesSource`, `f:validationMode` |
| [`f:reasoningDefaults`](setting-groups.md#reasoning-defaults) | OWL/RDFS reasoning | `f:reasoningModes`, `f:schemaSource` |
| [`f:datalogDefaults`](setting-groups.md#datalog-defaults) | Datalog rules | `f:datalogEnabled`, `f:rulesSource` |
| [`f:transactDefaults`](setting-groups.md#transact-defaults) | Transaction constraints | `f:uniqueEnabled`, `f:constraintsSource` |

Each group is resolved independently — locking down policy does not affect whether reasoning can be overridden.

### Per-graph overrides

Ledger-wide defaults apply to all graphs. For finer control, `f:graphOverrides` on the `f:LedgerConfig` contains `f:GraphConfig` entries that override settings for specific named graphs. See [Override control](override-control.md) for the full resolution model.

### Privileged system read

Config is read via a **privileged system read** that bypasses policy enforcement. This is necessary because config defines the policy — reading it through the policy-enforced path would create a circular dependency. User queries against the config graph still go through normal policy enforcement.

### Lagging config

Config changes take effect on the **next** transaction, not the current one. The transaction pipeline reads config from the pre-transaction state. This prevents a transaction from "authorizing itself" by changing config within its own payload.

## Common patterns

These recipes cover typical scenarios. Each assumes the ledger `mydb:main` — substitute your own ledger ID.

### Lock down a production ledger

Deny all access by default and require policy rules for every operation. Use `f:OverrideNone` so no query can bypass:

```trig
@prefix f: <https://ns.flur.ee/db#> .

GRAPH <urn:fluree:mydb:main#config> {
  <urn:fluree:mydb:main:config:ledger> a f:LedgerConfig ;
    f:policyDefaults [
      f:defaultAllow false ;
      f:policySource [
        a f:GraphRef ;
        f:graphSource [ f:graphSelector f:defaultGraph ]
      ] ;
      f:overrideControl f:OverrideNone
    ] .
}
```

After this transaction, the **next** transaction and all subsequent queries will require matching policy rules in the default graph. Make sure policy rules are already in place before enabling this — see [Config mutation governance](writing-config.md#config-mutation-governance).

### Enable SHACL validation in development (warn mode)

Validate data shapes but log warnings instead of rejecting — useful during development:

```trig
@prefix f: <https://ns.flur.ee/db#> .

GRAPH <urn:fluree:mydb:main#config> {
  <urn:fluree:mydb:main:config:ledger> a f:LedgerConfig ;
    f:shaclDefaults [
      f:shaclEnabled true ;
      f:shapesSource [
        a f:GraphRef ;
        f:graphSource [ f:graphSelector f:defaultGraph ]
      ] ;
      f:validationMode f:ValidationWarn
    ] .
}
```

Switch to `f:ValidationReject` when ready for production.

### Enforce unique emails

Two-step setup: annotate the property, then enable enforcement:

```trig
@prefix f:  <https://ns.flur.ee/db#> .
@prefix ex: <http://example.org/ns/> .

# Step 1: Annotate the property (in the default graph)
ex:email f:enforceUnique true .

# Step 2: Enable enforcement (in the config graph)
GRAPH <urn:fluree:mydb:main#config> {
  <urn:fluree:mydb:main:config:ledger> a f:LedgerConfig ;
    f:transactDefaults [
      f:uniqueEnabled true
    ] .
}
```

See [Unique constraints](unique-constraints.md) for full details including per-graph scoping and edge cases.

### Enable RDFS reasoning by default

Automatically expand `rdfs:subClassOf` and `rdfs:subPropertyOf` in all queries:

```trig
@prefix f: <https://ns.flur.ee/db#> .

GRAPH <urn:fluree:mydb:main#config> {
  <urn:fluree:mydb:main:config:ledger> a f:LedgerConfig ;
    f:reasoningDefaults [
      f:reasoningModes f:RDFS ;
      f:schemaSource [
        a f:GraphRef ;
        f:graphSource [ f:graphSelector f:defaultGraph ]
      ]
    ] .
}
```

With `f:OverrideAll` (the default), individual queries can still opt out by passing `"reasoning": "none"`.

### Different policy per graph

Allow open access to most graphs but lock down a sensitive one:

```trig
@prefix f: <https://ns.flur.ee/db#> .

GRAPH <urn:fluree:mydb:main#config> {
  <urn:fluree:mydb:main:config:ledger> a f:LedgerConfig ;
    f:policyDefaults [
      f:defaultAllow true ;
      f:overrideControl f:OverrideAll
    ] ;
    f:graphOverrides (
      [ a f:GraphConfig ;
        f:targetGraph <http://example.org/sensitive> ;
        f:policyDefaults [
          f:defaultAllow false ;
          f:policySource [
            a f:GraphRef ;
            f:graphSource [ f:graphSelector f:defaultGraph ]
          ] ;
          f:overrideControl f:OverrideNone
        ]
      ]
    ) .
}
```

The sensitive graph requires policy rules and cannot be overridden at query time. All other graphs remain open.

## Troubleshooting

### Config changes seem to have no effect

Config uses **lagging semantics** — changes take effect on the **next** transaction, not the current one. If you enable SHACL and insert invalid data in the same transaction, the data will be accepted. The next transaction will enforce the new config.

### Ledger became unmodifiable after policy misconfiguration

If you set `f:defaultAllow false` with `f:OverrideNone` before granting write access to the config graph, the ledger becomes locked — no transaction can modify it (including config changes). Recovery requires a ledger fork/restore. To prevent this:

1. **Always write policy rules first**, then enable restrictive policy in a subsequent transaction
2. **Test with `f:OverrideAll`** before switching to `f:OverrideNone`
3. **Ensure at least one identity** has write access to the config graph before locking down

### Multiple `f:LedgerConfig` resources

If the config graph contains more than one `f:LedgerConfig` resource, the system uses the one with the **lexicographically smallest `@id`** and logs a warning. Use the recommended subject IRI convention (`urn:fluree:{ledger_id}:config:ledger`) to avoid this.

### Config graph query returns empty results

User queries against the config graph go through **policy enforcement**. If `f:defaultAllow` is `false` and no policy explicitly grants read access to the config graph, queries will return empty results even though config is active. The system's internal privileged read is unaffected.

## CLI usage

The config graph is written and queried through normal CLI transaction and query commands:

```bash
# Write config via TriG
fluree insert --ledger mydb:main --format trig config.trig

# Query the config graph via SPARQL
fluree query --ledger mydb:main --format sparql \
  'PREFIX f: <https://ns.flur.ee/db#>
   SELECT ?s ?p ?o
   FROM <urn:fluree:mydb:main#config>
   WHERE { ?s ?p ?o }'
```

No special CLI commands are needed — config is data, written and queried like any other named graph.

## In this section

- [Writing config data](writing-config.md) — How to create and update config via TriG, SPARQL, or JSON-LD
- [Setting groups](setting-groups.md) — All setting groups with fields and examples
- [Override control](override-control.md) — Resolution precedence, identity gating, monotonicity
- [Unique constraints](unique-constraints.md) — Enforcing property value uniqueness with `f:enforceUnique`

## Related

- [Cross-ledger governance](../security/cross-ledger-policy.md) — Configure one model ledger to govern many data ledgers. Every `f:GraphRef`-shaped predicate (`f:policySource`, `f:constraintsSource`, `f:schemaSource`, `f:shapesSource`, `f:rulesSource`) accepts `f:ledger` to reference another ledger. Builds on the `f:GraphRef` shape documented in [setting groups](setting-groups.md#policy-defaults).


---

# ledger-config/writing-config.md

# Writing Config Data

The config graph is mutated using normal ledger transactions — config writes are signed, versioned, and replicable like any other write. The only difference is that the triples target the config graph IRI.

## Config graph IRI

Each ledger's config graph has a deterministic IRI:

```
urn:fluree:{ledger_id}#config
```

For a ledger named `mydb:main`, the config graph is `urn:fluree:mydb:main#config`.

## Writing via TriG

TriG is the most natural format for writing to named graphs. Wrap your config triples in a `GRAPH` block targeting the config graph IRI:

```trig
@prefix f: <https://ns.flur.ee/db#> .

GRAPH <urn:fluree:mydb:main#config> {
  <urn:fluree:mydb:main:config:ledger> a f:LedgerConfig ;
    f:policyDefaults [
      f:defaultAllow false
    ] ;
    f:shaclDefaults [
      f:shaclEnabled true ;
      f:validationMode f:ValidationReject
    ] .
}
```

## Writing via SPARQL UPDATE

Use `INSERT DATA` with a `GRAPH` clause:

```sparql
PREFIX f: <https://ns.flur.ee/db#>

INSERT DATA {
  GRAPH <urn:fluree:mydb:main#config> {
    <urn:fluree:mydb:main:config:ledger> a f:LedgerConfig ;
      f:reasoningDefaults [
        f:reasoningModes f:RDFS ;
        f:schemaSource [
          a f:GraphRef ;
          f:graphSource [ f:graphSelector f:defaultGraph ]
        ]
      ] .
  }
}
```

## Writing via JSON-LD

Use the `@graph` key with a named graph wrapper:

```json
{
  "@context": { "f": "https://ns.flur.ee/db#" },
  "@graph": [
    {
      "@id": "urn:fluree:mydb:main:config:ledger",
      "@type": "f:LedgerConfig",
      "@graph": "urn:fluree:mydb:main#config",
      "f:shaclDefaults": {
        "f:shaclEnabled": true,
        "f:validationMode": { "@id": "f:ValidationReject" }
      }
    }
  ]
}
```

## Updating config

Config changes are normal ledger operations. To change a setting, use a `DELETE/INSERT WHERE` pattern that binds the existing blank node:

```sparql
PREFIX f: <https://ns.flur.ee/db#>

DELETE {
  GRAPH <urn:fluree:mydb:main#config> {
    ?policy f:defaultAllow false .
  }
}
INSERT {
  GRAPH <urn:fluree:mydb:main#config> {
    ?policy f:defaultAllow true .
  }
}
WHERE {
  GRAPH <urn:fluree:mydb:main#config> {
    <urn:fluree:mydb:main:config:ledger> f:policyDefaults ?policy .
    ?policy f:defaultAllow false .
  }
}
```

This pattern binds `?policy` to the existing setting-group blank node, retracts the old value, and asserts the new one. It avoids the problem of `DELETE DATA` with blank nodes (which cannot match stored blank node identities).

Alternatively, give setting-group nodes explicit IRIs so they can be addressed directly:

```trig
@prefix f: <https://ns.flur.ee/db#> .

GRAPH <urn:fluree:mydb:main#config> {
  <urn:fluree:mydb:main:config:ledger> a f:LedgerConfig ;
    f:policyDefaults <urn:fluree:mydb:main:config:policy> .

  <urn:fluree:mydb:main:config:policy>
    f:defaultAllow false ;
    f:overrideControl f:OverrideAll .
}
```

With explicit IRIs, individual fields can be retracted by subject IRI without binding.

Retracting a field returns the ledger to the system default for that setting (as if the field were absent).

## Config mutation governance

Config writes go through the normal policy-enforced transaction path. This means:

- **Reading** config is privileged (system read, bypasses policy) — necessary to bootstrap.
- **Writing** config is **not** privileged — policy enforcement applies.

A `defaultAllow: false` config is self-protecting: the policy it defines must explicitly grant write access to the config graph for any changes to be possible.

If a ledger becomes unmodifiable due to a policy misconfiguration (no authorized config writers), recovery requires a ledger fork/restore — there is no superuser bypass.

## Recommended subject IRI

For operational simplicity, use a stable, conventional subject IRI:

```
urn:fluree:{ledger_id}:config:ledger
```

Colons (not a second `#` fragment) keep the IRI well-formed: the graph IRI already uses a fragment (`#config`), and RFC 3986 allows only one fragment per IRI. Using colons produces a valid URN (RFC 8141) that stays scoped to the ledger and avoids accidental multiple-config instances.

## Querying the config graph

The config graph is a named graph like any other — you can query it with SPARQL or JSON-LD to inspect the current configuration.

### SPARQL

```sparql
PREFIX f: <https://ns.flur.ee/db#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>

SELECT ?setting ?pred ?val
FROM <mydb:main#config>
WHERE {
  ?config rdf:type f:LedgerConfig ;
          ?setting ?group .
  ?group ?pred ?val .
  FILTER(?setting IN (
    f:policyDefaults, f:shaclDefaults, f:reasoningDefaults,
    f:datalogDefaults, f:transactDefaults
  ))
}
```

### JSON-LD query

```json
{
  "@context": { "f": "https://ns.flur.ee/db#" },
  "from": {
    "@id": "mydb:main",
    "graph": "urn:fluree:mydb:main#config"
  },
  "select": ["?config", "?pred", "?val"],
  "where": [
    { "@id": "?config", "@type": "f:LedgerConfig", "?pred": "?val" }
  ]
}
```

### Ledger-scoped endpoint

```bash
curl -X POST "http://localhost:8090/v1/fluree/query/mydb:main" \
  -H "Content-Type: application/sparql-query" \
  -d 'PREFIX f: <https://ns.flur.ee/db#>
      SELECT ?s ?p ?o
      FROM <urn:fluree:mydb:main#config>
      WHERE { ?s ?p ?o }'
```

### Policy applies to reads

User queries against the config graph go through normal **policy enforcement**. If `f:defaultAllow` is `false` and no policy grants read access to the config graph, user queries will return empty results. The *system* still reads config via a privileged path (bypassing policy), so config always takes effect regardless of policy.

### Time-travel

Config is part of the ledger's immutable commit chain. You can query config at any historical point:

```sparql
PREFIX f: <https://ns.flur.ee/db#>

SELECT ?setting ?val
FROM <mydb:main@t:5#config>
WHERE {
  ?config a f:LedgerConfig ;
          f:policyDefaults ?policy .
  ?policy ?setting ?val .
}
```

## Lagging semantics

Config changes take effect on the **next** transaction. The transaction pipeline reads config from the pre-transaction state (`t - 1`). This prevents a transaction from changing the rules it is validated against.

This means:
- Enabling SHACL in the same transaction as invalid data will **not** reject that data
- Enabling `f:uniqueEnabled` in the same transaction as duplicate values will **not** reject those duplicates
- The next transaction after the config change will be validated against the new config


---

# ledger-config/setting-groups.md

# Setting Groups

Each setting group configures a different subsystem. Groups are resolved independently — locking down one group does not affect others.

All setting groups can appear on both `f:LedgerConfig` (ledger-wide defaults) and `f:GraphConfig` (per-graph overrides), except where noted.

## System defaults

When no config graph is present (or a setting group is absent), the system defaults apply:

| Setting group | System default |
|---------------|----------------|
| Policy | `f:defaultAllow true` — all queries and transactions are permitted |
| SHACL | Disabled — no shape validation |
| Reasoning | Disabled — no OWL/RDFS inference |
| Datalog | Disabled — no rule evaluation |
| Transact constraints | Disabled — no uniqueness enforcement |
| Override control | `f:OverrideAll` — any request can override any setting |

In other words, an unconfigured ledger is **fully open**: no policy, no validation, no reasoning. This matches the behavior of a fresh ledger and ensures backward compatibility.

---

## Policy defaults

**Group predicate**: `f:policyDefaults`

Controls default policy enforcement behavior.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `f:defaultAllow` | boolean | `true` | Allow (`true`) or deny (`false`) when no policy rule matches |
| `f:policySource` | `f:GraphRef` | (none) | Graph containing policy rules (`f:Allow`, `f:Modify`, etc.) |
| `f:policyClass` | IRI or list | (none) | Default policy classes to apply |
| `f:overrideControl` | IRI or object | `f:OverrideAll` | Override gating (see [Override control](override-control.md)) |

`f:policySource` is non-overridable — it can only be changed by writing to the config graph, not at query time. `f:defaultAllow` and `f:policyClass` are overridable (subject to override control).

When `f:policySource` is set, the policy loader scans the specified graph for policy rules instead of the default graph. This keeps policy rules separate from end-user data. If `f:policySource` is not set, policies are loaded from the default graph (backward compatible).

**Cross-ledger references are supported on `f:policySource`.** The graph source can name another ledger via `f:ledger`, so a single model ledger can hold policy rules that govern many data ledgers. See [Cross-ledger policy](../security/cross-ledger-policy.md) for the configuration pattern and the contract on `f:policyClass` filtering, baseline `f:AccessPolicy` semantics, and the failure modes.

**Not yet honored on `f:policySource`** (parsed by the config layer but rejected at request time with a clear error): `f:atT` temporal pinning, `f:trustPolicy` verification, `f:rollbackGuard` freshness constraints. Cross-ledger references are also supported on `f:constraintsSource`, `f:schemaSource` (single graph only — transitive `owl:imports` recursion across ledgers is not yet supported), `f:shapesSource`, and `f:rulesSource`. See [Cross-ledger policy](../security/cross-ledger-policy.md) for the end-to-end configuration patterns and failure modes shared by all five subsystems.

### Example: policies in the default graph

```trig
@prefix f: <https://ns.flur.ee/db#> .

GRAPH <urn:fluree:mydb:main#config> {
  <urn:fluree:mydb:main:config:ledger> a f:LedgerConfig ;
    f:policyDefaults [
      f:defaultAllow false ;
      f:policySource [
        a f:GraphRef ;
        f:graphSource [ f:graphSelector f:defaultGraph ]
      ] ;
      f:overrideControl f:OverrideAll
    ] .
}
```

### Example: policies in a named graph

Storing policy rules in a dedicated named graph keeps them out of the default data graph. The identity's `f:policyClass` triples must also be in the policy graph.

```trig
@prefix f: <https://ns.flur.ee/db#> .

GRAPH <urn:fluree:mydb:main#config> {
  <urn:fluree:mydb:main:config:ledger> a f:LedgerConfig ;
    f:policyDefaults [
      f:defaultAllow false ;
      f:policySource [
        a f:GraphRef ;
        f:graphSource [ f:graphSelector <urn:fluree:mydb:main/policy> ]
      ]
    ] .
}
```

---

## SHACL defaults

**Group predicate**: `f:shaclDefaults`

Controls SHACL shape validation at transaction time.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `f:shaclEnabled` | boolean | `false` | Enable or disable SHACL validation |
| `f:shapesSource` | `f:GraphRef` | (none) | Graph containing SHACL shapes |
| `f:validationMode` | IRI | `f:ValidationReject` | `f:ValidationReject` (reject invalid data) or `f:ValidationWarn` (log warning, allow) |
| `f:overrideControl` | IRI or object | `f:OverrideAll` | Override gating |

`f:shapesSource` is non-overridable. `f:shaclEnabled` and `f:validationMode` are overridable.

### Example

```trig
@prefix f: <https://ns.flur.ee/db#> .

GRAPH <urn:fluree:mydb:main#config> {
  <urn:fluree:mydb:main:config:ledger> a f:LedgerConfig ;
    f:shaclDefaults [
      f:shaclEnabled true ;
      f:shapesSource [
        a f:GraphRef ;
        f:graphSource [ f:graphSelector f:defaultGraph ]
      ] ;
      f:validationMode f:ValidationReject ;
      f:overrideControl f:OverrideNone
    ] .
}
```

---

## Reasoning defaults

**Group predicate**: `f:reasoningDefaults`

Controls OWL/RDFS reasoning applied at query time.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `f:reasoningModes` | IRI or list | (none) | Reasoning modes: `f:RDFS`, `f:OWL2QL`, `f:OWL2RL`, `f:Datalog` |
| `f:schemaSource` | `f:GraphRef` | (none) | Graph containing schema triples (`rdfs:subClassOf`, etc.) |
| `f:reasoningMaxFacts` | integer | 1,000,000 | OWL2-RL materialization budget: max derived facts before the closure is capped |
| `f:reasoningMaxSeconds` | integer | 30 | OWL2-RL materialization budget: max wall-clock seconds before the closure is capped |
| `f:overrideControl` | IRI or object | `f:OverrideAll` | Override gating |

`f:schemaSource` is non-overridable. `f:reasoningModes` and the budget fields
are overridable.

The budget fields are a **correctness control**: a capped materialization is
an incomplete closure, so reasoning queries may silently miss entailments.
A capped run surfaces in tracked query responses as a `reasoning` block with
`"capped": true` (and in the `x-fdb-reasoning` response header). The budget
can be configured without `f:reasoningModes` — it then governs queries that
request reasoning themselves. Queries may supply their own budget via the
`"reasoningBudget"` key (JSON-LD) or `# PRAGMA reasoning-max-facts:` /
`# PRAGMA reasoning-max-seconds:` (SPARQL), subject to `f:overrideControl`.
Server-wide defaults come from `FLUREE_REASONING_MAX_FACTS` /
`FLUREE_REASONING_MAX_SECONDS` (lowest precedence above the built-ins).

### Example

```trig
@prefix f: <https://ns.flur.ee/db#> .

GRAPH <urn:fluree:mydb:main#config> {
  <urn:fluree:mydb:main:config:ledger> a f:LedgerConfig ;
    f:reasoningDefaults [
      f:reasoningModes f:RDFS ;
      f:schemaSource [
        a f:GraphRef ;
        f:graphSource [ f:graphSelector f:defaultGraph ]
      ] ;
      f:overrideControl f:OverrideAll
    ] .
}
```

---

## Datalog defaults

**Group predicate**: `f:datalogDefaults`

Controls Fluree's stored datalog rules (`f:rule`).

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `f:datalogEnabled` | boolean | `false` | Enable or disable datalog rule evaluation |
| `f:rulesSource` | `f:GraphRef` | (none) | Graph containing `f:rule` definitions |
| `f:allowQueryTimeRules` | boolean | `true` | Allow queries to supply ad-hoc rules |
| `f:overrideControl` | IRI or object | `f:OverrideAll` | Override gating |

`f:rulesSource` is non-overridable. `f:datalogEnabled` and `f:allowQueryTimeRules` are overridable.

### Example

```trig
@prefix f: <https://ns.flur.ee/db#> .

GRAPH <urn:fluree:mydb:main#config> {
  <urn:fluree:mydb:main:config:ledger> a f:LedgerConfig ;
    f:datalogDefaults [
      f:datalogEnabled true ;
      f:rulesSource [
        a f:GraphRef ;
        f:graphSource [ f:graphSelector f:defaultGraph ]
      ] ;
      f:allowQueryTimeRules false ;
      f:overrideControl f:OverrideNone
    ] .
}
```

---

## Transact defaults

**Group predicate**: `f:transactDefaults`

Controls transaction-time constraint enforcement, such as property value uniqueness.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `f:uniqueEnabled` | boolean | `false` | Enable unique constraint enforcement |
| `f:constraintsSource` | `f:GraphRef` or list | default graph | Graph(s) containing constraint annotations (e.g., `f:enforceUnique`) |
| `f:overrideControl` | IRI or object | `f:OverrideAll` | Override gating |

When `f:uniqueEnabled` is `true` and `f:constraintsSource` is omitted, the default graph is used as the constraint source.

### Additive merge semantics

Unlike other setting groups where per-graph values **replace** ledger-wide values field-by-field, transact defaults use **additive** merge semantics:

- **`f:uniqueEnabled`**: Once enabled at the ledger level, it stays enabled for all graphs. Per-graph configs cannot disable it.
- **`f:constraintsSource`**: Per-graph sources are **added to** ledger-wide sources, not substituted. A graph checks annotations from all sources (ledger-wide + graph-specific).

This prevents a per-graph override from accidentally disabling enforcement or dropping constraint sources.

Note: additive merge is still subject to override control. If the ledger-wide `f:overrideControl` for `f:transactDefaults` is `f:OverrideNone`, per-graph additions are blocked entirely — the ledger-wide settings are final.

### Example

```trig
@prefix f: <https://ns.flur.ee/db#> .
@prefix ex: <http://example.org/ns/> .

# Define constraint annotations in the default graph
ex:email f:enforceUnique true .
ex:ssn   f:enforceUnique true .

# Enable enforcement via config
GRAPH <urn:fluree:mydb:main#config> {
  <urn:fluree:mydb:main:config:ledger> a f:LedgerConfig ;
    f:transactDefaults [
      f:uniqueEnabled true ;
      f:constraintsSource [
        a f:GraphRef ;
        f:graphSource [ f:graphSelector f:defaultGraph ]
      ]
    ] .
}
```

See [Unique constraints](unique-constraints.md) for full details on `f:enforceUnique`.

---

## Full-text defaults

**Group predicate**: `f:fullTextDefaults`

Declares properties whose string values should be indexed for BM25 full-text
scoring without requiring the `@fulltext` datatype per value, and sets the
default analyzer language for untagged plain strings.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `f:defaultLanguage` | BCP-47 string | `"en"` | Analyzer language for plain (`xsd:string`) values on configured properties |
| `f:property` | `f:FullTextProperty` list | empty | One node per property to full-text index |
| `f:overrideControl` | IRI or object | `f:OverrideAll` | Override gating |

Each `f:property` entry is an `f:FullTextProperty` node carrying `f:target` —
the IRI of the property being indexed. Additional optional knobs (per-property
language, tokenizer, etc.) can be added to `f:FullTextProperty` in the future
without breaking the schema.

The `@fulltext` datatype retains its zero-config shortcut semantics: any value
tagged `@fulltext` always indexes as English, regardless of what
`f:fullTextDefaults` declares. Configured plain-string paths and
`@fulltext`-datatype English content share the same per-property English
arena — no duplication.

`rdf:langString` values auto-route to per-language arenas by their tag. An
unrecognized BCP-47 tag tokenizes + lowercases only (no stopwords, no
stemming) — consistent on both indexing and query sides.

### Additive merge semantics

Like `f:transactDefaults`, `f:fullTextDefaults` uses additive merge. Per-graph
`f:property` entries are appended to the ledger-wide list (deduping by
target IRI — per-graph wins on a collision). Per-graph `f:defaultLanguage`
shadows the ledger-wide value. Ledger-wide `f:OverrideNone` blocks per-graph
overrides entirely.

### Config changes require a manual reindex

Editing `f:fullTextDefaults` never triggers any indexing automatically. Arenas
reflect the config that was in effect at their build time; to pick up a
changed property list or default language, run a full reindex (`fluree
reindex …` or equivalent). Until then, existing arenas stay authoritative and
novelty written after the config change is scored with whatever language the
current effective config resolves to — which may produce temporarily
mismatched scoring until the reindex completes.

An in-flight reindex operates on a point-in-time snapshot and will not see a
config change committed during its run. Wait for the reindex to finish, then
trigger a new one against the post-change state.

### Example

```trig
@prefix f: <https://ns.flur.ee/db#> .
@prefix ex: <http://example.org/ns/> .

GRAPH <urn:fluree:mydb:main#config> {
  <urn:fluree:mydb:main:config:ledger> a f:LedgerConfig ;
    f:fullTextDefaults [
      a f:FullTextDefaults ;
      f:defaultLanguage "en" ;
      f:property [ a f:FullTextProperty ; f:target ex:title ] ,
                 [ a f:FullTextProperty ; f:target ex:body ]
    ] .
}
```

### Per-graph override example

```trig
GRAPH <urn:fluree:mydb:main#config> {
  <urn:fluree:mydb:main:config:ledger> a f:LedgerConfig ;
    f:fullTextDefaults [
      a f:FullTextDefaults ;
      f:defaultLanguage "en" ;
      f:property [ a f:FullTextProperty ; f:target ex:title ]
    ] ;
    f:graphOverrides [
      a f:GraphConfig ;
      f:targetGraph <urn:example:productCatalog> ;
      f:fullTextDefaults [
        a f:FullTextDefaults ;
        f:defaultLanguage "es" ;
        f:property [ a f:FullTextProperty ; f:target ex:productName ]
      ]
    ] .
}
```

Under this config, queries touching the `productCatalog` graph analyze
untagged plain strings as Spanish (`"es"`); other graphs keep English.
`ex:title` is full-text indexed everywhere (ledger-wide); `ex:productName`
is indexed only in the `productCatalog` graph.

See [Inline fulltext search](../indexing-and-search/fulltext.md) for the
end-user guide — when to pick this path over the `@fulltext` datatype,
supported languages, per-graph multilingual setups, the reindex workflow,
and how configured properties interact with `@fulltext`-datatype values.

---

## Ledger-scoped settings

Some settings are structurally tied to the ledger as a whole and are **not meaningful per-graph**. They live exclusively on `f:LedgerConfig` and are ignored if present on `f:GraphConfig`.

Override control does not apply to ledger-scoped settings — they are changed only by writing to the config graph.

> **Note:** `f:authzSource` (an identity/relationship graph used by policy evaluation) is planned as a ledger-scoped setting but is not yet implemented. When available, it will let the config graph specify which graph contains identity data (e.g., DID→role mappings) for policy resolution.

---

## `f:GraphRef`: referencing source graphs

Several fields (`f:policySource`, `f:shapesSource`, `f:schemaSource`, `f:rulesSource`, `f:constraintsSource`) use `f:GraphRef` to point at graphs containing rules, shapes, schema, or constraints.

A `f:GraphRef` has two levels: the outer node carries the type and optional trust/rollback settings, and a nested `f:graphSource` object carries the source coordinates:

| Field | Level | Type | Description |
|-------|-------|------|-------------|
| `f:graphSource` | `f:GraphRef` | object | Nested source coordinates (required) |
| `f:trustPolicy` | `f:GraphRef` | object | How to verify the referenced graph (future) |
| `f:rollbackGuard` | `f:GraphRef` | object | Freshness constraints (future) |
| `f:graphSelector` | `f:graphSource` | IRI | Target graph: `f:defaultGraph`, `f:txnMetaGraph`, or a named graph IRI |
| `f:ledger` | `f:graphSource` | IRI | Ledger identifier. Supported on `f:policySource`, `f:constraintsSource`, `f:schemaSource` (single graph only), `f:shapesSource`, and `f:rulesSource`. See [Cross-ledger policy](../security/cross-ledger-policy.md) for the shared resolver contract and failure modes. |
| `f:atT` | `f:graphSource` | integer | Pin to a specific transaction time (optional) |

For the common case of referencing a graph within the same ledger, only `f:graphSelector` is needed inside `f:graphSource`:

```trig
f:shapesSource [
  a f:GraphRef ;
  f:graphSource [ f:graphSelector f:defaultGraph ]
] .
```

For referencing the config graph itself (co-resident rules/shapes):

```trig
f:policySource [
  a f:GraphRef ;
  f:graphSource [ f:graphSelector <urn:fluree:mydb:main#config> ]
] .
```

Cross-ledger `f:GraphRef` (using `f:ledger` to reference another ledger) is honored by all five governance source predicates: `f:policySource`, `f:constraintsSource`, `f:schemaSource` (single graph only — transitive `owl:imports` recursion across ledgers is not yet supported on schema), `f:shapesSource`, and `f:rulesSource`. See [Cross-ledger policy](../security/cross-ledger-policy.md) for end-to-end patterns.


---

# ledger-config/override-control.md

# Override Control

Fluree's config resolution follows a three-tier precedence model. Each setting group is resolved independently, and an **override control** mechanism governs whether higher-priority sources can change values set at lower tiers.

## Resolution precedence

Settings are resolved from lowest to highest priority:

| Priority | Source | When it applies |
|----------|--------|-----------------|
| 4 (lowest) | System defaults | No config present (allow-all, no SHACL, no reasoning) |
| 3 | Ledger-wide config (`f:LedgerConfig`) | Fallback for any setting not overridden at higher tiers |
| 2 | Per-graph config (`f:GraphConfig`) | Only if ledger-wide override control permits |
| 1 (highest) | Query/transaction-time opts | Only if effective override control permits + identity check passes |

## Override control modes

Each setting group may include an `f:overrideControl` field controlling whether higher-priority sources can override the value.

| Mode | Value | Behavior |
|------|-------|----------|
| No overrides | `f:OverrideNone` | Config values are final. No per-graph or query-time overrides permitted. |
| All overrides | `f:OverrideAll` | Any request can override. Default when `f:overrideControl` is absent. |
| Identity-gated | Object with `f:controlMode: f:IdentityRestricted` | Only requests with a server-verified identity matching `f:allowedIdentities` can override. |

### Identity-gated example

```json
{
  "f:overrideControl": {
    "f:controlMode": { "@id": "f:IdentityRestricted" },
    "f:allowedIdentities": [
      { "@id": "did:key:z6MkhaXgBZDvotDkL5257faiztiGiC2QtKLGpbnnEGta2doK" }
    ]
  }
}
```

### Identity verification

Override identity is the **server-verified request identity** (canonical DID string), not a user-supplied query parameter. Specifically:

- With the `credential` feature: the DID from the verified JWS `kid` header
- With server auth middleware: the DID mapped from an OAuth token
- A caller **cannot** become an allowed identity by setting `"opts": {"identity": "..."}` in query JSON — that field is for policy evaluation context, not override authorization
- Anonymous requests (no verified identity) are always denied by `f:IdentityRestricted`

### Query-time vs transact-time overrides

- **Query-time overrides** (reasoning modes, policy opts): identity is the **query caller**
- **Transact-time overrides** (SHACL mode, validation settings): identity is the **transaction signer**

## Monotonicity: per-graph can only tighten

Ledger-wide `f:overrideControl` sets the **maximum permissiveness**. Per-graph configs may only restrict further, never loosen.

Permissiveness ordering: `f:OverrideNone` < `f:IdentityRestricted` < `f:OverrideAll`

The effective per-graph override control is **min(ledger-wide, per-graph)**:

| Ledger-wide | Per-graph | Effective | Why |
|-------------|-----------|-----------|-----|
| `OverrideNone` | `OverrideAll` | **OverrideNone** | Per-graph cannot loosen (warning logged) |
| `IdentityRestricted({alice})` | `OverrideAll` | **IdentityRestricted({alice})** | Per-graph cannot loosen |
| `IdentityRestricted({alice, bob})` | `IdentityRestricted({alice})` | **IdentityRestricted({alice})** | Intersection: per-graph tightens |
| `OverrideAll` | `OverrideNone` | **OverrideNone** | Per-graph tightens (valid) |
| `OverrideAll` | `IdentityRestricted({alice})` | **IdentityRestricted({alice})** | Per-graph tightens (valid) |
| `OverrideAll` | (absent) | **OverrideAll** | Inherits ledger-wide |

When both are `IdentityRestricted`, the effective `allowedIdentities` is the **intersection** of the two lists.

## Resolution algorithm

For each setting group independently:

```
1. Start with system defaults
2. Apply ledger-wide config for this group (if present)
3. Get ledger-wide overrideControl (default: OverrideAll)
4. If ledger-wide overrideControl is OverrideNone:
     → this group is final. Skip to step 8.
5. Apply per-graph config for this group (if present)
6. Compute effective overrideControl:
     = min(ledgerWide, perGraph)
     If both IdentityRestricted: allowedIdentities = intersection
7. Check effective overrideControl against query/txn-time opts:
     OverrideNone         → config values are final
     OverrideAll          → apply query-time opts
     IdentityRestricted   → apply only if request identity matches
8. Result is the effective setting for this group.
```

## Per-group truth tables

### Policy (`f:policyDefaults`)

| Ledger-wide | Per-graph | Query (identity) | Effective | Why |
|-------------|-----------|-------------------|-----------|-----|
| `defaultAllow: false`, OverrideNone | (none) | `defaultAllow: true` (any) | **deny** | No overrides allowed |
| `defaultAllow: false`, OverrideAll | (none) | `defaultAllow: true` (any) | **allow** | All overrides allowed |
| `defaultAllow: false`, IdentityRestricted({alice}) | (none) | `defaultAllow: true` (alice) | **allow** | Alice is authorized |
| `defaultAllow: false`, IdentityRestricted({alice}) | (none) | `defaultAllow: true` (bob) | **deny** | Bob not authorized |
| `defaultAllow: false`, IdentityRestricted({alice}) | (none) | `defaultAllow: true` (anon) | **deny** | No identity = no override |
| `defaultAllow: false`, OverrideNone | `defaultAllow: true` | (none) | **deny** | OverrideNone blocks per-graph |
| `defaultAllow: false`, OverrideAll | `defaultAllow: true` | (none) | **allow** | Per-graph overrides ledger-wide |
| `defaultAllow: true`, OverrideAll | `defaultAllow: false`, OverrideNone | `defaultAllow: true` (any) | **deny** | Per-graph OverrideNone blocks query |
| (none) | (none) | (none) | **allow** | System default (allow-all) |

### Reasoning (`f:reasoningDefaults`)

| Ledger-wide | Per-graph | Query (identity) | Effective | Why |
|-------------|-----------|-------------------|-----------|-----|
| `modes: [rdfs]`, OverrideNone | (none) | `reasoning: [owl2-rl]` (any) | **rdfs** | No overrides |
| `modes: [rdfs]`, OverrideAll | (none) | `reasoning: [owl2-rl]` (any) | **owl2-rl** | Override allowed |
| `modes: [rdfs]`, IdentityRestricted({alice}) | (none) | `reasoning: [owl2-rl]` (alice) | **owl2-rl** | Alice authorized |
| `modes: [rdfs]`, IdentityRestricted({alice}) | (none) | `reasoning: [owl2-rl]` (bob) | **rdfs** | Bob not authorized |
| `modes: [rdfs]`, OverrideAll | `modes: [owl2-rl]` | (none) | **owl2-rl** | Per-graph overrides |
| `modes: [rdfs]`, OverrideNone | `modes: [owl2-rl]` | (none) | **rdfs** | OverrideNone blocks per-graph |

### SHACL (`f:shaclDefaults`)

| Ledger-wide | Per-graph | Effective | Why |
|-------------|-----------|-----------|-----|
| `enabled: false`, OverrideNone | `enabled: true` | **disabled** | OverrideNone blocks per-graph |
| `enabled: true`, OverrideAll | `enabled: false` | **disabled** | Per-graph disables for its graph |
| `mode: warn`, OverrideAll | `mode: reject` | **reject** | Per-graph overrides |

### Transact (`f:transactDefaults`)

Transact defaults use **additive** merge semantics, unlike other groups. However, the general override control rule still applies: if the ledger-wide `f:overrideControl` is `f:OverrideNone`, per-graph transact defaults are blocked entirely.

| Ledger-wide | Per-graph | Effective | Why |
|-------------|-----------|-----------|-----|
| `uniqueEnabled: true` | `uniqueEnabled: false` | **enabled** | Monotonic OR — cannot disable |
| `uniqueEnabled: true`, sources: `[default]` | sources: `[schemaGraph]` | sources: **[default, schemaGraph]** | Additive — sources accumulate |
| `uniqueEnabled: false` | `uniqueEnabled: true` | **enabled** | Per-graph can enable |
| `uniqueEnabled: true`, OverrideNone | sources: `[schemaGraph]` | sources: **[default]** only | OverrideNone blocks per-graph additions |

## Overridable vs non-overridable fields

Not all fields in a setting group are overridable. Source pointers (where rules/shapes/schema come from) are always config-only:

| Subsystem | Overridable fields | Non-overridable (config-only) |
|-----------|-------------------|-------------------------------|
| `f:policyDefaults` | `f:defaultAllow`, `f:policyClass` | `f:policySource` |
| `f:shaclDefaults` | `f:validationMode`, `f:shaclEnabled` | `f:shapesSource` |
| `f:reasoningDefaults` | `f:reasoningModes` | `f:schemaSource` |
| `f:datalogDefaults` | `f:datalogEnabled`, `f:allowQueryTimeRules` | `f:rulesSource` |

Non-overridable fields can only be changed by writing to the config graph. This prevents a query from redirecting the engine to read rules or schema from an arbitrary graph.

## Per-graph overrides

Per-graph overrides target specific named graphs by IRI:

```trig
@prefix f: <https://ns.flur.ee/db#> .

GRAPH <urn:fluree:mydb:main#config> {
  <urn:fluree:mydb:main:config:ledger> a f:LedgerConfig ;
    f:policyDefaults [
      f:defaultAllow true ;
      f:overrideControl f:OverrideAll
    ] ;
    f:graphOverrides (
      [ a f:GraphConfig ;
        f:targetGraph <http://example.org/sensitive> ;
        f:policyDefaults [
          f:defaultAllow false ;
          f:overrideControl f:OverrideNone
        ]
      ]
    ) .
}
```

In this example:
- **All graphs** default to `defaultAllow: true` with `OverrideAll`
- **`http://example.org/sensitive`** overrides to `defaultAllow: false` with `OverrideNone` — no query can override policy for this graph
- `f:targetGraph` uses `f:defaultGraph` for the default graph


---

# ledger-config/unique-constraints.md

# Unique Constraints (`f:enforceUnique`)

Fluree supports transaction-time enforcement of property value uniqueness via `f:enforceUnique`. This is complementary to SHACL — it runs independently.

## How it works

Unique constraint enforcement has two parts:

1. **Annotation**: Mark properties as unique by asserting `f:enforceUnique true` on their IRIs in any graph
2. **Activation**: Enable enforcement in the config graph via `f:transactDefaults`

This separation follows the same pattern as SHACL (shapes + config activation) and reasoning (schema + config activation). Annotations alone do nothing — enforcement must be explicitly enabled.

## Step 1: Define unique properties

Assert `f:enforceUnique true` on any property IRI that should enforce uniqueness. These annotations can live in the default graph or any named graph:

```trig
@prefix f: <https://ns.flur.ee/db#> .
@prefix ex: <http://example.org/ns/> .

# In the default graph
ex:email f:enforceUnique true .
ex:ssn   f:enforceUnique true .
```

## Step 2: Enable enforcement

Enable unique constraint checking in the config graph:

```trig
@prefix f: <https://ns.flur.ee/db#> .

GRAPH <urn:fluree:mydb:main#config> {
  <urn:fluree:mydb:main:config:ledger> a f:LedgerConfig ;
    f:transactDefaults [
      f:uniqueEnabled true
    ] .
}
```

When `f:constraintsSource` is omitted, the default graph is used as the annotation source.

### Explicit constraint source

To read annotations from a specific graph:

```trig
@prefix f: <https://ns.flur.ee/db#> .

GRAPH <urn:fluree:mydb:main#config> {
  <urn:fluree:mydb:main:config:ledger> a f:LedgerConfig ;
    f:transactDefaults [
      f:uniqueEnabled true ;
      f:constraintsSource [
        a f:GraphRef ;
        f:graphSource [ f:graphSelector f:defaultGraph ]
      ]
    ] .
}
```

### Multiple constraint sources

Multiple sources can be specified — all are checked:

```trig
f:transactDefaults [
  f:uniqueEnabled true ;
  f:constraintsSource [
    a f:GraphRef ;
    f:graphSource [ f:graphSelector f:defaultGraph ]
  ] , [
    a f:GraphRef ;
    f:graphSource [ f:graphSelector <http://example.org/schema> ]
  ]
] .
```

### Cross-ledger constraint source

`f:constraintsSource` also supports **cross-ledger references** —
set `f:ledger` on the inner `f:graphSource` to load
`f:enforceUnique` annotations from another ledger at transaction
time. See
[Cross-ledger governance — Cross-ledger constraints](../security/cross-ledger-policy.md#cross-ledger-uniqueness-constraints)
for the end-to-end pattern and failure modes.

## What gets enforced

Once enabled, any transaction that would result in **two or more distinct subjects** holding the same value for a unique property **within the same graph** is rejected.

### Scoping: per-graph

Uniqueness is enforced **per graph**. The same value on the same property is allowed across different named graphs:

```
# Graph A: ex:alice ex:email "alice@example.com" — OK
# Graph B: ex:bob   ex:email "alice@example.com" — OK (different graph)
# Graph A: ex:carol ex:email "alice@example.com" — REJECTED (same graph as alice)
```

### Value identity

Uniqueness is determined by the **storage-layer value representation**, not by RDF strict equality. The uniqueness key is:

```
(graph, predicate, value)
```

where "value" is the internal storage representation (type discriminant + payload).

The enforcement query matches on `(predicate, object)` in the POST index without constraining by datatype or language tag. This means:

- Two values with different datatype IRIs but the **same internal representation** are treated as the same value. For example, `"hello"^^xsd:string` and `"hello"^^ex:customType` both store as the same string value internally, so they conflict.
- Two values with different language tags but the **same string content** conflict, because the language tag is metadata, not part of the value key.
- Two values with **different internal representations** are naturally distinct. For example, `"42"` (stored as a string) and `42` (stored as an integer) do not conflict because they are different value types at the storage layer.

This design matches how humans think about value identity and prevents circumventing uniqueness by attaching a different datatype annotation or language tag.

### Intra-transaction enforcement

Uniqueness is checked after staging, so conflicts within a single transaction are caught:

```json
{
  "@context": { "ex": "http://example.org/ns/" },
  "@graph": [
    { "@id": "ex:alice", "ex:email": "same@example.com" },
    { "@id": "ex:bob",   "ex:email": "same@example.com" }
  ]
}
```

This transaction is rejected because two subjects assert the same value for a unique property.

### Upsert safety

Upserts that change a value are handled correctly. When an upsert retracts the old value and asserts a new one in the same transaction, the old value is no longer active — no false positive.

### Idempotent re-insert

Re-asserting the same `(subject, property, value)` triple that already exists is allowed. One subject still holds the value — no violation.

## Inline `opts.uniqueProperties` per transaction

In addition to constraints stored in the ledger, a transaction can
supply **inline unique-property declarations** via the
`opts.uniqueProperties` field. The properties are enforced only for
that one transaction; the list itself never persists.

```json
{
  "@context": {"ex": "http://example.org/ns/"},
  "@id":      "ex:bob",
  "ex:email": "alice@example.org",
  "opts": {
    "uniqueProperties": [
      "http://example.org/ns/email"
    ]
  }
}
```

Each entry must be a **full IRI** (not a compact prefix form). IRIs
that the ledger's namespace map has never seen are dropped silently
— no instance of the property exists, so the constraint cannot be
violated either way; this matches the same-ledger contract.

Semantics:

- **Additive, not replacing.** Inline properties union with whatever
  `f:constraintsSource` already resolves to (same-ledger or
  cross-ledger). A property is enforced if either source declares it.
- **Transient.** The list is never written into the ledger. The next
  transaction without `opts.uniqueProperties` runs without it.
- **No-config enforcement.** Inline properties drive enforcement
  even on a ledger with no `f:transactDefaults` block — the inline
  list is itself the configuration for this transaction.
- **No audit trail.** Without persistence it's not reconstructable
  which constraints validated which commit. If auditability matters,
  declare the constraint in a `f:constraintsSource` graph instead.

Use cases that fit well: per-tenant constraints layered on top of
operator-set baselines; one-off bulk loads that need extra hygiene
without polluting `#config`; testing a candidate constraint before
committing the annotation.

## Error message

When a uniqueness violation is detected, the transaction fails with an error like:

```
Unique constraint violation: property <http://example.org/ns/email>
  value "alice@example.com" already exists for subject
  <http://example.org/ns/alice> in graph default
  (conflicting subject: <http://example.org/ns/bob>)
```

## Lagging config

Config is read from the **pre-transaction state**. This means:

- Enabling `f:uniqueEnabled` and inserting duplicate values in the **same transaction** will **not** reject the duplicates
- The **next** transaction will enforce the constraint

This is intentional and consistent with all other config graph features.

## Per-graph overrides

Transact defaults use **additive** merge semantics:

- `f:uniqueEnabled` uses monotonic OR — once enabled at the ledger level, per-graph configs cannot disable it
- `f:constraintsSource` is additive — per-graph sources are added to (not replace) ledger-wide sources

Note: additive merge is still subject to override control. If the ledger-wide `f:overrideControl` for `f:transactDefaults` is `f:OverrideNone`, per-graph additions are blocked entirely.

This means a per-graph override can add additional constraint sources but cannot remove ledger-wide ones:

```trig
GRAPH <urn:fluree:mydb:main#config> {
  <urn:fluree:mydb:main:config:ledger> a f:LedgerConfig ;
    f:transactDefaults [
      f:uniqueEnabled true ;
      f:constraintsSource [
        a f:GraphRef ;
        f:graphSource [ f:graphSelector f:defaultGraph ]
      ]
    ] ;
    f:graphOverrides (
      [ a f:GraphConfig ;
        f:targetGraph <http://example.org/graphX> ;
        f:transactDefaults [
          f:constraintsSource [
            a f:GraphRef ;
            f:graphSource [ f:graphSelector <http://example.org/schema> ]
          ]
        ]
      ]
    ) .
}
```

In this example, `graphX` checks unique annotations from **both** the default graph (ledger-wide) and `http://example.org/schema` (per-graph addition).

## Zero cost when not configured

When `f:uniqueEnabled` is not set or is `false`, uniqueness checking is completely skipped — no property scan, no index queries, no overhead. The enforcement code fast-paths out immediately.

## Complete example

```trig
@prefix f:  <https://ns.flur.ee/db#> .
@prefix ex: <http://example.org/ns/> .

# 1. Define unique annotations in the default graph
ex:email f:enforceUnique true .

# 2. Enable enforcement in the config graph
GRAPH <urn:fluree:mydb:main#config> {
  <urn:fluree:mydb:main:config:ledger> a f:LedgerConfig ;
    f:transactDefaults [
      f:uniqueEnabled true
    ] .
}
```

After this transaction, the **next** transaction that attempts to give two subjects the same `ex:email` value (within the same graph) will be rejected.


---

# security/README.md

# Security and Policy

Fluree provides comprehensive security features including authentication, fine-grained access control through policies, and transparent encryption of data at rest.

## Authentication

### [Authentication](authentication.md)

Fluree's authentication model, covering:
- Identity vs transport (DIDs, signed requests, Bearer tokens)
- Three auth modes: decentralized did:key, standalone server tokens, OIDC/OAuth2
- Bearer token claim set and scope definitions
- Replication vs query access boundary
- Token verification paths (Ed25519 + OIDC/JWKS)

## Data Encryption

### [Storage Encryption](encryption.md)

Protect data at rest with AES-256-GCM encryption:
- Transparent encryption/decryption
- Environment variable key configuration
- Portable ciphertext format
- Key rotation support

## Commit Integrity

### [Commit Signing and Attestation](commit-signing.md)

Cryptographic proof of which node wrote a commit:
- Ed25519 signatures over domain-separated commit digests
- Embedded signature blocks in commit files
- did:key signer identities
- Future: detached attestations and consensus policies

## Policy System

### [Policy Model and Inputs](policy-model.md)

Understanding Fluree's policy architecture:
- Policy structure and syntax
- Subject, action, resource model
- Policy evaluation order
- Input data for policy decisions
- Default allow vs default deny

### [Policy in Queries](policy-in-queries.md)

How policies affect query execution:
- Query-time filtering
- Result set restrictions
- Pattern-based filtering
- Performance considerations
- Policy debugging for queries

### [Policy in Transactions](policy-in-transactions.md)

How policies affect transaction operations:
- Transaction validation
- Authorization checks
- Entity-level permissions
- Property-level permissions
- Policy-based retractions

### [Programmatic Policy API (Rust)](programmatic-policy.md)

Using policies in Rust applications:
- `wrap_identity_policy_view` - Identity-based policy lookup via `f:policyClass`
- `wrap_policy_view` - Inline policies with `QueryConnectionOptions`
- Policy precedence rules
- Transaction-side policy enforcement
- Historical views with policy

### [Cross-ledger governance](cross-ledger-policy.md)

Govern many data ledgers from one model ledger via any of the
five `f:GraphRef`-shaped predicates with `f:ledger`:
- Two-ledger configuration pattern (model M, data D)
- Cross-ledger policy (`f:policySource`): `f:policyClass`
  filtering, `f:AccessPolicy` baseline, engaging the policy
  path over HTTP
- Cross-ledger uniqueness constraints (`f:constraintsSource`):
  tx-time enforcement of M's `f:enforceUnique` annotations on
  D's transactions
- Cross-ledger schema/ontology (`f:schemaSource`): M's RDFS/OWL
  axioms feed D's reasoner (single graph, transitive
  `owl:imports` deferred)
- Cross-ledger SHACL shapes (`f:shapesSource`): M's shape
  definitions compile against D's staged namespace at
  validation time
- Cross-ledger datalog rules (`f:rulesSource`): M's `f:rule`
  JSON bodies feed D's query-time datalog evaluator
- Failure modes and HTTP status mapping
- Cache and update semantics

## Key Concepts

### Data-Level Security

Fluree enforces security at the data level, not just the application level:
- Users see only authorized data
- Policies applied during query execution
- No unauthorized data leakage
- Transparent to applications

### Policy as Data

Policies are stored as RDF triples in the database:
- Version controlled with data
- Query policies like any data
- Time travel for policy history
- Policies can reference other data

### Identity-Based Access

Policies use decentralized identifiers (DIDs):
- did:key for cryptographic identity
- did:web for organization identity
- Signed requests link to DID
- Policies grant/deny based on DID

## Policy Structure

Basic policy format:

```json
{
  "@context": {
    "f": "https://ns.flur.ee/db#",
    "ex": "http://example.org/ns/"
  },
  "@id": "ex:read-policy",
  "@type": "f:Policy",
  "f:subject": "did:key:z6Mkh...",
  "f:action": "query",
  "f:resource": {
    "@type": "schema:Person"
  },
  "f:allow": true
}
```

**Subject:** Who (DID, role, group)
**Action:** What operation (query, transact)
**Resource:** Which data (type, predicate, specific entities)
**Allow/Deny:** Grant or deny access

## Policy Enforcement Points

### Query Time

Policies filter query results:

```sparql
SELECT ?name
WHERE {
  ?person schema:name ?name .
}
```

Policy filters results to only show authorized people.

### Transaction Time

Policies validate transaction operations:

```json
{
  "@graph": [
    { "@id": "ex:alice", "schema:age": 31 }
  ]
}
```

Policy checks if user can modify ex:alice.

## Common Policy Patterns

### Allow All (Development)

```json
{
  "@id": "ex:allow-all",
  "f:subject": "*",
  "f:action": "*",
  "f:allow": true
}
```

### Role-Based Access

```json
{
  "@id": "ex:admin-policy",
  "f:subject": { "ex:role": "admin" },
  "f:action": "*",
  "f:allow": true
}
```

### Resource-Type Based

```json
{
  "@id": "ex:public-data-policy",
  "f:subject": "*",
  "f:action": "query",
  "f:resource": { "@type": "ex:PublicData" },
  "f:allow": true
}
```

### Property-Level Access

```json
{
  "@id": "ex:sensitive-property-policy",
  "f:subject": { "ex:role": "hr" },
  "f:action": "query",
  "f:resource": {
    "f:predicate": "ex:salary"
  },
  "f:allow": true
}
```

### Owner-Based Access

```json
{
  "@id": "ex:owner-policy",
  "f:subject": "?user",
  "f:action": ["query", "transact"],
  "f:resource": {
    "ex:owner": "?user"
  },
  "f:allow": true
}
```

## Policy Evaluation

### Evaluation Order

1. **Collect applicable policies** based on subject, action, resource
2. **Evaluate each policy** against request context
3. **Combine results** using policy combining algorithm
4. **Apply default** if no policies match

### Combining Algorithms

**Deny Overrides** (default):
- If any policy denies, access denied
- Otherwise, allow if any policy allows
- Default: deny if no matches

**Allow Overrides:**
- If any policy allows, access granted
- Otherwise, deny if any policy denies
- Default: deny if no matches

## Policy Context

Policies have access to runtime context:

**Request Context:**
- Subject DID
- Action being performed
- Target resource/entity
- Timestamp

**Data Context:**
- Entity properties
- Related entities
- Graph structure
- Historical data

**Example using context:**
```json
{
  "f:subject": "?user",
  "f:resource": {
    "ex:department": "?dept"
  },
  "f:condition": "?user ex:department ?dept",
  "f:allow": true
}
```

Allows access if user is in same department as resource.

## Multi-Tenant Policies

Isolate data by tenant:

```json
{
  "@id": "ex:tenant-isolation-policy",
  "f:subject": "?user",
  "f:action": "*",
  "f:resource": {
    "ex:tenant": "?tenant"
  },
  "f:condition": "?user ex:tenant ?tenant",
  "f:allow": true
}
```

Users can only access data from their tenant.

## Policy Performance

### Efficient Policies

Good (specific):
```json
{
  "f:resource": { "@type": "ex:PublicData" },
  "f:allow": true
}
```

Less efficient (broad):
```json
{
  "f:resource": { "?pred": "?value" },
  "f:condition": "complex graph pattern",
  "f:allow": true
}
```

### Query Optimization

Policies are optimized during query planning:
- Type-based filters pushed down
- Property filters optimized
- Complex patterns may impact performance

## Policy Management

### Creating Policies

Policies are created via transactions:

```bash
curl -X POST "http://localhost:8090/v1/fluree/upsert?ledger=policies:main" \
  -H "Content-Type: application/json" \
  -d '{
    "@graph": [
      {
        "@id": "ex:new-policy",
        "@type": "f:Policy",
        "f:subject": "did:key:z6Mkh...",
        "f:action": "query",
        "f:allow": true
      }
    ]
  }'
```

### Updating Policies

Update using WHERE/DELETE/INSERT:

```json
{
  "where": [
    { "@id": "ex:policy-1", "f:allow": "?oldValue" }
  ],
  "delete": [
    { "@id": "ex:policy-1", "f:allow": "?oldValue" }
  ],
  "insert": [
    { "@id": "ex:policy-1", "f:allow": false }
  ]
}
```

### Policy Versioning

Policies are versioned with data:
- Time travel to see historical policies
- Audit who changed policies when
- Rollback policies if needed

## Security Best Practices

### 1. Principle of Least Privilege

Grant minimum necessary permissions:

```json
// Good: Specific permissions
{
  "f:subject": "did:key:z6Mkh...",
  "f:action": "query",
  "f:resource": { "@type": "ex:PublicData" },
  "f:allow": true
}

// Bad: Overly broad
{
  "f:subject": "did:key:z6Mkh...",
  "f:action": "*",
  "f:allow": true
}
```

### 2. Default Deny

Start with deny-all, add specific allows:

```json
// Default policy
{
  "@id": "ex:default",
  "f:subject": "*",
  "f:action": "*",
  "f:allow": false
}

// Specific allows
{
  "@id": "ex:public-read",
  "f:subject": "*",
  "f:action": "query",
  "f:resource": { "@type": "ex:PublicData" },
  "f:allow": true
}
```

### 3. Use Roles

Define roles, not individual permissions:

```json
{
  "@id": "ex:admin-role",
  "@type": "ex:Role",
  "ex:permissions": ["read", "write", "admin"]
}

{
  "@id": "ex:role-policy",
  "f:subject": { "ex:hasRole": "ex:admin-role" },
  "f:action": "*",
  "f:allow": true
}
```

### 4. Audit Policy Changes

Track who changes policies:

```json
{
  "@id": "ex:policy-audit",
  "ex:changedBy": "did:key:z6Mkh...",
  "ex:changedAt": "2024-01-22T10:00:00Z",
  "ex:reason": "Added read access for contractors"
}
```

### 5. Test Policies

Test policies before deploying:

```javascript
async function testPolicy(policy, testCases) {
  for (const testCase of testCases) {
    const result = await evaluatePolicy(policy, testCase);
    assert.equal(result.allowed, testCase.expected);
  }
}
```

## Related Documentation

- [Verifiable Data](../concepts/verifiable-data.md) - Cryptographic signatures
- [Signed Requests](../api/signed-requests.md) - Request authentication
- [Signed Transactions](../transactions/signed-transactions.md) - Transaction signing
- [Commit Signing and Attestation](commit-signing.md) - Commit-level signatures


---

# security/authentication.md

# Authentication

Fluree supports multiple authentication mechanisms to cover different deployment scenarios — from standalone servers with no external identity provider to managed platforms using OIDC.

This document describes the authentication model, the supported modes, the bearer token claim set, and the access boundary between replication and query operations.

## Identity vs transport

### Identity (who)

Fluree policy enforcement is based on an **identity**, ideally a DID:

- **Preferred**: `did:key:...` — portable across environments, no central identity server required
- **Also possible**: other DIDs or IRIs mapped into Fluree policy (e.g. `ex:alice`)

Policies are stored as RDF triples in the ledger and evaluated at query/transaction time against the requesting identity. See [Policy model](policy-model.md) for details.

### Transport (how requests authenticate)

Two "on-the-wire" mechanisms carry the identity:

| Mechanism | Format | When to use |
|-----------|--------|-------------|
| **Signed requests** | JWS/VC envelope containing the DID | Proof-of-possession; trustless environments |
| **Bearer tokens** | `Authorization: Bearer <JWT>` | Session-based; OIDC/OAuth2 flows |

Bearer tokens are a UX and deployment convenience — they do not replace the identity model. The server extracts the identity from the token claims and enforces the same dataset policies as signed requests.

## Three supported auth modes

### Mode 1 — Decentralized: `did:key` signed requests (no IdP)

- The client holds an Ed25519 keypair and derives a `did:key:...`
- Requests are signed using JWS or Verifiable Credential format
- The server verifies the signature and uses the DID as the principal
- Dataset policies decide allow/deny

This preserves Fluree's core value: no central identity server required.

See [Signed requests (JWS/VC)](../api/signed-requests.md) for the wire format.

### Mode 2 — Standalone server with offline-minted tokens

Designed for: "stand up a server somewhere" (local dev, single-node EC2, etc.).

- An admin generates an Ed25519 keypair with `fluree token keygen`
- The admin mints a scoped Bearer token with `fluree token create`
- The admin provides the token to CLI users or stores it in a secret manager
- The server validates the token's embedded JWK signature and enforces scopes + policy

The **policy identity** remains DID-based (`fluree.identity` claim), so authorization stays dataset/policy driven even though the transport is a Bearer token.

See [CLI token command](../cli/token.md) for minting instructions.

### Mode 3 — OIDC/OAuth2 with an external identity provider

Designed for: managed platforms (e.g., any application using an OIDC provider).

- The IdP authenticates the user (device flow, PKCE, etc.)
- The application knows the user's Fluree dataset entitlements
- The application issues (or exchanges for) a **Fluree-scoped token** carrying:
  - identity (`fluree.identity` — ideally a DID)
  - ledger read/write scopes
  - optional policy class
- The server verifies the token against the provider's JWKS endpoint

This preserves separation of concerns:

- **IdP**: authentication (who logged in)
- **Application**: authorization (what they can access in Fluree)

The server must be configured with `--jwks-issuer` to trust OIDC tokens. See [Configuration — OIDC](../operations/configuration.md#oidc--jwks-token-verification).

## Bearer token claim set

All Fluree Bearer tokens (Mode 2 and Mode 3) share the same claim set. The server extracts identity and scopes from these claims regardless of how the token was signed.

### Standard JWT claims

| Claim | Required | Description |
|-------|----------|-------------|
| `iss` | Yes | Issuer — `did:key:...` for Ed25519 tokens, URL for OIDC tokens |
| `sub` | No | Subject — human-readable identity of the token holder |
| `aud` | No | Audience — target service (e.g. server URL) |
| `exp` | Yes | Expiration time (Unix timestamp) |
| `iat` | Yes | Issued-at time (Unix timestamp) |

### Fluree-specific claims

| Claim | Type | Description |
|-------|------|-------------|
| `fluree.identity` | String (IRI/DID) | Identity for policy enforcement — takes precedence over `sub` |
| `fluree.policy.class` | String (IRI) | Optional policy class for identity-based policy lookup |

### Scope claims

Scopes control which endpoints and ledgers a token can access.

#### Query scopes (`fluree.ledger.*`)

| Claim | Type | Description |
|-------|------|-------------|
| `fluree.ledger.read.all` | Boolean | Read access to all ledgers via data API |
| `fluree.ledger.read.ledgers` | Array of strings | Read access to specific ledgers |
| `fluree.ledger.write.all` | Boolean | Write access to all ledgers via data API |
| `fluree.ledger.write.ledgers` | Array of strings | Write access to specific ledgers |

#### Replication scopes (`fluree.storage.*`)

| Claim | Type | Description |
|-------|------|-------------|
| `fluree.storage.all` | Boolean | Storage/replication access to all ledgers |
| `fluree.storage.ledgers` | Array of strings | Storage/replication access to specific ledgers |

**Back-compat:** `fluree.storage.*` claims also imply data API read access for the same ledgers.

#### Populating `fluree.storage.ledgers` (multi-tenant hint)

If you run an IdP or a request-router that exchanges IdP tokens for Fluree-scoped tokens, prefer populating `fluree.storage.ledgers` rather than granting `fluree.storage.all`.

Recommended conventions for mapping IdP group/role claims to ledger scopes:

- Treat group values like `fluree:storage:<ledger-id>` (example: `fluree:storage:books:main`) as permission to replicate that ledger.
- Optionally support wildcards at the router boundary (example: `fluree:storage:books:*` expands to the set of ledgers your router knows about under `books:`).
- Reserve `fluree.storage.all=true` for admin/service accounts.

#### Event scopes (`fluree.events.*`)

| Claim | Type | Description |
|-------|------|-------------|
| `fluree.events.all` | Boolean | SSE event stream for all ledgers |
| `fluree.events.ledgers` | Array of strings | SSE event stream for specific ledgers |

### Example token payload

```json
{
  "iss": "https://solo.example.com",
  "sub": "alice@example.com",
  "aud": "https://fluree.example.com",
  "exp": 1700000000,
  "iat": 1699996400,
  "fluree.identity": "did:key:z6MkhaXgBZDvotDkL5257faiztiGiC2QtKLGpbnnEGta2doK",
  "fluree.ledger.read.all": true,
  "fluree.ledger.write.ledgers": ["mydb:main", "mydb:staging"]
}
```

## Token verification paths

The server supports two verification paths, selected automatically based on the JWT header:

| JWT header | Path | Algorithm | Trust model |
|------------|------|-----------|-------------|
| Contains `jwk` (embedded key) | Ed25519 / did:key | EdDSA | Issuer trust checked against `--events-auth-trusted-issuer` (or admin/storage equivalents) |
| Contains `kid` (key ID) | OIDC / JWKS | RS256 | Issuer must match a `--jwks-issuer`; key fetched from JWKS endpoint |

This dual-path dispatch is transparent to callers — the same `Authorization: Bearer <token>` header works for both paths. The server applies the same scope and identity enforcement regardless of which path verified the signature.

## Replication vs query access boundary

Fluree draws a hard boundary between **replication-scoped** and **query-scoped** access.

### Replication access (`fluree.storage.*`)

Replication operations — nameservice sync, storage proxy reads, and CLI `fetch`/`pull`/`push` — require **root-level** `fluree.storage.*` claims. These operations transfer raw commit data and index blocks; they bypass dataset policy because the data must be bit-identical to what the transaction server wrote.

Replication tokens are intended for **operator and service-account use** (e.g. a peer server's storage-proxy token, or an admin's CLI pull/push workflow). They should never be issued to end users.

### Query access (`fluree.ledger.read/write.*`)

Query operations — `/v1/fluree/query/{ledger...}`, `/v1/fluree/insert/{ledger...}`, connection-scoped SPARQL, etc. — use `fluree.ledger.read/write.*` claims. These go through the full query engine and dataset policy enforcement. The server never exposes raw storage bytes through query endpoints.

Query tokens are appropriate for **end users and application service accounts**. Combined with a `fluree.identity` claim and dataset policies, the server enforces fine-grained row- and property-level access control.

### CLI consequence: `track` vs `pull`

| Command | Access type | Required scope | What happens |
|---------|-------------|----------------|--------------|
| `fluree pull` | Replication | `fluree.storage.*` | Downloads raw commits and indexes into local storage |
| `fluree track` | Query | `fluree.ledger.read/write.*` | Registers a remote ledger; queries forwarded to server |

If a user holds only query-scoped tokens, they **cannot** clone or pull a ledger. They can only `track` it and issue queries/transactions against the remote.

## Identity precedence

When multiple identity signals are present, the server uses this precedence (highest first):

1. **Signed request DID** — proof-of-possession from JWS/VC signature
2. **Bearer token `fluree.identity`** — identity claim in the token
3. **Client-provided headers/body** — only honored when the server is in unauthenticated mode

When auth is present, the server forces `opts.identity` (and optional policy class) from the token, ignoring any client-provided identity in headers or request bodies. This prevents identity spoofing.

## Endpoint coverage

All Bearer-token-authenticated endpoints support both Ed25519 and OIDC verification paths:

| Endpoint group | Extractor | Scopes checked |
|----------------|-----------|----------------|
| Data API (query/update/info/exists) | `MaybeDataBearer` | `fluree.ledger.read/write.*` |
| Admin (create/drop) | `require_admin_token` | Issuer trust |
| Events (SSE) | `MaybeBearer` | `fluree.events.*` |
| Storage proxy | `StorageProxyBearer` | `fluree.storage.*` |
| Nameservice refs | `StorageProxyBearer` | `fluree.storage.*` |

MCP endpoints currently use the Ed25519 path only.

## Security notes

- Tokens are validated server-side on every request; client-side validation is never trusted
- Out-of-scope ledgers return `404` (not `403`) to avoid existence leaks
- `fluree.storage.*` tokens grant raw data access — issue only to trusted operators
- Connection-scoped SPARQL (`FROM`/`FROM NAMED`) requires all referenced ledgers to be within the token's read scope

## See also

- [Signed requests (JWS/VC)](../api/signed-requests.md) — Wire format for signed requests
- [Configuration — OIDC](../operations/configuration.md#oidc--jwks-token-verification) — Server OIDC/JWKS setup
- [CLI auth command](../cli/auth.md) — Managing tokens on remotes
- [CLI token command](../cli/token.md) — Minting Ed25519 tokens
- [Auth contract (CLI ↔ Server)](../design/auth-contract.md) — Discovery, exchange, and refresh protocol
- [Policy model](policy-model.md) — Dataset-level access control


---

# security/encryption.md

# Storage Encryption

Fluree supports transparent encryption of data at rest using AES-256-GCM authenticated encryption. When enabled, all data written to storage is automatically encrypted, and data is decrypted transparently when read.

## Overview

**Key Features:**
- **AES-256-GCM**: Industry-standard authenticated encryption with integrity protection
- **Transparent Operation**: Encryption/decryption happens automatically on read/write
- **All Storage Backends**: Works natively with file, S3, and memory storage
- **Portable Ciphertext**: Encrypted data can be moved between storage backends (file ↔ S3)
- **Environment Variable Support**: Keys can be loaded from environment variables
- **Secure Key Handling**: Key material in `EncryptionKey` is zeroized on drop

## Quick Start

### Rust API

```rust
use fluree_db_api::FlureeBuilder;

// Option 1: Direct key (for testing)
let key: [u8; 32] = /* your 32-byte key */;
let fluree = FlureeBuilder::file("/data/fluree")
    .build_encrypted(key)?;

// Option 2: Base64-encoded key
let fluree = FlureeBuilder::file("/data/fluree")
    .with_encryption_key_base64("your-base64-encoded-32-byte-key")?
    .build_encrypted_from_config()?;

// Option 3: From JSON-LD config with env var
let config = serde_json::json!({
    "@context": {"@vocab": "https://ns.flur.ee/system#"},
    "@graph": [{
        "@type": "Connection",
        "indexStorage": {
            "@type": "Storage",
            "filePath": "/data/fluree",
            "AES256Key": {"envVar": "FLUREE_ENCRYPTION_KEY"}
        }
    }]
});
let fluree = FlureeBuilder::from_json_ld(&config)?
    .build_encrypted_from_config()?;
```

### Server Configuration

Set the encryption key via environment variable:

```bash
# Generate a secure 32-byte key and base64 encode it
export FLUREE_ENCRYPTION_KEY=$(openssl rand -base64 32)

# Start the server with JSON-LD config
./fluree-db-server --config config.jsonld
```

## Configuration

### JSON-LD Configuration

The encryption key is specified in the storage configuration using `AES256Key`:

```json
{
  "@context": {
    "@base": "https://example.org/config/",
    "@vocab": "https://ns.flur.ee/system#"
  },
  "@graph": [
    {
      "@id": "indexStorage",
      "@type": "Storage",
      "filePath": "/var/lib/fluree/data",
      "AES256Key": {
        "envVar": "FLUREE_ENCRYPTION_KEY"
      }
    },
    {
      "@id": "mainConnection",
      "@type": "Connection",
      "indexStorage": {"@id": "indexStorage"},
      "cacheMaxMb": 2000
    }
  ]
}
```

### Configuration Options

| Field | Type | Description |
|-------|------|-------------|
| `AES256Key` | string or object | Base64-encoded 32-byte encryption key |
| `AES256Key.envVar` | string | Environment variable containing the key |
| `AES256Key.defaultVal` | string | Fallback key if env var is not set |

### Environment Variable Indirection

You can load the encryption key from an environment variable:

```json
{
  "AES256Key": {
    "envVar": "FLUREE_ENCRYPTION_KEY"
  }
}
```

Or with a fallback default (not recommended for production):

```json
{
  "AES256Key": {
    "envVar": "FLUREE_ENCRYPTION_KEY",
    "defaultVal": "fallback-base64-key-for-dev-only"
  }
}
```

## Key Management

### Generating Keys

Generate a cryptographically secure 32-byte key:

```bash
# Using OpenSSL (recommended)
openssl rand -base64 32

# Using /dev/urandom
head -c 32 /dev/urandom | base64

# Example output: "K7gNU3sdo+OL0wNhqoVWhr3g6s1xYv72ol/pe/Unols="
```

### Key Storage Best Practices

1. **Never commit keys to version control**
2. **Use environment variables or secret managers**
3. **Rotate keys periodically** (see Key Rotation below)
4. **Limit access to key material**

Recommended secret management solutions:
- HashiCorp Vault
- AWS Secrets Manager
- Kubernetes Secrets
- Docker secrets

### Key Rotation

The encryption envelope format includes a `key_id` field to support key rotation:

1. **Existing data** continues to be readable with the old key
2. **New writes** use the new key
3. **Re-encrypt on read** (optional): Decrypt with old key, re-encrypt with new key

> **Note**: Full key rotation support with `KeyProvider` trait is planned for a future release. Currently, a single static key is used.

## Encryption Details

### Algorithm

- **Cipher**: AES-256-GCM (Galois/Counter Mode)
- **Key Size**: 256 bits (32 bytes)
- **Nonce Size**: 96 bits (12 bytes), randomly generated per write
- **Tag Size**: 128 bits (16 bytes)

### Ciphertext Envelope Format

All encrypted data uses a portable envelope format:

```
┌──────────────────────────────────────────────────────────────┐
│ Header (22 bytes)                                            │
├──────────┬─────────┬─────────┬──────────┬───────────────────┤
│ Magic    │ Version │ Alg     │ Key ID   │ Nonce             │
│ 4 bytes  │ 1 byte  │ 1 byte  │ 4 bytes  │ 12 bytes          │
│ "FLU\0"  │ 0x01    │ 0x01    │ uint32   │ random            │
├──────────┴─────────┴─────────┴──────────┴───────────────────┤
│ Ciphertext (variable length)                                 │
├──────────────────────────────────────────────────────────────┤
│ Authentication Tag (16 bytes)                                │
└──────────────────────────────────────────────────────────────┘
```

- **Magic bytes**: `FLU\0` (0x46 0x4C 0x55 0x00) for format detection
- **Version**: Format version (currently 0x01)
- **Algorithm**: 0x01 = AES-256-GCM
- **Key ID**: Identifier for key rotation support
- **Nonce**: Randomly generated per encryption operation
- **Authentication Tag**: GCM integrity tag (authenticates header + ciphertext)

### Security Properties

1. **Confidentiality**: AES-256 encryption protects data content
2. **Integrity**: GCM authentication tag detects tampering
3. **Authenticity**: Header is included in AAD (Additional Authenticated Data)
4. **Non-deterministic**: Random nonces mean same plaintext → different ciphertext

## Portability

Encrypted data is portable between storage backends:

```bash
# Encrypted files can be copied from local storage to S3
aws s3 sync /var/lib/fluree/data s3://my-bucket/fluree/

# And back again
aws s3 sync s3://my-bucket/fluree/ /var/lib/fluree/data
```

The same encryption key will decrypt data regardless of where it's stored.

## Performance Considerations

- **CPU overhead**: ~5-15% for encryption/decryption (depends on hardware AES support)
- **Storage overhead**: 22 bytes header + 16 bytes tag per object
- **Memory**: Keys are kept in memory while the connection is open

Modern CPUs with AES-NI instructions provide hardware acceleration, minimizing the performance impact.

## Troubleshooting

### Common Errors

**"Invalid encryption format"**
- The data doesn't have the expected magic bytes
- Possible causes: trying to read unencrypted data with encryption enabled, or corrupted data

**"Unknown encryption key ID"**
- The data was encrypted with a different key than what's configured
- Check that the correct key is being used

**"Decryption failed"**
- The encryption key doesn't match
- The data may be corrupted
- The authentication tag verification failed (data was tampered with)

**"Encryption key must be 32 bytes"**
- The provided key is the wrong length
- Base64-decode your key and verify it's exactly 32 bytes

### Verifying Encryption

Check if a file is encrypted by looking for the magic bytes:

```bash
# Check first 4 bytes of a file
xxd -l 4 /var/lib/fluree/data/some-file
# Encrypted: 00000000: 464c 5500  FLU.
# Unencrypted: will show different bytes (likely JSON or Avro magic)
```

## Changing Encryption Settings

### Enabling Encryption on Existing Data

To encrypt existing unencrypted data:

1. **Export** all ledgers to JSON-LD
2. **Delete** the old unencrypted data directory
3. **Configure** encryption with a new key
4. **Import** the JSON-LD data

```bash
# 1. Export (while running without encryption)
fluree export mydb:main --format json-ld > mydb-export.jsonld

# 2. Stop server and backup/delete old data
mv /var/lib/fluree/data /var/lib/fluree/data-unencrypted-backup

# 3. Configure encryption key
export FLUREE_ENCRYPTION_KEY=$(openssl rand -base64 32)
echo "Save this key securely: $FLUREE_ENCRYPTION_KEY"

# 4. Start server with encryption config and import
./fluree-db-server --config encrypted-config.jsonld
fluree create mydb --from mydb-export.jsonld
```

### Disabling Encryption

> **Warning**: This exposes your data. Only do this if absolutely necessary.

Follow the same export/import process, but configure without an encryption key.

## Related Documentation

- [Storage Modes](../operations/storage.md) - Storage backend configuration
- [Configuration](../operations/configuration.md) - General configuration reference
- [Policy Model](policy-model.md) - Access control and authorization


---

# security/commit-signing.md

# Commit Signing and Attestation

Fluree supports cryptographic signing at two levels:

1. **Transaction signatures** prove **who submitted** a transaction (user-facing). See [Signed Transactions](../transactions/signed-transactions.md).
2. **Commit signatures** prove **which node wrote** a commit (infrastructure-facing). This page covers commit signatures.

Both use **did:key** identifiers with **Ed25519** signatures, aligning with the credential infrastructure in `fluree-db-credential`.

**Note:** Requires the `credential` feature flag. See [Compatibility and Feature Flags](../reference/compatibility.md#fluree-db-api-features).

## Transaction Signatures vs Commit Signatures

These two signature types serve different purposes:

| | Transaction Signature | Commit Signature |
|---|---|---|
| **Proves** | Who submitted the transaction | Which node wrote the commit |
| **Signed by** | End user (client-side) | Fluree node (server-side) |
| **Trust model** | User authentication | Infrastructure integrity |
| **Format** | JWS / Verifiable Credential | Domain-separated Ed25519 over commit hash |
| **Stored in** | Commit envelope (`txn_signature`) | Trailing signature block after commit hash |

A single commit can have both: a transaction signature from the user who submitted it, and a commit signature from the node that wrote it.

## How Commit Signing Works

### Commit Digest

When a commit is written, its content is hashed with SHA-256 to produce a `commit_hash`. The signing digest is then computed with domain separation to prevent cross-protocol and cross-ledger replay:

```text
to_sign = SHA-256("fluree/commit/v1" || varint(ledger_id.len()) || ledger_id || commit_hash)
```

Where:
- `"fluree/commit/v1"` is a domain separator (18 bytes ASCII)
- `ledger_id` is the ledger ID (`name:branch`, length-prefixed)
- `commit_hash` is the 32-byte SHA-256 of the commit content

### Signature Block Layout

The signature block is appended **after** the commit hash and is not covered by it:

```text
+-------------------------------------+
| Header (32 bytes)                   |
|   flags: includes HAS_COMMIT_SIG    |
+-------------------------------------+
| Envelope + Ops + Dictionaries       |
+-------------------------------------+
| Footer (64 bytes)                   |
+-------------------------------------+
| commit_hash (32 bytes)              |
+-------------------------------------+
| Signature Block (optional)          |  <-- after hash boundary
|   sig_count: u16                    |
|   signatures: [CommitSignature]     |
+-------------------------------------+
```

This design means:
- `commit_hash` is stable regardless of signatures
- Signatures can be added without changing the commit's content address
- Existing verification (hash check) works unchanged

### Signature Entry Format

Each signature entry contains:

| Field | Type | Description |
|-------|------|-------------|
| `signer` | String | Signer identity (`did:key:z6Mk...`) |
| `algo` | u8 | Signing algorithm (`0x01` = Ed25519) |
| `signature` | [u8; 64] | Ed25519 signature bytes |
| `timestamp` | i64 | Signing time (epoch millis, informational only) |
| `metadata` | Option\<Vec\<u8\>\> | Optional metadata (node_id, region, role for consensus) |

The `algo` byte provides forward compatibility for new signature algorithms. Unknown `algo` values are rejected on decode (not silently skipped).

The `timestamp` is informational only and is **not** part of the signed digest. Ordering is determined by the commit chain, not by signature timestamps.

The `metadata` field is reserved for future consensus features (multi-node signing, quorum sets). It allows nodes to include identifying information like node ID, region, or role. Currently unused but present in the format to avoid future versioning.

## Enabling Commit Signing (Rust API)

Commit signing is opt-in via `CommitOpts` when using the Rust API:

```rust
use std::sync::Arc;
use fluree_db_novelty::SigningKey;

// Load or generate an Ed25519 signing key
let signing_key = Arc::new(SigningKey::from_bytes(&key_bytes));

// Attach to commit options
let opts = CommitOpts::default()
    .with_signing_key(signing_key);
```

When a signing key is present, the commit writer:
1. Computes the domain-separated digest from the commit hash and ledger ID
2. Signs the digest with Ed25519
3. Appends the signature block after the commit hash
4. Sets the `FLAG_HAS_COMMIT_SIG` bit in the header

## Verifying Commit Signatures

Verification recomputes the domain-separated digest and checks the Ed25519 signature:

```rust
use fluree_db_credential::verify_commit_digest;

verify_commit_digest(
    &signer_did,       // "did:key:z6Mk..."
    &signature_bytes,  // [u8; 64]
    &commit_hash,      // [u8; 32]
    ledger_id,           // "mydb:main"
)?;
```

The verifier:
1. Extracts the Ed25519 public key from the `did:key` identifier
2. Recomputes `to_sign = SHA-256("fluree/commit/v1" || varint(ledger_id.len()) || ledger_id || commit_hash)`
3. Verifies the signature over `to_sign`

No external key registry is needed for `did:key` identifiers — the public key is embedded in the DID itself.

## Wire Format

Each `CommitSignature` is encoded as:

```
signer_len:   u16 (LE)          - length of signer string
signer:       [u8; signer_len]  - UTF-8 did:key identifier
algo:         u8                - signature algorithm (0x01 = Ed25519)
signature:    [u8; 64]          - Ed25519 signature bytes
timestamp:    i64 (LE)          - signing timestamp (epoch millis)
meta_len:     u16 (LE)          - metadata length (0 if none)
metadata:     [u8; meta_len]    - optional metadata bytes
```

The signature block is prefixed with `sig_count: u16` (LE) containing the number of signatures.

## Security Properties

### Replay Prevention

- **Cross-ledger:** The ledger ID is part of the signed digest, so a signature from ledger A cannot be replayed on ledger B
- **Cross-protocol:** The domain separator `"fluree/commit/v1"` prevents signatures meant for other systems from being accepted
- **Version upgrade:** Changing the domain separator (e.g., `v1` to `v2`) invalidates old signatures

### What Commit Signatures Do Not Provide

- **Transaction authorization:** Use [transaction signatures](../transactions/signed-transactions.md) and [policies](policy-model.md) for user-level access control
- **Consensus:** A single commit signature proves one node wrote it. Multi-node consensus requires attestation policies (see below)
- **Encryption:** Commit signing provides integrity and authenticity, not confidentiality. See [Storage Encryption](encryption.md) for data-at-rest protection

## Future: Attestations and Consensus Policy

The following capabilities are designed but not yet implemented.

### Detached Attestations

For multi-node deployments, signatures can be collected as separate attestation objects rather than embedded in the commit:

- Commit file remains immutable and content-addressed
- Signatures collected asynchronously from multiple nodes
- No coordination needed during commit write
- Attestations from different nodes can arrive at different times

### Consensus Policy

Consensus policy will define how many signatures are required for a commit to be accepted:

- **None:** No signatures required (default)
- **Single signer:** One designated writer must sign
- **Threshold (K-of-N):** At least K signatures from an allowlist of N signers
- **Quorum set:** At least one signature from each required group

Policy validation runs after commit hash integrity check, before accepting the commit.

## Related Documentation

- [Signed Transactions](../transactions/signed-transactions.md) — User-facing transaction signing (JWS/VC)
- [Verifiable Data](../concepts/verifiable-data.md) — Cryptographic verification concepts
- [Storage Encryption](encryption.md) — Data-at-rest encryption
- [Commit Receipts](../transactions/commit-receipts.md) — Commit metadata and content hashes


---

# security/policy-model.md

# Policy Model and Inputs

This is the reference for Fluree's access-control policy model. For a conceptual introduction, see [Policy enforcement](../concepts/policy-enforcement.md). For worked examples, see the [policy cookbook](../guides/cookbook-policies.md). For Rust-side wiring (building a `PolicyContext`, `wrap_identity_policy_view`, transaction helpers), see [Programmatic policy API](programmatic-policy.md).

## Policy node shape

Every policy is a JSON-LD node. Required `@type`: `f:AccessPolicy` (the IRI is `https://ns.flur.ee/db#AccessPolicy`). A second class IRI (e.g. `ex:CorpPolicy`) is conventional and allows the policy to be loaded by `policy-class`.

```json
{
  "@id": "ex:somePolicy",
  "@type": ["f:AccessPolicy", "ex:CorpPolicy"],
  "f:required": true,
  "f:onProperty": [{"@id": "ex:salary"}],
  "f:onClass":    [{"@id": "ex:Employee"}],
  "f:onSubject":  [{"@id": "ex:alice"}],
  "f:action": [{"@id": "f:view"}, {"@id": "f:modify"}],
  "f:query": "<JSON-encoded WHERE>",
  "f:allow": true,
  "f:exMessage": "Reason returned to caller on denial"
}
```

### Predicate reference

| Predicate | Type | Required? | Description |
|-----------|------|-----------|-------------|
| `f:action` | array of IRIs (or single IRI string) | yes | Which operations the policy governs. Values: `f:view` (queries), `f:modify` (transactions). |
| `f:allow` | boolean | one of `f:allow` / `f:query` | Static decision. `true` permits, `false` denies. Takes precedence over `f:query` if both are present. |
| `f:query` | string (JSON-encoded JSON-LD WHERE) | one of `f:allow` / `f:query` | Dynamic decision. The targeted flake is permitted when the query returns at least one row. `?$this` and `?$identity` are pre-bound. |
| `f:onProperty` | array of `@id` references | no | Restrict the policy to flakes whose predicate is one of these IRIs. |
| `f:onClass` | array of `@id` references | no | Restrict the policy to flakes whose subject has one of these `rdf:type`s. |
| `f:onSubject` | array of `@id` references | no | Restrict the policy to flakes whose subject IRI is one of these. |
| `f:required` | boolean | no, defaults to `false` | When `true`, the policy MUST allow for access to its targets to be granted, regardless of `default-allow`. |
| `f:exMessage` | string | no | User-facing error message returned when this policy denies a transaction. |

If neither `f:allow` nor `f:query` is present, the policy is **deny by default**.

If multiple targeting predicates are present, they intersect: the policy applies only to flakes that match the property AND the class AND the subject sets.

If all targeting predicates are omitted, the policy is a **default policy** that applies to every flake of its `f:action`s.

### Action values

`f:action` carries IRIs in the `f:` namespace:

- `"f:view"` (or `{"@id": "f:view"}`) — queries.
- `"f:modify"` (or `{"@id": "f:modify"}`) — transactions.
- Both: `[{"@id": "f:view"}, {"@id": "f:modify"}]`.

A policy with no `f:action` defaults to applying to both view and modify.

## `f:query` syntax

`f:query` is a string containing a JSON-encoded JSON-LD query. The engine parses the string and runs the query as a subquery for each candidate flake, with two pre-bound variables:

| Variable | Binding |
|----------|---------|
| `?$this` | The IRI of the subject being read or written. |
| `?$identity` | The IRI of the requesting identity (resolved from `opts.identity`, `policy_values["?$identity"]`, or the verified bearer-token subject). |

Anything else binds via the embedded WHERE just like a normal Fluree query.

Because RDF can't carry structured JSON values natively, stored policies must JSON-encode the query (`serde_json::to_string`). For inline policies passed via `opts.policy`, you can also use the JSON-LD typed-literal form `{"@type": "@json", "@value": {...}}` to avoid manually escaping.

Example (string form, suitable for storing in a transaction):

```json
"f:query": "{\"where\": {\"@id\": \"?$identity\", \"http://example.org/role\": \"hr\"}}"
```

Example (typed-literal form, suitable for inline policies):

```json
"f:query": {
  "@type": "@json",
  "@value": {
    "where": {"@id": "?$identity", "http://example.org/role": "hr"}
  }
}
```

> **Inline policies must use full IRIs.** Compact IRIs (`schema:ssn`) inside an inline policy passed through `opts.policy` are not expanded against the request `@context`. Use full IRIs (`http://schema.org/ssn`).

## Combining algorithm

When more than one policy targets the same flake, the engine combines them as follows:

1. If any **required** policy (`f:required: true`) targets the flake and does not allow it (either `f:allow: false`, missing `f:allow`, or `f:query` returning no rows), access is **denied** for that flake. Required policies are *gates*: they cannot be overridden by other allows or by `default-allow`.
2. If at least one targeted (but not required) policy allows the flake, access is **granted**. Non-required allows combine with allow-overrides semantics.
3. If a targeted policy's `f:query` returns false (no rows), that policy *applied but did not permit* — the flake is denied even if `default-allow` is `true`. Default-allow only applies when **no** policy targets the flake.
4. If no policies target the flake, `default-allow` decides. `false` denies; `true` permits.

`f:allow` always takes precedence over `f:query`: if both are set on the same policy, `f:allow` wins.

For a deeper treatment, including the three-state identity resolution semantics (`FoundWithPolicies` / `FoundNoPolicies` / `NotFound`), see the [Policy combining algorithm](programmatic-policy.md#policy-combining-algorithm) section in the programmatic policy API reference.

## Default-allow

`default-allow` is the fallback decision for flakes that no policy targets:

| Setting | Behavior |
|---------|----------|
| `default-allow: false` | Fail-closed. A flake with no targeting policies is denied. **Recommended for production.** |
| `default-allow: true` | Fail-open. A flake with no targeting policies is allowed. Useful in development or in deployments where an application layer handles authorization and Fluree is recording signed transactions for provenance. |

Important: `default-allow: true` does **not** override required policies that fail. It only governs the no-policy case.

## Identity resolution

When `opts.identity` is set, Fluree resolves it to a `?$identity` SID and applies the identity's `f:policyClass` automatically — every stored policy of that class is loaded into the request's policy set.

The resolution path:

```
opts.identity  →  policy_class               →  policy             →  policy_values["?$identity"]
   (highest)                                                                  (lowest)
```

If multiple are set, the higher-priority binding wins. `policy_values["?$identity"]` is a manual escape hatch — useful when you want to test a specific identity SID without going through the full resolution path.

A request with no identity supplied uses an "anonymous" context: only inline policies, no class-based discovery, no `?$identity` binding.

## Where policies come from

Two delivery paths, often combined:

### Stored policies

Persist policies as data in the ledger. The policy node carries the class type alongside `f:AccessPolicy`:

```json
{
  "@id": "ex:salary-restriction",
  "@type": ["f:AccessPolicy", "ex:CorpPolicy"],
  ...
}
```

Identities tag themselves with `f:policyClass`:

```json
{
  "@id": "ex:aliceIdentity",
  "ex:user": {"@id": "ex:alice"},
  "f:policyClass": [{"@id": "ex:CorpPolicy"}]
}
```

When `opts.identity = "ex:aliceIdentity"`, every `f:AccessPolicy` whose `@type` includes `ex:CorpPolicy` is loaded for the request — no per-request policy listing needed. Stored policies are versioned, time-travelable, branchable, and consistent across all callers.

### Inline policies

Pass policies in `opts.policy` (an array of policy nodes) for ad-hoc requests:

```json
{
  "from": "mydb:main",
  "select": "?x",
  "where": [...],
  "opts": {
    "policy": [
      {"@id": "ex:adhoc", "@type": "f:AccessPolicy", "f:action": "f:view", "f:allow": true}
    ],
    "default-allow": false
  }
}
```

Useful for tests, admin scripts, and migration tooling. Inline policies and stored policies can coexist in a single request.

## Request-time options

Each request can supply these `opts` fields (JSON-LD form). Over SPARQL, the equivalent fluree-* HTTP headers carry the same values.

| `opts` field | HTTP header | Description |
|--------------|-------------|-------------|
| `identity` | `fluree-identity` | IRI of an identity entity. Drives `f:policyClass` discovery and binds `?$identity`. |
| `policy-class` | `fluree-policy-class` | Class IRI(s) to load stored policies by. Repeated header or comma-separated. |
| `policy-values` | `fluree-policy-values` | JSON object of additional `?$var` bindings injected into every policy's `f:query`. |
| `policy` | `fluree-policy` | Inline policy array (full JSON-LD). |
| `default-allow` | `fluree-default-allow` | `true` / `false`. Fallback decision for flakes that no policy targets. |

When the server is configured with `data_auth_default_policy_class`, a verified bearer token's identity claim is auto-applied to `policy-values` and the configured class to `policy-class` — no client-side opts needed. See [Configuration](../operations/configuration.md) and [Authentication](authentication.md) for the bearer-token flow.

## Read enforcement vs write enforcement

The same model governs both, distinguished by `f:action`:

- **`f:view`** — applied during query execution. Flakes that fail the policy are filtered before the query plan emits results. The query never sees them.
- **`f:modify`** — applied during transaction staging. The transaction is rejected — with `f:exMessage` if provided — when a write would touch flakes the identity isn't allowed to modify.

A single policy can govern both. See [Policy in queries](policy-in-queries.md) and [Policy in transactions](policy-in-transactions.md) for path-specific details.

## Performance notes

Two phases:

- **Load.** The relevant policies for a request are gathered once (from `policy-class` lookups + inline `policy`). Cost is small and proportional to the size of the policy set.
- **Apply.** During plan execution, each candidate flake is checked against the matching subset of the policy set. Cost is proportional to the number of touched flakes × the average per-flake check cost.

Two practical implications:

1. **Target every policy you can.** A policy with `f:onProperty` or `f:onClass` only runs on flakes whose predicate or rdf:type matches. Default policies (no targeting) run on every flake.
2. **Keep `f:query` cheap.** It runs once per targeted flake. Lean on identity-side properties already loaded (`@type`, `f:policyClass`, role flags) rather than deep traversals.

## Policies are queryable data

Because each policy is just a JSON-LD node, you can query the policies themselves:

```sparql
PREFIX f: <https://ns.flur.ee/db#>
PREFIX ex: <http://example.org/>

SELECT ?policy ?action ?onProperty
WHERE {
  ?policy a f:AccessPolicy ;
          a ex:CorpPolicy ;
          f:action ?action ;
          f:onProperty ?onProperty .
}
```

History queries against the same shape produce a complete audit trail of policy changes over time. See [Time travel](../concepts/time-travel.md) for query-at-t syntax.

## Related documentation

- [Policy enforcement (concepts)](../concepts/policy-enforcement.md) — model and architecture
- [Cookbook: Access control policies](../guides/cookbook-policies.md) — worked examples and patterns
- [Policy in queries](policy-in-queries.md) — read-time enforcement details
- [Policy in transactions](policy-in-transactions.md) — write-time enforcement details
- [Programmatic policy API (Rust)](programmatic-policy.md) — `PolicyContext`, builder helpers, combining algorithm
- [Cross-ledger policy](cross-ledger-policy.md) — one model ledger governs many data ledgers via `f:policySource` with `f:ledger`
- [Authentication](authentication.md) — identities, JWTs, bearer-token verification
- [Configuration](../operations/configuration.md) — server-side policy defaults (`data_auth_default_policy_class`, etc.)
- [Vocabulary reference](../reference/vocabulary.md#policy-vocabulary) — predicate IRIs


---

# security/policy-in-queries.md

# Policy in Queries

Query-time enforcement uses Fluree's [policy model](policy-model.md) to filter individual flakes during query execution. The query plan is the same regardless of policy — what changes is which flakes the engine returns. The application sees a query result; the policy filtering is invisible.

This page documents how query-time enforcement works, how patterns interact with the plan, and how to test policies from the CLI. For the policy node shape and combining algorithm, see the [policy model reference](policy-model.md). For the underlying concept, see [Policy enforcement](../concepts/policy-enforcement.md).

## How query-time filtering works

When a query is executed against a `PolicyContext`:

1. The engine resolves the request's policy set: identity-driven `f:policyClass` lookups + any inline `opts.policy` array.
2. The plan executes normally — same join order, same indices.
3. Each flake the plan would emit is checked against the policies whose target matches it (`f:onProperty`, `f:onClass`, `f:onSubject`, or default for untargeted policies).
4. A flake survives only if the [combining algorithm](policy-model.md#combining-algorithm) approves it.
5. Surviving flakes flow through the rest of the plan (joins, filters, aggregates) as normal.

Filtering is at the flake level — a single subject can appear in the result with some properties visible and others elided.

## Worked example

Two users in a `mydb:main` ledger:

```bash
fluree insert '{
  "@context": {"schema": "http://schema.org/", "ex": "http://example.org/"},
  "@graph": [
    {"@id": "ex:alice", "schema:name": "Alice", "ex:role": "engineer", "ex:salary": 130000},
    {"@id": "ex:bob",   "schema:name": "Bob",   "ex:role": "manager",  "ex:salary": 155000}
  ]
}'
```

A required policy that hides `ex:salary` unless the requester is a manager:

```bash
fluree insert '{
  "@context": {"f": "https://ns.flur.ee/db#", "ex": "http://example.org/"},
  "@graph": [
    {
      "@id": "ex:salary-restriction",
      "@type": ["f:AccessPolicy", "ex:CorpPolicy"],
      "f:required": true,
      "f:onProperty": [{"@id": "ex:salary"}],
      "f:action": [{"@id": "f:view"}],
      "f:query": "{\"where\": {\"@id\": \"?$identity\", \"http://example.org/role\": \"manager\"}}"
    },
    {
      "@id": "ex:default-view",
      "@type": ["f:AccessPolicy", "ex:CorpPolicy"],
      "f:action": [{"@id": "f:view"}],
      "f:allow": true
    },
    {"@id": "ex:aliceIdentity", "f:policyClass": [{"@id": "ex:CorpPolicy"}], "ex:role": "engineer"},
    {"@id": "ex:bobIdentity",   "f:policyClass": [{"@id": "ex:CorpPolicy"}], "ex:role": "manager"}
  ]
}'
```

The same query, executed as different identities:

```bash
# As Bob (manager) — sees salaries
fluree query --as ex:bobIdentity --policy-class ex:CorpPolicy \
  'SELECT ?name ?salary WHERE { ?p <http://schema.org/name> ?name ; <http://example.org/salary> ?salary }'
# → Alice 130000, Bob 155000

# As Alice (engineer) — salary flakes filtered out
fluree query --as ex:aliceIdentity --policy-class ex:CorpPolicy \
  'SELECT ?name ?salary WHERE { ?p <http://schema.org/name> ?name ; <http://example.org/salary> ?salary }'
# → no results: the join requires ?salary which is filtered for Alice
```

To get Alice's name back without the salary join, use `OPTIONAL`:

```sparql
SELECT ?name ?salary WHERE {
  ?p <http://schema.org/name> ?name .
  OPTIONAL { ?p <http://example.org/salary> ?salary }
}
```

Now Alice sees both names, with `?salary` unbound — exactly the behavior an application expects when a property is suppressed by policy.

## Enforcement across query operations

Filtering happens wherever the plan reads flakes, so every query operation enforces the same per-flake policy — not just basic triple patterns:

- **Triple patterns / BGPs** — the common case shown above; each matched flake is checked.
- **OPTIONAL, joins, aggregates, `COUNT`** — these operate over already-filtered flakes, so a hidden flake is absent from joins and excluded from counts. `SELECT (COUNT(?o) ...) WHERE { ?s ex:salary ?o }` counts only the salary flakes the identity may view.
- **Property paths** (`?x :knows+ ?y`, `:reportsTo*`, sequence/alternation) — traversal only follows edges the identity can view. A hidden `:knows` flake is not walked, so nodes reachable *only* through hidden edges do not appear in the result. You traverse exactly the graph you can see.
- **Full-text and vector search** (embedded indexes) — a search hit is returned only if the identity can view the indexed content that produced the match. If the policy hides the indexed property's flake for a subject, that subject is dropped from the search results — even when selecting only the hit id/score with no constraining pattern. (This applies to Fluree's embedded search indexes; an external/dedicated search service enforces its own access controls.)

In every case the rule is the same: the engine never emits — or traverses, or counts, or returns as a search hit — a flake the identity is not allowed to see.

## Reasoning (RDFS / OWL / datalog)

Reasoning and view policy compose, but the contract is specific:

- **OWL 2 QL** rewrites the query and runs it through the normal scan path under your identity, so it is filtered exactly like any other query.
- **OWL 2 RL and datalog** materialize *derived* facts into the query's overlay. Those derived facts are filtered by the **same per-flake view policy as base data** — a derived flake you may not view is dropped just like a stored one.

What the engine does **not** do is trace a derived fact's *provenance*: a derived flake is judged by its own `(subject, predicate, object)`, not by the base facts it was computed from. So if a rule or ontology axiom re-expresses hidden data under a different, viewable predicate — e.g. `ex:ssn rdfs:subPropertyOf ex:identifier`, or a rule that writes `ex:isHighEarner` from a hidden `ex:salary` — the derived value can surface even though the source is hidden.

**So when you enable reasoning under a non-root policy, your policy must cover the derived properties and classes.** Either deny them explicitly, or run with `default-allow: false` so any predicate you did not explicitly allow — including reasoning-introduced ones — is hidden by default. Inline per-query ontologies (`f:schemaSource`) are subject to the same rule: any class/property they entail must be covered by your policy.

Query-time rule injection (the query's `rules` field) is **admin-only**: under a non-root view policy, caller-supplied datalog rules are stripped before execution, because a rule with a viewable head could launder hidden data the policy author never anticipated. Database-stored rules (`f:rule`) and OWL/RDFS reasoning are administrator-controlled and continue to apply.

## Targeting patterns

### Property-level (`f:onProperty`)

Restricts a flake whose predicate matches:

```json
{
  "@id": "ex:hide-ssn",
  "@type": ["f:AccessPolicy", "ex:CorpPolicy"],
  "f:required": true,
  "f:onProperty": [{"@id": "http://schema.org/ssn"}],
  "f:action": [{"@id": "f:view"}],
  "f:query": "{\"where\": {\"@id\": \"?$identity\", \"http://example.org/role\": \"hr\"}}"
}
```

Flakes whose predicate is not `schema:ssn` are unaffected by this policy.

### Class-level (`f:onClass`)

Restricts flakes whose subject has one of the listed `rdf:type`s:

```json
{
  "@id": "ex:employee-data-only",
  "@type": ["f:AccessPolicy", "ex:CorpPolicy"],
  "f:required": true,
  "f:onClass": [{"@id": "http://example.org/Employee"}],
  "f:action": [{"@id": "f:view"}],
  "f:query": "{\"where\": {\"@id\": \"?$identity\", \"@type\": \"http://example.org/Employee\"}}"
}
```

Flakes about non-`Employee` subjects fall through to other policies.

### Subject-level (`f:onSubject`)

Restricts flakes about specific subjects:

```json
{
  "@id": "ex:hide-internal-doc",
  "@type": ["f:AccessPolicy", "ex:CorpPolicy"],
  "f:required": true,
  "f:onSubject": [{"@id": "http://example.org/secret-doc"}],
  "f:action": [{"@id": "f:view"}],
  "f:allow": false
}
```

### Default (no targeting)

A policy with no `f:onProperty` / `f:onClass` / `f:onSubject` applies to **every** flake. Use sparingly — default policies are evaluated against every emitted flake, which is more expensive than targeted policies.

## SPARQL queries

SPARQL queries have no `opts` block, so policy is delivered via headers:

```bash
curl -X POST 'http://localhost:8090/v1/fluree/query?ledger=mydb:main' \
  -H 'Content-Type: application/sparql-query' \
  -H "Authorization: Bearer $JWT" \
  -H 'fluree-identity: ex:aliceIdentity' \
  -H 'fluree-policy-class: ex:CorpPolicy' \
  -H 'fluree-default-allow: false' \
  -d 'SELECT ?name WHERE { ?p <http://schema.org/name> ?name }'
```

The full header set is documented in the [policy model](policy-model.md#request-time-options).

## JSON-LD queries

JSON-LD queries put policy in `opts`:

```json
{
  "from": "mydb:main",
  "select": ["?name", "?salary"],
  "where": [
    {"@id": "?p", "schema:name": "?name"},
    ["optional", {"@id": "?p", "ex:salary": "?salary"}]
  ],
  "opts": {
    "identity": "ex:aliceIdentity",
    "policy-class": ["ex:CorpPolicy"],
    "default-allow": false
  }
}
```

Inline policies, additional `policy-values`, and multiple `policy-class` entries all live under `opts`. The full vocabulary is in the [policy model reference](policy-model.md#request-time-options).

## Multi-graph queries

Policies apply per-flake, regardless of which named graph the flake came from. A query that pulls from multiple `from-named` graphs sees a uniformly filtered result — there's no per-graph policy override.

If different graphs need different policy regimes, use targeted policies (`f:onClass` for type-scoped restrictions, `f:onSubject` for explicit subject lists). For wholly separate access regimes, use separate ledgers.

## Time-travel queries

Policy evaluation honors the query's `t`. When you query `--at` a past `t`:

- The policy set itself is resolved at that `t` (so retired policies still apply when you time-travel back to when they were live).
- Identity attributes used in `f:query` are evaluated at that `t`.

This makes audit-style queries — *"What could Alice see on 2024-06-15?"* — directly expressible:

```bash
fluree query --as ex:aliceIdentity --policy-class ex:CorpPolicy --at 2024-06-15T00:00:00Z \
  'SELECT ?p ?o WHERE { <http://example.org/financial-report> ?p ?o }'
```

## Performance considerations

Two phases: load the policy set once per request; apply it to each touched flake.

- **Target policies whenever possible.** A policy with `f:onProperty` only runs against flakes whose predicate matches. Default policies (no targeting) run against every flake.
- **Keep `f:query` cheap.** It runs once per flake-target. Lean on identity-side properties already loaded (`@type`, `f:policyClass`, role flags) rather than deep traversals.
- **Avoid deep recursion in `f:query`.** Each level of indirection multiplies the per-flake cost.
- **Required policies short-circuit on the first failed gate.** Required policies are AND gates — every one must grant. As soon as any required gate fails (an explicit deny, or an `f:query` returning no rows), the flake is denied and the remaining required policies are skipped.
- **Targeted policies keep fast paths fast.** Cardinality and scalar-aggregate fast paths (e.g. a bare `COUNT` over a single predicate) still answer directly from index metadata when the active policy provably cannot restrict the scanned predicate — a policy that only targets `ex:salary` does not slow a `COUNT` over `schema:name`. A policy that *could* apply to the scanned predicate (including any default/subject-targeted policy) forces the per-flake filtered scan instead. This is another reason to prefer targeted policies.

For complex deployments, the [explain plan](../query/explain.md) shows whether a query is dominated by policy filtering and which policies contribute.

## Testing policies from the CLI

The `fluree` CLI supports policy-enforced queries so you can verify that the policies you've configured filter results as expected — without writing any client code.

### Flags

Available on `fluree query` (and on `fluree insert`, `upsert`, `update` for write-time enforcement):

| Flag | Purpose |
|------|---------|
| `--as <IRI>` | Execute as this identity. Resolves `f:policyClass` on the identity subject to collect applicable policies, and binds `?$identity`. |
| `--policy-class <IRI>` | Apply stored policies of the given class IRI. Repeatable. Narrows to the intersection with the identity's policies, or applies directly without `--as`. |
| `--default-allow` | Allow when no matching policy exists for the operation. Defaults to `false` (deny-by-default). |

### Workflow

1. Transact your policy rules (and the identities with their `f:policyClass` assignments) into the ledger, using any of the normal insert / upsert / update commands.
2. Re-run the same query as different identities to confirm results differ as the policies prescribe:

```bash
# Full result set (no policy enforcement)
fluree query 'SELECT ?name ?salary WHERE { ?p <http://schema.org/name> ?name ; <http://example.org/salary> ?salary }'

# As an HR user — should see all salaries
fluree query --as ex:hrIdentity --policy-class ex:CorpPolicy \
  'SELECT ?name ?salary WHERE { ?p <http://schema.org/name> ?name ; <http://example.org/salary> ?salary }'

# As a regular employee — policies should hide salary field
fluree query --as ex:engineerIdentity --policy-class ex:CorpPolicy \
  'SELECT ?name ?salary WHERE { ?p <http://schema.org/name> ?name ; <http://example.org/salary> ?salary }'
```

### Local vs remote

The flags work in both modes:

- **Local** (default, or with `--direct`): the CLI loads the ledger directly and applies policy via the in-process query engine.
- **Remote** (with `--remote <name>`, or auto-routed through a running local server): the CLI sends the flags to the server as HTTP headers (`fluree-identity`, `fluree-policy-class`, `fluree-default-allow`) and, for JSON-LD bodies, also injects them into `opts`. Multi-value `--policy-class` rides through the body opts only; SPARQL transport is single-valued via the header.

### Remote impersonation: how it's authorized

When you run against a remote server with `--as <iri>`, the server treats the request as **impersonation** and gates it as follows:

1. Your bearer token's identity is resolved on the target ledger.
2. If that identity has **no** `f:policyClass` assignments (the `FoundNoPolicies` outcome — your service account is unrestricted on this ledger), the server honors `--as` and runs the query as the target identity.
3. If your bearer identity is itself policy-constrained (`FoundWithPolicies`) or unknown to this ledger (`NotFound`), the server force-overrides `--as` with your bearer identity. You see your own filtered view, not the target's.

Each successful impersonation is logged at `info` level on the server:

```
policy impersonation: bearer=<svc-id> target=<as-iri> ledger=<name>
```

This is the standard service-account pattern: register your CLI/app-server identity in the ledger with no `f:policyClass`, and it gains the right to delegate to any end-user identity for testing or per-request enforcement. Assigning a policy class to that identity revokes the delegation right with no config change.

### Limitations

- Inline policy rules (`opts.policy`) and policy variable bindings (`opts.policy-values`) are not yet exposed as CLI flags — use a JSON-LD query body with an `"opts"` block when you need those.
- For SPARQL queries against a remote, only `--as`, single-value `--policy-class`, and `--default-allow` are wired (via headers). Multi-value `--policy-class` works on JSON-LD only.
- Proxy-mode servers fall back to the legacy non-impersonation behavior — the upstream server performs the impersonation check.

## Related documentation

- [Policy model and inputs](policy-model.md) — node shape, combining algorithm, request-time options
- [Policy enforcement (concepts)](../concepts/policy-enforcement.md) — model overview
- [Policy in transactions](policy-in-transactions.md) — write-time enforcement
- [Cross-ledger policy](cross-ledger-policy.md) — query-time engagement under cross-ledger `f:policySource`
- [Cookbook: Access control policies](../guides/cookbook-policies.md) — worked patterns
- [Programmatic policy API (Rust)](programmatic-policy.md) — building `PolicyContext` in code
- [Query reference](../query/README.md) — SPARQL and JSON-LD syntax
- [Explain plans](../query/explain.md) — diagnosing policy filter overhead


---

# security/policy-in-transactions.md

# Policy in Transactions

Transaction-time enforcement uses the same [policy model](policy-model.md) as queries, switched on by `f:action: f:modify`. Where query-time enforcement *filters* flakes from results, transaction-time enforcement *rejects* the transaction when a write would touch flakes the identity isn't allowed to modify.

This page documents how write-time enforcement integrates with the transaction lifecycle, the failure shape, and the patterns that come up most often. For the policy node shape and combining algorithm, see the [policy model reference](policy-model.md). For the conceptual frame, see [Policy enforcement](../concepts/policy-enforcement.md).

## How transaction-time enforcement works

When a transaction is staged against a `PolicyContext`:

1. The engine resolves the request's policy set: identity-driven `f:policyClass` lookups + any inline `opts.policy` array, restricted to policies whose `f:action` includes `f:modify`.
2. The transaction is staged into novelty (assertions and retractions are computed from `insert` / `delete` / `where` clauses).
3. Each staged flake is checked against the matching policies.
4. If any required policy denies a flake (or any non-required allow is missing where one would be needed), the **entire transaction is rejected**. Transactions are atomic — a partial write is never persisted.
5. On rejection, the response carries the policy's `f:exMessage` (when supplied), the offending flake, and the policy's `@id`.

The result: the requester gets a clear authorization failure rather than a silently incomplete write.

## Worked example

```bash
fluree insert '{
  "@context": {"f": "https://ns.flur.ee/db#", "ex": "http://example.org/"},
  "@graph": [
    {
      "@id": "ex:email-restriction",
      "@type": ["f:AccessPolicy", "ex:CorpPolicy"],
      "f:required": true,
      "f:onProperty": [{"@id": "http://schema.org/email"}],
      "f:action": [{"@id": "f:modify"}],
      "f:exMessage": "Users can only update their own email.",
      "f:query": "{\"where\": {\"@id\": \"?$identity\", \"http://example.org/user\": {\"@id\": \"?$this\"}}}"
    },
    {
      "@id": "ex:default-rw",
      "@type": ["f:AccessPolicy", "ex:CorpPolicy"],
      "f:action": [{"@id": "f:view"}, {"@id": "f:modify"}],
      "f:allow": true
    },
    {"@id": "ex:johnIdentity",  "ex:user": {"@id": "ex:john"},  "f:policyClass": [{"@id": "ex:CorpPolicy"}]},
    {"@id": "ex:janeIdentity",  "ex:user": {"@id": "ex:jane"},  "f:policyClass": [{"@id": "ex:CorpPolicy"}]}
  ]
}'
```

Now John attempts to update his own email — succeeds:

```bash
fluree update --as ex:johnIdentity --policy-class ex:CorpPolicy '
  PREFIX ex: <http://example.org/>
  PREFIX schema: <http://schema.org/>
  WHERE  { ex:john schema:email ?email }
  DELETE { ex:john schema:email ?email }
  INSERT { ex:john schema:email "new-john@flur.ee" }
'
```

John attempts to update Jane's email — rejected:

```bash
fluree update --as ex:johnIdentity --policy-class ex:CorpPolicy '
  PREFIX ex: <http://example.org/>
  PREFIX schema: <http://schema.org/>
  WHERE  { ex:jane schema:email ?email }
  DELETE { ex:jane schema:email ?email }
  INSERT { ex:jane schema:email "hacked@flur.ee" }
'
# Error: policy denied: Users can only update their own email. (ex:email-restriction)
```

## What gets enforced

Every modification path runs the same `f:modify` policy check on its staged flakes:

| Operation | Flakes checked |
|-----------|----------------|
| **Insert** | All asserted flakes. |
| **Upsert** | Asserted flakes + retractions for any pre-existing values being replaced. |
| **Update** (WHERE/DELETE/INSERT) | Both retracted flakes (DELETE) and asserted flakes (INSERT). |
| **Retraction** (`@type: f:Retraction`) | Retracted flakes. |

Crucially, the policy is checked against the **flakes**, not the operation type. A transaction that retracts a flake the identity can't modify is rejected just like an insert that asserts one.

Enforcement is also independent of the **wire format**: the check runs on the staged flakes, so JSON-LD, SPARQL UPDATE, and Turtle / TriG / N-Triples writes are all governed by the same `f:modify` policy. Sending data as Turtle is not a way to bypass write policy.

## Targeting patterns

### Whitelist a property to a role

```json
{
  "@id": "ex:salary-write",
  "@type": ["f:AccessPolicy", "ex:CorpPolicy"],
  "f:required": true,
  "f:onProperty": [{"@id": "http://example.org/salary"}],
  "f:action": [{"@id": "f:modify"}],
  "f:exMessage": "Only HR may write salary.",
  "f:query": "{\"where\": {\"@id\": \"?$identity\", \"http://example.org/role\": \"hr\"}}"
}
```

Combined with `default-allow: true` (or a permissive default `f:modify` policy), every other property remains writable.

### Owner-only edits

```json
{
  "@id": "ex:owner-edit",
  "@type": ["f:AccessPolicy", "ex:CorpPolicy"],
  "f:required": true,
  "f:action": [{"@id": "f:modify"}],
  "f:query": "{\"where\": {\"@id\": \"?$identity\", \"http://example.org/user\": {\"@id\": \"?$user\"}}, \"$where\": {\"@id\": \"?$this\", \"http://example.org/owner\": {\"@id\": \"?$user\"}}}"
}
```

The `f:query` resolves the identity's user and verifies that `?$this` (the entity being modified) has that user as its owner.

### Status-based gates

Prevent edits to records past a workflow gate:

```json
{
  "@id": "ex:no-edit-after-approval",
  "@type": ["f:AccessPolicy", "ex:CorpPolicy"],
  "f:required": true,
  "f:onClass": [{"@id": "http://example.org/Order"}],
  "f:action": [{"@id": "f:modify"}],
  "f:exMessage": "Approved orders cannot be modified.",
  "f:query": "{\"where\": [{\"@id\": \"?$this\", \"http://example.org/status\": \"?status\"}, [\"filter\", \"(!= ?status \\\"approved\\\")\"]]}"
}
```

Approved orders fail the gate — their flakes can't be retracted or modified.

### Workflow service exception

Combine targeting + identity-typed checks to limit a write to a single service:

```json
{
  "@id": "ex:approved-by-workflow-only",
  "@type": ["f:AccessPolicy", "ex:CorpPolicy"],
  "f:required": true,
  "f:onProperty": [{"@id": "http://example.org/approved"}],
  "f:action": [{"@id": "f:modify"}],
  "f:exMessage": "ex:approved is set by the workflow service only.",
  "f:query": "{\"where\": {\"@id\": \"?$identity\", \"@type\": \"http://example.org/WorkflowService\"}}"
}
```

End-user identities can read `ex:approved`, but only the workflow service can write it.

### Immutable records

```json
{
  "@id": "ex:audit-log-immutable",
  "@type": ["f:AccessPolicy", "ex:CorpPolicy"],
  "f:required": true,
  "f:onClass": [{"@id": "http://example.org/AuditEvent"}],
  "f:action": [{"@id": "f:modify"}],
  "f:exMessage": "Audit events are immutable.",
  "f:allow": false
}
```

Notice the absence of `f:query` — `f:allow: false` is a flat deny, applied to every modification of `ex:AuditEvent` instances. New events can still be inserted because the policy targets only existing-instance flakes; a fresh `@type: ex:AuditEvent` insertion creates a new subject and a new `rdf:type` flake, neither of which the targeting matches.

(For a hard "append-only" guarantee that forbids anything but new insertions, model the constraint with a SHACL shape that requires the property to be unset on prior commits — SHACL is a better fit for that pattern than policy.)

## Failure shape

When a transaction is rejected, the API returns:

```json
{
  "error": "policy_denied",
  "message": "Users can only update their own email.",
  "policy": "http://example.org/email-restriction",
  "subject": "http://example.org/jane",
  "property": "http://schema.org/email"
}
```

`f:exMessage` is the user-visible string. The policy `@id`, the offending subject, and the property are reported for diagnostics.

When no `f:exMessage` is set, a generic message is returned (`"policy denied"`); the structured fields are still present so a client can surface the right error to a user.

## WHERE/DELETE/INSERT semantics with policy

A WHERE/DELETE/INSERT transaction proceeds in three phases — match → retract → assert. The two policy actions apply at different phases:

- **`f:view` filters the match phase.** The WHERE clause is a *read*, so it binds only the flakes the requesting identity may view — exactly as the same patterns would in a query. An identity therefore cannot conditionally match on data it can't see: `INSERT { ?s ex:flag true } WHERE { ?s ex:salary ?v . FILTER(?v > 100000) }` binds nothing when `ex:salary` isn't viewable, so a conditional write can't be used to probe hidden values.
- **`f:modify` rejects the write.** Enforcement runs on the staged flakes from the retract/assert phases; any flake the identity can't modify rejects the whole transaction.

```sparql
PREFIX ex:     <http://example.org/>
PREFIX schema: <http://schema.org/>

WHERE  { ?u schema:email ?old . FILTER(?u = ex:jane) }
DELETE { ?u schema:email ?old }
INSERT { ?u schema:email "new@flur.ee" }
```

Run by an identity that **can view** Jane's email but lacks **modify** rights on it:

- The WHERE binds Jane's email (it is viewable), so the DELETE/INSERT are staged.
- The DELETE retraction stages a flake the identity can't modify — **rejected**.

Run by an identity that **cannot view** Jane's email:

- The WHERE binds nothing — the hidden value is never read, so it can't be inferred from whether the write succeeded. A WHERE/DELETE/INSERT that matches no rows produces no flakes and is reported as an empty transaction.

So pairing an `f:modify` policy with a same-shape `f:view` policy turns a would-be modify *rejection* into a clean no-op: the WHERE stops matching the protected rows, so nothing is staged to reject.

## Signed transactions and impersonation

When a transaction is signed (JWS or VC-wrapped), the signing key's identity replaces the bearer identity for policy purposes. The signed credential becomes the source of truth: the server verifies the signature, resolves the signer's identity entity, and applies that identity's `f:policyClass` policies.

For the impersonation rules — when `--as <iri>` is honored vs force-overridden — see [Policy in queries → Remote impersonation](policy-in-queries.md#remote-impersonation-how-its-authorized). The same gate applies to transactions.

See [Signed / credentialed transactions](../transactions/signed-transactions.md) for the wire format.

## Provenance

Every committed transaction carries the asserting identity in its commit metadata. Combined with policy enforcement, this gives a clean audit trail:

- The identity is recorded on the commit.
- The policies in effect at commit time are themselves time-travelable.
- Replay-from-commit produces the same policy decisions.

## Performance considerations

- **Stage cost dominates.** Most of the work is staging the transaction (computing assertions/retractions, building the novelty layer). Policy checks add a small per-flake cost on top.
- **Required policies short-circuit.** A failure rejects the transaction immediately without checking remaining flakes.
- **Batch transactions amortize loading.** Loading the policy set is per-transaction, not per-flake — large batched transactions pay the load cost once.
- **Cache identity properties.** The identity's `@type`, `f:policyClass`, and any role tags used in `f:query` are loaded once per transaction.

## Testing policies from the CLI

The same `--as`, `--policy-class`, and `--default-allow` flags used on `fluree query` are available on `fluree insert`, `fluree upsert`, and `fluree update` so you can verify write-time enforcement without any client code:

```bash
# Attempt a write as an identity that lacks the f:modify policy — expect failure
fluree insert --as ex:readOnlyIdentity --policy-class ex:CorpPolicy -f new-data.ttl

# Same write as an authorized identity — expect success
fluree insert --as ex:writerIdentity --policy-class ex:CorpPolicy -f new-data.ttl
```

The flags work locally and against remote servers. On remote, the CLI sends the policy options as HTTP headers (`fluree-identity`, `fluree-policy-class`, `fluree-default-allow`) and, for JSON-LD bodies, also injects them into `opts`. The server applies the **root-impersonation gate**: your bearer identity may delegate to `--as <iri>` only when the bearer identity itself has no `f:policyClass` on the target ledger. Restricted bearers have `--as` force-overridden back to their own identity (and writes only what their own policies permit).

This is the standard service-account pattern — see [Policy in queries → Remote impersonation](policy-in-queries.md#remote-impersonation-how-its-authorized) for the full authorization rules and audit-log format.

### Transaction enforcement is end-to-end

Unsigned bearer-authenticated transactions build a `PolicyContext` from the (post-header-merge) opts and route through the policy-enforcing `transact_tracked_with_policy` path. A non-root bearer's `f:modify` constraints apply to their writes, matching the long-standing query-side behavior. SPARQL UPDATE inherits the same enforcement, with identity sourced from either the bearer or the `fluree-identity` header (impersonation-gated).

## Related documentation

- [Policy model and inputs](policy-model.md) — node shape, combining algorithm, request-time options
- [Policy enforcement (concepts)](../concepts/policy-enforcement.md) — model overview
- [Policy in queries](policy-in-queries.md) — read-time enforcement
- [Cross-ledger policy](cross-ledger-policy.md) — transaction-time enforcement under cross-ledger `f:policySource`
- [Cookbook: Access control policies](../guides/cookbook-policies.md) — worked patterns
- [Programmatic policy API (Rust)](programmatic-policy.md) — building `PolicyContext` and using `transact_tracked_with_policy`
- [Signed / credentialed transactions](../transactions/signed-transactions.md) — JWS / VC transaction wrapping
- [Transaction overview](../transactions/overview.md) — transaction lifecycle


---

# security/programmatic-policy.md

# Programmatic Policy API (Rust)

This guide covers how to use Fluree's policy system programmatically in Rust applications.

## Overview

There are two main approaches to applying policies programmatically:

1. **Identity-based policies** (`wrap_identity_policy_view`): Policies stored in the database and loaded via `f:policyClass` on an identity subject
2. **Inline policies** (`wrap_policy_view` with `opts.policy`): Policies provided directly in the query/transaction options

## Identity-Based Policy Lookup

The recommended approach for production systems. Policies are stored in the ledger and loaded dynamically based on the identity's `f:policyClass` property.

### Storing Policies in the Database

First, insert policies with types that will be referenced by identities:

```rust
let policies = json!({
    "@context": {
        "f": "https://ns.flur.ee/db#",
        "ex": "http://example.org/ns/",
        "schema": "http://schema.org/"
    },
    "@graph": [
        // Identity with policy class assignment
        {
            "@id": "http://example.org/identity/alice",
            "f:policyClass": [{"@id": "ex:EmployeePolicy"}],
            "ex:user": {"@id": "ex:alice"}
        },

        // SSN restriction policy - only see your own SSN
        {
            "@id": "ex:ssnRestriction",
            "@type": ["f:AccessPolicy", "ex:EmployeePolicy"],
            "f:required": true,
            "f:onProperty": [{"@id": "schema:ssn"}],
            "f:action": {"@id": "f:view"},
            "f:query": serde_json::to_string(&json!({
                "where": {
                    "@id": "?$identity",
                    "http://example.org/ns/user": {"@id": "?$this"}
                }
            })).unwrap()
        },

        // Default allow policy for other properties
        {
            "@id": "ex:defaultAllowView",
            "@type": ["f:AccessPolicy", "ex:EmployeePolicy"],
            "f:action": {"@id": "f:view"},
            "f:allow": true
        }
    ]
});

// Prefer the lazy Graph API for transactions
fluree.graph("mydb:main")
    .transact()
    .insert(&policies)
    .commit()
    .await?;
```

### Using wrap_identity_policy_view

Create a policy-wrapped view using an identity IRI:

```rust
use fluree_db_api::{wrap_identity_policy_view, FlureeBuilder, GraphDb};

let fluree = FlureeBuilder::memory().build_memory();
let ledger = fluree.ledger("mydb:main").await?;

// Wrap the ledger with identity-based policy
let wrapped = wrap_identity_policy_view(
    &ledger,
    "http://example.org/identity/alice",  // identity IRI
    true  // default_allow: allow access when no policy matches
).await?;

// Check policy properties
assert!(!wrapped.is_root(), "Should not be root/unrestricted");

// Create a view with the policy applied, then query using the builder
let view = GraphDb::from_ledger_state(&ledger)
    .with_policy(std::sync::Arc::new(wrapped.policy().clone()));

let query = json!({
    "select": ["?s", "?ssn"],
    "where": {
        "@id": "?s",
        "@type": "ex:User",
        "schema:ssn": "?ssn"
    }
});

let result = view.query(&fluree)
    .jsonld(&query)
    .execute()
    .await?;
```

### How Identity Lookup Works

When you call `wrap_identity_policy_view`:

1. Fluree queries for policies via the identity's `f:policyClass`:
   ```sparql
   SELECT ?policy WHERE {
       <identity-iri> f:policyClass ?class .
       ?policy a ?class .
       ?policy a f:AccessPolicy .
   }
   ```

2. Each matching policy's properties are loaded (`f:action`, `f:allow`, `f:query`, `f:onProperty`, etc.)

3. The `?$identity` variable is automatically bound to the identity IRI for use in `f:query` policies

## Inline Policies with policy-values

For cases where policies should not be stored in the database, use inline policies with explicit `?$identity` binding.

### QueryConnectionOptions Pattern

```rust
use fluree_db_api::{QueryConnectionOptions, wrap_policy_view};
use std::collections::HashMap;

let policy = json!([{
    "@id": "ex:inlineSsnPolicy",
    "f:required": true,
    "f:onProperty": [{"@id": "http://schema.org/ssn"}],
    "f:action": "f:view",
    "f:query": serde_json::to_string(&json!({
        "where": {
            "@id": "?$identity",
            "http://example.org/ns/user": {"@id": "?$this"}
        }
    })).unwrap()
}]);

let opts = QueryConnectionOptions {
    policy: Some(policy),
    policy_values: Some(HashMap::from([(
        "?$identity".to_string(),
        json!({"@id": "http://example.org/identity/alice"}),
    )])),
    default_allow: true,
    ..Default::default()
};

let wrapped = wrap_policy_view(&ledger, &opts).await?;
```

### Using query_from with Inline Policy

For FROM-driven queries where policy options are embedded in the query body, use `query_from()`:

```rust
let query = json!({
    "@context": {
        "ex": "http://example.org/ns/",
        "schema": "http://schema.org/"
    },
    "from": "mydb:main",
    "opts": {
        "default-allow": true,
        "policy": [{
            "@id": "inline-ssn-policy",
            "f:required": true,
            "f:onProperty": [{"@id": "http://schema.org/ssn"}],
            "f:action": "f:view",
            "f:query": serde_json::to_string(&json!({
                "where": {
                    "@id": "?$identity",
                    "http://example.org/ns/user": {"@id": "?$this"}
                }
            })).unwrap()
        }],
        "policy-values": {
            "?$identity": {"@id": "http://example.org/identity/alice"}
        }
    },
    "select": ["?s", "?ssn"],
    "where": {
        "@id": "?s",
        "@type": "ex:User",
        "schema:ssn": "?ssn"
    }
});

let result = fluree.query_from()
    .jsonld(&query)
    .execute()
    .await?;
```

## Policy Options Precedence

When multiple policy options are provided, they follow this precedence:

| Priority | Option | Behavior |
|----------|--------|----------|
| 1 (highest) | `opts.identity` | Query `f:policyClass` policies, auto-bind `?$identity` |
| 2 | `opts.policy_class` | Query policies of specified types |
| 3 (lowest) | `opts.policy` | Use inline policy JSON directly |

**Important:** If `opts.identity` is set, inline `opts.policy` is ignored.

## Policy Structure Reference

### f:allow (Static Allow/Deny)

```json
{
    "@id": "ex:allowAll",
    "@type": ["f:AccessPolicy", "ex:MyPolicyClass"],
    "f:action": {"@id": "f:view"},
    "f:allow": true
}
```

### f:query (Dynamic Evaluation)

```json
{
    "@id": "ex:ownerOnly",
    "@type": ["f:AccessPolicy", "ex:MyPolicyClass"],
    "f:action": {"@id": "f:view"},
    "f:onProperty": [{"@id": "schema:ssn"}],
    "f:required": true,
    "f:query": "{\"where\": {\"@id\": \"?$identity\", \"ex:user\": {\"@id\": \"?$this\"}}}"
}
```

### Policy Properties

| Property | Type | Description |
|----------|------|-------------|
| `f:action` | `f:view` / `f:modify` | What action this policy applies to |
| `f:allow` | boolean | Static allow (true) or deny (false) |
| `f:query` | string (JSON) | Query that must return results for access to be granted |
| `f:onProperty` | IRI(s) | Restrict policy to specific properties |
| `f:onSubject` | IRI(s) | Restrict policy to specific subjects |
| `f:onClass` | IRI(s) | Restrict policy to instances of specific classes |
| `f:required` | boolean | If true, this policy MUST allow for access to be granted |
| `f:exMessage` | string | Custom error message when policy denies access |

### Special Variables

| Variable | Binding |
|----------|---------|
| `?$identity` | The identity IRI (from `opts.identity` or `policy_values["?$identity"]`) |
| `?$this` | The subject being accessed (for property-level policies) |

## Policy Combining Algorithm

When multiple policies match a flake, they are combined using **Deny Overrides**:

1. If **any** matching policy explicitly denies (`f:allow: false`), access is **denied**
2. If a targeted policy's `f:query` returns false, access is **denied** (doesn't fall through to Default policies)
3. If any policy allows (`f:allow: true` or `f:query` returns true), access is **granted**
4. If no policies match and `default_allow` is `true` → access is **granted**
5. Otherwise, access is **denied**

> **Identity resolution is three-state:** `FoundWithPolicies` (restrictions apply) → `FoundNoPolicies` (subject exists, no restrictions) → `NotFound` (subject absent, no restrictions). The three-state split determines whether a concrete identity SID is available to bind `?$identity` in policy queries; it does not gate `default_allow`. An unknown identity with `default_allow: true` is granted access — this is the intended behavior for deployments where an application layer handles authorization and Fluree records signed transactions for provenance. Set `default_allow: false` for fail-closed behavior.

**Important:** Inline policies must use full IRIs (e.g., `"http://schema.org/ssn"`), not compact IRIs (e.g., `"schema:ssn"`). Compact IRIs in inline policies are not expanded.

## Transactions with Policy

Policies can also be applied to transactions using the builder API:

```rust
use fluree_db_api::policy_builder;

let policy_ctx = policy_builder::build_policy_context_from_opts(
    &ledger.snapshot,
    ledger.novelty.as_ref(),
    Some(ledger.novelty.as_ref()),
    ledger.t(),
    &qc_opts,
    &[0], // default graph; use resolve_policy_source_g_ids() for config-driven graphs
).await?;

let txn = json!({
    "@context": {"ex": "http://example.org/ns/"},
    "insert": [
        {"@id": "ex:alice", "ex:data": "secret"}
    ]
});

// Use the transaction builder with policy
let result = fluree.graph("mydb:main")
    .transact()
    .update(&txn)
    .policy(policy_ctx)
    .commit()
    .await;

match result {
    Ok(txn_result) => println!("Transaction succeeded at t={}", txn_result.ledger.t()),
    Err(e) => println!("Policy denied: {}", e),
}
```

## Historical Views with Policy

For time-travel queries with policy, load a historical graph and apply policy as a view overlay:

```rust
use fluree_db_api::{GraphDb, QueryConnectionOptions};

// Load a historical view
let graph = fluree.view_at_t("mydb:main", 100).await?;

// Apply policy to create a view
let policy_ctx = policy_builder::build_policy_context_from_opts(
    &ledger.snapshot,
    ledger.novelty.as_ref(),
    Some(ledger.novelty.as_ref()),
    ledger.t(),
    &opts,
    &[0],
).await?;

let view = graph.with_policy(std::sync::Arc::new(policy_ctx));

// Query the historical view with policy applied
let result = view.query(&fluree)
    .jsonld(&query)
    .execute()
    .await?;
```

## API Reference

### wrap_identity_policy_view

```rust
pub async fn wrap_identity_policy_view<'a>(
    ledger: &'a LedgerState,
    identity_iri: &str,
    default_allow: bool,
) -> Result<PolicyWrappedView<'a>>
```

Creates a policy-wrapped view using identity-based `f:policyClass` lookup.

**Parameters:**
- `ledger`: The ledger state to wrap
- `identity_iri`: IRI of the identity subject (will query `f:policyClass`)
- `default_allow`: Whether to allow access when no policies match. Ignored (forced `false`) if the identity IRI has no subject node in the ledger — see combining algorithm step 5

### wrap_policy_view

```rust
pub async fn wrap_policy_view<'a>(
    ledger: &'a LedgerState,
    opts: &QueryConnectionOptions,
) -> Result<PolicyWrappedView<'a>>
```

Creates a policy-wrapped view from query connection options.

**QueryConnectionOptions fields:**
- `identity`: Identity IRI for `f:policyClass` lookup
- `policy`: Inline policy JSON
- `policy_class`: Policy class IRIs to query
- `policy_values`: Variable bindings for policy queries
- `default_allow`: Default access when no policies match

### PolicyWrappedView

```rust
impl PolicyWrappedView {
    /// Check if this is a root/unrestricted policy
    pub fn is_root(&self) -> bool;

    /// Get the underlying policy context
    pub fn policy(&self) -> &PolicyContext;

    /// Get the policy enforcer for query execution
    pub fn enforcer(&self) -> &Arc<QueryPolicyEnforcer>;
}
```

## Best Practices

### 1. Prefer Identity-Based Policies

Store policies in the database for:
- Version control with data
- Audit trail of policy changes
- Dynamic policy updates without code changes
- Time-travel to historical policy states

### 2. Use HTTP IRIs for Identities

HTTP IRIs are more portable than DIDs for identity subjects:

```rust
// Recommended
let identity = "http://example.org/identity/alice";

// Also works but may have encoding issues
let identity = "did:key:z6MkhaXgBZDvotDkL5257faiztiGiC2QtKLGpbnnEGta2doK";
```

### 3. Always Set default_allow Explicitly

```rust
// Be explicit about default behavior
let wrapped = wrap_identity_policy_view(&ledger, identity, false).await?;
//                                                          ^^^^^ explicit deny
```

### 4. Handle Policy Errors

```rust
let graph = GraphDb::from_ledger_state(&ledger)
    .with_policy(std::sync::Arc::new(policy_ctx));

match graph.query(&fluree).jsonld(&query).execute().await {
    Ok(result) => process_results(result),
    Err(ApiError::PolicyDenied { message, policy_id }) => {
        log::warn!("Access denied by {}: {}", policy_id, message);
        // Return empty or error to user
    }
    Err(e) => return Err(e),
}
```

## Related Documentation

- [Policy Model](policy-model.md) - Policy structure and evaluation
- [Policy in Queries](policy-in-queries.md) - Query-time enforcement
- [Policy in Transactions](policy-in-transactions.md) - Transaction-time enforcement
- [Cross-ledger policy](cross-ledger-policy.md) - Govern many data ledgers from one model ledger via `f:policySource` with `f:ledger`; `db_with_policy` dispatches automatically when the data ledger's `#config` is cross-ledger.
- [Rust API](../getting-started/rust-api.md) - General Rust API usage


---

# security/cross-ledger-policy.md

# Cross-ledger governance

A single **model ledger** can hold governance artifacts —
policy rules, uniqueness constraints, ontology/schema axioms,
SHACL shapes, and datalog rules — that govern many **data
ledgers** that reference it. Update the model once and every
governed data ledger sees the new artifacts on its next
request, with no per-dataset duplication.

All five `f:GraphRef`-shaped governance predicates support
cross-ledger references today:

- **Cross-ledger policy** (`f:policySource` with `f:ledger`) —
  M's policy rule set is applied to queries against D.
- **Cross-ledger constraints** (`f:constraintsSource` with
  `f:ledger`) — M's `f:enforceUnique` annotations are applied
  to transactions against D.
- **Cross-ledger schema** (`f:schemaSource` with `f:ledger`) —
  M's RDFS/OWL axioms (class hierarchy, property hierarchy,
  domain/range, equivalences, owl:imports declarations) feed
  D's reasoner. Single-graph only today; transitive
  `owl:imports` recursion across multiple model ledgers is
  reserved.
- **Cross-ledger SHACL shapes** (`f:shapesSource` with
  `f:ledger`) — M's shape definitions are compiled against D's
  staged namespace at validation time and rejected/accepted
  transactions on D accordingly.
- **Cross-ledger datalog rules** (`f:rulesSource` with
  `f:ledger`) — M's `f:rule` JSON bodies feed D's query-time
  datalog evaluator alongside any rules D stores locally.

This page covers configuration for all five. For the
underlying design (resolver contract, term-space translation,
cache shape, failure taxonomy) see
[Cross-ledger model enforcement](../design/cross-ledger-model-enforcement.md).

## When to use it

Cross-ledger policy is the right tool when:

- Multiple data ledgers share a common access-control model
  (e.g., every customer dataset enforces the same baseline
  policy on `Document` / `User` classes).
- Policy authoring needs to be decoupled from data authoring
  (a security team owns the model ledger; product teams own the
  data ledgers).
- Updates to policy rules must propagate atomically across all
  governed datasets — no per-dataset re-sync window.

If your policy lives entirely inside one ledger, stick with the
local pattern in
[Policy model and inputs](policy-model.md) — it's simpler.

## The two-ledger pattern

| Term            | Meaning |
|-----------------|---------|
| **Model ledger** (M) | The ledger holding the policy rule set. Identified by its canonical id (e.g., `org/governance:main`). |
| **Data ledger** (D) | The application ledger holding the data being protected. References M in its `#config`. |

Both ledgers must live on the same Fluree instance (same
nameservice, same storage namespace). Cross-instance federation is
out of scope.

## Setting up the model ledger

The model ledger holds policy resources just like a same-ledger
configuration would — there is nothing special about them on M's
side. The convention is to put them in a named graph so they
don't mix with any data that happens to live on M:

```trig
@prefix f:    <https://ns.flur.ee/db#> .
@prefix rdf:  <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix ex:   <http://example.org/ns/> .

GRAPH <http://example.org/governance/policies> {
    ex:denyUsers
        rdf:type    f:AccessPolicy ;
        f:action    f:view ;
        f:onClass   ex:User ;
        f:allow     false .
}
```

That graph IRI (`http://example.org/governance/policies` above) is
what the data ledger's config will name. Any number of policies
can live in the same graph; they're all loaded together on
resolution.

## Configuring the data ledger

D's `#config` declares an `f:policySource` whose `f:graphSource`
carries an explicit `f:ledger` field pointing at M:

```trig
@prefix f:   <https://ns.flur.ee/db#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .

GRAPH <urn:fluree:mydb:main#config> {
    <urn:cfg:main> rdf:type f:LedgerConfig ;
        f:policyDefaults <urn:cfg:policy> .

    <urn:cfg:policy>
        f:defaultAllow false ;
        f:policyClass  f:AccessPolicy ;
        f:policySource <urn:cfg:policy-ref> .

    <urn:cfg:policy-ref> rdf:type f:GraphRef ;
        f:graphSource <urn:cfg:policy-src> .

    <urn:cfg:policy-src>
        f:ledger        <org/governance:main> ;
        f:graphSelector <http://example.org/governance/policies> .
}
```

Three things to notice:

1. **`f:ledger`** carries the canonical id of the model ledger
   (`org/governance:main`). Use `nameservice.lookup()` if you
   need to confirm the canonical form — aliases are resolved
   into the canonical id before the resolver runs.
2. **`f:graphSelector`** names the graph IRI within M that
   holds the policies. It must match exactly what M used in
   its `GRAPH <...>` block — there's no fuzzy matching.
3. **`f:policyClass`** is what determines which rules from M
   actually apply. See below.

## How `f:policyClass` filtering works

When D's request reaches the resolver, every rule materialized
from M's policy graph is filtered against the data ledger's
configured `f:policyClass` set by exact IRI intersection. A rule
passes the filter if any of its `rdf:type` IRIs appears in D's
`f:policyClass` list.

| D's `f:policyClass`                  | Rules from M that apply |
|--------------------------------------|--------------------------|
| not set                              | Defaults to `{f:AccessPolicy}` — all `rdf:type f:AccessPolicy` rules apply. |
| `f:AccessPolicy`                     | All `rdf:type f:AccessPolicy` rules apply. |
| `ex:OrgPolicy`                       | Only rules typed `rdf:type ex:OrgPolicy`. |
| `f:AccessPolicy`, `ex:OrgPolicy`     | Rules typed as either. |

The match is **exact-IRI only**. There is no subclass entailment:
declaring `ex:OrgPolicy rdfs:subClassOf f:AccessPolicy` doesn't
make `ex:OrgPolicy`-typed rules match a config that asks for
`f:AccessPolicy`. This mirrors the same-ledger
`load_policies_by_class` behavior.

The `{f:AccessPolicy}` default makes "set `f:policySource` and
get baseline enforcement" the no-configuration path. Custom-typed
rules are opt-in — operators name the class to enroll them.

## Engaging policy enforcement

There's a subtlety in how the server's JSON-LD query route
chooses whether to invoke policy enforcement at all. Requests
without an `fluree-policy-class`, `fluree-identity`, or inline
`opts.policy` go through a no-policy fast path that bypasses the
cross-ledger dispatch. A configured `f:policySource` in `#config`
is **not** enough on its own to force enforcement at the HTTP
layer today.

To engage cross-ledger policy via HTTP, send a request with at
least one of:

- `fluree-policy-class: <iri>` — the policy class header (the
  cleanest way to declare "use the configured policy"). Matching
  the class in D's config (e.g., `f:AccessPolicy`) is the
  natural choice.
- `fluree-identity: <iri>` — an identity header. Identity-mode
  has a different contract; see below.
- `opts.policy` in the body — inline JSON-LD policy. This still
  merges with cross-ledger rules.

When using the in-process Rust API, calling
`fluree.db_with_policy(ledger_id, &opts)` always engages the
policy path, even with empty opts. Programmatic users don't see
this gating.

## Cross-ledger uniqueness constraints

Same two-ledger pattern, different subsystem. M holds an
`f:enforceUnique true` annotation on a property; D references
M's constraints graph in its `#config`. Every transaction
against D that would create a duplicate value on that property
is rejected.

```trig
# On model ledger M
@prefix f:  <https://ns.flur.ee/db#> .
@prefix ex: <http://example.org/ns/> .

GRAPH <http://example.org/governance/constraints> {
    ex:email f:enforceUnique true .
}
```

```trig
# On data ledger D — #config
@prefix f:   <https://ns.flur.ee/db#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .

GRAPH <urn:fluree:mydb:main#config> {
    <urn:cfg:main> rdf:type f:LedgerConfig ;
        f:transactDefaults <urn:cfg:transact> .

    <urn:cfg:transact>
        f:uniqueEnabled      true ;
        f:constraintsSource  <urn:cfg:cref> .

    <urn:cfg:cref> rdf:type f:GraphRef ;
        f:graphSource <urn:cfg:csrc> .

    <urn:cfg:csrc>
        f:ledger        <org/governance:main> ;
        f:graphSelector <http://example.org/governance/constraints> .
}
```

After this config lands on D, the next transaction that creates
a duplicate `ex:email` value is rejected with
`TransactError::UniqueConstraintViolation`. The annotation
itself never appears on D — D enforces it because its config
points at M.

A few specifics that differ from cross-ledger policy:

- Constraints are enforced at **transaction time**, not query
  time. No HTTP header gymnastics are needed to engage them —
  the staging pipeline picks them up automatically whenever the
  data ledger's config has `f:uniqueEnabled true` plus a
  cross-ledger `f:constraintsSource`.
- Failures during cross-ledger constraints resolution surface
  as `TransactError::Parse` (mapped to HTTP **400 Bad Request**
  at the API layer) rather than `ApiError::CrossLedger` (502).
  That's a staging-pipeline quirk — the error message preserves
  the underlying cross-ledger failure variant for diagnostics,
  but the HTTP status differs from the query-side policy path.
- The wire artifact for constraints is simpler than policy:
  just a list of property IRIs. There's no equivalent of
  `f:policyClass` filtering — every property M declares unique
  applies.

## Cross-ledger schema / ontology

Same two-ledger pattern, third subsystem. M holds an RDFS/OWL
schema in a named graph (class hierarchy, property hierarchy,
domain/range, equivalences); D references that graph in its
`#config` under `f:reasoningDefaults.f:schemaSource`. When D's
queries enable reasoning, M's axioms feed the reasoner exactly
as if they lived on D.

```trig
# On model ledger M — ontology in a named graph
@prefix owl:  <http://www.w3.org/2002/07/owl#> .
@prefix rdf:  <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix ex:   <http://example.org/ns/> .

GRAPH <http://example.org/ontology/core> {
    ex:Animal  rdf:type        owl:Class .
    ex:Dog     rdf:type        owl:Class ;
               rdfs:subClassOf ex:Animal .

    ex:knows   rdf:type           owl:ObjectProperty .
    ex:friend  rdf:type           owl:ObjectProperty ;
               rdfs:subPropertyOf ex:knows .
}
```

```trig
# On data ledger D — #config
@prefix f:   <https://ns.flur.ee/db#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .

GRAPH <urn:fluree:mydb:main#config> {
    <urn:cfg:main> rdf:type f:LedgerConfig ;
        f:reasoningDefaults <urn:cfg:reasoning> .

    <urn:cfg:reasoning>
        f:reasoningModes  ( "rdfs" "owl2-rl" ) ;
        f:schemaSource    <urn:cfg:schema-ref> .

    <urn:cfg:schema-ref> rdf:type f:GraphRef ;
        f:graphSource <urn:cfg:schema-src> .

    <urn:cfg:schema-src>
        f:ledger        <org/ontology:main> ;
        f:graphSelector <http://example.org/ontology/core> .
}
```

With this config, a query against D for `?s rdf:type ex:Animal`
returns every `ex:Dog` instance on D (subClassOf reasoning) —
even though `ex:Dog rdfs:subClassOf ex:Animal` never appears on
D itself.

The materializer pulls a **whitelisted subset** of axioms from
M's schema graph: the predicates `rdfs:subClassOf`,
`rdfs:subPropertyOf`, `rdfs:domain`, `rdfs:range`,
`owl:inverseOf`, `owl:equivalentClass`, `owl:equivalentProperty`,
`owl:sameAs`, `owl:imports`, plus `rdf:type` declarations for
the schema class set (`owl:Class`, `owl:ObjectProperty`,
`owl:DatatypeProperty`, the property characteristic classes,
`owl:Ontology`, `rdf:Property`). Instance data in the schema
graph is filtered out; only axioms cross over.

A few specifics that differ from cross-ledger policy:

- Reasoning must be **enabled** for cross-ledger schema to take
  effect. The data ledger's config can set
  `f:reasoningModes` (e.g., `["rdfs"]` or `["owl2-rl"]`), or
  the query can opt in via the `reasoning` option.
- Failures during cross-ledger schema resolution surface as
  `ApiError::OntologyImport` (with the underlying
  `CrossLedgerError` displayed in the message) rather than
  `ApiError::CrossLedger`. That preserves continuity with the
  same-ledger ontology-imports error path.
- Single graph only in this phase: `owl:imports` triples in
  M's schema graph are carried through to the wire (so a
  future reader can see them), but the resolver does NOT
  transitively follow them across ledger boundaries yet. If M's
  schema declares `owl:imports <X>` where X is on a different
  model ledger M2, that import isn't resolved.

## Cross-ledger SHACL shapes

Configure `f:shapesSource` on D's `#config` under
`f:shaclDefaults`. When the source carries `f:ledger`, the
resolver pulls M's shapes graph at transaction time, compiles
the SHACL whitelist against D's *staged* namespace registry
(post-stage, not pre-stage — so IRIs the in-flight transaction
introduced are encodable), and feeds the resulting overlay to
the same SHACL engine the same-ledger path uses.

```trig
# On model ledger M — shapes in a named graph
@prefix sh:   <http://www.w3.org/ns/shacl#> .
@prefix ex:   <http://example.org/ns/> .

GRAPH <http://example.org/governance/shapes> {
    ex:PersonShape a sh:NodeShape ;
        sh:targetClass ex:Person ;
        sh:property [
            sh:path     ex:name ;
            sh:minCount 1 ;
            sh:datatype xsd:string
        ] .
}
```

```trig
# On data ledger D — #config
@prefix f:   <https://ns.flur.ee/db#> .

GRAPH <urn:fluree:data:main#config> {
    <urn:cfg:main>      a f:LedgerConfig ;
                        f:shaclDefaults <urn:cfg:shacl> .
    <urn:cfg:shacl>     f:shaclEnabled  true ;
                        f:shapesSource  <urn:cfg:sref> .
    <urn:cfg:sref>      a f:GraphRef ;
                        f:graphSource   <urn:cfg:ssrc> .
    <urn:cfg:ssrc>      f:ledger        <model:main> ;
                        f:graphSelector <http://example.org/governance/shapes> .
}
```

Specifics:

- Shapes are **compiled against D's staged namespace registry**,
  not its pre-stage snapshot. The in-flight transaction's
  ns_registry sees the IRIs the transaction is introducing
  (e.g., `ex:Person` declared by the same tx the shape is
  validating), which the pre-stage snapshot doesn't.
- IRIs the staged registry has never seen are dropped silently —
  M-only IRIs can't apply to data D doesn't have, and
  allocating a fresh namespace code for an M-only term would
  introduce namespace churn into D for no benefit.
- Inline `opts.shapes` layers **additively** on top of the
  cross-ledger source: both shape sets enforce. See
  [Cookbook: SHACL validation — Inline shapes per
  transaction](../guides/cookbook-shacl.md#inline-shapes-per-transaction).

## Cross-ledger datalog rules

Configure `f:rulesSource` on D's `#config` under
`f:datalogDefaults`. When the source carries `f:ledger`, the
resolver pulls the `f:rule` JSON bodies from M's rules graph at
query time and merges them into D's query-time datalog evaluator
alongside any rules D stores locally and any rules supplied via
the top-level `rules` query field.

```trig
# On model ledger M — rules in a named graph
@prefix f:   <https://ns.flur.ee/db#> .
@prefix ex:  <http://example.org/> .

GRAPH <http://example.org/governance/rules> {
    ex:grandparentRule f:rule """
        {
          \"@context\": {\"ex\": \"http://example.org/\"},
          \"where\":   {\"@id\": \"?p\", \"ex:parent\": {\"ex:parent\": \"?g\"}},
          \"insert\":  {\"@id\": \"?p\", \"ex:grandparent\": {\"@id\": \"?g\"}}
        }
    """^^rdf:JSON .
}
```

```trig
# On data ledger D — #config
@prefix f:   <https://ns.flur.ee/db#> .

GRAPH <urn:fluree:data:main#config> {
    <urn:cfg:main>     a f:LedgerConfig ;
                       f:datalogDefaults <urn:cfg:datalog> .
    <urn:cfg:datalog>  f:datalogEnabled  true ;
                       f:rulesSource     <urn:cfg:rref> .
    <urn:cfg:rref>     a f:GraphRef ;
                       f:graphSource     <urn:cfg:rsrc> .
    <urn:cfg:rsrc>     f:ledger          <model:main> ;
                       f:graphSelector   <http://example.org/governance/rules> .
}
```

Specifics:

- Rules are **inherently term-portable** — the JSON body is a
  JSON-LD document whose IRIs resolve at parse time against
  D's snapshot, the same way query-time rules from the top-level
  `rules` field are handled. No term translation step on the
  wire.
- Engaging the rules still requires datalog reasoning on the
  query: either `"reasoning": "datalog"` on the JSON-LD query
  or `PRAGMA reasoning: datalog` on SPARQL.
- **Fail-closed on malformed rules.** A bad JSON body in M's
  rules graph errors the query rather than silently dropping
  the rule (this differs from query-time `rules` where a single
  bad entry only logs a warning — cross-ledger rules are
  admin-authored, so silently weakening the configured
  reasoning model is the worst failure mode).

## Limitations

The following behaviors are **not yet implemented** and fail
closed when configured:

| Configuration                              | Outcome |
|--------------------------------------------|---------|
| `f:atT` (temporal pinning of M)            | Request fails with `UnsupportedFeature { feature: "f:atT", phase: "Phase 3" }`. |
| `f:trustPolicy` (commit-signer allowlist)  | Request fails with `UnsupportedFeature`. |
| `f:rollbackGuard` (freshness constraints)  | Request fails with `UnsupportedFeature`. |
| `opts.identity` + cross-ledger `f:policySource` | Request fails with a config error. Identity-mode loads policies via the identity's `f:policyClass` triples, which would have to resolve in D (the identity isn't an M concept); combining the two modes ambiguously is rejected rather than silently choosing one. Use `opts.policy_class` with cross-ledger configs. |
| `f:policySource` with `f:graphSelector` naming M's `#config` or `#txn-meta` | Request fails with `ReservedGraphSelected` before any storage read on M. |
| Transitive `owl:imports` across model ledgers (`f:schemaSource` recursion) | Not yet honored. Imports inside M's schema graph are projected but the resolver doesn't follow them across ledger boundaries. |

The other reserved fields and source predicates may land in
later releases; the resolver's contract is shared across all of
them. See [Cross-ledger model enforcement → Scope](../design/cross-ledger-model-enforcement.md#scope).

## Failure modes

When cross-ledger resolution fails, the request returns HTTP
**502 Bad Gateway** with a structured JSON body naming the
specific failure:

```json
{
  "status": 502,
  "@type": "err:system/CrossLedgerError",
  "error": "model ledger 'org/governance:main' is not present on this instance"
}
```

The specific failure modes operators see:

| Variant                       | Trigger |
|-------------------------------|---------|
| `ModelLedgerMissing`          | The named model ledger isn't present or is retracted on this instance. |
| `GraphMissingAtT`             | The model ledger exists but the named graph IRI isn't in its graph registry. |
| `ReservedGraphSelected`       | The selector targets `#config` or `#txn-meta` on M. |
| `TranslationFailed`           | The policy graph was read but couldn't be projected to the wire format (typically corruption or a dictionary loss in M). |
| `UnsupportedFeature`          | A reserved field (`f:atT` / `f:trustPolicy` / `f:rollbackGuard`) was set. |
| `CrossInstanceUnsupported`    | `f:ledger` names a ledger on a different instance. |
| `CycleDetected`               | A model ledger graph transitively references itself. |

The choice of 502 rather than 500 is deliberate: the data ledger
isn't broken — its upstream governance dependency is — and
operators distinguishing those two cases in their dashboards
matters. The wrapped variant is preserved in the response body so
clients can branch on the specific failure.

## Behavior on model ledger updates

There is no explicit invalidation channel. The cache key includes
the model ledger's `resolved_t` (its commit head at the time of
capture), so new commits to M produce new cache keys
automatically. The next request after M advances captures the
new head; older entries age out under the cache's LRU/TinyLFU
policy.

Within a single request, every cross-ledger reference to the
same M reuses one `resolved_t` value. Policy and any future
shapes / schema lookups on the same M can never disagree about
which version they're enforcing for that request.

If M is dropped while D references it, the next request against
D that needs governance from M fails closed with
`ModelLedgerMissing`. D isn't proactively notified — the
failure surface is the next request.

## Related

- [Cross-ledger model enforcement](../design/cross-ledger-model-enforcement.md) — design rationale and the shared resolver contract.
- [Policy model and inputs](policy-model.md) — policy structure (the rules themselves look the same in cross-ledger configs as in same-ledger ones).
- [Setting groups](../ledger-config/setting-groups.md#policy-defaults) — `f:policySource` and the full `f:GraphRef` shape in the config schema.
- [Programmatic policy API (Rust)](programmatic-policy.md) — how cross-ledger interacts with the in-process Rust API.


---

# indexing-and-search/README.md

# Indexing and Search

Fluree provides powerful indexing and search capabilities beyond standard graph queries. This section covers background indexing, full-text search, and vector similarity search.

## Index Types

### [Background Indexing](background-indexing.md)

Core database indexing for query performance:
- SPOT, POST, OPST, PSOT indexes
- Automatic index maintenance
- Indexing configuration
- Performance tuning
- Monitoring and metrics

### [Reindex API](reindex.md)

Manual index rebuilding for recovery and maintenance:
- Memory-bounded batched processing
- Checkpointing for resumable operations
- Progress monitoring with callbacks
- Resume after interruption
- Index configuration options

### [Inline Fulltext Search](fulltext.md)

Inline BM25-ranked text scoring. Two entry points, same query surface:
- **`@fulltext` datatype** — per-value annotation (analogous to `@vector`), always English, zero config
- **`f:fullTextDefaults` config** — declare properties + language once at the ledger level; supports 18 languages with Snowball stemming and per-graph overrides for multilingual setups
- `fulltext(?var, "query")` scoring function in `bind` expressions (same for both paths)
- Automatic per-(graph, property, language) fulltext arena construction during background indexing
- Unified scoring across indexed and novelty documents
- Works immediately (no-index fallback) with optimal performance after indexing

### [BM25 Full-Text Search](bm25.md)

Dedicated full-text search indexes using BM25 ranking (for large-scale corpora):
- Creating BM25 indexes via Rust API
- Query-based field selection (indexing query defines what to index)
- BM25 scoring with configurable k1/b parameters
- Block-Max WAND for efficient top-k queries
- Incremental index updates via property-dependency tracking

### [Vector Search](vector-search.md)

Approximate nearest neighbor (ANN) search for embeddings:
- Vector index configuration
- Embedded HNSW indexes (in-process) or remote via dedicated search service
- Embedding storage with `@vector` datatype (resolves to `https://ns.flur.ee/db#embeddingVector`)
- Similarity queries via `f:*` syntax
- Deployment modes (embedded / remote)
- Use cases (semantic search, recommendations)

### [Geospatial](geospatial.md)

Geographic point data with native binary encoding:
- `geo:wktLiteral` datatype support (OGC GeoSPARQL)
- Automatic POINT geometry detection and optimization
- Packed 60-bit lat/lng encoding (~0.3mm precision)
- Foundation for proximity queries (latitude-band index scans)

## Indexing Architecture

Fluree maintains multiple index types for different query patterns:

**Core Indexes (automatic):**
- SPOT: Subject-Predicate-Object-Time
- POST: Predicate-Object-Subject-Time
- OPST: Object-Predicate-Subject-Time
- PSOT: Predicate-Subject-Object-Time

**Graph Source Indexes (explicit):**
- BM25: Full-text search indexes
- Vector: Embedding similarity indexes
- R2RML: Relational database views
- Iceberg: Data lake integrations

## Background Indexing

Core database indexing happens automatically:

```text
Transaction → Commit → Background Indexer → Index Published
```

**Process:**
1. Transaction committed (t assigned)
2. Commit published to nameservice
3. Background indexer detects new commit
4. Indexes updated (SPOT, POST, OPST, PSOT)
5. Index snapshot published

**Novelty Layer:**
- Gap between latest commit and latest index
- Queries combine indexed data + novelty
- Monitored via `commit_t - index_t`

See [Background Indexing](background-indexing.md) for details.

## Inline Fulltext Search

For small-to-medium corpora (up to hundreds of thousands of documents per predicate), inline fulltext search provides BM25-ranked scoring with zero configuration:

**Annotate data:**
```json
{
  "@id": "ex:article-1",
  "ex:content": {
    "@value": "Rust is a systems programming language focused on safety",
    "@type": "@fulltext"
  }
}
```

**Query with scoring:**
```json
{
  "select": ["?title", "?score"],
  "where": [
    { "@id": "?doc", "ex:content": "?content", "ex:title": "?title" },
    ["bind", "?score", "(fulltext ?content \"Rust programming\")"],
    ["filter", "(> ?score 0)"]
  ],
  "orderBy": [["desc", "?score"]],
  "limit": 10
}
```

See [Inline Fulltext Search](fulltext.md) for details.

## Full-Text Search (BM25 Graph Source)

For larger corpora (1M+ documents) with strict latency requirements, the BM25 graph source pipeline provides WAND-based top-k pruning, chunked posting lists, and incremental updates:

BM25 provides ranked full-text search:

**Creating Index (Rust API):**
```rust
use fluree_db_api::Bm25CreateConfig;
use serde_json::json;

let query = json!({
    "@context": { "schema": "http://schema.org/" },
    "where": [{ "@id": "?x", "@type": "schema:Product", "schema:name": "?name" }],
    "select": { "?x": ["@id", "schema:name", "schema:description"] }
});
let config = Bm25CreateConfig::new("products-search", "mydb:main", query);
let result = fluree.create_full_text_index(config).await?;
```

There are no HTTP endpoints for index management yet — indexes are managed via the Rust API.

**Searching:**
```json
{
  "@context": {"f": "https://ns.flur.ee/db#"},
  "from": "mydb:main",
  "select": ["?product", "?score"],
  "where": [
    {
      "f:graphSource": "products-search:main",
      "f:searchText": "laptop computer",
      "f:searchLimit": 20,
      "f:searchResult": { "f:resultId": "?product", "f:resultScore": "?score" }
    }
  ],
  "orderBy": ["-?score"]
}
```

See [BM25](bm25.md) for details.

## Vector Search

Similarity search using vector embeddings via HNSW indexes (embedded or remote).

**Important**: Embeddings must be stored with the vector datatype (`@type: "@vector"`, `@type: "f:embeddingVector"`, or full IRI `https://ns.flur.ee/db#embeddingVector`) to preserve array structure.

**Creating Index (Rust API):**
```rust
let config = VectorCreateConfig::new(
    "products-vector", "mydb:main", query, "ex:embedding", 384
);
fluree.create_vector_index(config).await?;
```

**Searching:**
```json
{
  "from": "mydb:main",
  "select": ["?product", "?score"],
  "where": [
    {
      "f:graphSource": "products-vector:main",
      "f:queryVector": [0.1, 0.2, ..., 0.9],
      "f:searchLimit": 10,
      "f:searchResult": {
        "f:resultId": "?product",
        "f:resultScore": "?score"
      }
    }
  ]
}
```

See [Vector Search](vector-search.md) for details.

## Index as Graph Sources

Search indexes are exposed as graph sources:

**Graph Source Names:**
- `products-search:main` - BM25 index
- `products-vector:main` - Vector index

**Query Like Regular Ledgers:**
```json
{
  "@context": {"f": "https://ns.flur.ee/db#"},
  "from": "mydb:main",
  "select": ["?product", "?name", "?score"],
  "where": [
    {
      "f:graphSource": "products-search:main",
      "f:searchText": "laptop",
      "f:searchLimit": 20,
      "f:searchResult": { "f:resultId": "?product", "f:resultScore": "?score" }
    },
    { "@id": "?product", "schema:name": "?name" }
  ]
}
```

Combines structured data with search results via the `f:graphSource` pattern.

## Index Management

### Creating Indexes

BM25 and vector indexes are created via the Rust API. See [BM25](bm25.md) and [Vector Search](vector-search.md) for details.

### Updating Indexes

BM25 indexes are **not** automatically updated when the source ledger changes. They must be explicitly synced:

```rust
// Incremental sync (detects changes since last watermark)
let result = fluree.sync_bm25_index("products-search:main").await?;

// Or use the Bm25MaintenanceWorker for automatic background syncing
```

The `Bm25MaintenanceWorker` can be configured to watch for ledger commits and sync automatically.

### Deleting Indexes

```rust
let result = fluree.drop_full_text_index("products-search:main").await?;
```

## Performance Characteristics

### Inline Fulltext Search

- **Indexed throughput**: ~625,000 docs/sec (50K paragraph-length docs in 80ms)
- **Novelty throughput**: ~85,000 docs/sec (50K docs in ~600ms, no index required)
- **Indexed speedup**: 7-7.5x faster than novelty-only
- **Scaling**: Near-linear; ~625K docs within a 1-second query budget
- **Arena build**: Adds minimal overhead to the normal binary index build

### BM25 Search

- **Index Build Time**: O(n) for n documents
- **Top-k Query Time**: Sub-linear via Block-Max WAND — skips posting list segments that cannot contribute to the top-k, with early termination. Falls back to O(total matching postings) when k approaches corpus size.
- **Space**: ~2-3x document size
- **Updates**: Incremental via property-dependency tracking, O(changed docs)

### Vector Search

- **Flat scan (inline functions)**: O(n) brute-force, viable up to ~100K vectors with binary indexing; binary index provides ~6x speedup over novelty-only scans and ~25x for filtered queries
- **HNSW index**: O(log n) approximate nearest neighbor, recommended for 100K+ vectors or strict latency requirements
- **Space**: ~1.5x embedding size
- **Updates**: Incremental, O(1) per vector
- See [Vector Search -- Performance and Scaling](vector-search.md#performance-and-scaling) for benchmark data and guidance on when to adopt HNSW

### Combined Queries

Combine search with graph queries:

```json
{
  "@context": {"f": "https://ns.flur.ee/db#"},
  "from": "mydb:main",
  "select": ["?product", "?category"],
  "where": [
    {
      "f:graphSource": "products-search:main",
      "f:searchText": "laptop",
      "f:searchLimit": 20,
      "f:searchResult": { "f:resultId": "?product" }
    },
    { "@id": "?product", "schema:category": "?category" }
  ]
}
```

Query optimizer handles joins between the search graph source and structured data efficiently.

## Use Cases

### Full-Text Search

**E-commerce Product Search:**
```json
{
  "@context": {"f": "https://ns.flur.ee/db#"},
  "from": "products:main",
  "select": ["?product", "?score"],
  "where": [
    {
      "f:graphSource": "products-search:main",
      "f:searchText": "wireless headphones",
      "f:searchLimit": 20,
      "f:searchResult": { "f:resultId": "?product", "f:resultScore": "?score" }
    }
  ],
  "orderBy": ["-?score"]
}
```

**Document Management:**
```json
{
  "@context": {"f": "https://ns.flur.ee/db#"},
  "from": "documents:main",
  "where": [
    {
      "f:graphSource": "documents-search:main",
      "f:searchText": "quarterly report 2024",
      "f:searchLimit": 20,
      "f:searchResult": { "f:resultId": "?doc" }
    },
    { "@id": "?doc", "ex:department": "finance" }
  ]
}
```

### Vector Similarity

**Semantic Search:**
```json
{
  "from": "articles:main",
  "values": [
    ["?queryVec"],
    [{"@value": [0.1, 0.2, 0.3], "@type": "https://ns.flur.ee/db#embeddingVector"}]
  ],
  "where": [
    {
      "f:graphSource": "articles-vector:main",
      "f:queryVector": "?queryVec",
      "f:searchLimit": 10,
      "f:searchResult": {
        "f:resultId": "?article",
        "f:resultScore": "?vecScore"
      }
    }
  ],
  "select": ["?article", "?vecScore"],
  "orderBy": [["desc", "?vecScore"]]
}
```

**Recommendation Engine:**
```json
{
  "from": "products:main",
  "where": [
    {
      "@id": "ex:product-123",
      "ex:embedding": "?queryVec"
    },
    {
      "f:graphSource": "products-vector:main",
      "f:queryVector": "?queryVec",
      "f:searchLimit": 5,
      "f:searchResult": { "f:resultId": "?similar", "f:resultScore": "?vecScore" }
    }
  ],
  "select": ["?similar", "?vecScore"],
  "orderBy": [["desc", "?vecScore"]]
}
```

### Hybrid Search

Combine text and vector search:

```json
{
  "@context": {"f": "https://ns.flur.ee/db#"},
  "from": "products:main",
  "values": [
    ["?queryVec"],
    [{"@value": [0.1, 0.2, 0.3], "@type": "https://ns.flur.ee/db#embeddingVector"}]
  ],
  "where": [
    {
      "f:graphSource": "products-search:main",
      "f:searchText": "laptop",
      "f:searchLimit": 100,
      "f:searchResult": { "f:resultId": "?product", "f:resultScore": "?textScore" }
    },
    {
      "f:graphSource": "products-vector:main",
      "f:queryVector": "?queryVec",
      "f:searchLimit": 100,
      "f:searchResult": { "f:resultId": "?product", "f:resultScore": "?vecScore" }
    }
  ],
  "bind": {
    "?finalScore": "(?textScore * 0.6) + (?vecScore * 0.4)"
  },
  "orderBy": ["-?finalScore"]
}
```

## Monitoring

### Check BM25 Staleness

Check whether a BM25 index is behind its source ledger:

```rust
let check = fluree.check_bm25_staleness("products-search:main").await?;
println!("Index at t={}, ledger at t={}, stale: {}, lag: {}",
    check.index_t, check.ledger_t, check.is_stale, check.lag);
```

### Background Maintenance

The `Bm25MaintenanceWorker` watches for source ledger commits and syncs indexes automatically:
- Debounces rapid commits (configurable interval)
- Bounded concurrency for concurrent sync operations
- Registers/unregisters graph sources dynamically

## Best Practices

### 1. Choose Appropriate Index Type

- **Structured queries**: Use core graph indexes
- **Keyword search (< 500K docs)**: Use inline `@fulltext` for zero-config BM25 scoring
- **Keyword search (1M+ docs)**: Use the BM25 graph source for WAND-optimized top-k retrieval
- **Semantic similarity**: Use vector search
- **Hybrid**: Combine multiple indexes

### 2. Tune BM25 Parameters

Adjust k1 and b for your corpus:

```rust
let config = Bm25CreateConfig::new("search", "docs:main", query)
    .with_k1(1.5)  // Higher = more weight to term frequency (default: 1.2)
    .with_b(0.5);   // Lower = less document length normalization (default: 0.75)
```

The indexing query controls **which properties** are indexed — all selected text properties contribute to the document's searchable content.

### 3. Monitor Index Staleness

Check staleness after bulk operations:

```rust
let check = fluree.check_bm25_staleness("search:main").await?;
if check.is_stale {
    fluree.sync_bm25_index("search:main").await?;
}
```

### 4. Sync After Bulk Updates

BM25 indexes require explicit sync. After bulk inserts, sync once at the end:

```rust
// Insert many documents...
for batch in batches {
    fluree.insert(ledger.clone(), &batch).await?;
}
// Sync the BM25 index once after all inserts
fluree.sync_bm25_index("products-search:main").await?;
```

### 5. Use Appropriate Limits

Limit results for performance:

```json
{
  "@context": {"f": "https://ns.flur.ee/db#"},
  "from": "docs:main",
  "where": [
    {
      "f:graphSource": "docs-search:main",
      "f:searchText": "search query",
      "f:searchLimit": 100,
      "f:searchResult": { "f:resultId": "?doc" }
    }
  ]
}
```

## Related Documentation

- [Background Indexing](background-indexing.md) - Core index details
- [Inline Fulltext Search](fulltext.md) - `@fulltext` datatype and `fulltext()` scoring
- [BM25](bm25.md) - Dedicated full-text search graph source
- [Vector Search](vector-search.md) - Similarity search
- [Graph Sources](../graph-sources/README.md) - Graph source concepts
- [Query](../query/README.md) - Query syntax


---

# indexing-and-search/background-indexing.md

# Background Indexing

Fluree maintains query-optimized indexes through a background indexing process. This document covers the indexing architecture, configuration, and monitoring.

## Index Architecture

Fluree maintains four index permutations for efficient query execution:

### SPOT (Subject-Predicate-Object-Time)

Organized by subject first:

```text
ex:alice → schema:name → "Alice" → [t=1, t=5]
ex:alice → schema:age → 30 → [t=1]
ex:alice → schema:age → 31 → [t=10]
```

**Optimized for:** "Give me all properties of this subject"

### POST (Predicate-Object-Subject-Time)

Organized by predicate first:

```text
schema:name → "Alice" → ex:alice → [t=1, t=5]
schema:age → 30 → ex:alice → [t=1]
schema:age → 31 → ex:alice → [t=10]
```

**Optimized for:** "Find all subjects with this property/value"

### OPST (Object-Predicate-Subject-Time)

Organized by object first:

```text
"Alice" → schema:name → ex:alice → [t=1, t=5]
30 → schema:age → ex:alice → [t=1]
31 → schema:age → ex:alice → [t=10]
```

**Optimized for:** "Find subjects with this object value"

### PSOT (Predicate-Subject-Object-Time)

Organized by predicate, then subject:

```text
schema:name → ex:alice → "Alice" → [t=1, t=5]
schema:age → ex:alice → 30 → [t=1]
schema:age → ex:alice → 31 → [t=10]
```

**Optimized for:** "Get all values for this predicate"

## Indexing Process

### 1. Transaction Commit

```text
t=42: Transaction committed
  - Flakes written to append-only log
  - Commit metadata created
  - Commit published to nameservice (commit_t=42)
```

### 2. Indexer Detection

Background indexing is triggered when the ledger’s novelty exceeds the configured threshold (see Configuration below):

```text
Indexer checks: commit_t=42, index_t=40
Indexer: Need to index t=41, t=42
```

### 3. Index Building

Background indexing builds a new index snapshot up to a specific `to_t` (typically the current `commit_t` when the job starts). During the job, new commits may arrive; those remain in novelty for the next cycle.

```text
Incremental indexing (default path):
  - Load the existing index root (CAS CID) from nameservice
  - Resolve only commits with t in (index_t, to_t]
  - Merge resolved novelty into only the affected leaf blobs (Copy-on-Write)
  - Update dictionaries (forward packs + reverse trees)
  - Assemble a new root referencing mostly-unchanged CAS artifacts

Fallback:
  - If incremental indexing cannot safely proceed, fall back to a full rebuild
```

### 4. Index Publishing

When complete:

```text
  - Upload new CAS blobs (leaves, branches, dict blobs) as needed
  - Upload the new index root (CAS CID)
  - Publish index_head_id to nameservice (atomic “commit point”)
  - Update index_t to to_t
```

## Novelty Layer

The **novelty layer** consists of transactions committed but not yet indexed:

```text
Current State:
  commit_t = 150
  index_t = 145
  novelty = [t=146, t=147, t=148, t=149, t=150]
```

### Query Execution with Novelty

Queries combine indexed data with novelty:

```text
Query for ex:alice's properties:

1. Check SPOT index (up to t=145)
2. Apply novelty layer (t=146 to t=150)
3. Combine results
```

### Impact of Large Novelty

**Small novelty** (< 10 transactions):
- Minimal query overhead
- Fast query execution

**Large novelty** (> 100 transactions):
- Significant query overhead
- Slower query execution
- Higher memory usage

## Configuration

Background indexing is **on by default**. Indexing is triggered based on novelty size thresholds:

- Enable/disable background indexing: `--indexing-enabled` / `FLUREE_INDEXING_ENABLED` (default `true`; disable only when a peer/indexer process owns this storage)
- Trigger threshold (soft): `--reindex-min-bytes` / `FLUREE_REINDEX_MIN_BYTES`
- Backpressure threshold (hard): `--reindex-max-bytes` / `FLUREE_REINDEX_MAX_BYTES`

See [Operations: Configuration](../operations/configuration.md#background-indexing) for the canonical flag/env/config-file reference.

### Incremental parallelism (per ledger)

Within a single incremental indexing job, Fluree can update multiple `(graph, index-order)` branches concurrently. This is bounded by:

- `IndexerConfig.incremental_max_concurrency` (default: 4)

This setting is part of the Rust `IndexerConfig` used by the indexer pipeline; it is not a server CLI flag. Increasing it can improve throughput on multi-graph ledgers and can run the four main index orders (SPOT/PSOT/POST/OPST) in parallel, at the cost of higher peak memory.

## Monitoring

### Check Index Status

```bash
curl http://localhost:8090/v1/fluree/info/mydb:main
```

Response:
```json
{
  "ledger_id": "mydb:main",
  "branch": "main",
  "commit_t": 150,
  "index_t": 145,
  "commit_id": "bafy...headCommit",
  "index_id": "bafy...indexRoot"
}
```

**Key Metrics:**
- **index lag (txns)**: `commit_t - index_t`

For byte-level novelty size and indexing trigger decisions, see the `indexing` block returned by transaction and replication endpoints (e.g. `POST /push/<ledger>`), documented in [API Endpoints](../api/endpoints.md).

### Key Log Messages

At `INFO`, background indexing now emits coarse-grained progress logs that make it easier to distinguish:

- request queued vs. worker started
- current wait status while `trigger_index()` is blocked
- incremental vs. rebuild path selection
- commit-chain walking progress
- commit resolution progress and phase completion

When background indexing is queued by an HTTP transaction request, the worker logs also include copied `request_id` and `trace_id` fields from the triggering request. This provides log-level correlation between the foreground request and the later background build without making the index build part of the original request trace.

At `DEBUG`, the same wait and commit-walk paths emit more frequent progress updates for incident debugging without changing behavior.

When you call indexing through the Rust API with `trigger_index()`, wait timeout
is optional and should generally be chosen by the caller. Leave
`TriggerIndexOptions.timeout_ms` unset to wait until completion, or set it
explicitly for bounded environments such as Lambda jobs, HTTP gateways, or
other workers with a fixed maximum runtime.

`TriggerIndexResult` carries `fuel: Option<f64>` — the total fuel charged for
the build that satisfied the trigger. The background indexer always meters
each build, so callers see `Some(N)` for a real build, `Some(0.0)` when the
requested t was already indexed (no work was needed), and the same value for
all coalesced trigger callers satisfied by a single underlying build. See
[Tracking and Fuel](../query/tracking-and-fuel.md#indexing-fuel) for the
cost-ladder and rate semantics.

### Health Indicators

**Healthy:**
```text
index_lag: 0-10 transactions
index_rate > transaction_rate
```

**Warning:**
```text
index_lag: 10-50 transactions
index_rate ≈ transaction_rate
```

**Critical:**
```text
index_lag: > 50 transactions
index_rate < transaction_rate
```

## Performance Tuning

### Optimize for Write-Heavy Loads

```bash
fluree-server \
  --indexing-enabled \
  --reindex-min-bytes 200000 \
  --reindex-max-bytes 2000000
```

Larger thresholds reduce indexing frequency (more novelty accumulation), trading some query-time overlay cost for reduced background indexing activity.

### Optimize for Read-Heavy Loads

```bash
fluree-server \
  --indexing-enabled \
  --reindex-min-bytes 50000
```

Smaller `reindex-min-bytes` keeps novelty smaller (better query performance) at the cost of more frequent background indexing cycles.

## Index Storage

### Index Snapshots

Indexes are stored as immutable, content-addressed snapshots:

```text
  - Leaf blobs (FLI3) and branch manifests (FBR3)
  - Dictionary blobs (forward packs, reverse tree leaves/branches)
  - An index root blob (FIR6) that references everything needed for queries
```

The nameservice stores the current index root CID (`index_head_id`) and its watermark (`index_t`). Peers fetch only the CAS objects they need on demand.

### Index Retention

Old index snapshots are retained for time-travel safety and concurrent query safety. Cleanup is performed by the binary index garbage collector, governed by:

- `IndexerConfig.gc_max_old_indexes`
- `IndexerConfig.gc_min_time_mins`

No standalone HTTP compaction endpoint is currently exposed. Use `POST
/v1/fluree/reindex` when you need to force a full index refresh.

## Troubleshooting

### High indexing lag

**Symptom:** `commit_t - index_t` grows continuously

**Causes:**
- Transaction rate exceeds indexing capacity
- Large transactions
- Insufficient resources

**Solutions:**
1. Reduce `reindex-min-bytes` so indexing triggers sooner
2. Increase resources for the indexer (CPU/memory and storage throughput)
3. Consider running a dedicated indexer process (separate from the transactor)
4. For incremental indexing, consider increasing `IndexerConfig.incremental_max_concurrency`

### Slow Indexing

**Symptom:** `index_t` advances slowly (or stops advancing)

**Causes:**
- Disk I/O bottleneck
- CPU bottleneck
- Large index size
- Storage backend latency

**Solutions:**
1. Use faster storage (SSD)
2. Increase CPU allocation
3. Optimize transaction patterns
4. Use local storage vs network storage

### Index Corruption

**Symptom:** Query errors, unexpected results

**Recovery:** Use the [Reindex API](reindex.md) to rebuild indexes from scratch if you suspect corruption or need to change index structure parameters.

## Best Practices

### 1. Monitor Novelty

```javascript
setInterval(async () => {
  const status = await fetch('http://localhost:8090/v1/fluree/info/mydb:main')
    .then(r => r.json());
  
  const lag = status.t - status.index.t;
  if (lag > 50) {
    console.warn(`High indexing lag: ${lag} transactions`);
  }
}, 30000);  // Check every 30 seconds
```

### 2. Tune for Workload

Match configuration to workload pattern:
- Write-heavy: Larger `reindex-min-bytes` (fewer indexing cycles)
- Read-heavy: Smaller `reindex-min-bytes` (less novelty overlay)
- Balanced: Default settings

### 3. Capacity Planning

Estimate indexing capacity:

```text
Transaction rate: 10 txn/second
Avg flakes per txn: 100
Total flakes: 1,000 flakes/second

Indexing capacity: 2,000 flakes/second (2× margin)
```

### 4. Alert on Lag

Set up alerting:

```javascript
const lag = status.commit_t - status.index_t;
if (lag > 100) {
  alertOps('Critical: Indexing lag > 100 transactions');
}
```

### 5. Scheduled Reindex

Run a full reindex during off-peak hours when you need to rebuild indexes:

```bash
# Cron job
0 2 * * * curl -X POST http://localhost:8090/v1/fluree/reindex -H "Content-Type: application/json" -d '{"ledger":"mydb:main"}'
```

## Related Documentation

- [Reindex API](reindex.md) - Manual index rebuilding and recovery
- [Indexing Side-Effects](../transactions/indexing-side-effects.md) - Transaction impact on indexing
- [Query Performance](../query/explain.md) - Query optimization
- [BM25](bm25.md) - Full-text search indexing
- [Vector Search](vector-search.md) - Vector indexing


---

# indexing-and-search/reindex.md

# Reindex API

The Reindex API provides full rebuilds of ledger indexes from the commit chain. Use this when you need to rebuild indexes from scratch, such as after suspected corruption or index configuration changes.

## Overview

Unlike [background indexing](background-indexing.md) which incrementally updates indexes as transactions commit, reindexing rebuilds the entire binary columnar index from the commit history.

Reindex publishes the new index root via `publish_index_allow_equal`, which means a reindex can produce a **new index root CID** even when `index_t` stays the same (same logical snapshot, different physical layout/config).

## When to Reindex

### Common Use Cases

1. **Index corruption** - Query errors or unexpected results suggest corrupted indexes
2. **Configuration changes** - Changing index parameters (leaf size, branch size)
3. **Storage backend changes** - If you move a deployment between storage backends or adopt a new index strategy/type.

### Before You Reindex

Consider these factors:

- **Duration**: Full reindex scales with ledger size; large ledgers may take hours
- **Resources**: Ensure adequate memory and storage during the operation
- **Availability**: Queries remain available during reindex, but may be slower
- **Backup**: Be sure to back up data before major reindex operations

## Rust API

The reindex API is exposed through the `Fluree` type in `fluree-db-api`. `Fluree` owns the storage backend, node cache, nameservice, and provides all ledger operations including queries, transactions, and admin functions like reindex.

### Basic Reindex

```rust
use fluree_db_api::{FlureeBuilder, ReindexOptions, ReindexResult};

// Create Fluree instance
let fluree = FlureeBuilder::file("/path/to/data")
    .build()
    .await?;

// Reindex with default options
let result: ReindexResult = fluree.reindex("mydb:main", ReindexOptions::default()).await?;

println!("Reindexed to t={}", result.index_t);
println!("Root ID: {}", result.root_id);
```

### Reindex with Custom Options

```rust
use fluree_db_api::{FlureeBuilder, ReindexOptions};
use fluree_db_indexer::IndexerConfig;

let fluree = FlureeBuilder::file("/path/to/data").build().await?;

let result = fluree.reindex("mydb:main", ReindexOptions::default()
    // Use custom index node sizes
    .with_indexer_config(IndexerConfig::large())
).await?;
```

## ReindexOptions Reference

| Option | Default | Description |
|--------|---------|-------------|
| `indexer_config` | `IndexerConfig::default()` | Controls output index structure (leaf/branch sizes, GC settings, memory budget) |

### `indexer_config`

Controls the output index structure and rebuild resources:

```rust
use fluree_db_indexer::IndexerConfig;

// For small datasets (< 100k flakes)
ReindexOptions::default()
    .with_indexer_config(IndexerConfig::small())

// For large datasets (> 10M flakes)
ReindexOptions::default()
    .with_indexer_config(IndexerConfig::large())

// Custom configuration
let config = IndexerConfig::default()
    .with_gc_max_old_indexes(10)       // Keep more old index versions
    .with_gc_min_time_mins(60)         // Retain for at least 60 minutes
    .with_run_budget_bytes(1 << 30)    // 1 GB memory budget for sort buffers
    .with_data_dir("/data/fluree");    // Directory for index artifacts

ReindexOptions::default()
    .with_indexer_config(config)
```

Key `IndexerConfig` fields:

| Field | Default | Description |
|-------|---------|-------------|
| `leaf_target_bytes` | 187,500 | Target bytes per leaf node |
| `leaf_max_bytes` | 375,000 | Maximum bytes per leaf node (triggers split) |
| `branch_target_children` | 100 | Target children per branch node |
| `branch_max_children` | 200 | Maximum children per branch node |
| `gc_max_old_indexes` | 5 | Old index versions to retain before GC |
| `gc_min_time_mins` | 30 | Minimum age (minutes) before an index can be GC'd |
| `run_budget_bytes` | 256 MB | Memory budget for sort buffers (split across all sort orders) |
| `data_dir` | System temp dir | Base directory for index artifacts |
| `incremental_enabled` | true | Background indexing: attempt incremental updates before full rebuild |
| `incremental_max_commits` | 10,000 | Background indexing: max commit window for incremental indexing |
| `incremental_max_concurrency` | 4 | Background indexing: max concurrent (graph, order) branch updates |
| `incremental_leaf_upload_concurrency` | 16 | Background indexing: global budget of concurrent leaf/sidecar uploads in Phase 2, shared across all order-tasks |

Note: Reindex is a full rebuild. The `incremental_*` fields are used by background indexing and are not relevant to the semantics of a reindex operation.

## ReindexResult

The reindex operation returns:

```rust
pub struct ReindexResult {
    /// Ledger ID
    pub ledger_id: String,
    /// Transaction time the index was built to
    pub index_t: i64,
    /// ContentId of the new index root
    pub root_id: ContentId,
    /// Index build statistics
    pub stats: IndexStats,
}
```

## Error Handling

### Common Errors

```rust
use fluree_db_api::ApiError;

match fluree.reindex("mydb:main", opts).await {
    Ok(result) => println!("Success: t={}", result.index_t),
    Err(ApiError::NotFound(msg)) => {
        // Ledger doesn't exist or has no commits
        println!("Ledger not found: {}", msg);
    }
    Err(ApiError::ReindexConflict { expected, found }) => {
        // Ledger advanced during reindex (new commits arrived)
        println!("Conflict: expected t={}, found t={}", expected, found);
    }
    Err(e) => {
        // Storage, indexing, or other errors
        println!("Reindex failed: {}", e);
    }
}
```

## How It Works

The reindex operation:

1. **Looks up** the current ledger state and captures `commit_t` for conflict detection
2. **Cancels** any active background indexing for the ledger
3. **Rebuilds** a fresh binary columnar index from the full commit chain using `rebuild_index_from_commits`:
   - **Phase A**: Walks the commit DAG once, reading only the envelope header of each commit via byte-range requests (`ContentStore::get_range`). Returns the chronological CID list plus the genesis-most `NsSplitMode` in a single pass, so per-commit bandwidth on remote storage is ~128 KiB rather than the full commit blob.
   - **Phase B**: Resolves commits into batched chunks with chunk-local dictionaries (subjects, strings) and shared global dictionaries (predicates, datatypes, graphs, languages, numbigs, vectors). Commit blobs are pre-fetched concurrently (`buffered(K)`, default `K=3`, env-tunable via `FLUREE_REBUILD_FETCH_CONCURRENCY`) so S3 round-trip latency overlaps with local decode cost.
   - **Phase C**: Merges per-chunk dictionaries into global dictionaries with remap tables
   - **Phase D**: Builds SPOT indexes from sorted commit files via k-way merge with graph-aware partitioning
   - **Phase E**: Builds secondary indexes (PSOT, POST, OPST) per-graph from partitioned run files
   - **Phase F**: Uploads dictionaries and index artifacts to CAS, creates `IndexRoot` (FIR6)
4. **Validates** that no new commits arrived during the build (conflict detection)
5. **Publishes** the new index root via `publish_index_allow_equal`
6. **Spawns** async garbage collection to clean up old index versions

The rebuilt index preserves full time-travel history: retract-winner events and their preceding asserts are stored in Region 3 (history) of leaf nodes, enabling `as-of` queries at any past transaction time.

## Best Practices

### 1. Schedule During Low-Traffic Periods

While queries continue to work during reindex, performance may be impacted. Schedule large reindex operations during maintenance windows when possible.

### 2. Tune Memory Budget for Large Ledgers

For ledgers with millions of flakes, increasing `run_budget_bytes` reduces the number of spill files and speeds up the merge phase:

```rust
let config = IndexerConfig::default()
    .with_run_budget_bytes(2 * 1024 * 1024 * 1024); // 2 GB
```

### 3. Tune Phase B Fetch Concurrency for Remote Storage

When reindexing from remote storage (S3) on latency-bound platforms like AWS Lambda, Phase B benefits from fetching several commit blobs in parallel so S3 round-trip latency (25–50 ms) overlaps with local decode cost.

```bash
# Default: 3. Increase for high-latency links; pin to 1 for strict serial behavior.
export FLUREE_REBUILD_FETCH_CONCURRENCY=4
```

In-flight memory is bounded by `K × avg_commit_blob_size`. For typical commits (< 1 MB) and `K=3`, the overhead is negligible against the `run_budget_bytes` pool. Pathologically large commits (hundreds of MB) should set `K=1` to avoid transient memory spikes.

### 4. Verify After Reindex

After reindex, verify the results:

```rust
// Get ledger info to check state
let info = fluree.ledger_info(ledger_id).execute().await?;
println!("Index rebuilt to t={}", info["index"]["t"]);

// Run a sample query to verify correctness
let db = fluree_db_api::GraphDb::from_ledger_state(&ledger);
let query_result = fluree.query(&db, &sample_query).await?;
```

### 5. Concurrent Operations

During reindex:
- Queries continue to work (using old index + novelty)
- Transactions continue to work (writes to novelty)
- Background indexing is paused for this ledger

## Related Documentation

- [Background Indexing](background-indexing.md) - Automatic incremental indexing
- [Admin and Health](../operations/admin-and-health.md) - Admin operations
- [Rust API](../getting-started/rust-api.md) - Using Fluree as a library
- [Storage](../operations/storage.md) - Storage configuration


---

# indexing-and-search/fulltext.md

# Inline Fulltext Search

Inline fulltext search enables BM25-ranked text scoring directly in queries, using the `@fulltext` datatype (or a ledger-level `f:fullTextDefaults` config) and the `fulltext()` scoring function. This follows the same pattern as `@vector` and inline similarity functions: declare what to index, persist as normal commits, and query with a scoring function in `bind` expressions. No external services, no separate ingestion pipeline.

Two ways to enable fulltext scoring on a property:

- **Per-value annotation** (`@fulltext` datatype) — zero-config, always English. Tag individual literal values at insert time. Good for a handful of obviously-fulltext fields where English is fine.
- **Property-level configuration** (`f:fullTextDefaults`) — declare once in the ledger's config graph which properties should be full-text indexed, and optionally which language to analyze them in. Plain-string values on those properties get indexed automatically — no `@type` annotation needed at insert time. Required when you want non-English stemming/stopwords, or when you want every value of a property indexed by default.

Both paths produce the same on-disk BM25 arenas and are queried with the same `fulltext(?var, "query")` function.

Use cases:

- **Document ranking**: Score and rank articles, product descriptions, or knowledge base entries by keyword relevance
- **Content discovery**: Find the most relevant documents for a natural language query
- **Faceted search**: Combine fulltext scoring with graph pattern filters (e.g., score only documents in a specific category)
- **Multilingual catalogs**: Index product descriptions in Spanish on one graph and English on another, with the right stemmer picked automatically per-language

## The `@fulltext` Datatype

### Why a dedicated datatype?

Plain strings in Fluree are stored as `xsd:string` values. They are indexed for exact matching and prefix queries, but not for full-text search. The `@fulltext` datatype tells Fluree that a string value should be analyzed (tokenized, stemmed, stopword-filtered) and indexed for relevance scoring.

`@fulltext` is a JSON-LD shorthand that resolves to the full IRI `https://ns.flur.ee/db#fullText`, which can also be written as `f:fullText` when the Fluree namespace prefix is declared in your `@context`.

### Inserting fulltext values (JSON-LD)

Use `"@type": "@fulltext"` to annotate a string as fulltext-searchable:

```json
{
  "@context": {
    "ex": "http://example.org/"
  },
  "@graph": [
    {
      "@id": "ex:article-1",
      "@type": "ex:Article",
      "ex:title": "Rust Programming",
      "ex:content": {
        "@value": "Rust is a systems programming language focused on safety and performance",
        "@type": "@fulltext"
      }
    }
  ]
}
```

You can also use the full IRI or `f:` prefix form:

```json
{
  "@context": {
    "ex": "http://example.org/",
    "f": "https://ns.flur.ee/db#"
  },
  "@graph": [
    {
      "@id": "ex:article-1",
      "ex:content": {
        "@value": "Rust is a systems programming language...",
        "@type": "f:fullText"
      }
    }
  ]
}
```

### Inserting fulltext values (Turtle / SPARQL UPDATE)

In Turtle and SPARQL UPDATE, the `@fulltext` shorthand is not available. Use the `f:fullText` datatype IRI with the standard `^^` typed-literal syntax.

**Turtle data file:**

```turtle
@prefix ex: <http://example.org/> .
@prefix f: <https://ns.flur.ee/db#> .

ex:article-1
  a ex:Article ;
  ex:title "Introduction to Rust" ;
  ex:content "Rust is a systems programming language focused on safety and performance"^^f:fullText .

ex:article-2
  a ex:Article ;
  ex:title "Database Design Patterns" ;
  ex:content "Modern database systems use columnar storage and immutable ledgers"^^f:fullText .
```

**SPARQL UPDATE:**

```sparql
PREFIX ex: <http://example.org/>
PREFIX f: <https://ns.flur.ee/db#>

INSERT DATA {
  ex:article-1 a ex:Article ;
    ex:title "Introduction to Rust" ;
    ex:content "Rust is a systems programming language focused on safety"^^f:fullText .
}
```

The `^^f:fullText` annotation is the Turtle/SPARQL equivalent of `"@type": "@fulltext"` in JSON-LD. Without it, the string is stored as a plain `xsd:string`.

### Multiple fulltext properties per entity

An entity can have `@fulltext` on multiple different properties:

```json
{
  "@id": "ex:article-1",
  "ex:title": {
    "@value": "Rust Programming Guide",
    "@type": "@fulltext"
  },
  "ex:content": {
    "@value": "Rust is a systems programming language focused on safety...",
    "@type": "@fulltext"
  }
}
```

Each property produces an independent fulltext index (arena). When you query with `fulltext()`, the function automatically uses the arena for the property bound to the variable.

### Portability

`@fulltext` annotations are fully portable across Fluree's data distribution pipeline. Import, export, push, and pull all preserve `@fulltext` type annotations, and indexes are rebuilt transparently on the receiving side.

## Configured Full-Text Properties (`f:fullTextDefaults`)

The `@fulltext` datatype is a per-value shortcut — you decide at insert time, one triple at a time, whether a string gets full-text indexed, and English is the only supported language. For many real-world workloads that's not what you want. You want to say once, at the ledger level, "index every value of `ex:title`", or "index `ex:productName` in the product catalog graph in Spanish." That's what `f:fullTextDefaults` gives you.

When a property is declared in `f:fullTextDefaults`, any plain `xsd:string` or `rdf:langString` value on that property gets full-text indexed — no `@type: @fulltext` needed on individual values. Language-tagged (`rdf:langString`) values automatically route to a per-language arena (French stemmer for `"fr"`, Spanish stopwords for `"es"`, and so on). Untagged plain strings fall back to the configured default language.

The `@fulltext` datatype continues to work exactly as before: any value tagged `@fulltext` is always indexed as English, regardless of what `f:fullTextDefaults` says about its property. You can mix both paths on the same property; English content from either path lands in a single shared arena.

### When to use which

| Need | Use |
|------|-----|
| English-only, a few obviously-fulltext fields, want the choice per-value | `@fulltext` datatype |
| Non-English (or mixed languages) | `f:fullTextDefaults` with `f:defaultLanguage` |
| Every value of a property should be searchable, no per-value opt-in | `f:fullTextDefaults` |
| Different languages per graph (e.g. multilingual catalog) | `f:fullTextDefaults` with per-graph overrides |
| Zero config, just works | `@fulltext` datatype |

### Setting it up

Write configuration into the ledger's `#config` named graph, alongside any other config groups (policy, SHACL, reasoning, etc.). The config is itself a transaction — it's versioned and auditable like any other data.

**Minimal — index `ex:title` and `ex:body`, English by default:**

```trig
@prefix f: <https://ns.flur.ee/db#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix ex: <http://example.org/> .

GRAPH <urn:fluree:mydb:main#config> {
  <urn:fluree:mydb:main:config:ledger> a f:LedgerConfig ;
    f:fullTextDefaults [
      a f:FullTextDefaults ;
      f:defaultLanguage "en" ;
      f:property [ a f:FullTextProperty ; f:target ex:title ] ,
                 [ a f:FullTextProperty ; f:target ex:body ]
    ] .
}
```

Or as JSON-LD:

```json
{
  "@context": {
    "f": "https://ns.flur.ee/db#",
    "ex": "http://example.org/"
  },
  "@graph": [
    {
      "@id": "urn:fluree:mydb:main:config:ledger",
      "@type": "f:LedgerConfig",
      "@graph": "urn:fluree:mydb:main#config",
      "f:fullTextDefaults": {
        "@type": "f:FullTextDefaults",
        "f:defaultLanguage": "en",
        "f:property": [
          { "@type": "f:FullTextProperty", "f:target": { "@id": "ex:title" } },
          { "@type": "f:FullTextProperty", "f:target": { "@id": "ex:body" } }
        ]
      }
    }
  ]
}
```

**HTTP / Docker:** the same JSON-LD config goes into a regular `/update` transaction. Wrap it in `@graph` and POST to the ledger:

```bash
curl -X POST 'http://localhost:8090/v1/fluree/update?ledger=mydb:main' \
  -H 'Content-Type: application/json' \
  -d @- <<'JSON'
{
  "@context": {
    "f": "https://ns.flur.ee/db#",
    "ex": "http://example.org/"
  },
  "@graph": [
    {
      "@id": "urn:fluree:mydb:main:config:ledger",
      "@type": "f:LedgerConfig",
      "@graph": "urn:fluree:mydb:main#config",
      "f:fullTextDefaults": {
        "@type": "f:FullTextDefaults",
        "f:defaultLanguage": "en",
        "f:property": [
          { "@type": "f:FullTextProperty", "f:target": { "@id": "ex:title" } },
          { "@type": "f:FullTextProperty", "f:target": { "@id": "ex:body" } }
        ]
      }
    }
  ]
}
JSON
```

The config is stored in the ledger's `#config` named graph (note the `"@graph": "urn:fluree:mydb:main#config"` placement directive on the resource). To verify, query the config graph:

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H 'Content-Type: application/json' \
  -d '{
    "@context": { "f": "https://ns.flur.ee/db#" },
    "from": "mydb:main",
    "from-named": ["urn:fluree:mydb:main#config"],
    "where": [{ "@graph": "urn:fluree:mydb:main#config",
                "@id": "?cfg", "f:fullTextDefaults": "?defaults" }],
    "select": ["?cfg", "?defaults"]
  }'
```

After writing config, trigger a reindex so existing values on `ex:title` and `ex:body` get indexed. See [Reindexing after a config change](#reindexing-after-a-config-change) below.

**Data writes don't change.** Once config is in place and the reindex has run, just insert plain strings the way you always would:

```json
{
  "@id": "ex:doc1",
  "ex:title": "Rust programming language guide",
  "ex:body": "Rust is a systems programming language..."
}
```

Both values flow into BM25 arenas automatically.

### Multiple languages

Fluree ships Snowball stemmers and curated stopwords for 18 languages. Pick one as your ledger default via `f:defaultLanguage`; any BCP-47 tag in the list below works.

| Tag | Language |
|-----|----------|
| `ar` | Arabic |
| `da` | Danish |
| `de` | German |
| `el` | Greek |
| `en` | English |
| `es` | Spanish |
| `fi` | Finnish |
| `fr` | French |
| `hu` | Hungarian |
| `it` | Italian |
| `nl` | Dutch |
| `no` (or `nb`, `nn`) | Norwegian |
| `pt` | Portuguese |
| `ro` | Romanian |
| `ru` | Russian |
| `sv` | Swedish |
| `ta` | Tamil |
| `tr` | Turkish |

A BCP-47 tag that isn't on this list still works — it just skips stemming and stopword removal (tokenize + lowercase only). Index and query sides agree on that behavior so scores remain consistent.

**Per-value language tagging via `rdf:langString`.** If a single property holds values in different languages, tag them with `@language` (JSON-LD) or `@lang` (Turtle):

```json
{
  "@id": "ex:doc1",
  "ex:title": [
    { "@value": "Rust programming", "@language": "en" },
    { "@value": "Programmation Rust", "@language": "fr" }
  ]
}
```

Fluree automatically builds per-language arenas (`ex:title` in English, `ex:title` in French) and queries against the arena whose language matches the row's tag. Untagged values fall back to the ledger's `f:defaultLanguage`.

### Per-graph overrides

Different graphs can have different full-text configuration. For example, a product catalog graph might index `ex:productName` in Spanish while the rest of the ledger uses English:

```trig
@prefix f: <https://ns.flur.ee/db#> .
@prefix ex: <http://example.org/> .

GRAPH <urn:fluree:mydb:main#config> {
  <urn:fluree:mydb:main:config:ledger> a f:LedgerConfig ;
    # Ledger-wide: English, index ex:title everywhere.
    f:fullTextDefaults [
      a f:FullTextDefaults ;
      f:defaultLanguage "en" ;
      f:property [ a f:FullTextProperty ; f:target ex:title ]
    ] ;
    # Catalog graph: also index ex:productName, default Spanish.
    f:graphOverrides [
      a f:GraphConfig ;
      f:targetGraph <urn:example:productCatalog> ;
      f:fullTextDefaults [
        a f:FullTextDefaults ;
        f:defaultLanguage "es" ;
        f:property [ a f:FullTextProperty ; f:target ex:productName ]
      ]
    ] .
}
```

The merge is **additive**: every property in the ledger-wide list applies to every graph (including `productCatalog`), and the per-graph override *adds* `ex:productName` on top of `ex:title`. The override's `f:defaultLanguage` shadows the ledger-wide language only for untagged plain strings on that specific graph.

**Targeting the default graph or txn-meta explicitly.** Use the `f:defaultGraph` sentinel to target only the default graph (`g_id = 0`), or `f:txnMetaGraph` for the ledger's txn-meta graph:

```trig
f:graphOverrides [
  a f:GraphConfig ;
  f:targetGraph f:defaultGraph ;
  f:fullTextDefaults [
    a f:FullTextDefaults ;
    f:property [ a f:FullTextProperty ; f:target ex:note ]
  ]
]
```

### Locking config (`f:overrideControl`)

If you want to prevent per-graph overrides from modifying the ledger-wide full-text defaults, set `f:overrideControl` to `f:OverrideNone` on the ledger-wide group:

```trig
<urn:fluree:mydb:main:config:ledger> f:fullTextDefaults [
  a f:FullTextDefaults ;
  f:defaultLanguage "en" ;
  f:overrideControl f:OverrideNone ;
  f:property [ a f:FullTextProperty ; f:target ex:title ]
] .
```

With `f:OverrideNone`, any `f:graphOverrides` entry targeting `f:fullTextDefaults` is ignored at resolution time — the ledger-wide group is final. See [Override control](../ledger-config/override-control.md) for the full model.

### Reindexing after a config change

Writing or editing `f:fullTextDefaults` does **not** automatically rebuild any arenas. You control when reindexing happens.

**What you need to know:**

1. **New commits after the config change** pick up the new config automatically during the next incremental index build — newly inserted values on configured properties flow into arenas as expected.
2. **Existing values** that were committed *before* the config change are not retroactively indexed until you run a full reindex.
3. **Removing or renaming a property** from `f:fullTextDefaults` drops it from the configured set for new commits, but the existing arena stays until you reindex.
4. **Changing `f:defaultLanguage`** doesn't rewrite existing arenas — they keep whatever language they were built with. New values get the new language; scores may be temporarily inconsistent across the old/new boundary until a reindex.

To force the full picture — pick up config changes for *all* existing data — run a manual reindex:

```bash
# CLI
fluree reindex mydb:main

# Or via the admin API
curl -X POST https://<fluree-server>/v1/fluree/reindex \
  -H 'Content-Type: application/json' \
  -d '{"ledger": "mydb:main"}'
```

The reindex reads the current `f:fullTextDefaults`, walks the entire commit chain, and rebuilds arenas with the new configuration applied consistently.

> **Note on concurrent reindex + config write.** A reindex already in progress operates on a point-in-time snapshot and will NOT pick up a config change committed during its run. If you change config during a reindex, wait for it to finish, then trigger another reindex. See [Reindex](reindex.md) for full semantics.

### How config-path and `@fulltext`-datatype coexist

If a value's datatype is `@fulltext`, the datatype wins: that value is indexed as English, even if the property is listed in `f:fullTextDefaults` with a different `f:defaultLanguage`. This keeps the `@fulltext` contract stable ("I tagged this value English, index it now") and guarantees no double-indexing.

In practice, a single property can mix:

- `@fulltext`-datatype values → English arena
- `rdf:langString` values tagged `"fr"` → French arena
- Plain `xsd:string` values → arena for the configured `f:defaultLanguage`

Each language becomes its own arena; queries automatically look up the right one based on the row's language tag (with English as the fallback). Ledger-wide English content from both paths shares a single arena — no wasted duplication.

## The `fulltext()` Scoring Function

The `fulltext()` function computes a BM25 relevance score for a bound text value against a query string. Use it in `bind` expressions within JSON-LD queries.

### Basic usage

```json
{
  "@context": {
    "ex": "http://example.org/"
  },
  "select": ["?title", "?score"],
  "where": [
    { "@id": "?doc", "ex:content": "?content", "ex:title": "?title" },
    ["bind", "?score", "(fulltext ?content \"Rust programming\")"],
    ["filter", "(> ?score 0)"]
  ],
  "orderBy": [["desc", "?score"]],
  "limit": 10
}
```

**Arguments:**
- First argument: a variable bound to a `@fulltext`-typed value
- Second argument: the search query string (natural language)

**Returns:** A numeric score (`xsd:double`). Higher scores indicate greater relevance. Returns `0.0` when the document contains none of the query terms.

### Alternative array syntax

The function also accepts array form:

```json
["bind", "?score", ["fulltext", "?content", "Rust programming"]]
```

This is equivalent to the S-expression string form.

### Filtering by score

Combine `bind` with `filter` to exclude non-matching documents:

```json
["bind", "?score", "(fulltext ?content \"search terms\")"],
["filter", "(> ?score 0)"]
```

### Combining with graph patterns

Fulltext scoring works naturally with standard graph patterns. Filter by type, category, or relationships before or after scoring:

```json
{
  "@context": {
    "ex": "http://example.org/"
  },
  "select": ["?title", "?score"],
  "where": [
    {
      "@id": "?doc",
      "@type": "ex:Article",
      "ex:content": "?content",
      "ex:title": "?title",
      "ex:category": "?cat"
    },
    ["filter", "(= ?cat \"technology\")"],
    ["bind", "?score", "(fulltext ?content \"distributed database systems\")"],
    ["filter", "(> ?score 0)"]
  ],
  "orderBy": [["desc", "?score"]],
  "limit": 10
}
```

Placing the category filter before the `fulltext()` bind reduces the number of documents scored, improving query performance.

## How Scoring Works

The `fulltext()` function uses **BM25** (Best Match 25), the standard information retrieval scoring algorithm used by search engines.

### BM25 formula

For each query term *t* in document *d*:

```
IDF(t)     = ln((N - df(t) + 0.5) / (df(t) + 0.5) + 1)
TF_norm(t) = tf(t,d) * (k1 + 1) / (tf(t,d) + k1 * (1 - b + b * |d| / avgdl))
score(q,d) = SUM( IDF(t) * TF_norm(t) )  for each query term t
```

### What makes the scoring effective

- **IDF (Inverse Document Frequency)** -- Downweights common terms ("the", "is") and boosts rare, discriminative terms. A query for "distributed database" gives more weight to "distributed" (rarer) than "database" (common in a tech corpus).

- **Document length normalization** -- Prevents long documents from dominating purely due to having more words. Controlled by parameter *b* (default 0.75). A 50-word abstract mentioning "database" twice scores comparably to a 500-word article mentioning it twice.

- **Term frequency saturation** -- Diminishing returns for repeated terms, controlled by parameter *k1* (default 1.2). The 5th occurrence of "database" in a document contributes less than the 1st.

- **Corpus-wide average document length** (`avgdl`) -- Anchors the length normalization across the entire collection.

### Text analysis pipeline

Both documents and queries go through the same analysis pipeline, and the index and query sides always use the same analyzer for a given arena — so query stems match document stems:

1. **Tokenization** -- Split text on whitespace and punctuation (Unicode-aware)
2. **Lowercasing** -- Normalize to lowercase
3. **Stopword removal** -- Remove common stopwords for the bucket's language ("the", "is", "and" in English; "le", "la", "et" in French; etc.)
4. **Stemming** -- Reduce words to stems using the Snowball stemmer for the bucket's language

This means a query for "programming" against an English arena matches documents containing "programmed", "programs", or "programmer". A French-language arena stems French word forms instead ("chantait" → "chant", matching "chanter", "chantons", and so on).

For the `@fulltext` datatype, the analyzer is always English. For properties declared in `f:fullTextDefaults`, the analyzer matches the arena's language (row's `rdf:langString` tag, or the configured `f:defaultLanguage`). An unrecognized BCP-47 tag skips steps 3 and 4 — tokenize + lowercase only — consistently on both sides.

## Indexing

### Automatic arena construction

During background binary index builds, Fluree automatically constructs a **FulltextArena** (FTA1 format) for each `(graph, predicate)` combination that has `@fulltext` values. Each arena stores:

- A sorted **term dictionary** of stemmed tokens
- Per-document **bag-of-words** (BoW) entries: `(term_id, tf)` pairs sorted by term ID
- **Corpus-level statistics**: document count (*N*), sum of document lengths (*sum_dl*), and per-term document frequency (*df*)

This precomputed representation enables fast scoring at query time -- the indexed path avoids per-row text analysis entirely, reading precomputed BoW entries via binary search.

### No-index fallback

If no binary index has been built yet (e.g., immediately after ledger creation), `fulltext()` still works using an on-the-fly analysis fallback. Documents are tokenized and scored using TF-saturation (a simplified scoring model). This is slower but ensures the feature works before background indexing catches up.

### Novelty overlay

Documents committed after the last index build (in the "novelty" layer) are automatically included in query results with consistent BM25 scores. Fluree computes effective corpus statistics by merging the persisted arena stats with a novelty delta:

- `N' = N_arena + delta_N_novelty`
- `avgdl' = (sum_dl_arena + delta_sum_dl_novelty) / N'`
- `df'(t) = df_arena(t) + delta_df_novelty(t)`

This ensures that indexed documents and novelty documents produce comparable, consistent scores in the same query.

### Retraction handling

When a `@fulltext` value is retracted, it is removed from the arena at the next index build. The retracted document no longer appears in fulltext query results and its statistics are excluded from corpus-level calculations.

## Performance

### Query-time benchmarks

All benchmarks measure the full end-to-end query path: JSON-LD parse, query plan, scan, BM25 score, sort, and limit 10. Documents are paragraph-length (~30-60 words), representative of article abstracts, product descriptions, or knowledge base entries.

| Documents | Novelty (no index) | Indexed (arena BM25) | Speedup |
|----------:|:------------------:|:--------------------:|:-------:|
| 1,000 | 11.6 ms | 1.7 ms | 6.7x |
| 5,000 | 57.0 ms | 7.9 ms | 7.2x |
| 10,000 | 115.8 ms | 15.5 ms | 7.5x |
| 50,000 | 601.9 ms | 80.2 ms | 7.5x |

**Indexed throughput: ~625,000 docs/sec** -- 50K documents scored and ranked in 80ms.

**Novelty throughput: ~85,000 docs/sec** -- 50K documents in ~600ms (no index required).

The indexed path is 7-7.5x faster because it reads precomputed BoW entries via binary search on sorted `(term_id, tf)` arrays, avoiding per-row text analysis and HashMap allocation.

Scaling is near-linear. Extrapolating, the indexed path handles approximately 625K documents within a 1-second query budget.

### When to consider the BM25 graph source pipeline

Inline `@fulltext` works well for **tens to hundreds of thousands of documents** per predicate. For larger corpora (1M+ documents), consider the dedicated [BM25 graph source pipeline](bm25.md), which provides:

- **WAND (Weak AND) top-k pruning** -- Skips documents that provably cannot enter the top-k results, critical for large corpora where scanning every document is prohibitive
- **Chunked posting list storage** -- Compressed, seekable posting lists with skip pointers for efficient I/O at scale
- **Incremental index updates** -- Updates posting lists in place without rebuilding the full index
- **Cross-property dependency tracking** -- BM25 scores can depend on fields from other properties
- **Configurable analyzers per property** -- Language-specific tokenizers, stemmers, and stopword lists
- **Multi-term query optimization** -- Term-at-a-time vs document-at-a-time evaluation strategies

| Corpus size | Recommendation |
|-------------|----------------|
| < 100K docs | Inline `@fulltext` works well, especially with binary indexing |
| 100K - 500K | Inline `@fulltext` remains viable; query times scale linearly |
| 500K - 1M | Evaluate based on latency requirements; WAND pruning may help |
| 1M+ | Use the [BM25 graph source](bm25.md) for production workloads |

## Comparison with `@vector`

Both `@fulltext` and `@vector` follow the same architectural pattern: annotate, commit, index, query.

| | `@vector` | `@fulltext` |
|---|---|---|
| **Annotation** | `"@type": "@vector"` | `"@type": "@fulltext"` |
| **Index artifact** | VAS1 arena (raw vectors) | FTA1 arena (BoW + corpus stats) |
| **Scoring function** | `dotProduct`, `cosineSimilarity`, `euclideanDistance` | `fulltext(?var, "query")` |
| **Query input** | Vector literal | Natural language string |
| **Per-row cost** | O(dims) float math | O(query_terms) integer lookups |
| **Portability** | Push/pull/import/export preserves `@vector` | Push/pull/import/export preserves `@fulltext` |

## Complete Example

**1. Insert documents with fulltext content:**

```json
{
  "@context": {
    "ex": "http://example.org/"
  },
  "@graph": [
    {
      "@id": "ex:article-1",
      "@type": "ex:Article",
      "ex:title": "Introduction to Rust",
      "ex:content": {
        "@value": "Rust is a systems programming language focused on safety, speed, and concurrency. It prevents segfaults and guarantees thread safety.",
        "@type": "@fulltext"
      }
    },
    {
      "@id": "ex:article-2",
      "@type": "ex:Article",
      "ex:title": "Database Design Patterns",
      "ex:content": {
        "@value": "Modern database systems use columnar storage and immutable ledgers. Graph databases model relationships as first-class citizens.",
        "@type": "@fulltext"
      }
    },
    {
      "@id": "ex:article-3",
      "@type": "ex:Article",
      "ex:title": "Rust for Systems Programming",
      "ex:content": {
        "@value": "Building high-performance systems in Rust requires understanding ownership, borrowing, and lifetime semantics. Rust's type system catches bugs at compile time.",
        "@type": "@fulltext"
      }
    }
  ]
}
```

**2. Query -- find articles about "Rust systems programming", ranked by relevance:**

```json
{
  "@context": {
    "ex": "http://example.org/"
  },
  "select": ["?title", "?score"],
  "where": [
    {
      "@id": "?doc",
      "@type": "ex:Article",
      "ex:content": "?content",
      "ex:title": "?title"
    },
    ["bind", "?score", "(fulltext ?content \"Rust systems programming\")"],
    ["filter", "(> ?score 0)"]
  ],
  "orderBy": [["desc", "?score"]],
  "limit": 10
}
```

Expected results (ordered by relevance):
1. "Rust for Systems Programming" -- highest score (most query terms, multiple occurrences)
2. "Introduction to Rust" -- mentions Rust and systems programming
3. "Database Design Patterns" -- excluded by `> 0` filter (no matching terms)

## SPARQL Support

### Inserting data

Fulltext annotation works in SPARQL UPDATE today using the `^^f:fullText` typed literal syntax (see the Turtle/SPARQL insertion examples above).

### Querying

The `fulltext()` scoring function is currently available in **JSON-LD Query only**. SPARQL query support is planned for a future release, with anticipated syntax like:

```sparql
PREFIX ex: <http://example.org/>
PREFIX f: <https://ns.flur.ee/db#>

SELECT ?title ?score
WHERE {
  ?doc a ex:Article ;
       ex:content ?content ;
       ex:title ?title .
  BIND(f:fulltext(?content, "Rust programming") AS ?score)
  FILTER(?score > 0)
}
ORDER BY DESC(?score)
LIMIT 10
```

This mirrors the pattern established by inline vector similarity functions (`dotProduct`, `cosineSimilarity`, `euclideanDistance`), which also support JSON-LD Query today with SPARQL planned.

## Related Documentation

- [Datatypes and Typed Values](../concepts/datatypes.md) -- All supported datatypes including `@fulltext`
- [Setting Groups](../ledger-config/setting-groups.md#full-text-defaults) -- Full reference for the `f:fullTextDefaults` schema (fields, additive merge, override control)
- [Override control](../ledger-config/override-control.md) -- Locking ledger-wide config against per-graph overrides
- [Reindex](reindex.md) -- When and how to reindex (required to pick up config changes for existing data)
- [JSON-LD Query](../query/jsonld-query.md) -- Full query language reference
- [BM25 Graph Source](bm25.md) -- Dedicated BM25 full-text search for large-scale corpora
- [Vector Search](vector-search.md) -- Inline similarity search with `@vector`
- [Background Indexing](background-indexing.md) -- How background indexing works


---

# indexing-and-search/bm25.md

# BM25 Full-Text Search

Fluree provides integrated full-text search using the BM25 (Best Matching 25) ranking algorithm. BM25 indexes are implemented as graph sources: they index text content from a source ledger and expose search results that can be joined with structured graph queries.

## What is BM25?

BM25 is a probabilistic ranking function that scores documents based on query term frequency and document length normalization. It's widely used in search engines and information retrieval systems.

**Key features:**
- Term frequency with saturation (controlled by k1)
- Inverse document frequency weighting
- Document length normalization (controlled by b)
- English stemming and stopword filtering (default analyzer)
- Block-Max WAND for efficient top-k queries (early termination)
- Incremental index updates
- Time-travel: query the index as of any past transaction

## Creating a BM25 Index

BM25 indexes are created via the Rust API using `Bm25CreateConfig`. There are no HTTP endpoints for index management yet — indexes are managed programmatically.

### Basic Index

```rust
use fluree_db_api::{Bm25CreateConfig, FlureeBuilder};
use serde_json::json;

let fluree = FlureeBuilder::file("/path/to/data").build()?;

// Create a ledger and insert some data
let ledger = fluree.create_ledger("docs:main").await?;
let tx = json!({
    "@context": { "ex": "http://example.org/" },
    "@graph": [
        { "@id": "ex:doc1", "@type": "ex:Article", "ex:title": "Rust programming guide" },
        { "@id": "ex:doc2", "@type": "ex:Article", "ex:title": "Python for beginners" },
        { "@id": "ex:doc3", "@type": "ex:Article", "ex:title": "Systems programming in Rust" }
    ]
});
let ledger = fluree.insert(ledger, &tx).await?.ledger;

// Define the indexing query
let query = json!({
    "@context": { "ex": "http://example.org/" },
    "where": [{ "@id": "?x", "@type": "ex:Article", "ex:title": "?title" }],
    "select": { "?x": ["@id", "ex:title"] }
});

// Create the BM25 index
let config = Bm25CreateConfig::new("article-search", "docs:main", query);
let result = fluree.create_full_text_index(config).await?;

println!("Indexed {} documents", result.doc_count);
println!("Graph source: {}", result.graph_source_id); // "article-search:main"
```

The graph source ID is `{name}:{branch}` — for example, `article-search:main`.

### Indexing Query

The indexing query defines **what** to index. It's a standard Fluree JSON-LD query with these requirements:

- Must include `@id` in the select (to identify documents)
- Must use `select` with a map form: `{"?x": ["@id", "ex:prop1", "ex:prop2"]}`
- All selected text properties are extracted and tokenized for search

The query can filter by type, filter by property values, or use any valid Fluree where clause:

```json
{
    "@context": { "ex": "http://example.org/" },
    "where": [
        { "@id": "?x", "@type": "ex:Article", "ex:title": "?title" },
        { "@id": "?x", "ex:status": "published" }
    ],
    "select": { "?x": ["@id", "ex:title", "ex:content", "ex:tags"] }
}
```

### Configuration Options

| Parameter | Default | Description |
|-----------|---------|-------------|
| `name` | (required) | Graph source name. Cannot contain `:`. |
| `ledger` | (required) | Source ledger alias (e.g., `"docs:main"`) |
| `query` | (required) | Indexing query (JSON-LD, must have `select`) |
| `branch` | `"main"` | Branch name for the graph source |
| `k1` | `1.2` | Term frequency saturation. Higher = more weight to term frequency. Must be > 0. Typical range: 1.2-2.0. |
| `b` | `0.75` | Document length normalization. 0 = no normalization, 1 = full normalization. Must be 0.0-1.0. |

```rust
let config = Bm25CreateConfig::new("search", "docs:main", query)
    .with_branch("dev")
    .with_k1(1.5)
    .with_b(0.5);
```

### Text Analysis

Fluree uses a default English analyzer that applies:

1. **Tokenization**: Unicode-aware word boundary splitting
2. **Lowercasing**: All tokens converted to lowercase
3. **Stopword filtering**: Common English words removed (the, a, an, is, etc.)
4. **Stemming**: Snowball English stemmer reduces words to root forms (e.g., "programming" -> "program")

The analyzer is not configurable — it always uses the English pipeline for consistency.

## Querying BM25 Indexes

### JSON-LD Query Syntax

BM25 search is integrated into Fluree's query system via the `f:` namespace predicates:

```json
{
    "@context": {
        "ex": "http://example.org/",
        "f": "https://ns.flur.ee/db#"
    },
    "from": "docs:main",
    "where": [
        {
            "f:graphSource": "article-search:main",
            "f:searchText": "rust programming",
            "f:searchLimit": 10,
            "f:searchResult": {
                "f:resultId": "?doc",
                "f:resultScore": "?score"
            }
        },
        { "@id": "?doc", "ex:author": "?author" }
    ],
    "select": ["?doc", "?score", "?author"]
}
```

**Pattern fields:**

| Field | Description |
|-------|-------------|
| `f:graphSource` | Graph source ID (e.g., `"article-search:main"`) |
| `f:searchText` | Query text (analyzed with same pipeline as indexing) |
| `f:searchLimit` | Maximum number of search results |
| `f:searchResult` | Binding object for results |
| `f:resultId` | Variable binding for the document IRI |
| `f:resultScore` | Variable binding for the BM25 relevance score |
| `f:resultLedger` | (Optional) Variable binding for ledger provenance |

### Combining Search with Structured Queries

The search pattern produces `?doc` and `?score` bindings. These can be joined with ledger data using normal where clauses:

```json
{
    "@context": {
        "ex": "http://example.org/",
        "f": "https://ns.flur.ee/db#"
    },
    "from": "docs:main",
    "where": [
        {
            "f:graphSource": "article-search:main",
            "f:searchText": "rust",
            "f:searchLimit": 20,
            "f:searchResult": { "f:resultId": "?doc", "f:resultScore": "?score" }
        },
        { "@id": "?doc", "ex:title": "?title" },
        { "@id": "?doc", "ex:author": "?author" }
    ],
    "select": ["?doc", "?title", "?author", "?score"]
}
```

The BM25 search runs first and produces candidate bindings. The subsequent where clauses join those candidates with the source ledger to retrieve additional properties.

### Rust API: Direct Search

You can also use the Rust API directly for programmatic search without the query engine:

```rust
use fluree_db_query::bm25::{Analyzer, Bm25Scorer};

// Load the index
let index = fluree.load_bm25_index("article-search:main").await?;

// Analyze query terms (same pipeline as indexing)
let analyzer = Analyzer::english_default();
let terms = analyzer.analyze_to_strings("rust programming");
let term_refs: Vec<&str> = terms.iter().map(|s| s.as_str()).collect();

// Score and rank
let scorer = Bm25Scorer::new(&index, &term_refs);
let results = scorer.top_k(10);

for (doc_key, score) in &results {
    println!("{}: {:.2}", doc_key.subject_iri, score);
}
```

### Rust API: Query with BM25

Use `query_connection_with_bm25` for integrated queries:

```rust
let query = json!({
    "@context": { "ex": "http://example.org/", "f": "https://ns.flur.ee/db#" },
    "from": "docs:main",
    "where": [
        {
            "f:graphSource": "article-search:main",
            "f:searchText": "rust",
            "f:searchLimit": 10,
            "f:searchResult": { "f:resultId": "?doc", "f:resultScore": "?score" }
        },
        { "@id": "?doc", "ex:author": "?author" }
    ],
    "select": ["?doc", "?score", "?author"]
});

let result = fluree.query_connection_with_bm25(&query).await?;
```

## Index Maintenance

### Syncing

BM25 indexes are **not** automatically updated when the source ledger changes. You must explicitly sync them:

```rust
// Incremental sync (detects changes since last watermark)
let sync_result = fluree.sync_bm25_index("article-search:main").await?;
println!("Upserted: {}, Removed: {}", sync_result.upserted, sync_result.removed);

// Force full resync (rebuilds the entire index)
let sync_result = fluree.resync_bm25_index("article-search:main").await?;
```

Incremental sync uses property dependency tracking to identify which subjects changed since the last indexed commit. Only affected documents are re-queried and re-indexed. If no affected subjects are detected, it falls back to a full resync.

### Background Maintenance Worker

For production use, the `Bm25MaintenanceWorker` can be configured to automatically sync indexes when source ledgers change:

- Watches for commit events on source ledgers
- Debounces rapid commits (configurable interval)
- Bounded concurrency for concurrent sync operations
- Registers/unregisters graph sources dynamically

### Staleness Checking

Check whether an index is behind its source ledger:

```rust
let check = fluree.check_bm25_staleness("article-search:main").await?;
println!("Index at t={}, ledger at t={}, stale: {}, lag: {}",
    check.index_t, check.ledger_t, check.is_stale, check.lag);
```

### Time-Travel

Load an index at a specific historical transaction time:

```rust
// Load index as of transaction t=5
let (index, actual_t) = fluree.load_bm25_index_at("article-search:main", 5).await?;
println!("Loaded snapshot at t={}, docs: {}", actual_t, index.num_docs());
```

BM25 maintains a manifest of historical snapshots. The manifest is stored in content-addressed storage and tracks all snapshot versions. `load_bm25_index_at` selects the snapshot with the largest `index_t <= as_of_t`.

### Dropping an Index

```rust
let drop_result = fluree.drop_full_text_index("article-search:main").await?;
println!("Deleted {} snapshots", drop_result.deleted_snapshots);

// Drop is idempotent
let drop_again = fluree.drop_full_text_index("article-search:main").await?;
assert!(drop_again.was_already_retracted);
```

Dropping marks the graph source as retracted in the nameservice and deletes all snapshot blobs from storage. The index can be recreated with the same name afterward.

## Scoring and Top-K Optimization

For top-k queries (the typical case via `f:searchLimit`), BM25 uses **Block-Max WAND** (Weak AND) to avoid scoring every matching document. Posting lists are divided into fixed-size blocks (128 postings each) with per-block metadata (maximum term frequency). WAND uses these to compute score upper bounds, skipping entire blocks that cannot contribute to the current top-k results.

This makes `top_k(10)` on a 100K-document index significantly faster than scoring all matches — the algorithm terminates early once it can prove no remaining document can displace the current top results.

When block metadata is unavailable (e.g., during index building before the first snapshot), scoring falls back to dense accumulation over all postings.

## Storage Format

### V4 Chunked Format

Large BM25 indexes use a chunked storage format (v4) that splits the index into:

- **Root blob**: Terms dictionary, document metadata, BM25 statistics, routing table
- **Posting leaflet blobs**: Compressed posting lists (~2MB each), stored as separate content-addressed objects. Each posting list includes block metadata (128 postings per block with `max_doc_id` and `max_tf`) used for WAND score upper bounds and block-level navigation.

This enables **selective loading**: queries only fetch the leaflets containing terms that match the search query, rather than loading the entire index.

### Leaflet Caching

Posting leaflets are cached in the global `LeafletCache` (shared with core index leaflets). Cache entries are keyed by content ID hash and are immutable (content-addressed data never changes). The cache uses moka's TinyLFU eviction and is governed by the global cache budget (`--cache-max-mb` / `FLUREE_CACHE_MAX_MB`, default tier: <4GB 30%, 4-8GB 40%, >=8GB 35%).

### Parallel I/O

Both reads and writes use bounded-concurrency parallel I/O (`buffer_unordered(32)`) for leaflet operations. This caps socket pressure when working with object stores like S3 while still providing significant throughput improvement over sequential access.

### Format Selection

The storage format is selected automatically based on the storage backend:
- **File storage**: V3 single-blob format (optimized for local filesystem)
- **Memory / S3 / object store**: V4 chunked format (enables selective loading and caching)

## Deployment Modes

### Embedded Mode (Default)

In embedded mode, the BM25 index is loaded and searched within the same process as Fluree. This is the default behavior.

### Remote Mode

In remote mode, search queries are delegated to a dedicated search service (`fluree-search-httpd`):

```bash
fluree-search-httpd \
  --storage-root file:///var/fluree/data \
  --nameservice-path file:///var/fluree/ns \
  --listen 0.0.0.0:9090
```

Both modes use identical analyzer configuration, BM25 scoring algorithm, and time-travel semantics — queries return identical results regardless of deployment mode.

See [BM25 Graph Source](../graph-sources/bm25.md) for details on the remote search protocol.

## Related Documentation

- [BM25 Graph Source](../graph-sources/bm25.md) - Graph source integration and remote search protocol
- [Background Indexing](background-indexing.md) - Core index architecture
- [Vector Search](vector-search.md) - Similarity search
- [Graph Sources Overview](../graph-sources/overview.md) - Graph source concepts


---

# indexing-and-search/vector-search.md

# Vector Search

Vector search enables similarity search using embedding vectors, supporting use cases like:

- **Semantic search**: Find similar meanings, not just keywords
- **Recommendations**: Find similar products, content, users
- **Image search**: Find similar images by visual features
- **Anomaly detection**: Find unusual patterns

Fluree supports two complementary approaches:

1. **Inline similarity functions** -- compute `dotProduct`, `cosineSimilarity`, or `euclideanDistance` directly in queries using `bind`. No external index required.
2. **HNSW vector indexes** -- build dedicated approximate-nearest-neighbor (ANN) indexes for large-scale similarity search using the `f:*` query pattern.

## The `@vector` Datatype

### Why a dedicated datatype?

In RDF, a plain JSON array like `[0.5, 0.5, 0.0]` is decomposed into individual values. Duplicate elements can be deduplicated, and ordering is not guaranteed. This breaks embedding vectors. The `@vector` datatype tells Fluree to store the array as a single, ordered, fixed-length vector.

`@vector` is a shorthand for the full IRI `https://ns.flur.ee/db#embeddingVector`, which can also be written as `f:embeddingVector` when the Fluree namespace prefix is declared in your `@context`.

### Storage: f32 precision contract

All `@vector` values are stored as **IEEE-754 binary32 (f32)** arrays. This means:

- Each element in your JSON array is quantized to f32 at ingest time
- Values that are not representable as finite f32 (NaN, Infinity, values exceeding f32 range) are rejected
- Round-trip reads return the f32-quantized values (e.g., `0.1` in JSON becomes `0.10000000149011612` after f32 quantization)
- This provides a compact, cache-friendly representation optimized for SIMD similarity computation

If you need higher precision (f64) or different vector formats (sparse, integer), store them as a custom RDF datatype string.

### Inserting vectors (JSON-LD)

Use `"@type": "@vector"` to annotate a numeric array as a vector:

```json
{
  "@context": {
    "ex": "http://example.org/"
  },
  "@graph": [
    {
      "@id": "ex:doc1",
      "@type": "ex:Document",
      "ex:embedding": {
        "@value": [0.1, 0.2, 0.3, 0.4],
        "@type": "@vector"
      }
    }
  ]
}
```

You can also use the full IRI or the `f:` prefix form, which is equivalent:

```json
{
  "@context": {
    "ex": "http://example.org/",
    "f": "https://ns.flur.ee/db#"
  },
  "@graph": [
    {
      "@id": "ex:doc1",
      "ex:embedding": {
        "@value": [0.1, 0.2, 0.3, 0.4],
        "@type": "f:embeddingVector"
      }
    }
  ]
}
```

**Incorrect -- plain array (will not work for similarity):**

```json
{
  "@id": "ex:doc1",
  "ex:embedding": [0.1, 0.2, 0.3, 0.4]
}
```

Plain arrays are decomposed into individual RDF values where duplicates may be removed and order is lost.

### Inserting vectors (Turtle / SPARQL UPDATE)

In Turtle and SPARQL UPDATE, the `@vector` shorthand is not available. Use the `f:embeddingVector` datatype IRI with the standard `^^` typed-literal syntax:

```sparql
PREFIX ex: <http://example.org/>
PREFIX f: <https://ns.flur.ee/db#>

INSERT DATA {
  ex:doc1 ex:embedding "[0.1, 0.2, 0.3, 0.4]"^^f:embeddingVector .
}
```

The vector is represented as a JSON array string with the `^^f:embeddingVector` datatype annotation.

### Multiple vectors per entity

An entity can have multiple vectors on the same property:

```json
{
  "@id": "ex:doc1",
  "ex:embedding": [
    {"@value": [0.1, 0.9], "@type": "@vector"},
    {"@value": [0.2, 0.8], "@type": "@vector"}
  ]
}
```

Each vector produces separate rows in query results.

### Vector literals in query VALUES clauses

When passing a vector literal in a query `values` clause, use the full IRI or the `f:` prefix form -- the `@vector` shorthand is only resolved in the transaction parser:

```json
"values": [
  ["?queryVec"],
  [{"@value": [0.7, 0.6], "@type": "f:embeddingVector"}]
]
```

Or with the full IRI:

```json
"values": [
  ["?queryVec"],
  [{"@value": [0.7, 0.6], "@type": "https://ns.flur.ee/db#embeddingVector"}]
]
```

## Inline Similarity Functions (JSON-LD Query)

Fluree provides three vector similarity functions that can be used in `bind` expressions within JSON-LD queries. These compute similarity scores directly during query execution without requiring a pre-built index.

Function names are case-insensitive; `dotProduct`, `dotproduct`, and `dot_product` are all equivalent.

### dotProduct

Computes the dot product (inner product) of two vectors. Higher scores indicate greater similarity when vectors represent aligned directions.

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "f": "https://ns.flur.ee/db#"
  },
  "select": ["?doc", "?score"],
  "values": [
    ["?queryVec"],
    [{"@value": [0.7, 0.6], "@type": "f:embeddingVector"}]
  ],
  "where": [
    {"@id": "?doc", "ex:embedding": "?vec"},
    ["bind", "?score", "(dotProduct ?vec ?queryVec)"]
  ],
  "orderBy": [["desc", "?score"]],
  "limit": 10
}
```

**Score range**: (-inf, +inf). Best when vector magnitude encodes importance.

### cosineSimilarity

Computes the cosine of the angle between two vectors. Ignores magnitude, focusing purely on directional similarity.

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "f": "https://ns.flur.ee/db#"
  },
  "select": ["?doc", "?score"],
  "values": [
    ["?queryVec"],
    [{"@value": [0.7, 0.6], "@type": "f:embeddingVector"}]
  ],
  "where": [
    {"@id": "?doc", "ex:embedding": "?vec"},
    ["bind", "?score", "(cosineSimilarity ?vec ?queryVec)"]
  ],
  "orderBy": [["desc", "?score"]],
  "limit": 10
}
```

**Score range**: [-1, 1] (1 = identical direction, 0 = orthogonal, -1 = opposite). Returns `null` if either vector has zero magnitude. Best for text embeddings and normalized vectors.

### euclideanDistance

Computes the L2 (straight-line) distance between two vectors. Lower scores indicate greater similarity.

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "f": "https://ns.flur.ee/db#"
  },
  "select": ["?doc", "?distance"],
  "values": [
    ["?queryVec"],
    [{"@value": [0.7, 0.6], "@type": "f:embeddingVector"}]
  ],
  "where": [
    {"@id": "?doc", "ex:embedding": "?vec"},
    ["bind", "?distance", "(euclideanDistance ?vec ?queryVec)"]
  ],
  "orderBy": "?distance",
  "limit": 10
}
```

**Score range**: [0, +inf) (0 = identical). Best for geometric similarity and when absolute position matters.

### Alternative array syntax

The similarity functions also accept array form instead of the S-expression string:

```json
["bind", "?score", ["dotProduct", "?vec", "?queryVec"]]
```

This is equivalent to:

```json
["bind", "?score", "(dotProduct ?vec ?queryVec)"]
```

### Filtering by score threshold

Combine `bind` with `filter` to return only results above a similarity threshold:

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "f": "https://ns.flur.ee/db#"
  },
  "select": ["?doc", "?score"],
  "values": [
    ["?queryVec"],
    [{"@value": [0.7, 0.6], "@type": "f:embeddingVector"}]
  ],
  "where": [
    {"@id": "?doc", "ex:embedding": "?vec"},
    ["bind", "?score", "(dotProduct ?vec ?queryVec)"],
    ["filter", "(> ?score 0.7)"]
  ]
}
```

### Combining with graph patterns

Vector similarity can be combined with standard graph patterns to filter by type, property values, or relationships:

```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "f": "https://ns.flur.ee/db#"
  },
  "select": ["?doc", "?title", "?score"],
  "values": [
    ["?queryVec"],
    [{"@value": [0.9, 0.1, 0.05], "@type": "f:embeddingVector"}]
  ],
  "where": [
    {"@id": "?doc", "@type": "ex:Article", "ex:title": "?title", "ex:embedding": "?vec"},
    ["bind", "?score", "(cosineSimilarity ?vec ?queryVec)"],
    ["filter", "(> ?score 0.5)"]
  ],
  "orderBy": [["desc", "?score"]],
  "limit": 5
}
```

### Using a stored vector as the query vector

Instead of providing a literal vector, you can use a stored entity's vector:

```json
{
  "@context": {
    "ex": "http://example.org/ns/"
  },
  "select": ["?similar", "?score"],
  "where": [
    {"@id": "ex:reference-doc", "ex:embedding": "?queryVec"},
    {"@id": "?similar", "ex:embedding": "?vec"},
    ["filter", "(!= ?similar ex:reference-doc)"],
    ["bind", "?score", "(cosineSimilarity ?vec ?queryVec)"]
  ],
  "orderBy": [["desc", "?score"]],
  "limit": 10
}
```

### Mixed datatypes

If a property contains both vector and non-vector values, the similarity functions return `null` for non-vector bindings:

```json
{
  "@graph": [
    {"@id": "ex:a", "ex:data": {"@value": [0.6, 0.5], "@type": "@vector"}},
    {"@id": "ex:b", "ex:data": "Not a vector"}
  ]
}
```

Querying with `dotProduct` on `?data` will return a numeric score for `ex:a` and `null` for `ex:b`.

### SPARQL support

Inline vector similarity functions (`dotProduct`, `cosineSimilarity`, `euclideanDistance`) are available in both JSON-LD Query and SPARQL. In SPARQL, use them as built-in function calls within `BIND` expressions:

#### dotProduct (SPARQL)

```sparql
PREFIX ex: <http://example.org/ns/>
PREFIX f: <https://ns.flur.ee/db#>

SELECT ?doc ?score
WHERE {
  VALUES ?queryVec { "[0.7, 0.6]"^^f:embeddingVector }
  ?doc ex:embedding ?vec ;
       ex:title ?title .
  BIND(dotProduct(?vec, ?queryVec) AS ?score)
}
ORDER BY DESC(?score)
LIMIT 10
```

#### cosineSimilarity (SPARQL)

```sparql
PREFIX ex: <http://example.org/ns/>
PREFIX f: <https://ns.flur.ee/db#>

SELECT ?doc ?score
WHERE {
  VALUES ?queryVec { "[0.88, 0.12, 0.08]"^^f:embeddingVector }
  ?doc a ex:Article ;
       ex:embedding ?vec ;
       ex:title ?title .
  BIND(cosineSimilarity(?vec, ?queryVec) AS ?score)
  FILTER(?score > 0.5)
}
ORDER BY DESC(?score)
LIMIT 5
```

#### euclideanDistance (SPARQL)

```sparql
PREFIX ex: <http://example.org/ns/>
PREFIX f: <https://ns.flur.ee/db#>

SELECT ?doc ?distance
WHERE {
  VALUES ?queryVec { "[0.7, 0.6]"^^f:embeddingVector }
  ?doc ex:embedding ?vec .
  BIND(euclideanDistance(?vec, ?queryVec) AS ?distance)
}
ORDER BY ?distance
LIMIT 10
```

#### Vector literals in SPARQL

In SPARQL, vectors are passed as JSON array strings with the `^^f:embeddingVector` typed literal syntax:

```sparql
VALUES ?queryVec { "[0.1, 0.2, 0.3]"^^f:embeddingVector }
```

Or with the full IRI:

```sparql
VALUES ?queryVec { "[0.1, 0.2, 0.3]"^^<https://ns.flur.ee/db#embeddingVector> }
```

#### Function name variants

Function names are case-insensitive in SPARQL. All of these are equivalent:

- `dotProduct`, `DOTPRODUCT`, `dot_product`
- `cosineSimilarity`, `COSINESIMILARITY`, `cosine_similarity`
- `euclideanDistance`, `EUCLIDEANDISTANCE`, `euclidean_distance`

## HNSW Vector Indexes

For large-scale similarity search, Fluree provides dedicated HNSW (Hierarchical Navigable Small World) vector indexes. These are approximate nearest-neighbor (ANN) indexes that trade exact results for dramatically faster query times on large datasets.

Vector indexes are implemented using embedded [usearch](https://github.com/unum-cloud/usearch) following the same architecture as BM25:

- Embedded in-process HNSW indexes (no external service required)
- Remote mode via dedicated search service (`fluree-search-httpd`)
- Snapshot-based persistence with watermarks
- Incremental sync for efficient updates
- Feature-gated via `vector` feature flag

**v1 limitation**: HNSW vector search is **head-only**. Time-travel queries (e.g. `@t:`) are not supported.

### Creating Vector Indexes

> **HTTP/Docker users:** there is no HTTP endpoint for creating vector indexes today. Index creation is Rust-API-only. To use HNSW vector search from an HTTP-only deployment, create the index using a Rust program (or the Rust API embedded in your application) against the same storage path your Fluree server reads, then run queries normally via `POST /v1/fluree/query`.

#### Rust API

```rust
use fluree_db_api::{FlureeBuilder, VectorCreateConfig};
use fluree_db_query::vector::DistanceMetric;

let fluree = FlureeBuilder::memory().build_memory();

// Create indexing query to select documents with embeddings
let indexing_query = json!({
    "@context": { "ex": "http://example.org/" },
    "where": [{ "@id": "?x", "@type": "ex:Document" }],
    "select": { "?x": ["@id", "ex:embedding"] }
});

// Create vector index
let config = VectorCreateConfig::new(
    "doc-embeddings",           // index name
    "mydb:main",                // source ledger
    indexing_query,             // what to index
    "ex:embedding",             // embedding property
    768                         // dimensions
)
.with_metric(DistanceMetric::Cosine);

let result = fluree.create_vector_index(config).await?;
println!("Indexed {} vectors", result.vector_count);
```

#### Configuration Options

| Option | Description | Default |
|--------|-------------|---------|
| `name` | Index name (creates graph source ID `name:branch`) | Required |
| `ledger` | Source ledger ID (`name:branch`) | Required |
| `query` | JSON-LD query selecting documents | Required |
| `embedding_property` | Property containing embeddings | Required |
| `dimensions` | Vector dimensions | Required |
| `metric` | Distance metric (Cosine, Dot, Euclidean) | Cosine |
| `connectivity` | HNSW M parameter | 16 |
| `expansion_add` | efConstruction parameter | 128 |
| `expansion_search` | efSearch parameter | 64 |

### Query Syntax

Vector index search uses the `f:*` pattern syntax in WHERE clauses:

```json
{
  "@context": {
    "ex": "http://example.org/",
    "f": "https://ns.flur.ee/db#"
  },
  "from": "mydb:main",
  "where": [
    {
      "f:graphSource": "doc-embeddings:main",
      "f:queryVector": [0.1, 0.2, 0.3],
      "f:distanceMetric": "cosine",
      "f:searchLimit": 10,
      "f:searchResult": {
        "f:resultId": "?doc",
        "f:resultScore": "?score"
      }
    }
  ],
  "select": ["?doc", "?score"]
}
```

#### Query Parameters

| Parameter | Description | Required |
|-----------|-------------|----------|
| `f:graphSource` | Vector index alias | Yes |
| `f:queryVector` | Query vector (array or variable) | Yes |
| `f:distanceMetric` | Distance metric ("cosine", "dot", "euclidean") | No (uses index default) |
| `f:searchLimit` | Maximum results | No |
| `f:searchResult` | Result binding (variable or object) | Yes |
| `f:syncBeforeQuery` | Wait for index sync before query | No (default: false) |
| `f:timeoutMs` | Query timeout in ms | No |

#### Result Binding

Simple variable binding:
```json
"f:searchResult": "?doc"
```

Structured binding with score and ledger:
```json
"f:searchResult": {
  "f:resultId": "?doc",
  "f:resultScore": "?similarity",
  "f:resultLedger": "?source"
}
```

#### Variable Query Vectors

Query vector can be a variable bound earlier:
```json
{
  "where": [
    { "@id": "ex:reference-doc", "ex:embedding": "?queryVec" },
    {
      "f:graphSource": "embeddings:main",
      "f:queryVector": "?queryVec",
      "f:searchLimit": 5,
      "f:searchResult": "?similar"
    }
  ]
}
```

### Index Maintenance

#### Sync Updates

After committing new data, sync the vector index:

```rust
let sync_result = fluree.sync_vector_index("doc-embeddings:main").await?;
println!("Upserted: {}, Removed: {}", sync_result.upserted, sync_result.removed);
```

#### Full Resync

Rebuild the entire index from scratch:

```rust
let resync_result = fluree.resync_vector_index("doc-embeddings:main").await?;
```

#### Check Staleness

```rust
let check = fluree.check_vector_staleness("doc-embeddings:main").await?;
if check.is_stale {
    println!("Index is {} commits behind", check.commits_behind);
}
```

#### Drop Index

```rust
fluree.drop_vector_index("doc-embeddings:main").await?;
```

## Distance Metrics

### Cosine (Default)

Measures angle between vectors. Best for:
- Text embeddings (e.g., sentence transformers)
- Normalized vectors
- When magnitude doesn't matter

Score range: [-1, 1] (1 = identical, 0 = orthogonal, -1 = opposite)

For unit-normalized vectors, cosine similarity equals dot product. Fluree's SIMD kernels exploit this for faster computation when vectors are pre-normalized.

### Dot Product

Measures alignment and magnitude. Best for:
- Maximum inner product search (MIPS)
- When vector magnitude encodes importance

Score range: (-inf, +inf)

### Euclidean (L2)

Measures straight-line distance. Best for:
- Geometric similarity
- Image feature vectors
- When absolute position matters

Raw score range: [0, +inf). In HNSW index results, normalized to (0, 1] via `1 / (1 + distance)`.

**Note**: In HNSW index results (`f:*` queries), all metrics are normalized to "higher is better". In inline similarity functions, `euclideanDistance` returns the raw L2 distance (lower = more similar).

## Deployment Modes

Vector indexes support two deployment topologies: searching in-process (embedded) or via a dedicated `fluree-search-httpd` service that mounts the same storage. Both topologies use identical distance-metric computation, score normalization, and snapshot serialization, so results are identical.

### Embedded Mode (Default)

The vector index is loaded and searched within the same process as the Fluree server. No additional services. This is the default and is appropriate for most deployments.

### Dedicated Search Service

For large indexes or when you want search traffic isolated from the main Fluree process, run the standalone `fluree-search-httpd` binary on the same storage volume and have your application send vector requests directly to it.

> **Note:** Today, vector search is invoked from a Fluree query (the `f:graphSource` / `f:queryVector` pattern) using the **embedded** path — the main Fluree server does not yet route those queries to a remote service. The dedicated service is reachable directly via its own `POST /v1/search` API (the same protocol BM25 uses), which is suitable for applications that issue vector queries outside of a Fluree query context. Transparent delegation from inside a Fluree query is a planned follow-up; the wiring is in place but the deployment config is not yet persisted by `create_vector_index`.

See [Remote Search Service](../graph-sources/bm25.md#remote-search-service) for `fluree-search-httpd` configuration, env vars, the request/response protocol (`vector` and `vector_similar_to` query kinds), and Docker deployment.

## Performance and Scaling

### The importance of binary indexing

Fluree's binary columnar index dramatically accelerates vector queries. Queries against novelty-only (unindexed) data perform a linear scan through the in-memory commit log, while indexed queries read pre-sorted, cache-friendly columnar data. **Ensure background indexing is running** for production workloads -- the difference is substantial.

The following benchmarks use 768-dimensional vectors (typical for transformer embeddings like sentence-transformers or OpenAI `text-embedding-3-small`) on Apple M-series hardware:

#### Novelty-only (no binary index)

| Scenario | Vectors | Query time | Throughput |
|----------|---------|-----------|------------|
| Scan all | 1,000 | 9.9 ms | ~101K vec/s |
| Scan all | 5,000 | 45.1 ms | ~111K vec/s |
| Filtered + score | 1,000 (75 pass filter) | 13.5 ms | ~5.5K vec/s |
| Filtered + score | 5,000 (402 pass filter) | 62.1 ms | ~6.5K vec/s |

#### With binary index

| Scenario | Vectors | Query time | Throughput | Speedup vs novelty |
|----------|---------|-----------|------------|-------------------|
| Scan all | 1,000 | 1.68 ms | ~595K vec/s | 5.9x |
| Scan all | 5,000 | 7.69 ms | ~650K vec/s | 5.9x |
| Filtered + score | 1,000 (75 pass filter) | 533 us | ~141K vec/s | 25x |
| Filtered + score | 5,000 (402 pass filter) | 2.40 ms | ~168K vec/s | 26x |

Key takeaways:

- **Unfiltered scans** are ~6x faster with the binary index
- **Filtered queries** (where graph patterns reduce the candidate set before scoring) are ~25x faster -- the index enables efficient predicate-first access that avoids loading irrelevant vectors entirely
- At 5,000 vectors, a filtered indexed query completes in **2.4 ms** -- well within interactive latency budgets

### Inline similarity functions (flat scan)

- **Best for**: Small to medium datasets, ad-hoc similarity queries, prototyping
- **Complexity**: O(n) linear scan -- computes similarity against every matching vector
- **Advantage**: No index setup required, works immediately after insert
- **SIMD acceleration**: Fluree uses runtime-detected SIMD kernels (SSE2/AVX on x86_64, NEON on ARM) for vectorized dot/cosine/L2 computation
- **Normalized embedding optimization**: For unit-normalized vectors (most transformer embeddings), cosine similarity reduces to a dot product, avoiding magnitude computation entirely

### When to consider HNSW

Inline similarity functions perform a brute-force scan over all candidate vectors. This scales linearly and remains fast for moderate datasets, but at larger scales an HNSW index provides O(log n) approximate nearest-neighbor search.

**Rule of thumb:**

| Vector count (per property) | Recommendation |
|----------------------------|----------------|
| < 100K | Flat scan works well, especially with binary indexing. Sub-100ms queries typical. |
| 100K -- 1M | **Start evaluating HNSW.** Flat scan may still be acceptable depending on latency target and hardware, but HNSW will provide more consistent low-latency results. |
| 1M -- 10M | HNSW strongly recommended for interactive latency. Flat scan can work if vectors are memory-resident and you can tolerate ~1-2 second queries. |
| > 10M | HNSW (or other ANN index) is the default recommendation. Flat scan becomes I/O- and cache-bound for low-latency use cases. |

Factors that shift the crossover:

- **Hardware**: Fast NVMe / large RAM pushes the threshold higher; object storage (S3) pulls it lower
- **Latency target**: A 50 ms budget favors HNSW earlier than a 2-second budget
- **Filter selectivity**: If graph patterns reduce candidates to a small fraction before scoring, flat scan remains viable at higher counts
- **Normalized embeddings**: Cosine-as-dot-product is faster, pushing the threshold higher
- **Binary indexing**: An indexed dataset scans ~6x faster than novelty-only, effectively raising the flat-scan ceiling

### HNSW vector indexes

- **Best for**: Large datasets (100K+ vectors), production similarity search with strict latency requirements
- **Complexity**: O(log n) approximate nearest neighbor
- **Space**: ~1.5x embedding size + IRI mapping overhead
- **Updates**: Incremental via affected-subject tracking

#### Tuning parameters

| Parameter | Effect | Trade-off |
|-----------|--------|-----------|
| `connectivity` (M) | Graph connectivity | Higher = better recall, more memory |
| `expansion_add` (efConstruction) | Build-time search width | Higher = better index quality, slower build |
| `expansion_search` (efSearch) | Query-time search width | Higher = better recall, slower queries |

## Feature Flag

The HNSW vector index functionality requires the `vector` feature:

```toml
[dependencies]
fluree-db-api = { version = "0.1", features = ["vector"] }
```

Inline similarity functions (`dotProduct`, `cosineSimilarity`, `euclideanDistance`) and the `@vector` datatype are available without feature flags.

## Complete Example: Semantic Search

**1. Insert documents with embeddings:**

```json
{
  "@context": {
    "ex": "http://example.org/",
    "f": "https://ns.flur.ee/db#"
  },
  "@graph": [
    {
      "@id": "ex:doc1",
      "@type": "ex:Article",
      "ex:title": "Introduction to Machine Learning",
      "ex:embedding": {"@value": [0.9, 0.1, 0.05], "@type": "@vector"}
    },
    {
      "@id": "ex:doc2",
      "@type": "ex:Article",
      "ex:title": "Database Design Patterns",
      "ex:embedding": {"@value": [0.1, 0.8, 0.1], "@type": "@vector"}
    },
    {
      "@id": "ex:doc3",
      "@type": "ex:Article",
      "ex:title": "Neural Network Architectures",
      "ex:embedding": {"@value": [0.85, 0.15, 0.1], "@type": "@vector"}
    }
  ]
}
```

**2. Query -- find articles similar to a "machine learning" embedding:**

```json
{
  "@context": {
    "ex": "http://example.org/",
    "f": "https://ns.flur.ee/db#"
  },
  "select": ["?title", "?score"],
  "values": [
    ["?queryVec"],
    [{"@value": [0.88, 0.12, 0.08], "@type": "f:embeddingVector"}]
  ],
  "where": [
    {"@id": "?doc", "@type": "ex:Article", "ex:title": "?title", "ex:embedding": "?vec"},
    ["bind", "?score", "(cosineSimilarity ?vec ?queryVec)"]
  ],
  "orderBy": [["desc", "?score"]],
  "limit": 5
}
```

Expected results (ordered by similarity):
1. "Introduction to Machine Learning" -- highest cosine similarity
2. "Neural Network Architectures" -- similar domain
3. "Database Design Patterns" -- different domain, lower score

## Related Documentation

- [Datatypes and Typed Values](../concepts/datatypes.md) - All supported datatypes including `@vector`
- [JSON-LD Query](../query/jsonld-query.md) - Full query language reference
- [BM25](bm25.md) - Full-text search
- [Background Indexing](background-indexing.md) - Core indexing
- [Graph Sources](../graph-sources/README.md) - Graph source concepts


---

# indexing-and-search/geospatial.md

# Geospatial Data

Fluree provides native support for geographic point data using the OGC GeoSPARQL standard. POINT geometries from `geo:wktLiteral` values are stored in an optimized binary format enabling efficient storage and index-accelerated proximity queries.

## Status

Geospatial support is implemented with:

- **Inline GeoPoint encoding**: POINT geometries stored as packed 60-bit lat/lng values
- **Automatic detection**: `geo:wktLiteral` POINT values automatically converted to native format
- **Full round-trip**: GeoPoints preserved through commit, index, and query paths
- **~0.3mm precision**: 30-bit encoding per coordinate provides sub-millimeter accuracy
- **Index-accelerated proximity queries**: POST latitude-band scans with haversine post-filtering
- **Time travel support**: Point-in-time geo queries via `from: "<ledger>@t:<t>"` (see examples below)

Non-POINT geometries (polygons, linestrings, multipolygons, etc.) are indexed using a separate S2 cell-based spatial index that enables efficient containment and intersection queries.

## Storing Geographic Data

### WKT Literal Format

Geographic data uses the Well-Known Text (WKT) format with the `geo:wktLiteral` datatype:

```json
{
  "@context": {
    "ex": "http://example.org/",
    "geo": "http://www.opengis.net/ont/geosparql#"
  },
  "@graph": [
    {
      "@id": "ex:eiffel-tower",
      "@type": "ex:Landmark",
      "ex:name": "Eiffel Tower",
      "ex:location": {
        "@value": "POINT(2.2945 48.8584)",
        "@type": "geo:wktLiteral"
      }
    }
  ]
}
```

**Important**: WKT uses `POINT(longitude latitude)` order (X, Y), which is the opposite of common lat/lng conventions.

### Coordinate Order

| Format | Order | Example |
|--------|-------|---------|
| WKT | longitude, latitude | `POINT(2.2945 48.8584)` |
| Common conventions | latitude, longitude | `48.8584, 2.2945` |

Fluree handles the conversion internally, storing coordinates in latitude-primary order for efficient latitude-band index scans.

### Valid POINT Syntax

Fluree recognizes these POINT formats:

```
POINT(2.2945 48.8584)           # Standard 2D point
POINT( 2.2945  48.8584 )        # Whitespace is flexible
POINT(-122.4194 37.7749)        # Negative coordinates (San Francisco)
```

The following are **not** supported for native GeoPoint storage (stored as strings instead):

```
POINT EMPTY                      # Empty point
POINT Z(2.2945 48.8584 100)     # 3D point with altitude
POINT M(2.2945 48.8584 1.0)     # Point with measure
POINT ZM(2.2945 48.8584 100 1)  # 3D point with measure
<http://...>POINT(...)          # SRID prefix
point(2.2945 48.8584)           # Lowercase (case-sensitive)
```

### Coordinate Validation

Coordinates must be within valid ranges:

- **Latitude**: -90.0 to 90.0 (degrees)
- **Longitude**: -180.0 to 180.0 (degrees)
- **Finite values only**: NaN and infinity are rejected

Invalid coordinates cause the value to be stored as a plain string rather than a native GeoPoint.

## Querying Geographic Data

### Basic Retrieval

GeoPoints are returned in WKT format in query results:

```json
{
  "@context": {
    "ex": "http://example.org/",
    "geo": "http://www.opengis.net/ont/geosparql#"
  },
  "from": "places:main",
  "where": [
    { "@id": "?place", "@type": "ex:Landmark" },
    { "@id": "?place", "ex:location": "?loc" }
  ],
  "select": ["?place", "?loc"]
}
```

Result:

```json
[
  ["ex:eiffel-tower", "POINT(2.2945 48.8584)"]
]
```

### SPARQL Queries

```sparql
PREFIX ex: <http://example.org/>
PREFIX geo: <http://www.opengis.net/ont/geosparql#>

SELECT ?place ?location
WHERE {
  ?place a ex:Landmark ;
         ex:location ?location .
}
```

### Output Formats

GeoPoints appear differently based on output format:

**JSON-LD (default):**
```json
{
  "@id": "ex:eiffel-tower",
  "ex:location": {
    "@value": "POINT(2.2945 48.8584)",
    "@type": "geo:wktLiteral"
  }
}
```

**SPARQL JSON:**
```json
{
  "type": "literal",
  "value": "POINT(2.2945 48.8584)",
  "datatype": "http://www.opengis.net/ont/geosparql#wktLiteral"
}
```

**Typed JSON:**
```json
{
  "@value": "POINT(2.2945 48.8584)",
  "@type": "geo:wktLiteral"
}
```

## Storage Encoding

### Binary Format

GeoPoints are stored using a compact 60-bit encoding:

- **Upper 30 bits**: Latitude scaled from [-90, 90] to [0, 2^30-1]
- **Lower 30 bits**: Longitude scaled from [-180, 180] to [0, 2^30-1]

This provides:

- **8 bytes total storage** per point (vs ~25+ bytes for WKT string)
- **~0.3mm precision** at the equator
- **Ordered encoding** enabling efficient range scans by latitude band

### Index Structure

GeoPoints use `ObjKind::GEO_POINT` (0x14) in the binary index:

| Component | Encoding |
|-----------|----------|
| Object kind | 1 byte (0x14) |
| Object key | 8 bytes (packed lat/lng) |

The latitude-primary encoding enables POST index scans that efficiently retrieve all points within a latitude band.

## Distance Queries

Fluree supports the `geof:distance` function (OGC GeoSPARQL) for calculating haversine distances between geographic points.

### geof:distance Function

Calculate the distance between two points in meters:

**JSON-LD Query (bind + filter):**
```json
{
  "@context": {
    "ex": "http://example.org/",
    "geo": "http://www.opengis.net/ont/geosparql#"
  },
  "from": "places:main",
  "where": [
    { "@id": "?place", "ex:location": "?loc" },
    { "@id": "ex:paris", "ex:location": "?parisLoc" },
    ["bind", "?distance", "(geof:distance ?loc ?parisLoc)"],
    ["filter", "(< ?distance 500000)"]
  ],
  "select": ["?place", "?distance"]
}
```

**SPARQL:**
```sparql
PREFIX ex: <http://example.org/>
PREFIX geo: <http://www.opengis.net/ont/geosparql#>
PREFIX geof: <http://www.opengis.net/def/function/geosparql/>

SELECT ?place ?distance
WHERE {
  ?place ex:location ?loc .
  ex:paris ex:location ?parisLoc .
  BIND(geof:distance(?loc, ?parisLoc) AS ?distance)
  FILTER(?distance < 500000)
}
ORDER BY ?distance
```

**Function aliases:** `geof:distance`, `geo_distance`, `geodistance`

**Arguments:**
- Two GeoPoint values (stored as `geo:wktLiteral` POINT)
- Or two WKT POINT strings

**Returns:** Distance in meters (Double)

**Calculation:** Uses the haversine formula with Earth's mean radius (6,371 km), accurate to within 0.3% for typical distances.

## Proximity Search

Fluree supports index-accelerated proximity queries that find points within a given distance of a center point.

### Index-Accelerated Point Proximity

Use a `geof:distance` bind + filter pattern to run an accelerated proximity search over inline GeoPoints. This pattern works identically in both JSON-LD and SPARQL queries — the query optimizer detects the Triple + Bind(geof:distance) + Filter combination and rewrites it into an index-accelerated scan.

**JSON-LD Query (find restaurants within 5km, include distance, limit to 10):**
```json
{
  "@context": {
    "ex": "http://example.org/",
    "geo": "http://www.opengis.net/ont/geosparql#"
  },
  "from": "places:main",
  "where": [
    { "@id": "?place", "@type": "ex:Restaurant" },
    { "@id": "?place", "ex:location": "?loc" },
    ["bind", "?distance", "(geof:distance ?loc \"POINT(2.35 48.85)\")"],
    ["filter", "(<= ?distance 5000)"]
  ],
  "select": ["?place", "?distance"],
  "orderBy": ["?distance"],
  "limit": 10
}
```

**SPARQL (same pattern, same acceleration):**
```sparql
PREFIX ex: <http://example.org/>
PREFIX geo: <http://www.opengis.net/ont/geosparql#>
PREFIX geof: <http://www.opengis.net/def/function/geosparql/>

SELECT ?station ?distance
WHERE {
  ?station a ex:GasStation ;
           ex:location ?loc .
  BIND(geof:distance(?loc, "POINT(2.35 48.85)"^^geo:wktLiteral) AS ?distance)
  FILTER(?distance < 10000)
}
ORDER BY ?distance
LIMIT 10
```

### How Index Acceleration Works

1. **Latitude-band scan**: The query planner converts the radius to latitude bounds and scans only points in `[lat - δ, lat + δ]`
2. **Haversine post-filter**: Results are filtered by exact haversine distance to eliminate false positives
3. **Distance sorting**: Results can be sorted by distance for k-nearest-neighbor queries

**Performance characteristics:**
- Uses POST index with latitude-primary encoding
- Scans only relevant latitude band (not full table scan)
- False positive rate: 22-70% depending on latitude and radius (eliminated by post-filter)
- Handles antimeridian crossing with multiple range scans

### Time Travel Support

Point proximity queries support time travel via the `from` ledger selector.

**JSON-LD with time travel:**
```json
{
  "@context": {
    "ex": "http://example.org/",
    "geo": "http://www.opengis.net/ont/geosparql#"
  },
  "from": "places:main@t:100",
  "where": [
    { "@id": "?place", "ex:location": "?loc" },
    ["bind", "?dist", "(geof:distance ?loc \"POINT(2.35 48.85)\")"],
    ["filter", "(<= ?dist 5000)"]
  ],
  "select": ["?place"]
}
```

**SPARQL with time travel:**
```sparql
PREFIX ex: <http://example.org/>
PREFIX fluree: <https://ns.flur.ee/ledger#>

SELECT ?place ?loc
FROM <ledger:places:main?t=100>
WHERE {
  ?place ex:location ?loc .
}
```

Time travel correctly handles:
- Points that existed at time `t` but were later retracted
- Points added after time `t` (excluded from results)
- Overlay novelty merging for recent uncommitted data

### Graph Scoping

Point proximity queries respect graph context. When used inside a GRAPH pattern, the query scans only the specified named graph:

```json
{
  "@context": {
    "ex": "http://example.org/",
    "geo": "http://www.opengis.net/ont/geosparql#"
  },
  "from": "world:main",
  "where": [
    ["graph", "http://example.org/france", [
      { "@id": "?city", "ex:location": "?loc" },
      ["bind", "?dist", "(geof:distance ?loc \"POINT(2.35 48.85)\")"],
      ["filter", "(<= ?dist 50000)"]
    ]]
  ],
  "select": ["?city"]
}
```

This returns only cities from the France graph within 50km of Paris, not cities from other named graphs.

## S2 Spatial Index (Complex Geometries)

Fluree provides an S2 cell-based spatial index for complex geometries (polygons, linestrings, multipolygons). This index enables efficient spatial predicate queries like "find all places within this region" or "find all regions that contain this point."

### Supported Operations

| Operation | Description | Use Case |
|-----------|-------------|----------|
| `within` | Find geometries that are completely inside a query geometry | "Find all buildings within this city boundary" |
| `contains` | Find geometries that completely contain a query geometry | "Find the district that contains this point" |
| `intersects` | Find geometries that overlap with a query geometry | "Find all parcels that touch this proposed road" |
| `nearby` | Find geometries within a radius (with distances) | "Find polygons within 10km of this point" |

### Query Syntax

**JSON-LD Query (find places within a polygon):**
```json
{
  "@context": {
    "ex": "http://example.org/",
    "geo": "http://www.opengis.net/ont/geosparql#",
    "idx": "https://ns.flur.ee/index#"
  },
  "from": "places:main",
  "where": [
    {
      "idx:spatial": "within",
      "idx:property": "ex:boundary",
      "idx:geometry": "POLYGON((2.0 48.0, 3.0 48.0, 3.0 49.0, 2.0 49.0, 2.0 48.0))",
      "idx:result": "?place"
    }
  ],
  "select": ["?place"]
}
```

**Find regions containing a point:**
```json
{
  "where": [
    {
      "idx:spatial": "contains",
      "idx:property": "ex:boundary",
      "idx:geometry": "POINT(2.35 48.85)",
      "idx:result": "?district"
    }
  ],
  "select": ["?district"]
}
```

**Find intersecting parcels:**
```json
{
  "where": [
    {
      "idx:spatial": "intersects",
      "idx:property": "ex:parcel",
      "idx:geometry": "LINESTRING(2.0 48.0, 3.0 49.0)",
      "idx:result": "?parcel"
    }
  ],
  "select": ["?parcel"]
}
```

**Find polygons near a point (with distances):**
```json
{
  "@context": {
    "ex": "http://example.org/",
    "idx": "https://ns.flur.ee/index#"
  },
  "from": "places:main",
  "where": [
    {
      "idx:spatial": "nearby",
      "idx:property": "ex:boundary",
      "idx:geometry": "POINT(2.35 48.85)",
      "idx:radius": 10000,
      "idx:result": {
        "idx:id": "?region",
        "idx:distance": "?dist"
      }
    }
  ],
  "select": ["?region", "?dist"],
  "orderBy": ["?dist"]
}
```

### How It Works

The S2 spatial index uses Google's S2 geometry library to map geometries to hierarchical cells on a sphere:

1. **Ingestion**: When a `geo:wktLiteral` polygon/linestring is committed, the indexer generates an S2 cell covering and stores cell entries in the spatial index.

2. **Query**: When you query with a spatial predicate, the system:
   - Generates an S2 covering for your query geometry
   - Scans the index for matching cell ranges
   - Applies bounding-box prefiltering
   - Performs exact geometry tests on candidates

3. **Time-Travel**: The index supports full time-travel semantics, so you can query spatial data at any historical point in time.

### Index Configuration

The S2 index is automatically created for predicates with `geo:wktLiteral` values. Configuration options:

| Parameter | Default | Description |
|-----------|---------|-------------|
| `min_level` | 4 | Minimum S2 cell level (coarser = faster build) |
| `max_level` | 16 | Maximum S2 cell level (finer = tighter coverage) |
| `max_cells` | 8 | Maximum cells per geometry covering |

Higher `max_cells` values produce tighter coverings (fewer false positives) but increase index size and build time.

### Performance Characteristics
Performance depends on data distribution, covering configuration, and result selectivity. See [Spatial Index Design](../design/spatial-index.md) for design rationale; a benchmark suite is recommended for deployment-specific measurements.

### Supported Geometry Types

| Geometry Type | S2 Index | Notes |
|---------------|----------|-------|
| POLYGON | ✅ Yes | Most common for region queries |
| MULTIPOLYGON | ✅ Yes | Multiple disjoint regions |
| LINESTRING | ✅ Yes | Routes, boundaries |
| MULTILINESTRING | ✅ Yes | Multiple line segments |
| POINT | ⚠️ Optional | Use inline GeoPoint for proximity; S2 available with `index_points=true` |
| GEOMETRYCOLLECTION | ✅ Yes | Mixed geometry types |

### Graph Scoping

Spatial indexes are scoped by named graph. Each graph has its own spatial index, and queries automatically use the correct index based on the graph context.

**Default graph query:**
```json
{
  "from": "mydb:main",
  "where": [
    {
      "idx:spatial": "within",
      "idx:property": "ex:boundary",
      "idx:geometry": "POLYGON(...)",
      "idx:result": "?region"
    }
  ]
}
```

**Named graph query (using GRAPH pattern):**
```json
{
  "from": "mydb:main",
  "where": [
    ["graph", "http://example.org/regions",
     {
       "idx:spatial": "within",
       "idx:property": "ex:boundary",
       "idx:geometry": "POLYGON(...)",
       "idx:result": "?region"
     }
    ]
  ]
}
```

When you enter a GRAPH pattern, the spatial query automatically switches to that graph's index. This ensures results are correctly scoped—a spatial query inside `GRAPH <http://example.org/france>` only searches geometries in the France graph, not geometries from other named graphs.

**Multiple named graphs:**

If you have data across multiple named graphs (e.g., countries), you can query each independently:

```json
{
  "from": "world:main",
  "where": [
    ["graph", "http://example.org/germany",
     {
       "idx:spatial": "within",
       "idx:property": "ex:boundary",
       "idx:geometry": "POLYGON(...)",
       "idx:result": "?germanCity"
     }
    ]
  ]
}
```

The same `idx:property` (e.g., `ex:boundary`) in different named graphs will query separate spatial indexes.

### Time-Travel Support

Spatial queries support time travel via the `from` ledger selector:

```json
{
  "from": "places:main@t:100",
  "where": [
    {
      "idx:spatial": "within",
      "idx:property": "ex:boundary",
      "idx:geometry": "POLYGON(...)",
      "idx:result": "?place"
    }
  ],
  "select": ["?place"]
}
```

This returns places as they existed at transaction time 100, correctly handling:
- Geometries added after t=100 (excluded)
- Geometries retracted before t=100 (excluded)
- Geometries modified between t=100 and now

**Note**: Time travel requires `t >= index.base_t`. Queries for times before the index was built will return an error.

**Note (v1)**: The historical-view API (`query_historical`) does not execute spatial index patterns. Use a time-pinned `from` selector (as above) against the current ledger state for spatial time travel.

## Choosing Between Point Proximity and S2 Spatial Queries

Fluree provides two spatial query paths. Use this guide to pick the right one:

| Use Case | Approach | Reason |
|----------|----------|--------|
| "Find restaurants near me" | `geof:distance` bind+filter | POINT proximity with distance ranking |
| "Find cities within 100km" | `geof:distance` bind+filter | POINT data with radius filter |
| "Find buildings in this district" | `idx:spatial` (within) | POLYGONs inside a boundary |
| "Which zone contains this address?" | `idx:spatial` (contains) | POLYGON containment test |
| "Find parcels crossing this road" | `idx:spatial` (intersects) | LINESTRING intersection |
| "Find regions near this location" | `idx:spatial` (nearby) | POLYGONs with distance from point |

**Quick rule**: Use `geof:distance` bind+filter for POINT locations with radius queries. Use `idx:spatial` for polygon/linestring containment, intersection, or region-based queries.

### End-to-End Example: Points and Polygons

This example shows storing both POINT locations and POLYGON boundaries, then querying each appropriately.

**1. Insert data with both geometry types:**
```json
{
  "@context": {
    "ex": "http://example.org/",
    "geo": "http://www.opengis.net/ont/geosparql#"
  },
  "@graph": [
    {
      "@id": "ex:central-paris",
      "@type": "ex:District",
      "ex:name": "Central Paris",
      "ex:boundary": {
        "@value": "POLYGON((2.3 48.8, 2.4 48.8, 2.4 48.9, 2.3 48.9, 2.3 48.8))",
        "@type": "geo:wktLiteral"
      }
    },
    {
      "@id": "ex:eiffel-tower",
      "@type": "ex:Landmark",
      "ex:name": "Eiffel Tower",
      "ex:location": {
        "@value": "POINT(2.2945 48.8584)",
        "@type": "geo:wktLiteral"
      }
    },
    {
      "@id": "ex:louvre",
      "@type": "ex:Landmark",
      "ex:name": "Louvre Museum",
      "ex:location": {
        "@value": "POINT(2.3376 48.8606)",
        "@type": "geo:wktLiteral"
      }
    }
  ]
}
```

**2. Find landmarks near Eiffel Tower (POINT proximity):**
```json
{
  "@context": {
    "ex": "http://example.org/",
    "geo": "http://www.opengis.net/ont/geosparql#"
  },
  "from": "places:main",
  "where": [
    { "@id": "?place", "ex:location": "?loc" },
    ["bind", "?dist", "(geof:distance ?loc \"POINT(2.2945 48.8584)\")"],
    ["filter", "(<= ?dist 5000)"],
    { "@id": "?place", "ex:name": "?name" }
  ],
  "select": ["?name", "?dist"],
  "orderBy": ["?dist"]
}
```

**3. Find which district contains the Louvre (POLYGON containment):**
```json
{
  "@context": {
    "ex": "http://example.org/",
    "idx": "https://ns.flur.ee/index#"
  },
  "from": "places:main",
  "where": [
    {
      "idx:spatial": "contains",
      "idx:property": "ex:boundary",
      "idx:geometry": "POINT(2.3376 48.8606)",
      "idx:result": "?district"
    },
    { "@id": "?district", "ex:name": "?name" }
  ],
  "select": ["?name"]
}
```

### MULTIPOLYGON Example

Store regions with multiple disjoint areas (e.g., archipelagos, non-contiguous territories):

```json
{
  "@context": {
    "ex": "http://example.org/",
    "geo": "http://www.opengis.net/ont/geosparql#"
  },
  "@id": "ex:hawaii",
  "@type": "ex:State",
  "ex:name": "Hawaii",
  "ex:territory": {
    "@value": "MULTIPOLYGON(((-160 22, -159 22, -159 21, -160 21, -160 22)), ((-156 20, -155 20, -155 19, -156 19, -156 20)))",
    "@type": "geo:wktLiteral"
  }
}
```

Query: "Find states that contain this coordinate"
```json
{
  "where": [
    {
      "idx:spatial": "contains",
      "idx:property": "ex:territory",
      "idx:geometry": "POINT(-155.5 19.5)",
      "idx:result": "?state"
    }
  ],
  "select": ["?state"]
}
```

### LINESTRING Example

Store routes, roads, or boundaries:

```json
{
  "@context": {
    "ex": "http://example.org/",
    "geo": "http://www.opengis.net/ont/geosparql#"
  },
  "@id": "ex:route-66",
  "@type": "ex:Highway",
  "ex:name": "Route 66",
  "ex:path": {
    "@value": "LINESTRING(-118.2 34.1, -112.0 35.2, -106.6 35.1, -97.5 35.5, -90.2 38.6, -87.6 41.9)",
    "@type": "geo:wktLiteral"
  }
}
```

Query: "Find highways that cross this region"
```json
{
  "where": [
    {
      "idx:spatial": "intersects",
      "idx:property": "ex:path",
      "idx:geometry": "POLYGON((-100 34, -95 34, -95 37, -100 37, -100 34))",
      "idx:result": "?highway"
    }
  ],
  "select": ["?highway"]
}
```

## Planned Capabilities

### R-tree Index

An ephemeral R-tree is planned for:
- Spatial joins between datasets
- Range queries across multiple properties

## Examples

### Storing Multiple Locations

```json
{
  "@context": {
    "ex": "http://example.org/",
    "geo": "http://www.opengis.net/ont/geosparql#"
  },
  "@graph": [
    {
      "@id": "ex:paris",
      "@type": "ex:City",
      "ex:name": "Paris",
      "ex:center": { "@value": "POINT(2.3522 48.8566)", "@type": "geo:wktLiteral" }
    },
    {
      "@id": "ex:london",
      "@type": "ex:City",
      "ex:name": "London",
      "ex:center": { "@value": "POINT(-0.1278 51.5074)", "@type": "geo:wktLiteral" }
    },
    {
      "@id": "ex:tokyo",
      "@type": "ex:City",
      "ex:name": "Tokyo",
      "ex:center": { "@value": "POINT(139.6917 35.6895)", "@type": "geo:wktLiteral" }
    }
  ]
}
```

### Turtle Format

```turtle
@prefix ex: <http://example.org/> .
@prefix geo: <http://www.opengis.net/ont/geosparql#> .

ex:sensor-1 a ex:WeatherStation ;
    ex:name "Central Park Station" ;
    ex:location "POINT(-73.9654 40.7829)"^^geo:wktLiteral .

ex:sensor-2 a ex:WeatherStation ;
    ex:name "Times Square Station" ;
    ex:location "POINT(-73.9855 40.7580)"^^geo:wktLiteral .
```

### Mixed Geometry Types

Non-POINT geometries are stored as strings:

```json
{
  "@context": {
    "ex": "http://example.org/",
    "geo": "http://www.opengis.net/ont/geosparql#"
  },
  "@graph": [
    {
      "@id": "ex:central-park",
      "@type": "ex:Park",
      "ex:name": "Central Park",
      "ex:entrance": {
        "@value": "POINT(-73.9654 40.7829)",
        "@type": "geo:wktLiteral"
      },
      "ex:boundary": {
        "@value": "POLYGON((-73.9819 40.7681, -73.9580 40.8006, -73.9493 40.7969, -73.9732 40.7644, -73.9819 40.7681))",
        "@type": "geo:wktLiteral"
      }
    }
  ]
}
```

The `ex:entrance` POINT is stored as a native GeoPoint, while the `ex:boundary` POLYGON is stored as a string.

## GeoSPARQL-related support (v1)

Fluree supports the GeoSPARQL `geo:wktLiteral` datatype and `geof:distance` function. Point proximity queries use a unified `geof:distance` bind+filter pattern in both JSON-LD and SPARQL. For complex geometry queries (`within`/`contains`/`intersects`/`nearby`), use the JSON-LD `idx:spatial` pattern described above.

| Feature | Status |
|---------|--------|
| `geo:wktLiteral` datatype | ✅ Supported |
| POINT geometry | ✅ Native encoding (60-bit packed) |
| LINESTRING geometry | ✅ S2 spatial index |
| POLYGON geometry | ✅ S2 spatial index |
| MULTIPOLYGON geometry | ✅ S2 spatial index |
| `geo:asWKT` property | ✅ Use any property with wktLiteral type |
| `geof:distance` function | ✅ Supported (haversine, ~0.3% accuracy) |
| Proximity queries (radius) | ✅ Index-accelerated via geof:distance bind+filter |
| Time travel | ✅ Support via `from: "<ledger>@t:<t>"` |
| k-NN queries (nearest K) | ✅ Via ORDER BY distance + LIMIT |
| `within` spatial predicate | ✅ Via JSON-LD `idx:spatial` |
| `contains` spatial predicate | ✅ Via JSON-LD `idx:spatial` |
| `intersects` spatial predicate | ✅ Via JSON-LD `idx:spatial` |
| Spatial join (two variables) | 🔜 Planned (R-tree) |

## Best Practices

### Use geo:wktLiteral for All Geometry

Always declare the datatype explicitly:

```json
// Correct
{ "@value": "POINT(2.3522 48.8566)", "@type": "geo:wktLiteral" }

// Incorrect - stored as plain string
{ "@value": "POINT(2.3522 48.8566)" }
```

### Coordinate Precision

While Fluree stores ~0.3mm precision, consider your source data accuracy:

```json
// Excessive precision (GPS typically ±3-5m)
"POINT(2.352219834765 48.856614892341)"

// Appropriate precision for most applications
"POINT(2.3522 48.8566)"
```

### Coordinate Validation

Validate coordinates before insertion:

- Latitude: -90 to 90
- Longitude: -180 to 180
- No NaN or infinity values

Invalid coordinates are stored as strings and won't benefit from native GeoPoint indexing.

## Troubleshooting

### Query returns no results

**Check the coordinate order.** WKT uses `POINT(longitude latitude)`, not `POINT(latitude longitude)`:
```json
// Correct: Paris (lng=2.35, lat=48.86)
"POINT(2.35 48.86)"

// Wrong: coordinates swapped
"POINT(48.86 2.35)"
```

**Check the datatype.** Geometry values must use `geo:wktLiteral`:
```json
// Correct
{ "@value": "POINT(2.35 48.86)", "@type": "geo:wktLiteral" }

// Wrong - no datatype, stored as plain string
{ "@value": "POINT(2.35 48.86)" }
```

**Check the predicate.** The property in the triple pattern must match the data exactly:
```json
// If data uses ex:location, the triple must use ex:location
{ "@id": "?place", "ex:location": "?loc" }    // Correct
{ "@id": "?place", "ex:geo": "?loc" }         // Wrong - different predicate
```

For S2 spatial queries, `idx:property` must also match:
```json
"idx:property": "ex:boundary"    // Correct
"idx:property": "ex:geo"         // Wrong - different predicate
```

### "No spatial index available" error

The spatial index is built asynchronously after commits. If querying immediately after insert:
- Wait for background indexing to complete, or
- Use `from: "<ledger>@t:<t>"` to query up to the indexed `t`

### Large polygons cause slow queries

Polygons crossing the antimeridian (±180° longitude) generate many S2 cells. Consider:
- Splitting the polygon at the antimeridian
- Using a simpler bounding region for initial filtering

### SPARQL spatial predicates not accelerated

In v1, SPARQL `geof:*` spatial predicates (like `geof:sfWithin`) evaluate as filters, not index operators. For accelerated spatial queries on complex geometries, use the JSON-LD `idx:spatial` pattern instead. Note: `geof:distance` bind+filter patterns **are** automatically accelerated in both SPARQL and JSON-LD.

## Related Documentation

- [Datatypes](../concepts/datatypes.md) - Type system overview
- [Vector Search](vector-search.md) - Similarity search
- [BM25](bm25.md) - Full-text search


---

# graph-sources/README.md

# Graph Sources and Integrations

Graph sources extend Fluree's query capabilities by integrating specialized indexes and external data sources. Graph sources appear as queryable ledgers but are backed by different storage and indexing systems.

## Graph Source Types

### [Overview](overview.md)

Introduction to graph sources:
- What are graph sources
- Architecture and design
- Use cases
- Performance characteristics
- Creating and managing graph sources

### [Iceberg / Parquet](iceberg.md)

Apache Iceberg data lake integration:
- Querying Iceberg tables
- Parquet file support
- Schema mapping
- Partition pruning
- Performance optimization

### [R2RML](r2rml.md)

Relational database mapping:
- R2RML standard
- Mapping relational data to RDF
- SQL query generation
- Join optimization
- Supported databases (PostgreSQL, MySQL, etc.)

### [BM25 Graph Source](bm25.md)

Full-text search as graph source:
- BM25 index as queryable ledger
- Search predicates
- Combining with structured queries
- Real-time index updates

## What are Graph Sources?

Graph sources are queryable data sources that appear as Fluree ledgers but are backed by specialized storage:

**Standard Ledger:**
```text
mydb:main → RDF triple store → SPOT/POST/OPST/PSOT indexes
```

**Graph Source:**
```text
products-search:main → BM25 index → Inverted text index
products-vector:main → HNSW → Vector similarity index
warehouse-data:main → Iceberg → Parquet files
sql-db:main → R2RML → PostgreSQL tables
```

## Query Transparency

Graph sources are queried like regular ledgers:

```json
{
  "@context": {"f": "https://ns.flur.ee/db#"},
  "from": "products:main",
  "select": ["?product", "?score"],
  "where": [
    {
      "f:graphSource": "products-search:main",
      "f:searchText": "laptop",
      "f:searchLimit": 20,
      "f:searchResult": { "f:resultId": "?product", "f:resultScore": "?score" }
    }
  ]
}
```

> **Note:** SPARQL queries use the same `f:` namespace pattern (`f:graphSource`, `f:searchText`, etc.) within JSON-LD query syntax.

## Multi-Graph Queries

Combine regular ledgers with graph sources:

```json
{
  "@context": {"f": "https://ns.flur.ee/db#"},
  "from": "products:main",
  "select": ["?product", "?name", "?price", "?score"],
  "where": [
    {
      "f:graphSource": "products-search:main",
      "f:searchText": "laptop",
      "f:searchLimit": 20,
      "f:searchResult": { "f:resultId": "?product", "f:resultScore": "?score" }
    },
    { "@id": "?product", "schema:name": "?name" },
    { "@id": "?product", "schema:price": "?price" }
  ],
  "orderBy": ["-?score"]
}
```

Joins structured data from products:main with search results from the products-search:main graph source.

## Graph Source Lifecycle

### 1. Create Graph Source

Define mapping/configuration:

```bash
curl -X POST http://localhost:8090/index/bm25?ledger=mydb:main \
  -d '{"name": "products-search", "fields": [...]}'
```

### 2. Initial Indexing

Build index from source data:
- Load data from source ledger
- Transform to target format
- Build specialized index
- Publish to nameservice

### 3. Incremental Updates

Keep synchronized with source:
- Monitor source ledger for changes
- Update graph source incrementally
- Maintain consistency

### 4. Query Execution

Execute queries against graph source:
- Parse query
- Route to appropriate backend
- Execute specialized query
- Return results

## Supported Graph Sources

### BM25 Full-Text Search

**Purpose:** Keyword search with relevance ranking

**Backend:** Inverted index

**Use Cases:**
- E-commerce product search
- Document search
- Knowledge base search

**Example:**
```json
{
  "@context": {"f": "https://ns.flur.ee/db#"},
  "from": "docs:main",
  "where": [
    {
      "f:graphSource": "docs-search:main",
      "f:searchText": "quarterly report",
      "f:searchLimit": 20,
      "f:searchResult": { "f:resultId": "?doc" }
    }
  ]
}
```

See [BM25 Graph Source](bm25.md) and [BM25 Indexing](../indexing-and-search/bm25.md).

### Vector Similarity Search

**Purpose:** Semantic search using embeddings

**Backend:** HNSW index (embedded or remote)

**Use Cases:**
- Semantic search
- Recommendations
- Image similarity
- Clustering

See [Vector Search](../indexing-and-search/vector-search.md) for details.

### Apache Iceberg

**Purpose:** Query data lake tables

**Backend:** Apache Iceberg / Parquet files

**Use Cases:**
- Analytics on historical data
- Data warehouse integration
- Large-scale batch data

**Example:**
```json
{
  "from": "warehouse-sales:main",
  "select": ["?date", "?revenue"],
  "where": [
    { "@id": "?sale", "warehouse:date": "?date" },
    { "@id": "?sale", "warehouse:revenue": "?revenue" }
  ],
  "filter": "?date >= '2024-01-01'"
}
```

See [Iceberg / Parquet](iceberg.md).

### R2RML (Relational Databases)

**Purpose:** Query relational databases as RDF

**Backend:** SQL databases (PostgreSQL, MySQL, etc.)

**Use Cases:**
- Existing database integration
- Incremental adoption of graph queries
- Unified queries across systems

**Example:**
```json
{
  "from": "sql-customers:main",
  "select": ["?name", "?email"],
  "where": [
    { "@id": "?customer", "schema:name": "?name" },
    { "@id": "?customer", "schema:email": "?email" }
  ]
}
```

See [R2RML](r2rml.md).

## Architecture

### Graph Source Registry

Graph sources registered in nameservice:

```json
{
  "graph_source_id": "products-search:main",
  "type": "bm25",
  "source": "products:main",
  "backend": "inverted_index",
  "status": "ready"
}
```

### Query Routing

Query engine routes to appropriate backend:

```text
Query: FROM <products-search:main>
  ↓
Nameservice lookup: type=bm25
  ↓
Route to BM25 query engine
  ↓
Execute against inverted index
  ↓
Return results
```

### Result Integration

Results from graph sources join with regular graphs:

```text
FROM <products:main>, <products-search:main>
  ↓
Execute subquery on products:main → Results A
Execute subquery on products-search:main → Results B
  ↓
Join Results A + B on ?product
  ↓
Return combined results
```

## Performance Considerations

### Query Planning

Graph sources affect query optimization:
- Specialized indexes enable efficient filtering
- Push filters down to graph source when possible
- Minimize data transfer between graphs

### Data Transfer

Minimize data movement:
- Filter in graph source before joining
- Use selective projections
- Leverage graph source's native capabilities

### Caching

Some graph source backends support caching:
- BM25: Results cacheable
- Vector: Similar queries share computation
- Iceberg: Parquet file caching
- R2RML: SQL query plan caching

## Best Practices

### 1. Choose Appropriate Graph Source Type

Match graph source to use case:
- Keyword search → BM25
- Semantic search → Vector
- Analytics → Iceberg
- Relational database integration → R2RML

### 2. Filter Early

Push filters to graph sources:

Good:
```json
{
  "@context": {"f": "https://ns.flur.ee/db#"},
  "from": "products:main",
  "where": [
    {
      "f:graphSource": "products-search:main",
      "f:searchText": "laptop",
      "f:searchLimit": 50,
      "f:searchResult": { "f:resultId": "?p" }
    },
    { "@id": "?p", "schema:price": "?price" }
  ],
  "filter": "?price < 1000"
}
```

### 3. Monitor Graph Source Lag

Check synchronization status:

```bash
curl http://localhost:8090/index/status/products-search:main
```

### 4. Use Appropriate Limits

Limit results from graph sources:

```json
{
  "@context": {"f": "https://ns.flur.ee/db#"},
  "from": "products:main",
  "where": [
    {
      "f:graphSource": "products-search:main",
      "f:searchText": "query",
      "f:searchLimit": 100,
      "f:searchResult": { "f:resultId": "?p" }
    }
  ]
}
```

### 5. Test Performance

Profile queries combining graph sources:

```bash
curl -X POST http://localhost:8090/v1/fluree/explain \
  -d '{...}'
```

## Troubleshooting

### Graph Source Not Found

```json
{
  "error": "GraphSourceNotFound",
  "message": "Graph source not found: products-search:main"
}
```

**Solution:** Create graph source or check name spelling.

### Synchronization Lag

Graph source out of sync with source:

```bash
# Check status
curl http://localhost:8090/index/status/products-search:main

# Trigger rebuild
curl -X POST http://localhost:8090/index/rebuild/products-search:main
```

### Poor Performance

Query combining graph sources is slow:

1. Check explain plan
2. Add filters to reduce result set
3. Ensure indexes are up-to-date
4. Consider query rewrite

## Related Documentation

- [Overview](overview.md) - Graph source concepts
- [BM25](bm25.md) - Full-text search
- [Vector Search](../indexing-and-search/vector-search.md) - Similarity search
- [Iceberg](iceberg.md) - Data lake integration
- [R2RML](r2rml.md) - Relational mapping
- [Query Datasets](../query/datasets.md) - Multi-graph queries


---

# graph-sources/overview.md

# Graph Sources Overview

Graph sources enable querying specialized indexes and external data sources using the same query interface as regular Fluree ledgers. This document provides a comprehensive overview of graph source architecture and capabilities.

## Concept

A **graph source** is anything you can address by a graph name/IRI and query as part of a single execution. Some graph sources are ledger-backed RDF graphs; others are backed by different systems optimized for specific query patterns.

**Regular Ledger:**
- Stored as RDF triples
- Indexed with SPOT, POST, OPST, PSOT
- Optimized for graph traversal

**Non-ledger Graph Source:**
- Stored in specialized format
- Custom indexing for specific queries
- Optimized for particular use cases

Both are queried using the same SPARQL or JSON-LD Query syntax.

## Architecture

### Components

```text
┌─────────────────────────────────────────┐
│         Fluree Query Engine             │
└─────────────────┬───────────────────────┘
                  │
      ┌───────────┴──────────┐
      │                      │
┌─────▼──────┐      ┌───────▼────────┐
│  Regular   │      │    Graph       │
│  Ledgers   │      │    Sources     │
└─────┬──────┘      └───────┬────────┘
      │                     │
      │             ┌───────┴────────┐
      │             │                │
┌─────▼──────┐ ┌───▼───┐     ┌─────▼──────┐
│ RDF Triple │ │ BM25  │     │  usearch   │
│   Store    │ │ Index │     │  Vector    │
└────────────┘ └───────┘     └────────────┘
```

### Graph Source Registry (Nameservice)

Non-ledger graph sources are registered in nameservice:

```json
{
  "graph_source_id": "products-search:main",
  "type": "graph-source",
  "backend": "bm25",
  "source": "products:main",
  "config": {
    "fields": [...]
  },
  "status": "ready",
  "last_sync": "2024-01-22T10:30:00Z"
}
```

## Graph Source Types

### 1. BM25 Full-Text Search

**Backend:** Inverted text index

**Purpose:** Keyword search with relevance ranking

**Configuration:**
```json
{
  "type": "bm25",
  "source": "products:main",
  "fields": [
    { "predicate": "schema:name", "weight": 2.0 },
    { "predicate": "schema:description", "weight": 1.0 }
  ]
}
```

**Query:**
```json
{
  "@context": {"f": "https://ns.flur.ee/db#"},
  "from": "products:main",
  "where": [
    {
      "f:graphSource": "products-search:main",
      "f:searchText": "laptop",
      "f:searchLimit": 20,
      "f:searchResult": { "f:resultId": "?product", "f:resultScore": "?score" }
    }
  ],
  "select": ["?product", "?score"]
}
```

### 2. Vector Similarity

**Backend:** HNSW index (embedded or remote)

**Purpose:** Semantic search using embeddings

**Configuration:**
```json
{
  "type": "vector",
  "source": "products:main",
  "embedding_property": "ex:embedding",
  "dimensions": 384,
  "metric": "cosine"
}
```

**Query:**
```json
{
  "from": "mydb:main",
  "where": [
    {
      "f:graphSource": "products-vector:main",
      "f:queryVector": [0.1, 0.2, ...],
      "f:distanceMetric": "cosine",
      "f:searchLimit": 10,
      "f:searchResult": {
        "f:resultId": "?product",
        "f:resultScore": "?score"
      }
    }
  ],
  "select": ["?product", "?score"]
}
```

### 3. Apache Iceberg

**Backend:** Iceberg tables / Parquet files via R2RML mapping

**Purpose:** Analytics on data lake

Iceberg graph sources require an [R2RML mapping](r2rml.md) that defines how table rows become RDF triples. Two catalog modes select how Iceberg metadata is discovered:

- **REST catalog**: connects to an Iceberg REST catalog API (e.g., Polaris)
- **Direct S3**: reads `metadata/version-hint.text` from the table’s S3 location (no catalog server required)

See [Iceberg / Parquet](iceberg.md) for full configuration details and examples.

**Query:**
```json
{
  "from": "warehouse-orders:main",
  "select": ["?orderId", "?total"],
  "where": [
    { "@id": "?order", "ex:orderId": "?orderId" },
    { "@id": "?order", "ex:total": "?total" }
  ]
}
```

## Creating Graph Sources

### Via Rust API

Graph sources are created and registered via the `fluree-db-api` Rust API, which publishes the graph source record into the nameservice.

```rust
use fluree_db_api::{FlureeBuilder, R2rmlCreateConfig};

let fluree = FlureeBuilder::default().build().await?;

let config = R2rmlCreateConfig::new_direct(
    "execution-log",
    "s3://bucket/warehouse/logs/execution_log",
    "fluree:file://mappings/execution_log.ttl",
)
.with_s3_region("us-east-1");

fluree.create_r2rml_graph_source(config).await?;
```

## Querying Graph Sources

Graph sources come in two flavors with different query models:

- **Iceberg sources** — queried transparently using standard SPARQL/JSON-LD patterns (FROM, GRAPH, or as a direct query target)
- **Search indexes** (BM25, Vector) — queried using the `f:graphSource` / `f:searchText` pattern

### Iceberg (Transparent)

Iceberg graph sources are queried just like ledgers. No special syntax is needed:

**As a direct target:**
```sparql
-- Query the graph source directly
SELECT ?s ?p ?o FROM <execution-log:main> WHERE { ?s ?p ?o } LIMIT 10
```

**Via GRAPH pattern (joining with ledger data):**
```json
{
  "from": "mydb:main",
  "select": ["?customer", "?orderId", "?total"],
  "where": [
    { "@id": "?customer", "schema:name": "?name" },
    { "@id": "?customer", "ex:customerId": "?custId" },
    {
      "graph": "warehouse-orders:main",
      "where": [
        { "@id": "?order", "ex:customerId": "?custId" },
        { "@id": "?order", "ex:orderId": "?orderId" },
        { "@id": "?order", "ex:total": "?total" }
      ]
    }
  ]
}
```

Iceberg graph sources use R2RML mappings to define how table rows become RDF triples. See [Iceberg / Parquet](iceberg.md) and [R2RML](r2rml.md) for details.

### Search Indexes (BM25, Vector)

Search indexes use the `f:graphSource` pattern:

### Single Graph Source

Query one graph source:

```json
{
  "@context": {"f": "https://ns.flur.ee/db#"},
  "from": "products:main",
  "select": ["?product", "?score"],
  "where": [
    {
      "f:graphSource": "products-search:main",
      "f:searchText": "laptop",
      "f:searchLimit": 20,
      "f:searchResult": { "f:resultId": "?product", "f:resultScore": "?score" }
    }
  ]
}
```

### Multiple Graph Sources

Combine multiple graph sources:

```json
{
  "@context": {"f": "https://ns.flur.ee/db#"},
  "from": "products:main",
  "select": ["?product", "?textScore", "?vecScore"],
  "values": [
    ["?queryVec"],
    [{"@value": [0.1, 0.2, 0.3], "@type": "https://ns.flur.ee/db#embeddingVector"}]
  ],
  "where": [
    {
      "f:graphSource": "products-search:main",
      "f:searchText": "laptop",
      "f:searchLimit": 100,
      "f:searchResult": { "f:resultId": "?product", "f:resultScore": "?textScore" }
    },
    {
      "f:graphSource": "products-vector:main",
      "f:queryVector": "?queryVec",
      "f:searchLimit": 100,
      "f:searchResult": { "f:resultId": "?product", "f:resultScore": "?vecScore" }
    }
  ]
}
```

### Graph Sources + Regular Graphs

Combine graph sources and regular ledgers:

```json
{
  "@context": {"f": "https://ns.flur.ee/db#"},
  "from": "products:main",
  "select": ["?product", "?name", "?price", "?score"],
  "where": [
    {
      "f:graphSource": "products-search:main",
      "f:searchText": "laptop",
      "f:searchLimit": 20,
      "f:searchResult": { "f:resultId": "?product", "f:resultScore": "?score" }
    },
    { "@id": "?product", "schema:name": "?name" },
    { "@id": "?product", "schema:price": "?price" }
  ]
}
```

## Synchronization

### Source Tracking

Graph sources track their source ledger:

```text
Source: products:main @ t=150
Graph Source: products-search:main @ source_t=150
```

### Update Modes

**Real-Time:**
- Updates immediately as source changes
- Low latency
- Higher overhead

**Batch:**
- Updates periodically
- Higher latency
- Lower overhead

**Manual:**
- Updates on demand
- Full control
- Requires manual triggering

### Checking Sync Status

```bash
curl http://localhost:8090/graph-source/products-search:main/status
```

Response:
```json
{
  "name": "products-search:main",
  "source": "products:main",
  "source_t": 150,
  "index_t": 148,
  "lag": 2,
  "last_sync": "2024-01-22T10:30:00Z",
  "status": "syncing"
}
```

## Query Execution

### Query Planning

Query planner handles graph sources:

1. **Parse Query:** Extract graph patterns
2. **Route Subqueries:** Identify which graphs handle which patterns
3. **Execute Subqueries:** Run against appropriate backends
4. **Join Results:** Combine results from multiple graphs
5. **Apply Filters:** Final filtering and sorting

### Example Execution

Query:
```json
{
  "@context": {"f": "https://ns.flur.ee/db#"},
  "from": "products:main",
  "where": [
    {
      "f:graphSource": "products-search:main",
      "f:searchText": "laptop",
      "f:searchLimit": 50,
      "f:searchResult": { "f:resultId": "?p" }
    },
    { "@id": "?p", "schema:price": "?price" }
  ],
  "filter": "?price < 1000"
}
```

Execution Plan:
```text
1. Execute BM25 search on products-search:main:
   f:searchText "laptop", f:searchLimit 50
   → Result: ?p = [ex:p1, ex:p2, ex:p3, ...]

2. Execute on products:main:
   SELECT ?p ?price WHERE {
     VALUES ?p { ex:p1 ex:p2 ex:p3 ... }
     ?p schema:price ?price
   }
   → Result: [(ex:p1, 899), (ex:p2, 1200), ...]

3. Join and filter:
   ?price < 1000
   → Result: [(ex:p1, 899)]
```

## Performance Characteristics

### BM25 Graph Sources

- **Index Build:** O(n × avg_doc_length)
- **Query:** O(log n) with inverted index
- **Space:** 2-3× source data
- **Update:** Incremental, O(doc_size)

### Vector Graph Sources

- **Index Build:** O(n log n) for HNSW
- **Query:** O(log n) approximate
- **Space:** 1.5× embedding size
- **Update:** Incremental, O(1)

### Iceberg Graph Sources

- **Index Build:** No index (direct file access)
- **Query:** O(partitions scanned)
- **Space:** Zero overhead (uses Parquet files)
- **Update:** Batch-oriented

## Best Practices

### 1. Choose Appropriate Type

Match graph source type to use case:
- **Keyword search** → BM25
- **Semantic search** → Vector
- **Analytics / data lake** → Iceberg (with R2RML mapping)

### 2. Monitor Synchronization

Check sync lag regularly:

```javascript
setInterval(async () => {
  const status = await getGraphSourceStatus('products-search:main');
  if (status.lag > 10) {
    console.warn(`Graph source lag: ${status.lag} transactions`);
  }
}, 60000);
```

### 3. Filter in Graph Sources

Push filters to graph sources when possible:

Good (graph source pattern first narrows results before graph traversal):
```json
{
  "@context": {"f": "https://ns.flur.ee/db#"},
  "from": "products:main",
  "where": [
    {
      "f:graphSource": "products-search:main",
      "f:searchText": "laptop",
      "f:searchLimit": 20,
      "f:searchResult": { "f:resultId": "?p" }
    },
    { "@id": "?p", "schema:name": "?name" }
  ]
}
```

Bad (graph traversal before graph source means scanning all products first):
```json
{
  "@context": {"f": "https://ns.flur.ee/db#"},
  "from": "products:main",
  "where": [
    { "@id": "?p", "schema:name": "?name" },
    {
      "f:graphSource": "products-search:main",
      "f:searchText": "laptop",
      "f:searchLimit": 20,
      "f:searchResult": { "f:resultId": "?p" }
    }
  ]
}
```

### 4. Use Explain Plans

Understand query execution:

```bash
curl -X POST http://localhost:8090/v1/fluree/explain \
  -d '{...}'
```

### 5. Limit Results

Always use LIMIT with graph sources:

```json
{
  "where": [...],
  "limit": 100
}
```

## Troubleshooting

### High Sync Lag

**Symptom:** `lag` increasing

**Causes:**
- Source ledger write rate too high
- Graph source indexing too slow
- Resource constraints

**Solutions:**
- Increase indexing resources
- Batch updates
- Use manual sync mode

### Query Performance Issues

**Symptom:** Slow queries combining graph sources

**Solutions:**
1. Check explain plan
2. Add filters to reduce intermediate results
3. Ensure graph source is synced
4. Consider query rewrite

### Missing Results

**Symptom:** Expected results not returned

**Causes:**
- Graph source not synced
- Mapping misconfiguration
- Filter too restrictive

**Solutions:**
- Check sync status
- Verify mapping configuration
- Test subqueries independently

## Related Documentation

- [BM25 Graph Source](bm25.md) - Full-text search
- [Iceberg](iceberg.md) - Data lake integration
- [R2RML](r2rml.md) - R2RML mapping reference
- [BM25 Indexing](../indexing-and-search/bm25.md) - BM25 details
- [Vector Search](../indexing-and-search/vector-search.md) - Vector details
- [Query Datasets](../query/datasets.md) - Multi-graph queries


---

# graph-sources/iceberg.md

# Iceberg / Parquet

Fluree integrates with Apache Iceberg to query data lake tables as graph sources. An [R2RML mapping](r2rml.md) defines how Iceberg table rows are materialized into RDF triples, enabling you to query large-scale analytical data stored in Parquet format using the same SPARQL / JSON-LD query interface as regular ledgers.

**Note:** Requires the `iceberg` feature flag. See [Compatibility and Feature Flags](../reference/compatibility.md#fluree-db-api-features).

## What is Apache Iceberg?

Apache Iceberg is an open table format for huge analytical datasets. It provides:
- ACID transactions on data lakes
- Time travel and versioning
- Schema evolution
- Partition management
- Optimized file organization (Parquet)

## Configuration

### Catalog Modes

Fluree supports two ways to discover Iceberg metadata:

- **REST catalog**: discover table metadata via an Iceberg REST catalog API (e.g., Polaris).
- **Direct S3 (no catalog server)**: bypass REST discovery and read `version-hint.text` from the table’s `metadata/` directory to resolve the current metadata file.

### CLI

The `fluree iceberg map` command creates Iceberg graph sources from the command line. An R2RML mapping is required to define how table rows become RDF triples.

```bash
# REST catalog with R2RML mapping
fluree iceberg map warehouse-orders \
  --catalog-uri https://polaris.example.com/api/catalog \
  --r2rml mappings/orders.ttl \
  --auth-bearer $POLARIS_TOKEN

# Direct S3 (no catalog server) with R2RML mapping
fluree iceberg map execution-log \
  --mode direct \
  --table-location s3://bucket/warehouse/logs/execution_log \
  --r2rml mappings/execution_log.ttl
```

Once mapped, graph sources appear in `fluree list`, can be inspected with `fluree info`, and removed with `fluree drop`. See [CLI iceberg reference](../cli/iceberg.md) for all options.

### HTTP API

When running the Fluree server (or Docker image) with the `iceberg` feature enabled, map a table by POSTing to `{api_base_url}/iceberg/map` (default: `/v1/fluree/iceberg/map`). The endpoint is admin-protected — include the admin Bearer token if admin auth is configured.

```bash
# REST catalog with R2RML mapping (mapping passed inline)
curl -X POST http://localhost:8090/v1/fluree/iceberg/map \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d @- <<'JSON'
{
  "name": "warehouse-orders",
  "mode": "rest",
  "catalog_uri": "https://polaris.example.com/api/catalog",
  "table": "sales.orders",
  "warehouse": "my-warehouse",
  "auth_bearer": "polaris-token-here",
  "r2rml": "@prefix rr: <http://www.w3.org/ns/r2rml#> . ...",
  "r2rml_type": "text/turtle"
}
JSON
```

```bash
# Direct S3 mode (no catalog server)
curl -X POST http://localhost:8090/v1/fluree/iceberg/map \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{
    "name": "execution-log",
    "mode": "direct",
    "table_location": "s3://bucket/warehouse/logs/execution_log",
    "r2rml": "...",
    "r2rml_type": "text/turtle",
    "s3_region": "us-east-1",
    "s3_path_style": true
  }'
```

R2RML can be omitted to auto-generate a direct mapping. AWS credentials for `direct` mode are read from the server's environment (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`, or an attached instance role). See the [Graph Source Endpoints](../api/endpoints.md#graph-source-endpoints) section in the API reference for the complete request/response schema.

### Rust API

`R2rmlCreateConfig::new` and `new_direct` take the R2RML mapping as a **content string** (Turtle or JSON-LD), not a file path — read the file yourself first. To reference an already-stored mapping by address instead, build the config directly with `R2rmlMappingInput::Address(...)`.

**REST catalog mode (Polaris-style):**

```rust
use fluree_db_api::R2rmlCreateConfig;

let mapping = std::fs::read_to_string("mappings/orders.ttl")?;

let config = R2rmlCreateConfig::new(
    "warehouse-orders",
    "https://polaris.example.com/api/catalog",
    "sales.orders",
    mapping,
)
.with_warehouse("my-warehouse")
.with_auth_bearer("my-token")
.with_vended_credentials(true);

fluree.create_r2rml_graph_source(config).await?;
```

**Direct S3 mode (no REST catalog):**

```rust
use fluree_db_api::R2rmlCreateConfig;

let mapping = std::fs::read_to_string("mappings/execution_log.ttl")?;

let config = R2rmlCreateConfig::new_direct(
    "execution-log",
    "s3://bucket/warehouse/logs/execution_log",
    mapping,
)
.with_s3_region("us-east-1")
.with_s3_path_style(true);

fluree.create_r2rml_graph_source(config).await?;
```

### Stored Configuration Format (Nameservice)

Iceberg graph sources are persisted as an `IcebergGsConfig` JSON document in the nameservice record’s `config` field.

Note the nesting: the graph source is “Iceberg” (this page), and `catalog.type` selects the **catalog mode** (`rest` vs `direct`) used to discover Iceberg metadata.

**REST catalog config:**

```json
{
  "catalog": {
    "type": "rest",
    "uri": "https://polaris.example.com/api/catalog",
    "warehouse": "my-warehouse",
    "auth": { "type": "bearer", "token": { "env_var": "POLARIS_TOKEN" } }
  },
  "table": "sales.orders",
  "io": {
    "vended_credentials": true,
    "s3_region": "us-east-1",
    "s3_endpoint": null,
    "s3_path_style": false
  }
}
```

**Direct S3 config:**

```json
{
  "catalog": {
    "type": "direct",
    "table_location": "s3://bucket/warehouse/logs/execution_log"
  },
  "table": "",
  "io": {
    "vended_credentials": false,
    "s3_region": "us-east-1",
    "s3_endpoint": null,
    "s3_path_style": true
  }
}
```

**Direct mode requirements:**

- `catalog.table_location` must be an S3 URI (`s3://` or `s3a://`) pointing to the table root directory.
- The table must contain a `metadata/` subdirectory with:
  - `version-hint.text` (containing the current metadata filename, e.g., `00001-abc-def.metadata.json`)
  - The referenced `.metadata.json` file
- Direct mode uses ambient AWS credentials (IAM roles, env vars, `~/.aws/credentials`). It does **not** support vended credentials.

**How Direct metadata resolution works:**

- Fluree does **not** require you to provide a path to `version-hint.text` in the config. You provide the **table root** (`table_location`), and Fluree reads:
  - `"{table_location}/metadata/version-hint.text"` to get the current metadata filename
  - `"{table_location}/metadata/{filename}"` as the table’s current metadata
- `version-hint.text` may contain a bare filename (e.g., `00001-abc.metadata.json`) or a full absolute path (`s3://...`).
- If `version-hint.text` is missing or empty, Direct mode fails with an error mentioning `version-hint.text`.

**Iceberg table setup must already exist:**

Direct mode assumes `table_location` points at a **valid Iceberg table layout** (created by `iceberg-rust`, Spark, etc.), including the `metadata/` directory and referenced metadata/manifest files. Fluree does not create or “bootstrap” Iceberg tables; it only reads them.

**When to use Direct vs REST:**
| Scenario | Recommended |
|----------|-------------|
| Shared catalog (multiple consumers) | REST |
| Writer and reader are the same system | Direct |
| `iceberg-rust` / Spark appending to known S3 path | Direct |
| Need catalog-managed credentials (vended) | REST |
| Minimizing infrastructure (no catalog server) | Direct |

## RDF Mapping (R2RML)

Every Iceberg graph source requires an [R2RML mapping](r2rml.md) (Turtle format) that defines how table rows become RDF triples — specifying subject IRI templates, predicate mappings, and type conversions. See [R2RML](r2rml.md) for the full mapping reference.

### Type Mapping

Iceberg types map to XSD types:

| Iceberg Type | RDF Type |
|--------------|----------|
| int, long | xsd:integer |
| float, double | xsd:decimal |
| string | xsd:string |
| boolean | xsd:boolean |
| date | xsd:date |
| timestamp | xsd:dateTime |
| uuid | xsd:string |

## Querying Iceberg Tables

Iceberg graph sources are queried using standard SPARQL and JSON-LD syntax. In the Rust API, mapped sources resolve transparently through the lazy query builders:

- `fluree.graph("warehouse-orders:main").query()` for a single target that may be either a native ledger or a mapped graph source
- `fluree.query_from()` when the query body itself carries the dataset (`"from"` / `FROM`) or when composing multiple sources

The lower-level materialized snapshot path (`let view = fluree.db(...).await?; fluree.query(&view, ...)`) is still native-ledger-oriented and should not be used for graph source aliases.

```rust
// Single-target lazy query
let result = fluree.graph("warehouse-orders:main")
    .query()
    .sparql("SELECT * WHERE { ?s ?p ?o } LIMIT 10")
    .execute()
    .await?;

// FROM-driven query
let result = fluree.query_from()
    .sparql("SELECT * FROM <warehouse-orders:main> WHERE { ?s ?p ?o } LIMIT 10")
    .execute()
    .await?;
```

### Basic Query

```json
{
  "@context": {
    "ex": "http://example.org/ns/"
  },
  "from": "warehouse-orders:main",
  "select": ["?orderId", "?total"],
  "where": [
    { "@id": "?order", "ex:orderId": "?orderId" },
    { "@id": "?order", "ex:total": "?total" }
  ],
  "limit": 100
}
```

### SPARQL Query

```sparql
PREFIX ex: <http://example.org/ns/>

SELECT ?orderId ?total ?date
FROM <warehouse-orders:main>
WHERE {
  ?order ex:orderId ?orderId .
  ?order ex:total ?total .
  ?order ex:orderDate ?date .
  FILTER (?date >= "2024-01-01"^^xsd:date)
}
ORDER BY DESC(?date)
LIMIT 100
```

## Partition Pruning

Iceberg's partition pruning optimizes queries:

```json
{
  "from": "warehouse-orders:main",
  "select": ["?orderId", "?total"],
  "where": [
    { "@id": "?order", "ex:orderId": "?orderId" },
    { "@id": "?order", "ex:total": "?total" },
    { "@id": "?order", "ex:orderDate": "?date" }
  ],
  "filter": "?date >= '2024-01-01' && ?date < '2024-02-01'"
}
```

If `orderDate` is a partition column, Iceberg only scans January 2024 partitions.

## Combining with Fluree Data

Join Iceberg data with Fluree ledgers:

```json
{
  "from": ["customers:main", "warehouse-orders:main"],
  "select": ["?customerName", "?orderTotal", "?orderDate"],
  "where": [
    { "@id": "?customer", "schema:name": "?customerName" },
    { "@id": "?customer", "ex:customerId": "?customerId" },
    { "@id": "?order", "ex:customerId": "?customerId" },
    { "@id": "?order", "ex:total": "?orderTotal" },
    { "@id": "?order", "ex:orderDate": "?orderDate" }
  ],
  "filter": "?orderDate >= '2024-01-01'",
  "orderBy": ["-?orderDate"]
}
```

Combines customer data from Fluree with order data from Iceberg.

## Time Travel

Query historical Iceberg snapshots:

```json
{
  "from": "warehouse-orders:main@snapshot:12345",
  "select": ["?orderId", "?total"],
  "where": [
    { "@id": "?order", "ex:orderId": "?orderId" },
    { "@id": "?order", "ex:total": "?total" }
  ]
}
```

Or by timestamp:

```json
{
  "from": "warehouse-orders:main@timestamp:2024-01-01T00:00:00Z",
  "select": ["?orderId", "?total"],
  "where": [...]
}
```

## Aggregations

Aggregate Iceberg data:

```sparql
PREFIX ex: <http://example.org/ns/>

SELECT ?date (SUM(?total) AS ?dailyRevenue) (COUNT(?order) AS ?orderCount)
FROM <warehouse-orders:main>
WHERE {
  ?order ex:orderDate ?date .
  ?order ex:total ?total .
  FILTER (?date >= "2024-01-01"^^xsd:date)
}
GROUP BY ?date
ORDER BY ?date
```

## Performance

### Query Planning

Fluree pushes filters to Iceberg:

```text
Query: SELECT ?id WHERE { ?order ex:orderDate ?date } FILTER (?date > "2024-01-01")
  ↓
Pushed to Iceberg:
  SELECT order_id FROM sales.orders WHERE order_date > '2024-01-01'
  ↓
Iceberg optimizations:
  - Partition pruning (only scan 2024 partitions)
  - File skipping (skip files outside date range)
  - Column pruning (only read order_id, order_date)
```

### Best Practices

1. **Partition by Common Filters:**
   ```sql
   -- Partition Iceberg table by date
   PARTITIONED BY (YEAR(order_date), MONTH(order_date))
   ```

2. **Use Filters:**
   ```json
   {
     "where": [...],
     "filter": "?date >= '2024-01-01'"  // Enables partition pruning
   }
   ```

3. **Limit Results:**
   ```json
   {
     "where": [...],
     "limit": 1000
   }
   ```

4. **Project Only Needed Columns:**
   ```json
   {
     "select": ["?orderId", "?total"],  // Only these columns read from Parquet
     "where": [...]
   }
   ```

## Schema Evolution

Iceberg supports schema evolution via metadata updates. If a schema change renames/removes columns used by your R2RML mapping, update the mapping accordingly.

## Configuration Options

### AWS Credentials

For S3-backed Iceberg (both REST and Direct modes):

```bash
export AWS_ACCESS_KEY_ID=your-key
export AWS_SECRET_ACCESS_KEY=your-secret
export AWS_REGION=us-east-1
```

REST catalog mode also supports vended credentials (credentials issued by the catalog). Direct mode uses only ambient AWS credentials (env vars, IAM roles, `~/.aws/credentials`).

## Use Cases

### Analytics on Historical Data

Query years of historical data:

```sparql
SELECT ?year (SUM(?revenue) AS ?totalRevenue)
FROM <warehouse-sales:main>
WHERE {
  ?sale ex:year ?year .
  ?sale ex:revenue ?revenue .
  FILTER (?year >= 2020 && ?year <= 2023)
}
GROUP BY ?year
ORDER BY ?year
```

### Data Warehouse Integration

Combine real-time Fluree data with warehouse analytics:

```json
{
  "from": ["products:main", "warehouse-sales:main"],
  "select": ["?productName", "?totalSold"],
  "where": [
    { "@id": "?product", "schema:name": "?productName" },
    { "@id": "?product", "ex:productId": "?pid" },
    { "@id": "?sale", "ex:productId": "?pid" }
  ]
}
```

### Large-Scale Reporting

Generate reports from petabyte-scale data:

```sparql
SELECT ?region ?category (SUM(?amount) AS ?total)
FROM <warehouse-transactions:main>
WHERE {
  ?txn ex:region ?region .
  ?txn ex:category ?category .
  ?txn ex:amount ?amount .
  FILTER (?year = 2024)
}
GROUP BY ?region ?category
ORDER BY DESC(?total)
```

## Limitations

1. **Read-Only:** Iceberg graph sources are read-only (no writes via Fluree)
2. **Complex Joins:** Large joins between Fluree and Iceberg may be slow
3. **No Full-Text Search:** Use Fluree's BM25 for text search

## Troubleshooting

### Connection Issues

```json
{
  "error": "IcebergConnectionError",
  "message": "Cannot connect to Glue catalog"
}
```

**Solutions:**
- Check AWS credentials
- Verify IAM permissions
- Check network connectivity

### Schema Mismatch

```json
{
  "error": "SchemaMismatchError",
  "message": "Column 'order_date' not found in Iceberg table"
}
```

**Solutions:**
- Update R2RML mapping configuration (if the mapping references missing columns)
- Verify table name and catalog

### Slow Queries

**Causes:**
- Large result sets
- No partition pruning
- Scanning many files

**Solutions:**
- Add date filters to enable partition pruning
- Use LIMIT clause
- Optimize Iceberg table partitioning
- Use Iceberg file compaction

## Related Documentation

- [Graph Sources Overview](overview.md) - Graph source concepts
- [R2RML](r2rml.md) - Relational database mapping
- [Query Datasets](../query/datasets.md) - Multi-graph queries


---

# graph-sources/r2rml.md

# R2RML (Relational to RDF Mapping)

R2RML (RDB to RDF Mapping Language) is a W3C standard for mapping tabular data into RDF triples. In Fluree, R2RML mappings are used to expose **Iceberg tables** as RDF graph sources, enabling you to query data lake tables using SPARQL or JSON-LD Query.

## What is R2RML?

R2RML defines how to map:
- Database tables to RDF classes
- Table columns to RDF properties
- Rows to RDF resources
- Foreign keys to RDF relationships

In Fluree, this enables querying Iceberg tables as if they were RDF graphs.

## Configuration

### Create R2RML Graph Source (Iceberg-backed)

Use `R2rmlCreateConfig` to register a graph source that combines:

- an Iceberg table (REST catalog or Direct S3), and
- an R2RML mapping (Turtle) that materializes table rows into RDF triples.

If you use **Direct S3** mode, Fluree resolves the current Iceberg metadata by reading `metadata/version-hint.text` under the configured `table_location`, then loading the metadata file referenced by the hint. The Iceberg table layout must already exist at that location.

```rust
use fluree_db_api::{FlureeBuilder, R2rmlCreateConfig};

let fluree = FlureeBuilder::default().build().await?;

let config = R2rmlCreateConfig::new_direct(
    "airlines-rdf",
    "s3://bucket/warehouse/openflights/airlines",
    "fluree:file://mappings/airlines.ttl",
)
.with_s3_region("us-east-1")
.with_s3_path_style(true)
.with_mapping_media_type("text/turtle");

fluree.create_r2rml_graph_source(config).await?;
```

## R2RML Mapping

### Basic Mapping

Map a table to RDF class:

```turtle
@prefix rr: <http://www.w3.org/ns/r2rml#> .
@prefix ex: <http://example.org/ns/> .
@prefix schema: <http://schema.org/> .

<#CustomerMapping>
  a rr:TriplesMap ;
  
  rr:logicalTable [
    rr:tableName "customers"
  ] ;
  
  rr:subjectMap [
    rr:template "http://example.org/customer/{id}" ;
    rr:class schema:Person
  ] ;
  
  rr:predicateObjectMap [
    rr:predicate schema:name ;
    rr:objectMap [ rr:column "name" ]
  ] ;
  
  rr:predicateObjectMap [
    rr:predicate schema:email ;
    rr:objectMap [ rr:column "email" ]
  ] ;
  
  rr:predicateObjectMap [
    rr:predicate ex:customerId ;
    rr:objectMap [ rr:column "id" ]
  ] .
```

This maps the `customers` table:

```sql
CREATE TABLE customers (
  id SERIAL PRIMARY KEY,
  name VARCHAR(255),
  email VARCHAR(255)
);
```

To RDF triples:

```turtle
<http://example.org/customer/1>
  a schema:Person ;
  schema:name "Alice" ;
  schema:email "alice@example.org" ;
  ex:customerId "1" .
```

### Foreign Key Mapping

Map relationships:

```turtle
<#OrderMapping>
  a rr:TriplesMap ;
  
  rr:logicalTable [
    rr:tableName "orders"
  ] ;
  
  rr:subjectMap [
    rr:template "http://example.org/order/{id}" ;
    rr:class ex:Order
  ] ;
  
  rr:predicateObjectMap [
    rr:predicate ex:orderId ;
    rr:objectMap [ rr:column "id" ]
  ] ;
  
  rr:predicateObjectMap [
    rr:predicate ex:customer ;
    rr:objectMap [
      rr:parentTriplesMap <#CustomerMapping> ;
      rr:joinCondition [
        rr:child "customer_id" ;
        rr:parent "id"
      ]
    ]
  ] ;
  
  rr:predicateObjectMap [
    rr:predicate ex:total ;
    rr:objectMap [ rr:column "total" ]
  ] .
```

Maps foreign key `customer_id` to RDF object property linking to customer resource.

### Complex Queries

Use SQL views for complex mappings:

```turtle
<#SalesReportMapping>
  a rr:TriplesMap ;
  
  rr:logicalTable [
    rr:sqlQuery """
      SELECT
        c.id as customer_id,
        c.name as customer_name,
        SUM(o.total) as total_spent,
        COUNT(o.id) as order_count
      FROM customers c
      JOIN orders o ON o.customer_id = c.id
      WHERE o.order_date >= '2024-01-01'
      GROUP BY c.id, c.name
    """
  ] ;
  
  rr:subjectMap [
    rr:template "http://example.org/customer/{customer_id}" ;
    rr:class ex:Customer
  ] ;
  
  rr:predicateObjectMap [
    rr:predicate schema:name ;
    rr:objectMap [ rr:column "customer_name" ]
  ] ;
  
  rr:predicateObjectMap [
    rr:predicate ex:totalSpent ;
    rr:objectMap [ rr:column "total_spent" ; rr:datatype xsd:decimal ]
  ] ;
  
  rr:predicateObjectMap [
    rr:predicate ex:orderCount ;
    rr:objectMap [ rr:column "order_count" ; rr:datatype xsd:integer ]
  ] .
```

## Querying R2RML Graph Sources

R2RML graph sources are queried using standard SPARQL and JSON-LD query syntax — no special query language is needed. In the Rust API, graph source resolution is wired into the lazy query builders:

- `fluree.graph("my-gs:main").query()` for a single target that may be either a native ledger or a mapped graph source
- `fluree.query_from()` when the query body specifies the dataset (`"from"` / `FROM`) or combines multiple sources

The raw materialized snapshot path (`fluree.db(&alias)` → `fluree.query(&view, ...)`) is still the wrong abstraction for graph source aliases because it assumes a native ledger snapshot has already been loaded.

Graph sources can be:
- **Queried directly** as the target: `fluree query my-gs 'SELECT * WHERE { ?s ?p ?o }'`
- **Referenced in FROM clauses**: `SELECT * FROM <my-gs:main> WHERE { ... }`
- **Referenced in GRAPH patterns**: `SELECT * WHERE { GRAPH <my-gs:main> { ... } }` (useful for joining with ledger data)

### Basic Query

```json
{
  "@context": {
    "schema": "http://schema.org/",
    "ex": "http://example.org/ns/"
  },
  "from": "warehouse-customers:main",
  "select": ["?name", "?email"],
  "where": [
    { "@id": "?customer", "@type": "schema:Person" },
    { "@id": "?customer", "schema:name": "?name" },
    { "@id": "?customer", "schema:email": "?email" }
  ]
}
```

The mapping controls how subjects and predicate/object values are produced from the scanned table columns.

### SPARQL Query

```sparql
PREFIX schema: <http://schema.org/>
PREFIX ex: <http://example.org/ns/>

SELECT ?name ?email
FROM <warehouse-customers:main>
WHERE {
  ?customer a schema:Person .
  ?customer schema:name ?name .
  ?customer schema:email ?email .
}
```

### Filters

```json
{
  "from": "warehouse-customers:main",
  "select": ["?name", "?email"],
  "where": [
    { "@id": "?customer", "schema:name": "?name" },
    { "@id": "?customer", "schema:email": "?email" },
    { "@id": "?customer", "ex:status": "?status" }
  ],
  "filter": "?status == 'active'"
}
```

### Joins

```json
{
  "from": "warehouse-orders:main",
  "select": ["?customerName", "?orderTotal"],
  "where": [
    { "@id": "?customer", "schema:name": "?customerName" },
    { "@id": "?order", "ex:customer": "?customer" },
    { "@id": "?order", "ex:total": "?orderTotal" }
  ]
}
```

## Combining with Fluree Data

Join Iceberg data with Fluree ledgers:

```json
{
  "from": ["products:main", "warehouse-inventory:main"],
  "select": ["?productName", "?stockLevel"],
  "where": [
    { "@id": "?product", "schema:name": "?productName" },
    { "@id": "?product", "ex:sku": "?sku" },
    { "@id": "?inventory", "ex:sku": "?sku" },
    { "@id": "?inventory", "ex:stockLevel": "?stockLevel" }
  ]
}
```

Combines product data from Fluree with inventory from an Iceberg-backed R2RML graph source.

## Performance

R2RML graph sources execute by scanning the underlying Iceberg table and materializing RDF terms according to the mapping.

### Best Practices

1. **Filter Early:** Filters are pushed down to Iceberg for partition pruning.
   ```json
   {
     "where": [...],
     "filter": "?date >= '2024-01-01'"
   }
   ```

2. **Limit Results:**
   ```json
   {
     "where": [...],
     "limit": 100
   }
   ```

3. **Project Only Needed Columns:** Only columns referenced in the query and mapping are read from Parquet files.

4. **Partition by Common Filters:** Partition your Iceberg tables by columns frequently used in filters (e.g., date).

## Use Cases

### Data Lake Analytics

Query Iceberg tables containing large-scale analytical data alongside Fluree ledgers:

```json
{
  "from": ["products:main", "warehouse-sales:main"],
  "select": ["?productName", "?totalSold"],
  "where": [
    { "@id": "?product", "schema:name": "?productName" },
    { "@id": "?product", "ex:productId": "?pid" },
    { "@id": "?sale", "ex:productId": "?pid" },
    { "@id": "?sale", "ex:quantity": "?totalSold" }
  ]
}
```

### Multi-Table Mapping

A single R2RML mapping file can define multiple `TriplesMap` entries, each targeting a different Iceberg table or logical view. This enables querying across related tables through a single graph source.

## Limitations

1. **Read-Only:** R2RML graph sources are read-only (no writes via Fluree)
2. **Performance:** Complex joins across Fluree + Iceberg may be slow
3. **Schema Changes:** Requires mapping updates when referenced columns change

## Troubleshooting

### Connection Errors

```json
{
  "error": "IcebergConnectionError",
  "message": "Cannot load table metadata"
}
```

**Solutions:**
- Check catalog configuration (REST vs Direct)
- Verify AWS credentials and S3 access
- Verify `version-hint.text` is present for Direct mode

### Mapping Errors

```json
{
  "error": "R2RMLMappingError",
  "message": "Invalid R2RML mapping: table 'customers' not found"
}
```

**Solutions:**
- Verify table name / location
- Check referenced column names in the mapping
- Validate R2RML syntax (Turtle)

### Slow Queries

**Causes:**
- Large result sets (many Parquet files scanned)
- No partition pruning
- Complex joins across Fluree + Iceberg

**Solutions:**
- Add date/partition filters to enable Iceberg partition pruning
- Use LIMIT clause
- Optimize R2RML mapping to project only needed columns
- Partition Iceberg tables by common filter columns

## Related Documentation

- [Graph Sources Overview](overview.md) - Graph source concepts
- [Iceberg](iceberg.md) - Data lake integration
- [Query Datasets](../query/datasets.md) - Multi-graph queries


---

# graph-sources/bm25.md

# BM25 Graph Source

BM25 indexes in Fluree are implemented as graph sources, allowing full-text search to be seamlessly integrated with structured graph queries through the standard query interface.

## Overview

A BM25 graph source:
- Indexes text content from a source ledger using a configurable query
- Provides relevance-ranked search results via BM25 scoring
- Integrates with JSON-LD queries through `f:` namespace predicates
- Supports time-travel (query the index at any historical point)
- Maintains a manifest of snapshots for incremental sync

For index creation, configuration, and lifecycle management, see [BM25 Full-Text Search](../indexing-and-search/bm25.md).

## Querying BM25 Graph Sources

### JSON-LD Search Pattern

BM25 search uses the `f:` (Fluree) namespace predicates in where clauses:

```json
{
    "@context": {
        "ex": "http://example.org/",
        "f": "https://ns.flur.ee/db#"
    },
    "from": "docs:main",
    "where": [
        {
            "f:graphSource": "article-search:main",
            "f:searchText": "rust programming",
            "f:searchLimit": 10,
            "f:searchResult": {
                "f:resultId": "?doc",
                "f:resultScore": "?score"
            }
        },
        { "@id": "?doc", "ex:title": "?title" }
    ],
    "select": ["?doc", "?title", "?score"]
}
```

### Pattern Fields

| Field | Required | Description |
|-------|----------|-------------|
| `f:graphSource` | Yes | Graph source ID (e.g., `"article-search:main"`) |
| `f:searchText` | Yes | Query text. Analyzed with the same tokenizer/stemmer as indexing. |
| `f:searchLimit` | Yes | Maximum number of search results to return |
| `f:searchResult` | Yes | Object with variable bindings for results |
| `f:resultId` | Yes | Variable for the matched document IRI (e.g., `"?doc"`) |
| `f:resultScore` | No | Variable for the BM25 relevance score (e.g., `"?score"`) |
| `f:resultLedger` | No | Variable for the source ledger alias (for multi-ledger provenance) |

### How It Works

1. The search pattern is parsed and turned into a `Bm25SearchOperator`
2. The operator loads the BM25 index from storage (using the leaflet cache when available)
3. Query text is analyzed (tokenized, lowercased, stopwords removed, stemmed)
4. The top-k results are computed using Block-Max WAND, which skips posting list segments whose upper-bound scores cannot enter the result set, then returns the highest-scoring documents
5. Results produce variable bindings (`?doc`, `?score`) that flow into subsequent where clauses
6. Subsequent patterns join against the source ledger to retrieve additional properties

## Joining with Ledger Data

The primary use case is combining search results with structured graph data:

```json
{
    "@context": {
        "ex": "http://example.org/",
        "f": "https://ns.flur.ee/db#"
    },
    "from": "docs:main",
    "where": [
        {
            "f:graphSource": "article-search:main",
            "f:searchText": "database design",
            "f:searchLimit": 20,
            "f:searchResult": { "f:resultId": "?doc", "f:resultScore": "?score" }
        },
        { "@id": "?doc", "ex:title": "?title" },
        { "@id": "?doc", "ex:author": "?author" },
        { "@id": "?doc", "ex:year": "?year" }
    ],
    "select": ["?doc", "?title", "?author", "?year", "?score"]
}
```

The BM25 search runs first, producing a set of `(?doc, ?score)` bindings. The remaining where clauses join those bindings against the source ledger to enrich results with structured data.

## Rust API

### Creating and Querying

```rust
use fluree_db_api::{Bm25CreateConfig, FlureeBuilder};
use serde_json::json;

let fluree = FlureeBuilder::memory().build_memory();

// Seed ledger
let ledger0 = fluree.create_ledger("docs:main").await?;
let tx = json!({
    "@context": { "ex": "http://example.org/" },
    "@graph": [
        { "@id": "ex:doc1", "@type": "ex:Doc", "ex:title": "Rust guide", "ex:author": "Alice" },
        { "@id": "ex:doc2", "@type": "ex:Doc", "ex:title": "Python intro", "ex:author": "Bob" }
    ]
});
let ledger = fluree.insert(ledger0, &tx).await?.ledger;

// Create index
let query = json!({
    "@context": { "ex": "http://example.org/" },
    "where": [{ "@id": "?x", "@type": "ex:Doc", "ex:title": "?title" }],
    "select": { "?x": ["@id", "ex:title"] }
});
let config = Bm25CreateConfig::new("search", "docs:main", query);
let created = fluree.create_full_text_index(config).await?;

// Query with BM25 search + ledger join
let search_query = json!({
    "@context": { "ex": "http://example.org/", "f": "https://ns.flur.ee/db#" },
    "from": "docs:main",
    "where": [
        {
            "f:graphSource": &created.graph_source_id,
            "f:searchText": "rust",
            "f:searchLimit": 10,
            "f:searchResult": { "f:resultId": "?doc", "f:resultScore": "?score" }
        },
        { "@id": "?doc", "ex:author": "?author" }
    ],
    "select": ["?doc", "?score", "?author"]
});

let result = fluree.query_connection_with_bm25(&search_query).await?;
```

### Using FlureeIndexProvider

The `FlureeIndexProvider` implements the `Bm25IndexProvider` and `Bm25SearchProvider` traits, used by the query engine for graph source resolution:

```rust
use fluree_db_api::FlureeIndexProvider;
use fluree_db_query::bm25::{Bm25IndexProvider, Bm25Scorer, Analyzer};

let provider = FlureeIndexProvider::new(&fluree);

// Load index through the provider (with optional sync and time-travel)
let index = provider
    .bm25_index("search:main", Some(ledger.t()), false, None)
    .await?;

// Direct search
let analyzer = Analyzer::english_default();
let terms = analyzer.analyze_to_strings("rust");
let term_refs: Vec<&str> = terms.iter().map(|s| s.as_str()).collect();
let scorer = Bm25Scorer::new(&index, &term_refs);
let results = scorer.top_k(10);
```

## Remote Search Service

For large indexes or multi-instance deployments, BM25 (and vector) search can be delegated to a standalone search service: the `fluree-search-httpd` binary.

> **Important:** the search service is a **separate process** with its own listen port and its own HTTP API. It is not mounted under the main Fluree server's `api_base_url` (`/v1/fluree/...`). It needs read access to the same storage and nameservice paths the main server writes to, so the typical deployment is to share a storage volume.

### Prerequisite: the index must already exist

`fluree-search-httpd` only **serves** queries against existing indexes; it does not create them. Today, BM25 and vector graph-source indexes are created via the **Rust API** (`Bm25CreateConfig` + `create_full_text_index`, or `VectorCreateConfig` + `create_vector_index`). HTTP endpoints for index creation are not yet available — see the note in [API endpoints](../api/endpoints.md#graph-source-endpoints).

The recommended workflow is:

1. Run the Fluree server (or use the Rust API directly) to create the BM25 / vector index on a shared storage path.
2. Run `fluree-search-httpd` against the same `--storage-root` and `--nameservice-path`.
3. Point clients (or the main Fluree server's `SearchDeploymentConfig`) at the search service's `/v1/search` endpoint.

### Running the Search Service

```bash
fluree-search-httpd \
  --storage-root file:///var/fluree/data \
  --nameservice-path file:///var/fluree/ns \
  --listen 0.0.0.0:9090
```

**Configuration options** (CLI flag / env var):

| Flag | Env var | Default | Description |
|------|---------|---------|-------------|
| `--storage-root` | `FLUREE_STORAGE_ROOT` | (required) | Path to Fluree storage (where indexes are persisted). `file://` prefix optional. |
| `--nameservice-path` | `FLUREE_NAMESERVICE_PATH` | (required) | Path to nameservice data. |
| `--listen` | `FLUREE_SEARCH_LISTEN` | `0.0.0.0:9090` | Address and port to bind. |
| `--cache-max-entries` | `FLUREE_SEARCH_CACHE_MAX_ENTRIES` | `100` | Maximum cached indexes. |
| `--cache-ttl-secs` | `FLUREE_SEARCH_CACHE_TTL_SECS` | `300` | Cache TTL in seconds. |
| `--max-limit` | `FLUREE_SEARCH_MAX_LIMIT` | `1000` | Maximum results per query. |
| `--default-timeout-ms` | `FLUREE_SEARCH_DEFAULT_TIMEOUT_MS` | `30000` | Default request timeout. |
| `--max-timeout-ms` | `FLUREE_SEARCH_MAX_TIMEOUT_MS` | `300000` | Maximum allowed request timeout. |

Vector search is feature-gated: build/run a binary that includes the `vector` feature to enable the vector backend. When enabled, `GET /v1/capabilities` reports `"vector"` in `supported_query_kinds`.

### Docker Deployment

Run the search service in Docker against a shared volume that the main Fluree server also mounts:

```bash
docker run -d --name fluree-search \
  -p 9090:9090 \
  -v fluree-data:/var/lib/fluree \
  -e FLUREE_STORAGE_ROOT=/var/lib/fluree/storage \
  -e FLUREE_NAMESERVICE_PATH=/var/lib/fluree/ns \
  fluree/search-httpd:latest
```

For a full Compose example showing the main server + search service sharing a volume, see [Running with Docker › Search service](../operations/docker.md#search-service-fluree-search-httpd).

### Search Protocol

The remote search service uses a JSON-based protocol on `POST /v1/search`. The request is the same shape regardless of backend; the `query.kind` discriminator selects BM25 vs. vector.

**BM25 request:**
```json
{
  "protocol_version": "1.0",
  "graph_source_id": "article-search:main",
  "query": { "kind": "bm25", "text": "rust programming" },
  "limit": 20,
  "as_of_t": 150,
  "sync": false,
  "timeout_ms": 5000
}
```

**Vector request** (requires the `vector` feature):
```json
{
  "protocol_version": "1.0",
  "graph_source_id": "doc-embeddings:main",
  "query": { "kind": "vector", "vector": [0.12, -0.34, ...], "metric": "cosine" },
  "limit": 10
}
```

A `vector_similar_to` variant takes a `to_iri` instead of an explicit vector — the server resolves the entity's embedding from the source ledger.

**Response:**
```json
{
  "protocol_version": "1.0",
  "index_t": 150,
  "hits": [
    { "iri": "http://example.org/doc1", "ledger_id": "docs:main", "score": 8.75 },
    { "iri": "http://example.org/doc2", "ledger_id": "docs:main", "score": 7.32 }
  ],
  "took_ms": 12
}
```

**Endpoints:**
- `POST /v1/search` — execute a search query (BM25 or vector)
- `GET /v1/capabilities` — protocol version, supported query kinds, max limit/timeout
- `GET /v1/health` — health check

**Time-travel:** BM25 supports `as_of_t` (the service walks the manifest to find the newest snapshot ≤ `t`). Vector indexes are head-only and reject `as_of_t`.

**Auth:** the standalone service does not enforce auth itself — front it with a reverse proxy (or a network policy) if it shouldn't be publicly reachable. The `auth_token` field on the main server's `SearchDeploymentConfig` is sent as a Bearer token, so any proxy you put in front can validate it.

### Where this fits in your architecture

Two ways to use the search service today:

1. **Direct client → search service.** Your application sends BM25 / vector requests straight to `fluree-search-httpd` and joins the resulting IRIs back to the main Fluree server's query API on the application side. This is the path that works end-to-end today and is appropriate when search traffic dominates and you want it isolated from your main Fluree process.
2. **Main Fluree server → search service (transparent delegation).** The query path inside the main server has the plumbing to consult a per-graph-source `SearchDeploymentConfig` and forward to a remote endpoint. This wiring is **not yet exposed** end-to-end through the create APIs — `Bm25CreateConfig` has no deployment builder, and the deployment field is not persisted to the nameservice config record by today's create flow. Track this as a near-term gap; until then, query the search service directly.

### Parity Guarantee

Both embedded and remote modes use identical:
- Analyzer configuration (tokenization, stemming, stopwords)
- BM25 scoring algorithm and parameters
- Time-travel and sync semantics

Queries return identical results regardless of deployment mode.

**Time-travel note**: BM25 time-travel selection is implemented by BM25 itself via a manifest/root in storage. The nameservice stores only a head pointer to the latest BM25 manifest (an opaque address) and does not store BM25 snapshot history.

## Graph Source Identity

BM25 graph sources are registered in the nameservice as `@type: "f:GraphSourceDatabase"` records:

- **ID format**: `{name}:{branch}` (e.g., `article-search:main`)
- **Name**: Cannot contain `:` (reserved for ID formatting)
- **Branch**: Defaults to `"main"`
- **Dependencies**: Tracked for the source ledger(s) the index draws from
- **Config**: Stores the indexing query and BM25 parameters (k1, b)

List ledgers and graph sources to discover BM25 graph sources:

```bash
curl http://localhost:8090/v1/fluree/ledgers
```

## Related Documentation

- [BM25 Full-Text Search](../indexing-and-search/bm25.md) - Index creation, configuration, maintenance, and storage internals
- [Graph Sources Overview](overview.md) - Graph source concepts
- [Query Datasets](../query/datasets.md) - Multi-graph queries


---

# ai/README.md

# Fluree for AI and agents

Fluree is a knowledge graph built for the way LLM agents actually work: a queryable, verifiable graph they can reason over, a Model Context Protocol surface they can call as a tool, a query output format tuned for token budgets, and persistent memory that survives across sessions.

## Why a knowledge graph for AI

Retrieval over a graph beats retrieval over flat chunks when the answer depends on *relationships*. An agent can traverse from a claim to its source, from an entity to its neighbors, or from a fact to the policy that governs it — in one query, with the joins done by the engine instead of stitched together by the model. And because every commit is immutable and content-addressed, an agent's answer can cite *exactly* the graph state it read (`t` or wallclock `iso`), and signed transactions let you prove who asserted what. See [Verifiable data](../concepts/verifiable-data.md) and [Time travel](../concepts/time-travel.md).

## The pieces

### Agent JSON — token-efficient query output

A self-describing query envelope designed for LLM consumption: datatypes declared once in a `schema` header instead of repeated per value, native JSON types for inferable values, byte-budget truncation with a `hasMore` flag, and a ready-to-run `resume` query for pagination. Request it over HTTP with `Accept: application/vnd.fluree.agent+json` and a `Fluree-Max-Bytes` budget.

→ [Output formats: Agent JSON](../query/output-formats.md#agent-json-format) · [HTTP headers](../api/headers.md)

### MCP server — Fluree as an agent tool

Fluree exposes Model Context Protocol in two distinct places:

- **Server `/mcp` endpoint** — turns a running ledger into a tool an agent can call. Exposes `sparql_query` (results returned as Agent JSON, byte-budgeted) and `get_data_model` (schema/stats discovery). Off by default; enable with `--mcp-enabled` and protect it with `--mcp-auth-trusted-issuer`. Tune the Agent JSON budget and query timeout per the config reference.

  → [MCP endpoint configuration](../operations/configuration.md#mcp-endpoint)

- **CLI `fluree mcp serve`** — a stdio MCP server for IDE agents, exposing the Fluree Memory tools (`memory_add`, `memory_recall`, `memory_update`, `memory_forget`, `memory_status`) plus `kg_query` for raw SPARQL over the memory graph.

  → [`fluree mcp`](../cli/mcp.md) · [Memory: MCP server](../memory/concepts/mcp.md)

### Fluree Memory — persistent project memory for coding agents

Long-term, searchable memory for tools like Claude Code, Cursor, and VS Code Copilot. Facts, decisions, and constraints are captured as structured memories in a local Fluree ledger, stored as plain-text TTL you can commit to git, and retrieved via ranked recall. Local-first, auditable, and tuned to keep recall small so it doesn't blow the context window.

→ [Fluree Memory](../memory/README.md) · [Getting started (per IDE)](../memory/getting-started/README.md)

### Vector and full-text search — the retrieval substrate

HNSW vector similarity and BM25 full-text search live *inside* the query engine, so semantic and keyword retrieval participate in the same joins, filters, and aggregations as the rest of your graph patterns — no separate vector store or search service to operate. This is the substrate for graph-aware RAG: retrieve by similarity, then traverse the graph from the hits.

→ [Vector search](../indexing-and-search/vector-search.md) · [BM25](../indexing-and-search/bm25.md) · [Cookbook: full-text and vector search](../guides/cookbook-search.md)

### Reasoning — derived facts without a materialization step

OWL/RDFS inference and Datalog rules run inside the query engine, so an agent querying the graph sees inferred facts (class hierarchies, transitive relationships, rule conclusions) without a separate build step. Useful when you want the model to reason over a domain ontology rather than re-deriving relationships itself.

→ [Reasoning and inference](../concepts/reasoning.md) · [Reasoning in queries](../query/reasoning.md) · [Datalog rules](../query/datalog-rules.md)

## Putting it together

A typical agent-over-Fluree stack:

1. **Ingest** your domain into a ledger as RDF (optionally with [edge annotations](../concepts/edge-annotations.md) for provenance/confidence on each statement).
2. **Index** it for [vector](../indexing-and-search/vector-search.md) and [BM25](../indexing-and-search/bm25.md) search.
3. **Expose** the ledger to the agent via the [server `/mcp` endpoint](../operations/configuration.md#mcp-endpoint), so it can call `get_data_model` to learn the schema and `sparql_query` to retrieve — getting back [Agent JSON](../query/output-formats.md#agent-json-format) sized to its context budget.
4. **Govern** access with [policy](../security/policy-in-queries.md) and prove provenance with [time travel](../concepts/time-travel.md) and [signed commits](../security/commit-signing.md).
5. For the **coding-assistant** use case, layer [Fluree Memory](../memory/README.md) so the agent remembers decisions and constraints across sessions.


---

# memory/README.md

# Fluree Memory

Persistent, searchable memory for AI coding assistants — built for real work.

Fluree Memory gives tools like Claude Code, Cursor, and VS Code Copilot a long-term project brain. Facts, decisions, and constraints are captured as structured memories, stored in a local Fluree ledger you control, and retrieved via ranked recall — either by the agent through MCP or directly from the CLI.

Because memories live in plain-text TTL files under your project (`.fluree-memory/repo.ttl` for the team, `.fluree-memory/.local/user.ttl` for you), they can be committed to git and shared across the team the same way code is. No cloud service, no opaque database, no data leaving your machine. Open the file, read it, grep it, diff it, review it in a PR.

## Design philosophy

We initiallly built Fluree Memory for us, with a goal to increase the velocity of development with LLMs, work seamlessly in a git workflow, and to reduce token usage -- in that order. We ended with a simple knowledge organization model (it started out more complex), and leaned into the speed and power of our knowledge graph database. We found most memory systems are designed for benchmarks or demos -- they optimize for recall scores on synthetic tasks, ship your data to a hosted service, or bury context in a format only the tool can read - often running LLMs over git hooks or conversation turns that can burn more tokens than your actual coding session.

Fluree Memory has been refined by running it daily across real repositories — a 37-crate Rust workspace, multi-service TypeScript apps, real teams — and iterating on what actually gets used. The schema started with five memory kinds, four sensitivity levels, six sub-type fields, and bi-temporal validity. Usage data showed that 85% of memories were facts, "architecture" covered 81% of sub-types, and most optional fields were never set. So we simplified. Three kinds. Tags instead of sub-type taxonomies. Scope instead of a redundant sensitivity axis. Fewer decisions for the agent to make on every save means more saves actually happen.

The principles that came out of this:

- **Your repo, your data.** Memories are local Turtle (TTL) files. They live alongside your code, flow through your existing review and version control, and never leave your infrastructure. There is no hosted component, no account, no telemetry.
- **Visible and auditable.** Every memory is a block of Turtle you can read in any text editor. `git diff` shows exactly what changed. `git blame` shows who (or what) added it. No black boxes.
- **Simple enough to actually use.** Three kinds — `fact`, `decision`, `constraint` — cover the real-world space. If a model has to deliberate over a five-way kind taxonomy plus sub-types on every save, it won't save. A system that gets used at 80% fidelity beats one that's theoretically perfect but sits idle.
- **Recalled, not regurgitated.** Seach with metadata re-ranking (tags, branch affinity, recency) pulls what's relevant to the current task. The agent gets a handful of targeted memories, not a dump of everything that was ever stored.
- **Optimized for context tokens.** Terse output, scoring thresholds, and explicit instructions of pagination telling the LLM whats next with enough context it can decide if useful to fetch more.
- **Iterated from production.** The schema, the recall ranking, the tool descriptions — all of it has been refined based on real agent behavior across real codebases. Features that earned usage stay. Features that didn't get cut.

## Why

Every AI coding session starts from zero. The model doesn't remember what was tried last week, which library the team chose and why, or the ten subtle gotchas that live in someone's head. You either re-explain each time, stuff it all into a `CLAUDE.md` / `AGENTS.md` that bloats context, or ship agents that repeat mistakes.

Fluree Memory is:

- **Structured**, not a wall of markdown. Memories have a kind (`fact`, `decision`, `constraint`), tags, scope, optional severity, rationale, and artifact references.
- **Recalled on demand** via BM25 keyword-scored search over memory content, with metadata-based re-ranking (tags, refs, kind, branch, recency). The agent pulls only what's relevant to the current task, keeping context small.
- **Versioned** via git — `update` modifies in place (same ID, only changed fields); `git log -p` shows the full history. Use `fluree create <name> --memory` to import git history into a time-travel-capable Fluree ledger.
- **Scoped** per-repo or per-user, so team knowledge stays shareable and personal preferences stay yours.
- **Local-first**, stored in `.fluree-memory/` as TTL — no cloud dependency, you own the data.
- **Secret-aware** — content is scanned on write against a set of known credential patterns, and matches are redacted automatically.

## Start here

- **New?** → [Quickstart](getting-started/quickstart.md) — install, init, store a memory, recall it.
- **Using Claude Code?** → [Set up Claude Code](getting-started/claude-code.md)
- **Using Cursor?** → [Set up Cursor](getting-started/cursor.md)
- **Want to understand the model?** → [What is a memory?](concepts/what-is-a-memory.md)
- **Looking for a command?** → [CLI reference](cli/README.md)

## How it fits

Fluree Memory is a feature of [Fluree DB](../docs/README.md) — installing the `fluree` CLI gives you both. If you only care about the memory tooling, you can still install and use Fluree as a single binary and never touch the rest of the database features.


---

# memory/getting-started/README.md

# Getting started

Three steps and you're running:

1. **[Install the `fluree` CLI](install.md)** — one binary, single command.
2. **[Run the quickstart](quickstart.md)** — initialize the memory store, add your first memory, recall it. 2 minutes.
3. **Wire it into your AI tool** — pick yours:
   - [Claude Code](claude-code.md)
   - [Cursor](cursor.md)
   - [VS Code (Copilot)](vscode.md)
   - [Windsurf](windsurf.md)
   - [Zed](zed.md)

Once the MCP server is configured, the AI tool gets `memory_add` and `memory_recall` tools and will start saving and retrieving memories without you having to reach for the CLI.

## What to read next

- [What is a memory?](../concepts/what-is-a-memory.md) — the kinds and when to use each
- [Repo vs user memory](../concepts/repo-vs-user.md) — how scope shapes the file layout
- [Team workflows](../guides/team-workflows.md) — sharing memories via git


---

# memory/getting-started/install.md

# Install Fluree

Fluree Memory ships as part of the `fluree` CLI. Install the binary once and you have both the database and the memory tooling.

## macOS / Linux (installer script)

```bash
curl --proto '=https' --tlsv1.2 -LsSf \
  https://github.com/fluree/db/releases/latest/download/fluree-db-cli-installer.sh | sh
```

## Homebrew (macOS / Linux)

```bash
brew install fluree/tap/fluree
```

## PowerShell (Windows)

Open PowerShell and run:

```powershell
irm https://github.com/fluree/db/releases/latest/download/fluree-db-cli-installer.ps1 | iex
```

Open a **new** PowerShell session and verify with `fluree --version`. The binary is unsigned, so Windows SmartScreen may prompt on first run — click **More info → Run anyway**.

## Pre-built binary

```bash
# Linux x86_64
curl -L https://github.com/fluree/db/releases/latest/download/fluree-db-cli-x86_64-unknown-linux-gnu.tar.xz | tar xJ

# macOS aarch64
curl -L https://github.com/fluree/db/releases/latest/download/fluree-db-cli-aarch64-apple-darwin.tar.xz | tar xJ
```

## Build from source

If you have Rust installed:

```bash
git clone https://github.com/fluree/db
cd db
cargo install --path fluree-db-cli
```

## Verify

```bash
fluree --version
fluree memory --help
```

You should see a list of `memory` subcommands: `init`, `add`, `recall`, `update`, `forget`, `status`, `export`, `import`, `mcp-install`.

Next: [quickstart](quickstart.md).


---

# memory/getting-started/quickstart.md

# Quickstart

2 minutes to your first memory.

## 1. Initialize the memory store

From the root of a project you'd like to give memory to:

```bash
cd my-project
fluree memory init
```

This creates:

- `.fluree-memory/repo.ttl` — **team memories**, meant to be committed to git
- `.fluree-memory/.local/user.ttl` — **your personal memories**, gitignored
- `.fluree-memory/.gitignore` — pre-configured to ignore `.local/` (which holds your user scope plus the MCP log)
- The `__memory` ledger inside your project's `.fluree/` store

`init` is idempotent; running it again is safe.

It will also detect any installed AI coding tools (Claude Code, Cursor, VS Code, Windsurf, Zed) and offer to wire up MCP. You can say no here and run [`fluree memory mcp-install`](../cli/mcp-install.md) later.

## 2. Add a memory

```bash
fluree memory add --kind fact \
  --text "Tests use cargo nextest, not cargo test" \
  --tags testing
```

Output:

```
Stored memory: mem:fact-01JDXYZ5A2B3C4D5E6F7G8H9J0
```

The ID is a ULID — sortable by creation time and unique across the store.

## 3. Recall it

```bash
fluree memory recall "how do I run tests"
```

Output:

```
Recall: "how do I run tests" (1 match)

1. [score: 13.0] mem:fact-01JDXYZ5A2B3C4D5E6F7G8H9J0
   Tests use cargo nextest, not cargo test
   Tags: testing
```

Recall is BM25-ranked over the memory content and tags. No embeddings, no network — fast and deterministic.

## 4. Check status

```bash
fluree memory status
```

```
Memory Store Status
  Total memories: 1
  Total tags:     1
  By kind:
    fact: 1
```

## That's the loop

Add memories as you learn things. Recall them when you need them. Commit `.fluree-memory/repo.ttl` to share team knowledge.

## Next

- **Wire to your AI tool** — [Claude Code](claude-code.md), [Cursor](cursor.md), or others — so the agent does this for you.
- **Learn the memory kinds** — [What is a memory?](../concepts/what-is-a-memory.md)
- **Understand scope** — [Repo vs user memory](../concepts/repo-vs-user.md)


---

# memory/getting-started/claude-code.md

# Set up Claude Code

Wire Fluree Memory into [Claude Code](https://claude.com/claude-code) so it saves and recalls memories for you.

## Automatic setup

Easiest path: run `init` from your project root and accept the Claude Code prompt.

```bash
cd my-project
fluree memory init
```

When you see:

```
Detected AI coding tools:
  - Claude Code

Install MCP config for Claude Code? [Y/n]
```

…press `Y`. This runs `claude mcp add` under the hood to register the Fluree Memory MCP server at local (user) scope, and appends a short section to your `CLAUDE.md` telling Claude when to use it.

If you already ran `init` and skipped it:

```bash
fluree memory mcp-install --ide claude-code
```

## What gets added

- **MCP server** registered in `~/.claude.json` — scope `local`
  - Command: `fluree mcp serve --transport stdio`
- **Project instructions** in `<repo>/CLAUDE.md` — a short block explaining the memory tools

## Verify

Restart Claude Code and start a session in the project. Ask:

> What project memories do you have?

Claude should call `memory_recall` and return whatever you've added (initially nothing).

Try:

> Remember: we use `cargo nextest` for tests, not `cargo test`.

Claude should call `memory_add` and report the stored ID.

## Troubleshooting

**The tool doesn't appear.** Confirm Claude Code sees the MCP server:

```bash
claude mcp list
```

You should see a `fluree-memory` entry. If not, re-run `fluree memory mcp-install --ide claude-code`.

**Memories aren't scoped to the repo.** The Claude Code MCP entry doesn't set `FLUREE_HOME` — the server walks up from its spawn CWD looking for a `.fluree/` directory. In normal use this matches the workspace, but if Claude Code launched the server from outside your repo, memories can land in a global store. Fix by editing `~/.claude.json` and adding an `env` block to the `fluree-memory` server entry:

```json
"env": { "FLUREE_HOME": "/absolute/path/to/your/repo/.fluree" }
```

Then restart Claude Code.

**The MCP log.** The MCP server logs to `<repo>/.fluree-memory/.local/mcp.log` (the file is truncated on each server start). Tail it if something's off:

```bash
tail -f .fluree-memory/.local/mcp.log
```


---

# memory/getting-started/cursor.md

# Set up Cursor

Wire Fluree Memory into [Cursor](https://cursor.com) so its agent mode saves and recalls memories for you.

## Automatic setup

From your project root:

```bash
cd my-project
fluree memory init
```

Accept the Cursor prompt:

```
Install MCP config for Cursor? [Y/n]
```

Or, at any time:

```bash
fluree memory mcp-install --ide cursor
```

## What gets written

- `<repo>/.cursor/mcp.json` — repo-scoped MCP server config
- `<repo>/.cursor/rules/fluree_rules.md` — a short rules file telling Cursor when to reach for `memory_recall`

```json
{
  "mcpServers": {
    "fluree-memory": {
      "type": "stdio",
      "command": "fluree",
      "args": ["mcp", "serve", "--transport", "stdio"],
      "env": {
        "FLUREE_HOME": "${workspaceFolder}/.fluree"
      }
    }
  }
}
```

`${workspaceFolder}` is a Cursor config-interpolation token — the MCP server is always launched with `FLUREE_HOME` pointing at the current project, so memories stay scoped to the repo even if Cursor spawns the process from a different working directory.

## Verify

Fully restart Cursor (Cmd-Q on macOS, not just reload window). Open the project and ask the agent:

> Recall project memories for testing.

The agent should call `memory_recall` with the tag `testing` and return what's in `.fluree-memory/repo.ttl`.

## Troubleshooting

**MCP isn't connecting.** Tail the MCP log:

```bash
tail -f .fluree-memory/.local/mcp.log
```

You should see a `client initialized` line within a few seconds of Cursor startup. If not, check `.cursor/mcp.json` exists and is valid JSON, then restart Cursor.

**Memories going to a global store on macOS.** If you see memories landing in `~/Library/Application Support/.fluree-memory/` instead of `<repo>/.fluree-memory/`, `FLUREE_HOME` isn't being honored. Re-run `fluree memory mcp-install --ide cursor` from inside the repo and restart Cursor fully.

**Rules file ignored.** Cursor picks up `.cursor/rules/*.md` on project open. After editing, reload the window.


---

# memory/getting-started/vscode.md

# Set up VS Code (Copilot)

Wire Fluree Memory into [VS Code](https://code.visualstudio.com) with GitHub Copilot Chat so it can save and recall memories through MCP.

## Automatic setup

From your project root:

```bash
fluree memory init
```

Accept the VS Code prompt, or run:

```bash
fluree memory mcp-install --ide vscode
```

## What gets written

- `<repo>/.vscode/mcp.json` — repo-scoped MCP server config (key: `servers`)
- `<repo>/.vscode/fluree_rules.md` — rules file you can reference from your prompts

```json
{
  "servers": {
    "fluree-memory": {
      "type": "stdio",
      "command": "fluree",
      "args": ["mcp", "serve", "--transport", "stdio"]
    }
  }
}
```

Unlike the Cursor config, this entry does not set `FLUREE_HOME` — VS Code normally spawns the server from the workspace root, so the walk-up logic in `fluree mcp serve` finds `.fluree/` on its own. If you need to pin the location explicitly (e.g. the server is ending up in a global store), add an `env` block pointing at the absolute path to `<repo>/.fluree/`.

## Verify

Open the project in VS Code with Copilot Chat enabled. In chat (agent mode), ask:

> Call memory_recall for "testing".

Copilot should invoke the tool and return matching memories. On first use VS Code may prompt to allow the MCP server — approve it.

## Troubleshooting

Tail `.fluree-memory/.local/mcp.log` and fully restart VS Code if something's off. If memory is landing in a global store rather than the repo, add an explicit `env.FLUREE_HOME` pointing at `<repo>/.fluree/` in `.vscode/mcp.json` and restart.


---

# memory/getting-started/windsurf.md

# Set up Windsurf

Wire Fluree Memory into [Windsurf](https://codeium.com/windsurf) (Codeium's IDE).

## Automatic setup

```bash
fluree memory init
```

Accept the Windsurf prompt, or run:

```bash
fluree memory mcp-install --ide windsurf
```

## What gets written

Windsurf uses a **global** MCP config:

- `~/.codeium/windsurf/mcp_config.json` — a `fluree-memory` entry is merged under `mcpServers`

```json
{
  "mcpServers": {
    "fluree-memory": {
      "command": "fluree",
      "args": ["mcp", "serve", "--transport", "stdio"]
    }
  }
}
```

Because the config is global, it's wired once and every Windsurf project can use it. The MCP server figures out which repo it's serving by walking up from its spawn CWD until it finds a `.fluree/` directory; in normal use Windsurf spawns it from the workspace root so this works without extra configuration. No `FLUREE_HOME` is set by default.

## Verify

Restart Windsurf and open your project. In Cascade (Windsurf's agent chat):

> Use memory_recall to find testing patterns.

The agent should invoke the tool.

## Troubleshooting

If memories end up in a global store instead of `<repo>/.fluree-memory/`, Windsurf is likely spawning the server from outside the workspace. Edit `~/.codeium/windsurf/mcp_config.json` and add an explicit absolute path:

```json
"env": { "FLUREE_HOME": "/absolute/path/to/repo/.fluree" }
```

`${workspaceFolder}` interpolation is not guaranteed in all Windsurf versions — when in doubt, use an absolute path and switch it per project.


---

# memory/getting-started/zed.md

# Set up Zed

Wire Fluree Memory into [Zed](https://zed.dev)'s agent via MCP.

## Automatic setup

```bash
fluree memory init
```

Accept the Zed prompt, or run:

```bash
fluree memory mcp-install --ide zed
```

## What gets written

- `<repo>/.zed/settings.json` — the `context_servers` key gets a `fluree-memory` entry

```json
{
  "context_servers": {
    "fluree-memory": {
      "command": "fluree",
      "args": ["mcp", "serve", "--transport", "stdio"]
    }
  }
}
```

No `FLUREE_HOME` is set by default — the MCP server walks up from Zed's spawn CWD to find the workspace's `.fluree/`. If you need to pin it explicitly, add an `env` block alongside `command`/`args` with an absolute path.

## Caveat: JSONC

Zed's `settings.json` often contains `//` comments (JSONC). `mcp-install` detects this and will skip the automatic write rather than risk corrupting your settings — it prints a hint telling you to add the block by hand.

If you'd like to pre-empt that, strip comments from `.zed/settings.json` before running `mcp-install`, or paste the block yourself.

## Verify

Restart Zed. In the agent panel:

> Recall project memories about testing.

The agent should call `memory_recall` via the `fluree-memory` context server.


---

# memory/concepts/README.md

# Concepts

Short, self-contained explanations of the ideas behind Fluree Memory. Read these once; they'll save you time when you're reading CLI reference or wiring a new IDE.

- [What is a memory?](what-is-a-memory.md) — the three kinds (`fact`, `decision`, `constraint`) and when to use each.
- [Repo vs user memory](repo-vs-user.md) — how scope decides whether a memory ends up in `repo.ttl` (shared with the team) or `.local/user.ttl` (yours).
- [Updates and forgetting](supersession.md) — how `update` modifies memories in place and how history is tracked via git.
- [Recall and ranking](recall-and-ranking.md) — how BM25 scores results and how tag / kind filters narrow them.
- [MCP server](mcp.md) — the tools Fluree Memory exposes to AI agents.
- [Secrets and sensitivity](secrets-and-sensitivity.md) — automatic redaction and scope-based privacy.


---

# memory/concepts/what-is-a-memory.md

# What is a memory?

A memory is a single structured record of something worth remembering about a project. Every memory has:

- **Content** — the text itself ("Tests use cargo nextest, not cargo test")
- **Kind** — what *sort* of thing it is
- **Tags** — free-form keywords for filtering
- **Scope** — repo (shared) or user (yours)
- **Refs** — optional file or artifact pointers
- **Timestamps** — when it was created

Everything else (severity, rationale, alternatives) is optional metadata that can appear on any kind.

## The three kinds

Memories are typed. The kind tells future-you (and future-agents) how to interpret the content.

### `fact`

Something that is objectively true about the project.

> "The indexer uses postcard encoding for on-disk format."
> "We run PostgreSQL 16 in production."
> "The BM25 code lives in `fluree-db-indexer/src/bm25.rs`."
> "Error pattern defined here -> `fluree-db-core/src/error.rs`"

Use facts liberally. They're the default and make up the bulk of a typical memory store. Use tags to categorize them (e.g. `architecture`, `dependency`, `configuration`). Facts can carry `--rationale` and `--alternatives` when you want to explain *why* something is the way it is.

### `decision`

A choice the team made, ideally with *why* and *what was considered*.

> "Use postcard for compact index encoding. **Why:** no_std compatible, smaller than bincode. **Alternatives:** bincode, CBOR, MessagePack."

Decisions are what distinguishes a project with institutional knowledge from one where people keep re-litigating settled choices. Capture them with `--rationale` and `--alternatives`:

```bash
fluree memory add --kind decision \
  --text "Use postcard for compact index encoding" \
  --rationale "no_std compatible, smaller output than bincode" \
  --alternatives "bincode, CBOR, MessagePack" \
  --refs fluree-db-indexer/
```

### `constraint`

A rule — something that *must*, *should*, or is *preferred*. Constraints carry a severity.

> **must** "Never commit secrets; use environment variables."
> **should** "Integration tests run in a real Postgres, not SQLite."
> **prefer** "Name errors with the module prefix (`QueryError`, not `Error`)."

```bash
fluree memory add --kind constraint \
  --text "Never suppress dead code with _underscore prefix; delete it" \
  --severity must \
  --tags code-style \
  --rationale "Underscore-prefixed names hide code from future discovery"
```

When an agent is about to do something, constraints are the first thing it should recall. Like facts and decisions, constraints can carry `--rationale` and `--alternatives` to explain the reasoning behind the rule.

## Which kind should I use?

| You have... | Use kind |
|---|---|
| A verifiable truth | `fact` |
| A choice and its reasoning | `decision` |
| A rule that must/should be followed | `constraint` |
| A pointer to code / a file | `fact` (with `--refs`) |
| A soft taste or convention | `fact` or `constraint --severity prefer` |

When in doubt: `fact`. The kind can always be refined later via `update`. All three kinds support `--rationale` and `--alternatives` for capturing the *why*.


---

# memory/concepts/repo-vs-user.md

# Repo vs user memory

Fluree Memory has two scopes, and they live in separate files:

| Scope | File | Git | Visible to |
|---|---|---|---|
| **repo** | `.fluree-memory/repo.ttl` | ✅ commit it | the whole team |
| **user** | `.fluree-memory/.local/user.ttl` | ❌ gitignored | just you |

Scope is set at write time (`--scope repo` or `--scope user`) and defaults to `repo`. Once set, it determines which TTL file the memory is written to.

## Layout

After `fluree memory init` inside a project:

```
my-project/
├── .fluree/                      # Fluree DB storage for the __memory ledger
├── .fluree-memory/
│   ├── .gitignore                # contents: ".local/"
│   ├── repo.ttl                  # team memories — COMMIT THIS
│   └── .local/                   # ignored by the .gitignore above
│       ├── user.ttl              # your personal memories
│       ├── mcp.log               # MCP server log
│       └── build-hash            # content hash used to detect external TTL edits
└── (your code)
```

The `.fluree-memory/.gitignore` is written by `init` and handles the split for you. Commit the whole `.fluree-memory/` directory; git will skip `.local/` automatically.

## When to use which

**Repo scope (default):**
- Facts about the codebase ("tests use cargo nextest")
- Team decisions with rationale
- Constraints everyone must follow
- File/symbol pointers via `--refs` ("X lives at Y")

**User scope:**
- Your IDE quirks
- Personal conventions the team hasn't agreed on
- Scratch notes while you're exploring
- Anything you'd be embarrassed to commit

## Changing scope after the fact

You can't move a memory between scopes directly. If you stored something as repo that should be user-only:

```bash
fluree memory forget <id>
fluree memory add --scope user --kind <kind> --text "..."
```

## Recall sees both

By default, `fluree memory recall` and the `memory_recall` MCP tool return matches from **both** scopes — your personal notes and the team's are merged in the result set. Filter with `--scope repo` or `--scope user` if you need to isolate one.

## Sharing with the team

Memory becomes a shared asset as soon as you commit `.fluree-memory/repo.ttl`. A teammate who clones the repo and runs `fluree memory init` gets the ledger populated from the committed TTL automatically — no manual import step.

Conflicts on `repo.ttl` resolve like any other text file. TTL is line-oriented per-triple, so most merges are clean; occasionally you'll see a merge mark in the middle of a memory's fields and need to pick one side.

See [Team workflows](../guides/team-workflows.md) for the full story.


---

# memory/concepts/supersession.md

# Updates and forgetting

Memories are updated **in place**. When you `update` a memory, the same ID is kept and only the changed fields are modified. History is tracked via git, not via internal versioning.

## `update` modifies in place

```bash
fluree memory update mem:fact-01JDXYZ... --text "Tests use cargo nextest with --no-fail-fast"
```

Output:

```
Updated: mem:fact-01JDXYZ...
```

The memory keeps its original ID. The TTL file is rewritten with the new content, and git records what changed:

```bash
git diff .fluree-memory/repo.ttl
```

```diff
 mem:fact-01JDXYZ a mem:Fact ;
-    mem:content "Tests use cargo nextest" ;
+    mem:content "Tests use cargo nextest with --no-fail-fast" ;
     mem:tag "cargo" ;
```

## `forget` retracts

`forget` is different from `update`. It **retracts** the memory's triples — the memory stops existing entirely.

```bash
fluree memory forget mem:fact-01JDXYZ...
```

```
Forgotten: mem:fact-01JDXYZ...
```

Rule of thumb:

| You think... | Use |
|---|---|
| "This was wrong from the start" | `forget` |
| "This was right but the world changed" | `update` |
| "I never want anyone to see this again" | `forget` |

## History via git

Both `update` and `forget` rewrite the TTL file, and git tracks the full history. To see how a memory evolved:

```bash
git log -p .fluree-memory/repo.ttl
```

This shows every change — what was added, updated, or forgotten, and when.

## Time-travel over memory history

If you want to query memory history with Fluree's time-travel capabilities, you can import your git-tracked memory history into a Fluree ledger:

```bash
fluree create my-memory-ledger --memory
```

This replays each git commit to `.fluree-memory/repo.ttl` as a Fluree transaction, giving you a full time-travel-capable ledger over your memory history. Use `--no-user` to exclude `user.ttl` from the import.


---

# memory/concepts/recall-and-ranking.md

# Recall and ranking

`recall` is how you get memories out. It's a keyword query against an inverted index with BM25 scoring — fast, local, and deterministic.

## The basics

```bash
fluree memory recall "how do I run tests"
```

The query string is tokenized and matched against each memory's **content** via a BM25-scored fulltext index. Tags, artifact refs, kind, branch, and recency contribute as **re-rank bonuses** on top of the BM25 score — they're not part of the fulltext match itself. Results are sorted by combined score (higher = better) and capped at `--limit` (default: 3).

```
Recall: "how do I run tests" (2 matches)

1. [score: 13.0] mem:fact-01JDXYZ...
   Tests use cargo nextest, not cargo test
   Tags: testing

2. [score: 8.0] mem:fact-01JDABC...
   Integration tests use assert_cmd + predicates
   Tags: testing
```

## What BM25 rewards

BM25 scores a memory's content higher when:

- **Query terms appear** in the content.
- Those terms are **rare in the overall store** — a match on "postcard" beats a match on "the".
- The matched terms are in a **shorter** memory — density matters.
- Multiple distinct terms from the query match (not the same term repeated).

There are no embeddings, no semantic matching — just lexical overlap with smart weighting. If you mean "tests" but phrase it as "unit tests" or "testing", BM25 catches that because the stems overlap; it won't catch "QA" unless the content mentions it.

## Re-rank bonuses

After BM25 produces content scores, Fluree Memory adds small bonuses:

- **Tag hit**: +10 per tag that contains a query word.
- **Artifact ref hit**: +8 per ref path that contains a query word.
- **Kind word in query**: +6 if the query mentions the memory's kind ("constraint", "decision", etc.).
- **Branch match**: +3 if the memory was captured on the current git branch.
- **Recency**: +2 for memories <7 days old, +1 for <30 days.

If BM25 returns no hits, recall falls back to metadata-only scoring using these same bonuses so a well-tagged memory can still surface on a content miss.

## Filters

Filters narrow the candidate set *before* scoring:

```bash
# Only constraints tagged "errors"
fluree memory recall "handling" --kind constraint --tags errors

# Only repo-scoped memories
fluree memory recall "deployment" --scope repo

# Page through results
fluree memory recall "tests" --limit 10 --offset 10
```

Common filter recipes:

| You want… | Flags |
|---|---|
| Team-only (ignore personal) | `--scope repo` |
| Just the hard rules | `--kind constraint` |
| Just the decisions with reasoning | `--kind decision` |
| Pointers to code | `--kind fact --tags <domain>` (with `--refs`) |

## Output formats

```bash
fluree memory recall "tests"                 # text — for humans
fluree memory recall "tests" --format json   # JSON — for scripts
fluree memory recall "tests" --format context  # XML — for LLM injection
```

The `context` format produces a compact XML block designed to be pasted into an agent's context window:

```xml
<memory-context>
  <memory id="mem:fact-01JDXYZ..." kind="fact" score="13.0">
    <content>Tests use cargo nextest, not cargo test</content>
    <tags>testing</tags>
  </memory>
  <pagination shown="1" offset="0" total_in_store="13" />
</memory-context>
```

When results are cut off, the pagination element embeds a human-readable hint telling the agent how to get more:

```xml
<pagination shown="3" offset="0" limit="3" total_in_store="13">
  Results 1–3. Use offset=3 to retrieve more.
</pagination>
```

This pattern is why Fluree Memory is practical to use with an agent: a small, ranked slice goes into context, and the agent can ask for more if the top hits aren't enough.

## How this compares to other approaches

| Approach | Cost | Quality | Works offline |
|---|---|---|---|
| BM25 (Fluree Memory) | free, instant | high for keyword overlap | yes |
| Embedding search | paid + latency | high for paraphrase | usually no |
| Stuff-it-all-in-CLAUDE.md | free | context blow-up | yes |

For developer memory — where the agent knows the words for what it's looking for — BM25 is a very good fit. If you later want semantic recall, Fluree DB itself ships a [vector search](../../docs/indexing-and-search/vector-search.md) feature that the memory store could layer on.


---

# memory/concepts/mcp.md

# MCP server

Fluree Memory exposes its functionality over [Model Context Protocol](https://modelcontextprotocol.io) so AI coding agents can use it natively. The MCP server is bundled with the `fluree` CLI — no separate install.

## Start it manually

```bash
fluree mcp serve --transport stdio
```

In practice you never start it manually — your IDE launches it. `fluree memory mcp-install` writes the IDE-specific config that does the spawning. See [mcp-install](../cli/mcp-install.md) for the per-IDE details.

## Tools exposed

The server exposes these tools to the agent:

### `memory_recall`

Search for relevant memories.

```json
{
  "name": "memory_recall",
  "arguments": {
    "query": "how do I run tests",
    "limit": 5,
    "offset": 0,
    "kind": "fact",
    "tags": ["testing"],
    "scope": "repo"
  }
}
```

Returns XML context-formatted output (see [Recall and ranking](recall-and-ranking.md)).

### `memory_add`

Store a new memory. The content field is named `content` (not `text`).

```json
{
  "name": "memory_add",
  "arguments": {
    "kind": "fact",
    "content": "Tests use cargo nextest, not cargo test",
    "tags": ["testing"],
    "scope": "repo"
  }
}
```

Other optional arguments: `refs`, `severity`, `rationale`, `alternatives`. Returns the new memory ID.

### `memory_update`

Patch an existing memory in place. The memory keeps its ID; only the fields you pass are changed. Use `content` (not `text`) for the new body.

```json
{
  "name": "memory_update",
  "arguments": {
    "id": "mem:fact-01JDXYZ...",
    "content": "Tests use cargo nextest with --no-fail-fast"
  }
}
```

Also accepts `tags`, `refs`, `rationale`, `alternatives`.

### `memory_forget`

Retract a memory permanently.

```json
{
  "name": "memory_forget",
  "arguments": { "id": "mem:fact-01JDXYZ..." }
}
```

### `memory_status`

Return a summary of the store — totals by kind and a preview of recent memories. Agents are encouraged to call this first to discover what topics to query.

### `kg_query`

Run a raw SPARQL SELECT against the `__memory` ledger. Advanced escape hatch — prefer `memory_recall` for ranked search.

```json
{
  "name": "kg_query",
  "arguments": {
    "query": "PREFIX mem: <https://ns.flur.ee/memory#> SELECT ?id ?content WHERE { ?id a mem:Constraint ; mem:content ?content } LIMIT 20"
  }
}
```

## Where the store lives

When the MCP server starts, it picks its Fluree directory the same way the CLI does:

1. If `$FLUREE_HOME` is set, that directory is used (unified mode).
2. Otherwise it walks up from the spawn CWD looking for an existing `.fluree/`.
3. If neither is found, it falls back to the platform's global config/data directories.

When the server is in unified mode (cases 1 and 2), the memory store lives in `<dir>/../.fluree-memory/` and is shared with the CLI. In global mode, file-based sync is disabled and memories live only in the global ledger.

This matters for IDE integrations: the **Cursor** config that `mcp-install` writes explicitly sets `FLUREE_HOME=${workspaceFolder}/.fluree` so memory stays scoped to the current repo regardless of Cursor's CWD. The other supported IDEs (Claude Code, VS Code, Windsurf, Zed) rely on the spawn CWD plus the walk-up behavior — which normally works, but can land in a global store if the IDE spawns the MCP server from outside the repo. If you see that, set `FLUREE_HOME` manually in the MCP config or re-run `mcp-install` from inside the repo root.

## The rules file

Alongside the MCP server, `mcp-install` writes (or appends to) a short rules file for IDEs that support one:

| IDE | Rules file |
|---|---|
| Claude Code | Short section appended to `<repo>/CLAUDE.md` |
| Cursor | `<repo>/.cursor/rules/fluree_rules.md` |
| VS Code | `<repo>/.vscode/fluree_rules.md` |
| Windsurf, Zed | None written — you can add your own guidance manually |

The file tells the agent *when* to reach for memory tools — e.g. at the start of a task (`memory_recall` first), after capturing something reusable (`memory_add`), and not to re-ask the user for things already memorized. You can edit it to customize the agent's instincts; see [Customizing the rules file](../guides/rules-file.md).


---

# memory/concepts/secrets-and-sensitivity.md

# Secrets and sensitivity

Memory is meant to be written freely and committed to git. That only works if secrets never land in there.

## Automatic redaction

Every `memory_add` / `fluree memory add` runs the input through a secret detector before storage. If the content matches patterns for API keys, passwords, tokens, or connection strings, the sensitive substrings are replaced with `[REDACTED]` and a warning is printed:

```
  warning: secrets detected in content — storing redacted version.
  Original content contained sensitive data that was replaced with [REDACTED].
Stored memory: mem:fact-01JDXYZ...
```

Patterns covered include:

- AWS access key IDs (`AKIA…`)
- GitHub personal access tokens (`ghp_…`, `gho_…`, `ghu_…`, `ghs_…`, `ghr_…`)
- OpenAI keys (`sk-…`) and Anthropic keys (`sk-ant-…`)
- Fluree API keys (`flk_…`)
- Generic `api_key=…` / `apikey: …` assignments
- `password=…` / `passwd: …` assignments
- Connection strings with inline credentials (`postgres://`, `mysql://`, `mongodb://`, `redis://`, `amqp://` containing `user:pass@host`)
- PEM private keys (`-----BEGIN … PRIVATE KEY-----`)
- Bearer tokens (`Bearer eyJ…`)
- JWT tokens (three base64 segments separated by dots)

Redaction preserves enough context that the memory still makes sense (e.g. "Use the API key `[REDACTED]` from 1Password") while the actual value never reaches the TTL file.

The detector is pattern-based, not entropy-based — well-disguised secrets outside these patterns can still slip through. Treat redaction as a safety net, not a guarantee.

## Scope as the privacy boundary

Memory visibility is controlled by `scope` (`repo` or `user`), not by a separate sensitivity level. Repo-scoped memories live in `.fluree-memory/repo.ttl` and are committed to git, so they're visible to anyone with repo access. User-scoped memories live in `.fluree-memory/.local/user.ttl`, which is gitignored.

If something is client-specific or team-internal, put it in `user` scope or use a private sub-repo. The scope mechanism plus secret-detection on ingest handles what a separate sensitivity field used to.

## What if I slip?

If something slipped past the detector and into `repo.ttl` before you noticed:

1. `fluree memory forget <id>` — retracts the memory.
2. Run `git log -p .fluree-memory/repo.ttl` and use `git filter-repo` (or the BFG) to scrub the history if the value leaked there too.
3. Rotate the credential at the source. Redaction in memory doesn't rotate keys.

Treat this the same way you'd treat accidentally committing `.env` — the git history is the hard part, the file is the easy part.


---

# memory/guides/README.md

# Guides

Task-oriented walkthroughs for common situations.

- [Team workflows: sharing memory via git](team-workflows.md) — how `repo.ttl` becomes shared knowledge and how conflicts resolve.
- [Customizing the rules file](rules-file.md) — tuning what your AI tool does with memory.
- [Migrating from plain-markdown memory](migrating.md) — moving from `CLAUDE.md` / `AGENTS.md` style blobs to structured memories.

Looking for end-to-end setup instead? See [Getting started](../getting-started/README.md).


---

# memory/guides/team-workflows.md

# Team workflows: sharing memory via git

The whole point of `repo.ttl` is that memory becomes a team asset — captured once by whoever learns it, available to every teammate and every AI agent forever.

## The happy path

1. **Someone runs `fluree memory init`** in the repo and commits `.fluree-memory/` (minus the gitignored `.local/`).
2. **Teammates pull** and run `fluree memory init` once. The init picks up the committed `repo.ttl` and populates the ledger from it. No manual import.
3. **As people add memories**, `.fluree-memory/repo.ttl` changes in the working tree. Commit it like any other file.
4. **Pulls bring in new memories automatically** — `fluree memory recall` (and the MCP server) read the ledger, which stays in sync with the TTL file.

That's it. No server, no sync daemon, no API tokens. Git is the sync mechanism.

## What to commit

✅ Commit:
- `.fluree-memory/repo.ttl`
- `.fluree-memory/.gitignore`
- Any IDE config MCP-install created: `.cursor/mcp.json`, `.cursor/rules/fluree_rules.md`, `.vscode/mcp.json`, `.vscode/fluree_rules.md`, `.zed/settings.json`

❌ Don't commit:
- `.fluree-memory/.local/user.ttl` — your personal memories *(handled by `.fluree-memory/.gitignore`)*
- `.fluree-memory/.local/mcp.log` — noisy and personal *(handled by `.fluree-memory/.gitignore`)*
- `.fluree/` — the Fluree storage dir, can be re-hydrated from `repo.ttl` *(add this to your project's root `.gitignore` — `.fluree-memory/.gitignore` only covers its own subtree)*

## Reviewing memory in PRs

Treat `repo.ttl` changes like documentation changes in code review:

- **New memory?** Is the kind right? Is the wording accurate? Are the tags useful?
- **Updated memory?** Is the new content better (not just different)?
- **Forgot memory?** Was that really wrong, or should it have been updated instead?

Memories are serialized as subject blocks with one predicate per line, so most diffs are readable.

## Merge conflicts

Memories in `repo.ttl` are sorted by `(branch, id)` — memories from the same git branch cluster together, and different branches land in different regions of the file. This means two feature branches that each add memories will almost never conflict, because their blocks insert at different positions in the file.

The branch name is captured automatically when a memory is created, so memories from `feature/auth` sort separately from memories created on `feature/indexer`. Within each branch group, memories are ordered chronologically (ULID encodes creation time).

When conflicts do occur, they're usually because two branches modified the same existing memory (via `update`) or both worked on the same branch. These are typically clean to resolve:

```ttl
<<<<<<< HEAD
mem:fact-01JD... a mem:Fact ;
    mem:content "Tests use cargo nextest" ;
    mem:tag "cargo" ;
    mem:tag "testing" ;
    ...
=======
mem:fact-01JD... a mem:Fact ;
    mem:content "Tests use cargo nextest with --no-fail-fast" ;
    mem:tag "testing" ;
    ...
>>>>>>> their-branch
```

Pick the version you want or combine them, then re-run `fluree memory status` to make sure the store parses cleanly. If the merged file is genuinely messy, a cleaner path is to accept one side wholesale and then apply the other side's changes via `fluree memory add` / `update` on top.

## Onboarding a new teammate

When someone new clones the repo:

```bash
git clone git@github.com:team/project
cd project
fluree memory init
```

After init, they can immediately:

```bash
fluree memory recall "testing" -n 10
```

…and get everything the team has captured. No setup beyond installing `fluree`.

## Going further

- Keep PR review short by tagging memories with their **domain** (`auth`, `indexer`, `docs`, etc.) so reviewers can filter.
- Use `constraint --severity must` sparingly — they're the "policy layer" of memory. Prefer `should` or `prefer` for taste.
- Periodically `fluree memory status` and prune stale memories with `forget`. The store should feel curated, not a dumping ground.


---

# memory/guides/rules-file.md

# Customizing the rules file

When you run `fluree memory mcp-install`, a short "rules file" gets written alongside the MCP server config. This file tells your AI tool *when and how* to use the memory tools — things the tool definitions alone don't express.

## Where it lives

| IDE | Rules file |
|---|---|
| Claude Code | Section appended to `<repo>/CLAUDE.md` |
| Cursor | `<repo>/.cursor/rules/fluree_rules.md` |
| VS Code | `<repo>/.vscode/fluree_rules.md` |
| Windsurf | Not written — add your own guidance to Windsurf's memory / rules UI |
| Zed | Not written — add your own guidance via Zed's assistant settings |

The canonical source for the default text lives in `fluree-db-memory/rules/fluree_rules.md` in the repo; the Cursor and VS Code installers copy it verbatim. The Claude Code installer appends a short variant directly to `CLAUDE.md`. Windsurf and Zed don't have a conventional per-project rules-file slot that `mcp-install` targets automatically — the paragraph below is a reasonable starting point if you want to paste one in yourself.

## What the default says

A minimal set of instructions along these lines:

> **Before starting a task:** call `memory_recall` with a query describing what you're about to do. Review the top matches for constraints, decisions, and relevant facts.
>
> **After learning something reusable:** call `memory_add` with the appropriate kind:
> - `fact` — verifiable truths about the codebase (use `--refs` for file pointers)
> - `decision` — choices with rationale (use `--rationale`)
> - `constraint` — rules with severity (use `--severity must/should/prefer`)
>
> **Don't re-ask the user** for things that are already in memory.

## Customizing

Edit the file freely. Common tweaks:

- **Add domain-specific guidance**: "When working on the indexer, always recall with the `indexer` tag first."
- **Tighten the defaults**: "Only call `memory_add` for memories that will apply in future sessions — not for task-specific scratch."
- **Shape the kinds**: "Use `fact` with `--refs` when the memory is really a pointer to a file or symbol."

## Reloading

- **Cursor / VS Code**: reload the window after editing.
- **Claude Code**: appending to `CLAUDE.md` takes effect on the next session.
- **Zed**: agent reads settings on connection — reload.

## Keeping team customizations shared

If you edit the rules file and like what you got, commit it. Teammates get your tuning automatically on their next pull. The rules file is just markdown — treat it like any other piece of team guidance.


---

# memory/guides/migrating.md

# Migrating from plain-markdown memory

Many teams start with one big markdown file that their AI tool reads on every session — `CLAUDE.md`, `AGENTS.md`, `.cursorrules`, `.windsurfrules`, or a section in `README.md`. These files work until they don't: they bloat context, mix levels (architectural rules next to "the CI flag is --all-features"), and rot silently.

Here's a pragmatic migration from that world to structured memory.

## Phase 1: leave the markdown alone

You don't have to delete anything to start using Fluree Memory. Add memories for **new** things you learn while keeping the old file around. After a week or two of active use, you'll have a sense of which things belong where.

```bash
fluree memory init
# ...work, capture things as they come up...
fluree memory add --kind constraint --severity must \
  --text "All public fns must have doc comments" --tags code-style
```

## Phase 2: categorize the markdown file

Open the old file and go paragraph by paragraph. For each chunk, ask:

| Chunk type | Where it goes |
|---|---|
| High-level overview / architecture prose | Stays in markdown (README, ARCHITECTURE.md) |
| Rules ("do this", "don't do that") | → `constraint` memories with `--severity` |
| Choices + reasoning | → `decision` memories with `--rationale` |
| Named quirks / gotchas | → `fact` memories |
| "Look here for X" | → `fact` memories with `--refs` |
| Personal preferences | → `fact` memories (`--scope user` usually) |

The markdown file that's left after this should be genuinely about framing — the 30-second project tour — not a knowledge base.

## Phase 3: move the categorized chunks

Turn each chunk into a `memory add` call. Tag consistently so things group later:

```bash
fluree memory add --kind constraint --severity must \
  --text "Never commit secrets; use environment variables" \
  --tags security,secrets

fluree memory add --kind decision \
  --text "Use postcard for index encoding" \
  --rationale "no_std compatible, smaller than bincode" \
  --alternatives "bincode, CBOR, MessagePack" \
  --refs fluree-db-indexer/ \
  --tags indexer,encoding

fluree memory add --kind fact \
  --text "Error pattern defined here" \
  --refs fluree-db-core/src/error.rs \
  --tags errors
```

If you want to script it, pipe content into `fluree memory add` on stdin (with `--kind` / `--tags` set per-line). `add` reads stdin when `--text` is omitted:

```bash
echo "The index format uses postcard encoding" \
  | fluree memory add --kind fact --tags indexer
```

## Phase 4: trim the old file

Once the chunks are in memory, delete them from the markdown. What's left is your high-level orientation doc, which is fine.

**Leave a pointer at the top:**

```markdown
> Detailed conventions, rules, and decisions are in Fluree Memory.
> Use `memory_recall` from an MCP-enabled IDE, or `fluree memory recall "..."` from the shell.
```

## Phase 5: review

Run `fluree memory status` and `fluree memory recall "" -n 50` to eyeball everything. Look for:

- Duplicates — memories that say nearly the same thing with different wording.
- Mis-categorized kinds — a "decision" with no rationale is really a `fact`.
- Over-long content — memories should be paragraphs at most, not pages. Break up if needed.

## Why this is worth doing

| Plain markdown | Structured memory |
|---|---|
| Entire file loaded every session | Only relevant matches loaded |
| No filtering | Filter by kind, tag, scope |
| No history | Full history via `git log -p` |
| Hard to share a slice | `export` + `jq` / curated `recall` |
| Drifts silently | `status` visibility + curation flow |

You get a knowledge base the team can actually maintain — and that costs fewer tokens per session than the markdown file it replaces.


---

# memory/cli/README.md

# CLI reference

The `fluree memory` subcommands, alphabetically-ish.

| Command | Purpose |
|---|---|
| [`init`](init.md) | Create the memory store and optionally configure MCP for detected AI tools |
| [`add`](add.md) | Store a new memory |
| [`recall`](recall.md) | Search and rank relevant memories |
| [`update`](update.md) | Update an existing memory in place |
| [`forget`](forget.md) | Retract a memory permanently |
| [`status`](status.md) | Summary of the store (totals, tags, kinds) |
| [`export` / `import`](export-import.md) | Round-trip memories as JSON |
| [`mcp-install`](mcp-install.md) | Install MCP config for an IDE |

Several subcommands take a `--format` flag (`text` for humans, `json` for scripts, and `context` on `recall` for XML intended for LLM injection). The default is always `text`.

## The common options

A few flags show up across many subcommands:

| Flag | Default | Where |
|---|---|---|
| `--scope <repo\|user>` | `repo` | `add`; filter on `recall` |
| `--tags <t1,t2>` | none | `add`, `update`; filter on `recall` |
| `--kind <kind>` | `fact` on `add` | `add`; filter on `recall` |
| `--format <text\|json>` | `text` | `add`, `update` |
| `--format <text\|json\|context>` | `text` | `recall` (XML `context` is for LLM injection) |

See [What is a memory?](../concepts/what-is-a-memory.md) for the kind taxonomy.

## Environment

| Variable | Effect |
|---|---|
| `FLUREE_HOME` | When set, the CLI and MCP server use this path as the unified Fluree directory. If unset, both walk up from CWD looking for an existing `.fluree/`; if none is found, they fall back to a platform-global config/data directory. |

Set `FLUREE_HOME=<repo>/.fluree` if you need to force repo-scoped operation from a shell that starts elsewhere. Among the IDE integrations, only the Cursor MCP config sets this automatically via `${workspaceFolder}`; the others rely on the walk-up behavior from spawn CWD.


---

# memory/cli/init.md

# fluree memory init

Initialize the memory store and optionally configure MCP for detected AI coding tools. Idempotent — safe to run repeatedly.

```bash
fluree memory init [OPTIONS]
```

## Options

| Option | Description |
|---|---|
| `--yes`, `-y` | Auto-confirm all MCP installations (non-interactive) |
| `--no-mcp` | Skip AI tool detection and MCP configuration entirely |

## What init does

1. **Creates the `__memory` ledger** inside `<repo>/.fluree/` and transacts the memory schema.
2. **Creates `.fluree-memory/`** at the project root:
   - `repo.ttl` — team memories (empty to start; meant to be committed)
   - `.local/user.ttl` — your personal memories (gitignored)
   - `.gitignore` — pre-configured with `.local/` (which holds your user scope and the MCP log)
3. **Migrates existing memories** — if the ledger already has memories (e.g. from before the TTL file layout), they're exported into the appropriate `.ttl` file.
4. **Detects AI coding tools** (Claude Code, Cursor, VS Code, Windsurf, Zed) and offers to install MCP for each.

## Detection signals

Each IDE is detected via a combination of these signals — any one is enough:

| Signal | Where it lives |
|---|---|
| Binary on `PATH` | `claude`, `code`, `cursor`, `windsurf`, `zed` |
| macOS app bundle | `/Applications/<App>.app` *and* `~/Applications/<App>.app` |
| Home-dir marker | `~/.claude`, `~/.claude.json`, `~/.cursor`, `~/.vscode`, `~/.codeium/windsurf`, `~/.zed`, `~/.config/zed` |
| User-config dir | `<config>/Code`, `<config>/Cursor`, `<config>/Windsurf`, `<config>/Zed` (where `<config>` is `~/.config` on Linux, `~/Library/Application Support` on macOS, `%APPDATA%` on Windows) |

If `init` reports **"No AI coding tools detected"** but you do have one installed, the most likely cause is that none of those markers exist yet (e.g. a fresh install that's never been launched). You can install MCP for it directly:

```bash
fluree memory mcp-install --ide <tool>
# tool ∈ claude-code, cursor, vscode, windsurf, zed
```

## Example

```bash
$ fluree memory init

Memory store initialized at /path/to/project/.fluree-memory

Repo memories are stored in .fluree-memory/repo.ttl (git-tracked).
Commit this directory to share project knowledge with your team.

Detected AI coding tools:
  - Claude Code (already configured)
  - Cursor
  - VS Code (Copilot) (already configured)

Install MCP config for Cursor? [Y/n] Y
  Installed: .cursor/mcp.json
  Installed: .cursor/rules/fluree_rules.md

Configured 1 tool.
```

With `--yes`: auto-confirms all installations without prompting. In a non-interactive shell (piped stdin) without `--yes`, MCP installation is skipped with a message.

## Re-running

`init` is safe to run again. It won't re-create or overwrite files that already exist; it just:

- Checks that the ledger and schema are current (migrating if not).
- Detects IDEs you've since installed and offers to configure them.
- Leaves existing memories untouched.

Run it again after:
- Installing a new AI tool you want to wire up.
- Cloning a repo someone else set up — `init` will pick up the committed `repo.ttl` into the ledger automatically.


---

# memory/cli/add.md

# fluree memory add

Store a new memory.

```bash
fluree memory add [OPTIONS]
```

## Options

| Option | Description |
|---|---|
| `--kind <KIND>` | `fact` (default), `decision`, `constraint` |
| `--text <TEXT>` | Content text (or provide via stdin) |
| `--tags <T1,T2>` | **Required.** Comma-separated tags — the primary recall signal |
| `--refs <R1,R2>` | Comma-separated file/artifact references |
| `--severity <SEV>` | For constraints: `must`, `should`, `prefer` |
| `--scope <SCOPE>` | `repo` (default) or `user` |
| `--rationale <TEXT>` | Why — the reasoning behind this memory (any kind) |
| `--alternatives <TEXT>` | Alternatives considered (any kind) |
| `--format <FMT>` | `text` (default) or `json` |

## Examples

```bash
# A simple fact
fluree memory add --kind fact \
  --text "Tests use cargo nextest" \
  --tags testing,cargo

# A hard constraint with rationale
fluree memory add --kind constraint \
  --text "Never suppress dead code with an underscore prefix" \
  --tags code-style \
  --severity must \
  --rationale "Underscore-prefixed names hide code from future discovery"

# From stdin (useful for piping from other tools)
echo "The index format uses postcard encoding" \
  | fluree memory add --kind fact --tags indexer

# A decision with full context
fluree memory add --kind decision \
  --text "Use postcard for compact index encoding" \
  --rationale "no_std compatible, smaller output than bincode" \
  --alternatives "bincode, CBOR, MessagePack" \
  --refs fluree-db-indexer/

# A fact pointing to a file (use --refs for artifact pointers)
fluree memory add --kind fact \
  --text "Error pattern defined here" \
  --refs fluree-db-core/src/error.rs \
  --tags errors

# A personal convention, user-scoped
fluree memory add --kind fact \
  --text "Always run clippy with --all-features" \
  --scope user \
  --tags code-style
```

## Output

Default (`text`):

```
Stored memory: mem:fact-01JDXYZ5A2B3C4D5E6F7G8H9J0
```

`json`:

```json
{
  "id": "mem:fact-01JDXYZ5A2B3C4D5E6F7G8H9J0",
  "kind": "fact",
  "scope": "repo",
  "created_at": "2026-04-14T16:45:12Z"
}
```

## Secret detection

If the content matches a known secret pattern (AWS keys, GitHub tokens, password-bearing URLs, etc.), the sensitive portions are replaced with `[REDACTED]` before storage and a warning is printed. See [Secrets and sensitivity](../concepts/secrets-and-sensitivity.md).

## Scope and file placement

| `--scope repo` (default) | Writes to `.fluree-memory/repo.ttl` — committable |
| `--scope user` | Writes to `.fluree-memory/.local/user.ttl` — gitignored |

See [Repo vs user memory](../concepts/repo-vs-user.md).

## See also

- [`recall`](recall.md) — search stored memories
- [`update`](update.md) — update an existing memory in place
- [What is a memory?](../concepts/what-is-a-memory.md) — choosing the right kind


---

# memory/cli/recall.md

# fluree memory recall

Search and retrieve relevant memories ranked by BM25 score.

```bash
fluree memory recall <QUERY> [OPTIONS]
```

## Arguments

| Argument | Description |
|---|---|
| `<QUERY>` | Natural-language search query (keyword-matched, not semantic) |

## Options

| Option | Description |
|---|---|
| `-n`, `--limit <N>` | Max results per page (default: 3) |
| `--offset <N>` | Skip the first N results — use for pagination (default: 0) |
| `--kind <KIND>` | Filter to a specific memory kind |
| `--tags <T1,T2>` | Filter to memories with these tags |
| `--scope <SCOPE>` | Filter by `repo` or `user` |
| `--format <FMT>` | `text` (default), `json`, or `context` (XML for LLM) |

## Examples

```bash
# Basic recall — returns top 3
fluree memory recall "how to run tests"

# Page through a longer result set
fluree memory recall "how to run tests" --offset 3
fluree memory recall "error handling" -n 10

# Narrow with filters
fluree memory recall "error handling" --kind constraint --tags errors
fluree memory recall "deployment" --scope repo

# XML output designed for LLM context injection
fluree memory recall "testing patterns" --format context
```

## Output

### `text`

```
Recall: "how to run tests" (2 matches)

1. [score: 13.0] mem:fact-01JDXYZ...
   Tests use cargo nextest
   Tags: testing, cargo

2. [score: 8.0] mem:fact-01JDABC...
   Integration tests use assert_cmd + predicates
   Tags: testing

  (showing results 1–3; use --offset 3 for more)
```

### `json`

```json
{
  "query": "how to run tests",
  "memories": [
    {
      "memory": {
        "id": "mem:fact-01JDXYZ...",
        "kind": "fact",
        "content": "Tests use cargo nextest",
        "tags": ["testing", "cargo"],
        "scope": "repo",
        "created_at": "2026-02-22T14:00:00Z"
      },
      "score": 13.0
    }
  ],
  "total_count": 13
}
```

`total_count` is the total number of memories in the store, not the number of matches — useful for UI context but not for pagination math.

### `context` (XML for LLM injection)

```xml
<memory-context>
  <memory id="mem:fact-01JDXYZ..." kind="fact" score="13.0">
    <content>Tests use cargo nextest</content>
    <tags>testing, cargo</tags>
  </memory>
  <pagination shown="1" offset="0" total_in_store="13" />
</memory-context>
```

When results are cut off, the pagination element embeds a hint:

```xml
<pagination shown="3" offset="0" limit="3" total_in_store="13">
  Results 1–3. Use offset=3 to retrieve more.
</pagination>
```

## How ranking works

See [Recall and ranking](../concepts/recall-and-ranking.md) for the full story — BM25 over content plus metadata bonuses for tag, ref, kind, branch, and recency matches. Filters (`--kind`, `--tags`, `--scope`) narrow the candidate set. All local, deterministic, and offline.

## See also

- [`add`](add.md) — store a new memory
- [`status`](status.md) — store summary
- [Recall and ranking](../concepts/recall-and-ranking.md)


---

# memory/cli/update.md

# fluree memory update

Update an existing memory in place. The memory keeps the same ID — only the changed fields are modified. History is tracked via git.

```bash
fluree memory update <ID> [OPTIONS]
```

## Options

| Option | Description |
|---|---|
| `--text <TEXT>` | New content text |
| `--tags <T1,T2>` | New tags (replaces all existing) |
| `--refs <R1,R2>` | New artifact refs (replaces all existing) |
| `--format <FMT>` | `text` (default) or `json` |

## Example

```bash
fluree memory update mem:fact-01JDXYZ... \
  --text "Tests use cargo nextest with --no-fail-fast"
```

Output:

```
Updated: mem:fact-01JDXYZ...
```

The TTL file is rewritten with the updated content. Use `git diff` to see what changed, or `git log -p .fluree-memory/repo.ttl` to review the full history.

## See also

- [`forget`](forget.md) — retract instead of update
- [Updates and forgetting](../concepts/supersession.md) — the update model


---

# memory/cli/forget.md

# fluree memory forget

Retract a memory permanently. Unlike [`update`](update.md), `forget` removes the memory entirely — it stops existing.

```bash
fluree memory forget <ID>
```

Output:

```
Forgotten: mem:fact-01JDXYZ...
```

## When to forget vs. update

| You think… | Use |
|---|---|
| "This was wrong from the start" | `forget` |
| "This was right but the world changed" | [`update`](update.md) |
| "I never want anyone to see this again" | `forget` |

See [Updates and forgetting](../concepts/supersession.md) for more detail.

## Forgetting accidentally-committed secrets

Forgetting removes the memory from the ledger and rewrites `repo.ttl` (or `.local/user.ttl`) immediately, so the deletion shows up in your next `git diff`. If a secret value also ended up in **git history**, you need to scrub the history separately — see [Secrets and sensitivity](../concepts/secrets-and-sensitivity.md#what-if-i-slip).


---

# memory/cli/explain.md

# Memory history via git

The `fluree memory explain` command has been removed. Memory history is now tracked via git.

## Viewing history

Since updates modify memories in place and the TTL file is rewritten on each change, `git log` shows the full history:

```bash
# Full history of all memory changes
git log -p .fluree-memory/repo.ttl

# Search for changes to a specific memory ID
git log -p -S "mem:fact-01JDXYZ" .fluree-memory/repo.ttl

# Compact one-line summary
git log --oneline .fluree-memory/repo.ttl
```

## Time-travel via Fluree

For richer querying over memory history, import your git history into a Fluree ledger:

```bash
fluree create my-memory-ledger --memory
```

Each git commit becomes a Fluree transaction, enabling time-travel queries over the full evolution of your project's memory.

See [Updates and forgetting](../concepts/supersession.md) for the update model.


---

# memory/cli/status.md

# fluree memory status

Show a summary of the memory store.

```bash
fluree memory status
```

Output:

```
  Directory: /path/to/project/.fluree-memory
Memory Store: 12 memories, 25 tags
  Kinds: 7 fact, 2 decision, 3 constraint

Recent memories:
  - [fact] Tests use cargo nextest, not cargo test [cargo, testing]
    ID: mem:fact-01JDXYZ...
  - [decision] Use postcard for compact index encoding [encoding, indexer]
    ID: mem:decision-01JDABC...

Use memory_recall with specific keywords from above to search.
```

`status` counts all memories in the store. The "Recent memories" list is included to help agents (or you) pick good keywords for `memory_recall`.

## When it's useful

- Confirming init worked and the store is live.
- Sanity-checking after an import.
- Quick "how much does this project remember?" check.

For per-memory detail, use [`recall`](recall.md) with a broad query (e.g. `fluree memory recall "" -n 100`) or [`export`](export-import.md).


---

# memory/cli/export-import.md

# fluree memory export / import

Round-trip memories as JSON.

## export

Write all memories to stdout as a JSON array.

```bash
fluree memory export > memories.json
```

`export` takes no options — it emits every memory, both scopes included. To get a single scope, filter with `jq` or use `recall` with `--scope` and a permissive limit.

Output is a flat array of full memory objects:

```json
[
  {
    "id": "mem:fact-01JDXYZ...",
    "kind": "fact",
    "content": "Tests use cargo nextest",
    "tags": ["testing", "cargo"],
    "scope": "repo",
    "severity": null,
    "artifact_refs": [],
    "branch": "main",
    "created_at": "2026-02-22T14:00:00Z"
  }
]
```

## import

Load memories from a JSON file produced by `export` (or a hand-crafted array of the same shape).

```bash
fluree memory import memories.json
```

Import is additive — every entry in the file is re-transacted into the ledger, with secret-detection applied to `content`, `rationale`, and `alternatives`. IDs and timestamps from the source file are preserved. There is no dedup step, so importing the same file twice will double-insert; `forget` the existing entries first (or import into a freshly-initialized store) if that's not what you want.

## When to use

- **Backup / portability** — export before a risky refactor.
- **Bootstrapping a new repo** from another project's knowledge.
- **Sharing a slice** of memory out-of-band (e.g. into an issue or wiki).

For normal team sharing, you don't need export/import — `.fluree-memory/repo.ttl` is committed to git and everyone who clones + runs `fluree memory init` picks it up automatically. See [Repo vs user memory](../concepts/repo-vs-user.md).


---

# memory/cli/mcp-install.md

# fluree memory mcp-install

Install MCP configuration for an IDE so its agent can use memory tools.

```bash
fluree memory mcp-install [--ide <IDE>]
```

## Options

| Option | Description |
|---|---|
| `--ide <IDE>` | Target IDE (auto-detected if omitted) |

## Supported IDEs

| Value | Config written | Extras |
|---|---|---|
| `claude-code` | `claude mcp add` → `~/.claude.json` (local scope) | Appends to `CLAUDE.md` |
| `vscode` | `<repo>/.vscode/mcp.json` (key: `servers`) | `.vscode/fluree_rules.md` |
| `cursor` | `<repo>/.cursor/mcp.json` (key: `mcpServers`) | `.cursor/rules/fluree_rules.md` |
| `windsurf` | `~/.codeium/windsurf/mcp_config.json` (global) | — |
| `zed` | `<repo>/.zed/settings.json` (key: `context_servers`) | Skips if JSONC detected |

If an existing target file isn't valid JSON (JSONC with comments, hand-edited corruption, etc.), `mcp-install` refuses to overwrite it and prints a snippet you can paste in manually. This applies to all per-repo configs above, plus Windsurf's global file.

Legacy aliases: `claude-vscode` and `github-copilot` both map to `vscode`.

When `--ide` is omitted, the first **unconfigured** detected tool is used; defaults to `claude-code` if nothing's detected.

## Example

```bash
fluree memory mcp-install --ide cursor
```

Output:

```
  Installed: .cursor/mcp.json
  Installed: .cursor/rules/fluree_rules.md
```

## Per-IDE config shape

The JSON `mcp-install` writes differs per IDE:

**Cursor** (`.cursor/mcp.json`) is the only target that sets `FLUREE_HOME` by default. It uses `${workspaceFolder}` interpolation to pin the memory store to the current workspace regardless of where Cursor spawns the process from:

```json
{
  "mcpServers": {
    "fluree-memory": {
      "type": "stdio",
      "command": "fluree",
      "args": ["mcp", "serve", "--transport", "stdio"],
      "env": { "FLUREE_HOME": "${workspaceFolder}/.fluree" }
    }
  }
}
```

**VS Code, Windsurf, Zed, Claude Code** get a simpler entry with no `env`:

```json
{
  "command": "fluree",
  "args": ["mcp", "serve", "--transport", "stdio"]
}
```

(The top-level wrapper key differs — `servers` for VS Code, `mcpServers` for Windsurf, `context_servers` for Zed. Claude Code's entry is registered globally via `claude mcp add`.)

These rely on the MCP server's walk-up behavior: on start, it looks for `.fluree/` beginning at its spawn CWD. That's usually the workspace, but if the IDE starts it elsewhere memory may land in a global store. See the troubleshooting section below.

## Troubleshooting: repo vs global memory

**Repo-scoped (the goal):**

- Memories: `<repo>/.fluree-memory/repo.ttl`
- MCP log: `<repo>/.fluree-memory/.local/mcp.log` (truncated on each server start — tail it while reproducing the issue)

**Global (something's wrong):**

- Memories under the platform default, e.g. `~/Library/Application Support/fluree/` on macOS
- **Fix:** add an explicit absolute `FLUREE_HOME` to the MCP config entry, pointing at your repo's `.fluree/`, and fully restart (not just reload) the IDE. For Cursor, the `${workspaceFolder}`-based default should already be in place — re-run `mcp-install` from inside the repo if it's missing.

## See also

- [Concepts: MCP server](../concepts/mcp.md) — what tools are exposed
- [Getting started: Claude Code / Cursor / VS Code / Windsurf / Zed](../getting-started/README.md) — per-IDE walkthroughs


---

# memory/reference/README.md

# Reference

Lookups — the things you open once to check a specific detail.

- [IDE support matrix](ide-matrix.md) — what's supported where, config paths, gotchas.
- [Schema (mem: vocabulary)](schema.md) — the RDF shape of a memory.
- [TTL file format](ttl-format.md) — structure of `repo.ttl` and `user.ttl`.


---

# memory/reference/ide-matrix.md

# IDE support matrix

Where each supported AI coding tool stores its MCP config and its rules file, and whether the config is scoped per-repo or global.

| IDE | MCP config | Config scope | `FLUREE_HOME` set? | Rules file | `mcp-install` value |
|---|---|---|---|---|---|
| Claude Code | `~/.claude.json` (via `claude mcp add`) | user (local) | no | section appended to `<repo>/CLAUDE.md` | `claude-code` |
| Cursor | `<repo>/.cursor/mcp.json` | **repo** | yes — `${workspaceFolder}/.fluree` | `<repo>/.cursor/rules/fluree_rules.md` | `cursor` |
| VS Code (Copilot) | `<repo>/.vscode/mcp.json` | **repo** | no | `<repo>/.vscode/fluree_rules.md` | `vscode` |
| Windsurf | `~/.codeium/windsurf/mcp_config.json` | global | no | none | `windsurf` |
| Zed | `<repo>/.zed/settings.json` | **repo** | no | none (skipped if JSONC) | `zed` |

Legacy aliases:
- `claude-vscode` → `vscode`
- `github-copilot` → `vscode`

## `FLUREE_HOME` and repo scoping

Only the Cursor config sets `FLUREE_HOME` automatically. For the other IDEs, the MCP server figures out which repo it's serving by walking up from its spawn CWD until it finds a `.fluree/` directory. In normal use the IDE spawns the server from the workspace root, so this works without extra configuration.

If memory ends up in a platform-global store instead of `<repo>/.fluree-memory/`, the fix is to add `FLUREE_HOME` manually to the relevant MCP config, pointing at an absolute path (or a variable the IDE interpolates — Cursor supports `${workspaceFolder}`; other IDEs' support varies). Then restart the IDE.

## Detection

`fluree memory init` detects each IDE via four signal classes (any one is enough):

1. **Binary on `PATH`** — `claude`, `code`, `cursor`, `windsurf`, `zed`. This is the most reliable cross-platform signal and the primary signal for Claude Code, which has no `.app` bundle.
2. **macOS app bundle** — `/Applications/<App>.app` or `~/Applications/<App>.app`.
3. **Home-dir marker** — `~/.cursor`, `~/.vscode`, `~/.codeium/windsurf`, `~/.zed`, `~/.config/zed`, `~/.claude`, `~/.claude.json`.
4. **User-config dir** — `<config>/Code`, `<config>/Cursor`, `<config>/Windsurf`, `<config>/Zed`, where `<config>` is `~/.config` (Linux), `~/Library/Application Support` (macOS), or `%APPDATA%` (Windows). VS Code on Linux, for example, stores its first-launch marker as `~/.config/Code/`, not `~/.vscode/`.

A freshly installed IDE that has never been launched **and** isn't on `PATH` may go undetected. If `init` misses your IDE, install MCP for it directly: `fluree memory mcp-install --ide <tool>`.

## Known gotchas

- **Zed + JSONC**: If `.zed/settings.json` contains `//` comments, `mcp-install` refuses to write to avoid corrupting your settings. Paste the snippet yourself or strip comments first. The same refusal now applies to `.cursor/mcp.json`, `.vscode/mcp.json`, and `~/.codeium/windsurf/mcp_config.json` — any non-empty config that fails to parse as plain JSON is left alone, and `mcp-install` prints a snippet for you to paste in.
- **Windsurf globals**: Windsurf's MCP config is user-global, not per-repo. If you work across multiple repos, you likely need to leave `FLUREE_HOME` unset and rely on walk-up — or switch the env var per project manually.
- **Cursor restarts**: Cursor caches MCP servers aggressively. If a change to `.cursor/mcp.json` doesn't take effect, fully quit Cursor (Cmd-Q on macOS) rather than just reloading the window.
- **Claude Code CLAUDE.md**: The rules section is appended at the end of `<repo>/CLAUDE.md` (only if one doesn't already mention `fluree memory` or `memory_recall`). If you have a large existing CLAUDE.md, make sure the agent is actually reading to the end.
- **Running from a subdir**: `init` resolves the project root by walking up from the cwd to the nearest `.git`. The Claude Code installer registers MCP keyed by that root (not by your subdir cwd), so running `fluree memory init` from `crates/foo/` behaves identically to running it from the repo root.


---

# memory/reference/schema.md

# Schema (mem: vocabulary)

Every memory is a set of RDF triples. The `mem:` vocabulary defines the classes and predicates.

## Namespace

```
@prefix mem: <https://ns.flur.ee/memory#> .
```

## Classes

A memory's kind is expressed via `rdf:type` (`a` in Turtle) — there is no `mem:kind` predicate.

| Class | Kind |
|---|---|
| `mem:Fact` | fact |
| `mem:Decision` | decision |
| `mem:Constraint` | constraint |

`mem:repo` and `mem:user` are additional IRIs used as the range of `mem:scope` (see below).

## Core predicates

| Predicate | Range | Required | Meaning |
|---|---|---|---|
| `mem:content` | `xsd:string` (indexed as `@fulltext`) | ✅ | The textual content; BM25-searchable |
| `mem:scope` | IRI — `mem:repo` or `mem:user` | ✅ | Which TTL file it lives in |
| `mem:createdAt` | `xsd:dateTime` | ✅ | Insertion timestamp |
| `mem:tag` | `xsd:string` (multi-valued) | optional | Free-form tags |
| `mem:artifactRef` | `xsd:string` (multi-valued) | optional | File / symbol / URL references |
| `mem:branch` | `xsd:string` | optional | Git branch captured at write time |

## Optional predicates (any kind)

These predicates can appear on any memory kind. All values are stored as plain string literals (not IRIs).

| Predicate | Range | Meaning |
|---|---|---|
| `mem:rationale` | `xsd:string` (indexed as `@fulltext`) | Why — the reasoning behind this memory |
| `mem:alternatives` | `xsd:string` | What else was considered |
| `mem:severity` | `xsd:string` — `"must"`, `"should"`, or `"prefer"` | How hard a constraint is (constraints only) |

## ID format

Memory IRIs take the shape:

```
mem:<kind>-<ULID>
```

Examples:

```
mem:fact-01JDXYZ5A2B3C4D5E6F7G8H9J0
mem:decision-01JDABC6D7E8F9G0H1I2J3K4L5
mem:constraint-01JDLMN7O8P9Q0R1S2T3U4V5W6
```

ULIDs are sortable by creation time, which is why memories display nicely in chronological order without an explicit index.

## Full example

```ttl
@prefix mem: <https://ns.flur.ee/memory#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

mem:decision-01JDABC a mem:Decision ;
    mem:content "Use postcard for compact index encoding" ;
    mem:tag "encoding" ;
    mem:tag "indexer" ;
    mem:scope mem:repo ;
    mem:artifactRef "fluree-db-indexer/" ;
    mem:createdAt "2026-02-22T14:00:00Z"^^xsd:dateTime ;
    mem:rationale "no_std compatible, smaller output than bincode" ;
    mem:alternatives "bincode, CBOR, MessagePack" .
```

See also: [TTL file format](ttl-format.md) for how this shows up on disk.


---

# memory/reference/ttl-format.md

# TTL file format

The `.fluree-memory/repo.ttl` and `.fluree-memory/.local/user.ttl` files hold the serialized form of every memory in their respective scope. Each memory is a block of [Turtle](https://www.w3.org/TR/turtle/) triples.

## Structure

Each memory is a Turtle subject block: the IRI, followed by `a mem:<Kind>` (RDF type), then a predicate list in a canonical order. Multi-valued predicates (`mem:tag`, `mem:artifactRef`) repeat once per value.

```ttl
# Fluree Memory — repo-scoped
# Auto-managed by `fluree memory`. Manual edits are supported.
@prefix mem: <https://ns.flur.ee/memory#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

mem:fact-01JDXYZ a mem:Fact ;
    mem:content "Tests use cargo nextest" ;
    mem:tag "cargo" ;
    mem:tag "testing" ;
    mem:scope mem:repo ;
    mem:createdAt "2026-02-22T14:00:00Z"^^xsd:dateTime .

mem:decision-01JDABC a mem:Decision ;
    mem:content "Use postcard for compact index encoding" ;
    mem:tag "encoding" ;
    mem:tag "indexer" ;
    mem:scope mem:repo ;
    mem:artifactRef "fluree-db-indexer/" ;
    mem:createdAt "2026-02-22T14:05:00Z"^^xsd:dateTime ;
    mem:rationale "no_std compatible, smaller output than bincode" ;
    mem:alternatives "bincode, CBOR, MessagePack" .
```

Tags and artifact refs are sorted alphabetically within a memory for deterministic diffs. When a memory is updated, the TTL file is rewritten with the changes in place and git tracks the history.

## Why TTL and not JSON

Three reasons:

- **Diff-friendly** — predicates are one per line within a subject block, so git diffs are readable. Memories are sorted by `(branch, id)`, which groups memories from the same branch together and reduces merge conflicts across feature branches.
- **Merge-friendly** — because the sort distributes memories by originating branch, two feature branches adding memories will insert into different regions of the file and won't conflict on merge.
- **Semantically exact** — Turtle is RDF, so there's no impedance mismatch between what's in the file and what's in the `__memory` ledger.

## Sync direction

The TTL file is the **canonical** store for a given scope. The `__memory` ledger is a derived cache rebuilt from the TTL files when they change.

When you `memory add`, the CLI / MCP server:

1. Rewrites the TTL file with the new memory inserted in sorted position (authoritative).
2. Transacts the new triples into the `__memory` ledger (so recall is fast).
3. Writes a content-hash watermark to `.fluree-memory/.local/build-hash`.

If the ledger write fails, the hash is left stale and the next `ensure_synced` call rebuilds the ledger from the files. When git pulls in a new version of `repo.ttl`, the hash mismatch triggers the same rebuild. In practice this is invisible.

## Concurrent access

It's normal for several processes to touch one project's memory at once — Cursor and Claude Code together, or several Claude sessions, each running its own `fluree mcp serve`. Two guarantees keep that safe:

- **The TTL file is the only shared mutable state, and writes are serialized.** Every mutation takes a cross-process file lock (`.fluree-memory/.local/rebuild.lock`) before rewriting the file, and re-reads the file under that lock, so two writers can't lose each other's memories.
- **Each process keeps its ledger cache private.** The `mcp serve` ledger is held **in-memory** and rebuilt from the TTL files on startup, so one process refreshing its cache can never disturb another's. Each process also tracks, in memory, the file content its own ledger reflects — the shared on-disk `build-hash` alone can't tell a long-running process that *another* process changed the files, so the per-process watermark is what triggers its rebuild. (Short-lived `fluree memory` CLI commands keep a file-backed ledger for `import` and legacy migration; they're not long-lived enough to drift.)

## Editing by hand

You *can* edit `repo.ttl` or `user.ttl` directly if you need to — fix a typo, reorder, batch-retag. After editing:

```bash
fluree memory status
```

…to verify the store parses cleanly. If there's a syntax error, `status` will point at it.

For most fixes, though, prefer `update` / `forget` — they'll produce cleaner git history than hand-edits.

## File size

TTL is compact. A project with ~200 memories typically lands under 50 KB. At that size, `repo.ttl` stays pleasant to review in a PR.

If a file grows past that, consider whether you're memorizing task state instead of durable knowledge — a `fluree memory status` + skim + cleanup pass is usually all it takes.


---

# operations/README.md

# Operations

This section covers operational aspects of running Fluree in production, including configuration, storage backends, monitoring, and administrative operations.

## Operation Guides

### [Configuration](configuration.md)

Server configuration options:
- Command-line flags
- Configuration files
- Environment variables
- Runtime settings
- Tuning parameters

### [Running with Docker](docker.md)

Configuring the official `fluree/server` image:
- Image internals (entrypoint, volumes, runtime user)
- Three configuration approaches: env vars, mounted JSON-LD/TOML config, CLI flags
- Common recipes: LRU cache sizing, background indexing, auth, S3+DynamoDB, query peers
- Full annotated Docker Compose example
- Troubleshooting (volume permissions, `RUST_LOG` vs `FLUREE_LOG_LEVEL`, cache auto-sizing under cgroup limits)

### [Storage Modes](storage.md)

Storage backend options:
- Memory storage (development)
- File system storage (single server)
- AWS S3/DynamoDB (distributed)
- IPFS / Kubo (decentralized)
- Storage selection criteria
- Switching between storage modes

### [Serverless Storage Choices](serverless-storage.md)

Cloud/serverless storage placement guidance:
- Standard S3 vs S3 Express One Zone for index storage
- Why commits should normally remain on Standard S3
- Expected transaction, query, and indexing latency ranges
- Lambda disk cache and S3 concurrency tuning notes

### [Hardware sizing: CPU vs disk (benchmark)](hardware-benchmarks.md)

A worked benchmark (SPARQLoscope on ~574 M-triple DBLP) showing how hardware maps to performance:
- Import is storage-bound — local NVMe imports faster even on slower CPUs
- Query latency is CPU-bound — served from cache, tracks single-thread speed
- Sizing guidance: when to prioritize disk vs core vs RAM

### [IPFS Storage](ipfs-storage.md)

IPFS-specific setup and configuration:
- Kubo node installation and setup
- JSON-LD configuration fields
- Content addressing and CID mapping
- Pinning strategies
- Operational considerations

### [DynamoDB Nameservice](dynamodb-guide.md)

DynamoDB-specific setup and configuration:
- Table creation (CLI, CloudFormation, Terraform)
- Schema reference (v2 attributes)
- AWS credentials and permissions
- Local development with LocalStack
- Production considerations

### [Telemetry and Logging](telemetry.md)

Monitoring and observability:
- Logging configuration
- Metrics collection
- Tracing
- Health monitoring
- Performance metrics
- Integration with monitoring systems

### [Admin, Health, and Stats](admin-and-health.md)

Administrative operations:
- Health check endpoints
- Server statistics
- Manual indexing triggers
- Backup and restore
- Maintenance operations

### [Query peers and replication](query-peers.md)

Run `fluree-server` as a read-only query peer:
- SSE nameservice events (`GET /v1/fluree/events`)
- Peer mode (refresh on stale + write forwarding)
- Storage proxy endpoints (`/v1/fluree/storage/*`) for private-storage deployments

## Deployment Patterns

### Development

Single-process, memory storage:

```bash
./fluree-db-server --storage memory --log-level debug
```

### Single Server Production

File-based storage:

```bash
./fluree-db-server \
  --storage file \
  --data-dir /var/lib/fluree \
  --port 8090 \
  --log-level info
```

### Distributed Production

AWS-backed distributed deployment:

```bash
./fluree-db-server \
  --storage aws \
  --s3-bucket fluree-prod-data \
  --s3-region us-east-1 \
  --dynamodb-table fluree-nameservice \
  --port 8090
```

## Key Configuration Areas

### Server Settings

- Port and host binding
- TLS/SSL certificates
- Request size limits
- Timeout values
- CORS configuration

### Storage Configuration

- Storage mode selection
- Data directory (file mode)
- AWS credentials (S3 mode)
- IPFS / Kubo connection (IPFS mode)
- Connection pooling
- Cache settings

### Indexing Configuration

- Index interval
- Batch size
- Memory allocation
- Number of threads
- Index retention

### Security Configuration

- Authentication mode
- API key requirements
- Signed request validation
- Policy enforcement
- Rate limiting

## Monitoring

### Health Checks

```bash
curl http://localhost:8090/health
```

Response:
```json
{
  "status": "healthy",
  "version": "0.1.0",
  "storage": "file",
  "uptime_ms": 3600000
}
```

### Server Statistics

```bash
curl http://localhost:8090/v1/fluree/stats
```

Response:
```json
{
  "version": "0.1.0",
  "uptime_ms": 3600000,
  "ledgers": 5,
  "queries": {
    "total": 12345,
    "active": 3,
    "avg_duration_ms": 45
  },
  "transactions": {
    "total": 567,
    "avg_duration_ms": 89
  },
  "indexing": {
    "active": true,
    "pending_ledgers": 1,
    "avg_lag_ms": 1500
  }
}
```

### Metrics Collection

Use `GET /v1/fluree/stats` for built-in server statistics. Prometheus-style
`/metrics` export is not currently part of the standalone server API.

## Operational Tasks

### Backup

File storage backup:

```bash
# Backup data directory
tar -czf fluree-backup-$(date +%Y%m%d).tar.gz /var/lib/fluree/
```

AWS storage backup:

```bash
# S3 versioning enabled - automatic backups
aws s3 ls s3://fluree-prod-data/ --recursive

# Point-in-time recovery via S3 versions
```

### Restore

File storage restore:

```bash
# Stop server
systemctl stop fluree

# Restore backup
tar -xzf fluree-backup-20240122.tar.gz -C /

# Start server
systemctl start fluree
```

### Manual Indexing

Trigger indexing manually:

```bash
curl -X POST http://localhost:8090/v1/fluree/reindex \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb:main"}'
```

### Compaction

There is no standalone HTTP compaction endpoint. Reindexing rebuilds index
artifacts when you need to force a full refresh.

## Performance Tuning

### Memory Settings

```bash
./fluree-db-server \
  --query-memory-mb 2048 \
  --cache-size-mb 1024
```

### Indexing Tuning

```bash
fluree-server \
  --indexing-enabled \
  --reindex-min-bytes 100000 \
  --reindex-max-bytes 1000000
```

### Query Tuning

```bash
./fluree-db-server \
  --query-timeout-ms 30000 \
  --max-query-size 1048576 \
  --query-threads 8
```

## High Availability

### Load Balancing

Run multiple Fluree instances behind load balancer:

```text
          ┌─────────────┐
          │   Clients   │
          └──────┬──────┘
                 │
          ┌──────▼──────┐
          │    Load     │
          │  Balancer   │
          └──────┬──────┘
                 │
    ┌────────────┼────────────┐
    │            │            │
┌───▼────┐  ┌───▼────┐  ┌───▼────┐
│Fluree 1│  │Fluree 2│  │Fluree 3│
└───┬────┘  └───┬────┘  └───┬────┘
    │           │           │
    └───────────┼───────────┘
                │
         ┌──────▼──────┐
         │  S3/Dynamo  │
         │  Nameservice│
         └─────────────┘
```

### Failover

Configure health checks in load balancer:

```yaml
health_check:
  path: /health
  interval: 10s
  timeout: 5s
  healthy_threshold: 2
  unhealthy_threshold: 3
```

## Security Hardening

### TLS/SSL

```bash
./fluree-db-server \
  --tls-cert /path/to/cert.pem \
  --tls-key /path/to/key.pem \
  --tls-ca /path/to/ca.pem
```

### Require Authentication

```bash
./fluree-db-server \
  --require-auth \
  --require-signed-requests
```

### Rate Limiting

```bash
./fluree-db-server \
  --rate-limit-queries 100 \
  --rate-limit-transactions 10 \
  --rate-limit-window 60
```

## Best Practices

### 1. Use Appropriate Storage Mode

- Development: memory
- Single server: file
- Production/Distributed: AWS
- Decentralized: IPFS

### 2. Enable Monitoring

Set up monitoring for:
- Health status
- Query latency
- Transaction rate
- Indexing lag
- Error rates

### 3. Regular Backups

Automate backups:

```bash
# Daily backup cron
0 2 * * * /usr/local/bin/backup-fluree.sh
```

### 4. Capacity Planning

Monitor growth:
- Storage usage
- Query volume
- Transaction rate
- Index sizes

### 5. Security Best Practices

- Use TLS in production
- Require authentication
- Enable rate limiting
- Regular security audits

### 6. Log Management

- Rotate logs regularly
- Ship logs to centralized system
- Set appropriate log levels
- Monitor error rates

## Related Documentation

- [Configuration](configuration.md) - Detailed configuration reference
- [Storage](storage.md) - Storage backend details
- [Telemetry](telemetry.md) - Monitoring and metrics
- [Admin and Health](admin-and-health.md) - Administrative operations
- [Getting Started: Server](../getting-started/quickstart-server.md) - Initial setup


---

# operations/configuration.md

# Configuration

Fluree server is configured via a configuration file, command-line flags, and environment variables.

## Configuration Methods

### Configuration File (TOML, JSON, or JSON-LD)

The server reads configuration from `.fluree/config.toml` (or `.fluree/config.jsonld`) — the same file used by the Fluree CLI. Server settings live under the `[server]` section (or `"server"` key in JSON/JSON-LD). The server walks up from the current working directory looking for `.fluree/config.toml` or `.fluree/config.jsonld`, falling back to the global Fluree **config** directory (`$FLUREE_HOME`, or the platform config directory — see table below).

#### Global Directory Layout

When `$FLUREE_HOME` is set, both config and data share that single directory. When it is not set, the platform's config and data directories are used:

| Content                     | Linux                   | macOS                                  | Windows                 |
| --------------------------- | ----------------------- | -------------------------------------- | ----------------------- |
| Config (`config.toml`)      | `~/.config/fluree`      | `~/Library/Application Support/fluree` | `%LOCALAPPDATA%\fluree` |
| Data (`storage/`, `active`) | `~/.local/share/fluree` | `~/Library/Application Support/fluree` | `%LOCALAPPDATA%\fluree` |

On Linux, config and data directories are separated per the XDG Base Directory specification. On macOS and Windows both resolve to the same directory. When directories are split, `fluree init --global` writes an absolute `storage_path` into `config.toml` so the server can locate the data directory regardless of working directory.

```bash
# Use default config file discovery
fluree-server

# Override config file path
fluree-server --config /etc/fluree/config.toml

# Activate a profile
fluree-server --profile prod
```

Example `config.toml`:

```toml
[server]
listen_addr = "0.0.0.0:8090"
storage_path = "/var/lib/fluree"
log_level = "info"
query_timeout_ms = 900000  # 15 minutes; set to 0 to disable
query_min_t_timeout_ms = 5000
# cache_max_mb = 4096  # global cache budget (MB); default: tiered by RAM (<4GB: 30%, 4-8GB: 40%, >=8GB: 35%)

[server.query_refresh]
enabled = false
ttl_ms = 1000

[server.indexing]
enabled = true
reindex_min_bytes = 100000
# reindex_max_bytes defaults to 20% of system RAM; override only if needed:
# reindex_max_bytes = 536870912  # 512 MB

[server.auth.data]
mode = "required"
trusted_issuers = ["did:key:z6Mk..."]
```

JSON is also supported (detected by `.json` file extension):

```json
{
  "server": {
    "listen_addr": "0.0.0.0:8090",
    "storage_path": "/var/lib/fluree",
    "indexing": { "enabled": true }
  }
}
```

#### JSON-LD Format

JSON-LD config files (`.jsonld` extension) add a `@context` that maps config keys to the Fluree config vocabulary (`https://ns.flur.ee/config#`), making the file valid JSON-LD. Generate one with:

```bash
fluree init --format jsonld
```

Example `.fluree/config.jsonld`:

```json
{
  "@context": {
    "@vocab": "https://ns.flur.ee/config#"
  },
  "_comment": "Fluree Configuration — JSON-LD format.",
  "server": {
    "listen_addr": "0.0.0.0:8090",
    "storage_path": ".fluree/storage",
    "log_level": "info",
    "indexing": {
      "enabled": true,
      "reindex_min_bytes": 100000
    }
  },
  "profiles": {
    "prod": {
      "server": {
        "log_level": "warn"
      }
    }
  }
}
```

The `@context` is validated at load time (using the JSON-LD parser) but does not affect config value resolution — `serde` ignores unknown keys like `@context` and `_comment`. If both `config.toml` and `config.jsonld` exist in the same directory, TOML takes precedence and a warning is logged.

### Profiles

Profiles allow environment-specific overrides. Define them in `[profiles.<name>.server]` and activate with `--profile <name>`:

```toml
[server]
log_level = "info"

[profiles.dev.server]
log_level = "debug"

[profiles.prod.server]
log_level = "warn"
[profiles.prod.server.indexing]
enabled = true
[profiles.prod.server.auth.data]
mode = "required"
```

Profile values are deep-merged onto `[server]` — only the fields present in the profile are overridden.

### Command-Line Flags

```bash
fluree-server \
  --listen-addr 0.0.0.0:8090 \
  --storage-path /var/lib/fluree \
  --log-level info
```

### Environment Variables

All CLI flags have corresponding environment variables with `FLUREE_` prefix:

```bash
export FLUREE_LISTEN_ADDR=0.0.0.0:8090
export FLUREE_STORAGE_PATH=/var/lib/fluree
export FLUREE_LOG_LEVEL=info

fluree-server
```

A few operational knobs are environment-only (no CLI flag):

| Variable | Default | Purpose |
|----------|---------|---------|
| `FLUREE_REASONING_MAX_FACTS` | 1,000,000 | Server-wide default OWL2-RL materialization budget (max derived facts). Overridden per ledger by `f:reasoningMaxFacts` and per query by `"reasoningBudget"`; see [Reasoning](../query/reasoning.md#materialization-budget). |
| `FLUREE_REASONING_MAX_SECONDS` | 30 | Server-wide default OWL2-RL materialization budget (wall-clock seconds). Same override chain as above. |

### Precedence

Configuration precedence (highest to lowest):

1. Command-line flags
2. Environment variables
3. Profile overrides (`[profiles.<name>.server]`)
4. Config file (`[server]`)
5. Built-in defaults

### Error Handling

If `--config` or `--profile` is specified and the configuration cannot be loaded (file not found, parse error, missing profile), the server **exits with an error**. This prevents silent misconfiguration in production.

If the config file is auto-discovered (no explicit `--config`) and cannot be parsed, the server logs a warning and continues with CLI/env/default values only.

## Server Configuration

### Listen Address

Address and port to bind to:

| Flag            | Env Var              | Default        |
| --------------- | -------------------- | -------------- |
| `--listen-addr` | `FLUREE_LISTEN_ADDR` | `0.0.0.0:8090` |

```bash
fluree-server --listen-addr 0.0.0.0:9090
```

### Storage Path

Path for file-based storage. If not specified, defaults to `.fluree/storage` relative to the working directory (the same location used by `fluree init`):

| Flag             | Env Var               | Default           |
| ---------------- | --------------------- | ----------------- |
| `--storage-path` | `FLUREE_STORAGE_PATH` | `.fluree/storage` |

```bash
# Explicit storage path (e.g. production)
fluree-server --storage-path /var/lib/fluree

# Default: uses .fluree/storage in the working directory
fluree-server
```

### Connection Configuration (S3, DynamoDB, etc.)

For storage backends beyond local files — S3, DynamoDB nameservice, split commit/index storage, encryption — use a JSON-LD connection config file:

| Flag                  | Env Var                    | Default |
| --------------------- | -------------------------- | ------- |
| `--connection-config` | `FLUREE_CONNECTION_CONFIG`  | None    |

When set, the server builds its storage and nameservice from the connection config file instead of using `--storage-path`. The file uses the same JSON-LD format as the [Fluree API connection config](../reference/connection-config-jsonld.md).

```bash
# S3 + DynamoDB via connection config
fluree server run --connection-config /etc/fluree/connection.jsonld

# Or via environment variable
FLUREE_CONNECTION_CONFIG=/etc/fluree/connection.jsonld fluree server run
```

Example connection config (`connection.jsonld`):

```json
{
  "@context": {
    "@base": "https://ns.flur.ee/config/connection/",
    "@vocab": "https://ns.flur.ee/system#"
  },
  "@graph": [
    {
      "@id": "commitStorage",
      "@type": "Storage",
      "s3Bucket": "fluree-commits",
      "s3Prefix": "fluree-data/"
    },
    {
      "@id": "indexStorage",
      "@type": "Storage",
      "s3Bucket": "fluree-indexes--use1-az4--x-s3"
    },
    {
      "@id": "publisher",
      "@type": "Publisher",
      "dynamodbTable": "fluree-nameservice",
      "dynamodbRegion": "us-east-1"
    },
    {
      "@id": "conn",
      "@type": "Connection",
      "commitStorage": { "@id": "commitStorage" },
      "indexStorage": { "@id": "indexStorage" },
      "primaryPublisher": { "@id": "publisher" }
    }
  ]
}
```

**Behavior notes:**

- `--connection-config` and `--storage-path` are mutually exclusive. If both are set, `--connection-config` takes precedence (a warning is logged).
- Server-level settings (`--cache-max-mb`, `--indexing-enabled`, `--reindex-min-bytes`, `--reindex-max-bytes`) override any equivalent values from the connection config.
- `--indexing-enabled` defaults to `true`. Pass `--indexing-enabled=false` only when a separate peer/indexer process owns index maintenance for the same storage.
- AWS credentials and region are resolved via the standard AWS SDK chain (env vars, instance profile, `~/.aws/config`, etc.) — they are not part of the connection config.
- The connection config can use `envVar` indirection for sensitive fields like S3 bucket names or encryption keys (see [ConfigurationValue](../reference/connection-config-jsonld.md#configurationvalue-env-var-indirection)).

**Config file equivalent:**

```toml
[server]
connection_config = "/etc/fluree/connection.jsonld"
```

#### Capabilities by Backend

Not all nameservice backends support all features. The server checks capabilities at runtime:

| Feature                 | File (local)   | DynamoDB       | Storage-backed |
| ----------------------- | -------------- | -------------- | -------------- |
| Query / transact        | Yes            | Yes            | Yes            |
| Event subscriptions     | Yes            | No             | No             |
| Default context (read)  | Yes            | Yes            | Yes            |
| Default context (write) | Yes            | Yes            | No             |

If a capability is not available, the server returns an appropriate error (e.g., 501 for event subscriptions with DynamoDB).

### CORS

Enable Cross-Origin Resource Sharing:

| Flag             | Env Var               | Default |
| ---------------- | --------------------- | ------- |
| `--cors-enabled` | `FLUREE_CORS_ENABLED` | `true`  |

When enabled, allows requests from any origin.

### Body Limit

Maximum request body size in bytes:

| Flag           | Env Var             | Default           |
| -------------- | ------------------- | ----------------- |
| `--body-limit` | `FLUREE_BODY_LIMIT` | `52428800` (50MB) |

### Query Timeout

Maximum query execution time in milliseconds. The server starts a timeout task
that signals query cancellation when the limit elapses; query execution observes
that signal at I/O boundaries — operator batch handoffs, leaflet refills in the
fused COUNT fast paths, and parallel-partition starts — never inside per-row
loops (in-loop checks measurably perturb hot-loop codegen). Cancellation
latency is bounded by one leaflet/batch of work, typically well under 10ms. Set to `0` to disable the server-side
timeout. If an HTTP client disconnects while a query is still running, the
server signals cancellation through the same cooperative handle so long-running
operators can stop at the next checkpoint.

| Flag                 | Env Var                    | Default                  |
| -------------------- | -------------------------- | ------------------------ |
| `--query-timeout-ms` | `FLUREE_QUERY_TIMEOUT_MS`  | `900000` (15 minutes)    |
| `--query-min-t-timeout-ms` | `FLUREE_QUERY_MIN_T_TIMEOUT_MS` | `5000` (5 seconds) |
| `--stream-heartbeat-ms` | `FLUREE_STREAM_HEARTBEAT_MS` | `15000` (15 seconds) |

`query_min_t_timeout_ms` caps HTTP read-after-write waits from `Fluree-Min-T`, `opts.min-t`, and numeric `@t` snapshot pins. It remains enforced even when `query_timeout_ms = 0`.

`stream_heartbeat_ms` is the keep-alive cadence for the [streaming query endpoint](../api/streaming-query.md) (`/stream/query`). Records flush at this interval during stalls so a long-running query survives a fronting proxy's idle timeout; set it below that timeout (e.g. under CloudFront/ALB's ~60s). `0` disables heartbeats.

### Query-Time Refresh

Long-running query servers can opt in to a bounded nameservice freshness check before current-head queries. When enabled, the server calls the same `Fluree::refresh()` API used by serverless query handlers, but gates the call by a per-process, per-ledger TTL so high-QPS traffic does not check DynamoDB on every request.

This is demand-driven, not a background poller: the first current-head query for a ledger after its TTL window expires pays the nameservice round-trip, and idle ledgers are not refreshed.

This is useful for split deployments where writers update a shared DynamoDB nameservice but query servers do not subscribe to a transaction server's SSE event stream.

For a hard read-after-write guarantee when the client knows the transaction `t`, send `Fluree-Min-T` (or JSON-LD `opts.min-t`) on the query request. TTL refresh bounds ordinary staleness; `min-t` waits for a specific transaction time.

| Flag | Env Var | Default | Description |
| ---- | ------- | ------- | ----------- |
| `--query-refresh-enabled` | `FLUREE_QUERY_REFRESH_ENABLED` | `false` | Enable TTL-gated nameservice refresh before current-head query execution |
| `--query-refresh-ttl-ms` | `FLUREE_QUERY_REFRESH_TTL_MS` | `1000` | Minimum interval between refresh checks for the same ledger in one server process; set `0` to check every request |

Config file equivalent:

```toml
[server.query_refresh]
enabled = true
ttl_ms = 200
```

### Log Level

Logging verbosity:

| Flag          | Env Var            | Default |
| ------------- | ------------------ | ------- |
| `--log-level` | `FLUREE_LOG_LEVEL` | `info`  |

Options: `trace`, `debug`, `info`, `warn`, `error`

### Cache Size

Global cache budget (MB):

| Flag              | Env Var              | Default                |
| ----------------- | -------------------- | ---------------------- |
| `--cache-max-mb`  | `FLUREE_CACHE_MAX_MB`| Tiered by RAM: `<4GB: 30%, 4-8GB: 40%, >=8GB: 35%`    |

### Background Indexing

Enable background indexing and configure novelty backpressure thresholds:

| Flag                  | Env Var                    | Default   | Description                                     |
| --------------------- | -------------------------- | --------- | ----------------------------------------------- |
| `--indexing-enabled`  | `FLUREE_INDEXING_ENABLED`  | `true`    | Enable background indexing (set `false` only when an external indexer process owns this storage) |
| `--reindex-min-bytes` | `FLUREE_REINDEX_MIN_BYTES` | `100000`  | Soft threshold (triggers background indexing)   |
| `--reindex-max-bytes` | `FLUREE_REINDEX_MAX_BYTES` | 20% of system RAM (256 MB fallback) | Hard threshold (blocks commits until reindexed) |

Config file equivalent:

```toml
[server.indexing]
enabled = true
reindex_min_bytes = 100000         # 100 KB — soft trigger
# reindex_max_bytes = 536870912    # 512 MB — defaults to 20% of system RAM if omitted
```

## Server Role Configuration

### Server Role

Operating mode: transaction server or query peer:

| Flag            | Env Var              | Default       |
| --------------- | -------------------- | ------------- |
| `--server-role` | `FLUREE_SERVER_ROLE` | `transaction` |

Options:

- `transaction`: Write-enabled, produces events stream
- `peer`: Read-only, subscribes to transaction server

### Transaction Server URL (Peer Mode)

Base URL of the transaction server (required in peer mode):

| Flag              | Env Var                |
| ----------------- | ---------------------- |
| `--tx-server-url` | `FLUREE_TX_SERVER_URL` |

```bash
fluree-server \
  --server-role peer \
  --tx-server-url http://tx.internal:8090
```

## Authentication Configuration

### Replication vs Query Access

Fluree enforces a hard boundary between **replication-scoped** and **query-scoped** access:

- **Replication** (`fluree.storage.*`): Raw commit and index block transfer for peer sync and CLI `fetch`/`pull`/`push`. These operations bypass dataset policy (data must be bit-identical). Replication tokens are operator/service-account credentials — never issue them to end users.
- **Query** (`fluree.ledger.read/write.*`): Application-level data access through the query engine with full dataset policy enforcement. Query tokens are appropriate for end users and application service accounts.

A user holding only query-scoped tokens **cannot** clone or pull a ledger. They can `fluree track` a remote ledger (forwarding queries/transactions to the server) but cannot replicate its storage locally.

### Events Endpoint Authentication

Protect the `/v1/fluree/events` SSE endpoint:

| Flag                           | Env Var                              | Default |
| ------------------------------ | ------------------------------------ | ------- |
| `--events-auth-mode`           | `FLUREE_EVENTS_AUTH_MODE`            | `none`  |
| `--events-auth-audience`       | `FLUREE_EVENTS_AUTH_AUDIENCE`        | None    |
| `--events-auth-trusted-issuer` | `FLUREE_EVENTS_AUTH_TRUSTED_ISSUERS` | None    |

Modes:

- `none`: No authentication
- `optional`: Accept tokens but don't require them
- `required`: Require valid Bearer token

Supports both Ed25519 (embedded JWK) and OIDC/JWKS (RS256) tokens when the `oidc` feature is enabled and `--jwks-issuer` is configured. For OIDC tokens, issuer trust is implicit — only tokens signed by keys from configured JWKS endpoints will verify. For Ed25519 tokens, the issuer must appear in `--events-auth-trusted-issuer`.

```bash
# Ed25519 tokens only
fluree-server \
  --events-auth-mode required \
  --events-auth-trusted-issuer did:key:z6Mk...

# OIDC + Ed25519 (both work simultaneously)
fluree-server \
  --events-auth-mode required \
  --jwks-issuer "https://auth.example.com=https://auth.example.com/.well-known/jwks.json" \
  --events-auth-trusted-issuer did:key:z6Mk...
```

### Data API Authentication

Protect query/transaction endpoints (including `/v1/fluree/query/{ledger...}`,
`/v1/fluree/insert/{ledger...}`, `/v1/fluree/upsert/{ledger...}`,
`/v1/fluree/update/{ledger...}`, `/v1/fluree/info/{ledger...}`, and
`/v1/fluree/exists/{ledger...}`):

| Flag                               | Env Var                                 | Default |
| ---------------------------------- | --------------------------------------- | ------- |
| `--data-auth-mode`                 | `FLUREE_DATA_AUTH_MODE`                 | `none`  |
| `--data-auth-audience`             | `FLUREE_DATA_AUTH_AUDIENCE`             | None    |
| `--data-auth-trusted-issuer`       | `FLUREE_DATA_AUTH_TRUSTED_ISSUERS`      | None    |
| `--data-auth-default-policy-class` | `FLUREE_DATA_AUTH_DEFAULT_POLICY_CLASS` | None    |

Modes:

- `none`: No authentication (default)
- `optional`: Accept tokens but don't require them (development only)
- `required`: Require either a valid Bearer token **or** a signed request (JWS/VC)

Bearer token scopes:

- **Read**: `fluree.ledger.read.all=true` or `fluree.ledger.read.ledgers=[...]`
- **Write**: `fluree.ledger.write.all=true` or `fluree.ledger.write.ledgers=[...]`

Back-compat: `fluree.storage.*` claims imply **read** scope for data endpoints.

```bash
fluree-server \
  --data-auth-mode required \
  --data-auth-trusted-issuer did:key:z6Mk...
```

### OIDC / JWKS Token Verification

When the `oidc` feature is enabled, the server can verify JWT tokens signed by external identity
providers (e.g., Fluree Cloud Service) using JWKS (JSON Web Key Set) endpoints. This is in addition to the
existing embedded-JWK (Ed25519 `did:key`) verification path.

**Dual-path dispatch**: The server inspects each Bearer token's header:

- **Embedded JWK** (Ed25519): Uses the existing `verify_jws()` path — no JWKS needed.
- **kid header** (RS256): Uses OIDC/JWKS path — fetches the signing key from the issuer's JWKS endpoint.

Both paths coexist; no configuration change is needed for existing Ed25519 tokens.

| Flag               | Env Var                 | Default | Description                       |
| ------------------ | ----------------------- | ------- | --------------------------------- |
| `--jwks-issuer`    | `FLUREE_JWKS_ISSUERS`   | None    | OIDC issuer to trust (repeatable) |
| `--jwks-cache-ttl` | `FLUREE_JWKS_CACHE_TTL` | `300`   | JWKS cache TTL in seconds         |

The `--jwks-issuer` flag takes the format `<issuer_url>=<jwks_url>`:

```bash
fluree-server \
  --data-auth-mode required \
  --jwks-issuer "https://solo.example.com=https://solo.example.com/.well-known/jwks.json"
```

For multiple issuers, repeat the flag or use comma separation in the env var:

```bash
# CLI flags (repeatable)
fluree-server \
  --jwks-issuer "https://issuer1.example.com=https://issuer1.example.com/.well-known/jwks.json" \
  --jwks-issuer "https://issuer2.example.com=https://issuer2.example.com/.well-known/jwks.json"

# Environment variable (comma-separated)
export FLUREE_JWKS_ISSUERS="https://issuer1.example.com=https://issuer1.example.com/.well-known/jwks.json,https://issuer2.example.com=https://issuer2.example.com/.well-known/jwks.json"
```

**Behavior details:**

- JWKS endpoints are fetched at startup (`warm()`) but the server starts even if they're unreachable.
- Keys are cached and refreshed when a `kid` miss occurs (rate-limited to one refresh per issuer every 10 seconds).
- The token's `iss` claim must exactly match a configured issuer URL — unconfigured issuers are rejected immediately with a clear error.
- Data API, events, admin, and storage proxy endpoints all support JWKS verification. A single `--jwks-issuer` flag enables OIDC tokens across all endpoint groups. MCP auth continues to use the existing Ed25519 path only.

#### Connection-Scoped SPARQL Scope Enforcement

When a Bearer token is present for connection-scoped SPARQL queries (`/v1/fluree/query` with
`Content-Type: application/sparql-query`), the server enforces ledger scope:

- FROM / FROM NAMED clauses are parsed to extract ledger IDs (`name:branch`).
- Each ledger ID is checked against the token's read scope (`fluree.ledger.read.all` or `fluree.ledger.read.ledgers`).
- Out-of-scope ledgers return 404 (no existence leak).
- If no FROM clause is present, the query proceeds normally (the engine handles missing dataset errors).

### Admin Endpoint Authentication

Protect `/v1/fluree/create`, `/v1/fluree/drop`, `/v1/fluree/reindex`, branch
administration, and Iceberg mapping endpoints:

| Flag                          | Env Var                             | Default |
| ----------------------------- | ----------------------------------- | ------- |
| `--admin-auth-mode`           | `FLUREE_ADMIN_AUTH_MODE`            | `none`  |
| `--admin-auth-trusted-issuer` | `FLUREE_ADMIN_AUTH_TRUSTED_ISSUERS` | None    |

Modes:

- `none`: No authentication (development)
- `required`: Require valid Bearer token (production)

Supports both Ed25519 (embedded JWK) and OIDC/JWKS (RS256) tokens when the `oidc` feature is enabled and `--jwks-issuer` is configured. For OIDC tokens, issuer trust is implicit — only tokens signed by keys from configured JWKS endpoints will verify. For Ed25519 tokens, the issuer must appear in `--admin-auth-trusted-issuer` or the fallback `--events-auth-trusted-issuer`.

```bash
# Ed25519 tokens only
fluree-server \
  --admin-auth-mode required \
  --admin-auth-trusted-issuer did:key:z6Mk...

# OIDC (trust comes from --jwks-issuer, no did:key issuers needed)
fluree-server \
  --admin-auth-mode required \
  --jwks-issuer "https://auth.example.com=https://auth.example.com/.well-known/jwks.json"
```

If no admin-specific issuers are configured, falls back to `--events-auth-trusted-issuer`.

### MCP Endpoint

Protect and tune the `/mcp` Model Context Protocol endpoint:

| Flag                         | Env Var                           | Default                |
| ---------------------------- | --------------------------------- | ---------------------- |
| `--mcp-enabled`              | `FLUREE_MCP_ENABLED`              | `false`                |
| `--mcp-auth-trusted-issuer`  | `FLUREE_MCP_AUTH_TRUSTED_ISSUERS` | None                   |
| `--mcp-agent-json-max-bytes` | `FLUREE_MCP_AGENT_JSON_MAX_BYTES` | `32768`                |
| `--mcp-query-timeout-ms`     | `FLUREE_MCP_QUERY_TIMEOUT_MS`     | `300000` (5 minutes)   |

`--mcp-agent-json-max-bytes` (config file: `[server.mcp] agent_json_max_bytes`) is the byte
budget for the MCP `sparql_query` tool's Agent JSON result. Results larger than this are
truncated and the envelope sets `hasMore: true`; an agent paginates by re-running with the
returned `t`, an `ORDER BY`, and `OFFSET` advanced by the returned `rowCount`.

`--mcp-query-timeout-ms` (config file: `[server.mcp] query_timeout_ms`) is the
server-side timeout for MCP `sparql_query` execution. It uses the same
cooperative cancellation mechanism as HTTP queries, but defaults lower because
MCP tool calls are usually interactive. Set to `0` to disable the MCP timeout.
This setting does not apply to `get_data_model`, which may perform schema/stat
collection without this query timeout.

```bash
fluree-server \
  --mcp-enabled \
  --mcp-auth-trusted-issuer did:key:z6Mk...
```

## Peer Mode Configuration

### Peer Subscription

Configure what the peer subscribes to:

| Flag                              | Description                                     |
| --------------------------------- | ----------------------------------------------- |
| `--peer-subscribe-all`            | Subscribe to all ledgers and graph sources      |
| `--peer-ledger <ledger-id>`       | Subscribe to specific ledger (repeatable)       |
| `--peer-graph-source <ledger-id>` | Subscribe to specific graph source (repeatable) |

```bash
fluree-server \
  --server-role peer \
  --tx-server-url http://tx:8090 \
  --peer-subscribe-all
```

Or subscribe to specific resources:

```bash
fluree-server \
  --server-role peer \
  --tx-server-url http://tx:8090 \
  --peer-ledger books:main \
  --peer-ledger users:main
```

### Peer Events Configuration

| Flag                  | Env Var                    | Description                                                     |
| --------------------- | -------------------------- | --------------------------------------------------------------- |
| `--peer-events-url`   | `FLUREE_PEER_EVENTS_URL`   | Custom events URL (default: `{tx_server_url}/v1/fluree/events`) |
| `--peer-events-token` | `FLUREE_PEER_EVENTS_TOKEN` | Bearer token for events (supports `@filepath`)                  |

### Peer Reconnection

| Flag                          | Default | Description             |
| ----------------------------- | ------- | ----------------------- |
| `--peer-reconnect-initial-ms` | `1000`  | Initial reconnect delay |
| `--peer-reconnect-max-ms`     | `30000` | Maximum reconnect delay |
| `--peer-reconnect-multiplier` | `2.0`   | Backoff multiplier      |

### Peer Storage Access

| Flag                    | Env Var                      | Default  |
| ----------------------- | ---------------------------- | -------- |
| `--storage-access-mode` | `FLUREE_STORAGE_ACCESS_MODE` | `shared` |

Options:

- `shared`: Direct storage access (requires `--storage-path` or `--connection-config`)
- `proxy`: Proxy reads through transaction server

For proxy mode:

| Flag                         | Env Var                           |
| ---------------------------- | --------------------------------- |
| `--storage-proxy-token`      | `FLUREE_STORAGE_PROXY_TOKEN`      |
| `--storage-proxy-token-file` | `FLUREE_STORAGE_PROXY_TOKEN_FILE` |

## Storage Proxy Configuration (Transaction Server)

Storage proxy provides **replication-scoped** access to raw storage for peer servers and CLI replication commands (`fetch`/`pull`/`push`). Tokens must carry `fluree.storage.*` claims — query-scoped tokens (`fluree.ledger.read/write.*`) are not sufficient. See [Replication vs Query Access](#replication-vs-query-access) above.

Enable storage proxy endpoints for peers without direct storage access:

| Flag                                   | Env Var                                     | Default |
| -------------------------------------- | ------------------------------------------- | ------- |
| `--storage-proxy-enabled`              | `FLUREE_STORAGE_PROXY_ENABLED`              | `false` |
| `--storage-proxy-trusted-issuer`       | `FLUREE_STORAGE_PROXY_TRUSTED_ISSUERS`      | None    |
| `--storage-proxy-default-identity`     | `FLUREE_STORAGE_PROXY_DEFAULT_IDENTITY`     | None    |
| `--storage-proxy-default-policy-class` | `FLUREE_STORAGE_PROXY_DEFAULT_POLICY_CLASS` | None    |
| `--storage-proxy-debug-headers`        | `FLUREE_STORAGE_PROXY_DEBUG_HEADERS`        | `false` |

```bash
# Ed25519 trust (did:key):
fluree-server \
  --storage-proxy-enabled \
  --storage-proxy-trusted-issuer did:key:z6Mk...

# OIDC/JWKS trust (same --jwks-issuer flag used by other endpoints):
fluree-server \
  --storage-proxy-enabled \
  --jwks-issuer "https://solo.example.com=https://solo.example.com/.well-known/jwks.json"
```

> **JWKS support**: When `--jwks-issuer` is configured, storage proxy endpoints accept RS256 OIDC tokens in addition to Ed25519 JWS tokens. The `--jwks-issuer` flag is shared with data, admin, and events endpoints — a single flag enables OIDC across all endpoint groups.

## Complete Configuration Examples

### Development (Memory Storage)

```bash
fluree-server \
  --log-level debug
```

### Single Server (File Storage)

```bash
fluree-server \
  --storage-path /var/lib/fluree \
  --indexing-enabled \
  --log-level info
```

### Production with Admin Auth

```bash
fluree-server \
  --storage-path /var/lib/fluree \
  --indexing-enabled \
  --admin-auth-mode required \
  --admin-auth-trusted-issuer did:key:z6Mk... \
  --log-level info
```

### Transaction Server with Events Auth

```bash
fluree-server \
  --storage-path /var/lib/fluree \
  --events-auth-mode required \
  --events-auth-trusted-issuer did:key:z6Mk... \
  --storage-proxy-enabled \
  --admin-auth-mode required
```

### Production with OIDC (All Endpoints)

```bash
fluree-server \
  --storage-path /var/lib/fluree \
  --indexing-enabled \
  --jwks-issuer "https://auth.example.com=https://auth.example.com/.well-known/jwks.json" \
  --data-auth-mode required \
  --events-auth-mode required \
  --admin-auth-mode required \
  --storage-proxy-enabled
```

### Query Peer (Shared Storage)

```bash
fluree-server \
  --server-role peer \
  --tx-server-url http://tx.internal:8090 \
  --storage-path /var/lib/fluree \
  --peer-subscribe-all \
  --peer-events-token @/etc/fluree/peer-token.jwt
```

### Query Peer (Proxy Storage)

```bash
fluree-server \
  --server-role peer \
  --tx-server-url http://tx.internal:8090 \
  --storage-access-mode proxy \
  --storage-proxy-token @/etc/fluree/storage-proxy.jwt \
  --peer-subscribe-all \
  --peer-events-token @/etc/fluree/peer-token.jwt
```

### S3 + DynamoDB (Connection Config)

```bash
fluree server run \
  --connection-config /etc/fluree/connection.jsonld \
  --indexing-enabled \
  --reindex-min-bytes 100000 \
  --reindex-max-bytes 5000000 \
  --cache-max-mb 4096
```

With a config file:

```toml
[server]
connection_config = "/etc/fluree/connection.jsonld"
cache_max_mb = 4096

[server.indexing]
enabled = true
reindex_min_bytes = 100000
reindex_max_bytes = 5000000

[server.auth.data]
mode = "required"
trusted_issuers = ["did:key:z6Mk..."]
```

### S3 Peer (Shared Storage via Connection Config)

```bash
fluree server run \
  --server-role peer \
  --tx-server-url http://tx.internal:8090 \
  --connection-config /etc/fluree/connection.jsonld \
  --peer-subscribe-all \
  --peer-events-token @/etc/fluree/peer-token.jwt
```

## Environment Variables Reference

| Variable                                | Description                                     | Default                                                                 |
| --------------------------------------- | ----------------------------------------------- | ----------------------------------------------------------------------- |
| `FLUREE_HOME`                           | Global Fluree directory (unified config + data) | Platform dirs (see [Global Directory Layout](#global-directory-layout)) |
| `FLUREE_CONFIG`                         | Config file path                                | `.fluree/config.{toml,jsonld}` (auto-discovered)                        |
| `FLUREE_PROFILE`                        | Configuration profile name                      | None                                                                    |
| `FLUREE_LISTEN_ADDR`                    | Server address:port                             | `0.0.0.0:8090`                                                          |
| `FLUREE_STORAGE_PATH`                   | File storage path                               | `.fluree/storage`                                                       |
| `FLUREE_CONNECTION_CONFIG`              | JSON-LD connection config file path             | None                                                                    |
| `FLUREE_CORS_ENABLED`                   | Enable CORS                                     | `true`                                                                  |
| `FLUREE_INDEXING_ENABLED`               | Enable background indexing                      | `true`                                                                  |
| `FLUREE_REINDEX_MIN_BYTES`              | Soft reindex threshold (bytes)                  | `100000`                                                                |
| `FLUREE_REINDEX_MAX_BYTES`              | Hard reindex threshold (bytes)                  | 20% of system RAM (256 MB fallback)                                      |
| `FLUREE_CACHE_MAX_MB`                   | Global cache budget (MB)                        | Tiered by RAM: `<4GB: 30%, 4-8GB: 40%, >=8GB: 35%`                                                     |
| `FLUREE_BODY_LIMIT`                     | Max request body bytes                          | `52428800`                                                              |
| `FLUREE_QUERY_TIMEOUT_MS`               | Max query execution time in milliseconds (`0` disables) | `900000`                                                     |
| `FLUREE_QUERY_MIN_T_TIMEOUT_MS`         | Max read-after-write min-t wait in milliseconds | `5000`                                                                  |
| `FLUREE_STREAM_HEARTBEAT_MS`            | Streaming-query heartbeat interval in ms (`0` disables) | `15000`                                                         |
| `FLUREE_QUERY_REFRESH_ENABLED`          | Enable TTL-gated nameservice refresh before current-head queries | `false`                                                       |
| `FLUREE_QUERY_REFRESH_TTL_MS`           | Query-time refresh interval per ledger per process (`0` checks every request) | `1000`                                           |
| `FLUREE_LOG_LEVEL`                      | Log level                                       | `info`                                                                  |
| `FLUREE_SERVER_ROLE`                    | Server role                                     | `transaction`                                                           |
| `FLUREE_TX_SERVER_URL`                  | Transaction server URL                          | None                                                                    |
| `FLUREE_EVENTS_AUTH_MODE`               | Events auth mode                                | `none`                                                                  |
| `FLUREE_EVENTS_AUTH_TRUSTED_ISSUERS`    | Events trusted issuers                          | None                                                                    |
| `FLUREE_DATA_AUTH_MODE`                 | Data API auth mode                              | `none`                                                                  |
| `FLUREE_DATA_AUTH_AUDIENCE`             | Data API expected audience                      | None                                                                    |
| `FLUREE_DATA_AUTH_TRUSTED_ISSUERS`      | Data API trusted issuers                        | None                                                                    |
| `FLUREE_DATA_AUTH_DEFAULT_POLICY_CLASS` | Data API default policy class                   | None                                                                    |
| `FLUREE_ADMIN_AUTH_MODE`                | Admin auth mode                                 | `none`                                                                  |
| `FLUREE_ADMIN_AUTH_TRUSTED_ISSUERS`     | Admin trusted issuers                           | None                                                                    |
| `FLUREE_MCP_ENABLED`                    | Enable MCP endpoint                             | `false`                                                                 |
| `FLUREE_MCP_AUTH_TRUSTED_ISSUERS`       | MCP trusted issuers                             | None                                                                    |
| `FLUREE_MCP_AGENT_JSON_MAX_BYTES`       | MCP `sparql_query` Agent JSON byte budget       | `32768`                                                                 |
| `FLUREE_MCP_QUERY_TIMEOUT_MS`           | MCP `sparql_query` execution timeout            | `300000`                                                                |
| `FLUREE_STORAGE_ACCESS_MODE`            | Peer storage mode                               | `shared`                                                                |
| `FLUREE_STORAGE_PROXY_ENABLED`          | Enable storage proxy                            | `false`                                                                 |

## Command-Line Reference

```bash
fluree-server --help
```

## Best Practices

### 1. Keep Secrets Out of Config Files

Tokens and credentials should not be stored as plaintext in config files (which may be committed to version control or readable by other processes). Three options, in order of preference:

**Environment variables** (recommended for production):

```bash
export FLUREE_PEER_EVENTS_TOKEN=$(cat /etc/fluree/token.jwt)
export FLUREE_STORAGE_PROXY_TOKEN=$(cat /etc/fluree/proxy-token.jwt)
```

**`@filepath` references** in config files or CLI flags (reads the file at startup):

```toml
[server.peer]
events_token = "@/etc/fluree/peer-token.jwt"
storage_proxy_token = "@/etc/fluree/proxy-token.jwt"
```

```bash
--peer-events-token @/etc/fluree/token.jwt
```

**Direct values** (development only): If a secret-bearing field contains a literal token in the config file, the server logs a warning at startup recommending `@filepath` or env vars.

The following config file fields support `@filepath` resolution:

| Config file key            | Env var alternative          |
| -------------------------- | ---------------------------- |
| `peer.events_token`        | `FLUREE_PEER_EVENTS_TOKEN`   |
| `peer.storage_proxy_token` | `FLUREE_STORAGE_PROXY_TOKEN` |

### 2. Enable Admin Auth in Production

Always protect admin endpoints in production:

```bash
fluree-server \
  --admin-auth-mode required \
  --admin-auth-trusted-issuer did:key:z6Mk...
```

### 3. Use File Storage for Persistence

Memory storage is lost on restart:

```bash
# Development only
fluree-server

# Production
fluree-server --storage-path /var/lib/fluree
```

### 4. Monitor Logs

Use structured logging for production:

```bash
fluree-server --log-level info 2>&1 | jq .
```

## Remote Connections

Remote connections enable SPARQL `SERVICE` federation against other Fluree instances. A remote connection maps a name to a server URL and bearer token. Once registered, queries can reference any ledger on that server using `SERVICE <fluree:remote:<name>/<ledger>> { ... }`.

### Rust API

Register remote connections on the `FlureeBuilder`:

```rust
let fluree = FlureeBuilder::file("./data")
    .remote_connection("acme", "https://acme-fluree.example.com", Some(token))
    .remote_connection("partner", "https://partner.example.com", None)
    .build()?;
```

Each call registers a named connection. The name is used in SPARQL queries:

```sparql
SERVICE <fluree:remote:acme/customers:main> { ?s ?p ?o }
SERVICE <fluree:remote:partner/inventory:main> { ?item ex:sku ?sku }
```

### Connection Parameters

| Parameter | Description |
|-----------|-------------|
| `name` | Alias used in `fluree:remote:<name>/...` URIs |
| `base_url` | Server URL (e.g., `https://acme-fluree.example.com`). The query path `/v1/fluree/query/{ledger}` is appended automatically. |
| `token` | Optional bearer token for authentication. Sent as `Authorization: Bearer <token>` on every request. |

The default per-request timeout is 30 seconds. Requests that exceed this produce a query error (or empty results with `SERVICE SILENT`).

### Security

Bearer tokens are stored in memory on the `Fluree` instance. They are never serialized to storage, included in nameservice records, or exposed through info/admin endpoints. If the token needs rotation, rebuild the `Fluree` instance with an updated token, or use `set_remote_service()` to inject a custom executor with token refresh logic.

### Feature Flag

The HTTP transport for remote SERVICE requires the `search-remote-client` Cargo feature (which enables `reqwest`). Without this feature, remote connections can be registered but queries against them will fail at runtime. The feature is enabled by default in the server binary.

See [SPARQL: Remote Fluree Federation](../query/sparql.md#remote-fluree-federation) for query syntax and examples.

## Memory Allocator (mimalloc)

`fluree-db-server` and `fluree-db-cli` expose an opt-in `mimalloc` Cargo feature
that swaps the global allocator to [mimalloc](https://github.com/microsoft/mimalloc).
It is **off by default** — enable it in your release build:

```bash
cargo build --release -p fluree-db-server --features mimalloc
```

**When it helps:** allocation-heavy, multicore query paths — most notably
R2RML/Iceberg graph-source scans, which materialize many RDF terms in parallel.
On those workloads the system allocator's lock contention caps the parallel
speedup; mimalloc roughly doubles the realized gain (e.g. TPC-H Q1 over a 6M-row
Iceberg table: ~62s with the system allocator vs ~31s with mimalloc).

**Tradeoffs (operational, not semantic):**

- Changes the allocator for *every* allocation in the process (HTTP, query,
  import, caches, AWS/Iceberg clients, tracing buffers).
- Usually better multicore throughput and lower fragmentation, but may retain
  more RSS due to per-thread heaps — relevant for large scans that already peak
  high. Track RSS, p95 query latency, and concurrent-query behavior during a
  soak before making it the default.
- New native dependency in the binary; verify it builds for your target
  (e.g. musl/cross builds) and keep a system-allocator fallback build.

Pair it with bounded query parallelism so a single large scan cannot monopolize
cores and allocator pressure under concurrent load.

## Related Documentation

- [Query Peers](query-peers.md) - Peer mode and replication
- [Storage Modes](storage.md) - Storage backend details
- [Telemetry](telemetry.md) - Monitoring configuration
- [Admin and Health](admin-and-health.md) - Health check endpoints


---

# operations/docker.md

# Running Fluree with Docker

The official image (`fluree/server`) ships the `fluree` binary on a slim Debian base. This guide covers what's inside the image, how to configure it (env vars, mounted config files, CLI flags), and worked recipes for the common production patterns.

## What's in the Image

| Aspect             | Value                                  |
| ------------------ | -------------------------------------- |
| Base               | `debian:trixie-slim`                   |
| Entrypoint         | `/usr/local/bin/fluree-entrypoint.sh`  |
| Default command    | `fluree server run`                    |
| `WORKDIR`          | `/var/lib/fluree`                      |
| `VOLUME`           | `/var/lib/fluree`                      |
| Exposed port       | `8090`                                 |
| Runtime user       | `fluree` (UID `1000`, GID `1000`)      |
| Healthcheck        | `GET /health` every 30s                |
| Default log filter | `RUST_LOG=info`                        |

**Entrypoint behavior:** on first start, if `/var/lib/fluree/.fluree/` does not exist, the entrypoint runs `fluree init` to create a default `.fluree/config.toml` and `.fluree/storage/` directory. Subsequent starts skip init. Any arguments passed to `docker run` after the image name are forwarded to `fluree server run`, so you can append CLI flags (e.g. `--log-level debug`) directly.

## Quick Start

```bash
docker run --rm -p 8090:8090 fluree/server:latest
```

Verify:

```bash
curl http://localhost:8090/health
```

Data lives inside the container's writable layer here — fine for trying things out, lost when the container is removed. For anything beyond a smoke test, mount a volume.

## Persisting Data

The image declares `VOLUME /var/lib/fluree`. Mount a host directory or named volume there:

```bash
# Named volume (recommended)
docker run -d --name fluree \
  -p 8090:8090 \
  -v fluree-data:/var/lib/fluree \
  fluree/server:latest

# Host bind mount — make sure the directory is writable by UID 1000
mkdir -p ./fluree-data && sudo chown 1000:1000 ./fluree-data
docker run -d --name fluree \
  -p 8090:8090 \
  -v "$PWD/fluree-data:/var/lib/fluree" \
  fluree/server:latest
```

The volume holds both `.fluree/config.toml` (config) and `.fluree/storage/` (ledger data) by default.

## Three Ways to Configure

Fluree resolves configuration with this precedence (highest wins):

1. **CLI flags** appended after the image name
2. **Environment variables** (`FLUREE_*`) set with `-e` or `environment:`
3. **Profile overrides** (`[profiles.<name>.server]`) when you pass `--profile`
4. **Config file** at `.fluree/config.toml` or `.fluree/config.jsonld`
5. **Built-in defaults**

You can use any one of these — or, more typically, layer them: bake a base config file into a volume, then tweak per-environment with env vars or compose overrides.

> **Heads up — log level:** The Dockerfile sets `ENV RUST_LOG=info`. The console log filter uses `RUST_LOG` if it is non-empty and only falls back to `FLUREE_LOG_LEVEL` when `RUST_LOG` is unset. Inside this image you must override `RUST_LOG` to change console verbosity:
>
> ```bash
> docker run -e RUST_LOG=debug fluree/server:latest
> ```

### 1. Environment Variables Only

Every CLI flag has a `FLUREE_*` env var equivalent (see [Configuration](configuration.md)). For simple deployments this is the lowest-friction path:

```bash
docker run -d --name fluree \
  -p 8090:8090 \
  -v fluree-data:/var/lib/fluree \
  -e FLUREE_LISTEN_ADDR=0.0.0.0:8090 \
  -e FLUREE_STORAGE_PATH=/var/lib/fluree/.fluree/storage \
  -e FLUREE_INDEXING_ENABLED=true \
  -e FLUREE_REINDEX_MIN_BYTES=1000000 \
  -e FLUREE_REINDEX_MAX_BYTES=10000000 \
  -e FLUREE_CACHE_MAX_MB=2048 \
  -e RUST_LOG=info \
  fluree/server:latest
```

### 2. Mounted Config File (JSON-LD or TOML)

Author a config file on the host, then mount it at `/var/lib/fluree/.fluree/config.jsonld` (or `.toml`). The server walks up from `WORKDIR=/var/lib/fluree` and picks it up automatically.

`./fluree-config/config.jsonld`:

```json
{
  "@context": { "@vocab": "https://ns.flur.ee/config#" },
  "server": {
    "listen_addr": "0.0.0.0:8090",
    "storage_path": "/var/lib/fluree/.fluree/storage",
    "log_level": "info",
    "cache_max_mb": 2048,
    "indexing": {
      "enabled": true,
      "reindex_min_bytes": 1000000,
      "reindex_max_bytes": 10000000
    }
  },
  "profiles": {
    "prod": {
      "server": {
        "log_level": "warn",
        "cache_max_mb": 8192
      }
    }
  }
}
```

```bash
docker run -d --name fluree \
  -p 8090:8090 \
  -v fluree-data:/var/lib/fluree \
  -v "$PWD/fluree-config/config.jsonld:/var/lib/fluree/.fluree/config.jsonld:ro" \
  fluree/server:latest --profile prod
```

If both `config.toml` and `config.jsonld` exist in the same directory, TOML wins and the server logs a warning. Pick one format.

The TOML equivalent (`./fluree-config/config.toml`):

```toml
[server]
listen_addr = "0.0.0.0:8090"
storage_path = "/var/lib/fluree/.fluree/storage"
log_level = "info"
cache_max_mb = 2048

[server.indexing]
enabled = true
reindex_min_bytes = 1000000
reindex_max_bytes = 10000000

[profiles.prod.server]
log_level = "warn"
cache_max_mb = 8192
```

You can also stash the config outside `WORKDIR` and point at it explicitly:

```bash
docker run -d --name fluree \
  -p 8090:8090 \
  -v fluree-data:/var/lib/fluree \
  -v "$PWD/fluree-config:/etc/fluree:ro" \
  fluree/server:latest --config /etc/fluree/config.jsonld
```

### 3. Layered: File + Env Var Overrides

The common production shape: bake the base config into the image or volume, then let the orchestrator override per-environment with `FLUREE_*` env vars. Env vars beat the file — no file edit needed to bump cache size in staging vs. prod.

```bash
docker run -d --name fluree \
  -p 8090:8090 \
  -v fluree-data:/var/lib/fluree \
  -v "$PWD/fluree-config/config.jsonld:/var/lib/fluree/.fluree/config.jsonld:ro" \
  -e FLUREE_CACHE_MAX_MB=4096 \
  -e RUST_LOG=warn \
  fluree/server:latest
```

## Common Configuration Recipes

### Tuning the LRU Cache

`cache_max_mb` is the global budget for the in-memory index/flake cache. The default is tiered by effective system memory: <4GB uses 30%, 4-8GB uses 40%, and >=8GB uses 35%. On a container with a hard memory limit, the auto-tier clamps to the cgroup limit when available; set this explicitly if you need a tighter cache budget.

```yaml
# docker-compose.yml fragment
services:
  fluree:
    image: fluree/server:latest
    mem_limit: 6g
    environment:
      FLUREE_CACHE_MAX_MB: 3072    # ~50% of the cgroup limit
```

Or in JSON-LD:

```json
{
  "@context": { "@vocab": "https://ns.flur.ee/config#" },
  "server": { "cache_max_mb": 3072 }
}
```

### Background Indexing

Indexing is **off by default**. Enable it for production write workloads — without it, every commit writes to novelty and queries get slower as novelty grows.

| Setting              | Meaning                                                      |
| -------------------- | ------------------------------------------------------------ |
| `indexing.enabled`   | Turn the background indexer on                               |
| `reindex_min_bytes`  | Soft threshold — novelty above this triggers a background reindex |
| `reindex_max_bytes`  | Hard threshold — commits **block** above this until reindexing catches up |

Tune `min`/`max` based on commit volume. Defaults (100 KB / 1 MB) are conservative; busy ledgers should raise both:

```toml
[server.indexing]
enabled = true
reindex_min_bytes = 5000000     # 5 MB — start indexing in the background
reindex_max_bytes = 50000000    # 50 MB — block commits at this point
```

```bash
docker run -d \
  -e FLUREE_INDEXING_ENABLED=true \
  -e FLUREE_REINDEX_MIN_BYTES=5000000 \
  -e FLUREE_REINDEX_MAX_BYTES=50000000 \
  fluree/server:latest
```

### CORS and Request Body Size

```toml
[server]
cors_enabled = true
body_limit = 104857600    # 100 MB — raise for bulk imports
```

### Authentication (Production)

Require a Bearer token on data and admin endpoints. The trusted issuer is the `did:key` of your token signer.

```toml
[server.auth.data]
mode = "required"
trusted_issuers = ["did:key:z6Mk..."]

[server.auth.admin]
mode = "required"
trusted_issuers = ["did:key:z6Mk..."]
```

For OIDC/JWKS (e.g. an external IdP), set `--jwks-issuer` or `FLUREE_JWKS_ISSUERS`:

```bash
docker run -d \
  -e FLUREE_DATA_AUTH_MODE=required \
  -e FLUREE_JWKS_ISSUERS="https://auth.example.com=https://auth.example.com/.well-known/jwks.json" \
  fluree/server:latest
```

See [Configuration → Authentication](configuration.md#authentication-configuration) for the full matrix.

### S3 + DynamoDB (Distributed Storage)

For multi-node or cloud deployments, point the server at a JSON-LD **connection config** describing your storage and nameservice. AWS credentials come from the standard SDK chain (env vars, IAM role, etc.) — they are **not** part of the connection config.

`./fluree-config/connection.jsonld`:

```json
{
  "@context": {
    "@base": "https://ns.flur.ee/config/connection/",
    "@vocab": "https://ns.flur.ee/system#"
  },
  "@graph": [
    { "@id": "commitStorage", "@type": "Storage",
      "s3Bucket": "fluree-prod-commits", "s3Prefix": "data/" },
    { "@id": "indexStorage", "@type": "Storage",
      "s3Bucket": "fluree-prod-indexes" },
    { "@id": "publisher", "@type": "Publisher",
      "dynamodbTable": "fluree-nameservice", "dynamodbRegion": "us-east-1" },
    { "@id": "conn", "@type": "Connection",
      "commitStorage": { "@id": "commitStorage" },
      "indexStorage":  { "@id": "indexStorage" },
      "primaryPublisher": { "@id": "publisher" } }
  ]
}
```

```bash
docker run -d --name fluree \
  -p 8090:8090 \
  -v "$PWD/fluree-config:/etc/fluree:ro" \
  -e AWS_REGION=us-east-1 \
  -e AWS_ACCESS_KEY_ID=... \
  -e AWS_SECRET_ACCESS_KEY=... \
  -e FLUREE_CONNECTION_CONFIG=/etc/fluree/connection.jsonld \
  -e FLUREE_INDEXING_ENABLED=true \
  fluree/server:latest
```

`--connection-config` and `--storage-path` are mutually exclusive. See [Configuration → Connection Configuration](configuration.md#connection-configuration-s3-dynamodb-etc) and the [DynamoDB guide](dynamodb-guide.md) for backend-specific setup.

### Search Service (`fluree-search-httpd`)

Run a dedicated BM25 / vector search service alongside the main server when search traffic is heavy enough that you want it isolated from the transactional path. The service is a separate binary with its own listen port — it is **not** mounted under the main server's `api_base_url`. It needs read access to the same storage and nameservice paths the main server writes to.

```bash
docker run -d --name fluree-search \
  -p 9090:9090 \
  -v fluree-data:/var/lib/fluree \
  -e FLUREE_STORAGE_ROOT=/var/lib/fluree/storage \
  -e FLUREE_NAMESERVICE_PATH=/var/lib/fluree/ns \
  fluree/search-httpd:latest
```

| Env var | Default | Purpose |
|---------|---------|---------|
| `FLUREE_STORAGE_ROOT` | (required) | Storage path (file:// optional) |
| `FLUREE_NAMESERVICE_PATH` | (required) | Nameservice path |
| `FLUREE_SEARCH_LISTEN` | `0.0.0.0:9090` | Listen address |
| `FLUREE_SEARCH_CACHE_MAX_ENTRIES` | `100` | Max cached indexes |
| `FLUREE_SEARCH_CACHE_TTL_SECS` | `300` | Cache TTL |
| `FLUREE_SEARCH_MAX_LIMIT` | `1000` | Max results per query |
| `FLUREE_SEARCH_DEFAULT_TIMEOUT_MS` | `30000` | Default request timeout |
| `FLUREE_SEARCH_MAX_TIMEOUT_MS` | `300000` | Maximum allowed timeout |

**Prerequisites.** The service only serves queries against indexes that already exist on the shared volume. BM25 / vector graph-source indexes are created via the Rust API today (`Bm25CreateConfig` + `create_full_text_index`, or `VectorCreateConfig` + `create_vector_index`). The `@fulltext` datatype and the `f:fullTextDefaults` config-graph paths are managed entirely through the main server's HTTP API and don't require this dedicated service.

**Compose example with both services sharing a volume:**

```yaml
services:
  fluree:
    image: fluree/server:latest
    ports:
      - "8090:8090"
    volumes:
      - fluree-data:/var/lib/fluree
    environment:
      RUST_LOG: info
      FLUREE_INDEXING_ENABLED: "true"

  fluree-search:
    image: fluree/search-httpd:latest
    depends_on:
      - fluree
    ports:
      - "9090:9090"
    volumes:
      - fluree-data:/var/lib/fluree:ro     # read-only is sufficient
    environment:
      RUST_LOG: info
      FLUREE_STORAGE_ROOT: /var/lib/fluree/storage
      FLUREE_NAMESERVICE_PATH: /var/lib/fluree/ns

volumes:
  fluree-data:
```

Clients send search requests to `POST http://fluree-search:9090/v1/search`. See [BM25 → Remote Search Service](../graph-sources/bm25.md#remote-search-service) for the request/response protocol.

### Query Peer

Run as a read-only peer that subscribes to a transaction server's event stream:

```bash
docker run -d --name fluree-peer \
  -p 8090:8090 \
  -v fluree-peer-data:/var/lib/fluree \
  -e FLUREE_SERVER_ROLE=peer \
  -e FLUREE_TX_SERVER_URL=http://tx.internal:8090 \
  fluree/server:latest --peer-subscribe-all
```

See [Query peers and replication](query-peers.md) for the proxy-mode and auth options.

## Docker Compose: Full Example

A production-leaning single-node setup with a mounted JSON-LD config, env-var overrides, named data volume, and resource limits:

```yaml
services:
  fluree:
    image: fluree/server:latest
    container_name: fluree
    restart: unless-stopped
    ports:
      - "8090:8090"
    volumes:
      - fluree-data:/var/lib/fluree
      - ./fluree-config/config.jsonld:/var/lib/fluree/.fluree/config.jsonld:ro
    environment:
      RUST_LOG: info
      FLUREE_CACHE_MAX_MB: 4096
      FLUREE_INDEXING_ENABLED: "true"
      FLUREE_REINDEX_MIN_BYTES: "5000000"
      FLUREE_REINDEX_MAX_BYTES: "50000000"
      # Auth — point at your trusted did:key signer
      FLUREE_DATA_AUTH_MODE: required
      FLUREE_DATA_AUTH_TRUSTED_ISSUERS: did:key:z6Mk...
      FLUREE_ADMIN_AUTH_MODE: required
      FLUREE_ADMIN_AUTH_TRUSTED_ISSUERS: did:key:z6Mk...
    mem_limit: 8g
    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8090/health"]
      interval: 30s
      timeout: 3s
      start_period: 15s
      retries: 3
    command: ["--profile", "prod"]

volumes:
  fluree-data:
```

```bash
docker compose up -d
docker compose logs -f fluree
```

## Troubleshooting

**Container restarts after `fluree init`.** First-run init only runs when `/var/lib/fluree/.fluree/` is missing. If the volume is owned by a non-`1000` UID, init fails. Fix with `sudo chown -R 1000:1000 ./fluree-data` on the host.

**Mounted config file is ignored.** Confirm the mount path and the file extension. The server only auto-discovers `.fluree/config.toml` or `.fluree/config.jsonld` under the working directory. Anything else needs `--config <path>` (or `FLUREE_CONFIG=<path>`). If both formats are present in the same directory, TOML wins — check the startup logs for the warning.

**Setting `FLUREE_LOG_LEVEL` doesn't change console output.** The image's `ENV RUST_LOG=info` shadows it. Override with `-e RUST_LOG=debug` instead.

**`cache_max_mb` auto-default is too large under a memory limit.** Current builds clamp the auto-tier to the cgroup limit when available. If logs still show an oversized cache, check for an older image/binary or an explicit `FLUREE_CACHE_MAX_MB` / `cache_max_mb` override.

**Health check failing.** `curl http://localhost:8090/health` from your host. If the server is up but the healthcheck fails, the listen address is probably bound to `127.0.0.1` inside the container — set `FLUREE_LISTEN_ADDR=0.0.0.0:8090`.

## Related Documentation

- [Configuration reference](configuration.md) — full flag/env/file matrix
- [Storage modes](storage.md) — memory / file / AWS / IPFS
- [JSON-LD connection configuration](../reference/connection-config-jsonld.md) — schema for `connection.jsonld`
- [Query peers and replication](query-peers.md) — peer-mode deployments
- [Quickstart: Server](../getting-started/quickstart-server.md) — first-run walkthrough


---

# operations/storage.md

# Storage Modes

Fluree supports four storage modes, each optimized for different deployment scenarios. This document provides detailed information about each storage mode and guidance for choosing the right one.

## Storage Modes

### Memory Storage

In-memory storage for development and testing:

```bash
./fluree-db-server --storage memory
```

**Characteristics:**
- Data stored in RAM only
- No persistence (data lost on restart)
- Fastest performance
- No external dependencies

**Use Cases:**
- Local development
- Unit testing
- Temporary/ephemeral databases
- Prototyping

**Limitations:**
- No durability (data lost on crash/restart)
- Limited by available RAM
- Single process only

### File Storage

Local file system storage:

```bash
./fluree-db-server \
  --storage file \
  --data-dir /var/lib/fluree
```

**Characteristics:**
- Data persisted to local disk
- Survives server restarts
- Good performance (SSD recommended)
- Simple setup

**Use Cases:**
- Single-server production
- Development with persistence
- Edge deployments
- Small to medium scale

**Limitations:**
- Single machine only
- No built-in replication
- Limited by disk capacity
- No cross-region support

### AWS Storage

Distributed storage using S3 and DynamoDB:

```bash
./fluree-db-server \
  --storage aws \
  --s3-bucket fluree-prod-data \
  --s3-region us-east-1 \
  --dynamodb-table fluree-nameservice \
  --dynamodb-region us-east-1
```

**Characteristics:**
- Distributed, scalable storage
- Multi-process coordination
- Cross-region replication
- High durability (99.999999999%)

**Use Cases:**
- Multi-server production
- High availability requirements
- Geographic distribution
- Cloud-native applications

**Limitations:**
- Requires AWS account
- Higher latency than local storage
- Usage costs
- More complex setup

For cloud and serverless deployments, commit and index storage can be split.
Prefer Standard S3 for commits because commits are the durable source of truth.
Indexes are reproducible from commits, so they can use either Standard S3 or S3
Express One Zone depending on latency and cost requirements. See
[Serverless Storage Choices](serverless-storage.md) for benchmark-backed guidance.

### IPFS Storage

Decentralized content-addressed storage via a local Kubo node:

```json
{
  "@context": {"@vocab": "https://ns.flur.ee/system#"},
  "@graph": [{
    "@type": "Connection",
    "indexStorage": {
      "@type": "Storage",
      "ipfsApiUrl": "http://127.0.0.1:5001",
      "ipfsPinOnPut": true
    }
  }]
}
```

**Characteristics:**
- Content-addressed (every blob identified by SHA-256 hash)
- Immutable, tamper-evident storage
- Decentralized replication via IPFS network
- Fluree's native CIDs work directly with IPFS

**Use Cases:**
- Decentralized / censorship-resistant deployments
- Content integrity verification
- Cross-organization data sharing
- Foundation for IPNS/ENS-based ledger discovery

**Limitations:**
- Requires a running Kubo node
- No prefix listing (manifest-based tracking needed)
- No native deletion (unpin + GC)
- Higher write latency than local file I/O

See [IPFS Storage Guide](ipfs-storage.md) for complete setup and configuration.

## Storage Architecture

### Memory Storage

```text
┌──────────────────────┐
│   Fluree Process     │
│  ┌────────────────┐  │
│  │  Hash Map      │  │
│  │  (In Memory)   │  │
│  └────────────────┘  │
└──────────────────────┘
```

All data in process memory.

### File Storage

```text
┌──────────────────────┐
│   Fluree Process     │
│  ┌────────────────┐  │
│  │   File I/O     │  │
│  └────────┬───────┘  │
└───────────┼──────────┘
            │
     ┌──────▼──────┐
     │ File System │
     │  /var/lib/  │
     │   fluree/   │
     └─────────────┘
```

Data persisted to local files.

### AWS Storage

```text
┌──────────────────────┐  ┌──────────────────────┐
│   Fluree Process 1   │  │   Fluree Process 2   │
│  ┌────────────────┐  │  │  ┌────────────────┐  │
│  │  AWS SDK       │  │  │  │  AWS SDK       │  │
│  └────────┬───────┘  │  │  └────────┬───────┘  │
└───────────┼──────────┘  └───────────┼──────────┘
            │                         │
            └────────┬────────────────┘
                     │
          ┌──────────▼──────────┐
          │     AWS Cloud       │
          │  ┌──────┐  ┌──────┐│
          │  │  S3  │  │Dynamo││
          │  └──────┘  └──────┘│
          └─────────────────────┘
```

Multiple processes coordinate via AWS.

### IPFS Storage

```text
┌──────────────────────┐
│   Fluree Process     │
│  ┌────────────────┐  │
│  │  IpfsStorage   │  │
│  │  (HTTP client) │  │
│  └────────┬───────┘  │
└───────────┼──────────┘
            │ HTTP RPC
     ┌──────▼──────┐
     │  Kubo Node  │
     │  (IPFS)     │
     └──────┬──────┘
            │ libp2p
     ┌──────▼──────┐
     │  IPFS P2P   │
     │  Network    │
     └─────────────┘
```

Data stored as content-addressed blocks in IPFS via Kubo.

## Storage Encryption

Fluree supports transparent AES-256-GCM encryption for data at rest. When enabled, all data is automatically encrypted before being written to storage.

### Enabling Encryption

```bash
# Generate a 32-byte encryption key
export FLUREE_ENCRYPTION_KEY=$(openssl rand -base64 32)
```

Configure via JSON-LD (file storage):

```json
{
  "@context": {"@vocab": "https://ns.flur.ee/system#"},
  "@graph": [{
    "@type": "Connection",
    "indexStorage": {
      "@type": "Storage",
      "filePath": "/var/lib/fluree",
      "AES256Key": {"envVar": "FLUREE_ENCRYPTION_KEY"}
    }
  }]
}
```

For S3 storage with encryption:

```json
{
  "@context": {"@vocab": "https://ns.flur.ee/system#"},
  "@graph": [{
    "@type": "Connection",
    "indexStorage": {
      "@type": "Storage",
      "s3Bucket": "my-fluree-bucket",
      "s3Endpoint": "https://s3.us-east-1.amazonaws.com",
      "AES256Key": {"envVar": "FLUREE_ENCRYPTION_KEY"}
    }
  }]
}
```

**Key Features:**
- AES-256-GCM authenticated encryption
- Works natively with all storage backends (memory, file, S3)
- Transparent encryption/decryption on read/write
- Portable ciphertext format (encrypted data can be moved between backends)
- Environment variable support for key configuration

See [Storage Encryption](../security/encryption.md) for full documentation.

## File Storage Details

### Directory Structure

```text
/var/lib/fluree/
├── ns@v2/                    # Nameservice records
│   ├── mydb/
│   │   ├── main.json        # Ledger metadata
│   │   └── dev.json
│   └── customers/
│       └── main.json
├── mydb/
│   ├── main/
│   │   ├── commit/          # Commit blobs (*.fcv2)
│   │   ├── txn/             # Transaction metadata (*.json)
│   │   ├── config/          # Ledger config blobs
│   │   └── index/
│   │       ├── roots/       # Index root descriptors (*.fir6)
│   │       ├── objects/
│   │       │   ├── branches/
│   │       │   ├── leaves/
│   │       │   └── history/
│   │       ├── garbage/
│   │       ├── stats/
│   │       └── spatial/
│   ├── dev/
│   │   └── ...
│   └── @shared/
│       └── dicts/           # Dictionaries shared by all branches
└── graph-sources/            # Graph sources
    └── products-search/
        └── main/
            ├── mapping/
            └── snapshots/
```

### File Formats

**Nameservice (JSON):**
```json
{
  "ledger_id": "mydb:main",
  "name": "mydb",
  "branch": "main",
  "commit_t": 150,
  "index_t": 145,
  "commit_head_id": "bafybeig...commitT150",
  "index_head_id": "bafybeig...indexRootT145",
  "retracted": false
}
```

**Commits (Binary):**
- Compressed flake data
- Transaction metadata
- Cryptographic signatures

**Indexes (Binary):**
- Root descriptors, branch manifests, leaf pages, and history sidecars
- Optimized for query performance

**Shared dictionaries (Binary):**
- Cross-branch dictionary blobs under `{ledger}/@shared/dicts/`
- May be referenced by more than one branch of the same ledger

### File System Requirements

**Minimum:**
- 10 GB free space
- SSD recommended (HDD acceptable)
- Sufficient IOPS for workload

**Recommended:**
- 100 GB+ free space
- NVMe SSD
- High IOPS capability
- Regular backups

## AWS Storage Details

### S3 Structure

```text
s3://fluree-prod-data/
├── mydb/
│   ├── main/
│   │   ├── commit/
│   │   ├── txn/
│   │   ├── config/
│   │   └── index/
│   │       ├── roots/
│   │       ├── objects/
│   │       │   ├── branches/
│   │       │   ├── leaves/
│   │       │   └── history/
│   │       ├── garbage/
│   │       ├── stats/
│   │       └── spatial/
│   └── @shared/
│       └── dicts/
└── graph-sources/
    └── products-search/
        └── main/
            ├── mapping/
            └── snapshots/
```

### DynamoDB Schema

The nameservice uses a DynamoDB table with a **composite primary key** (`pk` + `sk`) for ledger and graph source metadata coordination. Each ledger or graph source is stored as multiple items (one per concern) under the same partition key.

See [DynamoDB Nameservice Guide](dynamodb-guide.md) for:
- Complete table schema with composite-key layout
- Table creation scripts (AWS CLI, CloudFormation, Terraform)
- GSI setup for listing by kind
- Local development setup with LocalStack
- Production considerations and troubleshooting

**Quick Reference:**
```text
Table: fluree-nameservice
Primary Key: pk (String, ledger-id) + sk (String, concern)
Sort Key Values: meta, head, index, config, status
GSI1 (gsi1-kind): kind (HASH) + pk (RANGE)
Items per ledger: 5 (meta, head, index, config, status)
Items per graph source: 4 (meta, config, index, status)
```

### AWS Permissions

Required IAM permissions:

**S3:**
```json
{
  "Effect": "Allow",
  "Action": [
    "s3:GetObject",
    "s3:PutObject",
    "s3:ListBucket",
    "s3:DeleteObject"
  ],
  "Resource": [
    "arn:aws:s3:::fluree-prod-data",
    "arn:aws:s3:::fluree-prod-data/*"
  ]
}
```

**DynamoDB:**
```json
{
  "Effect": "Allow",
  "Action": [
    "dynamodb:GetItem",
    "dynamodb:PutItem",
    "dynamodb:UpdateItem",
    "dynamodb:DeleteItem",
    "dynamodb:Query",
    "dynamodb:BatchGetItem",
    "dynamodb:BatchWriteItem"
  ],
  "Resource": [
    "arn:aws:dynamodb:us-east-1:*:table/fluree-nameservice",
    "arn:aws:dynamodb:us-east-1:*:table/fluree-nameservice/index/gsi1-kind"
  ]
}
```

### Cost Considerations

**S3 Costs:**
- Storage: ~$0.023/GB/month (Standard)
- PUT requests: ~$0.005/1000 requests
- GET requests: ~$0.0004/1000 requests

**DynamoDB Costs:**
- Provisioned: ~$0.25/WCU/month + $0.05/RCU/month
- On-Demand: ~$1.25/million writes + $0.25/million reads

**Typical Monthly Costs (medium deployment):**
- S3: $50-200 (depending on data size)
- DynamoDB: $10-50 (depending on traffic)
- Total: $60-250/month

## Choosing a Storage Mode

### Decision Matrix

| Requirement | Memory | File | AWS | IPFS |
|-------------|--------|------|-----|------|
| **Development** | Best | Good | Overkill | Overkill |
| **Single server** | No | Best | Overkill | Good |
| **Multi-server** | No | No | Best | Good |
| **Persistence** | No | Yes | Yes | Yes |
| **Cloud-native** | No | No | Yes | No |
| **Decentralized** | No | No | No | Best |
| **Content integrity** | No | No | No | Best |
| **Cost** | Free | Free | Monthly | Free |
| **Setup complexity** | Trivial | Simple | Complex | Moderate |
| **Performance** | Fastest | Fast | Good | Good |
| **Durability** | None | Local | 11 9's | Network-wide |

### Recommendations

**Use Memory when:**
- Developing locally
- Running tests
- Data is temporary
- Maximum performance needed

**Use File when:**
- Single server deployment
- Local persistence needed
- Simple setup preferred
- Predictable costs important

**Use AWS when:**
- Multiple servers needed
- High availability required
- Geographic distribution needed
- Cloud-native architecture

**Use IPFS when:**
- Decentralized storage required
- Content integrity verification is critical
- Cross-organization data sharing
- Building toward IPNS/ENS-based ledger discovery
- Censorship resistance is a requirement

## Switching Storage Modes

### Memory to File

Export from the running system and import into the new one:

```bash
# Export from memory
curl -X POST http://localhost:8090/export?ledger=mydb:main > mydb-export.jsonld

# Stop memory server, start file server
./fluree-db-server --storage file --data-dir /var/lib/fluree

# Import to file storage
curl -X POST "http://localhost:8090/v1/fluree/insert?ledger=mydb:main" \
  --data-binary @mydb-export.jsonld
```

### File to AWS

Copy files to S3 and create the nameservice table:

```bash
# Copy data directory to S3
aws s3 sync /var/lib/fluree/ s3://fluree-prod-data/

# Create DynamoDB table (see docs/operations/dynamodb-guide.md for full schema)
aws dynamodb create-table \
  --table-name fluree-nameservice \
  --attribute-definitions \
    AttributeName=pk,AttributeType=S \
    AttributeName=sk,AttributeType=S \
    AttributeName=kind,AttributeType=S \
  --key-schema \
    AttributeName=pk,KeyType=HASH \
    AttributeName=sk,KeyType=RANGE \
  --billing-mode PAY_PER_REQUEST

# Start AWS-backed server
./fluree-db-server --storage aws --s3-bucket fluree-prod-data
```

### AWS to File

Download from S3:

```bash
# Download data from S3
aws s3 sync s3://fluree-prod-data/ /var/lib/fluree/

# Start file-backed server
./fluree-db-server --storage file --data-dir /var/lib/fluree
```

## Backup and Recovery

### Memory Storage

No native backup (data is ephemeral):

```bash
# Export ledger
curl -X POST http://localhost:8090/export?ledger=mydb:main > backup.jsonld
```

### File Storage

Backup data directory:

```bash
# Stop server (recommended)
systemctl stop fluree

# Backup
tar -czf fluree-backup-$(date +%Y%m%d).tar.gz /var/lib/fluree/

# Start server
systemctl start fluree
```

For online backups, prefer storage-level snapshots or object-store versioning.
The standalone server does not currently expose HTTP read-only toggle endpoints.

### AWS Storage

Use S3 versioning and lifecycle policies:

```bash
# Enable versioning
aws s3api put-bucket-versioning \
  --bucket fluree-prod-data \
  --versioning-configuration Status=Enabled

# Configure lifecycle
aws s3api put-bucket-lifecycle-configuration \
  --bucket fluree-prod-data \
  --lifecycle-configuration file://lifecycle.json
```

DynamoDB backups:

```bash
# Enable point-in-time recovery
aws dynamodb update-continuous-backups \
  --table-name fluree-nameservice \
  --point-in-time-recovery-specification PointInTimeRecoveryEnabled=true
```

## Troubleshooting

### File Storage

**Permission Errors:**
```bash
sudo chown -R fluree:fluree /var/lib/fluree
chmod -R 755 /var/lib/fluree
```

**Disk Full:**
```bash
# Check space
df -h /var/lib/fluree

# Force a full index refresh
curl -X POST http://localhost:8090/v1/fluree/reindex \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb:main"}'
```

### AWS Storage

**Connection Errors:**
- Verify AWS credentials
- Check IAM permissions
- Verify S3 bucket exists
- Check DynamoDB table exists

**Throttling:**
- Increase DynamoDB capacity
- Use provisioned capacity mode
- Implement retry logic

## Related Documentation

- [Configuration](configuration.md) - Configuration options
- [IPFS Storage Guide](ipfs-storage.md) - IPFS/Kubo setup and configuration
- [DynamoDB Nameservice Guide](dynamodb-guide.md) - DynamoDB-specific setup
- [Getting Started: Server](../getting-started/quickstart-server.md) - Initial setup
- [Admin and Health](admin-and-health.md) - Administrative operations


---

# operations/serverless-storage.md

# Serverless Storage Choices

This guide covers storage placement for cloud and serverless deployments, especially
AWS Lambda-style architectures where commit writes, background indexing, and query
execution may run in separate warm containers.

## Short Recommendation

Use **Standard S3 for commits**. Commits are the durable source of truth and should
use the strongest redundancy profile available. Indexes are derived from commits
and can be rebuilt, so index storage can be optimized separately.

For indexes:

- Choose **Standard S3** when cost, multi-AZ durability, and operational simplicity
  matter more than the lowest cold-read/indexing latency.
- Choose **S3 Express One Zone** when you need lower latency for cold queries,
  sustained incremental indexing, or workloads that touch many small index blobs.
- Keep a local disk artifact cache enabled for Lambda/query/indexer workers. It
  is what makes hot query performance largely independent of the remote index
  bucket.

## Commit Storage

Commit blobs are immutable and are the canonical history of the ledger. Indexes,
statistics, and derived query structures can always be reconstructed from commits.

For that reason, prefer Standard S3 for `commitStorage` even when using S3 Express
One Zone for `indexStorage`:

```json
{
  "@context": {
    "@base": "https://ns.flur.ee/config/connection/",
    "@vocab": "https://ns.flur.ee/system#"
  },
  "@graph": [
    {
      "@id": "commitStorage",
      "@type": "Storage",
      "s3Bucket": "fluree-commits",
      "s3Prefix": "commits/"
    },
    {
      "@id": "indexStorage",
      "@type": "Storage",
      "s3Bucket": "fluree-index--use1-az1--x-s3",
      "s3Prefix": "indexes/"
    },
    {
      "@id": "connection",
      "@type": "Connection",
      "commitStorage": { "@id": "commitStorage" },
      "indexStorage": { "@id": "indexStorage" }
    }
  ]
}
```

When using S3 Express One Zone, omit `s3Endpoint` and let the AWS SDK resolve the
directory-bucket endpoint.

## What To Expect

Every database and query shape is different. The ranges below come from internal
serverless benchmarks using identical Lambda binaries, one stack backed by
Standard S3 for indexes and one backed by S3 Express One Zone for indexes. Dataset
names and customer-specific identifiers are intentionally omitted.

### Transactions

Transactions showed no meaningful difference between Standard S3 and S3 Express
for index storage.

In the benchmark, transaction wall time was dominated by the synchronous commit
path through the transactor and queueing layer, not by index storage. Across a
medium staged JSON-LD workload, both backends landed in the same rough multi-second
range per commit, with differences lost in normal Lambda and queue variance.

### Queries

Hot queries should show little to no difference. Once the relevant root, branch,
leaf, dictionary, and sidecar artifacts are in local memory or the disk artifact
cache, the remote bucket is no longer on the critical path.

For cold or partially warm queries:

- Simple lookups that touch only a small number of index blobs are usually close.
  Expect roughly no measurable difference to about a 30% slowdown on Standard S3.
- Queries that touch many index blobs can show a larger cold-cache gap, because
  Standard S3 has higher per-object latency and the penalty compounds with the
  number of small reads.
- If background indexing falls behind, query latency can degrade for reasons
  unrelated to S3 Express vs Standard S3. The query engine may need to account
  for unindexed commits, and that work grows with the indexing gap.

In one small synthetic workload, Standard S3 query medians were about 10-35%
slower before cache effects dominated. In a medium staged workload with the
indexer keeping up, Standard and Express query medians were effectively
indistinguishable, with individual queries varying by normal runtime noise.

### Indexing

Indexing is the area where index bucket choice is most visible. Incremental
indexing reads and writes many small content-addressed artifacts: roots, branch
manifests, leaves, sidecars, dictionaries, sketches, and other derived blobs.
S3 Express One Zone is optimized for this small-object, low-latency access pattern.

In a medium staged workload of roughly 8.5 MB JSON-LD, about 10,000 subjects, and
five commits, Standard S3 indexing was consistently slower but completed cleanly:

| Index Event | Express Indexing | Standard S3 Indexing | Standard / Express |
|-------------|------------------|----------------------|--------------------|
| Initial build | ~0.5 s | ~1.2 s | ~2.5x |
| Incremental updates | ~0.7-1.1 s | ~1.5-1.9 s | ~1.7-2.5x |

Treat these as ballpark ranges, not guarantees. Larger ledgers, wider class and
property statistics, more named graphs, and colder caches can increase the gap.
On the other hand, hot repeated query traffic may see almost no difference.

## Choosing An Index Backend

Use Standard S3 for indexes when:

- You want multi-AZ S3 durability for both commits and indexes.
- You are cost-sensitive or want to avoid the S3 Express per-bucket cost floor.
- Your workload is mostly hot queries, modest indexing volume, or can tolerate
  indexing taking roughly twice as long.

Use S3 Express One Zone for indexes when:

- Cold query latency matters.
- Incremental indexing throughput is important.
- Queries touch many index segments before the local cache is warm.
- You can tolerate the single-AZ durability profile because indexes are
  reproducible from commits.

## Tuning Notes

- Size Lambda `/tmp` large enough for the disk artifact cache. The cache softens
  both query and indexing latency by avoiding repeated remote reads of immutable
  artifacts.
- Use `s3MaxConcurrentRequests` to cap per-process S3 SDK concurrency if a
  deployment shows signs of retry storms or HTTP-layer stalls.
- Use the indexer worker's own processing time logs for indexing measurements.
  Client-side polling includes nameservice propagation and scheduling delays.
- Watch `index_t` vs `commit_t`. If `index_t` lags behind `commit_t`, query
  latency may reflect catch-up work rather than the raw performance of the
  selected index bucket.


---

# operations/hardware-benchmarks.md

# Hardware sizing: CPU vs disk (a worked benchmark)

This guide is a concrete, reproducible example of how cloud hardware choices affect
Fluree, using a large public dataset. The headline result is that **import speed and
query speed respond to different levers**: bulk import is dominated by **storage
throughput**, while steady-state query latency is dominated by **single-thread CPU
performance**. Sizing a deployment well means knowing which one you are optimizing for.

The numbers below come from the open Fluree benchmark suite:

- **Repo:** <https://github.com/fluree/benchmark-db>
- **Query set:** [SPARQLoscope](https://purl.org/ad-freiburg/sparqloscope) — a
  105-query suite spanning joins, OPTIONAL/MINUS/EXISTS, UNION, aggregates, numeric
  and date functions, string/REGEX, transitive paths, and result-size/export.
- **Dataset:** DBLP-core (the standard DBLP bibliography), DROPS monthly archive
  `2026-06-01` — **574.2 M** N-Triples lines (**561.5 M** distinct after dedup),
  90 predicates, ~73.5 GB uncompressed. Fluree builds a **27 GB** inline-indexed
  ledger from it.

Method: `fluree create dblp --from dblp.ttl` for import; for queries, 1 warmup + 3
timed runs per query, median per query, then aggregated across the 105 queries.
Each machine has 16 cores and 64 GB RAM, differing in CPU family and disk.

## Import: storage-bound

| Instance | CPU | Disk | Import | Throughput | Peak RSS | Index |
|---|---|---|--:|--:|--:|--:|
| `m7a.4xlarge` | AMD EPYC Zen 4 (~3.7 GHz) | gp3 EBS (500 MB/s) | 504 s | 1.14 M tr/s | 21.9 GB | 27 GB |
| `m7gd.4xlarge` | Graviton3 (~2.6 GHz) | local NVMe | 430 s | 1.33 M tr/s | 24.6 GB | 27 GB |
| `m8gd.4xlarge` | Graviton4 (~2.8 GHz) | local NVMe | **382 s** | **1.50 M tr/s** | 24.4 GB | 27 GB |

The two NVMe machines import **15–24% faster** than the EBS machine **even though
their CPUs are slower per core**. That is the tell-tale sign that import is
**I/O-bound, not CPU-bound**: a slower processor still finishes sooner once it is
fed by a faster disk. During the dominant parse + sorted-commit phase the CPU sits
around 25–30% busy on all three machines — the bottleneck is how fast triples can
be streamed off disk and indexed onto disk. If your workload is ingest-heavy
(large initial loads, frequent bulk imports), **prioritize local NVMe / high disk
throughput** over raw core speed.

## Query: CPU-bound

After warmup the working set (the 27 GB index) is resident in the 64 GB page cache,
so the disk barely participates and query latency tracks the **CPU**:

| Instance | CPU | Average (mean) | Geo mean | Median |
|---|---|--:|--:|--:|
| `m7a.4xlarge` | AMD EPYC Zen 4 (~3.7 GHz) | **936 ms** | **119 ms** | **67 ms** |
| `m8gd.4xlarge` | Graviton4 (~2.8 GHz) | 1,003 ms | 145 ms | 94 ms |
| `m7gd.4xlarge` | Graviton3 (~2.6 GHz) | 1,153 ms | 167 ms | 110 ms |

Here the ordering is the reverse of import: the highest-clock core (Zen 4) is
fastest, and the ranking lines up with single-thread performance, not disk. The
gap is widest on the cheapest queries — where fixed per-query overhead (planning,
parsing, serialization) dominates and there is little parallel work to hide a
slower core — and narrows on the heavy joins and aggregates. Newer cores help:
Graviton4 is ~13% faster than Graviton3 on the same suite. If your workload is
query-latency-sensitive (interactive APIs, many small reads), **prioritize a
high-clock / high-IPC CPU** and ensure RAM comfortably holds the working index.

## Putting it together

| If you optimize for… | Choose… | Why |
|---|---|---|
| Bulk import / ingest throughput | local NVMe, high disk bandwidth | import is I/O-bound |
| Query latency | fast single-thread CPU + enough RAM to cache the index | queries are CPU-bound and served from cache |
| Mixed | a modern core **with** NVMe (e.g. Graviton4 + NVMe) | best balance of both |

A practical rule of thumb: **size RAM to hold the active index** (so queries stay
in cache), pick the **fastest core** you can for query latency, and reach for
**local NVMe** when import time matters. The same-hardware effect is real — moving
from network block storage to local NVMe shortens import with no change to the CPU
— so storage class is a first-class sizing decision, not an afterthought.

To reproduce these numbers or run the suite against your own hardware and datasets,
see <https://github.com/fluree/benchmark-db>.


---

# operations/ipfs-storage.md

# IPFS Storage

Fluree can use [IPFS](https://ipfs.tech/) as a content-addressed storage backend via the [Kubo](https://github.com/ipfs/kubo) HTTP RPC API. This enables decentralized, content-addressed data storage where every piece of data is identified by its cryptographic hash.

> **Feature flag:** Requires the `ipfs` feature to be enabled at compile time.
> Build with: `cargo build --features ipfs`

## Overview

IPFS storage maps naturally to Fluree's content-addressed architecture. Fluree already identifies every blob (commits, transactions, index nodes) with a CIDv1 content identifier using SHA-256 hashing and Fluree-specific multicodec values. When IPFS is used as the storage backend, these CIDs are stored directly into IPFS via a local Kubo node.

**Key properties:**

- Content-addressed: data is identified by its SHA-256 hash, providing built-in integrity verification
- Immutable: once written, data cannot be modified or deleted (only unpinned for garbage collection)
- Decentralized: data can be replicated across IPFS nodes without centralized coordination
- Compatible: Fluree's native CIDs work directly with IPFS (no translation layer needed)

## Kubo Setup

[Kubo](https://github.com/ipfs/kubo) (formerly go-ipfs) is the reference IPFS implementation. Fluree communicates with Kubo via its HTTP RPC API (default port 5001).

### Install Kubo

**macOS (Homebrew):**

```bash
brew install ipfs
```

**Linux (official binary):**

```bash
wget https://dist.ipfs.tech/kubo/v0.32.1/kubo_v0.32.1_linux-amd64.tar.gz
tar xvfz kubo_v0.32.1_linux-amd64.tar.gz
cd kubo
sudo ./install.sh
```

**Docker:**

```bash
docker run -d \
  --name ipfs \
  -p 4001:4001 \
  -p 5001:5001 \
  -p 8080:8080 \
  -v ipfs_data:/data/ipfs \
  ipfs/kubo:latest
```

### Initialize and Start

```bash
# Initialize IPFS (first time only)
ipfs init

# Start the daemon
ipfs daemon
```

Verify the node is running:

```bash
# Check node identity
curl -s -X POST http://127.0.0.1:5001/api/v0/id | jq .ID
```

### Security Note

The Kubo HTTP RPC API (port 5001) provides full administrative access to the IPFS node. By default, it listens only on `127.0.0.1`. **Do not expose port 5001 to the public internet.** If Fluree and Kubo run on different hosts, use SSH tunneling, a VPN, or a reverse proxy with authentication.

The IPFS gateway (port 8080) is read-only and can be exposed publicly if desired.

## Configuration

### JSON-LD Configuration

```json
{
  "@context": {
    "@base": "https://ns.flur.ee/config/connection/",
    "@vocab": "https://ns.flur.ee/system#"
  },
  "@graph": [
    {
      "@id": "ipfsStorage",
      "@type": "Storage",
      "ipfsApiUrl": "http://127.0.0.1:5001",
      "ipfsPinOnPut": true
    },
    {
      "@id": "connection",
      "@type": "Connection",
      "indexStorage": { "@id": "ipfsStorage" }
    }
  ]
}
```

### Flat JSON Configuration

```json
{
  "indexStorage": {
    "@type": "IpfsStorage",
    "ipfsApiUrl": "http://127.0.0.1:5001",
    "ipfsPinOnPut": true
  }
}
```

### Configuration Fields

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `ipfsApiUrl` | string | `http://127.0.0.1:5001` | Kubo HTTP RPC API base URL |
| `ipfsPinOnPut` | boolean | `true` | Pin blocks after writing (prevents garbage collection) |

Both fields support `ConfigurationValue` indirection (env vars):

```json
{
  "ipfsApiUrl": { "envVar": "FLUREE_IPFS_API_URL", "defaultVal": "http://127.0.0.1:5001" },
  "ipfsPinOnPut": true
}
```

## Architecture

```text
┌──────────────────────┐
│   Fluree Process     │
│  ┌────────────────┐  │
│  │  IpfsStorage   │  │
│  │  (HTTP client) │  │
│  └────────┬───────┘  │
└───────────┼──────────┘
            │ HTTP RPC
     ┌──────▼──────┐
     │  Kubo Node  │
     │  (port 5001)│
     └──────┬──────┘
            │ libp2p
     ┌──────▼──────┐
     │  IPFS P2P   │
     │  Network    │
     └─────────────┘
```

Fluree communicates with a local Kubo node via the HTTP RPC API. The Kubo node handles peer-to-peer networking, block storage, and replication with the broader IPFS network.

### API Endpoints Used

| Kubo Endpoint | Purpose |
|---------------|---------|
| `POST /api/v0/block/put` | Store a block with optional codec and hash type |
| `POST /api/v0/block/get` | Retrieve a block by CID |
| `POST /api/v0/block/stat` | Check if a block exists (metadata only) |
| `POST /api/v0/pin/add` | Pin a block to prevent garbage collection |
| `POST /api/v0/id` | Health check (verify node is reachable) |

## Content Addressing

### How Fluree CIDs Map to IPFS

Fluree uses CIDv1 with SHA-256 multihash and private-use multicodec values:

| Content Kind | Multicodec | Hex | Example |
|--------------|------------|-----|---------|
| Commit | `fluree-commit` | `0x300001` | `bafybeig...` |
| Transaction | `fluree-txn` | `0x300002` | `bafybeig...` |
| Index Root | `fluree-index-root` | `0x300003` | `bafybeig...` |
| Index Branch | `fluree-index-branch` | `0x300004` | `bafybeig...` |
| Index Leaf | `fluree-index-leaf` | `0x300005` | `bafybeig...` |
| Dict Blob | `fluree-dict-blob` | `0x300006` | `bafybeig...` |
| Garbage Record | `fluree-garbage` | `0x300007` | `bafybeig...` |
| Ledger Config | `fluree-ledger-config` | `0x300008` | `bafybeig...` |
| Stats Sketch | `fluree-stats-sketch` | `0x300009` | `bafybeig...` |
| Graph Source Snapshot | `fluree-graph-source-snapshot` | `0x30000A` | `bafybeig...` |
| Spatial Index | `fluree-spatial-index` | `0x30000B` | `bafybeig...` |

These are in the multicodec private-use range (`0x300000`+). Kubo accepts them via the `cid-codec` parameter and resolves blocks by multihash regardless of codec. This means Fluree's native CIDs work directly with IPFS without any translation layer.

### Cross-Codec Retrieval

IPFS block storage is keyed by multihash internally. A block stored with codec `0x300001` (Fluree commit) can be retrieved using a CID with codec `0x55` (raw) as long as the SHA-256 digest is the same. This simplifies the address-based `StorageRead` implementation: given a Fluree address containing a hash, we can construct any CID with that hash to fetch the block.

## Pinning

### What is Pinning?

IPFS nodes periodically garbage-collect unpinned blocks to free disk space. Pinning tells the node to keep specific blocks permanently. Without pinning, blocks may be removed from the local node (though they remain available on other nodes that have them).

### Default Behavior

Fluree pins every block on write when `ipfsPinOnPut` is `true` (the default). This ensures that:

- All committed data survives Kubo garbage collection
- The local node serves as a reliable storage backend
- Blocks remain available even if no other node has them

### When to Disable Pinning

Set `ipfsPinOnPut: false` when:

- Running integration tests (faster, less disk usage)
- Using a separate pinning service (Pinata, web3.storage, etc.)
- The Kubo node is configured with `--enable-gc=false`

### Pinning Services

For production deployments, consider using a remote pinning service for redundancy:

```bash
# Add a remote pinning service
ipfs pin remote service add pinata https://api.pinata.cloud/psa YOUR_JWT

# Pin a CID to the remote service
ipfs pin remote add --service=pinata bafybeig...
```

## Limitations

### No Prefix Listing

IPFS is a content-addressed store with no concept of directory listing or prefix enumeration. The `list_prefix()` operation returns an error. Operations that require listing (e.g., ledger discovery, GC scans) must use an alternative strategy such as manifest-based tracking.

### No Deletion

IPFS content is immutable. The `delete()` operation is a no-op. Data removal is handled through:

1. **Unpinning** the block on the local node
2. Waiting for Kubo's **garbage collector** to reclaim space
3. The block may still exist on other IPFS nodes

### Nameservice

IPFS storage currently requires a separate nameservice (file-based or DynamoDB) for ledger metadata. A future phase will add IPNS and/or ENS-based decentralized nameservices.

### Latency

Writes go through the Kubo HTTP RPC API, adding HTTP overhead compared to direct file I/O. For latency-sensitive workloads, ensure Kubo runs on the same host as Fluree (localhost communication).

### No Encryption

The IPFS storage backend does not currently support Fluree's `AES256Key` encryption. Blocks are stored unencrypted in IPFS. If encryption is needed, use a separate encryption layer or a private IPFS network.

## Storage Addresses

Fluree addresses for IPFS storage follow the standard format:

```
fluree:ipfs://{ledger_id}/{kind_dir}/{hash_hex}.{ext}
```

Examples:

```
fluree:ipfs://mydb/main/commit/a1b2c3...f6a1b2.fcv2
fluree:ipfs://mydb/main/index/roots/d4e5f6...c3d4e5.json
fluree:ipfs://mydb/main/index/spot/abc123...def456.fli
```

The hash hex in the filename is extracted and used to construct a CID for retrieval from IPFS.

## Operational Considerations

### Disk Usage

Kubo stores blocks in a local datastore (by default, a LevelDB-based flatfs at `~/.ipfs/blocks/`). Monitor disk usage:

```bash
# Check IPFS repo size
ipfs repo stat

# Run garbage collection (removes unpinned blocks)
ipfs repo gc
```

### Network Bandwidth

By default, Kubo participates in the IPFS DHT and may serve blocks to other nodes. For a private deployment:

```bash
# Disable DHT (private node)
ipfs config Routing.Type none

# Or use a private IPFS network with a swarm key
# See: https://github.com/ipfs/kubo/blob/master/docs/experimental-features.md#private-networks
```

### Performance Tuning

```bash
# Increase concurrent connections
ipfs config Swarm.ConnMgr.HighWater 300

# Adjust datastore cache
ipfs config Datastore.BloomFilterSize 1048576

# Disable automatic GC (if using external pinning)
ipfs config --json Datastore.GCPeriod '"0"'
```

### Monitoring

Check Kubo node health:

```bash
# Node identity and version
ipfs id

# Connected peers
ipfs swarm peers | wc -l

# Repo statistics
ipfs repo stat

# Bandwidth usage
ipfs stats bw
```

## Troubleshooting

### Connection Refused

```
IPFS node connection failed: http://127.0.0.1:5001
```

**Causes:**
- Kubo daemon is not running
- Kubo is listening on a different address/port
- Firewall blocking the connection

**Fix:**
```bash
# Start the daemon
ipfs daemon

# Or check what address it's listening on
ipfs config Addresses.API
```

### Block Not Found

```
IPFS block not found: bafybeig...
```

**Causes:**
- Block was never stored on this node
- Block was unpinned and garbage collected
- CID format mismatch

**Fix:**
```bash
# Check if block exists locally
ipfs block stat bafybeig...

# Try fetching from the network
ipfs block get bafybeig... > /dev/null
```

### Slow Writes

**Causes:**
- Kubo node under heavy load
- Network latency (if Kubo is remote)
- Disk I/O bottleneck

**Fix:**
- Run Kubo on the same host as Fluree
- Use SSD storage for the IPFS datastore
- Consider disabling DHT for private deployments

## Future Roadmap

### Phase 2: Decentralized Nameservice

The IPFS storage backend is designed as the foundation for decentralized Fluree deployments. Planned additions:

- **IPNS**: Publish mutable pointers to ledger state (commit head, index root)
- **ENS / L2 chain**: On-chain CID pointers for trustless ledger discovery
- **Two-tier nameservice**: Local nameservice for fast reads with async push to decentralized upstream (similar to `git push`)

### Content Pinning Strategy

Future versions may support:

- Automatic pinning profiles (pin commits only, pin everything, pin nothing)
- Integration with remote pinning services (Pinata, web3.storage)
- Manifest-based tracking for GC and prefix listing

## Related Documentation

- [Storage modes](storage.md) - Overview of all storage backends
- [Configuration](configuration.md) - Server configuration options
- [JSON-LD connection config](../reference/connection-config-jsonld.md) - Full config reference
- [ContentId and ContentStore](../design/content-id-and-contentstore.md) - Content addressing design
- [Storage traits](../design/storage-traits.md) - Storage backend architecture


---

# operations/dynamodb-guide.md

# DynamoDB Nameservice Guide

## Overview

Fluree supports Amazon DynamoDB as a nameservice backend for storing ledger and graph source metadata. The DynamoDB nameservice provides:

- **Item-per-concern independence**: Each concern (commit head, index, status, config) is a separate DynamoDB item, eliminating physical write contention between transactors and indexers
- **Atomic conditional updates**: Reduced logical contention via conditional expressions
- **Strong consistency reads**: Always see the latest data
- **High availability**: DynamoDB's built-in redundancy and durability
- **Unified ledger + graph source support**: Both ledgers and graph sources (BM25, Vector, Iceberg, etc.) share the same table with a composite key

### Why DynamoDB for Nameservice?

The nameservice stores metadata about ledgers and graph sources: commit IDs, index state, status, and configuration. In high-throughput scenarios, transactors and indexers may update this metadata concurrently.

DynamoDB solves this because:

1. **Item-per-concern layout**: Each concern (head, index, status, config) is a separate DynamoDB item under the same partition key, so writes to different concerns never contend at the physical level
2. **Conditional updates**: Each update only proceeds if the new watermark advances monotonically
3. **No read-modify-write cycles (for the write itself)**: Updates are atomic; callers should still expect occasional conditional-update conflicts under contention and retry where appropriate

### Graph Sources (non-ledger)

Graph sources (BM25, Vector, Iceberg, etc.) are stored in the same nameservice table as ledgers. Under the **graph-source-owned manifest** design, the nameservice does **not** store snapshot history for graph sources.

- For ledgers, `index_id` points to a ledger index root.
- For graph sources, `index_id` points to a **graph-source-owned root/manifest** in storage (opaque to nameservice).
- Snapshot history (if any) is stored in storage and managed by the graph source implementation.

This keeps DynamoDB schema stable: **no unbounded "snapshot history" list is stored in the DynamoDB item**.

## Table Setup

### Schema Overview

The table uses a **composite primary key** (`pk` + `sk`) with a Global Secondary Index (GSI) for listing by kind.

- **`pk`** (Partition Key, String): Alias in `name:branch` form (e.g., `mydb:main`)
- **`sk`** (Sort Key, String): Concern discriminator (`meta`, `head`, `index`, `config`, `status`)
- **GSI1** (`gsi1-kind`): Enables efficient listing of all ledgers or all graph sources

### AWS CLI

```bash
aws dynamodb create-table \
  --table-name fluree-nameservice \
  --attribute-definitions \
    AttributeName=pk,AttributeType=S \
    AttributeName=sk,AttributeType=S \
    AttributeName=kind,AttributeType=S \
  --key-schema \
    AttributeName=pk,KeyType=HASH \
    AttributeName=sk,KeyType=RANGE \
  --global-secondary-indexes '[
    {
      "IndexName": "gsi1-kind",
      "KeySchema": [
        {"AttributeName": "kind", "KeyType": "HASH"},
        {"AttributeName": "pk", "KeyType": "RANGE"}
      ],
      "Projection": {
        "ProjectionType": "INCLUDE",
        "NonKeyAttributes": ["name", "branch", "source_type", "dependencies", "retracted"]
      }
    }
  ]' \
  --billing-mode PAY_PER_REQUEST
```

### CloudFormation

```yaml
Resources:
  FlureeNameserviceTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: fluree-nameservice
      BillingMode: PAY_PER_REQUEST
      AttributeDefinitions:
        - AttributeName: pk
          AttributeType: S
        - AttributeName: sk
          AttributeType: S
        - AttributeName: kind
          AttributeType: S
      KeySchema:
        - AttributeName: pk
          KeyType: HASH
        - AttributeName: sk
          KeyType: RANGE
      GlobalSecondaryIndexes:
        - IndexName: gsi1-kind
          KeySchema:
            - AttributeName: kind
              KeyType: HASH
            - AttributeName: pk
              KeyType: RANGE
          Projection:
            ProjectionType: INCLUDE
            NonKeyAttributes:
              - name
              - branch
              - source_type
              - dependencies
              - retracted
      PointInTimeRecoverySpecification:
        PointInTimeRecoveryEnabled: true
      Tags:
        - Key: Application
          Value: Fluree
```

### Terraform

```hcl
resource "aws_dynamodb_table" "fluree_nameservice" {
  name         = "fluree-nameservice"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "pk"
  range_key    = "sk"

  attribute {
    name = "pk"
    type = "S"
  }

  attribute {
    name = "sk"
    type = "S"
  }

  attribute {
    name = "kind"
    type = "S"
  }

  global_secondary_index {
    name            = "gsi1-kind"
    hash_key        = "kind"
    range_key       = "pk"
    projection_type = "INCLUDE"
    non_key_attributes = [
      "name",
      "branch",
      "source_type",
      "dependencies",
      "retracted",
    ]
  }

  point_in_time_recovery {
    enabled = true
  }

  tags = {
    Application = "Fluree"
  }
}
```

### Programmatic Table Creation

Fluree's `DynamoDbNameService` also provides an `ensure_table()` method that creates the table with the correct schema if it doesn't already exist:

```rust
use fluree_db_storage_aws::dynamodb::DynamoDbNameService;

let ns = DynamoDbNameService::from_client(dynamodb_client, "fluree-nameservice".to_string());
ns.ensure_table().await?;
```

This is used by integration tests and can be used for bootstrapping development environments.

## Table Schema

### Primary Key

| Attribute | Type | Description |
|-----------|------|-------------|
| `pk` | String (Partition Key) | Alias in `name:branch` form (e.g., `mydb:main`) |
| `sk` | String (Sort Key) | Concern discriminator: `meta`, `head`, `index`, `config`, `status` |

### Items per Alias

Each ledger or graph source is represented as multiple items under the same `pk`:

**Ledger (5 items):**

| Sort Key (`sk`) | Description | Key Attributes |
|-----------------|-------------|----------------|
| `meta` | Identity and metadata | `kind`, `name`, `branch`, `retracted`, `schema` |
| `head` | Commit head pointer | `commit_id`, `commit_t` |
| `index` | Index head pointer | `index_id`, `index_t` |
| `config` | Ledger configuration | `default_context_id`, `config_v`, `config_meta` |
| `status` | Operational status | `status`, `status_v`, `status_meta` |

**Graph Source (4 items):**

| Sort Key (`sk`) | Description | Key Attributes |
|-----------------|-------------|----------------|
| `meta` | Identity and metadata | `kind`, `source_type`, `name`, `branch`, `dependencies`, `retracted`, `schema` |
| `config` | Source configuration | `config_json`, `config_v` |
| `index` | Index head pointer | `index_id`, `index_t` |
| `status` | Operational status | `status`, `status_v`, `status_meta` |

### Attribute Reference

All items share these common attributes:

| Attribute | Type | Description |
|-----------|------|-------------|
| `pk` | String | Record address (`name:branch`) |
| `sk` | String | Concern discriminator |
| `schema` | Number | Schema version (always `2`) |
| `updated_at_ms` | Number | Last update timestamp (epoch milliseconds) |

**`meta` item:**

| Attribute | Type | Description |
|-----------|------|-------------|
| `kind` | String | `ledger` or `graph_source` |
| `name` | String | Base name (reserved word — use `#name` in expressions) |
| `branch` | String | Branch name |
| `retracted` | Boolean | Soft-delete flag |
| `source_type` | String (graph source only) | Graph-source type (e.g., `f:Bm25Index`) |
| `dependencies` | List\<String\> (graph source only) | Dependent ledger IDs |

**`head` item (ledgers only):**

| Attribute | Type | Description |
|-----------|------|-------------|
| `commit_id` | String \| null | Latest commit ContentId (CIDv1) |
| `commit_t` | Number | Commit watermark (`t`). `0` = unborn. |

**`index` item (ledgers + graph sources):**

| Attribute | Type | Description |
|-----------|------|-------------|
| `index_id` | String \| null | Latest index ContentId (CIDv1) |
| `index_t` | Number | Index watermark (`t`). `0` = unborn. |

**`config` item:**

| Attribute | Type | Description |
|-----------|------|-------------|
| `default_context_id` | String \| null | Default JSON-LD context ContentId (ledger) |
| `config_json` | String \| null | Opaque JSON config string (graph source) |
| `config_v` | Number | Config version watermark |
| `config_meta` | Map \| null | Extensible config metadata (ledger) |

**`status` item:**

| Attribute | Type | Description |
|-----------|------|-------------|
| `status` | String | Current state (reserved word — use `#st` in expressions) |
| `status_v` | Number | Status version watermark |
| `status_meta` | Map \| null | Extensible status metadata |

### GSI1: `gsi1-kind`

Enables listing all entities of a given kind (ledger or graph source).

| GSI Attribute | Source Attribute | Description |
|---------------|------------------|-------------|
| Partition Key | `kind` | `ledger` or `graph_source` |
| Sort Key | `pk` | Record address |
| Projected | `name`, `branch`, `source_type`, `dependencies`, `retracted` | Meta fields for listing without additional reads |

Only `meta` items carry the `kind` attribute and project into the GSI.

### Initialization Semantics

**All concern items are created atomically at initialization time.** This is a key structural decision:

- `init` creates all 5 items (`meta`, `head`, `index`, `config`, `status`) via `TransactWriteItems`
- `publish_graph_source` creates all 4 items (`meta`, `config`, `index`, `status`) via `TransactWriteItems`

Subsequent writes usually use `UpdateItem` operations (`compare_and_set_ref`, `publish_index`, `push_status`, `push_config`). The one exception is commit-head CAS on an unknown ledger ID with `expected=None`, where the backend bootstraps the ledger atomically via `TransactWriteItems`.

### How Updates Work

**Commit updates** (transactor):
```
UpdateItem Key: { pk: "mydb:main", sk: "head" }
UpdateExpression: SET commit_id = :cid, commit_t = :t, updated_at_ms = :now
ConditionExpression: attribute_exists(pk) AND commit_t < :t
```

**Index updates** (indexer):
```
UpdateItem Key: { pk: "mydb:main", sk: "index" }
UpdateExpression: SET index_id = :cid, index_t = :t, updated_at_ms = :now
ConditionExpression: attribute_exists(pk) AND index_t < :t
```

Since commit and index updates target different items (different `sk`), they never contend at the DynamoDB physical level.

**Status updates** (CAS):
```
UpdateItem Key: { pk: "mydb:main", sk: "status" }
UpdateExpression: SET #st = :new_state, status_v = :new_v, updated_at_ms = :now
ConditionExpression: status_v = :expected_v AND #st = :expected_state
```

**Config updates** (CAS):
```
UpdateItem Key: { pk: "mydb:main", sk: "config" }
UpdateExpression: SET default_context_id = :ctx, config_v = :new_v, updated_at_ms = :now
ConditionExpression: config_v = :expected_v
```

**RefPublisher updates** (compare-and-set refs):

- `CommitHead` uses strict monotonic guard: `new.t > current.t`
- `IndexHead` allows same-watermark overwrite: `new.t >= current.t` (reindex at same `t`)

When a caller attempts `compare_and_set_ref(expected=None)` on an unknown ledger ID, the DynamoDB backend bootstraps the ledger by creating all 5 ledger concern items via `TransactWriteItems` and pre-setting the target ref to the requested value.

**Retract:**
```
UpdateItem Key: { pk: "mydb:main", sk: "meta" }
UpdateExpression: SET retracted = :true, updated_at_ms = :now
```

### DynamoDB Reserved Words

The attributes `name` and `status` are DynamoDB reserved words. All expressions (reads, updates, projections) must use `ExpressionAttributeNames`:

```
ExpressionAttributeNames: { "#name": "name", "#st": "status" }
```

## Trait Implementations

The DynamoDB nameservice implements all seven nameservice traits:

| Trait | Description |
|-------|-------------|
| `NameService` | Lookup, ledger ID resolution, list all records |
| `Publisher` | Initialize ledgers, publish indexes, retract |
| `AdminPublisher` | Admin index publishing (allows equal-t overwrites) |
| `RefPublisher` | Compare-and-set on commit/index refs |
| `StatusPublisher` | CAS-based status updates |
| `ConfigPublisher` | CAS-based config updates (ledgers only) |
| `GraphSourceLookup` | Read-only graph source discovery: lookup, list all records |
| `GraphSourcePublisher` | Graph source lifecycle (extends `GraphSourceLookup`): create, index, retract |

**Note:** `ConfigPublisher` is scoped to ledgers only. Graph source configuration is managed through `GraphSourcePublisher`, which stores config as an opaque JSON string (`config_json`). `GraphSourceLookup` is a supertrait of `NameService`, so all nameservice implementations automatically support graph source discovery. `GraphSourcePublisher` adds write operations and is required only by APIs that create or drop graph sources.

## Configuration

### JSON-LD Connection Configuration

```json
{
  "@context": {
    "@vocab": "https://ns.flur.ee/system#"
  },
  "@graph": [
    {
      "@id": "s3Storage",
      "@type": "Storage",
      "s3Bucket": "fluree-production-data",
      "s3Endpoint": "https://s3.us-east-1.amazonaws.com",
      "s3Prefix": "ledgers",
      "addressIdentifier": "prod-s3"
    },
    {
      "@id": "dynamodbNs",
      "@type": "Publisher",
      "dynamodbTable": "fluree-nameservice",
      "dynamodbRegion": "us-east-1"
    },
    {
      "@id": "connection",
      "@type": "Connection",
      "parallelism": 4,
      "cacheMaxMb": 1000,
      "commitStorage": {"@id": "s3Storage"},
      "indexStorage": {"@id": "s3Storage"},
      "primaryPublisher": {"@id": "dynamodbNs"}
    }
  ]
}
```

### Configuration Options

| Field | Required | Description | Default |
|-------|----------|-------------|---------|
| `dynamodbTable` | Yes | DynamoDB table name | - |
| `dynamodbRegion` | No | AWS region | `us-east-1` |
| `dynamodbEndpoint` | No | Custom endpoint URL (for LocalStack) | AWS default |
| `dynamodbTimeoutMs` | No | Request timeout in milliseconds | `5000` |

## AWS Credentials

### Authentication Methods

The DynamoDB nameservice uses the standard AWS SDK credential chain:

1. **Environment Variables**
   ```bash
   export AWS_ACCESS_KEY_ID=your_access_key
   export AWS_SECRET_ACCESS_KEY=your_secret_key
   export AWS_REGION=us-east-1
   ```

2. **AWS Credentials File** (`~/.aws/credentials`)
   ```ini
   [default]
   aws_access_key_id = your_access_key
   aws_secret_access_key = your_secret_key
   region = us-east-1
   ```

3. **IAM Roles** (when running on EC2/ECS/Lambda)
   - Automatically uses instance/task role credentials

4. **Session Tokens** (for temporary credentials)
   ```bash
   export AWS_SESSION_TOKEN=your_session_token
   ```

### Required IAM Permissions

Full permissions (recommended):

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:GetItem",
        "dynamodb:PutItem",
        "dynamodb:UpdateItem",
        "dynamodb:DeleteItem",
        "dynamodb:Query",
        "dynamodb:BatchGetItem",
        "dynamodb:BatchWriteItem"
      ],
      "Resource": [
        "arn:aws:dynamodb:*:*:table/fluree-nameservice",
        "arn:aws:dynamodb:*:*:table/fluree-nameservice/index/gsi1-kind"
      ]
    }
  ]
}
```

If you also use `ensure_table()` for automated table creation (development/testing):

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:GetItem",
        "dynamodb:PutItem",
        "dynamodb:UpdateItem",
        "dynamodb:DeleteItem",
        "dynamodb:Query",
        "dynamodb:BatchGetItem",
        "dynamodb:BatchWriteItem",
        "dynamodb:CreateTable",
        "dynamodb:DescribeTable"
      ],
      "Resource": [
        "arn:aws:dynamodb:*:*:table/fluree-nameservice",
        "arn:aws:dynamodb:*:*:table/fluree-nameservice/index/gsi1-kind"
      ]
    }
  ]
}
```

Minimal runtime permissions (if not using `all_records`, `all_graph_source_records`, graph sources, or hard-drop purge):

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:GetItem",
        "dynamodb:PutItem",
        "dynamodb:UpdateItem",
        "dynamodb:Query"
      ],
      "Resource": "arn:aws:dynamodb:*:*:table/fluree-nameservice"
    }
  ]
}
```

Hard drops and branch-drop cascade cleanup require `dynamodb:DeleteItem`. Backends that purge all nameservice items for a ledger partition also need `dynamodb:BatchWriteItem`.

## Local Development

### Using LocalStack

1. **Start LocalStack**
   ```bash
   docker run -d --name localstack \
     -p 4566:4566 \
     -e SERVICES=dynamodb \
     localstack/localstack
   ```

2. **Create Test Table**
   ```bash
   AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test \
   aws --endpoint-url=http://localhost:4566 dynamodb create-table \
     --table-name fluree-nameservice \
     --attribute-definitions \
       AttributeName=pk,AttributeType=S \
       AttributeName=sk,AttributeType=S \
       AttributeName=kind,AttributeType=S \
     --key-schema \
       AttributeName=pk,KeyType=HASH \
       AttributeName=sk,KeyType=RANGE \
     --global-secondary-indexes '[
       {
         "IndexName": "gsi1-kind",
         "KeySchema": [
           {"AttributeName": "kind", "KeyType": "HASH"},
           {"AttributeName": "pk", "KeyType": "RANGE"}
         ],
         "Projection": {
           "ProjectionType": "INCLUDE",
           "NonKeyAttributes": ["name", "branch", "source_type", "dependencies", "retracted"]
         }
       }
     ]' \
     --billing-mode PAY_PER_REQUEST
   ```

3. **Configure Fluree**
   ```json
   {
     "@id": "dynamodbNs",
     "@type": "Publisher",
     "dynamodbTable": "fluree-nameservice",
     "dynamodbEndpoint": "http://localhost:4566",
     "dynamodbRegion": "us-east-1"
   }
   ```

4. **Set Environment Variables**
   ```bash
   export AWS_ACCESS_KEY_ID=test
   export AWS_SECRET_ACCESS_KEY=test
   ```

### Using DynamoDB Local

1. **Start DynamoDB Local**
   ```bash
   docker run -d --name dynamodb-local \
     -p 8000:8000 \
     amazon/dynamodb-local
   ```

2. **Create Test Table** (same command as LocalStack, change `--endpoint-url` to `http://localhost:8000`)

## Production Considerations

### Performance

- DynamoDB provides single-digit millisecond latency
- The item-per-concern layout eliminates physical contention between transactors and indexers
- Use on-demand (PAY_PER_REQUEST) billing for variable workloads
- Consider provisioned capacity for predictable high-throughput scenarios
- Enable DynamoDB Accelerator (DAX) if sub-millisecond reads are needed

### Security

- Use IAM roles instead of access keys when possible
- Enable encryption at rest (default for new tables)
- Use VPC endpoints for private DynamoDB access
- Enable CloudTrail for audit logging

### Monitoring

Set up CloudWatch alarms for:

- `ConditionalCheckFailedRequests` - indicates contention (usually normal)
- `ThrottledRequests` - capacity issues
- `SystemErrors` - service issues
- `SuccessfulRequestLatency` - track latency

### Backup and Recovery

```bash
# Enable Point-in-Time Recovery
aws dynamodb update-continuous-backups \
  --table-name fluree-nameservice \
  --point-in-time-recovery-specification PointInTimeRecoveryEnabled=true

# Create on-demand backup
aws dynamodb create-backup \
  --table-name fluree-nameservice \
  --backup-name fluree-ns-backup-$(date +%Y%m%d)
```

### Cost Optimization

- On-demand pricing is cost-effective for variable workloads
- Table data is small (5 items per ledger, 4 per graph source), so costs are minimal
- Typical costs: $1-10/month for small deployments
- GSI storage adds minimal cost (only meta items project into it)

## Troubleshooting

### Authentication Failures

**Symptoms**: Access denied, credential errors

**Solutions**:
- Verify AWS credentials are configured
- Check IAM permissions for the table and GSI
- Test with AWS CLI:
  ```bash
  aws dynamodb describe-table --table-name fluree-nameservice
  ```

### Table Not Found

**Symptoms**: ResourceNotFoundException

**Solutions**:
- Verify table name is correct
- Check table is in the correct region
- Ensure table has finished creating (including GSI)

### Timeout Errors

**Symptoms**: Request timeout

**Solutions**:
- Increase `dynamodbTimeoutMs` configuration
- Check network connectivity to DynamoDB
- Verify endpoint URL is correct (especially for LocalStack)

### Conditional Check Failures

**Symptoms**: High rate of ConditionalCheckFailedException in logs

**Note**: This is usually normal and indicates the system is working correctly. The conditional check prevents overwriting newer data with older data. `publish_index` stale writes are silently ignored (the newer value is preserved). CAS operations (`compare_and_set_ref`, `push_status`, `push_config`) return the current value so the caller can retry or report a conflict.

### Unprocessed Keys (BatchGetItem)

**Symptoms**: Listing graph sources intermittently returns fewer results under load, or logs show throttling.

**Cause**: DynamoDB may return `UnprocessedKeys` in `BatchGetItem` responses under throttling.

**Behavior**: Fluree retries `UnprocessedKeys` with exponential backoff (bounded retries). If retries are exhausted, it returns an error rather than silently dropping items.

### Uninitialized Alias Errors

**Symptoms**: Publish operations fail with "not found" or storage errors

**Cause**: Attempting to `publish_index` or other non-bootstrap writes on a ledger ID that was never initialized with `init`.

**Solution**: Ensure ledger initialization happens before index/status/config writes. Normal Fluree transaction commit-head publication uses `RefPublisher` CAS and can bootstrap an unknown ledger ID when `expected=None`.

## Related Documentation

- [Storage Modes](storage.md) - Overview of all storage options
- [Configuration](configuration.md) - Full configuration reference
- [Nameservice Schema v2 Design](../design/nameservice-schema-v2.md) - Schema design details
- [Ledgers and the Nameservice](../concepts/ledgers-and-nameservice.md) - Conceptual overview


---

# operations/query-peers.md

# Query peers and replication

This document describes how to run `fluree-server` in **transaction** mode (event source + transactions) and **peer** mode (read replica). It also documents the **events stream** (`/v1/fluree/events`) and **storage proxy** endpoints (`/v1/fluree/storage/*`) used to keep peers up to date and/or to proxy storage reads.

This guide is written from an **operator / end-user** standpoint: what to deploy, how to configure it, and what to expect from each mode.

## Server roles

`fluree-server` supports two roles:

- **Transaction server** (`--server-role transaction`)
  - Write-enabled.
  - Produces the nameservice events stream at `GET /v1/fluree/events`.
  - Optionally exposes storage proxy endpoints at `/v1/fluree/storage/*`.
- **Query peer** (`--server-role peer`)
  - Read-only API surface for clients (queries, history, etc.).
  - Subscribes to `GET /v1/fluree/events` from a transaction server to learn about nameservice updates.
  - Reads ledger data from storage (shared-storage deployments), and refreshes on staleness based on the events stream.
  - Forwards write/admin operations to the configured transaction server.

## Events stream (SSE): `GET /v1/fluree/events`

The transaction server exposes a Server-Sent Events (SSE) stream that emits **nameservice changes** for ledgers and graph sources. Query peers use this stream to stay up to date.

### Query parameters

- **`all=true`**: subscribe to all ledgers and graph sources
- **`ledger=<ledger_id>`**: subscribe to a ledger ID (`name:branch`, repeatable)
- **`graph-source=<graph_source_id>`**: subscribe to a graph source ID (`name:branch`, repeatable)

### Authentication and authorization

The `/v1/fluree/events` endpoint can be configured to require Bearer tokens:

- **`--events-auth-mode none|optional|required`**
- **`--events-auth-audience <aud>`** (optional)
- **`--events-auth-trusted-issuer <did:key:...>`** (repeatable)

When authentication is enabled, the token can restrict what the client may subscribe to. Requests that ask for resources not covered by the token are **silently filtered** to the allowed scope.

The repo includes a token generator binary for operator workflows:

- **`fluree-events-token`**: generates Bearer tokens suitable for `GET /v1/fluree/events`

## Peer mode behavior

In peer mode:

- **Write forwarding**: write and admin endpoints are forwarded to the transaction server configured by `--tx-server-url`.
- **Read serving**: query endpoints are served locally, using ledger/index data obtained either from shared storage or via storage proxy reads (see below). History queries are executed via the standard `/query` endpoint with time range specifiers.

### Peer configuration (SSE subscription)

- **`--server-role peer`**
- **`--tx-server-url <base-url>`** (required)
- **`--peer-events-url <url>`** (optional; default is `{tx_server_url}/v1/fluree/events`)
- **`--peer-events-token <token-or-@file>`** (optional; Bearer token for `/v1/fluree/events`)
- Subscribe scope:
  - **`--peer-subscribe-all`** or
  - **`--peer-ledger <ledger_id>`** (repeatable) and/or **`--peer-graph-source <graph_source_id>`** (repeatable)

### Peer storage access modes

Peer servers support two storage access modes:

- **Shared storage** (`--storage-access-mode shared`, default)
  - The peer reads the same storage backend as the transaction server (shared filesystem, shared bucket credentials, etc.).
  - Requires `--storage-path`.
- **Proxy storage** (`--storage-access-mode proxy`)
  - The peer does **not** need direct storage credentials.
  - The peer proxies all storage reads through the transaction server’s `/v1/fluree/storage/*` endpoints.
  - Requires `--tx-server-url` and a **storage proxy token** via `--storage-proxy-token` or `--storage-proxy-token-file`.
  - `--storage-path` is ignored in this mode.

## Storage proxy endpoints (transaction server): `/v1/fluree/storage/*`

Storage proxy endpoints allow a peer to read storage **through** the transaction server, rather than holding storage credentials directly. This is intended for environments where storage is private and peers cannot access it.

Storage proxy supports two kinds of reads:

- **Raw bytes** reads (`Accept: application/octet-stream`) for any block type (commit blobs, branch nodes, leaf nodes).
- **Policy-filtered leaf flakes** reads (`Accept: application/x-fluree-flakes`) for ledger **leaf** nodes only.

### Enablement

Storage proxy endpoints are disabled by default. Enable them on the transaction server:

- **`--storage-proxy-enabled`**
- **`--storage-proxy-trusted-issuer <did:key:...>`** (repeatable; optional if you reuse `--events-auth-trusted-issuer`)
- **`--storage-proxy-default-identity <iri>`** (optional; used when token has no `fluree.identity`)
- **`--storage-proxy-default-policy-class <class-iri>`** (optional; applies policy in addition to identity-based policy)
- **`--storage-proxy-debug-headers`** (optional; debug only—can leak information)

### AuthZ claims (Bearer token)

Storage proxy endpoints require a Bearer token that grants storage proxy permissions:

- **`fluree.storage.all: true`**: access all ledgers (graph source artifacts are denied in v1)
- **`fluree.storage.ledgers: ["books:main", ...]`**: access specific ledgers
- **`fluree.identity: "ex:PeerServiceAccount"`** (optional): identity used for policy evaluation in policy-filtered read mode

Unauthorized requests return **404** (no existence leak).

### Endpoints

#### `GET /v1/fluree/storage/ns/{ledger-id}`

Fetch a nameservice record for a ledger ID. Requires storage proxy authorization for that ledger.

#### `POST /v1/fluree/storage/block`

Fetch a block/blob by **CID**. The request includes the **ledger ID** so the server can authorize the request and derive the physical storage address internally. Currently supports:

- `Accept: application/octet-stream` (raw bytes; always available)
- `Accept: application/x-fluree-flakes` (binary “FLKB” transport of policy-filtered **leaf** flakes only)
- `Accept: application/x-fluree-flakes+json` (debug-only JSON flake transport; leaf flakes only)

If the client requests a flakes format for a **non-leaf** block, the server returns **406 Not Acceptable**. Clients (and peers in proxy mode) should retry with `Accept: application/octet-stream` in that case.

Example request body:

```json
{
  "cid": "bafy...leafOrBranchCid",
  "ledger": "mydb:main"
}
```

##### Policy filtering semantics (leaf flakes)

When a flakes format is requested and the block is a ledger leaf:

- The transaction server loads policy restrictions using the **effective identity** and **effective policy class**:
  - **effective identity**: token `fluree.identity` if present, otherwise `--storage-proxy-default-identity` (if configured)
  - **effective policy class**: `--storage-proxy-default-policy-class` (if configured; token-driven policy class selection may be added later)
- If the resolved policy is **root/unrestricted**, the server returns all leaf flakes (still encoded as FLKB in `application/x-fluree-flakes` mode).
- If the resolved policy is **non-root**, the server filters leaf flakes before encoding them for transport.

> Note: the peer can still apply additional client-facing policy enforcement on top of this. Client-side policy can only further restrict results; it cannot “recover” facts filtered out upstream.

### Security notes and limitations

- **Branch/commit leakage (v1 limitation)**: filtering leaves without rewriting branches/commits can leak structure/existence information to the peer identity. This is currently an accepted v1 limitation.
- **Graph source artifacts (v1)**: storage proxy denies graph-source artifacts by returning 404 even when `fluree.storage.all` is present.

## Deployment examples

### Transaction server (events + storage proxy)

```bash
fluree-server \
  --listen-addr 0.0.0.0:8090 \
  --server-role transaction \
  --storage-path /var/lib/fluree \
  --events-auth-mode required \
  --events-auth-trusted-issuer did:key:z6Mk... \
  --storage-proxy-enabled
```

### Query peer (shared storage)

```bash
fluree-server \
  --listen-addr 0.0.0.0:8091 \
  --server-role peer \
  --tx-server-url http://tx.internal:8090 \
  --storage-path /var/lib/fluree \
  --peer-subscribe-all \
  --peer-events-token @/etc/fluree/peer-events.jwt
```

### Query peer (proxy storage mode)

In proxy storage mode, the peer does not need `--storage-path` and instead needs a storage proxy token:

```bash
fluree-server \
  --listen-addr 0.0.0.0:8091 \
  --server-role peer \
  --tx-server-url http://tx.internal:8090 \
  --storage-access-mode proxy \
  --storage-proxy-token @/etc/fluree/storage-proxy.jwt \
  --peer-subscribe-all \
  --peer-events-token @/etc/fluree/peer-events.jwt
```


---

# operations/raft-clusters.md

# Raft clusters (replicated transaction servers)

This guide is for operators who want to run `fluree-server` as a Raft-replicated cluster instead of a single-node transaction server. In this mode every node accepts client traffic, but writes are committed through a Raft log so the cluster survives the loss of any minority of nodes.

Coverage:

- [When to use Raft mode](#when-to-use-raft-mode)
- [Architecture](#architecture)
- [Per-node configuration](#per-node-configuration)
- [Bootstrapping a 3-node cluster](#bootstrapping-a-3-node-cluster)
- [Day-2 operations](#day-2-operations)
- [Admin HTTP surface](#admin-http-surface)
- [Client traffic and follower forwarding](#client-traffic-and-follower-forwarding)
- [Failure modes](#failure-modes)
- [Operational constraints and limits](#operational-constraints-and-limits)
- [Security notes](#security-notes)

For the deeper architecture (log entry types, snapshot model, queue/worker design), see the design doc: [Raft command queue and replicated state machine](../design/raft-command-queue.md).

## When to use Raft mode

Use it when:

- you need high availability for writes (single-node loss must not stop transactions),
- you can put all nodes behind one shared, content-addressed storage backend (typically S3 with a shared DynamoDB nameservice in the cloud profile, or a shared file mount in a homogeneous cluster),
- you can give each node a stable identity (node id) and a stable VPC-internal address for inter-node RPC.

Do **not** use Raft mode for:

- single-node deployments — the overhead of running a 1-node cluster is needless complexity. Use the default `transaction` server role instead.
- query-only replicas — `--server-role peer` already solves that, with lighter operational overhead.
- proxy-storage peers — Raft mode replicates writes through the log, which is the opposite of proxy mode (which forwards writes to a remote transaction server). Validation rejects the combination at startup.

## Architecture

A Fluree Raft cluster is a replicated state machine over an [openraft](https://github.com/databendlabs/openraft) log. Every node runs the same state machine; the leader is the only node that proposes new log entries. The state machine's job is to replicate **nameservice state** — branch heads, ledger lifecycle, and a deduplicating idempotency cache — not the commit blobs themselves. Commit blobs live in the shared content-addressed storage backend (CAS) and the log carries content identifiers, not payloads.

This split is the central architectural choice: the Raft log stays narrow (CIDs, branch ids, queue positions) while the heavy bytes (commit envelopes, index artifacts) ride the CAS that every node already needs to reach. A follower that has applied a log entry can serve reads of the new head as soon as the relevant blob arrives from CAS — without an openraft RPC.

### Two-listener topology

Each node listens on **two ports**:

| Port                                              | Default     | Faces        | Routes                                                | Auth                                                              |
| ------------------------------------------------- | ----------- | ------------ | ----------------------------------------------------- | ----------------------------------------------------------------- |
| Client (`--listen-addr`)                          | `0.0.0.0:8080` | Public / load balancer | `/fluree/*`, `/api/*`, `/admin/*`, etc.                | Same as single-node: `--events-auth-*`, `--data-auth-*`, `--admin-auth-*` |
| Raft (`--raft-listen-addr`)                       | (no default; required) | VPC-internal | `/raft/*` (inter-node RPC), `/cluster/*` (admin) | `/raft/*`: none (network trust). `/cluster/*`: `--admin-auth-mode` |

The client port is the public surface and behaves identically to a non-Raft `fluree-server` — reads work on every node, writes are accepted but transparently forwarded to the current leader by middleware. The Raft port is for inter-node openraft traffic plus operator-facing cluster admin; expose it only on a trusted network segment.

### Submission flow (writes)

Writes follow a four-stage path inside the cluster. The first two stages run wherever the request lands (follower or leader); the latter two run only on the leader:

1. **Forward.** The follower-forward middleware reads `raft.current_leader()`; if this node is not the leader, the request is re-issued as an HTTP call against the leader's `client_addr` (looked up from the membership the cluster replicates) and the leader's response is relayed back to the client verbatim.
2. **Enqueue.** On the leader, the `QueuedTransactor` builds a `QueuedRequest` envelope (the full request body), writes it to the shared CAS, and proposes a `Command::EnqueueCommand` carrying the envelope's CID + body hash. The state machine appends a `QueueEntry` to the target branch's FIFO and assigns a `queue_id`. The transactor registers a oneshot waiter on the per-process `WaiterMap`.
3. **Stage and commit.** The leader-only `CommitWorker` polls per-branch queues, fetches the envelope from CAS, stages the work via the in-process `Fluree` API, writes the resulting commit blob to CAS, stashes a typed `AppliedReceipt` in the per-process `StagedReceiptMap`, and proposes `Command::ApplyHead` to advance the head.
4. **Apply and resolve.** The state-machine adapter applies `ApplyHead` on every node (advancing the replicated head), and on the leader signals the waiter with the stashed receipt. The transactor's `await` returns the typed receipt to the client.

The reason for the queue + worker split is that **only the leader can propose log entries**, but the heavy work of staging a transaction (parsing, policy evaluation, indexing) must happen *before* the head moves. Decoupling proposal from staging keeps the openraft commit path free of blocking work, makes idempotent retries cheap (the second `EnqueueCommand` with the same idempotency key hits the cache and skips staging), and lets a leader change abandon in-flight stages without losing the queue.

### Storage layering

| Layer                          | Scope            | Backed by                                                                | Notes                                                                                                                                              |
| ------------------------------ | ---------------- | ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| Raft log + vote + snapshots    | Per node         | Local disk at `--raft-storage-path`                                       | Postcard-encoded, atomic writes (write-to-temp → fsync → rename → fsync parent). Losing this directory on one node is recoverable as long as a quorum survives. |
| Replicated state machine       | Per node, in-memory | `NameServiceState` (branch heads, ledger registry, idempotency cache, per-branch queues) | Persisted via openraft snapshots into `--raft-storage-path/snapshots/`. Restored on restart so log replay starts at `last_applied + 1`. |
| Ledger CAS (commit blobs, envelopes, index artifacts) | Cluster-wide | Whatever `--connection-config` / `--storage-path` resolves to (S3, file, memory) | Must be reachable from every node. The leader writes, every node reads. |

`--raft-storage-path` and `--storage-path` must point at **disjoint** filesystem subtrees. Validation rejects overlapping paths at startup because the Raft log/snapshot tree and the ledger CAS each manage their own layout; overlapping them lets either side blow away the other's files on compaction and tends to surface only after a restart corrupts state.

### Replicated nameservice

`RaftNameService` implements the same `NameServiceLookup` / `CommitPublisher` / `IndexPublisher` traits as the standalone nameservice, but its writes propose log entries and its reads observe the state machine the adapter writes to under apply. Followers see committed state immediately (no openraft RPC on the read path); only `current_leader()` is queried from openraft for the forwarding decision.

Three publish operations cross the log:

- `CommitPublisher::publish_commit` → `Command::ApplyHead { queue_id, head, ... }` — strict head advance, keyed by the queue position the worker is draining.
- `IndexPublisher::publish_index` → `Command::AdvanceIndexHead` — strict monotonic index pointer advance.
- Admin rebinds (`bind-ref` / `set-head`) → `Command::RewriteIndexHead` / `Command::CompareAndSetRef` — allow equal-or-different, gated by an expected-prior CID for CAS semantics.

### Idempotency cache

The state machine carries a hash map from `IdempotencyCacheKey` (ledger id + caller-supplied key) to `ApplyOutcome` (either `Applied { head, t, body_cid, ... }` or `Failed { reason, ... }`). On a re-proposal with the same key the state machine short-circuits to `IdempotencyHit` / `IdempotencyFailed` without re-staging. Entries age out via a leader-only `EvictIdempotency` scheduler (default TTL **1 hour**, eviction tick **60 s**). After the TTL elapses, a retry with the same key looks like a fresh submission and may execute again — clients that need stricter at-most-once semantics must use shorter retry windows.

### Snapshots

openraft drives snapshotting on a configured policy (log-entries-since-last-snapshot). The state-machine adapter serializes the full `NameServiceState` via postcard, plus `last_applied` and `last_membership`. Snapshots are stored as `<id>.meta` + `<id>.data` under `--raft-storage-path/raft/snapshots/`, with a `current` file naming the latest. On restart the adapter opens the latest snapshot first so log replay can skip everything before `last_applied`.

Snapshot ids arriving from peers are validated before any disk write: non-empty, ≤ 128 bytes, alphanumeric + `-_.` only, no `..`. Anything else is rejected as `StorageError::Corruption`.

## Per-node configuration

Raft mode is built behind a `raft` Cargo feature. A build without the feature has no `--raft-*` flags at all, so single-node binaries don't pull in openraft.

When the feature is enabled, four CLI/env switches govern Raft mode:

| CLI flag                | Env var                        | Required when raft is on |
| ----------------------- | ------------------------------ | ------------------------ |
| `--raft-enabled`        | `FLUREE_RAFT_ENABLED`          | toggle                   |
| `--raft-node-id`        | `FLUREE_RAFT_NODE_ID`          | yes (stable `u64`)       |
| `--raft-storage-path`   | `FLUREE_RAFT_STORAGE_PATH`     | yes                      |
| `--raft-listen-addr`    | `FLUREE_RAFT_LISTEN_ADDR`      | yes                      |

`--raft-storage-path` must point at a directory on **durable** local storage (not tmpfs). Losing this directory loses committed transactions for *this* node — though as long as a quorum still has its log, openraft can re-deliver the lost state when the node is re-added with the same id.

`--raft-storage-path` must also be disjoint from `--storage-path` (see [Storage layering](#storage-layering)).

The same options are accepted in `.fluree/config.toml`:

```toml
[server.raft]
enabled      = true
node_id      = 1
storage_path = "/var/lib/fluree/raft"
listen_addr  = "0.0.0.0:9090"
```

CLI / env override file values; file values override defaults.

### Storage backend choice

Raft mode is orthogonal to the content-addressed storage choice. In production we recommend shared cloud storage so the **commit blobs the leader writes are immediately visible to every follower's reads** through the same backend. Concretely:

- **AWS** — point all nodes at the same S3 bucket(s) via `--connection-config /etc/fluree/s3.jsonld`. Use a shared DynamoDB nameservice for non-Raft profiles; in Raft mode the **DynamoDB nameservice's writes are unused** (the Raft state machine is the authoritative nameservice), but the storage half stays shared.
- **On-prem** — point all nodes at the same NFS / object-store mount via `--storage-path /shared/fluree`.

The `RaftNameService` on each node sees committed log state; the CAS backend just has to be reachable from every node.

### A complete per-node startup

```bash
fluree-server \
  --listen-addr 0.0.0.0:8080 \
  --connection-config /etc/fluree/s3.jsonld \
  --raft-enabled \
  --raft-node-id 1 \
  --raft-storage-path /var/lib/fluree/raft \
  --raft-listen-addr 0.0.0.0:9090
```

After this the node is **idle** — it has joined no cluster yet. Subsequent admin calls form the cluster.

## Bootstrapping a 3-node cluster

The general shape is:

1. Start `fluree-server` on every node with its per-node config.
2. On **one** node — call it node 1 — initialize the cluster as a single-voter cluster (node 1 auto-elects itself).
3. Add nodes 2 and 3 as **learners**; the leader replicates the existing log to each new peer.
4. Promote the learners to **voters** so they participate in quorum.

The cluster is now a 3-node Raft group. Future membership changes reuse steps 3–4.

The cleanest way to drive these is the `fluree cluster` CLI subcommand, which wraps the private `/cluster/*` HTTP endpoints. The CLI does not need to be installed on the cluster nodes themselves — it just needs network reachability to each node's admin URL. Run it from your bastion / operator workstation over SSH tunnels or via VPC peering.

```bash
# 1. Start every node first (see "A complete per-node startup" above).

# 2. Initialize node 1 as the seed (single-voter cluster).
fluree cluster init \
  --addr       http://node-1-private:9090 \
  --node-id    1 \
  --raft-url   http://node-1-private:9090/raft \
  --client-url http://node-1:8080

# 3. Add nodes 2 and 3 as learners, blocking on catch-up so the
#    follow-up promote can't race replication.
fluree cluster add \
  --leader     http://node-1-private:9090 \
  --node-id    2 \
  --raft-url   http://node-2-private:9090/raft \
  --client-url http://node-2:8080

fluree cluster add \
  --leader     http://node-1-private:9090 \
  --node-id    3 \
  --raft-url   http://node-3-private:9090/raft \
  --client-url http://node-3:8080

# 4. Promote both learners into the voting set.
fluree cluster promote \
  --leader  http://node-1-private:9090 \
  --members 1,2,3

# 5. Confirm.
fluree cluster status --addr http://node-1-private:9090
```

After step 5 you should see `voters: 1, 2, 3`, a non-empty `leader`, and a non-zero `term`.

If you'd rather script the bootstrap directly against the HTTP surface, each CLI command is a single POST or GET — see the [Admin HTTP surface](#admin-http-surface) section below.

## Day-2 operations

### Adding a node

Same recipe as steps 3–4 of the bootstrap: start the new node, `fluree cluster add` against the current leader (blocking), then `promote` with the new voter set.

### Removing a node

`promote --members 1,2 --retain` keeps the dropped voter as a learner (useful for graceful drain); `promote --members 1,2` (no `--retain`) removes it entirely.

### Restarting a node

Restart in place. openraft re-reads its log and snapshots from `--raft-storage-path` and rejoins automatically once it can reach the leader. The node's HTTP listener is up immediately, but its `RaftNameService` returns stale data until the log catches up — clients targeting that node will see slightly older reads during the gap.

### Recovering a failed node

If a node's Raft storage is lost or corrupted, delete the contents of `--raft-storage-path` on that node, restart it, then on the current leader call `fluree cluster add` with the **same** `--node-id`. The leader will deliver the full log (or a snapshot if the log has been truncated past the join point) to bring the node current. `promote` to re-add it to the voter set when it's caught up.

### Configuration changes

Changes to client-facing config (auth, CORS, indexing thresholds) take effect on next restart and don't need cluster coordination — restart nodes one at a time, observing `fluree cluster status` between restarts to make sure quorum is preserved.

`--raft-node-id` and `--raft-listen-addr` **must not change** for a running node. If you really need to change them, treat it as remove + re-add.

### Rolling upgrades

The Raft log and snapshot encodings (postcard for `Command`, `Response`, and `NameServiceState`) are **not versioned** across binary versions. A node running a different binary may fail to deserialize entries from the leader and crash on startup. Roll upgrades one node at a time:

1. Check `fluree cluster status` — confirm the cluster is healthy (correct voter set, recent `last_applied`).
2. Stop one **follower**; upgrade and restart it. Wait for its `last_applied` to catch up to the others.
3. Repeat for the remaining followers.
4. Trigger a leadership transfer (stop and restart the current leader); upgrade and restart it.

Skipping versions is not supported. The safe path is N → N+1; for larger jumps, do them sequentially.

## Admin HTTP surface

All four `fluree cluster` actions map 1:1 to a single HTTP call. Useful when scripting in environments where the CLI isn't available.

When `--admin-auth-mode` is set on a node, these endpoints require an admin token. The CLI does not currently mint admin tokens itself — operators either disable admin auth on the Raft port (acceptable when the listener is behind a trusted network boundary) or front it with a proxy that injects the token.

### `POST /cluster/initialize`

Once, on the seed node, to bootstrap a single-voter cluster:

```bash
curl -X POST http://node-1-private:9090/cluster/initialize \
  -H 'Content-Type: application/json' \
  -d '{
        "members": {
          "1": {
            "raft_addr":   "http://node-1-private:9090/raft",
            "client_addr": "http://node-1:8080"
          }
        }
      }'
```

Returns `204 No Content` on success. `409 Conflict` if the cluster has already been initialized.

### `POST /cluster/add-learner`

Against the leader, for each new peer:

```bash
curl -X POST http://node-1-private:9090/cluster/add-learner \
  -H 'Content-Type: application/json' \
  -d '{
        "node_id":     2,
        "raft_addr":   "http://node-2-private:9090/raft",
        "client_addr": "http://node-2:8080",
        "blocking":    true
      }'
```

Returns `204 No Content` when the learner has caught up (with `blocking: true`). `421 Misdirected Request` if you hit a follower — the response body names the current leader.

### `POST /cluster/change-membership`

Against the leader:

```bash
curl -X POST http://node-1-private:9090/cluster/change-membership \
  -H 'Content-Type: application/json' \
  -d '{ "members": [1, 2, 3], "retain": false }'
```

`retain: true` keeps dropped voters as learners; `false` (default) removes them.

### `GET /cluster/status`

Against any node:

```bash
curl http://node-1-private:9090/cluster/status
```

Returns JSON:

```json
{
  "current_leader": 1,
  "current_term": 4,
  "last_applied_index": 1283,
  "voters": [1, 2, 3],
  "learners": []
}
```

## Client traffic and follower forwarding

Clients can hit **any** node on its client port. Reads always serve locally. Writes (transact, push, branch admin) are inspected by the follower-forward middleware: if the receiving node is the current leader, the request is handled in place; otherwise it's re-issued against the leader's `client_addr` (looked up from the membership the cluster replicates) and the leader's response is relayed back to the client verbatim. From the client's perspective it's a single round-trip — the extra hop lives inside the cluster.

A forwarded request carries an `x-fluree-raft-forward-hops` header. The forwarder bails with `508 Loop Detected` once the count reaches **2** — the slack absorbs an at-most-one stale membership view across a leader transition. Anything beyond is almost certainly a converging cluster, and the client should see a `503` + retry rather than a runaway forward chain.

If no leader is currently elected (mid-election), follower nodes return `503 Service Unavailable` with body `"no leader currently elected; retry shortly"`. Standard client retry/backoff handles this.

## Failure modes

### No leader elected (election in progress)

**Symptom:** Writes to any node return `503` with body `"no leader currently elected; retry shortly"`. Reads succeed against each node's last-applied state.

**When it happens:** Brief leadership transitions (heartbeat miss + election round). Normally bounded to a few hundred milliseconds on a healthy LAN. Persistent `503` means a real partition or majority loss.

**Operator action:** Standard client retry/backoff. Persistent failure → check `fluree cluster status` from every node to identify which side of the partition you're on.

### Quorum lost (minority survives)

**Symptom:** No leader elects. Writes are unavailable. Reads continue to serve from each surviving node's last-applied state but will fall behind whatever the partitioned-off nodes were still applying.

**When it happens:** Loss of `floor(N/2) + 1` voters simultaneously.

**Operator action:** Restore the lost nodes. There is no safe automated way to elect a leader from a minority — doing so risks a split-brain when the partitioned voters return. If the lost nodes are unrecoverable, a manual reset is possible (see [Recovering a failed node](#recovering-a-failed-node)) but should be treated as a recovery operation, not a routine fix.

### Stale leader pointer on a follower

**Symptom:** A follower attempts to forward, can't find a usable `client_addr` for the leader id it sees, and returns `503` with body `"leader node {id} has no usable client address"`.

**When it happens:** Brief window during a membership change where the follower's local view of the cluster lags the new membership.

**Operator action:** Client retry. Persistent failure → confirm via `fluree cluster status` that membership has converged.

### Forwarded request timeout

**Symptom:** Follower returns `504 Gateway Timeout`.

**When it happens:** The follower → leader forwarding budget is a hard-coded **60 seconds** for the full round-trip (connect through response body). Hit when the leader is alive enough to accept the connection but stuck behind a slow stage (e.g. a large transaction blocked on indexing) or a wedged CAS write.

**Operator action:** Client retry with backoff. If the leader is chronically slow, investigate leader-side metrics (queue depth, stage latency) before assuming a Raft-layer problem.

### Hop-limit exceeded

**Symptom:** Follower returns `508 Loop Detected`.

**When it happens:** Two nodes each believe the other is leader (e.g. across a membership-update race), or a misconfigured cluster.

**Operator action:** Confirm via `fluree cluster status` from each node. The hop-limit is **2**, so this requires sustained disagreement.

### Idempotent retry collision

**Symptom:** Two concurrent submissions with the same idempotency key both forward to the leader; the second receives the cached outcome of the first without re-executing.

**When it happens:** By design. The replicated idempotency cache deduplicates on `(ledger_id, idempotency_key)`.

**Operator action:** None — this is correct behavior. Note the cache TTL is **1 hour**; retries after that may execute a second time. Clients needing tighter at-most-once semantics should bound their retry window accordingly.

### Branch lifecycle abort

**Symptom:** Client receives a typed `SubmissionError::Execution { status, ... }` with HTTP status:

- `409 Conflict` — branch head was reset while the request was queued.
- `410 Gone` — branch was dropped, purged, or retracted while queued.
- `422 Unprocessable Entity` — staging failed (policy denial, malformed body, conflict outcome).

**When it happens:** The queue can outlive the branch state it targets. Admin operations and concurrent writes can invalidate queued work.

**Operator action:** None at the cluster layer — these surface to clients as ordinary application-level errors.

### Snapshot rejected

**Symptom:** A follower logs `StorageError::Corruption` while receiving an install-snapshot RPC and refuses to apply it.

**When it happens:** Either a corrupt snapshot stream, or a peer pushing a snapshot with an unsafe id (empty, `..`, > 128 bytes, or characters outside `[A-Za-z0-9_.-]`). The id validation is a path-traversal guard — a peer cannot make this happen accidentally on a clean cluster.

**Operator action:** Treat as a security signal — investigate the source peer before resuming.

### Single-node storage corruption

**Symptom:** A node fails on startup deserializing its snapshot or log.

**When it happens:** Disk corruption, partial writes that bypassed the atomic write (e.g. the underlying filesystem lost an fsync), or a binary downgrade across an incompatible postcard schema change.

**Operator action:** Wipe `--raft-storage-path` on the affected node and re-join as a learner (see [Recovering a failed node](#recovering-a-failed-node)). The cluster continues operating with `N-1` while you do this.

### Body too large on the Raft port

**Symptom:** Peer logs report `413 Payload Too Large` from the Raft listener.

**When it happens:** The per-route body limits on the Raft port are **hard-coded** (vote 1 MiB, append-entries 64 MiB, install-snapshot 1 GiB). A real append-entries batch that exceeds 64 MiB is unusual; reaching it usually means an oversized commit blob was somehow encoded inline (it shouldn't be — commit payloads ride CAS, not the log).

**Operator action:** Investigate the leader's queue for an over-sized envelope. There is no operator knob for these limits in the current build.

### Mixed-version cluster

**Symptom:** A newly-restarted node panics on deserialization of a log entry or snapshot.

**When it happens:** Binary versions across the cluster differ enough that `Command` / `NameServiceState` encodings disagree.

**Operator action:** Roll back the upgraded node; perform the upgrade one-node-at-a-time and only across adjacent versions. See [Rolling upgrades](#rolling-upgrades).

## Operational constraints and limits

### Cluster sizing

Raft requires an odd voter count for liveness. Recommended:

| Voters | Tolerates failed voters | Notes                                    |
| ------ | ----------------------- | ---------------------------------------- |
| 1      | 0                       | No fault tolerance. Prefer non-Raft mode. |
| 3      | 1                       | Smallest production-grade cluster.       |
| 5      | 2                       | Recommended for multi-AZ deployments.    |
| 7      | 3                       | Diminishing returns; commit latency rises with replication fan-out. |

Even-numbered voter counts are inadvisable: the failure budget is the same as the next-smaller odd cluster, but commit latency is worse and an exact split is possible.

Learners do not count toward quorum and can be added freely.

### Stable identities

Both `--raft-node-id` and `--raft-listen-addr` must be stable for a given node. The openraft log and snapshots are keyed by node id; changing either at runtime is treated as a new peer joining and a ghost peer remaining in membership. Bake them into each host's deployment config; if you really must change them, treat the operation as remove + re-add.

### Durable local storage

`--raft-storage-path` must be on durable storage — not tmpfs, not a ramdisk, not ephemeral container storage. The vote, log, and snapshots are all required to survive a restart; losing them on a node means re-joining that node from scratch.

### Disjoint storage paths

`--raft-storage-path` and `--storage-path` (the ledger CAS path, when using the file backend) must be different directories, with neither nested under the other. Startup validation rejects overlaps.

### Membership change ordering

The safe sequence:

1. Bootstrap node 1: `cluster init` (single-voter cluster, auto-elects).
2. Add learner: `cluster add --blocking true` against the leader, waits for catch-up.
3. Promote: `cluster promote --members <new voter set>` against the leader.
4. Repeat 2–3 for each additional node.

`--blocking true` (the default on `cluster add`) is what prevents a promote-before-replicated race that could leave the cluster un-quorumed.

### Idempotency cache TTL

- Cache TTL: **1 hour** (default).
- Eviction tick: **60 s** (leader-only).
- Marker TTL (admin clear): **5 minutes**.

Clients that need stricter at-most-once semantics must keep their retry windows well inside the cache TTL. These TTLs are compile-time defaults today; no operator-facing flags expose them.

### Network transport defaults

| Setting                       | Default        | Source                                              |
| ----------------------------- | -------------- | --------------------------------------------------- |
| RPC timeout (vote, append)    | 500 ms         | `NetworkConfig::default`                            |
| Snapshot install timeout      | 30 s           | `NetworkConfig::default`                            |
| Connect timeout               | 250 ms         | `NetworkConfig::default`                            |
| Vote body limit               | 1 MiB          | per-route hard limit                                 |
| Append-entries body limit     | 64 MiB         | per-route hard limit                                 |
| Install-snapshot body limit   | 1 GiB          | per-route hard limit                                 |
| Follower → leader forward     | 60 s           | hard-coded in the forwarder                          |
| Forward hop limit             | 2              | hard-coded in the forwarder                          |
| Per-attempt waiter timeout    | 8 s            | `QueuedTransactor` default                           |
| Idempotency cache TTL         | 1 h            | `EvictionScheduler` default                          |

These are not currently exposed as server-config flags. Larger clusters that hit the per-route body limits should raise the issue with the maintainers rather than patching local builds — the limits are tuned against a buffered HTTP body model and changing them blindly can OOM nodes during catch-up.

### What survives a leader change

| State                            | Survives? | Notes                                                                                       |
| -------------------------------- | --------- | ------------------------------------------------------------------------------------------- |
| Branch heads, ledger registry    | Yes       | Replicated via the log.                                                                     |
| Queue entries (`EnqueueCommand`) | Yes       | Replicated. The new leader's `CommitWorker` picks up where the old one left off.            |
| Idempotency cache                | Yes       | Replicated.                                                                                 |
| In-flight staged receipts        | **No**    | `StagedReceiptMap` is per-process. Receipts from in-flight stages on the old leader are lost on transition; the apply path falls back to `AppliedReceipt::Minimal` (head identity only). |
| Per-attempt waiters              | **No**    | `WaiterMap` is per-process. Clients waiting on a forwarded request see the forwarding hop time out (`504`) and must retry. Idempotent submissions retry safely; anonymous ones do not. |

## Security notes

- **`/raft/*` inter-node RPC has no built-in authentication.** Peers authenticate to one another via network trust (security groups, VPC ACLs, host firewall rules). Never expose the Raft port to the public internet — a compromised peer that can speak the RPC protocol is already inside the cluster's trust boundary.
- **`/cluster/*` admin endpoints are gated by `--admin-auth-mode` when set.** When admin auth is disabled (the default), the endpoints inherit the same network-trust assumption as the RPC. When admin auth is enabled, requests must carry a valid admin token; the `fluree cluster` CLI does not currently mint admin tokens, so operators using auth must either disable it on the Raft port or front it with a proxy that injects credentials.
- **Snapshot ids from peers are validated** before any path is constructed under the snapshots directory: empty, `..`, > 128 bytes, or characters outside `[A-Za-z0-9_.-]` are rejected. A peer cannot use a malformed snapshot id to escape the snapshots subtree.
- **Body size limits on the Raft port are enforced per-route** (vote 1 MiB, append-entries 64 MiB, install-snapshot 1 GiB) — bounded to prevent a compromised peer from forcing oversized buffer allocation. They are not configurable in the current build.
- **The `--listen-addr` (client) port is governed by the same auth knobs as the single-node transaction server** (`--events-auth-*`, `--data-auth-*`, `--admin-auth-*`). Forwarded requests preserve client auth headers across the follower→leader hop — the leader re-evaluates auth against the same trusted-issuer set, so the auth boundary is unchanged.
- **Outbound HTTP redirects on the Raft transport are disabled** to close SSRF via a 302 to an internal address (such as the EC2 instance-metadata service).


---

# operations/telemetry.md

# Telemetry and Logging

Fluree provides comprehensive logging, metrics, and tracing capabilities for monitoring and debugging production deployments.

## Logging

### Log Levels

Configure log verbosity:

```bash
--log-level error|warn|info|debug|trace
```

**error:** Critical errors only
**warn:** Warnings and errors
**info:** Informational messages (default)
**debug:** Detailed debugging information
**trace:** Very detailed tracing

### Log Formats

#### JSON Format (Recommended)

```bash
--log-format json
```

Output:
```json
{
  "timestamp": "2024-01-22T10:30:00.123Z",
  "level": "INFO",
  "target": "fluree_db_server",
  "message": "Transaction committed",
  "fields": {
    "ledger": "mydb:main",
    "t": 42,
    "duration_ms": 45,
    "flakes_added": 3
  }
}
```

Benefits:
- Machine-parseable
- Easy to index (Elasticsearch, etc.)
- Structured fields
- JSON query tools work

#### Text Format

```bash
--log-format text
```

Output:
```text
2024-01-22T10:30:00.123Z INFO  fluree_db_server] Transaction committed ledger=mydb:main t=42 duration_ms=45
```

Benefits:
- Human-readable
- Compact
- Easy to grep

### Log Output

#### Standard Output (Default)

```bash
./fluree-db-server
```

Logs to stdout/stderr.

#### Log File

```bash
--log-file /var/log/fluree/server.log
```

```toml
[logging]
file = "/var/log/fluree/server.log"
```

#### Log Rotation

Use logrotate:

```bash
# /etc/logrotate.d/fluree
/var/log/fluree/*.log {
    daily
    rotate 14
    compress
    delaycompress
    notifempty
    create 0644 fluree fluree
    sharedscripts
    postrotate
        systemctl reload fluree
    endscript
}
```

### Structured Logging

Add context to logs:

```rust
// Rust code (for reference)
info!(
    ledger = %ledger,
    t = transaction_time,
    duration_ms = duration.as_millis(),
    "Transaction committed"
);
```

Output:
```json
{
  "message": "Transaction committed",
  "ledger": "mydb:main",
  "t": 42,
  "duration_ms": 45
}
```

## Metrics

> **Planned — not yet implemented.** The metrics below are a design target for a future PR. Prometheus metrics are not currently exposed by the server. The tracing/OTEL instrumentation described in the rest of this document is the current observability mechanism.

### Prometheus Metrics (planned)

```bash
curl http://localhost:8090/metrics
```

**Planned metrics:**
- `fluree_transactions_total` - Total transactions (counter)
- `fluree_transaction_duration_seconds` - Transaction latency (histogram)
- `fluree_queries_total` - Total queries (counter)
- `fluree_query_duration_seconds` - Query latency (histogram)
- `fluree_query_errors_total` - Query errors (counter)
- `fluree_indexing_lag_transactions` - Novelty count (gauge)
- `fluree_index_duration_seconds` - Indexing time (histogram)
- `fluree_uptime_seconds` - Server uptime (gauge)

### Prometheus Integration (planned)

Configure Prometheus to scrape Fluree:

```yaml
# prometheus.yml
scrape_configs:
  - job_name: 'fluree'
    static_configs:
      - targets: ['localhost:8090']
    metrics_path: '/metrics'
    scrape_interval: 15s
```

## Distributed Tracing (OpenTelemetry)

Fluree supports OpenTelemetry (OTEL) distributed tracing, providing deep visibility into query, transaction, and indexing performance. Traces are exported to any OTLP-compatible backend (Jaeger, Grafana Tempo, AWS X-Ray, Datadog, etc.).

> **Integrating your application's traces with Fluree?** See [Distributed Tracing Integration](distributed-tracing.md) for how to correlate your spans with Fluree's -- both for the Rust library (`fluree-db-api`) and the HTTP server (`fluree-db-server` with W3C `traceparent`).

### Enabling OTEL

Build the server with the `otel` feature flag:

```bash
cargo build -p fluree-db-server --features otel --release
```

> **Building via the CLI crate?** Feature flags do not propagate across
> binaries: `cargo build -p fluree-db-cli --features otel` enables OTEL only
> for the CLI's import pipeline, not for the server the daemon launches. Use
> `--features "otel,fluree-db-server/otel"` to enable both.

Then set environment variables to configure the OTLP exporter:

```bash
OTEL_SERVICE_NAME=fluree-server \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
OTEL_EXPORTER_OTLP_PROTOCOL=grpc \
RUST_LOG=info,fluree_db_query=debug,fluree_db_transact=debug \
./target/release/fluree-db-server --data-dir ./data
```

| Environment Variable | Default | Description |
|---------------------|---------|-------------|
| `OTEL_SERVICE_NAME` | `fluree-db-server` | Service name in traces |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://localhost:4317` | OTLP receiver endpoint |
| `OTEL_EXPORTER_OTLP_PROTOCOL` | `grpc` | Protocol: `grpc` or `http/protobuf` |

### Quick Start with Jaeger

The repository includes a self-contained test harness in the `otel/` directory:

```bash
cd otel/
make all    # starts Jaeger, builds with --features otel, starts server, runs tests
make ui     # opens Jaeger UI at http://localhost:16686
```

See [Performance Investigation with Distributed Tracing](../troubleshooting/performance-tracing.md) for detailed usage.

### Dual-Layer Subscriber Architecture

The OTEL exporter uses its own `Targets` filter **independent of `RUST_LOG`**. This is a critical design choice: without it, enabling `RUST_LOG=debug` causes third-party crate spans (hyper, tonic, h2, tower-http) to flood the OTEL batch processor, which overwhelms the exporter and causes parent spans to be dropped.

```
┌──────────────────────────────────────────────────┐
│              tracing-subscriber registry          │
│                                                   │
│  ┌─────────────────────┐  ┌────────────────────┐ │
│  │   Console fmt layer  │  │   OTEL trace layer │ │
│  │   (EnvFilter from    │  │   (Targets filter: │ │
│  │    RUST_LOG)          │  │    fluree_* only)  │ │
│  └─────────────────────┘  └────────────────────┘ │
└──────────────────────────────────────────────────┘
```

- **Console layer:** Respects `RUST_LOG` as-is (all crates)
- **OTEL layer:** Exports only `fluree_*` crate targets at DEBUG level. Per-leaf-node TRACE spans (`binary_cursor_next_leaf`, `scan`) are excluded to prevent flooding the batch processor queue on large queries

This means `RUST_LOG=debug` produces verbose console output, but the OTEL exporter only receives Fluree spans -- no hyper/tonic/tower noise.

**Batch processor queue size:** The OTEL batch span processor queue is set to 1,000,000 spans. At ~200 bytes per span, this represents ~200MB of potential memory usage under sustained debug-level traffic. This is intentional to prevent span loss during investigation. At `RUST_LOG=info` without OTEL, no debug spans are created at all (true zero overhead). With OTEL enabled, the queue rarely exceeds a few thousand entries under normal operation.

### Shutdown

On server shutdown, the OTEL `SdkTracerProvider` is flushed and shut down to ensure all pending spans are exported. This is handled automatically by the server's shutdown hook.

### Dynamic Span Naming (otel.name)

Each HTTP request span is named dynamically via the `otel.name` field so that traces in Jaeger/Tempo show descriptive names instead of a generic `request`:

| Operation | otel.name examples |
|-----------|-------------------|
| Query | `query:json-ld`, `query:sparql`, `query:explain`, `query:multi-query` |
| Transact | `transact:json-ld`, `transact:sparql-update`, `transact:turtle` |
| Insert | `insert:json-ld`, `insert:turtle` |
| Upsert | `upsert:json-ld`, `upsert:turtle`, `upsert:trig` |
| Ledger mgmt | `ledger:create`, `ledger:drop`, `ledger:info`, `ledger:exists` |

The `operation` span attribute retains the handler-specific name for precise filtering when needed.

### Span Hierarchy

Fluree instruments queries, transactions, and indexing with structured tracing spans at two tiers. The only `info_span!` in the codebase is `request` (the HTTP request span). All operation spans use `debug_span!`, guaranteeing true zero overhead when OTEL is not compiled and `RUST_LOG` is at `info`.

#### Tier 1: DEBUG (operation and phase level)

All operation, phase, and operator spans. Visible when OTEL is enabled or when `RUST_LOG` includes debug:

```bash
RUST_LOG=info,fluree_db_query=debug,fluree_db_transact=debug,fluree_db_indexer=debug
```

Spans: `query_execute`, `query_prepare`, `query_run`, `txn_stage`, `txn_commit`, `commit_*` sub-spans, `index_build`, `build_all_indexes`, `build_index`, `sort_blocking`, `groupby_blocking`, core operators (`scan`, `join`, `filter`, `project`, `sort`), `format`, `policy_enforce`, etc.

#### Tier 2: TRACE (maximum detail)

Per-operator detail for deep performance analysis:

```bash
RUST_LOG=info,fluree_db_query=trace
```

Additional spans: `binary_cursor_next_leaf`, `property_join`, `group_by`, `aggregate`, `group_aggregate`, `distinct`, `limit`, `offset`, `union`, `optional`, `subquery`, `having`

#### Span Tree (Query)

```
query_execute (debug)
├── ledger_view_load (debug, ledger_id, cold_binary_store — view acquisition; the pre-prepare phase, multi-second on cold ledgers)
├── query_prepare (debug)
│   ├── reasoning_prep (debug)
│   ├── pattern_rewrite (debug, patterns_before, patterns_after)
│   └── plan (debug, pattern_count)
├── query_run (debug)
│   ├── scan (debug)
│   ├── join (debug)
│   │   └── join_next_batch (debug, per iteration)
│   ├── cyclic_scan_relation (debug, per cyclic edge: predicate, mode=full|probed, rows; cross-thread when scans parallelize)
│   ├── cyclic_index_build (debug, per cyclic edge: predicate, rows; cross-thread when scans parallelize)
│   ├── cyclic_prune (debug: rows_before, rows_after, passes)
│   ├── cyclic_wedge_build (debug, squares only)
│   ├── cyclic_enumerate (debug, per output batch: strategy, rows)
│   ├── filter (debug)
│   ├── project (debug)
│   ├── sort (debug)
│   ├── sort_blocking (debug, cross-thread via spawn_blocking)
│   └── ...
└── format (debug)
    └── inject_annotations (debug, edge_in_named_graph, path, annotation_count)
        └── annotation_arena_lookup (debug, live_count)  ← path = "arena" only
```

`inject_annotations` and `annotation_arena_lookup` fire only on
hydration responses that surface annotation bodies; both are skipped
on non-annotation ledgers via the formatter's zero-cost gate.
`path` is `"arena"` when the cached `AnnotationArenaReader` resolved
the lookup, `"scan"` when the M2a POST-scan fallback ran.

#### Span Tree (Multi-query envelope)

The `/multi-query` endpoint produces a single `request` span (otel.name = `query:multi-query`) with one `sub_query` child per envelope alias. Each `sub_query` carries the full single-query span tree shown above (`query_execute → query_prepare → query_run → ...`) under it, so per-alias execution remains drillable in trace tools.

```
request (info, otel.name=query:multi-query)
├── sub_query (debug, alias, language, effective_timeout_ms, result_status)
│   └── query_execute … (full single-query tree)
└── sub_query (debug, alias, language, effective_timeout_ms, result_status)
    └── query_execute … (full single-query tree)
```

`sub_query` attributes:

| Attribute | Set when | Description |
|-----------|----------|-------------|
| `alias` | span open | The user-supplied alias key from the envelope's `queries` map. |
| `language` | span open | `jsonld` or `sparql`. |
| `effective_timeout_ms` | permit acquired | `min(opts.timeoutMs, envelope deadline remaining)` at semaphore-acquisition time. |
| `result_status` | task end | `ok` \| `error` \| `timeout`. |

Sub-query tasks run on the tokio scheduler under a shared `JoinSet`, so the `request` span context propagates through `.instrument(span).await` to each `sub_query`. When the envelope wall deadline fires, in-flight `sub_query` spans close with `result_status = timeout`.

#### Span Tree (Transaction)

```
transact_execute (debug)
├── txn_stage (debug, insert_count, delete_count)
│   ├── where_exec (debug, pattern_count, binding_rows, retraction_count, assertion_count)
│   │   ├── delete_gen (debug, template_count, retraction_count)  ← per streaming-WHERE batch
│   │   └── insert_gen (debug, template_count, assertion_count)   ← per batch (mixed DELETE+INSERT only)
│   ├── cascade_reifies_bundle (debug, retract_input_count, lpg_edge_lifecycle, cascade_count)
│   │     ← only on annotation ledgers (gated by snapshot.has_annotations
│   │       || novelty.attachments.has_annotations())
│   ├── cancellation (debug)        ← mixed DELETE+INSERT path
│   ├── dedup_retractions (debug)   ← pure-DELETE path (no INSERT templates, not Upsert)
│   └── policy_enforce (debug)
└── txn_commit (debug, flake_count, delta_bytes)
    ├── commit_nameservice_lookup (debug)
    ├── commit_verify_sequencing (debug)
    ├── commit_namespace_delta (debug)
    ├── commit_write_raw_txn (debug)  ← await of upload task spawned at pipeline entry
    ├── commit_build_record (debug)
    ├── commit_write_commit_blob (debug)
    ├── commit_publish_nameservice (debug)
    ├── commit_generate_metadata_flakes (debug)
    ├── commit_populate_dict_novelty (debug)
    └── commit_apply_to_novelty (debug)
```

#### Span Tree (Indexing)

Indexing runs as a **separate top-level trace** (not nested under an HTTP request). Each index refresh cycle starts its own trace root:

```
index_build (debug, ledger_id)
├── commit_chain_walk (debug)
├── commit_resolve (debug, per commit)
├── dict_merge_and_remap (debug)
├── build_all_indexes (debug)
│   └── build_index (debug, per order: SPOT, PSOT, POST, OPST) [cross-thread]
├── secondary_partition (debug)
├── upload_dicts (debug)
├── upload_indexes (debug)
├── build_index_root (debug)
└── BinaryIndexStore::load (debug) [cross-thread]
```

`index_gc` is a separate top-level trace (fire-and-forget `tokio::spawn`):
```
index_gc (debug, separate trace)
├── gc_walk_chain (debug)
└── gc_delete_entries (debug)
```

#### Span Tree (Bulk Import / fluree-ingest)

Bulk import runs as a **standalone top-level trace** under the `fluree-cli` service (no HTTP server involved). The import pipeline instruments all major phases:

```
bulk_import (debug, alias)
├── import_chunks (debug, total_chunks, parse_threads)
│   ├── [resolver thread: inherits parent context]
│   ├── [ttl-parser-N threads: inherit parent context]
│   └── commit + run generation log events
├── import_index_build (debug)
│   ├── build_all_indexes (debug)
│   │   └── build_index (debug, per order: SPOT, PSOT, POST, OPST) [cross-thread]
│   ├── import_cas_upload (debug)
│   └── import_publish (debug)
└── cleanup log events
```

The `import_chunks` span covers the parse+commit loop. Spawned threads (resolver, parse workers) and async tasks (dict upload, index build) inherit the parent span context so their work appears nested in the trace waterfall.

### Tracker-to-Span Bridge

When tracked queries or transactions are executed (via the `/query` or `/update` endpoints with tracking enabled), the `tracker_time` and `tracker_fuel` fields are recorded as deferred attributes on the `query_execute` and `transact_execute` spans. These values appear as span attributes in OTEL backends (Jaeger, Tempo, etc.), enabling correlation between the Tracker's fuel accounting and the span waterfall.

### RUST_LOG Quick Reference

| Goal | Pattern | What you see |
|------|---------|--------------|
| Production default | `info` | HTTP `request` spans only (zero operation spans) |
| Debug slow queries | `info,fluree_db_query=debug` | + `query_execute`, `query_prepare`, `query_run`, operators |
| Debug slow transactions | `info,fluree_db_transact=debug` | + `txn_stage`, `txn_commit`, commit sub-spans |
| Full phase decomposition | `info,fluree_db_query=debug,fluree_db_transact=debug,fluree_db_indexer=debug` | All debug spans |
| Per-operator detail | `info,fluree_db_query=trace` | + per-leaf: `binary_cursor_next_leaf`, etc. |
| Console firehose | `debug` | Everything (OTEL still filters to `fluree_*`) |

**Note:** When OTEL is enabled, the OTEL `Targets` filter always captures `fluree_*` spans at DEBUG regardless of `RUST_LOG`. The table above describes console output visibility only.

### Further Reading

- [Distributed Tracing Integration](distributed-tracing.md) -- How to correlate your application's traces with Fluree (library and HTTP)
- [Performance Investigation with Distributed Tracing](../troubleshooting/performance-tracing.md) -- How to use tracing to find bottlenecks, including AWS deployment patterns (ECS, Lambda, X-Ray, Tempo)
- [Adding Tracing Spans](../contributing/tracing-guide.md) -- How contributors should instrument new code
- [otel/ README](../../otel/README.md) -- OTEL validation harness reference

## Monitoring Integration

### Grafana Dashboards

Import Fluree dashboard:

```json
{
  "dashboard": {
    "title": "Fluree Monitoring",
    "panels": [
      {
        "title": "Query Rate",
        "targets": [
          {
            "expr": "rate(fluree_queries_total[5m])"
          }
        ]
      },
      {
        "title": "Query Latency (p95)",
        "targets": [
          {
            "expr": "histogram_quantile(0.95, fluree_query_duration_seconds)"
          }
        ]
      },
      {
        "title": "Indexing Lag",
        "targets": [
          {
            "expr": "fluree_indexing_lag_transactions"
          }
        ]
      }
    ]
  }
}
```

### Datadog Integration

Send logs to Datadog:

```bash
./fluree-db-server \
  --log-format json | \
  datadog-agent stream --service=fluree
```

### New Relic Integration

Use New Relic agent:

```bash
export NEW_RELIC_LICENSE_KEY=your-key
export NEW_RELIC_APP_NAME=fluree-prod

./fluree-db-server
```

### Elasticsearch/Kibana

Ship logs to Elasticsearch:

```bash
./fluree-db-server \
  --log-format json | \
  filebeat -e -c filebeat.yml
```

Filebeat config:
```yaml
filebeat.inputs:
  - type: stdin
    json.keys_under_root: true

output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "fluree-logs-%{+yyyy.MM.dd}"
```

## Health Monitoring

### Health Check Endpoint

```bash
curl http://localhost:8090/health
```

Response (healthy):
```json
{
  "status": "healthy",
  "version": "0.1.0",
  "storage": "file",
  "uptime_ms": 3600000,
  "checks": {
    "storage": "healthy",
    "indexing": "healthy",
    "nameservice": "healthy"
  }
}
```

Response (unhealthy):
```json
{
  "status": "unhealthy",
  "checks": {
    "storage": "healthy",
    "indexing": "unhealthy",
    "nameservice": "healthy"
  },
  "errors": [
    {
      "component": "indexing",
      "message": "Indexing lag exceeds threshold"
    }
  ]
}
```

### Liveness Probe

For Kubernetes:

```yaml
livenessProbe:
  httpGet:
    path: /health
    port: 8090
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3
```

### Readiness Probe

```yaml
readinessProbe:
  httpGet:
    path: /ready
    port: 8090
  initialDelaySeconds: 10
  periodSeconds: 5
  timeoutSeconds: 3
```

## Alerting

### Alert Rules

Prometheus alert rules:

```yaml
groups:
  - name: fluree
    rules:
      - alert: HighQueryLatency
        expr: histogram_quantile(0.95, fluree_query_duration_seconds) > 1
        for: 5m
        annotations:
          summary: "High query latency"
          description: "95th percentile query latency is {{ $value }}s"
      
      - alert: HighIndexingLag
        expr: fluree_indexing_lag_transactions > 100
        for: 10m
        annotations:
          summary: "High indexing lag"
          description: "Indexing lag is {{ $value }} transactions"
      
      - alert: HighErrorRate
        expr: rate(fluree_query_errors_total[5m]) > 10
        for: 5m
        annotations:
          summary: "High query error rate"
          description: "Error rate is {{ $value }}/s"
```

### Alert Destinations

Configure alert routing:

```yaml
route:
  receiver: 'team-ops'
  group_by: ['alertname', 'ledger']
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty'
    - match:
        severity: warning
      receiver: 'slack'

receivers:
  - name: 'pagerduty'
    pagerduty_configs:
      - service_key: 'your-key'
  
  - name: 'slack'
    slack_configs:
      - api_url: 'https://hooks.slack.com/...'
        channel: '#alerts'
```

## Performance Monitoring

### Key Metrics to Track

1. **Query Performance:**
   - p50, p95, p99 latency
   - Queries per second
   - Error rate

2. **Transaction Performance:**
   - Commit time
   - Transactions per second
   - Error rate

3. **Indexing:**
   - Novelty count
   - Index time
   - Indexing lag

4. **Resource Usage:**
   - CPU utilization
   - Memory usage
   - Disk I/O
   - Network I/O

5. **Storage:**
   - Storage used
   - Storage growth rate
   - S3 request rate (if AWS)

### Dashboards

Create operational dashboards:

**Overview Dashboard:**
- Request rate
- Error rate
- Response times
- Active connections

**Performance Dashboard:**
- Query latency percentiles
- Transaction latency
- Indexing performance
- Resource utilization

**Capacity Dashboard:**
- Storage usage and growth
- Memory usage trends
- Indexing lag trends
- Projection to capacity limits

## Logging Best Practices

### 1. Use Structured Logging

JSON format with consistent fields:

```json
{
  "timestamp": "2024-01-22T10:30:00Z",
  "level": "INFO",
  "ledger": "mydb:main",
  "operation": "query",
  "duration_ms": 45
}
```

### 2. Log Request IDs

Include request IDs for tracing:

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "X-Request-ID: abc-123-def-456" \
  -d '{...}'
```

### 3. Appropriate Log Levels

- Production: `info`
- Debugging: `debug`
- Development: `debug` or `trace`

### 4. Sample High-Volume Logs

For high-traffic deployments, sample logs:

```toml
[logging]
sample_rate = 0.1  # Log 10% of requests
```

### 5. Sensitive Data

Never log sensitive data:
- API keys
- Passwords
- Personal information
- Financial data

## Related Documentation

- [Configuration](configuration.md) - Configuration options
- [Admin and Health](admin-and-health.md) - Health monitoring
- [Troubleshooting](../troubleshooting/README.md) - Debugging guides


---

# operations/distributed-tracing.md

# Distributed Tracing Integration

This guide explains how to correlate your application's traces and logs with Fluree's internal instrumentation, whether you use Fluree as an embedded Rust library (`fluree-db-api`) or as an HTTP server (`fluree-db-server`).

## Overview

Fluree instruments queries, transactions, and indexing with [tracing](https://docs.rs/tracing) spans. These spans can participate in your application's distributed traces so that a single trace shows the full picture: your application code, the Fluree call, and every internal phase (parsing, planning, execution, commit, etc.).

There are two integration paths depending on how you use Fluree:

| Integration mode | Mechanism | What you get |
|-----------------|-----------|--------------|
| **Rust library** (`fluree-db-api`) | Shared `tracing` subscriber | Fluree spans automatically nest under your application spans |
| **HTTP server** (`fluree-db-server`) | W3C Trace Context (`traceparent` header) | Fluree's request span becomes a child of your distributed trace |

## Rust Library Integration (`fluree-db-api`)

When you embed Fluree via `fluree-db-api`, trace correlation works automatically through the `tracing` crate's context propagation -- no special Fluree configuration required.

### How it works

The `tracing` crate uses task-local storage to track the "current span." When your code creates a span and then calls a Fluree API method, any spans Fluree creates internally become children of your span. This happens automatically as long as both your code and Fluree share the same `tracing` subscriber (which they do by default -- there's one global subscriber per process).

### Basic setup

```rust
use fluree_db_api::{FlureeBuilder, Result};
use tracing_subscriber::{EnvFilter, fmt};

#[tokio::main]
async fn main() -> Result<()> {
    // Initialize tracing -- Fluree's spans will appear here too
    tracing_subscriber::fmt()
        .with_env_filter(EnvFilter::from_default_env())
        .init();

    let fluree = FlureeBuilder::new()
        .with_storage_path("./data")
        .build()
        .await?;

    // Your application span wraps the Fluree call
    let span = tracing::info_span!("handle_request", user_id = %user_id);
    async {
        let db = fluree.db("my-ledger", None).await?;
        let result = fluree.query(&db, my_query).await?;
        Ok(result)
    }
    .instrument(span)
    .await
}
```

At the default `RUST_LOG=info`, Fluree's info-level log events appear within your span's context:

```
INFO handle_request{user_id=42}: fluree_db_api::view::query: parse_ms=0.12 plan_ms=0.45 exec_ms=3.21 query phases
```

With `RUST_LOG=info,fluree_db_query=debug`, you additionally see Fluree's operation spans nested under yours:

```
INFO  handle_request{user_id=42}: my_app: handling request
DEBUG handle_request{user_id=42}:query_execute: fluree_db_query: ...
DEBUG handle_request{user_id=42}:query_execute:query_prepare: fluree_db_query: ...
DEBUG handle_request{user_id=42}:query_execute:query_run: fluree_db_query: ...
INFO  handle_request{user_id=42}:query_execute: fluree_db_api: parse_ms=0.12 plan_ms=0.45 exec_ms=3.21 query phases
```

### With OpenTelemetry export

If your application exports traces to an OTEL backend (Jaeger, Tempo, Datadog, etc.), Fluree's spans appear in the same trace waterfall:

```rust
use opentelemetry::global;
use opentelemetry_otlp::WithExportConfig;
use tracing_opentelemetry::OpenTelemetryLayer;
use tracing_subscriber::{layer::SubscriberExt, EnvFilter, Registry};

fn init_tracing() {
    let exporter = opentelemetry_otlp::SpanExporter::builder()
        .with_tonic()
        .with_endpoint("http://localhost:4317")
        .build()
        .expect("OTLP exporter");

    let provider = opentelemetry_sdk::trace::SdkTracerProvider::builder()
        .with_simple_exporter(exporter)
        .build();

    global::set_tracer_provider(provider);

    let otel_layer = OpenTelemetryLayer::new(global::tracer("my-app"));

    let subscriber = Registry::default()
        .with(otel_layer)
        .with(EnvFilter::from_default_env())
        .with(tracing_subscriber::fmt::layer());

    tracing::subscriber::set_global_default(subscriber).unwrap();
}
```

In Jaeger/Tempo, you'll see a single trace containing both your application spans and Fluree's internal spans (`query_execute`, `query_prepare`, `query_run`, `scan`, `join`, etc.).

### Three tiers of visibility

Fluree uses a tiered logging strategy. At every tier, events and spans are correlated to your application's active span.

| Tier | `RUST_LOG` pattern | What you see from Fluree |
|------|-------------------|-------------------------|
| **Logs** | `info` (default) | Info-level log events: phase timings (`parse_ms`, `plan_ms`, `exec_ms`), commit summaries, errors. Zero span overhead. |
| **Operation spans** | `info,fluree_db_query=debug` | + `query_execute`, `query_prepare`, `query_run`, operator spans — timing waterfall in Jaeger/Tempo |
| **Deep tracing** | `info,fluree_db_query=trace` | + per-leaf, per-iteration detail (`binary_cursor_next_leaf`, `group_by`, etc.) |

At the default **INFO** level, you get Fluree's summary log events (timings, counts, errors) correlated inside your spans. This is sufficient for most production correlation needs.

At **DEBUG**, you additionally get the structured span hierarchy that produces the timing waterfall in OTEL backends. This is useful for performance investigation.

Useful `RUST_LOG` patterns:

| Pattern | Use case |
|---------|----------|
| `info` | Production: correlatable log events, zero span overhead |
| `info,fluree_db_query=debug` | Investigate slow queries |
| `info,fluree_db_transact=debug` | Investigate slow transactions |
| `info,fluree_db_query=debug,fluree_db_transact=debug` | Full operation visibility |
| `debug` | Everything, but includes third-party crate noise |

See [Telemetry and Logging](telemetry.md#span-hierarchy) for the full span hierarchy.

### Key span names and fields

These are the most useful spans and fields for application-level correlation:

| Span | Level | Key fields | When it appears |
|------|-------|-----------|-----------------|
| `query_execute` | DEBUG | `ledger_id` | Every query |
| `query_prepare` | DEBUG | `pattern_count` | Query planning phase |
| `query_run` | DEBUG | | Query execution phase |
| `transact_execute` | DEBUG | `ledger_id` | Every transaction |
| `txn_stage` | DEBUG | `insert_count`, `delete_count` | Transaction staging |
| `txn_commit` | DEBUG | `flake_count`, `delta_bytes` | Commit to storage |
| `format` | DEBUG | `output_format`, `result_count` | Result serialization |

### Adding your own context to Fluree spans

Since spans nest automatically, the simplest approach is to wrap Fluree calls with your own spans containing the context you need:

```rust
let span = tracing::info_span!(
    "api_query",
    user_id = %user_id,
    endpoint = %path,
    ledger = %ledger_alias,
);

let result = async {
    fluree.query(&db, query).await
}
.instrument(span)
.await?;
```

All of Fluree's internal spans inherit the `user_id`, `endpoint`, and `ledger` fields from the parent span in trace backends that support field inheritance.

## HTTP Server Integration (`fluree-db-server`)

When Fluree runs as a standalone HTTP server, your application connects over HTTP. Distributed trace correlation uses the [W3C Trace Context](https://www.w3.org/TR/trace-context/) standard.

### W3C `traceparent` header

When your application sends a `traceparent` header with an HTTP request, `fluree-db-server` automatically makes its `request` span a child of your trace. This requires the `otel` feature to be enabled on the server.

```
traceparent: 00-{trace-id}-{parent-span-id}-{trace-flags}
```

Example request:

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "Content-Type: application/json" \
  -H "traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01" \
  -d '{"from": "my-ledger", "select": {"?s": ["*"]}, "where": [["?s", "rdf:type", "schema:Person"]]}'
```

The resulting trace in Jaeger/Tempo:

```
your-service: handle_request          ─────────────────────────────
  fluree-server: request (query:json-ld) ──────────────────────────
    query_execute                           ─────────────────────
      query_prepare                         ────
      query_run                                 ───────────────
        scan                                    ─────
        join                                         ─────────
      format                                                   ──
```

### Server requirements

W3C trace context propagation requires:

1. **`otel` feature enabled** at build time:
   ```bash
   cargo build -p fluree-db-server --features otel --release
   ```

2. **OTEL environment variables set** at runtime:
   ```bash
   OTEL_SERVICE_NAME=fluree-server \
   OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
   ./fluree-server
   ```

Without the `otel` feature, the `traceparent` header is still parsed and the trace ID is recorded as a log field for text-based correlation, but the span is not linked as a child in the OTEL trace.

For background indexing triggered by a transaction request, note the distinction between logs and traces:

- The later indexing work still runs in its own background task and appears as a separate trace/span tree.
- Fluree copies the triggering request's `request_id` and `trace_id` into the queued indexing job, so the background worker's log lines can still be correlated back to the originating request.
- If multiple requests coalesce onto one queued indexing job, the latest queued request metadata is the one retained on the worker logs.

### `X-Request-ID` header (non-OTEL correlation)

For simpler log correlation without full distributed tracing, send an `X-Request-ID` header:

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "X-Request-ID: abc-123-def-456" \
  -d '...'
```

The server logs and echoes back this ID in the response headers. All log lines for the request include the `request_id` field, so you can correlate with:

```bash
# In JSON log output:
grep '"request_id":"abc-123-def-456"' /var/log/fluree/server.log
```

This works without the `otel` feature and is useful for text-based log correlation. The same `request_id` is also copied onto background indexing logs when that request queues an index build, which helps connect the foreground transaction and later worker activity in plain log search.

### Client examples

#### Python (OpenTelemetry)

```python
from opentelemetry import trace
from opentelemetry.propagate import inject
import requests

tracer = trace.get_tracer("my-app")

with tracer.start_as_current_span("fluree_query") as span:
    headers = {"Content-Type": "application/json"}
    inject(headers)  # adds traceparent header automatically

    response = requests.post(
        "http://localhost:8090/v1/fluree/query",
        headers=headers,
        json={
            "from": "my-ledger",
            "select": {"?s": ["*"]},
            "where": [["?s", "rdf:type", "schema:Person"]],
        },
    )
```

#### JavaScript / TypeScript (OpenTelemetry)

```typescript
import { trace, context, propagation } from "@opentelemetry/api";

const tracer = trace.getTracer("my-app");

await tracer.startActiveSpan("fluree_query", async (span) => {
  const headers: Record<string, string> = {
    "Content-Type": "application/json",
  };
  propagation.inject(context.active(), headers);

  const response = await fetch("http://localhost:8090/v1/fluree/query", {
    method: "POST",
    headers,
    body: JSON.stringify({
      from: "my-ledger",
      select: { "?s": ["*"] },
      where: [["?s", "rdf:type", "schema:Person"]],
    }),
  });

  span.end();
  return response;
});
```

#### Rust (reqwest + tracing-opentelemetry)

```rust
use opentelemetry::global;
use opentelemetry::propagation::Injector;
use reqwest::header::HeaderMap;

struct HeaderInjector<'a>(&'a mut HeaderMap);
impl Injector for HeaderInjector<'_> {
    fn set(&mut self, key: &str, value: String) {
        if let Ok(name) = key.parse() {
            if let Ok(val) = value.parse() {
                self.0.insert(name, val);
            }
        }
    }
}

let span = tracing::info_span!("fluree_query", ledger = "my-ledger");
let _guard = span.enter();

let mut headers = HeaderMap::new();
global::get_text_map_propagator(|propagator| {
    propagator.inject(&mut HeaderInjector(&mut headers));
});

let response = reqwest::Client::new()
    .post("http://localhost:8090/v1/fluree/query")
    .headers(headers)
    .json(&query)
    .send()
    .await?;
```

## Correlation Strategy Summary

| Scenario | Mechanism | Setup required |
|----------|-----------|---------------|
| Rust app embedding `fluree-db-api` | Shared `tracing` subscriber | None -- automatic |
| Rust app embedding `fluree-db-api` with OTEL | Shared subscriber + OTEL layer | Add `OpenTelemetryLayer` to subscriber |
| HTTP client → `fluree-db-server` (OTEL) | `traceparent` header | Server built with `otel` feature + OTEL env vars |
| HTTP client → `fluree-db-server` (log only) | `X-Request-ID` header | None -- works out of the box |

## Related Documentation

- [Telemetry and Logging](telemetry.md) -- Server-side logging, OTEL export, span hierarchy
- [Adding Tracing Spans](../contributing/tracing-guide.md) -- Contributor guide for instrumenting new code
- [Performance Investigation](../troubleshooting/performance-tracing.md) -- Using traces to find bottlenecks
- [Using Fluree as a Rust Library](../getting-started/rust-api.md) -- General library usage guide


---

# operations/pack-archive-restore.md

# Pack format: archive and restore

Fluree's `.flpack` format is a self-contained binary snapshot of an entire ledger -- commits, transaction payloads, and (optionally) binary index artifacts. It enables ledger portability: archive a ledger to cold storage, restore it later under the same or a different name, or move it between environments.

## Overview

The pack protocol (`fluree-pack-v1`) was designed for efficient bulk transfer between Fluree instances. The same format works equally well for file-based archive/restore workflows. Because all objects inside a pack are content-addressed (identified by `ContentId` / CIDv1), the ledger name only matters at the nameservice layer -- making rename-on-restore straightforward.

### What's in a `.flpack` file?

A `.flpack` file is a binary stream of frames:

```text
[Preamble: FPK1 + version(1)]
[Header frame]        -- JSON metadata (commit count, estimated size, etc.)
[Data frames...]      -- commits + txn blobs (oldest-first, topological order)
[Manifest frame]?     -- marks start of index artifact phase (if included)
[Data frames...]?     -- index branches, leaves, dict blobs, roots
[End frame]
```

Each data frame contains a CID (content identity) and the raw bytes of the object. On ingest, every frame is integrity-verified before being written to storage.

### With or without indexes

A pack can include just commits + txn blobs (compact, sufficient for full restore -- queries replay from commits), or it can also include binary index artifacts (larger, but the restored ledger is immediately queryable without reindexing).

## CLI usage

### Archive (export to `.flpack`)

```bash
# Local ledger
fluree export mydb --format ledger -o mydb.flpack

# Smaller archive without binary index artifacts (importer will reindex):
fluree export mydb --format ledger --no-indexes -o mydb.flpack

# Remote ledger (cold-archive a production ledger to local disk):
fluree export mydb --remote prod --format ledger -o mydb.flpack
```

`--format ledger` (alias `--format flpack`) writes the full `fluree-pack-v1` archive — commits, txn blobs, and (unless `--no-indexes`) index artifacts — plus a `phase: "nameservice"` manifest frame that lets the importer reconstruct commit/index head pointers.

`-o FILE` is required when stdout is a TTY (the archive is binary). Pipe-friendly forms work too: `fluree export mydb --format ledger > mydb.flpack`.

The local path calls `Fluree::archive_ledger`. The `--remote` path calls `GET /storage/ns/:ledger-id` to fetch the remote NsRecord, then streams `POST /pack/*ledger` and substitutes the nameservice manifest in place of the terminal End frame on the fly — so a remote-sourced archive is byte-compatible with a locally-generated one. Both endpoints require a Bearer token with `fluree.storage.*` permissions (same auth bracket as `fluree clone` / `pull`).

For non-CLI archive flows (S3 upload, custom storage), use `Fluree::archive_ledger` directly — see [Rust API usage](#rust-api-usage) below.

### Restore (import from `.flpack`)

```bash
# Restore into a new LOCAL ledger
fluree create my-restored-ledger --from /path/to/archive.flpack

# Restore onto a REMOTE server (streams the archive to POST /import)
fluree create my-restored-ledger --remote origin --from /path/to/archive.flpack
```

Both forms read the `.flpack` archive, ingest all CAS objects (verifying each), and create a new ledger pointing at the imported commit chain. The ledger name (`my-restored-ledger`) is independent of whatever the original ledger was called.

The remote form streams the file to the server's `POST /import/<ledger>` endpoint, so the server materializes the ledger itself — no local staging instance required. This makes `.flpack` the universal on-ramp for getting data onto a server: build a ledger locally in **any** supported format, export it, then import it wholesale.

```bash
# The generic "load any data onto a server" pattern
fluree create staging --from data.ttl          # any format: ttl, jsonld, nq, dir, jsonl, …
fluree export staging --format ledger -o snap.flpack
fluree create prod --remote origin --from snap.flpack
```

> **Restore vs. publish.** `fluree publish` folds a *local ledger's commits* into a remote, re-validated and reindexed (the `POST /push` path). `--remote --from <archive>.flpack` restores a *trusted snapshot* wholesale — byte-for-byte, index included, no replay. Use publish to merge ongoing work; use import to materialize a ledger from an archive.

## Rust API usage

All building blocks for archive/restore live in the API and core crates -- no CLI dependency required.

### Dependencies

```toml
[dependencies]
fluree-db-api = { version = "0.1", features = ["native"] }
fluree-db-core = "0.1"
fluree-db-nameservice-sync = "0.1"
tokio = { version = "1", features = ["full"] }
```

### Archive: generate a `.flpack` file

Use `stream_pack()` from `fluree-db-api` to generate pack frames, then write them to a file (or S3, GCS, etc.).

```rust
use fluree_db_api::{Fluree, FlureeBuilder};
use fluree_db_api::pack::{full_ledger_pack_request, stream_pack};
use tokio::sync::mpsc;
use tokio::io::AsyncWriteExt;

async fn archive_ledger(
    fluree: &Fluree<impl Storage + Clone + Send + Sync + 'static, impl NameService + RefPublisher + Send + Sync>,
    ledger_id: &str,
    output_path: &std::path::Path,
) -> Result<(), Box<dyn std::error::Error>> {
    let handle = fluree.ledger(ledger_id).await?;

    // Build a request that captures the current head commit (and index
    // root, if present). `include_indexes = true` gives the restored
    // ledger instant queryability; pass `false` for a smaller archive
    // that reindexes on import. Empty `want` is always rejected by
    // `stream_pack`, so always build via this helper.
    //
    // `full_ledger_pack_request` sets `include_txns = true` by default.
    // To produce an even smaller archive without original transaction
    // payloads (verifiable but not replayable), mutate the returned
    // request: `request.include_txns = false;`.
    let request = full_ledger_pack_request(&handle, /* include_indexes */ true).await?;

    let (tx, mut rx) = mpsc::channel(64);

    // Spawn the pack generator
    let fluree_clone = fluree.clone();
    let handle_clone = handle.clone();
    let req_clone = request.clone();
    tokio::spawn(async move {
        let _ = stream_pack(&fluree_clone, &handle_clone, &req_clone, tx).await;
    });

    // Write frames to file
    let mut file = tokio::fs::File::create(output_path).await?;
    while let Some(chunk) = rx.recv().await {
        file.write_all(&chunk.bytes).await?;
    }
    file.flush().await?;

    Ok(())
}
```

To archive to S3 instead of a local file, replace the file writer with your S3 upload (e.g., `aws_sdk_s3` multipart upload consuming chunks from `rx`).

### Restore: ingest a `.flpack` file

`Fluree::restore_ledger` does the whole restore in one streaming call: it creates the target ledger, ingests and verifies every CAS object, finalizes the commit/index heads from the embedded nameservice manifest, and rolls back on any failure. It is the import counterpart of `Fluree::archive_ledger`.

It reads from any `AsyncRead`, so the **source** of the archive is open-ended — a local file, an S3 `GetObject` body, an HTTP response, or a pipe. The archive is decoded one frame at a time and never buffered whole, so multi-gigabyte production archives restore without exhausting memory.

```rust
use fluree_db_api::Fluree;

async fn restore_from_file(
    fluree: &Fluree,
    new_ledger_id: &str,
    path: &std::path::Path,
) -> Result<(), Box<dyn std::error::Error>> {
    let mut file = tokio::fs::File::open(path).await?;
    let result = fluree.restore_ledger(new_ledger_id, &mut file).await?;
    println!(
        "restored {new_ledger_id}: {} commits, {} index artifacts",
        result.commits, result.index_artifacts,
    );
    Ok(())
}
```

To restore from a non-file source, adapt it into an `AsyncRead` and pass it to the same call:

```rust
// From an HTTP / S3 byte stream (Stream<Item = Result<Bytes, _>>):
use tokio_util::io::StreamReader;
let mut reader = StreamReader::new(byte_stream.map_err(std::io::Error::other));
fluree.restore_ledger(new_ledger_id, &mut reader).await?;

// From an aws-sdk-s3 GetObject body:
let mut reader = get_object_output.body.into_async_read();
fluree.restore_ledger(new_ledger_id, &mut reader).await?;
```

The server's `POST /import/<ledger>` endpoint is exactly this: it adapts the request body into an `AsyncRead` and calls `restore_ledger`.

### Key points

- **Rename on restore**: The `new_ledger_id` argument controls the ledger name (a bare name is normalized to `name:main`). CAS objects are content-addressed and name-agnostic; only the nameservice pointer uses the name. The one exception is the FIR6 **index root**, which carries the source `ledger_id` as an inline identity field: restoring under a new name re-stamps that field and re-writes the root under a fresh CID before pointing the index head at it. Without this, the restored ledger would query fine (cold load trusts the root's own name) but reject every write — `apply_loaded_db` asserts the loaded root's `ledger_id` matches the live ledger, so a mismatch loops writes as retryable and leaves the ledger silently read-only. Branches, leaves, and dict blobs are name-independent and ride along verbatim. (The historical txn-meta/config graph IRIs keep the source name — they live in the content-addressed dict tree, matching clone/pull semantics.)
- **Integrity**: Every data frame is verified (SHA-256) before writing, and the manifest's commit/index head CIDs are checked to be present in the archive before the heads are set — a corrupted, truncated, or mismatched archive is rejected rather than creating a dangling head.
- **Atomic on failure**: Any mid-stream error rolls back the half-created ledger, so a failed restore never leaves a live, partially-ingested ledger behind.
- **Indexes are optional**: Without indexes, the restored ledger is functional but replays from commits (or reindexes) before queries are efficient. With indexes, it's queryable immediately.
- **Default context preserved**: If the source ledger has a stored default JSON-LD context, its blob travels in the archive and the restored ledger keeps it — so queries that omit an inline `@context` and rely on the ledger's prefixes keep working. (Currently the local `archive_ledger` path; the remote `export --remote --format ledger` path does not yet carry it.)
- **Storage-agnostic destination**: The restored ledger can live on file storage, S3, or any backend that implements the `Storage` trait — independent of where the archive's bytes came from.
- **Concurrent CAS writes**: A large archive holds tens of thousands of CAS objects (DBLP: ~74k). Restore decodes frames sequentially (cheap) but dispatches the per-object storage writes into a bounded pool so they overlap — otherwise wall-clock is the *sum* of per-object round-trips (minutes of pure latency on S3). The pool is bounded by both count and bytes-in-flight, so a burst of large dictionary packs can't balloon memory. This still doesn't make a 20 GB+ restore instantaneous — the archive is read as one sequential stream — so a genuinely huge restore belongs on a long-lived worker, not a short-timeout serverless function.
- **Per-frame size**: each data frame carries one whole CAS object. The indexer packs dictionaries to a 256 MiB target, so a single pack can land just over 256 MiB. Restore decodes with a 1 GiB per-frame ceiling (the frame length is a `u32`; every frame is SHA-256-verified) so these oversized-but-legitimate objects load. The conservative 256 MiB pull-path default would reject them — restore deliberately uses the larger cap.
- **Large archives (> 5 GB) use multipart upload**: The `Fluree::restore_ledger` API itself streams an `AsyncRead` of any size with no cap. The size limit only appears in the *negotiated CLI upload* to a size-capped app server: a single S3 presigned PUT rejects bodies over 5 GiB, so the CLI/server negotiate a **multipart** upload (the app mints one presigned `UploadPart` URL per part, the CLI uploads parts, the app calls `CompleteMultipartUpload`). This is automatic — see the [Negotiated Upload Import Contract](../cli/server-integration.md#negotiated-upload-import-contract). Note that compressing the archive further is not a workaround: commit ops and index leaves are already compressed inside the `.flpack`, so an outer pass yields little.

## Wire format reference

For full protocol details including frame encoding, see:

- [Storage-agnostic commits and sync](../design/storage-agnostic-commits-and-sync.md) -- design rationale and protocol overview
- [ContentId and ContentStore](../design/content-id-and-contentstore.md) -- CID encoding and verification
- `fluree-db-core/src/pack.rs` -- wire format constants and encode/decode functions

## Architecture

| Concern | Crate | Key file |
|---------|-------|----------|
| Wire format (FPK1 frames, encode/decode) | `fluree-db-core` | `src/pack.rs` |
| Pack stream generation (export) | `fluree-db-api` | `src/pack.rs` |
| Archive export (`Fluree::archive_ledger`) | `fluree-db-api` | `src/lib.rs` |
| Streaming restore (`Fluree::restore_ledger`) + head finalization | `fluree-db-api` | `src/commit_transfer.rs` |
| HTTP export endpoint (`POST /v1/fluree/pack/*`) | `fluree-db-server` | `src/routes/pack.rs` |
| HTTP import endpoint (`POST /v1/fluree/import/*`) | `fluree-db-server` | `src/routes/import.rs` |
| Network sync ingestion (clone/pull) | `fluree-db-nameservice-sync` | `src/pack_client.rs` |
| CLI `.flpack` import (local + `--remote`) | `fluree-db-cli` | `src/commands/create.rs` |


---

# operations/admin-and-health.md

# Admin, Health, and Stats

This document covers administrative operations, health monitoring, and server statistics for Fluree deployments.

## Health Endpoints

### GET /health

Basic health check:

```bash
curl http://localhost:8090/health
```

**Response (200 OK):**
```json
{
  "status": "ok",
  "version": "0.1.0"
}
```

Use this endpoint for:
- Load balancer health checks
- Container orchestration (Kubernetes liveness/readiness probes)
- Monitoring systems

**Kubernetes Example:**
```yaml
livenessProbe:
  httpGet:
    path: /health
    port: 8090
  initialDelaySeconds: 5
  periodSeconds: 10
readinessProbe:
  httpGet:
    path: /health
    port: 8090
  initialDelaySeconds: 5
  periodSeconds: 5
```

## Statistics Endpoints

### GET /v1/fluree/stats

Server statistics:

```bash
curl http://localhost:8090/v1/fluree/stats
```

**Response:**
```json
{
  "uptime_secs": 3600,
  "storage_type": "file",
  "indexing_enabled": true,
  "cached_ledgers": 3,
  "version": "0.1.0"
}
```

| Field | Description |
|-------|-------------|
| `uptime_secs` | Server uptime in seconds |
| `storage_type` | Storage mode (`memory` or `file`) |
| `indexing_enabled` | Whether background indexing is enabled |
| `cached_ledgers` | Number of ledgers currently cached |
| `version` | Server version |

## Diagnostic endpoints

### GET /v1/fluree/whoami

Diagnostic endpoint for debugging Bearer tokens.

- If no token is present, returns `token_present=false`.
- If a token is present, attempts to **cryptographically verify** it using the same verification logic as authenticated endpoints (embedded-JWK Ed25519 and JWKS/OIDC when enabled/configured).
- On verification failure, returns `verified=false` and includes an `error` string. Some unverified decoded fields may be included for debugging.

```bash
curl http://localhost:8090/v1/fluree/whoami \
  -H "Authorization: Bearer eyJ..."
```

## CLI discovery

### GET /.well-known/fluree.json

Discovery document used by the CLI when adding a remote (`fluree remote add`) or when running `fluree auth login` with no configured auth type.

Standalone `fluree-server` returns:

- `{"version":1,"api_base_url":"/v1/fluree"}` when no auth is enabled
- `{"version":1,"api_base_url":"/v1/fluree","auth":{"type":"token"}}` when any server auth mode is enabled (data/events/admin)

OIDC-capable implementations can return `auth.type="oidc_device"` plus `issuer`, `client_id`, and `exchange_url`.
The CLI treats `oidc_device` as "OIDC interactive login": it uses device-code when the IdP supports it, otherwise authorization-code + PKCE (localhost callback).

Implementations MAY also return `api_base_url` to tell the CLI where the Fluree API is mounted (for example,
when the API is hosted under `/v1/fluree` or on a separate `data` subdomain).

See [Auth contract (CLI ↔ Server)](../design/auth-contract.md) for the full schema and behavior.

### GET /v1/fluree/info/<ledger...>

Get detailed ledger metadata:

```bash
curl "http://localhost:8090/v1/fluree/info/mydb:main"
```

**Minimum fields used by the Fluree CLI:**

- `t` (required)
- `commitId` (required for `fluree push` when `t > 0`)

**Optional query params:**

- By default, `ledger-info` returns the **full novelty-aware** stats view, including real-time datatype details and class ref edges.
- **`realtime_property_details=false`**: switch `ledger-info` to the lighter fast novelty-aware stats layer that keeps counts current but skips lookup-backed class/ref enrichment.
- **`include_property_datatypes=false`**: omit `stats.properties[*].datatypes` when you want a smaller payload.
- **`include_property_estimates=true`**: include index-derived `ndv-values`, `ndv-subjects`, and selectivity fields under `stats.properties[*]`.

Example:

```bash
curl "http://localhost:8090/v1/fluree/info/mydb:main"
```

**Response:**
```json
{
  "ledger": "mydb:main",
  "t": 150,
  "commitId": "bafybeig...commitT150",
  "indexId": "bafybeig...indexRootT145",
  "commit": {
    "commit_id": "bafybeig...commitT150",
    "t": 150
  },
  "index": {
    "id": "bafybeig...indexRootT145",
    "t": 145
  },
  "stats": {
    "flakes": 12345,
    "size": 1048576,
    "indexed": 145,
    "properties": {
      "ex:name": {
        "count": 3,
        "last-modified-t": 150
      }
    },
    "classes": {
      "ex:Person": {
        "count": 2,
        "properties": {
          "ex:worksFor": {
            "count": 2,
            "refs": { "ex:Organization": 2 },
            "ref-classes": { "ex:Organization": 2 }
          },
          "ex:name": {}
        },
        "property-list": ["ex:name", "ex:worksFor"]
      }
    }
  }
}
```

#### Stats freshness (real-time vs indexed)

- **Real-time (includes novelty)**:
  - `commit` and top-level `t` reflect the latest committed head.
  - `stats.flakes` and `stats.size` are derived from the current ledger stats view (indexed + novelty deltas).
  - `stats.classes[*].properties` / `property-list` will include properties introduced in novelty, even when the update does not restate `@type`.
  - `stats.properties[*].datatypes` is real-time by default.
  - `stats.classes[*].properties[*].refs` is real-time by default.

- **As-of last index**:
  - `stats.indexed` is the last index \(t\). If `commit.t > indexed`, the index is behind the head.
  - NDV-related fields in `stats.properties[*]` (`ndv-values`, `ndv-subjects`) and selectivity derived from them are only as current as the last index refresh, so they are omitted by default and only included when `include_property_estimates=true`.
  - `stats.properties[*].datatypes` are omitted only when `include_property_datatypes=false` is requested.
  - Class property ref-edge counts (`stats.classes[*].properties[*].refs`) fall back to the lighter indexed/fast path only when `realtime_property_details=false` is requested.

### GET /v1/fluree/exists/<ledger...>

Check if a ledger exists:

```bash
curl "http://localhost:8090/v1/fluree/exists/mydb:main"
```

**Response:**
```json
{
  "ledger": "mydb:main",
  "exists": true
}
```

This is a lightweight check that only queries the nameservice without loading the ledger.

## Administrative Operations

### POST /v1/fluree/create

Create a new ledger:

```bash
curl -X POST http://localhost:8090/v1/fluree/create \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb:main"}'
```

**Response (201 Created):**
```json
{
  "ledger": "mydb:main",
  "t": 0,
  "tx-id": "fluree:tx:sha256:abc123...",
  "commit": {
    "commit_id": "bafybeig...commitT0"
  }
}
```

**Authentication:** When `--admin-auth-mode=required`, requires Bearer token from a trusted issuer.

See [Admin Authentication](../api/endpoints.md#admin-authentication) for details.

### POST /v1/fluree/drop

Drop an **entire ledger** (every branch under the name, plus `@shared/dicts/` in hard mode) or, as a fallback, a graph source with the same name:

```bash
# Soft drop the whole "mydb" ledger (retract every branch)
curl -X POST http://localhost:8090/v1/fluree/drop \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb"}'

# Hard drop (delete every branch's artifacts + @shared/dicts - IRREVERSIBLE)
curl -X POST http://localhost:8090/v1/fluree/drop \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb", "hard": true}'
```

`ledger` accepts the bare ledger name. Any branch-qualified form
(including `"mydb:main"`) is **rejected** — use
[`POST /drop-branch`](../api/endpoints.md#post-drop-branch) to drop a
single branch.

**Response:**
```json
{
  "ledger_id": "mydb",
  "status": "dropped",
  "files_deleted": 73,
  "branches_dropped": ["mydb:feature-x", "mydb:dev", "mydb:main"]
}
```

| Status | Description |
|--------|-------------|
| `dropped` | Successfully dropped (aggregate across branches) |
| `already_retracted` | Every branch was previously retracted |
| `not_found` | No nameservice record exists for the name |

**Authentication:** When `--admin-auth-mode=required`, requires Bearer token from a trusted issuer.

**Drop Modes:**
- **Soft** (default): Marks every branch retracted in the nameservice; artifacts remain and aliases stay reserved.
- **Hard**: Deletes per-branch storage artifacts and `@shared/dicts/`, then purges the nameservice records so the ledger name can be reused. Branches are dropped leaf-first; on a per-branch nameservice failure the operation aborts with `500` before touching parents or shared cleanup (retry is safe — each step is idempotent).

If no nameservice record is found for the name, the endpoint tries the same name as a graph source on branch `main`. Truly orphaned storage with no nameservice pointer is **not** swept by `/drop`.

See [Dropping Ledgers](../getting-started/rust-api.md#dropping-ledgers) and [`POST /drop`](../api/endpoints.md#post-drop) for the full reference.

## API Specification

### GET /swagger.json

OpenAPI specification:

```bash
curl http://localhost:8090/swagger.json
```

Returns the OpenAPI 3.0 specification for the server API.

## Monitoring Best Practices

### 1. Use Health Checks

Configure your infrastructure to poll `/health`:

```bash
# Simple monitoring script
while true; do
  curl -sf http://localhost:8090/health > /dev/null || echo "ALERT: Server unhealthy"
  sleep 10
done
```

### 2. Track Server Stats

Periodically collect statistics:

```bash
curl http://localhost:8090/v1/fluree/stats | jq .
```

Key metrics to track:
- `uptime_secs`: Detect restarts
- `cached_ledgers`: Cache efficiency

### 3. Monitor Ledger Health

For each critical ledger:

```bash
curl "http://localhost:8090/v1/fluree/info/mydb:main" | jq .
```

Watch for:
- Index lag (`commit.t` vs `index.t`)
- Unexpected state changes

### 4. Set Up Alerts

Alert conditions:
- Health check failures
- Server restarts (low uptime)
- High index lag

### 5. Log Analysis

Enable structured logging:

```bash
fluree-server --log-level info 2>&1 | jq .
```

Search for:
- `level: "error"` - Errors
- `level: "warn"` - Warnings
- Slow query patterns

## Security Considerations

### Protect Admin Endpoints

In production, enable admin authentication:

```bash
fluree-server \
  --admin-auth-mode required \
  --admin-auth-trusted-issuer did:key:z6Mk...
```

This protects `/v1/fluree/create`, `/v1/fluree/drop`, and other admin-protected
API routes from unauthorized access.

### Limit Endpoint Exposure

Consider network-level restrictions:
- Health endpoint: Available to load balancers
- Stats endpoint: Internal monitoring only
- Admin endpoints: Restricted access

### Audit Logging

Admin operations are logged. Monitor for:
- Ledger creation
- Ledger drops
- Authentication failures

## Related Documentation

- [Configuration](configuration.md) - Server configuration options
- [Query Peers](query-peers.md) - Distributed deployment
- [Telemetry](telemetry.md) - Logging configuration
- [API Endpoints](../api/endpoints.md) - Full endpoint reference


---

# troubleshooting/README.md

# Troubleshooting

This section helps you diagnose and resolve common issues with Fluree deployments.

## Troubleshooting Guides

### [Common Errors](common-errors.md)

Reference for frequently encountered errors:
- Ledger not found
- Invalid IRI errors
- Transaction failures
- Query timeouts
- Permission errors
- Storage issues
- Indexing problems

### [Debugging Queries](debugging-queries.md)

Tools and techniques for query debugging:
- Using EXPLAIN plans
- Query tracing
- Performance profiling
- Identifying slow queries
- Optimizing query patterns

## Quick Diagnostics

### Health Check

First step for any issue:

```bash
curl http://localhost:8090/health
```

Check for unhealthy components.

### Server Status

Check overall server state:

```bash
curl http://localhost:8090/v1/fluree/stats
```

Look for:
- High error counts
- Active queries/transactions stuck
- High indexing lag
- Memory issues

### Logs

Check server logs:

```bash
# Recent errors
tail -f /var/log/fluree/server.log | grep ERROR

# Recent warnings
tail -f /var/log/fluree/server.log | grep WARN
```

## Common Issue Categories

### Connection Issues

**Symptoms:**
- Cannot connect to server
- Connection refused
- Connection timeout

**Common Causes:**
- Server not running
- Wrong port
- Firewall blocking
- Network issues

**Quick Checks:**
```bash
# Is server running?
ps aux | grep fluree-db-server

# Is port listening?
netstat -an | grep 8090

# Can you reach it?
curl http://localhost:8090/health
```

### Query Issues

**Symptoms:**
- Queries return no results
- Queries timeout
- Unexpected results
- Query errors

**Quick Checks:**
```bash
# Enable explain
curl -X POST http://localhost:8090/v1/fluree/explain \
  -d '{...}'

# Check server stats
curl http://localhost:8090/v1/fluree/stats
```

See [Debugging Queries](debugging-queries.md).

### Transaction Issues

**Symptoms:**
- Transactions fail
- Validation errors
- Policy denials
- Slow commits

**Quick Checks:**
```bash
# Validate JSON-LD
# Use online validator: json-ld.org/playground

# Check permissions
curl -X POST http://localhost:8090/v1/fluree/update?dryRun=true \
  -d '{...}'

# Check server stats
curl http://localhost:8090/v1/fluree/stats
```

### Performance Issues

**Symptoms:**
- Slow queries
- Slow transactions
- High latency
- Timeouts

**Quick Checks:**
```bash
# Check indexing lag
curl http://localhost:8090/v1/fluree/info/mydb:main | jq '.t - .index.t'

# Check resource usage
curl http://localhost:8090/v1/fluree/stats

# Check active operations
curl http://localhost:8090/v1/fluree/stats
```

### Storage Issues

**Symptoms:**
- Cannot write data
- Storage errors
- Disk full
- AWS errors

**Quick Checks:**
```bash
# Check disk space
df -h /var/lib/fluree

# Check AWS connectivity
aws s3 ls s3://fluree-prod-data/

# Check server stats
curl http://localhost:8090/v1/fluree/stats
```

## Error Code Reference

See [Common Errors](common-errors.md) for complete error code reference.

**Most Common:**
- `LEDGER_NOT_FOUND` - Ledger doesn't exist
- `PARSE_ERROR` - Invalid JSON-LD or SPARQL
- `INVALID_IRI` - Malformed IRI
- `QUERY_TIMEOUT` - Query took too long
- `POLICY_DENIED` - Not authorized

## Diagnostic Tools

### Enable Debug Logging

```bash
./fluree-db-server --log-level debug
```

Runtime log-level changes are not currently exposed through the standalone HTTP
API; restart with the desired `--log-level` or `RUST_LOG`.

### Enable Query Tracing

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "X-Fluree-Trace: true" \
  -d '{...}'
```

### Enable Policy Tracing

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "X-Fluree-Policy-Trace: true" \
  -d '{...}'
```

### Get Query Plan

```bash
curl -X POST http://localhost:8090/v1/fluree/explain \
  -d '{...}'
```

## Getting Help

### Diagnostic Information to Collect

When reporting issues, include:

1. **Server version:**
   ```bash
   curl http://localhost:8090/health
   ```

2. **Configuration:**
   ```bash
   ./fluree-db-server --help
   # Include relevant config values
   ```

3. **Error messages:**
   - Complete error response
   - Relevant log entries

4. **Reproduction steps:**
   - Minimal example to reproduce
   - Sample data if needed

5. **Environment:**
   - OS and version
   - Storage mode
   - Available resources (RAM, disk)

### Log Collection

Collect diagnostic logs:

```bash
# Last 1000 lines
tail -n 1000 /var/log/fluree/server.log > fluree-diagnostic.log

# Specific time range
grep "2024-01-22T10:" /var/log/fluree/server.log > issue-logs.log
```

## Best Practices

### 1. Check Logs First

Always check logs before deeper investigation:

```bash
tail -f /var/log/fluree/server.log
```

### 2. Start with Health Check

```bash
curl http://localhost:8090/health
```

### 3. Isolate the Issue

Test components independently:
- Can you connect?
- Can you query?
- Can you transact?

### 4. Use Debug Mode Carefully

Debug logging is verbose:
- Use temporarily
- Disable in production
- May impact performance

### 5. Test on Development

Reproduce on development environment before investigating production.

### 6. Keep Logs

Retain logs for historical analysis:

```bash
# Logrotate config
/var/log/fluree/*.log {
    daily
    rotate 30
    compress
}
```

## Related Documentation

- [Common Errors](common-errors.md) - Error reference
- [Debugging Queries](debugging-queries.md) - Query debugging
- [API Errors](../api/errors.md) - HTTP error codes
- [Operations](../operations/README.md) - Operational guides


---

# troubleshooting/common-errors.md

# Common Errors

This document provides solutions for the most frequently encountered Fluree errors.

## LEDGER_NOT_FOUND

```json
{
  "error": "NotFound",
  "message": "Ledger not found: mydb:main",
  "code": "LEDGER_NOT_FOUND"
}
```

### Causes

1. Ledger doesn't exist
2. Typo in ledger name
3. Wrong branch name
4. Nameservice not initialized

### Solutions

**Check ledger exists:**
```bash
curl http://localhost:8090/v1/fluree/ledgers
```

**Create ledger:**
```bash
curl -X POST "http://localhost:8090/v1/fluree/create" \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb:main"}'
```

**Verify spelling:**
- Check for typos in ledger name
- Verify branch name (default is `main`)
- Check case sensitivity

## PARSE_ERROR

```json
{
  "error": "ParseError",
  "message": "Invalid JSON-LD: unexpected token at line 5",
  "code": "PARSE_ERROR",
  "details": {
    "line": 5,
    "column": 12
  }
}
```

### Causes

1. Invalid JSON syntax
2. Invalid JSON-LD structure
3. Invalid SPARQL syntax
4. Missing required fields

### Solutions

**Validate JSON:**
```bash
# Use jq to validate
cat query.json | jq .
```

**Check JSON-LD:**
- Validate @context format
- Check @id and @type values
- Verify array vs object usage

**Check SPARQL:**
- Validate syntax online
- Check PREFIX declarations
- Verify quote matching

**Common JSON Mistakes:**
```json
// Bad: trailing comma
{
  "select": ["?name"],
  "where": [...],
}

// Good: no trailing comma
{
  "select": ["?name"],
  "where": [...]
}
```

## INVALID_IRI

```json
{
  "error": "ValidationError",
  "message": "Invalid IRI: not a valid URI",
  "code": "INVALID_IRI",
  "details": {
    "iri": "not a uri"
  }
}
```

### Causes

1. Malformed IRI
2. Missing namespace prefix
3. Invalid characters
4. Spaces in IRI

### Solutions

**Use valid IRIs:**
```json
// Good
{"@id": "http://example.org/alice"}
{"@id": "ex:alice"}

// Bad
{"@id": "not a uri"}
{"@id": "alice"}  // Missing namespace
{"@id": "ex:alice smith"}  // Space
```

**Define namespace:**
```json
{
  "@context": {
    "ex": "http://example.org/ns/"
  },
  "@graph": [
    {"@id": "ex:alice"}  // Now valid
  ]
}
```

**URL encode spaces:**
```json
{"@id": "ex:alice%20smith"}
```

## UNRESOLVED_COMPACT_IRI

```text
Unresolved compact IRI 'ex:Person': prefix 'ex' is not defined in @context.
If this is intended as an absolute IRI, use a full form (e.g. http://...)
or add the prefix to @context.
```

This error fires from the JSON-LD strict compact-IRI guard. A value that *looks* like a compact IRI (`prefix:suffix`) appeared in an IRI position, but `prefix` is not defined in `@context` and is not a recognised absolute scheme.

### Causes

1. Forgotten `@context` on a query or transaction
2. Misspelled or missing prefix in `@context`
3. Intentionally using a bare `prefix:suffix` string as an opaque identifier

### Solutions

**Add the missing prefix to @context** (most common fix):
```json
{
  "@context": {"ex": "http://example.org/ns/"},
  "@graph": [{"@id": "ex:alice", "ex:name": "Alice"}]
}
```

**Use a full absolute IRI** instead of the compact form:
```json
{
  "@graph": [
    {"@id": "http://example.org/ns/alice", "http://example.org/ns/name": "Alice"}
  ]
}
```

**Opt out of the guard** for legacy data where bare `prefix:suffix` strings are intentional:
```json
{
  "@context": {"ex": "http://example.org/ns/"},
  "opts": {"strictCompactIri": false},
  "@graph": [{"@id": "legacy:alice", "ex:name": "Alice"}]
}
```

The opt-out applies to both queries and transactions. See [IRIs and @context — Strict Compact-IRI Guard](../concepts/iri-and-context.md#strict-compact-iri-guard) for the full policy.

## QUERY_TIMEOUT

```json
{
  "error": "Timeout",
  "message": "Query execution exceeded timeout of 30000ms",
  "code": "QUERY_TIMEOUT",
  "details": {
    "timeout_ms": 30000,
    "elapsed_ms": 31245
  }
}
```

### Causes

1. Complex query
2. Large result set
3. High indexing lag
4. Insufficient resources

### Solutions

**Add LIMIT:**
```json
{
  "select": ["?name"],
  "where": [...],
  "limit": 100  // Add limit
}
```

**Add filters:**
```json
{
  "where": [...],
  "filter": "?age > 18"  // Reduce result set
}
```

**Check indexing lag:**
```bash
curl http://localhost:8090/v1/fluree/info/mydb:main
# If (t - index.t) is large, wait for indexing (or reduce write rate)
```

**Simplify query:**
- Break into smaller queries
- Remove unnecessary joins
- Use more specific patterns

**Increase timeout:**
```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "X-Fluree-Timeout: 60000" \
  -d '{...}'
```

## POLICY_DENIED

```json
{
  "error": "Forbidden",
  "message": "Policy denies access to ledger mydb:main",
  "code": "POLICY_DENIED",
  "details": {
    "subject": "did:key:z6Mkh...",
    "action": "query",
    "ledger": "mydb:main"
  }
}
```

### Causes

1. No permission for operation
2. Missing authentication
3. Policy misconfiguration
4. Wrong DID/identity

### Solutions

**Check authentication:**
```bash
# Are you sending credentials?
curl -H "Authorization: Bearer token" ...
```

**Verify policy:**
```sparql
# Query policies
SELECT ?policy ?subject ?action ?allow
WHERE {
  ?policy a f:Policy .
  ?policy f:subject ?subject .
  ?policy f:action ?action .
  ?policy f:allow ?allow .
}
```

**Test with policy trace:**
```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "X-Fluree-Policy-Trace: true" \
  -d '{...}'
```

**Check DID:**
- Verify DID in signed request
- Check DID is registered
- Verify public key

## TYPE_ERROR

```json
{
  "error": "TypeError",
  "message": "Expected integer, got string",
  "code": "TYPE_ERROR",
  "details": {
    "expected": "xsd:integer",
    "actual": "xsd:string",
    "value": "not a number"
  }
}
```

### Causes

1. Wrong datatype
2. Type mismatch in comparison
3. Invalid type conversion

### Solutions

**Use correct types:**
```json
// Good
{"ex:age": 30}
{"ex:age": {"@value": "30", "@type": "xsd:integer"}}

// Bad
{"ex:age": "30"}  // String, not integer
```

**Check type constraints:**
- Verify expected types
- Use explicit @type
- Validate before submitting

## PAYLOAD_TOO_LARGE

```json
{
  "error": "PayloadTooLarge",
  "message": "Transaction exceeds maximum size of 10485760 bytes",
  "code": "PAYLOAD_TOO_LARGE",
  "details": {
    "max_size": 10485760,
    "actual_size": 15000000
  }
}
```

### Causes

1. Transaction too large
2. Query result too large
3. Large embedded data

### Solutions

**Batch large transactions:**
```javascript
const batchSize = 1000;
for (let i = 0; i < entities.length; i += batchSize) {
  const batch = entities.slice(i, i + batchSize);
  await transact({"@graph": batch});
}
```

**Use LIMIT for queries:**
```json
{
  "select": ["?name"],
  "where": [...],
  "limit": 1000  // Paginate
}
```

**Increase limits (if appropriate):**
```bash
./fluree-db-server --max-transaction-size 20971520
```

## STORAGE_ERROR

```json
{
  "error": "StorageError",
  "message": "Cannot write to storage",
  "code": "STORAGE_ERROR"
}
```

### Causes

1. Disk full (file storage)
2. Permission errors
3. AWS connectivity (AWS storage)
4. Storage backend down

### Solutions

**File Storage:**
```bash
# Check disk space
df -h /var/lib/fluree

# Check permissions
ls -la /var/lib/fluree
sudo chown -R fluree:fluree /var/lib/fluree
```

**AWS Storage:**
```bash
# Check AWS credentials
aws sts get-caller-identity

# Check S3 access
aws s3 ls s3://fluree-prod-data/

# Check DynamoDB
aws dynamodb describe-table --table-name fluree-nameservice
```

## HIGH_INDEXING_LAG

Not an error, but a warning condition.

### Symptoms

```bash
curl http://localhost:8090/v1/fluree/info/mydb:main
```

```json
{
  "commit_t": 150,
  "index_t": 0
}
```

### Causes

1. Transaction rate exceeds indexing capacity
2. Large transactions
3. Insufficient resources
4. Storage bottleneck

### Solutions

**Tune indexing:**
```bash
fluree-server \
  --indexing-enabled \
  --reindex-min-bytes 100000 \
  --reindex-max-bytes 1000000
```

**Reduce transaction rate:**
```javascript
// Add delay between transactions
await transact(data);
await sleep(100);
```

**Wait for catchup:**
```javascript
async function waitForIndexing() {
  while (true) {
    const status = await getStatus();
    const lag = status.commit_t - status.index_t;
    if (lag < 10) break;
    await sleep(1000);
  }
}
```

**Add resources:**
- More CPU
- More memory
- Faster disk

## CONCURRENT_MODIFICATION

```json
{
  "error": "Conflict",
  "message": "Concurrent modification detected",
  "code": "CONCURRENT_MODIFICATION"
}
```

### Causes

1. Multiple processes updating same entity
2. Nameservice contention
3. Race condition

### Solutions

**Implement retry:**
```javascript
async function transactWithRetry(data, maxRetries = 3) {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await transact(data);
    } catch (err) {
      if (err.code === 'CONCURRENT_MODIFICATION' && i < maxRetries - 1) {
        await sleep(Math.pow(2, i) * 100);
        continue;
      }
      throw err;
    }
  }
}
```

**Use upsert for retry-friendly transactions:**
```bash
# Upsert is more retry-friendly for idempotent entity transactions
POST /upsert?ledger=mydb:main
```

## SIGNATURE_VERIFICATION_FAILED

```json
{
  "error": "SignatureVerificationFailed",
  "message": "Invalid signature",
  "code": "INVALID_SIGNATURE"
}
```

### Causes

1. Wrong private key
2. Payload modified after signing
3. Incorrect algorithm
4. Key not registered

### Solutions

**Verify signing process:**
```javascript
// Ensure payload not modified
const payload = JSON.stringify(transaction);
const jws = await sign(payload, privateKey);
// Don't modify payload after signing
```

**Check algorithm:**
```json
{
  "alg": "EdDSA",  // Must match key type
  "kid": "did:key:z6Mkh..."
}
```

**Verify public-key material:** standalone server signed requests use the key
material embedded in supported JWS/JWT headers (or configured OIDC JWKS). There
is no `/admin/keys` registration endpoint.

## Memory Issues

### Symptoms

- Out of memory errors
- Server crashes
- Slow performance
- Swap usage

### Solutions

**Check memory:**
```bash
curl http://localhost:8090/v1/fluree/stats
```

**Reduce memory usage:**
```bash
# See docs/operations/configuration.md for current memory-related flags.
# In general: reduce write/query load, reduce indexing lag, and provision more RAM.
```

**Add more RAM:**
- Upgrade server
- Use cloud instance with more memory

**Reduce novelty:**
- Index more frequently
- Reduce transaction size

## Troubleshooting Checklist

When encountering issues, check:

1. [ ] Server is running
2. [ ] Can connect to server
3. [ ] Health endpoint returns healthy
4. [ ] Logs show no errors
5. [ ] Ledger exists
6. [ ] Correct ledger name/branch
7. [ ] Valid JSON-LD/SPARQL syntax
8. [ ] Sufficient resources (disk, memory)
9. [ ] No network issues
10. [ ] Authentication working (if required)

## Related Documentation

- [Debugging Queries](debugging-queries.md) - Query-specific debugging
- [API Errors](../api/errors.md) - HTTP error reference
- [Operations](../operations/README.md) - Operational guides
- [Telemetry](../operations/telemetry.md) - Monitoring and logging


---

# troubleshooting/debugging-queries.md

# Debugging Queries

This guide provides tools and techniques for debugging query performance and correctness issues in Fluree.

## Query Explain Plans

### Enable Explain

Get query execution plan:

```bash
curl -X POST http://localhost:8090/v1/fluree/explain \
  -H "Content-Type: application/json" \
  -d '{
    "from": "mydb:main",
    "select": ["?name", "?age"],
    "where": [
      { "@id": "?person", "schema:name": "?name" },
      { "@id": "?person", "schema:age": "?age" }
    ],
    "filter": "?age > 25"
  }'
```

**Response:**
```json
{
  "plan": {
    "type": "join",
    "left": {
      "type": "scan",
      "index": "POST",
      "predicate": "schema:name",
      "estimated_rows": 1000
    },
    "right": {
      "type": "scan",
      "index": "POST",
      "predicate": "schema:age",
      "estimated_rows": 1000
    },
    "join_variable": "?person",
    "filter": {
      "expression": "?age > 25",
      "selectivity": 0.6
    },
    "estimated_result_rows": 600
  },
  "execution": {
    "duration_ms": 45,
    "rows_scanned": 2000,
    "rows_returned": 573,
    "index_hits": 2000,
    "filter_applications": 1000
  }
}
```

### Understanding Explain Output

**Scan Operations:**
- Which index used (SPOT, POST, OPST, PSOT)
- Estimated rows
- Actual rows scanned

**Join Operations:**
- Join type (hash, merge, nested loop)
- Join variable
- Join order

**Filter Operations:**
- Filter expression
- Estimated selectivity
- Rows filtered

**Execution Stats:**
- Total duration
- Rows scanned vs returned
- Index efficiency

## Query Tracing

### Enable Tracing

Get detailed execution trace:

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "X-Fluree-Trace: true" \
  -d '{...}'
```

**Response:**
```json
{
  "results": [...],
  "trace": {
    "total_duration_ms": 45,
    "phases": [
      {
        "phase": "parse",
        "duration_ms": 2
      },
      {
        "phase": "plan",
        "duration_ms": 3
      },
      {
        "phase": "execute",
        "duration_ms": 38,
        "steps": [
          {
            "step": "scan_POST_schema:name",
            "duration_ms": 12,
            "rows": 1000
          },
          {
            "step": "scan_POST_schema:age",
            "duration_ms": 15,
            "rows": 1000
          },
          {
            "step": "join",
            "duration_ms": 8,
            "rows": 1000
          },
          {
            "step": "filter",
            "duration_ms": 3,
            "rows_in": 1000,
            "rows_out": 573
          }
        ]
      },
      {
        "phase": "serialize",
        "duration_ms": 2
      }
    ]
  }
}
```

### Trace Analysis

Look for:
- **Slow phases:** Which phase takes longest?
- **Excessive scans:** Too many rows scanned?
- **Inefficient joins:** Large intermediate results?
- **Ineffective filters:** Filters applied too late?

## Common Query Issues

### No Results

**Symptom:** Query returns empty array

**Debugging Steps:**

1. **Check data exists:**
   ```sparql
   SELECT (COUNT(*) as ?count)
   WHERE { ?s ?p ?o }
   ```

2. **Test each pattern separately:**
   ```json
   // Test pattern 1
   {"select": ["?person"], "where": [{"@id": "?person", "schema:name": "?name"}]}
   
   // Test pattern 2
   {"select": ["?person"], "where": [{"@id": "?person", "schema:age": "?age"}]}
   ```

3. **Check IRI matching:**
   ```json
   // Query with full IRI
   {"@id": "http://example.org/ns/alice"}
   
   // Or with prefix
   {"@id": "ex:alice"}
   ```

4. **Verify time specifier:**
   ```bash
   # Current data
   "from": "mydb:main"
   
   # Historical might be empty
   "from": "mydb:main@t:1"
   ```

### Unexpected Results

**Symptom:** Results don't match expectations

**Debugging Steps:**

1. **Check each variable:**
   ```json
   {
     "select": ["?person", "?name", "?age"],  // See all bindings
     "where": [...]
   }
   ```

2. **Verify types:**
   ```sparql
   SELECT ?person ?name (DATATYPE(?name) as ?nameType)
   WHERE {
     ?person schema:name ?name
   }
   ```

3. **Check for duplicates:**
   ```sparql
   SELECT ?person (COUNT(?name) as ?count)
   WHERE {
     ?person schema:name ?name
   }
   GROUP BY ?person
   HAVING (?count > 1)
   ```

4. **Test without filters:**
   ```json
   // Remove filter temporarily
   {"where": [...] }  // No filter
   ```

### Slow Queries

**Symptom:** Query takes too long

**Debugging Steps:**

1. **Check explain plan:**
   ```bash
   curl -X POST http://localhost:8090/v1/fluree/explain -d '{...}'
   ```

2. **Check indexing lag:**
   ```bash
   curl http://localhost:8090/v1/fluree/info/mydb:main
   # High indexing lag (commit_t - index_t) can slow queries
   ```

3. **Add LIMIT:**
   ```json
   {"where": [...], "limit": 100}
   ```

4. **Check pattern specificity:**
   ```json
   // Specific (fast)
   {"@id": "ex:alice", "schema:name": "?name"}
   
   // General (slow)
   {"@id": "?entity", "?pred": "?value"}
   ```

5. **Verify index usage:**
   - Subject-based patterns use SPOT (fast)
   - Broad patterns may scan many triples (slow)

## Query Optimization

### Automatic Pattern Reordering

The query planner automatically reorders WHERE-clause patterns for optimal
join order. You do not need to manually order patterns from most to least
selective — the planner does this for you using a greedy algorithm that
considers cardinality estimates and which variables are already bound at each
step.

When database statistics are available (after at least one indexing cycle),
estimates use HLL-derived property counts and distinct-value counts.
Without statistics, the planner falls back to heuristic constants. You can
verify the planner's decisions using explain plans (see
[Explain Plans](../query/explain.md)).

Both of these queries produce the same execution plan:

```json
{
  "where": [
    {"@id": "?company", "schema:name": "?companyName"},
    {"@id": "?person", "schema:worksFor": "?company"},
    {"@id": "ex:alice", "schema:name": "?name"}
  ]
}
```

```json
{
  "where": [
    {"@id": "ex:alice", "schema:name": "?name"},
    {"@id": "ex:alice", "schema:worksFor": "?company"},
    {"@id": "?company", "schema:name": "?companyName"}
  ]
}
```

The planner recognizes that `ex:alice` patterns are highly selective (bound
subject), and that `?company` becomes bound after those patterns execute,
making the final pattern a cheap per-subject lookup rather than a full scan.

### Filter and BIND Placement

Filters and BINDs are placed during the greedy reordering loop, as soon as all
their input variables are bound. You do not need to manually position them for
efficiency. For BIND patterns, only the expression's input variables must be
bound — the target variable is an output that feeds back into the bound set,
enabling cascading placement of dependent patterns.

When a filter or BIND becomes ready immediately after a compound pattern
(UNION, Graph, or Service), the planner pushes it *into* the compound
pattern's inner lists rather than placing it after. For UNION, the filter is
cloned into every branch. This means filters execute within each branch,
benefiting from optimal placement, range pushdown, and inline evaluation — the
same optimizations available to top-level filters.

Additionally, filters whose variables are all bound by a join operator are
evaluated inline during the join itself, avoiding the overhead of a separate
filter pass. Filters that depend on a BIND's output variable are fused into
the BindOperator and evaluated inline after computing each row's BIND value,
similarly eliminating a separate filter pass. Range-safe filters (comparisons
like `>`, `<` on indexed properties) are pushed down to the index scan.

### Use LIMIT

Always limit large result sets:

```json
{
  "where": [...],
  "orderBy": ["?name"],
  "limit": 100,
  "offset": 0
}
```

Implement pagination for UI.

### Avoid Cartesian Products

Ensure patterns are connected:

**Bad (Cartesian product):**
```json
{
  "where": [
    {"@id": "?person", "schema:name": "?name"},
    {"@id": "?company", "schema:name": "?companyName"}
    // Not connected! Returns person × company combinations
  ]
}
```

**Good (connected):**
```json
{
  "where": [
    {"@id": "?person", "schema:name": "?name"},
    {"@id": "?person", "schema:worksFor": "?company"},
    {"@id": "?company", "schema:name": "?companyName"}
  ]
}
```

## Policy Debugging

### Enable Policy Trace

See which policies apply:

```bash
curl -X POST http://localhost:8090/v1/fluree/query \
  -H "X-Fluree-Policy-Trace: true" \
  -d '{...}'
```

Response:
```json
{
  "results": [...],
  "policy_trace": [
    {
      "policy": "ex:department-policy",
      "matched": true,
      "condition_met": true,
      "decision": "allow",
      "patterns_added": [
        {"@id": "?person", "ex:department": "engineering"}
      ]
    },
    {
      "policy": "ex:role-policy",
      "matched": false,
      "reason": "subject_mismatch"
    }
  ],
  "final_decision": "allow"
}
```

### Policy Impact on Query

Compare query with and without policies:

```javascript
// With policies (authenticated)
const authResult = await queryWithAuth(query);

// Without policies (admin override)
const fullResult = await queryAsAdmin(query);

console.log(`Auth sees ${authResult.length} rows`);
console.log(`Admin sees ${fullResult.length} rows`);
console.log(`Policy filtered ${fullResult.length - authResult.length} rows`);
```

## Testing Queries

### Isolate Components

Test query components separately:

```javascript
// Test each WHERE pattern
for (const pattern of wherePatterns) {
  const result = await query({
    select: ["?s"],
    where: [pattern]
  });
  console.log(`Pattern ${JSON.stringify(pattern)}: ${result.length} results`);
}
```

### Use Smaller Datasets

Test on small dataset first:

```bash
# Create test ledger
curl -X POST "http://localhost:8090/v1/fluree/insert?ledger=test:main" \
  -d '{"@graph": [small test data]}'

# Test query
curl -X POST http://localhost:8090/v1/fluree/query \
  -d '{"from": "test:main", ...}'
```

### Compare with Expected Results

```javascript
const expected = [
  { name: "Alice", age: 30 },
  { name: "Bob", age: 25 }
];

const actual = await query({...});

assert.deepEqual(actual, expected);
```

## Diagnostic Queries

### Check Index Usage

```sparql
# Count triples per index
SELECT (COUNT(*) as ?count)
WHERE { ?s ?p ?o }
```

### Find Large Entities

```sparql
SELECT ?entity (COUNT(?triple) as ?tripleCount)
WHERE {
  ?entity ?p ?o .
  BIND(?entity AS ?triple)
}
GROUP BY ?entity
ORDER BY DESC(?tripleCount)
LIMIT 10
```

### Find Common Predicates

```sparql
SELECT ?predicate (COUNT(*) as ?count)
WHERE {
  ?s ?predicate ?o
}
GROUP BY ?predicate
ORDER BY DESC(?count)
```

### Check Data Types

```sparql
SELECT ?type (COUNT(*) as ?count)
WHERE {
  ?entity a ?type
}
GROUP BY ?type
ORDER BY DESC(?count)
```

## Performance Profiling

### Measure Query Time

```javascript
const start = Date.now();
const result = await query({...});
const duration = Date.now() - start;

console.log(`Query returned ${result.length} rows in ${duration}ms`);
```

### Identify Bottlenecks

Use trace to find slow operations:

```javascript
const response = await queryWithTrace({...});
const trace = response.trace;

const slowSteps = trace.phases
  .flatMap(p => p.steps || [])
  .filter(s => s.duration_ms > 100)
  .sort((a, b) => b.duration_ms - a.duration_ms);

console.log('Slow steps:', slowSteps);
```

### Compare Approaches

Test different query formulations:

```javascript
// Approach 1
const start1 = Date.now();
const result1 = await query(approach1);
const time1 = Date.now() - start1;

// Approach 2
const start2 = Date.now();
const result2 = await query(approach2);
const time2 = Date.now() - start2;

console.log(`Approach 1: ${time1}ms, Approach 2: ${time2}ms`);
```

## Best Practices

### 1. Use Explain Early

Run explain on new queries:

```bash
curl -X POST http://localhost:8090/v1/fluree/explain -d '{...}'
```

### 2. Test with Representative Data

Test queries with production-like data volume:

```javascript
// Load realistic test data
await loadTestData(10000);  // Similar to production size

// Test query performance
const result = await query({...});
```

### 3. Monitor Query Patterns

Track slow queries:

```javascript
if (duration > 1000) {
  logger.warn(`Slow query: ${duration}ms`, {
    query: queryText,
    resultCount: result.length
  });
}
```

### 4. Profile Before Optimizing

Measure before optimizing:

```javascript
console.time('query');
const result = await query({...});
console.timeEnd('query');
```

### 5. Use Query Logs

Enable query logging:

```toml
[logging]
level = "debug"
log_queries = true
```

## Common Query Antipatterns

### Antipattern 1: Overly Broad Patterns

Bad:
```json
{"@id": "?entity", "?predicate": "?value"}
```

Good:
```json
{"@id": "?person", "@type": "schema:Person"},
{"@id": "?person", "schema:name": "?name"}
```

### Antipattern 2: Disconnected Patterns (Cartesian Products)

Ensure all patterns share at least one variable with the rest of the query.
Disconnected patterns produce a Cartesian product:

Bad:
```json
{
  "where": [
    {"@id": "?person", "schema:name": "?name"},
    {"@id": "?dept", "schema:budget": "?budget"}
  ]
}
```

Good:
```json
{
  "where": [
    {"@id": "?person", "schema:name": "?name"},
    {"@id": "?person", "schema:department": "?dept"},
    {"@id": "?dept", "schema:budget": "?budget"}
  ]
}
```

Note: filter placement is handled automatically by the planner. Filters are
applied as soon as all their referenced variables are bound, regardless of
where they appear in the query.

### Antipattern 3: Missing LIMIT

Bad:
```json
{
  "select": ["?name"],
  "where": [...]  // Could return millions
}
```

Good:
```json
{
  "select": ["?name"],
  "where": [...],
  "limit": 1000  // Always limit
}
```

### Antipattern 4: Redundant Patterns

Bad:
```json
{
  "where": [
    {"@id": "ex:alice", "schema:name": "?name"},
    {"@id": "ex:alice", "schema:name": "Alice"}  // Redundant
  ]
}
```

Good:
```json
{
  "where": [
    {"@id": "ex:alice", "schema:name": "Alice"}
  ]
}
```

## Tools

### Query Validation

Validate before sending:

```javascript
function validateQuery(query) {
  if (!query.select) {
    throw new Error('Missing select clause');
  }
  if (!query.where || query.where.length === 0) {
    throw new Error('Missing where clause');
  }
  if (!query.limit && estimateResultSize(query) > 1000) {
    console.warn('Query missing LIMIT clause');
  }
}
```

### Query Builder

Use query builder for complex queries:

```javascript
const query = new QueryBuilder()
  .from('mydb:main')
  .select('?name', '?age')
  .where('?person', 'schema:name', '?name')
  .where('?person', 'schema:age', '?age')
  .filter('?age > 25')
  .limit(100)
  .build();
```

### Query Templates

Create reusable templates:

```javascript
function findPersonByName(name) {
  return {
    from: 'mydb:main',
    select: ['?person', '?email'],
    where: [
      { '@id': '?person', 'schema:name': name },
      { '@id': '?person', 'schema:email': '?email' }
    ]
  };
}
```

## Related Documentation

- [Common Errors](common-errors.md) - Error reference
- [Explain Plans](../query/explain.md) - Query optimization
- [JSON-LD Query](../query/jsonld-query.md) - Query syntax
- [SPARQL](../query/sparql.md) - SPARQL syntax
- [Telemetry](../operations/telemetry.md) - Logging and monitoring


---

# troubleshooting/performance-tracing.md

# Performance Investigation with Distributed Tracing

Fluree includes deep instrumentation that decomposes every query, transaction, and indexing operation into a span waterfall visible in [Jaeger](https://www.jaegertracing.io/), [Grafana Tempo](https://grafana.com/oss/tempo/), AWS X-Ray, or any OpenTelemetry-compatible backend. This guide explains how to use that instrumentation to find and fix performance bottlenecks.

## When to Use Deep Tracing

| Symptom | Start with | Escalate to |
|---------|-----------|-------------|
| Single slow query | `/v1/fluree/explain` plan | Deep tracing at `debug` level |
| Slow queries in general, unclear which phase | Deep tracing at `debug` level | `trace` level for operator detail |
| Slow transactions / commits | Deep tracing at `debug` level | Check `txn_commit` sub-spans |
| Indexing taking too long | Deep tracing at `debug` level | Check `build_index` per-order timing |
| Intermittent latency spikes | Sustained tracing + Jaeger search by duration | Correlate with indexing traces |
| Production regression | Compare Jaeger traces before/after deploy | Filter by `tracker_time` span attribute |

Deep tracing is complementary to explain plans, not a replacement. Explain plans show the *shape* of a query plan; tracing shows *where wall-clock time actually went*.

## Quick Start: Local Investigation

The `otel/` directory at the repository root provides a self-contained Makefile-driven harness for local trace investigation.

### Prerequisites

- Docker (for Jaeger)
- Rust toolchain
- curl, bash

### One-liner setup

```bash
cd otel/
make all    # starts Jaeger, builds with --features otel, starts server, runs smoke tests
make ui     # opens Jaeger UI in browser
```

This gives you a running Fluree server exporting traces to a local Jaeger instance with pre-loaded test data.

### Investigate a specific query

Once the server is running (via `make server` or `make all`):

```bash
# Run your problematic query against the server
curl -s -X POST http://localhost:8090/v1/fluree/query/otel-test:main \
  -H 'Content-Type: application/json' \
  -d '{
    "select": ["?name", "?price"],
    "where": [
      {"@id": "?p", "@type": "ex:Product"},
      {"@id": "?p", "ex:name": "?name"},
      {"@id": "?p", "ex:price": "?price"}
    ],
    "orderBy": [{"desc": "?price"}],
    "limit": 100
  }'
```

Then open Jaeger (`make ui` or `http://localhost:16686`), select service `fluree-server`, and find the trace. The waterfall shows exactly where time was spent.

### Teardown

```bash
make clean-all   # stops server, stops Jaeger, removes data
```

## Writing Custom Scenario Scripts

The `otel/scripts/` directory contains scenario scripts you can use as templates. To investigate a specific performance issue:

### 1. Create a scenario script

```bash
#!/usr/bin/env bash
# otel/scripts/my-investigation.sh
set -euo pipefail

PORT="${PORT:-8090}"
LEDGER="${LEDGER:-otel-test:main}"
BASE="http://localhost:${PORT}"

echo "=== My investigation scenario ==="

# Step 1: Insert data that triggers the problem
curl -sf -X POST "${BASE}/${LEDGER}/insert" \
  -H 'Content-Type: application/json' \
  -d '{
    "@context": {"ex": "http://example.org/ns/"},
    "@graph": [
      ... your test data ...
    ]
  }' > /dev/null

sleep 0.5  # let OTEL batch exporter flush

# Step 2: Run the problematic query multiple times
for i in $(seq 1 5); do
  echo "  Query iteration $i..."
  curl -sf -X POST "${BASE}/${LEDGER}/query" \
    -H 'Content-Type: application/json' \
    -d '{ ... your query ... }' > /dev/null
  sleep 0.3
done

echo "=== Done. Check Jaeger for traces. ==="
```

### 2. Run it

```bash
cd otel/
make up build server          # ensure infrastructure is running
bash scripts/my-investigation.sh
make ui                        # inspect traces
```

### 3. Add a Makefile target (optional)

```makefile
# In otel/Makefile
my-investigation: _data/storage
	bash scripts/my-investigation.sh
```

### Tips for effective scenario scripts

- **Pause between requests** (`sleep 0.3-0.5`) to let the OTEL batch exporter flush. Without this, spans from adjacent requests may interleave in Jaeger, making waterfall analysis harder.
- **Run the query multiple times** to see variance. Sort by duration in Jaeger to find the worst case.
- **Use different `RUST_LOG` levels** for different investigations. Override when starting the server: `make server RUST_LOG=info,fluree_db_query=trace`
- **Isolate variables**: test with and without indexing (`INDEXING=false`), with different data volumes, or with different query patterns.

## Reading Jaeger Waterfalls

### Anatomy of a query trace

```
request (info)                              ─────────────────────────── 834ms
  query_execute (debug)                     ─────────────────────────── 832ms
    query_prepare (debug)                   ──── 12ms
      reasoning_prep (debug)                ── 3ms
      pattern_rewrite (debug)               ── 2ms
      plan (debug)                          ── 5ms
    query_run (debug)                       ──────────────────────── 818ms
      scan (debug)                          ── 4ms
      join (debug)                          ─────────────────── 780ms
        join_flush_scan_spot (debug)        ────────────────── 775ms
      filter (debug)                        ── 2ms
      sort (debug)                          ── 15ms
        sort_blocking (debug)               ── 14ms
      project (debug)                       ── 1ms
    format (debug)                          ── 2ms
```

In this example, the bottleneck is immediately visible: the `join_flush_scan_spot` span accounts for 775ms of the 834ms total. This tells you the query is doing a large range scan during the join phase.

### Key span attributes to check

| Span | Attribute | What it tells you |
|------|-----------|-------------------|
| `query_execute` | `tracker_time`, `tracker_fuel` | Total tracked time and fuel consumption |
| `pattern_rewrite` | `patterns_before`, `patterns_after` | Whether pattern rewriting is effective |
| `plan` | `pattern_count` | Complexity of the query plan |
| `scan` | (trace level) | How long individual scans take |
| `join_flush_scan_spot` | `unique_subjects`, `total_leaves` | Join scan size — large values indicate broad scans |
| `sort_blocking` | `input_rows`, `sort_ms` | Sort cost — are you sorting a huge result set? |
| `txn_stage` | `insert_count`, `delete_count` | Transaction size |
| `txn_commit` | `flake_count`, `delta_bytes` | Commit I/O volume |

> **Note:** Span attributes like `tracker_time`, `tracker_fuel`, `patterns_before`, `assertion_count`, `template_count`, and `pattern_count` are verified by acceptance tests (`it_tracing_spans.rs`). Other attributes in this table (`unique_subjects`, `total_leaves`, `sort_ms`, `flake_count`, `delta_bytes`) are documented from code inspection but not programmatically verified — they may drift if span instrumentation is refactored.

### Anatomy of a transaction trace

```
request (info)                              ─────────────── 245ms
  transact_execute (debug)                  ─────────────── 243ms
    txn_stage (debug)                       ────── 45ms
      where_exec (debug)                    ── 8ms
      delete_gen (debug)                    ── 3ms
      insert_gen (debug)                    ── 12ms
      cancellation (debug)                  ── 5ms
      policy_enforce (debug)                ── 2ms
    txn_commit (debug)                      ──────────── 195ms
      commit_nameservice_lookup (debug)     ── 2ms
      commit_verify_sequencing (debug)      ── 1ms
      commit_write_raw_txn (debug)          ── 5ms  (await of spawned upload)
      commit_build_record (debug)           ── 3ms
      commit_write_commit_blob (debug)      ────── 65ms
      commit_publish_nameservice (debug)    ────── 35ms
```

When `store_raw_txn` is opted in, the raw-transaction bytes are uploaded on a Tokio task spawned at the top of the pipeline (see `PendingRawTxnUpload`). `commit_write_raw_txn` then measures just the **await** of that task — usually a few ms, even on S3, because the upload overlapped staging CPU work. If you see `commit_write_raw_txn` approaching the upload's intrinsic latency (50-100ms on S3), staging finished faster than the upload; otherwise the overlap has absorbed it. The bottleneck on S3 is now typically `commit_write_commit_blob` alone.

### Anatomy of an indexing trace

Indexing runs as a **separate trace** (not nested under an HTTP request). Search Jaeger for operation name `index_build`:

If the index build was queued by an HTTP transaction request, use logs to bridge the two views: the background worker now copies the triggering `request_id` and `trace_id` onto its log lines, but the OTEL/Jaeger indexing trace remains separate by design.

```
index_build (debug)                         ─────────────────── 12.5s
  commit_chain_walk (debug)                 ── 50ms
  commit_resolve (debug, commits=2)         ── 27ms
  build_all_indexes (debug)                 ─────────────────── 12.4s
    build_index (debug, order=SPOT)         ──────── 3.1s
    build_index (debug, order=PSOT)         ──────── 3.2s
    build_index (debug, order=POST)         ──────── 3.0s
    build_index (debug, order=OPST)         ──────── 3.1s
```

## Common Bottleneck Patterns

### 1. Join scan too broad

**Symptom:** `join_flush_scan_spot` has `unique_subjects` in the thousands and dominates the waterfall.

**Cause:** A join pattern that matches too many subjects, forcing a large range scan.

**Fix:** Add more selective patterns or filters to narrow the join. Check the explain plan for join order.

### 2. Sort on large result set

**Symptom:** `sort_blocking` shows `input_rows` > 10,000 and `sort_ms` dominates.

**Cause:** Sorting happens after all joins/filters, on the full result set.

**Fix:** Add `LIMIT` if possible, or ensure filters run before the sort by placing restrictive patterns first.

### 3. Commit I/O on S3

**Symptom:** `commit_write_commit_blob` takes 50-200ms. `commit_write_raw_txn` may also show time if staging completed before the parallel upload finished.

**Cause:** S3 PutObject latency (~50-100ms per call). The raw-txn upload is parallelized with staging, so its cost is usually absorbed, but the commit-blob write is serial on the critical path.

**Fix:** S3 latency is inherent. Batch multiple small transactions into fewer larger ones. Consider file storage for latency-sensitive workloads. If `commit_write_raw_txn` is non-trivial, it indicates staging finished faster than the raw-txn upload — the overlap helped but couldn't fully hide it.

### 4. Indexing backlog

**Symptom:** Multiple `index_build` traces in quick succession, each taking 10+ seconds.

**Cause:** Transaction volume exceeds indexing throughput, building up novelty.

**Fix:** Increase the novelty reindex threshold, or reduce transaction frequency. Check `build_index` sub-spans to see which index order is slowest.

### 5. Policy evaluation overhead

**Symptom:** `policy_eval` or `policy_enforce` takes a significant fraction of query/transaction time.

**Cause:** Complex policy rules that require additional queries to evaluate.

**Fix:** Simplify policy rules, or pre-compute policy decisions where possible.

## Controlling Trace Verbosity

### RUST_LOG patterns

| Goal | Pattern | Visible spans |
|------|---------|---------------|
| Production default | `info` | HTTP `request` spans only (zero operation spans) |
| Query investigation | `info,fluree_db_query=debug` | + `query_execute`, `query_prepare`, `query_run`, operators |
| Transaction investigation | `info,fluree_db_transact=debug` | + `txn_stage`, `txn_commit`, commit sub-spans |
| Full debug | `info,fluree_db_query=debug,fluree_db_transact=debug,fluree_db_indexer=debug` | All debug spans |
| Operator-level detail | `info,fluree_db_query=trace` | + per-leaf: `binary_cursor_next_leaf`, etc. |
| Everything | `debug` | Console firehose (OTEL layer still filters to `fluree_*` only) |

**Note:** When OTEL is enabled, all `fluree_*` debug spans flow to the OTEL collector regardless of `RUST_LOG`. The table above describes console output only.

### With the otel/ harness

```bash
# Override RUST_LOG when starting the server
make server RUST_LOG='info,fluree_db_query=trace'

# Then run your scenario
make query
```

### In production

Set `RUST_LOG` via your container orchestrator's environment variables. Start at `info` and increase selectively:

```bash
# ECS task definition (environment section)
RUST_LOG=info,fluree_db_query=debug,fluree_db_transact=debug
```

## Production Tracing: AWS Deployments

### Architecture: fluree-db-server on ECS/Fargate

```
┌─────────────┐     OTLP/gRPC (4317)     ┌───────────────────┐
│  ECS Task   │ ─────────────────────────▶│  OTEL Collector   │
│  fluree-srv │                           │  (sidecar or      │
│  --features │                           │   Daemon/Service) │
│    otel     │                           └────────┬──────────┘
└─────────────┘                                    │
                                         ┌─────────▼──────────┐
                                         │  Grafana Tempo /    │
                                         │  AWS X-Ray /        │
                                         │  Jaeger             │
                                         └─────────────────────┘
```

**ECS task definition snippet:**

```json
{
  "containerDefinitions": [
    {
      "name": "fluree-server",
      "image": "your-ecr-repo/fluree-db-server:latest",
      "environment": [
        {"name": "RUST_LOG", "value": "info,fluree_db_query=debug,fluree_db_transact=debug"},
        {"name": "OTEL_SERVICE_NAME", "value": "fluree-server"},
        {"name": "OTEL_EXPORTER_OTLP_ENDPOINT", "value": "http://localhost:4317"},
        {"name": "OTEL_EXPORTER_OTLP_PROTOCOL", "value": "grpc"}
      ]
    },
    {
      "name": "otel-collector",
      "image": "amazon/aws-otel-collector:latest",
      "essential": true,
      "command": ["--config=/etc/otel-config.yaml"]
    }
  ]
}
```

**OTEL Collector config (for X-Ray export):**

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  awsxray:
    region: us-east-1
  # Or for Grafana Tempo:
  # otlp:
  #   endpoint: tempo.internal:4317
  #   tls:
  #     insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [awsxray]
```

### Architecture: fluree-db-api as a Rust crate in AWS Lambda

When using `fluree-db-api` directly (not through fluree-db-server), you initialize OTEL yourself. The key pattern is the same dual-layer subscriber with a `Targets` filter on the OTEL layer.

```rust
use opentelemetry_otlp::SpanExporter;
use opentelemetry_sdk::trace::SdkTracerProvider;
use tracing_subscriber::{filter::Targets, layer::SubscriberExt, Registry};

fn init_tracing() {
    // OTEL exporter — Lambda uses HTTP/protobuf to the collector sidecar
    let exporter = SpanExporter::builder()
        .with_http()
        .with_endpoint("http://localhost:4318")  // collector sidecar
        .build()
        .expect("Failed to create OTLP exporter");

    let resource = opentelemetry_sdk::Resource::builder()
        .with_service_name("my-lambda-fn")
        .build();

    let provider = SdkTracerProvider::builder()
        .with_batch_exporter(exporter)
        .with_resource(resource)
        .build();

    let tracer = provider.tracer("fluree-db");
    let otel_layer = tracing_opentelemetry::layer().with_tracer(tracer);

    // Critical: filter OTEL layer to fluree_* crates only
    let otel_filter = Targets::new()
        .with_target("fluree_db_api", tracing::Level::DEBUG)
        .with_target("fluree_db_query", tracing::Level::DEBUG)
        .with_target("fluree_db_transact", tracing::Level::DEBUG)
        .with_target("fluree_db_indexer", tracing::Level::DEBUG)
        .with_target("fluree_db_core", tracing::Level::DEBUG);

    let subscriber = Registry::default()
        .with(tracing_subscriber::fmt::layer())
        .with(otel_layer.with_filter(otel_filter));

    tracing::subscriber::set_global_default(subscriber).ok();
}
```

**Lambda deployment with ADOT (AWS Distro for OpenTelemetry):**

Add the ADOT Lambda layer and set:

```bash
AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler
OTEL_SERVICE_NAME=my-fluree-lambda
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
```

The ADOT layer runs a collector sidecar that receives OTLP spans and exports them to X-Ray, Tempo, or any configured backend.

### Grafana Tempo + Grafana UI

For production trace exploration, Grafana Tempo with the Grafana UI provides the best experience:

1. **Search by attributes**: Find all queries with `tracker_time > 500ms`
2. **Service graph**: Visualize call patterns between services
3. **Trace-to-logs**: Jump from a slow span to the corresponding log lines
4. **Trace-to-metrics**: Correlate latency spikes with metric dashboards

**Tempo query examples (TraceQL):**

```
# Find slow queries
{ resource.service.name = "fluree-server" && name = "query_execute" && duration > 500ms }

# Find large commits
{ name = "txn_commit" && span.flake_count > 1000 }

# Find indexing operations
{ name = "index_build" }
```

### AWS X-Ray

X-Ray works with OTEL traces exported via the AWS OTEL Collector. Key differences from Jaeger/Tempo:

- X-Ray automatically creates a **service map** showing request flow
- Subsegment annotations map to OTEL span attributes
- X-Ray sampling rules can be configured server-side (no code changes)
- Use **X-Ray Insights** for anomaly detection on latency patterns

## Using the otel/ Harness for Regression Testing

The `otel/` directory is designed for reproducible trace validation. Use it to verify that tracing instrumentation works correctly after code changes:

```bash
cd otel/

# Clean slate
make fresh

# After all scenarios complete, check Jaeger:
# 1. Transaction traces should show txn_stage > txn_commit with sub-spans
# 2. Query traces should show query_prepare > query_run with operator spans
# 3. Index traces should appear as separate traces (not under a request)
```

### Specific test scenarios

| Scenario | Command | What to verify in Jaeger |
|----------|---------|-------------------------|
| All transaction types | `make transact` | 5 traces, each with `txn_stage` + `txn_commit` |
| All query types | `make query` | 7 traces with `query_prepare` + `query_run` |
| Background indexing | `make index` | Separate `index_build` trace (not under a request) |
| Bulk import | `make import` | Many commit traces, possibly indexing traces |
| Full end-to-end | `make smoke` | All of the above |
| Multi-cycle stress | `make cycle` | 3 full cycles, multiple `index_build` traces |

## Analyzing Exported Traces

Jaeger allows exporting traces as JSON files (click the download icon on any trace or search result). These exports are useful for offline analysis, sharing with teammates, and archiving evidence of performance issues.

### Exporting from Jaeger

1. Open Jaeger UI (`http://localhost:16686` or your deployment)
2. Search for traces (by service, operation, duration, tags)
3. Click the download/export icon to save as JSON

### What's in the export

The JSON file contains `data[].spans[]` with full span details: operation names, tags (key-value attributes), parent-child references, durations (in microseconds), and timestamps. Files range from ~100KB for a few traces to 50MB+ for large search exports.

### Analyzing with Claude Code

The repository includes Claude Code skills for trace analysis:

```
/trace-inspect /path/to/traces.json    # Drill into a single trace's span tree and timing
/trace-overview /path/to/traces.json   # Aggregate stats and anomaly detection across all traces
```

These skills analyze the export file using targeted Python scripts (to avoid loading the full JSON into context) and cross-reference the results against the expected span hierarchy to produce a diagnosis with concrete code-level fix recommendations.

### Manual analysis with Python

For quick one-off checks without Claude Code, the Jaeger JSON structure is straightforward:

```python
import json

with open("traces.json") as f:
    data = json.load(f)

for trace in data["data"]:
    for span in trace["spans"]:
        tags = {t["key"]: t["value"] for t in span.get("tags", [])}
        print(f"{span['operationName']}  dur={span['duration']/1000:.1f}ms  {tags}")
```

Key fields: `operationName` (span name), `duration` (microseconds), `tags` (span attributes), `references` (parent-child links with `refType: "CHILD_OF"`).

## Related Documentation

- [Telemetry and Logging](../operations/telemetry.md) -- OTEL configuration reference
- [Adding Tracing Spans](../contributing/tracing-guide.md) -- How to instrument new code paths
- [Debugging Queries](debugging-queries.md) -- Query-specific debugging (explain plans, etc.)
- [otel/README.md](../../otel/README.md) -- Full otel/ harness reference


---

# reference/README.md

# Reference

Reference materials for Fluree developers and operators.

## Reference Guides

### [Glossary](glossary.md)

Definitions of key terms and concepts:
- RDF terminology
- Fluree-specific terms
- Database concepts
- Query terminology
- Index terminology

### [Fluree System Vocabulary](vocabulary.md)

Complete reference for Fluree's system vocabulary under `https://ns.flur.ee/db#`:
- Commit metadata predicates (`f:t`, `f:address`, `f:time`, `f:previous`, etc.)
- Search query vocabulary (BM25 and vector search patterns)
- Nameservice record fields and type taxonomy
- Policy vocabulary
- Namespace codes

### [Standards and Feature Flags](compatibility.md)

Standards and feature-flag reference:
- SPARQL 1.1 compliance
- JSON-LD specifications
- W3C standards support
- Feature flags
- Deprecated features

### [Graph Identities and Naming](graph-identities.md)

Naming conventions for graphs, ledgers, and identifiers:
- User-facing terminology (ledger, graph IRI, graph source, graph snapshot)
- Time pinning syntax (`@t:`, `@iso:`, `@commit:`)
- Named graphs within a ledger
- Base resolution for graph references

### [Crate Map](crate-map.md)

Overview of Fluree's Rust crate architecture:
- Core crates
- API crates
- Query engine crates
- Storage crates
- Dependency relationships

## Quick Reference

### Common Namespaces

```turtle
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix schema: <http://schema.org/> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
@prefix dc: <http://purl.org/dc/terms/> .
```

### Fluree Namespaces

```turtle
@prefix f: <https://ns.flur.ee/db#> .
```

### Time Specifiers

```text
ledger:branch@t:123             # Transaction number
ledger:branch@iso:2024-01-22    # ISO timestamp
ledger:branch@commit:bafybeig...  # Commit ContentId
```

### HTTP Status Codes

| Code | Meaning | Common Cause |
|------|---------|--------------|
| 200 | OK | Success |
| 400 | Bad Request | Invalid syntax |
| 401 | Unauthorized | Missing auth |
| 403 | Forbidden | Policy denied |
| 404 | Not Found | Ledger not found |
| 408 | Timeout | Query too slow |
| 413 | Payload Too Large | Request too big |
| 429 | Too Many Requests | Rate limited |
| 500 | Internal Error | Server error |
| 503 | Unavailable | Overloaded |

### Index Types

| Index | Order | Optimized For |
|-------|-------|---------------|
| SPOT | Subject-Predicate-Object-Time | Entity properties |
| POST | Predicate-Object-Subject-Time | Property values |
| OPST | Object-Predicate-Subject-Time | Value lookups |
| PSOT | Predicate-Subject-Object-Time | Predicate scans |

## Standards Compliance

### RDF Standards

- **RDF 1.1:** Full compliance
- **Turtle:** Full support
- **JSON-LD 1.1:** Full compliance
- **N-Triples:** Support (future)

### Query Standards

- **SPARQL 1.1 Query:** Full compliance
- **SPARQL 1.1 Update:** Partial support
- **GeoSPARQL:** Planned

### Security Standards

- **JWS (RFC 7515):** Full support
- **JWT (RFC 7519):** Full support
- **Verifiable Credentials:** W3C compliant
- **DIDs:** did:key, did:web support

## Performance Benchmarks

Typical performance characteristics:

### Query Performance

| Query Type | Small DB | Medium DB | Large DB |
|------------|----------|-----------|----------|
| Simple lookup | < 1ms | < 5ms | < 10ms |
| Pattern match | < 10ms | < 50ms | < 100ms |
| Complex join | < 50ms | < 200ms | < 500ms |
| Aggregation | < 100ms | < 500ms | < 2s |

### Transaction Performance

| Operation | Typical Time |
|-----------|--------------|
| Small insert (< 10 triples) | < 10ms |
| Medium insert (< 100 triples) | < 50ms |
| Large insert (< 1000 triples) | < 200ms |
| Update | < 20ms |
| Upsert | < 30ms |

### Indexing Performance

| Workload | Rate |
|----------|------|
| Light | 1,000 flakes/sec |
| Medium | 5,000 flakes/sec |
| Heavy | 10,000 flakes/sec |

## Related Documentation

- [Glossary](glossary.md) - Term definitions
- [Compatibility](compatibility.md) - Standards compliance
- [Crate Map](crate-map.md) - Code architecture


---

# reference/glossary.md

# Glossary

Definitions of key terms and concepts used throughout Fluree documentation.

## Core Concepts

### Ledger

A versioned graph database instance in Fluree, equivalent to a database in traditional systems. Ledgers are identified by ledger IDs like `mydb:main`.

Example: `customers:main`, `inventory:prod`

### Branch

A variant of a ledger, allowing multiple independent versions of the same logical database. Branches are part of the ledger ID after the colon.

Example: In `mydb:dev`, "dev" is the branch.

### Transaction Time (t)

A monotonically increasing integer assigned to each transaction, representing the logical time of the transaction.

Example: `t=42` is transaction number 42.

### Flake

Fluree's internal representation of an RDF triple with temporal information. A flake is a tuple: (subject, predicate, object, transaction-time, operation, metadata).

### Novelty Layer

The set of transactions that have been committed but not yet indexed. The gap between `commit_t` and `index_t`.

Example: If `commit_t=150` and `index_t=145`, the novelty layer contains transactions 146-150.

### Nameservice

Fluree's metadata registry that tracks ledger state, including commit and index locations. Enables discovery and coordination across distributed deployments.

## RDF Terminology

### IRI (Internationalized Resource Identifier)

A globally unique identifier for resources, predicates, and graphs. The internationalized version of URI supporting Unicode.

Example: `http://example.org/alice`, `http://例え.jp/人物/アリス`

### Triple

The fundamental unit of RDF data: a subject-predicate-object statement.

Example: `ex:alice schema:name "Alice"`

### Subject

The entity being described in a triple (first position).

Example: In `ex:alice schema:name "Alice"`, `ex:alice` is the subject.

### Predicate

The property or relationship in a triple (second position).

Example: In `ex:alice schema:name "Alice"`, `schema:name` is the predicate.

### Object

The value or target entity in a triple (third position).

Example: In `ex:alice schema:name "Alice"`, `"Alice"` is the object.

### Literal

A data value in a triple (string, number, date, etc.), as opposed to an IRI reference.

Example: `"Alice"`, `30`, `"2024-01-22"^^xsd:date`

### Blank Node

An anonymous resource without an explicit IRI.

Example: `[ schema:streetAddress "123 Main St" ]`

### Named Graph

A set of triples identified by an IRI, allowing data partitioning within a ledger.

Example: `ex:graph1` containing specific triples.

### Dataset

A collection of graphs (one default graph and zero or more named graphs) used for query execution.

## Transaction Terms

### Assertion

Adding a new triple to the database.

Example: Asserting `ex:alice schema:age 30` adds this triple.

### Retraction

Removing an existing triple from the current database state.

Example: Retracting `ex:alice schema:age 30` removes this triple.

### Commit

A persisted transaction with assigned transaction time and cryptographic signature.

### Commit ContentId

Content-addressed identifier (CIDv1) for a commit, providing storage-agnostic identity and integrity verification. The SHA-256 digest is embedded in the CID.

Example: `bafybeig...commitT42`

### Replace Mode

Transaction mode where all properties of an entity are replaced, enabling idempotent writes.

Also called: Upsert mode

### WHERE/DELETE/INSERT

Update pattern for targeted modifications: match data (WHERE), remove old data (DELETE), add new data (INSERT).

## Index Terms

### SPOT Index

Subject-Predicate-Object-Time index, optimized for retrieving all properties of a subject.

### POST Index

Predicate-Object-Subject-Time index, optimized for finding subjects with specific property values.

### OPST Index

Object-Predicate-Subject-Time index, optimized for finding subjects that reference specific objects.

### PSOT Index

Predicate-Subject-Object-Time index, optimized for scanning all values of a predicate.

### Index Snapshot

A complete, query-optimized snapshot of the database at a specific transaction time.

### Background Indexing

Asynchronous process that builds index snapshots from committed transactions.

## Query Terms

### Variable

A placeholder in a query pattern that matches actual values in the data, prefixed with `?`.

Example: `?person`, `?name`, `?age`

### Binding

The association of a variable with a specific value during query execution.

Example: `?name` binds to `"Alice"`

### Pattern

A triple template with variables that matches actual triples in the database.

Example: `{ "@id": "?person", "schema:name": "?name" }`

### Filter

A condition that restricts which variable bindings are included in query results.

Example: `"filter": "?age > 25"`

### CONSTRUCT

A SPARQL query form that generates RDF triples rather than variable bindings.

### Graph Crawl

Following relationships recursively to explore connected entities.

## Graph Source Terms

### Graph Source

An addressable query source that participates in execution and can be named in SPARQL via `FROM`, `FROM NAMED`, and `GRAPH <…>`.

Graph sources include:
- Ledger graph sources (default graph and named graphs stored in a ledger)
- Index graph sources (BM25 and vector/HNSW indexes)
- Mapped graph sources (R2RML and Iceberg-backed graph mappings)

### Graph Source (Non-Ledger)

A non-ledger graph source is a queryable data source that appears in graph queries but is backed by specialized storage (BM25 index, vector index, Iceberg table, SQL database).

Example: `products-search:main`, `products-vector:main`

### BM25

Best Matching 25, a ranking algorithm for full-text search. Scores documents by relevance to query terms.

### Vector Embedding

A numerical representation of data (text, images, etc.) as a high-dimensional vector, enabling similarity search.

Example: 384-dimensional vector for text embeddings

### HNSW

Hierarchical Navigable Small World, a graph-based algorithm for approximate nearest neighbor search in high-dimensional spaces.

### R2RML

RDB to RDF Mapping Language, a W3C standard for mapping relational databases to RDF.

### Iceberg

Apache Iceberg, an open table format for huge analytical datasets with ACID guarantees.

## Security Terms

### Policy

A rule specifying who can perform what operations on which data.

### DID (Decentralized Identifier)

A globally unique identifier that doesn't require a central authority, used for cryptographic identity.

Example: `did:key:z6MkhaXgBZDvotDkL5257faiztiGiC2QtKLGpbnnEGta2doK`

### JWS (JSON Web Signature)

An IETF standard (RFC 7515) for representing digitally signed content as JSON.

### Verifiable Credential (VC)

A W3C standard for cryptographically verifiable digital credentials.

### Public Key

Cryptographic key used to verify signatures, shared publicly.

### Private Key

Cryptographic key used to create signatures, kept secret.

## Storage Terms

### ContentId

A CIDv1 (multiformats) value that uniquely identifies any immutable artifact in Fluree. Encodes the content kind (multicodec) and a SHA-256 digest. The canonical string form is base32-lower multibase (e.g., `bafybeig...`).

See [ContentId and ContentStore](../design/content-id-and-contentstore.md) for details.

### ContentKind

An enum identifying the type of content a ContentId refers to: `Commit`, `Txn`, `IndexRoot`, `IndexBranch`, `IndexLeaf`, `DictBlob`, or `DefaultContext`. Encoded as a multicodec tag within the CID.

### ContentStore

The content-addressed storage trait providing `get(ContentId)`, `put(ContentKind, bytes)`, and `has(ContentId)` operations. All immutable artifacts are stored and retrieved via ContentStore.

### Commit ID

A ContentId identifying a committed transaction. Derived by hashing the canonical commit bytes with SHA-256.

Example: `bafybeig...commitT42`

### Index ID

A ContentId identifying an index root snapshot. Derived by hashing the index root descriptor bytes with SHA-256.

Example: `bafybeig...indexRootT145`

### Storage Backend

The underlying system storing Fluree data (memory, file system, AWS S3/DynamoDB).

### Nameservice Record

Metadata about a ledger stored in the nameservice, including commit and index ContentIds.

## Time Travel Terms

### Time Specifier

A suffix on a ledger reference indicating which point in time to query.

Examples: `@t:100`, `@iso:2024-01-22`, `@commit:bafybeig...`

### Point-in-Time Query

A query executed against database state at a specific transaction time.

### History Query

A query that returns changes to entities over a time range, showing assertions and retractions.

### Temporal Database

A database that maintains complete history of all changes, enabling queries at any past state.

## JSON-LD Terms

### @context

JSON-LD mechanism for defining namespace prefixes and term mappings.

Example:
```json
{
  "@context": {
    "ex": "http://example.org/ns/",
    "schema": "http://schema.org/"
  }
}
```

### @id

JSON-LD property for specifying the IRI of a resource.

Example: `"@id": "ex:alice"`

### @type

JSON-LD property for specifying the type(s) of a resource.

Example: `"@type": "schema:Person"`

### @graph

JSON-LD property containing an array of entities.

Example:
```json
{
  "@graph": [
    { "@id": "ex:alice", "schema:name": "Alice" }
  ]
}
```

### @value

JSON-LD property for specifying a literal value explicitly.

Example: `{"@value": "30", "@type": "xsd:integer"}`

### Compact IRI

A shortened IRI using namespace prefix.

Example: `ex:alice` (compact) vs `http://example.org/ns/alice` (full)

### IRI Expansion

Converting compact IRIs to full IRIs using @context mappings.

Example: `ex:alice` expands to `http://example.org/ns/alice`

### IRI Compaction

Converting full IRIs to compact form using @context.

Example: `http://schema.org/name` compacts to `schema:name`

## Query Execution Terms

### Fuel

A decimal measure of query/transaction execution cost, reported to 3 decimal places. Cost reflects actual work — dominated by I/O "touches" (an index leaflet or persisted-dict read costs `0.010` fuel each) rather than output cardinality. Every query is also charged a one-time `1.000` floor, so a fuel-tracked query never reports less than `1.000`. Used to prevent runaway queries from consuming excessive resources. See [Tracking and Fuel](../query/tracking-and-fuel.md) for the full cost ladder.

Example: `"opts": {"max-fuel": 10000}` limits a query to 10,000 fuel.

### Tracking

Query/transaction execution monitoring that provides visibility into performance metrics. When enabled, returns time (execution duration), fuel (items processed), and policy statistics.

Example: `"opts": {"meta": true}` enables all tracking metrics.

### TrackingTally

The result of tracking, containing time (formatted as "12.34ms"), fuel (total count), and policy stats (`{policy-id: {executed, allowed}}`).

## Acronyms

- **ANN:** Approximate Nearest Neighbor
- **API:** Application Programming Interface
- **CORS:** Cross-Origin Resource Sharing
- **CAS:** Compare-And-Swap
- **CID:** Content Identifier (multiformats)
- **DID:** Decentralized Identifier
- **HTTP:** Hypertext Transfer Protocol
- **HNSW:** Hierarchical Navigable Small World
- **IRI:** Internationalized Resource Identifier
- **JSON:** JavaScript Object Notation
- **JSON-LD:** JSON for Linked Data
- **JWT:** JSON Web Token
- **JWS:** JSON Web Signature
- **RDF:** Resource Description Framework
- **REST:** Representational State Transfer
- **SHA:** Secure Hash Algorithm
- **SPARQL:** SPARQL Protocol and RDF Query Language
- **SSL/TLS:** Secure Sockets Layer / Transport Layer Security
- **URI:** Uniform Resource Identifier
- **URL:** Uniform Resource Locator
- **VC:** Verifiable Credential
- **W3C:** World Wide Web Consortium
- **XSD:** XML Schema Definition

## Related Documentation

- [Standards and feature flags](compatibility.md) - Standards compliance and feature flags
- [Crate Map](crate-map.md) - Code architecture
- [Concepts](../concepts/README.md) - Core concepts


---

# reference/vocabulary.md

# Fluree System Vocabulary Reference

All Fluree system vocabulary lives under a single canonical namespace:

```
https://ns.flur.ee/db#
```

Users declare a prefix in their JSON-LD `@context` to use compact forms:

```json
{ "@context": { "f": "https://ns.flur.ee/db#" } }
```

Any prefix works (`f:`, `db:`, `fluree:`, etc.) as long as it expands to the canonical IRI. Internally, Fluree always compares on fully expanded IRIs.

The `@vector` and `@fulltext` shorthands are exceptions: they are JSON-LD convenience aliases that resolve to `f:embeddingVector` and `f:fullText` respectively without requiring a prefix declaration.

> **Source of truth**: All constants are defined in the `fluree-vocab` crate. This document is the user-facing reference.

---

## Commit metadata predicates

These predicates appear on commit subjects in the txn-meta graph. Each commit produces 7-10 flakes describing the commit.

| Predicate | Full IRI | Datatype | Description |
|-----------|----------|----------|-------------|
| `f:address` | `https://ns.flur.ee/db#address` | `xsd:string` | Commit ContentId (CID string) |
| `f:alias` | `https://ns.flur.ee/db#alias` | `xsd:string` | Ledger ID (e.g. `mydb:main`) |
| `f:v` | `https://ns.flur.ee/db#v` | `xsd:int` | Commit format version |
| `f:time` | `https://ns.flur.ee/db#time` | `xsd:long` | Commit timestamp (epoch milliseconds) |
| `f:t` | `https://ns.flur.ee/db#t` | `xsd:int` | Transaction number (watermark) |
| `f:size` | `https://ns.flur.ee/db#size` | `xsd:long` | Cumulative data size in bytes |
| `f:flakes` | `https://ns.flur.ee/db#flakes` | `xsd:long` | Cumulative flake count |
| `f:previous` | `https://ns.flur.ee/db#previous` | `@id` (ref) | Reference to previous commit (optional) |
| `f:identity` | `https://ns.flur.ee/db#identity` | `xsd:string` | Authenticated identity acting on the transaction (system-controlled — verified DID for signed requests, otherwise `opts.identity` / `CommitOpts.identity`). |
| `f:author` | `https://ns.flur.ee/db#author` | `xsd:string` | Author claim — user-supplied via `f:author` in the transaction body (optional). Distinct from `f:identity`. |
| `f:txn` | `https://ns.flur.ee/db#txn` | `xsd:string` | Transaction ContentId (CID string, optional) |
| `f:message` | `https://ns.flur.ee/db#message` | `xsd:string` | Commit message — user-supplied via `f:message` in the transaction body (optional). |
| `f:asserts` | `https://ns.flur.ee/db#asserts` | `xsd:long` | Assertion count in this commit |
| `f:retracts` | `https://ns.flur.ee/db#retracts` | `xsd:long` | Retraction count in this commit |

### Querying commit metadata

Commit metadata lives in the `#txn-meta` named graph within each ledger. To query it:

```json
{
  "@context": { "f": "https://ns.flur.ee/db#" },
  "select": ["?t", "?time", "?author"],
  "where": {
    "@graph": "mydb:main#txn-meta",
    "f:t": "?t",
    "f:time": "?time",
    "f:author": "?author"
  }
}
```

### Commit subject identifiers

Commit subjects use the scheme `fluree:commit:<content-id>` (e.g. `fluree:commit:bafybeig...`). This is a subject identifier scheme, not part of the `db#` predicate vocabulary.

---

## Datalog rules

| Predicate | Full IRI | Description |
|-----------|----------|-------------|
| `f:rule` | `https://ns.flur.ee/db#rule` | Datalog rule definition predicate |

---

## Edge-annotation predicates (reserved)

These seven predicates encode the edge that an [edge annotation](../concepts/edge-annotations.md) reifies. They are the durable, system-controlled representation behind the `@annotation` / `@reifies` (JSON-LD) and `{| ... |}` / `~` / `rdf:reifies` (SPARQL 1.2) surfaces. Together they form a *reifies bundle* on the annotation subject.

| Predicate | Full IRI | Datatype | Description |
|-----------|----------|----------|-------------|
| `f:reifiesGraph` | `https://ns.flur.ee/db#reifiesGraph` | `@id` (ref) | Named graph the reified edge lives in. **Optional** — omitted for default-graph edges. |
| `f:reifiesSubject` | `https://ns.flur.ee/db#reifiesSubject` | `@id` (ref) | Subject of the reified edge. **Required.** |
| `f:reifiesPredicate` | `https://ns.flur.ee/db#reifiesPredicate` | `@id` (ref) | Predicate of the reified edge. **Required.** |
| `f:reifiesObject` | `https://ns.flur.ee/db#reifiesObject` | any | Object of the reified edge — ref, literal, or language-tagged string. **Required.** |
| `f:reifiesDatatype` | `https://ns.flur.ee/db#reifiesDatatype` | `@id` (ref) | Datatype IRI of a literal object. **Optional** — the JSON-LD lowering omits it (recovered from the object flake's datatype); a hand-authored bundle may carry it. |
| `f:reifiesLang` | `https://ns.flur.ee/db#reifiesLang` | `xsd:string` | BCP-47 language tag, present only when the object is a language-tagged string. **Optional.** |
| `f:reifiesListIndex` | `https://ns.flur.ee/db#reifiesListIndex` | `xsd:int` | List-occurrence index. **Reserved/deferred** — always omitted in this release. |

**These predicates are reserved.** User-authored mention of any `f:reifies*` IRI (compact or full form) is rejected at parse time on every write surface (JSON-LD insert/upsert/update, SPARQL UPDATE, Turtle/raw ingest), and they are filtered out of variable-predicate (`?p`) scans and wildcard (`select: "*"`) hydration so they never surface as ordinary RDF. Mint and manage annotations only through `@annotation` / `@edge` (JSON-LD) or the annotation tail (`{| ... |}` / `~` / `rdf:reifies`) in SPARQL 1.2. See [Edge annotations](../concepts/edge-annotations.md) for the full surface and the [storage-internals design doc](../design/edge-annotations.md) for the bundle encoding and invariants.

---

## Vector datatype

| Term | IRI | Description |
|------|-----|-------------|
| `f:embeddingVector` | `https://ns.flur.ee/db#embeddingVector` | f32-precision embedding vector datatype |
| `@vector` | (shorthand) | JSON-LD alias that resolves to `f:embeddingVector` |

Example usage in a transaction:

```json
{
  "@context": { "f": "https://ns.flur.ee/db#", "ex": "http://example.org/" },
  "insert": {
    "@id": "ex:doc1",
    "ex:embedding": { "@value": [0.1, 0.2, 0.3], "@type": "f:embeddingVector" }
  }
}
```

Or with the `@vector` shorthand:

```json
{
  "insert": {
    "@id": "ex:doc1",
    "ex:embedding": { "@value": [0.1, 0.2, 0.3], "@type": "@vector" }
  }
}
```

A property declared with `@type: "@vector"` (or `@type: "f:embeddingVector"`) in the `@context` may also use a bare JSON array as its value — equivalent to the explicit `@value` form above:

```json
{
  "@context": {
    "ex": "http://example.org/",
    "ex:embedding": { "@type": "@vector" }
  },
  "insert": {
    "@id": "ex:doc1",
    "ex:embedding": [0.1, 0.2, 0.3]
  }
}
```

### Validation rules

- **Element type:** every element must be a JSON number; non-numeric elements are rejected.
- **Element range:** values are quantized to `f32` at ingest. Non-finite values (`NaN`, `±Infinity`) and values outside the representable `f32` range are rejected.
- **Non-empty:** vectors must have at least one element. The empty vector (`[]`) is reserved as an internal max-bound sentinel and is rejected by both the coercion layer and the write-path guard.
- **Scalar values are rejected:** a single number paired with the `f:embeddingVector` datatype (e.g. `{"@value": 0.1, "@type": "@vector"}`) is rejected; the value must be an array.

The same rules apply to the SPARQL typed-literal form `"[0.1, 0.2, 0.3]"^^f:embeddingVector` and to Turtle.

---

## Fulltext datatype

| Term | IRI | Description |
|------|-----|-------------|
| `f:fullText` | `https://ns.flur.ee/db#fullText` | Inline full-text search datatype |
| `@fulltext` | (shorthand) | JSON-LD alias that resolves to `f:fullText` |

Example usage in a transaction:

```json
{
  "@context": { "f": "https://ns.flur.ee/db#", "ex": "http://example.org/" },
  "insert": {
    "@id": "ex:article-1",
    "ex:content": { "@value": "Rust is a systems programming language", "@type": "f:fullText" }
  }
}
```

Or with the `@fulltext` shorthand:

```json
{
  "insert": {
    "@id": "ex:article-1",
    "ex:content": { "@value": "Rust is a systems programming language", "@type": "@fulltext" }
  }
}
```

Values annotated with `@fulltext` are analyzed (tokenized, stemmed) and indexed into per-predicate fulltext arenas during background index builds. Query with the `fulltext()` function in `bind` expressions for BM25 relevance scoring.

See [Inline Fulltext Search](../indexing-and-search/fulltext.md) for details.

### Fulltext configuration predicates

These predicates live in the ledger's `#config` named graph and declare which properties to full-text index (no per-value `@fulltext` annotation needed). See [Configured full-text properties](../indexing-and-search/fulltext.md#configured-full-text-properties-ffulltextdefaults) for the end-user guide and [Setting groups](../ledger-config/setting-groups.md#full-text-defaults) for the full schema reference.

| Term | IRI | Description |
|------|-----|-------------|
| `f:fullTextDefaults` | `https://ns.flur.ee/db#fullTextDefaults` | Setting group on `f:LedgerConfig` / `f:GraphConfig` |
| `f:FullTextDefaults` | `https://ns.flur.ee/db#FullTextDefaults` | Class (type) of the setting-group node |
| `f:defaultLanguage` | `https://ns.flur.ee/db#defaultLanguage` | BCP-47 tag used for untagged plain strings on configured properties |
| `f:property` | `https://ns.flur.ee/db#property` | One entry per full-text-indexed property (cardinality 0..n) |
| `f:FullTextProperty` | `https://ns.flur.ee/db#FullTextProperty` | Class of each `f:property` entry |
| `f:target` | `https://ns.flur.ee/db#target` | IRI of the property being indexed (on `f:FullTextProperty`) |

---

## Search query vocabulary

These predicates are used in WHERE clause patterns for BM25 and vector search. Users write compact forms like `"f:searchText"` (with `"f"` in their `@context`) or full IRIs.

### BM25 search

| Predicate | Full IRI | Required | Description |
|-----------|----------|----------|-------------|
| `f:graphSource` | `https://ns.flur.ee/db#graphSource` | Yes | Graph source ID (`name:branch`, e.g. `"my-search:main"`) |
| `f:searchText` | `https://ns.flur.ee/db#searchText` | Yes | Search query text (string or variable) |
| `f:searchResult` | `https://ns.flur.ee/db#searchResult` | Yes | Result binding (variable or nested object) |
| `f:searchLimit` | `https://ns.flur.ee/db#searchLimit` | No | Maximum results |
| `f:syncBeforeQuery` | `https://ns.flur.ee/db#syncBeforeQuery` | No | Wait for index sync before querying (boolean) |
| `f:timeoutMs` | `https://ns.flur.ee/db#timeoutMs` | No | Query timeout in milliseconds |

### Vector search

| Predicate | Full IRI | Required | Description |
|-----------|----------|----------|-------------|
| `f:graphSource` | `https://ns.flur.ee/db#graphSource` | Yes | Graph source ID (`name:branch`) |
| `f:queryVector` | `https://ns.flur.ee/db#queryVector` | Yes | Query vector (array of numbers or variable) |
| `f:searchResult` | `https://ns.flur.ee/db#searchResult` | Yes | Result binding |
| `f:distanceMetric` | `https://ns.flur.ee/db#distanceMetric` | No | Distance metric: `"cosine"`, `"dot"`, `"euclidean"` (default: `"cosine"`) |
| `f:searchLimit` | `https://ns.flur.ee/db#searchLimit` | No | Maximum results |
| `f:syncBeforeQuery` | `https://ns.flur.ee/db#syncBeforeQuery` | No | Wait for index sync (boolean) |
| `f:timeoutMs` | `https://ns.flur.ee/db#timeoutMs` | No | Query timeout in milliseconds |

### Nested result objects

Both BM25 and vector search support nested result bindings:

| Predicate | Full IRI | Description |
|-----------|----------|-------------|
| `f:resultId` | `https://ns.flur.ee/db#resultId` | Document/subject ID binding |
| `f:resultScore` | `https://ns.flur.ee/db#resultScore` | Search score binding |
| `f:resultLedger` | `https://ns.flur.ee/db#resultLedger` | Source ledger ID (multi-ledger disambiguation) |

Example BM25 search with nested result:

```json
{
  "@context": { "f": "https://ns.flur.ee/db#" },
  "select": ["?doc", "?score"],
  "where": {
    "f:graphSource": "my-search:main",
    "f:searchText": "software engineer",
    "f:searchLimit": 10,
    "f:searchResult": {
      "f:resultId": "?doc",
      "f:resultScore": "?score"
    }
  }
}
```

---

## Nameservice record vocabulary

### Ledger record fields

These predicates appear on ledger nameservice records (the metadata Fluree stores about each ledger).

| Predicate | Full IRI | Description |
|-----------|----------|-------------|
| `f:ledger` | `https://ns.flur.ee/db#ledger` | Ledger name/identifier |
| `f:branch` | `https://ns.flur.ee/db#branch` | Branch name (e.g. `main`) |
| `f:t` | `https://ns.flur.ee/db#t` | Current transaction watermark |
| `f:ledgerCommit` | `https://ns.flur.ee/db#ledgerCommit` | Pointer to latest commit ContentId |
| `f:ledgerIndex` | `https://ns.flur.ee/db#ledgerIndex` | Pointer to latest index root |
| `f:status` | `https://ns.flur.ee/db#status` | Record status (`ready`, etc.) |
| `f:defaultContextCid` | `https://ns.flur.ee/db#defaultContextCid` | Default JSON-LD context ContentId |

### Graph source record fields

| Predicate | Full IRI | Description |
|-----------|----------|-------------|
| `f:name` | `https://ns.flur.ee/db#name` | Graph source base name |
| `f:branch` | `https://ns.flur.ee/db#branch` | Branch |
| `f:status` | `https://ns.flur.ee/db#status` | Status |
| `f:graphSourceConfig` | `https://ns.flur.ee/db#graphSourceConfig` | Configuration JSON |
| `f:graphSourceDependencies` | `https://ns.flur.ee/db#graphSourceDependencies` | Dependent ledger IDs |
| `f:graphSourceIndex` | `https://ns.flur.ee/db#graphSourceIndex` | Index ContentId reference |
| `f:graphSourceIndexT` | `https://ns.flur.ee/db#graphSourceIndexT` | Index watermark (commit t) |
| `f:graphSourceIndexAddress` | `https://ns.flur.ee/db#graphSourceIndexAddress` | Index ContentId (string form) |

### Record type taxonomy

Nameservice records use `@type` to classify what kind of graph source a record represents.

**Required kind types** (exactly one per record):

| Type | Full IRI | Description |
|------|----------|-------------|
| `f:LedgerSource` | `https://ns.flur.ee/db#LedgerSource` | Ledger-backed knowledge graph |
| `f:IndexSource` | `https://ns.flur.ee/db#IndexSource` | Index-backed graph source (BM25/HNSW/GEO) |
| `f:MappedSource` | `https://ns.flur.ee/db#MappedSource` | Mapped database (Iceberg, R2RML) |

**Optional subtype `@type` values** (further classify the record):

| Type | Full IRI | Description |
|------|----------|-------------|
| `f:Bm25Index` | `https://ns.flur.ee/db#Bm25Index` | BM25 full-text search index |
| `f:HnswIndex` | `https://ns.flur.ee/db#HnswIndex` | HNSW vector similarity search index |
| `f:GeoIndex` | `https://ns.flur.ee/db#GeoIndex` | Geospatial index |
| `f:IcebergMapping` | `https://ns.flur.ee/db#IcebergMapping` | Iceberg-mapped database |
| `f:R2rmlMapping` | `https://ns.flur.ee/db#R2rmlMapping` | R2RML relational mapping |

---

## Policy vocabulary

These predicates are used to define access control policies.

| Predicate | Full IRI | Description |
|-----------|----------|-------------|
| `f:policyClass` | `https://ns.flur.ee/db#policyClass` | Marks a class as policy-governed |
| `f:allow` | `https://ns.flur.ee/db#allow` | Allow/deny flag on a policy rule |
| `f:action` | `https://ns.flur.ee/db#action` | Action this rule governs (view or modify) |
| `f:view` | `https://ns.flur.ee/db#view` | View action IRI |
| `f:modify` | `https://ns.flur.ee/db#modify` | Modify action IRI |
| `f:onProperty` | `https://ns.flur.ee/db#onProperty` | Property-level policy targeting |
| `f:onSubject` | `https://ns.flur.ee/db#onSubject` | Subject-level policy targeting |
| `f:onClass` | `https://ns.flur.ee/db#onClass` | Class-level policy targeting |
| `f:query` | `https://ns.flur.ee/db#query` | Policy query (determines applicability) |
| `f:required` | `https://ns.flur.ee/db#required` | Whether the policy is required (boolean) |
| `f:exMessage` | `https://ns.flur.ee/db#exMessage` | Error message when policy denies access |

See [Policy model and inputs](../security/policy-model.md) for usage details.

---

## Config graph vocabulary

These predicates define ledger-level configuration stored in the config graph. See [Ledger configuration](../ledger-config/README.md) for full documentation.

### Core types

| Type | Full IRI | Description |
|------|----------|-------------|
| `f:LedgerConfig` | `https://ns.flur.ee/db#LedgerConfig` | Ledger-wide configuration resource |
| `f:GraphConfig` | `https://ns.flur.ee/db#GraphConfig` | Per-graph configuration override |
| `f:GraphRef` | `https://ns.flur.ee/db#GraphRef` | Reference to a graph source |

### Setting group predicates

| Predicate | Full IRI | Description |
|-----------|----------|-------------|
| `f:policyDefaults` | `https://ns.flur.ee/db#policyDefaults` | Policy enforcement defaults |
| `f:shaclDefaults` | `https://ns.flur.ee/db#shaclDefaults` | SHACL validation defaults |
| `f:reasoningDefaults` | `https://ns.flur.ee/db#reasoningDefaults` | OWL/RDFS reasoning defaults |
| `f:datalogDefaults` | `https://ns.flur.ee/db#datalogDefaults` | Datalog rule defaults |
| `f:transactDefaults` | `https://ns.flur.ee/db#transactDefaults` | Transaction constraint defaults |

### Policy fields

| Predicate | Full IRI | Description |
|-----------|----------|-------------|
| `f:defaultAllow` | `https://ns.flur.ee/db#defaultAllow` | Default allow/deny when no policy matches (boolean) |
| `f:policySource` | `https://ns.flur.ee/db#policySource` | Graph containing policy rules (GraphRef) |
| `f:policyClass` | `https://ns.flur.ee/db#policyClass` | Default policy classes to apply |

### SHACL fields

| Predicate | Full IRI | Description |
|-----------|----------|-------------|
| `f:shaclEnabled` | `https://ns.flur.ee/db#shaclEnabled` | Enable/disable SHACL validation (boolean) |
| `f:shapesSource` | `https://ns.flur.ee/db#shapesSource` | Graph containing SHACL shapes (GraphRef) |
| `f:validationMode` | `https://ns.flur.ee/db#validationMode` | `f:ValidationReject` or `f:ValidationWarn` |

### Reasoning fields

| Predicate | Full IRI | Description |
|-----------|----------|-------------|
| `f:reasoningModes` | `https://ns.flur.ee/db#reasoningModes` | Reasoning modes: `f:RDFS`, `f:OWL2QL`, `f:OWL2RL`, `f:Datalog` |
| `f:schemaSource` | `https://ns.flur.ee/db#schemaSource` | Graph containing schema triples (GraphRef) |
| `f:reasoningMaxFacts` | `https://ns.flur.ee/db#reasoningMaxFacts` | OWL2-RL materialization budget: max derived facts (integer) |
| `f:reasoningMaxSeconds` | `https://ns.flur.ee/db#reasoningMaxSeconds` | OWL2-RL materialization budget: max wall-clock seconds (integer) |

### Datalog fields

| Predicate | Full IRI | Description |
|-----------|----------|-------------|
| `f:datalogEnabled` | `https://ns.flur.ee/db#datalogEnabled` | Enable/disable datalog rules (boolean) |
| `f:rulesSource` | `https://ns.flur.ee/db#rulesSource` | Graph containing `f:rule` definitions (GraphRef) |
| `f:allowQueryTimeRules` | `https://ns.flur.ee/db#allowQueryTimeRules` | Allow ad-hoc query-time rules (boolean) |

### Transact / uniqueness fields

| Predicate | Full IRI | Description |
|-----------|----------|-------------|
| `f:uniqueEnabled` | `https://ns.flur.ee/db#uniqueEnabled` | Enable unique constraint enforcement (boolean) |
| `f:constraintsSource` | `https://ns.flur.ee/db#constraintsSource` | Graph(s) containing constraint annotations (GraphRef) |
| `f:enforceUnique` | `https://ns.flur.ee/db#enforceUnique` | Annotation on property IRIs: enforce value uniqueness (boolean) |

### Override control

| Term | Full IRI | Description |
|------|----------|-------------|
| `f:overrideControl` | `https://ns.flur.ee/db#overrideControl` | Override gating on a setting group |
| `f:OverrideNone` | `https://ns.flur.ee/db#OverrideNone` | No overrides permitted |
| `f:OverrideAll` | `https://ns.flur.ee/db#OverrideAll` | Any request can override (default) |
| `f:IdentityRestricted` | `https://ns.flur.ee/db#IdentityRestricted` | Only verified identities can override |
| `f:controlMode` | `https://ns.flur.ee/db#controlMode` | Control mode (for identity-restricted objects) |
| `f:allowedIdentities` | `https://ns.flur.ee/db#allowedIdentities` | List of DIDs authorized to override |

### Graph targeting

| Predicate | Full IRI | Description |
|-----------|----------|-------------|
| `f:graphOverrides` | `https://ns.flur.ee/db#graphOverrides` | List of `f:GraphConfig` per-graph overrides |
| `f:targetGraph` | `https://ns.flur.ee/db#targetGraph` | Target graph IRI for a `f:GraphConfig` |
| `f:graphSelector` | `https://ns.flur.ee/db#graphSelector` | Graph selector within a `f:GraphRef` |
| `f:defaultGraph` | `https://ns.flur.ee/db#defaultGraph` | Sentinel IRI for the default graph |
| `f:txnMetaGraph` | `https://ns.flur.ee/db#txnMetaGraph` | Sentinel IRI for the txn-meta graph |

See [Ledger configuration](../ledger-config/README.md) for usage details.

---

## RDF-Star annotation predicates

Fluree supports RDF-Star annotations for transaction metadata. These predicates can appear in annotation triples:

| Predicate | Full IRI | Description |
|-----------|----------|-------------|
| `f:t` | `https://ns.flur.ee/db#t` | Transaction number on an annotated triple |
| `f:op` | `https://ns.flur.ee/db#op` | Operation type (assert/retract) |

---

## Namespace codes (internal)

Fluree encodes namespace IRIs as integer codes for compact storage. These are internal implementation details but useful for contributors working on the core.

| Code | Namespace | IRI |
|------|-----------|-----|
| 0 | (empty) | `""` |
| 1 | JSON-LD | `@` |
| 2 | XSD | `http://www.w3.org/2001/XMLSchema#` |
| 3 | RDF | `http://www.w3.org/1999/02/22-rdf-syntax-ns#` |
| 4 | RDFS | `http://www.w3.org/2000/01/rdf-schema#` |
| 5 | SHACL | `http://www.w3.org/ns/shacl#` |
| 6 | OWL | `http://www.w3.org/2002/07/owl#` |
| 7 | Fluree DB | `https://ns.flur.ee/db#` |
| 8 | DID Key | `did:key:` |
| 9 | Fluree Commit | `fluree:commit:` |
| 10 | Blank Node | `_:` |
| 11 | OGC GeoSPARQL | `http://www.opengis.net/ont/geosparql#` |
| 100+ | User-defined | (allocated at first use) |

---

## Standard W3C namespaces

Fluree also recognizes these standard W3C namespaces:

| Prefix | IRI | Common predicates |
|--------|-----|-------------------|
| `rdf:` | `http://www.w3.org/1999/02/22-rdf-syntax-ns#` | `rdf:type`, `rdf:first`, `rdf:rest` |
| `rdfs:` | `http://www.w3.org/2000/01/rdf-schema#` | `rdfs:label`, `rdfs:subClassOf`, `rdfs:range` |
| `xsd:` | `http://www.w3.org/2001/XMLSchema#` | `xsd:string`, `xsd:int`, `xsd:dateTime` |
| `owl:` | `http://www.w3.org/2002/07/owl#` | `owl:sameAs`, `owl:inverseOf` |
| `sh:` | `http://www.w3.org/ns/shacl#` | `sh:path`, `sh:datatype`, `sh:minCount` |

See [IRIs, namespaces, and JSON-LD @context](../concepts/iri-and-context.md) for details on prefix declarations and IRI resolution.


---

# reference/connection-config-jsonld.md

# JSON-LD Connection Configuration (Rust)

This page documents the **JSON-LD connection config** supported by the Rust implementation.

This config uses the same `@context` + `@graph` model as other Fluree JSON-LD config surfaces.

## Using with the Fluree server

The server accepts a connection config file via `--connection-config`:

```bash
fluree server run --connection-config /path/to/connection.jsonld
```

This replaces `--storage-path` for S3, DynamoDB, and other non-filesystem backends. The server builds its storage and nameservice from the config file at startup. Server-level settings (`--cache-max-mb`, `--indexing-enabled`, etc.) override connection config defaults. See [Configuration](../operations/configuration.md#connection-configuration-s3-dynamodb-etc) for full details and examples.

## Entry points (Rust API)

All construction flows through `FlureeBuilder`:

- `FlureeBuilder::from_json_ld(&json)?` — parses JSON-LD config into builder settings
  - Then call `.build_client().await` for a type-erased `FlureeClient`
  - Or use typed terminal methods (`.build()`, `.build_memory()`, `.build_s3()`) for compile-time type safety
## JSON-LD shape

At minimum, your document contains:

- `@context` with `@base` and `@vocab`
- `@graph` with:
  - one `Connection` node
  - one or more `Storage` nodes
  - optional `Publisher` nodes (nameservice backends)

```json
{
  "@context": {
    "@base": "https://ns.flur.ee/config/connection/",
    "@vocab": "https://ns.flur.ee/system#"
  },
  "@graph": [
    { "@id": "storage1", "@type": "Storage", "filePath": "./data" },
    {
      "@id": "connection",
      "@type": "Connection",
      "indexStorage": { "@id": "storage1" }
    }
  ]
}
```

## ConfigurationValue (env var indirection)

Many fields can be provided as direct literals **or** as a `ConfigurationValue` object:

```json
{
  "s3Bucket": { "envVar": "FLUREE_S3_BUCKET", "defaultVal": "my-bucket" },
  "cacheMaxMb": { "envVar": "FLUREE_CACHE_MAX_MB", "defaultVal": "1024" }
}
```

Notes:
- `envVar`: reads from environment (non-wasm targets)
- `defaultVal`: fallback string value
- `javaProp`: accepted for compatibility; Rust treats it like another env var key (best-effort)

## Connection node fields

Supported:
- `parallelism` (default 4)
- `cacheMaxMb` (supports `ConfigurationValue`)
- `indexStorage` (required): reference to a `Storage` node
- `commitStorage` (optional): reference to a `Storage` node
- `primaryPublisher` (optional): reference to a `Publisher` node
- `addressIdentifiers` (read routing): map of identifier → storage reference
- `defaults` (partial):
  - `defaults.indexing.reindexMinBytes` / `reindexMaxBytes` are applied as the default `IndexConfig` for writes
  - `defaults.indexing.indexingEnabled=false` suppresses background index triggers
  - `defaults.indexing.maxOldIndexes` sets the maximum number of old index versions to retain before GC (default: 5)
  - `defaults.indexing.gcMinTimeMins` sets the minimum age in minutes before an index can be garbage collected (default: 30)

### addressIdentifiers (read routing)

The `addressIdentifiers` field maps identifier strings to storage backends, enabling read routing based on the identifier segment in Fluree addresses.

```json
{
  "@id": "connection",
  "@type": "Connection",
  "indexStorage": {"@id": "indexS3"},
  "commitStorage": {"@id": "commitS3"},
  "addressIdentifiers": {
    "commit-storage": {"@id": "commitS3"},
    "index-storage": {"@id": "indexS3"}
  }
}
```

**Routing behavior:**
- `fluree:commit-storage:s3://db/commit/abc.fcv2` → routes to `commitS3`
- `fluree:index-storage:s3://db/index/xyz.json` → routes to `indexS3`
- `fluree:s3://db/index/xyz.json` (no identifier) → routes to default storage
- `fluree:unknown-id:s3://db/file.json` (unknown identifier) → fallback to default storage

**Notes:**
- Writes always go to the default storage (TieredStorage or indexStorage), regardless of identifier
- This is a read-only routing mechanism for addresses that already contain identifiers
- Use `addressIdentifier` (singular) on storage nodes to **write** addresses with identifier segments

Not yet supported (parsed/ignored or absent):
- `remoteSystems` — not supported

## Storage node fields

### Memory storage

```json
{ "@id": "mem", "@type": "Storage" }
```

### File storage (requires `native`)

Supported:
- `filePath`
- `AES256Key` (supports `ConfigurationValue`)

Notes:
- Rust expects `AES256Key` to be **base64-encoded** and decode to exactly 32 bytes.
- This encrypts the **index/commit blobs** written via the storage layer. The file-based
  nameservice remains plaintext, matching the existing builder behavior.

```json
{
  "@id": "fileStorage",
  "@type": "Storage",
  "filePath": "/var/lib/fluree",
  "AES256Key": { "envVar": "FLUREE_ENCRYPTION_KEY" }
}
```

### S3 storage (requires `aws`)

Supported fields (parsed and **applied** by Rust):
- `s3Bucket`
- `s3Prefix`
- `s3Endpoint` (optional; recommended **only** for LocalStack/MinIO/custom endpoints)
- `s3ReadTimeoutMs`, `s3WriteTimeoutMs`, `s3ListTimeoutMs`
  - Rust applies a single **operation timeout** of `max(read, write, list)`
- `s3MaxRetries`, `s3RetryBaseDelayMs`, `s3RetryMaxDelayMs`
  - Rust maps `s3MaxRetries` to AWS SDK `max_attempts = max_retries + 1`
- `s3MaxConcurrentRequests`
  - Optional per-storage-instance cap on concurrent S3 SDK operations

#### Standard S3 (AWS)

```json
{
  "@id": "s3",
  "@type": "Storage",
  "s3Bucket": "fluree-prod-data",
  "s3Prefix": "fluree/"
}
```

#### LocalStack / MinIO (custom endpoint)

```json
{
  "@id": "s3",
  "@type": "Storage",
  "s3Bucket": "fluree-test",
  "s3Endpoint": "http://localhost:4566",
  "s3Prefix": "fluree/"
}
```

#### S3 Express One Zone

Rust relies on the AWS SDK’s native support for directory buckets. We also provide bucket-name
detection (`--x-s3` + `-azN`) for diagnostics.

```json
{
  "@id": "s3Express",
  "@type": "Storage",
  "s3Bucket": "my-index--use1-az1--x-s3",
  "s3Prefix": "indexes/"
}
```

Note: omit `s3Endpoint` for Express directory buckets and let the AWS SDK handle endpoint
resolution. `FlureeBuilder::s3()` is designed for standard and LocalStack
endpoints; for Express buckets, use `FlureeBuilder::from_json_ld()` with a config that omits
`s3Endpoint`.

Guidance:
- **Standard S3 in AWS**: omit `s3Endpoint` (let the SDK pick defaults)
- **Express One Zone**: omit `s3Endpoint`
- **LocalStack/MinIO/custom**: set `s3Endpoint`

For serverless index-storage tradeoffs, including Standard S3 vs S3 Express One
Zone benchmark ranges, see
[Serverless Storage Choices](../operations/serverless-storage.md).

#### addressIdentifier

Rust parses `addressIdentifier` on storage nodes and uses it to rewrite **published**
commit/index ContentIds so they include the identifier segment, e.g.:
`fluree:{addressIdentifier}:s3://...`.

This is mainly useful when you have multiple storage backends and want addresses to
carry an explicit storage identifier.

## Split commit vs index storage (tiered S3)

Rust supports the tiered `commitStorage` + `indexStorage` format via `FlureeBuilder::from_json_ld()` / `build_client()`.
Internally, Rust routes:
- `.../commit/...` and `.../txn/...` → commit storage
- everything else → index storage

```json
{
  "@context": {"@base": "https://ns.flur.ee/config/connection/", "@vocab": "https://ns.flur.ee/system#"},
  "@graph": [
    { "@id": "commitStorage", "@type": "Storage", "s3Bucket": "commits-bucket", "s3Prefix": "fluree-data/" },
    { "@id": "indexStorage",  "@type": "Storage", "s3Bucket": "index--use1-az1--x-s3" },
    { "@id": "publisher", "@type": "Publisher", "dynamodbTable": "fluree-nameservice", "dynamodbRegion": "us-east-1" },
    {
      "@id": "connection",
      "@type": "Connection",
      "commitStorage": {"@id": "commitStorage"},
      "indexStorage":  {"@id": "indexStorage"},
      "primaryPublisher": {"@id": "publisher"}
    }
  ]
}
```

### IPFS storage (requires `ipfs`)

Supported:
- `ipfsApiUrl` (default `http://127.0.0.1:5001`): Kubo HTTP RPC API base URL
- `ipfsPinOnPut` (default `true`): pin blocks after writing

```json
{
  "@id": "ipfsStorage",
  "@type": "Storage",
  "ipfsApiUrl": "http://127.0.0.1:5001",
  "ipfsPinOnPut": true
}
```

With env var indirection:

```json
{
  "@id": "ipfsStorage",
  "@type": "Storage",
  "ipfsApiUrl": { "envVar": "FLUREE_IPFS_API_URL", "defaultVal": "http://127.0.0.1:5001" },
  "ipfsPinOnPut": true
}
```

Notes:
- Requires a running Kubo node at the specified URL
- Fluree's CIDs (SHA-256 + private-use multicodec) are stored directly into IPFS
- No encryption support (`AES256Key` is not applicable)
- See [IPFS Storage Guide](../operations/ipfs-storage.md) for Kubo setup and operational details

## Publisher (nameservice) node fields

### Storage-backed nameservice

Supported:
- `storage` (reference to a `Storage` node)

```json
{
  "@id": "publisher",
  "@type": "Publisher",
  "storage": { "@id": "s3" }
}
```

### DynamoDB nameservice (requires `aws`)

Supported (and applied):
- `dynamodbTable`
- `dynamodbRegion`
- `dynamodbEndpoint`
- `dynamodbTimeoutMs`

## Compatibility notes

This Rust JSON-LD model is intended to stay aligned with existing Fluree docs:
- `../db/docs/S3_STORAGE_GUIDE.md`
- `../db/docs/FILE_STORAGE_GUIDE.md`
- `../db/docs/DYNAMODB_NAMESERVICE_GUIDE.md`

Current intentional gaps in Rust:
- `remoteSystems` not supported
- `defaults.identity` is parsed but not currently applied
- `defaults.indexing.trackClassStats` is parsed but not currently applied



---

# reference/compatibility.md

# Standards and Feature Flags

This document covers Fluree's compliance with standards and feature flags.

## Standards Compliance

### RDF 1.1

**Status:** Fully compliant

Fluree implements the W3C RDF 1.1 specification:
- RDF triples (subject-predicate-object)
- IRI identifiers
- Typed literals
- Language tags
- Blank nodes
- RDF datasets

**Specification:** https://www.w3.org/TR/rdf11-concepts/

### JSON-LD 1.1

**Status:** Fully compliant

Fluree supports JSON-LD 1.1:
- @context for namespace mappings
- @id for resource identification
- @type for type specification
- @graph for multiple entities
- @value and @type for literals
- @language for language tags
- Nested objects
- Arrays

**Specification:** https://www.w3.org/TR/json-ld11/

### SPARQL 1.1 Query

**Status:** In progress toward full compliance

Supported SPARQL features:
- SELECT queries
- CONSTRUCT queries
- ASK queries
- DESCRIBE queries
- FROM and FROM NAMED clauses
- GRAPH patterns
- OPTIONAL patterns
- UNION patterns
- FILTER expressions
- BIND expressions
- Aggregations (COUNT, SUM, AVG, MIN, MAX, SAMPLE, GROUP_CONCAT) with DISTINCT modifier
- GROUP BY (variables and expressions)
- ORDER BY
- LIMIT and OFFSET
- Subqueries
- Property paths (partial: `+`, `*`, `^`, `|`, `/`; see [SPARQL docs](../query/sparql.md#property-paths))

**Aggregate result types:** COUNT and SUM of integers return `xsd:integer` (per W3C spec), not `xsd:long`. SUM of mixed types and AVG return `xsd:double`.

**W3C Compliance Testing:** Fluree runs the official W3C SPARQL test suite via the `testsuite-sparql` crate. The suite automatically discovers and runs 700+ test cases from W3C manifest files. See the [compliance test guide](../contributing/sparql-compliance.md) for details.

**Specification:** https://www.w3.org/TR/sparql11-query/

### SPARQL 1.1 Update

**Status:** Partial support

Supported:
- INSERT DATA (via JSON-LD transactions)
- DELETE/INSERT WHERE (via WHERE/DELETE/INSERT)

Not yet supported:
- DELETE DATA
- LOAD
- CLEAR
- DROP
- COPY, MOVE, ADD

Use JSON-LD transactions for transaction operations.

**Specification:** https://www.w3.org/TR/sparql11-update/

### Turtle

**Status:** Fully supported

Fluree parses Turtle 1.1:
- @prefix declarations
- Base IRIs
- Abbreviated syntax (a, ;, ,)
- Literals with datatypes and language tags
- Collections
- Blank nodes

**Specification:** https://www.w3.org/TR/turtle/

### JSON Web Signature (JWS)

**Status:** Partial (EdDSA only)

Supported algorithms:
- EdDSA (Ed25519) - **Only supported algorithm**

Not yet supported:
- ES256, ES384, ES512 (ECDSA)
- RS256 (RSA)
- HS256, HS384, HS512 (HMAC)

**Specification:** RFC 7515

**Note:** Requires the `credential` feature flag.

### Verifiable Credentials

**Status:** Planned (not yet implemented)

The credential module currently supports JWS verification only. Full VC support
(proof verification, JSON-LD canonicalization) is planned but not yet available.

**Specification:** https://www.w3.org/TR/vc-data-model/

### Decentralized Identifiers (DIDs)

**Status:** Partial support

Supported DID methods:
- did:key (Ed25519 keys only)

Not yet supported:
- did:web
- did:ion
- did:ethr

**Specification:** https://www.w3.org/TR/did-core/

**Note:** Requires the `credential` feature flag.

## Compile-Time Feature Flags (Cargo)

These features are controlled at compile time via Cargo:

### `fluree-db-api` Features

| Feature | Default | Description |
|---------|---------|-------------|
| `native` | Yes | File storage support |
| `aws` | No | AWS-backed storage support (S3, storage-backed nameservice). Enables `FlureeBuilder::s3()` and S3-based JSON-LD configs. |
| `credential` | No | DID/JWS/VerifiableCredential support for signed queries/transactions. Pulls in crypto dependencies (`ed25519-dalek`, `bs58`). |
| `iceberg` | No | Apache Iceberg/R2RML graph source support |
| `shacl` | No | SHACL constraint validation (requires fluree-db-transact + fluree-db-shacl). Default in server/CLI. |
| `vector` | No | Embedded vector similarity search (HNSW indexes via usearch) |
| `ipfs` | No | IPFS-backed storage via Kubo HTTP RPC |
| `search-remote-client` | No | HTTP client for remote BM25 and vector search services |
| `aws-testcontainers` | No | Opt-in LocalStack-backed S3/DynamoDB tests (auto-start via testcontainers) |
| `full` | No | Convenience bundle: `native`, `credential`, `iceberg`, `shacl`, `ipfs` |

Example:
```toml
[dependencies]
fluree-db-api = { path = "../fluree-db-api", features = ["native", "credential"] }
```

### `fluree-db-server` Features

| Feature | Default | Description |
|---------|---------|-------------|
| `native` | Yes | File storage support (forwards to `fluree-db-api/native`) |
| `credential` | Yes | Signed request verification (forwards to `fluree-db-api/credential`) |
| `shacl` | Yes | SHACL constraint validation (forwards to `fluree-db-api/shacl`) |
| `iceberg` | Yes | Apache Iceberg/R2RML graph source support (forwards to `fluree-db-api/iceberg`) |
| `aws` | No | AWS S3 storage + DynamoDB nameservice (forwards to `fluree-db-api/aws`) |
| `oidc` | No | OIDC JWT verification via JWKS (RS256 tokens from external IdPs) |
| `swagger-ui` | No | Swagger UI endpoint |
| `otel` | No | OpenTelemetry tracing |

To build the server without credential support (faster compile):
```bash
cargo build -p fluree-db-server --no-default-features --features native
```

## Runtime Behavior

Reasoning, SPARQL property paths, and GeoSPARQL functions are always
available in any build that links the corresponding crate features (see
the build-time feature tables above). They are not gated behind a runtime
flag.

Reasoning is opted into per query (via the `reasoning` parameter or the
SPARQL `PRAGMA reasoning` directive) or per ledger (via
`f:reasoningDefaults` in the ledger configuration graph). See
[Query-time reasoning](../query/reasoning.md) and
[Setting groups](../ledger-config/setting-groups.md).

## Parsing Modes

### Strict Mode (Default)

Enforces strict compliance with standards:
- Invalid IRIs rejected
- Type mismatches rejected
- Strict JSON-LD parsing

```bash
./fluree-db-server --strict-mode true
```

### Lenient Mode

More permissive parsing:
- Attempts to fix malformed IRIs
- Coerces types when possible
- Accepts non-standard syntax

```bash
./fluree-db-server --strict-mode false
```

Use lenient mode only when you fully control inputs and explicitly want permissive parsing behavior.

## API Versioning

Current API version: v1

**Version Header:**
```http
X-Fluree-API-Version: 1
```

## Supported Data Formats

### JSON-LD

Supported JSON-LD versions:
- JSON-LD 1.0: Yes
- JSON-LD 1.1: Yes

### SPARQL

Supported SPARQL versions:
- SPARQL 1.0: Yes
- SPARQL 1.1: Yes

### RDF Formats

| Format | Read | Write |
|--------|------|-------|
| JSON-LD | Yes | Yes |
| Turtle | Yes | Yes |
| N-Triples | Planned | Planned |
| N-Quads | Planned | Planned |
| RDF/XML | Planned | No |
| TriG | Planned | Planned |

## Protocol Support

### HTTP Versions

- HTTP/1.1: Fully supported
- HTTP/2: Supported
- HTTP/3: Planned

### TLS Versions

- TLS 1.2: Supported
- TLS 1.3: Supported
- SSL 3.0: Not supported (deprecated)
- TLS 1.0/1.1: Not supported (deprecated)

## Client Support

Fluree works with:

**HTTP Clients:**
- curl
- Postman
- Insomnia
- Any HTTP client library

**RDF Libraries:**
- Apache Jena (Java)
- RDF4J (Java)
- rdflib (Python)
- N3.js (JavaScript)

**SPARQL Clients:**
- Apache Jena ARQ
- RDF4J SPARQLRepository
- Any SPARQL 1.1 client

## Platform Support

### Operating Systems

**Server:**
- Linux (x86_64, aarch64)
- macOS (Intel, Apple Silicon)
- Windows (x86_64)

**Clients:**
- Any OS with HTTP support

### Cloud Platforms

- AWS (native support)
- Google Cloud Platform (via file storage)
- Azure (via file storage)
- Self-hosted / on-premises

### Container Support

- Docker: Full support
- Kubernetes: Full support
- Podman: Supported
- Docker Compose: Full support

## Database Support

### Import Sources

Fluree can import from:

**RDF Databases:**
- Apache Jena TDB
- Virtuoso
- Stardog
- GraphDB
- Any RDF export

**Graph Databases:**
- Neo4j (via RDF export)
- Amazon Neptune (via RDF export)

**Relational Databases:**
- Via R2RML mapping
- Direct SQL query

### Export Formats

Export Fluree data to:
- Turtle files
- JSON-LD documents
- SPARQL CONSTRUCT results
- Any RDF format

## Feature Roadmap

### Planned Features

**Query:**
- SPARQL property paths: remaining operators (`?` zero-or-one, `!` negated set)
- GeoSPARQL
- SPARQL 1.1 Federation
- Full SPARQL UPDATE

**Storage:**
- Additional cloud providers (GCP, Azure)
- Hybrid storage modes

**Security:**
- OAuth 2.0 integration
- SAML support
- Additional DID methods

**Graph Sources:**
- BigQuery integration
- Snowflake integration
- Elasticsearch integration

### Feature Discovery

Feature availability is documented in this compatibility matrix and by
crate feature flags; the standalone server does not expose a `/features`
HTTP endpoint.

## Browser Support

For web applications using Fluree API:

**Supported Browsers:**
- Chrome/Edge 90+
- Firefox 88+
- Safari 14+

**Requirements:**
- Fetch API support
- CORS support
- WebSocket support (for future streaming)

## Tool Support

### RDF Tools

Compatible with standard RDF tools:
- Protégé (ontology editor)
- TopBraid Composer
- RDF validators
- SPARQL editors

### Data Tools

Works with data engineering tools:
- Apache Airflow (via HTTP operators)
- dbt (via SQL proxy with R2RML)
- Apache Spark (via Iceberg)
- Pandas (via query API)

## Version Requirements

### Rust Version

Building from source requires:
- Rust 1.75.0 or later
- Cargo 1.75.0 or later

### Dependencies

Runtime dependencies:
- None (statically linked binary)

Optional dependencies:
- AWS SDK (for AWS storage)

## Related Documentation

- [Glossary](glossary.md) - Term definitions
- [Crate Map](crate-map.md) - Code architecture
- [Getting Started](../getting-started/README.md) - Installation


---

# reference/graph-identities.md

# Graph Identities and Naming
This document defines **names → things** and recommended naming conventions for Fluree.
It is split into:
- **User-facing naming**: what we say in docs, examples, and APIs.
- **Internal naming**: how we name types/components in Rust so the implementation stays clear.
\
The goal is to make these simultaneously true:
- Users can think in the familiar model: **“database as a value”** (immutable, time-travelable).
- SPARQL semantics remain correct: `GRAPH <…>` identifies a **graph by IRI**.
- Fluree can seamlessly query across:
  - graphs inside a ledger (default + named graphs),
  - across ledgers (federation),
  - and non-ledger sources (BM25, vector, Iceberg/R2RML, etc.).

## Summary: the model in one paragraph
In Fluree, you query **graphs** and often load a **graph snapshot** (an immutable point-in-time view you can query repeatedly). In SPARQL, graph scoping uses `GRAPH <iri> { … }`, where `<iri>` is a **graph identifier** (a graph IRI). Fluree supports multiple kinds of **graph sources** (ledger graphs and non-ledger sources like BM25/vector indexes and tabular mappings). Users may refer to graphs with short, friendly **aliases** that Fluree resolves against a configured base into canonical **graph IRIs**. Not all graph sources support the same time-travel semantics — time pinning and “as-of” behavior is a **graph-source capability**, not a universal guarantee.

Under the hood, this “graph snapshot” corresponds to the same semantic idea many temporal systems describe as **“database as a value”**: immutable, time-travelable, and safe to pass around.

---

## User-facing naming (recommended)

### Core terms
- **Ledger**: A durable data product with a commit chain, identified by a *ledger ID* like `mydb:main`.
  - A ledger is what users create/manage.
  - A ledger can contain multiple graphs (default graph + named graphs).
- **Graph Source ID**: A canonical `name:branch` identifier used in APIs/CLI/config to refer to a graph source, e.g. `products-search:main`.
  - This is an *alias-style* identifier (not a full IRI).
  - In SPARQL contexts it may appear inside `<…>` and can be resolved against a configured base into a canonical Graph IRI.
- **Graph**: A query scope (SPARQL term).
  - In a query, a “graph” is identified by an IRI and used to scope patterns (`GRAPH <iri> { … }`).
- **Graph Snapshot**: An immutable point-in-time view of a graph that can be queried repeatedly.
- **LedgerSnapshot (database value)**: The underlying semantic model: an immutable value at a point in time.
  - In product/docs we usually say “graph snapshot” because it aligns with SPARQL and the Rust API.
  - Internally the type is `LedgerSnapshot` in `fluree-db-core`.
- **Graph IRI**: The canonical identity of a graph. This is what SPARQL uses.
- **Graph reference (GraphRef)**: What a user types (often an alias-like string), which Fluree resolves to a Graph IRI.
- **Graph Source**: Anything addressable by a Graph IRI that can participate in query execution.
- **Federation** (preferred) / **Dataset** (SPARQL term): A query executed over a set of graphs.
  - We prefer “federation” when describing the product feature to non-SPARQL users.

### “Graph snapshot” vs “Graph IRI” (how to talk about it)
- **Graph Snapshot (value)** answers: “What immutable point-in-time graph am I querying?”
- **Graph IRI (identifier)** answers: “Which graph does this part of the query run against?”

In practice, you query a graph snapshot by naming its graph:
- When you write `FROM <…>` or `GRAPH <…>`, you are naming a **graph IRI**.
- That graph IRI resolves to a **graph snapshot** (an immutable value) at execution time.

### Time pinning syntax (“the part after `@` pins the snapshot”)
Fluree supports time pinning in graph references.

**Current syntax (implemented today):**
- `<ledger>:<branch>@t:<t>` — pin to transaction time
- `<ledger>:<branch>@iso:<rfc3339>` — pin to ISO datetime
- `<ledger>:<branch>@commit:<commit-content-id>` — pin to commit ContentId (prefix allowed)

Note: you may see an `=` form in older design notes (`@t=100`, etc.). That form is **not** the supported user-facing syntax today; use the `@t:` / `@iso:` / `@commit:` forms in docs and examples.

From a user perspective:
- The `@…` portion selects **which snapshot value** you mean for that ledger graph.

Important nuance:
- For **ledger graph sources**, `@…` selects a pinned point-in-time view.
- For **non-ledger graph sources**, `@…` support is **capability-specific**:
  - Some sources may support time pinning by selecting an appropriate snapshot/root.
  - Some sources are **head-only** and should reject time-pinned requests with a clear error.

### Named graphs within a ledger
We support multiple named graphs inside a single ledger (shared commit chain, distinct graph identities/indexes).

#### System graphs

Every ledger reserves system named graphs for internal use:

| Graph | IRI pattern | Purpose |
|-------|-------------|---------|
| Default graph | (implicit) | Application data |
| Txn-meta | `urn:fluree:{ledger_id}#txn-meta` | Commit metadata |
| Config graph | `urn:fluree:{ledger_id}#config` | Ledger configuration |

User-defined named graphs (created via TriG) are identified by their IRI and allocated after the system graphs.

The **config graph** stores ledger-level operational defaults (policy, SHACL, reasoning, uniqueness constraints) as RDF triples. See [Ledger configuration](../ledger-config/README.md) for details.

#### Naming convention

Recommended user-facing convention (alias-friendly, URL-friendly, avoids `/` as a delimiter inside the ledger namespace):

```
<ledger>:<branch>[ @time-spec ] [ #<named-graph-alias> ]
```

Examples:
- Default graph, latest: `<acme/people:main>`
- Default graph at \(t=1000\): `<acme/people:main@t:1000>`
- Txn metadata graph at latest: `<acme/people:main#txn-meta>`
- Txn metadata graph at \(t=1000\): `<acme/people:main@t:1000#txn-meta>`

#### Important note about `#` fragments
Using `#<named-graph-alias>` is idiomatic RDF identity, but **HTTP clients do not send fragments to servers**.
That’s fine for graph identity and query semantics, but if you want a dereferenceable HTTP endpoint for a named graph,
plan to expose a server-visible selector (e.g., `?graph=txn-meta`) in addition to the canonical identity.

### Full IRIs are always allowed
Semantic web users may prefer full IRIs:
- `https://data.flur.ee/acme/people:main@t:1000#txn-meta`

These should be used as-is (no resolution needed).

### Base resolution (“make aliases globally identifiable”)
Many users prefer short names like `people:main` or `acme/people:main`. To make them globally identifiable:
- Allow a configured base (SPARQL `BASE <…>` or a connection/query base configuration).
- Treat alias-style graph references as **relative IRI references** resolved against that base.

Example:
- Base: `https://data.flur.ee/`
- Ref: `<acme/people:main@t:1000#txn-meta>`
- Graph IRI: `https://data.flur.ee/acme/people:main@t:1000#txn-meta`

### Character and encoding rules (user-facing)
To avoid ambiguity and URL pitfalls:
- **Reserved delimiters**:
  - `@` separates the time specifier
  - `#` separates a named-graph alias (fragment)
  - `:` is used inside the ledger ID as `ledger:branch`
- **Do not use raw `@` or `#` inside ledger names, branch names, or named-graph aliases**.
  - If needed, percent-encode them.
- RFC3339 / ISO timestamps must be URL-safe:
  - Prefer UTC with `Z` (e.g., `2026-02-03T17:02:11Z`).
  - If offsets are used (`+05:00`), they should be percent-encoded in IRI contexts.

---

## Graph Sources (user-facing)

We use **Graph Source** as the umbrella term for anything you can name in `FROM`, `FROM NAMED`, or `GRAPH <…>` and query as part of a single execution.

Graph sources differ in capabilities:
- Some behave like RDF triple stores (ledger graphs, some mappings).
- Some provide specialized operators/patterns (BM25 and vector search).
- Some support time pinning / time travel; others are head-only.

### Categories
- **Ledger Graph Sources**: RDF graphs stored in a ledger (default graph or named graph).
- **Index Graph Sources**: persisted indexes queried through graph-integrated patterns (BM25, Vector/HNSW).
- **Mapped Graph Sources**: non-ledger data mapped into an RDF-shaped graph (R2RML, Iceberg).

---

## Conventions and examples

### SPARQL: base + pinned graphs
```sparql
BASE <https://data.flur.ee/>

SELECT ?s ?p ?o
FROM <acme/people:main@t:1000>
WHERE {
  ?s ?p ?o .
}
```

### SPARQL: txn metadata named graph
```sparql
BASE <https://data.flur.ee/>

SELECT ?commit ?t
FROM NAMED <acme/people:main@t:1000#txn-meta>
WHERE {
  GRAPH <acme/people:main@t:1000#txn-meta> {
    ?commit <https://ns.flur.ee/db#t> ?t .
  }
}
```

### JSON-LD Query: pinned graph reference
```json
{
  "from": "acme/people:main@t:1000",
  "select": ["?name"],
  "where": [{ "@id": "?p", "http://schema.org/name": "?name" }]
}
```

---

## Internal naming conventions (Rust code)

These conventions govern how we name types and variables in Rust code related to graphs, ledgers, and identifiers.

### Canonical identities: prefer explicit types, not raw `String`
Internally we should distinguish:
- **`LedgerId`**: the durable ledger identifier (e.g., `acme/people:main`)
- **`GraphIri`**: canonical graph identity used by SPARQL (`Arc<str>` or validated URL/IRI type)
- **`GraphRef`**: user input token, resolved to a `GraphIri` using base rules

Even if these are initially just `struct LedgerId(Arc<str>)` newtypes, they prevent accidental mixing and make APIs self-documenting.

### Naming rules (make each word mean one thing)
This repo reserves `_id` for **content identifiers** (e.g. `commit_id`, `index_id`, `default_context_id`) and **identity/lookup keys** (`name:branch` tokens). The recommended rule set:

- **`id`**: canonical identifier used as a cache key / stable identity.
  - For ledgers this is the full `name:branch` form (e.g., `people:main`).
  - For graph sources this is the full `name:branch` form (e.g., `products-search:main`).
- **`_id` (for content references)**: a content identifier (ContentId) used by the storage layer to fetch immutable artifacts.
  - Examples: `commit_id`, `index_id`, `default_context_id`.
- **`name`**: a base name without branch (e.g., `people`).
- **`branch`**: the branch name (e.g., `main`).
- **`alias`**: a human-friendly label, and only that. For ledger identifiers, prefer `ledger_id`.

Practical guidelines:
- If a string is used to load/cache/lookup a ledger, call it **`ledger_id`** (not `ledger_alias`, not `ledger_address`).
- If a string is used to load/cache/lookup a graph source by `name:branch`, call it **`graph_source_id`** (not `gs_alias`, not `graph_source_alias`).
- If a value identifies a content-addressed artifact (commit, index, context), call it **`*_id`** (e.g., `commit_id`, `index_id`, `default_context_id`).
- If a string is used to identify a graph in SPARQL (`FROM`, `GRAPH`), call it **`graph_iri`** (canonical) or **`graph_ref`** (user input).
- Avoid having two different meanings for the same field name across crates.

### Ledger vs Db vs Graph (internal meaning)
- **Ledger**: the durable identity + commit chain + publication state (nameservice record).
- **LedgerSnapshot**: the indexed snapshot value used for range/scan (hot path).
- **Graph**: query scoping mechanism (active graph, dataset graph selection, `GRAPH` operator).

Avoid using "graph" as a synonym for "db" in code comments. Prefer:
- "ledger graph" (RDF graph inside a ledger)
- "graph IRI" (identifier)
- "graph source" (registry/resolution concept)

### "Dataset" naming internally
SPARQL calls the set-of-graphs a "dataset", and the code already models a `DataSet`.
For product-facing APIs and docs, prefer "federation", but internally keep `DataSet` as the SPARQL-aligned term.

---

## Related docs
- `docs/concepts/time-travel.md` (time pinning syntax)
- `docs/concepts/datasets-and-named-graphs.md` (SPARQL dataset semantics)
- `docs/graph-sources/overview.md` and `docs/concepts/graph-sources.md` (graph source overview)
- `docs/reference/glossary.md` (terms glossary)



---

# reference/owl-rdfs-support.md

# OWL & RDFS Support Reference

This page lists every OWL and RDFS construct that Fluree's reasoning engine
supports. For conceptual background see
[Reasoning and inference](../concepts/reasoning.md); for query syntax see
[Query-time reasoning](../query/reasoning.md).

## Quick orientation

Fluree implements reasoning via two techniques:

- **Query rewriting** (RDFS and OWL 2 QL modes) — patterns are expanded at
  compile time; no facts are materialized.
- **Forward-chaining materialization** (OWL 2 RL mode) — derived facts are
  computed before query execution using the OWL 2 RL rule set.

The tables below indicate which technique handles each construct.

---

## RDFS constructs

These constructs are handled by **query rewriting** in RDFS mode (and also by
materialization in OWL 2 RL mode).

### rdfs:subClassOf

Declares that every instance of one class is also an instance of another.

```turtle
ex:Student  rdfs:subClassOf  ex:Person .
```

**Effect:** A query for `?x rdf:type ex:Person` also returns instances typed as
`ex:Student` (and any subclass of `Student`, transitively).

**JSON-LD transaction:**
```json
{"@id": "ex:Student", "rdfs:subClassOf": {"@id": "ex:Person"}}
```

### rdfs:subPropertyOf

Declares that one property is a specialization of another.

```turtle
ex:hasMother  rdfs:subPropertyOf  ex:hasParent .
```

**Effect:** A query for `?x ex:hasParent ?y` also returns triples asserted with
`ex:hasMother`.

**JSON-LD transaction:**
```json
{"@id": "ex:hasMother", "rdfs:subPropertyOf": {"@id": "ex:hasParent"}}
```

### rdfs:domain

Declares that the subject of a property is an instance of a class.

```turtle
ex:teaches  rdfs:domain  ex:Professor .
```

**Effect (OWL 2 QL / OWL 2 RL):** If `ex:alice ex:teaches ex:cs101`, then
`ex:alice rdf:type ex:Professor` is inferred.

**JSON-LD transaction:**
```json
{"@id": "ex:teaches", "rdfs:domain": {"@id": "ex:Professor"}}
```

### rdfs:range

Declares that the object of a property is an instance of a class.

```turtle
ex:teaches  rdfs:range  ex:Course .
```

**Effect (OWL 2 QL / OWL 2 RL):** If `ex:alice ex:teaches ex:cs101`, then
`ex:cs101 rdf:type ex:Course` is inferred.

**JSON-LD transaction:**
```json
{"@id": "ex:teaches", "rdfs:range": {"@id": "ex:Course"}}
```

> **Note:** Range inference applies to IRI-valued objects only. Literal values
> (strings, numbers, etc.) are not assigned a type via `rdfs:range`.

---

## OWL property constructs

These are handled by **materialization** in OWL 2 RL mode (some also by query
rewriting in OWL 2 QL mode, as noted).

### owl:inverseOf

Declares that two properties are inverses of each other.

```turtle
ex:hasMother  owl:inverseOf  ex:motherOf .
```

**Effect:** If `ex:alice ex:hasMother ex:carol`, then
`ex:carol ex:motherOf ex:alice` is inferred (and vice versa).

**Handled by:** OWL 2 QL (query rewriting) *and* OWL 2 RL (materialization).

**OWL 2 RL rule:** `prp-inv`

**JSON-LD transaction:**
```json
{"@id": "ex:hasMother", "owl:inverseOf": {"@id": "ex:motherOf"}}
```

### owl:SymmetricProperty

Declares that a property holds in both directions.

```turtle
ex:livesWith  a  owl:SymmetricProperty .
```

**Effect:** If `ex:alice ex:livesWith ex:bob`, then
`ex:bob ex:livesWith ex:alice` is inferred.

**OWL 2 RL rule:** `prp-symp`

**JSON-LD transaction:**
```json
{"@id": "ex:livesWith", "@type": "owl:SymmetricProperty"}
```

### owl:TransitiveProperty

Declares that a property chains through intermediate nodes.

```turtle
ex:hasAncestor  a  owl:TransitiveProperty .
```

**Effect:** If `ex:a ex:hasAncestor ex:b` and `ex:b ex:hasAncestor ex:c`, then
`ex:a ex:hasAncestor ex:c` is inferred.

**OWL 2 RL rule:** `prp-trp`

**JSON-LD transaction:**
```json
{"@id": "ex:hasAncestor", "@type": "owl:TransitiveProperty"}
```

### owl:FunctionalProperty

Declares that a property can have at most one value per subject.

```turtle
ex:hasBirthDate  a  owl:FunctionalProperty .
```

**Effect:** If `ex:alice ex:hasBirthDate ex:d1` and
`ex:alice ex:hasBirthDate ex:d2`, then `ex:d1 owl:sameAs ex:d2` is inferred.

**OWL 2 RL rule:** `prp-fp`

**JSON-LD transaction:**
```json
{"@id": "ex:hasBirthDate", "@type": "owl:FunctionalProperty"}
```

### owl:InverseFunctionalProperty

Declares that a property's object uniquely identifies the subject.

```turtle
ex:hasSSN  a  owl:InverseFunctionalProperty .
```

**Effect:** If `ex:alice ex:hasSSN "123"` and `ex:bob ex:hasSSN "123"`, then
`ex:alice owl:sameAs ex:bob` is inferred.

**OWL 2 RL rule:** `prp-ifp`

**JSON-LD transaction:**
```json
{"@id": "ex:hasSSN", "@type": "owl:InverseFunctionalProperty"}
```

### owl:equivalentProperty

Declares that two properties have identical extensions.

```turtle
ex:author  owl:equivalentProperty  ex:writtenBy .
```

**Effect:** Treated as mutual `rdfs:subPropertyOf` — queries and rules see both
properties' triples when either is used.

### owl:propertyChainAxiom

Declares that a chain of properties implies another property.

```turtle
ex:hasUncle  owl:propertyChainAxiom  ( ex:hasParent  ex:hasBrother ) .
```

**Effect:** If `ex:alice ex:hasParent ex:bob` and
`ex:bob ex:hasBrother ex:charlie`, then `ex:alice ex:hasUncle ex:charlie` is
inferred.

**OWL 2 RL rule:** `prp-spo2`

Chains can be of arbitrary length (2 or more properties) and can include
inverse elements:

```turtle
ex:hasNephew  owl:propertyChainAxiom  (
    [ owl:inverseOf ex:hasBrother ]
    ex:hasChild
) .
```

**JSON-LD transaction:**
```json
{
  "@id": "ex:hasUncle",
  "owl:propertyChainAxiom": {
    "@list": [{"@id": "ex:hasParent"}, {"@id": "ex:hasBrother"}]
  }
}
```

---

## OWL class constructs

### owl:equivalentClass

Declares that two classes have identical extensions.

```turtle
ex:Pupil  owl:equivalentClass  ex:Student .
```

**Effect:** Instances of either class are inferred to be instances of both.

**OWL 2 RL rule:** `cax-eqc`

### owl:hasKey

Declares a set of properties that uniquely identify instances of a class.

```turtle
ex:Person  owl:hasKey  ( ex:hasSSN ) .
```

**Effect:** If two `ex:Person` instances share the same `ex:hasSSN` value, they
are inferred to be `owl:sameAs`.

**OWL 2 RL rule:** `prp-key`

---

## OWL restrictions (class expressions)

OWL restrictions define classes based on property constraints. They are used with
OWL 2 RL materialization.

### owl:hasValue

Defines a class of all subjects that have a specific value for a property.

```turtle
ex:RedThings  a  owl:Restriction ;
    owl:onProperty  ex:color ;
    owl:hasValue     ex:Red .
```

**Effect (forward — cls-hv1):** If `?x rdf:type ex:RedThings`, then
`?x ex:color ex:Red` is inferred.

**Effect (backward — cls-hv2):** If `?x ex:color ex:Red`, then
`?x rdf:type ex:RedThings` is inferred.

> **Limitation:** Currently supports IRI-valued `hasValue` only. Literal values
> (strings, numbers) are not yet supported.

### owl:someValuesFrom

Defines a class of subjects that have at least one value of a given type for a
property.

```turtle
ex:Parent  a  owl:Restriction ;
    owl:onProperty      ex:hasChild ;
    owl:someValuesFrom  ex:Person .
```

**Effect (cls-svf1):** If `?x ex:hasChild ?y` and `?y rdf:type ex:Person`,
then `?x rdf:type ex:Parent` is inferred.

### owl:allValuesFrom

Defines a class where all values of a property belong to a given type.

```turtle
ex:VeganRestaurant  a  owl:Restriction ;
    owl:onProperty     ex:servesFood ;
    owl:allValuesFrom  ex:VeganDish .
```

**Effect (cls-avf):** If `?x rdf:type ex:VeganRestaurant` and
`?x ex:servesFood ?y`, then `?y rdf:type ex:VeganDish` is inferred.

### owl:maxCardinality (= 1)

When a restriction specifies `maxCardinality` of 1, it acts like a
context-specific functional property.

```turtle
ex:SingleChild  a  owl:Restriction ;
    owl:onProperty      ex:hasChild ;
    owl:maxCardinality  1 .
```

**Effect (cls-maxc2):** If `?x rdf:type ex:SingleChild`,
`?x ex:hasChild ?y1`, and `?x ex:hasChild ?y2`, then
`?y1 owl:sameAs ?y2` is inferred.

### owl:maxQualifiedCardinality (= 1)

Like `maxCardinality` but restricted to objects of a specific class.

```turtle
ex:MonogamousPerson  a  owl:Restriction ;
    owl:onProperty                  ex:marriedTo ;
    owl:maxQualifiedCardinality     1 ;
    owl:onClass                     ex:Person .
```

**Effect (cls-maxqc3/4):** If `?x` is typed as this restriction class, has two
`ex:marriedTo` values, and both are `ex:Person`, they are inferred to be
`owl:sameAs`.

---

## OWL class operations

### owl:intersectionOf

Defines a class as the intersection of member classes.

```turtle
ex:WorkingStudent  owl:intersectionOf  ( ex:Student  ex:Employee ) .
```

**Effect (forward — cls-int1):** If `?x rdf:type ex:Student` and
`?x rdf:type ex:Employee`, then `?x rdf:type ex:WorkingStudent` is inferred.

**Effect (backward — cls-int2):** If `?x rdf:type ex:WorkingStudent`, then
both `?x rdf:type ex:Student` and `?x rdf:type ex:Employee` are inferred.

### owl:unionOf

Defines a class as the union of member classes.

```turtle
ex:PersonOrOrg  owl:unionOf  ( ex:Person  ex:Organization ) .
```

**Effect (cls-uni):** If `?x rdf:type ex:Person` (or `ex:Organization`), then
`?x rdf:type ex:PersonOrOrg` is inferred.

### owl:oneOf

Defines an enumerated class — a fixed set of individuals.

```turtle
ex:PrimaryColor  owl:oneOf  ( ex:Red  ex:Blue  ex:Yellow ) .
```

**Effect (cls-oo):** `ex:Red`, `ex:Blue`, and `ex:Yellow` are each inferred to
be of type `ex:PrimaryColor`.

---

## owl:sameAs

`owl:sameAs` declares that two IRIs refer to the same real-world entity.

```turtle
ex:alice  owl:sameAs  ex:aliceSmith .
```

**Effect:** All facts about `ex:alice` and `ex:aliceSmith` are merged. Queries
for either IRI return the combined set of properties.

### How sameAs is produced

`owl:sameAs` can be asserted explicitly or inferred by these rules:

| Rule | Trigger |
|------|---------|
| `prp-fp` | Functional property with multiple objects |
| `prp-ifp` | Inverse functional property with multiple subjects |
| `prp-key` | owl:hasKey match across instances |
| `cls-maxc2` | maxCardinality = 1 violation |
| `cls-maxqc3/4` | maxQualifiedCardinality = 1 violation |

### Equivalence properties

`owl:sameAs` is handled as an equivalence relation:
- **Symmetric:** if `a sameAs b` then `b sameAs a`
- **Transitive:** if `a sameAs b` and `b sameAs c` then `a sameAs c`
- **Reflexive:** every resource is same-as itself (implicit)

The engine uses a union-find data structure to efficiently track equivalence
classes and select a canonical representative for each.

---

## OWL 2 RL rule index

For reference, the complete set of OWL 2 RL rules implemented by Fluree:

### Identity-producing rules (Phase B)

These rules produce `owl:sameAs` facts and run before other rules to ensure
proper canonicalization.

| Rule | Construct | Description |
|------|-----------|-------------|
| `prp-fp` | `owl:FunctionalProperty` | Same subject + different objects → sameAs |
| `prp-ifp` | `owl:InverseFunctionalProperty` | Same object + different subjects → sameAs |
| `prp-key` | `owl:hasKey` | Same class + matching key values → sameAs |
| `cls-maxc2` | `owl:maxCardinality = 1` | Over-cardinality → sameAs |
| `cls-maxqc3` | `owl:maxQualifiedCardinality = 1` | Qualified over-cardinality → sameAs |
| `cls-maxqc4` | `owl:maxQualifiedCardinality = 1` | Variant for `owl:Thing` |

### Non-identity rules (Phase C)

| Rule | Construct | Description |
|------|-----------|-------------|
| `prp-symp` | `owl:SymmetricProperty` | P(x,y) → P(y,x) |
| `prp-trp` | `owl:TransitiveProperty` | P(x,y) ∧ P(y,z) → P(x,z) |
| `prp-inv` | `owl:inverseOf` | P(x,y) → Q(y,x) |
| `prp-dom` | `rdfs:domain` | P(x,y) → type(x,C) |
| `prp-rng` | `rdfs:range` | P(x,y) → type(y,C) |
| `prp-spo1` | `rdfs:subPropertyOf` | P1(x,y) → P2(x,y) |
| `prp-spo2` | `owl:propertyChainAxiom` | Chain match → P(first,last) |
| `cax-sco` | `rdfs:subClassOf` | type(x,C1) → type(x,C2) |
| `cax-eqc` | `owl:equivalentClass` | type(x,C1) ↔ type(x,C2) |
| `cls-hv1` | `owl:hasValue` (backward) | type(x,C) → P(x,v) |
| `cls-hv2` | `owl:hasValue` (forward) | P(x,v) → type(x,C) |
| `cls-svf1` | `owl:someValuesFrom` | P(x,y) ∧ type(y,D) → type(x,C) |
| `cls-avf` | `owl:allValuesFrom` | type(x,C) ∧ P(x,y) → type(y,D) |
| `cls-int1` | `owl:intersectionOf` (forward) | All member types → intersection type |
| `cls-int2` | `owl:intersectionOf` (backward) | Intersection type → all member types |
| `cls-uni` | `owl:unionOf` | Any member type → union type |
| `cls-oo` | `owl:oneOf` | Listed individual → enumerated type |

---

## Known limitations

| Area | Limitation |
|------|-----------|
| **Literal hasValue** | `owl:hasValue` with literal values (strings, numbers) is not yet supported; only IRI-valued restrictions work. |
| **Negation** | `owl:complementOf` and negation-as-failure are not supported. OWL 2 RL is a positive-only fragment. |
| **Disjointness** | `owl:disjointWith` and `owl:AllDisjointClasses` do not trigger inconsistency detection. |
| **Cardinality > 1** | Only `maxCardinality = 1` and `maxQualifiedCardinality = 1` are implemented (these are the only identity-producing cardinalities in OWL 2 RL). |
| **Datatype reasoning** | No inference over datatypes (e.g., `xsd:integer` subtype of `xsd:decimal`). |

## Namespaces

For reference, the standard namespace prefixes:

| Prefix | URI |
|--------|-----|
| `rdf` | `http://www.w3.org/1999/02/22-rdf-syntax-ns#` |
| `rdfs` | `http://www.w3.org/2000/01/rdf-schema#` |
| `owl` | `http://www.w3.org/2002/07/owl#` |
| `xsd` | `http://www.w3.org/2001/XMLSchema#` |


---

# reference/crate-map.md

# Crate Map

Fluree is organized into multiple Rust crates, each with a specific purpose. This document provides an overview of the crate architecture and dependencies.

## Crate Organization

```text
fluree-db/
├── Core
│   ├── fluree-vocab/              # RDF vocabulary constants and namespace codes
│   ├── fluree-db-core/            # Runtime-agnostic core types and queries
│   └── fluree-db-novelty/         # Novelty overlay and commit types
│
├── Graph Processing
│   ├── fluree-graph-ir/           # Format-agnostic RDF intermediate representation
│   ├── fluree-graph-json-ld/      # JSON-LD processing
│   ├── fluree-graph-turtle/       # Turtle parser
│   └── fluree-graph-format/       # RDF formatters (JSON-LD, Turtle, etc.)
│
├── Query & Transaction
│   ├── fluree-db-query/           # Query engine (JSON-LD Query)
│   ├── fluree-db-sparql/          # SPARQL parser and lowering
│   └── fluree-db-transact/        # Transaction processing
│
├── Storage & Connection
│   ├── fluree-db-connection/      # Storage backends and connection management
│   ├── fluree-db-storage-aws/     # AWS storage (S3, S3 Express, DynamoDB)
│   ├── fluree-db-nameservice/     # Nameservice implementations
│   └── fluree-db-nameservice-sync/# Git-like remote sync for nameservice
│
├── Indexing
│   ├── fluree-db-binary-index/    # Binary index formats + read-side runtime
│   ├── fluree-db-indexer/         # Index building
│   └── fluree-db-ledger/          # Ledger state (indexed DB + novelty)
│
├── Security & Validation
│   ├── fluree-db-policy/          # Policy enforcement
│   ├── fluree-db-credential/      # JWS/VerifiableCredential verification
│   ├── fluree-db-crypto/          # Storage encryption (AES-256-GCM)
│   └── fluree-db-shacl/           # SHACL validation engine
│
├── Reasoning
│   └── fluree-db-reasoner/        # OWL2-RL reasoning engine
│
├── Graph Sources
│   ├── fluree-db-tabular/         # Tabular column batch types
│   ├── fluree-db-iceberg/         # Apache Iceberg integration
│   └── fluree-db-r2rml/           # R2RML mapping support
│
├── Search
│   ├── fluree-search-protocol/    # Search service protocol types
│   ├── fluree-search-service/     # Search backend implementations
│   └── fluree-search-httpd/       # Standalone HTTP search server
│
├── Networking
│   ├── fluree-sse/                # Server-Sent Events parser
│   └── fluree-db-peer/            # SSE protocol for peer mode
│
└── Top-Level
    ├── fluree-db-api/             # Public API and high-level operations
    └── fluree-db-server/          # HTTP server (binary)
```

## Foundation Crates

### fluree-vocab

**Purpose:** RDF vocabulary constants and namespace codes

**Responsibilities:**
- Standard RDF namespace definitions (rdf:, rdfs:, xsd:, owl:, etc.)
- Fluree-specific namespace codes
- IRI constants for common predicates

**Dependencies:** None (foundation crate)

### fluree-db-core

**Purpose:** Runtime-agnostic core library for Fluree DB

**Responsibilities:**
- Core types (Flake, Sid, IndexType, etc.)
- Index structures (SPOT, POST, OPST, PSOT)
- Range query operations
- Database snapshot representation
- Statistics and cardinality tracking
- Content-addressed identity (`ContentId`, `ContentKind`)
- Content store trait (`ContentStore`)

**Key Types:**
- `Flake` - Indexed triple representation
- `Sid` - Subject identifier
- `LedgerSnapshot` - Database snapshot at a point in time
- `IndexType` - Index selection enum
- `StatsView` - Query statistics
- `ContentId` - CIDv1 content-addressed identifier
- `ContentKind` - Content type enum (Commit, Txn, IndexRoot, etc.)
- `ContentStore` - Content-addressed storage trait
- `BranchedContentStore` - Recursive content store with namespace fallback for branches

**Dependencies:**
- fluree-vocab

### fluree-db-novelty

**Purpose:** Novelty overlay and commit types

**Responsibilities:**
- In-memory novelty (uncommitted/unindexed flakes)
- Commit metadata and structure
- Novelty application and slicing

**Key Types:**
- `Novelty` - In-memory flake overlay
- `Commit` - Commit metadata
- `FlakeId` - Novelty flake identifier

**Dependencies:**
- fluree-db-core
- fluree-db-binary-index
- fluree-vocab

## Graph Processing Crates

### fluree-graph-ir

**Purpose:** Format-agnostic RDF intermediate representation

**Responsibilities:**
- Generic graph IR for RDF data
- Triple/quad representation
- Format-independent graph operations

**Dependencies:**
- fluree-vocab

### fluree-graph-json-ld

**Purpose:** Minimal JSON-LD processing

**Responsibilities:**
- JSON-LD expansion
- JSON-LD compaction
- @context handling
- IRI resolution

**Dependencies:**
- fluree-graph-ir
- fluree-vocab

### fluree-graph-turtle

**Purpose:** Turtle (TTL) parser

**Responsibilities:**
- Turtle syntax parsing
- Triple generation from Turtle

**Dependencies:**
- fluree-graph-ir
- fluree-vocab

### fluree-graph-format

**Purpose:** RDF graph formatters

**Responsibilities:**
- Output formatting (JSON-LD, Turtle, N-Triples)
- Serialization utilities

**Dependencies:**
- fluree-graph-ir

## Query & Transaction Crates

### fluree-db-query

**Purpose:** Query engine for JSON-LD Query

**Responsibilities:**
- Query parsing and planning
- Statistics-driven pattern reordering across all WHERE-clause pattern types
  (triples, UNION, OPTIONAL, MINUS, search patterns, Graph, Service, etc.)
- Bound-variable-aware selectivity estimation using HLL-derived property
  statistics (with heuristic fallbacks)
- Query execution
- Filter pushdown (index-level range filters, inline join/BIND evaluation,
  dependency-based placement, compound pattern nesting)
- Aggregations
- BM25 and vector search integration
- Explain plan generation for optimization debugging

**Key Types:**
- `Query` - Parsed query
- `VarRegistry` - Variable management
- `Pattern` - Query patterns
- `TriplePattern` - Subject–predicate–object pattern with optional `DatatypeConstraint`
- `Ref` - Variable or constant in subject/predicate position (no literals)
- `Term` - Variable or constant in object position (includes literals)
- `DatatypeConstraint` - Explicit datatype (`Explicit(Sid)`) or language tag
  (`LangTag`; implies `rdf:langString` datatype)
- `PatternEstimate` - Cardinality classification (Source, Reducer, Expander, Deferred)

**Dependencies:**
- fluree-db-core

### fluree-db-sparql

**Purpose:** SPARQL parsing and execution

**Responsibilities:**
- SPARQL lexing and parsing
- AST construction
- Lowering to internal IR
- Diagnostic reporting

**Key Types:**
- `Query` - SPARQL query AST
- `Pattern` - Graph pattern
- `Diagnostic` - Parse/validation errors

**Dependencies:**
- fluree-db-query
- fluree-db-core

### fluree-db-transact

**Purpose:** Transaction processing

**Responsibilities:**
- JSON-LD transaction parsing
- RDF triple generation
- Flake generation
- Commit creation

**Dependencies:**
- fluree-graph-json-ld
- fluree-db-core

## Storage & Connection Crates

### fluree-db-connection

**Purpose:** Storage backends and connection management

**Responsibilities:**
- Storage abstraction trait
- Memory, file, and cloud storage
- Address resolution
- Commit storage and retrieval

**Key Types:**
- `Storage` trait
- `MemoryStorage`
- `FileStorage`

**Dependencies:**
- fluree-db-core
- fluree-graph-json-ld
- fluree-db-storage-aws (optional)
- fluree-db-nameservice

### fluree-db-storage-aws

**Purpose:** AWS storage backends

**Responsibilities:**
- S3 storage implementation
- S3 Express One Zone support
- DynamoDB integration

**Dependencies:**
- fluree-db-core
- fluree-db-nameservice

### fluree-db-nameservice

**Purpose:** Nameservice implementations

**Responsibilities:**
- Nameservice abstraction
- Ledger metadata management
- Publish/lookup operations
- Branch creation and listing
- File and DynamoDB backends

**Key Types:**
- `NameService` trait (includes `list_branches`, `create_branch`, `drop_branch`)
- `Publisher` trait (commit/index publishing)
- `NsRecord` - Nameservice record (includes `source_branch` for ancestry and `branches` child count for reference counting)
- `FileNameService`

**Dependencies:**
- fluree-db-core

### fluree-db-nameservice-sync

**Purpose:** Git-like remote sync for nameservice

**Responsibilities:**
- Remote nameservice synchronization (fetch/push refs)
- Multi-origin CAS object fetching with integrity verification
- Pack protocol client (streaming binary transport for clone/pull)
- SSE-based change streaming
- Sync driver (fetch/pull/push orchestration)

**Key Types:**
- `MultiOriginFetcher` - Priority-ordered HTTP origin fallback
- `HttpOriginFetcher` - Single-origin CAS object + pack fetcher
- `SyncDriver` - Orchestrates fetch/pull/push with remote clients
- `PackIngestResult` - Result of streaming pack import

**Dependencies:**
- fluree-db-core
- fluree-db-nameservice
- fluree-db-novelty
- fluree-sse

## Indexing Crates

### fluree-db-binary-index

**Purpose:** Binary index wire formats and read-side runtime

**Responsibilities:**
- Binary index format codecs (FIR6 root, FBR3 branch, FLI3 leaf, leaflet layout)
- Dictionary artifacts and readers (inline dicts, dict trees, arenas)
- Query-time read types (`BinaryIndexStore`, `BinaryGraphView`, cursors)

**Dependencies:**
- fluree-db-core

### fluree-db-indexer

**Purpose:** Index building for Fluree DB

**Responsibilities:**
- Incremental index updates
- Full reindexing
- Index refresh orchestration

**Dependencies:**
- fluree-db-core
- fluree-db-binary-index
- fluree-db-novelty
- fluree-db-nameservice
- fluree-vocab

### fluree-db-ledger

**Purpose:** Ledger state management

**Responsibilities:**
- Combining indexed DB with novelty overlay
- Ledger snapshot creation
- State transitions
- Building `BranchedContentStore` trees from branch ancestry

**Key Types:**
- `LedgerState` - Complete ledger snapshot

**Dependencies:**
- fluree-db-core
- fluree-db-novelty
- fluree-db-nameservice

## Security & Validation Crates

### fluree-db-policy

**Purpose:** Policy enforcement

**Responsibilities:**
- Policy parsing and evaluation
- Query augmentation for policy
- Transaction authorization

**Dependencies:**
- fluree-db-query
- fluree-db-core

### fluree-db-credential

**Purpose:** Credential verification

**Responsibilities:**
- JWS signature verification
- VerifiableCredential processing
- DID resolution

**Dependencies:** None (standalone)

### fluree-db-crypto

**Purpose:** Storage encryption

**Responsibilities:**
- AES-256-GCM encryption/decryption
- Key management
- Encrypted storage layer

**Dependencies:**
- fluree-db-core

### fluree-db-shacl

**Purpose:** SHACL validation engine

**Responsibilities:**
- SHACL shapes parsing
- Constraint validation
- Validation reports

**Dependencies:**
- fluree-db-core
- fluree-db-query
- fluree-vocab

## Reasoning

### fluree-db-reasoner

**Purpose:** OWL2-RL reasoning engine

**Responsibilities:**
- OWL2-RL rule application
- Inference generation
- Materialization

**Dependencies:**
- fluree-db-core
- fluree-vocab

## Graph Source Crates

### fluree-db-tabular

**Purpose:** Tabular column batch types

**Responsibilities:**
- Arrow-compatible column batches
- Graph source data abstraction

**Dependencies:** None (foundation for graph sources)

### fluree-db-iceberg

**Purpose:** Apache Iceberg integration

**Responsibilities:**
- Iceberg REST catalog support
- Iceberg table scanning
- Parquet file reading

**Dependencies:**
- fluree-db-core
- fluree-db-tabular

### fluree-db-r2rml

**Purpose:** R2RML mapping support

**Responsibilities:**
- R2RML mapping parsing
- Relational-to-RDF mapping
- Graph source generation

**Dependencies:**
- fluree-graph-ir
- fluree-graph-turtle (optional)
- fluree-db-tabular
- fluree-vocab

## Search Crates

### fluree-search-protocol

**Purpose:** Search service protocol types

**Responsibilities:**
- Request/response structs
- Error model and codes
- Protocol version constants
- BM25 and vector query definitions

**Dependencies:** serde, thiserror

### fluree-search-service

**Purpose:** Search backend implementations

**Responsibilities:**
- `SearchBackend` trait
- BM25 backend (tantivy)
- Vector backend (usearch, feature-gated)
- Index caching with TTL

**Dependencies:**
- fluree-search-protocol
- fluree-db-query
- fluree-db-core

### fluree-search-httpd

**Purpose:** Standalone HTTP search server

**Responsibilities:**
- HTTP API for search queries
- Index loading from storage
- Health and capabilities endpoints

**Dependencies:**
- fluree-search-protocol
- fluree-search-service
- axum, tokio

## Networking Crates

### fluree-sse

**Purpose:** Lightweight SSE parser

**Responsibilities:**
- Server-Sent Events parsing
- Event stream handling

**Dependencies:** None (foundation)

### fluree-db-peer

**Purpose:** SSE protocol for peer mode

**Responsibilities:**
- Peer protocol types
- SSE client for peer communication

**Dependencies:**
- fluree-sse

## Top-Level Crates

### fluree-db-api

**Purpose:** Public API and orchestration

**Responsibilities:**
- Ledger lifecycle (create, load, drop, branch)
- Query execution coordination
- Transaction execution
- Time travel resolution
- Policy application
- Dataset and view composition

**Key Types:**
- `Fluree` - Main entry point
- `Graph` - Lazy handle for chaining
- `GraphSnapshot` - Materialized snapshot
- `LedgerState` - Loaded ledger state
- `QueryResult` - Query results
- `TransactResult` - Commit receipt

**Dependencies:**
- fluree-db-query
- fluree-db-sparql
- fluree-db-transact
- fluree-db-connection
- fluree-db-nameservice
- fluree-db-policy
- fluree-db-reasoner
- fluree-db-shacl

### fluree-db-server

**Purpose:** HTTP server (binary)

**Responsibilities:**
- HTTP API endpoints
- Request routing
- Response formatting
- TLS/SSL, CORS handling

**Dependencies:**
- fluree-db-api
- axum

## Dependency Layers

```text
Layer 5 (Top)        fluree-db-server
                            │
                     fluree-db-api
                            │
Layer 4 (Features)   ┌──────┼──────┬──────────┬───────────┐
                     │      │      │          │           │
                  policy  shacl reasoner  credential  crypto
                     │      │      │
Layer 3 (Query)      └──────┴──────┴──────────┐
                                              │
                     fluree-db-query ←── fluree-db-sparql
                            │
Layer 2 (Data)       ledger, binary-index, indexer, novelty, connection
                            │
Layer 1 (Core)       fluree-db-core
                            │
Layer 0 (Foundation) fluree-vocab, fluree-sse, fluree-db-tabular
```

## External Dependencies

### Key External Crates

**Web Framework:**
- `axum` - HTTP server framework
- `tokio` - Async runtime
- `tower` - Service abstractions

**Serialization:**
- `serde` - Serialization framework
- `serde_json` - JSON support

**RDF:**
- `oxiri` - IRI parsing and validation

**Storage:**
- `aws-sdk-s3` - AWS S3 client
- `aws-sdk-dynamodb` - AWS DynamoDB client

**Search:**
- `tantivy` - BM25 full-text search
- `usearch` - Vector similarity search (HNSW indexes)

**Analytics:**
- `iceberg-rust` - Apache Iceberg support
- `parquet` - Parquet file reading

**Cryptography:**
- `ed25519-dalek` - Ed25519 signatures
- `ring` - Cryptographic operations

## Building

### Build All

```bash
cargo build --release
```

### Build Server Only

```bash
cargo build --release --bin fluree-db-server
```

### Run Tests

```bash
cargo test
```

### Build with Features

```bash
cargo build --features native,vector
```

## Crate Versions

All crates use synchronized versioning and are updated together.

Check versions:

```bash
cargo tree | grep fluree
```

## Related Documentation

- [Contributing: Dev Setup](../contributing/dev-setup.md) - Development environment
- [Contributing: Tests](../contributing/tests.md) - Testing guide
- [Glossary](glossary.md) - Term definitions


---

# contributing/README.md

# Contributing

Welcome to the Fluree contributor documentation! This section provides everything you need to contribute to Fluree.

## Getting Started

### [Dev Setup](dev-setup.md)

Set up your development environment:
- Install dependencies
- Clone repository
- Build from source
- Run development server
- IDE configuration

### [Tests](tests.md)

Testing guide:
- Running tests
- Writing tests
- Test organization
- Integration tests
- Benchmarking
- Continuous integration

### [Adding Tracing Spans](tracing-guide.md)

How to instrument new code paths with tracing spans:
- The two-tier span strategy (info / debug / trace)
- Code patterns for sync and async spans
- Deferred field recording
- Testing spans with SpanCaptureLayer
- Common gotchas (`!Send` guards, OTEL floods, etc.)

### [W3C SPARQL Compliance Suite](sparql-compliance.md)

Guide to the manifest-driven W3C compliance test suite:
- Running and interpreting results
- Debugging failures
- From failure to issue/PR workflow
- Using Claude Code for compliance work
- Architecture overview

### [SHACL Implementation](shacl-implementation.md)

How SHACL validation is wired into Fluree, for contributors adding
constraints or fixing bugs:
- Pipeline: compile → cache → validate
- Crate layout (`fluree-db-shacl` / `-transact` / `-api`)
- Shared post-stage helper and its call sites
- Per-graph config, `f:shapesSource`, target-type resolution
- Adding a new constraint (walkthrough)
- Testing patterns (unit + integration + temp-revert regression trick)
- Known gaps (`sh:uniqueLang`, `sh:qualifiedValueShape`, cross-txn cache)


## How to Contribute

### Ways to Contribute

1. **Report Bugs:** File issues with reproduction steps
2. **Suggest Features:** Propose enhancements with use cases
3. **Fix Bugs:** Submit pull requests for bug fixes
4. **Add Features:** Implement new capabilities
5. **Improve Documentation:** Fix typos, clarify explanations, add examples
6. **Review Pull Requests:** Help review others' contributions
7. **Answer Questions:** Help users in discussions

### Before Contributing

1. **Check existing issues:** Search for duplicate issues
2. **Read documentation:** Understand the feature area
3. **Discuss major changes:** Open issue before large PRs
4. **Follow style guide:** Match existing code style
5. **Add tests:** Include tests for new features
6. **Update docs:** Document new features

## Contribution Workflow

### 1. Fork Repository

```bash
# Fork on GitHub, then clone
git clone https://github.com/YOUR-USERNAME/db.git
cd db
```

### 2. Create Branch

```bash
git checkout -b feature/my-feature
```

Branch naming:
- `feature/` - New features
- `fix/` - Bug fixes
- `docs/` - Documentation
- `refactor/` - Code refactoring
- `test/` - Test additions

### 3. Make Changes

Edit code, following style guidelines.

### 4. Add Tests

```bash
# Run existing tests
cargo test

# Add new tests
# Edit tests/test_my_feature.rs
```

### 5. Run Checks

```bash
# Format code
cargo fmt

# Lint code
cargo clippy

# Run all tests
cargo test --all
```

### 6. Commit Changes

```bash
git add .
git commit -m "Add feature: description"
```

Commit message format:
```text
Short summary (50 chars or less)

More detailed explanation if needed. Wrap at 72 characters.

- Key point 1
- Key point 2

Fixes #123
```

### 7. Push and Create PR

```bash
git push origin feature/my-feature
```

Create pull request on GitHub.

### 8. Address Review Comments

Respond to reviewer feedback, make requested changes.

## Code Style

### Rust Style

Follow Rust standard style:

```bash
# Format all code
cargo fmt

# Check style
cargo clippy
```

### Naming Conventions

**Types:** PascalCase
```rust
struct Dataset { ... }
enum QueryResult { ... }
```

**Functions:** snake_case
```rust
fn execute_query() { ... }
fn parse_json_ld() { ... }
```

**Constants:** SCREAMING_SNAKE_CASE
```rust
const MAX_QUERY_SIZE: usize = 1_048_576;
```

**Modules:** snake_case
```rust
mod query_engine;
mod storage_backend;
```

### Documentation

Document public APIs:

```rust
/// Executes a query against the dataset.
///
/// # Arguments
///
/// * `query` - The query to execute
/// * `context` - Execution context
///
/// # Returns
///
/// Query results or error
///
/// # Examples
///
/// ```
/// let results = dataset.query(&query, &context)?;
/// ```
pub fn query(&self, query: &Query, context: &Context) -> Result<Vec<Solution>> {
    // Implementation
}
```

### Error Handling

Use Result types:

```rust
// Good
pub fn parse_query(input: &str) -> Result<Query, ParseError> {
    // ...
}

// Bad
pub fn parse_query(input: &str) -> Query {
    // No error handling
}
```

### Testing

Write tests for new code:

```rust
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_query_execution() {
        let query = Query::parse("...").unwrap();
        let result = execute(&query).unwrap();
        assert_eq!(result.len(), 2);
    }
}
```

## Pull Request Guidelines

### PR Title

Format: `category: short description`

Examples:
- `feat: Add SPARQL property paths support`
- `fix: Correct transaction time ordering`
- `docs: Update query examples`
- `test: Add integration tests for time travel`
- `refactor: Simplify index scan logic`

### PR Description

Include:

1. **Summary:** What does this PR do?
2. **Motivation:** Why is this needed?
3. **Changes:** What changed?
4. **Testing:** How was it tested?
5. **Breaking Changes:** Any breaking changes?

Example:
```markdown
## Summary
Adds support for SPARQL property paths, enabling recursive graph traversal.

## Motivation
Many users need to query hierarchical data structures. Property paths are a standard SPARQL feature.

## Changes
- Added property path parser to fluree-db-sparql
- Implemented path evaluation in query engine
- Added tests for various path patterns

## Testing
- Unit tests for parser
- Integration tests for path queries
- Benchmarks show acceptable performance

## Breaking Changes
None
```

### PR Checklist

- [ ] Code follows style guidelines
- [ ] Tests added/updated
- [ ] Documentation updated
- [ ] All tests pass
- [ ] No clippy warnings
- [ ] Commit messages clear
- [ ] PR description complete

## Review Process

### What Reviewers Look For

1. **Correctness:** Does it work as intended?
2. **Tests:** Adequate test coverage?
3. **Style:** Follows conventions?
4. **Documentation:** Properly documented?
5. **Performance:** No obvious performance issues?
6. **Breaking Changes:** Backward compatible?

### Responding to Reviews

- Be receptive to feedback
- Ask questions if unclear
- Make requested changes promptly
- Explain your reasoning when appropriate
- Say thanks for helpful reviews

## Community Guidelines

### Code of Conduct

- Be respectful and inclusive
- Assume good intentions
- Give constructive feedback
- Welcome newcomers
- No harassment or discrimination

### Communication

- **GitHub Issues:** Bug reports, feature requests
- **Pull Requests:** Code contributions
- **Discussions:** Questions, ideas, help

### Getting Help

- Read documentation first
- Search existing issues
- Ask specific questions
- Provide reproduction steps
- Be patient and respectful

## License

Contributions licensed under Apache 2.0.

By contributing, you agree to license your contributions under the same license.

## Recognition

Contributors are recognized in:
- CONTRIBUTORS.md file
- Release notes
- GitHub contributors page

Thank you for contributing to Fluree!

## Related Documentation

- [Dev Setup](dev-setup.md) - Development environment
- [Tests](tests.md) - Testing guide
- [Graph Identities and Naming](../reference/graph-identities.md) - Naming conventions


---

# contributing/dev-setup.md

# Development Setup

This guide walks through setting up a development environment for contributing to Fluree.

## Prerequisites

### Required

**Rust:**
```bash
# Install rustup
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Verify installation
rustc --version  # Should be 1.75.0 or later
cargo --version
```

**Git:**
```bash
git --version  # Should be 2.0 or later
```

### Recommended

**IDE/Editor:**
- Visual Studio Code with rust-analyzer
- IntelliJ IDEA with Rust plugin
- Vim/Neovim with rust-analyzer LSP

**Tools:**
- `cargo-watch` - Auto-rebuild on changes
- `cargo-nextest` - Faster test runner
- `cargo-flamegraph` - Performance profiling

```bash
cargo install cargo-watch cargo-nextest cargo-flamegraph
```

## Clone Repository

```bash
# Clone main repository
git clone https://github.com/fluree/db.git
cd db

# Or clone your fork
git clone https://github.com/YOUR-USERNAME/db.git
cd db
```

## Build from Source

### Development Build

```bash
# Build all crates
cargo build

# Build specific crate
cd fluree-db-query
cargo build
```

### Release Build

```bash
# Optimized build
cargo build --release

# Server binary at: target/release/fluree-db-server
```

### Build Server Only

```bash
cargo build --release --bin fluree-db-server
```

## Run Development Server

### Quick Start

```bash
# Run with default settings (memory storage)
cargo run --bin fluree-db-server
```

Server starts on http://localhost:8090

### With Custom Settings

```bash
cargo run --bin fluree-db-server -- \
  --storage file \
  --data-dir ./dev-data \
  --log-level debug
```

### Watch Mode

Auto-rebuild and restart on changes:

```bash
cargo watch -x 'run --bin fluree-db-server'
```

## Run Tests

### All Tests

```bash
cargo test --all
```

### Specific Crate Tests

```bash
cd fluree-db-query
cargo test
```

### Specific Test

```bash
cargo test test_query_execution
```

### With Output

```bash
cargo test -- --nocapture
```

### Integration Tests

```bash
cargo test --test integration_tests
```

### With Nextest (Faster)

```bash
cargo nextest run
```

## IDE Setup

### Visual Studio Code

**Install Extensions:**
- rust-analyzer
- CodeLLDB (debugging)
- Even Better TOML

**Settings (.vscode/settings.json):**
```json
{
  "rust-analyzer.cargo.features": "all",
  "rust-analyzer.checkOnSave.command": "clippy",
  "rust-analyzer.inlayHints.enable": true
}
```

**Launch Config (.vscode/launch.json):**
```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "lldb",
      "request": "launch",
      "name": "Debug server",
      "cargo": {
        "args": ["build", "--bin=fluree-db-server"],
        "filter": {
          "name": "fluree-db-server",
          "kind": "bin"
        }
      },
      "args": ["--storage", "memory", "--log-level", "debug"],
      "cwd": "${workspaceFolder}"
    }
  ]
}
```

### IntelliJ IDEA

**Install Plugin:**
- Rust plugin (official)

**Configure:**
- File → Settings → Languages & Frameworks → Rust
- Set toolchain location
- Enable external linter (clippy)

### Vim/Neovim

**Install rust-analyzer:**

For Neovim with built-in LSP:
```lua
-- init.lua
require'lspconfig'.rust_analyzer.setup{}
```

For Vim with CoC:
```vim
" Install coc-rust-analyzer
:CocInstall coc-rust-analyzer
```

## Development Workflow

### Make Changes

```bash
# Create branch
git checkout -b feature/my-feature

# Edit code
vim fluree-db-query/src/execute.rs

# Format
cargo fmt

# Check
cargo clippy
```

### Test Changes

```bash
# Run affected tests
cargo test -p fluree-db-query

# Run all tests
cargo test --all
```

### Verify Build

```bash
# Development build
cargo build

# Release build
cargo build --release

# Check all features compile
cargo build --all-features
```

### Run Server Locally

```bash
cargo run --bin fluree-db-server -- \
  --storage memory \
  --log-level debug
```

Test your changes:
```bash
# In another terminal
curl http://localhost:8090/health

curl -X POST http://localhost:8090/v1/fluree/query -d '{...}'
```

## Debugging

### With rust-lldb

```bash
# Build with debug symbols
cargo build

# Run with lldb
rust-lldb target/debug/fluree-db-server

# Set breakpoint
(lldb) b fluree_db_query::execute::execute_query
(lldb) run --storage memory

# Debug commands
(lldb) continue
(lldb) step
(lldb) print variable_name
```

### With VS Code

Use launch.json configuration from above, then F5 to debug.

### Print Debugging

```rust
// Quick debugging
println!("Debug: value = {:?}", value);

// Better: use tracing
tracing::debug!(?value, "Processing query");
```

### Logging

Enable debug logs:

```bash
RUST_LOG=debug cargo run --bin fluree-db-server
```

Or trace specific module:

```bash
RUST_LOG=fluree_db_query=trace cargo run --bin fluree-db-server
```

## Performance Profiling

### Criterion Benchmarks

Run benchmarks:

```bash
cargo bench
```

View results: `target/criterion/report/index.html`

### Flamegraphs

> **A stock `--release` flamegraph is blank.** `[profile.release]` sets
> `strip = true` with no `debug`, so symbols are gone and every frame shows as
> `[unknown]`. Override `strip`/`debug` for the build and add frame pointers —
> this keeps release's `lto = true, codegen-units = 1` (so you profile the same
> program CI ships) while making stacks symbolize:

```bash
# Install tools (Linux)
sudo apt install linux-tools-common linux-tools-generic
cargo install flamegraph

# Generate flamegraph with symbols (env overrides, no Cargo.toml change needed)
CARGO_PROFILE_RELEASE_STRIP=false \
CARGO_PROFILE_RELEASE_DEBUG=line-tables-only \
RUSTFLAGS="-C force-frame-pointers=yes" \
  cargo flamegraph --bin fluree-db-server

# Open flamegraph.svg in browser
```

To isolate one operation inside a long-lived process, attach `perf` to a
window: `perf record -g --call-graph fp -F 997 -p <pid> -- sleep <secs>`, then
`perf script | inferno-collapse-perf | inferno-flamegraph > out.svg`.

### perf (Linux)

```bash
# Record
cargo build --release
perf record -g target/release/fluree-db-server

# Report
perf report
```

## Common Development Tasks

### Add New Query Feature

1. Add to query parser (fluree-db-query/src/parse/)
2. Add to query executor (fluree-db-query/src/execute/)
3. Add tests (fluree-db-query/tests/)
4. Update documentation (docs/query/)

### Add New Transaction Feature

1. Add to transaction parser (fluree-db-transact/src/parse/)
2. Add to staging logic (fluree-db-transact/src/stage.rs)
3. Add tests (fluree-db-transact/tests/)
4. Update documentation (docs/transactions/)

### Add New Storage Backend

1. Implement Storage trait (fluree-db-storage/src/)
2. Add backend-specific logic
3. Add tests
4. Update configuration options
5. Document in docs/operations/storage.md

## Code Organization

### Module Structure

```text
fluree-db-query/
├── src/
│   ├── lib.rs           # Public API and re-exports
│   ├── triple.rs        # TriplePattern, Ref, Term, DatatypeConstraint
│   ├── parse/           # Query parsing
│   │   ├── mod.rs
│   │   ├── ast.rs       # Unresolved AST (before IRI resolution)
│   │   ├── lower.rs     # AST → IR lowering
│   │   └── node_map.rs  # JSON-LD node-map → AST
│   ├── execute/         # Query execution
│   │   ├── mod.rs
│   │   ├── runner.rs
│   │   ├── operator_tree.rs
│   │   └── where_plan.rs  # WHERE-clause planning (pattern types, reordering)
│   ├── bind.rs          # Variable binding
│   └── filter.rs        # Filter evaluation
├── tests/               # Integration tests
└── benches/             # Benchmarks
```

### Import Organization

```rust
// Standard library
use std::collections::HashMap;

// External crates
use serde::{Deserialize, Serialize};

// Internal crates
use fluree_db_common::{Iri, Literal};

// Current crate
use crate::parse::Query;
```

## Documentation

### Code Documentation

Use Rustdoc:

```rust
/// Executes a query against a dataset.
///
/// This function parses the query, generates an execution plan,
/// and runs the plan against the dataset's indexes.
///
/// # Arguments
///
/// * `dataset` - The dataset to query
/// * `query` - The query to execute
///
/// # Returns
///
/// A vector of solutions (variable bindings)
///
/// # Errors
///
/// Returns error if query is invalid or execution fails
///
/// # Examples
///
/// ```
/// use fluree_db_api::query;
///
/// let results = query(&dataset, &query)?;
/// assert_eq!(results.len(), 10);
/// ```
pub fn query(dataset: &Dataset, query: &Query) -> Result<Vec<Solution>> {
    // Implementation
}
```

Generate docs:

```bash
cargo doc --open
```

### User Documentation

Update relevant docs in `docs/` directory when adding user-facing features.

## Dependencies

### Adding Dependencies

Add to Cargo.toml:

```toml
[dependencies]
serde = { version = "1.0", features = ["derive"] }
tokio = { version = "1.35", features = ["full"] }
```

### Updating Dependencies

```bash
# Update all dependencies
cargo update

# Update specific dependency
cargo update -p serde
```

### Checking for Outdated

```bash
cargo install cargo-outdated
cargo outdated
```

## Troubleshooting Development Issues

### Build Fails

```bash
# Clean and rebuild
cargo clean
cargo build
```

### Tests Fail

```bash
# Run with output
cargo test -- --nocapture

# Run specific test
cargo test test_name -- --nocapture
```

### Clippy Warnings

```bash
# Fix automatically where possible
cargo clippy --fix
```

### rustfmt Issues

```bash
# Format all code
cargo fmt
```

## Development Tools

### Cargo Commands

```bash
cargo build          # Build
cargo test           # Test
cargo run            # Run
cargo bench          # Benchmark
cargo doc            # Documentation
cargo clean          # Clean
cargo check          # Quick check (no binary)
cargo clippy         # Lint
cargo fmt            # Format
```

### Useful Cargo Plugins

```bash
# Install useful plugins
cargo install cargo-watch      # Auto-rebuild
cargo install cargo-nextest    # Faster tests
cargo install cargo-outdated   # Check deps
cargo install cargo-audit      # Security audit
cargo install cargo-expand     # Expand macros
```

## Performance Tips

### Development Builds

Use development builds during development:
- Faster compilation
- Slower execution
- Debug symbols included

### Release Builds

Use release builds for testing performance:

```bash
cargo build --release
cargo test --release
```

### Link Time Optimization

For maximum performance:

```toml
[profile.release]
lto = true
codegen-units = 1
```

Warning: Significantly slower compile times.

## Related Documentation

- [Tests](tests.md) - Testing guide
- [Graph Identities and Naming](../reference/graph-identities.md) - Naming conventions
- [Crate Map](../reference/crate-map.md) - Code architecture


---

# contributing/tests.md

# Tests

This guide covers testing practices, test organization, and how to run tests in the Fluree codebase.

## Test Organization

### Unit Tests

Tests in the same file as code:

```rust
// src/query.rs
pub fn execute_query(query: &Query) -> Result<Vec<Solution>> {
    // Implementation
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_execute_query() {
        let query = Query::parse("SELECT ?s WHERE { ?s ?p ?o }").unwrap();
        let results = execute_query(&query).unwrap();
        assert!(!results.is_empty());
    }
}
```

### Integration Tests

Tests in `tests/` directory:

```rust
// tests/integration_test.rs
use fluree_db_api::{Dataset, query};

#[test]
fn test_query_workflow() {
    let dataset = Dataset::new_memory();
    
    // Insert data
    dataset.transact(test_data()).unwrap();
    
    // Query data
    let results = query(&dataset, test_query()).unwrap();
    
    // Verify
    assert_eq!(results.len(), 5);
}
```

#### Grouped integration binaries (`fluree-db-api`)

Every top-level `tests/*.rs` file is a **separate test binary** that links the
entire crate dependency graph. With ~160 integration files that meant ~160 full
link steps per `cargo test` — the dominant cost of building the test suite.

`fluree-db-api` therefore **groups** its integration files into a handful of
binaries by domain (`grp_query`, `grp_policy`, `grp_import`, `grp_index`,
`grp_transact`, `grp_ledger`, `grp_graphsource`, `grp_reasoning`, `grp_misc`,
…). The crate sets `autotests = false` and declares each binary explicitly via
`[[test]]` in `Cargo.toml`. Each case file is included as a module:

```rust
// tests/grp_query.rs
#[path = "support/mod.rs"]
mod support;            // shared harness, compiled once per binary

#[path = "it_query_jsonld.rs"]
mod it_query_jsonld;
#[path = "it_query_sparql.rs"]
mod it_query_sparql;
// …
```

Within a case file, refer to the shared harness as `crate::support` (not
`mod support;`). Each file becomes its own module, so item names never collide
across files.

To **add a new integration test**: create `tests/it_<name>.rs`, then add a
`#[path = "it_<name>.rs"] mod it_<name>;` line to the appropriate `grp_*.rs`
binary. No new `[[test]]` entry is needed unless you are creating a new group.

**Kept standalone** (their own `[[test]]` binary, *not* grouped):
- Feature-gated suites (`credential`, `iceberg`, `aws-testcontainers`, `vector`)
  — declared with `required-features`.
- Tests that mutate **process-global env vars** (e.g. `FLUREE_*` toggles). A
  grouped binary runs its tests as threads in one process under plain
  `cargo test`, so env mutation must stay in a dedicated binary to remain
  isolated. (Under `cargo nextest`, every test already runs in its own process,
  so this only matters for bare `cargo test`.)

### Example Tests

Tests in `examples/`:

```rust
// examples/basic_query.rs
fn main() -> Result<()> {
    let dataset = Dataset::new_memory();
    dataset.transact(sample_data())?;
    let results = dataset.query(sample_query())?;
    println!("Results: {:?}", results);
    Ok(())
}
```

Run with:
```bash
cargo run --example basic_query
```

## Running Tests

### All Tests

```bash
cargo test --all
```

### Opt-in LocalStack (S3/DynamoDB) tests

Some AWS/S3 tests are intentionally **opt-in** and will not run during typical `cargo test` runs.
They require Docker and start LocalStack automatically.

```bash
cargo test -p fluree-db-connection --features aws-testcontainers --test aws_testcontainers_test -- --nocapture
```

### Specific Crate

```bash
cargo test -p fluree-db-query
```

### Specific Test

```bash
cargo test test_query_execution
```

### With Output

```bash
cargo test -- --nocapture
```

### Integration Tests Only

```bash
cargo test --test '*'
```

### Doc Tests

```bash
cargo test --doc
```

### With Nextest (Faster)

```bash
cargo nextest run
```

## Writing Tests

### Unit Test Example

```rust
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_parse_simple_query() {
        let input = r#"{"select": ["?s"], "where": [{"@id": "?s"}]}"#;
        let query = parse_query(input).unwrap();
        
        assert_eq!(query.select_vars.len(), 1);
        assert_eq!(query.where_patterns.len(), 1);
    }

    #[test]
    fn test_parse_invalid_query() {
        let input = "invalid json";
        let result = parse_query(input);
        
        assert!(result.is_err());
        assert!(matches!(result.unwrap_err(), ParseError::InvalidJson));
    }
}
```

### Integration Test Example

```rust
// tests/it_query.rs
use fluree_db_api::*;

#[tokio::test]
async fn test_basic_query() {
    // Setup
    let dataset = Dataset::new_memory().await.unwrap();
    
    // Insert test data
    let txn = r#"{
        "@context": {"ex": "http://example.org/ns/"},
        "@graph": [{"@id": "ex:alice", "ex:name": "Alice"}]
    }"#;
    dataset.transact(txn).await.unwrap();
    
    // Execute query
    let query = r#"{
        "from": "test:main",
        "select": ["?name"],
        "where": [{"@id": "?s", "ex:name": "?name"}]
    }"#;
    let results = dataset.query(query).await.unwrap();
    
    // Verify
    assert_eq!(results.len(), 1);
    assert_eq!(results[0]["name"], "Alice");
}
```

### Async Tests

Use tokio test runtime:

```rust
#[tokio::test]
async fn test_async_operation() {
    let result = async_function().await.unwrap();
    assert_eq!(result, expected);
}
```

### Property-Based Tests

Use proptest for property-based testing:

```rust
use proptest::prelude::*;

proptest! {
    #[test]
    fn test_parse_roundtrip(s in "\\PC*") {
        let iri = Iri::parse(&s)?;
        let serialized = iri.to_string();
        let reparsed = Iri::parse(&serialized)?;
        assert_eq!(iri, reparsed);
    }
}
```

## Test Fixtures

### Test Data

Create reusable test data:

```rust
// tests/fixtures/mod.rs
pub fn sample_person_data() -> &'static str {
    r#"{
        "@context": {"schema": "http://schema.org/"},
        "@graph": [
            {"@id": "ex:alice", "@type": "schema:Person", "schema:name": "Alice"},
            {"@id": "ex:bob", "@type": "schema:Person", "schema:name": "Bob"}
        ]
    }"#
}

pub fn sample_query() -> &'static str {
    r#"{
        "select": ["?name"],
        "where": [{"@id": "?p", "schema:name": "?name"}]
    }"#
}
```

Use in tests:

```rust
#[test]
fn test_with_fixtures() {
    let dataset = Dataset::new_memory();
    dataset.transact(fixtures::sample_person_data()).unwrap();
    let results = dataset.query(fixtures::sample_query()).unwrap();
    assert_eq!(results.len(), 2);
}
```

### Test Helpers

```rust
// tests/helpers/mod.rs
pub async fn setup_test_dataset() -> Dataset {
    let dataset = Dataset::new_memory().await.unwrap();
    dataset.transact(sample_data()).await.unwrap();
    dataset
}

pub fn assert_query_results(results: &[Solution], expected: &[(&str, &str)]) {
    assert_eq!(results.len(), expected.len());
    for (result, (var, value)) in results.iter().zip(expected) {
        assert_eq!(result.get(var).unwrap().to_string(), *value);
    }
}
```

## Test Categories

### Fast Tests

Quick unit tests:

```rust
#[test]
fn test_fast_operation() {
    // < 1ms execution
}
```

### Slow Tests

Tests that take longer:

```rust
#[test]
#[ignore]  // Ignored by default
fn test_slow_operation() {
    // > 1s execution
}
```

Run slow tests:
```bash
cargo test -- --ignored
```

### Integration Tests

End-to-end workflows:

```rust
// tests/it_full_workflow.rs
#[tokio::test]
async fn test_complete_workflow() {
    let dataset = setup_test_dataset().await;
    
    // Multiple operations
    transact_initial_data(&dataset).await;
    query_and_verify(&dataset).await;
    update_data(&dataset).await;
    query_history(&dataset).await;
}
```

## Benchmarking

### Criterion Benchmarks

Create benchmarks:

```rust
// benches/query_bench.rs
use criterion::{black_box, criterion_group, criterion_main, Criterion};
use fluree_db_query::*;

fn benchmark_query_execution(c: &mut Criterion) {
    let dataset = setup_benchmark_dataset();
    let query = parse_query(QUERY).unwrap();
    
    c.bench_function("query execution", |b| {
        b.iter(|| {
            execute_query(black_box(&dataset), black_box(&query))
        });
    });
}

criterion_group!(benches, benchmark_query_execution);
criterion_main!(benches);
```

Run benchmarks:

```bash
cargo bench
```

### Comparison Benchmarks

Compare different approaches:

```rust
fn benchmark_approaches(c: &mut Criterion) {
    let mut group = c.benchmark_group("approach_comparison");
    
    group.bench_function("approach_1", |b| {
        b.iter(|| approach_1(black_box(&data)))
    });
    
    group.bench_function("approach_2", |b| {
        b.iter(|| approach_2(black_box(&data)))
    });
    
    group.finish();
}
```

## Continuous Integration

### GitHub Actions

Tests run automatically on:
- Pull requests
- Commits to main
- Scheduled (nightly)

Workflow: `.github/workflows/test.yml`

```yaml
name: Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions-rs/toolchain@v1
        with:
          toolchain: stable
      - run: cargo test --all
      - run: cargo clippy -- -D warnings
      - run: cargo fmt -- --check
```

### Pre-commit Checks

Run before committing:

```bash
#!/bin/bash
# .git/hooks/pre-commit

cargo fmt --check || exit 1
cargo clippy -- -D warnings || exit 1
cargo test --all || exit 1
```

Make executable:
```bash
chmod +x .git/hooks/pre-commit
```

## Test Best Practices

### 1. Test One Thing

Each test should verify one behavior:

Good:
```rust
#[test]
fn test_query_returns_correct_count() {
    let results = query(&dataset, &query).unwrap();
    assert_eq!(results.len(), 5);
}

#[test]
fn test_query_returns_correct_values() {
    let results = query(&dataset, &query).unwrap();
    assert_eq!(results[0]["name"], "Alice");
}
```

Bad:
```rust
#[test]
fn test_query() {
    let results = query(&dataset, &query).unwrap();
    assert_eq!(results.len(), 5);
    assert_eq!(results[0]["name"], "Alice");
    assert_eq!(results[1]["name"], "Bob");
    // Too many assertions
}
```

### 2. Use Descriptive Names

```rust
#[test]
fn test_query_with_filter_returns_only_matching_results() {
    // Clear what's being tested
}
```

### 3. Arrange-Act-Assert

Structure tests clearly:

```rust
#[test]
fn test_example() {
    // Arrange: Setup
    let dataset = setup_test_dataset();
    let query = parse_query(TEST_QUERY);
    
    // Act: Execute
    let results = execute_query(&dataset, &query).unwrap();
    
    // Assert: Verify
    assert_eq!(results.len(), 3);
}
```

### 4. Test Error Cases

```rust
#[test]
fn test_invalid_query_returns_error() {
    let result = parse_query("invalid");
    assert!(result.is_err());
}

#[tokio::test]
async fn test_missing_ledger_returns_ledger_not_found() {
    let result = fluree.ledger("nonexistent:main").await;
    assert!(matches!(result.unwrap_err(), Error::LedgerNotFound(_)));
}
```

### 5. Avoid Flaky Tests

Don't depend on:
- Timing
- Random values (use seeded RNG)
- External services
- File system state

### 6. Clean Up Resources

```rust
#[test]
fn test_with_temp_file() {
    let temp_dir = tempfile::tempdir().unwrap();
    let file_path = temp_dir.path().join("test.db");
    
    // Test with file_path
    
    // temp_dir automatically cleaned up
}
```

### 7. Use Test Utilities

```rust
// tests/common/mod.rs
pub fn assert_solution_contains(solutions: &[Solution], var: &str, value: &str) {
    let found = solutions.iter().any(|s| {
        s.get(var).map(|v| v.to_string() == value).unwrap_or(false)
    });
    assert!(found, "Expected to find {}={} in results", var, value);
}
```

## W3C SPARQL Compliance Tests

The `testsuite-sparql` crate runs official W3C SPARQL test cases against Fluree's parser and query engine. Tests are discovered automatically from W3C manifest files — zero hand-written test cases.

```bash
# Run all W3C SPARQL tests
cargo test -p testsuite-sparql

# Run with verbose output
cargo test -p testsuite-sparql -- --nocapture 2>&1
```

The suite covers SPARQL 1.0 and 1.1 syntax tests (293 tests) plus query evaluation tests across 12 categories (233 tests). Eval tests are `#[ignore]`'d by default — run with `--include-ignored` or via `make test-eval` in `testsuite-sparql/`.

For the full guide on interpreting results, debugging failures, and contributing fixes, see the [W3C SPARQL Compliance Suite](sparql-compliance.md) guide.

## Test Coverage

### Generate Coverage Report

Using tarpaulin:

```bash
cargo install cargo-tarpaulin

cargo tarpaulin --out Html --output-dir coverage/
```

View: `coverage/index.html`

### Coverage Goals

- Core functionality: 90%+ coverage
- Edge cases: Tested
- Error paths: Tested
- Public APIs: 100% covered

## Related Documentation

- [Dev Setup](dev-setup.md) - Development environment
- [Graph Identities and Naming](../reference/graph-identities.md) - Naming conventions
- [Contributing](README.md) - Contribution guidelines


---

# contributing/sparql-compliance.md

# W3C SPARQL Compliance Test Suite

The `testsuite-sparql` crate runs **official W3C SPARQL test cases** against Fluree's parser and query engine. Every test is discovered automatically from W3C manifest files — there are zero hand-written test cases.

This guide covers how to run the suite, interpret results, and turn failures into fixes.

## Why This Exists

The W3C publishes its SPARQL test suite as RDF data. Each `manifest.ttl` file declares test entries: a query file, optional input data, and expected results. Every serious SPARQL implementation (Oxigraph, Apache Jena, Eclipse RDF4J) runs these manifests programmatically. We do the same.

The ratio is extraordinary: ~700 lines of Rust infrastructure drive 700+ W3C test cases. Each failure is a spec-backed bug report with built-in test data and expected results.

**Philosophy: failures are features.** When a test fails, the default response is to fix Fluree, not skip the test. Skip entries are reserved for documented, deliberate design divergences reviewed by the team.

## Quick Start

> **Important:** The `testsuite-sparql` crate is **excluded from the Cargo workspace** (see root `Cargo.toml`). You must `cd testsuite-sparql/` before running any `cargo` or `make` commands. Using `cargo test -p testsuite-sparql` from the workspace root will fail.

All commands below assume you are already in `testsuite-sparql/`.

### Run All Tests

```bash
cd testsuite-sparql
cargo test
```

This runs all non-ignored W3C test suites. Currently that includes SPARQL 1.0 and 1.1 syntax tests. Query evaluation tests (12 categories, 327 tests) are registered but `#[ignore]`'d — run them with `--include-ignored` or via the Makefile.

### Run a Specific Suite

```bash
# SPARQL 1.1 syntax only
cargo test sparql11_syntax_query_tests

# SPARQL 1.0 syntax only
cargo test sparql10_syntax_tests

# Full query evaluation (~5 min, includes all 12 categories)
cargo test sparql11_query_w3c_testsuite -- --include-ignored

# Single evaluation category
cargo test sparql11_functions -- --include-ignored
```

### Run With Verbose Output

```bash
cargo test -- --nocapture 2>&1
```

The suite writes progress to stderr (`Running test N: <test_id> ...`) and a summary at the end.

### Using the Makefile

The `testsuite-sparql/Makefile` provides convenience targets:

```bash
# --- Running tests ---
make test              # Run syntax tests (live output)
make test-syntax11     # SPARQL 1.1 syntax tests only
make test-syntax10     # SPARQL 1.0 syntax tests only
make test-eval         # Full eval suite, all 12 categories
make test-eval-cat CAT=functions
                       # Run one eval category
make test-eval10       # Run SPARQL 1.0 eval tests

# --- Reports ---
make count-eval        # Quick pass/fail counts for eval tests
make report-eval-json  # JSON report for 1.1 eval → report-eval.json
make report-10-json    # JSON report for 1.0 eval → report-10.json
make cat-json CAT=functions
                       # JSON report for a single category

# --- Analysis (requires report-eval.json) ---
make summary           # Per-category pass/fail breakdown
make classify          # Group failures by error type
make failures-eval     # List all eval failures with type
make failures-eval CAT=functions
                       # Filter failures to one category

# --- Investigating specific tests ---
make investigate-eval TEST=substring01
                       # Search eval report for a test
make show-query TEST=syntax-select-expr-04.rq
                       # Print the .rq file for a test
make clean             # Remove generated report files
```

## Understanding the Output

### Test Summary

After running, the suite prints:

```
=== Test Summary ===
Total:   94
Passed:  79
Ignored: 0
Failed:  15
```

- **Total**: Number of W3C test cases discovered from manifest files
- **Passed**: Tests where Fluree's behavior matched the W3C expectation
- **Ignored**: Tests in the skip list (should be near zero)
- **Failed**: Tests where Fluree diverged from the spec — these are bugs or gaps

### Failure Messages

Each failure includes the test ID, type, and error details:

```
https://w3c.github.io/rdf-tests/sparql/sparql11/syntax-query/manifest.ttl#test_34:
  Positive syntax test failed — parser rejected valid query.
  Test: ...#test_34
  File: .../syntax-query/syntax-select-expr-04.rq
```

For syntax tests, failures fall into three categories:

| Failure Type            | What It Means                 | Example                                                 |
| ----------------------- | ----------------------------- | ------------------------------------------------------- |
| **Positive test fails** | Parser rejects valid SPARQL   | Missing feature (subqueries, property path `\|`)        |
| **Negative test fails** | Parser accepts invalid SPARQL | Missing validation (BIND scope, GROUP BY scope)         |
| **Parser timeout**      | Parser enters infinite loop   | Bug in grammar handling (mitigated by safety-net forward-progress check) |

### Test IDs

Every test has a unique IRI like:

```
https://w3c.github.io/rdf-tests/sparql/sparql11/syntax-query/manifest.ttl#test_34
```

The fragment (`#test_34`) identifies the specific test within that manifest. The path tells you the W3C category (`syntax-query`, `aggregates`, `bind`, etc.).

## Analyzing Results

### Per-Category Breakdown

Use `make summary` to see pass/fail rates by W3C category:

```bash
make summary
```

This requires `report-eval.json` (generated automatically if missing). Output looks like:

```
Category                  Pass  Fail Total    Rate
----------------------------------------------------
syntax-query                80    14    94     85%
subquery                     8     6    14     57%
functions                   27    48    75     36%
...
----------------------------------------------------
TOTAL                      167   160   327   51.1%
```

### Error Classification

Use `make classify` to group failures by root cause:

```bash
make classify
```

Error types:
- **RESULT MISMATCH** — Query runs but returns wrong values
- **INTERNAL ERROR** — Execution fails with an internal error
- **PARSE/LOWERING** — SPARQL parsing or IR lowering fails
- **NEGATIVE SYNTAX** — Parser accepts a query it should reject
- **POSITIVE SYNTAX** — Parser rejects a query it should accept
- **EMPTY RESULTS** — Query returns no results when some were expected
- **NOT IMPLEMENTED** — Feature not yet implemented
- **PANIC** — Subprocess crashed (usually an index/unwrap bug)
- **TIMEOUT** — Test exceeded 5s (syntax) or 10s (eval) timeout

### Listing Failures

Use `make failures-eval` to list all failures with their type and first error line:

```bash
make failures-eval               # All failures
make failures-eval CAT=functions # Just one category
```

### JSON Reports

For programmatic analysis, generate a JSON report:

```bash
make report-eval-json    # → report-eval.json
make report-10-json      # → report-10.json
make cat-json CAT=bind   # → report-bind.json
```

Report format:

```json
{
  "total": 327, "passed": 167, "failed": 160, "pass_rate": "51.1%",
  "tests": [
    { "test_id": "http://...#agg01", "status": "pass", "error": null, "timeout": false },
    { "test_id": "http://...#agg02", "status": "fail", "error": "Results not isomorphic...", "timeout": false }
  ]
}
```

The analysis script at `scripts/analyze_report.py` can also be used directly:

```bash
python3 scripts/analyze_report.py summary report-eval.json
python3 scripts/analyze_report.py classify report-eval.json
python3 scripts/analyze_report.py failures report-eval.json --category functions
```

## From Failure to Fix: The Workflow

### Step 1: Identify the Failure Category

Run the suite and look at the failure message:

```bash
cargo test sparql11_syntax_query_tests -- --nocapture 2>&1 | tail -40
```

Determine which category:

- **Parser timeout** → Bug in `fluree-db-sparql` grammar rules causing infinite loop (mitigated by safety-net forward-progress check in `parse_group_graph_pattern()`, but can still occur in other parse entry points)
- **Positive syntax rejected** → Missing parser feature or incorrect grammar
- **Negative syntax accepted** → Missing semantic validation pass
- **Query evaluation mismatch** → Bug in query engine, data loading, or result formatting

### Step 2: Find the Test Query

Every W3C test references a `.rq` (query) or `.ru` (update) file. The failure message includes the file URL. Map it to a local path:

```
URL:   https://w3c.github.io/rdf-tests/sparql/sparql11/syntax-query/syntax-select-expr-04.rq
Local: testsuite-sparql/rdf-tests/sparql/sparql11/syntax-query/syntax-select-expr-04.rq
```

The pattern: strip `https://w3c.github.io/rdf-tests/` and prepend `testsuite-sparql/rdf-tests/`.

Read the query to understand what SPARQL feature is being tested:

```bash
cat testsuite-sparql/rdf-tests/sparql/sparql11/syntax-query/syntax-select-expr-04.rq
```

### Step 3: Reproduce in Isolation

Try parsing the query directly to see the exact error:

```rust
// Quick test in fluree-db-sparql
let output = fluree_db_sparql::parse_sparql("SELECT (1 + ?x AS ?y) WHERE { ?x ?p ?o }");
println!("has_errors: {}", output.has_errors());
for err in output.errors() {
    println!("  error: {err:?}");
}
```

If you suspect an infinite loop, the subprocess timeout will catch it automatically when run via the harness.

### Step 4: Investigate the Root Cause

For **parser issues**, the relevant code is in `fluree-db-sparql/`. Start with:

- `src/parser/` — Grammar rules and parser combinators
- `src/ast/` — AST types the parser emits

For **query evaluation issues**, the chain is:

1. `fluree-db-sparql` → parses to `SparqlAst`
2. `fluree-db-query` → evaluates the AST against a ledger
3. `fluree-db-api` → orchestrates ledger creation and query execution

### Step 5: Create an Issue

Use this template:

```markdown
## W3C SPARQL Compliance: [short description]

**Test ID:** `https://w3c.github.io/rdf-tests/sparql/sparql11/[category]/manifest.ttl#[test_name]`
**Category:** [syntax-query | aggregates | bind | etc.]
**Failure type:** [parser timeout | positive syntax rejected | negative syntax accepted | evaluation mismatch]

### Test Query

\`\`\`sparql
[paste the .rq file contents]
\`\`\`

### Expected Behavior

[For positive syntax: should parse successfully]
[For negative syntax: should be rejected]
[For evaluation: expected results from the .srx/.srj file]

### Actual Behavior

[Error message or incorrect output]

### Root Cause Analysis

[What part of the code needs to change and why]

### W3C Spec Reference

[Link to relevant section of https://www.w3.org/TR/sparql11-query/]
```

### Step 6: Fix and Verify

After making code changes:

```bash
# Verify the specific test passes (from testsuite-sparql/)
cargo test sparql11_syntax_query_tests -- --nocapture 2>&1 | grep "test_34"

# Verify you haven't regressed other tests
make count-eval

# Run the parser's own tests (from workspace root)
cd .. && cargo test -p fluree-db-sparql

# Full CI parity check
cargo clippy -p fluree-db-sparql --all-features -- -D warnings
```

## Using Claude Code for Debugging

Claude Code is particularly effective for SPARQL compliance work because each failure is self-contained: a query file, an expected behavior, and a specific error. Here's how to give a session full context.

### Prompt Template for Parser Failures

```
I'm working on W3C SPARQL compliance in Fluree. The following test is failing:

Test ID: https://w3c.github.io/rdf-tests/sparql/sparql11/syntax-query/manifest.ttl#test_34
Category: Positive syntax test (parser should accept this query but rejects it)

The query file is at: testsuite-sparql/rdf-tests/sparql/sparql11/syntax-query/syntax-select-expr-04.rq

The SPARQL parser is in fluree-db-sparql/. The parse entry point is
`parse_sparql()` which returns `ParseOutput<SparqlAst>` — check `has_errors()`.

Please:
1. Read the failing query file
2. Understand what SPARQL feature it tests
3. Find the relevant parser grammar in fluree-db-sparql/src/parser/
4. Identify why the parser rejects this input
5. Propose a fix
```

### Prompt Template for Query Evaluation Failures

```
I'm working on W3C SPARQL compliance. This query evaluation test is failing:

Test ID: https://w3c.github.io/rdf-tests/sparql/sparql11/aggregates/manifest.ttl#agg01
Test data: testsuite-sparql/rdf-tests/sparql/sparql11/aggregates/agg01.ttl
Query file: testsuite-sparql/rdf-tests/sparql/sparql11/aggregates/agg01.rq
Expected results: testsuite-sparql/rdf-tests/sparql/sparql11/aggregates/agg01.srx

The test harness creates an in-memory Fluree ledger, loads the data via
stage_owned().insert_turtle(), executes the query via query_sparql(), and
compares results.

Actual output: [paste actual output]
Expected output: [paste expected from .srx file]

Please investigate why the results differ and propose a fix.
```

### Key Files to Reference

When asking Claude Code for help, these files provide essential context:

| Context Needed                                  | File(s)                                           |
| ----------------------------------------------- | ------------------------------------------------- |
| Test harness architecture                       | `testsuite-sparql/src/lib.rs`, `src/evaluator.rs` |
| Subprocess timeout isolation                    | `testsuite-sparql/src/subprocess.rs`              |
| Subprocess worker binary                        | `testsuite-sparql/src/bin/run_w3c_test.rs`        |
| How manifests are parsed                        | `testsuite-sparql/src/manifest.rs`                |
| Syntax test handlers                            | `testsuite-sparql/src/sparql_handlers.rs`         |
| Eval test handler (data load + query + compare) | `testsuite-sparql/src/query_handler.rs`           |
| Expected result parsing (.srx/.srj)             | `testsuite-sparql/src/result_format.rs`           |
| Isomorphic result comparison                    | `testsuite-sparql/src/result_comparison.rs`       |
| SPARQL parser entry point                       | `fluree-db-sparql/src/lib.rs` (`parse_sparql()`)  |
| Parser grammar rules                            | `fluree-db-sparql/src/parser/`                    |
| SPARQL AST types                                | `fluree-db-sparql/src/ast/`                       |
| Query engine                                    | `fluree-db-query/src/`                            |
| API orchestration                               | `fluree-db-api/src/`                              |
| W3C SPARQL test categories                      | `testsuite-sparql/tests/w3c_sparql.rs`            |

### Batch Processing Tips

When multiple tests fail for the same root cause (e.g., "all BIND tests timeout"), group them:

```
These 3 tests all timeout in the parser on BIND expressions:
- test_34: SELECT (1 + ?x AS ?y)
- test_40: SELECT (CONCAT(?x, "!") AS ?label)
- test_65: subquery with SELECT expression

All are in testsuite-sparql/rdf-tests/sparql/sparql11/syntax-query/.

The parser code for BIND is in fluree-db-sparql/src/parser/. Please find the
common root cause and fix all three.
```

## JSON-LD Query Parity

SPARQL and JSON-LD queries in Fluree compile to the **same intermediate representation** (`fluree-db-query/src/ir.rs`) and share the entire execution engine. This means:

1. **Shared code changes affect both languages.** If you add a new `Expression` variant, `Pattern` variant, or `AggregateFn` for SPARQL, it automatically becomes available to JSON-LD query as well. Ensure JSON-LD tests still pass.

2. **New SPARQL features may need JSON-LD test coverage.** If a feature you're implementing for SPARQL compliance (e.g., a new built-in function, a new filter operator) is also expressible in JSON-LD query syntax, add corresponding JSON-LD integration tests.

3. **Some features are SPARQL-only.** Property paths, RDF-star, ASK query form, and SPARQL Update don't have JSON-LD equivalents. These don't require parity testing.

### Where to add parity tests

| Language | Test files |
| -------- | ---------- |
| SPARQL   | `fluree-db-api/tests/it_query_sparql.rs` |
| JSON-LD  | `fluree-db-api/tests/it_query.rs`, `it_query_analytical.rs`, `it_query_grouping.rs` |
| Shared   | Unit tests in `fluree-db-query/src/` modules |

### Validation after shared-code changes

```bash
# SPARQL W3C tests (from testsuite-sparql/)
make test-eval-cat CAT=<category>

# JSON-LD query tests (from workspace root) — integration tests are grouped
# into grp_* binaries; run a whole group or filter to one file's module.
cargo test -p fluree-db-api --test grp_query
cargo test -p fluree-db-api --test grp_query_sparql
# e.g. just one file's tests: cargo test -p fluree-db-api --test grp_query it_query_jsonld
```

## Architecture Overview

### Crate Structure

```
testsuite-sparql/
├── Cargo.toml                      # Excluded from workspace, publish = false
├── Makefile                        # Developer convenience targets
├── scripts/
│   └── analyze_report.py           # JSON report analysis (summary, classify, failures)
├── src/
│   ├── lib.rs                      # check_testsuite() entry point
│   ├── vocab.rs                    # W3C namespace constants (mf:, qt:, etc.)
│   ├── files.rs                    # URL → local file path mapping
│   ├── manifest.rs                 # TestManifest: Iterator<Item=Test>
│   ├── evaluator.rs                # TestEvaluator: type → handler dispatch
│   ├── sparql_handlers.rs          # Handler registration (syntax + eval)
│   ├── query_handler.rs            # QueryEvaluationTest: load data, run query, compare
│   ├── subprocess.rs               # Subprocess isolation for timeout enforcement
│   ├── result_format.rs            # Parse .srx/.srj expected result files
│   ├── result_comparison.rs        # Isomorphic result comparison (blank node mapping)
│   ├── report.rs                   # JSON report generation
│   └── bin/
│       └── run_w3c_test.rs         # Subprocess worker binary
├── tests/
│   └── w3c_sparql.rs               # Test entry points (syntax + 12 eval categories)
└── rdf-tests/                      # Git submodule → github.com/w3c/rdf-tests
```

### How It Works

**1. Manifest Parsing** (`manifest.rs`): `TestManifest` implements `Iterator<Item = Result<Test>>`. It loads `manifest.ttl` files using Fluree's own Turtle parser, follows `mf:include` links recursively, and extracts per-test metadata: type, query file, data file, expected results.

**2. Handler Dispatch** (`evaluator.rs`): `TestEvaluator` maps test type URIs (e.g., `mf:PositiveSyntaxTest11`) to handler functions. For each test, it finds the matching handler and invokes it.

**3. SPARQL Handlers** (`sparql_handlers.rs` + `query_handler.rs`): The Fluree-specific logic. Both syntax and evaluation tests run in isolated **subprocesses** via the `run-w3c-test` binary (`subprocess.rs`). For syntax tests, the subprocess calls `parse_sparql()` + `validate()` and reports whether errors were found (5-second timeout). For evaluation tests, the subprocess creates an in-memory Fluree ledger, loads Turtle test data, executes the SPARQL query, and compares results against expected `.srx`/`.srj` files using isomorphic matching (10-second timeout). If a test exceeds its timeout, the parent kills the child process — no zombie threads.

**4. Test Entry Points** (`tests/w3c_sparql.rs`): Each test function is ~5 lines — just a manifest URL and a skip list. The harness does the rest.

### Key Design Decisions

- **Subprocess isolation** for all test execution. Each syntax and eval test runs in a child process (`run-w3c-test` binary) that can be killed on timeout. This prevents zombie threads from parser infinite loops or runaway queries.
- **Syntax timeout: 5 seconds, eval timeout: 10 seconds.** If a test exceeds its limit, the subprocess is killed and the test is marked as a timeout failure.
- **Uses Fluree's own Turtle parser** for manifest files. If our parser can't handle well-formed W3C manifests, that's a bug worth knowing about.
- **Fluree's `list_index`** approach (instead of `rdf:first/rdf:rest`) simplifies manifest list handling.
- **`@base` prepended** to manifest files since they use `<>` (empty relative IRI) which requires a base.

## Test Categories

### Syntax Tests (Phase 1)

| Suite             | What It Tests                             | Manifest                    |
| ----------------- | ----------------------------------------- | --------------------------- |
| SPARQL 1.1 syntax | Parser correctness for SPARQL 1.1 grammar | `syntax-query/manifest.ttl` |
| SPARQL 1.0 syntax | Backward compatibility with SPARQL 1.0    | `manifest-syntax.ttl`       |

### Query Evaluation Tests (Phase 2)

Each test creates an in-memory Fluree ledger, loads RDF data, executes a SPARQL query, and compares results against W3C expected outputs. Run with `make test-eval-cat CAT=<name>`.

| Suite              | What It Tests                                   | Manifest                          |
| ------------------ | ----------------------------------------------- | --------------------------------- |
| Aggregates         | COUNT, SUM, AVG, MIN, MAX, GROUP_CONCAT, SAMPLE | `aggregates/manifest.ttl`         |
| BIND               | BIND expressions, variable assignment           | `bind/manifest.ttl`               |
| Bindings           | VALUES inline data                              | `bindings/manifest.ttl`           |
| Cast               | xsd:integer(), xsd:double(), xsd:string()       | `cast/manifest.ttl`               |
| Construct          | CONSTRUCT query form                            | `construct/manifest.ttl`          |
| Exists             | FILTER EXISTS, FILTER NOT EXISTS                | `exists/manifest.ttl`             |
| Functions          | String, numeric, date/time, hash, IRI functions | `functions/manifest.ttl`          |
| Grouping           | GROUP BY semantics, error handling              | `grouping/manifest.ttl`           |
| Negation           | MINUS, NOT EXISTS                               | `negation/manifest.ttl`           |
| Project-Expression | SELECT expressions, AS aliases                  | `project-expression/manifest.ttl` |
| Property-Path      | `/`, `\|`, `^`, `+`, `*`, `?` operators         | `property-path/manifest.ttl`      |
| Subquery           | Nested SELECT within WHERE                      | `subquery/manifest.ttl`           |

### BIND / VALUES Compliance Notes

**BIND** (10/10 — 100%):
- Fixed lexer to tokenize `+`/`-` as separate operators per the SPARQL spec (`INTEGER` is unsigned; `INTEGER_POSITIVE`/`INTEGER_NEGATIVE` are grammar-level). This fixed `?o+10` being mis-tokenized as `Var, Integer(10)` instead of `Var, Plus, Integer(10)`.
- BIND input variable liveness is handled by `precompute_suffix_vars` (cross-block) and `pending_binds.expr.variables()` (within-block) in the WHERE planner — no special handling needed in `compute_variable_deps`.
- Explicitly nested `{ }` blocks inside WHERE are lowered as anonymous subqueries (`SubqueryPattern`) to preserve SPARQL scope boundaries (bind10).

**VALUES / Bindings** (10/11 — 91%):
- Post-query VALUES (`WHERE { ... } VALUES ?x { ... }`) is now parsed and lowered. Added `values` field on `SelectQuery` AST and `post_values` field on `ParsedQuery` to prevent the planner from reordering it relative to OPTIONAL/UNION.
- `NestedLoopJoinOperator::combine_rows` fixed to handle `Unbound`/`Poisoned` left-side shared variables by falling back to right-side values. This fixes VALUES with UNDEF (values4, values5, values8).
- `ValuesOperator` updated to treat `Poisoned` (from failed OPTIONAL) as wildcard in `is_compatible` and `merge_rows`, fixing values7 (OPTIONAL + VALUES).
- Remaining failure: `graph` test requires named graph support (GRAPH keyword) — tracked separately.

## Managing the Skip List

Skip entries are the `ignored_tests` parameter in `check_testsuite()` calls:

```rust
check_testsuite(
    "https://w3c.github.io/rdf-tests/sparql/sparql11/syntax-query/manifest.ttl",
    &[
        // Deliberately accept bare `1` as integer literal (RDF 1.1 vs 1.0)
        // Spec: https://www.w3.org/TR/sparql11-query/#rNumericLiteral
        // Reviewed: 2025-02-15 by @ajohnson, @bsmith
        "https://...#test_99",
    ],
)
```

**Rules:**

1. Start with an **empty** skip list. Expect full compliance.
2. Only add entries after investigation confirms a _deliberate_ design choice, not a bug.
3. Every skip entry must have a comment explaining why, linking to the relevant spec section.
4. Skip entries require review by 2+ team members.
5. The total skip list should be <5% of tests (Oxigraph skips ~25 out of 700+).
6. Review skip entries periodically — remove them as features are added.

## Updating the rdf-tests Submodule

The W3C test data lives in a git submodule at `testsuite-sparql/rdf-tests/`. To update to the latest W3C tests:

```bash
cd testsuite-sparql/rdf-tests
git pull origin main
cd ../..
git add testsuite-sparql/rdf-tests
git commit -m "chore: update W3C rdf-tests submodule"
```

After updating, run the full suite to check for new tests or changed expectations:

```bash
cd testsuite-sparql
cargo test
```

## Related Documentation

- [Tests guide](tests.md) — General testing practices
- [SPARQL query docs](../query/sparql.md) — User-facing SPARQL feature documentation
- [Compatibility](../reference/compatibility.md) — Standards compliance status
- [Crate map](../reference/crate-map.md) — Workspace architecture


---

# contributing/shacl-implementation.md

# SHACL Implementation

This is the contributor-facing guide to how SHACL validation is wired into Fluree. It covers the pipeline, the crate layout, and the places you'll want to touch when fixing a bug or adding a constraint.

User-facing docs: [Cookbook: SHACL Validation](../guides/cookbook-shacl.md) and [Setting Groups — SHACL](../ledger-config/setting-groups.md#shacl-defaults).

## Pipeline at a glance

```
Transaction flakes
        │
        ▼
┌─────────────────────────────────────────────────────────────────┐
│ fluree-db-transact :: stage()                                   │
│   stages flakes into a StagedLedger (novelty overlay)             │
└─────────────────────────────────────────────────────────────────┘
        │
        ▼
┌─────────────────────────────────────────────────────────────────┐
│ fluree-db-api :: apply_shacl_policy_to_staged_view()            │
│   (shared post-stage helper — called from every write surface)  │
│                                                                 │
│  1. load_transaction_config(ledger)                             │
│  2. build_per_graph_shacl_policy(config, graph_delta)           │
│     → HashMap<GraphId, ShaclGraphPolicy>                        │
│  3. resolve_shapes_source_g_ids(config, snapshot)               │
│     → Vec<GraphId>  (where to compile shapes from)              │
│  4. ShaclEngine::from_dbs_with_overlay(&[GraphDbRef], ledger)   │
│  5. validate_view_with_shacl(view, cache, ..., per_graph_policy)│
│     → ShaclValidationOutcome { reject, warn }                   │
│  6. log warn bucket; propagate ShaclViolation for reject bucket │
└─────────────────────────────────────────────────────────────────┘
```

## Crate layout

| Crate | Role |
|-------|------|
| `fluree-db-shacl` | SHACL engine: shape compilation, cache, per-node validation, constraint evaluators. **No transaction-layer concerns.** |
| `fluree-db-transact` | Staged-validation plumbing: `validate_view_with_shacl`, `validate_staged_nodes`. Knows about `StagedLedger`, staged flakes, and graph routing. Defines the per-graph policy types. |
| `fluree-db-api` | Config resolution, policy building, and the shared helper that every write surface (JSON-LD, Turtle, commit replay) calls through. |

SHACL is feature-gated (`shacl`). See [Standards and feature flags](../reference/compatibility.md).

## The shared post-stage helper

All SHACL-enforced write surfaces route through **`apply_shacl_policy_to_staged_view`** in `fluree-db-api/src/tx.rs`:

```rust
pub(crate) async fn apply_shacl_policy_to_staged_view(
    view: &StagedLedger,
    ctx: StagedShaclContext<'_>,
) -> Result<(), TransactError>
```

`StagedShaclContext` carries everything that varies between call sites:

| Field | Populated by JSON-LD txn | Populated by Turtle insert | Populated by commit replay |
|-------|-------------------------|----------------------------|----------------------------|
| `graph_delta`  | `Some(&txn.graph_delta)` (IRIs) | `None` | `Some(&routing.graph_iris)` |
| `graph_sids`   | `Some(&graph_sids)` | `None` | `Some(&routing.graph_sids)` |
| `tracker`      | `options.tracker` | `None` | `None` |

Why not fold this into `fluree-db-transact`? Config resolution (three-tier merge, override control, per-graph lookup) is API-layer policy, not a staging primitive. Keeping the helper in `tx.rs` lets `fluree-db-transact` stay focused on staging mechanics.

Call sites:
- `fluree-db-api/src/tx.rs::stage_with_config_shacl` (JSON-LD / SPARQL UPDATE txns)
- `fluree-db-api/src/tx.rs::stage_turtle_insert` (plain Turtle)
- `fluree-db-api/src/commit_transfer.rs` (push / replay)

## Config resolution

### Ledger-wide and per-graph policy

`build_per_graph_shacl_policy(config, graph_delta)` returns `Option<HashMap<GraphId, ShaclGraphPolicy>>`:

- Graphs **absent from the map** are **disabled** — their staged subjects are skipped by the validator.
- `ShaclGraphPolicy { mode: ValidationMode }` controls warn vs reject for that graph.
- The default graph (g_id=0) always gets the ledger-wide resolved policy when SHACL is enabled.
- Every graph in `graph_delta` is resolved independently via `config_resolver::resolve_effective_config(config, Some(graph_iri))`, which applies the three-tier merge (query-time → per-graph → ledger-wide) under override-control rules.
- Returns `None` when every graph resolves to disabled → the helper short-circuits before building the SHACL engine.

The transact layer's `validate_view_with_shacl` signature:

```rust
pub async fn validate_view_with_shacl(
    view: &StagedLedger,
    shacl_cache: &ShaclCache,
    graph_sids: Option<&HashMap<GraphId, Sid>>,
    tracker: Option<&Tracker>,
    per_graph_policy: Option<&HashMap<GraphId, ShaclGraphPolicy>>,
) -> Result<ShaclValidationOutcome>
```

- `per_graph_policy = None`: treat every graph with staged flakes as `Reject` (legacy / shapes-exist-heuristic path).
- `per_graph_policy = Some(map)`: only graphs in the map participate; their mode drives the warn/reject split.

Output:

```rust
pub struct ShaclValidationOutcome {
    pub reject_violations: Vec<ValidationResult>,
    pub warn_violations: Vec<ValidationResult>,
}
```

The API helper logs the warn bucket and returns `TransactError::ShaclViolation` for the reject bucket.

### `f:shapesSource` resolution

`resolve_shapes_source_g_ids(config, snapshot)` in `tx.rs` is the sibling of `policy_builder::resolve_policy_source_g_ids` — identical shape, different namespace. Both:

1. Start with `[0]` (default graph) when the source field is unset.
2. Map `f:defaultGraph` → `[0]`.
3. Map a named graph IRI to its registered `GraphId` via `snapshot.graph_registry.graph_id_for_iri`.
4. Reject unsupported dimensions: `f:atT`, `f:trustPolicy`, `f:rollbackGuard`, cross-ledger `f:ledger` (these surface as `TransactError::Parse`).

`f:shapesSource` is **authoritative, not additive** — when set, shapes come exclusively from the configured graph(s). It's intentionally non-overridable at query/txn time; it can only be changed via a config-graph transaction.

## Shape compilation from multiple graphs

`ShapeCompiler::compile_from_dbs(&[GraphDbRef])` in `fluree-db-shacl/src/compile.rs` scans each input graph for every SHACL predicate (see the `shacl_predicates` list), accumulates into a single `ShapeCompiler`, then finalizes. Cross-graph `sh:and` / `sh:or` / `sh:xone` / `sh:in` list references still resolve because finalization runs once after all graphs are consumed.

`ShaclEngine::from_dbs_with_overlay(&[GraphDbRef], ledger_id)` is the corresponding engine constructor. `from_db_with_overlay(db, ledger_id)` is a single-graph convenience that delegates to the multi-graph path via `slice::from_ref(&db)`.

The engine's `SchemaHierarchy` is taken from the first graph's snapshot — hierarchy is schema-level and not graph-scoped.

## Target-type resolution

The cache (`fluree-db-shacl/src/cache.rs`) holds four indexes:

| Field | Keyed by | Used for |
|-------|----------|----------|
| `by_target_class` | class Sid (with `rdfs:subClassOf*` expansion) | `sh:targetClass` |
| `by_target_node` | subject Sid | `sh:targetNode` |
| `by_target_subjects_of` | predicate Sid | `sh:targetSubjectsOf` |
| `by_target_objects_of` | predicate Sid | `sh:targetObjectsOf` |

`ShaclEngine::validate_node` assembles applicable shapes for a focus node by:

1. `shapes_for_node(focus)` — O(1) hashmap hit.
2. `shapes_for_class(type)` for each of the focus's `rdf:type` values — O(1) per type.
3. For each key `p` in `by_target_subjects_of`: existence check `db.range(SPOT, s=focus, p=p)` — if non-empty, shape applies.
4. For each key `p` in `by_target_objects_of`: existence check `db.range(OPST, p=p, o=focus)` — if non-empty, shape applies.

Why the live db check for steps 3/4 instead of precomputed staged-flake hints? Three scenarios a hint-only approach can't cover:

- **Base-state edge**: the triggering edge is already indexed; the current txn only touches another property.
- **Retraction-only**: the staged flake set for a focus contains retractions that don't remove the last matching edge.
- **Cross-graph routing**: a subject's edge exists in graph A but we're validating the subject in graph B — the per-graph db ref sees only B.

`db.range()` returns only post-state assertions (retractions are filtered in the range pipeline — see `fluree-db-core/src/range.rs`), so the check is exactly "is this edge present in the post-txn view of this graph".

Cost is bounded by the number of predicate-targeted shapes in the cache, not by data size — typically 0–10 per ledger.

## Staged validation loop

`validate_staged_nodes` in `fluree-db-transact/src/stage.rs`:

1. Partition staged flakes into `subjects_by_graph: HashMap<GraphId, HashSet<Sid>>`.
   - Every flake's subject is added (including retractions — class/node targets still need to see them).
   - Every **assert** flake's Ref-object is also added to the graph's focus set (ensures `sh:targetObjectsOf` shapes fire on newly-referenced nodes).
2. For each `(g_id, subjects)`:
   - If `enabled_graphs` is `Some` and `g_id` is not in it: **skip**.
   - Build a per-graph `GraphDbRef` with `view` as overlay and `view.staged_t()` as `t`.
   - Attach the tracker (if any) — fuel accounting works for SHACL range scans too.
   - For each subject: fetch `rdf:type` flakes, then call `engine.validate_node(db, subject, &types)`.
   - Tag each returned `ValidationResult` with `graph_id = Some(g_id)` so the caller can partition reject vs warn.

### RDFS subclass fallback (`is_subclass_of`)

When the indexed `SchemaHierarchy` doesn't know about a `rdfs:subClassOf` edge (e.g. asserted in the same or a recent unindexed transaction), `validate_class_constraint` calls `is_subclass_of(db, start, target)` which walks `rdfs:subClassOf` upward via BFS.

Two invariants in that walk:

- **Always scope to `g_id=0`** via `rescope_to_schema_graph(db)` — schema lives in the default graph, matching how `SchemaHierarchy::from_db_root_schema` is built. Subject may be in graph G but the `subClassOf` edge must be looked up in the schema graph.
- **Preserve tracker + other `GraphDbRef` fields** — `rescope_to_schema_graph` uses `db` copy + `g_id = 0` mutation rather than `GraphDbRef::new(..)`, which would reset `tracker`, `runtime_small_dicts`, and `eager`. There's a unit test pinning this (`rescope_to_schema_graph_preserves_tracker_and_other_fields`).

## Adding a new constraint

### 1. Compile

In `fluree-db-shacl/src/compile.rs`:

- Add a variant to the `Constraint` enum (or `NodeConstraint` for node-level).
- Add the predicate name to the `shacl_predicates` array in `ShapeCompiler::compile_from_dbs`.
- Handle the predicate in `process_flake` (sets the right field on the intermediate shape builder).
- If the constraint takes arguments via an RDF list, extend `expand_rdf_lists`.

### 2. Validate

Pure per-value constraints (no db access) go in `fluree-db-shacl/src/constraints/`:

- Add a `validate_<name>(values, ..) -> Option<ConstraintViolation>` helper next to the similar ones in `cardinality.rs` / `value.rs` / etc.
- Wire it into the big match in `validate_constraint` in `fluree-db-shacl/src/validate.rs`.

Constraints that need database access (`sh:class`, pair constraints) are handled **before** the pure dispatch, inside `validate_property_shape`. Pattern:

```rust
Constraint::MyConstraint(target) => {
    let helper_violations = validate_my_constraint(db, &values, target).await?;
    for v in helper_violations {
        results.push(ValidationResult {
            focus_node: focus_node.clone(),
            result_path: Some(prop_shape.path.clone()),
            source_shape: parent_shape.id.clone(),
            source_constraint: Some(prop_shape.id.clone()),
            severity: prop_shape.severity,
            message: v.message,
            value: v.value,
            graph_id: None, // tagged later in validate_staged_nodes
        });
    }
}
```

### 3. Advertise

Update `fluree-db-shacl/src/lib.rs`:
- Add the constraint to the **Supported Constraints** list.
- Remove from the **Not Yet Supported** section if it was listed.

### 4. Test

- Add a unit test next to your `validate_<name>` helper for the pure logic.
- Add an integration test in `fluree-db-api/src/shacl_tests.rs` that transacts a shape + violating data + valid data.
- For a bug fix: temp-revert the fix, confirm the test fails, restore, confirm it passes. This pins the regression into the test.

## Testing patterns

### Integration tests

Most SHACL integration tests live in `fluree-db-api/src/shacl_tests.rs` and use the `assert_shacl_violation(err, "substring")` helper. Pattern:

```rust
let shape = json!({ /* sh:NodeShape with the constraint under test */ });
let ledger = fluree.create_ledger("shacl/foo:main").await.unwrap();
let ledger = fluree.upsert(ledger, &shape).await.unwrap().ledger;

// Negative case
let err = fluree.upsert(ledger, &violating_data).await.unwrap_err();
assert_shacl_violation(err, "expected message fragment");

// Positive case
fluree.upsert(ledger, &valid_data).await.expect("must pass");
```

### Cross-graph / per-graph tests

See `fluree-db-api/tests/it_config_graph.rs` for patterns that write config via TriG into the config graph, then stage transactions across multiple graphs. Examples:

- `shacl_shapes_source_points_to_named_graph` — `f:shapesSource` wiring.
- `shacl_per_graph_disable_honored` — per-graph `shaclEnabled: false`.
- `shacl_per_graph_mode_warn_vs_reject` — mixed modes across graphs.
- `shacl_target_subjects_of_fires_on_base_state_edge` — base-state predicate-target discovery.

### The temp-revert trick

For every correctness-fix PR, confirm the regression test actually covers the bug:

1. Apply the minimum temp-revert in the production code (comment out the fix with a `// TEMP REVERT:` marker).
2. Run the new test — it should **fail** with the expected symptom.
3. Restore the fix — test passes.
4. Commit the fix + the test together.

This is how we guard against tests that pass trivially but don't actually exercise the fix.

## Known gaps

- **`sh:uniqueLang`, `sh:languageIn`** — parsed but not evaluated. Needs language-tag metadata on flakes, which isn't yet threaded through the validation path.
- **`sh:qualifiedValueShape` (+ `sh:qualifiedMinCount` / `sh:qualifiedMaxCount`)** — parsed but not evaluated. Needs recursive nested-shape counting.
- **Cross-transaction shape cache** — every call to `from_dbs_with_overlay` recompiles from scratch. `ShaclCacheKey` has a `schema_epoch` field that's ready to drive a shared `Arc<ShaclCache>` cache on the connection, but nothing populates it yet. Low priority until perf regressions are observed.

## Where to look in the code

| What | File |
|------|------|
| Shape compilation (Turtle/JSON-LD → `CompiledShape`) | `fluree-db-shacl/src/compile.rs` |
| Shape cache with target indexes | `fluree-db-shacl/src/cache.rs` |
| Per-focus validation engine | `fluree-db-shacl/src/validate.rs` |
| Per-constraint validators (pure values) | `fluree-db-shacl/src/constraints/` |
| Staged-validation loop (per-graph) | `fluree-db-transact/src/stage.rs::validate_staged_nodes` |
| Public transact entry + outcome split | `fluree-db-transact/src/stage.rs::validate_view_with_shacl` |
| Policy types (`ShaclGraphPolicy`, `ShaclValidationOutcome`) | `fluree-db-transact/src/stage.rs` |
| Shared post-stage helper | `fluree-db-api/src/tx.rs::apply_shacl_policy_to_staged_view` |
| Per-graph policy builder | `fluree-db-api/src/tx.rs::build_per_graph_shacl_policy` |
| `f:shapesSource` resolver | `fluree-db-api/src/tx.rs::resolve_shapes_source_g_ids` |
| JSON-LD / SPARQL txn call site | `fluree-db-api/src/tx.rs::stage_with_config_shacl` |
| Turtle insert call site | `fluree-db-api/src/tx.rs::stage_turtle_insert` |
| Commit replay call site | `fluree-db-api/src/commit_transfer.rs` |
| Config field definition | `fluree-db-core/src/ledger_config.rs::ShaclDefaults` |
| Config graph parser | `fluree-db-api/src/config_resolver.rs::read_shacl_defaults` |
| Effective-config merge | `fluree-db-api/src/config_resolver.rs::merge_shacl_opts` |

## Related

- [Cookbook: SHACL Validation](../guides/cookbook-shacl.md) — user-facing usage guide
- [Setting Groups — SHACL](../ledger-config/setting-groups.md#shacl-defaults) — config reference
- [Override Control](../ledger-config/override-control.md) — three-tier precedence and monotonicity rules
- [Crate map](../reference/crate-map.md) — layering overview


---

# contributing/tracing-guide.md

# Adding Tracing Spans to New Code

When you add or modify code paths in Fluree, you should instrument them with tracing spans so that performance investigations can decompose wall-clock time into meaningful phases. This guide explains the conventions, patterns, and gotchas.

## The Two-Tier Span Strategy

Fluree uses a tiered approach so that tracing is **zero-overhead by default** but **deeply informative on demand**.

### The `request` span: `info_span!` (the one exception)

The HTTP `request` span in `telemetry.rs::create_request_span()` is the only `info_span!` in the codebase. It provides operators with HTTP request visibility at the production default `RUST_LOG=info`. All other operation spans are `debug_span!` — this guarantees true zero overhead when the `otel` feature is not compiled and `RUST_LOG` is at `info`.

### Tier 1: `debug_span!` -- operation and phase level

All operation spans (`query_execute`, `transact_execute`, `txn_stage`, `txn_commit`, `index_build`, `sort_blocking`, etc.) and their phases use `debug_span!`. They are visible when OTEL is enabled (the OTEL `Targets` filter registers interest at DEBUG for `fluree_*` crates) or when a developer sets `RUST_LOG=debug` or `RUST_LOG=info,fluree_db_query=debug`. Without either, `debug_span!` short-circuits to a single atomic load (~1-2ns, unmeasurable).

### Tier 2: `trace_span!` -- maximum detail

Per-operator, per-item, or per-iteration spans. Visible at `RUST_LOG=info,fluree_db_query=trace`. Use for fine-grained instrumentation in hot paths where you only want visibility during deep investigation. The OTEL `Targets` filter intentionally excludes TRACE to prevent flooding the batch processor.

### Decision guide

| You're adding... | Span level | Example |
|-------------------|-----------|---------|
| New top-level operation, phase, or operator | `debug_span!` | `query_execute`, `reasoning_prep`, `join`, `binary_open_leaf` |
| Detail or per-iteration instrumentation | `trace_span!` | `group_by`, `distinct`, `binary_cursor_next_leaf` |

**Do not use `info_span!`** for new operation spans. The `request` span is the sole exception.

## Code Patterns

### Sync phases (no `.await`)

Use `span.enter()` which creates a guard dropped at end of scope:

```rust
let span = tracing::debug_span!(
    "pattern_rewrite",
    patterns_before = patterns.len() as u64,
    patterns_after = tracing::field::Empty,  // recorded later
);
let _guard = span.enter();

// ... do the rewriting ...

span.record("patterns_after", rewritten.len() as u64);
// _guard dropped here, span ends
```

### Async phases (contains `.await`)

**Never** hold a `span.enter()` guard across an `.await` point. In tokio's multi-threaded runtime, `span.enter()` enters the span on the current thread. When the task yields at `.await`, the span remains "entered" on that thread. Other tasks polled on the same thread will then inherit this span as their parent, causing **cross-request trace contamination** — completely unrelated operations become nested under each other in Jaeger. This was the root cause of a critical trace corruption bug in the HTTP route handlers.

**Symptoms in Jaeger**: If you see sequential, independent requests nested as children of an earlier request (especially where child spans outlive their parents), the cause is almost certainly `span.enter()` held across `.await`.

Instead, use `.instrument(span)`:

```rust
let span = tracing::debug_span!(
    "format",
    output_format = %format_name,
    result_count = total_rows as u64,
);
format_results(batch, format).instrument(span).await
```

If you need to record deferred fields on a span that wraps an async block, use `Span::current()` inside the instrumented block:

```rust
let span = tracing::debug_span!(
    "txn_stage",
    insert_count = tracing::field::Empty,
    delete_count = tracing::field::Empty,
);
async {
    // ... do staging work ...
    let current = tracing::Span::current();
    current.record("insert_count", inserts as u64);
    current.record("delete_count", deletes as u64);
    Ok(result)
}.instrument(span).await
```

### HTTP route handlers (axum)

Route handlers are async and **must** use `.instrument()`. The standard pattern wraps the entire handler body in an `async move` block instrumented with the request span, then uses `Span::current()` inside:

```rust
pub async fn query(
    State(state): State<Arc<AppState>>,
    headers: FlureeHeaders,
) -> Result<impl IntoResponse> {
    let span = create_request_span("query", request_id.as_deref(), ...);

    async move {
        let span = tracing::Span::current(); // Same span, safe to .record() on
        tracing::info!(status = "start", "query request received");

        let alias = get_ledger_alias(...)?;
        span.record("ledger_alias", alias.as_str());

        execute_query(&state, &alias, &query_json).await
    }
    .instrument(span)
    .await
}
```

**Why `async move` + `Span::current()` instead of just `.instrument()`**: Route handlers need to record deferred fields (like `ledger_alias`, `error_code`) on the span after creation. By obtaining `Span::current()` inside the instrumented block, you get a handle to the same span that `.instrument()` entered, letting you call `.record()` and pass it to `set_span_error_code()`.

### spawn_blocking

For `tokio::task::spawn_blocking`, enter the span *inside* the closure:

```rust
let span = tracing::debug_span!("heavy_compute");
tokio::task::spawn_blocking(move || {
    let _guard = span.enter();
    // ... sync work ...
}).await
```

### std::thread::scope (parallel OS threads)

`std::thread::scope` spawned threads do NOT inherit `tracing` span context from the parent thread. Capture the current span before spawning and enter it inside each closure:

```rust
let parent_span = tracing::Span::current();

std::thread::scope(|s| {
    for item in &work_items {
        let thread_span = parent_span.clone();
        s.spawn(move || {
            let _guard = thread_span.enter();
            // ... work that creates child spans ...
        });
    }
});
```

This is safe because scoped threads are pure sync (no `.await`). The same pattern applies to any OS thread spawning (`std::thread::spawn`, `rayon`, etc.).

### Lightweight operators (hot path)

For simple operators that just need a span marker, use the terse `.entered()` pattern:

```rust
fn open(&mut self, ctx: &mut Context<S, C>) -> Result<()> {
    let _span = tracing::trace_span!("filter").entered();
    self.child.open(ctx)?;
    // ...
    Ok(())
}
```

## Deferred Fields

Declare fields as `tracing::field::Empty` at span creation, then record values later. This is essential for fields whose values aren't known until the operation completes.

```rust
let span = tracing::debug_span!(
    "plan",
    pattern_count = tracing::field::Empty,
);
let _guard = span.enter();

let plan = build_plan(&patterns)?;
span.record("pattern_count", plan.patterns.len() as u64);
```

**Gotcha:** `tracing::Span::current().record(...)` records on the *current innermost* span. If you've entered a child span, `.record()` targets the child, not the parent. Get a handle to the parent span before entering children:

```rust
let parent_span = tracing::debug_span!("outer", total = tracing::field::Empty);
let _parent_guard = parent_span.enter();

{
    let _child = tracing::trace_span!("inner").entered();
    // Span::current() is now "inner", NOT "outer"
}

// Back to "outer" scope -- safe to record on parent
parent_span.record("total", count as u64);
```

## `#[tracing::instrument]` vs Manual Spans

**Use `#[tracing::instrument]`** for simple functions where:
- You want span entry/exit to match the function boundary
- The function name is a good span name
- You don't need deferred field recording

Always use `skip_all` and explicitly list fields:

```rust
#[tracing::instrument(level = "debug", name = "parse", skip_all, fields(input_format, input_bytes))]
fn parse_query(input: &[u8], format: &str) -> Result<Query> {
    // ...
}
```

**Use manual spans** when:
- The span covers only *part* of a function
- You need a different name than the function
- You need deferred fields
- The function is a hot path (the `#[instrument]` macro captures all arguments by default unless you `skip_all`)

## Where to Add Spans

### New query feature

If you add a new phase to query execution (e.g., a new optimization pass):

1. Add a `debug_span!` in the code path
2. Add the span name to the hierarchy in `docs/operations/telemetry.md`
3. Add a test in `fluree-db-api/tests/it_tracing_spans.rs` verifying the span emits

### New operator

If you add a new query operator:

1. For core structural operators (scan, join, filter, project, sort), use `debug_span!` in `open()`
2. For detail operators (group_by, distinct, limit, offset, etc.), use `trace_span!` in `open()`
3. If it's a blocking/buffering operator (like sort), add a `debug_span!` timing span in `next_batch()`
4. Add a test verifying the span emits at the correct level

For lower-level remote storage diagnostics on the binary path, prefer short-lived `debug_span!` blocks around:
- leaf-open strategy selection (`binary_open_leaf`)
- remote leaf metadata reads (`binary_fetch_header_dir`)
- individual range reads (`binary_range_fetch`)
- leaflet cache hit/miss points (`binary_load_leaflet`)

These spans are intended for investigation of repeated remote I/O and cache effectiveness under query load.

### New transaction phase

If you add a new phase to transaction processing:

1. Add a `debug_span!` in the phase code
2. Record relevant counts/sizes as deferred fields
3. Add the span to the hierarchy in `docs/operations/telemetry.md`

### New background task

If you add a new background task (like indexing, garbage collection, compaction):

1. Add a `debug_span!` as the **trace root** (these are independent traces, not children of HTTP requests)
2. Add debug sub-spans for phases within the task
3. Ensure the crate target is listed in the OTEL `Targets` filter in `telemetry.rs`

## Testing Spans

All new spans should have at least one test verifying they emit with expected fields at the right level.

### Test utilities

The test infrastructure lives in `fluree-db-api/tests/support/span_capture.rs`:

```rust
mod support;
use support::span_capture;

#[tokio::test]
async fn my_new_span_emits_at_debug_level() {
    let (store, _guard) = span_capture::init_test_tracing(); // captures ALL levels

    // ... run the code that emits the span ...

    assert!(store.has_span("my_new_phase"));
    let span = store.find_span("my_new_phase").unwrap();
    assert_eq!(span.level, tracing::Level::DEBUG);
    assert!(span.fields.contains_key("some_field"));
}

#[tokio::test]
async fn my_new_span_not_visible_at_info() {
    let (store, _guard) = span_capture::init_info_only_tracing(); // captures only INFO+

    // ... run the code ...

    assert!(!store.has_span("my_new_phase")); // zero noise at info
}
```

### Test helpers available

- `span_capture::init_test_tracing()` -- captures all spans regardless of level (for verifying span existence)
- `span_capture::init_info_only_tracing()` -- captures only INFO+ (for verifying zero-noise at default level)
- `SpanStore::has_span(name)` -- check if a span was emitted
- `SpanStore::find_span(name)` -- get span details (level, fields, parent)
- `SpanStore::find_spans(name)` -- find all spans with a given name
- `SpanStore::span_names()` -- list all captured span names

### Where to put tests

- Tracing integration tests go in `fluree-db-api/tests/it_tracing_spans.rs`
- The test utilities are in `fluree-db-api/tests/support/span_capture.rs`

## OTEL Layer Configuration

If you add a new crate that emits spans that should be exported via OTEL, add it to the `Targets` filter in `fluree-db-server/src/telemetry.rs`:

```rust
let otel_filter = Targets::new()
    .with_target("fluree_db_server", Level::DEBUG)
    .with_target("fluree_db_api", Level::DEBUG)
    // ... existing targets ...
    .with_target("my_new_crate", Level::DEBUG);  // ADD THIS
```

Without this, spans from the new crate will appear in console logs but not in Jaeger/Tempo.

**Important:** All OTEL targets are set to DEBUG level. Do **not** set any target to TRACE in the OTEL filter — TRACE-level spans (e.g., `binary_cursor_next_leaf`, per-scan spans) can generate thousands of spans per query, overwhelming the `BatchSpanProcessor` queue and causing parent spans to be dropped. Users who need TRACE-level detail should use `RUST_LOG` for console output; the OTEL exporter intentionally excludes TRACE spans.

## Checklist for New Instrumentation

- [ ] Used `debug_span!` (not `info_span!`) for all new operation spans
- [ ] Used `span.enter()` only in sync code, `.instrument(span)` for async
- [ ] Propagated span context into spawned threads (`spawn_blocking`, `std::thread::scope`, etc.)
- [ ] Added deferred fields for values computed after span creation
- [ ] Tested span emission with `SpanCaptureLayer`
- [ ] Verified zero overhead at INFO level (no debug/trace spans appear without OTEL or `RUST_LOG=debug`)
- [ ] Updated span hierarchy in `docs/operations/telemetry.md` if adding spans
- [ ] Updated `.claude/skills/*/references/span-hierarchy.md` (both copies)
- [ ] Added new crate to OTEL `Targets` filter if applicable

## Common Gotchas

1. **`span.enter()` across `.await` causes cross-request contamination** -- This is the most dangerous tracing bug. In tokio's multi-threaded runtime, `span.enter()` sets the span on the current thread. When the task suspends at `.await`, the span stays "entered" on that thread. Other tasks polled on the same thread inherit it as their parent. **Result**: unrelated requests cascade into each other's traces in Jaeger, with child spans that outlive their parents. Always use `.instrument(span)` in async code. This was a real bug in the HTTP route handlers and took Jaeger analysis to identify.
2. **`Span::current().record()` targets the innermost span** -- not necessarily the one you intend. Hold a reference to the span you want to record on.
3. **OTEL exporter floods** -- if you set `RUST_LOG=debug` globally, third-party crates (hyper, tonic, h2) emit debug spans that overwhelm the OTEL batch processor. The `Targets` filter on the OTEL layer prevents this.
4. **Tower-HTTP `TraceLayer` removed** -- tower-http's `TraceLayer` was removed entirely because it created a duplicate `request` span that collided with Fluree's own `request` span in `create_request_span()`. If you re-add tower-http tracing, ensure it does not conflict.
5. **`set_global_default` in tests** -- can only be called once per process. Use `set_default()` which returns a guard scoped to the test.
6. **Compiler won't catch `span.enter()` across `.await`** -- Unlike what the tracing docs suggest, `Entered` may actually be `Send` (since `&Span` is `Send` when `Span: Sync`). The code compiles fine but produces incorrect traces at runtime. The only way to detect this is visual inspection in Jaeger. Grep for `span.enter()` in async functions as part of code review.
7. **`std::thread::scope` / `std::thread::spawn` drops span context** -- New OS threads start with empty thread-local span context, so any spans created on them become orphaned root traces. You must capture `Span::current()` and `.enter()` it inside the thread closure. This same issue applies to `tokio::task::spawn_blocking`, `rayon`, and any other thread-spawning API.

## Claude Code Trace Analysis Skills

Two Claude Code skills are available for analyzing Jaeger trace exports:

### `/trace-inspect`

Drills into a **single trace**: span tree visualization, timing breakdown, structural health checks. Use when you have a specific slow request and want to understand where time went.

```
/trace-inspect path/to/traces.json
```

### `/trace-overview`

Analyzes **all traces** in an export: aggregate statistics, anomaly detection across the corpus, comparison of query vs transaction patterns. Use when you want a high-level understanding of system behavior.

```
/trace-overview path/to/traces.json
```

### Exporting traces from Jaeger

1. Open Jaeger UI (default: `http://localhost:16686`)
2. Search for traces of interest (by service name, operation, duration, etc.)
3. Click the JSON download button on a trace or search result
4. Save to a file and pass to either skill

See the [OTEL dev harness](../../otel/README.md) for running a local Jaeger instance.

## Related Documentation

- [Performance Investigation](../troubleshooting/performance-tracing.md) -- How operators use deep tracing
- [Telemetry and Logging](../operations/telemetry.md) -- Configuration reference
- [Deep Tracing Playbook](../../dev-docs/deep-tracing-replay-playbook.md) -- Comprehensive implementation reference


---

# contributing/releasing.md

# Releasing

Cutting a release is two phases: a release PR that bumps the version, then a tag pushed after the PR merges. The tag triggers cargo-dist, which builds binaries and publishes the GitHub Release. Release notes are auto-generated by GitHub from merged PR titles since the previous tag.

## The flow

```bash
# 1. Branch off main and bump the workspace version.
git checkout main && git pull
git checkout -b release/v4.0.3
$EDITOR Cargo.toml                      # update [workspace.package].version
cargo update --workspace                # refresh Cargo.lock
git commit -am "release v4.0.3"
git push -u origin release/v4.0.3
gh pr create --title "release v4.0.3"

# 2. After the PR is reviewed and merged to main, tag the merge commit.
git checkout main && git pull
git tag v4.0.3
git push origin v4.0.3                  # ← triggers .github/workflows/release.yml
```

Watch the Actions tab. cargo-dist builds platform binaries, creates the GitHub Release with auto-generated notes, publishes the Homebrew formula, and pushes the multi-arch Docker image.

## Why the two-phase split

cargo-dist's release workflow triggers on **any** pushed `vX.Y.Z` tag regardless of which branch the tag points at. Holding the tag step until after merge ensures every release goes through PR review.

## How release notes are generated

`.github/workflows/release.yml` calls `gh release create --generate-notes`. GitHub builds the release body automatically from PRs merged since the previous tag, categorized per `.github/release.yml`:

| Label                              | Section            |
|------------------------------------|--------------------|
| `breaking-change`, `semver:major`  | Breaking Changes   |
| `feature`, `enhancement`, `feat`   | Features           |
| `bug`, `fix`                       | Bug Fixes          |
| `performance`, `perf`              | Performance        |
| `documentation`, `docs`            | Documentation      |
| (anything else)                    | Other Changes      |

Apply one of these labels to each PR before merging. Unlabeled PRs still appear under "Other Changes" with their full title and author credit, so categorization is nice-to-have, not required.

PR titles already follow the `feat:` / `fix:` / `docs:` convention from `CLAUDE.md`, which makes the unlabeled "Other Changes" list readable on its own.

## Rolling back

**During Phase 1, before pushing:**

```bash
git checkout main
git branch -D release/vX.Y.Z
```

**After the release PR is opened but before merge:**

Close the PR and delete the branch on GitHub. Nothing has shipped.

**After the tag is pushed but cargo-dist still running:**

```bash
git push origin :refs/tags/vX.Y.Z       # delete the remote tag
git tag -d vX.Y.Z                       # delete the local tag
```

Cancel the in-progress `Release` workflow run from the Actions tab.

**After cargo-dist created the GitHub Release:**

Delete the GitHub Release from the UI, then delete the tag (commands above). The merge commit on `main` stays in place — re-tag it once the issue is fixed, or supersede it with another release PR.

## Configuration files

- `.github/release.yml` — categorization for GitHub's auto-generated release notes.
- `dist-workspace.toml` — cargo-dist's distribution targets and installers.
- `.github/workflows/release.yml` — autogenerated by cargo-dist (with `allow-dirty = ["ci"]` set so our `--generate-notes` edit survives `dist init`).