Storage Modes

Fluree supports four storage modes, each optimized for different deployment scenarios. This document provides detailed information about each storage mode and guidance for choosing the right one.

Storage Modes

Memory Storage

In-memory storage for development and testing:

./fluree-db-server --storage memory

Characteristics:

Data stored in RAM only
No persistence (data lost on restart)
Fastest performance
No external dependencies

Use Cases:

Local development
Unit testing
Temporary/ephemeral databases
Prototyping

Limitations:

No durability (data lost on crash/restart)
Limited by available RAM
Single process only

File Storage

Local file system storage:

./fluree-db-server \
  --storage file \
  --data-dir /var/lib/fluree

Characteristics:

Data persisted to local disk
Survives server restarts
Good performance (SSD recommended)
Simple setup

Use Cases:

Single-server production
Development with persistence
Edge deployments
Small to medium scale

Limitations:

Single machine only
No built-in replication
Limited by disk capacity
No cross-region support

AWS Storage

Distributed storage using S3 and DynamoDB:

./fluree-db-server \
  --storage aws \
  --s3-bucket fluree-prod-data \
  --s3-region us-east-1 \
  --dynamodb-table fluree-nameservice \
  --dynamodb-region us-east-1

Characteristics:

Distributed, scalable storage
Multi-process coordination
Cross-region replication
High durability (99.999999999%)

Use Cases:

Multi-server production
High availability requirements
Geographic distribution
Cloud-native applications

Limitations:

Requires AWS account
Higher latency than local storage
Usage costs
More complex setup

For cloud and serverless deployments, commit and index storage can be split. Prefer Standard S3 for commits because commits are the durable source of truth. Indexes are reproducible from commits, so they can use either Standard S3 or S3 Express One Zone depending on latency and cost requirements. See Serverless Storage Choices for benchmark-backed guidance.

IPFS Storage

Decentralized content-addressed storage via a local Kubo node:

{
  "@context": {"@vocab": "https://ns.flur.ee/system#"},
  "@graph": [{
    "@type": "Connection",
    "indexStorage": {
      "@type": "Storage",
      "ipfsApiUrl": "http://127.0.0.1:5001",
      "ipfsPinOnPut": true
    }
  }]
}

Characteristics:

Content-addressed (every blob identified by SHA-256 hash)
Immutable, tamper-evident storage
Decentralized replication via IPFS network
Fluree’s native CIDs work directly with IPFS

Use Cases:

Decentralized / censorship-resistant deployments
Content integrity verification
Cross-organization data sharing
Foundation for IPNS/ENS-based ledger discovery

Limitations:

Requires a running Kubo node
No prefix listing (manifest-based tracking needed)
No native deletion (unpin + GC)
Higher write latency than local file I/O

See IPFS Storage Guide for complete setup and configuration.

Storage Architecture

Memory Storage

┌──────────────────────┐
│   Fluree Process     │
│  ┌────────────────┐  │
│  │  Hash Map      │  │
│  │  (In Memory)   │  │
│  └────────────────┘  │
└──────────────────────┘

All data in process memory.

File Storage

┌──────────────────────┐
│   Fluree Process     │
│  ┌────────────────┐  │
│  │   File I/O     │  │
│  └────────┬───────┘  │
└───────────┼──────────┘
            │
     ┌──────▼──────┐
     │ File System │
     │  /var/lib/  │
     │   fluree/   │
     └─────────────┘

Data persisted to local files.

AWS Storage

┌──────────────────────┐  ┌──────────────────────┐
│   Fluree Process 1   │  │   Fluree Process 2   │
│  ┌────────────────┐  │  │  ┌────────────────┐  │
│  │  AWS SDK       │  │  │  │  AWS SDK       │  │
│  └────────┬───────┘  │  │  └────────┬───────┘  │
└───────────┼──────────┘  └───────────┼──────────┘
            │                         │
            └────────┬────────────────┘
                     │
          ┌──────────▼──────────┐
          │     AWS Cloud       │
          │  ┌──────┐  ┌──────┐│
          │  │  S3  │  │Dynamo││
          │  └──────┘  └──────┘│
          └─────────────────────┘

Multiple processes coordinate via AWS.

IPFS Storage

┌──────────────────────┐
│   Fluree Process     │
│  ┌────────────────┐  │
│  │  IpfsStorage   │  │
│  │  (HTTP client) │  │
│  └────────┬───────┘  │
└───────────┼──────────┘
            │ HTTP RPC
     ┌──────▼──────┐
     │  Kubo Node  │
     │  (IPFS)     │
     └──────┬──────┘
            │ libp2p
     ┌──────▼──────┐
     │  IPFS P2P   │
     │  Network    │
     └─────────────┘

Data stored as content-addressed blocks in IPFS via Kubo.

Storage Encryption

Fluree supports transparent AES-256-GCM encryption for data at rest. When enabled, all data is automatically encrypted before being written to storage.

Enabling Encryption

# Generate a 32-byte encryption key
export FLUREE_ENCRYPTION_KEY=$(openssl rand -base64 32)

Configure via JSON-LD (file storage):

{
  "@context": {"@vocab": "https://ns.flur.ee/system#"},
  "@graph": [{
    "@type": "Connection",
    "indexStorage": {
      "@type": "Storage",
      "filePath": "/var/lib/fluree",
      "AES256Key": {"envVar": "FLUREE_ENCRYPTION_KEY"}
    }
  }]
}

For S3 storage with encryption:

{
  "@context": {"@vocab": "https://ns.flur.ee/system#"},
  "@graph": [{
    "@type": "Connection",
    "indexStorage": {
      "@type": "Storage",
      "s3Bucket": "my-fluree-bucket",
      "s3Endpoint": "https://s3.us-east-1.amazonaws.com",
      "AES256Key": {"envVar": "FLUREE_ENCRYPTION_KEY"}
    }
  }]
}

Key Features:

AES-256-GCM authenticated encryption
Works natively with all storage backends (memory, file, S3)
Transparent encryption/decryption on read/write
Portable ciphertext format (encrypted data can be moved between backends)
Environment variable support for key configuration

See Storage Encryption for full documentation.

File Storage Details

Directory Structure

/var/lib/fluree/
├── ns@v2/                    # Nameservice records
│   ├── mydb/
│   │   ├── main.json        # Ledger metadata
│   │   └── dev.json
│   └── customers/
│       └── main.json
├── mydb/
│   ├── main/
│   │   ├── commit/          # Commit blobs (*.fcv2)
│   │   ├── txn/             # Transaction metadata (*.json)
│   │   ├── config/          # Ledger config blobs
│   │   └── index/
│   │       ├── roots/       # Index root descriptors (*.fir6)
│   │       ├── objects/
│   │       │   ├── branches/
│   │       │   ├── leaves/
│   │       │   └── history/
│   │       ├── garbage/
│   │       ├── stats/
│   │       └── spatial/
│   ├── dev/
│   │   └── ...
│   └── @shared/
│       └── dicts/           # Dictionaries shared by all branches
└── graph-sources/            # Graph sources
    └── products-search/
        └── main/
            ├── mapping/
            └── snapshots/

File Formats

Nameservice (JSON):

{
  "ledger_id": "mydb:main",
  "name": "mydb",
  "branch": "main",
  "commit_t": 150,
  "index_t": 145,
  "commit_head_id": "bafybeig...commitT150",
  "index_head_id": "bafybeig...indexRootT145",
  "retracted": false
}

Commits (Binary):

Compressed flake data
Transaction metadata
Cryptographic signatures

Indexes (Binary):

Root descriptors, branch manifests, leaf pages, and history sidecars
Optimized for query performance

Shared dictionaries (Binary):

Cross-branch dictionary blobs under {ledger}/@shared/dicts/
May be referenced by more than one branch of the same ledger

File System Requirements

Minimum:

10 GB free space
SSD recommended (HDD acceptable)
Sufficient IOPS for workload

Recommended:

100 GB+ free space
NVMe SSD
High IOPS capability
Regular backups

Durability

Filesystem syncing (FSYNC) is on by default. Fluree flushes committed data to durable storage so acknowledged commits survive a process crash or power loss. No configuration is needed to enable it.

To turn FSYNC off, set this environment variable before starting Fluree:

export FLUREE_STORAGE_FSYNC=0

With FSYNC off, writes reach the operating system’s page cache without waiting for a disk flush. This can improve performance for development, benchmarks, or restartable imports, but a power loss or kernel panic can lose acknowledged commits. Keep FSYNC on for data you need to retain.

To turn it back on, set FLUREE_STORAGE_FSYNC=1 and restart Fluree. The environment variable overrides the storage node’s durability setting and applies only to local file storage; it does not disable Raft log flushing.

Writes remain atomic with either setting: readers do not see partially written files.

Because the staged file is moved into place, each write gives the destination a new inode. Ownership, permissions, ACLs and hard links applied to a path are therefore dropped on the next write to that path — set them on the containing directory instead. This is immaterial for content-addressed blobs, which are written once and never rewritten, but it applies to the paths that are updated in place: nameservice head refs and ledger config.

Configuration details: Connection config (JSON-LD) and Configuration.

Staging files left by a crash

Because writes stage alongside the destination, a process killed between staging the bytes and moving them into place leaves the staged copy behind, named <file>.<pid>.<token>.<seq>.tmp. These are never served — listings skip them, so they can’t be read back as content — but they are a full copy of the object being written, and a crash loop produces one per attempt.

Starting a file-backed Fluree instance — opening a connection, or building an API client — reclaims them. The sweep is a deliberate startup action, taken explicitly by those startup paths: merely constructing a storage handle (as a test or an inspection tool might, on a directory it does not own) never deletes anything. The sweep is also deliberately timid, because the directory it walks is shared — by other instances in a multi-instance deployment, and by other subsystems even in a single process:

Only files named the way this backend’s own staging writer names them are considered at all. .tmp is a suffix, not a namespace — the indexer, the disk cache, the nameservice and the Raft log all stage under it, and the nameservice writes into this same tree. Anything whose name doesn’t parse as ours is ignored outright, whatever its age.
A staging file carrying this process’s token is never removed, at any age. In flight and already-leaked look identical from a directory entry. Staging files written by a pre-token build (v4.1.5/v4.1.6) carry a pid where the token now sits, so this rule cannot recognize them as anyone’s — for those, the 24-hour rule below is the only protection. The deployment where that matters is a rolling upgrade, with an old-format process still staging into the shared tree.
A staging file modified within the last 24 hours is never removed. A staging write is a single write of one in-memory buffer, so a day is far past any real one.
Anything the sweep can’t classify — an unparseable name, an entry it can’t stat, an mtime in the future — is kept.

What this does not do is coordinate with other processes. It is an age heuristic, not a lease: it compares another host’s clock against this one’s, and a foreign staging write that somehow stayed open for over a day would be unlinked. Even then nothing is corrupted — on POSIX the writer keeps its open descriptor, so only its final rename fails and the write reports an error.

The walk runs at most once per directory per process, and is handed to a background thread when one is available, so startup never waits on it.

It is also bounded, at 100,000 directory entries by default — a walk bounded in entries is not bounded in wall-clock on a network mount, where every directory read is a round trip. Exhausting that budget is not a deferral. The walk restarts from the top each time with no cursor and never removes content files, so if the first 100,000 entries it encounters are content, every subsequent start re-walks those same entries and the orphans beyond them are never reached. That case logs at warn; if you see it, raise FLUREE_STORAGE_TMP_SWEEP_BUDGET past the number of files under the directory. That variable only ever sizes the walk — an unparseable value, or 0, keeps the 100,000 default rather than meaning “don’t walk”. Turning the sweep off is the other variable’s job.

Set FLUREE_STORAGE_TMP_SWEEP=0 to skip the sweep entirely — worth doing if you want a crash’s leftovers preserved for a post-mortem.

AWS Storage Details

S3 Structure

s3://fluree-prod-data/
├── mydb/
│   ├── main/
│   │   ├── commit/
│   │   ├── txn/
│   │   ├── config/
│   │   └── index/
│   │       ├── roots/
│   │       ├── objects/
│   │       │   ├── branches/
│   │       │   ├── leaves/
│   │       │   └── history/
│   │       ├── garbage/
│   │       ├── stats/
│   │       └── spatial/
│   └── @shared/
│       └── dicts/
└── graph-sources/
    └── products-search/
        └── main/
            ├── mapping/
            └── snapshots/

DynamoDB Schema

The nameservice uses a DynamoDB table with a composite primary key (pk + sk) for ledger and graph source metadata coordination. Each ledger or graph source is stored as multiple items (one per concern) under the same partition key.

See DynamoDB Nameservice Guide for:

Complete table schema with composite-key layout
Table creation scripts (AWS CLI, CloudFormation, Terraform)
GSI setup for listing by kind
Local development setup with LocalStack
Production considerations and troubleshooting

Quick Reference:

Table: fluree-nameservice
Primary Key: pk (String, ledger-id) + sk (String, concern)
Sort Key Values: meta, head, index, config, status
GSI1 (gsi1-kind): kind (HASH) + pk (RANGE)
Items per ledger: 5 (meta, head, index, config, status)
Items per graph source: 4 (meta, config, index, status)

AWS Permissions

Required IAM permissions:

S3:

{
  "Effect": "Allow",
  "Action": [
    "s3:GetObject",
    "s3:PutObject",
    "s3:ListBucket",
    "s3:DeleteObject"
  ],
  "Resource": [
    "arn:aws:s3:::fluree-prod-data",
    "arn:aws:s3:::fluree-prod-data/*"
  ]
}

DynamoDB:

{
  "Effect": "Allow",
  "Action": [
    "dynamodb:GetItem",
    "dynamodb:PutItem",
    "dynamodb:UpdateItem",
    "dynamodb:DeleteItem",
    "dynamodb:Query",
    "dynamodb:BatchGetItem",
    "dynamodb:BatchWriteItem"
  ],
  "Resource": [
    "arn:aws:dynamodb:us-east-1:*:table/fluree-nameservice",
    "arn:aws:dynamodb:us-east-1:*:table/fluree-nameservice/index/gsi1-kind"
  ]
}

Cost Considerations

S3 Costs:

Storage: ~$0.023/GB/month (Standard)
PUT requests: ~$0.005/1000 requests
GET requests: ~$0.0004/1000 requests

DynamoDB Costs:

Provisioned: ~$0.25/WCU/month + $0.05/RCU/month
On-Demand: ~$1.25/million writes + $0.25/million reads

Typical Monthly Costs (medium deployment):

S3: $50-200 (depending on data size)
DynamoDB: $10-50 (depending on traffic)
Total: $60-250/month

Choosing a Storage Mode

Decision Matrix

Requirement	Memory	File	AWS	IPFS
Development	Best	Good	Overkill	Overkill
Single server	No	Best	Overkill	Good
Multi-server	No	No	Best	Good
Persistence	No	Yes	Yes	Yes
Cloud-native	No	No	Yes	No
Decentralized	No	No	No	Best
Content integrity	No	No	No	Best
Cost	Free	Free	Monthly	Free
Setup complexity	Trivial	Simple	Complex	Moderate
Performance	Fastest	Fast	Good	Good
Durability	None	Local	11 9’s	Network-wide

Recommendations

Use Memory when:

Developing locally
Running tests
Data is temporary
Maximum performance needed

Use File when:

Single server deployment
Local persistence needed
Simple setup preferred
Predictable costs important

Use AWS when:

Multiple servers needed
High availability required
Geographic distribution needed
Cloud-native architecture

Use IPFS when:

Decentralized storage required
Content integrity verification is critical
Cross-organization data sharing
Building toward IPNS/ENS-based ledger discovery
Censorship resistance is a requirement

Switching Storage Modes

Memory to File

Export from the running system and import into the new one:

# Export from memory
curl -X POST http://localhost:8090/export?ledger=mydb:main > mydb-export.jsonld

# Stop memory server, start file server
./fluree-db-server --storage file --data-dir /var/lib/fluree

# Import to file storage
curl -X POST "http://localhost:8090/v1/fluree/insert?ledger=mydb:main" \
  --data-binary @mydb-export.jsonld

File to AWS

Copy files to S3 and create the nameservice table:

# Copy data directory to S3
aws s3 sync /var/lib/fluree/ s3://fluree-prod-data/

# Create DynamoDB table (see docs/operations/dynamodb-guide.md for full schema)
aws dynamodb create-table \
  --table-name fluree-nameservice \
  --attribute-definitions \
    AttributeName=pk,AttributeType=S \
    AttributeName=sk,AttributeType=S \
    AttributeName=kind,AttributeType=S \
  --key-schema \
    AttributeName=pk,KeyType=HASH \
    AttributeName=sk,KeyType=RANGE \
  --billing-mode PAY_PER_REQUEST

# Start AWS-backed server
./fluree-db-server --storage aws --s3-bucket fluree-prod-data

AWS to File

Download from S3:

# Download data from S3
aws s3 sync s3://fluree-prod-data/ /var/lib/fluree/

# Start file-backed server
./fluree-db-server --storage file --data-dir /var/lib/fluree

Backup and Recovery

Memory Storage

No native backup (data is ephemeral):

# Export ledger
curl -X POST http://localhost:8090/export?ledger=mydb:main > backup.jsonld

File Storage

Backup data directory:

# Stop server (recommended)
systemctl stop fluree

# Backup
tar -czf fluree-backup-$(date +%Y%m%d).tar.gz /var/lib/fluree/

# Start server
systemctl start fluree

For online backups, prefer storage-level snapshots or object-store versioning. The standalone server does not currently expose HTTP read-only toggle endpoints.

AWS Storage

Use S3 versioning and lifecycle policies:

# Enable versioning
aws s3api put-bucket-versioning \
  --bucket fluree-prod-data \
  --versioning-configuration Status=Enabled

# Configure lifecycle
aws s3api put-bucket-lifecycle-configuration \
  --bucket fluree-prod-data \
  --lifecycle-configuration file://lifecycle.json

DynamoDB backups:

# Enable point-in-time recovery
aws dynamodb update-continuous-backups \
  --table-name fluree-nameservice \
  --point-in-time-recovery-specification PointInTimeRecoveryEnabled=true

Troubleshooting

File Storage

Permission Errors:

sudo chown -R fluree:fluree /var/lib/fluree
chmod -R 755 /var/lib/fluree

Disk Full:

# Check space
df -h /var/lib/fluree

# Force a full index refresh
curl -X POST http://localhost:8090/v1/fluree/reindex \
  -H "Content-Type: application/json" \
  -d '{"ledger": "mydb:main"}'

AWS Storage

Connection Errors:

Verify AWS credentials
Check IAM permissions
Verify S3 bucket exists
Check DynamoDB table exists

Throttling:

Increase DynamoDB capacity
Use provisioned capacity mode
Implement retry logic

Configuration - Configuration options
IPFS Storage Guide - IPFS/Kubo setup and configuration
DynamoDB Nameservice Guide - DynamoDB-specific setup
Getting Started: Server - Initial setup
Admin and Health - Administrative operations

Keyboard shortcuts

Fluree DB