fluree iceberg
Manage Apache Iceberg table connections.
Subcommands
| Subcommand | Description |
|---|---|
map | Map an Iceberg table as a graph source |
list | List Iceberg-family graph sources (Iceberg and R2RML) |
info | Show details for an Iceberg-family graph source |
drop | Drop an Iceberg-family graph source |
fluree iceberg map
Map an Iceberg table as a queryable graph source.
Usage
fluree iceberg map <NAME> --r2rml <PATH> [OPTIONS]
fluree iceberg map <NAME> --remote <REMOTE> --stored-mapping <ID> [OPTIONS]
Arguments
| Argument | Description |
|---|---|
<NAME> | Graph source name (e.g., “warehouse-orders”) |
Options
Catalog mode:
| Option | Description |
|---|---|
--mode <MODE> | Catalog mode, case-insensitive: rest (default), direct, glue (AWS Glue Data Catalog), or s3tables (AWS S3 Tables) |
REST catalog mode options:
| Option | Description |
|---|---|
--catalog-uri <URI> | REST catalog URI (required for rest mode) |
--table <ID> | Table identifier in namespace.table format. Optional: the --r2rml mapping’s rr:tableName entries name the tables |
--warehouse <NAME> | Warehouse (or catalog) name as the REST catalog knows it; the catalog’s own route prefix is looked up from it |
--no-vended-credentials | Disable vended credentials (enabled by default); the same as --storage-identity ambient, and not given with it |
Direct S3 mode options:
| Option | Description |
|---|---|
--table-location <URI> | S3 table location (required for direct mode, e.g., s3://bucket/warehouse/ns/table) |
AWS Glue / S3 Tables mode options (native AWS SDK; reads S3 with the ambient AWS credential chain — no vended credentials):
| Option | Description |
|---|---|
--table <ID> | Table identifier namespace.table. Optional: the --r2rml mapping’s rr:tableName entries name the tables. For glue, namespace is the Glue database. |
--region <REGION> | AWS region of the Glue / S3 Tables API. Glue falls back to --s3-region, then the AWS SDK’s region chain; S3 Tables defaults to the table bucket ARN’s region and refuses a --region that contradicts it. S3 data reads use --s3-region when given, else this region. Must be an AWS region code (e.g. us-east-1). |
--catalog-id <ID> | Glue catalog id for cross-account access (glue mode; defaults to the caller’s account) |
--table-bucket-arn <ARN> | S3 Tables table-bucket ARN, arn:aws:s3tables:<region>:<account>:bucket/<name> (required for s3tables mode; any AWS partition) |
R2RML mapping:
| Option | Description |
|---|---|
--r2rml <PATH> | R2RML mapping file (Turtle format). Defines how table rows become RDF triples. Table references come from the mapping’s rr:tableName entries. Required, unless --stored-mapping names a mapping the remote stores |
--r2rml-type <TYPE> | Mapping media type (e.g., text/turtle); inferred from extension if omitted |
Stored on the remote (--remote only; see Stored connections and mappings):
| Option | Description |
|---|---|
--stored-catalog <ID> | A catalog connection the remote stores, by its id there. Replaces every connection, credential and storage option; needs a mapping (--r2rml or --stored-mapping) |
--stored-mapping <ID> | An R2RML mapping the remote stores, by its id there. Replaces --r2rml and implies its catalog connection; --table defaults to the mapping’s first table |
Authentication:
These options, and --warehouse, configure a REST catalog: any other --mode refuses them rather than ignore them. With --remote, the CLI sends the server the structured connection its local map would store (see Iceberg connection), so the two register the same record.
| Option | Description |
|---|---|
--auth-bearer <TOKEN> | Bearer token for REST catalog authentication. Stored with the graph source; prefer the option below |
--auth-bearer-env <VAR> | Environment variable holding the bearer token, read by the process that reads the tables. The token is not stored. With --remote, the server must list the variable in FLUREE_GRAPH_SOURCE_SECRET_ENV_VARS |
--oauth2-token-url <URL> | OAuth2 token URL for client credentials auth |
--oauth2-client-id <ID> | OAuth2 client ID |
--oauth2-client-secret <SECRET> | OAuth2 client secret. Stored with the graph source; prefer the option below |
--oauth2-client-secret-env <VAR> | Environment variable holding the OAuth2 client secret; as --auth-bearer-env |
--oauth2-scope <SCOPE> | OAuth2 scope (e.g. session:role:ICEBERG_READER for Snowflake Horizon / Polaris) |
--oauth2-audience <AUDIENCE> | OAuth2 audience |
--auth-google-metadata | Authenticate as the GCE/GKE metadata server’s service account (GKE Workload Identity). A host identity: an operator can refuse it with FLUREE_GRAPH_SOURCE_HOST_IDENTITIES |
--auth-google-adc | Authenticate with the host’s application-default credentials (GOOGLE_APPLICATION_CREDENTIALS, else gcloud’s, e.g. after gcloud auth application-default login). A host identity. User credentials carry the user’s own cloud-platform scope, which a refresh cannot narrow |
--auth-google-service-account-env <VAR> | Authenticate as the service account whose key JSON this environment variable holds, read by the process that reads the tables. The key is not stored, and is never taken literally |
--auth-google-wif-config <FILE> | Authenticate through Workload Identity Federation from AWS: the credential configuration that gcloud iam workload-identity-pools create-cred-config … --aws prints. Not a secret: the file is read and stored with the graph source |
--auth-google-quota-project <PROJECT> | The Google Cloud project catalog requests bill to (x-goog-user-project). BigLake requires one for every identity; default: the project a bq://projects/<P>/… or bl://projects/<P>/… warehouse names. It bills the catalog only, never storage reads |
--auth-google-scopes <SCOPES> | OAuth scopes, comma-separated (default: cloud-platform) |
Give one way to authenticate: a bearer token, OAuth2 client credentials, or one Google source. Two at once are refused, and so is a partial OAuth2 set: OAuth2 needs --oauth2-token-url and the client secret (--oauth2-client-id defaults to empty, as Snowflake Horizon needs).
Storage identity:
| Option | Description |
|---|---|
--storage-identity <IDENTITY> | Which identity reads the table’s files: vended (the REST catalog’s per-table vended credentials, required), ambient (the ambient AWS credential chain), catalog (the REST catalog’s own Google identity reads gs://, as BigLake federation needs), or google (a Google identity of the source’s own, the options below). Default: catalog for a bq:// warehouse, vended for a bl:// one; otherwise vended credentials for a REST catalog (unless --no-vended-credentials) and the ambient chain for the other modes |
--storage-google-metadata | Read the files as the metadata server’s service account |
--storage-google-adc | Read the files with the host’s application-default credentials. User credentials read with the user’s own cloud-platform scope (a refresh grant cannot narrow it to read-only storage); a service-account key file is minted read-only |
--storage-google-service-account-env <VAR> | Read the files as the service account whose key JSON this variable holds |
--storage-google-wif-config <FILE> | Read the files through this Workload Identity Federation configuration |
--storage-google-quota-project <PROJECT> | The project storage requests bill to (requester pays). Never defaulted, and never the catalog’s quota project |
The default is the CLI’s (Python’s map_iceberg has the same one); the server stores what it is sent, so a remote map sends the identity the default chose. A --storage-google-* source alone means --storage-identity google. A bq:// warehouse never vends, so its default needs a Google catalog identity: without one, map refuses and names the options to give.
S3 overrides:
| Option | Description |
|---|---|
--s3-region <REGION> | S3 region override, in every mode (for glue / s3tables, defaults to the catalog’s region) |
--s3-endpoint <URL> | S3 endpoint override (e.g. MinIO, LocalStack, or https://storage.googleapis.com for Google Cloud Storage) |
--s3-path-style | Use path-style S3 URLs |
Other:
| Option | Description |
|---|---|
--remote <NAME> | Execute against a remote server (by remote name) |
--branch <NAME> | Branch name (defaults to “main”) |
--model <LEDGER> | Model ledger (name:branch) whose default graph supplies the source’s view policies and class/property hierarchy. Must exist. See Access policy |
--default-allow <BOOL> | Fallback for governed requests that match no policy; true keeps the source readable under authentication without a model (unset: deny) |
Description
Maps an Apache Iceberg table as a graph source that can be queried using SPARQL or JSON-LD queries. The table is accessed read-only; Fluree does not modify the Iceberg table.
An R2RML mapping is required to define how Iceberg table rows are transformed into RDF triples: a file (--r2rml), or, on a remote that stores mappings, one of its own (--stored-mapping; see Stored connections and mappings). Without either, map is refused before it registers anything or contacts a remote.
Four catalog modes are supported:
- REST mode (default): Connects to an Iceberg REST catalog (e.g., Apache Polaris) to discover table metadata. Supports vended credentials and warehouse selection.
- Direct S3 mode: Reads table metadata directly from S3 by resolving
version-hint.textin the table’smetadata/directory. No catalog server required. Also reads Google Cloud Storage (--s3-endpoint https://storage.googleapis.com); GCS reads are signed with AWS SigV4 using GCS HMAC interop keys (the standardAWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY) — see the Iceberg guide. - AWS Glue mode (
--mode glue): Looks the table up in the AWS Glue Data Catalog (GlueGetTable) and reads the metadata file itsmetadata_locationparameter names. The Glue database is the namespace;--catalog-idreads another account’s catalog. - AWS S3 Tables mode (
--mode s3tables): Resolves the table’s metadata file in an S3 Tables table bucket (--table-bucket-arn) withGetTableMetadataLocation.
Glue and S3 Tables use the ambient AWS credential chain (environment, shared config / AWS_PROFILE / SSO, container or instance role) for the catalog call and the S3 reads; vended credentials are not supported, and the CLI turns them off for these modes. map loads each table the mapping names the way a query will: through the catalog (REST, Glue, S3 Tables), or by resolving its current metadata at the location (Direct). It reads table metadata only, never data files, and stops after 10 s. A table that cannot be loaded in that time does not fail map: it registers the source and reports Connection: not verified: followed by why: the table that did not load and the catalog’s or storage’s reason, a catalog that did not answer, a catalog client that could not be built, or a check that ran past 10 s (in which case the tables may only be slow). Re-registering a source with the same config can be answered from the catalog caches rather than a new call: a loadTable response for up to 60 s in memory (FLUREE_ICEBERG_LOADTABLE_TTL_SECS) and its metadata pointer for up to 300 s on disk (FLUREE_ICEBERG_LOADTABLE_PTR_TTL_SECS). A mapping that defines no rr:TriplesMap (most often an rr: prefix without its trailing #) is refused. See Catalog Modes and AWS Credentials.
Stored connections and mappings
Some remotes store catalog connections and R2RML mappings themselves, under ids of their own. There, map can name them instead of restating them:
# A mapping the remote stores; it implies its catalog connection and first table
fluree iceberg map orders --remote origin --stored-mapping <MAPPING-ID>
# A connection the remote stores, read with a local mapping file
fluree iceberg map orders --remote origin \
--stored-catalog <CATALOG-ID> \
--r2rml mappings/orders.ttl --table sales.orders
The request carries the ids (stored_catalog_id, stored_mapping_id) and the source’s own options (--table, --branch, --model, --default-allow), plus the mapping file’s text with --stored-catalog. It never carries a connection, so no credential is sent again. The CLI refuses before sending anything when:
- a connection, credential or storage option (
--mode,--catalog-uri,--auth-*,--storage-*,--s3-*, …) is given beside either reference; --r2rmlor--r2rml-typeis given beside--stored-mapping;--stored-catalogis given with--r2rmlbut no--table(a stored connection is mapped one named table at a time);- an id is empty;
- no
--remoteis given: only a named remote stores connections or mappings, and neither a local map nor one routed through a running local server has any.
Both references together must name the mapping’s own connection; the remote checks. A server that does not store connections or mappings, a Fluree server among them, refuses the request. The CLI then shows the server’s message and adds a hint: if the remote does not store catalog connections or mappings, it refuses references to them; otherwise the id may name nothing it has.
Examples
# REST catalog with R2RML mapping
fluree iceberg map airlines \
--catalog-uri https://polaris.example.com/api/catalog \
--r2rml mappings/airlines.ttl \
--auth-bearer $POLARIS_TOKEN
# REST catalog with explicit table and warehouse
fluree iceberg map warehouse-orders \
--catalog-uri https://polaris.example.com/api/catalog \
--table sales.orders \
--r2rml mappings/orders.ttl \
--auth-bearer $POLARIS_TOKEN \
--warehouse my-warehouse
# Direct S3 (no catalog server)
fluree iceberg map execution-log \
--mode direct \
--table-location s3://my-bucket/warehouse/logs/execution_log \
--r2rml mappings/execution_log.ttl \
--s3-region us-east-1
# Direct mode on Google Cloud Storage.
# Set AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY to your GCS HMAC interop keys
# in the server environment (used to SigV4-sign GCS reads).
fluree iceberg map orders \
--mode direct \
--table-location s3://my-bucket/warehouse/sales/orders \
--r2rml mappings/orders.ttl \
--s3-endpoint https://storage.googleapis.com \
--s3-region europe-west1 \
--s3-path-style
# AWS Glue Data Catalog (ambient AWS credentials; the Glue database is the namespace).
# Add --catalog-id 123456789012 to read another account's catalog.
fluree iceberg map glue-orders \
--mode glue \
--region us-east-1 \
--table sales.orders \
--r2rml mappings/orders.ttl
# OAuth2 authentication
fluree iceberg map orders \
--catalog-uri https://polaris.example.com/api/catalog \
--table sales.orders \
--r2rml mappings/orders.ttl \
--oauth2-token-url https://auth.example.com/token \
--oauth2-client-id my-client \
--oauth2-client-secret $CLIENT_SECRET
# BigLake federation (bq://): a service account's key in LAKE_SA_KEY
# authenticates the catalog, and the same identity reads the gs:// files
# (--storage-identity catalog, the bq:// default)
fluree iceberg map orders \
--catalog-uri https://biglake.googleapis.com/iceberg/v1/restcatalog \
--warehouse bq://projects/lake/locations/us/catalogs/sales \
--table sales.orders \
--r2rml mappings/orders.ttl \
--auth-google-service-account-env LAKE_SA_KEY
# The same through Workload Identity Federation from AWS: no stored secret
fluree iceberg map orders \
--catalog-uri https://biglake.googleapis.com/iceberg/v1/restcatalog \
--warehouse bq://projects/lake/locations/us/catalogs/sales \
--table sales.orders \
--r2rml mappings/orders.ttl \
--auth-google-wif-config wif-aws.json
# Create the graph source on a remote Fluree server
fluree iceberg map warehouse-orders \
--remote origin \
--catalog-uri https://polaris.example.com/api/catalog \
--table sales.orders \
--r2rml mappings/orders.ttl
Output
Mapped Iceberg table as R2RML graph source 'airlines:main'
Table: openflights.airlines
Catalog: https://polaris.example.com/api/catalog
R2RML: mappings/airlines.ttl
TriplesMaps: 3
Connection: verified
Mapping: validated
advisory: openflights.airlines: TriplesMap <http://example.org/mapping#Airline>: no key of its logical table 'openflights.airlines' lies within its subject's columns (airline_id), and it has no known key, so queries over it remove repeated solutions as they run. …
An advisory: line on standard error names each triples map that neither an
f:key nor the table’s identifier fields prove subject-unique, and the
f:key that would. The source is registered and answers correctly, but
COUNTs, same-subject stars and ORDER BY … LIMIT over such a map cost more;
see R2RML → A map with no key. Advisories are reported only when every table loaded.
After Mapping
Once mapped, the graph source appears in standard commands:
# Listed alongside ledgers
fluree list
# Inspect configuration
fluree iceberg info warehouse-orders
# Query the graph source directly (single-target): name it as the target, with
# no FROM. The CLI resolves it as a graph source and runs it through the
# R2RML/Iceberg engine. Requires the `iceberg` feature.
fluree query warehouse-orders 'SELECT ?id ?total WHERE { ?o ex:id ?id ; ex:total ?total }'
# Federate into it: a query whose FROM names the source routes to the
# connection-scoped path automatically (or force it with --connection).
fluree query --connection 'SELECT ?id ?total FROM <warehouse-orders:main> WHERE { ?o ex:id ?id ; ex:total ?total }'
# Remove the mapping
fluree drop warehouse-orders --force
Feature Flag
Requires the iceberg feature flag. Without it, the command returns:
error: Iceberg support not compiled. Rebuild with `--features iceberg`.
See Also
- Iceberg / Parquet - Iceberg integration details
- R2RML - R2RML mapping reference
- list - List ledgers and graph sources
- info - Show graph source details
- drop - Remove a graph source
fluree iceberg list
List Iceberg-family graph sources (Iceberg and R2RML types).
Usage
fluree iceberg list [--remote <NAME>]
Examples
# Local
fluree iceberg list
# Remote
fluree iceberg list --remote origin
fluree iceberg info
Show details for an Iceberg-family graph source.
Usage
fluree iceberg info <NAME> [--remote <NAME>]
Examples
# Local
fluree iceberg info warehouse-orders
# Remote
fluree iceberg info warehouse-orders --remote origin
fluree iceberg drop
Drop an Iceberg-family graph source. This command only targets Iceberg/R2RML graph sources; it does not fall back to dropping ledgers of the same name.
Usage
fluree iceberg drop <NAME> --force [--remote <NAME>]
Examples
# Local
fluree iceberg drop warehouse-orders --force
# Remote
fluree iceberg drop warehouse-orders --force --remote origin