Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Cypher Sequential Writes (Multi-Clause MERGE Composition)

Status: implemented (v1) Owner: query/transact Related: docs/query/cypher.md, docs/reference/cypher-support-matrix.md

Problem

A Cypher write statement is a pipeline: each clause observes the writes of the clauses before it, within one atomic statement. The canonical idiom:

UNWIND $rows AS row
MERGE (a:Person {id: row.src})
MERGE (b:Person {id: row.dst})
MERGE (a)-[:KNOWS]->(b)

The second MERGE’s find-or-create check must see nodes the first MERGE just created; the relationship MERGE must see both. A single Fluree Txn evaluates its whole WHERE (including every NOT EXISTS MERGE guard) against one snapshot horizon — the pre-transaction state — so it cannot express “later clause observes earlier clause.” That is the real architectural constraint behind the v1 restriction (“MERGE must be the only write clause”), and no rewriting into a single Txn can lift it.

Approach: sequential staging within one commit

Fluree already solves “N ordered operations, each observing the previous, one atomic commit” for SPARQL 1.1 ;-separated updates (roadmap D-10, Fluree::stage_transaction_from_txns): each operation stages against a virtual state (base + prior operations’ flakes applied as a commit-record-less overlay via LedgerState::apply_staged_flakes_for_sequential_staging), flakes fold last-wins per fact identity re-stamped to the single final t+1 (a fact a later operation reversed back to how the base had it is dropped), and one commit publishes.

The Cypher driver builds on the same machinery, factored into a reusable SequentialStager (fluree-db-api/src/tx.rs). What Cypher adds over SPARQL ; is row threading: clauses pipe rows (variable bindings), not just graph state. The driver is therefore an execute-time loop, not a static decomposition:

  1. Row table init — the read prefix (MATCH / UNWIND / WITH / desugared InlineRows) is evaluated as an ordinary read query against the base state, projecting every variable later clauses or the RETURN need. Entity bindings materialize as IRIs, literals as typed values. Read-path evaluation gives the full read surface (including WITH aggregation) for free.

  2. Per-clause staging loop — each write clause becomes one or two sub-Txns seeded with the current row table (a VALUES join), staged against the current virtual state via the full staging pipeline (policy, SHACL, uniqueness — per-clause validation, the same semantics choice D-10 made; the config, shapes and constraints enforced are read once, from the state before the statement, and the config checks run once on its result):

    • CREATE: one Txn; created-entity columns extend the row table as skolem IRIs (_:fdb-{clause_skolem}-{row}-{label} — solution order equals VALUES row order, pinned by test).
    • MERGE: (a) probe matched rows (VALUES ⋈ pattern query on the virtual state); (b) absent rows dedup by their engine-evaluated identity-value tuple, first row per key creates (create-branch Txn with ON CREATE SET); (c) flakes apply; (d) re-bind all rows (VALUES ⋈ pattern on the updated state) — every row now matches, created or pre-existing, with Cypher’s multi-match row multiplication; (e) rows that did not create run ON MATCH SET (a seeded SET Txn). In-batch duplicate keys thus converge to sequential semantics: first occurrence creates + ON CREATE SET, later occurrences match + ON MATCH SET. Computed branch values that read graph state (for example n.count = n.count + 1) evaluate after re-bind and stage row-by-row against the evolving virtual state; this lets each duplicate observe the preceding row’s update. Simple literals and threaded row-column values remain batched.
    • SET / REMOVE: one seeded Txn; row table unchanged. A SET whose items write one property key twice is split at the repeat into consecutive SETs, since its items apply in order.
    • DETACH DELETE of a node: one seeded Txn, like SET; its lowering finds the node’s triples in the state it stages on.
    • A relationship’s DELETE (DELETE r, or DETACH DELETE r): one seeded Txn that re-matches the edge r is on, in the state it stages on, so the LPG cleanup takes the properties earlier clauses set on r. The re-match is MATCH (s)-[r:T]->(o), seeded with r and the variables that hold its endpoints at the end of the read clauses, so it is a point lookup of the edge. A walk over the read clauses’ scope decides which variables are relationships and which hold their endpoints, following WITH projections and aliases. The read prefix names an unnamed endpoint, and a WITH that keeps a relationship but drops or rebinds an endpoint also projects that endpoint under a name of the driver’s own; neither changes what the statement matches (an endpoint is a function of its relationship, so no grouping or DISTINCT changes). So the names the statement’s MATCH gave the endpoints never point the re-match at another edge. A pattern read in either direction has no subject to carry; its re-match takes endpoint variables of its own, which the seeded r and its f:reifies* bundle pin to the edge, at the cost of a scan of the type’s relationships per row.
    • Seeding: a SET, REMOVE or DELETE is seeded only with the rows that bind its target. When a row (from an OPTIONAL MATCH) leaves one of a clause’s targets unbound, the clause stages once per target, each part seeded with the rows that bind it, so the row skips only its writes to the unbound target. Items on different targets are independent once a repeated key is split, so the parts write what the clause would; a DELETE’s relationship targets stage after its node targets.
    • A bare node DELETE in a multi-clause chain: rejected (clear error) in v1 — its relationship-existence probe stays single-statement.

    A Cypher DELETE r marks its Txn with the delete template that retracts r’s edge (TxnOpts::relationship_deletes), in the driver and in the single Txn alike. Staging notes the edges those templates retract and, where the f:reifies* cascade reads each edge’s relationships, refuses one of them that carries more than one: retracting the edge would take them all. The check reads the state the Txn stages on (in the driver, the virtual state, so a sibling an earlier clause created counts) and has no view of r’s property map or WHERE. It covers r’s edges and no others: an edge another delete template retracts too (a DETACH DELETE of one of its endpoints, which takes every relationship on the node) is left to the cascade.

  3. Finish — SequentialStager::finish() produces one StageResult over the original base: single commit at t+1, union namespace delta, folded flakes. Zero staged flakes → the ordinary no-op update path.

  4. RETURN — answered by one final read query (VALUES(final rows) RETURN items) against the final virtual state: full expression surface (projections, DISTINCT, ORDER BY, SKIP/LIMIT) rather than the created-entity-only reconstruction single-Txn writes use. The RETURN is the only reader of the state after the last write clause, so a statement without one stages its last clause (unless it’s a MERGE, which probes its own writes) without building that state, as SPARQL’s last ; operation does. The final state reaches a caller only with the answered RETURN (AnsweredReturn); the cypher_unread_final_state stamp records which way a statement went.

Row threading representation

Rows re-enter lowering through ReadClause::InlineRows whose cells now admit an internal-only Expr::SeededValue(UnresolvedValue, span) variant — never produced by the parser, carrying entity IRIs (including stable _:fdb- blank node labels) and typed literals losslessly into both the read-side lowering (probe/re-bind/RETURN queries) and the write-side lowering (seeded Txns). InlineRows vars now register as bound in the write lowering (bound_vars), so seeded rows satisfy SET/REMOVE target checks.

Dispatch

cypher_write_plan classification order:

  1. detect_conditional — the existing probe/branch shapes (standalone MERGE … ON MATCH SET, per-row relationship MERGE pair-staging, DELETE guards) keep their proven paths.
  2. detect_sequential (new) — multi-write-clause statements containing a MERGE (MERGE chains, CREATE+MERGE mixes, interleaved SET/REMOVE), and the previously-rejected node MERGE under a plain leading MATCH (the driver’s per-row partition makes it correct: one create per distinct absent key, every row binds). It also takes statements without a MERGE whose write clauses can see or undo one another’s writes (interacting_writes): two clauses writing one property key or one label (on any variables, since two can bind one node), a DETACH DELETE or a relationship’s DELETE with any other write, or a CREATE with a clause whose lowering reads stored values (a SET or REMOVE of a property, a map SET, DETACH DELETE — the OPTIONAL lookup adds rows the CREATE would fire on), plus a SET writing one key twice. The single-Txn lowering evaluates every clause against the state before the statement and stages them as one operation, so for these it neither lets a later clause see an earlier one nor lets the last write hold; the driver stages clause by clause and the stager folds each fact to its last write, as for SPARQL ; updates. Clauses that can’t meet (CREATE chains, SETs of different keys) keep the single Txn, as do statements with a member the driver doesn’t stage (a bare node DELETE, CALL { … } before the writes, an OPTIONAL MATCH before a CREATE), which the single Txn’s lowering refuses in turn.
  3. Single-Txn lowering — everything else, including its existing clear errors.

WritePlan gains a Sequential variant. Call sites:

  • Autocommit (transact_cypher_statement) and Bolt explicit transactions (cypher_transaction_write) call the driver directly; the Bolt path stages against the transaction’s private state, so statement pipelining and read-your-writes compose.
  • Consensus (LocalCommitter): the plan resolves under the write lock per retry attempt (same head-check/refresh contract as conditional MERGE). TransactionReceipt carries an optional Cypher RETURN table (serde default None); Raft mode rejects multi-clause RETURN in v1.

The UNWIND $batch desugar’s last-wins row dedup (dedup_rows_by_merge_identity) now applies only to single-write-clause statements — it keys on the first node MERGE and would drop legitimate rows in a chain; the driver owns duplicate-key semantics for multi-clause statements.

Non-impact guarantees

  • JSON-LD / Turtle / SPARQL transactions write what they wrote before. Two changes reach code they share:
    • stage()’s f:reifies* cascade refuses a parallel relationship on an edge a Cypher DELETE r retracts (TxnOpts::relationship_deletes). No other transaction sets the marker, and without it staging notes nothing.
    • The sequential stager builds the planner stats once, for the state a write begins on, and pins them (estimates only) onto every state it advances to, so SPARQL’s ;-separated operations after the first plan from those stats rather than assembling their own.
  • stage_transaction_from_txns keeps its len == 1 fast path; only its N ≥ 2 loop body moved into SequentialStager.
  • Single-clause Cypher writes keep their existing plans byte-for-byte, except a SET that writes one property key twice. Multi-clause writes without a MERGE keep the single Txn unless their clauses can meet (above): those wrote the wrong facts there.

Costs

Per MERGE clause: two or three VALUES-seeded engine queries (match probe, identity-tuple evaluation for dedup, re-bind) plus one or two stagings. The common path is row-bulk. A computed ON CREATE / ON MATCH value that must read the current graph uses a correctness-first per-row query/stage loop so duplicate rows observe prior updates. Row tables are capped (MAX_SEQUENTIAL_ROWS) with a clear error.