Cypher Sequential Writes (Multi-Clause MERGE Composition)
Status: implemented (v1)
Owner: query/transact
Related: docs/query/cypher.md, docs/reference/cypher-support-matrix.md
Problem
A Cypher write statement is a pipeline: each clause observes the writes of the clauses before it, within one atomic statement. The canonical idiom:
UNWIND $rows AS row
MERGE (a:Person {id: row.src})
MERGE (b:Person {id: row.dst})
MERGE (a)-[:KNOWS]->(b)
The second MERGE’s find-or-create check must see nodes the first MERGE just
created; the relationship MERGE must see both. A single Fluree Txn evaluates
its whole WHERE (including every NOT EXISTS MERGE guard) against one
snapshot horizon — the pre-transaction state — so it cannot express “later
clause observes earlier clause.” That is the real architectural constraint
behind the v1 restriction (“MERGE must be the only write clause”), and no
rewriting into a single Txn can lift it.
Approach: sequential staging within one commit
Fluree already solves “N ordered operations, each observing the previous, one
atomic commit” for SPARQL 1.1 ;-separated updates (roadmap D-10,
Fluree::stage_transaction_from_txns): each operation stages against a
virtual state (base + prior operations’ flakes applied as a commit-record-less
overlay via LedgerState::apply_staged_flakes_for_sequential_staging), flakes
fold last-wins per fact identity re-stamped to the single final t+1 (a fact a
later operation reversed back to how the base had it is dropped), and one
commit publishes.
The Cypher driver builds on the same machinery, factored into a reusable
SequentialStager (fluree-db-api/src/tx.rs). What Cypher adds over SPARQL
; is row threading: clauses pipe rows (variable bindings), not just
graph state. The driver is therefore an execute-time loop, not a static
decomposition:
-
Row table init — the read prefix (
MATCH/UNWIND/WITH/ desugaredInlineRows) is evaluated as an ordinary read query against the base state, projecting every variable later clauses or theRETURNneed. Entity bindings materialize as IRIs, literals as typed values. Read-path evaluation gives the full read surface (includingWITHaggregation) for free. -
Per-clause staging loop — each write clause becomes one or two sub-
Txns seeded with the current row table (aVALUESjoin), staged against the current virtual state via the full staging pipeline (policy, SHACL, uniqueness — per-clause validation, the same semantics choice D-10 made; the config, shapes and constraints enforced are read once, from the state before the statement, and the config checks run once on its result):- CREATE: one Txn; created-entity columns extend the row table as
skolem IRIs (
_:fdb-{clause_skolem}-{row}-{label}— solution order equalsVALUESrow order, pinned by test). - MERGE: (a) probe matched rows (
VALUES ⋈ patternquery on the virtual state); (b) absent rows dedup by their engine-evaluated identity-value tuple, first row per key creates (create-branch Txn withON CREATE SET); (c) flakes apply; (d) re-bind all rows (VALUES ⋈ patternon the updated state) — every row now matches, created or pre-existing, with Cypher’s multi-match row multiplication; (e) rows that did not create runON MATCH SET(a seeded SET Txn). In-batch duplicate keys thus converge to sequential semantics: first occurrence creates +ON CREATE SET, later occurrences match +ON MATCH SET. Computed branch values that read graph state (for examplen.count = n.count + 1) evaluate after re-bind and stage row-by-row against the evolving virtual state; this lets each duplicate observe the preceding row’s update. Simple literals and threaded row-column values remain batched. - SET / REMOVE: one seeded Txn; row table unchanged. A SET whose items write one property key twice is split at the repeat into consecutive SETs, since its items apply in order.
- DETACH DELETE of a node: one seeded Txn, like SET; its lowering finds the node’s triples in the state it stages on.
- A relationship’s DELETE (
DELETE r, orDETACH DELETE r): one seeded Txn that re-matches the edgeris on, in the state it stages on, so the LPG cleanup takes the properties earlier clauses set onr. The re-match isMATCH (s)-[r:T]->(o), seeded withrand the variables that hold its endpoints at the end of the read clauses, so it is a point lookup of the edge. A walk over the read clauses’ scope decides which variables are relationships and which hold their endpoints, followingWITHprojections and aliases. The read prefix names an unnamed endpoint, and aWITHthat keeps a relationship but drops or rebinds an endpoint also projects that endpoint under a name of the driver’s own; neither changes what the statement matches (an endpoint is a function of its relationship, so no grouping orDISTINCTchanges). So the names the statement’sMATCHgave the endpoints never point the re-match at another edge. A pattern read in either direction has no subject to carry; its re-match takes endpoint variables of its own, which the seededrand itsf:reifies*bundle pin to the edge, at the cost of a scan of the type’s relationships per row. - Seeding: a SET, REMOVE or DELETE is seeded only with the rows that bind
its target. When a row (from an
OPTIONAL MATCH) leaves one of a clause’s targets unbound, the clause stages once per target, each part seeded with the rows that bind it, so the row skips only its writes to the unbound target. Items on different targets are independent once a repeated key is split, so the parts write what the clause would; a DELETE’s relationship targets stage after its node targets. - A bare node DELETE in a multi-clause chain: rejected (clear error) in v1 — its relationship-existence probe stays single-statement.
A Cypher
DELETE rmarks its Txn with the delete template that retractsr’s edge (TxnOpts::relationship_deletes), in the driver and in the singleTxnalike. Staging notes the edges those templates retract and, where thef:reifies*cascade reads each edge’s relationships, refuses one of them that carries more than one: retracting the edge would take them all. The check reads the state the Txn stages on (in the driver, the virtual state, so a sibling an earlier clause created counts) and has no view ofr’s property map orWHERE. It coversr’s edges and no others: an edge another delete template retracts too (aDETACH DELETEof one of its endpoints, which takes every relationship on the node) is left to the cascade. - CREATE: one Txn; created-entity columns extend the row table as
skolem IRIs (
-
Finish —
SequentialStager::finish()produces oneStageResultover the original base: single commit att+1, union namespace delta, folded flakes. Zero staged flakes → the ordinary no-op update path. -
RETURN — answered by one final read query (
VALUES(final rows) RETURN items) against the final virtual state: full expression surface (projections,DISTINCT,ORDER BY,SKIP/LIMIT) rather than the created-entity-only reconstruction single-Txn writes use. The RETURN is the only reader of the state after the last write clause, so a statement without one stages its last clause (unless it’s a MERGE, which probes its own writes) without building that state, as SPARQL’s last;operation does. The final state reaches a caller only with the answered RETURN (AnsweredReturn); thecypher_unread_final_statestamp records which way a statement went.
Row threading representation
Rows re-enter lowering through ReadClause::InlineRows whose cells now admit
an internal-only Expr::SeededValue(UnresolvedValue, span) variant — never
produced by the parser, carrying entity IRIs (including stable _:fdb- blank
node labels) and typed literals losslessly into both the read-side lowering
(probe/re-bind/RETURN queries) and the write-side lowering (seeded Txns).
InlineRows vars now register as bound in the write lowering (bound_vars),
so seeded rows satisfy SET/REMOVE target checks.
Dispatch
cypher_write_plan classification order:
detect_conditional— the existing probe/branch shapes (standaloneMERGE … ON MATCH SET, per-row relationship MERGE pair-staging, DELETE guards) keep their proven paths.detect_sequential(new) — multi-write-clause statements containing a MERGE (MERGE chains, CREATE+MERGE mixes, interleaved SET/REMOVE), and the previously-rejected node MERGE under a plain leading MATCH (the driver’s per-row partition makes it correct: one create per distinct absent key, every row binds). It also takes statements without a MERGE whose write clauses can see or undo one another’s writes (interacting_writes): two clauses writing one property key or one label (on any variables, since two can bind one node), a DETACH DELETE or a relationship’s DELETE with any other write, or a CREATE with a clause whose lowering reads stored values (a SET or REMOVE of a property, a map SET, DETACH DELETE — theOPTIONALlookup adds rows the CREATE would fire on), plus a SET writing one key twice. The single-Txnlowering evaluates every clause against the state before the statement and stages them as one operation, so for these it neither lets a later clause see an earlier one nor lets the last write hold; the driver stages clause by clause and the stager folds each fact to its last write, as for SPARQL;updates. Clauses that can’t meet (CREATE chains, SETs of different keys) keep the singleTxn, as do statements with a member the driver doesn’t stage (a bare node DELETE,CALL { … }before the writes, anOPTIONAL MATCHbefore a CREATE), which the singleTxn’s lowering refuses in turn.- Single-Txn lowering — everything else, including its existing clear errors.
WritePlan gains a Sequential variant. Call sites:
- Autocommit (
transact_cypher_statement) and Bolt explicit transactions (cypher_transaction_write) call the driver directly; the Bolt path stages against the transaction’s private state, so statement pipelining and read-your-writes compose. - Consensus (
LocalCommitter): the plan resolves under the write lock per retry attempt (same head-check/refresh contract as conditional MERGE).TransactionReceiptcarries an optional Cypher RETURN table (serdedefaultNone); Raft mode rejects multi-clause RETURN in v1.
The UNWIND $batch desugar’s last-wins row dedup
(dedup_rows_by_merge_identity) now applies only to single-write-clause
statements — it keys on the first node MERGE and would drop legitimate rows
in a chain; the driver owns duplicate-key semantics for multi-clause
statements.
Non-impact guarantees
- JSON-LD / Turtle / SPARQL transactions write what they wrote before. Two
changes reach code they share:
stage()’sf:reifies*cascade refuses a parallel relationship on an edge a CypherDELETE rretracts (TxnOpts::relationship_deletes). No other transaction sets the marker, and without it staging notes nothing.- The sequential stager builds the planner stats once, for the state a
write begins on, and pins them (estimates only) onto every state it
advances to, so SPARQL’s
;-separated operations after the first plan from those stats rather than assembling their own.
stage_transaction_from_txnskeeps itslen == 1fast path; only its N ≥ 2 loop body moved intoSequentialStager.- Single-clause Cypher writes keep their existing plans byte-for-byte, except
a SET that writes one property key twice. Multi-clause writes without a
MERGE keep the single
Txnunless their clauses can meet (above): those wrote the wrong facts there.
Costs
Per MERGE clause: two or three VALUES-seeded engine queries (match probe,
identity-tuple evaluation for dedup, re-bind) plus one or two stagings. The
common path is row-bulk. A computed ON CREATE / ON MATCH value that must
read the current graph uses a correctness-first per-row query/stage loop so
duplicate rows observe prior updates. Row tables are capped
(MAX_SEQUENTIAL_ROWS) with a clear error.