watershed Collaborative data structures for Gleam

Edit upstream.
Converge downstream.

watershed lets people edit shared data without losing concurrent changes. Clients apply edits at once; the server sequences them; each data structure resolves conflicts under its stated policy. The same Gleam core runs on the BEAM and in the browser, on a distributed-data-structure (DDS) runtime and sequencer drawn from Fluid Framework’s design.

View on GitHub Watch three clients converge ↓
Skip past the interactive demo

Watch three clients converge

On a photorevised survey sheet, magenta marks updates not yet field-checked. This demo borrows that color code: edits that aren’t confirmed yet are drawn in magenta, confirmed state in ink. All three clients run watershed’s compiled map_kernel Gleam code and share one ordered list of changes. That is a shared map, the structure most collaborative apps start with.

Nudge a gauge, have two clients write at once, stretch the network delay. The copies agree once queued writes arrive; the most recent write to a key wins.

Try cutting Client B’s link mid-edit: its writes park locally, the others keep converging, and restoring the link catches B up and resubmits what it wrote offline.

  1. 1 · local edit prints in magenta
  2. 2 · the sequencer stamps it with an SN
  3. 3 · every client lands the same ink state

Merge rule: last write wins. Race two writes to the same key and the op the sequencer stamps later overwrites the earlier one, identically on every replica.

Client A

0 pending
mill-race 24
kettle-run 61
low-ford 42
sandbags-placed 120
earthwork-balance · yd³ +44
fill Σ
74
cut Σ
30
inspection-count 18
A Σ
0
B Σ
0
C Σ
0
north-levee —
spillway-gate —
pump-house Survey
spoil-north 18
borrow-pit-7 -6
wash-fill 12
north-stake marked
sluice-tag marked
borrow-flag clear
BM-17 recorded
BM-22 unrecorded
BM-31 unrecorded
stake-3 active
gate-pin unplaced
silt-flag retired
register atomic LWW revise
north-bench Survey Survey
gate-setpoint — —
pump-mode — —
sheet state op
queue grade-stakes, pump-check
held jobs none
pact accepted pending op
datum-grid Survey datum —
gate-policy — —
inspection-window — —
task assigned waiters op
sluice-inspection — empty
pump-watch — empty
crest-walk — empty

Client B

0 pending
mill-race 24
kettle-run 61
low-ford 42
sandbags-placed 120
earthwork-balance · yd³ +44
fill Σ
74
cut Σ
30
inspection-count 18
A Σ
0
B Σ
0
C Σ
0
north-levee —
spillway-gate —
pump-house Survey
spoil-north 18
borrow-pit-7 -6
wash-fill 12
north-stake marked
sluice-tag marked
borrow-flag clear
BM-17 recorded
BM-22 unrecorded
BM-31 unrecorded
stake-3 active
gate-pin unplaced
silt-flag retired
register atomic LWW revise
north-bench Survey Survey
gate-setpoint — —
pump-mode — —
sheet state op
queue grade-stakes, pump-check
held jobs none
pact accepted pending op
datum-grid Survey datum —
gate-policy — —
inspection-window — —
task assigned waiters op
sluice-inspection — empty
pump-watch — empty
crest-walk — empty

Client C

0 pending
mill-race 24
kettle-run 61
low-ford 42
sandbags-placed 120
earthwork-balance · yd³ +44
fill Σ
74
cut Σ
30
inspection-count 18
A Σ
0
B Σ
0
C Σ
0
north-levee —
spillway-gate —
pump-house Survey
spoil-north 18
borrow-pit-7 -6
wash-fill 12
north-stake marked
sluice-tag marked
borrow-flag clear
BM-17 recorded
BM-22 unrecorded
BM-31 unrecorded
stake-3 active
gate-pin unplaced
silt-flag retired
register atomic LWW revise
north-bench Survey Survey
gate-setpoint — —
pump-mode — —
sheet state op
queue grade-stakes, pump-check
held jobs none
pact accepted pending op
datum-grid Survey datum —
gate-policy — —
inspection-window — —
task assigned waiters op
sluice-inspection — empty
pump-watch — empty
crest-walk — empty
Sequencer SN 0

    Loading booting watershed kernels…

    Latency and jitter affect simulated arrival order; animation speed changes playback only. “Ops in flight” counts every hop still travelling: one client → sequencer leg, then one sequencer → replica leg per client.

    Every other structure runs the same way on its own page. Start with the field atlas, the full catalog →

    One ordered stream, one clear rule per structure

    watershed doesn’t paper over conflicts with one catch-all merge. Each structure states a small, clear rule: what can show up instantly, what has to wait for the server to confirm it, and what gets saved so a client can reload later.

    Only a few of watershed’s structures are shown here. Find the rest, grouped by family, in the field atlas. The full catalog includes a live demo for each structure.

    SharedMap

    map_kernel
    Merge rule
    for each key, the most recent write wins, decided by server order
    Before it’s confirmed
    your writes show instantly, then lock in once the server confirms them
    What gets saved
    confirmed entries reload with their keys and insertion order intact
    Read in depth →

    SharedCounter

    counter_kernel
    Merge rule
    everyone sends +/− changes instead of overwriting, so simultaneous edits just add up
    Before it’s confirmed
    your change shows next to the confirmed total right away
    What gets saved
    only one number needs saving, since every change is an add
    Read in depth →

    OrSet

    or_set_kernel
    Merge rule
    add, remove, and add again all work; if an add and a remove race, the add wins
    Before it’s confirmed
    your change overlays the list in magenta until it’s confirmed
    What gets saved
    current members and their removal history reload intact
    Read in depth →

    Claims

    claims_kernel
    Merge rule
    the first client to claim a slot owns it; every later claim is refused
    Before it’s confirmed
    a claim only shows as yours once it has actually won, never before
    What gets saved
    who owns what reloads intact
    Read in depth →

    Every structure uses the same channel

    Every structure above rides the same ordered stream of changes, the very one the live SharedMap demo runs on. Open any family to switch which structure is on screen; they all stay hosted together as channels sharing one sequenced document.

    1. 01

      Submit

      A client makes a change and updates just its own local copy for now.

    2. 02

      Sequence

      A server like floodgate puts everyone’s changes in one order and sends that order to every client.

    3. 03

      Confirm / apply

      The client that made the change locks it in; everyone else applies it through the exact same code.

    4. 04

      Summarize

      A saved snapshot lets any client reload the current state, the same path used after a reconnect.

    Every structure,
    mapped by family.

    The demo above runs one kernel. The atlas is the whole field guide: every structure watershed ships, grouped by how it converges, each with its own live demo and merge rule.

    Open the field atlas →

    Counters

    Numbers that many hands move at once.

    • SharedCounter DDS
    • GCounter CRDT
    • PnCounter CRDT

    Sets

    Lists of things, as people add and remove at the same time.

    • GSet CRDT
    • TwoPSet CRDT
    • OrSet CRDT

    Maps

    Keyed state, resolved two different ways.

    • SharedMap DDS
    • OrMap CRDT
    • SharedDirectory DDS

    Sequences

    Ordered lists that stay ordered while everyone rearranges them.

    • SharedSequence CRDT
    • SharedText CRDT

    Coordination

    Deciding who owns what, and agreeing before acting.

    • Claims DDS
    • RegisterCollection DDS
    • OrderedCollection DDS
    • TaskManager DDS
    • PactMap DDS

    Transforms

    One shared document, kept in agreement as everyone edits.

    • JsonOt OT
    • SharedRichText OT

    One pure core, two runtimes

    watershed is written once and runs in two places. runtime_core, channel, wire, and the kernels compile for both targets. Erlang and JavaScript keep separate facades, runtimes, and socket bindings only where the platform forces them to differ.

    The boundary is enforced by target gates: watershed_beam and runtime_beam are @target(erlang); watershed, runtime, and transport_js are @target(javascript). The ungated core underneath is the compiled code running in the browser demo.

    A map over one sequenced stream

    SharedMap exposes familiar operations: set, get, subscribe. There is no merge callback to install: local writes apply optimistically, the server assigns their sequence numbers, and the highest-sequenced write wins per key.

    The same core runs on the Erlang target as an OTP actor and on the JavaScript target inside a Lustre single-page app. The facades differ at the edge: watershed_beam blocks and returns a Subject, while watershed uses callbacks. See the examples for both, verified converging against a live floodgate sequencer.

    Build an app, step by step: the field guide → 

    import gleam/erlang/process
import gleam/io
import gleam/json
import watershed_beam
import watershed/map_kernel.{type MapEvent, ValueChanged}

type Msg {
  MapChanged(MapEvent)
}

pub fn main() {
  // Connect, blocking until the op history has replayed locally. `connect`
  // returns a Result — asserted here for brevity; a supervised app matches
  // the Error case and lets OTP restart the link.
  let assert Ok(doc) =
    watershed_beam.connect(
      host: "127.0.0.1",
      port: 4000,
      tenant: "dev-tenant",
      document: "river-gauges",
      token: token,
      user_id: "gauge-1",
    )
  let gauges = watershed_beam.root(doc)

  let selector =
    process.new_selector()
    |> process.select_map(watershed_beam.subscribe(gauges), MapChanged)

  // Optimistic write: applies now and emits locally, then goes to the server.
  watershed_beam.set(gauges, "mill-race", json.int(24))

  let MapChanged(event) = process.selector_receive_forever(selector)
  case event {
    ValueChanged(key:, ..) -> io.println("revised: " <> key)
    _ -> Nil
  }
}
    examples · erlang + javascript targets

    What is implemented and tested

    The runtime is tested three ways: against reference models, against a live floodgate server, and against deliberately nasty orderings of operations — 1,392 test functions across 105 test files at last count. The ledger is what those checks cover: everything watershed ships.

    Property-tested convergence
    Eleven reference models — map, directory, counters, claims, registers, ordered collection, pact, sequence, text — each replay randomized multi-client command traces, 200 per property by default (FUZZ_ITERATIONS raises it), including partial sequencing and delivery, reconnect, rollback, stashed ops, and summary-loaded joins. Invariants are checked after each command and convergence after a final sync; qcheck shrinks failures and the harness dumps a replayable JSON trace. Read the harness →
    Reference-checked map ops
    SharedMap’s inner set, delete, and clear payloads are checked against 26 oracle scenarios generated from @fluidframework/map itself, including state, events, and convergence. Container, attach, and summary formats remain watershed’s own. Read the corpus →
    Reconnect safety
    Buffered out-of-order delivery, in-band op requests, and client-id remapping on reconnect. Pending edits survive the round trip — the fuzz harness disconnects clients mid-trace to prove it. Read the disconnect suite →
    What ships today
    01 Maps SharedMap, OR-map, and SharedDirectory: Fluid-derived op semantics, summaries, nested handles, and folder hierarchies
    02 Counters and sets SharedCounter, G-counter, PN counter, G-set, 2P-set, and OR-set: delta merge, duplicate-safe delivery, add-wins and permanent removal
    03 Sequences and text SharedSequence and SharedText: grapheme-indexed edits, convergent move, and anchors that survive concurrent edits
    04 Transforms json_ot and SharedRichText: one client-transform protocol over the ottypes json0 and quill-delta algebras
    05 Coordination Claims, TaskManager, ordered collection, register collection, and pact map: ownership, work queues with failover, and quorum agreement
    06 Runtime Server-sequenced delivery, requestOps catch-up, reconnect with client-id remap, nack policy, heartbeat, resubmit, and automatic summaries
    07 App layer Typed schema decoding, ensure_* bootstrap, presence ripples, Lustre effect bindings, and the in-memory sluice for tests

    Not yet a Hex release — installs are git-pinned — and container, attach, and summary formats are watershed’s own rather than Fluid’s. Wire compatibility is at the operation layer.