watershed Collaborative data structures for Gleam

← watershed · Data structures / Sets

Sets

Lists of things, as people add and remove at the same time.

A set looks simple until two clients disagree about whether an element belongs. These are a short course in that problem: the more removal you want, the more causal bookkeeping you pay for.

Start with add-only union, add irreversible tombstones, then reach the observed-remove set that lets you add, remove, and add again while staying convergent.

Skip past the interactive demo

Watch sets converge

The structures from this family that fit the shared gauge rig run below on watershed’s Gleam kernels. Structures with dedicated interactions link to their own demos. Pending edits print in magenta; once the sequencer stamps an SN, every replica lands the same ink state. The picker switches among the loaded views, which share one ordered stream.

Nudge a value, race a concurrent write, or stretch the link latency to watch each merge rule hold.

  1. 1 · local edit prints in magenta
  2. 2 · the sequencer stamps it with an SN
  3. 3 · every client lands the same ink state
Data structure shown in all clients

Merge rule: grow-only union. Marking a benchmark is a permanent fact. Race two marks or deliver one twice: the registry only grows, and duplicate deltas are absorbed.

Client A

0 pending
Shared map replica on Client A
mill-race 24
kettle-run 61
low-ford 42
sandbags-placed 120
earthwork-balance · yd³ +44
fill Σ
74
cut Σ
30
inspection-count 18
A Σ
0
B Σ
0
C Σ
0
Claims replica on Client A
north-levee
spillway-gate
pump-house Survey
OR-map stockpile ledger replica on Client A
spoil-north 18
borrow-pit-7 -6
wash-fill 12
OR-set field marker roster replica on Client A
north-stake marked
sluice-tag marked
borrow-flag clear
G-set permanent benchmark registry replica on Client A
BM-17 recorded
BM-22 unrecorded
BM-31 unrecorded
2P-set retired marker ledger replica on Client A
stake-3 active
gate-pin unplaced
silt-flag retired
RegisterCollection replica on Client A
register atomic LWW revise
north-bench Survey Survey
gate-setpoint
pump-mode
OrderedCollection replica on Client A
sheet state op
queue grade-stakes, pump-check
held jobs none
PactMap replica on Client A
pact accepted pending op
datum-grid Survey datum
gate-policy
inspection-window
TaskManager replica on Client A
task assigned waiters op
sluice-inspection empty
pump-watch empty
crest-walk empty

Client B

0 pending
Shared map replica on Client B
mill-race 24
kettle-run 61
low-ford 42
sandbags-placed 120
earthwork-balance · yd³ +44
fill Σ
74
cut Σ
30
inspection-count 18
A Σ
0
B Σ
0
C Σ
0
Claims replica on Client B
north-levee
spillway-gate
pump-house Survey
OR-map stockpile ledger replica on Client B
spoil-north 18
borrow-pit-7 -6
wash-fill 12
OR-set field marker roster replica on Client B
north-stake marked
sluice-tag marked
borrow-flag clear
G-set permanent benchmark registry replica on Client B
BM-17 recorded
BM-22 unrecorded
BM-31 unrecorded
2P-set retired marker ledger replica on Client B
stake-3 active
gate-pin unplaced
silt-flag retired
RegisterCollection replica on Client B
register atomic LWW revise
north-bench Survey Survey
gate-setpoint
pump-mode
OrderedCollection replica on Client B
sheet state op
queue grade-stakes, pump-check
held jobs none
PactMap replica on Client B
pact accepted pending op
datum-grid Survey datum
gate-policy
inspection-window
TaskManager replica on Client B
task assigned waiters op
sluice-inspection empty
pump-watch empty
crest-walk empty

Client C

0 pending
Shared map replica on Client C
mill-race 24
kettle-run 61
low-ford 42
sandbags-placed 120
earthwork-balance · yd³ +44
fill Σ
74
cut Σ
30
inspection-count 18
A Σ
0
B Σ
0
C Σ
0
Claims replica on Client C
north-levee
spillway-gate
pump-house Survey
OR-map stockpile ledger replica on Client C
spoil-north 18
borrow-pit-7 -6
wash-fill 12
OR-set field marker roster replica on Client C
north-stake marked
sluice-tag marked
borrow-flag clear
G-set permanent benchmark registry replica on Client C
BM-17 recorded
BM-22 unrecorded
BM-31 unrecorded
2P-set retired marker ledger replica on Client C
stake-3 active
gate-pin unplaced
silt-flag retired
RegisterCollection replica on Client C
register atomic LWW revise
north-bench Survey Survey
gate-setpoint
pump-mode
OrderedCollection replica on Client C
sheet state op
queue grade-stakes, pump-check
held jobs none
PactMap replica on Client C
pact accepted pending op
datum-grid Survey datum
gate-policy
inspection-window
TaskManager replica on Client C
task assigned waiters op
sluice-inspection empty
pump-watch empty
crest-walk empty
Sequencer SN 0

    Loading booting watershed kernels…

    Latency and jitter affect simulated arrival order; animation speed changes playback only. “Ops in flight” counts every hop still travelling: one client → sequencer leg, then one sequencer → replica leg per client.

    GSet

    CRDT
    g_set_kernel

    A set you can only add to (the simplest one that always agrees).

    The simplest set CRDT. Elements can be added but never removed. Merging two replicas is a plain union (commutative, associative, and idempotent), so adds arrive in any order, any number of times, and everyone converges on the same membership.

    Removal is not expressible, which is what makes a G-set trivially correct. When you need removal, you layer tombstones on top: the 2P-set and OR-set below.

    Best for

    • Append-only registries: recorded events, observed device IDs, seen keys
    • Deduplicated event logs where membership only ever grows
    • The base layer for removable set CRDTs
    Merge rule
    an add-only set; once something is in, it stays, and merging is a plain union
    Optimistic behavior
    your addition shows in magenta until it’s confirmed
    Summary shape
    the confirmed set reloads as a permanent record
    Model
    Conflict-free replicated data type: converges by merge

    TwoPSet

    CRDT
    two_p_set_kernel

    A set you can remove from, but a removed item never comes back.

    A two-phase set layers a grow-only set of tombstones over a grow-only set of adds. An element is a member when it is in the add-set and absent from the tombstone-set.

    Both halves only grow, so merge is two unions and convergence is guaranteed. The trade-off is stark: once removed, an element can never be re-added, because the tombstone always wins. Concurrent add-versus-remove resolves remove-wins on every replica. A reset needs a fresh set rather than a shrinking op.

    Best for

    • Membership where retirement is final: revoked credentials, decommissioned assets
    • Audit or compliance sets where a removal must never silently reverse
    • Cases where remove-wins is correct and re-adding is genuinely disallowed
    Merge rule
    supports removal, but once an item is removed it can never be added again
    Optimistic behavior
    adds and removals show in magenta until they’re confirmed
    Summary shape
    current items and removed ones reload together
    Model
    Conflict-free replicated data type: converges by merge

    OrSet

    CRDT
    or_set_kernel

    A set where add, remove, and add-again all work: the everyday choice.

    An observed-remove set fixes the 2P-set’s fatal flaw: you can add, remove, and add again. Every add attaches a unique causal tag (a dot), and a remove only tombstones the tags it has actually observed.

    If one client removes an element while another concurrently adds it under a fresh tag, the new tag survives and the element stays (add-wins). That bookkeeping is why the OR-set is the workhorse removable set across collaborative apps.

    Best for

    • Collaborative selections, tags, labels, and shopping carts
    • Durable roster membership edited concurrently by many clients; transient online presence belongs in ripples
    • Any removable set where re-adding a just-removed item must work
    Merge rule
    add, remove, and add again all work; if an add and a remove race, the add wins
    Optimistic behavior
    your change overlays the list in magenta until it’s confirmed
    Summary shape
    current members and their removal history reload intact
    Model
    Conflict-free replicated data type: converges by merge