watershed Collaborative data structures for Gleam

← watershed · Foundations / Documents and handles

Documents and handles

A watershed document is not one tree — it's a root map plus whatever channels its values point to, each one independently addressed, attached, and resolved.

Traces root_typed · handle_of · create_map · resolve

The root is one map, not one tree

Every document has exactly one root: a SharedMap, reached with root(document), and viewed through your schema with root_typed. Its tag comes from the Document(root) value itself, fixed at the one place your app first holds it — a Msg constructor, usually. Everything your schema declares hangs off that one map. 

/// The root map of the document, viewed through the schema of that document.
///
/// One document has one tag. The tag comes from the `Document(root)` value that
/// you pass, so it is fixed at the position where your application writes the
/// type concretely. That position is the `Msg` constructor that carries the
/// handle, or the `Model` field that holds it:
///
/// ```gleam
/// GotHandle(Document(document_schema.Survey))
/// ```
///
/// Every `root_typed` call on that document then agrees. A second schema at the
/// root is a compile error, and not a key namespace that two schemas share
/// quietly.
///
/// A component that is generic in `root` can still call this function. But an
/// abstract tag has no field, so that component cannot read or write the root.
/// A nested panel is thus structurally unable to reach past its own child map.
///
/// `typed(root(document))` is still available, and it is still unchecked. It is
/// the deliberate way to view the root through a foreign schema. Unlike the old
/// signature, you must now write it explicitly.
pub fn root_typed(document: Document(root)) -> TypedMap(root) {
  typed(root(document))
}

A plain field stores a value; a handle points at a channel

A Field writes its value straight into the map: a string is a string, on the wire and in the map's entries, no indirection. A ChildField or a ChannelField stores something else entirely — a small JSON object that marks a reference to another channel, elsewhere in the same document. Reading that key back gives you the marker, not the child's contents; you resolve it separately to reach the channel it points to. 

/// The Fluid handle marker that references `map`. Store it as a value in
/// another map. Its shape is
/// `{"type": "__fluid_handle__", "url": "/<address>"}`.
pub fn handle_of(map: SharedMap) -> Json {
  handle.encode_handle(map.address)
}

Detached, attached, resolved

A new channel starts life detached: local only, its edits producing no op, invisible to anyone else. It stays that way until you store its handle into a map that's already attached to the document. The runtime then attaches the new channel too, sends its snapshot, and only from that point on does it synchronize with the rest of the world. Create a map and forget to store its handle anywhere, and it just quietly stays a local orphan. 

/// Create a new map channel. The map starts *detached*, which means that it is
/// local only and its edits produce no operation. It stays detached until a
/// caller stores its handle, from `handle_of`, into an attached map. The
/// runtime then attaches it, with its snapshot, and it starts to synchronize
/// the edits of that map. The connection must be ready, which `on_ready`
/// reports.
pub fn create_map(document: Document(root)) -> Result(SharedMap, String) {
  runtime.create_map(document.runtime)
  |> result.map(fn(address) {
    SharedMap(runtime: document.runtime, address: address)
  })
}

Going the other way, resolve turns a handle you read back out of a map into the live channel it names. 

/// Resolve a handle value, from `get` or from `entries`, to the SharedMap that
/// it references. A caller can retry after an error. A handle from a remote
/// value can stay unresolved for a short time, while the attach operation of
/// the channel that it references is still in flight.
pub fn resolve(
  document: Document(root),
  value: Json,
) -> Result(SharedMap, String) {
  case handle.parse_handle(value) {
    Error(Nil) -> Error("value is not a handle marker")
    Ok(address) ->
      runtime.resolve_address(document.runtime, address)
      |> result.map(fn(_) {
        SharedMap(runtime: document.runtime, address: address)
      })
  }
}

ChildField versus ChannelField: same handle, different destination

Both store a handle. The difference is only in what kind of thing the handle points to, and the schema layer tracks that at compile time. ChildField(tag, child) points at another typed map — a nested document with its own schema, its own tag, its own set of fields. ChannelField(tag, kind) points at a channel that isn't a map at all: a counter, an OR-set, a claims channel, whatever kind names. Neither one is a subtree of the parent. Both are peer channels that happen to be reachable from it.

Independently addressed pieces, not one atomic tree

A watershed document collects data in a set of channels, each with its own address, each syncing on its own terms, tied together only by handles stored as ordinary values. Your application decides what that collection represents: the build guide makes each document a retro board, while the dice-scoreboard example makes each one a scoreboard. Its roster of players, keyed by id, looks like this on the wire — a root map, a child map, and a further map nested under that: 

root
├─ "game"      = "watershed dice scores"   (Field(GameRoot, String))
├─ "die_sides" = 6                         (Field(GameRoot, Int))
└─ "players"   = handle ──▶ roster         (ChildField(GameRoot, Roster))

roster
└─ "player-1234" = handle ──▶ player map

player map
├─ "name"
├─ "last_roll"
├─ "total"
└─ "rolls"
diagram · shape of examples/scoreboard_cli/src/scoreboard_cli.gleam

Nothing about that layout is a single data structure the way a JSON document or an in-memory tree is. Each named piece is its own channel, addressed on its own, with its own merge behavior — the field atlas covers what each kind actually does when two clients touch it at once.

What that layout costs you

Two honest limits fall straight out of this design, and neither one is a bug to route around:

  • No transaction across channels. Writing to two channels is two separate operations. There's no way to make both succeed or both fail together — if you need that, keep the related values in one channel instead of splitting them across two.
  • A handle can be briefly unresolved. Right after a channel is created, its attach operation has to reach every peer before that peer can resolve the handle pointing to it. Until then, resolving returns an error you can retry, not a broken document. 
/// Resolve the nested typed map that a child field references. The result is
/// `Ok(None)` when the key is absent. The function returns an error from
/// `resolve` without a change, and a caller can retry after it. That includes
/// the short-lived error for a channel that is not attached yet.
pub fn resolve_child(
  document: Document(root),
  typed_map: TypedMap(s),
  field: ChildField(s, c),
) -> Result(Option(TypedMap(c)), String) {
  case get(typed_map.map, schema.child_key(field)) {
    Error(Nil) -> Ok(None)
    Ok(value) ->
      resolve(document, value)
      |> result.map(fn(resolved) { Some(typed(resolved)) })
  }
}

That retry isn't an edge case to special-case away. Every client that creates a channel and immediately hands its handle to a peer has to expect this. The next sheet covers the pattern watershed uses to wait it out cleanly at startup, and runtime · reconnect covers what a dropped connection does to an already-resolved handle.

In short

A watershed document is a root map plus whatever channels its handles reach — attached on creation, resolved through the handle rather than nested by containment, with no cross-channel transaction and a short resolve window right after creation. Next: how a client gets from an empty connection to a document it can build on.