Barrel
FR GitHub

Vision: the database for agents

This note defines what the barrel application becomes: not a wrapper over barrel_docdb and barrel_vectordb, but the layer that makes them one database, positioned for agent workloads. Read it before working on any layer feature; the decision log at the end records what is settled and why.

Positioning

Turso's agent story reduces to five ideas: database-as-file multi-tenancy, copy-on-write branching, MVCC with commit-time conflict detection, asymmetric sync with an authoritative remote, and an audit-log packaging (AgentFS). The agent-infrastructure field splits into a memory layer (Letta, Mem0, Zep: extraction, consolidation, decay) and a substrate (Neon, Turso, Tiger, Cloudflare: cheap isolated DBs, branches, time travel, hybrid search, MCP). Nobody owns both. Barrel does: this umbrella is the substrate, and the memory layer sits on top.

Barrel's wedges, which the incumbents cannot copy:

  • In-VM embedding for BEAM applications: no network hop, no serialization.
  • True edge and multi-master sync with causal (HLC) conflict resolution; Turso retreated from edge replicas and uses last-push-wins.
  • Branch merge. Turso and Neon branches are one-way forks; a CouchDB-heritage engine gets merge from its replication machinery.
  • Provenance at the database level ("what did the agent know when").
  • Open source and self-hosted end to end, EU-friendly.

The record

A barrel record is one unit: document body, blobs (attachments), vectors, and provenance (HLC, actor, source, session). One write path owns it.

  • Write = document + embed-pending marker in a single docdb WriteBatch (atomic). Vector indexing is driven from the changes feed with exactly-once semantics; a re-driven consumer heals missing vectors after a crash.
  • Per-database embedding policy: which fields embed, which model, sync or async. Sync mode gives read-your-write search; async is the default.
  • barrel_vectordb stores only vectors and ANN indexes. Text and metadata are read through the barrel_vectordb_docstore seam pointed at the record's database (the seam's direction inverts: the document is primary, the vector is derived).
  • Endgame (later phase): vector column families move into the record store's RocksDB so vector and document commit in the same batch with no marker.

Query: BQL, a PartiQL dialect

One query surface over documents, vectors, and full-text.

  • PartiQL semantics: MISSING vs NULL, path expressions, UNNEST. Adopt the spec, do not invent semantics.
  • leex/yecc parser (riak_ql is the architectural template). The planner compiles onto the existing machinery: automatic path indexing, roaring64 postings, cardinality-based index selection, snapshot cursors.
  • Vector and FTS enter SQL as table functions first: vector_top_k(...), bm25_top_k(...) (the libSQL pattern, no planner magic). Hybrid = RRF. An ORDER BY distance LIMIT k planner rewrite can come later.
  • Live queries: the same query + SUBSCRIBE, backed by the existing query subscriptions; a live query is an Erlang process pushing deltas.
  • v1 scope: SELECT, WHERE, ORDER BY, LIMIT, paths, UNNEST, table functions. Not v1: joins, GROUP BY, SQL-92 completeness.

Sync

Server-authoritative for light clients, multi-master between servers.

  • HLC version vectors replace rev-tree semantics (the Couchbase Lite 4.0 move). Conflict default: last-write-wins by HLC, which respects causality. Losing versions are superseded, not deleted, and stay queryable in history. A merge hook allows custom resolution.
  • Asymmetric protocol: push logical HLC-stamped mutations, pull a compacted change log.
  • Partial sync is a write-time index (channels derived from the existing path posting lists), never a read-time filter over the global feed.
  • Blobs sync out-of-band, content-addressed. Vectors sync quantized; ANN indexes are never shipped, they rebuild locally.
  • A per-database retention window bounds tombstones, the PITR window, and the merge window in one knob.

Timeline: branches and point-in-time recovery

Branching, PITR, and merge are one subsystem built on two primitives: RocksDB checkpoints (fork mechanics: hardlinked SSTs, near-instant) and a retained HLC change log (semantics). The changes feed must retain history within the retention window; today it keeps only the latest change per doc.

  • Branch = fork at now: checkpoint both stores, record {parent, fork_hlc}.
  • PITR = fork at a past T: nearest checkpoint before T, replay the log to T. Restore creates a new database (Turso parity), and it can be merged back.
  • Merge = one-shot replication of branch changes since fork_hlc into the parent through the normal sync conflict machinery, with a policy of lww | fail_on_conflict | Fun. Merge is sync; no separate engine.
  • v1 lineage is linear parent-child. Branch DAGs and cross-lineage merges are explicitly later.

Agent layer

  • Database-per-agent: open/create cheap enough for hundreds of ephemeral databases per task; one supervised process per database, binary names end to end (no atom leak), and a lifecycle manager (barrel_dbs) with idle close and LRU eviction.
  • Spaces: shared context containers with capability tokens; context is shared by reference, not copied. A space IS a barrel database; a capability is a hashed random token with read < write < admin rights, revocable locally.
  • Sessions with TTL and handoffs are barrel primitives; a handoff is a shared space plus a capability whose possession is the right to accept. Sessions ride a general document-TTL primitive (expires_at + opt-in sweeper).
  • MCP as a first-class surface: databases and live queries exposed as MCP resources with subscriptions (specced but unimplemented industry-wide), plus tools for query, write, branch, and merge, and the agent layer.
  • Audit: provenance fields plus the retained change log answer "what did the agent know when" at the database level.
  • SHIPPED (phase 8): see decision log row 6.

Concurrency and encryption

  • Point reads move off the per-database writer onto RocksDB snapshots (queries already bypass it; get_doc must too). Group commit in the writer. Document-level rev CAS stays as commit-time conflict detection. Write scale-out comes from many databases, which the agent model provides.
  • Encryption at rest: a barrel_keyprovider behaviour with per-database keys (BYOK) resolved on the KEYSPACE (a branch opens with its parent's key), RocksDB EncryptedEnv for every store, and a sector cipher (offset-addressable AES-256-CTR + GCM envelopes for term files) for the mmap'd flat files EncryptedEnv cannot cover: BM25 disk and DiskANN. HNSW/FAISS state persists through the encrypted stores. Fail-closed opens via cleartext key-check markers. Per-database keys double as agent isolation. SHIPPED (phase 7).

Browser: barrel-lite

A TypeScript protocol client, not a WASM port of the engine.

  • OPFS storage with Web-Locks leader election (single writer per origin).
  • HLC-stamped local mutations; push logical, pull the change log; local data is a cache (Safari evicts), sync is the durability story.
  • Local BQL subset for the synced set; no ANN in the browser: brute-force cosine over the synced subset, optional transformers.js embeddings, offload heavy or global queries to the server.
  • SHIPPED (phases 9a, 9b, 9c): see decision log row 7.

Non-goals

  • No symmetric CRDT replication (the field's post-mortems are conclusive).
  • No SQL-92 completeness before the document subset earns it.
  • No WASM port of the Erlang engine.
  • No embedded DuckDB/DataFusion query engine (cannot see barrel's indexes).

Decision log

#DecisionChoiceRationale
1Sync coreHLC version vectors, not rev-treesRev-tree wounds (tombstones, arbitrary winners, metadata in read path); CBL 4.0 precedent. SHIPPED as phases 3+5: VV protocol + retained history (phase 3); network transport over HTTP (JSON + base64 wire, /db/:db/_sync/*), static bearer auth, write-time channels (0x1E feed) for partial sync, content-addressed attachment sync (feed in the blob store, LWW on origin HLC, streamed wire), event-driven continuous tasks (changes-stream wake, adaptive remote polling, error backoff). Quantized vector sync deferred to phase 9
2Doc+vector atomicityOutbox marker in one WriteBatch, changes-feed indexing; storage unification laterAtomic now without blocking vectordb's standalone life. SHIPPED as record mode (phase 2): tagged outbox + supervised indexer + sync mode + read-through docstore
3Query dialectPartiQL semanticsJSON-first SQL semantics already specified; nothing to invent. SHIPPED as BQL v1 (phase 4): leex/yecc parser lowering onto the query engine, vector_top_k/bm25_top_k/hybrid_top_k table functions, UNNEST, SUBSCRIBE live queries, streaming REST endpoint
4BranchingTimeline subsystem: checkpoint + retained HLC log; branch, PITR, merge-as-sync; linear lineage v1Merge is the leapfrog over Turso/Neon; PITR parity required. SHIPPED (phase 6): O(1) hard-link forks via rocksdb checkpoints + keyspace indirection (branch keeps the parent's name in keys, TIMELINE sidecar identity, fresh source_id); PITR rewind from the retained history log incl. conflict-window reconstruction; merge = incremental replication since fork through put_version (LWW/conflict_merger) with attachment phase; record-mode branches backfill their vector store from stored embeddings; REST /db/:db/_timeline/*
5Encryption at restPer-database keys via barrel_keyprovider, keyspace-resolved; EncryptedEnv + sector cipher for flat filesKeys double as agent isolation; a branch shares its parent's key by keyspace. SHIPPED (phase 7): barrel_crypto app (GCM envelope, offset-addressable CTR, key-check tokens, env/custom providers); EncryptedEnv on docdb docs+attachments and every vectordb RocksDB; BM25 disk (static postings nonce rotated per compaction, embedded blockmax) and DiskANN (embedded graph sectors, static vectors/pq, sealed meta/pq_state); fail-closed open matrix everywhere; one encryption spec at barrel:open covers the whole logical db; server via open_opts
6Agent layerSpaces = databases + capability tokens; sessions/handoffs folded in; MCP inside barrel_serverSharing by reference beats copying; possession-of-token beats caller identity for handoffs; one lifecycle manager serves REST, MCP, and spaces. SHIPPED (phase 8): binary db names + vectordb via-registry (atom leak closed); barrel_dbs lifecycle manager (idle close, LRU cap, pinning); provenance write option (actor/session/source, entity column + history TLV tail, local-only wire) with audit reads (history/doc_versions/version_body) and REST endpoints; doc TTL (expires_at, 0x1F expiry index, lazy reads + opt-in sweeper); apps/barrel_spaces (registry db, sp_ ids, per-space encryption), barrel_caps (bsp_ tokens, hash-only storage, rights ladder), sessions (sliding TTL, chronological messages, janitor), handoffs (CAS accept, complete revokes, chains); REST /spaces + /handoffs with capability bearers; MCP endpoint at /mcp (livery_mcp bridge, own auth provider for server + capability tokens, 26 tools, 3 resource templates, live-query bridge with debounced updated notifications, session GC, caps)
7Browser clientbarrel-lite: a TS protocol client in the umbrella (clients/barrel-lite), not a WASM portThe wire is plain JSON + base64, so the client re-implements the codecs and keeps local data as a cache. SHIPPED (phase 9a): byte-exact HLC/version/VV codecs (golden fixtures generated from the Erlang via an escript); local store over a StorageAdapter (memory + async main-thread OPFS) with a persisted source id and client-side checkpoints; one-shot and adaptive-polling live sync (apply mirrors do_put_version: skip/fast-forward/LWW with an onConflict hook); Web-Locks leader election with BroadcastChannel follower-proxy RPC (leader owns the store and sync); server enablers = CORS middleware (livery_cors wrapper) and capability tokens scoped to their space's /db routes (read = pull leg, write = push leg, fail-closed classifier); integration vs a real barrel_server and a Playwright chromium smoke. SHIPPED (phase 9b): a server-side continuous /changes SSE mode (subscribe push, ack backpressure, heartbeat) with a client stream that wakes the poller and degrades on drop; attachment sync (content-addressed OPFS blob store, sha256- digest, LWW on origin HLC, separate feed + checkpoint, put/get/remove/gc API); and a local BQL subset ported byte-for-byte (lexer + recursive-descent parser + lowering + evaluator) matching the server document subset, pinned by end-to-end golden fixtures generated from the Erlang, with queryRemote delegating table functions to POST /db/:db/query. SHIPPED (phase 9c): browser vector search = a dedicated embedding pull (include_embedding on GET doc + POST /db/:db/_bulk_get, the only server change; vectors ride lossless base64 float32 LE, byte-exact with barrel_doc:encode_embedding), a persisted single-dim vector index in the local store (vectors.json snapshot), and db.searchLocal running brute-force cosine top-k over the synced vectors with an optional BQL WHERE pre-filter; db.searchVector/searchText delegate to the server /search endpoints (a separate model-dimension ANN corpus from the per-doc emb column), and transformers.js is an optional example-only adapter, never a runtime dependency
8Scale-outContexts: bounded portable datasets that agents discover, attach, and query together; federation over composition, not transparent shardsSHIPPED in barrel 1.10.0 and barrel_server 1.10.0 (#50 to #59), design in contexts.md. Remote composition: context catalog, remote query client, federated BQL executor with per-source status, version and coverage, bounded query route (max_rows, deadline_ms, observed version), REST /contexts and MCP context_* tools. Ranking: retrieval is grouped per context; vector_top_k merges by score only when every member reports the same embedding fingerprint and cosine distance (measured equal to one brute-force index); BM25, hybrid and RRF scores never merge; rerank refused until a code-aware reranker is validated. Portability: read-only opens, export and verified read-only import of a database generation, working sets with retrieved-set slices and offline coverage (guide). Deferred: ngram index packages, pending a compact segment format

From docs/architecture/vision.md in the barrel repository at 794415d, updated 2026-09-26 Edit this page View as Markdown

Up and down to moveEnter to open