Barrel
FR GitHub

Contexts and working sets

Data rarely sits in one database. A support team may keep European tickets on one server and American tickets on another, and an agent that answers "which tickets took the longest" needs both. Querying each server in turn and stitching the answers together is work, and when one server is down you also need to know what the answer is missing. Contexts let you ask once, by name, and get one answer that says what it covers.

A context is a named dataset, one Barrel database today, that lives on this node, on another barrel_server, or in an imported snapshot. You register a card for it, then run one BQL statement over several contexts by name and get one answer that says which contexts answered and what each answer covers. A working set is the list of contexts you work with, some of them saved locally so they answer offline. Read this to know the model and its limits before you follow Query several datasets.

The model

Each term below is one piece of that flow: cards say where data lives, a context query asks the question, and a working set keeps the contexts you need close at hand.

TermWhat it is
CardA document in the node's catalog (_barrel_catalog): a name such as support/eu, an id ctx_..., one or more locations, and optional title, topics and embedding. Cards never hold secrets.
Locationlocal (a database on this node), remote (an endpoint and a database on another server) or snapshot (an imported export)
Context queryOne BQL statement, the list of contexts (or a working_set), and how to merge the answers
Working setA list of contexts. A member can be remote, a saved slice of query results, or an imported snapshot
SliceThe documents a query returned from one context, saved into a local database
SnapshotA read-only copy of a database made with barrel_ctx_export:export/3 and imported on another node

How a query runs

BQL has no syntax for contexts. You name them in the request, next to the statement. Every member runs the same statement (the collection name after FROM is ignored), in parallel, within budgets, and the node merges the rows.

This query asks the two support contexts for their two longest tickets and merges them into one list, longest first:

{ok, R} = barrel_ctx:query(#{
    query => <<"SELECT id, subject, minutes FROM c "
               "ORDER BY minutes DESC LIMIT 2">>,
    contexts => [<<"support/eu">>, <<"support/us">>]}).

The answer carries execution (succeeded, partial or failed), rows or groups, one sources entry per context, coverage, and a one-paragraph summary. A context that does not answer is reported in its source entry, not as a failed request. If support/us is down, you still get the European rows, with execution set to partial and the reason in the support/us source entry.

Rules:

  • A row query needs LIMIT (at most 1000). Add ORDER BY on a selected field to merge rows in order.
  • bm25_top_k and hybrid_top_k answers stay grouped per context; they are never merged into one ranking. BM25 scores depend on each database's own statistics, so scores from two contexts cannot be compared.
  • Refused: SUBSCRIBE, OFFSET, UNNEST, continuations, and the rrf and rerank merges.
  • Default budgets: 8 contexts per query, a 5 s deadline (60 s at most), 4 s per member, 8 members in parallel (16 at most).

What is implemented today

Works today
  • The catalog: register, list, inspect, discover by topic, unregister.
  • Context queries over local, remote and snapshot members, from Erlang (barrel_ctx), REST (/contexts/_query) and MCP (context_query).
  • Working sets: attach and detach members, materialize slices from a query, import snapshots, offline mode that skips remote members.
  • Export and read-only import of snapshots.
  • Credentials for remote servers in node configuration (ctx_credentials), never in cards.

What is planned

Planned

None of these exist yet. They are listed in the contexts design under "Not supported yet".

  • Live federation (SUBSCRIBE across contexts).
  • BM25 and hybrid ranking across contexts, which needs shared BM25 statistics; rrf and rerank merges.
  • Enforced working-set ownership: the owner label is stored, not checked. Any global principal can read or delete any working set.
  • A query-only permission: a grant that allows querying a source but not copying from it.
  • A shared or replicated catalog; today each node has its own, filled by hand.
  • Publishing generations and ngram index packages to object storage.
  • Connection reuse and TLS options in the remote client.
  • Continuous subsets, predicate snapshots, pagination, and deduplication across replicas.
  • REST and MCP routes to update a card (barrel_ctx:update/2 is Erlang only), and MCP tools to register and unregister.

Notes

  • barrel_ctx_export:export/3 refuses a database with no vector store ({error, no_vector_store}).
  • An imported snapshot opens read only; it is not a restore.
  • Registering, deleting and working-set routes need the global token on barrel_server; querying is open to capability tokens, and each local member is then checked as a query on that database.
  • The design is in Contexts design.

Written for the site Edit this page View as Markdown

Up and down to moveEnter to open