Barrel
FR GitHub

The record

An application that searches its own content usually keeps three things in three places: the data in a database, the files in object storage, and the vectors in a vector store. Each update then means three writes, and sooner or later a deleted article still shows up in search, or a file points to a row that is gone. Barrel avoids that drift by keeping all three under one id.

A record is everything Barrel keeps under that id: a JSON document, the files attached to it, and in record mode the vector that represents it. They are read, written, replicated and deleted by that id. Read this to know what a document looks like on the way in and on the way out, which fields Barrel adds, and what moves with the record.

id: kb-142 document JSON, every version kept attachments files, stored in chunks vector embedded from its fields
One id: the document on top, attachments on the left, the vector on the right.

Take the support article kb-142, "Reset a locked account". Its fields are the document, the file steps.md is an attachment, and the vector computed from its text is what lets a search by meaning find it. This page follows that one article.

The three parts

Each part of a record behaves a little differently when it changes or when it travels to another copy of the database. The table sums it up; the sections below explain each row.

PartWhat it isVersionedReplicated
DocumentA JSON object with an idyes, every write is a versionyes
AttachmentsNamed binary blobs, by document id and nameno, last write wins on its timestampyes, on their own feed
VectorThe embedding of the fields you choosefollows the documentno, each copy rebuilds it

The document

A document is the part you query. You write a map with an id. Every path in it is indexed, so you can query any field without declaring an index.

Erlang shell
barrel:put_doc(Db, #{<<"id">> => <<"kb-142">>,
  <<"title">> => <<"Reset a locked account">>,
  <<"_note">> => <<"dropped">>}),
barrel:get_doc(Db, <<"kb-142">>).
Output from a run on barrel 1.12.0
{ok,#{<<"_rev">> =>
        <<"000001a1ba45b14e00000000@wddc4e1c028733ad0">>,
    <<"id">> => <<"kb-142">>,
    <<"title">> => <<"Reset a locked account">>}}
What just happened

The read returns your id and title, plus a _rev that Barrel added. The _note field you sent is gone: Barrel dropped it because it starts with _.

  • _rev is the revision: the version token Barrel adds on read. Send it back to update the document; a missing or stale _rev returns {error, conflict}. This stops you from overwriting a change you never read. See Versions, conflicts, convergence.
  • Top-level fields that start with _ are reserved. Barrel drops the ones you send (_note above) and never indexes them.
  • A delete writes a version that marks the document deleted. A read then returns {error, not_found}, and the vector goes too.

Fields Barrel adds on request

Besides _rev, Barrel adds a few fields to a read when you ask for them. You need them when a document has conflicts to resolve, or when you want to see the vector behind a search result.

FieldRead optionWhat it holds
_revalwaysThe winning version token
_conflictsconflicts => trueThe other concurrent versions
_observedconflicts => trueEvery version the read showed, to pass back when you resolve
_embeddinginclude_embedding => truevector and source (computed when Barrel embedded it)

Attachments

An attachment is a named file that belongs to a document id, such as the steps.md file of kb-142. Use attachments for content that is not JSON. Writing one does not change the document's _rev, so updating a file never conflicts with an edit of the document.

Erlang shell
barrel:put_attachment(Db, <<"kb-142">>, <<"steps.md">>,
                    <<"1. Open the console">>),
barrel:list_attachments(Db, <<"kb-142">>).
Output from a run on barrel 1.12.0, digest trimmed
{ok,#{name => <<"steps.md">>,length => 19,
    digest => <<"sha256-dd490eb840644ed69f89e41c5...">>,
    content_type => <<"application/octet-stream">>,
    chunked => false}}
[<<"steps.md">>]
What just happened

The write returns the attachment's metadata: its name, its size in bytes (19), a SHA-256 digest of the content, and a generic content type since none was given. The listing then shows steps.md as the only attachment of kb-142.

See Attachments.

The vector

A vector is a list of numbers that stands for the meaning of a text. It is what lets a search find kb-142 when the query uses other words than the article.

In record mode you give the database an embedding policy: which fields to embed, with which provider, at which dimension. Barrel computes the embedding of those fields on every write and drops the vector on delete. Search results read their text and metadata from the current document, so they never show a stale copy.

In a plain database you manage vectors yourself with vector_add, under the same id or another one. The document and its vector are then two separate writes, and keeping them in step is up to you.

What else a write can carry

A write can carry options beyond the document itself. They say who made the change, how long the document lives, and whether to wait for the disk.

Write optionEffect
provenance => #{actor, session, source}Stored with the version, shown in history
expires_at => UnixMsThe document disappears after that time
sync => trueWaits for the write to reach disk

Here an agent, agent-7, records itself and its session as the author of a write. The version history then shows who changed the article.

Erlang shell
barrel:put_doc(Db, Doc,
  #{provenance => #{actor => <<"agent-7">>,
                    session => <<"ses_42">>}}),
barrel:doc_versions(Db, <<"kb-142">>).
Output from a run on barrel 1.12.0
{ok,[#{status => current,
     version =>
         <<"000001a1ba45b14e00000000@wddc4e1c028733ad0">>,
     deleted => false,
     provenance => #{session => <<"ses_42">>,
                     actor => <<"agent-7">>},
     vv => #{<<"wddc4e1c028733ad0">> =>
                 {timestamp,1794126491982,0}}}]}
What just happened

The history holds one version, marked current and not deleted. Its provenance holds the actor and session you passed. vv is the version vector: for each writer, the latest clock this version had seen.

Provenance is not replicated. See Audit and provenance.

Written for the site Edit this page View as Markdown

Up and down to moveEnter to open