Barrel
FR GitHub

Embed barrel in an Erlang app

barrel is the embeddable edge database. It composes the document layer (barrel_docdb) and the vector layer (barrel_vectordb) behind one API, where a document, its attachments (blobs), and its vector share a single id. Read this when you want a database inside your Erlang release without running a server.

When to use it

  • You want documents, vectors, BM25/hybrid search, attachments, and a changes feed from one handle, in-process.
  • You do not need a network endpoint. For HTTP access, run barrel_server instead.

Open and close

{ok, _} = application:ensure_all_started(barrel_docdb),
{ok, _} = application:ensure_all_started(barrel_vectordb),

{ok, Db} = barrel:open(mydb),
%% pass options per layer:
{ok, Db2} = barrel:open(mydb, #{
    vectordb => #{dimension => 768, bm25_backend => memory}
}),
ok = barrel:close(Db).

A database links its vector store to the process that opened it. Open it from a long-lived process (a gen_server or supervisor), not a transient one.

Open without a vector store

A database that only holds documents does not need the vector store and its RocksDB. Pass vectordb => none:

{ok, Db} = barrel:open(notes, #{vectordb => none}),
{ok, _} = barrel:put_doc(Db, #{<<"id">> => <<"a">>, <<"lang">> => <<"en">>}),
{ok, _Rows, _Meta} = barrel:query(Db, "SELECT * FROM db WHERE lang = 'en'"),
{error, no_vector_store} = barrel:search(Db, <<"anything">>, #{}).
  • Documents, attachments, queries, changes and replication work as usual.
  • Vector calls (vector_*, search*, embed*, embedder_info/1, vector_stats/1) and the vector_top_k, bm25_top_k and hybrid_top_k table functions answer {error, no_vector_store}.
  • Record mode needs a vector store: embedding with vectordb => none answers {error, {invalid_option, vectordb}}.
  • A branch of such a database has no vector store either. A context export refuses it ({error, no_vector_store}); a slice of it holds documents only.
  • barrel_spaces:create_space(#{vectordb => none}) creates a space without one; the space reopens without one on every node.

Open read only

Open with read_only => true to serve a database without writing any file, for example an imported copy that several nodes share. Every write answers {error, read_only}.

{ok, Ro} = barrel:open(mydb, #{read_only => true}),
{error, read_only} = barrel:put_doc(Ro, #{<<"id">> => <<"z">>}),
{ok, _} = barrel:get_doc(Ro, <<"a">>),
ok = barrel:close(Ro).
  • The database must exist: a missing one fails with read_only_store_missing. One written by an older version that needs an upgrade fails with read_only_upgrade_needed; open it writable once.
  • Record mode persists no policy and starts no indexer. Pass embedding => stored to use the policy the database persisted ({error, no_stored_policy} on a plain database).

Share handles through barrel_dbs

barrel_dbs caches open handles by name, closes idle ones and evicts past a cap. Take a lease when a process needs a database to stay open while it works; take a hold when you need its files closed (an export does).

{error, not_found} = barrel_dbs:ensure(<<"nope">>, #{must_exist => true}),
{ok, Db, Lease} = barrel_dbs:lease(<<"mydb">>, #{}),
%% ... Db stays open against idle close and eviction ...
ok = barrel_dbs:release(Lease),

{ok, #{was_open := _}} = barrel_dbs:hold(<<"mydb">>, #{}),
%% ... every ensure of mydb is refused until unhold ...
ok = barrel_dbs:unhold(<<"mydb">>).
  • Leases are counted and end when the holder exits; barrel_dbs:leases/0 lists them. lookup/1 returns the pinned flag, owner and open options.
  • A hold is refused on a pinned or leased database, or one opened outside the manager.

Documents

{ok, _}   = barrel:put_doc(Db, #{<<"id">> => <<"a">>, <<"title">> => <<"hello">>}),
{ok, Doc} = barrel:get_doc(Db, <<"a">>),
{ok, _}   = barrel:delete_doc(Db, <<"a">>),
{ok, Rows, _Meta} = barrel:find(Db, #{where => [{path, [<<"title">>], <<"hello">>}]}).

Batches

[{ok, _}, {ok, _}] = barrel:put_docs(Db, [#{<<"id">> => <<"a">>}, #{<<"id">> => <<"b">>}]),
Results = barrel:get_docs(Db, [<<"a">>, <<"b">>]),     %% one result per id, in order
_       = barrel:delete_docs(Db, [<<"a">>, <<"b">>]).

Attachments (blobs)

Blobs are document attachments; the storage backend is pluggable per database via the docdb barrel_att_backend seam (RocksDB BlobDB by default; barrel_att_s3 stores them in an S3-compatible bucket).

{ok, _}          = barrel:put_attachment(Db, <<"a">>, <<"f.txt">>, <<"bytes">>),
{ok, <<"bytes">>} = barrel:get_attachment(Db, <<"a">>, <<"f.txt">>),
[<<"f.txt">>]    = barrel:list_attachments(Db, <<"a">>),
ok               = barrel:delete_attachment(Db, <<"a">>, <<"f.txt">>).

Stream large blobs:

{ok, W0} = barrel:open_attachment_writer(Db, <<"a">>, <<"big">>, <<"application/octet-stream">>),
{ok, W1} = barrel:write_attachment(W0, <<"chunk">>),
{ok, _}  = barrel:finish_attachment(W1),

{ok, R0} = barrel:open_attachment_reader(Db, <<"a">>, <<"big">>),
{ok, _Chunk, R1} = barrel:read_attachment(R0),    %% eof at the end
ok = barrel:close_attachment_reader(R1).

To keep vectors in sync with documents automatically, open the database with an embedding policy instead of managing them by hand: see record mode. The direct vector API below applies to plain databases.

ok = barrel:vector_add(Db, <<"a">>, <<"hello world">>, #{}, Vector),
{ok, #{inserted := 2}} = barrel:vector_add_batch(Db, [
    {<<"a">>, <<"t1">>, #{}, V1},
    {<<"b">>, <<"t2">>, #{}, V2}
]),
{ok, Hits}  = barrel:search_vector(Db, Vector, #{k => 5}),
{ok, BHits} = barrel:search_bm25(Db, <<"hello">>, #{k => 5}).

Notes:

  • vector_add_batch/2 takes {Id, Text, Metadata} (text embedded by the store) or {Id, Text, Metadata, Vector} (explicit) tuples; a batch must be all one shape.
  • BM25 is opt-in: open with vectordb => #{bm25_backend => memory} (or disk). A disk index survives a reopen; a memory index is rebuilt from the stored text at open.
  • search_hybrid/3 and auto-embedding adds need an embedder configured via barrel_embed; without one they return {error, embedder_not_configured}.
  • To embed text yourself with the database's own embedder (same model and dimension as the indexed vectors), use barrel:embed/2, barrel:embed_batch/2 and barrel:embedder_info/1 on the handle; keep the handle opaque rather than reading its fields.

Changes

{ok, Changes, Last} = barrel:changes(Db, first),
Cursor  = barrel:hlc_encode(Last),         %% JSON/URL-safe cursor
Last2   = barrel:hlc_decode(Cursor),
{ok, More, _} = barrel:changes(Db, Last2),  %% changes since the cursor
{ok, Pid} = barrel:subscribe(Db, Last).

From docs/guides/embedding.md in the barrel repository at 794415d, updated 2026-10-02 Edit this page View as Markdown

Up and down to moveEnter to open