Embed barrel in an Erlang app
barrel is the embeddable edge database. It composes the document layer
(barrel_docdb) and the vector layer (barrel_vectordb) behind one API, where a
document, its attachments (blobs), and its vector share a single id. Read this
when you want a database inside your Erlang release without running a server.
When to use it
- You want documents, vectors, BM25/hybrid search, attachments, and a changes feed from one handle, in-process.
- You do not need a network endpoint. For HTTP access, run
barrel_serverinstead.
Open and close
{ok, _} = application:ensure_all_started(barrel_docdb),
{ok, _} = application:ensure_all_started(barrel_vectordb),
{ok, Db} = barrel:open(mydb),
%% pass options per layer:
{ok, Db2} = barrel:open(mydb, #{
vectordb => #{dimension => 768, bm25_backend => memory}
}),
ok = barrel:close(Db).
A database links its vector store to the process that opened it. Open it from a long-lived process (a gen_server or supervisor), not a transient one.
Open without a vector store
A database that only holds documents does not need the vector store and its
RocksDB. Pass vectordb => none:
{ok, Db} = barrel:open(notes, #{vectordb => none}),
{ok, _} = barrel:put_doc(Db, #{<<"id">> => <<"a">>, <<"lang">> => <<"en">>}),
{ok, _Rows, _Meta} = barrel:query(Db, "SELECT * FROM db WHERE lang = 'en'"),
{error, no_vector_store} = barrel:search(Db, <<"anything">>, #{}).
- Documents, attachments, queries, changes and replication work as usual.
- Vector calls (
vector_*,search*,embed*,embedder_info/1,vector_stats/1) and thevector_top_k,bm25_top_kandhybrid_top_ktable functions answer{error, no_vector_store}. - Record mode needs a vector store:
embeddingwithvectordb => noneanswers{error, {invalid_option, vectordb}}. - A branch of such a database has no vector store either. A context export
refuses it (
{error, no_vector_store}); a slice of it holds documents only. barrel_spaces:create_space(#{vectordb => none})creates a space without one; the space reopens without one on every node.
Open read only
Open with read_only => true to serve a database without writing any file,
for example an imported copy that several nodes share. Every write answers
{error, read_only}.
{ok, Ro} = barrel:open(mydb, #{read_only => true}),
{error, read_only} = barrel:put_doc(Ro, #{<<"id">> => <<"z">>}),
{ok, _} = barrel:get_doc(Ro, <<"a">>),
ok = barrel:close(Ro).
- The database must exist: a missing one fails with
read_only_store_missing. One written by an older version that needs an upgrade fails withread_only_upgrade_needed; open it writable once. - Record mode persists no policy and starts no indexer. Pass
embedding => storedto use the policy the database persisted ({error, no_stored_policy}on a plain database).
Share handles through barrel_dbs
barrel_dbs caches open handles by name, closes idle ones and evicts past a
cap. Take a lease when a process needs a database to stay open while it
works; take a hold when you need its files closed (an export does).
{error, not_found} = barrel_dbs:ensure(<<"nope">>, #{must_exist => true}),
{ok, Db, Lease} = barrel_dbs:lease(<<"mydb">>, #{}),
%% ... Db stays open against idle close and eviction ...
ok = barrel_dbs:release(Lease),
{ok, #{was_open := _}} = barrel_dbs:hold(<<"mydb">>, #{}),
%% ... every ensure of mydb is refused until unhold ...
ok = barrel_dbs:unhold(<<"mydb">>).
- Leases are counted and end when the holder exits;
barrel_dbs:leases/0lists them.lookup/1returns the pinned flag, owner and open options. - A hold is refused on a pinned or leased database, or one opened outside the manager.
Documents
{ok, _} = barrel:put_doc(Db, #{<<"id">> => <<"a">>, <<"title">> => <<"hello">>}),
{ok, Doc} = barrel:get_doc(Db, <<"a">>),
{ok, _} = barrel:delete_doc(Db, <<"a">>),
{ok, Rows, _Meta} = barrel:find(Db, #{where => [{path, [<<"title">>], <<"hello">>}]}).
Batches
[{ok, _}, {ok, _}] = barrel:put_docs(Db, [#{<<"id">> => <<"a">>}, #{<<"id">> => <<"b">>}]),
Results = barrel:get_docs(Db, [<<"a">>, <<"b">>]), %% one result per id, in order
_ = barrel:delete_docs(Db, [<<"a">>, <<"b">>]).
Attachments (blobs)
Blobs are document attachments; the storage backend is pluggable per database via
the docdb barrel_att_backend seam (RocksDB BlobDB by default; barrel_att_s3
stores them in an S3-compatible bucket).
{ok, _} = barrel:put_attachment(Db, <<"a">>, <<"f.txt">>, <<"bytes">>),
{ok, <<"bytes">>} = barrel:get_attachment(Db, <<"a">>, <<"f.txt">>),
[<<"f.txt">>] = barrel:list_attachments(Db, <<"a">>),
ok = barrel:delete_attachment(Db, <<"a">>, <<"f.txt">>).
Stream large blobs:
{ok, W0} = barrel:open_attachment_writer(Db, <<"a">>, <<"big">>, <<"application/octet-stream">>),
{ok, W1} = barrel:write_attachment(W0, <<"chunk">>),
{ok, _} = barrel:finish_attachment(W1),
{ok, R0} = barrel:open_attachment_reader(Db, <<"a">>, <<"big">>),
{ok, _Chunk, R1} = barrel:read_attachment(R0), %% eof at the end
ok = barrel:close_attachment_reader(R1).
Vectors and search
To keep vectors in sync with documents automatically, open the database with an embedding policy instead of managing them by hand: see record mode. The direct vector API below applies to plain databases.
ok = barrel:vector_add(Db, <<"a">>, <<"hello world">>, #{}, Vector),
{ok, #{inserted := 2}} = barrel:vector_add_batch(Db, [
{<<"a">>, <<"t1">>, #{}, V1},
{<<"b">>, <<"t2">>, #{}, V2}
]),
{ok, Hits} = barrel:search_vector(Db, Vector, #{k => 5}),
{ok, BHits} = barrel:search_bm25(Db, <<"hello">>, #{k => 5}).
Notes:
vector_add_batch/2takes{Id, Text, Metadata}(text embedded by the store) or{Id, Text, Metadata, Vector}(explicit) tuples; a batch must be all one shape.- BM25 is opt-in: open with
vectordb => #{bm25_backend => memory}(ordisk). A disk index survives a reopen; a memory index is rebuilt from the stored text at open. search_hybrid/3and auto-embedding adds need an embedder configured viabarrel_embed; without one they return{error, embedder_not_configured}.- To embed text yourself with the database's own embedder (same model and
dimension as the indexed vectors), use
barrel:embed/2,barrel:embed_batch/2andbarrel:embedder_info/1on the handle; keep the handle opaque rather than reading its fields.
Changes
{ok, Changes, Last} = barrel:changes(Db, first),
Cursor = barrel:hlc_encode(Last), %% JSON/URL-safe cursor
Last2 = barrel:hlc_decode(Cursor),
{ok, More, _} = barrel:changes(Db, Last2), %% changes since the cursor
{ok, Pid} = barrel:subscribe(Db, Last).