barrel_vectordb
An embedded vector store. Vectors, their text and metadata live in RocksDB, with an approximate nearest neighbour index, optional BM25 and hybrid fusion.
What it solves
Retrieval for agents needs nearest-neighbour search, keyword search for exact terms, and a way to combine both, without running a separate search service. barrel_vectordb gives you HNSW (pure Erlang, default), DiskANN or FAISS, BM25 in memory or on disk, RRF or linear fusion, quantization, encryption and a persisted index.
When to use it
- You need vector or keyword retrieval without documents around it, for example an agent memory.
- You bring your own vectors, or want text embedded through a barrel_embed provider.
- You want hybrid search with tunable weights.
When not to
- Your vectors belong to documents: use barrel, which keeps the two under one id.
- You need exact substring or regex matches: use barrel_ngram.
How it fits into Barrel
- Builds on
barrel_crypto,barrel_embed- Works with, when present
barrel_faiss:backend => faiss;barrel_rerank: reranks results- Used by
barrel- Other dependencies
- rocksdb, hackney, iommap; a C NIF built with CMake; OTP 26 or later
- On its own
- Yes. barrel_embed is always pulled in, even when you supply every vector.
Install
{barrel_vectordb, "~> 2.5"} Configuration
Store config
| Key | Default | What it does |
|---|---|---|
name | required | Store name used by every call |
path / db_path | "priv/barrel_vectordb_<name>" | Relative to the current directory. Set it. |
dimensions | 768 | Vector size; mismatches return {dimension_mismatch, _, _} |
embedder | none | Without one, text calls return embedder_not_configured |
backend | hnsw | hnsw, diskann or faiss |
hnsw.m, hnsw.ef_construction | 16, 200 | Graph degree and build effort |
hnsw.quantization | scalar | turboquant, subspace_turboquant or none |
bm25_backend | none | memory or disk to enable keyword search |
read_only | false | Writes return {error, read_only} |
Search options
| Key | Default | What it does |
|---|---|---|
k | 5 vector, 10 BM25 and hybrid | Number of results |
filter | none | fun(Metadata) -> boolean(), applied after retrieval |
fusion | rrf | linear uses bm25_weight and vector_weight (0.5 each) |
ef_search | max(k, 50) | Search effort |
The most used keys only. For every option, see the configuration reference and the API reference.
The API by task
Lifecycle
| Task | Call | Returns |
|---|---|---|
| Start a store | barrel_vectordb:start_link/1, start_supervised/1 | {ok, Pid} |
| Stop, remove | barrel_vectordb:stop/1, destroy/1 | ok |
| Persist the index | barrel_vectordb:persist_index/1, checkpoint/1 | ok |
Write and read
| Task | Call | Returns |
|---|---|---|
| Add text (embedded) | barrel_vectordb:add/4,5, add_batch/2 | ok |
| Add with a vector | barrel_vectordb:add_vector/5, add_vector_batch/2 | ok, {ok, #{inserted}} |
| Replace text | barrel_vectordb:update/4, upsert/4 | ok; re-embeds the text |
| Delete | barrel_vectordb:delete/2 | ok |
| Read | barrel_vectordb:get/2, peek/2, count/1, stats/1 | {ok, #{key, text, metadata, vector}} or not_found |
Search
| Task | Call | Returns |
|---|---|---|
| By text | barrel_vectordb:search/3 | {ok, [#{key, score, text, metadata}]} |
| By vector | barrel_vectordb:search_vector/3 | same shape |
| Keyword | barrel_vectordb:search_bm25/3 | {ok, [{Id, Score}]} |
| Hybrid | barrel_vectordb:search_hybrid/3 | {ok, [Hit]} |
A small agent memory with your own vectors
From a verified run. No embedder is configured, so text search reports it.
{ok, _} = application:ensure_all_started(barrel_vectordb).
Cfg = #{name => <<"agent_mem">>, path => Path,
dimensions => 4, bm25_backend => disk,
hnsw => #{m => 8, ef_construction => 100}}.
{ok, _} = barrel_vectordb:start_link(Cfg).
ok = barrel_vectordb:add_vector(<<"agent_mem">>, <<"m1">>,
<<"User prefers metric units in reports">>,
#{kind => preference}, [0.9, 0.1, 0.0, 0.0]).
barrel_vectordb:add_vector_batch(<<"agent_mem">>, [
{<<"m2">>, <<"Quarterly report is due on the 15th">>,
#{kind => fact}, [0.7, 0.3, 0.1, 0.0]},
{<<"m3">>, <<"Customer Acme renewed for two years">>,
#{kind => fact}, [0.0, 0.1, 0.9, 0.2]}]).
barrel_vectordb:search_vector(<<"agent_mem">>,
[0.8, 0.2, 0.0, 0.0],
#{k => 2,
filter => fun(M) -> maps:get(kind, M) =:= fact end}).
barrel_vectordb:search_bm25(<<"agent_mem">>, <<"report">>,
#{k => 5}).
barrel_vectordb:search(<<"agent_mem">>, <<"report">>, #{}). {ok,#{inserted => 2}}
{ok,[#{metadata => #{kind => fact},
text => <<"Quarterly report is due on the 15th">>,
key => <<"m2">>,score => 0.9788330406946226}]}
{ok,[{<<"m2">>,0.9403363021993156}]}
{error,embedder_not_configured} Limits to know
- The metadata filter runs after retrieval, so heavy filtering can return fewer than
kresults (ak => 2search returned 1). - BM25 is off unless
bm25_backendis set. A memory BM25 index rebuilds from the stored text at every open. - There is no update with an explicit vector:
update/4andupsert/4re-embed the text. If embedding fails, the old entry stays and the call returns the error (before 2.5.3 the entry was lost). - A read-only open of a store that needs an upgrade fails with
read_only_upgrade_needed. - The app env keys in
barrel_vectordb.app.src(default_dimensions,hnsw_m...) are never read; defaults come from code.
A store started with barrel_vectordb:start_link/1 and no path lands in ./priv/barrel_vectordb_<name> under the current directory. Through barrel, it goes next to the documents instead.