Barrel
FR GitHub

barrel_vectordb

An embedded vector store. Vectors, their text and metadata live in RocksDB, with an approximate nearest neighbour index, optional BM25 and hybrid fusion.

What it solves

Retrieval for agents needs nearest-neighbour search, keyword search for exact terms, and a way to combine both, without running a separate search service. barrel_vectordb gives you HNSW (pure Erlang, default), DiskANN or FAISS, BM25 in memory or on disk, RRF or linear fusion, quantization, encryption and a persisted index.

When to use it

  • You need vector or keyword retrieval without documents around it, for example an agent memory.
  • You bring your own vectors, or want text embedded through a barrel_embed provider.
  • You want hybrid search with tunable weights.

When not to

  • Your vectors belong to documents: use barrel, which keeps the two under one id.
  • You need exact substring or regex matches: use barrel_ngram.

How it fits into Barrel

Works with, when present
barrel_faiss: backend => faiss; barrel_rerank: reranks results
Used by
barrel
Other dependencies
rocksdb, hackney, iommap; a C NIF built with CMake; OTP 26 or later
On its own
Yes. barrel_embed is always pulled in, even when you supply every vector.

Install

rebar.config deps
{barrel_vectordb, "~> 2.5"}

Configuration

Store config

KeyDefaultWhat it does
name required Store name used by every call
path / db_path "priv/barrel_vectordb_<name>" Relative to the current directory. Set it.
dimensions 768 Vector size; mismatches return {dimension_mismatch, _, _}
embedder none Without one, text calls return embedder_not_configured
backend hnsw hnsw, diskann or faiss
hnsw.m, hnsw.ef_construction 16, 200 Graph degree and build effort
hnsw.quantization scalar turboquant, subspace_turboquant or none
bm25_backend none memory or disk to enable keyword search
read_only false Writes return {error, read_only}

Search options

KeyDefaultWhat it does
k 5 vector, 10 BM25 and hybrid Number of results
filter none fun(Metadata) -> boolean(), applied after retrieval
fusion rrf linear uses bm25_weight and vector_weight (0.5 each)
ef_search max(k, 50) Search effort

The most used keys only. For every option, see the configuration reference and the API reference.

The API by task

Lifecycle

TaskCallReturns
Start a store barrel_vectordb:start_link/1, start_supervised/1 {ok, Pid}
Stop, remove barrel_vectordb:stop/1, destroy/1 ok
Persist the index barrel_vectordb:persist_index/1, checkpoint/1 ok

Write and read

TaskCallReturns
Add text (embedded) barrel_vectordb:add/4,5, add_batch/2 ok
Add with a vector barrel_vectordb:add_vector/5, add_vector_batch/2 ok, {ok, #{inserted}}
Replace text barrel_vectordb:update/4, upsert/4 ok; re-embeds the text
Delete barrel_vectordb:delete/2 ok
Read barrel_vectordb:get/2, peek/2, count/1, stats/1 {ok, #{key, text, metadata, vector}} or not_found

Search

TaskCallReturns
By text barrel_vectordb:search/3 {ok, [#{key, score, text, metadata}]}
By vector barrel_vectordb:search_vector/3 same shape
Keyword barrel_vectordb:search_bm25/3 {ok, [{Id, Score}]}
Hybrid barrel_vectordb:search_hybrid/3 {ok, [Hit]}

A small agent memory with your own vectors

From a verified run. No embedder is configured, so text search reports it.

Erlang shell
{ok, _} = application:ensure_all_started(barrel_vectordb).
Cfg = #{name => <<"agent_mem">>, path => Path,
        dimensions => 4, bm25_backend => disk,
        hnsw => #{m => 8, ef_construction => 100}}.
{ok, _} = barrel_vectordb:start_link(Cfg).
ok = barrel_vectordb:add_vector(<<"agent_mem">>, <<"m1">>,
    <<"User prefers metric units in reports">>,
    #{kind => preference}, [0.9, 0.1, 0.0, 0.0]).
barrel_vectordb:add_vector_batch(<<"agent_mem">>, [
  {<<"m2">>, <<"Quarterly report is due on the 15th">>,
   #{kind => fact}, [0.7, 0.3, 0.1, 0.0]},
  {<<"m3">>, <<"Customer Acme renewed for two years">>,
   #{kind => fact}, [0.0, 0.1, 0.9, 0.2]}]).
barrel_vectordb:search_vector(<<"agent_mem">>,
    [0.8, 0.2, 0.0, 0.0],
    #{k => 2,
      filter => fun(M) -> maps:get(kind, M) =:= fact end}).
barrel_vectordb:search_bm25(<<"agent_mem">>, <<"report">>,
                            #{k => 5}).
barrel_vectordb:search(<<"agent_mem">>, <<"report">>, #{}).
Output, from a run on barrel_vectordb 2.5.1
{ok,#{inserted => 2}}

{ok,[#{metadata => #{kind => fact},
       text => <<"Quarterly report is due on the 15th">>,
       key => <<"m2">>,score => 0.9788330406946226}]}

{ok,[{<<"m2">>,0.9403363021993156}]}

{error,embedder_not_configured}

Limits to know

  • The metadata filter runs after retrieval, so heavy filtering can return fewer than k results (a k => 2 search returned 1).
  • BM25 is off unless bm25_backend is set. A memory BM25 index rebuilds from the stored text at every open.
  • There is no update with an explicit vector: update/4 and upsert/4 re-embed the text. If embedding fails, the old entry stays and the call returns the error (before 2.5.3 the entry was lost).
  • A read-only open of a store that needs an upgrade fails with read_only_upgrade_needed.
  • The app env keys in barrel_vectordb.app.src (default_dimensions, hnsw_m...) are never read; defaults come from code.
Known gap

A store started with barrel_vectordb:start_link/1 and no path lands in ./priv/barrel_vectordb_<name> under the current directory. Through barrel, it goes next to the documents instead.

Up and down to moveEnter to open