Barrel
FR GitHub

Vector, keyword and hybrid

Barrel searches a database three ways: by vector similarity (meaning), by BM25 (keywords), and hybrid, which runs both and fuses the two rankings. The same searches run as Erlang calls, as BQL table functions joined to your documents, over REST and as an MCP tool. Use this page when you set up search over your documents and need to pick a mode, filter results, or read the scores.

Set up a searchable database

Record mode is the simple path: Barrel embeds the fields you name on every write, and turns on a disk BM25 index.

{ok, Db} = barrel:open(<<"runbooks">>, #{
    embedding => #{
        fields => [<<"title">>, <<"body">>],
        mode => sync,
        embedder => {ollama,
                     #{model => <<"nomic-embed-text">>}},
        dimensions => 768,
        metadata_fields => [<<"team">>]},
    vectordb => #{db_path => "data/vectors/runbooks"}}),
barrel:put_docs(Db, [
  #{<<"id">> => <<"rb:pg">>, <<"team">> => <<"data">>,
    <<"title">> => <<"Postgres failover">>,
    <<"body">> => <<"Promote the replica when the primary "
                    "stops answering.">>},
  #{<<"id">> => <<"rb:redis">>, <<"team">> => <<"data">>,
    <<"title">> => <<"Redis eviction">>,
    <<"body">> => <<"Set maxmemory-policy to allkeys-lru "
                    "when memory is full.">>},
  #{<<"id">> => <<"rb:tls">>, <<"team">> => <<"edge">>,
    <<"title">> => <<"Renew TLS certificates">>,
    <<"body">> => <<"Run the renewal job before the "
                    "certificate expires.">>}]).

metadata_fields picks what search results carry as metadata. In a plain database, add vectors yourself with vector_add and set bm25_backend => memory | disk in the vectordb options; BM25 is off otherwise.

Search by meaning

barrel:search(Db, <<"the database is down">>,
              #{k => 2, include_text => false}).
Output from a run on barrel 1.12.0 with Ollama nomic-embed-text
{ok,[#{metadata => #{<<"team">> => <<"data">>},
     key => <<"rb:pg">>,score => 0.5337445455915225},
   #{metadata => #{<<"team">> => <<"data">>},
     key => <<"rb:redis">>,score => 0.4549723605217504}]}

The score is cosine similarity. Pass a vector you computed with search_vector/3 instead of text.

Search by keyword

barrel:search_bm25(Db, <<"replica">>, #{k => 2}).
Output from a run on barrel 1.12.0
{ok,[{<<"rb:pg">>,1.0187153614971578}]}

BM25 returns {Id, Score} pairs, only for documents that contain the terms. The score is the raw BM25 value, not bounded.

Hybrid

barrel:search_hybrid(Db, <<"cache is out of memory">>,
                     #{k => 2, include_text => false}).
Output from a run on barrel 1.12.0 with Ollama nomic-embed-text
{ok,[#{metadata => #{<<"team">> => <<"data">>},
     key => <<"rb:redis">>,score => 0.01639344262295082},
   #{metadata => #{<<"team">> => <<"data">>},
     key => <<"rb:pg">>,score => 0.008536214242239805}]}

The default fusion is reciprocal rank fusion (rrf_k 60): the score comes from the ranks, so 1/61 means first in one list. For a weighted sum of the two scores, pass fusion => linear with bm25_weight and vector_weight (0.5 each by default).

Search from BQL

Table functions run the search and join each hit to its document, so WHERE and the projection see every field:

barrel:query(Db,
  <<"SELECT h.id, h._score, title "
    "FROM hybrid_top_k($q, k => 3) AS h "
    "WHERE h.team = 'data'">>,
  #{params => #{<<"q">> => <<"primary database failed">>}}).
Output from a run on barrel 1.12.0 with Ollama nomic-embed-text
{ok,[#{<<"_score">> => 0.01639344262295082,
     <<"id">> => <<"rb:pg">>,
     <<"title">> => <<"Postgres failover">>},
   #{<<"_score">> => 0.008536214242239805,
     <<"id">> => <<"rb:redis">>,
     <<"title">> => <<"Redis eviction">>}],
  #{count => 2, ...}}
Function_score
vector_top_k(text, k => N)1 minus the distance; _distance is also selectable
bm25_top_k(text, k => N)the BM25 score
hybrid_top_k(text, k => N)the fused score

k defaults to 10. A table function fetches 3 times k candidates (1000 at most), applies WHERE, and returns up to k rows. Table functions need the barrel API: barrel_docdb:query/2 refuses them.

Over REST and MCP

barrel:search_hybrid(Db, <<"replica lag">>, #{k => 5}).

The REST routes are /search/vector (with "vector"), /search/bm25 and /search/hybrid; they answer {"hits":[...]}. A database the server opens with default options has a 768-dimension vector store, no embedder and no BM25: /search/bm25 answers 500 bm25_not_enabled and /search/hybrid 500 embedder_not_configured until you set open_opts or an embedding policy.

Filters

filter => fun(Metadata) -> boolean() end keeps only the hits whose metadata passes. It runs on the hits the index returns, so it can return fewer than k results, or none:

barrel:search(Db, <<"the database is down">>,
  #{k => 2, include_text => false,
    filter => fun(M) ->
                  maps:get(<<"team">>, M, undefined)
                      =:= <<"edge">>
              end}).
Output from a run on barrel 1.12.0 with Ollama nomic-embed-text
{ok,[]}

rb:tls is the only edge document, and it is not in the top two. Filter in BQL WHERE instead, which fetches 3 times k first, or raise k.

Limit

Default k differs by entry point: 5 for search/3 and search_vector/3, 10 for search_bm25/3, search_hybrid/3 and BQL table functions. Hybrid and BM25 results are never merged across contexts.

Notes

  • A search needs the embedder to be reachable: text queries embed the query first.
  • Vectors are not replicated. Each copy in record mode embeds its own documents.
  • For exact strings, identifiers and error codes, see Substring and regex. To reorder results with a cross-encoder, see Reranking.

Written for the site Edit this page View as Markdown

Up and down to moveEnter to open