Find it by meaning, by word, or by the exact string.
Barrel keeps a document and its vector under one id, so one query can rank by meaning, by keyword, or both, and come back as rows with the document's fields. For identifiers and error strings that meaning misses, a trigram index finds the exact text.
Search runs where the database runs: in your Erlang node, behind the REST and MCP server, or over vectors synced into a browser.
Six ways to find a record
Each one is a separate index or step you turn on. The mark says whether it works with no setup.
-
Vector
Nearest neighbours by meaning. HNSW by default, DiskANN, or FAISS through
barrel_faiss. Text queries need an embedder; you can also pass the query vector yourself. -
Keyword (BM25)
Ranked keyword search. Off on a plain database until you set
bm25_backendtomemoryordisk; record mode turns it on withdisk. -
Hybrid
Vector and BM25 fused into one ranking: reciprocal rank fusion by default (
rrf_k60), or a linear mix withbm25_weightandvector_weight. Needs BM25 enabled. -
Substring and regex
Exact strings and PCRE patterns over documents with
barrel_ngram, a trigram index fed by the changes feed. Every hit is confirmed against the text and carries its byte spans. -
Reranking
A second pass with a cross-encoder in
barrel_rerank: you hand it the query and the texts of your top hits, it returns them reordered with a score each. -
BQL table functions
vector_top_k,bm25_top_kandhybrid_top_kas the FROM of a query: hits join their documents, then WHERE, ORDER BY and projections apply.
How a hybrid query runs
Both indexes answer the same text; their ranks are fused, and a BQL table function joins each hit to its document before WHERE applies. Table functions fetch 3 times k (at most 1000) so a WHERE still has rows to keep.
- Query text or a query vector
embed with the store’s provider
- Vector index HNSW, DiskANN or FAISSBM25 index memory or disk
fuse ranks (RRF, k 60)
- Hybrid hits id, score, text, metadata
BQL: join to documents, then WHERE
- Rows fields, _score
How substring and regex search runs
- barrel_docdb documents
changes feed
- Trigram segments immutable, checksummed
candidates from postings
- Confirm on the text drops false positives
hits
- id and spans byte offset, length
What holds, and what tests it
Properties you can rely on, with the test that checks each one in the repository.
Exact means exact
Substring and regex search confirm every candidate against the stored text, so trigram collisions never show up as hits.
Checked by oracle_equivalence in barrel_ngram_SUITE, which compares every answer with a brute-force scan.
The vector follows the document
In record mode an edit re-embeds, a delete removes the vector, and the marker for the indexer is written in the same batch as the document.
Checked by barrel_record_SUITE: indexer_update_reembeds, indexer_delete_removes_vector, indexer_crash_restart.
Read your write, when you ask for it
Record mode with mode => sync embeds before the write returns. If the embedder fails, the write fails and nothing is stored.
Checked by barrel_record_SUITE: sync_put_immediate_search, sync_embed_failure_fails_put.
A disk BM25 index survives a kill
Acknowledged writes are still in the index after the process is killed and the store reopened. A memory index rebuilds from the stored text at open.
Checked by barrel_vectordb_bm25_persistence_tests (disk: kill then reopen loses no acked write).
Replicated documents get indexed
Documents that arrive by replication are embedded on the receiving copy; the version that lost a conflict is not.
Checked by barrel_record_SUITE: replicated_docs_embedded, replicated_loser_not_reembedded.
Same model, or no open
Each store records its embedder’s fingerprint: provider, model, dimensions, distance. A record-mode space opened with another model is refused.
Checked by embedder_fingerprint in barrel_record_SUITE.
Use it
Open a database with a vector store and BM25
The recorded run below used kb_embed, a 64-dimension feature-hashing provider written for the test: it matches shared words, it is not semantic.
{ok, Db} = barrel:open(<<"research">>, #{vectordb => #{
db_path => "data/research_vec",
dimensions => 64,
embedder => {kb_embed, #{}},
bm25_backend => memory}}),
{ok, #{inserted := 4}} = barrel:vector_add_batch(Db, Notes). Vector, keyword and hybrid
barrel:search(Db, <<"leader election">>, #{k => 2}),
barrel:search_bm25(Db, <<"keyword search">>, #{k => 2}),
barrel:search_hybrid(Db, <<"nearest neighbor search">>,
#{k => 2}). curl -X POST localhost:8080/db/research/search/bm25 \
-H 'content-type: application/json' \
-d '{"query":"keyword search","k":2}'
curl -X POST localhost:8080/db/research/search/hybrid \
-H 'content-type: application/json' \
-d '{"query":"nearest neighbor search","k":2}' {"jsonrpc":"2.0","id":3,"method":"tools/call",
"params":{"name":"search","arguments":{
"db":"research","mode":"hybrid",
"query":"nearest neighbor search","k":2}}} {ok,[{<<"paper:bm25">>,1.9587592623731485},
{<<"paper:hnsw">>,0.6586049622596158}]} Search inside a query
_score is 1 minus the distance for vectors, the raw BM25 score, or the fused value for hybrid. It can be selected and ordered, not filtered.
barrel:query(Db,
<<"SELECT h.id, h._score, topic "
"FROM hybrid_top_k($q, k => 3) AS h "
"ORDER BY h._score DESC">>,
#{params => #{<<"q">> => <<"replicated consensus">>}}). curl -X POST localhost:8080/db/research/query \
-H 'content-type: text/plain' \
--data-binary "SELECT b.id, b._score
FROM bm25_top_k('search', k => 3) AS b
WHERE b.year > 2000" {"jsonrpc":"2.0","id":4,"method":"tools/call",
"params":{"name":"query","arguments":{
"db":"research",
"query":"SELECT v.id FROM vector_top_k($q) AS v",
"params":{"q":"consensus"}}}} {ok,[#{<<"_score">> => 0.01639344262295082,
<<"id">> => <<"paper:raft">>,
<<"topic">> => <<"distributed-systems">>},
#{<<"_score">> => 0.016001024065540194,
<<"id">> => <<"paper:crdt">>, ...},
#{<<"_score">> => 0.008536214242239805,
<<"id">> => <<"paper:bm25">>, ...}], ...} Record mode: the policy embeds for you
Name the fields to embed. With mode => sync the write returns after the vector is indexed; the default, async, indexes in the background and heals after a crash.
Embedder = {ollama, #{model => <<"nomic-embed-text">>}},
Policy = #{fields => [<<"title">>, <<"body">>],
mode => sync, embedder => Embedder,
dimensions => 768},
{ok, Db} = barrel:open(<<"notes">>, #{embedding => Policy,
vectordb => #{db_path => "data/notes_vec"}}),
{ok, _} = barrel:put_doc(Db, #{<<"id">> => <<"n1">>,
<<"title">> => <<"Raft">>,
<<"body">> => <<"Leader election and log replication">>}),
barrel:search(Db, <<"how do replicas agree?">>, #{k => 1}). Exact substrings and regular expressions
The MCP tool opens the index for the database on first use and refreshes it before each search. There is no REST route.
ok = barrel_ngram:open(<<"rb_idx">>, #{db => <<"runbooks">>,
fields => [<<"title">>, <<"body">>]}),
barrel_ngram:search(<<"rb_idx">>, <<"ERR_CONN_TIMEOUT">>),
barrel_ngram:search(<<"rb_idx">>, <<"redis">>,
#{case_sensitive => false}),
barrel_ngram:regex(<<"rb_idx">>,
<<"pg-(primary|replica)-\\d+">>). {"jsonrpc":"2.0","id":5,"method":"tools/call",
"params":{"name":"ngram_search","arguments":{
"db":"runbooks","mode":"regex",
"query":"pg-(primary|replica)-\\d+"}}} {ok,[#{id => <<"rb:db">>,spans => [{11,16}]}]}
{ok,[#{id => <<"rb:cache">>,spans => [{68,5}]}]}
{ok,[#{id => <<"rb:db">>,spans => [{35,13},{58,13}]}]} Rerank the top hits optional
Ranker is a barrel_rerank:start_link(#) server (default model cross-encoder/ms-marco-MiniLM-L-6-v2).
{ok, Hits} = barrel:search(Db, Q, #{k => 20}),
Texts = [maps:get(text, H) || H <- Hits],
{ok, R} = barrel_rerank:rerank(Ranker, Q, Texts,
#{top_k => 5}),
%% R is [{Index, Score}], best first; Index is 0-based
Top = [lists:nth(I + 1, Hits) || {I, _Score} <- R]. {ok,[{0,9.581769943237305},
{2,-5.255744457244873},
{1,-9.625640869140625}]} Defaults that differ by entry point
| Entry point | Results when you pass nothing |
|---|---|
barrel:search/3, search_vector/3 | k 5 |
barrel:search_bm25/3, search_hybrid/3 | k 10 |
BQL vector_top_k, bm25_top_k, hybrid_top_k | k 10 |
REST /db/:db/search/* and the MCP search tool | k 10 (REST caps it at 10000) |
MCP ngram_search | limit 50 (cap 1000) |
Limits
- BM25 is off on a plain database, including the databases a default barrel_server opens (a 768-dimension cosine store, BM25 off). BM25 and hybrid then answer
bm25_not_enabledorembedder_not_configured, over REST as a 500. - There is no built-in offline embedder. Bring your own vectors, run Ollama locally, use the local Python provider, or write a provider module.
- The metadata filter (
filter => fun) runs after the nearest-neighbour step, so a tight filter can return fewer thankhits. - Substring and regex search are not in the
barrelAPI or in BQL: callbarrel_ngramor the MCP toolngram_search. The index takes about 34 times the size of the text, and its fields are fixed when you create it. - Reranking needs Python 3 with transformers and torch, and downloads its model on first use. Scores are raw cross-encoder logits, not values between 0 and 1.
- In plain mode the document and its vector are two writes with nothing tying them; record mode is the path that keeps them together.
- In the browser, barrel-lite ranks synced vectors locally by brute-force cosine; keyword and hybrid queries go to the server.