Quickstart: embedded
In this quickstart you add Barrel to an Erlang application and build a small help desk knowledge base: three support articles, such as kb-142 "Reset a locked account". You store them as JSON documents, select them with a query, then find the right article from a question worded differently, such as "I cannot log in". Use this path when Barrel will run inside your own OTP release, with no server in between: the database is a library your code calls directly, so there is no network hop and no separate process to deploy.
What you will do10 minutes
- Create an OTP release with barrel as a dependency
- Write and read JSON documents
- Select documents with a BQL query
- Search the articles by meaning with local embeddings
Before you start
- Erlang/OTP 26 or later and rebar3, to build and run the release.
- CMake and a C++ compiler: the vector store builds a native library.
- For step 5 only: Ollama running locally with the
nomic-embed-textmodel (ollama pull nomic-embed-text). Ollama turns text into an embedding, a list of numbers that places text with a similar meaning close together.
1. Create the project
You start from an empty OTP release, so Barrel runs inside your own application the way it will in production.
rebar3 new release kb
cd kb
Add barrel to the dependencies, and tell it where to keep its files.
data_dir is the directory where the document store writes to disk.
{deps, [{barrel, "~> 1.14"}]}. [{barrel_docdb, [{data_dir, "data/docdb"}]}]. 2. Open a database
A database is a named store of documents, and opening it gives you a
handle you pass to every later call. Start a shell with rebar3 shell.
vectordb => none opens the document store only, so nothing needs a
model yet.
{ok, _} = application:ensure_all_started(barrel),
{ok, Db} = barrel:open(<<"kb">>, #{vectordb => none}). 3. Write documents
Each article is a JSON document, written in Erlang as a map with binary
keys. The id field names the document; you read it back by that id.
Here you write the three articles, then read kb-207.
Articles = [
#{<<"id">> => <<"kb-142">>, <<"product">> => <<"console">>,
<<"title">> => <<"Reset a locked account">>},
#{<<"id">> => <<"kb-207">>, <<"product">> => <<"console">>,
<<"title">> => <<"Rotate an API key">>},
#{<<"id">> => <<"kb-311">>, <<"product">> => <<"mobile">>,
<<"title">> => <<"Sign in with a passkey">>}],
[{ok, _} = barrel:put_doc(Db, A) || A <- Articles],
barrel:get_doc(Db, <<"kb-207">>). {ok,#{<<"_rev">> =>
<<"000001a1ba3fd05d00000000@w4f0cb98700ad5ffc">>,
<<"id">> => <<"kb-207">>,<<"product">> => <<"console">>,
<<"title">> => <<"Rotate an API key">>}} Barrel stored the document as you wrote it and added _rev, the
revision token of this write. The part
before @ is a clock value and the part after it names the writer. To
update the document, send _rev back with your change: Barrel then knows
which version you edited, so an edit made from a stale copy cannot
silently overwrite a newer one. See The record.
4. Query with BQL
BQL is Barrel's query language, close to SQL. Every field is indexed, so you can filter and sort on any field without declaring an index first. This query selects the console articles.
{ok, Rows, _Meta} = barrel:query(Db,
<<"SELECT id, title FROM kb "
"WHERE product = 'console' ORDER BY id">>). {ok,[#{<<"id">> => <<"kb-142">>,
<<"title">> => <<"Reset a locked account">>},
#{<<"id">> => <<"kb-207">>,
<<"title">> => <<"Rotate an API key">>}],
#{count => 2, ...}} The result is a list of rows, one map per document, holding only the
fields you selected, in id order. kb-311 is missing because its
product is mobile. The last map is metadata about the answer;
count is the number of rows.
5. Search by meaning
A BQL filter finds "console", but not "I cannot log in": none of the titles contains those words. Search by meaning compares embeddings instead, so a question finds the article that answers it.
Open a second database in record
mode. In record mode Barrel embeds
the fields you name (here title) of every document you write, and keeps
the vector index in step with later edits and deletes, so you never
manage vectors yourself. mode => sync computes the embedding before the
write returns, so a search right after the write finds it. Record mode
only embeds documents written after it is on, so you write the articles
again into this fresh database.
{ok, Kb} = barrel:open(<<"kb_search">>, #{
embedding => #{
fields => [<<"title">>],
mode => sync,
embedder => {ollama,
#{model => <<"nomic-embed-text">>}}},
vectordb => #{db_path => "data/vectors/kb_search"}}),
[{ok, _} = barrel:put_doc(Kb, A) || A <- Articles],
barrel:search(Kb, <<"I cannot log in">>, #{k => 2}). {ok,[#{text => <<"Sign in with a passkey">>,
metadata =>
#{<<"product">> => <<"mobile">>,
<<"title">> => <<"Sign in with a passkey">>},
key => <<"kb-311">>,score => 0.5656703092487099},
#{text => <<"Reset a locked account">>,
metadata =>
#{<<"product">> => <<"console">>,
<<"title">> => <<"Reset a locked account">>},
key => <<"kb-142">>,score => 0.5460874853125287}]} You asked for the 2 closest articles (k => 2). Each result gives the
document id in key, the embedded text in text, the other fields in
metadata, and a similarity score, highest first. Neither title
contains "log in", yet both sign-in articles come back and the API key
article does not.
The same search runs from BQL, joined to the documents, so you can select document fields next to the score:
barrel:query(Kb,
<<"SELECT v.id, v._score, title "
"FROM vector_top_k('lost my credentials', k => 2) AS v">>). {ok,[#{<<"_score">> => 0.5919581721812395,
<<"id">> => <<"kb-142">>,
<<"title">> => <<"Reset a locked account">>},
#{<<"_score">> => 0.47384916313011594,
<<"id">> => <<"kb-311">>,
<<"title">> => <<"Sign in with a passkey">>}],
#{count => 2, ...}} vector_top_k is a table function: it returns the nearest documents as
rows, aliased v, with their id and _score. title comes from the
document itself. With a different question, "lost my credentials", kb-142
now ranks first.
Record mode embeds what you write after it is on; it does not index documents already in a database opened without it. No Ollama? Any provider works here: a local model through a Python virtualenv, or a cloud API with a key. See Choose an embedding provider.
Set db_path as above to choose where the vector store lives. Without
it the store goes next to the documents, in
<data_dir>/kb_search.vectordb.
6. Close it
Closing a database releases its handle. The data stays on
disk, and the next barrel:open/2 with the same name finds it again.
ok = barrel:close(Db),
ok = barrel:close(Kb).