Barrel
FR GitHub

Quickstart: embedded

In this quickstart you add Barrel to an Erlang application and build a small help desk knowledge base: three support articles, such as kb-142 "Reset a locked account". You store them as JSON documents, select them with a query, then find the right article from a question worded differently, such as "I cannot log in". Use this path when Barrel will run inside your own OTP release, with no server in between: the database is a library your code calls directly, so there is no network hop and no separate process to deploy.

What you will do10 minutes

  • Create an OTP release with barrel as a dependency
  • Write and read JSON documents
  • Select documents with a BQL query
  • Search the articles by meaning with local embeddings

Before you start

  • Erlang/OTP 26 or later and rebar3, to build and run the release.
  • CMake and a C++ compiler: the vector store builds a native library.
  • For step 5 only: Ollama running locally with the nomic-embed-text model (ollama pull nomic-embed-text). Ollama turns text into an embedding, a list of numbers that places text with a similar meaning close together.

1. Create the project

You start from an empty OTP release, so Barrel runs inside your own application the way it will in production.

rebar3 new release kb
cd kb

Add barrel to the dependencies, and tell it where to keep its files. data_dir is the directory where the document store writes to disk.

rebar.config
{deps, [{barrel, "~> 1.14"}]}.
config/sys.config
[{barrel_docdb, [{data_dir, "data/docdb"}]}].

2. Open a database

A database is a named store of documents, and opening it gives you a handle you pass to every later call. Start a shell with rebar3 shell. vectordb => none opens the document store only, so nothing needs a model yet.

Erlang shell
{ok, _} = application:ensure_all_started(barrel),
{ok, Db} = barrel:open(<<"kb">>, #{vectordb => none}).

3. Write documents

Each article is a JSON document, written in Erlang as a map with binary keys. The id field names the document; you read it back by that id. Here you write the three articles, then read kb-207.

Erlang shell
Articles = [
#{<<"id">> => <<"kb-142">>, <<"product">> => <<"console">>,
  <<"title">> => <<"Reset a locked account">>},
#{<<"id">> => <<"kb-207">>, <<"product">> => <<"console">>,
  <<"title">> => <<"Rotate an API key">>},
#{<<"id">> => <<"kb-311">>, <<"product">> => <<"mobile">>,
  <<"title">> => <<"Sign in with a passkey">>}],
[{ok, _} = barrel:put_doc(Db, A) || A <- Articles],
barrel:get_doc(Db, <<"kb-207">>).
Output from a run on barrel 1.12.0
{ok,#{<<"_rev">> =>
        <<"000001a1ba3fd05d00000000@w4f0cb98700ad5ffc">>,
    <<"id">> => <<"kb-207">>,<<"product">> => <<"console">>,
    <<"title">> => <<"Rotate an API key">>}}
What just happened

Barrel stored the document as you wrote it and added _rev, the revision token of this write. The part before @ is a clock value and the part after it names the writer. To update the document, send _rev back with your change: Barrel then knows which version you edited, so an edit made from a stale copy cannot silently overwrite a newer one. See The record.

4. Query with BQL

BQL is Barrel's query language, close to SQL. Every field is indexed, so you can filter and sort on any field without declaring an index first. This query selects the console articles.

Erlang shell
{ok, Rows, _Meta} = barrel:query(Db,
  <<"SELECT id, title FROM kb "
    "WHERE product = 'console' ORDER BY id">>).
Output from a run on barrel 1.12.0
{ok,[#{<<"id">> => <<"kb-142">>,
     <<"title">> => <<"Reset a locked account">>},
   #{<<"id">> => <<"kb-207">>,
     <<"title">> => <<"Rotate an API key">>}],
  #{count => 2, ...}}
What just happened

The result is a list of rows, one map per document, holding only the fields you selected, in id order. kb-311 is missing because its product is mobile. The last map is metadata about the answer; count is the number of rows.

5. Search by meaning

A BQL filter finds "console", but not "I cannot log in": none of the titles contains those words. Search by meaning compares embeddings instead, so a question finds the article that answers it.

Open a second database in record mode. In record mode Barrel embeds the fields you name (here title) of every document you write, and keeps the vector index in step with later edits and deletes, so you never manage vectors yourself. mode => sync computes the embedding before the write returns, so a search right after the write finds it. Record mode only embeds documents written after it is on, so you write the articles again into this fresh database.

Erlang shell
{ok, Kb} = barrel:open(<<"kb_search">>, #{
  embedding => #{
      fields   => [<<"title">>],
      mode     => sync,
      embedder => {ollama,
                   #{model => <<"nomic-embed-text">>}}},
  vectordb => #{db_path => "data/vectors/kb_search"}}),
[{ok, _} = barrel:put_doc(Kb, A) || A <- Articles],
barrel:search(Kb, <<"I cannot log in">>, #{k => 2}).
Output from a run on barrel 1.12.0 with Ollama nomic-embed-text
{ok,[#{text => <<"Sign in with a passkey">>,
     metadata =>
         #{<<"product">> => <<"mobile">>,
           <<"title">> => <<"Sign in with a passkey">>},
     key => <<"kb-311">>,score => 0.5656703092487099},
   #{text => <<"Reset a locked account">>,
     metadata =>
         #{<<"product">> => <<"console">>,
           <<"title">> => <<"Reset a locked account">>},
     key => <<"kb-142">>,score => 0.5460874853125287}]}
What just happened

You asked for the 2 closest articles (k => 2). Each result gives the document id in key, the embedded text in text, the other fields in metadata, and a similarity score, highest first. Neither title contains "log in", yet both sign-in articles come back and the API key article does not.

The same search runs from BQL, joined to the documents, so you can select document fields next to the score:

Erlang shell
barrel:query(Kb,
  <<"SELECT v.id, v._score, title "
    "FROM vector_top_k('lost my credentials', k => 2) AS v">>).
Output from a run on barrel 1.12.0 with Ollama nomic-embed-text
{ok,[#{<<"_score">> => 0.5919581721812395,
     <<"id">> => <<"kb-142">>,
     <<"title">> => <<"Reset a locked account">>},
   #{<<"_score">> => 0.47384916313011594,
     <<"id">> => <<"kb-311">>,
     <<"title">> => <<"Sign in with a passkey">>}],
  #{count => 2, ...}}
What just happened

vector_top_k is a table function: it returns the nearest documents as rows, aliased v, with their id and _score. title comes from the document itself. With a different question, "lost my credentials", kb-142 now ranks first.

Note

Record mode embeds what you write after it is on; it does not index documents already in a database opened without it. No Ollama? Any provider works here: a local model through a Python virtualenv, or a cloud API with a key. See Choose an embedding provider.

Set db_path as above to choose where the vector store lives. Without it the store goes next to the documents, in <data_dir>/kb_search.vectordb.

6. Close it

Closing a database releases its handle. The data stays on disk, and the next barrel:open/2 with the same name finds it again.

Erlang shell
ok = barrel:close(Db),
ok = barrel:close(Kb).

Written for the site Edit this page View as Markdown

Up and down to moveEnter to open