Barrel
FR GitHub

barrel_ngram

An exact substring and regex index over a barrel_docdb database. Every candidate is confirmed against the real text, so a hit is a real match.

What it solves

Semantic search misses identifiers, error strings, config keys and code symbols. barrel_ngram keeps a byte-level trigram index, built from immutable checksummed segments and kept live by the changes feed, and answers literal and PCRE regex queries with match spans.

When to use it

  • Agents search code, logs or runbooks for exact names such as ERR_CONN_TIMEOUT or pg-replica-02.
  • You need regex search over documents with byte offsets of each match.

When not to

  • You search by meaning: use vector or hybrid search in barrel_vectordb.
  • Disk is tight: the index is about 34 times the text size.

How it fits into Barrel

Builds on
barrel_docdb
Other dependencies
a C NIF (vendored CRoaring) built with CMake on Linux, macOS and Windows
On its own
Yes, next to a barrel_docdb database. The barrel facade and BQL do not expose it.

Install

rebar.config deps
{barrel_ngram, "~> 0.11"}

Configuration

open/2 options

KeyDefaultWhat it does
db required The barrel_docdb database to index
fields all Fields to index; fixed for the corpus
shards 1 Up to 4096; fixed for the corpus
postings varint roaring is the other encoding; fixed
freeze_threshold 1000 Documents buffered before a segment is written
compact_threshold 16 Segments before compaction, or infinity
verify_segments checksum layout checks structure only
on_legacy fail reindex rebuilds a 0.10 or older index

App env (barrel_ngram)

KeyDefaultWhat it does
data_dir "data/barrel_ngram" Relative to the current directory

The most used keys only. For every option, see the configuration reference.

The API by task

Corpus

TaskCallReturns
Create or attach barrel_ngram:open/2 ok, or {error, {config_mismatch, Field, Old, New}}
Catch up with the feed barrel_ngram:index/1 (alias refresh/1) {ok, #{segments, watermark}}
Compact barrel_ngram:compact/1 {ok, #{segments, doc_count}} or {error, busy}
Close, delete barrel_ngram:close/1, delete_corpus/1,2 ok

Search

TaskCallReturns
Literal substring barrel_ngram:search/2,3 {ok, [#{id, spans => [{Offset, Length}]}]}
Regex barrel_ngram:regex/2,3 same shape, or {error, {bad_regex, _}}
Ignore case option case_sensitive => false default is case sensitive

Find identifiers in runbooks

From a verified run. Spans are byte offsets into the indexed fields, concatenated in sorted field order (body before title).

Erlang shell
{ok, _} = application:ensure_all_started(barrel_ngram).
{ok, _} = barrel_docdb:create_db(<<"runbooks">>).
barrel_docdb:put_docs(<<"runbooks">>, [
  #{<<"id">> => <<"rb:db">>,
    <<"title">> => <<"Postgres failover">>,
    <<"body">> => <<"If you see ERR_CONN_TIMEOUT=30s on "
                    "pg-primary-01, promote pg-replica-02.">>},
  #{<<"id">> => <<"rb:api">>, <<"title">> => <<"API 502s">>,
    <<"body">> => <<"Check upstream connect_timeout and "
                    "read_timeout in nginx.conf">>}]).
ok = barrel_ngram:open(<<"rb_idx">>,
    #{db => <<"runbooks">>,
      fields => [<<"title">>, <<"body">>]}).
barrel_ngram:index(<<"rb_idx">>).
barrel_ngram:search(<<"rb_idx">>, <<"ERR_CONN_TIMEOUT">>).
barrel_ngram:regex(<<"rb_idx">>,
                   <<"pg-(primary|replica)-\\d+">>).
barrel_ngram:regex(<<"rb_idx">>, <<"\\w+_timeout">>).
barrel_ngram:regex(<<"rb_idx">>, <<"(unclosed">>).
Output, from a run on barrel_ngram 0.11.4
{ok,#{segments => 1,watermark => <<...>>}}

{ok,[#{id => <<"rb:db">>,spans => [{11,16}]}]}

{ok,[#{id => <<"rb:db">>,spans => [{35,13},{58,13}]}]}

{ok,[#{id => <<"rb:api">>,spans => [{15,15},{35,12}]}]}

{error,{bad_regex,{"missing closing parenthesis",9}}}

Limits to know

  • The index is about 34 times the text size.
  • db, fields, shards and postings are fixed for the corpus. Changing them means delete_corpus and a rebuild.
  • index/1 is the synchronous catch-up point. Between calls, new writes reach the index through the background feed.
  • Sharding and roaring postings exist and have tests, but were not part of the verified run.

Up and down to moveEnter to open