barrel_ngram
An exact substring and regex index over a barrel_docdb database. Every candidate is confirmed against the real text, so a hit is a real match.
What it solves
Semantic search misses identifiers, error strings, config keys and code symbols. barrel_ngram keeps a byte-level trigram index, built from immutable checksummed segments and kept live by the changes feed, and answers literal and PCRE regex queries with match spans.
When to use it
- Agents search code, logs or runbooks for exact names such as
ERR_CONN_TIMEOUTorpg-replica-02. - You need regex search over documents with byte offsets of each match.
When not to
- You search by meaning: use vector or hybrid search in barrel_vectordb.
- Disk is tight: the index is about 34 times the text size.
How it fits into Barrel
- Builds on
barrel_docdb- Used by
barrel_server- Other dependencies
- a C NIF (vendored CRoaring) built with CMake on Linux, macOS and Windows
- On its own
- Yes, next to a barrel_docdb database. The barrel facade and BQL do not expose it.
Install
{barrel_ngram, "~> 0.11"} Configuration
open/2 options
| Key | Default | What it does |
|---|---|---|
db | required | The barrel_docdb database to index |
fields | all | Fields to index; fixed for the corpus |
shards | 1 | Up to 4096; fixed for the corpus |
postings | varint | roaring is the other encoding; fixed |
freeze_threshold | 1000 | Documents buffered before a segment is written |
compact_threshold | 16 | Segments before compaction, or infinity |
verify_segments | checksum | layout checks structure only |
on_legacy | fail | reindex rebuilds a 0.10 or older index |
App env (barrel_ngram)
| Key | Default | What it does |
|---|---|---|
data_dir | "data/barrel_ngram" | Relative to the current directory |
The most used keys only. For every option, see the configuration reference.
The API by task
Corpus
| Task | Call | Returns |
|---|---|---|
| Create or attach | barrel_ngram:open/2 | ok, or {error, {config_mismatch, Field, Old, New}} |
| Catch up with the feed | barrel_ngram:index/1 (alias refresh/1) | {ok, #{segments, watermark}} |
| Compact | barrel_ngram:compact/1 | {ok, #{segments, doc_count}} or {error, busy} |
| Close, delete | barrel_ngram:close/1, delete_corpus/1,2 | ok |
Search
| Task | Call | Returns |
|---|---|---|
| Literal substring | barrel_ngram:search/2,3 | {ok, [#{id, spans => [{Offset, Length}]}]} |
| Regex | barrel_ngram:regex/2,3 | same shape, or {error, {bad_regex, _}} |
| Ignore case | option case_sensitive => false | default is case sensitive |
Find identifiers in runbooks
From a verified run. Spans are byte offsets into the indexed fields, concatenated in sorted field order (body before title).
{ok, _} = application:ensure_all_started(barrel_ngram).
{ok, _} = barrel_docdb:create_db(<<"runbooks">>).
barrel_docdb:put_docs(<<"runbooks">>, [
#{<<"id">> => <<"rb:db">>,
<<"title">> => <<"Postgres failover">>,
<<"body">> => <<"If you see ERR_CONN_TIMEOUT=30s on "
"pg-primary-01, promote pg-replica-02.">>},
#{<<"id">> => <<"rb:api">>, <<"title">> => <<"API 502s">>,
<<"body">> => <<"Check upstream connect_timeout and "
"read_timeout in nginx.conf">>}]).
ok = barrel_ngram:open(<<"rb_idx">>,
#{db => <<"runbooks">>,
fields => [<<"title">>, <<"body">>]}).
barrel_ngram:index(<<"rb_idx">>).
barrel_ngram:search(<<"rb_idx">>, <<"ERR_CONN_TIMEOUT">>).
barrel_ngram:regex(<<"rb_idx">>,
<<"pg-(primary|replica)-\\d+">>).
barrel_ngram:regex(<<"rb_idx">>, <<"\\w+_timeout">>).
barrel_ngram:regex(<<"rb_idx">>, <<"(unclosed">>). {ok,#{segments => 1,watermark => <<...>>}}
{ok,[#{id => <<"rb:db">>,spans => [{11,16}]}]}
{ok,[#{id => <<"rb:db">>,spans => [{35,13},{58,13}]}]}
{ok,[#{id => <<"rb:api">>,spans => [{15,15},{35,12}]}]}
{error,{bad_regex,{"missing closing parenthesis",9}}} Limits to know
- The index is about 34 times the text size.
db,fields,shardsandpostingsare fixed for the corpus. Changing them meansdelete_corpusand a rebuild.index/1is the synchronous catch-up point. Between calls, new writes reach the index through the background feed.- Sharding and roaring postings exist and have tests, but were not part of the verified run.