Barrel
FR GitHub

Troubleshooting

This page lists symptoms you may see, what causes them and how to fix them. Find your error with the search box, or scan the area it comes from. For the meaning of each error code on its own, see Error codes.

Clocks

SymptomCauseFix
Writes fail with {error, clock_skew} after a restartThe node's clock went back below the newest write the database made. Data is kept and the writer stays up.Fix the clock (NTP). Writes resume once it passes that point.
Replication from a peer fails with clock_skewThe peer's clock is ahead by more than hlc_max_offset milliseconds. Over HTTP, a 409 is reported as clock_skew only when its body says {"error":"clock_skew"}.Sync the clocks.
Replication over HTTP fails with {error, {http, 409, Body}}The peer, or a proxy in front of it, answered 409 for another reason.Read Body.

hlc_max_offset defaults to 0, which turns the offset check off. Set it in the barrel_docdb application environment only if you want peers with a drifting clock refused.

Replication

SymptomCauseFix
replicate returns {error, not_found} against a serverThe remote database does not exist. Replication never creates one.PUT /db/:db on the remote first.
{error, {checkpoint_failed, target, not_found}} on one nodeThe local target database was not created.Call barrel_docdb:create_db(Target) first.
{error, att_flush_unsupported}The target cannot confirm attachments are on disk, and durability is required by default.Upgrade the target, or pass att_durability => best_effort and accept the risk on power loss.
{error, att_filter_unsupported} on a filtered runThe source server cannot filter attachments.Upgrade the source.
start_task returns {error, {inline_credentials, Keys}}Tasks refuse credentials in their configuration.Set sync_auth, sync_signing or sync_ssl in the barrel_docdb application environment.
A continuous task shows running but nothing arrivesIt is retrying; the error is stored in last_error.Read barrel_rep_tasks:get_task(Id) and fix last_error.
A task is paused with db_closedIts local database is not open.Open the database, or set task_db_opener so restored tasks open it.
{error, too_many_failed_docs}More than 1000 documents failed in one run; the checkpoint did not move.Read the log for the failing documents and fix them, then run again.
After upgrading from docdb 2.0, a replica misses documents or holds empty bodiesBugs in 2.0.0. Upgrading stops new damage but repairs nothing.Run barrel_rep:reconcile(Healthy, Replica, #{}) once per replica.
{ok, Report} = barrel_rep:reconcile(<<"kb">>,
    <<"kb_replica">>, #{}),
maps:with([docs_checked, bodies_repaired,
           atts_resent], Report).
SymptomCauseFix
500 bm25_not_enabledKeyword search is off on a plain database.Open with vectordb => #{bm25_backend => disk}, or use record mode, which turns it on.
500 embedder_not_configuredText or hybrid search with no embedder.Configure an embedder, or send a vector.
500 {dimension_mismatch,768,N}The store expects 768 dimensions by default.Send vectors of the store's size, or open it with the model's dimensions.
Vectors have the wrong lengthdimensions is not set and defaults to 768.Set dimensions to the model's size (bge-small is 384).
The first local embed is slow or fails offlinePython providers install packages and download the model on first use.Warm it up once online, then run with HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1.
barrel_vectordb:update/4 or upsert made an entry disappearA bug before barrel_vectordb 2.5.3: the entry was deleted before the new text was embedded, so a failed embed lost it.Upgrade to barrel_vectordb 2.5.3. A failed embed then keeps the old entry and returns the error.

Databases and the server

SymptomCauseFix
open_db returns {error, {db_failed, Reason}}The writer crashed 3 times in 60 seconds. That database stays down; others are not affected.Read the log for Reason, fix it, then barrel_docdb:reopen_db(Name).
{error, db_is_encrypted}Opened without its encryption settings.Pass the same encryption spec on every open.
Opening fails with Snappy is not availablebarrel_docdb 2.2.0 compressed with Snappy, and rocksdb was built without the system Snappy library.Upgrade to barrel_docdb 2.2.1, which uses zstd. Or install libsnappy-dev (or your system's Snappy package) and rebuild rocksdb.
A REST call answers 404 db_not_foundThe database does not exist. Since barrel_server 1.14.0 only PUT /db/:db creates one; before, any route did, so a typo left an empty database.Check the name, or create the database with PUT /db/:db first. DELETE /db/:db?purge=true removes an empty one left by an older server.
PUT /db/MyDb answers 400 invalid_nameDatabase names use lowercase letters, digits, _ and -, 1 to 63 bytes, and must not start with _. Before barrel_server 1.14.0 this answered 500 invalid_db_name.Use a name that follows the rule.
Vector files appear in ./privThe vector store was created before barrel 1.13.0 without db_path, under the working directory. It keeps being used there, with a notice in the log; new stores go to <data_dir>/<name>.vectordb.Back that folder up, or set vectordb => #{db_path => ...} to place it yourself.
A capability token gets 403It is for another space, or lacks the right.Use a token for that space with the right it needs.
Prometheus finds nothing at /metricsThere is no such route.Push with the OTLP exporter, see Monitoring.
barrel_db_documents_total_db is missing just after an upgradeA database written before barrel_docdb 2.4.0 is counted once in the background after it opens.Wait for the first count; db_info reports documents as undefined until then.
barrel_db_pending_compaction_bytes_db keeps growingCompaction is falling behind the writes.Raise max_background_jobs in store_opts, or spread the writes.
A warning says compaction_interval or prune_depth no longer has an effectThese options were removed in barrel_docdb 2.2.0.Remove them from your configuration.

Branches

SymptomCauseFix
400 pitr_window_exceededThe instant is older than the retained history (30 days by default).Pick a later instant, or raise retention_period.
409 cannot_branch_a_branchLineage is one level deep.Branch the parent.

Python helpers

SymptomCauseFix
barrel_rerank creates a venv in a folder named undefinedA bug in barrel_rerank 1.0.2: the app declares venv_path as undefined and the code used it as a path.Upgrade to barrel_rerank 1.0.3, which uses priv/.venv when venv_path is unset. Remove the undefined folder.
barrel_rerank:start_link/1 returns {error, {port_open_failed, enoent}}python is not an absolute path and the venv setup failed.Pass an absolute python, or a working venv_path.
Note

If your symptom is not here, look up the error on Error codes, then check the node's log for the reason term.

Written for the site Edit this page View as Markdown

Up and down to moveEnter to open