Barrel
FR GitHub

Monitoring

This page covers what Barrel measures, how you read it, and how you watch replication tasks. Read it when you put a node in production and need to know it is healthy and keeping up.

What Barrel measures

The document layer, barrel_docdb, records metrics on every node, whether or not you export them. They follow Prometheus conventions: a counter ends in _total, and labels are exported as labels. Write your queries against the names below.

Exported nameTypeWhat it counts
barrel_doc_operations_total{db,operation}counterDocument puts, gets and deletes
barrel_doc_operation_duration_seconds{db,operation}histogramTime per document operation
barrel_write_group_size{db}histogramWrites committed together in one batch
barrel_query_operations_total{db}counterQueries run
barrel_query_duration_seconds{db}histogramTime per query
barrel_query_results_count{db}histogramRows returned per query
barrel_query_timeouts_totalcounterQueries that hit their deadline
barrel_replication_docs_total{direction}counterDocuments written by replication
barrel_replication_errors_total{task_id}counterReplication errors, per replication id
barrel_replication_active{task_id}gauge1 while the task runs, 0 when it is paused, failed, completed or deleted
barrel_replication_lag_seconds{task_id}gaugeSeconds since the last copied change was written on the source; 0 once caught up
barrel_db_documents_total{db}gaugeLive documents (deleted ones are not counted), exact
barrel_db_attachments_total{db}gaugeAttachments, exact, for both the blob and S3 backends
barrel_db_size_bytes{db}gaugeDocument store size on disk
barrel_db_pending_compaction_bytes{db}gaugeBytes compaction still has to rewrite
barrel_db_compactions_running{db}, barrel_db_compactions_queued{db}gaugeCompactions running now, and waiting for a thread
barrel_db_stubs_total{db}, barrel_db_stubs_bytes{db}gaugeDeleted documents kept as stubs, after each retention sweep

The size and compaction gauges refresh every minute (stats_interval), and once within a second of opening. A pending backlog that keeps growing, or queued compactions that stay above zero, means compaction is falling behind your writes.

A metric appears in the export once it has a value: a node that never replicated prints no replication metrics.

Changed in barrel_docdb 3.0

Before barrel_docdb 3.0 the labels were folded into the name: document operations were exported as barrel_doc_operations_db_operation_total{db,operation}, and the same went for every labeled metric. Update dashboards and alerts that use the old names.

The document and attachment counts are exact: every write that adds or removes one updates the count in the same atomic write (barrel_docdb 2.4.0). A database written before 2.4.0 is counted once in the background after it opens; until then its counts are not published and db_info reports them as undefined.

barrel_vectordb, barrel, barrel_server, barrel_spaces and barrel_embed record no metrics.

Read the metrics

Inside the node, barrel_metrics:export_text/0 returns the Prometheus text format as a binary (export/0 returns iodata).

io:format("~s", [barrel_metrics:export_text()]).
Output, trimmed, after one write, one read and one query on db mon (live run of the Barrel 2026.11.9 image)
barrel_doc_operations_total{db="mon",operation="put"} 1.0
barrel_doc_operations_total{db="mon",operation="get"} 1.0
barrel_query_operations_total{db="mon"} 1.0
barrel_doc_operation_duration_seconds_count{db="mon",operation="put"} 1

Send them to a collector

barrel_server does not serve a /metrics route. Push the metrics over OTLP to your collector, which Prometheus or another backend then reads. Set the exporters in the barrel_docdb application environment:

[{barrel_docdb, [
   {metrics, [{exporter, otlp},
              {otlp_endpoint, "http://collector:4318"}]},
   {tracing, [{enabled, true}, {exporter, otlp},
              {otlp_endpoint, "http://collector:4318"}]}
 ]}].

The exporter is none by default; console and {Mod, Map} are also accepted. Metrics, tracing and logging each take their own exporter, see Configuration.

Tracing covers document operations, queries and replication. Outgoing replication requests carry trace headers to the peer.

Note

The export_interval key shown in some barrel_docdb docs is never read.

If you want a scrape endpoint, serve barrel_metrics:export_text() from a handler in your own application.

Stats you can ask for

CallReturns
barrel:info(Db), GET /db/:dbConfiguration, retention, history floor, write-group counters, embedder, and storage: size on disk, compaction backlog, and the documents and attachments counts.
barrel:vector_stats(Db)Vector count, dimension, index type and whether the index was loaded or rebuilt
barrel_docdb:stub_stats(Name)Stub count and bytes. Scans the change feed in the caller.
barrel_rep_tasks:list_tasks()Every replication task with its status
barrel_rep_tasks:get_task(Id)One task: status, last_seq, last_error, paused_reason
{ok, S} = barrel:vector_stats(Db),
maps:with([count, dimension, index_origin], S).

The MCP tool db_info says it returns a document count; it returns the same map as barrel:info/1, which has none.

Watch replication tasks

A continuous task keeps the status running while it retries, with exponential backoff. The error is on the task, not in its status, so alert on last_error.

{ok, Tasks} = barrel_rep_tasks:list_tasks(#{mode => continuous}),
[{Id, Status, maps:get(last_error, T, none)}
 || #{id := Id, status := Status} = T <- Tasks].
  • Statuses: pending, running, paused, completed, failed.
  • paused with paused_reason <<"db_closed">> means the task's local database is not open. It resumes when the database opens.
  • A one-shot task fails fast; only continuous tasks retry.

Health checks

GET /health answers 200 {"status":"ok"} without authentication. It says the HTTP listener is up; it does not open a database or check a store.

curl -s localhost:8080/health

GET /ready is the readiness check. It also needs no token. It answers 200 {"status":"ready"} when the database and replication processes run, and 503 {"status":"not_ready","failed":[...]} otherwise, where failed names the checks that are down among databases, stores and replication. Point a load balancer or orchestrator at it.

curl -s localhost:8080/ready

For a check on one database, read GET /db/:db with a token. It answers 404 db_not_found when the database does not exist; it does not create it.

curl -s -H "authorization: Bearer $TOKEN" \
  localhost:8080/db/kb
Known gap

barrel_server records no HTTP request metrics and serves no /metrics route yet.

Written for the site Edit this page View as Markdown

Up and down to moveEnter to open