# Monitoring

This page covers what Barrel measures, how you read it, and how you watch replication tasks. Read it when you put a node in production and need to know it is healthy and keeping up.

## What Barrel measures

The document layer, barrel_docdb, records metrics on every node, whether or not you export them. They follow Prometheus conventions: a counter ends in `_total`, and labels are exported as labels. Write your queries against the names below.

| Exported name | Type | What it counts |
|---|---|---|
| `barrel_doc_operations_total{db,operation}` | counter | Document puts, gets and deletes |
| `barrel_doc_operation_duration_seconds{db,operation}` | histogram | Time per document operation |
| `barrel_write_group_size{db}` | histogram | Writes committed together in one batch |
| `barrel_query_operations_total{db}` | counter | Queries run |
| `barrel_query_duration_seconds{db}` | histogram | Time per query |
| `barrel_query_results_count{db}` | histogram | Rows returned per query |
| `barrel_query_timeouts_total` | counter | Queries that hit their deadline |
| `barrel_replication_docs_total{direction}` | counter | Documents written by replication |
| `barrel_replication_errors_total{task_id}` | counter | Replication errors, per replication id |
| `barrel_replication_active{task_id}` | gauge | 1 while the task runs, 0 when it is paused, failed, completed or deleted |
| `barrel_replication_lag_seconds{task_id}` | gauge | Seconds since the last copied change was written on the source; 0 once caught up |
| `barrel_db_documents_total{db}` | gauge | Live documents (deleted ones are not counted), exact |
| `barrel_db_attachments_total{db}` | gauge | Attachments, exact, for both the blob and S3 backends |
| `barrel_db_size_bytes{db}` | gauge | Document store size on disk |
| `barrel_db_pending_compaction_bytes{db}` | gauge | Bytes compaction still has to rewrite |
| `barrel_db_compactions_running{db}`, `barrel_db_compactions_queued{db}` | gauge | Compactions running now, and waiting for a thread |
| `barrel_db_stubs_total{db}`, `barrel_db_stubs_bytes{db}` | gauge | Deleted documents kept as stubs, after each retention sweep |

The size and compaction gauges refresh every minute (`stats_interval`), and once within a second of opening. A pending backlog that keeps growing, or queued compactions that stay above zero, means compaction is falling behind your writes.

A metric appears in the export once it has a value: a node that never replicated prints no replication metrics.

> **Changed in barrel_docdb 3.0:** Before barrel_docdb 3.0 the labels were folded into the name: document operations were exported as `barrel_doc_operations_db_operation_total{db,operation}`, and the same went for every labeled metric. Update dashboards and alerts that use the old names.

The document and attachment counts are exact: every write that adds or removes one updates the count in the same atomic write (barrel_docdb 2.4.0). A database written before 2.4.0 is counted once in the background after it opens; until then its counts are not published and `db_info` reports them as `undefined`.

barrel_vectordb, barrel, barrel_server, barrel_spaces and barrel_embed record no metrics.

## Read the metrics

Inside the node, `barrel_metrics:export_text/0` returns the Prometheus text format as a binary (`export/0` returns iodata).

```erlang
io:format("~s", [barrel_metrics:export_text()]).
```

Output, trimmed, after one write, one read and one query on db mon (live run of the Barrel 2026.11.9 image):

```
barrel_doc_operations_total{db="mon",operation="put"} 1.0
barrel_doc_operations_total{db="mon",operation="get"} 1.0
barrel_query_operations_total{db="mon"} 1.0
barrel_doc_operation_duration_seconds_count{db="mon",operation="put"} 1
```

## Send them to a collector

barrel_server does not serve a `/metrics` route. Push the metrics over OTLP to your collector, which Prometheus or another backend then reads. Set the exporters in the `barrel_docdb` application environment:

```erlang
[{barrel_docdb, [
   {metrics, [{exporter, otlp},
              {otlp_endpoint, "http://collector:4318"}]},
   {tracing, [{enabled, true}, {exporter, otlp},
              {otlp_endpoint, "http://collector:4318"}]}
 ]}].
```

The exporter is `none` by default; `console` and `{Mod, Map}` are also accepted. Metrics, tracing and logging each take their own exporter, see [Configuration](/docs/reference/configuration#barrel_docdb).

Tracing covers document operations, queries and replication. Outgoing replication requests carry trace headers to the peer.

> **Note:** The `export_interval` key shown in some barrel_docdb docs is never read.

If you want a scrape endpoint, serve `barrel_metrics:export_text()` from a handler in your own application.

## Stats you can ask for

| Call | Returns |
|---|---|
| `barrel:info(Db)`, `GET /db/:db` | Configuration, retention, history floor, write-group counters, embedder, and `storage`: size on disk, compaction backlog, and the `documents` and `attachments` counts. |
| `barrel:vector_stats(Db)` | Vector count, dimension, index type and whether the index was loaded or rebuilt |
| `barrel_docdb:stub_stats(Name)` | Stub count and bytes. Scans the change feed in the caller. |
| `barrel_rep_tasks:list_tasks()` | Every replication task with its status |
| `barrel_rep_tasks:get_task(Id)` | One task: `status`, `last_seq`, `last_error`, `paused_reason` |

```erlang
{ok, S} = barrel:vector_stats(Db),
maps:with([count, dimension, index_origin], S).
```

The MCP tool `db_info` says it returns a document count; it returns the same map as `barrel:info/1`, which has none.

## Watch replication tasks

A continuous task keeps the status `running` while it retries, with exponential backoff. The error is on the task, not in its status, so alert on `last_error`.

```erlang
{ok, Tasks} = barrel_rep_tasks:list_tasks(#{mode => continuous}),
[{Id, Status, maps:get(last_error, T, none)}
 || #{id := Id, status := Status} = T <- Tasks].
```

- Statuses: `pending`, `running`, `paused`, `completed`, `failed`.
- `paused` with `paused_reason` `<<"db_closed">>` means the task's local database is not open. It resumes when the database opens.
- A one-shot task fails fast; only continuous tasks retry.

## Health checks

`GET /health` answers `200 {"status":"ok"}` without authentication. It says the HTTP listener is up; it does not open a database or check a store.

```shell
curl -s localhost:8080/health
```

`GET /ready` is the readiness check. It also needs no token. It answers `200 {"status":"ready"}` when the database and replication processes run, and `503 {"status":"not_ready","failed":[...]}` otherwise, where `failed` names the checks that are down among `databases`, `stores` and `replication`. Point a load balancer or orchestrator at it.

```shell
curl -s localhost:8080/ready
```

For a check on one database, read `GET /db/:db` with a token. It answers 404 `db_not_found` when the database does not exist; it does not create it.

```shell
curl -s -H "authorization: Bearer $TOKEN" \
  localhost:8080/db/kb
```

> **Known gap:** barrel_server records no HTTP request metrics and serves no `/metrics` route yet.
