Metrics

The metrics server is separate from the REST API server. A scrape of the REST port (8092) returns 404 by design; the same applies to the gRPC (8091) and MCP (4041) ports.

The monitoring port is deliberately unauthenticated, so it is not something you want a browser or a script outside your network to reach. For those callers the same figures are available as JSON on the ordinary API port:

This is what the built-in monitoring screen reads. It requires the READ_METRIC permission, and it serves the loom_ meters only — the JVM and framework families stay on the scrape endpoint, where a Prometheus reads them.

Series are named exactly as in a scrape, suffixes included, so a name from the tables below works in both places. Narrow the response with ?prefix=loom_node_tasks_.

A rising loom_pipeline_runs_rejected_total{reason="no_processor"} is the clearest signal that your Cortex fleet does not advertise a node kind the pipeline needs.

loom_node_tasks_dispatched_total rises just as fast whether the fleet is quick or wedged, because it counts work leaving Loom. loom_node_task_latency_seconds is what closes that loop: it is the first meter that only moves when work comes back.

Retries and dead-letters are best read together. Retries without dead-letters is a flaky fleet that is recovering on its own; dead-letters climbing alongside them means the retries are not helping and media is being dropped.

Depth on its own cannot tell you whether the fleet is busy or backed up — 200 outstanding tasks is comfortable against a ceiling of 512 and a queue against a ceiling of 200. Compare the two:

Sustained near 1 means runs are being throttled by their own in-flight limit, and adding workers will help. Well below 1 with poor throughput means the bottleneck is elsewhere — a parked node kind, a slow source, or workers that are not accepting the kinds the graph needs.

Loom stops dispatching a node kind that is failing on every worker — a missing model file, a bad GPU driver, an expired API key. A parked kind produces no dispatches, no failures and no errors, so without these two meters the only symptom is throughput quietly going flat.

The gauge says whether a kind is parked right now; the counter distinguishes one bad deployment from a kind that has been flapping all afternoon.

Connected is not the same as usable: only online workers receive new work. A rolling restart that leaves the fleet in terminating looks perfectly healthy on loom_processors_connected while nothing is being dispatched, so alert on the breakdown rather than the total:

Per-worker CPU and memory are reported by the workers themselves — scrape those from each Cortex instance (cortex_cpu_load, cortex_memory_used_bytes, see Cortex Metrics) rather than from Loom.

Vert.x built-in metrics and the Micrometer JVM binders share the same registry, so the scrape also carries:

In Kubernetes, expose the monitoring port on the Loom Service and annotate the pod:

Metric names follow Prometheus conventions: counters carry a _total suffix, timers are exported in base units as _seconds, and labels are deliberately low-cardinality — node kind, status, message type or a stable worker id. Asset UUIDs, file paths, run UUIDs and user ids are never used as label values.

Timers are exported without a percentile histogram, so each one yields _seconds_count, _seconds_sum and _seconds_max series. Derive averages from sum / count; histogram_quantile() is not available for these meters.