Documentation

Pipeline Mechanism

A pipeline is a directed acyclic graph (DAG) of processing nodes applied to media assets. Pipelines are authored in the Loom UI, stored on Loom (versioned in the database), and executed by Loom’s pipeline engine, which delegates the individual node tasks to registered Cortex workers.

Execution Model: Loom Owns the Graph

Earlier versions of MetaLoom pushed a whole pipeline definition to Cortex and let Cortex execute the graph. That is no longer the case. Today:

Loom owns the DAG. When a run starts, Loom parses the stored definition into an executable graph and drives it with its own pipeline run engine.
Loom selects a worker. It picks a registered, online Cortex instance whose advertised node kinds accept the pipeline’s source kind, and sends it a source task to enumerate media items.
Loom dispatches node tasks. For each discovered item, Loom walks the DAG and dispatches one node task per node to a suitable worker; affinity-grouped work goes out as segment tasks. Cortex only ever sees one node (or segment) at a time and replies with a result.
Loom tracks the run. Run state (status, counters, durations) is persisted on Loom, so a run survives the process that started it and is reported back to the UI live.

  Loom UI ──POST /run──▶ Loom pipeline engine ──SOURCE_TASK──▶ Cortex worker
                              │  owns the DAG      ◀─SOURCE_ITEMS─┘
                              ├──NODE_TASK────────▶ Cortex worker ─NODE_TASK_RESULT─▶
                              └──SEGMENT_TASK─────▶ Cortex worker ─SEGMENT_TASK_RESULT─▶

This is what makes processing horizontally scalable: add Cortex workers to add capacity, and pin heavy node kinds to specific hardware — Loom routes each task to a worker that accepts its kind.

Pipeline Structure

A pipeline version has the following properties:

Field Type Description

Field	Type	Description
`name`	String	Human-readable pipeline name.
`description`	String	Human-readable description.
`priority`	int	When multiple pipelines are candidates, higher priority wins.
`enabled`	boolean	Whether this pipeline is active.
`dryRun`	boolean	When `true`, nodes run but do not persist results.
`definition`	JSON	The node graph — `nodes[]` and `edges[]`.

name

String

Human-readable pipeline name.

description

String

Human-readable description.

priority

int

When multiple pipelines are candidates, higher priority wins.

enabled

boolean

Whether this pipeline is active.

dryRun

boolean

When true, nodes run but do not persist results.

definition

JSON

The node graph — nodes[] and edges[].

Pipelines are versioned: every save creates a new immutable pipeline_version; older versions can be viewed and restored. See REST API for the endpoints.

Node Graph

The definition is a graph of nodes[] connected by edges[]:

{
  "nodes": [
    { "id": "pn1", "type": "filesystem-source", "name": "File Source" },
    { "id": "pn2", "type": "filter-mimetype",   "name": "MIME Filter" },
    { "id": "pn3", "type": "sha256",            "name": "SHA-256 Hash" }
  ],
  "edges": [
    { "id": "pe1", "source": "pn1", "sourcePort": "media",
      "target": "pn2", "targetPort": "media" },
    { "id": "pe2", "source": "pn2", "sourcePort": "passed",
      "target": "pn3", "targetPort": "media", "branch": "PASS" }
  ]
}

Each node carries a graph id and a type — the node kind that Loom uses to route the task to a worker that can run it.
Each edge names a port at both ends: sourcePort on the producing node, targetPort on the consuming one. Both are mandatory — an edge that only names two nodes does not say which value flows, and is rejected.
A pipeline has exactly one source node (e.g. filesystem-source) that enumerates media.
Filter nodes (e.g. filter-mimetype) route items down PASS / REJECT branches so, for example, a video-only branch never receives images. The key is branch, and it defaults to ANY.
Node kinds map to the built-in Nodes (or your own custom nodes).

Ports

A node declares named, typed input and output ports, and the edges say which output port feeds which input port. Nothing addresses another node by its graph id, so renaming a node in the editor cannot quietly starve its consumer.

A port carries three things:

Property Meaning

Property	Meaning
Id	A name local to its node — `media`, `transcript`, `detections`. The same id on two different nodes means nothing; only the edge connects them.
Content type	Always `family/subtype`. The eight families are `media`, `text`, `detection`, `hash`, `scalar`, `artifact`, `struct` and `control`, and `family/*` accepts (or produces) the whole family. Assignability never crosses families.
Cardinality	`ONE` element or `MANY`. A `MANY` output fans out — a downstream node whose input takes one element runs once per element — and a `MANY` input gathers that branch back into a single task per asset. Both are automatic; there is no merge node.

A name local to its node — media, transcript, detections. The same id on two different nodes means nothing; only the edge connects them.

Content type

Always family/subtype. The eight families are media, text, detection, hash, scalar, artifact, struct and control, and family/* accepts (or produces) the whole family. Assignability never crosses families.

Cardinality

ONE element or MANY. A MANY output fans out — a downstream node whose input takes one element runs once per element — and a MANY input gathers that branch back into a single task per asset. Both are automatic; there is no merge node.

The full port list of every built-in kind is on its page under Nodes, and GET /api/v1/pipeline/node-descriptors returns the same thing for the kinds your server knows.

Wiring is checked when the pipeline is saved, not when it runs. An edge naming a port that does not exist, two ports whose content types do not agree, a required input with nothing connected, or two edges into an input that takes a single element — each is rejected with a message naming the offending nodes and ports, and no run is created.

Node results with sync enabled are written back to Loom as asset metadata; see Cortex Examples for the persistence path.

Running a Pipeline

Trigger a run over REST (or from the UI’s Run button):

POST /api/v1/pipelines/:uuid/run     # body: { mediaUuids, path, pathGlobs, dryRun }
GET  /api/v1/pipelines/:uuid/runs    # run history with status + counters

The response reports whether the run was dispatched, the assigned worker, and the runUuid. If no suitable online worker exists the run is rejected (503) rather than silently queued.

Choosing what to process

The run request narrows what the pipeline’s source enumerates. The most specific selector wins: mediaUuids, then pathGlobs, then path.

Field Behaviour

Field	Behaviour
`path`	A single folder or file. The source compares it against its index and processes only media that is new, modified or moved — so re-running over a large library is cheap.
`pathGlobs`	One or more glob patterns. Always walks the filesystem in full, so use it when you want a complete re-scan or need pattern matching.
`mediaUuids`	Specific assets, resolved to their stored files.

path

A single folder or file. The source compares it against its index and processes only media that is new, modified or moved — so re-running over a large library is cheap.

pathGlobs

One or more glob patterns. Always walks the filesystem in full, so use it when you want a complete re-scan or need pattern matching.

mediaUuids

Specific assets, resolved to their stored files.

From the command line these are --dir, --glob and --asset respectively:

metaloom pipeline run my-pipeline --dir /media/library

Pausing and resuming a run

A run in progress can be suspended and later resumed, or stopped for good:

POST /api/v1/pipelines/:uuid/runs/:runUuid/pause
POST /api/v1/pipelines/:uuid/runs/:runUuid/resume
POST /api/v1/pipelines/:uuid/runs/:runUuid/cancel

A paused run keeps everything it has done. No further work is handed to workers, and the source scan stops as well, so a pause genuinely halts the pipeline rather than merely hiding it. Work already handed to a worker finishes normally. The run’s status becomes PAUSED, and its counters are preserved.

Cancelling is final: a cancelled run cannot be resumed.

A paused run continues to occupy a worker, so it should be resumed or cancelled rather than left indefinitely.

Note	Resuming requires the run to still be live on the server. A run whose server was restarted while it was paused is restored in the paused state and can be resumed as normal; one that was lost entirely must be started again.

From the command line:

metaloom run pause  <run-uuid>
metaloom run resume <run-uuid>
metaloom run cancel <run-uuid>

Dry-Run Mode

When dryRun = true, nodes execute but do not persist results — useful for validating a pipeline against real media without side effects. It is a property of the pipeline definition (or the run request).

Live Progress (WebSocket)

While a run executes, the engine emits structured tracking events which the UI consumes live:

ws://<loom-host>:8092/api/v1/pipelines/events/ws     # optional ?pipeline=<name> and ?run=<uuid> filters

Each event identifies the pipeline, node, media item, status (STARTED, COMPLETED, SKIPPED, FAILED) and duration. The Loom UI subscribes to this stream to display live processing progress and per-node statistics.

Two optional filters narrow the stream: ?pipeline=<name> restricts it to one pipeline, and ?run=<uuid> to a single run. Both may be combined. The stream carries no history — it delivers only what happens after the connection opens.

The command line consumes the same stream:

metaloom run follow <run-uuid>
metaloom pipeline run my-pipeline --dir /media --follow

Custom Nodes

Any node kind can be added by a custom Cortex worker that advertises it. See Cortex Examples for authoring a node in Java, assembling a custom daemon, or implementing a worker in Python.

Looking for something else?

All documentation Feature overview