On this page

Transcribing for the Chat Agent

Turn a library of video and audio into stored transcripts, and make the chat agent able to answer from them.

The chat agent can only answer from what has been extracted. Ask it "which clip mentions the launch date?" and it calls search_transcript — which returns nothing at all until something has actually transcribed your media.

This playbook closes that loop: a pipeline that turns a library of video and audio into stored transcripts, and the chat questions those transcripts then make answerable.

How the Pieces Line Up

From files on disk to an answer in the chat A filesystem-source node walks the library and its media output port feeds two branches. One goes to a whisper node wired to its video and audio media input ports, which accept the item only when its real type matches; the other goes to a sha512 node that establishes asset identity alongside it. The transcript is stored in Loom as an asset transcript component, which the chat agent reads through the search_transcript tool to answer a user question. Pipeline — Loom dispatches one node task at a time filesystem-source media sha512 hash — asset identity whisper audio | video → transcript media/* /media/library — files on disk asset_transcript_comp + asset_node_result ledger row persist over REST Chat & AI Agent search_transcript · get_asset · search_assets “which clips mention the launch date?” read

Two links in that chain are easy to get wrong, and both are about identity:

  1. The asset must already exist in Loom. Nodes resolve the asset by looking it up by SHA-512 (loadAsset(sha512)). A file on disk that is not a known asset still gets processed, but the result has nothing to attach to. Ingest first — upload through the UI, POST /api/v1/assets, or POST /api/v1/assets/bulk/create — then run the pipeline over the same files.

  2. The graph needs a hash node. That SHA-512 lookup needs the hash, so a sha512 node belongs in any graph whose node output you want persisted. It reads the same media port the rest of the graph reads — a hash port carries hash/sha512 and can only be connected to something that declares a hash input, so it is never a step between two media nodes.

Important

whisper is not yet registered with the stock worker’s pipeline node factory, so a run containing it is rejected with 503 until you register the kind and rebuild the worker image — see Which node kinds a worker can actually run. Verify what your workers accept with GET /api/v1/processors before authoring against them.

Prerequisites

  • A running stack — Docker or Kubernetes.

  • Assets in Loom for the media you want transcribed (see above).

  • A worker that can run whisper: a Whisper runtime and a model file. The model is a file on the worker, not something Loom ships.

Step 1 — Prepare the Whisper Worker

Whisper settings live in the worker’s cortex.yml, read from ~/.config/metaloom/cortex.yml (/cortex/.config/metaloom/cortex.yml in the container image):

metaPath: /meta
nodes:
  whisper:
    modelPath: /models/ggml-large-v3-turbo.bin
    language: de          # omit to auto-detect
    useGpu: true
    gpuDevice: 0
    temperature: 0.0
    temperatureInc: 0.2

Mount that file and the model directory into the worker:

docker run -d --name cortex-gpu --network metaloom \
  -e LOOM_HOST=loom -e LOOM_PORT=8092 -e LOOM_TOKEN="$TOKEN" \
  -e CORTEX_NODE_ID=cortex-gpu-1 \
  -e CORTEX_META_PATH=/meta \
  -e CORTEX_NODE_WHITELIST=filesystem-source,sha512,whisper \
  -v cortex-gpu-meta:/meta \
  -v /srv/models:/models:ro \
  -v ./cortex.yml:/cortex/.config/metaloom/cortex.yml:ro \
  -v /srv/media:/media:ro \
  metaloom/cortex-server:latest

The whitelist is doing real work here: it keeps cheap hashing jobs from queueing behind a half-hour transcription on your only GPU box, while a second CPU worker (with whisper blacklisted) handles the rest of the library.

Transcription is slow by design — Cortex gives whisper a default per-task timeout of 600 s and a default concurrency of 1. Long recordings need the timeout raised in the graph (see below).

Step 2 — The Pipeline

{
  "nodes": [
    { "id": "source",   "type": "filesystem-source", "name": "Media library",
      "options": { "path": "/media/library" } },
    { "id": "hash",     "type": "sha512",            "name": "SHA-512 identity" },
    { "id": "speech-v", "type": "whisper",           "name": "Transcribe video",
      "timeoutMs": 3600000 },
    { "id": "speech-a", "type": "whisper",           "name": "Transcribe audio",
      "timeoutMs": 3600000 }
  ],
  "edges": [
    { "id": "e1", "source": "source", "sourcePort": "media",
      "target": "hash",     "targetPort": "media" },
    { "id": "e2", "source": "source", "sourcePort": "media",
      "target": "speech-v", "targetPort": "video" },
    { "id": "e3", "source": "source", "sourcePort": "media",
      "target": "speech-a", "targetPort": "audio" }
  ]
}

Every edge names both ends — sourcePort on the producing node, targetPort on the consuming one — and both are mandatory; there is no positional fallback. What a node reads is decided by the edge drawn into its input port, never by the graph id of whoever produced the value.

What each part is for:

filesystem-source

The one source node. It walks the library and, when given a path, compares against its persisted index so a re-run only picks up new, modified or moved files. Its single output port media is what everything downstream connects to.

`whisper’s two media inputs

Nothing routes by file type here, because nothing has to. Whisper declares audio and video as alternatives — exactly one may be wired — and each carries a concrete content type. A source cannot know an item’s real type when the graph is drawn, so it emits the family wildcard media/*; the connection is accepted when you draw it and settled per item at run time with the file in hand. An image reaching a speech model is prevented by the type, not by a node you had to add.

sha512

The identity step described above. Its hash port carries hash/sha512, so it can only be connected to something that declares a matching hash input, such as a dedup node. It persists the hash itself, so nothing has to be connected to it at all. It hangs off the source rather than sitting in the middle of the media path.

whisper

Persists an asset_transcript_comp plus an asset_node_result ledger row against the asset, and carries the same text on its transcript output port for anything downstream — a sentiment score, a translation, a tts dub. timeoutMs overrides the 600 s default.

Note

Why two whisper nodes. whisper declares audio and video as alternatives: exactly one of the two may be wired, because they are one input with two shapes rather than two independent inputs. A library holding both therefore routes each kind down its own branch. Connect the second one in the editor and it disables the first — see Whisper.
Important

Where a setting goes. Only some keys are read from the pipeline definition. The rest come from the worker’s cortex.yml.

Read from the definition Read from the worker’s cortex.yml

id, type, name, blocking, concurrency, timeoutMs, mode, affinity; and for the source node path, pathGlobs, emitStates

Everything node-specific: whisper.modelPath, whisper.language, llm.prompts, tts.voice, …

Putting "options": {"language": "en"} on a whisper node has no effect — set nodes.whisper.language in the worker config instead. Node IDs must match ^[a-z0-9]([a-z0-9-]{0,62}[a-z0-9])?$; the graph must be acyclic and every node reachable, or the definition is rejected on save.

Step 3 — Create It

LOOM=http://localhost:8092

curl -s -X POST $LOOM/api/v1/pipelines \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "Transcribe Library",
    "description": "Transcribe audio and video so the agent can search speech",
    "enabled": true,
    "priority": 10,
    "definition": { "nodes": [ ... ], "edges": [ ... ] }
  }' | jq -r .uuid

Every save creates a new immutable version; older versions can be listed and restored. Authoring the same graph in the UI’s pipeline editor works too — it is the same stored definition.

Step 4 — Run It

PIPELINE=<uuid-from-above>

curl -s -X POST $LOOM/api/v1/pipelines/$PIPELINE/run \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"path": "/media/library/podcasts"}' | jq

The response carries the assigned worker and a runUuid. If no online worker accepts every kind in the graph, the run is rejected with 503 rather than queued — that is the failure mode to expect when the whisper worker is down.

The selector narrows what the source enumerates; the most specific wins:

Field Behaviour

path

One folder or file, compared against the source’s index — cheap to re-run over a large library.

pathGlobs

Patterns, always a full filesystem walk. Use for a complete re-scan.

mediaUuids

Specific assets, resolved to their stored files.

From the CLI, with live progress:

metaloom pipeline run "Transcribe Library" --dir /media/library/podcasts --follow

A long run can be paused and resumed (metaloom run pause|resume|cancel <run-uuid>); a paused run keeps its progress but continues to occupy a worker.

Step 5 — Check the Transcripts Landed

# Run history with per-status counters
curl -s -H "Authorization: Bearer $TOKEN" $LOOM/api/v1/pipelines/$PIPELINE/runs | jq

# Items that did not make it
metaloom run items <run-uuid> --state FAILED

Then open an asset in the UI — the transcript appears on the asset detail view. If a run reports success but no transcript is attached, the asset lookup failed: the file’s SHA-512 does not match any asset in Loom. Re-check the ingest step.

Step 6 — Ask the Agent

With transcripts stored, the chat has something to search:

"Which clips mention the launch date?"

"Summarize the interview with the product team."

"Find the part where they discuss pricing and tell me which video it is in."

The agent resolves these by calling search_transcript (alongside search_assets, get_asset, list_collections and asset_statistics), all permission-checked as the calling user. Results come back with asset references the UI renders as clickable chips.

To make summaries consistent, activate a skill — the transcript-summarizer skill on the Chat page is written for exactly this pipeline: resolve the asset, pull the transcript, produce 5–8 sourced bullets, and say so plainly when no transcript exists rather than inventing one.

Note
 The agent’s domain tools are read-only. It answers, summarizes and cites from stored transcripts; it does not write results back onto the asset. Persisting derived text is the pipeline’s job, not the chat’s.

Keeping It Current

Re-run the same pipeline on a schedule with path rather than pathGlobs. The filesystem source compares against its persisted index and only emits new, modified or moved media, so a nightly run over a large library is cheap — as long as CORTEX_META_PATH is on a volume that survives restarts.

Gotchas

Symptom Cause

Run rejected with 503

No online worker accepts every node kind in the graph. Check GET /api/v1/processors and the worker’s whitelist/blacklist.

Node succeeds, nothing stored

The file’s SHA-512 matches no asset in Loom, or the graph has no sha512 node.

The pipeline will not save: requires one of [audio, video] to be connected

`whisper’s two media inputs are alternatives — exactly one must be wired, and wiring both is rejected the same way.

Transcriptions time out

The 600 s default is too low for long recordings. Raise timeoutMs on the node.

language / modelPath ignored

They were set in the pipeline definition. Node-specific options come from the worker’s cortex.yml.

Images are being handed to Whisper

The edge lands on the wrong input port. Wire the source into whisper’s `audio or video port — never both, they are alternatives — and the concrete content type keeps everything else out.

Every run re-transcribes everything

The worker’s meta path is not persistent, so the filesystem index is rebuilt empty each start.

Next Steps

Looking for something else?

All documentation Feature overview