On this page

Scene-Level Video Analysis

Cut video into scenes and describe them — from the soundtrack, from the picture, or both — so a two-hour recording stops being a single unit of retrieval.

A two-hour recording is one asset, and that is the problem: "show me the part about pricing" has no answer when the only unit of retrieval is the whole file. This playbook cuts video into scenes and then describes those scenes — from the soundtrack, from the picture, or both.

What the Platform Gives You

Node Output port Produces Stored as

scene-detection

scenes

Scene boundaries from optical-flow analysis

asset_segment_comp (whole-set replace)

whisper

transcript

Transcript with per-utterance start/end times

asset_transcript_comp

thumbnail

thumbnail

Contact-sheet preview rendered from video frames

local thumbnail_bin cache + binary upload

quality

metrics (plus blurriness, width, height, fps, frame_count)

Resolution / blurriness / frame metrics

quality component

consistency

is_complete, zero_chunk_count

Is the file whole and decodable

consistency component

Scenes and transcript are both timestamped against the same asset, which is what makes them joinable: a scene is [start, end), an utterance is [start, end), and "what was said in scene 4" is an overlap query over the two stored components.

Important

scene-detection, whisper, consistency and quality are not yet registered with the stock worker’s pipeline node factory, so a run containing them is rejected with 503 until you register the kinds and rebuild the worker image — see Which node kinds a worker can actually run. thumbnail is registered.

Prerequisites

  • A running stack, and assets already in Loom for the video files (nodes resolve the asset by SHA-512 — see Transcribing for the Chat Agent for why that matters).

  • A worker with the OpenCV / video4j native runtime, which scene-detection, thumbnail and quality all need. It is present in the metaloom/cortex-server image.

  • For the audio route, a Whisper runtime and model — same worker setup as the transcription playbook.

Route A — Describe Scenes from the Soundtrack

Cut the video, transcribe the speech, join on time. This is the cheaper route and it works on anything with dialogue: interviews, lectures, podcasts with video, meeting recordings.

{
  "nodes": [
    { "id": "source",      "type": "filesystem-source", "name": "Video library",
      "options": { "path": "/media/video" } },
    { "id": "hash",        "type": "sha512",            "name": "SHA-512 identity" },
    { "id": "consistency", "type": "consistency",       "name": "Decodable?" },
    { "id": "scenes",      "type": "scene-detection",   "name": "Scene boundaries",
      "timeoutMs": 900000 },
    { "id": "speech",      "type": "whisper",           "name": "Transcribe",
      "timeoutMs": 3600000 }
  ],
  "edges": [
    { "id": "e1", "source": "source", "sourcePort": "media",
      "target": "hash",        "targetPort": "media" },
    { "id": "e2", "source": "source", "sourcePort": "media",
      "target": "consistency", "targetPort": "media" },
    { "id": "e3", "source": "source", "sourcePort": "media",
      "target": "scenes",      "targetPort": "media" },
    { "id": "e4", "source": "source", "sourcePort": "media",
      "target": "speech",      "targetPort": "video" }
  ]
}

Every edge names the port at each end; both are mandatory. whisper is wired on its video port because that is the type this graph is about — audio and video are alternatives and exactly one of them may be connected. The source emits the family wildcard media/*, so the connection is accepted when you draw it and settled per item at run time: a stray audio file is simply not handed to a node declaring media/video.

scenes and speech depend on the same source and nothing on each other, so Loom dispatches them independently — potentially to different workers, in parallel. That is the whole reason to model this as a graph rather than a script: the GPU box transcribes while the CPU box runs optical flow.

consistency is worth the extra node for what it records: a truncated download produces a plausible-looking file, and is_complete on the asset is how you find the ones that wasted a half-hour of transcription. It is not a gate you wire in front of the expensive nodes, though — its verdict is a scalar, and routing is what the Filter node’s branch ports do.

Reading the result

ASSET=<asset-uuid>

curl -s -H "Authorization: Bearer $TOKEN" $LOOM/api/v1/assets/$ASSET/segments    | jq
curl -s -H "Authorization: Bearer $TOKEN" $LOOM/api/v1/assets/$ASSET/transcripts | jq

Joining them is a query-side operation — no node does it for you today:

# For each scene, the utterances that overlap it
jq -n --slurpfile s scenes.json --slurpfile t transcript.json '
  $s[0][] as $scene
  | { scene: $scene,
      said: [ $t[0].segments[]
              | select(.start < $scene.end and .end > $scene.start)
              | .text ] }'

Feed that per-scene text to an LLM, or simply ask the chat agent — it already reaches the transcript through search_transcript and can quote the passage and the asset it came from.

Note
 Scene segments are a whole-set replace. Re-running scene-detection on an asset deletes the previous segments, so anything you stored keyed to old segment identifiers must be regenerated too.

Route B — Describe Scenes from the Picture

For content with little or no dialogue — b-roll, surveillance, silent product footage — the soundtrack tells you nothing and you need the frames.

What works today:

{
  "nodes": [
    { "id": "source", "type": "filesystem-source", "name": "Video library",
      "options": { "path": "/media/video" } },
    { "id": "hash",   "type": "sha512",          "name": "SHA-512 identity" },
    { "id": "scenes", "type": "scene-detection", "name": "Scene boundaries" },
    { "id": "sheet",  "type": "thumbnail",       "name": "Contact sheet" },
    { "id": "grade",  "type": "quality",         "name": "Quality metrics" }
  ],
  "edges": [
    { "id": "e1", "source": "source", "sourcePort": "media",
      "target": "hash",   "targetPort": "media" },
    { "id": "e2", "source": "source", "sourcePort": "media",
      "target": "scenes", "targetPort": "media" },
    { "id": "e3", "source": "source", "sourcePort": "media",
      "target": "sheet",  "targetPort": "media" },
    { "id": "e4", "source": "source", "sourcePort": "media",
      "target": "grade",  "targetPort": "media" }
  ]
}

That gives you scene boundaries, a contact-sheet preview and quality metrics per video.

To keep the contact sheet, wire sheet’s `thumbnail port into an S3 Sink: the file it renders lives in the worker’s local cache until something uploads it.

Warning

Per-scene visual captioning of video is not implemented today. The captioning node accepts a video asset but skips it (not implemented); it captions images only, and vlm is image-only as well. There is no node that samples a frame per scene and captions it.

Two ways forward until such a node exists:

  • Extract frames as image assets. Register keyframes as their own assets in Loom and run an image pipeline (captioning or vlm) over them, linking them back to the source video in your own metadata. Every node in that path is a stock node.

  • Write the node. A frame-sampling caption node is a natural custom node: declare a struct/segments input port, wire scene-detection’s `scenes port into it, grab one frame per segment, call the vision model. See A Custom Node in Python for the wire protocol, or Cortex Examples for the Java path.

Fan-out, and Why Scene Detection Does Not Do It

A port carries either a single element or a sequence, and a sequence is what makes a node run more than once per asset. When a MANY output is wired into an input that takes one element, the engine dispatches one task per element — and when that branch is later wired into a MANY input, it gathers the elements belonging to the same asset back into a single task. Neither needs a merge node or any configuration; it follows from the two ports' cardinalities.

scene-detection deliberately does not work that way. Its scenes port carries one element: a single timeline with the cuts inside it, not one element per scene. So nothing downstream runs per scene by connecting to it — which is exactly why per-scene captioning needs the custom node described above rather than a wiring trick.

The built-in node that does fan out is facedetect: its detections port emits one element per detected face, so facedescription — whose input takes one detection — is dispatched once per face. On the gathering side, s3-sink declares its artifacts input as a sequence, so every produced file wired into it arrives in one upload task per asset.

Doing Both

The two routes are the same graph with both branches attached — that is what a DAG is for:

Both routes in one graph A filesystem source emits media on one output port that feeds six independent nodes — sha512 and consistency, and then scene detection, whisper, thumbnail and quality — each wired into its own media input port, which Loom dispatches in parallel, each producing its own stored component. source media scene-detection optical flow whisper speech-to-text thumbnail contact sheet quality resolution · blur sha512 · consistency identity · decodable? → asset_segment_comp → asset_transcript_comp → binary + local cache → quality component every edge leaves the source's media port and enters that node's own media input

Costs to keep in mind when you fan out like this:

  • scene-detection decodes the whole video for optical flow — CPU-heavy, default task timeout 300 s. Long files need it raised.

  • whisper defaults to concurrency 1 and a 600 s timeout.

  • Both read the file from the shared media mount, so a single-spindle NFS export becomes the ceiling long before the CPUs do.

Pin the expensive kinds to the machines that suit them with CORTEX_NODE_WHITELIST — see Docker → dedicated workers or the GPU pool.

Gotchas

Symptom Cause

Run rejected with 503

A kind in the graph is not registered on any online worker — see node availability.

scene-detection fails with UnsatisfiedLinkError

The worker lacks the OpenCV / video4j native runtime, or a mismatched OpenCV version is on the library path.

Captions never appear for videos

Expected: captioning skips video. Caption extracted frames as image assets instead.

Segments disappeared after a re-run

Whole-set replace: a shorter re-run deletes the surplus segments.

Scenes and transcript do not line up

They are separate components joined on time by the consumer. Nothing in the pipeline aligns them.

Nothing runs per scene

scene-detection emits one element — a timeline — not one per scene, so no downstream node fans out from it. See Fan-out, and why scene detection does not do it.

The pipeline will not save: incompatible content types

An edge connects two ports whose types do not agree. consistency’s `is_complete is a boolean and sha512’s `hash is a hash — neither can feed a node that wants media. Wire those nodes off the source’s media port, beside the others.

Next Steps

Looking for something else?

All documentation Feature overview