Loom ships with a built-in AI assistant — an OpenAI-style chat, backed by a server-side agentic loop that runs inside the Loom backend. Users converse with their asset domain in natural language ("find beach videos", "what tasks are open for me?", "summarize this clip’s transcript") and the agent answers by calling Loom’s own tools with the caller’s permissions.
The chat is more than a single conversation: work is organised into Sessions (publishable snapshots of a chat’s working state, including any coding workspace), the agent draws on a persistent Memory bank of scoped notes that survive across conversations, its behaviour is steered by user-owned Skills (versioned, shareable instruction packages), and — when enabled — it can run code inside an isolated per-chat coding sandbox. Those four parts are described below.
The chat is the landing page of the Loom UI (route /).
The Agentic Loop
The backend runs a tool-calling loop against a Large Language Model. On each user message the loop:
-
Builds the conversation from the persisted transcript plus the active system prompt.
-
Streams the model’s answer — assistant text, hidden reasoning ("thinking") chunks, and tool calls — back to the browser as Server-Sent Events.
-
Dispatches any tool calls in-process against Loom’s tool registry (permission-checked), feeding the results back to the model.
-
Repeats until the model produces a final answer or the turn limit is reached.
Tool failures become error tool results rather than aborting the run, so the model can react and retry — a run is resilient to a single failing tool.
Built-in domain tools include search_assets, get_asset, search_transcript, list_collections,
asset_statistics, list_pipelines and get_pipeline. Results carry structured references
(asset · collection · task · comment · pipeline · annotation), which the UI renders as clickable
chips inside the conversation — and, for some results, a diagram the chat draws inline (see
Pipeline diagrams).
The LLM provider is pluggable and configured separately from the chat — see the provider settings in Loom Configuration.
Streaming Endpoint
| Method | Path | Description |
|---|---|---|
POST |
|
Send a user message; receive the agent run as an SSE event stream (text, reasoning, tool activity, references, final message). |
DELETE |
|
Cancel the active run (204). |
A chat may have only one active run at a time; a concurrent request returns 409 AGENT_BUSY.
Building a Pipeline by Asking
The assistant can do more than describe a pipeline — it can build one. Ask for what you want processed ("set up a pipeline that transcribes every new video and tags the language") and it works the way a person would: it looks up which processing nodes this Loom actually offers, checks what each one accepts and produces, drafts the pipeline, validates the draft without saving it, fixes whatever the validator objects to, and only then stores it.
The result is an ordinary pipeline. It appears in the pipeline list, opens in the editor, and can be reviewed and changed by hand before you run it. Nothing is started automatically — starting a run stays your decision.
Changes work the same way. Asking for a node to be added or a setting changed stores a new version of the pipeline; earlier versions are kept and can be restored, so a change you did not want is never destructive.
|
Tip
|
Ask the assistant to show you the pipeline before you run it. It draws the node graph as a diagram in the conversation (see below), which is the quickest way to check that it understood you. |
Permissions
Letting the assistant write pipelines is granted separately from letting a person edit them in the pipeline editor, so you can allow one without the other. Three permissions cover it, all in the Pipeline (assistant) group of the role editor:
| Permission | What it allows |
|---|---|
Create pipelines (assistant) |
The assistant may store a new pipeline. Granted in addition to the ordinary create permission — both are needed. |
Edit pipelines (assistant) |
The assistant may store a new version of an existing pipeline. Again in addition to the ordinary edit permission. |
Validate pipelines (assistant) |
The assistant may check a draft. Nothing is stored, so this one is safe to grant on its own — the assistant can design and check a pipeline for you to save yourself. |
Without these, the assistant is not told the pipeline-writing tools exist and will not offer to build one.
Pipeline Diagrams in Chat
Ask the agent about a processing pipeline — "show me the current pipeline for media transcription" — and it answers with a diagram, not just a paragraph. The agent looks the pipeline up by name or ID, and the chat draws its node graph as a compact card right inside the conversation:
The card shows the pipeline as it would run today: the graph of its latest version, with the
version number, a marker when the pipeline is disabled, and an Open action that jumps into the
full pipeline editor. Filter branches are labelled (PASS / REJECT), and very large graphs are
clipped with a note pointing to the editor.
The diagram appears as soon as the agent has looked the pipeline up — before the written answer finishes streaming — and it is stored with the conversation, so reopening the chat later shows it again. The written answer always describes the pipeline in full as well, so nothing is lost when a transcript is copied elsewhere.
Chats
A chat is a stored conversation (/api/v1/chats CRUD). Its transcript — user and assistant
messages, tool calls, reasoning and references — is persisted so a session can be reloaded and
continued. The active skill set is stored per chat.
Sessions
A session is a publishable snapshot of a chat’s working state. Beyond the transcript, a session captures:
-
the coding sandbox filesystem (
/sessionworkspace contents), when the coding sandbox is used; -
the pinned skill versions active at snapshot time;
-
context references to other published sessions, so one session can build on another.
Sessions are owned per user, can be named, tagged, described and published for other users to
reference. They are managed via /api/v1/chat-sessions. Publishing a session shares a reproducible
working context, not just a chat log.
Skills
A skill is a SKILL.md-style markdown instruction package (name, description and a full
markdown body) that steers the agent for a particular task.
A few skills ship with Loom and are always available — you do not create them, activate them or see them in your skill list. Today that is the pipeline authoring skill described below. Everything else is yours:
-
Skills are per user; each user owns their own set (
/api/v1/skillsCRUD). -
Skills are activated per chat — toggled in the chat’s Skills panel and sent with every message.
-
Skills use progressive disclosure: only the name + description are placed in the system prompt; the model pulls the full body on demand via a
load_skilltool, keeping the context window small. -
Skills are versioned and can be published to a shared library (
GET /api/v1/skills/library). Another user installs a published skill (POST /api/v1/skills/:uuid/install), which copies it into their own set with provenance — so a later edit by the author never silently changes an installed copy.
Skills are managed in the UI at /skills (table, markdown editor, publish toggle, Library tab with
install).
Example Skill: memory-keeper
A skill that teaches the agent to use the memory tools — list_memory, get_memory,
put_memory, delete_memory — to remember durable facts across conversations:
---
name: memory-keeper
description: Remember durable facts about the user and their assets across chats using the memory tools.
---
# Memory Keeper
You maintain a long-term memory of facts that stay useful beyond the current chat.
## Before answering
- Call `list_memory` to see what is already stored for the active scope.
- Call `get_memory` to read any note whose title looks relevant to the question.
## When you learn something durable
Persist it with `put_memory` — a short, single-fact note with a descriptive title. Good candidates:
- stable preferences ("prefers 16:9 thumbnails")
- project facts ("the 'Summer2026' collection is the campaign master set")
- decisions the user made that future chats should respect
Keep one fact per note. Do **not** store secrets or transient chat state.
## Housekeeping
If a stored fact becomes wrong, correct it by overwriting the note with `put_memory`, or remove it
with `delete_memory`. Never let a note contradict what the user just told you.Example Skill: transcript-summarizer
A domain skill that turns a request like "summarize this clip" into a tight, sourced summary:
---
name: transcript-summarizer
description: Summarize a video/audio asset's transcript into a short, sourced bullet list.
---
# Transcript Summarizer
When the user asks to summarize a clip:
1. Resolve the asset with `get_asset` (or `search_assets` if only a name was given).
2. Call `search_transcript` to pull the relevant transcript passages.
3. Produce **5–8 bullets**, most important first. Each bullet is one sentence.
4. End with a one-line **TL;DR**.
If no transcript exists yet, say so plainly and suggest running a Whisper pipeline on the asset —
do not invent content.Memory
The agent has a persistent memory bank — scoped markdown notes it can read and write across conversations, so useful facts survive between chats.
-
Notes are scoped (e.g. per user / shared scopes) and quota-limited.
-
During a run the agent reads and writes memory through four tools:
list_memory(enumerate notes in a scope),get_memory(read one note),put_memory(create/overwrite a note), anddelete_memory(remove a note). An optional per-scope index note (memory.md) is inlined into the system prompt for fast recall. -
Users can browse and edit the memory bank from the UI. REST surface (
/api/v1/memory):
| Method | Path | Description |
|---|---|---|
GET |
|
List the caller’s memory scopes with usage and quota. |
GET |
|
List the notes of a scope. |
GET |
|
Read one note. |
POST / PUT |
|
Create / upsert a note. |
DELETE |
|
Delete a note. |
When the coding sandbox is enabled, the memory bank can also be materialized read-only into the runner so coding tasks can consult it.
Coding Sandbox
Beyond querying the domain, the agent can run coding tasks (run_shell, write_file,
read_file, list_files) inside a hardened, per-chat Session Runner container. This turns the
chat into a lightweight coding agent operating on an isolated /workspace. It is off by default.
-
One Session Runner is provisioned lazily on the first coding tool call of a chat, and reaped on idle-TTL or when a hard session time-box elapses.
-
The runner’s only process is
runnerd, a small standard-library HTTP daemon. No credentials ever enter the runner — Loom drives it over HTTP with a per-session bearer token, confined to its/workspace. -
Loom holds the registry of live runners, applies a per-deployment concurrency cap, waits for readiness, and destroys runners on chat reset / idle / max-session.
Backends
The backend is selected with LOOM_AGENT_SANDBOX_BACKEND:
-
podman(local dev) — runs a rootless, hardened container and reaches it on127.0.0.1:<published-port>. -
kubernetes(production, including OpenShift) — creates a Pod using the Loom pod’s mounted service-account token, and reaches it on the pod IP.
Kubernetes RBAC & Isolation
The Kubernetes backend spawns runner Pods at runtime, so the Loom service account needs permission to manage them:
-
RBAC allowing the Loom service account to
create/get/deletepods inLOOM_AGENT_SANDBOX_NAMESPACE. -
Recommended guardrails on the
app: loom-session-runnerlabel: aResourceQuota, aLimitRange, and a default-denyNetworkPolicy(runners should have no egress).
The runner Pod is hardened: runAsNonRoot, all capabilities dropped, read-only root filesystem,
automountServiceAccountToken: false, seccomp RuntimeDefault, CPU/memory limits, a hard
activeDeadlineSeconds backstop, and emptyDir workspace/tmp volumes. A reference Pod manifest ships
at loom/agent/deploy/session-runner-pod-template.yaml; the backend POSTs the equivalent at runtime.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: loom-session-runner-manager
namespace: loom-runners
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["create", "get", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: loom-session-runner-manager
namespace: loom-runners
subjects:
- kind: ServiceAccount
name: loom # the service account the Loom pod runs as
namespace: loom
roleRef:
kind: Role
name: loom-session-runner-manager
apiGroup: rbac.authorization.k8s.ioThe Runner Image
Build and push the runner image so it matches LOOM_AGENT_SANDBOX_IMAGE:
loom/agent/session-runner/build.sh
# default reference: metaloom/loom-session-runner:latestSee Container Images for the image overview.
Sandbox Configuration (LOOM_AGENT_SANDBOX_*)
| Variable | Default | Meaning |
|---|---|---|
|
|
Master switch for the coding tools. |
|
|
|
|
|
Runner image reference. |
|
(SA namespace) |
Kubernetes namespace for runner pods. |
|
|
Reap a runner after this much inactivity. |
|
|
Hard per-session time-box. |
|
|
Default per-exec wall-clock timeout. |
|
|
Per-deployment cap on live runners. |
|
|
How long to wait for a runner to become healthy. |
|
|
Runner CPU request / limit. |
|
|
Runner memory request / limit. |
|
|
Workspace |
Full operator notes live in loom/agent/deploy/README.md in the source repository.
Configuration
The agent is configured via LOOM_AI_* (LLM provider + loop), the coding sandbox via
LOOM_AGENT_SANDBOX_* (above), and the memory bank via LOOM_AGENT_MEMORY_*. The agent is enabled
by default but needs a reachable LLM provider to answer; the coding sandbox and memory bank are
off by default. See Loom Configuration for the full tables.