@mux/ai

Easy to use, purpose-driven, cost effective, configurable workflow functions in a TypeScript SDK for building AI-powered video and audio workflows on the server, powered by Mux, with support for popular AI/LLM providers (OpenAI, Anthropic, Google, Baseten, and any OpenAI-compatible endpoint).

• Examples: getSummaryAndTags, getModerationScores, hasBurnedInCaptions, generateChapters, generateEmbeddings, generateEngagementInsights, translateCaptions, editCaptions, translateAudio
• Workflows automatically ship with "use workflow" compatability with Workflow DevKit

Turn your Mux video and audio assets into structured, actionable data — summaries, chapters, moderation scores, translations, embeddings, and more — with a single function call. @mux/ai handles fetching media data from Mux, formatting it for AI providers, and returning typed results so you can focus on building your product instead of wrangling prompts and media pipelines.

npm install @mux/ai

Add your credentials to a .env file (we support dotenv):

MUX_TOKEN_ID=your_mux_token_id
MUX_TOKEN_SECRET=your_mux_token_secret
OPENAI_API_KEY=your_openai_api_key          # or ANTHROPIC_API_KEY, GOOGLE_GENERATIVE_AI_API_KEY, BASETEN_API_KEY

You only need credentials for the AI provider you're using. See the Credentials guide for full setup details including signed playback, S3 storage, and all supported providers.

For multi-tenant apps or cases where you need to provide API keys at runtime rather than through environment variables, every workflow accepts a credentials option. You can also register a global credentials provider with setWorkflowCredentialsProvider() for dynamic key resolution (e.g. per-tenant secrets). When using Workflow DevKit, credentials can be encrypted before crossing workflow boundaries so plaintext secrets never appear in serialized payloads.

import { getSummaryAndTags } from "@mux/ai/workflows";

const result = await getSummaryAndTags("your-asset-id", {
  provider: "openai",
  tone: "professional",
  includeTranscript: true
});

console.log(result.title);        // "Getting Started with TypeScript"
console.log(result.description);  // "A comprehensive guide to..."
console.log(result.tags);         // ["typescript", "tutorial", "programming"]

⚠️ Note: Many workflows rely on transcripts for best results. Consider enabling auto-generated captions on your Mux assets to unlock the full potential of transcript-based workflows like summarization, chapters, and embeddings. This applies to both video and audio-only assets.

• Pre-built workflows for media AI tasks. Common multi-step operations (transcript access, frame analysis, LLM calls, and structured parsing) are available as high-level functions.
• Support for video and audio assets. The same workflows work with video and audio-only assets, including summarization, moderation, chaptering, and more.
• Provider-flexible API. Choose OpenAI, Anthropic, Google, Baseten, or any OpenAI-compatible endpoint through workflow options while keeping the same workflow interface.
• Published evaluation coverage. Workflows include evals for quality, latency, and cost, with results published publicly on pushes to main.
• Sensible default models. Defaults (gpt-5.6-luna at medium reasoning, claude-sonnet-4-5, gemini-3-flash-preview) are selected to balance output quality and runtime cost. Baseten and OpenAI-compatible endpoints serve user-deployed models, so they use an explicit model or the BASETEN_MODEL/OPENAI_COMPATIBLE_MODEL environment variables.
• Typed end-to-end. Workflow inputs, options, and outputs are fully typed in TypeScript.
• Operational defaults included. Retry handling, error handling, signed playback support, and Workflow DevKit compatibility are built in.
• Prompt customization support. Use promptOverrides to adjust sections of workflow prompts for your domain or product requirements.
• Composable abstractions. Start with full workflows and drop down to lower-level primitives when you need more control.

Workflows are high-level functions that handle complete media AI tasks end-to-end — fetching data from Mux, calling AI providers, and returning structured results. Most workflows support both video and audio-only assets.

See the Workflows guide for detailed documentation, options, and examples for each workflow. See the API Reference for complete parameter and return type details.

Content moderation:

import { getModerationScores } from "@mux/ai/workflows";

const result = await getModerationScores("your-asset-id", {
  provider: "openai",
  thresholds: { sexual: 0.7, violence: 0.8 }
});

if (result.exceedsThreshold) {
  console.log("Content flagged for review");
}

Chapter generation:

import { generateChapters } from "@mux/ai/workflows";

const result = await generateChapters("your-asset-id", {
  provider: "anthropic"
});

// [{ startTime: 0, title: "Introduction" }, { startTime: 45, title: "Main Content" }, ...]

Caption editing:

import { editCaptions } from "@mux/ai/workflows";

const result = await editCaptions("your-asset-id", "track-id", {
  provider: "anthropic",
  autoCensorProfanity: {
    mode: "blank", // "blank" ([____]), "remove", or "mask" (????)
    alwaysCensor: ["brandname"],
    neverCensor: ["damn"],
  },
  replacements: [
    { find: "Mucks", replace: "Mux" },
  ],
});

console.log(result.totalReplacementCount); // 8
console.log(result.autoCensorProfanity?.replacements); // [{cueStartTime: 5, before: "shit", after: "[____]"}, ...]

Semantic search embeddings:

import { generateEmbeddings } from "@mux/ai/workflows";

const result = await generateEmbeddings("your-asset-id", {
  provider: "openai",
  chunkingStrategy: { type: "token", maxTokens: 500, overlap: 100 }
});

for (const chunk of result.chunks) {
  await vectorDB.insert({ embedding: chunk.embedding, startTime: chunk.metadata.startTime });
}

Baseten provider:

BASETEN_API_KEY=your_baseten_api_key
BASETEN_MODEL=your-language-model-id
BASETEN_URL=https://model-id.api.baseten.co/environments/production/sync/v1
BASETEN_EMBEDDING_MODEL=your-embedding-model-id
BASETEN_EMBEDDING_URL=https://model-id.api.baseten.co/environments/production/sync

Omit BASETEN_URL to use Baseten's shared Model APIs; for a dedicated deployment, set it to the deployment's /sync/v1 URL. Embeddings always run against a dedicated deployment, so BASETEN_EMBEDDING_URL (its /sync or /sync/v1 URL) is required.

OpenAI-compatible provider:

OPENAI_COMPATIBLE_BASE_URL=https://my-endpoint.example.com/v1
OPENAI_COMPATIBLE_API_KEY=optional-api-key
OPENAI_COMPATIBLE_MODEL=your-language-model-id
OPENAI_COMPATIBLE_EMBEDDING_MODEL=your-embedding-model-id

Use provider: "openai-compatible" with any endpoint that speaks the OpenAI API protocol — vLLM, Ollama, Together AI, Fireworks, OpenRouter, or a self-hosted gateway. OPENAI_COMPATIBLE_API_KEY is optional for endpoints that don't require authentication, and OPENAI_COMPATIBLE_EMBEDDING_BASE_URL can point embeddings at a separate endpoint (defaults to the shared base URL). Workflows request schema-constrained output (response_format: json_schema), so the endpoint must support structured outputs.

For both providers, workflows that analyze storyboards or frames (such as getSummaryAndTags, hasBurnedInCaptions, and askQuestions) require the deployed model to be vision-capable. See Vision-Capable Models for known-good models and how to verify one.

Every workflow prompt is built from a structured template of named sections. The promptOverrides option lets you swap out individual sections with your own instructions while keeping the battle-tested defaults for everything else — no need to rewrite entire prompts.

const result = await getSummaryAndTags(assetId, {
  provider: "openai",
  promptOverrides: {
    title: "Create a search-optimized title (50-60 chars) with the primary keyword front-loaded.",
    keywords: "Focus on high search volume terms and long-tail keyword phrases.",
    // task, description, qualityGuidelines → keep defaults
  },
});

This works with getSummaryAndTags, generateChapters, and hasBurnedInCaptions. The Prompt Customization guide has ready-to-use presets for SEO, social media, e-commerce, and technical analysis, along with tips for writing effective overrides.

Choosing between OpenAI, Anthropic, and Google for a given workflow isn't guesswork. Every workflow in @mux/ai ships with eval coverage that benchmarks those providers and models against three dimensions. Baseten and generic OpenAI-compatible endpoints serve user-deployed models, so they are supported at runtime but intentionally skipped by eval runs.

• Efficacy — Does it produce accurate, high-quality results?
• Efficiency — How fast is it and how many tokens does it consume?
• Expense — What does each request cost?

Evals run automatically on every push to main and results are published to a public dashboard so you can compare providers side-by-side before choosing one for your use case.

You can also run evals locally against your own assets:

npm run test:eval

See the Evaluations guide for details on the 3 E's framework, adding your own evals, and cross-provider testing.

Primitives are low-level building blocks that give you direct access to Mux media data — transcripts, storyboards, thumbnails, and text chunking utilities. Use them when you need full control over your AI prompts or want to build custom workflows.

import { fetchTranscriptForAsset, getStoryboardUrl } from "@mux/ai/primitives";

const transcript = await fetchTranscriptForAsset(asset, playbackId, { languageCode: "en" });
const storyboard = getStoryboardUrl(playbackId, 640);

All pre-built workflows are composed from these primitives internally, so you can always drop down a level when you need to customize behavior.

See the Primitives guide for the full list of available functions and examples of building custom workflows.

// Import specific workflows
import { getSummaryAndTags, generateChapters } from "@mux/ai/workflows";

// Import specific primitives
import { fetchTranscriptForAsset, getStoryboardUrl } from "@mux/ai/primitives";

// Or import everything via namespace
import { workflows, primitives } from "@mux/ai";

• Node.js (>= 21.0.0)
• A Mux account (free to sign up)
• An API key for at least one supported AI provider

• Mux Video API Docs — Learn about Mux Video features
• Auto-generated Captions — Enable transcripts for your assets
• npm Package — Package page and version history

We welcome contributions! Please see the Contributing Guide for details on setting up your development environment, running tests, and submitting pull requests.

For questions or discussions, feel free to open an issue.

Apache 2.0