Realtime API - Decart API Platform

The Realtime API enables you to transform live video streams with minimal latency using WebRTC. Perfect for building camera effects, video conferencing filters, VR/AR applications, and interactive live streaming.

Quick Start

import { createDecartClient, models } from "@decartai/sdk";

const model = models.realtime("lucy_2_rt");

// Get user's camera stream
const stream = await navigator.mediaDevices.getUserMedia({
  audio: true,
  video: {
    frameRate: model.fps,
    width: model.width,
    height: model.height,
  },
});

// Create client and connect
const client = createDecartClient({
  apiKey: "your-api-key-here",
  telemetry: true, // optional, defaults to true
});

const realtimeClient = await client.realtime.connect(stream, {
  model,
  onRemoteStream: (transformedStream) => {
    videoElement.srcObject = transformedStream;
  },
  initialState: {
    prompt: {
      text: "Transform into this character",
      enhance: true,
    },
  },
});

const characterPhoto = document.querySelector("input[type=file]").files[0];

// Update prompt or reference image in one atomic call
await realtimeClient.set({
  prompt: "Add silver futuristic sunglasses",
  image: characterPhoto,
  enhance: true,
});

// Disconnect when done
realtimeClient.disconnect();

Client-Side Authentication

For browser applications, use client tokens instead of your permanent API key. Client tokens are short-lived tokens safe to expose in client-side code.

Learn more about client tokens and why they’re important for security.

Step 1: Create a backend endpoint

Your server creates client tokens using the SDK:

import express from "express";
import { createDecartClient } from "@decartai/sdk";

const app = express();

const client = createDecartClient({
  apiKey: process.env.DECART_API_KEY,
});

// Endpoint to generate client tokens for authenticated users
app.post("/api/realtime-token", async (req, res) => {
  try {
    const token = await client.tokens.create();
    res.json(token);
  } catch (error) {
    console.error("Token generation error:", error);
    res.status(500).json({ error: "Failed to generate token" });
  }
});

app.listen(3000);

Step 2: Use the client token in your frontend

Fetch the client token from your backend and use it to connect:

import { createDecartClient, models } from "@decartai/sdk";

async function connectToRealtime() {
  // 1. Get client token from your backend
  const tokenResponse = await fetch("/api/realtime-token", { method: "POST" });
  const { apiKey } = await tokenResponse.json();

  // 2. Get user's camera stream
  const model = models.realtime("mirage_v2");
  const stream = await navigator.mediaDevices.getUserMedia({
    video: {
      frameRate: model.fps,
      width: model.width,
      height: model.height,
    },
  });

  // 3. Connect using the client token
  const client = createDecartClient({
    apiKey, // Use client token, not your permanent key!
  });

  const realtimeClient = await client.realtime.connect(stream, {
    model,
    onRemoteStream: (transformedStream) => {
      document.getElementById("output-video").srcObject = transformedStream;
    },
    initialState: {
      prompt: { text: "Anime style" },
    },
  });

  return realtimeClient;
}

Client tokens only prevent new connections after expiry. Active WebRTC sessions continue working even after the token expires.

Connecting

Getting Camera Access

Request access to the user’s camera using the WebRTC getUserMedia API:

const stream = await navigator.mediaDevices.getUserMedia({
  audio: true,
  video: {
    frameRate: 25,
    width: 1280,
    height: 704,
  },
});

Use the model’s fps, width, and height properties to ensure optimal performance.

Establishing Connection

Connect to the Realtime API with your media stream:

const realtimeClient = await client.realtime.connect(stream, {
  model: models.realtime("mirage_v2"),
  onRemoteStream: (transformedStream: MediaStream) => {
    // Display the transformed video
    const videoElement = document.getElementById("output-video");
    videoElement.srcObject = transformedStream;
  },
  initialState: {
    prompt: {
      text: "Lego World",
      enhance: true, // Let Decart enhance the prompt (recommended)
    },
  },
});

Parameters:

stream (required) - MediaStream from getUserMedia
model (required) - Realtime model from models.realtime()
onRemoteStream (required) - Callback that receives the transformed video stream
initialState.prompt (required) - Initial style prompt
- text - Style description
- enhance - Whether to auto-enhance the prompt (default: true)

Managing Prompts

Change the transformation style dynamically without reconnecting:

// Simple prompt with automatic enhancement
realtimeClient.setPrompt("Anime style");

// Custom detailed prompt without enhancement
realtimeClient.setPrompt(
  "A detailed artistic style with vibrant colors and dramatic lighting",
  { enhance: false }
);

Parameters:

prompt (required) - Text description of desired style
options.enhance (optional) - Whether to enhance the prompt (default: true)

Prompt enhancement uses Decart’s AI to expand simple prompts for better results. Disable it if you want full control over the exact prompt.

Unified State Update

The set() method lets you update the prompt, reference image, or both in a single atomic call. For Lucy 2 (lucy_2_rt), it ensures prompt and image changes are applied together.

import type { SetInput } from "@decartai/sdk";

// Update prompt only
await realtimeClient.set({ prompt: "Anime style", enhance: true });

// Update reference image only
const file = document.querySelector("input[type=file]").files[0];
await realtimeClient.set({ image: file });

// Update both atomically — prompt and image are applied together
await realtimeClient.set({
  prompt: "Transform into this character",
  image: "https://example.com/character.jpg",
  enhance: true,
});

Parameters (SetInput):

prompt (optional) - Text description of desired style (at least one of prompt or image is required)
enhance (optional) - Whether to enhance the prompt (default: true)
image (optional) - Reference image as a Blob, File, URL string, or null to clear

Use set() instead of separate setPrompt() and setImage() calls when you need to change both at once. This avoids intermediate states where only one value has updated.

Avatar Live

The Avatar Live model animates portrait images with audio input. Unlike other realtime models that transform camera streams, Avatar Live takes a static image and audio to generate animated video.

Connecting with Avatar Image

import { createDecartClient, models } from "@decartai/sdk";

const model = models.realtime("live_avatar");

// Load avatar image (Blob, File, or URL string)
const avatarImage = await fetch("/portrait.jpg").then(r => r.blob());

const client = createDecartClient({
  apiKey: "your-api-key-here"
});

// Connect with avatar image (pass null for stream)
const realtimeClient = await client.realtime.connect(null, {
  model,
  initialState: { image: avatarImage },
  onRemoteStream: (animatedStream) => {
    videoElement.srcObject = animatedStream;
  }
});

Avatar Live doesn’t require a camera stream. Pass null as the first argument to connect().

Playing Audio

Send audio to animate the avatar:

// Play audio file
const audioFile = document.querySelector("input[type=file]").files[0];
await realtimeClient.playAudio(audioFile);

// Play from ArrayBuffer
const audioBuffer = await audioFile.arrayBuffer();
await realtimeClient.playAudio(audioBuffer);

// Play recorded audio blob
await realtimeClient.playAudio(recordedBlob);

Supported formats:

Blob - Audio blob (e.g., from MediaRecorder)
File - Audio file from file input
ArrayBuffer - Raw audio data

Avatar Behavior Prompts

Control how the avatar behaves and expresses:

// Set avatar behavior
await realtimeClient.setPrompt("Smile warmly and nod occasionally");

// With prompt enhancement
await realtimeClient.setPrompt("Look excited", { enhance: true });

Complete Avatar Live Example

import { createDecartClient, models, type DecartSDKError } from "@decartai/sdk";

async function setupAvatarLive() {
  const model = models.realtime("live_avatar");

  const client = createDecartClient({
    apiKey: process.env.DECART_API_KEY
  });

  // Load avatar image
  const avatarImage = await fetch("/portrait.jpg").then(r => r.blob());

  const realtimeClient = await client.realtime.connect(null, {
    model,
    initialState: { image: avatarImage },
    onRemoteStream: (stream) => {
      document.getElementById("avatar-video").srcObject = stream;
    }
  });

  // Handle connection state
  realtimeClient.on("connectionChange", (state) => {
    console.log(`Connection: ${state}`);
  });

  // Handle errors
  realtimeClient.on("error", (error: DecartSDKError) => {
    console.error("Error:", error.message);
  });

  // Audio file input handler
  document.getElementById("audio-input").addEventListener("change", async (e) => {
    const file = (e.target as HTMLInputElement).files[0];
    if (file) {
      await realtimeClient.playAudio(file);
    }
  });

  return realtimeClient;
}

Connection State

Monitor and react to connection state changes:

// Check state synchronously
const isConnected = realtimeClient.isConnected(); // boolean
const state = realtimeClient.getConnectionState(); // "connected" | "connecting" | "generating" | "reconnecting" | "disconnected"

// Listen to state changes
realtimeClient.on("connectionChange", (state) => {
  console.log(`Connection state: ${state}`);
  
  if (state === "disconnected") {
    // Handle disconnection
    showReconnectButton();
  } else if (state === "connected") {
    // Handle successful connection
    hideReconnectButton();
  }
});

Connection States:

"connecting" — Initial connection in progress
"connected" — Connected and ready to send prompts
"generating" — Actively generating transformed video (sticky until disconnected)
"reconnecting" — Connection was lost unexpectedly; the SDK is automatically retrying
"disconnected" — Not connected (initial state, after disconnect(), or after reconnect failure)

The SDK automatically reconnects when an unexpected disconnection occurs (e.g., network interruption). During auto-reconnect, the state transitions to "reconnecting" while the SDK retries with exponential backoff (up to 5 attempts). If all retries fail, the state moves to "disconnected" and an error event is emitted.

Error Handling

Handle errors with the error event:

import type { DecartSDKError } from "@decartai/sdk";

realtimeClient.on("error", (error: DecartSDKError) => {
  console.error("SDK error:", error.code, error.message);

  switch (error.code) {
    case "INVALID_API_KEY":
      showError("Invalid API key. Please check your credentials.");
      break;
    case "WEB_RTC_ERROR":
      showError("Connection error. Please check your network.");
      break;
    case "MODEL_NOT_FOUND":
      showError("Model not found. Please check the model name.");
      break;
    default:
      showError(`Error: ${error.message}`);
  }
});

Error Codes:

INVALID_API_KEY - API key is invalid or missing
WEB_RTC_ERROR - WebRTC connection failed
MODEL_NOT_FOUND - Specified model doesn’t exist
INVALID_INPUT - Invalid input parameters

Telemetry and Observability

The JavaScript SDK provides realtime observability in two ways:

SDK telemetry forwarding (enabled by default) sends stats and diagnostics to Decart’s telemetry ingestion pipeline.
Local event stream lets you consume observability events in your app via on("stats") and on("diagnostic").

Telemetry forwarding

Telemetry is enabled by default. You can disable it when creating the client:

const client = createDecartClient({
  apiKey: process.env.DECART_API_KEY,
  telemetry: false,
});

Local observability events

realtimeClient.on("stats", (stats) => {
  // WebRTCStats snapshot (fps, bitrate, packet loss, RTT, etc.)
  console.log("stats", stats);
});

realtimeClient.on("diagnostic", (event) => {
  // Typed diagnostic events from the connection lifecycle
  console.log(event.name, event.data);
});

Available diagnostic event names:

phaseTiming
iceCandidate
iceStateChange
peerConnectionStateChange
signalingStateChange
selectedCandidatePair
reconnect
videoStall

diagnostic events are emitted by the connection lifecycle. stats events are collected automatically while telemetry is enabled.

Session Management

Access the current session ID:

const sessionId = realtimeClient.sessionId;
console.log(`Current session: ${sessionId}`);

This can be useful for logging, analytics, or debugging. You can let other clients watch an active realtime session as read-only viewers using the subscribe feature. The producer session exposes a subscribeToken that viewers use to connect. After connecting, the producer’s subscribeToken is automatically populated:

const realtimeClient = await client.realtime.connect(stream, {
  model,
  onRemoteStream: (transformedStream) => {
    videoElement.srcObject = transformedStream;
  },
  initialState: {
    prompt: { text: "Anime style" },
  },
});

// Share this token with viewers (e.g., via your backend)
const token = realtimeClient.subscribeToken;
console.log(`Subscribe token: ${token}`);

Subscribing to a Session

Viewers use the subscribe token to receive the transformed video stream in read-only mode:

import { createDecartClient } from "@decartai/sdk";

const client = createDecartClient({ apiKey: "viewer-api-key" });

const subscriber = await client.realtime.subscribe({
  token: subscribeToken, // Token from the producer
  onRemoteStream: (stream) => {
    viewerVideoElement.srcObject = stream;
  },
});

// Monitor connection state
subscriber.on("connectionChange", (state) => {
  console.log(`Viewer connection: ${state}`);
});

// Handle errors
subscriber.on("error", (error) => {
  console.error("Viewer error:", error.message);
});

// Disconnect when done
subscriber.disconnect();

Subscribe clients are receive-only — they cannot send prompts or modify the session. The subscriber sees exactly what the producer’s onRemoteStream receives.

Cleanup

Always disconnect when done to free up resources:

// Disconnect from the service
realtimeClient.disconnect();

// Remove event listeners
realtimeClient.off("connectionChange", onConnectionChange);
realtimeClient.off("error", onError);

// Stop the local media stream
stream.getTracks().forEach(track => track.stop());

Failing to disconnect can leave WebRTC connections open and waste resources.

Complete Example

Here’s a full application with all features:

import { createDecartClient, models, type DecartSDKError } from "@decartai/sdk";

async function setupRealtimeVideo() {
  try {
    // Get camera stream with optimal settings
    const model = models.realtime("mirage_v2");
    const stream = await navigator.mediaDevices.getUserMedia({
      audio: true,
      video: {
        frameRate: model.fps,
        width: model.width,
        height: model.height,
      },
    });

    // Display input video
    const inputVideo = document.getElementById("input-video") as HTMLVideoElement;
    inputVideo.srcObject = stream;

    // Create client
    const client = createDecartClient({
      apiKey: process.env.DECART_API_KEY,
    });

    // Connect to Realtime API
    const realtimeClient = await client.realtime.connect(stream, {
      model,
      onRemoteStream: (transformedStream) => {
        const outputVideo = document.getElementById("output-video") as HTMLVideoElement;
        outputVideo.srcObject = transformedStream;
      },
      initialState: {
        prompt: {
          text: "Studio Ghibli animation style",
          enhance: true,
        },
      },
    });

    // Handle connection state changes
    realtimeClient.on("connectionChange", (state) => {
      const statusElement = document.getElementById("status");
      statusElement.textContent = `Status: ${state}`;
      statusElement.className = `status-${state}`;
    });

    // Handle errors
    realtimeClient.on("error", (error: DecartSDKError) => {
      console.error("Realtime error:", error);
      const errorElement = document.getElementById("error");
      errorElement.textContent = error.message;
      errorElement.style.display = "block";
    });

    // Allow user to change styles
    const styleInput = document.getElementById("style-input") as HTMLInputElement;
    styleInput.addEventListener("change", (e) => {
      const prompt = (e.target as HTMLInputElement).value;
      realtimeClient.setPrompt(prompt, { enhance: true });
    });

    // Cleanup on page unload
    window.addEventListener("beforeunload", () => {
      realtimeClient.disconnect();
      stream.getTracks().forEach(track => track.stop());
    });

    return realtimeClient;
  } catch (error) {
    console.error("Failed to setup realtime video:", error);
    throw error;
  }
}

// Initialize
setupRealtimeVideo();

Best Practices

Use model properties for video constraints

Always use the model’s fps, width, and height properties when calling getUserMedia to ensure optimal performance and compatibility.

const model = models.realtime("mirage_v2");
const stream = await navigator.mediaDevices.getUserMedia({
  video: {
    frameRate: model.fps,
    width: model.width,
    height: model.height,
  },
});

Enable prompt enhancement

For best results, keep enhance: true (default) to let Decart’s AI enhance your prompts. Only disable it if you need exact prompt control.

Handle connection state changes

Always listen to connectionChange events to update your UI and handle reconnection logic gracefully.

Clean up properly

Always call disconnect() and stop media tracks when done to avoid memory leaks and unnecessary resource usage.

API Reference

`client.realtime.connect(stream, options)`

Connects to the realtime transformation service. Parameters:

stream: MediaStream | null - MediaStream from getUserMedia, or null for Avatar Live
options.model: ModelDefinition - Realtime model from models.realtime()
options.onRemoteStream: (stream: MediaStream) => void - Callback for transformed video stream
options.initialState.prompt: { text: string; enhance?: boolean } - Initial transformation prompt
options.initialState.image: Blob | File | string - Portrait image for Avatar Live (passed via initialState)

Returns: Promise<RealtimeClient> - Connected realtime client instance

`realtimeClient.setPrompt(prompt, options?)`

Changes the transformation style. Parameters:

prompt: string - Text description of desired style
options.enhance?: boolean - Whether to enhance the prompt (default: true)

`realtimeClient.set(input)`

Updates the prompt, reference image, or both atomically. Parameters:

input: SetInput - Object with at least one of:
- prompt?: string - Text description of desired style
- enhance?: boolean - Whether to enhance the prompt (default: true)
- image?: Blob | File | string | null - Reference image, or null to clear

Returns: Promise<void>

`realtimeClient.setImage(image)`

Sets a reference image to guide the transformation. For Mirage, this guides style transformation. For Lucy, this sets a character reference for identity transformation, allowing you to transform yourself into a different character. Parameters:

image: Blob | File | string - Image as a Blob, File object, or URL string

Returns: Promise<void>

`realtimeClient.isConnected()`

Check if currently connected. Returns: boolean

`realtimeClient.getConnectionState()`

Get current connection state. Returns: "connected" | "connecting" | "generating" | "reconnecting" | "disconnected"

`realtimeClient.sessionId`

The ID of the current realtime inference session. Type: string | null

`realtimeClient.subscribeToken`

An opaque token that can be shared with other clients to let them watch this session in read-only mode via client.realtime.subscribe(). Type: string | null

`realtimeClient.disconnect()`

Closes the connection and cleans up resources.

Avatar Live Methods

These methods are only available when using the live_avatar model.

`realtimeClient.playAudio(audio)`

Sends audio to animate the avatar. Parameters:

audio: Blob | File | ArrayBuffer - Audio data to play

Returns: Promise<void> - Resolves when audio finishes playing Connects to an existing realtime session as a read-only viewer. Parameters:

options.token: string - Subscribe token from the producer’s realtimeClient.subscribeToken
options.onRemoteStream: (stream: MediaStream) => void - Callback for the video stream

Returns: Promise<RealTimeSubscribeClient> - Connected subscribe client The subscribe client exposes:

isConnected() - Check connection status
getConnectionState() - Get current state
disconnect() - Close the connection
on(event, listener) / off(event, listener) - Listen to connectionChange and error events

Events

`connectionChange`

Fired when connection state changes. Callback: (state: "connected" | "connecting" | "generating" | "reconnecting" | "disconnected") => void

`error`

Fired when an error occurs. Callback: (error: DecartSDKError) => void

`generationTick`

Fired periodically during generation with billing information. Use this to track session duration and display usage to users. Callback: (data: { seconds: number }) => void

realtimeClient.on("generationTick", ({ seconds }) => {
  console.log(`Generation running for ${seconds} seconds`);
  updateBillingUI(seconds);
});

`stats`

Fired continuously with parsed WebRTC transport/media metrics. Callback: (stats: WebRTCStats) => void

`diagnostic`

Fired for typed connection lifecycle diagnostics. Callback: (event: DiagnosticEvent) => void

Next Steps

Avatar Live

Create lifelike talking avatars from portrait images

Process API

Learn how to generate and transform media with the Process API

Examples

See complete example applications

Getting started

Realtime

Video

Image

Libraries & SDKs

Resources

​Quick Start

​Client-Side Authentication

​Step 1: Create a backend endpoint

​Step 2: Use the client token in your frontend

​Connecting

​Getting Camera Access

​Establishing Connection

​Managing Prompts

​Unified State Update

​Avatar Live

​Connecting with Avatar Image

​Playing Audio

​Avatar Behavior Prompts

​Complete Avatar Live Example

​Connection State

​Error Handling

​Telemetry and Observability

​Telemetry forwarding

​Local observability events

​Session Management

​Session Viewing (Subscribe)

​Getting a Subscribe Token

​Subscribing to a Session

​Cleanup

​Complete Example

​Best Practices

​API Reference

​client.realtime.connect(stream, options)

​realtimeClient.setPrompt(prompt, options?)

​realtimeClient.set(input)

​realtimeClient.setImage(image)

​realtimeClient.isConnected()

​realtimeClient.getConnectionState()

​realtimeClient.sessionId

​realtimeClient.subscribeToken

​realtimeClient.disconnect()

​Avatar Live Methods

​realtimeClient.playAudio(audio)

​client.realtime.subscribe(options)

​Events

​connectionChange

​error

​generationTick

​stats

​diagnostic

​Next Steps

Avatar Live

Process API

Examples

Quick Start

Client-Side Authentication

Step 1: Create a backend endpoint

Step 2: Use the client token in your frontend

Connecting

Getting Camera Access

Establishing Connection

Managing Prompts

Unified State Update

Avatar Live

Connecting with Avatar Image

Playing Audio

Avatar Behavior Prompts

Complete Avatar Live Example

Connection State

Error Handling

Telemetry and Observability

Telemetry forwarding

Local observability events

Session Management

Session Viewing (Subscribe)

Getting a Subscribe Token

Subscribing to a Session

Cleanup

Complete Example

Best Practices

API Reference

`client.realtime.connect(stream, options)`

`realtimeClient.setPrompt(prompt, options?)`

`realtimeClient.set(input)`

`realtimeClient.setImage(image)`

`realtimeClient.isConnected()`

`realtimeClient.getConnectionState()`

`realtimeClient.sessionId`

`realtimeClient.subscribeToken`

`realtimeClient.disconnect()`

Avatar Live Methods

`realtimeClient.playAudio(audio)`

`client.realtime.subscribe(options)`

Events

`connectionChange`

`error`

`generationTick`

`stats`

`diagnostic`

Next Steps