> ## Documentation Index
> Fetch the complete documentation index at: https://docs.platform.decart.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Realtime API

> Transform video streams in realtime with WebRTC

The Realtime API enables you to transform live video streams with minimal latency using WebRTC. Perfect for building camera effects, video conferencing filters, VR/AR applications, and interactive live streaming.

## Quick Start

```typescript theme={null}
import { createDecartClient, models } from "@decartai/sdk";

const model = models.realtime("lucy-2.1");

// Get user's camera stream
const stream = await navigator.mediaDevices.getUserMedia({
  audio: true,
  video: {
    frameRate: model.fps,
    width: model.width,
    height: model.height,
  },
});

// Create client and connect
const client = createDecartClient({
  apiKey: "your-api-key-here",
});

const characterPhoto = document.querySelector("input[type=file]").files[0];

const realtimeClient = await client.realtime.connect(stream, {
  model,
  onRemoteStream: (transformedStream) => {
    videoElement.srcObject = transformedStream;
  },
  initialState: {
    prompt: {
      text: "Transform into this character",
      enhance: true,
    },
    image: characterPhoto,
  },
});

// Update prompt or reference image later in one atomic call
await realtimeClient.set({
  prompt: "Add silver futuristic sunglasses",
  image: characterPhoto,
  enhance: true,
});

// Disconnect when done
realtimeClient.disconnect();
```

## Client-Side Authentication

For browser applications, use client tokens instead of your permanent API key. Client tokens are short-lived tokens safe to expose in client-side code.

<Info>
  Learn more about [client tokens](/getting-started/client-tokens) and why they're important for security.
</Info>

### Step 1: Create a backend endpoint

Your server creates client tokens using the SDK:

<CodeGroup>
  ```javascript Express.js theme={null}
  import express from "express";
  import { createDecartClient } from "@decartai/sdk";

  const app = express();

  const client = createDecartClient({
    apiKey: process.env.DECART_API_KEY,
  });

  // Endpoint to generate client tokens for authenticated users
  app.post("/api/realtime-token", async (req, res) => {
    try {
      const token = await client.tokens.create({
        expiresIn: 300,              // 5 minutes
        allowedModels: ["lucy-2.1"], // restrict to this model
      });
      res.json(token);
    } catch (error) {
      console.error("Token generation error:", error);
      res.status(500).json({ error: "Failed to generate token" });
    }
  });

  app.listen(3000);
  ```

  ```javascript Next.js API Route theme={null}
  // app/api/realtime-token/route.ts
  import { createDecartClient } from "@decartai/sdk";
  import { NextResponse } from "next/server";

  const client = createDecartClient({
    apiKey: process.env.DECART_API_KEY,
  });

  export async function POST() {
    try {
      const token = await client.tokens.create({
        expiresIn: 300,
        allowedModels: ["lucy-2.1"],
      });
      return NextResponse.json(token);
    } catch (error) {
      return NextResponse.json(
        { error: "Failed to generate token" },
        { status: 500 }
      );
    }
  }
  ```
</CodeGroup>

### Step 2: Use the client token in your frontend

Fetch the client token from your backend and use it to connect:

```typescript theme={null}
import { createDecartClient, models } from "@decartai/sdk";

async function connectToRealtime() {
  // 1. Get client token from your backend
  const tokenResponse = await fetch("/api/realtime-token", { method: "POST" });
  const { apiKey } = await tokenResponse.json();

  // 2. Get user's camera stream
  const model = models.realtime("lucy-restyle-2");
  const stream = await navigator.mediaDevices.getUserMedia({
    video: {
      frameRate: model.fps,
      width: model.width,
      height: model.height,
    },
  });

  // 3. Connect using the client token
  const client = createDecartClient({
    apiKey, // Use client token, not your permanent key!
  });

  const realtimeClient = await client.realtime.connect(stream, {
    model,
    onRemoteStream: (transformedStream) => {
      document.getElementById("output-video").srcObject = transformedStream;
    },
    initialState: {
      prompt: { text: "Anime style" },
    },
  });

  return realtimeClient;
}
```

<Warning>
  Client tokens only prevent **new** connections after expiry. Active WebRTC sessions continue working even after the token expires.
</Warning>

## Connecting

### Getting Camera Access

Request access to the user's camera using the WebRTC getUserMedia API:

```typescript theme={null}
const stream = await navigator.mediaDevices.getUserMedia({
  audio: true,
  video: {
    frameRate: 25,
    width: 1280,
    height: 704,
  },
});
```

<Tip>Use the model's `fps`, `width`, and `height` properties to ensure optimal performance.</Tip>

### Establishing Connection

Connect to the Realtime API with your media stream:

```typescript theme={null}
const realtimeClient = await client.realtime.connect(stream, {
  model: models.realtime("lucy-2"),
  onRemoteStream: (transformedStream: MediaStream) => {
    // Display the transformed video
    const videoElement = document.getElementById("output-video");
    videoElement.srcObject = transformedStream;
  },
  initialState: {
    prompt: {
      text: "Lego World",
      enhance: true, // Let Decart enhance the prompt (recommended)
    },
    image: characterPhoto,
  },
});
```

**Parameters:**

* `stream` (required) - MediaStream from getUserMedia
* `model` (required) - Realtime model from `models.realtime()`
* `onRemoteStream` (required) - Callback that receives the transformed video stream
* `mirror` (optional) - Pre-flip the input video before it's sent. `false` (default), `"auto"`, or `true`. See [Front-camera mirroring](#front-camera-mirroring).
* `initialState` (optional) - Pre-configure the session before the first frame
  * `prompt.text` - Style description
  * `prompt.enhance` - Whether to auto-enhance the prompt (default: true)
  * `image` - Reference image (`Blob`, `File`, or URL string)

<Tip>
  Set `initialState.image` and/or `initialState.prompt` so the first frame is already transformed — otherwise viewers briefly see the raw camera feed.
</Tip>

### Front-camera mirroring

Selfie streams should be pre-flipped on the input so server-baked pixels (watermarks, overlays) come out in display orientation. Pass `mirror`:

```typescript theme={null}
const realtimeClient = await client.realtime.connect(stream, {
  model,
  mirror: "auto",
  onRemoteStream: (s) => { videoElement.srcObject = s; },
});
```

* `false` (default) — never mirror.
* `"auto"` — mirror when the input track reports `facingMode: "user"`.
* `true` — always mirror. Use this on desktop webcams (they usually don't report `facingMode`) or when your app already knows the camera is front-facing.

## Managing Prompts

Change the transformation style dynamically without reconnecting:

```typescript theme={null}
// Simple prompt with automatic enhancement
realtimeClient.setPrompt("Anime style");

// Custom detailed prompt without enhancement
realtimeClient.setPrompt(
  "A detailed artistic style with vibrant colors and dramatic lighting",
  { enhance: false }
);
```

**Parameters:**

* `prompt` (required) - Text description of desired style
* `options.enhance` (optional) - Whether to enhance the prompt (default: true)

<Note>Prompt enhancement uses Decart's AI to expand simple prompts for better results. Disable it if you want full control over the exact prompt.</Note>

## Unified State Update

The `set()` method replaces the entire session state in a single atomic call. The new state contains only the fields you provide — any previously set values not included in the call are cleared.

```typescript theme={null}
import type { SetInput } from "@decartai/sdk";

// Set prompt only (clears any previously set image)
await realtimeClient.set({ prompt: "Anime style", enhance: true });

// Set image only (clears any previously set prompt)
const file = document.querySelector("input[type=file]").files[0];
await realtimeClient.set({ image: file });

// Set both prompt and image together
await realtimeClient.set({
  prompt: "Transform into this character",
  image: "https://example.com/character.jpg",
  enhance: true,
});
```

**Parameters (`SetInput`):**

* `prompt` (optional) - Text description of desired style (at least one of `prompt` or `image` is required)
* `enhance` (optional) - Whether to enhance the prompt (default: true)
* `image` (optional) - Reference image as a `Blob`, `File`, URL string, or `null` to clear

<Tip>`set()` replaces the entire state — fields you omit are cleared. Always include every field you want to keep. Use `set()` instead of separate `setPrompt()` and `setImage()` calls to avoid intermediate states.</Tip>

## Connection State

Monitor and react to connection state changes:

```typescript theme={null}
// Check state synchronously
const isConnected = realtimeClient.isConnected(); // boolean
const state = realtimeClient.getConnectionState(); // "connected" | "connecting" | "generating" | "reconnecting" | "disconnected"

// Listen to state changes
realtimeClient.on("connectionChange", (state) => {
  console.log(`Connection state: ${state}`);
  
  if (state === "disconnected") {
    // Handle disconnection
    showReconnectButton();
  } else if (state === "connected") {
    // Handle successful connection
    hideReconnectButton();
  }
});
```

**Connection States:**

* `"connecting"` — Initial connection in progress
* `"connected"` — Connected and ready to send prompts
* `"generating"` — Actively generating transformed video (sticky until disconnected)
* `"reconnecting"` — Connection was lost unexpectedly; the SDK is automatically retrying
* `"disconnected"` — Not connected (initial state, after `disconnect()`, or after reconnect failure)

<Info>
  The SDK automatically reconnects when an unexpected disconnection occurs (e.g., network interruption). During auto-reconnect, the state transitions to `"reconnecting"` while the SDK retries with exponential backoff (up to 5 attempts). If all retries fail, the state moves to `"disconnected"` and an `error` event is emitted.
</Info>

## Error Handling

Handle errors with the error event:

```typescript theme={null}
import type { DecartSDKError } from "@decartai/sdk";

realtimeClient.on("error", (error: DecartSDKError) => {
  console.error("SDK error:", error.code, error.message);

  switch (error.code) {
    case "INVALID_API_KEY":
      showError("Invalid API key. Please check your credentials.");
      break;
    case "WEB_RTC_ERROR":
      showError("Connection error. Please check your network.");
      break;
    case "MODEL_NOT_FOUND":
      showError("Model not found. Please check the model name.");
      break;
    default:
      showError(`Error: ${error.message}`);
  }
});
```

**Error Codes:**

* `INVALID_API_KEY` - API key is invalid or missing
* `WEB_RTC_ERROR` - WebRTC connection failed
* `MODEL_NOT_FOUND` - Specified model doesn't exist
* `INVALID_INPUT` - Invalid input parameters

## Telemetry and Observability

The JavaScript SDK provides realtime observability in two ways:

1. **SDK telemetry forwarding** (enabled by default) sends stats and diagnostics to Decart's telemetry ingestion pipeline.
2. **Local event stream** lets you consume observability events in your app via `on("stats")` and `on("diagnostic")`.

### Telemetry forwarding

Telemetry is enabled by default. You can disable it when creating the client:

```typescript theme={null}
const client = createDecartClient({
  apiKey: process.env.DECART_API_KEY,
  telemetry: false,
});
```

### Local observability events

```typescript theme={null}
realtimeClient.on("stats", (stats) => {
  // WebRTCStats snapshot (fps, bitrate, packet loss, RTT, etc.)
  console.log("stats", stats);
});

realtimeClient.on("diagnostic", (event) => {
  // Typed diagnostic events from the connection lifecycle
  console.log(event.name, event.data);
});
```

Available diagnostic event names:

* `phaseTiming`
* `iceCandidate`
* `iceStateChange`
* `peerConnectionStateChange`
* `signalingStateChange`
* `selectedCandidatePair`
* `reconnect`
* `videoStall`

<Note>
  `diagnostic` events are emitted by the connection lifecycle. `stats` events are collected automatically while telemetry is enabled.
</Note>

## Session Management

Access the current session ID:

```typescript theme={null}
const sessionId = realtimeClient.sessionId;
console.log(`Current session: ${sessionId}`);
```

This can be useful for logging, analytics, or debugging.

## Session Viewing (Subscribe)

You can let other clients watch an active realtime session as read-only viewers using the subscribe feature. The producer session exposes a `subscribeToken` that viewers use to connect.

### Getting a Subscribe Token

After connecting, the producer's `subscribeToken` is automatically populated:

```typescript theme={null}
const realtimeClient = await client.realtime.connect(stream, {
  model,
  onRemoteStream: (transformedStream) => {
    videoElement.srcObject = transformedStream;
  },
  initialState: {
    prompt: { text: "Anime style" },
  },
});

// Share this token with viewers (e.g., via your backend)
const token = realtimeClient.subscribeToken;
console.log(`Subscribe token: ${token}`);
```

### Subscribing to a Session

Viewers use the subscribe token to receive the transformed video stream in read-only mode:

```typescript theme={null}
import { createDecartClient } from "@decartai/sdk";

const client = createDecartClient({ apiKey: "viewer-api-key" });

const subscriber = await client.realtime.subscribe({
  token: subscribeToken, // Token from the producer
  onRemoteStream: (stream) => {
    viewerVideoElement.srcObject = stream;
  },
});

// Monitor connection state
subscriber.on("connectionChange", (state) => {
  console.log(`Viewer connection: ${state}`);
});

// Handle errors
subscriber.on("error", (error) => {
  console.error("Viewer error:", error.message);
});

// Disconnect when done
subscriber.disconnect();
```

<Note>Subscribe clients are receive-only — they cannot send prompts or modify the session. The subscriber sees exactly what the producer's `onRemoteStream` receives.</Note>

## Cleanup

Always disconnect when done to free up resources:

```typescript theme={null}
// Disconnect from the service
realtimeClient.disconnect();

// Remove event listeners
realtimeClient.off("connectionChange", onConnectionChange);
realtimeClient.off("error", onError);

// Stop the local media stream
stream.getTracks().forEach(track => track.stop());
```

<Warning>Failing to disconnect can leave WebRTC connections open and waste resources.</Warning>

## Complete Example

Here's a full application with all features:

```typescript theme={null}
import { createDecartClient, models, type DecartSDKError } from "@decartai/sdk";

async function setupRealtimeVideo() {
  try {
    // Get camera stream with optimal settings
    const model = models.realtime("lucy-restyle-2");
    const stream = await navigator.mediaDevices.getUserMedia({
      audio: true,
      video: {
        frameRate: model.fps,
        width: model.width,
        height: model.height,
      },
    });

    // Display input video
    const inputVideo = document.getElementById("input-video") as HTMLVideoElement;
    inputVideo.srcObject = stream;

    // Create client
    const client = createDecartClient({
      apiKey: process.env.DECART_API_KEY,
    });

    // Connect to Realtime API
    const realtimeClient = await client.realtime.connect(stream, {
      model,
      onRemoteStream: (transformedStream) => {
        const outputVideo = document.getElementById("output-video") as HTMLVideoElement;
        outputVideo.srcObject = transformedStream;
      },
      initialState: {
        prompt: {
          text: "Studio Ghibli animation style",
          enhance: true,
        },
      },
    });

    // Handle connection state changes
    realtimeClient.on("connectionChange", (state) => {
      const statusElement = document.getElementById("status");
      statusElement.textContent = `Status: ${state}`;
      statusElement.className = `status-${state}`;
    });

    // Handle errors
    realtimeClient.on("error", (error: DecartSDKError) => {
      console.error("Realtime error:", error);
      const errorElement = document.getElementById("error");
      errorElement.textContent = error.message;
      errorElement.style.display = "block";
    });

    // Allow user to change styles
    const styleInput = document.getElementById("style-input") as HTMLInputElement;
    styleInput.addEventListener("change", (e) => {
      const prompt = (e.target as HTMLInputElement).value;
      realtimeClient.setPrompt(prompt, { enhance: true });
    });

    // Cleanup on page unload
    window.addEventListener("beforeunload", () => {
      realtimeClient.disconnect();
      stream.getTracks().forEach(track => track.stop());
    });

    return realtimeClient;
  } catch (error) {
    console.error("Failed to setup realtime video:", error);
    throw error;
  }
}

// Initialize
setupRealtimeVideo();
```

## Best Practices

<AccordionGroup>
  <Accordion title="Use model properties for video constraints">
    Always use the model's `fps`, `width`, and `height` properties when calling getUserMedia to ensure optimal performance and compatibility.

    ```typescript theme={null}
    const model = models.realtime("lucy-restyle-2");
    const stream = await navigator.mediaDevices.getUserMedia({
      video: {
        frameRate: model.fps,
        width: model.width,
        height: model.height,
      },
    });
    ```
  </Accordion>

  <Accordion title="Enable prompt enhancement">
    For best results, keep `enhance: true` (default) to let Decart's AI enhance your prompts. Only disable it if you need exact prompt control.
  </Accordion>

  <Accordion title="Handle connection state changes">
    Always listen to `connectionChange` events to update your UI and handle reconnection logic gracefully.
  </Accordion>

  <Accordion title="Clean up properly">
    Always call `disconnect()` and stop media tracks when done to avoid memory leaks and unnecessary resource usage.
  </Accordion>
</AccordionGroup>

## API Reference

### `client.realtime.connect(stream, options)`

Connects to the realtime transformation service.

**Parameters:**

* `stream: MediaStream` - MediaStream from getUserMedia
* `options.model: ModelDefinition` - Realtime model from `models.realtime()`
* `options.onRemoteStream: (stream: MediaStream) => void` - Callback for transformed video stream
* `options.mirror?: "auto" | boolean` - Pre-flip the input video before sending. `false` (default), `"auto"` to mirror when the track reports `facingMode: "user"`, or `true` to always mirror.
* `options.initialState.prompt: { text: string; enhance?: boolean }` - Initial transformation prompt
* `options.initialState.image: Blob | File | string` - Reference image

**Returns:** `Promise<RealtimeClient>` - Connected realtime client instance

### `realtimeClient.setPrompt(prompt, options?)`

Changes the transformation style.

**Parameters:**

* `prompt: string` - Text description of desired style
* `options.enhance?: boolean` - Whether to enhance the prompt (default: true)

### `realtimeClient.set(input)`

Replaces the entire session state atomically. Fields not included in the input are cleared.

**Parameters:**

* `input: SetInput` - Object with at least one of:
  * `prompt?: string` - Text description of desired style
  * `enhance?: boolean` - Whether to enhance the prompt (default: true)
  * `image?: Blob | File | string | null` - Reference image, or `null` to clear

**Returns:** `Promise<void>`

### `realtimeClient.setImage(image)`

Sets a reference image to guide the transformation. For Lucy Restyle Live, this guides style transformation. For Lucy, this sets a character reference for identity transformation, allowing you to transform yourself into a different character.

**Parameters:**

* `image: Blob | File | string` - Image as a Blob, File object, or URL string

**Returns:** `Promise<void>`

### `realtimeClient.isConnected()`

Check if currently connected.

**Returns:** `boolean`

### `realtimeClient.getConnectionState()`

Get current connection state.

**Returns:** `"connected" | "connecting" | "generating" | "reconnecting" | "disconnected"`

### `realtimeClient.sessionId`

The ID of the current realtime inference session.

**Type:** `string | null`

### `realtimeClient.subscribeToken`

An opaque token that can be shared with other clients to let them watch this session in read-only mode via `client.realtime.subscribe()`.

**Type:** `string | null`

### `realtimeClient.disconnect()`

Closes the connection and cleans up resources.

### `client.realtime.subscribe(options)`

Connects to an existing realtime session as a read-only viewer.

**Parameters:**

* `options.token: string` - Subscribe token from the producer's `realtimeClient.subscribeToken`
* `options.onRemoteStream: (stream: MediaStream) => void` - Callback for the video stream

**Returns:** `Promise<RealTimeSubscribeClient>` - Connected subscribe client

The subscribe client exposes:

* `isConnected()` - Check connection status
* `getConnectionState()` - Get current state
* `disconnect()` - Close the connection
* `on(event, listener)` / `off(event, listener)` - Listen to `connectionChange` and `error` events

### Events

#### `connectionChange`

Fired when connection state changes.

**Callback:** `(state: "connected" | "connecting" | "generating" | "reconnecting" | "disconnected") => void`

#### `error`

Fired when an error occurs.

**Callback:** `(error: DecartSDKError) => void`

#### `generationTick`

Fired periodically during generation with billing information. Use this to track session duration and display usage to users.

**Callback:** `(data: { seconds: number }) => void`

```typescript theme={null}
realtimeClient.on("generationTick", ({ seconds }) => {
  console.log(`Generation running for ${seconds} seconds`);
  updateBillingUI(seconds);
});
```

#### `stats`

Fired continuously with parsed WebRTC transport/media metrics.

**Callback:** `(stats: WebRTCStats) => void`

#### `diagnostic`

Fired for typed connection lifecycle diagnostics.

**Callback:** `(event: DiagnosticEvent) => void`

## Next Steps

<CardGroup cols={2}>
  <Card title="Process API" icon="wand-magic-sparkles" href="/sdks/javascript-process">
    Learn how to generate and transform media with the Process API
  </Card>

  <Card title="Examples" icon="code" href="/examples/real-time-mobile-app">
    See complete example applications
  </Card>
</CardGroup>