Transform video streams in realtime with WebRTC on iOS and macOS
The Realtime API enables you to transform live video streams with minimal latency using WebRTC. Perfect for building iOS camera effects, video conferencing filters, AR applications, and interactive live streaming.
For iOS and macOS applications, use ephemeral keys instead of embedding your permanent API key in the app bundle. Ephemeral keys are short-lived tokens safe to include in client applications.
Learn more about client tokens and why they’re important for security.
Realtime media uses LiveKit tracks. Build a LocalVideoTrack from LiveKit’s camera-track factory, size it from the model registry, and attach a MirroringVideoProcessor to pre-flip the front camera.
Pre-flipping the selfie input keeps server-baked pixels (watermarks, overlays) correctly oriented when you render the remote stream as-is.MirrorMode values:
.off (default) — never mirror.
.auto — mirror only when the active camera is .front. Update mirror.cameraPosition = cameraCapturer.position on camera switch.
.on — always mirror.
With mirroring enabled, render both the local preview and the remote stream with RTCMLVideoViewWrapper(track:) — no mirror: argument.
referenceImageData - Optional reference image Data
connection (optional) - Connection configuration
iceServers - STUN/TURN server URLs (default: Google STUN)
connectionTimeout - Connection timeout in seconds (default: 15)
rtcConfiguration - Custom RTCConfiguration for advanced WebRTC tuning
media (optional) - Media configuration
video.maxBitrate - Max bitrate in bps (default: 2,500,000)
video.minBitrate - Min bitrate in bps (default: 300,000)
video.maxFramerate - Max framerate (default: 30)
video.preferredCodec - Preferred video codec (default: “VP8”)
Returns:RealtimeMediaStream — the transformed remote stream containing an optional videoTrack you can renderFor image-capable models, pass the reference image data on DecartPrompt:
let imageData = try Data(contentsOf: characterImageURL)let manager = try client.createRealtimeManager( options: RealtimeConfiguration( model: Models.realtime(.lucy_v2v_14b_rt), initialPrompt: DecartPrompt( text: "Substitute the character in the video with the person in the reference image.", referenceImageData: imageData, enrich: true ) ))
Set initialPrompt (with referenceImageData for image-capable models) so the first frame is already transformed — otherwise viewers briefly see the raw camera feed.
Monitor connection state, service status, generation ticks, and session ID using the events AsyncStream:
// Observe state changesTask { for await state in manager.events { switch state.connectionState { case .connecting: showLoadingIndicator() case .connected: hideLoadingIndicator() case .generating: showGeneratingIndicator() case .reconnecting: showReconnectingIndicator() case .disconnected: showReconnectButton() case .idle: break case .error: showError() } // Track generation progress if let tick = state.generationTick { showGenerationTime("\(tick)s") } // Track session ID if let sessionId = state.sessionId { print("Session: \(sessionId)") } // Track queue position if let position = state.queuePosition, let size = state.queueSize { showQueueStatus("Position \(position) of \(size)") } // Track service status switch state.serviceStatus { case .enteringQueue: showQueueMessage() case .ready: hideQueueMessage() case .unknown: break } }}
The SDK automatically reconnects when an unexpected disconnection occurs (e.g., network interruption). During auto-reconnect, the connection state transitions to .reconnecting while the SDK retries with exponential backoff (up to 5 attempts, max 10s delay).When auto-reconnect succeeds, a new RealtimeMediaStream is emitted via remoteStreamUpdates. You must rebind your UI to the new stream’s video track:
Task { for await newRemoteStream in manager.remoteStreamUpdates { // Update your UI with the new video track self.remoteVideoTrack = newRemoteStream.videoTrack }}
Auto-reconnect is not triggered on user-initiated disconnect(), permanent errors (401/403, invalid key, expired session), or after all retries are exhausted. If all retries fail, the state moves to .error.
Two layers report network health on a shared .good | .fair | .poor | .critical scale: a preflight check before connecting, and an in-session signal while connected.
While connected, the SDK derives a smoothed verdict from live connection stats (latency, packet loss, upstream bandwidth, frame rate) and tells you the limiting factor. The stream yields on debounced level changes; getConnectionQuality() returns the live snapshot whose metrics refresh on every poll:
Task { for await report in manager.connectionQualityUpdates { // report.limitingFactor: .bandwidth | .latency | .loss | .stall | .cpu | .none // report.metrics: rttMs, fps, packetLoss, availableUpstreamKbps, ... updateBadge(report.quality) }}let latest = manager.getConnectionQuality() // ConnectionQualityReport? — nil before the first sample
In-session quality is on by default. Opt out by passing observability: .init(connectionQualityEnabled: false) in RealtimeConfiguration.
Network RTT alone doesn’t reflect the latency users actually feel — a session can read .good while still feeling laggy. Set debugQuality: true to measure the real camera→display latency: the SDK stamps a pixel marker into each outgoing frame and reads it back off the rendered output, surfacing startup (ttffMs), steady-state (g2gMs), and end-to-end frame drops (g2gDropRatio). When present, glass-to-glass drives the latency verdict instead of RTT.
Diagnostic only. The marker is visible (bottom-left of the published and rendered video) and adds per-frame pixel work — don’t enable it for production / end-user sessions. The debugQuality flag must match on both createLocalCameraStream and RealtimeConfiguration.
For a measured verdict before connecting, use the deep probe — it briefly opens a real session with a synthetic source, measures glass-to-glass, then tears it down. Requires a model and costs a short session:
LiveKit factory that opens the camera and produces a LocalVideoTrack. Capture starts immediately — there is no separate startCapture() call.Parameters:
Toggles between front and back cameras. Access via videoTrack.capturer as? CameraCapturer. After switching, update mirror.cameraPosition = cameraCapturer.position to keep MirrorMode.auto in sync.