Mascotbot Lipsync SDK Quickstart

This gets you from an installed package to a talking avatar. If you have not installed yet, do Installation first.

1. Mount the provider

<MascotProvider> initializes a single LipsyncClient for your app and exposes it through context.

// app/layout.tsx (or wherever you mount providers)
"use client";
import { MascotProvider } from "@mascotbot/react";

export default function Layout({ children }: { children: React.ReactNode }) {
  return <MascotProvider apiKey="mascot_pub_…">{children}</MascotProvider>;
}

Anywhere inside it, useMascot() gives you the client and status, and useProcessAudio() fetches a URL, decodes, resamples to 16 kHz, and runs inference once:

"use client";
import { useMascot, useProcessAudio } from "@mascotbot/react";

export function Demo() {
  const { status, error } = useMascot();
  const { result, loading } = useProcessAudio("/audio/greeting.wav");

  if (status === "initializing") return <p>Loading SDK…</p>;
  if (status === "error") return <p>{error?.message}</p>;
  if (loading || !result) return <p>Processing audio…</p>;

  // result.timeline   — serializable viseme timeline (the offline artifact)
  // result.durationMs — total audio duration
  // result.speechMs   — non-silent ms detected
  return <pre>{JSON.stringify(result.timeline, null, 2)}</pre>;
}

result.timeline is a VisemeTimeline — hand it to playback below, or JSON.stringify it to persist and replay later with zero reprocessing.

2. Drive a Rive avatar

Wire the timeline into a Rive avatar’s mouth state machine with the /rive subpath:

"use client";
import { MascotProvider, useMascot, useProcessAudio } from "@mascotbot/react";
import {
  MascotProvider,
  Mascot,
  MascotRive,
  useMascotPlayback,
  Fit,
  Alignment,
} from "@mascotbot/react/rive";

function App() {
  return (
    <MascotProvider apiKey="mascot_pub_…">
      <MascotProvider>
        <Mascot
          src="/mascot-fox.riv"
          layout={{ fit: Fit.Contain, alignment: Alignment.Center }}
        >
          <MascotRive />
          <SamplePlayer />
        </Mascot>
      </MascotProvider>
    </MascotProvider>
  );
}

function SamplePlayer() {
  const { status } = useMascot();
  const { result } = useProcessAudio("/audio/greeting.wav");
  const playback = useMascotPlayback({ enableNaturalLipSync: true });

  function play() {
    if (status !== "ready" || !result) return;
    new Audio("/audio/greeting.wav").play().catch(() => {});
    playback.setTimeline(result.timeline); // offline timeline → mouth
    playback.play();
  }

  return (
    <button onClick={play} disabled={status !== "ready" || !result}>
      Play
    </button>
  );
}

useProcessAudio runs inference once. Persist result.timeline (it is plain JSON) and on later loads skip decode + inference entirely: playback.setTimeline(parseTimeline(JSON.parse(stored))). See Offline lip sync.

Rive avatar requirements

If you author your own .riv file:

Element	Requirement
Artboard name	`Character`
State machine name	`mascotStateMachine`
Mouth inputs	Number inputs `100`–`118` (viseme ids)
Emotion inputs (optional)	`is_speaking`, `eyes_smile`
Stress input (optional)	`stress` (number)

The SDK writes only those inputs. Any other input, data binding, or event on the file is yours to drive on the raw rive instance (useMascotRive().rive) — see Rive co-existence. Or skip authoring entirely and use a ready-made mascot.

3. Live microphone input

Drive the avatar from the user’s microphone in real time with useLipsyncStream:

"use client";
import { useEffect, useState } from "react";
import { useMascot } from "@mascotbot/react";
import { useMascotPlayback, useLipsyncStream } from "@mascotbot/react/rive";

function MicAvatar() {
  const { client, status } = useMascot();
  const playback = useMascotPlayback({ stream: true, enableNaturalLipSync: true });
  const [active, setActive] = useState(false);
  const isLive = active && status === "ready" && !!client;

  const { error } = useLipsyncStream({
    client,
    playback,
    source: { kind: "mic" },
    enabled: isLive, // gates getUserMedia + the audio graph without unmount
  });

  useEffect(() => {
    if (status !== "ready") setActive(false);
  }, [status]);

  return (
    <>
      <button onClick={() => setActive((v) => !v)} disabled={status !== "ready"}>
        {active ? "Stop mic" : "Start mic"}
      </button>
      {error && <p>{error.message}</p>}
    </>
  );
}

The audio worklet is embedded in the SDK and served from a Blob URL by default — there is no file to copy. Pass workletUrl only if your CSP forbids worker-src blob:. The same hook handles realtime AI providers via source: { kind: "mediaStream", stream } — see Realtime providers.

4. Vanilla JavaScript

Not using React? Use @mascotbot/core directly:

import { LipsyncClient, parseTimeline } from "@mascotbot/core";

const client = await LipsyncClient.init({
  apiKey: "mascot_pub_…",
  userId: "user_42", // optional, for accurate usage attribution
});

// Pre-recorded audio (16 kHz mono Float32 in [-1, 1])
const { timeline } = await client.processAudio(audioBuffer);
localStorage.setItem("greeting.vtl", JSON.stringify(timeline)); // persist
// later: parseTimeline(JSON.parse(localStorage.getItem("greeting.vtl")!))

// Live streaming: push 25 ms (400-sample) windows one at a time
const session = client.createStreamingSession();
const frame = await session.pushWindow(audioWindow);
console.log(frame.visemeId, frame.silenceDetected);

await client.stop(); // release resources when done

For the Rive engine without React, import from @mascotbot/core/rive — see Core SDK.

Error handling

useMascot() exposes status and error. Branch on error.code, not the subclass:

import { RefusedError, NetworkError, EngineError } from "@mascotbot/react";

function StatusUI() {
  const { status, error } = useMascot();
  if (status !== "error" && status !== "refused") return null;
  if (error instanceof RefusedError) {
    if (error.code === "key_disabled") return <ReSubscribe />;
    if (error.code === "dev_key_on_public_domain") return <UseProdKey />;
  }
  if (error instanceof NetworkError) return <p>Network issue — retry</p>;
  if (error instanceof EngineError) return <p>Inference error — refresh</p>;
  return <p>{error?.message}</p>;
}

Full matrix: Error codes.

Offline lip sync

Generate → persist → replay.

React hooks

The full hook reference.

Realtime providers

OpenAI, Gemini, ElevenLabs.

Getting Started

Core concepts

React SDK

Core SDK (vanilla)

Realtime providers

Reference

Ready-made Mascots

Mascotbot Lipsync SDK Quickstart - Animate a Rive Avatar in Minutes

1. Mount the provider

2. Drive a Rive avatar

Rive avatar requirements

3. Live microphone input

4. Vanilla JavaScript

Error handling

Next

Offline lip sync

React hooks

Realtime providers

​1. Mount the provider

​2. Drive a Rive avatar

​Rive avatar requirements

​3. Live microphone input

​4. Vanilla JavaScript

​Error handling

​Next

Offline lip sync

React hooks

Realtime providers

1. Mount the provider

2. Drive a Rive avatar

Rive avatar requirements

3. Live microphone input

4. Vanilla JavaScript

Error handling

Next