Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.mascot.bot/llms.txt

Use this file to discover all available pages before exploring further.

This gets you from an installed package to a talking avatar. If you have not installed yet, do Installation first.

​
1. Mount the provider

<MascotProvider> initializes a single LipsyncClient for your app and exposes it through context. 
// app/layout.tsx (or wherever you mount providers)
"use client";
import { MascotProvider } from "@mascotbot/react";

export default function Layout({ children }: { children: React.ReactNode }) {
  return <MascotProvider apiKey="mascot_pub_…">{children}</MascotProvider>;
}
Anywhere inside it, useMascot() gives you the client and status, and useProcessAudio() fetches a URL, decodes, resamples to 16 kHz, and runs inference once: 
"use client";
import { useMascot, useProcessAudio } from "@mascotbot/react";

export function Demo() {
  const { status, error } = useMascot();
  const { result, loading } = useProcessAudio("/audio/greeting.wav");

  if (status === "initializing") return <p>Loading SDK…</p>;
  if (status === "error") return <p>{error?.message}</p>;
  if (loading || !result) return <p>Processing audio…</p>;

  // result.timeline   — serializable viseme timeline (the offline artifact)
  // result.durationMs — total audio duration
  // result.speechMs   — non-silent ms detected
  return <pre>{JSON.stringify(result.timeline, null, 2)}</pre>;
}
result.timeline is a VisemeTimeline — hand it to playback below, or JSON.stringify it to persist and replay later with zero reprocessing.

​
2. Drive a Rive avatar

Wire the timeline into a Rive avatar’s mouth state machine with the /rive subpath: 
"use client";
import { MascotProvider, useMascot, useProcessAudio } from "@mascotbot/react";
import {
  MascotProvider,
  Mascot,
  MascotRive,
  useMascotPlayback,
  Fit,
  Alignment,
} from "@mascotbot/react/rive";

function App() {
  return (
    <MascotProvider apiKey="mascot_pub_…">
      <MascotProvider>
        <Mascot
          src="/mascot-fox.riv"
          layout={{ fit: Fit.Contain, alignment: Alignment.Center }}
        >
          <MascotRive />
          <SamplePlayer />
        </Mascot>
      </MascotProvider>
    </MascotProvider>
  );
}

function SamplePlayer() {
  const { status } = useMascot();
  const { result } = useProcessAudio("/audio/greeting.wav");
  const playback = useMascotPlayback({ enableNaturalLipSync: true });

  function play() {
    if (status !== "ready" || !result) return;
    new Audio("/audio/greeting.wav").play().catch(() => {});
    playback.setTimeline(result.timeline); // offline timeline → mouth
    playback.play();
  }

  return (
    <button onClick={play} disabled={status !== "ready" || !result}>
      Play
    </button>
  );
}
useProcessAudio runs inference once. Persist result.timeline (it is plain JSON) and on later loads skip decode + inference entirely: playback.setTimeline(parseTimeline(JSON.parse(stored))). See Offline lip sync.

​
Rive avatar requirements

If you author your own .riv file:
ElementRequirement
Artboard nameCharacter
State machine namemascotStateMachine
Mouth inputsNumber inputs 100–118 (viseme ids)
Emotion inputs (optional)is_speaking, eyes_smile
Stress input (optional)stress (number)
The SDK writes only those inputs. Any other input, data binding, or event on the file is yours to drive on the raw rive instance (useMascotRive().rive) — see Rive co-existence. Or skip authoring entirely and use a ready-made mascot.

​
3. Live microphone input

Drive the avatar from the user’s microphone in real time with useLipsyncStream: 
"use client";
import { useEffect, useState } from "react";
import { useMascot } from "@mascotbot/react";
import { useMascotPlayback, useLipsyncStream } from "@mascotbot/react/rive";

function MicAvatar() {
  const { client, status } = useMascot();
  const playback = useMascotPlayback({ stream: true, enableNaturalLipSync: true });
  const [active, setActive] = useState(false);
  const isLive = active && status === "ready" && !!client;

  const { error } = useLipsyncStream({
    client,
    playback,
    source: { kind: "mic" },
    enabled: isLive, // gates getUserMedia + the audio graph without unmount
  });

  useEffect(() => {
    if (status !== "ready") setActive(false);
  }, [status]);

  return (
    <>
      <button onClick={() => setActive((v) => !v)} disabled={status !== "ready"}>
        {active ? "Stop mic" : "Start mic"}
      </button>
      {error && <p>{error.message}</p>}
    </>
  );
}
The audio worklet is embedded in the SDK and served from a Blob URL by default — there is no file to copy. Pass workletUrl only if your CSP forbids worker-src blob:. The same hook handles realtime AI providers via source: { kind: "mediaStream", stream } — see Realtime providers.

​
4. Vanilla JavaScript

Not using React? Use @mascotbot/core directly: 
import { LipsyncClient, parseTimeline } from "@mascotbot/core";

const client = await LipsyncClient.init({
  apiKey: "mascot_pub_…",
  userId: "user_42", // optional, for accurate usage attribution
});

// Pre-recorded audio (16 kHz mono Float32 in [-1, 1])
const { timeline } = await client.processAudio(audioBuffer);
localStorage.setItem("greeting.vtl", JSON.stringify(timeline)); // persist
// later: parseTimeline(JSON.parse(localStorage.getItem("greeting.vtl")!))

// Live streaming: push 25 ms (400-sample) windows one at a time
const session = client.createStreamingSession();
const frame = await session.pushWindow(audioWindow);
console.log(frame.visemeId, frame.silenceDetected);

await client.stop(); // release resources when done
For the Rive engine without React, import from @mascotbot/core/rive — see Core SDK.

​
Error handling

useMascot() exposes status and error. Branch on error.code, not the subclass: 
import { RefusedError, NetworkError, EngineError } from "@mascotbot/react";

function StatusUI() {
  const { status, error } = useMascot();
  if (status !== "error" && status !== "refused") return null;
  if (error instanceof RefusedError) {
    if (error.code === "key_disabled") return <ReSubscribe />;
    if (error.code === "dev_key_on_public_domain") return <UseProdKey />;
  }
  if (error instanceof NetworkError) return <p>Network issue — retry</p>;
  if (error instanceof EngineError) return <p>Inference error — refresh</p>;
  return <p>{error?.message}</p>;
}
Full matrix: Error codes.

​
Next

Offline lip sync

Generate → persist → replay.

React hooks

The full hook reference.

Realtime providers

OpenAI, Gemini, ElevenLabs.