> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mascot.bot/llms.txt
> Use this file to discover all available pages before exploring further.

# Natural Lip Sync

> Create more realistic mouth movements with intelligent viseme processing

# Natural Lip Sync

Natural lip sync post-processes the viseme stream to produce more natural
speech animation. Instead of snapping to every phoneme, it merges similar
adjacent shapes and preserves the distinctive ones — the way a real mouth
blends sounds during fast speech.

## Why

Hitting every phoneme precisely looks robotic. In natural speech the mouth
does not have time to fully form each shape; shapes blend, sometimes to the
point of not changing at all. The processor mimics that blending while
protecting the visually critical shapes (w/u, o, r, l, f/v, and bilabials
p/b/m) so articulation still reads.

## Enable it

It is one option on `useMascotPlayback`. This works for every path — offline,
mic, and realtime:

```tsx theme={null}
import { useMascotPlayback } from "@mascotbot/react/rive";

const playback = useMascotPlayback({ enableNaturalLipSync: true });
```

That uses `DEFAULT_NATURAL_LIPSYNC_CONFIG`. To tune it, pass a
`naturalLipSyncConfig`:

```tsx theme={null}
const playback = useMascotPlayback({
  enableNaturalLipSync: true,
  naturalLipSyncConfig: CONVERSATION, // a STABLE reference — see warning below
});
```

<Warning>
  `naturalLipSyncConfig` must be a **stable reference** — a module constant, or
  memoized with `useState` / `useMemo`. A new object literal on every render
  reinitializes playback and lip sync breaks after the first audio chunk. This
  is the single most common integration bug; see
  [Troubleshooting](/libraries/react-troubleshooting).
</Warning>

## Configuration

`NaturalLipSyncConfig` (all fields optional — unspecified fields fall back to
`DEFAULT_NATURAL_LIPSYNC_CONFIG`):

| Field                           | Type                | Default                       | Meaning                                                                                                                                                    |
| ------------------------------- | ------------------- | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `minVisemeInterval`             | `number` (ms)       | `60`                          | Minimum time between visemes; closer ones merge (\~16 visemes/s max).                                                                                      |
| `mergeWindow`                   | `number` (ms)       | `80`                          | Look-ahead window for finding similar visemes to merge.                                                                                                    |
| `keyVisemePreference`           | `number` 0–1        | `0.7`                         | Strength of preference for distinctive shapes. Higher keeps more.                                                                                          |
| `preserveSilence`               | `boolean`           | `true`                        | Keep all silence visemes (recommended).                                                                                                                    |
| `similarityThreshold`           | `number` 0–1        | `0.6`                         | How similar two visemes must be to merge. Higher merges less.                                                                                              |
| `preserveCriticalVisemes`       | `boolean`           | `true`                        | Never skip critical shapes (u/o/l/v/p/b/m).                                                                                                                |
| `criticalVisemeMinDuration`     | `number` (ms)       | `0`                           | Hold critical visemes at least this long (opt-in; `0` disables).                                                                                           |
| `criticalVisemeAbsorbThreshold` | `number` (ms)       | `30`                          | If holding a critical viseme shrinks the next one below this, drop the successor instead of flashing it. Only active when `criticalVisemeMinDuration > 0`. |
| `criticalVisemeIds`             | `readonly number[]` | `DEFAULT_CRITICAL_VISEME_IDS` | Which viseme ids are "critical".                                                                                                                           |

`DEFAULT_CRITICAL_VISEME_IDS` is `[7, 8, 13, 14, 18, 21]` — w/u, o, r, l, f/v,
and bilabials. Both `DEFAULT_NATURAL_LIPSYNC_CONFIG` and
`DEFAULT_CRITICAL_VISEME_IDS` are exported so you can derive from them:

```ts theme={null}
import { DEFAULT_NATURAL_LIPSYNC_CONFIG, DEFAULT_CRITICAL_VISEME_IDS } from "@mascotbot/react/rive";

// Hold s/z too, otherwise defaults
export const WITH_SIBILANTS = {
  ...DEFAULT_NATURAL_LIPSYNC_CONFIG,
  criticalVisemeIds: [...DEFAULT_CRITICAL_VISEME_IDS, 15],
};
```

## Presets

Define presets as **module-level constants** so the reference is stable:

```ts theme={null}
// Natural conversation — a good default for voice AI
export const CONVERSATION = {
  minVisemeInterval: 60,
  mergeWindow: 80,
  keyVisemePreference: 0.7,
  preserveSilence: true,
  similarityThreshold: 0.6,
  preserveCriticalVisemes: true,
} as const;

// Fast / excited speech — coarser merging
export const FAST_SPEECH = {
  minVisemeInterval: 90,
  mergeWindow: 120,
  keyVisemePreference: 0.6,
  preserveSilence: true,
  similarityThreshold: 0.4,
  preserveCriticalVisemes: true,
  criticalVisemeMinDuration: 70,
} as const;

// Clear articulation — education / language learning
export const EDUCATIONAL = {
  minVisemeInterval: 40,
  mergeWindow: 50,
  keyVisemePreference: 0.9,
  preserveSilence: true,
  similarityThreshold: 0.8,
  preserveCriticalVisemes: true,
} as const;
```

```tsx theme={null}
const playback = useMascotPlayback({ enableNaturalLipSync: true, naturalLipSyncConfig: CONVERSATION });
```

Start from `CONVERSATION`. Raise `minVisemeInterval` / `mergeWindow` for
smoother (lazier) motion; lower them for crisper articulation.

## Without React

The processor is exported from `@mascotbot/core/rive` as a class
and a function:

```ts theme={null}
import { processNaturalLipSync, NaturalLipSyncProcessor } from "@mascotbot/core/rive";
```

When you enable `enableNaturalLipSync` on `MascotPlayback` /
`useMascotPlayback`, this runs internally — you only call it directly if you
post-process visemes outside the playback engine.

## Next

<Columns cols={3}>
  <Card title="React hooks" icon="react" href="/libraries/react-hooks">
    `useMascotPlayback` options.
  </Card>

  <Card title="Visemes & the timeline" icon="waveform-lines" href="/concepts/visemes-and-timeline">
    What is being processed.
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/libraries/react-troubleshooting">
    The stable-reference bug.
  </Card>
</Columns>