Skip to content

Let the mascot join Google Meet as a rendered character avatar #1273

@graycyrus

Description

@graycyrus

Summary

Allow OpenHuman's mascot to join Google Meet calls as a visible character avatar that other participants can see and interact with.

Problem

The mascot cannot yet participate inside a real meeting as a visible character.

What exists today:

  • The mascot renders as SVG in app/src/features/human/Mascot/Ghosty.tsx
  • The Tauri layer has Google Meet-specific recipe scaffolding in app/src-tauri/recipes/google-meet/**
  • The account-webview capability allowlist includes https://meet.google.com/*
  • The desktop shell enables WebRTC/browser features for embedded call surfaces

What is missing:

  • No pipeline to turn the mascot SVG into a live video feed for the call
  • No Meet-specific integration flow for choosing the mascot as the active camera surface
  • No call UX for interacting with the mascot as a speaking/listening character during the meeting

Solution

Build a Google Meet avatar-call flow that uses the mascot SVG as the rendered participant video feed.

High-level approach:

  • Open/join Google Meet through the existing embedded webview/account path
  • Render the mascot into a live video surface that Meet can consume as the outgoing camera feed
  • Keep the mascot expressive during the call (listening, thinking, speaking)
  • Let the mascot act as the visible face of the agent in the meeting

Technical scope:

  • Convert the mascot SVG render into a live video stream suitable for Meet camera input (canvas → MediaStream, virtual camera, or another browser-compatible path)
  • Coordinate mascot expression/lip-sync state with meeting audio/interaction state
  • Integrate with the existing embedded webview system

Acceptance criteria

  • The mascot can be rendered as the outgoing visible participant/avatar surface in a Google Meet call
  • The outgoing call video is derived from the mascot/SVG rendering system rather than a static placeholder
  • The mascot reflects basic live call states (listening, speaking) while in the meeting
  • The design integrates with the existing Meet webview/recipe infrastructure
  • Failure cases and unsupported environments degrade cleanly
  • Tests and integration coverage added for the Meet-specific avatar flow
  • Diff coverage ≥ 80%
  • Documented so contributors understand the Meet avatar architecture

Related

Metadata

Metadata

Assignees

Labels

featureNet-new user-facing capability or product behavior.priority: mediumDefault priority for planned work.react-uiReact app work in app/src: pages, components, providers, store, and UX.tauri-shellDesktop host work in app/src-tauri: windowing, relay, sidecar lifecycle.voiceVoice features and audio workflows in src/openhuman/voice/ and app/src/features/voice/.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions