Daydream Cloud: Remote GPU Processing in Scope

[emranemran]

Daydream Cloud: Remote GPU Processing in Scope

Summary

This PR adds a Cloud Mode toggle to Scope, enabling users to run video generation pipelines on remote cloud GPUs (H100s for now) instead of requiring a local GPU. Aside from Plugins and LoRAs, Local and Cloud mode are at feature parity.

Architecture Overview

Local Mode (Current)

┌─────────────────────────────────────────────────────────────┐
│                      User's Machine                          │
│  ┌─────────────┐     WebRTC      ┌────────────────────────┐ │
│  │   Browser   │ ◄─────────────► │    Scope Backend       │ │
│  │  (Frontend) │                 │  (Local GPU - RTX/etc) │ │
│  └─────────────┘                 └────────────────────────┘ │
│                                            │                 │
│                                     ┌──────┴──────┐          │
│                                     │    Spout    │          │
│                                     │  (Windows)  │          │
│                                     └─────────────┘          │
└─────────────────────────────────────────────────────────────┘

Cloud Mode

When to use: Desktop app where Spout I/O or local frame processing is needed.

How to enable: Start Scope, sign in, click "Connect to Cloud" in UI.

┌───────────────────────────────────────────────────────────────────────┐
│                           User's Machine                               │
│                                                                        │
│  ┌───────────┐        ┌─────────────────────────┐        ┌─────────┐  │
│  │  Browser  │ WebRTC │    Scope Backend        │ Spout  │ TouchD  │  │
│  │ (Frontend)│◄──────►│    (Client Mode)        │◄──────►│ Resolume│  │
│  └───────────┘        │                         │        │ etc.    │  │
│                       │  ┌───────────────────┐  │        └─────────┘  │
│                       │  │ CloudConnection   │  │                     │
│                       │  │ Manager           │  │                     │
│                       │  └────────┬──────────┘  │                     │
│                       └───────────┼─────────────┘                     │
└───────────────────────────────────┼───────────────────────────────────┘
                                    │
                          WebSocket │ + WebRTC
                          (signaling│ + video)
                                    │
                                    ▼
                       ┌─────────────────────────────────┐
                       │         Daydream Cloud            │
                       │  ┌───────────────────────────┐  │
                       │  │       fal_app.py          │  │
                       │  │  ┌─────────────────────┐  │  │
                       │  │  │   Scope Backend     │  │  │
                       │  │  │   (H100 GPU)        │  │  │
                       │  │  └─────────────────────┘  │  │
                       │  └───────────────────────────┘  │
                       └─────────────────────────────────┘

Flow:
1. User clicks "Connect to Cloud" in Settings panel
2. Local backend connects to Cloud backend via WebSocket
3. Local backend acts as WebRTC CLIENT to Cloud backend (role reversal!)
4. Frames from browser/Spout → local backend → Cloud → processed → back
5. Spout I/O works transparently (local backend handles it)

Key Components

Backend (scope-server)

File	Purpose
cloud_connection.py	WebSocket connection manager with request/response correlation
cloud_webrtc_client.py	WebRTC peer connection (client role) to fal.ai
cloud_track.py	MediaStreamTrack for bidirectional video relay
frame_processor.py	Modified to route frames to cloud when connected
webrtc.py	Extended to handle cloud mode offers

Cloud (fal.ai)

File	Purpose
fal_app.py	Serverless wrapper running Scope on fal.ai H100
FAL_DEPLOYMENT.md	Deployment guide for fal.ai

Frontend

File	Purpose
cloudAdapter.ts	WebSocket-based API client for fal.ai
cloudContext.tsx	React context for cloud mode state
useUnifiedWebRTC.ts	Unified hook supporting both local and cloud modes
useApi.ts	API hook that routes to local or cloud
CloudModeToggle.tsx	UI component for enabling cloud mode

API Endpoints

New endpoints for cloud connection management:

POST /api/v1/cloud/connect    - Connect to fal.ai cloud
POST /api/v1/cloud/disconnect - Disconnect from cloud
GET  /api/v1/cloud/status     - Get connection status and stats

When connected to cloud, existing endpoints are automatically proxied:

• POST /api/v1/pipeline/load → proxied to cloud
• GET /api/v1/pipeline/status → proxied to cloud
• GET /api/v1/pipelines/schemas → proxied to cloud
• POST /api/v1/webrtc/offer → handled locally (relay mode)

Features

Core Functionality

• Connect/disconnect to fal.ai cloud
• WebRTC video relay through local backend
• API call proxying via WebSocket
• Spout input/output in cloud mode
• Reference image upload to cloud
• Recording download from cloud

Authentication

Cloud mode requires Daydream authentication:

1. User signs in via OAuth flow
2. Frontend passes user ID to backend
3. Backend includes user ID in cloud connection
4. Events are tracked with user context in Kafka (only on Cloud backend - Local backend does not track anything)