Skip to content

fix(viewer): eliminate dual-decoder race and seek-flood jerkiness#13

Draft
Tailspin45 wants to merge 4 commits into
mainfrom
fix/video-player-jerky-halfframes
Draft

fix(viewer): eliminate dual-decoder race and seek-flood jerkiness#13
Tailspin45 wants to merge 4 commits into
mainfrom
fix/video-player-jerky-halfframes

Conversation

@Tailspin45

Copy link
Copy Markdown
Owner

Problem

  • Filmstrip + playback (▶▶/◀◀) in the file viewer is jerky on det_*.mp4 clips.
  • Neighbor panels in the 5-panel strip frequently render only the top half or stay empty (screenshot: panels 185/187/189 fill, 186/188 blank).
  • Frame counter and scrub slider mis-track on 24 / 25 / 29.97 / 60 fps clips.

Root cause

  • Dual-decoder race. A hidden `_thumbExtractorVid` seeked the same MP4 concurrently with the playback `hiddenVid`. The single hardware decoder serializes them; `onseeked` fires before the post-seek frame is fully decoded, so `drawImage` captures a half-decoded frame (top-half artifact). When the extractor's `seeked` never fires because `hiddenVid` preempts it, `_thumbExtractionBusy` stays true forever — empty panels.
  • Seek flood. `_scrubPlayTick` scheduled a new seek every `1000/fps` ms regardless of whether the previous seek had completed. The browser coalesces; playback stutters.
  • Duplicate renders. Both `timeupdate` and `seeked` ran `_updateScrubPosition` + `_updateFivePanel`, so every logical step produced multiple re-renders.
  • Hardcoded fps. `_initFrameScrubber` set `_videoFps = 30` unconditionally; `_frameTotalCount` and all `frame/fps` conversions drifted on non-30 fps clips.

Fix summary

  • Share the playback `hiddenVid` decoder for thumb capture. Remove the hidden extractor video, its canvas/ctx, and the busy-flag state machine.
  • Serialize thumb capture on a Promise chain (`_thumbChain = _thumbChain.then(...)`), with a generation counter so opening a new video aborts old chains. Thumb capture restores the user's playhead after a batch so the video doesn't jitter around.
  • Replace `_scrubPlayTick` with an async `_scrubPlayLoop` that `await`s each seek (via a new `_seekToFrame` helper that resolves on `seeked` + `requestVideoFrameCallback`, with a fallback settle and a 1.5 s safety timeout), then throttles to 1/fps only if the decoder beat real-time.
  • Consolidate rendering: `timeupdate` now only handles loop wrap-around; `seeked` drives a `requestAnimationFrame`-batched render so multiple seeked events in one frame collapse to one paint.
  • Detect real fps via requestVideoFrameCallback median-delta sampling; fall back to a new `GET /api/video/fps` Flask route that shells out to `ffprobe`; last resort 30. `_initFrameScrubber` no longer hardcodes 30.
  • Coerce `_jumpToFrame(f)` with `Math.round`; guard `scrubPlayToggle` against double-start.

`ffprobe` is not guaranteed on every host (bundled Electron, headless Pi). The route returns 404 on absence and the frontend falls back to the rVFC probe → 30 chain, so missing ffprobe is non-fatal.

Files changed

  • `static/telescope.js` — +165 / −113 (dual decoder removal, async scrub loop, rVFC seek, fps probe, render batcher).
  • `src/telescope_routes.py` — +83 / −0 (new `video_fps_route` + `/api/video/fps` registration).

Manual test plan

  • 29.97 fps det_*.mp4: click neighbor panel in filmstrip → both neighbor panels fill, no top-half artifact.
  • 29.97 fps det_*.mp4: click ▶▶ → smooth playback at decoder-native rate, no stutter.
  • Same clip: ◀◀ → smooth reverse.
  • Rapid-click panel 186 then 188 → both render fully; no stuck-busy panels.
  • 24 fps clip: frame counter matches `ffprobe`-reported `duration × fps`.
  • 60 fps clip: same.
  • Long MP4 (60+ s): click near end → no freeze.
  • Safari fallback path: temporarily gate rVFC off with `if (false && 'requestVideoFrameCallback' in HTMLVideoElement.prototype)` — confirm no regressions.
  • Host without ffprobe: `GET /api/video/fps?path=...` returns 404, frontend logs nothing alarming and fps defaults to 30.

- Share a single HTMLVideoElement decoder for both playback and filmstrip
  thumb capture. Drop the hidden extractor <video>, its canvas/ctx, and
  the onseeked-based busy flag. Thumb drain is now a serialized Promise
  chain with a generation counter so new-video loads abort old chains.
- Add _seekToFrame helper that resolves on seeked + requestVideoFrameCallback
  (with a timeout/safety fallback). Used by _stepFrame, _jumpToFrame,
  slider input, scrub-play, and thumb capture.
- Replace fixed-interval _scrubPlayTick with an async while-loop that
  awaits each seek, then throttles to 1/fps only if the decoder is faster
  than real-time. scrubPlayToggle guards against double-start.
- Render via a requestAnimationFrame batcher driven by 'seeked'; timeupdate
  is now loop-wrap only. Avoids duplicate _updateFivePanel calls per step.
- Detect real fps: rVFC median-delta probe first, then GET /api/video/fps
  (new ffprobe-backed Flask route), then fall back to 30. _initFrameScrubber
  no longer hardcodes 30.
- Round _jumpToFrame argument defensively.

ffprobe is optional on the host: the route returns 404 on absence and the
frontend falls back cleanly.
Comment thread src/telescope_routes.py
return jsonify({"error": "ffprobe timeout"}), 404
except Exception as exc:
logger.warning(f"[Telescope] video fps probe error: {exc}")
return jsonify({"error": str(exc)}), 404
- Opportunistic thumb capture on every _seekToFrame completion
- Drain queue during playback (in-place capture only, no extra seeks)
- Await initial seek(0) before extracting thumbs; wait for videoWidth>0
- Prime the pump: capture frame 0 synchronously on load
- Guard 5-panel render against premature canvas access
- _VIDDBG flag for silent debug logging
@Tailspin45

Copy link
Copy Markdown
Owner Author

Follow-up commit pushed: 8d4f9d4 — addresses the "all 5 panels blank" regression reported on a 4K ~14.9 fps clip.

What changed (static/telescope.js only, +96 / −11)

  • _seekToFrame now opportunistically captures the landed frame into the thumb cache on every seek completion. Playback, stepping, slider scrub, and programmatic jumps all populate the filmstrip for free.
  • _drainThumbQueue no longer bails outright during playback. While playing, it captures only thumbs that happen to match the current playhead and re-queues the rest (lets the play loop / _seekToFrame opportunistic capture do the work). While paused, behavior is unchanged (seek if needed, then capture).
  • _initAfterMeta now await _seekToFrame(vid, 0) before calling _extractFrameThumbs, so the playhead is settled and a real frame is decoded before the canvas is sized.
  • _extractFrameThumbs polls up to 1 s for videoWidth > 0 on large (4K) sources where the probe's play/pause dance leaves dimensions temporarily 0.
  • After extraction, frame 0 is captured synchronously via a new _captureLandedThumb helper → centre panel never opens empty.
  • _updateFivePanel guards against _thumbCanvas not being ready yet: draws placeholders and schedules one requestAnimationFrame retry.
  • New _VIDDBG const (default false) with a _vdbg(...) helper — silent in production, one-line flip for future regressions.

Test plan

  • Open 4K ~14.9 fps det_*.mp4 → all 5 panels render within ~1 s of open.
  • Click ▶▶ → filmstrip fills out as playback advances, no jerkiness.
  • Click ◀◀ after a forward pass → gaps fill in as the playhead reaches each frame.
  • Scrub slider rapidly → panels catch up, no stuck-blank neighbors.
  • Re-test 29.97 and 60 fps clips from PR fix(viewer): eliminate dual-decoder race and seek-flood jerkiness #13's original test plan — no regressions.
  • Flip _VIDDBG = true, reload, open a clip → console shows probe fps, canvas size, seek start/settle, and capture sizes.

- Capture happens after seeked event; previous render sees empty thumb
- Schedule a second render from capture paths when frame is in visible window
- Reject suspiciously small captures (decoder not ready) and re-queue
- Consolidate on single _scheduleFivePanelRender helper
@Tailspin45

Copy link
Copy Markdown
Owner Author

Follow-up commit pushed: `0f90278` — fixes "center panel stays blank after step/play" regression.

Root cause

The `seeked` listener fires BEFORE the rVFC callback that does the actual `drawImage` capture. So the rAF-batched `_updateFivePanel` ran with `_frameThumbs[N]` still empty for the frame just landed on, and nothing scheduled a second render after the capture completed — the center panel stayed blank until the next seek.

Fix (static/telescope.js only, +60 / −17)

  • New top-level `_scheduleFivePanelRender()` helper, replacing the inline closure inside `_initFrameScrubber`. Both the `seeked` listener and thumb-capture paths now call the same function.
  • `_captureLandedThumb` and `_captureCurrentAsThumb` schedule a render after a successful capture when the captured frame is within ±2 of `_currentFrame` (the visible filmstrip window). Out-of-window captures stay silent.
  • Reject captures whose JPEG data URL is < 500 bytes — those come from a not-ready decoder and would otherwise poison the cache with a blank thumb. Rejected frames are re-queued so the drainer retries once the decoder catches up.
  • `_seekToFrame` now fires an opportunistic 250 ms fallback inside the rVFC path: if rVFC hasn't fired by then, try drawing anyway (decoder is almost always ready). The outer 1500 ms safety timeout still applies as last resort.
  • Debug helpers under `_VIDDBG` (default false): `late render scheduled`, `blank capture rejected`, `captured`, `capture failed`.

Test plan

  • Open a 4K det_*.mp4. Centre panel populates within a frame of open.
  • Click ▶▶: centre panel always shows the current decoded frame (no blanks). Earlier panels populated; later panels populate as playback advances.
  • Click ◀◀: same behaviour in reverse.
  • Click a thumbnail two panels to the right → destination becomes the centre, immediately populated; its neighbours populate within ~500 ms.
  • Set `_VIDDBG = true` and reload: console shows `captured` lines with sane lengths, any `blank capture rejected` (< 500 bytes) is followed by a later successful capture at the same index, and `late render scheduled` fires once per in-window capture.

…ward play

- Guard _currentFrame against overwrite during programmatic seeks (fixes step-by-2)
- Forward play now uses native <video>.play() + requestVideoFrameCallback (smooth at native fps)
- Reverse play keeps step-based loop (seek-serialized)
- Center panel no longer blanks on first pass (capture fires on every rVFC tick)
@Tailspin45

Copy link
Copy Markdown
Owner Author

Follow-up commit: step-by-2 + smooth forward play

What changed

  • updateAfterSeek no longer overwrites _currentFrame from vid.currentTime while a programmatic seek is in flight (floating-point drift was pushing N → N+1, so the play loop advanced by two).
  • Forward play (▶▶) now calls vid.play() and drives the filmstrip from requestVideoFrameCallback — decode-native framerate, no per-frame seeks.
  • Reverse play (◀◀) stays step-based (no cross-codec negative playbackRate); it now prefetches the N-1 thumb each tick so the filmstrip stays ahead of the cursor.
  • _updateFivePanel skips _queueFivePanelThumbs while forward-playing — rVFC captures each landed frame, so the center populates organically without fighting the decoder.

Test plan (4K det_*.mp4)

  • Open 4K det_*.mp4 in the viewer — all 5 panels populate within 1–2 s.
  • Tap ▶▶: playback runs at native speed, center panel stays filled, no hesitation on the second pass.
  • Tap ◀◀: reverse steps frame-by-frame (slower than forward by design); center panel stays filled.
  • Tap single-frame and : advance exactly one frame each click (not two).
  • Drag the scrubber to an arbitrary frame: 5-panel snaps to the new center within ~1 frame.
  • Flip _VIDDBG = true locally and confirm logs: native play started, rVFC frame N (first 3), native play stopped.

Commit: 7f742fb

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants