Voice-conversion benchmark

Tools to evaluate real-time voice-conversion (VC) programs end to end: capture each tool's converted audio, then score its quality.

Demo page for results.

What it does

1. Capture (record_vc.py, capture/) — a black-box harness. It plays each clean source clip out through a virtual audio cable into a running VC tool, records what comes back, trims the silence, and saves it. Works with any tool that takes audio in and produces audio out.

2. Score (score_vc.py, benchmarks/) — rates the captured outputs on four reference-free metrics and writes per-file and summary CSVs:

   Metric	Measures	Direction
   WER / CER	intelligibility (faster-whisper vs. ground-truth transcript)	lower
   UTMOSv2	naturalness (neural MOS predictor)	higher
   DNSMOS	signal quality / artifacts (SIG/BAK/OVRL/P.808)	higher
   SECS	speaker identity (ECAPA embeddings)	see below

   SECS reports src_sim (distance from the source voice, lower = identity moved) and tgt_cons (consistency of a tool's output voice, higher = stable). Drop reference clips of the target voice into a folder and set target_secs = true to also get tgt_sim (cosine to the target, higher = better).

Requirements

Python 3.13 suggested for running the project.

pip install -r requirements.txt

Capture also needs two virtual audio cables (e.g. VB-CABLE) wired to the VC tool, set in config.toml.

UTMOSv2 auto-downloads its weights via wget, which isn't installed on Windows by default, so place them once by hand: download fold0_s42_best_model.pth from the UTMOSv2 HuggingFace repo into ~/.cache/utmosv2/models/fusion_stage3/.

Layout

config.toml              config
record_vc.py/capture/    capture harness
score_vc.py/benchmarks/  quality benchmark
data/corpus/             clean sources: <corpus>/{Female,Male,transcripts}
data/converted/          captured VC outputs: <program>/<model>/<id>.wav
data/scores/             benchmark CSV outputs

Usage

Capture a tool's outputs (the tool must be running and routed through the cables):

python record_vc.py --list-devices                 # inspect audio devices first
python record_vc.py --program seedvc --model base
python record_vc.py --program seedvc --limit 3     # quick test run
python record_vc.py --help                         # see all configs

Score everything in data/converted/:

python score_vc.py
python score_vc.py --program seedvc         # one tool
python score_vc.py --metrics wer,dnsmos     # subset of metrics
python score_vc.py --help                   # see all configs

Output

data/scores/ gets per_file.csv (one row per output, all metrics + ASR text), summary.csv (mean per tool), and summary_by_corpus.csv. A ranked table is also printed to the console.

Configuration

All defaults live in config.toml: audio devices and capture timing under [capture.*], the metric set and SECS target options under [benchmark.*], and shared input/output paths under [paths].