feat: augmentation robustness pass for image and video benchmarks

[dylanuys]

feat: augmentation robustness pass for image and video benchmarks

Summary

Adds a second evaluation pass to `gasbench run` that re-evaluates the same samples under a fixed suite of real-world image/video degradations, producing a dedicated `aug_sn34_score`, a per-sample probability-degradation signal, and an `augmentation_robustness` ratio alongside the existing base score.

The motivation is that deepfake detectors deployed in the wild never see pristine generated content — every image or video passes through at least one lossy encode/decode cycle before a human or system can flag it. A model that achieves a high base sn34_score but collapses under JPEG recompression or platform re-encoding is not actually useful; this PR makes that failure mode visible and scoreable.

---

Background and motivation

Several detection robustness studies establish that codec and JPEG compression are by far the dominant real-world degradation sources for forensic detectors:

FaceForensics++ (Rössler et al., ICCV 2019) introduced the canonical evaluation protocol: train and test at two H.264 compression levels — CRF 23 ("c23", light, YouTube-tier) and CRF 40 ("c40", heavy, WhatsApp/Messenger-tier). Nearly every subsequent deepfake detection paper reports results on these two tracks because platform re-encoding is the single most common reason detector performance degrades between the lab and production.

DeeperForensics-1.0 (Jiang et al., CVPR 2020) extended this with a seven-distortion robustness benchmark and showed that blur and noise have far less impact on detector performance degradation than codec/JPEG artifacts — motivating our decision to drop Gaussian blur from the suite.

Social media pipeline studies (Verdoliva et al., 2020; Gragnaniello et al., 2021) showed that Instagram, WhatsApp, and Twitter each apply chroma-subsampled JPEG at platform-specific quality settings (roughly q55–85) and that this single operation destroys the high-frequency DCT-domain fingerprints most detectors rely on.

---

What changes

New augmentation functions (processing/transforms.py)

apply_robustness_augmentations (image)

A fixed pipeline grounded in the social media compression literature. All parameters are pinned constants — not CLI args — to ensure aug_sn34_score values are comparable across runs:

Step	Operation	Rationale
1	Bilinear downscale 0.5× then upscale	Thumbnail/CDN resize pipeline; destroys sub-pixel forensic traces
2	JPEG roundtrip at quality 55	First platform upload (WhatsApp ≈ 55, Telegram ≈ 55–65)
3	WebP roundtrip at quality 75	CDN re-host and browser delivery (Google, Twitter/X serve WebP directly to browsers; our browser extension receives WebP from CDNs)
4	JPEG roundtrip at quality 80	Re-share / re-host recompression (Instagram ≈ 78–85, Twitter ≈ 85)

Chroma subsampling is pinned to 4:2:0 throughout (matching platform behavior).

Gaussian blur was removed because it models optical/focus degradation at capture time, not anything in a social media distribution pipeline, and because DeeperForensics empirically showed it has lower impact on detector performance than compression artifacts.

apply_video_robustness_augmentations (video)

Implements the FaceForensics++ evaluation protocol — full H.264 encode/decode roundtrip at CRF 23 (YouTube-tier). Cascade of backends:

1. ffmpeg (primary) — real -crf flag, forces even W/H for libx264 yuv420p compatibility
2. cv2 avc1 (fallback) — maps CRF [18, 51] → cv2 quality [100, 0]
3. Per-frame JPEG (last resort) — preserves chroma subsampling artifacts if no video codec available

CRF	Platform analogue
23	YouTube, high-quality streaming (FF++ c23)
40	WhatsApp, Messenger heavy compression (FF++ c40)

Suite params (CRF, FPS, scale factor) are pinned constants in code for benchmark integrity.

Second evaluation pass (image_bench.py, video_bench.py)

When --n-aug-per-dataset N is set, a second pass iterates the same N samples per dataset (same seed → same files → same sample_ids) through the robustness transform before inference. Aug-pass rows are written to the same tracker and parquet with aug_pass=True.

The two passes are fully decoupled — base metrics, per-dataset accuracy tables, and the existing sn34_score are computed on base-pass rows only. Aug rows are filtered out before any existing aggregation path.

Deterministic aug seed: the aug transform seed defaults to 42 when --seed is not provided, ensuring reproducible results without requiring an explicit seed. DatasetIterator seed is unchanged so sample selection and the sample_id join remain consistent.

Metrics (recording.py)

When aug rows are present, compute_metrics_from_df computes:

Field	Meaning
aug_sn34_score	sn34 computed on augmented samples (uses same provenance weights as base)
augmentation_robustness	aug_sn34 / base_sn34 — ratio close to 1.0 = robust
aug_paired_samples	Samples matched across base and aug passes via sample_id
aug_mean_prob_degradation	Mean drop in P(correct class) per paired sample
aug_p95_prob_degradation	95th-percentile degradation (captures tail behaviour)

Per-sample degradation is computed by joining on sample_id (SHA-256 hash of file-identity fields), so the same file evaluated clean and augmented is always paired correctly.

CLI flags

--n-aug-per-dataset N    Number of samples per dataset for the robustness pass (default: 0, disabled)
--aug-weight W           Weight of aug_sn34_score in any blended downstream scoring (default: 0.2)
--aug-cache-dir PATH     Cache pre-augmented arrays to disk; subsequent runs load from cache
                         instead of re-augmenting (recommended for repeated bmcore runs)

Suite-spec parameters (JPEG quality, scale factor, WebP quality, CRF) are intentionally not exposed as CLI flags — they are pinned constants in code to ensure aug_sn34_score values are comparable across runs.

Example — image benchmark with aug robustness pass, 100 samples per dataset, results cached:

gasbench run --image-model ./my_model/ --n-aug-per-dataset 100 --aug-cache-dir /cache/aug

Aug array cache (--aug-cache-dir)

First run: augmented arrays written to {aug_cache_dir}/{sample_id[:2]}/{sample_id}_v1.npy (2-char prefix sharding for filesystem scalability). Subsequent runs load from disk — no ffmpeg, no PIL, no recompute. Atomic write via temp file + os.replace, safe under concurrent worker threads. Version suffix (img_v1 / vid_v1) auto-invalidates on suite changes.

---

What was explicitly excluded

• Gaussian blur — models optical degradation at capture, not distribution. DeeperForensics shows it is among the least impactful distortions on detector performance.
• Additive noise — same reasoning; not present in platform pipelines.
• Adversarial perturbations — out of scope; these are not real-world transforms.
• Color jitter / random flips — change the forensic content, not the distribution channel artifacts.

---

References

1. Rössler, A. et al. FaceForensics++: Learning to Detect Manipulated Facial Images. ICCV 2019.
2. Jiang, L. et al. DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection. CVPR 2020.
3. Verdoliva, L. Media Forensics and DeepFakes: an Overview. IEEE J-STSP 2020.
4. Gragnaniello, D. et al. Are GAN generated images easy to detect? A critical analysis of the state-of-the-art. ICME 2021.

🤖 Generated with Claude Code