Add pyro-detector-baseline experiment

experiments/temporal-models/pyro-detector-baseline/.dvc/.gitignore

@@ -0,0 +1,3 @@
+/config.local
+/tmp
+/cache

experiments/temporal-models/pyro-detector-baseline/.dvc/config

@@ -0,0 +1,5 @@
+[core]
+    remote = s3remote
+    analytics = false
+['remote "s3remote"']
+    url = s3://pyro-vision-rd/dvc/experiments/pyro-detector-baseline/

experiments/temporal-models/pyro-detector-baseline/.dvcignore

@@ -0,0 +1,3 @@
+# Add patterns of files dvc should ignore, which could improve
+# the performance. Learn more at
+# https://dvc.org/doc/user-guide/dvcignore

experiments/temporal-models/pyro-detector-baseline/.python-version

@@ -0,0 +1 @@
+3.11

experiments/temporal-models/pyro-detector-baseline/Makefile

@@ -0,0 +1,17 @@
+.PHONY: install lint format test help
+
+help: ## Show this help
+	@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-15s\033[0m %s\n", $$1, $$2}'
+
+install: ## Install dependencies and set up notebook output stripping
+	uv sync
+	uv run nbstripout --install
+
+lint: ## Run ruff linter on code and notebooks
+	uv run ruff check .
+
+format: ## Format code and notebooks with ruff
+	uv run ruff format .
+
+test: ## Run tests with pytest
+	uv run pytest tests/ -v

experiments/temporal-models/pyro-detector-baseline/README.md

@@ -0,0 +1,102 @@
+# Pyro-Detector Baseline
+
+## Objective
+
+Evaluate the production pyro-engine detection pipeline (`pyro-predictor`) as a
+`TemporalModel` baseline. Provides a reference point for comparing new temporal
+models against the actual production system.
+
+## Approach
+
+1. Wrap `pyro-predictor`'s `Predictor` class (YOLO ONNX detection + per-camera
+   sliding-window temporal smoothing) as a pyrocore `TemporalModel` subclass.
+2. Run YOLO inference once per frame and cache detections (`infer` stage).
+3. Replay cached detections through the Predictor's temporal logic (`predict`
+   stage) — classifies a sequence as positive when the aggregated confidence
+   exceeds `conf_thresh`.
+4. Grid-search `conf_thresh` x `nb_consecutive_frames` on cached detections
+   (`sweep` stage) to find optimal parameters.
+5. Compute sequence-level metrics (precision, recall, F1, FPR, TTD).
+
+The Predictor mirrors production behavior:
+- YOLO ONNX model for single-frame detection (weights from
+  `pyronear/yolo11s_nimble-narwhal_v6.0.0`)
+- Per-camera sliding window for temporal smoothing
+- Aggregated confidence output in [0, 1]
+
+## Data
+
+Imported from [pyro-dataset](https://github.com/pyronear/pyro-dataset) v2.2.0
+via `dvc import` (sequential_train_val split):
+
+- **Train**: 1,467 wildfire + 1,467 FP sequences
+- **Val**: 151 wildfire + 151 FP sequences
+- Layout: `data/01_raw/datasets/{train,val}/{wildfire,fp}/sequence_name/{images,labels}/`
+- Ground truth: inferred from parent directory name (`wildfire/` = positive, `fp/` = negative)
+
+## Results
+
+### Evaluate (optimized params: conf=0.2, nb_frames=4)
+
+| Split | Method | Precision | Recall | F1 | FPR | Median TTD |
+|---|---|---|---|---|---|---|
+| val/all | Single-frame | 0.853 | 0.960 | 0.903 | 0.166 | -- |
+| val/all | Predictor | 0.892 | 0.934 | **0.913** | 0.113 | 9s |
+| train/all | Single-frame | 0.820 | 0.981 | 0.893 | 0.215 | -- |
+| train/all | Predictor | 0.849 | 0.971 | **0.906** | 0.172 | 20s |
+
+### Sweep top configs (val/all, ranked by F1)
+
+| conf | nb_frames | Precision | Recall | F1 | FPR | Mean TTD |
+|---|---|---|---|---|---|---|
+| 0.20 | 4 | 0.892 | 0.934 | **0.913** | 0.113 | 90s |
+| 0.25 | 3 | 0.897 | 0.927 | 0.912 | 0.106 | 80s |
+| 0.20 | 5 | 0.892 | 0.927 | 0.909 | 0.113 | 100s |
+| 0.30 | 2 | 0.892 | 0.927 | 0.909 | 0.113 | 72s |
+
+Production defaults (conf=0.35, nb_frames=7) scored F1=0.825 on val/all.
+The sweep-optimized params improve F1 by ~9pp.
+
+## How to Reproduce
+
+```bash
+cd experiments/temporal-models/pyro-detector-baseline
+make install
+
+# Dataset is imported via DVC from pyro-dataset v2.2.0:
+#   uv run dvc import https://github.com/pyronear/pyro-dataset \
+#       data/processed/sequential_train_val/train \
+#       -o data/01_raw/datasets/train --rev v2.2.0
+#   uv run dvc import https://github.com/pyronear/pyro-dataset \
+#       data/processed/sequential_train_val/val \
+#       -o data/01_raw/datasets/val --rev v2.2.0
+# The .dvc files are committed — just pull:
+uv run dvc pull
+
+uv run dvc repro
+uv run dvc metrics show
+```
+
+## Pipeline
+
+```
+prepare (download ONNX model from HuggingFace)
+  -> infer (01_raw -> 03_primary)          # YOLO-only, cached per-frame detections
+    -> predict (03_primary -> 07_model_output)  # replay temporal logic (fixed params)
+      -> evaluate (-> 08_reporting)
+    -> sweep (03_primary -> 08_reporting/sweep) # grid-search temporal params
+  -> package (01_raw -> 06_models)
+```
+
+## Parameters
+
+Current values are sweep-optimized. Production defaults from
+[pyro-engine](https://github.com/pyronear/pyro-engine) are shown for
+reference.
+
+| Parameter | Value | Production | Description |
+|---|---|---|---|
+| `predict.conf_thresh` | **0.2** | 0.35 | **Alert threshold.** The Predictor accumulates detections across the sliding window and computes an aggregated confidence score. An alert fires when this score exceeds `conf_thresh`. Also used to filter individual detections before aggregation. |
+| `predict.model_conf_thresh` | 0.05 | 0.05 | **YOLO detection threshold.** Minimum confidence for the YOLO model to report a bounding box. Kept low so the temporal layer sees all candidate detections. |
+| `predict.nb_consecutive_frames` | **4** | 7 | **Sliding window size.** Number of recent frames the Predictor keeps in its per-camera state for temporal smoothing. A detection must persist across enough of these frames to trigger an alert. |
+| `predict.max_bbox_size` | 0.4 | 0.4 | **Max detection width.** Detections wider than this fraction of the image are discarded (filters out large false positives like clouds). |

experiments/temporal-models/pyro-detector-baseline/data/01_raw/.gitignore

@@ -0,0 +1 @@
+/models

experiments/temporal-models/pyro-detector-baseline/data/01_raw/datasets/.gitignore

@@ -0,0 +1,2 @@
+/train
+/val

experiments/temporal-models/pyro-detector-baseline/data/01_raw/datasets/train.dvc

@@ -0,0 +1,14 @@
+md5: ceb8e6b2a605cfe4d97815aeee233176
+frozen: true
+deps:
+- path: data/processed/sequential_train_val/train
+  repo:
+    url: https://github.com/pyronear/pyro-dataset
+    rev: v2.2.0
+    rev_lock: ca3b7bbfc9f072d101849e99a5ec91791a3e7c7a
+outs:
+- md5: 22ee5d57943a0ff900121ca28b18f2a8.dir
+  size: 7149007108
+  nfiles: 135742
+  hash: md5
+  path: train

experiments/temporal-models/pyro-detector-baseline/data/01_raw/datasets/val.dvc

@@ -0,0 +1,14 @@
+md5: 180164308103dcd9b3ebdbb2395deac6
+frozen: true
+deps:
+- path: data/processed/sequential_train_val/val
+  repo:
+    url: https://github.com/pyronear/pyro-dataset
+    rev: v2.2.0
+    rev_lock: ca3b7bbfc9f072d101849e99a5ec91791a3e7c7a
+outs:
+- md5: 2ca47638a4b0cc9ca7d4011376c54e45.dir
+  size: 631259421
+  nfiles: 12448
+  hash: md5
+  path: val

experiments/temporal-models/pyro-detector-baseline/data/03_primary/.gitignore

@@ -0,0 +1,2 @@
+/val
+/train

experiments/temporal-models/pyro-detector-baseline/data/06_models/.gitignore

@@ -0,0 +1 @@
+/model.zip

experiments/temporal-models/pyro-detector-baseline/data/07_model_output/.gitignore

@@ -0,0 +1,2 @@
+/val
+/train

experiments/temporal-models/pyro-detector-baseline/data/08_reporting/sweep/train/.gitignore

@@ -0,0 +1,2 @@
+/all
+/pyronear

experiments/temporal-models/pyro-detector-baseline/data/08_reporting/sweep/val/.gitignore

@@ -0,0 +1,2 @@
+/all
+/pyronear

experiments/temporal-models/pyro-detector-baseline/data/08_reporting/train/all/.gitignore

@@ -0,0 +1 @@
+/plots

experiments/temporal-models/pyro-detector-baseline/data/08_reporting/train/all/metrics.json

@@ -0,0 +1,40 @@
+{
+  "single_frame": {
+    "num_sequences": 2933,
+    "num_positive_gt": 1466,
+    "num_negative_gt": 1467,
+    "tp": 1438,
+    "fp": 315,
+    "fn": 28,
+    "tn": 1152,
+    "tp_pct": 98.09,
+    "fp_pct": 21.47,
+    "fn_pct": 1.91,
+    "tn_pct": 78.53,
+    "precision": 0.8203,
+    "recall": 0.9809,
+    "f1": 0.8934,
+    "fpr": 0.2147,
+    "mean_ttd_seconds": 0.0,
+    "median_ttd_seconds": 0.0
+  },
+  "predictor": {
+    "num_sequences": 2933,
+    "num_positive_gt": 1466,
+    "num_negative_gt": 1467,
+    "tp": 1424,
+    "fp": 253,
+    "fn": 42,
+    "tn": 1214,
+    "tp_pct": 97.14,
+    "fp_pct": 17.25,
+    "fn_pct": 2.86,
+    "tn_pct": 82.75,
+    "precision": 0.8491,
+    "recall": 0.9714,
+    "f1": 0.9061,
+    "fpr": 0.1725,
+    "mean_ttd_seconds": 80.6,
+    "median_ttd_seconds": 20.0
+  }
+}

experiments/temporal-models/pyro-detector-baseline/data/08_reporting/train/pyronear/.gitignore

@@ -0,0 +1 @@
+/plots

experiments/temporal-models/pyro-detector-baseline/data/08_reporting/train/pyronear/metrics.json

@@ -0,0 +1,40 @@
+{
+  "single_frame": {
+    "num_sequences": 2536,
+    "num_positive_gt": 1069,
+    "num_negative_gt": 1467,
+    "tp": 1058,
+    "fp": 315,
+    "fn": 11,
+    "tn": 1152,
+    "tp_pct": 98.97,
+    "fp_pct": 21.47,
+    "fn_pct": 1.03,
+    "tn_pct": 78.53,
+    "precision": 0.7706,
+    "recall": 0.9897,
+    "f1": 0.8665,
+    "fpr": 0.2147,
+    "mean_ttd_seconds": 0.0,
+    "median_ttd_seconds": 0.0
+  },
+  "predictor": {
+    "num_sequences": 2536,
+    "num_positive_gt": 1069,
+    "num_negative_gt": 1467,
+    "tp": 1047,
+    "fp": 253,
+    "fn": 22,
+    "tn": 1214,
+    "tp_pct": 97.94,
+    "fp_pct": 17.25,
+    "fn_pct": 2.06,
+    "tn_pct": 82.75,
+    "precision": 0.8054,
+    "recall": 0.9794,
+    "f1": 0.8839,
+    "fpr": 0.1725,
+    "mean_ttd_seconds": 41.7,
+    "median_ttd_seconds": 7.0
+  }
+}

experiments/temporal-models/pyro-detector-baseline/data/08_reporting/val/all/.gitignore

@@ -0,0 +1 @@
+/plots

experiments/temporal-models/pyro-detector-baseline/data/08_reporting/val/all/metrics.json

@@ -0,0 +1,40 @@
+{
+  "single_frame": {
+    "num_sequences": 302,
+    "num_positive_gt": 151,
+    "num_negative_gt": 151,
+    "tp": 145,
+    "fp": 25,
+    "fn": 6,
+    "tn": 126,
+    "tp_pct": 96.03,
+    "fp_pct": 16.56,
+    "fn_pct": 3.97,
+    "tn_pct": 83.44,
+    "precision": 0.8529,
+    "recall": 0.9603,
+    "f1": 0.9034,
+    "fpr": 0.1656,
+    "mean_ttd_seconds": 0.0,
+    "median_ttd_seconds": 0.0
+  },
+  "predictor": {
+    "num_sequences": 302,
+    "num_positive_gt": 151,
+    "num_negative_gt": 151,
+    "tp": 141,
+    "fp": 17,
+    "fn": 10,
+    "tn": 134,
+    "tp_pct": 93.38,
+    "fp_pct": 11.26,
+    "fn_pct": 6.62,
+    "tn_pct": 88.74,
+    "precision": 0.8924,
+    "recall": 0.9338,
+    "f1": 0.9126,
+    "fpr": 0.1126,
+    "mean_ttd_seconds": 90.4,
+    "median_ttd_seconds": 9.0
+  }
+}

experiments/temporal-models/pyro-detector-baseline/data/08_reporting/val/pyronear/.gitignore

@@ -0,0 +1 @@
+/plots

experiments/temporal-models/pyro-detector-baseline/data/08_reporting/val/pyronear/metrics.json

@@ -0,0 +1,40 @@
+{
+  "single_frame": {
+    "num_sequences": 267,
+    "num_positive_gt": 116,
+    "num_negative_gt": 151,
+    "tp": 113,
+    "fp": 25,
+    "fn": 3,
+    "tn": 126,
+    "tp_pct": 97.41,
+    "fp_pct": 16.56,
+    "fn_pct": 2.59,
+    "tn_pct": 83.44,
+    "precision": 0.8188,
+    "recall": 0.9741,
+    "f1": 0.8898,
+    "fpr": 0.1656,
+    "mean_ttd_seconds": 0.0,
+    "median_ttd_seconds": 0.0
+  },
+  "predictor": {
+    "num_sequences": 267,
+    "num_positive_gt": 116,
+    "num_negative_gt": 151,
+    "tp": 111,
+    "fp": 17,
+    "fn": 5,
+    "tn": 134,
+    "tp_pct": 95.69,
+    "fp_pct": 11.26,
+    "fn_pct": 4.31,
+    "tn_pct": 88.74,
+    "precision": 0.8672,
+    "recall": 0.9569,
+    "f1": 0.9098,
+    "fpr": 0.1126,
+    "mean_ttd_seconds": 21.9,
+    "median_ttd_seconds": 5.0
+  }
+}