Skip to content

Add criterion and Python benchmark baselines for doeff-vm#326

Open
proboscis wants to merge 1 commit intomainfrom
issue/vm-perf-bench/run-20260314-154932
Open

Add criterion and Python benchmark baselines for doeff-vm#326
proboscis wants to merge 1 commit intomainfrom
issue/vm-perf-bench/run-20260314-154932

Conversation

@proboscis
Copy link
Copy Markdown
Owner

Summary

  • add a reusable Python benchmark workload module plus a rewritten benchmarks/benchmark_runner.py that validates results and writes JSON/CSV artifacts
  • add a criterion benchmark for packages/doeff-vm covering module_run and pyvm_fresh baseline cases
  • add make bench-python / make bench-vm, a benchmark README, benchmark artifact ignores, and smoke tests for the Python benchmark runner

Verification

  • uv run pytest tests/test_benchmark_runner.py packages/doeff-vm/tests/test_package_exports.py packages/doeff-vm/tests/test_pyvm.py::test_module_level_run_basic
    • 6 passed in 0.20s
  • make bench-python BENCH_PYTHON_ARGS='--runs 3 --iterations 4 --output-dir benchmarks/results/smoke'
    • public_run:pure: runs=3 iterations=4 | min=0.16ms median=0.17ms mean=0.17ms max=0.19ms
    • public_run:state: runs=3 iterations=4 | min=0.63ms median=0.64ms mean=0.65ms max=0.66ms
    • public_run:state_writer: runs=3 iterations=4 | min=0.96ms median=1.00ms mean=0.99ms max=1.01ms
    • generated artifacts:
      • benchmarks/results/smoke/doeff_vm_benchmark_results.json
      • benchmarks/results/smoke/doeff_vm_benchmark_results.csv
  • make bench-vm BENCH_VM_ARGS='--sample-size 10 --measurement-time 0.1 --warm-up-time 0.1 --noplot'
    • doeff_vm_baseline/module_run/pure time: [31.529 µs 34.816 µs 37.774 µs]
    • doeff_vm_baseline/pyvm_fresh/pure time: [30.211 µs 31.943 µs 34.707 µs]
    • doeff_vm_baseline/module_run/state time: [2.6103 ms 2.6416 ms 2.6787 ms]
    • doeff_vm_baseline/pyvm_fresh/state time: [2.5980 ms 2.6341 ms 2.6877 ms]
    • doeff_vm_baseline/module_run/state_writer time: [4.2105 ms 4.6090 ms 5.1725 ms]
    • doeff_vm_baseline/pyvm_fresh/state_writer time: [4.1763 ms 4.3576 ms 4.6610 ms]
    • criterion artifacts confirmed under packages/doeff-vm/target/criterion/doeff_vm_baseline/...
  • uv run ruff check benchmarks tests/test_benchmark_runner.py
    • All checks passed!

Issue

  • refs vm-perf-bench

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant