Skip to content

feat: add Docker Alpine benchmark suite#4

Closed
Lets7512 wants to merge 4 commits into
AlpinDale:masterfrom
Lets7512:feat/benchmarks
Closed

feat: add Docker Alpine benchmark suite#4
Lets7512 wants to merge 4 commits into
AlpinDale:masterfrom
Lets7512:feat/benchmarks

Conversation

@Lets7512
Copy link
Copy Markdown

@Lets7512 Lets7512 commented Mar 6, 2026

Summary

  • Add a mock-based benchmark binary (benches/mock_sync.rs) that measures parallel sync performance across four workload profiles with 1–16 workers
  • Add Dockerfile.bench (Alpine multi-stage) for reproducible, portable benchmarking
  • Add scripts/docker_bench.sh to build and run benchmarks in Docker with JSON + markdown output
  • Update README.md with a comprehensive benchmark results table and reproduction instructions

Benchmark Results

Tested on Docker alpine:latest with simulated per-read latency (no real SSH):

Scenario 1 job 2 jobs 4 jobs 8 jobs 16 jobs
Many small (1000 x 4KB) 3203ms 1763ms (1.8x) 1120ms (2.9x) 806ms (4.0x) 718ms (4.5x)
Medium (40 x 128KB) 488ms 287ms (1.7x) 171ms (2.9x) 99ms (4.9x) 107ms (4.6x)
Few large (5 x 10MB) 8302ms 5025ms (1.7x) 3347ms (2.5x) 1713ms (4.9x) 1722ms (4.8x)
Mixed (200 varied) 8509ms 4478ms (1.9x) 2224ms (3.8x) 1206ms (7.1x) 727ms (11.7x)

Key findings

  • Mixed workloads benefit the most from parallelism — up to 11.7x speedup with 16 jobs
  • Large-file scenarios plateau around 8 jobs (I/O bound on chunk writes)
  • Small-file scenarios show diminishing returns past 8 jobs (scheduling overhead)
  • Medium workloads peak at 8 jobs

How to reproduce

./scripts/docker_bench.sh

Test plan

  • Benchmark binary compiles and runs in Docker Alpine
  • JSON output is valid and parseable
  • Markdown table renders correctly
  • All four scenarios produce consistent results across 5 runs
  • Speedup values are reasonable (no regressions)

🤖 Generated with Claude Code

Add a mock-based benchmark binary that measures parsync's parallel sync
performance across four workload profiles (many small, medium, few large,
mixed) with 1-16 workers. Includes a Dockerfile for reproducible Alpine
builds and a helper script to run benchmarks in Docker.

Results show up to 11.7x speedup on mixed workloads with 16 jobs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@Lets7512
Copy link
Copy Markdown
Author

Lets7512 commented Mar 6, 2026

Implementation Notes

Benchmark approach: Uses a DelayedMockRemote that simulates network latency with thread::sleep() per read call. This isolates parsync's scheduling/parallelism from real network variance, giving reproducible results.

Why not Criterion: Criterion adds ~2 min compile time in Alpine. The custom harness compiles fast, runs 5 iterations per config, and outputs both JSON (for tooling) and markdown (for humans).

Dockerfile strategy: Multi-stage build keeps the final image minimal (~15MB). The builder stage uses rust:alpine with vendored OpenSSL for musl compatibility.

Interesting findings:

  • The mixed workload shows near-linear scaling up to 16 jobs (11.7x) because it has enough parallelism diversity — small files finish fast and free workers for large files
  • Large files plateau at 8 jobs because with only 5 files, workers beyond 5 are idle, and the remaining speedup comes from chunk-level parallelism
  • Medium files show a slight regression at 16 vs 8 jobs — thread scheduling overhead exceeds the parallelism benefit for this workload size

@Lets7512
Copy link
Copy Markdown
Author

Lets7512 commented Mar 6, 2026

Hi @AlpinDale , Just wanted to add benchmarking with docker so anyone can test it and compare results and have a small results table in the README.

Comment thread docs/plans/2026-03-06-benchmarks-design.md Outdated
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@Lets7512
Copy link
Copy Markdown
Author

Lets7512 commented Mar 6, 2026

Removed the planning doc in bfc6aa3. PR should be clean now — only benchmark code, Dockerfile, script, and README updates remain.

@Lets7512 Lets7512 requested a review from AlpinDale March 6, 2026 14:42
Lets7512 and others added 2 commits March 6, 2026 16:50
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove unused `sync::Mutex` import
- Change `fn median` parameter from `&mut Vec<f64>` to `&mut [f64]` (clippy::ptr_arg)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@Lets7512
Copy link
Copy Markdown
Author

Lets7512 commented Mar 7, 2026

CI run #53 failed on both test and test-windows jobs due to clippy errors (cargo clippy --all-targets --all-features -- -D warnings):

  1. Unused importstd::sync::Mutex was imported but never used
  2. clippy::ptr_argfn median(values: &mut Vec<f64>) should take &mut [f64] instead

Fixed in c4b1298. CI should pass now.

@AlpinDale
Copy link
Copy Markdown
Owner

Unnecessary for now

@AlpinDale AlpinDale closed this Mar 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants