add :benchmarks module comparing kmapper vs hand-written mappings

.github/workflows/build.yml

@@ -22,4 +22,46 @@ jobs:
       uses: gradle/actions/setup-gradle@v3
 
     - name: Build and test
-      run: ./gradlew build
+      run: ./gradlew build
+
+  benchmarks:
+    # Smoke-tests the :benchmarks module on every PR. The threshold gate fails
+    # the build only on catastrophic regressions (kmapper >5x slower than the
+    # hand-written equivalent) — GitHub-hosted runners are too noisy to detect
+    # smaller deltas reliably. The full JMH JSON report is uploaded as an
+    # artifact for human review.
+    runs-on: ubuntu-latest
+    needs: build
+
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v4
+
+    - name: Set up JDK
+      uses: actions/setup-java@v4
+      with:
+        java-version: '21'
+        distribution: 'temurin'
+
+    - name: Setup Gradle
+      uses: gradle/actions/setup-gradle@v3
+
+    - name: Publish kmapper artifacts to mavenLocal
+      run: |
+        ./gradlew \
+          :compiler-plugin:publishToMavenLocal \
+          :compiler-runtime:publishToMavenLocal \
+          :gradle-plugin:publishToMavenLocal
+
+    - name: Run benchmarks with threshold gate
+      run: |
+        ./gradlew -Pkmapper.benchmarks=true \
+          :benchmarks:verifyBenchmarkThresholds
+
+    - name: Upload JMH report
+      if: always()
+      uses: actions/upload-artifact@v4
+      with:
+        name: jmh-report
+        path: benchmarks/build/reports/benchmarks/
+        if-no-files-found: warn

README.md

@@ -304,6 +304,12 @@ val dto: PersonDto = Person("John").mapper {
 
 The plugin automatically generates the mapping implementation at compile time, replacing the `mapper` function call with the actual object construction code.
 
+### Performance
+
+Because KMapper is a compile-time codegen plugin (no reflection, no runtime registry), the generated code is equivalent to a hand-written constructor call and runtime overhead is essentially zero.
+
+This is verified by JMH benchmarks in [`benchmarks/`](benchmarks/), which compare `mapper { ... }` against hand-written equivalents for both a flat 3-field mapping and a nested mapping with value classes, lists, enums, and numeric widening. The CI pipeline runs the suite on every PR and fails the build if a kmapper benchmark is more than 5× slower than its manual counterpart, guarding against accidental regressions.
+
 
 
 Notes:

benchmarks/README.md

@@ -0,0 +1,152 @@
+# benchmarks
+
+Performance comparison between **kmapper-generated** mappings and
+**hand-written** constructor calls between two data classes.
+
+Because kmapper is a compile-time codegen plugin (no reflection at runtime),
+the expected runtime overhead is essentially zero. This module exists to
+prove that empirically, to make the claim replicable, and to catch
+regressions if a future change to the IR generator accidentally introduces
+runtime cost.
+
+## What is measured
+
+Two scenarios, each with a `kmapper` variant and a `manual` variant:
+
+| Scenario  | Source shape                                                                                                                  | Target shape                                          |
+| --------- | ----------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
+| `simple`  | flat 3-field data class                                                                                                       | flat 2-field data class with one derived field        |
+| `complex` | nested data classes, two `@JvmInline value class`es, a list of 10 nested objects, an enum, an `Int → Long` widening, derived field | flat-equivalent DTO graph with the same value/enum/list structure |
+
+The two variants do the same work: same constructor calls, same string
+concatenation for the derived name, same `List.map` for the line items.
+Only the *source* of the mapper code differs (kmapper IR vs hand-written
+Kotlin).
+
+## How to run
+
+There are two runners. **Use both** — they answer different questions.
+
+### Bootstrap (one-time, plus after every change to the compiler plugin)
+
+This module applies the `community.flock.kmapper` Gradle plugin to its own
+sources, so the plugin must be in `mavenLocal` *before* the module can even
+be configured. To avoid breaking the parent build for everyone else,
+`:benchmarks` is only included when the opt-in property is set.
+
+```bash
+./gradlew :compiler-plugin:publishToMavenLocal \
+          :compiler-runtime:publishToMavenLocal \
+          :gradle-plugin:publishToMavenLocal
+```
+
+Subsequent commands all need `-Pkmapper.benchmarks=true` to include the module.
+
+### 1. JMH via kotlinx-benchmark (authoritative)
+
+```bash
+./gradlew -Pkmapper.benchmarks=true :benchmarks:benchmark
+```
+
+This forks a fresh JVM, runs a configurable warmup, measures average time
+per op in nanoseconds, and writes a JSON report to `build/reports/benchmarks/main/`.
+Defaults are tuned for CI (warmups=2, iterations=3, 1s each) so the whole
+run completes in under a minute. For trustworthy numbers on a quiet
+workstation, override:
+
+```bash
+./gradlew -Pkmapper.benchmarks=true :benchmarks:benchmark \
+          -PbenchmarkWarmups=5 -PbenchmarkIterations=10
+```
+
+JMH guards against dead-code elimination, constant folding, and on-stack
+replacement skew. This is the number to quote in PRs and release notes.
+
+### 2. Standalone nanoTime runner (replicable smoke test)
+
+```bash
+./gradlew -Pkmapper.benchmarks=true :benchmarks:run
+```
+
+Prints `ns/op` and a `% overhead` figure for each scenario in a few
+seconds. Implementation lives in [`ManualRunner.kt`](src/main/kotlin/community/flock/kmapper/benchmarks/ManualRunner.kt).
+Numbers are *indicative only*: a hand-rolled loop can't fully defeat JIT
+optimizations the way JMH can. Useful for:
+
+- a 10-second sanity check while iterating on the compiler plugin
+- a CI step that fails only on order-of-magnitude regressions
+- demoing the comparison without asking reviewers to learn JMH
+
+If JMH and the manual runner disagree by more than a few percent, **trust
+JMH**.
+
+### 3. Threshold gate (CI / regression guard)
+
+```bash
+./gradlew -Pkmapper.benchmarks=true :benchmarks:verifyBenchmarkThresholds
+```
+
+Runs `:benchmark`, then parses the JSON report and **fails the build** if any
+kmapper benchmark is more than `benchmarkMaxRatio`x slower than its manual
+counterpart. The default ratio is `5.0` — deliberately loose, because
+GitHub-hosted runners are noisy and small-percent differences are not
+distinguishable from variance. Override locally for a stricter check:
+
+```bash
+./gradlew -Pkmapper.benchmarks=true :benchmarks:verifyBenchmarkThresholds \
+          -PbenchmarkMaxRatio=2.0
+```
+
+The CI job in [`.github/workflows/build.yml`](../.github/workflows/build.yml)
+runs this task on every PR and uploads the JMH report as a build artifact.
+
+## Other approaches considered
+
+The current setup picks **kotlinx-benchmark + a simple `main()` runner**
+because they are the most idiomatic Kotlin choices and cover both
+"authoritative" and "replicable" needs. Alternatives, with the tradeoffs
+that pushed them down the list:
+
+- **Raw JMH with the `me.champeau.jmh` Gradle plugin.** Equally accurate;
+  more Java-flavored configuration. Pick this if you want to drop the
+  kotlinx wrapper and depend only on JMH itself.
+- **JUnit `@Test` with `measureTimeMillis`.** Reuses the existing test
+  infra but produces unreliable numbers (no JIT control, no fork isolation,
+  no DCE protection). Adequate only for catching catastrophic regressions.
+- **JFR / async-profiler over a long-running workload.** Excellent for
+  flame-graphing where time is spent inside a generated mapping; not a
+  substitute for `ns/op` numbers. Worth pairing with JMH once a regression
+  is suspected:
+  ```bash
+  ./gradlew :benchmarks:benchmark -Pjvmargs="-XX:+FlightRecorder -XX:StartFlightRecording=duration=60s,filename=kmapper.jfr"
+  ```
+- **Compile-time overhead benchmark.** A separate question: how much does
+  *adding* the kmapper plugin slow down `kotlinc`? Out of scope for this
+  module (it would need clean-build timing of synthetic projects, not
+  micro-benchmarks). Track it separately if it becomes a concern.
+
+## How to add a new scenario
+
+1. Add domain + dto data classes and two mapper functions
+   (`toDtoKMapper()` using `mapper { ... }`, `toDtoManual()` using a plain
+   constructor) to a new file under `src/main/kotlin/community/flock/kmapper/benchmarks/`.
+2. Add a fixture instance to [`Fixtures.kt`](src/main/kotlin/community/flock/kmapper/benchmarks/Fixtures.kt).
+3. Add a `@State` class with two `@Benchmark` methods (`kmapper` and
+   `manual`) following the pattern of [`SimpleMappingBenchmark.kt`](src/main/kotlin/community/flock/kmapper/benchmarks/SimpleMappingBenchmark.kt).
+4. Optionally add a `runScenario(...)` call in
+   [`ManualRunner.kt`](src/main/kotlin/community/flock/kmapper/benchmarks/ManualRunner.kt).
+
+Both runners pick up new benchmarks automatically — no extra wiring.
+
+## Reproducibility notes
+
+- Pin a Java toolchain (`jvmToolchain(21)`) so JIT differences between
+  Java versions don't pollute results.
+- Run on a quiet machine. Close browsers, IDE indexers, Spotlight, etc.
+- Disable CPU frequency scaling if you have access to it
+  (`sudo cpupower frequency-set -g performance` on Linux).
+- Quote three numbers per scenario: `manual ns/op`, `kmapper ns/op`,
+  `% overhead`. A single number hides whether the absolute cost is
+  meaningful for your workload.
+- Re-run on the same hardware before claiming a regression is real.
+  Single-run noise on a laptop is often ±5–10%.

benchmarks/build.gradle.kts

@@ -0,0 +1,140 @@
+plugins {
+    kotlin("jvm") version "2.3.20"
+    kotlin("plugin.allopen") version "2.3.20"
+    id("org.jetbrains.kotlinx.benchmark") version "0.4.13"
+    id("community.flock.kmapper") version "0.0.0-SNAPSHOT"
+    application
+}
+
+group = rootProject.group
+version = rootProject.version
+
+repositories {
+    mavenLocal()
+    mavenCentral()
+}
+
+dependencies {
+    implementation("org.jetbrains.kotlinx:kotlinx-benchmark-runtime:0.4.13")
+}
+
+kotlin {
+    jvmToolchain(21)
+}
+
+// JMH requires @State classes to be open (non-final). The all-open plugin
+// rewrites them at compile time so we don't have to write `open class`.
+allOpen {
+    annotation("org.openjdk.jmh.annotations.State")
+}
+
+// CI-friendly defaults: keep total wall time under ~1 min so benchmarks can
+// run on every PR. Override locally for trustworthy numbers, e.g.:
+//   ./gradlew -Pkmapper.benchmarks=true :benchmarks:benchmark \
+//             -PbenchmarkWarmups=5 -PbenchmarkIterations=10
+val benchmarkWarmups = (project.findProperty("benchmarkWarmups") as String? ?: "2").toInt()
+val benchmarkIterations = (project.findProperty("benchmarkIterations") as String? ?: "3").toInt()
+
+benchmark {
+    targets {
+        register("main")
+    }
+    configurations {
+        named("main") {
+            warmups = benchmarkWarmups
+            iterations = benchmarkIterations
+            iterationTime = 1
+            iterationTimeUnit = "s"
+            outputTimeUnit = "ns"
+            mode = "avgt"
+            reportFormat = "json"
+        }
+    }
+}
+
+application {
+    mainClass.set("community.flock.kmapper.benchmarks.ManualRunnerKt")
+}
+
+// The kmapper Gradle plugin and the compiler-runtime are resolved from
+// mavenLocal. They must be published BEFORE this module compiles, so wire
+// the dependency explicitly (mirrors test-integration/build.gradle.kts).
+val publishKmapperLocally = listOf(
+    ":compiler-plugin:publishToMavenLocal",
+    ":compiler-runtime:publishToMavenLocal",
+    ":gradle-plugin:publishToMavenLocal",
+)
+tasks.named("compileKotlin") { publishKmapperLocally.forEach { dependsOn(it) } }
+tasks.matching { it.name == "mainBenchmarkCompile" || it.name == "mainBenchmark" }
+    .configureEach { publishKmapperLocally.forEach { dependsOn(it) } }
+
+// Threshold gate: parse the JMH JSON report and fail if any kmapper benchmark
+// is more than `benchmarkMaxRatio`x slower than its hand-written counterpart.
+// The default ratio is deliberately loose (5x) because GitHub-hosted runners
+// are noisy. Override locally for tighter checks:
+//   ./gradlew -Pkmapper.benchmarks=true :benchmarks:verifyBenchmarkThresholds \
+//             -PbenchmarkMaxRatio=2.0
+val verifyBenchmarkThresholds by tasks.registering {
+    group = "verification"
+    description = "Fails if kmapper-generated mappers are >Nx slower than hand-written equivalents"
+    dependsOn("benchmark")
+
+    val maxRatio = (project.findProperty("benchmarkMaxRatio") as String? ?: "5.0").toDouble()
+    val reportRoot = layout.buildDirectory.dir("reports/benchmarks/main")
+
+    doLast {
+        val root = reportRoot.get().asFile
+        val jsonFile = root.walkTopDown()
+            .filter { it.isFile && it.extension == "json" }
+            .maxByOrNull { it.lastModified() }
+            ?: error("No JMH JSON report found under $root — did `:benchmark` run with reportFormat=\"json\"?")
+
+        // Minimal hand-rolled parse: each entry has
+        //   "benchmark": "<fqcn>.<method>", "primaryMetric": { "score": <num>, ... }
+        val text = jsonFile.readText()
+        val entryRegex = Regex(
+            """"benchmark"\s*:\s*"([^"]+)"[\s\S]*?"primaryMetric"\s*:\s*\{[\s\S]*?"score"\s*:\s*([0-9.eE+-]+)"""
+        )
+        val scores: Map<String, Double> = entryRegex.findAll(text).associate { m ->
+            // benchmark id is "...SimpleMappingBenchmark.kmapper" — key by ClassName.method
+            val id = m.groupValues[1].split('.').takeLast(2).joinToString(".")
+            id to m.groupValues[2].toDouble()
+        }
+        require(scores.isNotEmpty()) {
+            "Could not parse any benchmark scores from $jsonFile"
+        }
+
+        val pairs = listOf(
+            "SimpleMappingBenchmark" to ("manual" to "kmapper"),
+            "ComplexMappingBenchmark" to ("manual" to "kmapper"),
+        )
+        val report = StringBuilder().appendLine(
+            "Benchmark threshold report (max ratio = ${maxRatio}x, source: ${jsonFile.name}):"
+        )
+        val failures = mutableListOf<String>()
+        for ((cls, methods) in pairs) {
+            val (manualMethod, kmapperMethod) = methods
+            val manualKey = "$cls.$manualMethod"
+            val kmapperKey = "$cls.$kmapperMethod"
+            val manualScore = scores[manualKey]
+            val kmapperScore = scores[kmapperKey]
+            if (manualScore == null || kmapperScore == null) {
+                failures += "missing scores for $cls (manual=$manualScore, kmapper=$kmapperScore)"
+                continue
+            }
+            val ratio = kmapperScore / manualScore
+            report.appendLine(
+                "  %s: kmapper=%.2f ns, manual=%.2f ns, ratio=%.2fx".format(
+                    cls, kmapperScore, manualScore, ratio,
+                )
+            )
+            if (ratio > maxRatio) {
+                failures += "$cls: kmapper is ${"%.2f".format(ratio)}x slower than manual (limit=${maxRatio}x)"
+            }
+        }
+        logger.lifecycle(report.toString())
+        if (failures.isNotEmpty()) {
+            error("Benchmark threshold violations:\n  - " + failures.joinToString("\n  - "))
+        }
+    }
+}