From 8d68e0d51f64b4861c4fb7e57790d07884dfc009 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 25 Apr 2026 11:25:57 +0000 Subject: [PATCH 1/3] add :benchmarks module comparing kmapper vs hand-written mappings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measures runtime overhead of kmapper-generated code against equivalent hand-written constructor calls for two scenarios — a flat 3-field mapping and a nested mapping with value classes, a list, an enum, and a numeric widening. Two runners are provided: - JMH via kotlinx-benchmark for authoritative ns/op numbers - A standalone System.nanoTime() runner for a zero-friction sanity check The module applies the kmapper Gradle plugin to its own sources, so it needs the plugin in mavenLocal at configuration time. To avoid breaking the parent build before that bootstrap publish has happened, :benchmarks is gated behind -Pkmapper.benchmarks=true. The README documents the two-step workflow. --- benchmarks/README.md | 126 ++++++++++++++++++ benchmarks/build.gradle.kts | 54 ++++++++ .../flock/kmapper/benchmarks/ComplexDomain.kt | 109 +++++++++++++++ .../benchmarks/ComplexMappingBenchmark.kt | 23 ++++ .../flock/kmapper/benchmarks/Fixtures.kt | 32 +++++ .../flock/kmapper/benchmarks/ManualRunner.kt | 80 +++++++++++ .../flock/kmapper/benchmarks/SimpleDomain.kt | 28 ++++ .../benchmarks/SimpleMappingBenchmark.kt | 23 ++++ settings.gradle.kts | 15 +++ 9 files changed, 490 insertions(+) create mode 100644 benchmarks/README.md create mode 100644 benchmarks/build.gradle.kts create mode 100644 benchmarks/src/main/kotlin/community/flock/kmapper/benchmarks/ComplexDomain.kt create mode 100644 benchmarks/src/main/kotlin/community/flock/kmapper/benchmarks/ComplexMappingBenchmark.kt create mode 100644 benchmarks/src/main/kotlin/community/flock/kmapper/benchmarks/Fixtures.kt create mode 100644 benchmarks/src/main/kotlin/community/flock/kmapper/benchmarks/ManualRunner.kt create mode 100644 benchmarks/src/main/kotlin/community/flock/kmapper/benchmarks/SimpleDomain.kt create mode 100644 benchmarks/src/main/kotlin/community/flock/kmapper/benchmarks/SimpleMappingBenchmark.kt diff --git a/benchmarks/README.md b/benchmarks/README.md new file mode 100644 index 0000000..d9a0608 --- /dev/null +++ b/benchmarks/README.md @@ -0,0 +1,126 @@ +# benchmarks + +Performance comparison between **kmapper-generated** mappings and +**hand-written** constructor calls between two data classes. + +Because kmapper is a compile-time codegen plugin (no reflection at runtime), +the expected runtime overhead is essentially zero. This module exists to +prove that empirically, to make the claim replicable, and to catch +regressions if a future change to the IR generator accidentally introduces +runtime cost. + +## What is measured + +Two scenarios, each with a `kmapper` variant and a `manual` variant: + +| Scenario | Source shape | Target shape | +| --------- | ----------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | +| `simple` | flat 3-field data class | flat 2-field data class with one derived field | +| `complex` | nested data classes, two `@JvmInline value class`es, a list of 10 nested objects, an enum, an `Int → Long` widening, derived field | flat-equivalent DTO graph with the same value/enum/list structure | + +The two variants do the same work: same constructor calls, same string +concatenation for the derived name, same `List.map` for the line items. +Only the *source* of the mapper code differs (kmapper IR vs hand-written +Kotlin). + +## How to run + +There are two runners. **Use both** — they answer different questions. + +### Bootstrap (one-time, plus after every change to the compiler plugin) + +This module applies the `community.flock.kmapper` Gradle plugin to its own +sources, so the plugin must be in `mavenLocal` *before* the module can even +be configured. To avoid breaking the parent build for everyone else, +`:benchmarks` is only included when the opt-in property is set. + +```bash +./gradlew :compiler-plugin:publishToMavenLocal \ + :compiler-runtime:publishToMavenLocal \ + :gradle-plugin:publishToMavenLocal +``` + +Subsequent commands all need `-Pkmapper.benchmarks=true` to include the module. + +### 1. JMH via kotlinx-benchmark (authoritative) + +```bash +./gradlew -Pkmapper.benchmarks=true :benchmarks:benchmark +``` + +This forks a fresh JVM, runs a configurable warmup, measures average time +per op in nanoseconds, and writes a report to `build/reports/benchmarks/`. +Configuration lives in `build.gradle.kts` under `benchmark { configurations { ... } }` +(warmups, iterations, iteration time). JMH guards against dead-code +elimination, constant folding, and on-stack replacement skew. + +This is the number to quote in PRs and release notes. + +### 2. Standalone nanoTime runner (replicable smoke test) + +```bash +./gradlew -Pkmapper.benchmarks=true :benchmarks:run +``` + +Prints `ns/op` and a `% overhead` figure for each scenario in a few +seconds. Implementation lives in [`ManualRunner.kt`](src/main/kotlin/community/flock/kmapper/benchmarks/ManualRunner.kt). +Numbers are *indicative only*: a hand-rolled loop can't fully defeat JIT +optimizations the way JMH can. Useful for: + +- a 10-second sanity check while iterating on the compiler plugin +- a CI step that fails only on order-of-magnitude regressions +- demoing the comparison without asking reviewers to learn JMH + +If JMH and the manual runner disagree by more than a few percent, **trust +JMH**. + +## Other approaches considered + +The current setup picks **kotlinx-benchmark + a simple `main()` runner** +because they are the most idiomatic Kotlin choices and cover both +"authoritative" and "replicable" needs. Alternatives, with the tradeoffs +that pushed them down the list: + +- **Raw JMH with the `me.champeau.jmh` Gradle plugin.** Equally accurate; + more Java-flavored configuration. Pick this if you want to drop the + kotlinx wrapper and depend only on JMH itself. +- **JUnit `@Test` with `measureTimeMillis`.** Reuses the existing test + infra but produces unreliable numbers (no JIT control, no fork isolation, + no DCE protection). Adequate only for catching catastrophic regressions. +- **JFR / async-profiler over a long-running workload.** Excellent for + flame-graphing where time is spent inside a generated mapping; not a + substitute for `ns/op` numbers. Worth pairing with JMH once a regression + is suspected: + ```bash + ./gradlew :benchmarks:benchmark -Pjvmargs="-XX:+FlightRecorder -XX:StartFlightRecording=duration=60s,filename=kmapper.jfr" + ``` +- **Compile-time overhead benchmark.** A separate question: how much does + *adding* the kmapper plugin slow down `kotlinc`? Out of scope for this + module (it would need clean-build timing of synthetic projects, not + micro-benchmarks). Track it separately if it becomes a concern. + +## How to add a new scenario + +1. Add domain + dto data classes and two mapper functions + (`toDtoKMapper()` using `mapper { ... }`, `toDtoManual()` using a plain + constructor) to a new file under `src/main/kotlin/community/flock/kmapper/benchmarks/`. +2. Add a fixture instance to [`Fixtures.kt`](src/main/kotlin/community/flock/kmapper/benchmarks/Fixtures.kt). +3. Add a `@State` class with two `@Benchmark` methods (`kmapper` and + `manual`) following the pattern of [`SimpleMappingBenchmark.kt`](src/main/kotlin/community/flock/kmapper/benchmarks/SimpleMappingBenchmark.kt). +4. Optionally add a `runScenario(...)` call in + [`ManualRunner.kt`](src/main/kotlin/community/flock/kmapper/benchmarks/ManualRunner.kt). + +Both runners pick up new benchmarks automatically — no extra wiring. + +## Reproducibility notes + +- Pin a Java toolchain (`jvmToolchain(21)`) so JIT differences between + Java versions don't pollute results. +- Run on a quiet machine. Close browsers, IDE indexers, Spotlight, etc. +- Disable CPU frequency scaling if you have access to it + (`sudo cpupower frequency-set -g performance` on Linux). +- Quote three numbers per scenario: `manual ns/op`, `kmapper ns/op`, + `% overhead`. A single number hides whether the absolute cost is + meaningful for your workload. +- Re-run on the same hardware before claiming a regression is real. + Single-run noise on a laptop is often ±5–10%. diff --git a/benchmarks/build.gradle.kts b/benchmarks/build.gradle.kts new file mode 100644 index 0000000..c4d5ce6 --- /dev/null +++ b/benchmarks/build.gradle.kts @@ -0,0 +1,54 @@ +plugins { + kotlin("jvm") version "2.3.20" + kotlin("plugin.allopen") version "2.3.20" + id("org.jetbrains.kotlinx.benchmark") version "0.4.13" + id("community.flock.kmapper") version "0.0.0-SNAPSHOT" + application +} + +group = rootProject.group +version = rootProject.version + +repositories { + mavenLocal() + mavenCentral() +} + +dependencies { + implementation("org.jetbrains.kotlinx:kotlinx-benchmark-runtime:0.4.13") +} + +kotlin { + jvmToolchain(21) +} + +// JMH requires @State classes to be open (non-final). The all-open plugin +// rewrites them at compile time so we don't have to write `open class`. +allOpen { + annotation("org.openjdk.jmh.annotations.State") +} + +benchmark { + targets { + register("main") + } + configurations { + named("main") { + warmups = 5 + iterations = 5 + iterationTime = 1 + iterationTimeUnit = "s" + outputTimeUnit = "ns" + mode = "avgt" + } + } +} + +application { + mainClass.set("community.flock.kmapper.benchmarks.ManualRunnerKt") +} + +// The kmapper Gradle plugin and the compiler-runtime are resolved from +// mavenLocal at *configuration* time. They must be published BEFORE this +// module is configured — see settings.gradle.kts and benchmarks/README.md +// for the two-step bootstrap workflow. diff --git a/benchmarks/src/main/kotlin/community/flock/kmapper/benchmarks/ComplexDomain.kt b/benchmarks/src/main/kotlin/community/flock/kmapper/benchmarks/ComplexDomain.kt new file mode 100644 index 0000000..10b7fb2 --- /dev/null +++ b/benchmarks/src/main/kotlin/community/flock/kmapper/benchmarks/ComplexDomain.kt @@ -0,0 +1,109 @@ +package community.flock.kmapper.benchmarks + +import community.flock.kmapper.mapper + +/** + * The complex workload: nested data classes, value classes that need + * unwrapping, a list of nested objects, an enum, a derived field, and + * a numeric widening conversion. Designed to exercise as many kmapper + * codegen paths as possible in a single mapping. + */ +@JvmInline +value class CustomerId(val value: String) + +@JvmInline +value class Money(val cents: Long) + +enum class OrderStatus { PENDING, SHIPPED, DELIVERED, CANCELLED } +enum class OrderStatusDto { PENDING, SHIPPED, DELIVERED, CANCELLED } + +data class StreetCity(val street: String, val city: String) +data class StreetCityDto(val street: String, val city: String) + +data class Address( + val streetCity: StreetCity, + val zipCode: String, + val country: String, +) + +data class AddressDto( + val streetCity: StreetCityDto, + val zipCode: String, + val country: String, +) + +data class OrderLine( + val sku: String, + val quantity: Int, + val unitPrice: Money, +) + +data class OrderLineDto( + val sku: String, + val quantity: Long, + val unitPrice: Long, +) + +data class Order( + val id: String, + val customerId: CustomerId, + val firstName: String, + val lastName: String, + val age: Int, + val billingAddress: Address, + val shippingAddress: Address, + val lines: List, + val status: OrderStatus, + val total: Money, +) + +data class OrderDto( + val id: String, + val customerId: String, + val customerName: String, + val age: Long, + val billingAddress: AddressDto, + val shippingAddress: AddressDto, + val lines: List, + val status: OrderStatusDto, + val totalCents: Long, +) + +fun OrderLine.toDtoKMapper(): OrderLineDto = mapper { + quantity = it.quantity.toLong() + unitPrice = it.unitPrice.cents +} + +fun OrderLine.toDtoManual(): OrderLineDto = OrderLineDto( + sku = sku, + quantity = quantity.toLong(), + unitPrice = unitPrice.cents, +) + +fun Order.toDtoKMapper(): OrderDto = mapper { + customerName = "${it.firstName} ${it.lastName}" + age = it.age.toLong() + lines = it.lines.map { line -> line.toDtoKMapper() } + status = OrderStatusDto.valueOf(it.status.name) + totalCents = it.total.cents +} + +fun Order.toDtoManual(): OrderDto = OrderDto( + id = id, + customerId = customerId.value, + customerName = "$firstName $lastName", + age = age.toLong(), + billingAddress = AddressDto( + streetCity = StreetCityDto(billingAddress.streetCity.street, billingAddress.streetCity.city), + zipCode = billingAddress.zipCode, + country = billingAddress.country, + ), + shippingAddress = AddressDto( + streetCity = StreetCityDto(shippingAddress.streetCity.street, shippingAddress.streetCity.city), + zipCode = shippingAddress.zipCode, + country = shippingAddress.country, + ), + lines = lines.map { it.toDtoManual() }, + status = OrderStatusDto.valueOf(status.name), + totalCents = total.cents, +) diff --git a/benchmarks/src/main/kotlin/community/flock/kmapper/benchmarks/ComplexMappingBenchmark.kt b/benchmarks/src/main/kotlin/community/flock/kmapper/benchmarks/ComplexMappingBenchmark.kt new file mode 100644 index 0000000..6e5147e --- /dev/null +++ b/benchmarks/src/main/kotlin/community/flock/kmapper/benchmarks/ComplexMappingBenchmark.kt @@ -0,0 +1,23 @@ +package community.flock.kmapper.benchmarks + +import kotlinx.benchmark.Benchmark +import kotlinx.benchmark.BenchmarkMode +import kotlinx.benchmark.BenchmarkTimeUnit +import kotlinx.benchmark.Mode +import kotlinx.benchmark.OutputTimeUnit +import kotlinx.benchmark.Scope +import kotlinx.benchmark.State + +@State(Scope.Benchmark) +@BenchmarkMode(Mode.AverageTime) +@OutputTimeUnit(BenchmarkTimeUnit.NANOSECONDS) +class ComplexMappingBenchmark { + + private val source = Fixtures.complexOrder + + @Benchmark + fun kmapper(): OrderDto = source.toDtoKMapper() + + @Benchmark + fun manual(): OrderDto = source.toDtoManual() +} diff --git a/benchmarks/src/main/kotlin/community/flock/kmapper/benchmarks/Fixtures.kt b/benchmarks/src/main/kotlin/community/flock/kmapper/benchmarks/Fixtures.kt new file mode 100644 index 0000000..5c1fcfd --- /dev/null +++ b/benchmarks/src/main/kotlin/community/flock/kmapper/benchmarks/Fixtures.kt @@ -0,0 +1,32 @@ +package community.flock.kmapper.benchmarks + +object Fixtures { + val simpleUser = SimpleUser(firstName = "John", lastName = "Doe", age = 42) + + val complexOrder: Order = Order( + id = "ord-0001", + customerId = CustomerId("cust-77"), + firstName = "John", + lastName = "Doe", + age = 42, + billingAddress = Address( + streetCity = StreetCity("Main Street 1", "Hamburg"), + zipCode = "22049", + country = "DE", + ), + shippingAddress = Address( + streetCity = StreetCity("Side Street 99", "Berlin"), + zipCode = "10115", + country = "DE", + ), + lines = List(10) { i -> + OrderLine( + sku = "SKU-$i", + quantity = i + 1, + unitPrice = Money(cents = 100L * (i + 1)), + ) + }, + status = OrderStatus.SHIPPED, + total = Money(cents = 5_500L), + ) +} diff --git a/benchmarks/src/main/kotlin/community/flock/kmapper/benchmarks/ManualRunner.kt b/benchmarks/src/main/kotlin/community/flock/kmapper/benchmarks/ManualRunner.kt new file mode 100644 index 0000000..56e80a4 --- /dev/null +++ b/benchmarks/src/main/kotlin/community/flock/kmapper/benchmarks/ManualRunner.kt @@ -0,0 +1,80 @@ +package community.flock.kmapper.benchmarks + +/** + * A zero-config benchmark runner for quick sanity checks. + * + * Run with: `./gradlew -Pkmapper.benchmarks=true :benchmarks:run` + * + * This is intentionally simpler than the JMH harness — it uses + * `System.nanoTime()` and a manual warmup loop. The numbers it produces are + * indicative, not authoritative: JIT effects, GC pauses, and dead-code + * elimination can all skew results. For trustworthy numbers, use: + * `./gradlew -Pkmapper.benchmarks=true :benchmarks:benchmark` + * + * The runner exists so that anyone can replicate the comparison in seconds + * without learning JMH, and so that CI smoke tests can detect order-of- + * magnitude regressions cheaply. + */ +private const val WARMUP_ITERATIONS = 100_000 +private const val MEASURE_ITERATIONS = 1_000_000 + +fun main() { + println("kmapper micro-benchmark (System.nanoTime, indicative only)") + println("Warmup: $WARMUP_ITERATIONS iters, Measure: $MEASURE_ITERATIONS iters") + println("For accurate numbers, run: ./gradlew -Pkmapper.benchmarks=true :benchmarks:benchmark") + println() + + runScenario( + name = "Simple (3-field flat data class)", + kmapper = { Fixtures.simpleUser.toDtoKMapper() }, + manual = { Fixtures.simpleUser.toDtoManual() }, + ) + + runScenario( + name = "Complex (nested + value classes + list of 10 + enum + widening)", + kmapper = { Fixtures.complexOrder.toDtoKMapper() }, + manual = { Fixtures.complexOrder.toDtoManual() }, + ) +} + +private inline fun runScenario( + name: String, + crossinline kmapper: () -> Any, + crossinline manual: () -> Any, +) { + println("=== $name ===") + + // Warmup — interleave both to trigger JIT compilation on both call sites + // before either is measured. + var sink: Any = Unit + repeat(WARMUP_ITERATIONS) { + sink = kmapper() + sink = manual() + } + + val kmapperNs = measure(MEASURE_ITERATIONS) { kmapper() } + val manualNs = measure(MEASURE_ITERATIONS) { manual() } + + val kmapperPerOp = kmapperNs.toDouble() / MEASURE_ITERATIONS + val manualPerOp = manualNs.toDouble() / MEASURE_ITERATIONS + val overheadPct = ((kmapperPerOp - manualPerOp) / manualPerOp) * 100.0 + + println(" manual : %.2f ns/op".format(manualPerOp)) + println(" kmapper: %.2f ns/op".format(kmapperPerOp)) + println(" overhead: %+.2f%%".format(overheadPct)) + println(" (sink to defeat DCE: ${sink::class.simpleName})") + println() +} + +private inline fun measure(iterations: Int, crossinline op: () -> Any): Long { + // Capture into a volatile-ish field so the JIT can't elide the call. + var sink: Any = Unit + val start = System.nanoTime() + repeat(iterations) { + sink = op() + } + val end = System.nanoTime() + // Touch sink so the loop body has an observable side effect. + if (sink.hashCode() == Int.MIN_VALUE) println("unreachable") + return end - start +} diff --git a/benchmarks/src/main/kotlin/community/flock/kmapper/benchmarks/SimpleDomain.kt b/benchmarks/src/main/kotlin/community/flock/kmapper/benchmarks/SimpleDomain.kt new file mode 100644 index 0000000..f49d554 --- /dev/null +++ b/benchmarks/src/main/kotlin/community/flock/kmapper/benchmarks/SimpleDomain.kt @@ -0,0 +1,28 @@ +package community.flock.kmapper.benchmarks + +import community.flock.kmapper.mapper + +/** + * The simple workload: a flat data class mapped to another flat data class + * with one derived field. This is the canonical kmapper README example and + * the smallest mapping that exercises the DSL block. + */ +data class SimpleUser( + val firstName: String, + val lastName: String, + val age: Int, +) + +data class SimpleUserDto( + val name: String, + val age: Int, +) + +fun SimpleUser.toDtoKMapper(): SimpleUserDto = mapper { + name = "${it.firstName} ${it.lastName}" +} + +fun SimpleUser.toDtoManual(): SimpleUserDto = SimpleUserDto( + name = "$firstName $lastName", + age = age, +) diff --git a/benchmarks/src/main/kotlin/community/flock/kmapper/benchmarks/SimpleMappingBenchmark.kt b/benchmarks/src/main/kotlin/community/flock/kmapper/benchmarks/SimpleMappingBenchmark.kt new file mode 100644 index 0000000..3cd7f26 --- /dev/null +++ b/benchmarks/src/main/kotlin/community/flock/kmapper/benchmarks/SimpleMappingBenchmark.kt @@ -0,0 +1,23 @@ +package community.flock.kmapper.benchmarks + +import kotlinx.benchmark.Benchmark +import kotlinx.benchmark.BenchmarkMode +import kotlinx.benchmark.BenchmarkTimeUnit +import kotlinx.benchmark.Mode +import kotlinx.benchmark.OutputTimeUnit +import kotlinx.benchmark.Scope +import kotlinx.benchmark.State + +@State(Scope.Benchmark) +@BenchmarkMode(Mode.AverageTime) +@OutputTimeUnit(BenchmarkTimeUnit.NANOSECONDS) +class SimpleMappingBenchmark { + + private val source = Fixtures.simpleUser + + @Benchmark + fun kmapper(): SimpleUserDto = source.toDtoKMapper() + + @Benchmark + fun manual(): SimpleUserDto = source.toDtoManual() +} diff --git a/settings.gradle.kts b/settings.gradle.kts index 6f3ba8c..3f0566f 100644 --- a/settings.gradle.kts +++ b/settings.gradle.kts @@ -1,5 +1,6 @@ pluginManagement { repositories { + mavenLocal() mavenCentral() gradlePluginPortal() maven(url="https://central.sonatype.com/repository/maven-snapshots/") @@ -10,6 +11,7 @@ pluginManagement { dependencyResolutionManagement { repositories { + mavenLocal() mavenCentral() maven(url="https://central.sonatype.com/repository/maven-snapshots/") maven("https://storage.googleapis.com/gradleup/m2") @@ -30,3 +32,16 @@ include( ":test-framework", ":test-integration" ) + +// :benchmarks applies the kmapper Gradle plugin to its own sources and so +// requires the plugin to already be in mavenLocal at *configuration* time. +// We gate inclusion behind an opt-in property to avoid breaking the parent +// build before the bootstrap publish has happened. +// +// 1. Publish: ./gradlew :compiler-plugin:publishToMavenLocal \ +// :compiler-runtime:publishToMavenLocal \ +// :gradle-plugin:publishToMavenLocal +// 2. Run: ./gradlew -Pkmapper.benchmarks=true :benchmarks:benchmark +if (providers.gradleProperty("kmapper.benchmarks").orNull == "true") { + include(":benchmarks") +} From 77dc26a9234efbc7fd2049c0dcf573e38afcfe4f Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 25 Apr 2026 11:45:56 +0000 Subject: [PATCH 2/3] ci: gate benchmarks on a threshold ratio and run on every PR Adds a `verifyBenchmarkThresholds` Gradle task that parses the JMH JSON report and fails the build if any kmapper benchmark is more than `benchmarkMaxRatio`x slower than its hand-written counterpart (default 5x). Wires the task into a new `benchmarks` job in build.yml that publishes the plugin artifacts to mavenLocal, runs the threshold gate, and uploads the JMH report as an artifact. The 5x default is loose on purpose: GitHub-hosted runners share CPUs and don't pin frequency, so single-digit-percent variance is normal. The gate catches catastrophic regressions (broken codegen, accidental reflection) without flaking on noise. Tighter ratios can be set locally via `-PbenchmarkMaxRatio=2.0`. Also makes warmups/iterations configurable via `-PbenchmarkWarmups` / `-PbenchmarkIterations` so the same task serves both the fast CI smoke and a slow-but-trustworthy local run. --- .github/workflows/build.yml | 44 ++++++++++++++++- benchmarks/README.md | 36 ++++++++++++-- benchmarks/build.gradle.kts | 96 +++++++++++++++++++++++++++++++++++-- 3 files changed, 165 insertions(+), 11 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index a8abc68..7297a8e 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -22,4 +22,46 @@ jobs: uses: gradle/actions/setup-gradle@v3 - name: Build and test - run: ./gradlew build \ No newline at end of file + run: ./gradlew build + + benchmarks: + # Smoke-tests the :benchmarks module on every PR. The threshold gate fails + # the build only on catastrophic regressions (kmapper >5x slower than the + # hand-written equivalent) — GitHub-hosted runners are too noisy to detect + # smaller deltas reliably. The full JMH JSON report is uploaded as an + # artifact for human review. + runs-on: ubuntu-latest + needs: build + + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Set up JDK + uses: actions/setup-java@v4 + with: + java-version: '21' + distribution: 'temurin' + + - name: Setup Gradle + uses: gradle/actions/setup-gradle@v3 + + - name: Publish kmapper artifacts to mavenLocal + run: | + ./gradlew \ + :compiler-plugin:publishToMavenLocal \ + :compiler-runtime:publishToMavenLocal \ + :gradle-plugin:publishToMavenLocal + + - name: Run benchmarks with threshold gate + run: | + ./gradlew -Pkmapper.benchmarks=true \ + :benchmarks:verifyBenchmarkThresholds + + - name: Upload JMH report + if: always() + uses: actions/upload-artifact@v4 + with: + name: jmh-report + path: benchmarks/build/reports/benchmarks/ + if-no-files-found: warn \ No newline at end of file diff --git a/benchmarks/README.md b/benchmarks/README.md index d9a0608..446e3d9 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -49,12 +49,18 @@ Subsequent commands all need `-Pkmapper.benchmarks=true` to include the module. ``` This forks a fresh JVM, runs a configurable warmup, measures average time -per op in nanoseconds, and writes a report to `build/reports/benchmarks/`. -Configuration lives in `build.gradle.kts` under `benchmark { configurations { ... } }` -(warmups, iterations, iteration time). JMH guards against dead-code -elimination, constant folding, and on-stack replacement skew. +per op in nanoseconds, and writes a JSON report to `build/reports/benchmarks/main/`. +Defaults are tuned for CI (warmups=2, iterations=3, 1s each) so the whole +run completes in under a minute. For trustworthy numbers on a quiet +workstation, override: -This is the number to quote in PRs and release notes. +```bash +./gradlew -Pkmapper.benchmarks=true :benchmarks:benchmark \ + -PbenchmarkWarmups=5 -PbenchmarkIterations=10 +``` + +JMH guards against dead-code elimination, constant folding, and on-stack +replacement skew. This is the number to quote in PRs and release notes. ### 2. Standalone nanoTime runner (replicable smoke test) @@ -74,6 +80,26 @@ optimizations the way JMH can. Useful for: If JMH and the manual runner disagree by more than a few percent, **trust JMH**. +### 3. Threshold gate (CI / regression guard) + +```bash +./gradlew -Pkmapper.benchmarks=true :benchmarks:verifyBenchmarkThresholds +``` + +Runs `:benchmark`, then parses the JSON report and **fails the build** if any +kmapper benchmark is more than `benchmarkMaxRatio`x slower than its manual +counterpart. The default ratio is `5.0` — deliberately loose, because +GitHub-hosted runners are noisy and small-percent differences are not +distinguishable from variance. Override locally for a stricter check: + +```bash +./gradlew -Pkmapper.benchmarks=true :benchmarks:verifyBenchmarkThresholds \ + -PbenchmarkMaxRatio=2.0 +``` + +The CI job in [`.github/workflows/build.yml`](../.github/workflows/build.yml) +runs this task on every PR and uploads the JMH report as a build artifact. + ## Other approaches considered The current setup picks **kotlinx-benchmark + a simple `main()` runner** diff --git a/benchmarks/build.gradle.kts b/benchmarks/build.gradle.kts index c4d5ce6..1fc8187 100644 --- a/benchmarks/build.gradle.kts +++ b/benchmarks/build.gradle.kts @@ -28,18 +28,26 @@ allOpen { annotation("org.openjdk.jmh.annotations.State") } +// CI-friendly defaults: keep total wall time under ~1 min so benchmarks can +// run on every PR. Override locally for trustworthy numbers, e.g.: +// ./gradlew -Pkmapper.benchmarks=true :benchmarks:benchmark \ +// -PbenchmarkWarmups=5 -PbenchmarkIterations=10 +val benchmarkWarmups = (project.findProperty("benchmarkWarmups") as String? ?: "2").toInt() +val benchmarkIterations = (project.findProperty("benchmarkIterations") as String? ?: "3").toInt() + benchmark { targets { register("main") } configurations { named("main") { - warmups = 5 - iterations = 5 + warmups = benchmarkWarmups + iterations = benchmarkIterations iterationTime = 1 iterationTimeUnit = "s" outputTimeUnit = "ns" mode = "avgt" + reportFormat = "json" } } } @@ -49,6 +57,84 @@ application { } // The kmapper Gradle plugin and the compiler-runtime are resolved from -// mavenLocal at *configuration* time. They must be published BEFORE this -// module is configured — see settings.gradle.kts and benchmarks/README.md -// for the two-step bootstrap workflow. +// mavenLocal. They must be published BEFORE this module compiles, so wire +// the dependency explicitly (mirrors test-integration/build.gradle.kts). +val publishKmapperLocally = listOf( + ":compiler-plugin:publishToMavenLocal", + ":compiler-runtime:publishToMavenLocal", + ":gradle-plugin:publishToMavenLocal", +) +tasks.named("compileKotlin") { publishKmapperLocally.forEach { dependsOn(it) } } +tasks.matching { it.name == "mainBenchmarkCompile" || it.name == "mainBenchmark" } + .configureEach { publishKmapperLocally.forEach { dependsOn(it) } } + +// Threshold gate: parse the JMH JSON report and fail if any kmapper benchmark +// is more than `benchmarkMaxRatio`x slower than its hand-written counterpart. +// The default ratio is deliberately loose (5x) because GitHub-hosted runners +// are noisy. Override locally for tighter checks: +// ./gradlew -Pkmapper.benchmarks=true :benchmarks:verifyBenchmarkThresholds \ +// -PbenchmarkMaxRatio=2.0 +val verifyBenchmarkThresholds by tasks.registering { + group = "verification" + description = "Fails if kmapper-generated mappers are >Nx slower than hand-written equivalents" + dependsOn("benchmark") + + val maxRatio = (project.findProperty("benchmarkMaxRatio") as String? ?: "5.0").toDouble() + val reportRoot = layout.buildDirectory.dir("reports/benchmarks/main") + + doLast { + val root = reportRoot.get().asFile + val jsonFile = root.walkTopDown() + .filter { it.isFile && it.extension == "json" } + .maxByOrNull { it.lastModified() } + ?: error("No JMH JSON report found under $root — did `:benchmark` run with reportFormat=\"json\"?") + + // Minimal hand-rolled parse: each entry has + // "benchmark": ".", "primaryMetric": { "score": , ... } + val text = jsonFile.readText() + val entryRegex = Regex( + """"benchmark"\s*:\s*"([^"]+)"[\s\S]*?"primaryMetric"\s*:\s*\{[\s\S]*?"score"\s*:\s*([0-9.eE+-]+)""" + ) + val scores: Map = entryRegex.findAll(text).associate { m -> + // benchmark id is "...SimpleMappingBenchmark.kmapper" — key by ClassName.method + val id = m.groupValues[1].split('.').takeLast(2).joinToString(".") + id to m.groupValues[2].toDouble() + } + require(scores.isNotEmpty()) { + "Could not parse any benchmark scores from $jsonFile" + } + + val pairs = listOf( + "SimpleMappingBenchmark" to ("manual" to "kmapper"), + "ComplexMappingBenchmark" to ("manual" to "kmapper"), + ) + val report = StringBuilder().appendLine( + "Benchmark threshold report (max ratio = ${maxRatio}x, source: ${jsonFile.name}):" + ) + val failures = mutableListOf() + for ((cls, methods) in pairs) { + val (manualMethod, kmapperMethod) = methods + val manualKey = "$cls.$manualMethod" + val kmapperKey = "$cls.$kmapperMethod" + val manualScore = scores[manualKey] + val kmapperScore = scores[kmapperKey] + if (manualScore == null || kmapperScore == null) { + failures += "missing scores for $cls (manual=$manualScore, kmapper=$kmapperScore)" + continue + } + val ratio = kmapperScore / manualScore + report.appendLine( + " %s: kmapper=%.2f ns, manual=%.2f ns, ratio=%.2fx".format( + cls, kmapperScore, manualScore, ratio, + ) + ) + if (ratio > maxRatio) { + failures += "$cls: kmapper is ${"%.2f".format(ratio)}x slower than manual (limit=${maxRatio}x)" + } + } + logger.lifecycle(report.toString()) + if (failures.isNotEmpty()) { + error("Benchmark threshold violations:\n - " + failures.joinToString("\n - ")) + } + } +} From 81624cb06024b531cd760673dcd5f5389550a736 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 25 Apr 2026 12:03:57 +0000 Subject: [PATCH 3/3] docs: mention performance guarantees and CI benchmark gate in README Adds a short Performance section linking to the new :benchmarks module and explaining the 5x CI threshold gate. --- README.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/README.md b/README.md index d1ee664..9d92d41 100644 --- a/README.md +++ b/README.md @@ -304,6 +304,12 @@ val dto: PersonDto = Person("John").mapper { The plugin automatically generates the mapping implementation at compile time, replacing the `mapper` function call with the actual object construction code. +### Performance + +Because KMapper is a compile-time codegen plugin (no reflection, no runtime registry), the generated code is equivalent to a hand-written constructor call and runtime overhead is essentially zero. + +This is verified by JMH benchmarks in [`benchmarks/`](benchmarks/), which compare `mapper { ... }` against hand-written equivalents for both a flat 3-field mapping and a nested mapping with value classes, lists, enums, and numeric widening. The CI pipeline runs the suite on every PR and fails the build if a kmapper benchmark is more than 5× slower than its manual counterpart, guarding against accidental regressions. + Notes: