Replace Intel Pin and implement eBPF-based coverage#116
Merged
Conversation
Drop Intel Pin (proprietary, ~100MB SDK download, C++ build step) in favor of bpftrace eBPF tracing and a native Go ELF shim binary that transparently replaces the target binary at install time. - delete FuncTracer.cpp/hpp, makefile/makefile.rules, dynamorio/, wrapunwrap.go - add cmd/elfutil.go: shared ELF helpers (isELF, hasDebugInfo, build-id, move) - add cmd/enumerate.go: DWARF-based function enumeration with demangling - add cmd/shim.go: install/uninstall + setcap setup for bpftrace - add cmd/trace.go: inline tracing without permanent install - add cmd/shim_binary/main.go: ELF shim with race-free fork-exec coordination - update cmd/report.go: dual log format (FUNC + CALLED) - add tests/sample/: 100-function C++ test binary in 4 groups - add tests/e2e/test_coverage.py: bpftrace E2E suite (replaces Pin test_calc.py) - update build.sh, run_unit_tests.sh, CI workflows Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The previous parseLibs/unmarshalStringSlice was dead code: declared an unused variable, took a redundant *[]string param, and hand-rolled JSON parsing that broke on library paths containing commas. Replaced with a straightforward encoding/json.Unmarshal in a single readLibsSidecar helper. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously the binary/library path was substituted into the printf format string itself. A path containing '%' would be interpreted as a format directive and break the script (or worse, read garbage from the stack). Now the path is emitted as a quoted bpftrace string argument with backslash and double-quote escaping. Also extracted writeUprobeBlock to remove the duplicate per-library block code. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
elfutil.getBuildID: check binary.Read errors on the three uint32 note header fields and bounds-check the descsz before slicing. shim child: a failed wait-pipe read (parent died, fd closed early) now exits with an explicit error instead of silently exec'ing the real binary untraced. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
printTxtReport iterated CalledFunctions and TotalFunctions maps directly, producing non-deterministic output. Added splitCalledUncalled helper that returns sorted slices, and use it from printTxtReport. The helper will also be reused by the XML and HTML report generators. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- safeNameRe + safeImageName helper at package level (was rebuilt on every generateXUnitReport / generateHTMLReport call) - generateXUnitReport and generateHTMLReport now use splitCalledUncalled instead of duplicating the called/uncalled bucketing logic - HTML report now lists called functions before uncalled, both sorted Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace sort.Slice (Go 1.8) with slices.SortFunc + cmp.Compare (Go 1.21+), the modern idiomatic form. Drops the sort import. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The manual string-splitting approach (Contains, SplitN, Index, TrimSpace) was hard to follow and fragile. Replaced with a single regexp that captures the absolute library path from either form of ldd line. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The function had six near-identical error-cleanup blocks (close pipe, kill child, wait child) repeated at every fail point. Replaced with a defer that runs LIFO cleanup callbacks, each appended only after the matching resource is acquired. Also extracted three single-purpose helpers from the body: - writeBpftraceScript: render+temp-file - openCalledLog: timestamped log creation - captureCalledLog: stdout-capture goroutine - waitForAttach: stderr "Attaching" detection with timeout Also remove the trivial buildBpftraceArgs helper (replaced by inline exec.Command call) since CAP_BPF is documented in the script header. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…able Each subcommand now lives in its own cmdX(args []string) error function. main() is reduced to: parse arg, look up command, run it, format error. Common error formatting moves up; per-command os.Exit(1) calls go away. Aliases (-h, --help, -v, --version, -r) are registered alongside their canonical names. The "wrap"/"unwrap" legacy aliases stay as explicit guards because they emit a redirect message rather than running anything. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both cmd/ and cmd/shim_binary/ are package main, so they cannot share unexported identifiers directly. Each was independently defining default LOG_DIR / SAFE_BIN_DIR constants, getEnv-with-default boilerplate, the @Version suffix stripper, and the libs.json read/write logic. Move these into internal/funkutil: - DefaultLogDir, DefaultSafeBinDir constants - EnvOr, LogDir, SafeBinDir helpers - StripVersion (was stripVersionSuffix in two places) - LibsSidecarPath, ReadLibsSidecar, WriteLibsSidecar Side benefits: - shim.go and trace.go now use lowerCamel locals (logDir, safeBinDir) instead of LOG_DIR/SAFE_BIN_DIR - trace.go uses errors.As for the ExitError check (was a type assertion) - shim.go uninstall now goes through WriteLibsSidecar(safe, nil) for delete, so the "remove .libs.json" pathway lives in one place Adds funkutil_test.go covering EnvOr, StripVersion, and the sidecar roundtrip / delete / malformed-input cases. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two trace-killing bugs in the bpftrace script generator:
1. waitForAttach matched only "Attaching N probes..." (bpftrace ≤0.24).
bpftrace ≥0.25 prints "Attached N probes" (past tense) — every run
hit FUNKOVERAGE_ATTACH_TIMEOUT and the child unblocked untraced.
2. sched_process_exit { delete(@watched[pid]); } deleted the watched
entry mid-execve. When a non-leader Go thread did syscall.Exec,
the kernel's de_thread() killed the old TGID leader; that fired
sched_process_exit with args->pid == TGID, wiping @watched[<our PID>]
before any uprobe in the new program could fire. Result: zero CALLED
lines on every shim invocation.
Map cleanup is unnecessary because bpftrace's lifetime equals the shim
child's lifetime; the map is freed when bpftrace exits.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
ParseLddLibraries returned every shared library, including libc.so, libstdc++, libpthread etc. The shim then asked bpftrace to attach uprobe wildcards on each — libc alone has ~7600 symbols, so attach exceeded FUNKOVERAGE_ATTACH_TIMEOUT and tracing started untraced (or not at all). System libs are rarely useful for application coverage; users wanting them can drop the filter or run with --no-libs == false on a wrapper. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
bpftrace's `uprobe:<path>:*` wildcard expands via the ELF symbol table, not DWARF, so symtab is the source of truth for which functions we can actually trace. Reading DWARF first also breaks on dwz-compressed distros (openSUSE, Fedora): most DW_TAG_subprogram entries lack inline name+low_pc and instead reference abstract DIEs in a separate .gnu_debugaltlink file that Go's debug/dwarf package cannot follow. unzip on openSUSE yielded 2 of 70 symtab functions. New order: binary .symtab → external .debug .symtab → DWARF (last resort for binaries with no symtab anywhere). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
When external .debug files aren't in .build-id/ layout, check for .gnu_debugaltlink section (dwz-compressed DWARF on openSUSE/Fedora). externalDebugPath now tries in order: 1. .build-id/<XX>/<YYY>.debug (standard Fedora/Debian) 2. Path from .gnu_debugaltlink section + search heuristics 3. /usr/lib/debug/.dwz/ brute-force by binary name prefix Limitation: dwz files are pure DWARF supplements (no symtab, no symbols). Go's debug/dwarf package doesn't follow alt-link references, so DWARF-only functions remain uncovered. openssh strips binaries and orphans its dwz file (no debugaltlink left in binary), making it unsupported by this approach.
When installing the shim, mergeDebugIfExternal now searches for dwz files using the same heuristics as externalDebugPath: 1. .build-id path (standard) 2. .gnu_debugaltlink section (unstripped binaries with alt-link) 3. /usr/lib/debug/.dwz/ brute-force by name contains() match This handles fully-stripped packages like openssh which have orphaned dwz files in .dwz/ with no .build-id or .gnu_debugaltlink to guide us. eu-unstrip merges the dwz DWARF directly into the binary at install time, making the function names available for enumeration. Tested: openssh /usr/bin/ssh now yields 3332 functions (vs 2 unmerged). Also refactored duplicate readGnuDebugAltLink/findDebugFile logic into elfutil.go helpers for both install and enumerate paths.
EnumerateFunctions must run after mergeDebugIfExternal so that all merged DWARF symbols are available. Previously, functions were enumerated from the original stripped binary, yielding only symtab entries (e.g., openssh: 2 funcs). Now enumerate runs post-merge, capturing the full DWARF (openssh: 3332 funcs).
dwz debug files are ET_REL (relocatable objects), while PIE binaries are ET_DYN. eu-unstrip requires --force to merge them. Applied the fix for better robustness across ELF type combinations.
…openssh openssh's dwz file is named 'openssh-10.3p1-2.1.x86_64', not 'ssh-*'. externalDebugPath was checking HasPrefix(dwzName, binName) which fails. Switch to same heuristic as hasDebugInfo: check both HasPrefix and Contains. Result: /usr/bin/ssh now enumerates 725 functions instead of 2. Tested: ssh -V via shim traces 30 functions, 4.1% coverage.
Remove hasDebugInfo check from library enumeration. Symbols are enumerable from .symtab / .dynsym even without DWARF debug sections. bpftrace can attach uprobes to any symbol in the dynamic symbol table, regardless of debug info. Result: /usr/bin/ssh now enumerates 8038 functions (ssh + 11 libraries) instead of 725 (ssh only). Libraries without embedded debug sections (libcrypto, libz, etc.) are now fully traced. Tested: ssh -V via shim traces 436 function calls (lib functions + ssh funcs).
…-flight events to drain The ringbuf.Reader.Read() was blocking until close(), so closing it immediately (before waiting for the drain) would discard events still in the kernel queue. Events produced by uprobes fire into the ringbuf until detach happens; the reader goroutine consumes them asynchronously. To avoid losing events on short-lived programs: 1. Detach all links first (stops new events from queuing) 2. Start a goroutine that closes the reader after 100ms (gives in-flight events time to drain) 3. Wait for the reader goroutine to exit on close 4. Close log and BPF objects Also fix the trace command to pass the real binary name via env var, since the shim can be invoked directly with its own name (trace doesn't require permanent install). Test results: sample --strings now captures 25 unique str_ functions (>15 threshold). All E2E tests pass (16/16, 2 skipped).
Contributor
There was a problem hiding this comment.
Pull request overview
This PR replaces the previous Intel Pin / wrapper-based approach with a new “shim + eBPF” implementation for function-level coverage on Linux, introduces new enumeration/sidecar formats, and updates tests/docs/packaging accordingly.
Changes:
- Remove Intel Pin tooling (FuncTracer, wrap/unwrap logic, old tests) and introduce shim-based install/uninstall + inline trace flows.
- Add native eBPF tracer (cilium/ebpf + uprobe.multi + ringbuf) plus function enumeration and sidecar helpers.
- Add a new C++ sample binary and Python E2E tests; refresh CI workflows and packaging metadata.
Reviewed changes
Copilot reviewed 44 out of 49 changed files in this pull request and generated 16 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_func_tracer.cpp | Removed old Catch2 unit test for Pin-era relevance filtering. |
| tests/sample/Makefile | Adds build recipe for the new sample binary used in E2E. |
| tests/sample/main.cpp | Sample driver that exercises many functions by category. |
| tests/sample/functions.h | Declares sample functions grouped by prefix. |
| tests/sample/functions.cpp | Implements sample functions used to generate traceable coverage. |
| tests/e2e/test_coverage.py | New E2E test suite for install/uninstall/enumerate/report flows. |
| tests/e2e/test_calc.py | Removed old Pin wrapper E2E test. |
| run_unit_tests.sh | Drops C++ unit tests; keeps Go unit tests. |
| rpm/coverage-tools.spec | New RPM spec for the eBPF-based tooling/shim packaging. |
| README.md | Updates project description and usage to the new approach. |
| makefile.rules | Removed Intel Pin build/test rules. |
| makefile | Removed Intel Pin top-level makefile integration. |
| internal/funkutil/funkutil.go | Adds shared helpers (env defaults, symbol/version stripping, sidecars). |
| internal/funkutil/funkutil_test.go | Unit tests for funkutil helpers. |
| internal/funkutil/funclist.go | Adds .funcs.json sidecar helpers for shim seeding. |
| internal/funkutil/funclist_test.go | Unit tests for function-list sidecar helpers. |
| go.sum | Adds deps for cilium/ebpf and x/sys. |
| go.mod | Updates Go version and adds indirect eBPF-related deps. |
| FuncTracer.hpp | Removed Pin-era tracer header. |
| FuncTracer.cpp | Removed Pin-era tracer implementation. |
| dynamorio/test_program.c | Removed DynamoRIO experiment artifacts. |
| dynamorio/Makefile | Removed DynamoRIO experiment artifacts. |
| dynamorio/drcov.cpp | Removed DynamoRIO experiment artifacts. |
| dynamorio/drcov_tool_test/drtool_cov.txt | Removed DynamoRIO experiment artifacts. |
| dynamorio/drcov_tool_test/coverage.info | Removed DynamoRIO experiment artifacts. |
| dynamorio/CMakeLists.txt | Removed DynamoRIO experiment artifacts. |
| dynamorio/build.sh | Removed DynamoRIO experiment artifacts. |
| docs/incremental-tracing.md | Adds design notes for scaling tracing to very large function counts. |
| docs/ebpf-scaling.md | Adds scaling constraints analysis and proposed solutions. |
| cmd/wrapunwrap.go | Removed Pin wrapper implementation. |
| cmd/trace.go | Adds inline trace mode that runs shim without permanent install. |
| cmd/templates.go | Updates CLI help text for new commands/flow. |
| cmd/shim.go | Implements install/uninstall/setup for shim-based coverage. |
| cmd/shim_binary/tracer.go | New native eBPF tracer implementation (uprobes + ringbuf). |
| cmd/shim_binary/tracer_x86_bpfel.go | Generated bpf2go bindings for embedded BPF object. |
| cmd/shim_binary/tracer_test.go | Unit tests for tracer function flattening and validation. |
| cmd/shim_binary/main.go | Shim entrypoint: fork/attach tracer/exec real binary. |
| cmd/shim_binary/bpf/tracer.bpf.c | eBPF programs (uprobe.multi + fork tracepoint) and maps. |
| cmd/report.go | Updates report parsing to new FUNC / CALLED log formats. |
| cmd/funkoverage.go | Refactors CLI into subcommands (setup/install/uninstall/trace/…). |
| cmd/funkoverage_test.go | Updates/expands tests for new enumeration, logs, install/uninstall, etc. |
| cmd/enumerate.go | Adds ELF/DWARF enumeration and _functions.log writer. |
| cmd/elfutil.go | Adds ELF helpers (debug discovery, unstrip, build-id helpers, move). |
| build.sh | Reworks build to Go-only build and optional BPF regeneration. |
| .gitignore | Updates ignored artifacts list for new binaries/build outputs. |
| .github/workflows/test.yml | Updates CI deps and adds sample/E2E steps. |
| .github/workflows/build.yml | Updates CI build dependencies. |
Files not reviewed (1)
- cmd/shim_binary/tracer_x86_bpfel.go: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…ace refs - .gitignore: negation for embedded BPF .o so fresh clones build - tracer.go: sync.Once for idempotent Stop(), -cflags in go:generate - trace.go: return (int, error) instead of os.Exit bypassing defers - shim.go: hard error + rollback on enum failure, preserve file perms, clean multicall symlinks on uninstall - shim_binary/main.go: preserve FUNKOVERAGE_ARG0 across cleanEnv - templates.go, funkutil.go: bpftrace refs → eBPF - test_coverage.py: bpftrace checks → BTF/shim cap checks - CI: actions/setup-go@v5, go 1.26, drop bpftrace - README: kernel 6.6+, Go 1.26+, add bpftool/libbpf-devel - rpm spec: fix URL, add commented regen deps - functions.cpp: fix UB in toupper/tolower casts - Regenerated tracer_x86_bpfel.o
Add standalone test scripts for bzip2, squid, and openssl that exercise funkoverage against real system binaries on openSUSE. Each script is atomic (installs deps, cleans state, shims, exercises, reports, uninstalls) with shared helpers in lib_test_helpers.sh. - test_bzip2.sh: basic shim lifecycle, 9 compression modes - test_squid.sh: C++ daemon with demangling verification - test_openssl.sh: multi-library tracing across 5+ images - Delete tests/catch2/ (unused since Intel Pin removal)
Filter which functions get uprobed by demangled name regex. Available on enumerate, install, and trace commands. Inspired by maxgio92/xcover's filtering approach.
Comment on lines
+11
to
+17
| "structs" | ||
|
|
||
| "github.com/cilium/ebpf" | ||
| ) | ||
|
|
||
| type tracerEvent struct { | ||
| _ structs.HostLayout |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Agent-Logs-Url: https://github.com/ilmanzo/BinaryCoverage/sessions/409819a2-ce86-4b00-896e-76ae5c5876ae Co-authored-by: ilmanzo <1872816+ilmanzo@users.noreply.github.com>
Agent-Logs-Url: https://github.com/ilmanzo/BinaryCoverage/sessions/1f1edb35-5f43-4843-a0fb-a1e03fa0284b Co-authored-by: ilmanzo <1872816+ilmanzo@users.noreply.github.com>
Agent-Logs-Url: https://github.com/ilmanzo/BinaryCoverage/sessions/1f1edb35-5f43-4843-a0fb-a1e03fa0284b Co-authored-by: ilmanzo <1872816+ilmanzo@users.noreply.github.com>
BPF programs require GPL to use kernel helpers like bpf_get_attach_cookie and bpf_ringbuf_reserve. Userspace Go code remains MIT. Updated LICENSE, spec file, and SPDX header.
ilmanzo
changed the title
- Consolidate duplicate Row/CoverageSummary, AggregateData/CoverageTotals - Generic writeJSON/readJSON for libs + funcs sidecars (dedupe ~50 LOC) - Extract addFilterFlags helper (dedupe across install/trace/enumerate) - Extract acceptFunc helper (dedupe in symtab + DWARF enumerators) - Extract scanLog from analyzeLogs (fix defer-in-loop accumulation) - Extract locateExternalDebugForMerge (single defer f.Close()) - Extract shimSearchPaths from findShimBinary - Use slices.Sorted+maps.Keys in summarizeCoverage, flattenFuncs - Use cmp.Or for debug path fallback - Use strings.SplitSeq for format dispatch - funcBlacklist: map → slice + slices.Contains - Named struct literals in ensureCoverage - Pre-size cleanEnv slice - Drop obsolete bpftrace/Pin references in comments
…refs - New docs/design.md: full architecture walkthrough with ASCII diagrams covering install/run/report lifecycle, eBPF program structure, data formats, concurrency model, permissions, and licensing rationale. - README: trim Pin reference, point to docs/design.md.
- docs/install.md: comprehensive fresh-system setup guide - System requirements (kernel 6.6+, BTF, Go 1.26+) - Package installation (zypper/dnf/apt) - Build verification (unit tests, E2E, system binaries) - Common issues (setcap, debug packages, BTF) - Verified on openSUSE Tumbleweed 20260511 kernel 7.0.5 - README.md: link to install.md, add dlopen() TODO - Libraries loaded via dlopen() not currently traced (only DT_NEEDED libs) - Possible fixes: uprobe on dlopen() or /proc/pid/maps polling
Node.js 20 EOL April 2026; forced migration June 2nd. Addresses deprecation warning for actions/checkout@v4 and actions/setup-go@v5.
- Add arm64 bpf2go generate directive
- Ship pre-generated tracer_{x86,arm64}_bpfel.{go,o}
- Build tags auto-select correct variant (no runtime wrapper needed)
- Update docs: README, CLAUDE.md, design.md, install.md
- Update .gitignore to track arm64 BPF object
Architecture detection is handled by Go build tags embedded in generated files.
ebpf-scaling.md and incremental-tracing.md described constraints of the old bpftrace architecture (bytecode size limits, 8K-10K probe ceiling). Current cilium/ebpf + uprobe_multi implementation: - Single syscall per image (link.UprobeMulti), not per-probe script - Tiny BPF program (dedup + ringbuf), no probe-count bytecode bloat - Scales to 100K+ functions (kernel perf_event limit, not BPF complexity) Both docs no longer apply.
This was referenced May 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A massive refactor, mostly an experiment. Can we drop PIN and just use eBPF ?
see also https://github.com/maxgio92/xcover (presented at FOSDEM 2026)