Add xcover userspace mode#96
Closed
maxgio92 wants to merge 81 commits into
Closed
Conversation
maxgio92
force-pushed
the
xcover-userspace
branch
2 times, most recently
from
8fb1415 to
073e00d
Compare
…ection helpers Introduces pkg/bpftime with two core primitives: - EnsureSyscallServer: extracts the embedded syscall-server .so into a memfd and re-execs the current process with LD_PRELOAD set, so the library is loaded before any BPF syscall is issued. Uses an env sentinel (XCOVER_BPFTIME_LOADED) to avoid infinite re-exec loops. - ExtractAgent: writes the embedded agent .so to a temp file and returns its path, ready to be used as LD_PRELOAD by the tracee operator. The .so files under libs/ are placeholder stubs. Build instructions are in the package doc comment. The embed directives keep them as part of the xcover binary so no external files are needed at runtime.
…jection When --userspace-bpf is passed, EnsureSyscallServer() is called before any BPF operation. On the first invocation it writes the embedded syscall-server library to a memfd and re-execs xcover with LD_PRELOAD set; on the re-exec'd invocation the sentinel env var is set and the call is a no-op. The flag is marked experimental in its usage string.
'xcover agent extract' writes the embedded bpftime-agent.so to a temporary file and prints its path. Intended for short-term userspace BPF workflow where the user preloads the agent manually: export LD_PRELOAD=$(xcover agent extract) ./my-binary & xcover run --path ./my-binary --userspace-bpf
…o files 'make bpftime-libs' clones the bpftime source tree, builds it with CMake, and copies the resulting shared libraries into pkg/bpftime/libs/ so that 'go build' picks them up via //go:embed. Builds with BPFTIME_UBPF_JIT=ON and BPFTIME_LLVM_JIT=OFF to statically embed ubpf into the .so files while keeping LLVM out of the build. A clean-bpftime target removes the source clone.
Documents the experimental bpftime integration: how the two libraries are injected, the build prerequisite (make bpftime-libs), the manual LD_PRELOAD workflow for the agent, known limitations, and the binary support matrix (LD_PRELOAD requires a dynamic linker; static and pure Go binaries fall back to kernel uprobe mode).
GetFuncOffsets and GetFuncCookies iterated the funcs map in two separate calls. Go map iteration order is random and differs between calls, so offset[i] and cookie[i] ended up referring to different functions. Both functions also used make([]uint64, len) followed by append, which pre-filled the slice with zeros before the real values. Replace both with a single GetFuncOffsetsAndCookies that builds both slices in one map iteration, guaranteeing they stay in sync.
…WithOpts Switch the userspace BPF (bpftime) attach path from hand-rolled perf_event_open + BPF_LINK_CREATE syscalls to bpf_program__attach_uprobe_opts via the maxgio92 fork of libbpfgo (PR aquasecurity/libbpfgo#523). The raw syscall code in bpftime_attach.go is deleted entirely. attachSingleUprobes now calls p.bpfProg.AttachUprobeWithOpts for each (offset, cookie) pair and stores the returned BPFLink in p.links, so cleanup goes through the same Destroy() path as the kernel path. The rawFds []int field and its unix.Close loop are removed. go.mod gains a replace directive pointing github.com/aquasecurity/libbpfgo at maxgio92/libbpfgo@cc9e1ae (v0.0.0-20260324172613-cc9e1ae113eb). go.sum needs refreshing locally with: go mod tidy The bpftime link_handler.hpp bpf_cookie patch is still required since the cookie propagation bug lives in bpftime, not in the Go attach path.
Mirror the existing kernel-mode hit/idle/miss benchmarks with userspace equivalents that run BPF programs via bpftime instead of the kernel. Key design decisions: - runTarget gains a variadic env parameter; userspace benchmarks pass LD_PRELOAD for the bpftime agent and BPFTIME_SHM_MEMORY_MB - startTracer gains a userspace bool that enables WithTracerUserspaceBPF - cleanBptimeSHM wipes /dev/shm/bpftime_* before each userspace run to prevent stale shared memory from corrupting measurements - The bpftime agent .so is extracted once in TestMain via bpftime.ExtractAgent and reused across all userspace benchmark iterations - Report extended with HitUserspace/IdleUserspace/MissUserspace summaries and kernel-vs-userspace overhead comparisons - Report path driven by BENCH_REPORT_PATH env var so the Makefile can direct kernel and userspace runs to separate JSON files - New Makefile targets: bench-userspace (sets LD_PRELOAD for the syscall-server process-wide) and bench-all (runs both modes sequentially) The two modes are kept as strictly separate make invocations to avoid the syscall-server LD_PRELOAD contaminating kernel-mode BPF syscalls. Signed-off-by: maxgio92 <maxgio92@pm.me>
Adds automated-demo-userspace.sh alongside the existing kernel uprobe demo. Follows the same structure but covers the bpftime-powered userspace BPF flow: agent extraction, shm cleanup, xcover run with --userspace-bpf, and tracee invocations with LD_PRELOAD.
Both were accidentally dropped in the demo commit. Restores: - pkg/cmd/run/run.go: --userspace-bpf flag + EnsureSyscallServer() call - pkg/cmd/cmd.go: agent subcommand registration
…n demo commit Restores the full userspace BPF attach path dropped by the demo commit: - probe.go: WithUserspaceBPF(), userspaceBPF field, attachSingleUprobes() via AttachUprobeWithOpts (from 6c610ef) - tracer-options.go: userspaceBPF field + WithTracerUserspaceBPF() (from 64b89f9) - tracer.go: probe.WithUserspaceBPF() wiring in Init() (from 9b5120f)
daemonize() manually rebuilds the args slice for the spawned child process but never included --userspace-bpf, so the daemon started without bpftime syscall-server injection. This broke the userspace BPF demo, which runs xcover with --detach. Forward the flag when set so the child process calls EnsureSyscallServer() and re-execs with the syscall-server library in LD_PRELOAD.
Pure Go binaries are statically linked by default. LD_PRELOAD is silently ignored on static binaries, so the bpftime agent never injects into the tracee. Pass -linkmode external to force dynamic linking against libc.
'export $PATH' expands the value as a variable name rather than exporting the PATH variable itself. Use 'export PATH' instead.
The demo commit accidentally dropped the replace directive and swapped the aquasecurity module for a pseudo-version pointing back to upstream. AttachUprobeWithOpts only exists in maxgio92/libbpfgo@cc9e1ae (PR aquasecurity/libbpfgo#523, not yet merged), so without the replace the build fails. Restore go.mod and go.sum to the state from dbd54a4.
…demo commit The demo commit replaced GetFuncOffsetsAndCookies with the old split GetFuncOffsets/GetFuncCookies helpers, which iterate the map separately and produce mismatched orderings. The restore commits missed this. Bring back the single-iteration method and drop the split helpers.
The demo commit wiped the entire bpftime-libs target. Restores it verbatim from 1a8a720: - Clones eunomia-bpf/bpftime with submodules - Patches libbpf const-qualifier discards (GCC 14 hard errors) - Patches bpf_cookie propagation bug in bpf_link_handler constructor - Builds with cmake (UBPF_JIT=ON, LLVM_JIT=OFF) - Copies syscall-server and agent .so into pkg/bpftime/libs/ Also restores clean-bpftime target.
…6.15+ bpftime's bundled bootstrap libbpf declares bpf_stream_vprintk with an older signature. On kernel 6.15+ the vmlinux.h generated by bpftool declares it differently, causing a hard clang error when building the bpftool skeleton BPF programs. Guard the bootstrap declaration with #ifndef so the vmlinux.h declaration wins when both headers are included in the same translation unit.
bootstrap/ is generated during bpftool's build and doesn't exist yet when the patch runs. Patch the source header instead: third_party/bpftool/src/libbpf/include/bpf/bpf_helpers.h which bpftool copies into bootstrap/ during its two-pass build.
Two issues prevented make bpftime-libs from succeeding: 1. The bpf_stream_vprintk patch targeted a non-existent path (third_party/bpftool/src/libbpf/include/bpf/bpf_helpers.h). The real source lives at third_party/bpftool/libbpf/src/bpf_helpers.h, so python3 raised FileNotFoundError and make aborted before configure. 2. Even at the right path, the #ifndef bpf_stream_vprintk guard was a no-op. Both the bundled libbpf header and the auto-generated vmlinux.h declare bpf_stream_vprintk as an extern function, not a macro, so #ifndef was always true and the type conflict stayed. Delete the bundled declaration outright instead. The bpftool skeleton sources (pid_iter.bpf.c, profiler.bpf.c) do not reference bpf_stream_printk or bpf_stream_vprintk, so removal is safe. Also export LD_LIBRARY_PATH with llvm-config --libdir for the cmake build step. When the C/C++ toolchain is Homebrew clang, the built bpftool links against libLLVM.so.<ver> in a non-standard prefix and cannot find it at runtime when invoked to generate bpf_tracer.skel.h. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
bpftime defaults to the LLVM JIT backend. We build with BPFTIME_LLVM_JIT=OFF and BPFTIME_UBPF_JIT=ON, so LLVM is not available. Set BPFTIME_VM_NAME=ubpf so the agent uses the uBPF backend when intercepting uprobe hits.
Poll(timeout) returns after timeoutMs even when no events arrive. Without a loop, the goroutine exited after one 60ms window and any events written after that were silently dropped. The loop exits cleanly when CloseEventBuf() frees the ring buffer, causing Poll to return an error.
…inks libbpf's probe_perf_link() calls bpf_link_create(prog_fd, -1, BPF_PERF_EVENT) with target_fd=-1 to detect FEAT_PERF_LINK support. It expects EBADF back - an invalid fd means the syscall exists but the args are bad. On the real kernel this works naturally (fdget(-1) fails). In bpftime, which uses a SHM handler table instead of real fds, -1 was accepted and a handler was created, so probe_perf_link() got a valid fd back and concluded the feature was absent. Consequence: kernel_supports(FEAT_PERF_LINK) returned false, libbpf fell back to ioctl(PERF_EVENT_IOC_SET_BPF), which rejects non-zero bpf_cookie with EOPNOTSUPP. All uprobe attachments failed silently, leaving funcs_ack=0. Fix: check bpftime_is_perf_event_fd(target_fd) before creating the link. If it is not a valid perf event fd, return -1/EBADF - matching kernel behaviour. Real attaches with a valid perf fd proceed normally and carry the cookie via the cookie-aware bpf_link_handler constructor.
…short functions
Frida-Gum requires at least 5 bytes at the function entry to place its
trampoline. With compiler optimisations enabled, Go collapses tiny functions
such as add(a, b int) int { return a+b } to a single add+ret (4 bytes),
which makes gum_interceptor_attach fail with GUM_ATTACH_WRONG_SIGNATURE and
aborts the entire agent attach cycle - leaving funcs_ack=0.
-N disables optimisations and -l disables inlining, forcing the compiler to
emit a full frame prologue on every function and giving Frida enough bytes to
work with.
…bility Frida-GUM's interception is incompatible with Go 1.17+'s register-based calling convention (ABI internal), where R14 holds the current goroutine pointer. After the Frida trampoline returns control to the Go function the register state is corrupted, causing a SIGSEGV inside the tracee. C binaries use the standard System V x86-64 ABI which Frida handles correctly. Replace demo-app.go with demo-app.c (same arithmetic + greet functions, same CLI interface) and update the demo script accordingly: - build with gcc -O0 (ensures proper function prologues for Frida) - update --include filter from Go-style '^main\.' to C function names
Signed-off-by: Massimiliano Giovagnoli <maxgio92@pm.me>
Signed-off-by: Massimiliano Giovagnoli <maxgio92@pm.me>
bpftime's LLVM JIT is compatible with LLVM 15-18 only. System LLVM 22
(linuxbrew default) breaks the build: PointerType::get API changed and
PassPlugin.h moved. When llvm@18 is present (brew install llvm@18),
point cmake to it explicitly via LLVM_DIR and CMAKE_{C,CXX}_COMPILER.
Falls back gracefully when llvm@18 is absent.
BPFTIME_MAX_FD_COUNT=25000 gives enough handler slots for large N runs.
BPFTIME_VM_NAME=ubpf is removed now that LLVM JIT is enabled in the
bpftime build; the default vm_name ("llvm") takes effect automatically.
Label mounted workspace directory with :z to relabel bind-mounted host directories to container_file_t because SELinux requires this specific type for container processes to access the files Signed-off-by: Massimiliano Giovagnoli <maxgio92@pm.me>
Make expands a rule's prerequisite list when the rule is parsed. The $(BPFTIME) reference in the $(PROGRAM) prereq list expanded to empty because BPFTIME was defined further down, so bpftime-libs was never built as a dependency of xcover. Signed-off-by: Massimiliano Giovagnoli <maxgio92@pm.me>
Add patch files for 0004 (GCC 14 const-qualifier fix in bundled libbpf) and 0005 (conflicting bpf_stream_vprintk declaration vs kernel 6.15+). Add README entries for 0003, 0004, and 0005.
bpftime's llvm-jit cli builds a vendored libbpf via ExternalProject with plain "make -C src -j", which never sets -fPIC. Fedora's clang defaults to PIE for executables, so linking bpftime-vm against that libbpf.a fails with R_X86_64_32 relocations. We only ship the bpftime .so files, not the executables, so dropping PIE on bpftime's executables is harmless and avoids patching the vendored libbpf build. Signed-off-by: Massimiliano Giovagnoli <maxgio92@pm.me>
…target Move bench-report-kernel.json and bench-report-userspace.json into a results/ subdirectory so both outputs land in one place. Create the directory automatically before writing. Add a bench-report Make target that runs both kernel and userspace benchmarks in sequence. Add results/ to the clean target.
Signed-off-by: Massimiliano Giovagnoli <maxgio92@pm.me>
Signed-off-by: Massimiliano Giovagnoli <maxgio92@pm.me>
Signed-off-by: Massimiliano Giovagnoli <maxgio92@pm.me>
Save raw go test output alongside JSON reports and run benchstat to print a side-by-side ns/call comparison between kernel and userspace benchmark runs.
Signed-off-by: Massimiliano Giovagnoli <maxgio92@pm.me>
Signed-off-by: Massimiliano Giovagnoli <maxgio92@pm.me>
Signed-off-by: Massimiliano Giovagnoli <maxgio92@pm.me>
maxgio92
force-pushed
the
xcover-userspace
branch
from
073e00d to
5de59ab
Compare
Signed-off-by: Massimiliano Giovagnoli <maxgio92@pm.me>
Signed-off-by: Massimiliano Giovagnoli <maxgio92@pm.me>
Signed-off-by: Massimiliano Giovagnoli <maxgio92@pm.me>
Signed-off-by: Massimiliano Giovagnoli <maxgio92@pm.me>
Signed-off-by: Massimiliano Giovagnoli <maxgio92@pm.me>
Signed-off-by: Massimiliano Giovagnoli <maxgio92@pm.me>
dodoazzurro
force-pushed
the
xcover-userspace
branch
from
16ac24d to
b05bae0
Compare
maxgio92
force-pushed
the
xcover-userspace
branch
from
b05bae0 to
916b00b
Compare
Introduce a 'userspace' build tag that controls whether bpftime support is compiled in. Without the tag, the two embedded .so files are not baked into the binary and the 'xcover agent' subcommand is not registered. - pkg/bpftime: guard bpftime.go with //go:build linux && userspace; add bpftime_stub.go (!userspace) with no-op EnsureSyscallServer and ExtractAgent that return a clear 'rebuild with -tags userspace' error - pkg/cmd: split agent subcommand registration into cmd_userspace.go (userspace) and cmd_nouserspace.go (!userspace) so it only appears in --help for binaries that actually support it - Makefile: add xcover-userspace target (go build -tags userspace) with bpftime-libs as a dep; drop bpftime-libs dep from default xcover target Default 'make xcover' produces a lean binary with no bpftime footprint. Userspace BPF support requires 'make xcover-userspace'.
maxgio92
force-pushed
the
xcover-userspace
branch
from
0814b37 to
d36eaac
Compare
Owner
Author
|
Supersed by #111 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This experimental feature enables xcover to run completely in userspace, leveraging eunomia/bpftime as BPF engine.
The function handler BPF probe runs completely in userspace, so the uprobe are implemented with Frida Gum interceptor as trampolines.
The BPF probe is JIT-ed with the LLVM-based BPFTime VM.
Limitations
Static binaries
Since BPFTime needs two libraries
Static binaries are not supported. bpftime requires LD_PRELOADing agent.so into the tracee and syscall_server.so into xcover. Static binaries can't be preloaded.
Setuid/setgid
setuid/setgid binaries are not supported. Linux silently ignores LD_PRELOAD for privileged executables.
Exec boundaries
Coverage doesn't cross exec boundaries. LD_PRELOAD is inherited across fork() but not exec(), so child processes that exec a new binary won't be instrumented.
dlopen
Dynamically loaded libraries may be missed. Functions in libraries opened via dlopen() after bpftime sets up its interceptors may not be traced.