feat(major): [sc-23696] replace jemalloc dependency with custom malloc interposer#333
Open
supersonicbyte wants to merge 43 commits into
Open
feat(major): [sc-23696] replace jemalloc dependency with custom malloc interposer#333supersonicbyte wants to merge 43 commits into
supersonicbyte wants to merge 43 commits into
Conversation
hassila
requested changes
Sep 1, 2025
hassila
left a comment
hassila
left a comment
Contributor
There was a problem hiding this comment.
Some initial feedback, will ping you offline about the initial bootstrap problem...
Contributor
|
Also I see a difference in which metrics are output between main and this branch, e.g. : Main: this PR: Got some additional malloc related metrics there, would be good to understand root cause. |
Contributor
|
(that was from running |
Contributor
|
Also one issue when testing with package-benchmarks-samples repo with lost capture of allocation: Main gives: PR: |
Contributor
|
(Memory (allocated resident) (K) was missing from the PR output too?) |
Contributor
|
This test also misses capture: PR: |
Contributor
|
And this test should show a leak, but does not on the PR: PR: |
Contributor
Author
Thanks for the feedback! That's strange will look into it.. |
Contributor
|
This one failed to find the Malloc/free delta leak on Linux: |
Contributor
|
For Linux: cp .build/x86_64-unknown-linux-gnu/debug/libMallocInterposerC-tool.so .build/x86_64-unknown-linux-gnu/debug/libMallocInterposerC.so
cp .build/x86_64-unknown-linux-gnu/debug/libMallocInterposerSwift-tool.so .build/x86_64-unknown-linux-gnu/debug/libMallocInterposerSwift.so(need to rebuild in between too, so build, first copy, build, second copy, ...) |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #333 +/- ##
==========================================
+ Coverage 68.85% 69.49% +0.65%
==========================================
Files 33 34 +1
Lines 3152 3245 +93
==========================================
+ Hits 2170 2255 +85
- Misses 982 990 +8
... and 2 files with indirect coverage changes
... and 2 files with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
This was referenced Sep 3, 2025
Contributor
Author
|
Currently blocked by this: swiftlang/swift#87696 (comment) |
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
Contributor
|
Pull request had an unknown failure |
Contributor
Author
|
@codex review |
|
To use Codex here, create a Codex account and connect to github. |
Contributor
Author
|
@claude review |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR replaces
jemallocwith a custom malloc interposer. The interposer logic is mostly inspired by the great work done by theswift-nioteam.On macOS/Darwin we ulitise the interposing feature of dydl while on Linux we use
LD_PRELOADin order to interpose.The interposition logic is written in C and located in the
MallocInterposerCpackage.For actual interaction with the interposition is done through a wrapper package
MallocInterposerSwift, which exposes a simple interface tohookandunhookthe interposition logic and also to actually count the statistics we need.The interposition requirers the the interposer library to be linked dynamically, since that's crucial to the nature of how the interposition works (both on Linux and Darwin). The SwiftPM Command plugin
BenchmarkCommandPluginhas a dependency onBenchmarkwhich has a dependency on the interposer libs. This causes a problem for SwiftPM and intitial tiggers ofswift package benchmarkcommand will fail.The issue is described in more detail here.
The current workaround is to trigger the
swift package benchmarkcommand once and let it fail. Then to copy thelibMallocInterposerC-tool.dylibandlibMallocInterposerSwift-tool.dyliband name them without the-toolsuffix.e.g.
The same applies for Linux but the build location will be different and the dynamic library extension is
.so.Since jemalloc is removed the
mallocSmallandmallocLargemetrics are removed. New metricsmallocByteCountandfreeTotalCountare introducted.How Has This Been Tested?
When running the test the executable is being linked statically to the test target. I didn't discover a way yet to dynamically link the interposer in order for it to work inside tests.
Minimal checklist:
DocCcode-level documentation for any public interfaces exported by the package