Skip to content

feat(pkg): use the relocatable compiler#14357

Merged
Alizter merged 3 commits into
ocaml:mainfrom
Alizter:push-vkkkwyxwsotw
May 8, 2026
Merged

feat(pkg): use the relocatable compiler#14357
Alizter merged 3 commits into
ocaml:mainfrom
Alizter:push-vkkkwyxwsotw

Conversation

@Alizter
Copy link
Copy Markdown
Collaborator

@Alizter Alizter commented Apr 27, 2026 •
edited by shonfeder
Loading

This PR enables the relocatable compiler by making it the default. It does so by using (a mirror of) David's relocatable opam-repository as a second set of overlays.

The order is now overlay, relocatable, upstream.

This is part of the work on:

The following need to be done:

  • changelog
  • documentation
  • testing with:
    • ocamlfind - some warnings but builds and works fine
    • ocamlbuild
    • ocamlformat dev tool
    • ocamllsp devtool
    • utop devtool

In order to test,

Ensure you have build dune from source using this branch or opam pin dune.dev git@github.com:Alizter/dune.git#push-vkkkwyxwsotw into your switch. Then

  1. clone your favourite OCaml project.
  2. Please set DUNE_TRACE=+cache and DUNE_CACHE=enabled
  3. Run dune build --pkg enabled and wait.
  4. Then when it's finished, backup the _build/trace.sexp file as trace.sexp.old and delete _build/
  5. Run the build again.

If all goes well (and your cache is actually enabled) it should restore from cache.

Any issues please report here together with your first and second trace files. _build/trace.csexp and the backed-up trace.csexp.old.

@Alizter Alizter mentioned this pull request Apr 27, 2026
Closed
16 tasks
@Alizter Alizter force-pushed the push-vkkkwyxwsotw branch 2 times, most recently from dba99db to 10f294f Compare April 28, 2026 10:33
@Alizter Alizter self-assigned this Apr 29, 2026
@Leonidas-from-XIV
Copy link
Copy Markdown
Collaborator

Leonidas-from-XIV commented Apr 30, 2026

Can you fill me in with the current state? The way I was explained this many months ago is that OCaml 5.5 is relocatable by default and the relocatable patches will be backported to older compilers in point releases. in such case it would just work out of the box for everyone without need for more overlays.

Is that not the case anymore?

@Alizter
Copy link
Copy Markdown
Collaborator Author

Alizter commented Apr 30, 2026 •
edited
Loading

@Leonidas-from-XIV That is the case for OCaml 5.5 which is yet to be released. You can constrain your dependencies to use it. The support provided by David's repository is for older versions to at least 4.14. It takes time for patches to make their way to the compiler branch, and I'm not even certain that it has been agreed yet that this will be the case. For users of dune pkg, we can and should make use of it however.

@Leonidas-from-XIV
Copy link
Copy Markdown
Collaborator

Leonidas-from-XIV commented Apr 30, 2026

Yes, but my question was about the backports to older compiler releases. Are these not planned anymore?

@Alizter
Copy link
Copy Markdown
Collaborator Author

Alizter commented Apr 30, 2026

I don't know. I don't know if it has been discussed with the compiler developers yet and I also don't know if these discussions have happened with opam. My point is that we don't need to know any of this to make use of it in dune.

@Leonidas-from-XIV
Copy link
Copy Markdown
Collaborator

Leonidas-from-XIV commented Apr 30, 2026

@dra27 could you provide us with some insight on what the plan is for the backporting (or lack thereof) for older compilers?

@Alizter Alizter mentioned this pull request May 3, 2026
Open
@Alizter Alizter force-pushed the push-vkkkwyxwsotw branch from 10f294f to c98d4e6 Compare May 4, 2026 16:15
rgrinberg
rgrinberg reviewed May 4, 2026
View reviewed changes
Comment thread src/dune_cache/shared.ml Outdated
@Alizter Alizter force-pushed the push-vkkkwyxwsotw branch 3 times, most recently from 9bd5654 to 53584cc Compare May 4, 2026 20:47
shonfeder
shonfeder reviewed May 6, 2026
View reviewed changes
Comment thread src/dune_pkg/workspace.ml Outdated
ElectreAAS
ElectreAAS approved these changes May 7, 2026
View reviewed changes
Copy link
Copy Markdown
Collaborator

@ElectreAAS ElectreAAS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just tested this with DUNE_TRACE=+cache DUNE_CACHE=enabled, and it works! 🎉
More specifically, what works:
After building the compiler once and then deleting the build directory, the next build is instantaneous!
This also applies when completely changing projects

One thing to note is that when building with dev-tools (ocamllsp and the like), it doesn't seem to work, the compiler is rebuilt. This isn't a regression, just an existing bug in dev-tools from what I understand. However once I have a normal compiler and a "dev-tool compiler" in the cache, installing more dev tools doesn't need to rebuild the compiler, just the new dev tools

@Alizter Alizter marked this pull request as ready for review May 7, 2026 13:57
@Alizter Alizter force-pushed the push-vkkkwyxwsotw branch from 53584cc to d7d32de Compare May 8, 2026 09:37
@Alizter Alizter force-pushed the push-vkkkwyxwsotw branch from d7d32de to e78157c Compare May 8, 2026 09:42
@Alizter Alizter requested a review from rgrinberg May 8, 2026 09:42
@Leonidas-from-XIV
Copy link
Copy Markdown
Collaborator

Leonidas-from-XIV commented May 8, 2026

I've tested this on mucppo (not a big project but that's what I had at hand which wasn't in a complete state of disrepair) with the current state of the branch.

Ran PATH=$DUNE_REPO/_build/install/default/bin:$PATH DUNE_CACHE=+cache DUNE_CACHE=enabled dune build --pkg enabled. It locked and built a compiler fine. After deleting _build and re-running the same command it restored the compiler near-instantly. So I can confirm that it is working fine on my machine.

Leonidas-from-XIV
Leonidas-from-XIV reviewed May 8, 2026
View reviewed changes
Comment thread src/dune_digest/digest.mli Outdated
Comment thread src/dune_digest/digest.ml
Alizter added 2 commits May 8, 2026 14:36
@Alizter
feat(pkg): use the relocatable compiler
9c118f2 
Add ocaml-dune/opam-repository-relocatable as a default repository for
package management. This is a mirror of dra27/opam-repository#relocatable
which provides overlay packages for building a relocatable OCaml compiler.

When this repository is included, the solver will pick up the
relocatable-compiler meta-package, allowing toolchain builds to be
cached as regular packages instead of using the toolchain cache.

Signed-off-by: Ali Caglayan <alizter@gmail.com>
@Alizter
fix(pkg): add relocatable-compiler to compiler_package_names
ac212b0 
The relocatable-compiler package installs libraries (including threads,
str, unix) under target/lib/ocaml/, matching the layout of other compiler
packages. Without being listed in compiler_package_names, install_roots
did not append the /ocaml subdirectory to its lib_root, causing dev-tools
like utop to fail with "Library threads not found".

Signed-off-by: Ali Caglayan <alizter@gmail.com>
@Alizter Alizter force-pushed the push-vkkkwyxwsotw branch 2 times, most recently from 5c47d0f to 28794a4 Compare May 8, 2026 14:44
@Alizter Alizter requested a review from Leonidas-from-XIV May 8, 2026 14:44
@Alizter
fix(digest): throttle file_async to avoid EMFILE
0140223 
file_async opens an fd eagerly and for large files holds it open across
a fiber yield while queued for the background thread pool. When called
via unbounded parallel_map over many targets (e.g. the relocatable
compiler with thousands of files), fds accumulate faster than they are
closed, exhausting the process fd limit.

Add a global Fiber.Throttle in file_async to bound the number of
concurrent digest operations.

Signed-off-by: Ali Caglayan <alizter@gmail.com>
@Alizter Alizter force-pushed the push-vkkkwyxwsotw branch from 28794a4 to 0140223 Compare May 8, 2026 15:15
@shonfeder
Copy link
Copy Markdown
Member

shonfeder commented May 8, 2026

The basic test worked for me on a pet project depending on fstar (which also shows the fruits of @Alizter and @ElectreAAS's work on #13792)!

I'll also try a more involved test exercising the dev tools and report back. But great work!

@Alizter Alizter mentioned this pull request May 8, 2026
Open
@Alizter
Copy link
Copy Markdown
Collaborator Author

Alizter commented May 8, 2026

One thing to note is that when building with dev-tools (ocamllsp and the like), it doesn't seem to work, the compiler is rebuilt. This isn't a regression, just an existing bug in dev-tools from what I understand. However once I have a normal compiler and a "dev-tool compiler" in the cache, installing more dev tools doesn't need to rebuild the compiler, just the new dev tools

This is because dev tools use non-portable lock directories. I'm tracking this issue here. #14456. As you've said it just means you have to build one compiler with the portable lock dir and one without but both are cached.

@Alizter Alizter merged commit 549dd57 into ocaml:main May 8, 2026
30 checks passed
@Alizter Alizter deleted the push-vkkkwyxwsotw branch May 8, 2026 21:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@shonfeder shonfeder shonfeder left review comments

@rgrinberg rgrinberg rgrinberg approved these changes

@ElectreAAS ElectreAAS ElectreAAS approved these changes

@punchagan punchagan Awaiting requested review from punchagan

@Leonidas-from-XIV Leonidas-from-XIV Awaiting requested review from Leonidas-from-XIV

Assignees

@Alizter Alizter

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Always use the relocatable compiler over the older OCaml ones.

5 participants

@Alizter @Leonidas-from-XIV @shonfeder @rgrinberg @ElectreAAS