Support distributed build cache

[Lupus]

In large CI environments caching plays an important role for making things run fast. Dune has build cache support that is using local filesystem. Would be awesome to extend this to support distributed caching storage accessed over the network. This could be dune caching daemon that is run over tcp elsewhere, or some HTTP transport (WebDAV support and "just use nginx" maybe?).

[bobot]

I believe it is already possible even if perhaps in an experimental way, @snowleopard .

[snowleopard]

Indeed, I am working on this and we have a cloud cache prototype for our internal build system Jenga. Porting the implementation to Dune is the next step. We don't expect it to take too much time, however, we have a couple of more urgent tasks on our stack at the moment. (Plus I'm on parental leave until September.)

[Lupus]

@snowleopard @rgrinberg I've heard some rumors that distributed cache is close to being open-sourced. Are there any official updates on that?

[rgrinberg]

There's no official plan to do it, but it will surely come eventually. The distributed cache is still being worked on so we're not ready to open source it yet. As for an ETA, I wouldn't count on anything being released this year.

[snowleopard]

Just to add some details:

• The distributed cache is implemented and used in production.
• We've only used it with Jenga (our old build system) so far. We plan to integrate it with Dune fairly soon.
• Once that is done, we'll be ready to opensource the distributed cache. That will not be trivial but that's our end goal.

[shunueda]

Is there any update on this, or any projects I can look into? Especially with dune-pkg managed projects, building ocaml-compiler takes a long time and can't afford to do this for every CI run.

[Alizter]

@shunueda The work on the relocatable compile #13229 will make it possible to cache the compiler more generally for package management. There is a PR #13321, available since 3.22, that added support together with instructions on how to use it and another #14357 that makes it the default compiler for all projects, this is still being tested and is yet to be merged.

Together with (cache enabled) this should allow full restoration of packages from the dune shared cache. I think this current issue is a little different as it is about sharing the cache over network.

[shunueda]

@shunueda The work on the relocatable compile #13229 will make it possible to cache the compiler more generally for package management. There is a PR #13321, available since 3.22, that added support together with instructions on how to use it and another #14357 that makes it the default compiler for all projects, this is still being tested and is yet to be merged.

Together with (cache enabled) this should allow full restoration of packages from the dune shared cache. I think this current issue is a little different as it is about sharing the cache over network.

Thank you for the great work on relocatable compiler, this feels very promising. Relying on local cache alone still has limitations in large-scale CI setups, so I’m very interested in any community project / possibility of Dune's plan to implement this natively.

[rgrinberg]

It doesn't seem likely that this is something that we'll be able to open source in the near term. However, if there's a distributed cache solution from the community, this is something that dune can easily integrate with. On dune's side, it's a matter of:

1. connecting how to connect to some distributed cache daemon
2. sending download requests whenever it tries to restore something from the cache
3. sending upload requests whenever it has something to put in the cache

If you have a cache that supports the above (using dune rpc for 2. and 3. preferably), then we can have a distributed cache.