Cleaning up old nightly conda packages

[jakirkham]

Having support for dask nightlies has been wonderful from a development standpoint

Recently we noticed that we are getting quite a lot of them tracking back over the years (back to 2021!)

As a result this means there are a lot of packages in the dask channel

This can make the channel repodata get quite large (and unfortunately a bit unwieldy)

So am wondering if we can clean this up a bit. Namely would it be reasonable to remove all nightlies after making a release?

Presumably users only care about nightlies newer than the last release. So dropping the old nightlies shouldn't matter much. Perhaps we can do this in an automated GHA script when a git tag is detected

Would be interested to hear thoughts on this 🙂

[jrbourbeau]

Thanks for the issue @jakirkham. Cleaning up old nightly builds seems very reasonable to me.

Presumably users only care about nightlies newer than the last release

I don't use these builds myself, but that seems sensible. To my knowledge the main consumers are various RAPIDS-related users / projects (I could be wrong about that), so if you / those folks are happy with that policy, then 👍

[jakirkham]

Thanks James! 🙏

Cool we can look into this then

Ok it might be worth double checking at Coiled. Recall hearing these were used previously ( for example: coiled/benchmarks#311 (comment) ). Though don't have much visibility over there

[jrbourbeau]

Fair point. Though I think things have changed in that repo since then (cc @hendrikmakait in case you happen to know if the dask conda channel is used)

[fjetter]

we're not using nightlies

[keewis]

you might want to check with the scientific-python people, as far as I can tell they have something similar working for scientific-python-nightly-wheels and the upload-nightly-wheels action.

cc @matthewfeickert, in case you have any hints

[matthewfeickert]

Dask nightlies sound pretty great to me. :) If the Dask team is interested, either give me the GitHub handle of a team member to work with and I'll setup a Issue on https://github.com/scientific-python/upload-nightly-action, or have a team member take a look at scientific-python/upload-nightly-action#87 and then self start a similar issue.

edit: Ah, I only read the last comment before posting. I guess the Dask team already has nightlies and wants a way to clean them up? If so, we have a system that works for https://anaconda.org/scientific-python-nightly-wheels using https://github.com/scientific-python/upload-nightly-action/blob/cb172cd73e045bb8b9fdaf271a183ffe4b5814d3/.github/workflows/remove-wheels.yml, but that's currently (c.f. scientific-python/upload-nightly-action#62) only for scientific-python-nightly-wheels.

[charlesbluca]

Opened dask/dask#11294 with a first pass at what these changes could look like

Also given recent inactivity on dask-sql (dask-contrib/dask-sql#1344), think we should also be good to outright purge dask/dask-sql now