Align repo, crawlers, and schemas for packages and libraries

[braver]

I don't know enough details to fully describe the discussion. The meat of it though, as I understand it right now, is ...

• packages are still on the v3 schema because the old website doesn't know any better (and cannot be updated)
• libraries are however already on v4.
• thecrawl crawls packages, https://github.com/packagecontrol/channel crawls libraries (?)
• there is crawler-like tech in the PC client package to resolve package

Perhaps more. But there is some consolidation that can happen here, especially once the old site is no longer needed.

[deathaxe]

The primary change between v3 and v4 is new wording for "dependencies", which are named "libraries". This wording applies to all aspects of those JSON files.

Another change is switching from SemVer to PEP440 versions, which was required to support all official python packages. To the extend versioning is used, PEP440 actually defines better and more conistent semantics for pre-releases. Ignore its oddities with local releases etc.

For packages v3 vs. v4 schema is probably less important as intented new keys, like python_version to override local .python-version files for unmaintained packages becomes more or less obsolete with new ST releases and dependencies vs. libraries keys within releases to specify dependencies upstream has never been practically used, yet being supported by PC3 and 4.

The important part for libraries is python_version being used to specify dedicated release branches for each supported python version. load_order key is no longer needed (and therefore disallowed by PC client).

Even though the long term goal may be to teach PC to resolve libraries on client-side, libraries have to be resolved from GH, BB, GL, CB and pypi for compatibility reasons with existing PC4 clients AND because there are a couple of ST-only python packages, such as mdpopups, sublime_aio, sublime_lib, lsp_utils, which are unlikely to be deployed via pypi.

To get rid of external libraries channel, thecrawl must learn to crawl them on its own.

Yes, PC has its own crawler logic as it is basically used for both, the client and server-side (packagecontrol.io) functionality. The code however is primarily python 3.3 and if it is to be continued to support all ST4 builds it can be lifted to py3.8 only. thecrawl uses py3.10+ code and thus can't be used for the client in its current form.

The primary question however is how much "crawling" logic the client needs. Most likely a couple of users will complain about any feature drop. Being able to specify a git repo as repository without a dedicated repository.json is a neat niche feature, despite suffering from its own edge cases.

It may be needed for version pinning, if PC needs to pull available versions of specified installed packages.

I never wanted to maintain and manage 2 code bases to handle all the logic. Despite a bit hard to keep track of, I always found the way package_control is used on both ends a very affective and efficiant solution, ensuring consistency between server and client part. Appears those days are over. At least it is not useful to bundle all required dependencies for aiohttp or httpx to PC client. That's just too much overhead and would require platform-specific package releases.

[kaste]

As of right now, the libraries are served from thecrawl. 🤞

[michaelblyons]

It seems to have reset stats and "first seen" data, possibly for every package.

[kaste]

It dropped the main db. LOL. Probably was scared. I think it has them now. It doesn't have the stats right now. Don't know why tbh as I can download the stats just fine and have them locally as well.

[kaste]

It doesn't rebuild the website because it is told "Nothing has changed". Someone need to push a change.

[kaste]

Next step is -- I think -- to move the library registry/repository to the main one. Just the file; the file doesn't have to be part of the packages channel. It can be just a file as generate_registry from here works with multiple input repositories and merges them.

For security, I wouldn't necessarily put it into the main "channel" as we fetch the declared sub-repositories (via the includes) and I don't think we need to start that for libraries. crawl_libraries has an option to only trust specific sources anyway. (And this is currently on for the living GitHub action.)

After that we have the registry in one place. We still loose -- or don't have a dedicated solution for -- tombstoned entries/packages. We have that information, I guess, in the git repo as separate commits, but we don't have a digestible json file or tooling to ask: was this name foo used at any point in time.