Packaging with Poetry and other PEP-517 based packages

[SimonBiggs]

I am attempting to package the pypi library pymedphys:

[[packaging_rule]]
type="pip-install-simple"
package="pymedphys"

When I run pyoxidizer build

This fails in the following way:

LookupError: https://files.pythonhosted.org/packages/20/e9/bae28bcfcb9942600b1e206d013499c291367ee8ddf9f455ff54729bc6e4/tomlkit-0.5.8.tar.gz#sha256=32c10cc16ded7e4101c79f269910658cc2a0be5913f1252121c3cd603051c269 (from https://pypi.org/simple/tomlkit/)

Out of interest when I try to build tomlkit I get the following:

LookupError: https://files.pythonhosted.org/packages/9d/6b/ad69758a595b6e0be4583d8ab298e43c4c0680278beceaa4a5e308f09a3c/clikit-0.4.0.tar.gz#sha256=6819a5b2a78523be485ea9e1ef380ae8d02da4bffed540f7881fdaafbff41390 (from https://pypi.org/simple/clikit/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*) is already being built: clikit<0.5.0,>=0.4.0 from https://files.pythonhosted.org/packages/9d/6b/ad69758a595b6e0be4583d8ab298e43c4c0680278beceaa4a5e308f09a3c/clikit-0.4.0.tar.gz#sha256=6819a5b2a78523be485ea9e1ef380ae8d02da4bffed540f7881fdaafbff41390 (from poetry>=0.12a3)

pymedphys and tomlkit both use Poetry with its PEP-517 based build system. Is that potentially what is causing this issue?

[jayvdb]

I encountered these problems, and #199 is at least a partial solution for that.

[indygreg]

Sadly, I can make no guarantees for PyOxidizer's compatibility with PEP-517 build systems at this time. Even the presence of these alternative build systems can change pip's/distutils's behavior. I need to spend time researching the matter further. Until then, things may or may not just work.

[SimonBiggs]

Sadly, I can make no guarantees for PyOxidizer's compatibility with PEP-517 build systems at this time. Even the presence of these alternative build systems can change pip's/distutils's behavior. I need to spend time researching the matter further. Until then, things may or may not just work.

All good :). Makes sense.

[jayvdb]

See also #170 for PEP-517/PEP-518 . I believe it is fixed by #199 , and that venvs are a mandatory part of solving it, due to the problems of pip's --target argument. The alternative is to return to pip < 1.0, and then many packages wont install properly for other reasons.

The other problem with poetry is PyOxidizer requires source packages , and this breaks poetry because of python-poetry/poetry#760 . The fix is to allow wheels (#193), and I have a fix for that which I will push up tonight.

[jayvdb]

tomlkit and poetry can be installed using PipInstallSimple("...") with the following change to the main branch. This works fine as the only built extension they need is pyrsistent, and it doesnt provide wheels.

diff --git a/pyoxidizer/src/app_packaging/packaging_rule.rs b/pyoxidizer/src/app_packaging/packaging_rule.rs
index 032066e..b86e214 100644
--- a/pyoxidizer/src/app_packaging/packaging_rule.rs
+++ b/pyoxidizer/src/app_packaging/packaging_rule.rs
@@ -511,8 +511,8 @@ fn resolve_pip_install_simple(
     verbose: bool,
 ) -> Vec<PythonResourceAction> {
     let mut install_args = vec![
-        "--no-binary".to_string(),
-        ":all:".to_string(),
+        //"--no-binary".to_string(),
+        //":all:".to_string(),
         rule.package.clone(),
     ];
 

However pymedphys is going to be large chunk of work to get it PyOxidized. There are a lot of dependencies in https://github.com/pymedphys/pymedphys/blob/master/pyproject.toml which fail in PyOxidizer, mostly because they use __file__. I have a broad solution to the __file__ problem, and I have scipy almost working, but there are lots of deps in that list which I know are seriously incompatible.

And pymedphys appears to need a lot of its dependencies at startup based on
https://github.com/pymedphys/pymedphys/blob/master/pymedphys/__init__.py . The big question is which dependencies are really really needed, and which are optional for basic usage. The current list in pyproject.toml includes many which are likely to be unnecessary, such as pytest. One way to experimentally find this out is create a new venv, install pymedphys, and your own code which uses pymedphys, and then delete sub-directories in the venv's site-packages for dependencies which are likely to be not needed. Dont delete the .egg-info/.dist-info directories/files -- they are needed even if you delete the source for the package they refer to.