upgrade Drake to latest monthy release

[RussTedrake]

update poetry, as well. the currently pinned drake fails on poetry export (due to mosek) with the latest poetry.

[RussTedrake-LBM]

Drake v1.43 works fine. Drake v1.44 fails with mosek library errors.

==================== Test output for //book/robot:inspector:
Traceback (most recent call last):
  File "/private/var/tmp/_bazel_russt/95c44f817cf1739792ac5cabe36eb430/sandbox/darwin-sandbox/888/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/book/robot/inspector.runfiles/_main/book/robot/inspector.ipynb.py", line 4, in <module>
    from pydrake.common.deprecation import DrakeDeprecationWarning
  File "/private/var/tmp/_bazel_russt/95c44f817cf1739792ac5cabe36eb430/sandbox/darwin-sandbox/888/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/book/robot/inspector.runfiles/rules_python~~pip~htmlbook_pip_313_drake/site-packages/pydrake/__init__.py", line 146, in <module>
    from . import common
ImportError: dlopen(/private/var/tmp/_bazel_russt/95c44f817cf1739792ac5cabe36eb430/sandbox/darwin-sandbox/888/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/book/robot/inspector.runfiles/rules_python~~pip~htmlbook_pip_313_drake/site-packages/pydrake/common/__init__.so, 0x0002): Library not loaded: @loader_path/../../mosek/libmosek64.11.0.dylib
  Referenced from: <67FFC657-A5B6-3690-953A-3EA6AF5E2D8E> /private/var/tmp/_bazel_russt/95c44f817cf1739792ac5cabe36eb430/external/rules_python~~pip~htmlbook_pip_313_drake/site-packages/pydrake/common/__init__.so
  Reason: tried: '/private/var/tmp/_bazel_russt/95c44f817cf1739792ac5cabe36eb430/external/rules_python~~pip~htmlbook_pip_313_drake/site-packages/pydrake/common/../../mosek/libmosek64.11.0.dylib' (no such file)

@jwnimmer-tri -- is this a known challenge? it seems like the libmosek importing trick is not playing well with bazel's rules_python now hermetic python?

[jwnimmer-tri]

Interesting. Sorry, I didn't foresee this. The problem isn't which python interpreter is being used, it's that the pip_compile doesn't respect a standard venv layout (bazel-contrib/rules_python#2156). This is the announcement in the Drake 1.44 release notes:

Wheel packages now depend on Mosek as a PyPI requirement instead of embedding it inside the Drake wheel file (23128, 23243).

It's required that Drake and Mosek are installed in the same place. The default pip_compile behavior is to unpack them each separately, and then set up a sys.path that finds them both.

I'm hoping that annotating the two as https://rules-python.readthedocs.io/en/latest/pypi/circular-dependencies.html would fix it, by annotating experimental_requirement_cycles on Drake+Mosek. If you can point me to a branch, I'm happy to try it out myself.

Alternatively, I believe there is experimental support for respecting the venv layout, described here: bazel-contrib/rules_python#2156 (comment).

[jwnimmer-tri]

So the experiment_require_cycles doesn't help. It only affects analysis-time -- it doesn't actually change the physical layout on disk to put drake and mosek next to each other.

Therefore, the next best solution would have to be switching to the standard venv layout. I tried that in jwnimmer-tri#1 but it has two problems:

(1) Linux fails because the nvidia python packages are troublesome in a way that rules_python venv logic doesn't handle yet:

ERROR: One of the output paths '_test_gym.venv/lib/python3.12/site-packages/nvidia' and '_test_gym.venv/lib/python3.12/site-packages/nvidia/nccl/lib' is a prefix of the other. These actions cannot be simultaneously present; please rename one of the output files or build just one of them

I believe it's the same problem as bazel-contrib/rules_python#3204 (comment). Still, this doesn't get in the way on macOS (no nvidia there) so plausibly we could only enable the venv option on macOS. However:

(2) The venv doesn't quite fix it.

   File "/private/var/tmp/_bazel_runner/7b4dc6001ecdbeb1b5ef1e14edba8c37/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/manipulation/prefetch_remotes.runfiles/_main/manipulation/_prefetch_remotes.venv/lib/python3.13/site-packages/pydrake/__init__.py", line 146, in <module>
    from . import common
ImportError: dlopen(/private/var/tmp/_bazel_runner/7b4dc6001ecdbeb1b5ef1e14edba8c37/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/manipulation/prefetch_remotes.runfiles/_main/manipulation/_prefetch_remotes.venv/lib/python3.13/site-packages/pydrake/common/__init__.so, 0x0002): Library not loaded: @loader_path/../../mosek/libmosek64.11.0.dylib
  Referenced from: <67FFC657-A5B6-3690-953A-3EA6AF5E2D8E> /private/var/tmp/_bazel_runner/7b4dc6001ecdbeb1b5ef1e14edba8c37/external/rules_python~~pip~htmlbook_pip_313_drake/site-packages/pydrake/common/__init__.so
  Reason: tried: '/private/var/tmp/_bazel_runner/7b4dc6001ecdbeb1b5ef1e14edba8c37/external/rules_python~~pip~htmlbook_pip_313_drake/site-packages/pydrake/common/../../mosek/libmosek64.11.0.dylib' (no such file)

I looked at the disk layout and it's actually what we want, however the problem is that the relative link to ../../mosek/libmosek64.11.0.dylib is being resolved w.r.t. the repository directory rules_python~~pip~htmlbook_pip_313_drake instead of the venv directory _prefetch_remotes.venv/lib/python3.12/site-packages/drake. One of the symlinks got resolved back into an absolute path a little too early. I think the solution could be to symlink the files in drake instead of the whole drake directory, but I don't know how to do that.

---

Stepping back, I'm not sure any of this is a hill worth climbing. The support for pip dependencies in rules_python is experimental at best, and even experts struggle to get it right. I recommend against anyone using it in production until it has matured, and certainly in a project without a full-time maintainer it probably causes more headaches than it's worth.

You could give up on using Bazel entirely, or you could do what we do in Drake and Anzu -- outside of Bazel, create a venv with your dependencies. I see you already have a poetry file anyway. Have your setup script call poetry to prepare a venv. Then, tell Bazel to use the manipulation_venv/bin/python3 as its interpreter, and change the requirement() macro to be a no-op.

That would is the best of both worlds:

• Use a robust and battle-tested tool to manage Python dependencies.
• Use Bazel for test running with safe caching, and other additional automations.

[RussTedrake]

Thanks Jeremy. That does seem like a big loss... right now my tests are pretty carefully annotated with the pip dependencies, which I find particularly valuable for the non-default dependencies (like torch, etc). There are a lot of complicated pip packages that are getting pulled in here and are working fine, it's unfortunate that Drake is the only one that can't comply...no?

[jwnimmer-tri]

If you upgrade rules_python to its most recent version, it is possible that setting build --@rules_python//python/config_settings:venvs_site_packages=yes in .bazelrc might work around the problem.

[RussTedrake]

I just gave it a spin, but no luck. Using build --@rules_python//python/config_settings:venvs_site_packages=yes in rules_python version 1.6.3 didn't work at all (seemingly it couldn't import any library successfully). Updating to rules_python 1.7.0-rc6 gets me back to the same failure

Reason: tried: '/private/var/tmp/_bazel_russt/95c44f817cf1739792ac5cabe36eb430/external/rules_python~~pip~htmlbook_pip_313_drake/site-packages/pydrake/common/../../mosek/libmosek64.11.0.dylib' (no such file)

[RussTedrake]

I've gone ahead and purged bazel completely in favor of a simpler life (using just pytest). #569