Off-topic: python bridge & IDL_DLM_PATH

[steinhh]

Sorry @znorman-harris for bringing up another non-plugin-related thing, but the normal bug reporting... argh. We're starting to use the python bridge and have stumbled upon a problem:

IDL> print,getenv("IDL_DLM_PATH")
<IDL_DEFAULT>:/mn/sertan/u1/osdcapps/excluded/solarsoft/site/dlm/linux.x86_64
IDL> >>>2
% /astro/local/opt/nv5/idl/bin/bin.linux.x86_64:/mn/sertan/u1/osdcapps/excluded/solarsoft/site/dlm/linux.x86_64/idl-python/bin does not exist, reverting to system Python.
% PYUTILS::LOAD: Python numpy is not installed. numpy can be installed by running "python -m pip install numpy==1.*" or "conda install numpy==1.*".
% Execution halted at: $MAIN$

However, with no IDL_DLM_PATH set upon invocation of IDL:

IDL> >>>2
% Python is embedded version 3.13.
% Loaded DLM: PYTHON313.
% PYTHON_INIT: /astro/local/opt/nv5/idl/bin/bin.linux.x86_64/idl-python313.
2

To me it seems like an extremely simple bug: IDL_DLM_PATH is not handled correctly when looking for the embedded python, it is simply used verbatim without splitting on ":".

[steinhh]

(Not that I'm sure we'll use the embedded python in the end, could not get pyutils.pipinstall,'scipy' to work (it was deemed incorrectly installed when trying to use it). Using conda to install everything (python3.13 + scipy etc) worked (though quite a laborious process!).

Cheers, and I hope all is well over there :)

[chris-torrence]

hey, thanks for reporting these issues! We had a few "growing pains" with the embedded Python on Linux and Mac. You're absolutely correct about the IDL_DLM_PATH - it just needed to be split on the ":", and that has been fixed for IDL 9.3.

We actually switched the Python that we're embedding, from a custom compiled one to a prebuilt one, and that also fixed a lot of the other issues. I just tried your pyutils.pipinstall,'scipy' and that seems to now work fine as well.

We're hoping to do a beta release of IDL 9.3 soon (no promises yet!) so keep any eye out for that.

In the meantime, if you wanted to install "scipy" in IDL's embedded Python, you could try setting the environment variable:
PYTHONUSERBASE=~/.idl/idl-python313
Then, in your shell, do:
python -m pip install scipy--user --force

I think that should install scipy in expected location for IDL's embedded Python, and then it should work within IDL. The --force is just so it doesn't notice that scipy is already installed in your standard Python and try to not install it...

[steinhh]

Great, thanks for the responsivity! Will try your hints when/if I get problems moving on. Forgot to say that the pyutils.pipinstall of scipy was an issue on MacOS, not on linux. Go figure. We'll deal with the growing pains as long as we manage to hack something together on both Mac (solved) and linux (I think I got it now, with the embedded version).

[chris-torrence]

Yeah, both Mac and Linux had some issues in IDL 9.2, sorry about that. Our test machines passed (of course) but the real world is always different. Let me know if you run into any other quirks, so I can see if we already have a fix for those, or if I need to fix something else.

[steinhh]

About to call it for the day, but: on a RHEL8 machine, pytools.pipinstall,'scipy' appears to be successful, however on a RHEL9 it's not:

IDL> pyutils.pipinstall,'scipy'            
% Python is embedded version 3.13.
% Loaded DLM: PYTHON313.
% PYTHON_INIT: /astro/local/opt/nv5/idl92/bin/bin.linux.x86_64/idl-python313.
% Looking in indexes: https://pypi.python.org/simple
% Collecting scipy
%   Using cached scipy-1.16.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (61 kB)
% Requirement already satisfied: numpy<2.6,>=1.25.2 in /astro/local/opt/nv5/idl92/bin/bin.linux.x86_64/idl-python313/lib/python3.13/site-packages (from
   scipy) (2.3.1)
% Using cached scipy-1.16.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (35.2 MB)
% Installing collected packages: scipy
IDL> s=python.import('scipy')    
% PYTHON_RUN: Exception: The `scipy` install you are using seems to be broken, (extension modules cannot be imported), please try reinstalling..
% Execution halted at: PYTHON::RUN       418 /astro/local/opt/nv5/idl92/lib/bridges/python__define.pro
%                      PYTHON::IMPORT    223 /astro/local/opt/nv5/idl92/lib/bridges/python__define.pro
%                      $MAIN$          

will test your PYTHONUSERBASE trick next

[steinhh]

Just for fun: the "broken" scipy installed on the RHEL9 machine does work when loaded on the RHEL8 machine (same homedir on both). Argh...

[steinhh]

Ditto for an installation using PYTHONUSERBASE, no matter where I install it works on the RHEL8 machine, not on RHEL9. I suspect system libraries are missing (RHEL8 is my workstation, RHEL9 is a server and does not have the same complement of packages). Any tips on how to proceed?

[chris-torrence]

I just tried on RHEL8 and it worked fine, just as you found. I don't currently have an RHEL9 box. Your idea of the missing system libraries is probably correct. You could try going into the package folder and running "ldd" on the libraries:

cd ~/.idl/idl-python313/lib/python3.13/site-packages/scipy.libs
ls -l
ldd libscipy*

On my RHEL8 this gives a list of about 10 system libraries. There are 4 other .so's within that "scipy.libs" folder, and I would guess that all of them rely on system libraries.
Let me know (no rush!) if you find out anything.

[steinhh]

Oops, didn't hit send on this one yesterday (will try your route as well now):

I did an lsof on IDL before and after loading any python (on RHEL8). A number of files were opened, but filtering out those that belong to the IDL distribution and those that are on my home directory (~/.idl/idl-python313), only one system library had been added: /usr/lib64/libutil-2.28.so.

On RHEL9 where it doesn't work, there is only /usr/lib64/libutil.so.1.

However, on RHEL8 there is a link from libutil.so -> libutil.so.1 -> libutil-2.28.so. So presumably the library is opened as "libutil.so" on both.

[steinhh]

This is the output from ldd on RHEL8

        linux-vdso.so.1 (0x00007ffc7f75a000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f43cef85000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f43ced65000)
        libgfortran-040039e1-0352e75f.so.5.0.0 => /mn/sertan/u1/steinhh/.idl/idl-python313/lib/python3.13/site-packages/scipy.libs/./libgfortran-040039e1-0352e75f.so.5.0.0 (0x00007f43ce8b4000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f43ce4dd000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f43d0785000)
        libquadmath-96973f99-934c22de.so.0.0.0 => /mn/sertan/u1/steinhh/.idl/idl-python313/lib/python3.13/site-packages/scipy.libs/./libquadmath-96973f99-934c22de.so.0.0.0 (0x00007f43ce29f000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f43ce087000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f43cde6f000)

And this is from RHEL9:

        linux-vdso.so.1 (0x00007ffeb59a3000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f2b8c11f000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f2b8c11a000)
        libgfortran-040039e1-0352e75f.so.5.0.0 => /mn/sertan/u1/steinhh/.idl/idl-python313/lib/python3.13/site-packages/scipy.libs/./libgfortran-040039e1-0352e75f.so.5.0.0 (0x00007f2b8bc69000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f2b8ba00000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f2b8d689000)
        libquadmath-96973f99-934c22de.so.0.0.0 => /mn/sertan/u1/steinhh/.idl/idl-python313/lib/python3.13/site-packages/scipy.libs/./libquadmath-96973f99-934c22de.so.0.0.0 (0x00007f2b8b7c2000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f2b8bc4d000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f2b8bc33000)

Nothing reported as missing. Not an expert in interpreting that output, though.

[steinhh]

So, I finally got it to work - even on on RHEL9, and on both IDL9.1 and 9.2:

conda create -n py312 python=3.12
conda activate py312
conda install numpy==1.26.4
conda install scipy

and

conda create -n py313 python=3.13
conda activate py313
conda install numpy
conda install scipy

Then I have this in my .tcshrc:

  alias conda_py_312 "conda activate py312"
  alias conda_py_313 "conda activate py313"

  alias conda_ld_312 "setenv LD_LIBRARY_PATH $HOME/.conda/envs/py312/lib"
  alias conda_ld_313 "setenv LD_LIBRARY_PATH $HOME/.conda/envs/py313/lib"

  alias conda_312 "conda_py_312 ; conda_ld_312"
  alias conda_313 "conda_py_313 ; conda_ld_313"
  
  alias idldir_91 "setenv IDL_DIR /astro/local/opt/nv5/idl91"
  alias idldir_92 "setenv IDL_DIR /astro/local/opt/nv5/idl92"

  alias IDL91 "conda_312 ; idldir_91 ; $IDL_DIR/bin/idl"
  alias IDL92 "conda_313 ; idldir_92 ; $IDL_DIR/bin/idl"

and I can do:

steinhh:sertan /mn/sertan/u1/steinhh/> IDL91
IDL> sci = python.import('scipy')
% Loaded DLM: PYTHON312.
% PYTHON_INIT: /mn/sertan/u1/steinhh/.conda/envs/py312.
IDL> 

However, with IDL92, attempting to import scipy right away loads the embedded python 3.13:

IDL> sci = python.import('scipy')
% Python is embedded version 3.13.
% Loaded DLM: PYTHON313.
% PYTHON_INIT: /astro/local/opt/nv5/idl92/bin/bin.linux.x86_64/idl-python313.
% PYTHON_RUN: Exception: No module named 'scipy'.
% Execution halted at: PYTHON::RUN       418 /astro/local/opt/nv5/idl92/lib/bridges/python__define.pro
%                      PYTHON::IMPORT    223 /astro/local/opt/nv5/idl92/lib/bridges/python__define.pro
%                      $MAIN$          

But somehow this picks up the right python:

steinhh:sertan /mn/sertan/u1/steinhh/> IDL92
IDL> python.load,'python313'
% Loaded DLM: PYTHON313.
% PYTHON_INIT: /mn/sertan/u1/steinhh/.conda/envs/py313.
IDL> sci = python.import('scipy')
IDL>

So, it works. With the external vs. embedded python in IDL92, is there simply a path problem?

Cheers,
Stein

[chris-torrence]

Great! Thanks for the update. The docs are perhaps not entirely clear, but that python.load,"python313" is behaving as expected. If you start IDL and immediately issue a Python command, it will automatically load the embedded Python. If you do python.load then it will use the first Python that it finds on the path, which might also be Python 3.13 (as in your case).

As to why the scipy doesn't work in the embedded Python, I think our embedded Python was missing some required library in IDL 9.2. Regardless, it works in IDL 9.3 with both our embedded Python. In the meantime, it sounds like you have a decent workaround until the next version comes out.

Thanks so much for being a guinea pig on all of this embedded Python testing! It really helps to have someone who is willing to try different things and report back what works and what doesn't. I'm going to close this discussion for now. If you find other Python bridge issues, feel free to open up a new discussion.

[steinhh]

Yeah, the python.load was a bit unclear. I had the impression that IDL would always use whichever python was first in the path (which it seems to do under IDL91?). Well, yes, the python bridge works well for our needs now, and I'll certainly let you know if there are other things that crop up :)

[steinhh]

Hi again @chris-torrence!

Now that we've gotten to use the python function we needed to, it seems to take a long time. I'm 99.9% certain it's not the bridge, we're simply doing a call out_array = clip.dosomething(input_array). But I thought I'd hear with you if you know off-hand of any performance issues with things like that. I.e.,, would the bridge overhead of such a call be significantly bigger than shuffling a large data array from one IDL variable to another, twice. I'm talking a factor of 50 or more!

Or whether python code itself runs significantly slower inside the bridge than as a standalone process, of course.

[chris-torrence]

Hmm, there will be a little bit of overhead when passing the arguments back and forth and calling the Python routine. However, any arrays should be passed by reference from IDL to Python (the values are not copied) so there is no overhead for that. There also shouldn't be any slowdown when running inside the bridge - it's just a Python interpreter that happens to be within the same process as IDL. So I don't understand what could be happening. Can you create a simple reproduce case that I can try on my machine?

[steinhh]

Just did a quick test - shortcutting the python function so it does nothing other than returning the data array goes as fast as can be expected (i.e. lightning speed). Indeed, it's the python routine, and after a quick look at the code I understand why. Thanks - case closed :)

[ct6502]

Whew! Glad to hear you tracked it down. Let us know if you see anything else strange, or have any feature requests for the Python bridge.