Isotropix Forums

Hello! We're refining our Python environment, and we've been having some troubles getting Clarisse to load our Python interpreter as expected on Windows. No matter what we do in clarisse.env and environment variables, it loads the runtime library at:

C:\WINDOWS\SYSTEM32\python27.dll

even though it's loading standard library modules correctly from our installation (os, platform, glob, etc.)

We've set our clarisse.env files to point to our installation:

Code: Select all

PYTHONHOME=OUR_INSTALL
PYTHONPATH=OUR_INSTALL\Lib;OUR_INSTALL\DLLs;OUR_INSTALL\Lib\lib-tk;$CLARISSE_BIN_DIR\python
...


The OUR_INSTALL path is the first path on the PATH environment variable. If I run the python interpreter from the command prompt, everything is fine. If I then run Clarisse from that same environment, it loads from the system module. When I compare the PATH variable in those two environments, it looks like Clarisse is:

• Adding these 3 paths to the beginning:
  

  Code: Select all

  c:/Program Files/Isotropix/Clarisse iFX 4.0 SP10/Clarisse/python
c:/Program Files/Isotropix/Clarisse iFX 4.0 SP10/Clarisse
c:\Program Files\Isotropix\Clarisse iFX 4.0 SP10\Clarisse

• Changing OUR_INSTALL\python\python27.zip to C:\WINDOWS\SYSTEM32\python27.zip
• Adding these right before OUR_INSTALL/python in the PATH:
  

  Code: Select all

  c:\Program Files\Isotropix\Clarisse iFX 4.0 SP10\Clarisse
%APPDATA%\Python\Python27\site-packages

Is this correct, and is this by design? We'd like to have Clarisse loading the module from our installation.

If it's useful, this is the code I'm using to inspect which module is loaded:

python code

import sys
import ctypes
from ctypes import wintypes

GetModuleFileName = ctypes.windll.kernel32.GetModuleFileNameA
GetModuleFileName.argtypes = (
    wintypes.HMODULE,
    wintypes.LPSTR,
    wintypes.DWORD
)
GetModuleFileName.restype = wintypes.DWORD

buffer_size = 1024
buffer = ctypes.create_string_buffer(buffer_size)
filled_char_count = ctypes.windll.kernel32.GetModuleFileNameA(
    sys.dllhandle, 
    buffer, 
    buffer_size
)

print buffer.value

Hi,

Quick notes:
- The dll is loaded by Windows depending on the PATH env var. PYTHONHOME and PYTHONPATH have no impact on where the dll is loaded from.
- Clarisse does not modify the PATH env var by itself.
- The Python dll is not installed in the same directory as the interpreter, but rather in C:\Windows\System32, so even if you were to add your custom Python directory to the PATH env var, the dll would still be loaded from the system folder (this is for Python2: starting with Python3 the dlls are next to the interpreter)

Now that being said, I just tested a few things, and I had some reaaaally weird behavior:
Whatever the content of the PATH env var, the python27.dll is ALWAYS loaded from the system. I even tried renaming python27.dll to python27.dll.back ... and even like this, the .dll.back got loaded !!!! It only worked as expected when I removed it from the system folder... so my guess is that it's something to do with Windows.

TLDL:

- move C:\Windows\System32\python27.dll to your custom Python install folder (e.g. you should no longer have any Python dll in the system folder!)
- make sure your custom Python install folder is coming first in your PATH env var list

> that worked for me, I could make Clarisse load the correct dll as soon as I removed the Python dll from the system.

Hey, thanks for looking into this!

Indeed, that matches the behaviour I'm seeing. Our Python installation has had its own copy of python27.dll, and that directory is first on the PATH, which is why I don't understand how the system library is getting picked up. (That's crazy it even picks up the .dll.back file, haha!)

I was able to get it to work, using your solution of moving the python27.dll out of the system folder. Unfortunately that's a difficult solution to use, because our pipeline boostrapping system does depend on that system Python (and then the bootstrap sets up the PATH to our pipeline Python installation).

I'm wondering if it may also have something to do with the way Python is embedded into Clarisse? I noticed this:

python code

import sys
print sys.executable

outputs the Clarisse executable (instead of the Python interpreter). Also, running the Python interpreter directly from our install does pick up the correct python27.dll (even when the system library is there).

Hi,

We're embedding Python the usual way, I can't see anything special on our side, but that being said on some other softwares which have support for Python scripting I don't notice the same behavior, so I'm not sure :/

As for the result of `print(sys.executable)`, that's normal. Do the same test on other softwares with python scripting, you'll get the path to the software.

I'm going to take a deeper look at how we link against the Python libraries.

Hey! Yes indeed on sys.executable -- looks like that's standard for embedded Python.

Thanks a lot for checking in on it, look forward to hearing what you discover!

We've actually managed to get it to work by moving python27.dll out of C:/Windows/System32 and into the local Python installation at C:/Python27

With that done, our pipeline bootstrap still works, and Clarisse picks up python27.dll from the other (pipeline) installation of Python. So there seems to be something automagical about the Windows system directory location.

Hi again,

So, 2 things:

- first: I checked, we don't do anything special. We just link against the python libraries, and that's all
- second: when I tested the other softwares, I missed the fact that they include their own python dll next to their binary, that's why the system picks the correct dll

To sum up, this is most likely a Windows thing: e.g. it looks like when a process needs a dll, Windows does:
1/ check in the directory of the main process's executable (and maybe in the folders of the plugins in case of DSOs ? Didn't check)
2/ check in the C:/Windows/System32 (and who knows what other system folders)
3/ finally, if the dll is still not found, checks the PATH env var.

I didn't find a doc where this is clearly explained, but from my experiment, seems like this is how it's done.

So, based on that, 2 solutions (although you seem to have solved your problems, it might helps others :p)
1/ the one you mentioned in your last answer
2/ copy the correct python27.dll to the Clarisse's folder (next to the Clarisse executable)

Regards,
Damien

For future reference, here is the official Microsoft documentation about DLL search order in Windows:
https://docs.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-search-order

It confirms what Damien said, with additional lookup locations.
You can see that PATH comes last.