Isotropix Forums

Multiprocessing supported in Clarisse Python?

Clarisse Scripting related topics

Multiprocessing supported in Clarisse Python?

Unread postby mhamid3d » Mon Dec 23, 2019 11:27 pm

I'm running a very basic ```multiprocessing``` command in python. One is from a Clarisses python interpreter and the other is from my IDE.

```
from multiprocessing import Pool

mylist = ['Hello', 'You', 'Are', 'Human']

def make_upper(x):
return x.upper()

pool = Pool(processes=4)
results = [pool.apply_async(make_upper, args=(x,)) for x in mylist]
output = [p.get() for p in results]

print output
```

On my IDE it returns the following:

``` >>> ['HELLO', 'YOU', 'ARE', 'HUMAN'] ```

On Clarisse it returns:
```
Traceback (most recent call last):
File "<string>", line 10, in <module>
File "/builds/python/2.7.8/523d45d733/lib/python2.7/multiprocessing/pool.py", line 558, in get
raise self._value
cPickle.PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

```

My IDE interpreter is pointing to this folder: ```/builds/python/2.7.8/523d45d733/bin```

Which is the same dir(pardir) that ```multiprocessing``` lives in. So it's the same version. Any clues as to why I'm getting this strange error?
mhamid3d
 
Posts: 4
Joined: Wed Dec 04, 2019 5:20 pm

Re: Multiprocessing supported in Clarisse Python?

Unread postby sam » Tue Dec 24, 2019 10:21 am

Hi there,

I'm afraid that's not possible. We have to explicitly lock the GIL before executing a Python Script in Clarisse since event callbacks implemented in Python can be called from C++ worker threads during the evaluation. If the GIL wasn't properly locked by Clarisse Python Engine that would lead to crashes since Clarisse application entirely runs in parallel to Python.

Python is probably the worst scripting language when it comes to work with a modern multi-threaded application such as Clarisse.
Sam Assadian
Isotropix
CEO/Founder
User avatar
sam
 
Posts: 1463
Joined: Fri Jan 25, 2013 11:33 pm

Re: Multiprocessing supported in Clarisse Python?

Unread postby mati » Tue Jan 07, 2020 4:32 am

Just to clarify that multiprocessing should work because GIL is related to multithreading. What I think you're hitting here is that the Pool class uses threads under the hood for communicating between the procs. This isn't pretty, but proves my point:

Code: Select all
from multiprocessing import Process, Manager

mylist = ['Hello', 'You', 'Are', 'Human']
def make_upper(reflist, i, return_dict):
    return_dict[i] = reflist[i].upper()

manager = Manager()
return_dict = manager.dict()
jobs = []
for i in range(len(mylist)):
    p = Process(target=make_upper, args=(mylist, i, return_dict))
    p.start()
    jobs.append(p)

for job in jobs:
    p.join()

print return_dict.values()

Workflows built around the subprocess module should also work.

Locking the GIL seems heavy-handed. Threading in Clarisse is only really a problem if you're running ix code, no? Could ix functions be adapted to raise an error if they are not called from the main python thread?
mati
 
Posts: 46
Joined: Fri Apr 19, 2019 8:35 pm

Re: Multiprocessing supported in Clarisse Python?

Unread postby sam » Wed Jan 08, 2020 1:55 pm

Locking at each ix calls makes Python unusable since the GIL is locked/unlocked for each single call. This is the best we could have done since 99% of the case, python scripts are single proc/threaded.
Sam Assadian
Isotropix
CEO/Founder
User avatar
sam
 
Posts: 1463
Joined: Fri Jan 25, 2013 11:33 pm

Re: Multiprocessing supported in Clarisse Python?

Unread postby Refik » Mon Jan 13, 2020 4:05 pm

Good time!
What about the Python GPU (Numba or PyCUDA)
Is there any chance of this?
Refik
 
Posts: 37
Joined: Mon May 14, 2018 3:01 pm

Re: Multiprocessing supported in Clarisse Python?

Unread postby sam » Mon Jan 13, 2020 4:28 pm

Hi there,

There are no plans to support them.
Sam Assadian
Isotropix
CEO/Founder
User avatar
sam
 
Posts: 1463
Joined: Fri Jan 25, 2013 11:33 pm


Return to Scripting
cron