python - les deux multiprocessing.map et joblib utiliser seulement 1 cpu après mise à niveau de Ubuntu 10.10 à 12.04 - Stack Overflow

I had some perfectly working python code which used multiprocessing module and loaded all 8 CPUs on my machine at 100%.

After I upgraded from Ubuntu 10.10 to 12.04 (the most evident thing, maybe I did something else that broke everything), it stopped working. After lots of debugging, I found that even in the simplest use case, both modules are only using 1 CPU:

from pylab import *
import multiprocessing as mp
from joblib import Parallel, delayed

def f(i):
    # Slow calculation
    x = 1
    for j in range(100000): x = cos(x)
    print i, x

if __name__ == '__main__':
    # Try to multiprocess with multiprocessing
    mp.Pool(processes=8).map(f, range(100))
    # Try to multiprocess with joblib
    Parallel(n_jobs=8)(delayed(f)(i) for i in range(100))

I need to use all 8 CPUs in my system. Any ideas of what I should look at to fix the issue?

EDIT:

As ali_m pointed out in a comment here and in the answer to What determines whether different Python processes are assigned to the same or different cores? the problem is related to numpy messing up with cpu affinity. Calling

os.system('taskset -p 0xffffffff %d' % os.getpid())

Before I do any multiprocessing solved the problem for me.

I had some perfectly working python code which used multiprocessing module and loaded all 8 CPUs on my machine at 100%.

After I upgraded from Ubuntu 10.10 to 12.04 (the most evident thing, maybe I did something else that broke everything), it stopped working. After lots of debugging, I found that even in the simplest use case, both modules are only using 1 CPU:

from pylab import *
import multiprocessing as mp
from joblib import Parallel, delayed

def f(i):
    # Slow calculation
    x = 1
    for j in range(100000): x = cos(x)
    print i, x

if __name__ == '__main__':
    # Try to multiprocess with multiprocessing
    mp.Pool(processes=8).map(f, range(100))
    # Try to multiprocess with joblib
    Parallel(n_jobs=8)(delayed(f)(i) for i in range(100))

I need to use all 8 CPUs in my system. Any ideas of what I should look at to fix the issue?

EDIT:

As ali_m pointed out in a comment here and in the answer to What determines whether different Python processes are assigned to the same or different cores? the problem is related to numpy messing up with cpu affinity. Calling

os.system('taskset -p 0xffffffff %d' % os.getpid())

Before I do any multiprocessing solved the problem for me.

Source