Error using python multiprocessing in Azure Machine Learning

Question

I want to do some parallel data manipulation in Azure ML but I can't seem to get multiprocessing to work. 
So when I try this simple example in Azure ML Python Script Module
import multiprocessing
def azureml_main(dataframe1 = None, dataframe2 = None):
process_pool = multiprocessing.Pool(10)
l = []
for i in range(10):
l.append((0, 1000))
r = process_pool.map(do, l)
process_pool.close()
foo = 0
for a in r:
foo = foo + a
print(foo)
return dataframe1
def do(arg):
foo = 0
for i in range(arg[0], arg[1]):
foo = foo + 1
return foo
I get this error:
    Error 0085: The following error occurred during script evaluation, please view the output log for more information:
     ---------- Start of error message from Python interpreter ----------
     data:text/plain,Caught exception while executing function: Traceback (most recent call last):
    File "C:\server\invokepy.py", line 167, in batch
     odfs = mod.azureml_main(*idfs)
    File "C:\temp\azuremod.py", line 19, in azureml_main
     process_pool = multiprocessing.Pool(10)
    File "C:\pyhome\lib\multiprocessing\__init__.py", line 232, in Pool
     return Pool(processes, initializer, initargs, maxtasksperchild)
    File "C:\pyhome\lib\multiprocessing\pool.py", line 159, in __init__
     self._repopulate_pool()
    File "C:\pyhome\lib\multiprocessing\pool.py", line 223, in _repopulate_pool
     w.start()
    File "C:\pyhome\lib\multiprocessing\process.py", line 130, in start
     self._popen = Popen(self)
    File "C:\pyhome\lib\multiprocessing\forking.py", line 280, in __init__
     to_child.close()
    IOError: [Errno 22] Invalid argument
     ---------- End of error message from Python  interpreter  ----------
When I try the same code in a local regular python file it works just fine
Any ideas? Or is there another way?
 

Answer 1

Hello Grahamaaa,
(a) Regarding map: Yes, vanilla map, list comprehensions, generators, etc. are all sequential.
(b) NumPy: You can call into multi-threaded native code (via OpenMP/TBB/etc.) as long as they follow the usual GIL locking/unlocking protocol. There are some calls such as SetCpuAffinity that are not supported.
There are essentially two ways to around the GIL in CPython: write C extensions that use threads or spawn multiple Python interpreters. The first approach should work modulo calls like SetCpuAffinity.
We are looking into why the fork call in Python is not working.
Regards,
Sudarshan (AzureML)