The pprocess
+module provides several mechanisms for running Python code concurrently
+in several processes. The most straightforward way of making a program
+parallel-aware - that is, where the program can take advantage of more
+than one processor to simultaneously process data - is to use the
+pmap function. Consider the following Python code:
+ t = time.time() + + sequence = [] + for i in range(0, N): + for j in range(0, N): + sequence.append((i, j)) + + results = map(calculate, sequence) + + print "Time taken:", time.time() - t + for i in range(0, N): + for result in results[i*N:i*N+N]: + print result, + print ++ +
(This code in context with import statements and functions is found in the examples/simple_map.py file.)
Here, we initialise a sequence of inputs for an N by N grid, perform a
+calculation on each element, then print the resulting sequence as a
+grid. Since the map function performs the calculations sequentially, even if the calculate
+function could be invoked independently for each input value, we have
+to wait for each calculation to complete before initiating a new
+one.
In order to reduce the processing time - to speed the code up, +in other words - we can make this code use several processes instead of +just one. Here is the modified code:
+ ++ t = time.time() + + sequence = [] + for i in range(0, N): + for j in range(0, N): + sequence.append((i, j)) + + results = pprocess.pmap(calculate, sequence, limit=limit) + + print "Time taken:", time.time() - t + for i in range(0, N): + for result in results[i*N:i*N+N]: + print result, + print ++ +
(This code in context with import statements and functions is found in the examples/simple_pmap.py file.)
By replacing usage of the map function with the pprocess.pmap
+function, and specifying the limit on the number of processes to be active at any
+given time, several calculations can now be performed in parallel.