1.1 --- a/docs/reference.html Fri Jun 05 01:46:07 2009 +0200
1.2 +++ b/docs/reference.html Fri Jun 05 21:23:25 2009 +0200
1.3 @@ -32,6 +32,7 @@
1.4 <li><a href="#Map">Map-style Processing</a></li>
1.5 <li><a href="#Reusing">Reusing Processes and Channels</a></li>
1.6 <li><a href="#MakeReusable">Making Existing Functions Parallel and Reusable</a></li>
1.7 +<li><a href="#Continuous">Continuous Processes and Channels</a></li>
1.8 <li><a href="#BackgroundCallable">Background Processes and Callables</a></li>
1.9 <li><a href="#PersistentQueue">Background and Persistent Queues</a></li>
1.10 <li><a href="#Implementation">Implementation Notes</a></li>
2.1 --- a/docs/tutorial.html Fri Jun 05 01:46:07 2009 +0200
2.2 +++ b/docs/tutorial.html Fri Jun 05 21:23:25 2009 +0200
2.3 @@ -18,7 +18,12 @@
2.4 <ul>
2.5 <li><a href="#pmap">Converting Map-Style Code</a></li>
2.6 <li><a href="#Map">Converting Invocations to Parallel Operations</a></li>
2.7 -<li><a href="#Queue">Converting Arbitrarily-Ordered Invocations</a></li>
2.8 +<li><a href="#Queue">Converting Arbitrarily-Ordered Invocations</a>
2.9 + <ul>
2.10 + <li><a href="#Exchange">Replacing Queues with Exchanges</li></a>
2.11 + <li><a href="#channel">Using Channels in Callables</li></a>
2.12 + </ul>
2.13 +</li>
2.14 <li><a href="#create">Converting Inline Computations</a></li>
2.15 <li><a href="#MakeReusable">Reusing Processes in Parallel Programs</a></li>
2.16 <li><a href="#continuous">Supporting Continuous Processes in Parallel Programs</a></li>
2.17 @@ -309,6 +314,8 @@
2.18 taking advantage of the more relevant "positional" information emerging from
2.19 the queue.</p>
2.20
2.21 +<h3 id="Exchange">Replacing Queues with Exchanges</h3>
2.22 +
2.23 <p>We can take this example further, illustrating some of the mechanisms
2.24 employed by <code>pprocess</code>. Instead of collecting results in a queue,
2.25 we can define a class containing a method which is called when new results
2.26 @@ -382,6 +389,8 @@
2.27 other structure so that they may be processed and assigned to the correct
2.28 positions in the result array.</p>
2.29
2.30 +<h3 id="channel">Using Channels in Callables</h3>
2.31 +
2.32 <p>For the curious, we may remove some of the remaining conveniences of the
2.33 above program to expose other features of <code>pprocess</code>. First, we
2.34 define a slightly modified version of the <code>calculate</code> function:</p>
2.35 @@ -592,6 +601,85 @@
2.36 creating a new process for each computation and then discarding it, only to
2.37 create a new process for the next computation.</p>
2.38
2.39 +<h2 id="continuous">Supporting Continuous Processes in Parallel Programs</h2>
2.40 +
2.41 +<p>Although reusable processes offer the opportunity to invoke a callable over
2.42 +and over within the same created process, they do not fully support the
2.43 +potential of the underlying mechanisms in <code>pprocess</code>: created
2.44 +processes can communicate multiple values to the creating process and can
2.45 +theoretically run within the same callable forever.</p>
2.46 +
2.47 +<p>Consider this modified form of the <code>calculate</code> function:</p>
2.48 +
2.49 +<pre>
2.50 +def calculate(ch, i):
2.51 +
2.52 + """
2.53 + A supposedly time-consuming calculation on 'i'.
2.54 + """
2.55 +
2.56 + for j in range(0, N):
2.57 + time.sleep(delay)
2.58 + ch.send((i, j, i * N + j))
2.59 +</pre>
2.60 +
2.61 +<p>This function accepts a channel <code>ch</code> together with an argument
2.62 +<code>i</code> corresponding to an entire row of the input array, as opposed
2.63 +to having two arguments (<code>i</code> and <code>j</code>) corresponding to a
2.64 +single cell in the input array. In this function, a series of calculations are
2.65 +performed and a number of values are returned through the channel, without the
2.66 +function terminating until all values have been returned for the row data.</p>
2.67 +
2.68 +<p>To use this modified function, a modified version of the
2.69 +<a href="#simple_managed_queue">simple_managed_queue</a> program is used:</p>
2.70 +
2.71 +<pre>
2.72 + t = time.time()
2.73 +
2.74 + # Initialise the communications queue with a limit on the number of
2.75 + # channels/processes.
2.76 +
2.77 + queue = pprocess.Queue(limit=limit<strong>, continuous=1</strong>)
2.78 +
2.79 + # Initialise an array.
2.80 +
2.81 + results = [0] * N * N
2.82 +
2.83 + # Manage the calculate function.
2.84 +
2.85 + calc = queue.manage(<strong>calculate</strong>)
2.86 +
2.87 + # Perform the work.
2.88 +
2.89 + print "Calculating..."
2.90 + for i in range(0, N):
2.91 + <strong>calc(i)</strong>
2.92 +
2.93 + # Store the results as they arrive.
2.94 +
2.95 + print "Finishing..."
2.96 + for i, j, result in queue:
2.97 + results[i*N+j] = result
2.98 +
2.99 + # Show the results.
2.100 +
2.101 + for i in range(0, N):
2.102 + for result in results[i*N:i*N+N]:
2.103 + print result,
2.104 + print
2.105 +
2.106 + print "Time taken:", time.time() - t
2.107 +</pre>
2.108 +
2.109 +<p>(This code in context with <code>import</code> statements and functions is
2.110 +found in the <code>examples/simple_continuous_queue.py</code> file.)</p>
2.111 +
2.112 +<p>Although the inner loop in the work section relocated to the
2.113 +<code>calculate</code> function, the queue still receives outputs from that
2.114 +function with positional information and a result for the result array. Thus,
2.115 +no change is needed for the retrieval of the results: they arrive in the queue
2.116 +as before.</p>
2.117 +
2.118 <h2 id="BackgroundCallable">Performing Computations in Background Processes</h2>
2.119
2.120 <p>Occasionally, it is desirable to initiate time-consuming computations and to
2.121 @@ -714,10 +802,6 @@
2.122 be collected by the queue: a list containing all of the results produced in the
2.123 computation.</p>
2.124
2.125 -<h2 id="continuous">Supporting Continuous Processes in Parallel Programs</h2>
2.126 -
2.127 -<p><strong>To be written.</strong></p>
2.128 -
2.129 <h2 id="ManagingBackgroundProcesses">Managing Several Background Processes</h2>
2.130
2.131 <p>In the above example, a single background process was used to manage a number
2.132 @@ -887,11 +971,15 @@
2.133 <td>MakeParallel, Map, manage</td>
2.134 </tr>
2.135 <tr>
2.136 - <td rowspan="5">simple2</td>
2.137 + <td rowspan="6">simple2</td>
2.138 <td>simple_managed_queue</td>
2.139 <td>MakeParallel, Queue, manage</td>
2.140 </tr>
2.141 <tr>
2.142 + <td>simple_continuous_queue</td>
2.143 + <td>Queue, manage (continuous)</td>
2.144 + </tr>
2.145 + <tr>
2.146 <td>simple_managed</td>
2.147 <td>MakeParallel, Exchange (subclass), manage, finish</td>
2.148 </tr>