1 Introduction
2 ------------
3
4 The pprocess module provides elementary support for parallel programming in
5 Python using a fork-based process creation model in conjunction with a
6 channel-based communications model implemented using socketpair and poll. On
7 systems with multiple CPUs or multicore CPUs, processes should take advantage
8 of as many CPUs or cores as the operating system permits.
9
10 Quick Start
11 -----------
12
13 Try running the simple examples:
14
15 PYTHONPATH=. python examples/simple.py
16 PYTHONPATH=. python examples/simple_create.py
17 PYTHONPATH=. python examples/simple_managed.py
18 PYTHONPATH=. python examples/simple_map.py
19 PYTHONPATH=. python examples/simple_pmap.py
20 PYTHONPATH=. python examples/simple_queue.py
21
22 (A simple example which shows how a limited number of processes can be used to
23 perform a parallel computation. The simple_map.py program is a sequential
24 version of the other programs.)
25
26 The tutorial provides some information about the examples: docs/tutorial.xhtml
27
28 The PyGmy raytracer modified to use pprocess can be run to investigate the
29 potential for speed increases:
30
31 cd examples/PyGmy
32 PYTHONPATH=../..:. python scene.py
33
34 (This should produce a file called test.tif - a TIFF file containing a
35 raytraced scene image.)
36
37 There are some elementary tests:
38
39 PYTHONPATH=. python tests/create_loop.py
40 PYTHONPATH=. python tests/start_loop.py
41
42 (Simple loop demonstrations which use two different ways of creating and
43 starting the parallel processes.)
44
45 PYTHONPATH=. python tests/start_indexer.py <directory>
46
47 (A text indexing demonstration, where <directory> should be a directory
48 containing text files to be indexed, although HTML files will also work well
49 enough. After indexing the files, a prompt will appear, words or word
50 fragments can be entered, and matching words and their locations will be
51 shown. Run the program without arguments to see more information.)
52
53 Contact, Copyright and Licence Information
54 ------------------------------------------
55
56 No Web page has yet been made available for this work, but the author can be
57 contacted at the following e-mail address:
58
59 paul@boddie.org.uk
60
61 Copyright and licence information can be found in the docs directory - see
62 docs/COPYING.txt, docs/lgpl-3.0.txt and docs/gpl-3.0.txt for more information.
63
64 For the PyGmy raytracer example, different copyright and licence information
65 is provided in the docs directory - see docs/COPYING-PyGmy.txt and
66 docs/LICENCE-PyGmy.txt for more information.
67
68 Dependencies
69 ------------
70
71 This software depends on standard library features which are stated as being
72 available only on "UNIX"; it has only been tested on a GNU/Linux system.
73
74 New in parallel 0.3 (Changes since parallel 0.2.5)
75 --------------------------------------------------
76
77 * Added managed callables: wrappers around callables which cause them to be
78 automatically managed by the exchange from which they were acquired.
79 * Added MakeParallel: a wrapper instantiated around a normal function which
80 sends the result of that function over the supplied channel when invoked.
81 * Added a Map class which attempts to emulate the built-in map function,
82 along with a pmap function using this class.
83 * Added a Queue class which provides a simpler iterator-style interface to
84 data produced by created processes.
85 * Added a create method to the Exchange class and an exit convenience
86 function to the module.
87 * Changed the Exchange implementation to not block when attempting to start
88 new processes beyond the process limit: such requests are queued and
89 performed as running processes are completed. This permits programs using
90 the start method to proceed to consumption of results more quickly.
91 * Extended and updated the examples.
92 * Added Ubuntu Feisty (7.04) package support.
93
94 New in parallel 0.2.5 (Changes since parallel 0.2.4)
95 ----------------------------------------------------
96
97 * Added a start method to the Exchange class for more convenient creation of
98 processes.
99 * Relicensed under the LGPL (version 3 or later) - this also fixes the
100 contradictory situation where the GPL was stated in the pprocess module
101 (which was not, in fact, the intention) and the LGPL was stated in the
102 documentation.
103
104 New in parallel 0.2.4 (Changes since parallel 0.2.3)
105 ----------------------------------------------------
106
107 * Set buffer sizes to zero for the file object wrappers around sockets: this
108 may prevent deadlock issues.
109
110 New in parallel 0.2.3 (Changes since parallel 0.2.2)
111 ----------------------------------------------------
112
113 * Added convenient message exchanges, offering methods handling common
114 situations at the cost of having to define a subclass of Exchange.
115 * Added a simple example of performing a parallel computation.
116 * Improved the PyGmy raytracer example to use the newly added functionality.
117
118 New in parallel 0.2.2 (Changes since parallel 0.2.1)
119 ----------------------------------------------------
120
121 * Changed the status testing in the Exchange class, potentially fixing the
122 premature closure of channels before all data was read.
123 * Fixed the PyGmy raytracer example's process accounting by relying on the
124 possibly more reliable Exchange behaviour, whilst also preventing
125 erroneous creation of "out of bounds" processes.
126 * Added a removed attribute on the Exchange to record which channels were
127 removed in the last call to the ready method.
128
129 New in parallel 0.2.1 (Changes since parallel 0.2)
130 --------------------------------------------------
131
132 * Added a PyGmy raytracer example.
133 * Updated copyright and licensing details (FSF address, additional works).
134
135 New in parallel 0.2 (Changes since parallel 0.1)
136 ------------------------------------------------
137
138 * Changed the name of the included module from parallel to pprocess in order
139 to avoid naming conflicts with PyParallel.
140
141 Release Procedures
142 ------------------
143
144 Update the pprocess __version__ attribute.
145 Change the version number and package filename/directory in the documentation.
146 Update the release notes (see above).
147 Check the release information in the PKG-INFO file.
148 Tag, export.
149 Archive, upload.
150 Update PyPI.
151
152 Making Packages
153 ---------------
154
155 To make Debian-based packages:
156
157 1. Create new package directories under packages if necessary.
158 2. Make a symbolic link in the distribution's root directory to keep the
159 Debian tools happy:
160
161 ln -s packages/ubuntu-hoary/python2.4-parallel-pprocess/debian/
162
163 Or:
164
165 ln -s packages/ubuntu-feisty/python-pprocess/debian/
166
167 3. Run the package builder:
168
169 dpkg-buildpackage -rfakeroot
170
171 4. Locate and tidy up the packages in the parent directory of the
172 distribution's root directory.