paulb@22 | 1 | Introduction
|
paulb@22 | 2 | ------------
|
paulb@22 | 3 |
|
paulb@40 | 4 | The pprocess module provides elementary support for parallel programming in
|
paulb@22 | 5 | Python using a fork-based process creation model in conjunction with a
|
paulb@68 | 6 | channel-based communications model implemented using socketpair and poll. On
|
paulb@68 | 7 | systems with multiple CPUs or multicore CPUs, processes should take advantage
|
paulb@68 | 8 | of as many CPUs or cores as the operating system permits.
|
paulb@22 | 9 |
|
paulb@22 | 10 | Quick Start
|
paulb@22 | 11 | -----------
|
paulb@22 | 12 |
|
paulb@105 | 13 | Try running the simple examples. For example:
|
paulb@68 | 14 |
|
paulb@100 | 15 | PYTHONPATH=. python examples/simple_create.py
|
paulb@105 | 16 |
|
paulb@105 | 17 | (These examples show in different ways how limited number of processes can be
|
paulb@105 | 18 | used to perform a parallel computation. The simple.py and simple_map.py
|
paulb@105 | 19 | programs are sequential versions of the other programs.)
|
paulb@105 | 20 |
|
paulb@105 | 21 | The following table summarises the features used in the programs:
|
paulb@105 | 22 |
|
paulb@105 | 23 | Program (.py) pmap MakeParallel manage start create Queue Exchange
|
paulb@105 | 24 | ------------- ---- ------------ ------ ----- ------ ----- --------
|
paulb@105 | 25 | simple
|
paulb@105 | 26 | simple_create Yes Yes
|
paulb@105 | 27 | simple_create_queue Yes Yes
|
paulb@105 | 28 | simple_managed Yes Yes Yes
|
paulb@105 | 29 | simple_managed_queue Yes Yes Yes
|
paulb@105 | 30 | simple_map
|
paulb@105 | 31 | simple_pmap Yes
|
paulb@105 | 32 | simple_start Yes Yes
|
paulb@105 | 33 | simple_start_queue Yes Yes Yes
|
paulb@68 | 34 |
|
paulb@105 | 35 | The simplest parallel program is simple_pmap.py which employs the pmap
|
paulb@105 | 36 | function resembling the built-in map function in Python.
|
paulb@105 | 37 |
|
paulb@105 | 38 | Other simple programs are those employing the Queue class, together with those
|
paulb@105 | 39 | using the manage method which associates functions or callables with Queue or
|
paulb@105 | 40 | Exchange objects for convenient invocation of those functions and the
|
paulb@105 | 41 | management of their communications.
|
paulb@105 | 42 |
|
paulb@105 | 43 | The most technically involved program is simple_start.py which uses the
|
paulb@105 | 44 | Exchange class together with a calculation function which is aware of the
|
paulb@105 | 45 | parallel environment and which communicates over the supplied communications
|
paulb@105 | 46 | channel directly to the creating process.
|
paulb@105 | 47 |
|
paulb@105 | 48 | It should be noted that with the exception of simple_start.py, those examples
|
paulb@105 | 49 | employing calculation functions (as opposed to doing a calculation inline in a
|
paulb@105 | 50 | loop body) all use MakeParallel to make those functions parallel-aware, thus
|
paulb@105 | 51 | permitting the conversion of "normal" functions to a form usable in the
|
paulb@105 | 52 | parallel environment.
|
paulb@100 | 53 |
|
paulb@100 | 54 | The tutorial provides some information about the examples: docs/tutorial.xhtml
|
paulb@100 | 55 |
|
paulb@105 | 56 | Parallel Raytracing with PyGmy
|
paulb@105 | 57 | ------------------------------
|
paulb@105 | 58 |
|
paulb@100 | 59 | The PyGmy raytracer modified to use pprocess can be run to investigate the
|
paulb@105 | 60 | potential for speed increases in "real world" programs:
|
paulb@68 | 61 |
|
paulb@100 | 62 | cd examples/PyGmy
|
paulb@100 | 63 | PYTHONPATH=../..:. python scene.py
|
paulb@100 | 64 |
|
paulb@100 | 65 | (This should produce a file called test.tif - a TIFF file containing a
|
paulb@100 | 66 | raytraced scene image.)
|
paulb@100 | 67 |
|
paulb@105 | 68 | Test Programs
|
paulb@105 | 69 | -------------
|
paulb@105 | 70 |
|
paulb@100 | 71 | There are some elementary tests:
|
paulb@22 | 72 |
|
paulb@22 | 73 | PYTHONPATH=. python tests/create_loop.py
|
paulb@22 | 74 | PYTHONPATH=. python tests/start_loop.py
|
paulb@22 | 75 |
|
paulb@22 | 76 | (Simple loop demonstrations which use two different ways of creating and
|
paulb@22 | 77 | starting the parallel processes.)
|
paulb@22 | 78 |
|
paulb@36 | 79 | PYTHONPATH=. python tests/start_indexer.py <directory>
|
paulb@22 | 80 |
|
paulb@36 | 81 | (A text indexing demonstration, where <directory> should be a directory
|
paulb@36 | 82 | containing text files to be indexed, although HTML files will also work well
|
paulb@36 | 83 | enough. After indexing the files, a prompt will appear, words or word
|
paulb@36 | 84 | fragments can be entered, and matching words and their locations will be
|
paulb@36 | 85 | shown. Run the program without arguments to see more information.)
|
paulb@22 | 86 |
|
paulb@22 | 87 | Contact, Copyright and Licence Information
|
paulb@22 | 88 | ------------------------------------------
|
paulb@22 | 89 |
|
paulb@22 | 90 | No Web page has yet been made available for this work, but the author can be
|
paulb@22 | 91 | contacted at the following e-mail address:
|
paulb@22 | 92 |
|
paulb@22 | 93 | paul@boddie.org.uk
|
paulb@22 | 94 |
|
paulb@22 | 95 | Copyright and licence information can be found in the docs directory - see
|
paulb@78 | 96 | docs/COPYING.txt, docs/lgpl-3.0.txt and docs/gpl-3.0.txt for more information.
|
paulb@22 | 97 |
|
paulb@48 | 98 | For the PyGmy raytracer example, different copyright and licence information
|
paulb@48 | 99 | is provided in the docs directory - see docs/COPYING-PyGmy.txt and
|
paulb@48 | 100 | docs/LICENCE-PyGmy.txt for more information.
|
paulb@48 | 101 |
|
paulb@22 | 102 | Dependencies
|
paulb@22 | 103 | ------------
|
paulb@22 | 104 |
|
paulb@22 | 105 | This software depends on standard library features which are stated as being
|
paulb@22 | 106 | available only on "UNIX"; it has only been tested on a GNU/Linux system.
|
paulb@22 | 107 |
|
paulb@100 | 108 | New in parallel 0.3 (Changes since parallel 0.2.5)
|
paulb@100 | 109 | --------------------------------------------------
|
paulb@84 | 110 |
|
paulb@84 | 111 | * Added managed callables: wrappers around callables which cause them to be
|
paulb@84 | 112 | automatically managed by the exchange from which they were acquired.
|
paulb@84 | 113 | * Added MakeParallel: a wrapper instantiated around a normal function which
|
paulb@84 | 114 | sends the result of that function over the supplied channel when invoked.
|
paulb@89 | 115 | * Added a Map class which attempts to emulate the built-in map function,
|
paulb@89 | 116 | along with a pmap function using this class.
|
paulb@100 | 117 | * Added a Queue class which provides a simpler iterator-style interface to
|
paulb@100 | 118 | data produced by created processes.
|
paulb@100 | 119 | * Added a create method to the Exchange class and an exit convenience
|
paulb@100 | 120 | function to the module.
|
paulb@100 | 121 | * Changed the Exchange implementation to not block when attempting to start
|
paulb@100 | 122 | new processes beyond the process limit: such requests are queued and
|
paulb@100 | 123 | performed as running processes are completed. This permits programs using
|
paulb@100 | 124 | the start method to proceed to consumption of results more quickly.
|
paulb@105 | 125 | * Extended and updated the examples. Added a tutorial.
|
paulb@100 | 126 | * Added Ubuntu Feisty (7.04) package support.
|
paulb@84 | 127 |
|
paulb@78 | 128 | New in parallel 0.2.5 (Changes since parallel 0.2.4)
|
paulb@78 | 129 | ----------------------------------------------------
|
paulb@78 | 130 |
|
paulb@78 | 131 | * Added a start method to the Exchange class for more convenient creation of
|
paulb@78 | 132 | processes.
|
paulb@78 | 133 | * Relicensed under the LGPL (version 3 or later) - this also fixes the
|
paulb@78 | 134 | contradictory situation where the GPL was stated in the pprocess module
|
paulb@78 | 135 | (which was not, in fact, the intention) and the LGPL was stated in the
|
paulb@78 | 136 | documentation.
|
paulb@78 | 137 |
|
paulb@73 | 138 | New in parallel 0.2.4 (Changes since parallel 0.2.3)
|
paulb@73 | 139 | ----------------------------------------------------
|
paulb@73 | 140 |
|
paulb@73 | 141 | * Set buffer sizes to zero for the file object wrappers around sockets: this
|
paulb@73 | 142 | may prevent deadlock issues.
|
paulb@73 | 143 |
|
paulb@68 | 144 | New in parallel 0.2.3 (Changes since parallel 0.2.2)
|
paulb@68 | 145 | ----------------------------------------------------
|
paulb@68 | 146 |
|
paulb@68 | 147 | * Added convenient message exchanges, offering methods handling common
|
paulb@68 | 148 | situations at the cost of having to define a subclass of Exchange.
|
paulb@68 | 149 | * Added a simple example of performing a parallel computation.
|
paulb@68 | 150 | * Improved the PyGmy raytracer example to use the newly added functionality.
|
paulb@68 | 151 |
|
paulb@55 | 152 | New in parallel 0.2.2 (Changes since parallel 0.2.1)
|
paulb@55 | 153 | ----------------------------------------------------
|
paulb@55 | 154 |
|
paulb@55 | 155 | * Changed the status testing in the Exchange class, potentially fixing the
|
paulb@55 | 156 | premature closure of channels before all data was read.
|
paulb@55 | 157 | * Fixed the PyGmy raytracer example's process accounting by relying on the
|
paulb@55 | 158 | possibly more reliable Exchange behaviour, whilst also preventing
|
paulb@55 | 159 | erroneous creation of "out of bounds" processes.
|
paulb@58 | 160 | * Added a removed attribute on the Exchange to record which channels were
|
paulb@58 | 161 | removed in the last call to the ready method.
|
paulb@55 | 162 |
|
paulb@48 | 163 | New in parallel 0.2.1 (Changes since parallel 0.2)
|
paulb@48 | 164 | --------------------------------------------------
|
paulb@48 | 165 |
|
paulb@48 | 166 | * Added a PyGmy raytracer example.
|
paulb@53 | 167 | * Updated copyright and licensing details (FSF address, additional works).
|
paulb@48 | 168 |
|
paulb@40 | 169 | New in parallel 0.2 (Changes since parallel 0.1)
|
paulb@40 | 170 | ------------------------------------------------
|
paulb@40 | 171 |
|
paulb@40 | 172 | * Changed the name of the included module from parallel to pprocess in order
|
paulb@40 | 173 | to avoid naming conflicts with PyParallel.
|
paulb@40 | 174 |
|
paulb@22 | 175 | Release Procedures
|
paulb@22 | 176 | ------------------
|
paulb@22 | 177 |
|
paulb@40 | 178 | Update the pprocess __version__ attribute.
|
paulb@22 | 179 | Change the version number and package filename/directory in the documentation.
|
paulb@22 | 180 | Update the release notes (see above).
|
paulb@22 | 181 | Check the release information in the PKG-INFO file.
|
paulb@22 | 182 | Tag, export.
|
paulb@22 | 183 | Archive, upload.
|
paulb@68 | 184 | Update PyPI.
|
paulb@26 | 185 |
|
paulb@26 | 186 | Making Packages
|
paulb@26 | 187 | ---------------
|
paulb@26 | 188 |
|
paulb@44 | 189 | To make Debian-based packages:
|
paulb@26 | 190 |
|
paulb@44 | 191 | 1. Create new package directories under packages if necessary.
|
paulb@26 | 192 | 2. Make a symbolic link in the distribution's root directory to keep the
|
paulb@26 | 193 | Debian tools happy:
|
paulb@26 | 194 |
|
paulb@44 | 195 | ln -s packages/ubuntu-hoary/python2.4-parallel-pprocess/debian/
|
paulb@26 | 196 |
|
paulb@100 | 197 | Or:
|
paulb@100 | 198 |
|
paulb@100 | 199 | ln -s packages/ubuntu-feisty/python-pprocess/debian/
|
paulb@100 | 200 |
|
paulb@26 | 201 | 3. Run the package builder:
|
paulb@26 | 202 |
|
paulb@26 | 203 | dpkg-buildpackage -rfakeroot
|
paulb@26 | 204 |
|
paulb@26 | 205 | 4. Locate and tidy up the packages in the parent directory of the
|
paulb@26 | 206 | distribution's root directory.
|