source: thirdparty/blosc/README_THREADED.rst @ 00587dc

Revision 00587dc, 1.5 KB checked in by Hal Finkel <hfinkel@…>, 9 years ago (diff)

Initial Commit (gio-base-20150317)

  • Property mode set to 100644
Line 
1Blosc supports threading
2========================
3
4Threads are the most efficient way to program parallel code for
5multi-core processors, but also the more difficult to program well.
6Also, they has a non-negligible start-up time that does not fit well
7with a high-performance compressor as Blosc tries to be.
8
9In order to reduce the overhead of threads as much as possible, I've
10decided to implement a pool of threads (the workers) that are waiting
11for the main process (the master) to send them jobs (basically,
12compressing and decompressing small blocks of the initial buffer).
13
14Despite this and many other internal optimizations in the threaded
15code, it does not work faster than the serial version for buffer sizes
16around 64/128 KB or less.  This is for Intel Quad Core2 (Q8400 @ 2.66
17GHz) / Linux (openSUSE 11.2, 64 bit), but your mileage may vary (and
18will vary!) for other processors / operating systems.
19
20In contrast, for buffers larger than 64/128 KB, the threaded version
21starts to perform significantly better, being the sweet point at 1 MB
22(again, this is with my setup).  For larger buffer sizes than 1 MB,
23the threaded code slows down again, but it is probably due to a cache
24size issue and besides, it is still considerably faster than serial
25code.
26
27This is why Blosc falls back to use the serial version for such a
28'small' buffers.  So, you don't have to worry too much about deciding
29whether you should set the number of threads to 1 (serial) or more
30(parallel).  Just set it to the number of cores in your processor and
31your are done!
32
33Francesc Alted
Note: See TracBrowser for help on using the repository browser.