Python’s multiprocessing module offers a
Surprisingly enough (for me at least), its performance varies greatly if used in a blocking or non-blocking fashion. In particular, the non-blocking way is about ten times slower than the blocking one!
Consider this code, which feeds a Queue and then drains it in a blocking and non-blocking way:
Running it on my machine, with Python 2.7, gives me these results (Python 3.4 results are similar, but less dramatic):
feed=0.599231004715 drain=0.964730024338 drain_nowait=8.39909696579
As you can see,
drain_nowait is about 10x slower than
drain. I suspect there is some serious lock-contention going on, but I have not yet pinpointed where, exactly! In the meantime, I think I’ll stick with the blocking code…