Home | Articles | CV (pdf | short)
<2016-04-25> by Lorenzo

Comparison of blocking and non-blocking Python's multiprocessing Queue

Python's multiprocessing module offers a Queue implementation. Surprisingly enough (for me at least), its performance varies greatly if used in a blocking or non-blocking fashion. In particular, the non-blocking way is about ten times slower than the blocking one!

Consider this code, which feeds a Queue and then drains it in a blocking and non-blocking way:

Running it on my machine, with Python 2.7, gives me these results (Python 3.4 results are similar, but less dramatic):

feed=0.599231004715 drain=0.964730024338 drain_nowait=8.39909696579

As you can see, drain_nowait is about 10x slower than drain. I suspect there is some serious lock-contention going on, but I have not yet pinpointed where, exactly! In the meantime, I think I'll stick with the blocking code…