Long running processes manager
Class to manage long running processes in Python, restarting them when necessary


If you want to run multiple processes in Python, the nicest way, in my opinion, is to use concurrent.futures (there is a backport for Python 2.x, too).

A problem that came up recently was to manage multiple long running processes, intended to run forever, from the same Python process. Using something like supervisord wouldn’t be ideal, because each individual application would have its own config file and deployment procedure.

Instead, having just one “master” Python process to drive several other processes doing the job seemed the easiest solution to manage. The only caveat is that, being the processes managed long running, you want to restart them as soon as they die.

See below a take to solve the problem using concurrent.futures. The class maintain a reference to the running futures and as soon as one finishes, it checks if it was successful or not. If not, it re-submits. Callbacks on_error and on_success can be overwritten to execute some actions on the finished futures.

January 17, 2014
163 words


Categories
Tags
futures python

Contact me

Hi! I'm Lorenzo Bolla, Senior Software Architect in London, UK.

When I'm not programming in Python, Rust, Go or Haskell, I am playing either basketball, piano or chess.