Long running processes manager

If you want to run multiple processes in Python, the nicest way, in my opinion, is to use concurrent.futures (there is a backport for Python 2.x, too).

A problem that came up recently was to manage multiple long running processes, intended to run forever, from the same Python process. Using something like supervisord wouldn't be ideal, because each individual application would have its own config file and deployment procedure.

Instead, having just one "master" Python process to drive several other processes doing the job seemed the easiest solution to manage. The only caveat is that, being the processes managed long running, you want to restart them as soon as they die.

See below a take to solve the problem using concurrent.futures. The class maintain a reference to the running futures and as soon as one finishes, it checks if it was successful or not. If not, it re-submits. Callbacks on_error and on_success can be overwritten to execute some actions on the finished futures.