A problem that came up recently was to manage multiple long running processes, intended to run forever, from the same Python process. Using something like supervisord wouldn’t be ideal, because each individual application would have its own config file and deployment procedure.
Instead, having just one “master” Python process to drive several other processes doing the job seemed the easiest solution to manage. The only caveat is that, being the processes managed long running, you want to restart them as soon as they die.
See below a take to solve the problem using concurrent.futures. The class
maintain a reference to the running futures and as soon as one finishes, it
checks if it was successful or not. If not, it re-submits. Callbacks
on_success can be overwritten to execute some actions on the finished