I would recommend Dask for this sort of work as it is easy to produce a task graph and minimise computation overhead
import dask
def A(argument):
print('Running A with', argument)
return argument
DomHudson
DomHudson
/ dask.md
Created
February 25, 2021 12:24
https://stackoverflow.com/questions/66319724/task-chaining-with-shift-using-celery?noredirect=1#comment117332393_66319724
DomHudson
/ custom_worker_processes.py
Last active
July 18, 2023 17:12
Multiprocessing in Python with Custom Workers
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import random | |
| import multiprocessing | |
| context = multiprocessing.get_context() | |
| class Worker(context.Process): | |
| def __init__(self, *args, **kwargs): |