celery-flask-redis-cloudrun-docker.MD

Running a Docker on Google Cloud Run with Celery + Flask + Redis

Tricks

1. Set everything up normally and get things working in testing.
2. Have the docker run something like

CMD ["./server.sh"]

at the end

3. In server.sh have something like

redis-server  &
P1=$!

celery -A app.celery worker --loglevel=info &
P2=$!

gunicorn --bind :5000 --workers 4 --threads 8 --timeout 0 app:app
P3=$!

wait $P1 $P2 $P3

4. And run that locally... Does it work? Cool.
5. When you deploy to Cloud Run, your tasks might fuck up if they're long lasting or making external API calls. In my case, I'm making a call to a Google API that takes a long time. Before I made a specific change I got an SSL errors galore.
6. Make sure you switch the "CPU Allocation and Pricing" option to CPU is always allocated
7. Should work

Yes I know I should use a VM or Sidecars or something. I don't care right now. I just need something that works. Remember, tech debt can be a privelage.

That didn't work for me with Celery making API calls to LLm APIs.

@lazaroborges Are you deploying to cloud run? Steps 5/6 are important for it to work.

Happy to provide my 2 cents if you provide more details.

Hey, Luke! Thanks. I ended up doing an old school VM. More or less the same thing (later I added a django-rq to avoid database locks in Sqlite).

BUt the "tech debt can be a privilege" hasn't been forgot till this day. :D