Ray.get on single task not working properly on cluster

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

I am currently executing a time consuming task (>10 minutes) from a rabbitmq consumer which processes only one task at a time.
So basically I’m doing ray.get() on each task individually, but the processes only gets executed on a single machine and that too on a single worker processes in my cluster.

I wish to execute these incoming task on multiple nodes in my cluster and different worker processes should execute these task.

Hi @jeffy. How does the rabbitmq consumer provide you with tasks? It sounds to me that you need a way to invoke Ray tasks asynchronously so that the rabbitmq consumer can get back to consuming.

1 Like