Hello, we are using ray cluster to optimize resource usage in kubernetes cluster. We have lots of “client” pods scheduling jobs to one shared cluster. Cluster is configured to autoscale as much as possible. Usually every job gets its own worker. Problem is that ray is mixing logs from all jobs together. All client pods are running the same sources but with different configuration.
i.e. client pod X sends jobs A and B to cluster and client pod Y sends jobs C and D to cluster. All run at the same time. X is receiving logs from A and B jobs but also some of the logs created by C and D jobs scheduled by client Y. Y is also receiving logs from all jobs. Those jobs are scheduled on different worker pods.
Is there any configuration option we have missed to prevent this behavior? We are running ray v 1.5.2.