RayCluster does not limit the total job info stored in redis

Ray Clusters
1

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

I find that when I submit many job to RayCluster(run with redis), the job table & workers table will not clear the oldest items, which 1) cost much more memory for redis; 2) make dashboard GET /api/jobs interface run slowly.

Is any plan to fix it ?

2

Hi there wangxin,

There doesn’t seem to be a direct fix mentioned in the docs, but there are a few things you can try to keep things running smoother. First, check that your cluster is handling memory efficiently - Ray has some built-in tools for managing memory and avoiding out-of-memory issues. The dashboard can also give you a good sense of what’s eating up resources, so keeping an eye on that might help.

If old job and worker entries are piling up, you could look into ways to periodically clear them out or tweak caching settings to keep memory usage in check. I’ll attach some docs that might be helpful down below. Do you have any sort of garbage collection right now in your jobs?

I’ll also go take a look at the Github issues and see if anyone’s mentioned it.

Docs:

3

Thanks for you reply.

I also do not find any workround for this issue, and I think the memory issue is not serious, but GET /api/jobs interface becomes slowly will bother me. I will clear the job info in redis periodically to avoid this issue.

I think it will be prefect if job&worker info can be cleared periodically by gcs.

1 Like