CPU and memory usage on ray-head pod is still increasing and has to be restarted every 3 days.
I have checked that it is not caused by storing objects in cluster but it is probably caused by redis database used by GCS. Records in database are being created but they are never deleted.
I have tried to clean database manually and some of the record can be safely deleted. i.e. “DASHBOARD*” keys are not needed and deleting them delays the time when head node needs to be restarted.
Do you know if this is ray bug or some configuration issue on our side?