I notice currently it uses Python based operator. I am asking any plan to extract this part and have a separate operator writter by golang.
The advantage is that most of golang client tools like controller-runtime, k8s.io/client etc are mainly written by golang. Even there’s official python tools, matureness and update frequency is not that good as golang’s.
The other part is I feel the ray operator currently doesn’t provide enough handling. For example, if the object fetching failed (404), operator is down directly and relies on k8s restart to recover.
operator pod is started as single pod which is not safe. k8s golang controller has enough tools like leader election, code generator to easily help us handle this issues.
If you think this is something worth to do, I can help on this issue
We’re working on stability improvements. Did you encounter a 404 error when using the operator?
We’re planning in the near future to at the very least back the operator pod by a deployment.
We’re probably going to stick to Python for the foreseeable future. Right now, the operator is a thin layer around the Ray autoscaler, which is a key component of the Ray codebase and will definitely stay in Python.
In principle, we could arrange things so that the Ray autoscaler makes requests to a Go-based operator that handles the actual creation and deletion of pods and other API resources, but I don’t think we have the bandwidth for such a re-architecture right now.