Some issues when using the Ray Client

1

Hey everyone, I have been trying to connect to a ray cluster I setup on Kubernetes (using the relevant instructions in the docs) from a Jupyter notebook. I was directed to the Ray Client feature still in beta), which proved to be the perfect thing for what I needed.

I had to run the command specified in the docs, make sure that the 50051 port was exposed and then kubectl port-forwarded to my local machine. There I used the ray.util.connect function to connect to the ray server.

I initially ran into some issues that I would like to share. I encountered the following error:

image
image2351×774 239 KB

The connection to the client seemed to work fine but actually trying to run something failed.

Turns out the issue was that both the python and ray versions need to be the same between the client and server (python 3.7.7 and ray 2.0.0.dev0 in this case). Just wanted to let everyone know in case someone uses this feature! Is this expected behaviour? Do people know when this feature might come out of beta?

Thanks a lot!
Andy

1 Like
2

I’d be surprised if the Python version mattered (if it does, that’d be a really good bug to reproduce!) but the client versions needing to match is, for the moment, expected – part of why it’s beta is that the protocol between the two is maturing rapidly. The error you posted looks (without having a full reproduction) like exactly that problem – that, I think, the server looks like it’s a little older than the client.

We’re evolving it quickly and we’ll remove beta when there’s a release (not a nightly) that we’re comfortable supporting long-term.

Hang in there :slight_smile: – but let me know how it works for you, and if you hit bugs do file issues!

1 Like