How to connect to Ray cluster?

Ray Clusters Kubernetes
1

I am new to Ray. I am trying to create a private cluster on-prem.

I installed ray on host1 and started as master, host2 joined a node.

I can connect to the cluster from master (host1) without any problem and could use all the resources from master and worker. But when I try to connect from python from worker node, it takes a long time to connect

ray.init(address='***')
2021-07-09 14:14:06,484	INFO worker.py:735 -- Connecting to existing Ray cluster at address: 

In [3]: /worker/lib/python3.9/site-packages/jupyter_console/ptshell.py:783: UserWarning: The kernel did not respond to an is_complete_request. Setting `use_kernel_is_complete` to False.
  warn('The kernel did not respond to an is_complete_request. '

The kernel stops responding. Can I only run the job from master node?

This has been addressed here, but I just want to make sure if I am using it correctly.

2

Is it possible that the redis port is not open? or that you have to use a private IP address?

Alternatively, this should be doable with the Ray Client: https://docs.ray.io/en/master/cluster/ray-client.html

Note that the address port will be different from the Ray Init command you’re currently using.

3

I can connect to Redis from workers. What port should I be using to connect to cluster in client mode?

4

I found out 10001 is the port. I could connect from the workers successfully, does ray support connecting from arbitrary servers, I could connect to redis from arbitrary machines but not as ray client?

5

Using the client would allow you to connect from any machine that has port access.

6

Got it, I figured out. This is truly amazing. Keep up the good work

1 Like
7

Thanks for the kind words :slight_smile:

What are you doing with Ray, if you don’t mind me asking?

8

Creating a on-prem cluster for every day data pipelines. It’s so trivial to parallelize them for any general purpose python lib. Spark cluster is little hard to setup (yarn/mesos)

2 Likes