# Testing Ray Cluster via Manual Setup

**URL:** <https://discuss.ray.io/t/testing-ray-cluster-via-manual-setup/367>\
**Category:** Uncategorized\
**Created:** [January 1, 2021, 10:43pm UTC](https://discuss.ray.io/t/testing-ray-cluster-via-manual-setup/367 "2021-01-01T22:43:40Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gregory](https://avatars.discourse-cdn.com/v4/letter/g/e56c9b/32.png) [@Gregory](https://discuss.ray.io/u/Gregory)\
**Post date:** [January 1, 2021, 10:43pm UTC](https://discuss.ray.io/t/testing-ray-cluster-via-manual-setup/367/1 "2021-01-01T22:43:40Z")

</div>

Happy 2021 everyone 😃

Today I made it my goal to speed test ray with cluster setup. It didn’t go so well, maybe I can submit to the docs a use-case example like this one when I have it working?

**My setup as follows:**

1. Head Node: VPC with 4 cores, no firewall, Ray 1.1.0, Python 3.6.9
2. Worker Node: Google Colab with 2 cores, firewall, Ray 1.1.0, Python 3.6.9

**Steps taken:**

1. on head node - ray start --head
2. on worker node - ray start --address=‘8.8.8.8:6379’ --redis-password=‘0000000000000000’ # note address has been obfuscated
3. on head node - python custom\_tf\_policy.py

custom\_tf\_policy.py is located at [https://github.com/ray-project/ray/blob/master/rllib/examples/custom\_tf\_policy.py#L48](https://github.com/ray-project/ray/blob/master/rllib/examples/custom_tf_policy.py#L48)

I changed line 48 to ray.init(address=‘auto’, \_redis\_password=‘0000000000000000’)  
I changed line 56 to “num\_workers”: 5,

**Observations**  
The head node starts perfectly, as does connecting the worker to the head. The output looks normal when executing the script “custom\_tf\_policy.py”, but it doesn’t make it to training:

> 2021-01-01 22:08:29,908 INFO worker.py:657 – Connecting to existing Ray cluster at address: 8.8.8.8:6379  
> == Status ==  
> Memory usage on this node: 1.6/7.8 GiB  
> Using FIFO scheduling algorithm.  
> Resources requested: 6/14 CPUs, 0/4 GPUs, 0.0/33.84 GiB heap, 0.0/10.16 GiB objects  
> Result logdir: /home/ray/ray\_results/MyCustomTrainer\_2021-01-01\_22-08-29  
> Number of trials: 1/1 (1 RUNNING)  
> ±----------------------------------------±---------±------+  
> | Trial name | status | loc |  
> |-----------------------------------------±---------±------|  
> | MyCustomTrainer\_CartPole-v0\_e153c\_00000 | RUNNING | |  
> ±----------------------------------------±---------±------+

There are a few INFO and WARNING messages but nothing related to workers, network, or anything unusual. A ctrl-c yields the following:

> File “/home/ray/.local/lib/python3.6/site-packages/ray/tune/tune.py”, line 419, in run  
> runner.step()  
> File “/home/ray/.local/lib/python3.6/site-packages/ray/tune/trial\_runner.py”, line 360, in step  
> self.\_process\_events() # blocking  
> File “/home/ray/.local/lib/python3.6/site-packages/ray/tune/trial\_runner.py”, line 469, in \_process\_events  
> trial = self.trial\_executor.get\_next\_available\_trial() # blocking  
> File “/home/ray/.local/lib/python3.6/site-packages/ray/tune/ray\_trial\_executor.py”, line 472, in get\_next\_available\_trial  
> [result\_id], \_ = ray.wait(shuffled\_results)  
> File “/home/ray/.local/lib/python3.6/site-packages/ray/worker.py”, line 1513, in wait  
> worker.current\_task\_id,

Here’s the full script to save you from opening the page:

> import argparse  
> import os
> 
> import ray  
> from ray import tune  
> from ray.rllib.agents.trainer\_template import build\_trainer  
> from ray.rllib.evaluation.postprocessing import discount\_cumsum  
> from ray.rllib.policy.tf\_policy\_template import build\_tf\_policy  
> from ray.rllib.utils.framework import try\_import\_tf
> 
> tf1, tf, tfv = try\_import\_tf()
> 
> parser = argparse.ArgumentParser()  
> parser.add\_argument(“–stop-iters”, type=int, default=200)  
> parser.add\_argument(“–num-cpus”, type=int, default=0)
> 
> def policy\_gradient\_loss(policy, model, dist\_class, train\_batch):  
> logits, \_ = model.from\_batch(train\_batch)  
> action\_dist = dist\_class(logits, model)  
> return -tf.reduce\_mean(  
> action\_dist.logp(train\_batch[“actions”]) \* train\_batch[“returns”])
> 
> def calculate\_advantages(policy,  
> sample\_batch,  
> other\_agent\_batches=None,  
> episode=None):  
> sample\_batch[“returns”] = discount\_cumsum(sample\_batch[“rewards”], 0.99)  
> return sample\_batch
> 
> MyTFPolicy = build\_tf\_policy(  
> name=“MyTFPolicy”,  
> loss\_fn=policy\_gradient\_loss,  
> postprocess\_fn=calculate\_advantages,  
> )
> 
> MyTrainer = build\_trainer(  
> name=“MyCustomTrainer”,  
> default\_policy=MyTFPolicy,  
> )
> 
> if \ __name_\_ == “\__ main_\_”:  
> args = parser.parse\_args()  
> ray.init(address=‘auto’, \_redis\_password=‘0000000000000000’)  
> tune.run(  
> MyTrainer,  
> stop={“training\_iteration”: args.stop\_iters},  
> config={  
> “env”: “CartPole-v0”,  
> # Use GPUs iff `RLLIB_NUM_GPUS` env var set to \> 0.  
> “num\_gpus”: int(os.environ.get(“RLLIB\_NUM\_GPUS”, “0”)),  
> “num\_workers”: 5,  
> “framework”: “tf”,  
> })

---

<div class="post-metadata">

**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\
**Post date:** [January 4, 2021, 5:40pm UTC](https://discuss.ray.io/t/testing-ray-cluster-via-manual-setup/367/2 "2021-01-04T17:40:40Z")

</div>

When you run this driver in a head node

```auto
import ray
ray.init(address='auto')
print(ray.nodes())

```

Does it show worker nodes?

---

<div class="post-metadata">

**Author:** ![Gregory](https://avatars.discourse-cdn.com/v4/letter/g/e56c9b/32.png) [@Gregory](https://discuss.ray.io/u/Gregory)\
**Post date:** [January 4, 2021, 6:20pm UTC](https://discuss.ray.io/t/testing-ray-cluster-via-manual-setup/367/3 "2021-01-04T18:20:43Z")

</div>

Yes, I’ve got 3 connected and it shows them in the JSON records when calling ray.nodes()

```
[{'NodeID': '18490838782c70', 'Alive': True, 'NodeManagerAddress': '8.8.8.8', 'NodeManagerHostname': 'head_node', 'NodeManagerPort': 52159, 'ObjectManagerPort': 40277, 'ObjectStoreSocketName': '/tmp/ray/session_2021-01-04_18-05-21_391764_1834/sockets/plasma_store', 'RayletSocketName': '/tmp/ray/session_2021-01-04_18-05-21_391764_1834/sockets/raylet', 'MetricsExportPort': 53464, 'alive': True, 'Resources': {'object_store_memory': 30.0, 'node:8.8.8.8': 1.0, 'memory': 89.0, 'CPU': 4.0}}, 
{'NodeID': '17a0e68d826f7', 'Alive': True, 'NodeManagerAddress': '4.4.4.4', 'NodeManagerHostname': 'worker1', 'NodeManagerPort': 65239, 'ObjectManagerPort': 34027, 'ObjectStoreSocketName': '/tmp/ray/session_2021-01-04_18-05-21_391764_1834/sockets/plasma_store', 'RayletSocketName': '/tmp/ray/session_2021-01-04_18-05-21_391764_1834/sockets/raylet', 'MetricsExportPort': 62493, 'alive': True, 'Resources': {'object_store_memory': 51.0, 'memory': 174.0, 'CPU': 2.0, 'node:172.28.0.2': 1.0}}, 
{'NodeID': '8718f31416623', 'Alive': True, 'NodeManagerAddress': '192.168.0.14', 'NodeManagerHostname': 'worker2', 'NodeManagerPort': 55809, 'ObjectManagerPort': 40157, 'ObjectStoreSocketName': '/tmp/ray/session_2021-01-04_18-05-21_391764_1834/sockets/plasma_store', 'RayletSocketName': '/tmp/ray/session_2021-01-04_18-05-21_391764_1834/sockets/raylet', 'MetricsExportPort': 40028, 'alive': True, 'Resources': {'memory': 219.0, 'accelerator_type:GTX': 1.0, 'node:192.168.0.14': 1.0, 'GPU': 4.0, 'CPU': 8.0, 'object_store_memory': 64.0}}]
```

---

<div class="post-metadata">

**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\
**Post date:** [January 4, 2021, 6:29pm UTC](https://discuss.ray.io/t/testing-ray-cluster-via-manual-setup/367/4 "2021-01-04T18:29:23Z")

</div>

Hmm maybe there’s an issue with the script in this case? cc @sven1977

---

<div class="post-metadata">

**Author:** ![Gregory](https://avatars.discourse-cdn.com/v4/letter/g/e56c9b/32.png) [@Gregory](https://discuss.ray.io/u/Gregory)\
**Post date:** [January 4, 2021, 6:46pm UTC](https://discuss.ray.io/t/testing-ray-cluster-via-manual-setup/367/5 "2021-01-04T18:46:22Z")

</div>

I’m very thankful for the responses 🙏  
And happy to submit a PR to add what we find to the docs regarding manual cluster setup.

Another user in the slack channel also said they could never get a manual cluster working.

---

<div class="post-metadata">

**Author:** ![Gregory](https://avatars.discourse-cdn.com/v4/letter/g/e56c9b/32.png) [@Gregory](https://discuss.ray.io/u/Gregory)\
**Post date:** [January 6, 2021, 3:30pm UTC](https://discuss.ray.io/t/testing-ray-cluster-via-manual-setup/367/6 "2021-01-06T15:30:40Z")

</div>

Solved thanks to the help of a friend, the problem was ports not being open. A note to new users testing their cluster… first test with computers having all necessary ports open.

I recommend adding to the docs the need to check the following file for clues about any issues with the cluster: /tmp/ray/session\_latest/logs/raylet.out

---

<div class="post-metadata">

**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\
**Post date:** [January 22, 2021, 10:58pm UTC](https://discuss.ray.io/t/testing-ray-cluster-via-manual-setup/367/7 "2021-01-22T22:58:26Z")

</div>

We have a port specification here; [Configuring Ray — Ray v1.1.0](https://docs.ray.io/en/latest/configure.html#ports-configurations), but I think the visibility was poor enough that you didn’t understand. What sort of document would help you to resolve this issue on your own? I’d love to hear recommendation.