Large data transfer stuck on windows

I have two windows servers(windows server 2019).
so i deploy ray[default] 2.3.0.
i make a simple test, but not success.

@ray.remote
def run_ray_demo(i):
return np.array(range(500000))
ray.get([run_ray_demo.remote(w) for w in range(20)])

program will stuck forever for The above code.
but when i change 500000 to 400000, it will success immediately(about 2~3 seconds).

in my opinion, 500000 is not a large dataset. but why it’s not success ?

@loracle Ray on windows is beta.

Note you also create 20 invocations of run_ray_demo.remote() in the loop, each putting an nparray of 500M. A total of 10GB in the object store. You can see the amount of memory in the object store
with a small np.array in the dashoard or using ray memory or ray status

@Jules_Damji

no, each func is 500K, also 0.5M, not 500M. it’s a small dataset.
and i have used ray memory or ray status to check object store already.

Node status

Healthy:
1 node_4d13888d2f4c958cef57fbebdc2bb0d2b5dc5a2cee407e877f044815
1 node_ea63569c43ac57fea0caf0c1edc79ca00f3f09fa6917ac5814965484
Pending:
(no pending nodes)
Recent failures:
(no failures)

Resources

Usage:
0.0/24.0 CPU
0.0/1.0 GPU
0.00/52.777 GiB memory
0.03/24.919 GiB object_store_memory

@loracle Sorry, you are right. it’s 500K, my eyes saws loads of zeros… :slight_smile:

Curious, do you have another machine, like a laptop with Linux or MacOS? Want to see if you can
run to completion. This seems a bit odd.

Does anything in the logs show up that looks suspicious. ray logs --help gives instructions how to scan the logs.
ray logs cluster. May be peek inside raylet.err and raylet.out

i maybe solve this problem. i opened all tcp ports of two windows machines finally. although it may be dangerous. but it’s success at last.

@loracle Excellent.

Perhaps only open TCP ports that Ray needs.

Btw we only support a single node Window use case, and multi nodes use cases may not work properly. I believe if you use the latest version, this will be clear from the ray start command.