Ray and python versions

Hi Ray Team,

Is there restrictions or limitations of running ray >=1.10 with python 3.6.9?
for some reason in ray version above 1.8 I’m getting the following error: (with 1.6 it’s work fine)

2022-07-21 08:54:30,177 INFO server.py:842 – Starting Ray Client server on 0.0.0.0:10001
2022-07-21 10:15:17,379 INFO proxier.py:670 – New data connection from client ace70b4b753146babdea12418dbbb528:
2022-07-21 10:15:18,403 INFO proxier.py:341 – SpecificServer started on port: 23000 with PID: 415 for client: ace70b4b753146babdea12418dbbb528
2022-07-21 10:15:48,405 ERROR proxier.py:379 – Timeout waiting for channel for ace70b4b753146babdea12418dbbb528
Traceback (most recent call last):
File “/usr/local/lib/python3.6/dist-packages/ray/util/client/server/proxier.py”, line 375, in get_channel
timeout=CHECK_CHANNEL_TIMEOUT_S
File “/usr/local/lib/python3.6/dist-packages/grpc/_utilities.py”, line 140, in result
self._block(timeout)
File “/usr/local/lib/python3.6/dist-packages/grpc/_utilities.py”, line 86, in _block
raise grpc.FutureTimeoutError()
grpc.FutureTimeoutError
2022-07-21 10:15:48,405 ERROR proxier.py:379 – Timeout waiting for channel for ace70b4b753146babdea12418dbbb528
Traceback (most recent call last):
File “/usr/local/lib/python3.6/dist-packages/ray/util/client/server/proxier.py”, line 375, in get_channel
timeout=CHECK_CHANNEL_TIMEOUT_S
File “/usr/local/lib/python3.6/dist-packages/grpc/_utilities.py”, line 140, in result
self._block(timeout)
File “/usr/local/lib/python3.6/dist-packages/grpc/_utilities.py”, line 86, in _block
raise grpc.FutureTimeoutError()
grpc.FutureTimeoutError
2022-07-21 10:15:48,406 ERROR proxier.py:692 – Channel not found for ace70b4b753146babdea12418dbbb528
2022-07-21 10:15:48,406 WARNING proxier.py:777 – Retrying Logstream connection. 1 attempts failed.

python version 3.6.9
ray version > 1.10

Thanks

hi @ray1, welcome to the community!

I suspect this is to do with the python grpc library version you are using. would you mind sharing your pip freeze result?

On another note, can you try ray==1.13.0 to see if this problem still exists?

Hi @Chen_Shen, Thanks for replying so quickly
output form pip freeze | grep grpc , both head\worker and job is:
grpcio==1.39.0
grpcio-tools==1.39.0
unfortunately, 1.13.0 not reolving my issue.
double checked that all my pods is with ray==1.13.0

I’m in k8s deployment

ah, I realized it’s a ray client/server connection issue. Just checking you are using the same python/ray version on ray client/server?

yes both pods (client\server) are with the same ray & python version. I’m using my one docker and with ray-1.13.0-py36 wheel install on it