[Core] [Client] Cloud pickle library missing in ray client dependency

Ray Core Ray Client
1

Hello, I tried using ray client library to connect to minikube cluster by following steps in documentation :

https://docs.ray.io/en/master/cluster/kubernetes.html

I ran:

python ray/doc/kubernetes/example_scripts/run_local_example.py

I get the below error:

$ python ray/doc/kubernetes/example_scripts
Traceback (most recent call last):
File “ray/doc/kubernetes/example_scripts/run_local_example.py”, line 57, in
ray.util.connect(f"127.0.0.1:{LOCAL_PORT}")
File “/home/asmalvan/anaconda3/envs/ray/lib/python3.7/site-packages/ray/util/client_connect.py”, line 26, in connect
conn_str, secure=secure, metadata=metadata, connection_retries=3)
File “/home/asmalvan/anaconda3/envs/ray/lib/python3.7/site-packages/ray/util/client/init.py”, line 40, in connect
from ray.util.client.worker import Worker
File “/home/asmalvan/anaconda3/envs/ray/lib/python3.7/site-packages/ray/util/client/worker.py”, line 22, in
from ray.util.client.client_pickler import convert_to_arg
File “/home/asmalvan/anaconda3/envs/ray/lib/python3.7/site-packages/ray/util/client/client_pickler.py”, line 24, in
import cloudpickle
ModuleNotFoundError: No module named ‘cloudpickle’
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File “”, line 983, in _find_and_load
File “”, line 953, in _find_and_load_unlocked
File “”, line 219, in _call_with_frames_removed
File “/home/asmalvan/anaconda3/envs/ray/lib/python3.7/site-packages/ray/util/client/server/init.py”, line 1, in
from ray.util.client.server.server import serve # noqa
File “/home/asmalvan/anaconda3/envs/ray/lib/python3.7/site-packages/ray/util/client/server/server.py”, line 20, in
from ray.util.client.server.server_pickler import convert_from_arg
File “/home/asmalvan/anaconda3/envs/ray/lib/python3.7/site-packages/ray/util/client/server/server_pickler.py”, line 14, in
import cloudpickle
ModuleNotFoundError: No module named ‘cloudpickle’

For now, I did a conda install of cloudpickle not sure which version of cloudpickle is recommended

2

Can you verify this is still the case on the newest ray install (1.3)?

3

I confirm that this error does not exists for ray 1.3.0 but now I get a different error @rliaw , please find below the trace:

$ python ray/doc/kubernetes/example_scripts/run_local_example.py
/home/asmalvan/anaconda3/envs/ray130/lib/python3.7/site-packages/ray/autoscaler/_private/cli_logger.py:61: FutureWarning: Not all Ray CLI dependencies were found. In Ray 1.4+, the Ray CLI, autoscaler, and dashboard will only be usable via pip install 'ray[default]'. Please update your install command.
“update your install command.”, FutureWarning)
Got Error from data channel – shutting down: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.UNKNOWN
details = “Exception iterating responses: "
debug_error_string = “{“created”:”@1619207923.847024876”,“description”:“Error received from peer ipv4:127.0.0.1:10001”,“file”:“src/core/lib/surface/call.cc”,“file_line”:1067,“grpc_message”:"Exception iterating responses: “,“grpc_status”:2}”

Exception in thread Thread-5:
Traceback (most recent call last):
File “/home/asmalvan/anaconda3/envs/ray130/lib/python3.7/threading.py”, line 926, in _bootstrap_inner
self.run()
File “/home/asmalvan/anaconda3/envs/ray130/lib/python3.7/threading.py”, line 870, in run
self._target(*self._args, **self._kwargs)
File “/home/asmalvan/anaconda3/envs/ray130/lib/python3.7/site-packages/ray/util/client/dataclient.py”, line 90, in _data_main
raise e
File “/home/asmalvan/anaconda3/envs/ray130/lib/python3.7/site-packages/ray/util/client/dataclient.py”, line 64, in _data_main
for response in resp_stream:
File “/home/asmalvan/anaconda3/envs/ray130/lib/python3.7/site-packages/grpc/_channel.py”, line 426, in next
return self._next()
File “/home/asmalvan/anaconda3/envs/ray130/lib/python3.7/site-packages/grpc/_channel.py”, line 826, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.UNKNOWN
details = “Exception iterating responses: "
debug_error_string = “{“created”:”@1619207923.847024876”,“description”:“Error received from peer ipv4:127.0.0.1:10001”,“file”:“src/core/lib/surface/call.cc”,“file_line”:1067,“grpc_message”:"Exception iterating responses: “,“grpc_status”:2}”

Traceback (most recent call last):
File “ray/doc/kubernetes/example_scripts/run_local_example.py”, line 57, in
ray.util.connect(f"127.0.0.1:{LOCAL_PORT}")
File “/home/asmalvan/anaconda3/envs/ray130/lib/python3.7/site-packages/ray/util/client_connect.py”, line 32, in connect
ignore_version=ignore_version)
File “/home/asmalvan/anaconda3/envs/ray130/lib/python3.7/site-packages/ray/util/client/init.py”, line 70, in connect
conn_info = self.client_worker.connection_info()
File “/home/asmalvan/anaconda3/envs/ray130/lib/python3.7/site-packages/ray/util/client/worker.py”, line 156, in connection_info
data = self.data_client.ConnectionInfo()
File “/home/asmalvan/anaconda3/envs/ray130/lib/python3.7/site-packages/ray/util/client/dataclient.py”, line 124, in ConnectionInfo
resp = self._blocking_send(datareq)
File “/home/asmalvan/anaconda3/envs/ray130/lib/python3.7/site-packages/ray/util/client/dataclient.py”, line 109, in _blocking_send
"Cannot send request due to data channel "
ConnectionError: Cannot send request due to data channel shutting down. Request: req_id: 1
connection_info {