Using ray.cloudpickle methods in the serve deployment - error

Ray Clusters
1

As soon as I use ray.cloudpickle.dumps/loads methods, the error about Ray Client is not connected is reported.

Ray Client is not connected. Please connect by calling `ray.init`.

Env: a ray cluster (head with 2 workers) on k3s installed with ray operator. Code is run from a juypyter notebook on the ray head.

In contrast, running similar code on a local machine ray does not have issue using ray.cloudpickle methods.

Here’s the serve deployment. In the current state, it produces the error. If I comment out the lines with cloudpickle and run the ones with ray.get/put - no problems.

import ray
from ray import serve
import json

# Connect to cluster
try:
    # runtime_env = {"pip":["ruptures","pathos"]}
    ray_client = ray.init(f"ray://localhost:10001", namespace="serve")
except Exception as e:
    print("Exception",e)
    pass
finally:
    print("Ray serve started")
    client = serve.start(detached=True, http_options={"host": "0.0.0.0"})

@serve.deployment(route_prefix="/lastchunker")
class LastChunker:
    async def __call__(self, request):
        try:
            raw_json = await request.json()
            id_str = raw_json.get('store_id')
            id_pkld = base64.b64decode(id_str.encode())
            id_restored = ray.cloudpickle.loads(id_pkld)
            ray_data = ray.get(id_restored
            # one = ray.put(1)
            
            return {"ray_data": ray_data}
            # return {"ray_data": ray.get(one)}
        except Exception as ex:
            return {"error": ex}

LastChunker.deploy()

Here is the calling code:

import requests
import base64

f = ray.put("foo")
print(f"f: {f}")
fpikl = ray.cloudpickle.dumps(f)
print(f"fpikl: {fpikl}")

b64fpkl = base64.b64encode(fpikl).decode()
print(f"b64fpkl: {b64fpkl}")

data='{"store_id": "' + f"{b64fpkl}" +'"}'
print(f"data: {data}")

id_str = json.loads(data).get('store_id')
print (f"id_str: {id_str}")

id_pkld = base64.b64decode(id_str.encode())
print(f"id_pkld: {id_pkld}")

id_restored = ray.cloudpickle.loads(id_pkld)
print(f"id_restored: {id_restored}")

resp = requests.get("http://localhost:8000/lastchunker", data=data)
resp.json()
2

I was able to get past the Ray is not connected error by calling ray.shutdown() after deploying the model and then doing ray.init() in the calling code.

The ray.cloudpickle.get() call works, but the ray.get() is not able to find the object, but rather now have the error

(raylet, ip=10.42.1.28) [2022-03-04 10:07:57,747 E 127 127] object_manager.cc:261: 
Couldn't send pull request from 
143716f6b723ac70e631505b86f8adbf0334549cc7a75edc7fe932ec 
to 
38ba8defd158a418425b79207dd50532e8e6953cecd58d8992631e49 
of object ffffffffffffffffffffffffffffffffffffffff0100000001000000 , 
setup rpc connection failed.
3

I think I figured out why the objects cannot be found… SHHH… Don’t tell anyone. The python that was running the client code had not called ray start to connect to the cluster.