Ray serve + FastAPI + APM: TypeError: cannot pickle '_thread.lock' object

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

I’m trying to host a ML model with Ray Serve + FastAPI, and it’s working fine up until I try to add APM server:

from ray import serve
from fastapi import FastAPI
from elasticapm.contrib.starlette import ElasticAPM, make_apm_client

app = FastAPI(
    title="My ML API Server",

# ===========
# newly added
apm_config = {
    "SERVICE_NAME": "My ML API Server",
apm = make_apm_client(config=apm_config)
app.add_middleware(ElasticAPM, client=apm)
# ===========

    ray_actor_options={"num_cpus": 1, "num_gpus": 0}
class MLDeployment:

The error is:

  File "~/Documents/Projects/xxx/./app/server.py", line 46, in <module>
    class MLDeployment:
  File "~/miniconda3/envs/env/lib/python3.10/site-packages/ray/serve/api.py", line 225, in decorator
    frozen_app = cloudpickle.loads(cloudpickle.dumps(app))
  File "~/miniconda3/envs/env/lib/python3.10/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 88, in dumps
  File "~/miniconda3/envs/env/lib/python3.10/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 733, in dump
    return Pickler.dump(self, obj)
TypeError: cannot pickle '_thread.lock' object

I initially started with this doc: Starlette/FastAPI Support | APM Python Agent Reference [6.x] | Elastic, but got error saying TypeError: ElasticAPM.__init__() missing 1 required positional argument: 'client', so I added an apm client, and resulted in above error trace regarding cloudpickle.

Any help to overcome this issue is appreciated!

Could you file this as a bug report on GitHub? It looks like the apm middleware fails to get serialized.

Thank you, just submitted: [Ray Serve] Ray Serve + FastAPI + Elastic APM: TypeError: cannot pickle '_thread.lock' object · Issue #42255 · ray-project/ray · GitHub