RayServe: Failed to serialize the FastAPI app

Ray Serve
1

Hi eveyone,
I try to use RayServe with FastAPI but it didn’t work. This code work well like normal FastAPI app.

app = FastAPI()

app.include_router(user_access_router, prefix='/user/access', tags=['user'])

 if __name__=="__main__":
     import uvicorn
     uvicorn.run("app:app",reload=True)

But this not :

app = FastAPI()
app.include_router(user_access_router, prefix='/user/access', tags=['user'])
@serve.deployment(name='datastore_app')
@serve.ingress(app)
class DatastoreApp:
    def __init__(self):
        pass

 datastore_app = DatastoreApp.bind()
 serve.run(datastore_app)

The summary of error log like this:

TypeError: cannot pickle '_thread.lock' object
...
    @serve.ingress(app)
     ^^^^^^^^^^^^^^^^^^
TypeError: Failed to serialize the FastAPI app.
...
=======================================
!!! FAIL serialization: cannot pickle '_thread.lock' object
    Serializing 'add_api_route' <bound method FastAPI.add_api_route of <fastapi.applications.FastAPI object at 0x7bdb259e12b0>>...
    !!! FAIL serialization: cannot pickle '_thread.lock' object
        Serializing '__func__' <function FastAPI.add_api_route at 0x7bdb210b1e40>...
    WARNING: Did not find non-serializable object in <bound method FastAPI.add_api_route of <fastapi.applications.FastAPI object at 0x7bdb259e12b0>>. This may be an oversight.
========================================

I inspected with:

inspect_serializability(serve.ingress(app))

The return:

================================================================================
Checking Serializability of <function ingress.<locals>.decorator at 0x7a16395b79c0>
================================================================================
!!! FAIL serialization: cannot pickle '_thread.lock' object
Detected 11 global variables. Checking serializability...
    Serializing 'inspect' <module 'inspect' from '/usr/lib/python3.12/inspect.py'>...
    Serializing 'collections' <module 'collections' from '/usr/lib/python3.12/collections/__init__.py'>...
    Serializing 'Callable' typing.Callable...
    Serializing 'FastAPI' <class 'fastapi.applications.FastAPI'>...
    Serializing 'APIRouter' <class 'fastapi.routing.APIRouter'>...
    Serializing 'make_fastapi_class_based_view' <function make_fastapi_class_based_view at 0x7a163a45ad40>...
    Serializing 'ensure_serialization_context' <function ensure_serialization_context at 0x7a163c3c87c0>...
    Serializing 'cloudpickle' <module 'ray.cloudpickle' from '/home/hieu/Workspace/projects/chatbone/.venv/lib/python3.12/site-packages/ray/cloudpickle/__init__.py'>...
    Serializing 'pickle_dumps' <function pickle_dumps at 0x7a163cb00d60>...
    Serializing 'ASGIAppReplicaWrapper' <class 'ray.serve._private.http_util.ASGIAppReplicaWrapper'>...
    Serializing '__name__' ray.serve.api...
Detected 1 nonlocal variables. Checking serializability...
    Serializing 'app' <fastapi.applications.FastAPI object at 0x7a163f4e4920>...
    !!! FAIL serialization: cannot pickle '_thread.lock' object
        Serializing 'add_api_route' <bound method FastAPI.add_api_route of <fastapi.applications.FastAPI object at 0x7a163f4e4920>>...
        !!! FAIL serialization: cannot pickle '_thread.lock' object
            Serializing '__func__' <function FastAPI.add_api_route at 0x7a163aab9da0>...
        WARNING: Did not find non-serializable object in <bound method FastAPI.add_api_route of <fastapi.applications.FastAPI object at 0x7a163f4e4920>>. This may be an oversight.
================================================================================
Variable: 

	FailTuple(add_api_route [obj=<bound method FastAPI.add_api_route of <fastapi.applications.FastAPI object at 0x7a163f4e4920>>, parent=<fastapi.applications.FastAPI object at 0x7a163f4e4920>])

was found to be non-serializable. There may be multiple other undetected variables that were non-serializable. 
Consider either removing the instantiation/imports of these variables or moving the instantiation into the scope of the function/class. 
================================================================================

I tried to serve the same endpoints fastapi without “include_router” (using only FastAPI() and it still didn’t work. Am i missing something?

Thank you.

UPDATE:
I traced my error and found that it happends because create sqlalchemy.AsyncSession in each FastApi endpoint (both through Dependencies or manual creation give error). But i want to do something like open session (connection), interact with database and close it in each endpoint and i still haven’t found the way to do that with Ray.

2

You can use the route_prefix configuration in Ray Serve applications to achieve the same functionality instead

3

@Akshay_Malik Thanks for your fast response. Can you give me some words about the practice? I already have FastAPI app with some APIRouter and i need to integrate it with RayServe. How to do that? I tried to change one of APIRouter with FastAPI app (no router anymore) but ingress(app) even didn’t work.

4

yes, APIRouter is not supported right now - [Serve] ingress decorator does not work with fastapi.APIRouter arg · Issue #50372 · ray-project/ray · GitHub . Can you share your code snippet? It should use the FastAPI object in ingress instead of Router

5

@Akshay_Malik I solved it. Actually FastApi dependency and router features work very well. I think RayServe-FastAPI integration is very good at it.

I thought the reason it fails before is that i injected the AsyncSession Dependency (Dep that yield Sqlalchemy AsyncSession) directly to endpoint parameters, and maybe Ray try to serialize the Session instead of create the object for endpoint. So i now i just yield it inside the endpoint instead of dependencies in the parameters. And it works.

Example code that run successfully:

@router.delete('/delete')
async def delete_old(schema: ChatSVCDeleteOld):
	async with get_async_session() as session:
		svc = MyService(session)
		await svc.delete_old_messages(schema)
		return dict(info="Old messages deleted successfully.")
1 Like
6

That’s good to hear. Can you share some code snippets of the router setup?