gRPC service doesn't work for nested directory structures?

Ray Libraries (Data, Train, Tune, Serve) Ray Serve
1
  • High: It blocks me to complete my task.

Hi, I am following the guide to set up a gRPC service. It works when I put the service definition (service.py) and the proto generated py files under the same directory. However, when I moved the proto generated files to a sub-directory proto/, I got some weird python import errors.

Here is my directory tree structure (skipping __pycache__ folders)

.
├── proto
│   ├── __init__.py
│   ├── user_defined_protos.proto
│   ├── user_defined_protos_pb2.py
│   └── user_defined_protos_pb2_grpc.py
└── service.py

Since I moved the proto files under proto directory, I added proto. to the import paths.

# service.py
import time
import subprocess
from proto.user_defined_protos_pb2 import (
    UserDefinedMessage,
    UserDefinedResponse,
)

import ray
from ray import serve
from ray.serve.config import gRPCOptions

@serve.deployment
class GrpcDeployment:
    def __call__(self, user_message: UserDefinedMessage) -> UserDefinedResponse:
        greeting = f"Hello {user_message.name} from {user_message.origin}"
        num = user_message.num * 2
        user_response = UserDefinedResponse(
            greeting=greeting,
            num=num,
        )
        return user_response

g = GrpcDeployment.bind()

subprocess.run(["ray", "start", "--head", "--dashboard-host=0.0.0.0"], check=True)
ray.init()
grpc_port = 9000
grpc_servicer_functions = [
    "proto.user_defined_protos_pb2_grpc.add_UserDefinedServiceServicer_to_server",
    "proto.user_defined_protos_pb2_grpc.add_ImageClassificationServiceServicer_to_server",
]

serve.start(
    http_options={"host": "0.0.0.0"},   
    grpc_options=gRPCOptions(
        port=grpc_port,
        grpc_servicer_functions=grpc_servicer_functions,
    ),
)
serve.run(GrpcDeployment.bind(), blocking=True)

This gives an error when I ran python service.py (The full service.py is in the reproduction script section):

TypeError: Could not serialize the deployment <class '__main__.GrpcDeployment'>:
...
	FailTuple(DESCRIPTOR [obj=<google._upb._message.Descriptor object at 0x104fd9260>, parent=<class 'user_defined_protos_pb2.UserDefinedResponse'>])
was found to be non-serializable. There may be multiple other undetected variables that were non-serializable.

I then tried to move the import statement into the __call__ function body according to the discussion here:

# Top-level import removed
@serve.deployment
class GrpcDeployment:
    # type hint removed
    def __call__(self, user_message):
        # Move import here
        from proto.user_defined_protos_pb2 import (
            UserDefinedMessage,
            UserDefinedResponse,
        )
        greeting = f"Hello {user_message.name} from {user_message.origin}"
        num = user_message.num * 2
        user_response = UserDefinedResponse(
            greeting=greeting,
            num=num,
        )
        return user_response

Then I got a different error:

ModuleNotFoundError: proto.user_defined_protos_pb2_grpc.add_UserDefinedServiceServicer_to_server can't be imported! Please make sure there are no typo in those functions. Or you might want to rebuild service definitions if .proto file is changed. [repeated 5x across cluster]

Is this expected? Due to my project setup, I need to put the proto generated files under a different directory. How can I make it work? Thanks!

Versions / Dependencies

python 3.11
rayserve 2.32.0

2

I found the reason for the error. When I move the proto files to the proto/ directory, I need to rerun the protoc tool to regenerate the py files:

python -m grpc_tools.protoc -I=. --python_out=. --pyi_out=. --grpc_python_out=.  ./proto/service.proto

This way the generated py files will have the right import paths. The original server script in the tutorial would work without any modification.

1 Like