"Received message larger than max" error when sending request with GRPC

How severe does this issue affect your experience of using Ray?

  • High

Hello, I have been able to create a deployment using ray serve, which takes documents and processes them (NLP inference). I followed the examples from Set Up a gRPC Service — Ray 2.9.1 and was able to implement grpc, with the server running at localhost:9000.

I am able to send requests from the client using the following:

# Running on osx 14.2
# grpc==1.5.9
# protobuf==4.25.1
# ray[serve]==2.9.1
import grpc
from my_service_pb2_grpc import MyServiceStub
from my_service_pb2 import MyMessage, MyResponse

options = [
        ('grpc.max_message_length', 1024 * 1024 * 1024), 
        ('grpc.max_send_message_length', 1024 * 1024 * 1024), 
        ('grpc.max_receive_message_length', 1024 * 1024 * 1024),
channel = grpc.insecure_channel("localhost:9000", options=options)
stub = MyServiceStub(channel)

# data is a list of dicts that conforms to the protobuf schema
request = MyMessage(documents = data)

response, call = stub.__call__.with_call(request=request)

When data is sufficiently small (<4 mb), everything runs well. However when data is >4 mb I receive an error such as:

_InactiveRpcError                         Traceback (most recent call last)
Cell In[12], line 1
----> 1 response, call = stub.__call__.with_call(request=request)

File ~/miniconda3/envs/myservice/lib/python3.10/site-packages/grpc/_channel.py:1178, in _UnaryUnaryMultiCallable.with_call(self, request, timeout, metadata, credentials, wait_for_ready, compression)
   1163 def with_call(
   1164     self,
   1165     request: Any,
   1170     compression: Optional[grpc.Compression] = None,
   1171 ) -> Tuple[Any, grpc.Call]:
   1172     (
   1173         state,
   1174         call,
   1175     ) = self._blocking(
   1176         request, timeout, metadata, credentials, wait_for_ready, compression
   1177     )
-> 1178     return _end_unary_response_blocking(state, call, True, None)

File ~/miniconda3/envs/myservice/lib/python3.10/site-packages/grpc/_channel.py:1004, in _end_unary_response_blocking(state, call, with_call, deadline)
   1002         return state.response
   1003 else:
-> 1004     raise _InactiveRpcError(state)

_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.RESOURCE_EXHAUSTED
	details = "Received message larger than max (8468525 vs. 4194304)"
	debug_error_string = "UNKNOWN:Error received from peer ipv6:%5B::1%5D:9000 {created_time:"2024-01-31T14:09:19.248513-05:00", grpc_status:8, grpc_message:"Received message larger than max (8468525 vs. 4194304)"}"

Things I have done

  • As you can see above, I have set 'grpc.max_send_message_length' and 'grpc.max_receive_message_length' when creating the channel and passing in options.
  • I also ran into errors with receiving large data, but on the client side setting 'grpc.max_receive_message_length' in options fixed this. This leads me to believe this is an issue with ray serve
  • I have looked at the code in gcs_util.py and ray_constants.py and it seems that all the constants point to message sizes of 250 - 500 mbs, and these options should be passed to the grpc server
  • I have also ran into this error on a windows 10 machine in which my service is able to receive and pass back responses without issue as long as the initial MyMessage size is <4 mb.
  • I have tried downgrading to ray 2.9.0 and am receiving the same errors.

I am really stuck as this seems to be coming from ray serve, and would appreciate help in solving this as it is preventing me from moving forward. Thanks in advance!

Hi @jjinho thanks for reporting this issue. I was able to reproduce. The issue is tracked in this ticket [Serve] gRPC proxy server is not respecting Ray's gRPC options · Issue #43109 · ray-project/ray · GitHub I will try to fix it ASAP. Should be fixed by Ray 2.10

1 Like

Hi @Gene thank you for looking into that! For what it’s worth, while trying to figure this bug out, I also found another bug on the grpc client side ("Sent message larger than max (X vs. Y)" - Error Message shows correct value for Y only if X is close to value, otherwise reverts to 4194304 · Issue #35805 · grpc/grpc · GitHub) but it seems that this may be a separate issue.

Do you know when Ray 2.10 is anticipated to be released? Thank you!

Awesome! Will keep that issue in mind.

Ray 2.10 will be cut end of this month (2/29) and the target release date is 3/18.

The fix is merged in master at this point if you want to give it a try early :slight_smile:

Hi @Gene I installed a nightly and was able to confirm that it works. Thank you!

Thanks for reporting back, glad it helped :smile: