[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/message_lite.cc:402] ray.rpc.GetCoreWorkerStatsReply exceeded maximum protobuf size of 2GB: 2250430536

Ray Core
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

ray version is 2.7.2. protobuf version is 3.20.3. when I run task in ray cluster, I got the error about [libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/message_lite.cc:402] ray.rpc.GetCoreWorkerStatsReply exceeded maximum protobuf size of 2GB: 2250430536.

How I can fix this error?

2

Hi @Skylarking

Are you using the state api? If so, you can set the limit so it returns less data.

3

No, this function is called by Ray core. And so, how I can set the limit of returns? I can not find the docs of this api.

4

This API should be triggered transitively when you call a state API. Also, could you try latest version of Ray. 2.7.2 is pretty old and this problem might have been fixed already.