Our environment is : ray-2.10.0 on windows 64, python 3.10 anaconda env
We created a detached named actor in one jupyter session, but new jupyter seesions can’t connect to the same single-node cluster with error as following:
e[33m(raylet)e[0m E0000 00:00:1711958744.976248 16196 wire_format_lite.cc:626] String field ‘ray.rpc.WorkerTableData.exit_detail’ contains invalid UTF-8 data when serializing a protocol buffer. Use the ‘bytes’ type if you intend to send raw bytes.
e[36m(pid=gcs_server)e[0m E0000 00:00:1711958744.976625 1088 wire_format_lite.cc:626] String field ‘ray.rpc.WorkerTableData.exit_detail’ contains invalid UTF-8 data when parsing a protocol buffer. Use the ‘bytes’ type if you intend to send raw bytes.
e[36m(pid=gcs_server)e[0m E0000 00:00:1711958744.976708 1088 wire_format_lite.cc:626] String field ‘ray.rpc.WorkerTableData.exit_detail’ contains invalid UTF-8 data when parsing a protocol buffer. Use the ‘bytes’ type if you intend to send raw bytes.
e[36m(pid=gcs_server)e[0m E0000 00:00:1711958744.976858 1088 wire_format_lite.cc:626] String field ‘ray.rpc.WorkerTableData.exit_detail’ contains invalid UTF-8 data when parsing a protocol buffer. Use the ‘bytes’ type if you intend to send raw bytes.
e[36m(pid=gcs_server)e[0m E0000 00:00:1711958745.990428 1088 wire_format_lite.cc:626] String field ‘ray.rpc.WorkerTableData.exit_detail’ contains invalid UTF-8 data when parsing a protocol buffer. Use the ‘bytes’ type if you intend to send raw bytes.
e[36m(pid=gcs_server)e[0m E0000 00:00:1711958745.990640 1088 wire_format_lite.cc:626] String field ‘ray.rpc.WorkerTableData.exit_detail’ contains invalid UTF-8 data when parsing a protocol buffer. Use the ‘bytes’ type if you intend to send raw bytes.
e[36m(pid=gcs_server)e[0m E0000 00:00:1711958745.990788 1088 wire_format_lite.cc:626] String field ‘ray.rpc.WorkerTableData.exit_detail’ contains invalid UTF-8 data when parsing a protocol buffer. Use the ‘bytes’ type if you intend to send raw bytes.
e[36m(pid=gcs_server)e[0m E0000 00:00:1711958746.992214 1088 wire_format_lite.cc:626] String field ‘ray.rpc.WorkerTableData.exit_detail’ contains invalid UTF-8 data when parsing a protocol buffer. Use the ‘bytes’ type if you intend to send raw bytes.
e[36m(pid=gcs_server)e[0m E0000 00:00:1711958746.992465 1088 wire_format_lite.cc:626] String field ‘ray.rpc.WorkerTableData.exit_detail’ contains invalid UTF-8 data when parsing a protocol buffer. Use the ‘bytes’ type if you intend to send raw bytes.
e[36m(pid=gcs_server)e[0m E0000 00:00:1711958746.992606 1088 wire_format_lite.cc:626] String field ‘ray.rpc.WorkerTableData.exit_detail’ contains invalid UTF-8 data when parsing a protocol buffer. Use the ‘bytes’ type if you intend to send raw bytes.
e[36m(pid=gcs_server)e[0m E0000 00:00:1711958748.006436 1088 wire_format_lite.cc:626] String field ‘ray.rpc.WorkerTableData.exit_detail’ contains invalid UTF-8 data when parsing a protocol buffer. Use the ‘bytes’ type if you intend to send raw bytes.
e[36m(pid=gcs_server)e[0m E0000 00:00:1711958748.006539 1088 wire_format_lite.cc:626] String field ‘ray.rpc.WorkerTableData.exit_detail’ contains invalid UTF-8 data when parsing a protocol buffer. Use the ‘bytes’ type if you intend to send raw bytes.
e[36m(pid=gcs_server)e[0m E0000 00:00:1711958748.006779 1088 wire_format_lite.cc:626] String field ‘ray.rpc.WorkerTableData.exit_detail’ contains invalid UTF-8 data when parsing a protocol buffer. Use the ‘bytes’ type if you intend to send raw bytes.
e[36m(pid=gcs_server)e[0m E0000 00:00:1711958749.018213 1088 wire_format_lite.cc:626] String field ‘ray.rpc.WorkerTableData.exit_detail’ contains invalid UTF-8 data when parsing a protocol buffer. Use the ‘bytes’ type if you intend to send raw bytes.
e[36m(pid=gcs_server)e[0m E0000 00:00:1711958749.018364 1088 wire_format_lite.cc:626] String field ‘ray.rpc.WorkerTableData.exit_detail’ contains invalid UTF-8 data when parsing a protocol buffer. Use the ‘bytes’ type if you intend to send raw bytes.
e[36m(pid=gcs_server)e[0m E0000 00:00:1711958749.018567 1088 wire_format_lite.cc:626] String field ‘ray.rpc.WorkerTableData.exit_detail’ contains invalid UTF-8 data when parsing a protocol buffer. Use the ‘bytes’ type if you intend to send raw bytes.
e[36m(pid=gcs_server)e[0m E0000 00:00:1711958750.032062 1088 wire_format_lite.cc:626] String field ‘ray.rpc.WorkerTableData.exit_detail’ contains invalid UTF-8 data when parsing a protocol buffer. Use the ‘bytes’ type if you intend to send raw bytes.
e[36m(pid=gcs_server)e[0m E0000 00:00:1711958750.032132 1088 wire_format_lite.cc:626] String field ‘ray.rpc.WorkerTableData.exit_detail’ contains invalid UTF-8 data when parsing a protocol buffer. Use the ‘bytes’ type if you intend to send raw bytes.
e[36m(pid=gcs_server)e[0m E0000 00:00:1711958750.032271 1088 wire_format_lite.cc:626] String field ‘ray.rpc.WorkerTableData.exit_detail’ contains invalid UTF-8 data when parsing a protocol buffer. Use the ‘bytes’ type if you intend to send raw bytes.
And we found the following stack trace messge in log file of the actor
{“result”: false, “msg”: “Traceback (most recent call last):\n File "C:\Users\guxiaobo\AppData\Local\anaconda3\envs\python310\lib\site-packages\ray\dashboard\optional_utils.py", line 94, in _handler_route\n return await handler(bind_info.instance, req)\n File "C:\Users\guxiaobo\AppData\Local\anaconda3\envs\python310\lib\site-packages\ray\dashboard\modules\reporter\reporter_head.py", line 428, in get_traceback\n reply = await reporter_stub.GetTraceback(\n File "C:\Users\guxiaobo\AppData\Local\anaconda3\envs\python310\lib\site-packages\grpc\aio\_call.py", line 318, in await\n raise _create_rpc_error(\ngrpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNKNOWN\n\tdetails = "Unexpected <class ‘FileNotFoundError’>: [WinError 2] \u7cfb\u7edf\u627e\u4e0d\u5230\u6307\u5b9a\u7684\u6587\u4ef6\u3002"\n\tdebug_error_string = "UNKNOWN:Error received from peer {grpc_message:"Unexpected <class \‘FileNotFoundError\’>: [WinError 2] \xe7\xb3\xbb\xe7\xbb\x9f\xe6\x89\xbe\xe4\xb8\x8d\xe5\x88\xb0\xe6\x8c\x87\xe5\xae\x9a\xe7\x9a\x84\xe6\x96\x87\xe4\xbb\xb6\xe3\x80\x82", grpc_status:2, created_time:"2024-04-01T08:19:45.537912+00:00"}"\n>\n”, “data”: {}}