Job specific log option missing in dashboard

  • Low: It annoys or frustrates me for a moment.

I have upgrade ray version from 2.3.0 to 2.6.0 recently and found that there is no option in dashboard to see the job specific logs.

In ray 2.3.0 there was a option named “logs” at marked position in above image but it’s not there in new version. And It’s giving **Failed to Load** under Logs section.

There is an error in dashboard.log file:

2023-09-14 19:32:39,459	INFO state_head.py:421 -- Streaming logs with options: GetLogOptions(timeout=30, node_id='b677f4256cdb12b2dfbfbc055396f3aa859b4dd901d4ad65d4a08cc9', node_ip=None, media_type='file', filename='job-driver-raysubmit_XYcqmYJvTtnu4ui7.log', actor_id=None, task_id=None, attempt_number=0, pid=None, lines=50000, interval=None, suffix='out', submission_id=None)
2023-09-14 19:32:39,459	INFO log_manager.py:442 -- Resolved log file: node_id='b677f4256cdb12b2dfbfbc055396f3aa859b4dd901d4ad65d4a08cc9' filename='job-driver-raysubmit_XYcqmYJvTtnu4ui7.log' start_offset=None end_offset=None
2023-09-14 19:32:39,461	ERROR state_head.py:439 -- <AioRpcError of RPC that terminated with:
	status = StatusCode.UNKNOWN
	details = "Unexpected <class 'AttributeError'>: 'grpc._cython.cygrpc._ServicerContext' object has no attribute 'done'"
	debug_error_string = "{"created":"@1694700159.461056331","description":"Error received from peer ipv4:10.60.62.206:63358","file":"src/core/lib/surface/call.cc","file_line":1062,"grpc_message":"Unexpected <class 'AttributeError'>: 'grpc._cython.cygrpc._ServicerContext' object has no attribute 'done'","grpc_status":2}"
>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/ray/dashboard/modules/state/state_head.py", line 427, in get_logs
    async for logs_in_bytes in self._log_api.stream_logs(options):
  File "/usr/local/lib/python3.8/dist-packages/ray/dashboard/modules/log/log_manager.py", line 125, in stream_logs
    async for streamed_log in stream:
  File "/usr/local/lib/python3.8/dist-packages/grpc/aio/_call.py", line 321, in _fetch_stream_responses
    await self._raise_for_status()
  File "/usr/local/lib/python3.8/dist-packages/grpc/aio/_call.py", line 231, in _raise_for_status
    raise _create_rpc_error(await self.initial_metadata(), await
grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
	status = StatusCode.UNKNOWN
	details = "Unexpected <class 'AttributeError'>: 'grpc._cython.cygrpc._ServicerContext' object has no attribute 'done'"
	debug_error_string = "{"created":"@1694700159.461056331","description":"Error received from peer ipv4:10.60.62.206:63358","file":"src/core/lib/surface/call.cc","file_line":1062,"grpc_message":"Unexpected <class 'AttributeError'>: 'grpc._cython.cygrpc._ServicerContext' object has no attribute 'done'","grpc_status":2}"
>

What grpc version are you using?

It’s grpcio==1.34.1.

@cade Can you please guide on this?

this version is a bit old, but not sure if it’s the cause.

cc @Ruiyang_Wang , do you know what’s happening here?

@shyampatel how are you running your job?