Failed to get the system config from Raylet: IOError: 14: failed to connect to all addresses

crystalww · July 7, 2021, 2:31pm

docker image: rayproject/ray:1.4.1

# cat /tmp/ray/session_latest/logs/python-core-driver-01000000ffffffffffffffffffffffffffffffffffffffffffffffff_638.log 
[2021-07-07 07:26:59,166 I 638 638] core_worker.cc:139: Constructing CoreWorkerProcess. pid: 638
[2021-07-07 07:27:08,170 C 638 647] core_worker.cc:223: Failed to get the system config from Raylet: IOError: 14: failed to connect to all addresses
[2021-07-07 07:27:08,170 E 638 647] logging.cc:441: *** Aborted at 1625668028 (unix time) try "date -d @1625668028" if you are using GNU date ***
[2021-07-07 07:27:08,174 E 638 647] logging.cc:441: PC: @                0x0 (unknown)
[2021-07-07 07:27:08,174 E 638 647] logging.cc:441: *** SIGABRT (@0x27e) received by PID 638 (TID 0x7fba63fff700) from PID 638; stack trace: ***
[2021-07-07 07:27:08,177 E 638 647] logging.cc:441:     @     0x7fba832d6d8f google::(anonymous namespace)::FailureSignalHandler()
[2021-07-07 07:27:08,180 E 638 647] logging.cc:441:     @     0x7fba843323c0 (unknown)
[2021-07-07 07:27:08,182 E 638 647] logging.cc:441:     @     0x7fba8417118b gsignal
[2021-07-07 07:27:08,184 E 638 647] logging.cc:441:     @     0x7fba84150859 abort
[2021-07-07 07:27:08,186 E 638 647] logging.cc:441:     @     0x7fba832c64ce ray::SpdLogMessage::Flush()
[2021-07-07 07:27:08,188 E 638 647] logging.cc:441:     @     0x7fba832c659d ray::RayLog::~RayLog()
[2021-07-07 07:27:08,189 E 638 647] logging.cc:441:     @     0x7fba82e93599 _ZNSt17_Function_handlerIFvRKN3ray6StatusERKNS0_3rpc20GetSystemConfigReplyEEZZZNS0_17CoreWorkerProcess22InitializeSystemConfigEvENKUlvE_clEvENKUllE_clElEUlS3_S7_E_E9_M_invokeERKSt9_Any_dataS3_S7_
[2021-07-07 07:27:08,191 E 638 647] logging.cc:441:     @     0x7fba82f23aef ray::rpc::ClientCallImpl<>::OnReplyReceived()
[2021-07-07 07:27:08,192 E 638 647] logging.cc:441:     @     0x7fba82e2299b _ZNSt17_Function_handlerIFvvEZN3ray3rpc17ClientCallManager29PollEventsFromCompletionQueueEiEUlvE_E9_M_invokeERKSt9_Any_data
[2021-07-07 07:27:08,194 E 638 647] logging.cc:441:     @     0x7fba83272238 boost::asio::detail::completion_handler<>::do_complete()
[2021-07-07 07:27:08,196 E 638 647] logging.cc:441:     @     0x7fba8336fbf1 boost::asio::detail::scheduler::do_run_one()
[2021-07-07 07:27:08,199 E 638 647] logging.cc:441:     @     0x7fba8336fd21 boost::asio::detail::scheduler::run()
[2021-07-07 07:27:08,201 E 638 647] logging.cc:441:     @     0x7fba83371820 boost::asio::io_context::run()
[2021-07-07 07:27:08,202 E 638 647] logging.cc:441:     @     0x7fba82e52d84 _ZZN3ray17CoreWorkerProcess22InitializeSystemConfigEvENKUlvE_clEv.isra.0
[2021-07-07 07:27:08,203 E 638 647] logging.cc:441:     @     0x7fba83610a40 execute_native_thread_routine
[2021-07-07 07:27:08,205 E 638 647] logging.cc:441:     @     0x7fba84326609 start_thread
[2021-07-07 07:27:08,206 E 638 647] logging.cc:441:     @     0x7fba8424d293 clone

github.com/ray-project/ray

[core] Failed to get the system config from Raylet: IOError: 14: failed to connect to all addresses

opened 09:19AM - 23 May 21 UTC

closed 07:46AM - 27 May 21 UTC

auderson

bug triage

Hi, I'm using ray 1.3.0 and I can't connect to a remote ray server. However i…f I downgrade ray version to 1.2.0, it will work fine. ``` [2021-05-23 17:06:12,083 I 541046 541046] core_worker.cc:136: Constructing CoreWorkerProcess. pid: 541046 [2021-05-23 17:06:21,087 C 541046 541153] core_worker.cc:219: Failed to get the system config from Raylet: IOError: 14: failed to connect to all addresses [2021-05-23 17:06:21,087 E 541046 541153] logging.cc:435: *** Aborted at 1621760781 (unix time) try "date -d @1621760781" if you are using GNU date *** [2021-05-23 17:06:21,090 E 541046 541153] logging.cc:435: PC: @ 0x0 (unknown) [2021-05-23 17:06:21,091 E 541046 541153] logging.cc:435: *** SIGABRT (@0x3f000084176) received by PID 541046 (TID 0x7fd73375b700) from PID 541046; stack trace: *** [2021-05-23 17:06:21,092 E 541046 541153] logging.cc:435: @ 0x7fd7660be20f google::(anonymous namespace)::FailureSignalHandler() [2021-05-23 17:06:21,095 E 541046 541153] logging.cc:435: @ 0x7fd7a5ced980 (unknown) [2021-05-23 17:06:21,098 E 541046 541153] logging.cc:435: @ 0x7fd7a5928fb7 gsignal [2021-05-23 17:06:21,101 E 541046 541153] logging.cc:435: @ 0x7fd7a592a921 abort [2021-05-23 17:06:21,102 E 541046 541153] logging.cc:435: @ 0x7fd7660adb05 ray::SpdLogMessage::Flush() [2021-05-23 17:06:21,103 E 541046 541153] logging.cc:435: @ 0x7fd7660adb3d ray::RayLog::~RayLog() [2021-05-23 17:06:21,104 E 541046 541153] logging.cc:435: @ 0x7fd765c89f89 _ZNSt17_Function_handlerIFvRKN3ray6StatusERKNS0_3rpc20GetSystemConfigReplyEEZZZNS0_17CoreWorkerProcess22InitializeSystemConfigEvENKUlvE_clEvENKUllE_clElEUlS3_S7_E_E9_M_invokeERKSt9_Any_dataS3_S7_ [2021-05-23 17:06:21,105 E 541046 541153] logging.cc:435: @ 0x7fd765d1540f ray::rpc::ClientCallImpl<>::OnReplyReceived() [2021-05-23 17:06:21,106 E 541046 541153] logging.cc:435: @ 0x7fd765c1facb _ZNSt17_Function_handlerIFvvEZN3ray3rpc17ClientCallManager29PollEventsFromCompletionQueueEiEUlvE_E9_M_invokeERKSt9_Any_data [2021-05-23 17:06:21,107 E 541046 541153] logging.cc:435: @ 0x7fd76605ca08 boost::asio::detail::completion_handler<>::do_complete() [2021-05-23 17:06:21,108 E 541046 541153] logging.cc:435: @ 0x7fd7661629a1 boost::asio::detail::scheduler::do_run_one() [2021-05-23 17:06:21,109 E 541046 541153] logging.cc:435: @ 0x7fd766162ad1 boost::asio::detail::scheduler::run() [2021-05-23 17:06:21,110 E 541046 541153] logging.cc:435: @ 0x7fd7661645d0 boost::asio::io_context::run() [2021-05-23 17:06:21,111 E 541046 541153] logging.cc:435: @ 0x7fd765c4e1b4 _ZZN3ray17CoreWorkerProcess22InitializeSystemConfigEvENKUlvE_clEv.isra.0 [2021-05-23 17:06:21,112 E 541046 541153] logging.cc:435: @ 0x7fd766407d10 execute_native_thread_routine [2021-05-23 17:06:21,113 E 541046 541153] logging.cc:435: @ 0x7fd7a5ce26db start_thread [2021-05-23 17:06:21,114 E 541046 541153] logging.cc:435: @ 0x7fd7a5a0b71f clone ``` on another machine which has already started the ray service and connected to the head node ```python import ray ray.init(address='auto', _redis_password='5241590000000000') ``` ``` 2021-05-23 17:06:12,070 INFO worker.py:640 -- Connecting to existing Ray cluster at address: ****** Process finished with exit code 134 ```

That not works for me.

sangcho · July 28, 2021, 6:38pm

Did you just run the container and call ray start there? CAn you explain the setup for me?

Topic		Replies	Views
How to use ray for multi-node training? Ray Clusters	1	1038	November 22, 2022
Ray init failed, but ray status success Ray Core	4	576	March 16, 2024
Ray Init causes error in the docker container due to non-mapped port Ray Core	6	1337	August 3, 2021
Error connecting Ray Cluster (Docker) using Java Ray Client	4	497	March 19, 2023
Failed to lease worker from node Ray Core	9	559	February 23, 2022

Failed to get the system config from Raylet: IOError: 14: failed to connect to all addresses

Related Topics