How severe does this issue affect your experience of using Ray?
- High: It blocks me to complete my task.
Couter example is not working on my 2-node ray cluster. Below is my setup:
- 2 node ray cluster provisioned in AWS EKS, 1 head node and 1 worker node.
- on my macos, kubectl port-forward is configured at port 10001.
- Built example.so and library-api.so linux container and copy to both head node and worker node.
- Ran Counter example binary from my mac and below is the code.
ray::RayConfig config;
config.address = "127.0.0.1:10001";
config.code_search_path.push_back("/home/ray/lib/example.so");
ray::Init(config);
... ...
Error:
[2025-02-25 17:20:24,593 I 58323 10855964] gcs_client.cc:98: GcsClient has no Cluster ID set, and won't fetch from GCS.
[2025-02-25 17:20:24,596 C 58323 10855964] process_helper.cc:109: Check failed: status Status not OK: RpcError: RPC Error message: Method not found!; RPC Error details:
*** StackTrace Information ***
0 libray_api.so 0x0000000103a66138 _ZN3raylsERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEERKNS_10StackTraceE + 84 ray::operator<<()
1 libray_api.so 0x0000000103a69688 _ZN3ray6RayLogD2Ev + 84 ray::RayLog::~RayLog()
2 libray_api.so 0x0000000102ee09e4 _ZN3ray8internal13ProcessHelper8RayStartENSt3__18functionIFNS_6StatusERKNS_3rpc7AddressENS5_8TaskTypeENS2_12basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEERKNS_4core11RayFunctionERKNS2_13unordered_mapISF_dNS2_4hashISF_EENS2_8equal_toISF_EENSD_INS2_4pairIKSF_dEEEEEERKNS2_6vectorINS2_10shared_ptrINS_9RayObjectEEENSD_ISZ_EEEERKNSW_INS5_15ObjectReferenceENSD_IS14_EEEERSQ_S19_PNSW_INSP_INS_8ObjectIDESZ_EENSD_IS1B_EEEES1E_PNSW_INSP_IS1A_bEENSD_IS1F_EEEERNSX_INS_17LocalMemoryBufferEEEPbPSF_RKNSW_INS_16ConcurrencyGroupENSD_IS1O_EEEESF_bbbxEEE + 3668 ray::internal::ProcessHelper::RayStart()
3 libray_api.so 0x0000000102e9d99c _ZN3ray8internal18AbstractRayRuntime6DoInitEv + 120 ray::internal::AbstractRayRuntime::DoInit()
4 libray_api.so 0x0000000102e84f54 _ZN3ray4InitERNS_9RayConfigE + 60 ray::Init()
5 example 0x0000000100ba3f30 main + 152 main
6 dyld 0x0000000199d20274 start + 2840 start
Seems the error happened during Init and have not reached locating dynamic libs. Can I get any help here. Thanks!
BTW, I read this question C++ Examples not works well and still can’t make it working.