Ray fails to start when building python only

Ray Core
1

Hello,

I follow the instructions to build python only code from here:

https://docs.ray.io/en/master/development.html#using-a-local-repository-for-dependencies

then I run : ray start --head on RHEL machine with version:

$ cat /etc/os-release
NAME=“Red Hat Enterprise Linux”
VERSION=“8.4 (Ootpa)”
ID=“rhel”
ID_LIKE=“fedora”
VERSION_ID=“8.4”
PLATFORM_ID=“platform:el8”
PRETTY_NAME=“Red Hat Enterprise Linux 8.4 (Ootpa)”
ANSI_COLOR=“0;31”
CPE_NAME=“cpe:/o:redhat:enterprise_linux:8.4:GA”
HOME_URL=“https://www.redhat.com/”
DOCUMENTATION_URL=“Product Documentation for Red Hat Enterprise Linux 8 - Red Hat Customer Portal”
BUG_REPORT_URL=“https://bugzilla.redhat.com/”

REDHAT_BUGZILLA_PRODUCT=“Red Hat Enterprise Linux 8”
REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
REDHAT_SUPPORT_PRODUCT=“Red Hat Enterprise Linux”
REDHAT_SUPPORT_PRODUCT_VERSION=“8.4”

I get the below error:

$ ray start --head
Local node IP: 10.208.109.88
2021-07-08 14:16:49,074 ERROR services.py:1324 – Failed to start the dashboard: Failed to start the dashboard, return code 1. The last 10 lines of /tmp/ray/session_2021-07-08_14-16-43_336081_3751274/logs/dashboard.log:
2021-07-08 14:16:44,062 INFO dashboard.py:92 – Setup static dir for dashboard: /home/asmalvan/anaconda3/envs/ray/lib/python3.7/site-packages/ray/new_dashboard/client/build
2021-07-08 14:16:44,065 ERROR head.py:87 – Connect to GCS failed: GCS address not found., retry…
2021-07-08 14:16:46,067 ERROR head.py:87 – Connect to GCS failed: GCS address not found., retry…
2021-07-08 14:16:48,070 ERROR head.py:87 – Connect to GCS failed: GCS address not found., retry…

[2021-07-08 14:16:54,022 C 3751274 3751274] service_based_gcs_client.cc:72: Check failed: GetGcsServerAddressFromRedis( redis_client_->GetPrimaryContext()->sync_context(), &current_gcs_server_address_, RayConfig::instance().gcs_service_connect_retries()) Failed to get gcs server address when init gcs client.
*** StackTrace Information ***
@ 0x7f1092df87f5 google::GetStackTraceToString()
@ 0x7f1092dc74be ray::GetCallTrace()
@ 0x7f1092dec434 ray::SpdLogMessage::Flush()
@ 0x7f1092dec5bd ray::RayLog::~RayLog()
@ 0x7f1092a42c87 ray::gcs::ServiceBasedGcsClient::Connect()
@ 0x7f109290c44f ray::gcs::GlobalStateAccessor::Connect()
@ 0x7f1092841b5b __pyx_pw_3ray_7_raylet_19GlobalStateAccessor_3connect()
@ 0x5645f0cc4d51 _PyMethodDef_RawFastCallKeywords
@ 0x5645f0cda9f3 _PyMethodDescr_FastCallKeywords
@ 0x5645f0cdb48e call_function
@ 0x5645f0cef11a _PyEval_EvalFrameDefault
@ 0x5645f0c906ad _PyEval_EvalCodeWithName
@ 0x5645f0cbeaf1 _PyFunction_FastCallKeywords
@ 0x5645f0cdb3a5 call_function
@ 0x5645f0cef11a _PyEval_EvalFrameDefault
@ 0x5645f0cbe758 _PyFunction_FastCallKeywords
@ 0x5645f0cdb3a5 call_function
@ 0x5645f0cef11a _PyEval_EvalFrameDefault
@ 0x5645f0cbe758 _PyFunction_FastCallKeywords
@ 0x5645f0cdb3a5 call_function
@ 0x5645f0cef11a _PyEval_EvalFrameDefault
@ 0x5645f0c906ad _PyEval_EvalCodeWithName
@ 0x5645f0cbeaf1 _PyFunction_FastCallKeywords
@ 0x5645f0cdb3a5 call_function
@ 0x5645f0cf26d5 _PyEval_EvalFrameDefault
@ 0x5645f0c906ad _PyEval_EvalCodeWithName
@ 0x5645f0c91771 _PyFunction_FastCallDict
@ 0x5645f0cdf37f slot_tp_init
@ 0x5645f0cdae47 _PyObject_FastCallKeywords
@ 0x5645f0cdb591 call_function
@ 0x5645f0cf0166 _PyEval_EvalFrameDefault
@ 0x5645f0c906ad _PyEval_EvalCodeWithName

can you please help?

2

If you try ray stop --force and ray start again, does this have the same issue?

3

@sangcho I think this was due to usage of incorrect python wheel version