Hi there,
This is my first time trying Ray (ray-1.5.0), . I`m trying to setup Ray with ray.init(address=…)
Here I have a pre-created Redis instance (well, it is Azure Redis Cache), and I give the azure redis address to address parameter.
And I see the following error from Ray:
RuntimeError: Could not read ‘session_name’ from GCS (redis). Has redis started correctly on the head node?
By looking the codes, I am sure the redis is connected.
May I know how I can connect Ray to existing redis?
Sorry, I forgot to attach the full stack trace:
RuntimeError Traceback (most recent call last)
in
----> 1 ray.init(address=“10.216.188.132:6379”, _redis_password=“password=”)
/opt/conda/envs/ACQAPython/lib/python3.6/site-packages/ray/_private/client_mode_hook.py in wrapper(*args, **kwargs)
80 if client_mode_should_convert():
81 return getattr(ray, func.name)(*args, **kwargs)
—> 82 return func(*args, **kwargs)
83
84 return wrapper
/opt/conda/envs/ACQAPython/lib/python3.6/site-packages/ray/worker.py in init(address, num_cpus, num_gpus, resources, object_store_memory, local_mode, ignore_reinit_error, include_dashboard, dashboard_host, dashboard_port, job_config, configure_logging, logging_level, logging_format, log_to_driver, namespace, runtime_env, internal_config, _enable_object_reconstruction, _redis_max_memory, _plasma_directory, _node_ip_address, _driver_object_store_memory, _memory, _redis_password, _temp_dir, _lru_evict, _metrics_export_port, _system_config, _tracing_startup_hook, **kwargs)
907 shutdown_at_exit=False,
908 spawn_reaper=False,
→ 909 connect_only=True)
910
911 if driver_mode == SCRIPT_MODE and job_config:
/opt/conda/envs/ACQAPython/lib/python3.6/site-packages/ray/node.py in init(self, ray_params, head, shutdown_at_exit, spawn_reaper, connect_only)
179 else:
180 redis_client = self.create_redis_client()
→ 181 session_name = _get_with_retry(redis_client, “session_name”)
182 self.session_name = ray._private.utils.decode(session_name)
183
/opt/conda/envs/ACQAPython/lib/python3.6/site-packages/ray/node.py in _get_with_retry(redis_client, key, num_retries)
46 time.sleep(2)
47 if not result:
—> 48 raise RuntimeError(f"Could not read ‘{key}’ from GCS (redis). "
49 “Has redis started correctly on the head node?”)
50 return result
RuntimeError: Could not read ‘session_name’ from GCS (redis). Has redis started correctly on the head node?
Hi @CraigMeng,
this seems to be a bug on our side - it seems that ray.init() does not initialize new ray cluster correctly.
The workaround here is to first start Ray via the CLI: ray start --head --address 10.216.188.132:6379 --redis-password "password". After that you can use ray.init() as you do in your script.
cc @Alex
I created an issue to track this: [core] External Redis does not work for new clusters · Issue #17400 · ray-project/ray · GitHub
Thank you, Kai.
I got following output by running
ray start --head --address 10.216.188.132:6379 --redis-password "password"`--head` starts a new Redis server, `--address` should not be set.
Sorry, I was wrong. I run the command in my windows machine, which only installed version 1.4.
On Linux machine, I got following errors：
/opt/conda/envs/ACQAPython/lib/python3.6/site-packages/ray/autoscaler/_private/cli_logger.py:61: FutureWarning: Not all Ray CLI dependencies were found. In Ray 1.4+, the Ray CLI, autoscaler, and dashboard will only be usable via
pip install 'ray[default]'. Please update your install command.
“update your install command.”, FutureWarning)
Will use value of--addressas remote Redis server address(es). If the primary one is not reachable, we starts new one(s) with--portin local.
Local node IP: 10.240.16.215
2021-07-29 01:47:53,602 ERROR services.py:1254 – Failed to start the dashboard: Failed to start the dashboard, return code 1. The last 10 lines of /tmp/ray/session_2021-07-29_01-47-51_520908_774/logs/dashboard.log:
2021-07-29 01:47:53,398 INFO dashboard.py:92 – Setup static dir for dashboard: /opt/conda/envs/ACQAPython/lib/python3.6/site-packages/ray/new_dashboard/client/build
2021-07-29 01:47:53,432 ERROR head.py:87 – Connect to GCS failed: GCS address not found., retry…2021-07-29 01:47:53,618 WARNING services.py:1716 – WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 67108864 bytes available. This will harm performance! You may be able to free up space by deleting files in /dev/shm. If you are inside a Docker container, you can increase /dev/shm size by passing ‘–shm-size=2.40gb’ to ‘docker run’ (or add it to the run_options list in a Ray cluster config). Make sure to set this to more than 30% of available RAM.
[2021-07-29 01:47:53,795 C 774 774] redis_client.cc:126: Check failed: _s.ok() Bad status: RedisError:
*** StackTrace Information ***
ray::SpdLogMessage::Flush()
ray::RayLog::~RayLog()
ray::gcs::RedisClient::Connect()
ray::gcs::RedisClient::Connect()
ray::gcs::ServiceBasedGcsClient::Connect()
ray::gcs::GlobalStateAccessor::Connect()
__pyx_pw_3ray_7_raylet_19GlobalStateAccessor_3connect()
_PyCFunction_FastCallDict
It seems that connecting to Redis failed.
Connecting to pre-created Redis instance support was recently added, and not well tested. Btw, did you find any documentation about connecting external redis, or are you just trying to do this now?
@sangcho I didn`t find any documentation about connecting external redis. I thought Ray supports connecting external redis by nature.
I see. This wasn’t supported, but we recently received a contribution. Maybe the feature is not well tested