How can I connect to pre-created redis instance?

Hi there,

This is my first time trying Ray (ray-1.5.0), . I`m trying to setup Ray with ray.init(address=…)
Here I have a pre-created Redis instance (well, it is Azure Redis Cache), and I give the azure redis address to address parameter.

And I see the following error from Ray:
RuntimeError: Could not read ‘session_name’ from GCS (redis). Has redis started correctly on the head node?

By looking the codes, I am sure the redis is connected.
May I know how I can connect Ray to existing redis?

Sorry, I forgot to attach the full stack trace:

2021-07-28 08:24:50,274 INFO – Connecting to existing Ray cluster at address:

RuntimeError Traceback (most recent call last)
----> 1 ray.init(address=“”, _redis_password=“password=”)

/opt/conda/envs/ACQAPython/lib/python3.6/site-packages/ray/_private/ in wrapper(*args, **kwargs)
80 if client_mode_should_convert():
81 return getattr(ray,*args, **kwargs)
—> 82 return func(*args, **kwargs)
84 return wrapper

/opt/conda/envs/ACQAPython/lib/python3.6/site-packages/ray/ in init(address, num_cpus, num_gpus, resources, object_store_memory, local_mode, ignore_reinit_error, include_dashboard, dashboard_host, dashboard_port, job_config, configure_logging, logging_level, logging_format, log_to_driver, namespace, runtime_env, internal_config, _enable_object_reconstruction, _redis_max_memory, _plasma_directory, _node_ip_address, _driver_object_store_memory, _memory, _redis_password, _temp_dir, _lru_evict, _metrics_export_port, _system_config, _tracing_startup_hook, **kwargs)
907 shutdown_at_exit=False,
908 spawn_reaper=False,
→ 909 connect_only=True)
911 if driver_mode == SCRIPT_MODE and job_config:

/opt/conda/envs/ACQAPython/lib/python3.6/site-packages/ray/ in init(self, ray_params, head, shutdown_at_exit, spawn_reaper, connect_only)
179 else:
180 redis_client = self.create_redis_client()
→ 181 session_name = _get_with_retry(redis_client, “session_name”)
182 self.session_name = ray._private.utils.decode(session_name)

/opt/conda/envs/ACQAPython/lib/python3.6/site-packages/ray/ in _get_with_retry(redis_client, key, num_retries)
46 time.sleep(2)
47 if not result:
—> 48 raise RuntimeError(f"Could not read ‘{key}’ from GCS (redis). "
49 “Has redis started correctly on the head node?”)
50 return result

RuntimeError: Could not read ‘session_name’ from GCS (redis). Has redis started correctly on the head node?

Hi @CraigMeng,

this seems to be a bug on our side - it seems that ray.init() does not initialize new ray cluster correctly.

The workaround here is to first start Ray via the CLI: ray start --head --address --redis-password "password". After that you can use ray.init() as you do in your script.

cc @Alex

I created an issue to track this: [core] External Redis does not work for new clusters · Issue #17400 · ray-project/ray · GitHub

Thank you, Kai.

I got following output by running ray start --head --address --redis-password "password"

`--head` starts a new Redis server, `--address` should not be set.

Sorry, I was wrong. I run the command in my windows machine, which only installed version 1.4.

On Linux machine, I got following errors:

/opt/conda/envs/ACQAPython/lib/python3.6/site-packages/ray/autoscaler/_private/ FutureWarning: Not all Ray CLI dependencies were found. In Ray 1.4+, the Ray CLI, autoscaler, and dashboard will only be usable via pip install 'ray[default]'. Please update your install command.
“update your install command.”, FutureWarning)
Will use value of --address as remote Redis server address(es). If the primary one is not reachable, we starts new one(s) with --port in local.
Local node IP:
2021-07-29 01:47:53,602 ERROR – Failed to start the dashboard: Failed to start the dashboard, return code 1. The last 10 lines of /tmp/ray/session_2021-07-29_01-47-51_520908_774/logs/dashboard.log:
2021-07-29 01:47:53,398 INFO – Setup static dir for dashboard: /opt/conda/envs/ACQAPython/lib/python3.6/site-packages/ray/new_dashboard/client/build
2021-07-29 01:47:53,432 ERROR – Connect to GCS failed: GCS address not found., retry…

2021-07-29 01:47:53,618 WARNING – WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 67108864 bytes available. This will harm performance! You may be able to free up space by deleting files in /dev/shm. If you are inside a Docker container, you can increase /dev/shm size by passing ‘–shm-size=2.40gb’ to ‘docker run’ (or add it to the run_options list in a Ray cluster config). Make sure to set this to more than 30% of available RAM.
[2021-07-29 01:47:53,795 C 774 774] Check failed: _s.ok() Bad status: RedisError:
*** StackTrace Information ***

It seems that connecting to Redis failed.

Connecting to pre-created Redis instance support was recently added, and not well tested. Btw, did you find any documentation about connecting external redis, or are you just trying to do this now?

@sangcho I didn`t find any documentation about connecting external redis. I thought Ray supports connecting external redis by nature.

I see. This wasn’t supported, but we recently received a contribution. Maybe the feature is not well tested