About Dashboard inside Docker

Dashboard, Monitoring & Debugging
1

Hi, I met a problem when running a ray program in docker and checking the dashboard in the host.

  1. I run a docker with docker run -p 9999:9999 option.
  2. I use ray.init(dashboard_port=9999,dashboard_host='0.0.0.0') in the docker and the console prints INFO services.py:1270 -- View the Ray dashboard at http://172.17.0.2:9999, which is expected.
  3. In the docker I check the dashboard is working:
root@03dc0ca63666:/home/game# lsof -i:9999
COMMAND PID USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
python3 192 root   27u  IPv4 165722973      0t0  TCP *:9999 (LISTEN)
  1. Now I want to check the dashboard in the host. If all works right, I could open localhost:9999 in the host. However, it failed, and on host I check whether the port forwarding succeed. lsof -i:9999 gives empty output.
    Besides, I use docker ps to confirm that:
$docker ps
fd54703aabb2   test "bash"    About a minute ago   Up About a minute    0.0.0.0:8265->8265/tcp, :::8265->8265/tcp

I am using:

pip install ray[default] -i https://pypi.tuna.tsinghua.edu.cn/simple    #(ray==1.8.0)
pip install 'aiohttp<3.8' -i https://pypi.tuna.tsinghua.edu.cn/simple

to avoid other issues.

Does anyone know what might go wrong here? Or how do you guys check the dashboard when running ray in the docker? Any comments are welcomed!

2

Hi, guys. Please feel free free to leave any comments

3

Are you binding the host:container ports so that port 9999 is exposed to the host? What does your docker run command look like or your docker compose?

4

Hi Sir, thanks for the reply!
I run a docker with docker run -p 9999:9999 option. Also, docker ps in the host gives:

CONTAINER ID   IMAGE                   COMMAND   CREATED       STATUS       PORTS                                                                                                                          NAMES
a92c0414d070   hnb                     "bash"    4 days ago    Up 4 days    0.0.0.0:9999->9999/tcp, ::::9999->9999/tcp   hnb

I suppose that means I have successfully export the docker port to the corresponding host port. I wonder if I make something wrong!

5

oddly enough I’m seeing something similar, even on the default port 8265 and if I try to access the ray dashboard, I get an ERR_CONNECTION_RESET in both docker and without docker

6

That sounds not good. Hope guys from the Ray team could give some suggestions.

7

@LoveRay I had to downgrade from aioredis 2.0.0 to aioredis@1.3.1 and then when I cleared by browser cache, I was able to get the dashboard to work again. YMMV, bu hope that helps or gives you something else to try

Also, I upgraded to ray 1.9

1 Like
8

@puntime_error Thanks for the reply. I have confirmed that worked. The only strange thing is even if I could access host_ip:9999 for the dashboard, lsof -i:9999 gives nothing. But anyway, this is the workout.

9

I encounter the same question. I construct a on-primise cluster with two physical servers. docker ray-head is run on server one, and docker ray-worker is run on server two. I build network connection by docker swarm. and I can get cluster information by command “ray status”. But I cannot visit Ray web UI when I enter http://:8265 on my laptop chrome
I make the port map too.
But if when I run container without “–network” configuation, I can see Ray web UI, in that case , my other server cannot join the Ray cluster.
Is there any help?
Thanks in advance

10

Sorry Sir, I am not familiar with the networking part. Maybe cc @puntime_error has some suggestions?

11

@LoveRay and @zhonglism (saw your other post), I FINALLY got the dashboard and the metrics working in my env. I had to manually install the deps listed in <python_install>/site-packages/ray/dashboard/optional_deps.py

I didn’t need to install all of them, but add to install aiohttp_cors, prometheus_client, opencensus after that I could see get to the dashboard…ymmv, but good luck

12

Got it. Thanks Ser. Maybe Ray team should mention it in the docs.

13

In Ray 2.0, we made the previous experimental dashboard the new default dashboard. Are you still running into the same issue?

14

@Huaiwei_Sun I can confirm that the problem persists with ray 2.5.0: Exposing ports and mapping them outside of the docker results in “The connection was reset” in Firefox. The problem also persists when installing the optional dependencies for the dashboard. Interestingly enough, the problem is solved when using the --network host flag in docker run, but that is applicable only in very specific cases.

This answer from SO suggests that it might have to do with the HTTP server technology/isolation level, but I have nearly 0 knowledge on the subject.

15

I couldn’t find an answer to this so for anyone that stumbles upon this issue: adding --dashboard-host 0.0.0.0 to my ray start... command in my container solved the issue.

1 Like
16

@tipmethewink what Ray version were you on?

17

2.34.0

I wrote a comprehensive post with details of my compose file and log excepts but as I’m new here it disallowed it. I’ve cut and paste the bits it didn’t moan about l:

My compose entry for ray looks like this:

ray:
image: python:3.11
…
ports:

  • 8265:8265
  • 10001:10001
    command: ‘sh -c "ray stop && ray start --head --block --num-cpus=1 --resources=’‘{"trade_data": 64}’’ -v --dashboard-host 0.0.0.0"’

Without the --dashboard-host option it listens on localhost and accepts the connection but immediately drops it. With the option above, it binds to the container network interface and works as you’d expect.

1 Like
18

You need to pass this to ray start:
–dashboard-host 0.0.0.0
Reasons: