About Dashboard inside Docker

Dashboard
#1

Hi, I met a problem when running a ray program in docker and checking the dashboard in the host.

  1. I run a docker with docker run -p 9999:9999 option.
  2. I use ray.init(dashboard_port=9999,dashboard_host='0.0.0.0') in the docker and the console prints INFO services.py:1270 -- View the Ray dashboard at http://172.17.0.2:9999, which is expected.
  3. In the docker I check the dashboard is working:
root@03dc0ca63666:/home/game# lsof -i:9999
COMMAND PID USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
python3 192 root   27u  IPv4 165722973      0t0  TCP *:9999 (LISTEN)
  1. Now I want to check the dashboard in the host. If all works right, I could open localhost:9999 in the host. However, it failed, and on host I check whether the port forwarding succeed. lsof -i:9999 gives empty output.
    Besides, I use docker ps to confirm that:
$docker ps
fd54703aabb2   test "bash"    About a minute ago   Up About a minute    0.0.0.0:8265->8265/tcp, :::8265->8265/tcp

I am using:

pip install ray[default] -i https://pypi.tuna.tsinghua.edu.cn/simple    #(ray==1.8.0)
pip install 'aiohttp<3.8' -i https://pypi.tuna.tsinghua.edu.cn/simple

to avoid other issues.

Does anyone know what might go wrong here? Or how do you guys check the dashboard when running ray in the docker? Any comments are welcomed!

#2

Hi, guys. Please feel free free to leave any comments

#3

Are you binding the host:container ports so that port 9999 is exposed to the host? What does your docker run command look like or your docker compose?

#4

Hi Sir, thanks for the reply!
I run a docker with docker run -p 9999:9999 option. Also, docker ps in the host gives:

CONTAINER ID   IMAGE                   COMMAND   CREATED       STATUS       PORTS                                                                                                                          NAMES
a92c0414d070   hnb                     "bash"    4 days ago    Up 4 days    0.0.0.0:9999->9999/tcp, ::::9999->9999/tcp   hnb

I suppose that means I have successfully export the docker port to the corresponding host port. I wonder if I make something wrong!

#5

oddly enough I’m seeing something similar, even on the default port 8265 and if I try to access the ray dashboard, I get an ERR_CONNECTION_RESET in both docker and without docker

#6

That sounds not good. Hope guys from the Ray team could give some suggestions.

#7

@LoveRay I had to downgrade from aioredis 2.0.0 to aioredis@1.3.1 and then when I cleared by browser cache, I was able to get the dashboard to work again. YMMV, bu hope that helps or gives you something else to try

Also, I upgraded to ray 1.9

1 Like
#8

@puntime_error Thanks for the reply. I have confirmed that worked. The only strange thing is even if I could access host_ip:9999 for the dashboard, lsof -i:9999 gives nothing. But anyway, this is the workout.

#9

I encounter the same question. I construct a on-primise cluster with two physical servers. docker ray-head is run on server one, and docker ray-worker is run on server two. I build network connection by docker swarm. and I can get cluster information by command “ray status”. But I cannot visit Ray web UI when I enter http://:8265 on my laptop chrome
I make the port map too.
But if when I run container without “–network” configuation, I can see Ray web UI, in that case , my other server cannot join the Ray cluster.
Is there any help?
Thanks in advance

#10

Sorry Sir, I am not familiar with the networking part. Maybe cc @puntime_error has some suggestions?

#11

@LoveRay and @zhonglism (saw your other post), I FINALLY got the dashboard and the metrics working in my env. I had to manually install the deps listed in <python_install>/site-packages/ray/dashboard/optional_deps.py

I didn’t need to install all of them, but add to install aiohttp_cors, prometheus_client, opencensus after that I could see get to the dashboard…ymmv, but good luck

#12

Got it. Thanks Ser. Maybe Ray team should mention it in the docs.