Ray Flow Insight Usage Problem

Dashboard, Monitoring & Debugging
1

1. Severity of the issue: (select one)
None: I’m just curious or want clarification.
Low: Annoying but doesn’t hinder my work.
Medium: Significantly affects my productivity but can find a workaround.
High: Completely blocks me.

2. Environment:

  • Ray version: 2.44
  • Python version: 3.10
  • OS: Ubuntu
  • Cloud/Infrastructure:
  • Other libs/tools (if relevant): ant-ray 2.44.1.1

3. Repro steps / sample code: (optional, but helps a lot!)

4. What happened vs. what you expected:

  • Expected: I would like to run Ray Flow Insight ( Ray Flow Insight：让分布式系统调试不再"黑盒" (qq.com), a visualization tool developed by AntGroup 2 weeks ago) but had some issues running it. It would be really helpful if someone has successfully run it and provide some guidance.
  • Actual:
    I had both ray and ant-ray installed on my PC. And I followed their instructions by
    $ export PYTHONPATH=/usr/local/lib/python3.10/dist-packages/ant-ray:$PYTHONPATH
    $ export RAY_FLOW_INSIGHT=1
    $ ray start --head my_host_ip
    The ant ray started as normal, with grafana/prometheus on the left upper corner. However, I didnt see the RAY FLOW INSIGHT tool that I would like to see. I would really appreciate it if anyone figures out any misleading steps in my command lines and tells me the correct ones. Thank you very much in advance!
2

Hi, actually this isn’t developed by the Ray team but rather someone from AntGroup (@jovany-wang maybe?)

3

Yes, thank you. I edited the post and changed that to AntRay team.
Sorry for the confusion it may cause.

4

I tried it yesterday and it worked for me. The entrypoint is in job list page. One of the actions is insight

You can try uninstalling ray and only keep anti-ray as well

5

Could you help me check if the following command lines are incorrect to run ray flow insight? I am using Ubuntu 22 with the following directory for ant-ray installation. Thank you very much for help!
$ export PYTHONPATH=/usr/local/lib/python3.10/dist-packages/ant_ray-2.44.1.1.dist-info/
$ export RAY_FLOW_INSIGHT=1
$ ray start --head --dashboard-host=0.0.0.0 --dashboard-port=8265
Not sure if the installation directory I entered was wrong or not, I got the above directory by running:
$ pip show ant-ray
It returns version 2.44.1.1 and the above location. Thank you.

6

I don’t know how ant-ray works exactly, so I cannot really help check.

However, the dockerfile below works for me locally. You can build an image and start the container locally and forward the port. It should work.

I modified this anyscale image example: Image requirements | Anyscale Docs

# syntax=docker/dockerfile:1.3-labs

# Specify the Ray Version and Python Version to use. You can override this setting at build time.
# Note that the ubuntu:22.04 base image is always Python 3.10.
# You are responsible for installing other versions of Python based on your application requirements.
ARG RAY_VERSION=2.44.1
ARG PYTHON_MAJOR_VERSION=3
ARG PYTHON_MINOR_VERSION=10

# Define the Anyscale image as a named build stage
ARG ANYSCALE_RAY_IMAGE=anyscale/ray:$RAY_VERSION-py$PYTHON_MAJOR_VERSION$PYTHON_MINOR_VERSION
FROM $ANYSCALE_RAY_IMAGE as anyscale-ray

# Base image
FROM ubuntu:22.04 as base

# Redeclare the RAY_VERSION and PYTHON_VERSION variables to make it available in the base image
ARG RAY_VERSION
ARG PYTHON_MAJOR_VERSION
ARG PYTHON_MINOR_VERSION

ENV DEBIAN_FRONTEND=noninteractive

# Install basic dependencies and setup `ray` user with sudoer permissions.
# Note that `ray` user should be (uid: 1000, gid: 100) to work with shared file
# systems.
# Add gdb since Ray dashboard uses `memray attach`, which requires gdb.
RUN <<EOF
#!/bin/bash
set -euxo pipefail

apt-get update -y
apt-get install -y --no-install-recommends sudo tzdata openssh-client openssh-server rsync zip unzip git gdb supervisor curl
# Install Python -- you can replace this with whatever Python installation method
# you want (i.e. conda, etc...), as long as `python` is on PATH. At runtime
# we'll source `/home/ray/.bashrc` in case you modify PATH there. This example uses
# virtualenv
apt-get install -y python3-venv

apt-get clean
rm -rf /var/lib/apt/lists/*

# Work around for https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/45234
mkdir -p /var/run/sshd

useradd -ms /bin/bash -d /home/ray ray --uid 1000 --gid 100
usermod -aG sudo ray
echo 'ray ALL=NOPASSWD: ALL' >> /etc/sudoers
EOF

# Switch to `ray` user
USER ray
ENV HOME=/home/ray
ENV PATH=/home/ray/virtualenv/bin:$PATH

RUN <<EOF
#!/bin/bash
# Run as user `ray` from here.
su --login ray
python3 -m venv --system-site-packages /home/ray/virtualenv
export PATH=/home/ray/virtualenv/bin:$PATH

# You only need jupyterlab if you want to access Jupyter notebooks from the Anyscale workspace web UI.
# Note that this only installs `ray[default]` to minimize the amount of dependencies.
# You can add extra libraries such as Ray Tune with `ray[default,tune]` or all of them with `ray[all]`.
# See the Ray docs for more info: https://docs.ray.io/en/latest/ray-overview/installation.html
pip install --no-cache-dir anyscale jupyterlab
pip install "ray[default]"
pip uninstall "ray[default]"
pip install ant-ray==2.44.1.1

# If you want to run your cluster on Amazon Web Services, uncomment out the following lines.
# curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
# unzip awscliv2.zip && sudo ./aws/install && rm awscliv2.zip

# If you want to run your cluster on Google Cloud Platform, uncomment the following line.
# pip install --no-cache-dir google google-cloud-storage

# Start of workspace dependencies: this section is only needed if you want your image to run on workspaces.

# This flushes bash history after each command, so that workspaces can persist it.
echo 'PROMPT_COMMAND="history -a"' >> /home/ray/.bashrc

# If the workspacerc exists, load it.
echo '[ -e ~/.workspacerc ] && source ~/.workspacerc' >> /home/ray/.bashrc

EOF


ENV RAY_FLOW_INSIGHT=1


RUN echo "Testing Ray Import..." && python -c "import ray"