Ray start --head failing due to not valid Sentinel

Ray Core
1

Container running ray is failing (which used to work with no code changes)
Container logs show not valid Sentinel Error
Deleted tmp_dir, still same error…

1. Severity of the issue: (select one)
None: I’m just curious or want clarification.
Low: Annoying but doesn’t hinder my work.
Medium: Significantly affects my productivity but can find a workaround.
x High: Completely blocks me.

2. Environment:

  • Ray version:2.47.0 and 2.49.1
  • Python version:3.10.12
  • OS:linux/amd64
  • Cloud/Infrastructure:aws
  • Other libs/tools (if relevant):
  • psutil==6.1.0
    ray[client]==2.49.1
    pydantic==2.11.7
    pandas==2.2.3
    numpy==1.25.2

3. What happened vs. what you expected:

  • Expected: ray start working

  • Actual: Getting exceptions:

  • Traceback (most recent call last):

    File “/usr/local/bin/ray”, line 3, in 

    from ray.scripts.scripts import main

    File “/usr/local/lib/python3.10/site-packages/ray/scripts/scripts.py”, line 2739, in 

    add_command_alias(up, name="create_or_update", hidden=True)

    File “/usr/local/lib/python3.10/site-packages/ray/scripts/scripts.py”, line 2729, in add_command_alias

    new_command = copy.deepcopy(command)

    File “/usr/local/lib/python3.10/copy.py”, line 172, in deepcopy

    y = \_reconstruct(x, memo, \*rv)

    File “/usr/local/lib/python3.10/copy.py”, line 271, in _reconstruct

    state = deepcopy(state, memo)

    File “/usr/local/lib/python3.10/copy.py”, line 146, in deepcopy

    y = copier(x, memo)

    File “/usr/local/lib/python3.10/copy.py”, line 231, in _deepcopy_dict

    y\[deepcopy(key, memo)\] = deepcopy(value, memo)

    File “/usr/local/lib/python3.10/copy.py”, line 146, in deepcopy

    y = copier(x, memo)

    File “/usr/local/lib/python3.10/copy.py”, line 206, in _deepcopy_list

    append(deepcopy(a, memo))

    File “/usr/local/lib/python3.10/copy.py”, line 172, in deepcopy

    y = \_reconstruct(x, memo, \*rv)

    File “/usr/local/lib/python3.10/copy.py”, line 271, in _reconstruct

    state = deepcopy(state, memo)

    File “/usr/local/lib/python3.10/copy.py”, line 146, in deepcopy

    y = copier(x, memo)

    File “/usr/local/lib/python3.10/copy.py”, line 231, in _deepcopy_dict

    y\[deepcopy(key, memo)\] = deepcopy(value, memo)

    File “/usr/local/lib/python3.10/copy.py”, line 172, in deepcopy

    y = \_reconstruct(x, memo, \*rv)

    File “/usr/local/lib/python3.10/copy.py”, line 265, in _reconstruct

    y = func(\*args)

    File “/usr/local/lib/python3.10/enum.py”, line 385, in _call_

    return cls.\__new_\_(cls, value)

    File “/usr/local/lib/python3.10/enum.py”, line 710, in _new_

    raise ve_exc

    ValueError: <object object at 0x73e5e5ec71f0> is not a valid Sentinel

requirements.ray:
psutil==6.1.0
ray[client]==2.49.1
pydantic==2.11.7
pandas==2.2.3
numpy==1.25.2
DockerFile contents:
FROM --platform=$BUILDPLATFORM python:3.10.12
ARG BLUE_BUILD_CACHE_ARG
ARG BLUE_BUILD_LIB_ARG

Set workdir

WORKDIR /app

Install ping

RUN apt-get update
RUN apt-get install iputils-ping -y

Update pip

RUN pip install --upgrade pip
ADD src/requirements.ray /app/requirements.ray
RUN pip install -r requirements.ray

Environment variables

ENV dashboard_host ‘0.0.0.0’
ENV port ‘6380’
CMD [“sh”, “-c”, “ray start --head --verbose --dashboard-host ${dashboard_host} --port ${port} --block”]

3

Hi Eser, I’m hitting the exact same ValueError: ... is not a valid Sentinelerror with Ray 2.49.1 and Python 3.10.12. Did you ever find a solution or workaround?

4

same here, but it seems this problem is not related to ray version. Still trying to find out the cause.

5

Hi I found the cause. Package click required by ray caused this problem. Specifically, click==8.3.0 will cause this problem. Downgrading it to 8.2.1 solves this problem.

By the way, upgrading python to python3.11 will solve this problem either. This probably indicates click==8.3.0 is not compatible with python3.10. Ray team should set a limit on the version of click.

6

That is exactly what I did (click to version 8.2.1.) and solved the problem.

1 Like