Specify the python and version of the dependencies for ray job

Ray Core
1

Hi, there seems to be a mistake in the documentation: Environment Dependencies — Ray 2.4.0

{"packages":["tensorflow", "requests"], "pip_check": False, "pip_version": "==22.0.2;python_version=='3.8.11'"}

With this command: ray job submit --address="http://127.0.0.1:8265" --runtime-env-json='{ "working_dir": "./", "excludes": [".joblib", "/.venv-py3.10/", "/data/", ".yaml"], "pip": {"packages":["tensorflow", "requests"], "pip_check": False, "pip_version": "==22.0.2;python_version=='3.8.11'"}, "py_modules": ["training_with_ray_tasks"]}' -- python training_with_ray_tasks/driver.py

I’m getting json.decoder.JSONDecodeError: Expecting value: line 1 column 146 (char 145)

What is the best practice for specifying my dependencies for a training job?

I noticed this limitation when I tried to specify scikit-learn version to 1.1.3 just by providing a requirements.txt cause the default ray image uses python3.7. ray/Dockerfile at master · ray-project/ray · GitHub

So I guess to use python3.10 I will need to specify the ray image to point to this one instead of the default 3.7 one? Installing Ray — Ray 2.4.0
However I’m not able to pull any of these images:
docker pull rayproject/ray:py310, the tags also can’t be loaded from docker hub: Docker

2

It turns out to be: docker pull docker.io/rayproject/ray:2.4.0-py310

3

@Y_C Thanks for posting your issue. Can you try small chunks of --runtime-env-json = ‘{…}’ to see where it fails.

cc: @architkulkarni Do you see anything I don’t?

4

This particular JSON error usually means the JSON parsing isn’t working. Can you try this suggestion from GPT, which switches out the types of quotes:

ray job submit --address="http://127.0.0.1:8265" --runtime-env-json="{ \"working_dir\": \"./\", \"excludes\": [\".joblib\", \"/.venv-py3.10/\", \"/data/\", \".yaml\"], \"pip\": {\"packages\":[\"tensorflow\", \"requests\"], \"pip_check\": False, \"pip_version\": \"==22.0.2;python_version=='3.8.11'\"}, \"py_modules\": [\"training_with_ray_tasks\"]}" -- python training_with_ray_tasks/driver.py