The error messages are automatically propagated to the CLI on the nightly wheels as well, in case it speeds up debugging:
❯ ray job submit --runtime-env-json='{"pip": ["doesnotexist"]}' -- echo hello
Job submission server address: http://127.0.0.1:8265
-------------------------------------------------------
Job 'raysubmit_SVzZdUCRfFcaaG8y' submitted successfully
-------------------------------------------------------
Next steps
Query the logs of the job:
ray job logs raysubmit_SVzZdUCRfFcaaG8y
Query the status of the job:
ray job status raysubmit_SVzZdUCRfFcaaG8y
Request the job to be stopped:
ray job stop raysubmit_SVzZdUCRfFcaaG8y
Tailing logs until the job exits (disable with --no-wait):
---------------------------------------
Job 'raysubmit_SVzZdUCRfFcaaG8y' failed
---------------------------------------
Status message: runtime_env setup failed: Failed to setup runtime environment.
Could not create the actor because its associated runtime env failed to be created.
Traceback (most recent call last):
File "/Users/archit/ray/python/ray/dashboard/modules/runtime_env/runtime_env_agent.py", line 260, in CreateRuntimeEnv
runtime_env_context = await _setup_runtime_env(
File "/Users/archit/ray/python/ray/dashboard/modules/runtime_env/runtime_env_agent.py", line 149, in _setup_runtime_env
size_bytes = await manager.create(
File "/Users/archit/ray/python/ray/_private/runtime_env/pip.py", line 423, in create
return await task
File "/Users/archit/ray/python/ray/_private/runtime_env/pip.py", line 414, in _create_for_hash
await PipProcessor(target_dir, runtime_env, logger)
File "/Users/archit/ray/python/ray/_private/runtime_env/pip.py", line 326, in _run
await self._install_pip_packages(
File "/Users/archit/ray/python/ray/_private/runtime_env/pip.py", line 302, in _install_pip_packages
await check_output_cmd(pip_install_cmd, logger=logger, cwd=cwd, env=pip_env)
File "/Users/archit/ray/python/ray/_private/runtime_env/utils.py", line 101, in check_output_cmd
raise SubprocessCalledProcessError(
ray._private.runtime_env.utils.SubprocessCalledProcessError: Run cmd[18] failed with the following details.
Command '['/tmp/ray/session_2022-03-17_15-22-36_124070_59807/runtime_resources/pip/8df9437b29c3f273fa6587bdffec5e399e705087/virtualenv/bin/python', '-m', 'pip', 'install', '--disable-pip-version-check', '--no-cache-dir', '-r', '/tmp/ray/session_2022-03-17_15-22-36_124070_59807/runtime_resources/pip/8df9437b29c3f273fa6587bdffec5e399e705087/requirements.txt']' returned non-zero exit status 1.
Last 50 lines of stdout:
ERROR: Could not find a version that satisfies the requirement doesnotexist (from versions: none)
ERROR: No matching distribution found for doesnotexist
Or
❯ ray job submit --runtime-env-json='{"working_dir": "doesnotexist"}' -- echo hello
Job submission server address: http://127.0.0.1:8265
Traceback (most recent call last):
File "/Users/archit/ray/python/ray/_private/runtime_env/working_dir.py", line 61, in upload_working_dir_if_needed
working_dir_uri = get_uri_for_directory(working_dir, excludes=excludes)
File "/Users/archit/ray/python/ray/_private/runtime_env/packaging.py", line 359, in get_uri_for_directory
raise ValueError(f"directory {directory} must be an existing" " directory")
ValueError: directory /Users/archit/ray/doesnotexist must be an existing directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/archit/anaconda3/envs/ray-py38/bin/ray", line 33, in <module>
sys.exit(load_entry_point('ray', 'console_scripts', 'ray')())
File "/Users/archit/ray/python/ray/scripts/scripts.py", line 2264, in main
return cli()
File "/Users/archit/anaconda3/envs/ray-py38/lib/python3.8/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/Users/archit/anaconda3/envs/ray-py38/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/Users/archit/anaconda3/envs/ray-py38/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/archit/anaconda3/envs/ray-py38/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/archit/anaconda3/envs/ray-py38/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/archit/anaconda3/envs/ray-py38/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/Users/archit/ray/python/ray/autoscaler/_private/cli_logger.py", line 808, in wrapper
return f(*args, **kwargs)
File "/Users/archit/ray/python/ray/dashboard/modules/job/cli.py", line 152, in submit
job_id = client.submit_job(
File "/Users/archit/ray/python/ray/dashboard/modules/job/sdk.py", line 71, in submit_job
self._upload_working_dir_if_needed(runtime_env)
File "/Users/archit/ray/python/ray/dashboard/modules/dashboard_sdk.py", line 324, in _upload_working_dir_if_needed
upload_working_dir_if_needed(runtime_env, upload_fn=_upload_fn)
File "/Users/archit/ray/python/ray/_private/runtime_env/working_dir.py", line 65, in upload_working_dir_if_needed
raise ValueError(
ValueError: directory doesnotexist must be an existing directory or a zip package