Recommended way to submit ray job programmatically?

matthew.cox · December 16, 2020, 5:58pm

I’ve been using the ray submit command to launch new tune jobs, but I’m realizing that I want to adjust the arguments programmatically (eg use a configuration file to build up the arguments). In looking at https://github.com/ray-project/ray/blob/master/python/ray/scripts/scripts.py , it appears that the exec command is a pretty thin wrapper around exec_cluster, so I’m thinking of leveraging that. However, I notice that exec_cluster is located in ray.autoscaler._private.commands. Is the usage of that function discouraged and if so, is there another function in Ray that I should use instead to kick off my tuning job on a cluster?

rliaw · December 16, 2020, 6:57pm

Hey @matthew.cox, thanks a bunch for reaching out!

We actually have an interface that we’re currently working on - ray.autoscaler.sdk. It may have backwards incompatible changes. You’ll find a run_on_cluster command which is going to soon be more stable than _private.commands.

github.com

ray-project/ray/blob/fdb4c6eb1cfc40c0680f3b276394e8ba1d4c5c6d/python/ray/autoscaler/sdk.py

"""IMPORTANT: this is an experimental interface and not currently stable."""

from contextlib import contextmanager
from typing import Any, Callable, Dict, Iterator, List, Optional, Union
import json
import os
import tempfile

from ray.autoscaler._private import commands
from ray.autoscaler._private.event_system import (  # noqa: F401
    CreateClusterEvent,  # noqa: F401
    global_event_system)
from ray.autoscaler._private.cli_logger import cli_logger


def create_or_update_cluster(cluster_config: Union[dict, str],
                             *,
                             no_restart: bool = False,
                             restart_only: bool = False,
                             no_config_cache: bool = False) -> Dict[str, Any]:

This file has been truncated. show original

matthew.cox · December 16, 2020, 8:35pm

Thanks! I’ve give that a whirl, knowing that there will likely be breaking changes in the future. Would it be valuable to the development for me to provide feedback if I experience anything notable, or is it early enough stage that I should wait until the design has solidified first?

rliaw · December 16, 2020, 10:39pm

I think it would be valuable to the development to receive any feedback

Topic		Replies	Views
Submitting jobs programatically Ray Clusters	1	370	August 31, 2023
[raycluster, k8s] How to interact with ray cluster from inside a k8s pod Ray Clusters	4	735	June 17, 2022
Python API for creating Ray autoscaling cluster? Ray Clusters	1	393	September 24, 2021
Ray exec multiple scripts w/ tune.run() to same ray cluster Ray Tune	18	1381	February 14, 2021
Ray.init not work, but ray job submit is	3	81	July 29, 2024

Recommended way to submit ray job programmatically?

Related topics