Recommended way to submit ray job programmatically?

matthew.cox · December 16, 2020, 5:58pm

I’ve been using the ray submit command to launch new tune jobs, but I’m realizing that I want to adjust the arguments programmatically (eg use a configuration file to build up the arguments). In looking at https://github.com/ray-project/ray/blob/master/python/ray/scripts/scripts.py , it appears that the exec command is a pretty thin wrapper around exec_cluster, so I’m thinking of leveraging that. However, I notice that exec_cluster is located in ray.autoscaler._private.commands. Is the usage of that function discouraged and if so, is there another function in Ray that I should use instead to kick off my tuning job on a cluster?

rliaw · December 16, 2020, 6:57pm

Hey @matthew.cox, thanks a bunch for reaching out!

We actually have an interface that we’re currently working on - ray.autoscaler.sdk. It may have backwards incompatible changes. You’ll find a run_on_cluster command which is going to soon be more stable than _private.commands.

github.com

ray-project/ray/blob/fdb4c6eb1cfc40c0680f3b276394e8ba1d4c5c6d/python/ray/autoscaler/sdk.py

"""IMPORTANT: this is an experimental interface and not currently stable."""

from contextlib import contextmanager
from typing import Any, Callable, Dict, Iterator, List, Optional, Union
import json
import os
import tempfile

from ray.autoscaler._private import commands
from ray.autoscaler._private.event_system import (  # noqa: F401
    CreateClusterEvent,  # noqa: F401
    global_event_system)
from ray.autoscaler._private.cli_logger import cli_logger


def create_or_update_cluster(cluster_config: Union[dict, str],
                             *,
                             no_restart: bool = False,
                             restart_only: bool = False,
                             no_config_cache: bool = False) -> Dict[str, Any]:

This file has been truncated. show original

matthew.cox · December 16, 2020, 8:35pm

Thanks! I’ve give that a whirl, knowing that there will likely be breaking changes in the future. Would it be valuable to the development for me to provide feedback if I experience anything notable, or is it early enough stage that I should wait until the design has solidified first?

rliaw · December 16, 2020, 10:39pm

I think it would be valuable to the development to receive any feedback

Topic		Replies	Views
Submitting jobs programatically Ray Clusters	1	260	August 31, 2023
Python API for creating Ray autoscaling cluster? Ray Clusters	1	357	September 24, 2021
How to submit a job to a local_mode cluster	3	453	February 27, 2021
Can I use ray autoscaler to control a manually launched ray cluster Kubernetes	3	507	July 15, 2021
Programmatic creation of ray cluster Ray Clusters	3	336	September 16, 2021

Recommended way to submit ray job programmatically?

Related Topics