Set Job ID (or Submission ID?) to group tasks under separate Jobs

1

I have a web server that runs ray.init() during the initialization. The web server will sit and wait for users to submit a request, and our server will launch a ray workflow for each request.

When I look at the ray dashboard, I see only a single job ID created when the web server first runs ray.init(). All of the jobs from different requests (from different users) are placed under the same Job ID.

Instead, I’d like to organize each request(workflow) under separate Job IDs. I see that there is an API for RuntimeContext.get_job_id but not set_job_id. Is there a way to instruct ray to place tasks under a different JobID for each request that our web server receives?

The only way that I can think of is to create a new process to handle each request and call ray.init() from the each sub-process. I’d like to avoid this approach as spawning a new process and initializing our application slows down our processing.

2

Unfortunately, it is not possible from the core level as the job == the script that runs ray.init() (from the dashboard).

cc @yic do we have any request similar to this and have a solution for it?

3

Would Ray namespaces help?
Using Namespaces — Ray 2.2.0

4

I don’t think so. I feel namespace potentially should be the one to work with.
Basically, you can have actors/jobs in different ns. And later when you send traffic, pick the right ns to send the traffic.

5

@zoe_tsekas @yic Thank you for suggesting ray namespace. I tried to use it, but it turned out that I can only use it for Ray actor - not Ray remote.

When I tried to set namespace for Ray remote, I get the following error message.

ValueError: Invalid option keyword namespace for remote functions. Valid ones are [‘accelerator_type’, ‘memory’, ‘name’, ‘num_cpus’, ‘num_gpus’, ‘object_store_memory’, ‘placement_group’, ‘placement_group_bundle_index’, ‘placement_group_capture_child_tasks’, ‘resources’, ‘runtime_env’, ‘scheduling_strategy’, ‘_metadata’, ‘max_calls’, ‘max_retries’, ‘num_returns’, ‘retry_exceptions’].

I could also set namespace in ray.init(), but I can only call ray.init() once when our server starts up, so it doesn’t help either.

As far as I can tell, Actor namespace is just to help with making sure that actor names won’t collide between different workspaces. I wonder why Ray assumes that there is only 1 job submitted per ray.init()?

6

@Soichi_Hayashi Job is a ray layer concept, and whenever you call ray.init, it’s a job.

The job you mean in the post, it’s an app layer job and it should be managed by yourself, for example, you can have multiple nodes and each running a job and you need to do LB and routing on your side. What you tried to distribute is tasks.

Namespace is for isolation. Actors can have names, and they might conflict in two jobs and this is where namespace comes for.