Say I have a function copy_data that copies data from an outside source (e.g., s3, gs, database) to a VM, and I want to run that function exactly once on each node in a cluster. Is there a good way to do that? If I have, say, 5 nodes, and each node has 32 cores, I’ve been decorating copy_data with @…

My recommendation is @ray.remote(num_cpus=1) class FunctionExecutor def get_ip_addr(self): return "haha" # write your code num_nodes = len(ray.nodes()) bundles = [{"CPU": 1} for _ in num_nodes] pg = placement_group(bundles=bundles, strategy="STRICT_SPREAD") ray.get(pg.ready()) executors = [Fun…

How to run a function exactly once on each node?

Ray Core

mannyv May 17, 2021, 9:03pm 2

Hi @dutch,

Check out this recent post for one way to address this problem.

1 Like

Topic		Replies	Views
How to create @ray.remote jobs that will only run on the workers from the local node? Ray Core	6	2271	May 9, 2021
Ray Cluster: ensure each new task goes to different node Ray Core	1	227	February 8, 2021
Parallelize function across nodes but never within nodes Ray Core	4	390	February 1, 2022
Execute function on each worker process in the cluster Ray Core	4	776	October 5, 2023
Assign specific nodes to remote functions in ray cluster Ray Clusters	8	1999	June 8, 2022

How to run a function exactly once on each node?

Related topics