Say I have a function copy_data that copies data from an outside source (e.g., s3, gs, database) to a VM, and I want to run that function exactly once on each node in a cluster. Is there a good way to do that? If I have, say, 5 nodes, and each node has 32 cores, I’ve been decorating copy_data with @…

My recommendation is @ray.remote(num_cpus=1) class FunctionExecutor def get_ip_addr(self): return "haha" # write your code num_nodes = len(ray.nodes()) bundles = [{"CPU": 1} for _ in num_nodes] pg = placement_group(bundles=bundles, strategy="STRICT_SPREAD") ray.get(pg.ready()) executors = [Fun…

How to run a function exactly once on each node?

Ray Core

mannyv May 17, 2021, 9:03pm 2

Hi @dutch,

Check out this recent post for one way to address this problem.

1 Like

Topic		Replies	Views
Execute function on each worker process in the cluster Ray Core	4	306	October 5, 2023
Parallelize function across nodes but never within nodes Ray Core	4	294	February 1, 2022
How to create @ray.remote jobs that will only run on the workers from the local node? Ray Core	6	1868	May 9, 2021
How does ray decide where to run a function? Ray Clusters	1	386	December 23, 2021
Ray Cluster: ensure each new task goes to different node Ray Core	1	175	February 8, 2021

How to run a function exactly once on each node?

Related Topics