Equivalent of @ray.remote which can work at pool submitting time not at the function definition?

Ray Core
1

I re-discovered Ray after starting to write a functional wrapper for multiprocessing’s pool which turns function arguments into shared memory references using python’s inspect module. One thing I would like to have is to efficiently submit functions to a pool in a way that does not modify or wrap around a function’s def definition, but rather ad-hoc without affecting other uses of the same original function definition. So that the same function is used in someplaces with Ray, and in others without Ray, within my codebase.

As I understand Ray’s code for its remote decorator, this decorator cannot be applied other than to a function def. Is there nonetheless a way?

2

Not sure if I’m answering your question but these 2 can be used:

@ray.remote
def foo():
    pass

or

def foo():
    pass
foo_remote = ray.remote(foo)

The second is called using foo_remote.remote

hope this helps

3

Thanks a lot @liberabaci!

I got it to work without any function definition outside of the pool dispatch:

ray.get([partial(ray.remote(f).remote, some_fixed_args_to_f)(dynamic_args) for dynamic_arg in range(dynamic_args)])

Of course the partial function can be defined once before this expression to yield the same.

I’ll check how I didn’t come with it myself …

This is great.