How severe does this issue affect your experience of using Ray?
- Low: It annoys or frustrates me for a moment.
I’m running Ray in a relatively constrained environment, with a single GPU, running soft-realtime ML inference tasks. These tasks use known amounts of GPU-memory. They are all instantiated into an Actor so that the we only pay initialization cost once, and the models stay in memory.
I’m tracking GPU memory as a custom gpu_memory resource that is set when starting Ray. Let’s pretend that’s a value in GB to make this example easier.
I’ve set the Actor to need a custom resource:
@ray.remote(max_concurrency=10, resources={'gpu_memory': 1})
class Actor:
def __init__(self):
# Load models into memory...
def do_work(...):
...
That covers the steady-state GPU memory usage, but each task also requires more GPU memory to actually run, based on its inputs. What I’d like to do is something like this:
@ray.remote(max_concurrency=10, resources={'gpu_memory': 1})
class Actor:
@ray.remote(resources={'gpu_memory': 2}) # or similar
def do_work(...):
...
This doesn’t work, I assume because the @ray.remote decorator is not designed to work on class methods.
I was hoping specifying options on the method call would work like it does with a normal task:
actor.do_work.options(resources={'gpu_memory': 2}).remote(...)
This also doesn’t work.
The workaround I’m using right now is to create a separate task that calls into the actor to reserve the resources:
@ray.remote(resources={'gpu_memory': 2})
def do_work(actor, ...):
ref = actor.do_work.remote(...)
ray.wait(ref)
return ref
This works, but is a little odd as a pattern. Is there a simpler built-in way to do this?
Thanks!