Problems with file dependencies in serve

Ray Libraries (Data, Train, Tune, Serve) Ray Serve
1

I am trying to deploy some functions that provide access to a trained model. The environment of the model relies on external files as dependencies. When training and evaluation normally everything works fine. But when i try to load an rllib trainer within serve it seems to fail while loading these external files. Can i use a runtime_env to solve this? And how exactly can i provide it to serve? Any help would be appreciated!

Serve code

serve.start()

@serve.deployment(route_prefix="/model-ppo")
class ModelServer:
	def __init__(self) -> None:
		self.agent = PPOTrainer(config=config, env=MyEnv)

ModelServer.deploy()
2

Hi @Blubberblub, yes Serve can accept runtime environment dictionary via either:

  • passed through ray.init(runtime_env=..) before calling rerve.start
  • either configure per deployment dependency (working_dir won’t be supported via this way) via @serve.deployment(ray_actor_options={"runtime_env": {...}}).

Generally if the external files exist in the same directory as you call ray start or serve start, it should be able to load them directly!

1 Like
3

@simon-mo Thanks for the clarification. I found the second option you posted and as you said working_dir isn’t supported like this so i was left a little confused. I will try out your other suggestions. Thanks!