Development environment/tool chain

1

Hi all,

I have been using ray and it seems to work quite well. Now my development environment for a kaggie competition is a combination of jupyter notebook and a number of python source files that are loaded by the notebook. I am editing these python files with pycharm because it works 100 times better than jupyter. Basically, jupyter is only convenient for running on a server with a strong gpu. However, ray also provides this functionality. Therefore, my question is what do other people use for developing efficiently for for instance data science challenges? What does your workflow/tool chain look like? Is it a good idea to work locally and use ray for remote execution?

Cheers
Erik

1 Like
2

Hey, we (the Ray team) are actively working to improve our local development/remote execution story.

We just released the Ray Client, which is meant to solve this exact problem. There are currently some rough edges (around dependency management and placement groups), so you may need to pay attention to how pickling works (rule of thumb: things within a file are usually serialized, whereas things in other files are pointers), but it sounds like what you’re looking for.

https://docs.ray.io/en/master/ray-client.html

1 Like