I am sure this is a solved problem, but I want to confirm what I intend to do is correct.
I am building my python project using Bazel and I want to run some distributed workload on Ray Clusters on GKE. The issue is when new raylet spawns in new containters, they will not have access to code or extra deps as those are packaged and managed by Bazel and are NOT available in the python env.
My solution to this is to build a wheel of my project and installing it in my container. This should solve it, but then I have to add the extra deps manually by hand (so far did not find a good way to collect these).
I want to understand if this is the correct way to make a Ray Application in a Bazel project.
Yes, your approach is correct: building a wheel of your project and installing it in your Ray container is the recommended way to ensure all nodes have access to your code and dependencies when using Bazel for packaging. This is a common solution for Ray clusters where dependencies are not available in the Python environment by default. You will need to manually specify and install extra dependencies in your container, as Ray expects all dependencies to be pre-installed on all nodes; there is no built-in Ray mechanism to automatically collect Bazel-managed dependencies for you. This method is also suggested for similar setups in Spark and is considered best practice for Ray clusters running in Docker containers on Kubernetes or GKE environments, as it ensures consistency across all nodes and avoids serialization issues with dependencies at runtime (discuss.ray.io: Ray runtime env from docker image or bazel target, discuss.ray.io: Python code with large dependencies).
If you want to automate collecting extra dependencies, you may need to script the extraction of Bazel-managed dependencies into your Docker build process, but this is outside Ray’s scope. Using a custom Docker image with all dependencies pre-installed is the most robust and maintainable approach (discuss.ray.io: Python code with large dependencies, discuss.ray.io: Ray runtime env from docker image or bazel target). Would you like more detail on automating dependency collection or Docker image setup?
What is py_executable used for? It mentions in the documentation briefly that it can be used for projects using bazel build systems.
Customizing the command the worker runs in: On the workers, you might want to customize uv with some special arguments that aren’t used for the driver. Or, you might want to run processes using poetry run, a build system like bazel, a profiler, or a debugger. In these cases, you can explicitly specify the executable the worker should run in via py_executable. It could even be a shell script that is stored in working_dir if you are trying to wrap multiple processes in more complex ways.