Capturing Ray Worker Logs in Papermill-Executed Jupyter Notebooks

Ray Core Ray Client
1
  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

env

  • python version: 3.10.14
  • Ray version: 2.20.0
  • KubeRay: 1.10

Hi,

I’m using Ray on a Jupyter notebook and trying to run it with the Papermill CLI like papermill my_ray_exec.ipynb output.ipynb. However, I couldn’t capture print logs from worker nodes in output.ipynb. When I run the notebook my_ray_exec.ipynb on Jupyter, all such logs are captured in the output cell below the cell where the Ray context is initialized. Is there any way to capture logs from worker nodes in the Papermill output notebook file? A sample code is like as follows:

my_ray_exec.ipynb

# ---- notebook cell: begin
import ray
ray.init()
# ---- notebook cell: end

# ---- notebook cell: begin
import time

@ray.remote
def log_check():
    print("Hello")
    time.sleep(2)
    print("Bye")

ray.get(log_check.remote())
# ---- notebook cell: end

# ---- notebook cell: begin
time.sleep(4)
ray.shutdown()
#  ---- notebook cell: end

command

papermill my_ray_exec.ipynb output.ipynb