Logging to stdout for ray serve

Ray Libraries (Data, Train, Tune, Serve) Ray Serve
1

Hello Team,
I have my rayserve application running inside a docker container. It’s a single node application with both the ray head and workers running inside the same container.

I have an entrypoint.sh file containing the commands to start the ray server and the deployments (shown below)

python -m rayserve.data_helpers.download_resources
ray start --head --node-ip-address=0.0.0.0 --include-dashboard=true --dashboard-host=0.0.0.0
serve build rayserve.deployer:deployment_graph -o config.yaml
serve deploy config.yaml
python -m rayserve.wait

Details of the commands:
download_resources → Downloads all the model files from S3 into a folder from which they are loaded into memory when the deployment objects are called

ray start --head … → Starts the ray server

serve build … → Creates a config file

serve deploy config.yaml → Starts the workers from the config file

Since serve deploy starts the deployments in the background , I invoke a wait.py file which basically does this

import time

if __name__=="__main__":
    while True:
        time.sleep(0.00001)

Since serve deploy runs everything in the background, I am not able to see any application logs in the console of my container (Screenshot given below)

Screenshot 2022-11-18 at 12.53.25 AM
Screenshot 2022-11-18 at 12.53.25 AM1349×422 94.1 KB

How do you suggest I overcome this?
And is this the recommended way of deploying the application?

Please help

2

Based on the entrypoint.sh file, you seem to be starting Ray and then immediately deploying the Serve config to the new, local Ray cluster. This should be sufficient for your single-node use case. Generally, users put Serve in production using KubeRay, with a long-running Ray cluster on VMs, or on a managed Ray service like Anyscale.

There’s a couple possible ways to access logs for your setup.

  1. Use serve run: Since you’re already waiting for the deployments to start with python -m rayserve.wait, you could use serve run rayserve.deployer:deployment_graph instead of serve deploy config.yaml. serve run will block and wait for the config to deploy. Afterwards, the logs should stream back to the container console.

  2. Access them manually: Ray Serve stores all its logs in the /tmp/ray/session_latest/logs/serve/ directory on each node.

I’d recommend starting with (1) and see if it fulfills your use case.