Ray and Ray Serve are up on a single (Windows 10) laptop.
In the (server) terminal, a python application is loaded. A json request is sent from another (client) terminal to the server application on Ray and there is an error in the program.
The (client) terminal shows a traceback. But, not on the server terminal from the python application running under Ray - from the exception handler or the print statements.
How do you make this happen?
The server side should log and print the same information. Can I ask what did you run in the server terminal? Did you use ray start --head command and the deploy script exit?
Here are the steps:
(server) terminal:
server> ray start --address 127.0.0.1 --port 8787 --num-cpus 2 --dashboard-host localhost --dashboard-port 8265
server> serve start
program raytest.py
import ray
from ray import serve
from fastapi import Request, FastAPI
app = FastAPI()
http_options = {'host': "127.0.0.1", 'port': 8787, 'location': "HeadOnly", 'num_cpus': 2}
ray.init(address="127.0.0.1:8787", namespace="serve")
serve.start(detached=True, http_options=http_options)
@serve.deployment(route_prefix="/api")
@serve.ingress(app)
class Deployment:
def __init__(self):
...
...
Deployment.deploy()
(server) terminal:
server> python raytest.py
2022-01-12 18:37:23,345 INFO worker.py:842 -- Connecting to existing Ray cluster at address: 127.0.0.1:8787
2022-01-12 18:37:25,551 INFO api.py:414 -- Connecting to existing Serve instance in namespace 'serve'.
ray is initialized
2022-01-12 18:37:25,674 INFO api.py:242 -- Updating deployment 'Deployment'. component=serve deployment=Deployment
pid=6500) 2022-01-12 18:37:27,958 INFO deployment_state.py:912 -- Adding 1 replicas to deployment 'Deployment'. component=serve deployment=Deployment
2022-01-12 18:37:33,850 INFO api.py:249 -- Deployment 'Deployment' is ready at `http://127.0.0.1:8000/api`. component=serve deployment=Deployment
server> # program raytest.py exits
(client) terminal
client> python clienttest.py
Traceback (most recent call last):
File "C:\clienttest.py", line 605, in <module>
score = results['data']['score']
KeyError: 'data'
The client terminal and program is behaving. The server terminal and program isn’t.
Ah here you are deploying to a background cluster. The logs should be available in the temp directory’s ray session logs.
You can find that directory via the output of ray.init(…)'s session_dir field, for example:
In [2]: ray.init(address="auto")
Out[2]:
{'node_ip_address': '127.0.0.1',
'raylet_ip_address': '127.0.0.1',
'redis_address': '127.0.0.1:6379',
'object_store_address': '/tmp/ray/session_2022-01-12_10-58-34_145187_60075/sockets/plasma_store',
'raylet_socket_name': '/tmp/ray/session_2022-01-12_10-58-34_145187_60075/sockets/raylet',
'webui_url': '127.0.0.1:8265',
**'session_dir': '/tmp/ray/session_2022-01-12_10-58-34_145187_60075',**
'metrics_export_port': 62094,
'gcs_address': '127.0.0.1:59225',
'address': '127.0.0.1:6379',
'node_id': 'e5aeec0a381ee5a6f4744b335f89434f4156c3ac621428ba2776b060'}
This is set up like this so the log from multiple process can be scraped and ingested properly via log aggregation tool.
Alternatively, you can also keep the raytest.py running by adding a sleep loop in the end, the log should stream through as well (given it is the first time you created serve, so i would recommend no detached=True). With detached=False, your server also shutdown when the script exits.
If you see Connecting to existing Serve instance, then the log won’t be streamed because the HTTP processes have been already started by previous program.
Deployment.deploy()
# add
while True:
time.sleep(2)
Ok, got it working. Thank-you
a. Is the sleep-loop a temporary solution?
b. I couldn’t find an autoreload function to automatically reload code changes during development. Breaking out of Serve and restarting it again is a pain. Is autoreload planned?
Yes! We are currently working on a proper solution to replace the sleep loop with reload. Stay tuned!
Q about the C++ API:
I’ve read the intro at Anyscale - Modern Distributed C++ with Ray.
If the application is written in Python then does it have to re-written in C++ to take advantage of the Ray C++ API ?
Right. C++ here is the language frontend. Internally Python API actually use the C++ core API so the performance is similar. The difference here is whether your application code is in C++