ray version == 2.7.1, and ray cluster run in docker.
Here is my fastapi.py
@ray.remote
def run_task(entrypoint, tid, metadata):
entrypoint += f" --job='{json.dumps(metadata)}'"
cmd = shlex.split(entrypoint)
logging.info(f"run {cmd}")
with Popen(args=cmd) as p:
...
@app.post("/tasks/")
async def fop_jobs(req: FopReq):
a = run_fop.options(
num_cpus=req.entrypoint_num_cpus,
num_gpus=req.entrypoint_num_gpus,
runtime_env=req.runtime_env
).remote(req.entrypoint, req.tid, req.metadata)
return req
if __name__ == "__main__":
uvicorn.run(app, host='0.0.0.0', port=8000)
I use memray tool get memory leak report:
After 1w+ request “/tasks”:
0x1 56.3 MiB malloc 1239346 invocation at /usr/local/lib/python3.10/site-packages/ray/remote_function.py:411
After 2w+ request “/tasks”:
0x27 150.3 MiB malloc 3198029 invocation at /usr/local/lib/python3.10/site-packages/ray/remote_function.py:411
Is there anyone who can give some constructive advice?
Additional information:
flamegraph
