Fix PR is here: [Serve] Address incremental memory leak due to _PyObjScanner by simon-mo · Pull Request #31317 · ray-project/ray · GitHub
Once the PR is merged, you can install the Ray nightly wheel to confirm. Installing Ray — Ray 3.0.0.dev0
Edit: PR merged!