Assigning Different Endpoints to the same Route but Different Backends

Hi there,

right now we are switching from aiohttp to ray serve one thing we have in place rigth now is that depending on the request method the backend is different, that means we have different backends assigned to the same route like this:

app.router.add_get('/my endpoint', function_1)
app.router.add_put("/my_endpoint", function_2)

In ray serve I now tried it like that:

client_.create_endpoint('get_endpoint', backend=f'{func.__name__}_backend_1',
                                methods=['GET'], route='/my_endpoint')
client_.create_endpoint('put_endpoint', backend=f'{func.__name__}_backend_2',
                                       methods=['PUT'], route='/my_endpoint')

But I get the error:

E      ValueError: Cannot create endpoint. Route '/my_endpoint' is already registered.

So is there a way to overcome this issue? Or are you planning to support that in the future?

If you did not plan it right now let me give you some reasons why this is would be a nice feature in my opinion:

  • when implementing authentication (OAuth2), some users might only be allowed to perform read (GET) but not write (DELETE/PUT), the authentication is often handled with decoraters, which check the users permissions, but they do not provide functionallity for checking the type of request method, so they need to be extendend/changed, normally they are not written by the developers using it, which makes it complicated
  • by not beeing able to assign different backends based on the combination of route/request method makes the backends more complicated, because you always need to provide different functionallity for GET/PUT/DELETE/POST, which will cause in (in my opinion) not nice if/else checks for the request method
  • other frameworks support it, not having it might be a reason for some to not switch to ray serve

Thanks for the question @TanjaBayer! This seems like a pretty significant painpoint. There is nothing in progress to support this, but we’re actively rethinking how best to do ingress to offer more flexibility. The major challenge is that we want to avoid completely re-writing a web framework from scratch and would prefer to leverage existing open source tools like FastAPI.

One solution we’ve been thinking of is allowing each backend to be a standalone FastAPI server (you could think of it like a FastAPI router). You could have multiple paths, methods, etc. that point to the same backend. For example, for your issue you could do something like:

class MyBackend:
    @serve.get("/")
    def get_endpoint(self, *args):
        pass

    @serve.put("/")
    def get_endpoint(self, *args):
        pass

client.create_backend("my_backend", MyBackend)
client.create_endpoint("my_endpoint", backend="my_backend", route="my_endpoint")

Assuming you had full feature parity with FastAPI for defining the routes, would that address your problem?

1 Like

I actually did not use FastApi so far, only aiohttp and flask.

In general the approach you suggest sounds reasonable, but it still restricts how free you are in assigning endpoints to functions. Because with that approach you are not able to have different endpoints pointing to the same function anymore, e.g.:

GET /user_request/user_id → get_userdata()
PUT /user_request/user_id (Not applicable)

GET /admin/user_id → get_userdata()
PUT /admin/user_id → update_userdata()

I think with the approach you mentioned above PUT /user_request/user_id will also be redirected to update_user_data(), but that function should not be allowed at all in my example.

Maybe that is a bad example because you would handle that with access rights but we have a similar example where the PUT of two different routes would call the same function while the GET goes to different backends.

Does it get clear what I want to say there?