Hey all
I have the following issue:
I am running the following code:
#############################################
import ray
from ray import serve
from allennlp.predictors.predictor import Predictor
from transformers import BertTokenizer
from flask import Flask
def predict_remote(request):
try:
data = request.get_data(as_text=True)
itms = ray.get([predictor_id, tokenizer_id])
tok = itms[1]
pr = itms[0]
train_encoding = tok(data)
train_encoding = crop(train_encoding[‘input_ids’])
inp = tok.decode(train_encoding)
probs = pr.predict(inp)[‘probs’]
return probs
except Exception as e:
return "Problem detected
Ray initialization, serve registration.
ray.init(_memory=6 * 10**7)
client = serve.start(detached=True)
Backends and endpoints registration.
predictor = Predictor.from_path("/app/models/stanford-sentiment-treebank-roberta.2021-03-11.tar.gz")
tokenizer = BertTokenizer.from_pretrained(“bert-base-cased”, do_lower_case=False)
predictor_id = ray.put(predictor)
tokenizer_id = ray.put(tokenizer)
client.create_backend(“prediction_backend”, predict_remote)
client.create_endpoint(“prediction_endpoint”, backend=“prediction_backend”, methods=[‘POST’], route="/api/v1/sentallen")
Register a health endpoint.
app = Flask(name)
@app.route(’/api/v1/health’)
def health():
return client.list_endpoints()
app.run(host=‘0.0.0.0’, port=8086)
#################################################
Ones this is ran, I am getting the issue with the routing of endpoint. When I try to make a request it returns with (not always):
“Path /api/v1/sentallen not found. Please ping http://…/-/routes for routing table”
The GET request to mentioned endpoint return with empty json.
At the same time GET request to “api/v1/health” endpoint returns the single endpoint api/v1/snetallen.
Looks like a http proxy issue.
Can someone look into this?
Thanks