Invalid method encountered or duplicate "Content-Length" header?

Ray Libraries (Data, Train, Tune, Serve) Ray Serve
21

Sorry I should have mention when you are installing the wheel you need to install related serve dependencies as well. You can fix it by pip install the following as well

 "uvicorn==0.16.0" "requests" "starlette" "fastapi" "aiorwlock"
22

No worries, I installed all the above packs and tested.

  • There is no duplicate content-length now
āÆ curl -ki -H "Content-Type:application/json" -d '{"text":"I live in california"}' http://127.0.0.1:8000/api/entities
HTTP/1.1 200 OK
date: Thu, 27 Jan 2022 02:32:34 GMT
server: uvicorn
content-length: 43
content-type: application/json

{"Response":{"predictions":["california"]}}%
  • Postman is happy too. :slight_smile:

Thanks @simon-mo for your relentless efforts and support.

23

Hi @simon-mo ā€¦ did anything change?


I created complete cluster with increased resources, followed all the steps to setup and I am seeing errors related GCS (redis) again.

(base) ray@example-cluster-ray-head-type-zhjst:~$ pip list | grep -E "uvicorn|requests|starlette|fastapi|aiorwlock|h11|httptools"
aiorwlock                              1.3.0
fastapi                                0.70.0
h11                                    0.13.0
requests                               2.26.0
requests-oauthlib                      1.3.0
starlette                              0.16.0
uvicorn                                0.16.0
(base) ray@example-cluster-ray-head-type-zhjst:~$ python
Python 3.7.7 (default, May  7 2020, 21:25:33)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ray
>>> ray.__commit__
'8ebc50f844f42e283f125792d630ea6d0a2a7000'

āÆ ray submit example-full.yaml deploy_test.py
2022-01-27 12:00:18,679	INFO util.py:278 -- setting max workers for head node type to 0
Loaded cached provider configuration
If you experience issues with the cloud provider, try re-running the command with --no-config-cache.
2022-01-27 12:00:24,344	INFO util.py:278 -- setting max workers for head node type to 0
2022-01-27 12:00:24,456	INFO command_runner.py:172 -- NodeUpdater: example-cluster-ray-head-type-zhjst: Running kubectl -n ray exec -it example-cluster-ray-head-type-zhjst -- bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (python ~/deploy_test.py)'
Traceback (most recent call last):
  File "/home/ray/deploy_test.py", line 15, in <module>
    ray.init(address="auto", namespace="ray", log_to_driver=True, logging_level=logging.INFO)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 969, in init
    connect_only=True)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/node.py", line 177, in __init__
    "session_name", ray_constants.KV_NAMESPACE_SESSION)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/node.py", line 1470, in _internal_kv_get_with_retry
    raise RuntimeError(f"Could not read '{key}' from GCS (redis). "
RuntimeError: Could not read 'b'session_name'' from GCS (redis). If using Redis, did Redis start successfully?
command terminated with exit code 1
2022-01-27 12:01:08,160	ERROR command_runner.py:184 -- NodeUpdater: example-cluster-ray-head-type-zhjst: Command failed:

  kubectl -n ray exec -it example-cluster-ray-head-type-zhjst --'bash --login -c -i '"'"'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (python ~/deploy_test.py)'"'"''

Am I missing something here?

24

Is Ray running on the head node? What does ray status return? You might need ray up --restart-only to restart the ray runtime on those node.

25

Here it is

(base) ray@example-cluster-ray-head-type-zhjst:~$ ray status
======== Autoscaler status: 2022-01-27 12:18:43.670684 ========
Node status
---------------------------------------------------------------
Healthy:
 1 rayHeadType
 2 rayWorkerType
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 6.0/8.0 CPU (6.0 used of 6.0 reserved in placement groups)
 0.00/24.500 GiB memory
 0.00/10.439 GiB object_store_memory

Demands:
 (no resource demands)
26

This is weird. What about restarting the Ray cluster?

ray up --restart-only  example-full.yaml
ray submit example-full.yaml deploy_test.py
27

Got thisā€¦

āÆ ray up example-full.yaml --restart-only
Cluster: example-cluster

2022-01-27 13:34:48,488	INFO util.py:278 -- setting max workers for head node type to 0
Loaded cached provider configuration
If you experience issues with the cloud provider, try re-running the command with --no-config-cache.
Updating cluster configuration and restarting the cluster Ray runtime. Setup commands will not be run due to `--restart-only`.
Confirm [y/N]: y

Currently running head node is out-of-date with cluster configuration
  Current hash is 4c58d362002d40200ef7cfa0e35f7ef36c5de827, expected b708b84c6d058c6f224d86579ecea46193081dca
  Current head node type is rayHeadType, expected head_node
Acquiring an up-to-date head node
  Relaunching the head node. Confirm [y/N]: y
2022-01-27 13:34:57,039	INFO node_provider.py:171 -- KubernetesNodeProvider: calling delete_namespaced_pod
  Terminated head node example-cluster-ray-head-type-kl8vh
2022-01-27 13:34:57,303	INFO node_provider.py:137 -- KubernetesNodeProvider: calling create_namespaced_pod (count=1).
Traceback (most recent call last):
  File "/Users/macpro/anaconda3/envs/iris3.7.7ray/bin/ray", line 8, in <module>
    sys.exit(main())
  File "/Users/macpro/anaconda3/envs/iris3.7.7ray/lib/python3.7/site-packages/ray/scripts/scripts.py", line 1958, in main
    return cli()
  File "/Users/macpro/anaconda3/envs/iris3.7.7ray/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/macpro/anaconda3/envs/iris3.7.7ray/lib/python3.7/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/macpro/anaconda3/envs/iris3.7.7ray/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/macpro/anaconda3/envs/iris3.7.7ray/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/macpro/anaconda3/envs/iris3.7.7ray/lib/python3.7/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/macpro/anaconda3/envs/iris3.7.7ray/lib/python3.7/site-packages/ray/autoscaler/_private/cli_logger.py", line 808, in wrapper
    return f(*args, **kwargs)
  File "/Users/macpro/anaconda3/envs/iris3.7.7ray/lib/python3.7/site-packages/ray/scripts/scripts.py", line 973, in up
    use_login_shells=use_login_shells)
  File "/Users/macpro/anaconda3/envs/iris3.7.7ray/lib/python3.7/site-packages/ray/autoscaler/_private/commands.py", line 237, in create_or_update_cluster
    override_cluster_name, no_monitor_on_head)
  File "/Users/macpro/anaconda3/envs/iris3.7.7ray/lib/python3.7/site-packages/ray/autoscaler/_private/commands.py", line 635, in get_or_create_head_node
    provider.create_node(head_node_config, head_node_tags, 1)
  File "/Users/macpro/anaconda3/envs/iris3.7.7ray/lib/python3.7/site-packages/ray/autoscaler/_private/_kubernetes/node_provider.py", line 140, in create_node
    pod = core_api().create_namespaced_pod(self.namespace, pod_spec)
  File "/Users/macpro/anaconda3/envs/iris3.7.7ray/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 7320, in create_namespaced_pod
    return self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)  # noqa: E501
  File "/Users/macpro/anaconda3/envs/iris3.7.7ray/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 7429, in create_namespaced_pod_with_http_info
    collection_formats=collection_formats)
  File "/Users/macpro/anaconda3/envs/iris3.7.7ray/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 353, in call_api
    _preload_content, _request_timeout, _host)
  File "/Users/macpro/anaconda3/envs/iris3.7.7ray/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 184, in __call_api
    _request_timeout=_request_timeout)
  File "/Users/macpro/anaconda3/envs/iris3.7.7ray/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 397, in request
    body=body)
  File "/Users/macpro/anaconda3/envs/iris3.7.7ray/lib/python3.7/site-packages/kubernetes/client/rest.py", line 281, in POST
    body=body)
  File "/Users/macpro/anaconda3/envs/iris3.7.7ray/lib/python3.7/site-packages/kubernetes/client/rest.py", line 234, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'e8cb5fb3-6646-4a34-a6ae-d783429aeaa5', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'e91d3839-905b-40ec-afee-7d6f728bc70e', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'b949bafe-816d-4caa-9208-f8fe085f268d', 'Date': 'Thu, 27 Jan 2022 21:34:57 GMT', 'Content-Length': '312'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"example-cluster-ray-head-\" is forbidden: error looking up service account ray/autoscaler: serviceaccount \"autoscaler\" not found","reason":"Forbidden","details":{"name":"example-cluster-ray-head-","kind":"pods"},"code":403}
  • I reverted changes to example-full.yaml, now above error is no longer there.

Butā€¦

28

Now below commands are running fine and I could submit all relevant deployments.

ray up --restart-only  example-full.yaml
ray submit example-full.yaml deploy_test.py

  • But received duplicate content-length header, again.
āÆ curl -ki -H "Content-Type:application/json" -d '{"text":"I live in california"}' http://127.0.0.1:8000/api/entities
HTTP/1.1 200 OK
date: Thu, 27 Jan 2022 22:54:37 GMT
server: uvicorn
content-length: 43
content-length: 43
content-type: application/json

{"Response":{"predictions":["california"]}}%

  • Upon checking head and worker node I found thisā€¦ - Looks like Nightly version has been reverted back by ray up --restart-only example-full.yaml.
(base) ray@example-cluster-ray-head-type-8s69v:~/app$ pip list | grep ray
ray                                    1.9.2
ray-cpp                                1.9.2
29

Have we found a resolution to this? I actually am using ray behind a FastAPI implementation.

using ray=1.9.2

pip list | grep -E ā€œuvicorn|requests|starlette|fastapi|aiorwlock|h11|httptoolsā€
fastapi 0.73.0
h11 0.9.0
httptools 0.3.0
requests 2.23.0
starlette 0.17.1
uvicorn 0.17.3

It seems like in order to receive any response without the server closing I need httptools. However I am still receiving the duplicate content-length header.

Iā€™ve tried a couple different configurations from this thread, but still havenā€™t resolved it.

30

This is how it worked for me.

  1. Installing/updating ray-nightly to cluster directly is somehow broken/not-working for me via ray commands.

  2. Work-around that worked for me is, I created new image with from rayproject/ray:latest with updated libs like ray-nightly and "uvicorn==0.16.0" "requests" "starlette" "fastapi" "aiorwlock" on top of it.

  3. Use this newly created image for your cluster deployment.

Hope this helps.

2 Likes
31

Hi @Macklin_Fluehr , as @lihost just posted above, first hand experience is likely most effective :slight_smile:

Only thing i want to add is ensure youā€™re running ray-nightly rather than 1.9.2, since Simonā€™s fix was only recently introduced and not included on stable release.

1 Like
32

yup, that did it. Thank you.