Inside the celery task call ray remote function: system error: Missing :authority header got this error here

Ray Core
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.
@ray.remote(num_gpus=1)  # Specify the number of GPUs required
def heavy_computation(data):
        # Here goes your GPU-intensive logic, for example:
        time.sleep(10)  # Simulate a GPU-intensive task
        return data * 2
    

@celery_app.task
def gpu_intensive_task(data):
    # This function will be executed on a Ray worker with GPU support
    # Submit the task to Ray and get the result
    if not ray.is_initialized():
        ray.init("ray_head:6385") 
    result_ref = heavy_computation.remote(data)
    result =ray.get(result_ref)
    return result

I got error in above code

elery_service        | [2024-04-01 10:09:36,315: INFO/MainProcess] Task celery_service.ray_task.pinecone_task.gpu_intensive_task[e63932c9-5790-45ca-abf3-09776f7ef2ff] received
celery_service        | [2024-04-01 10:09:41,090: WARNING/MainProcess] Exception in thread 
celery_service        | [2024-04-01 10:09:41,091: WARNING/MainProcess] ray_print_logs
celery_service        | [2024-04-01 10:09:41,091: WARNING/MainProcess] :
celery_service        | [2024-04-01 10:09:41,091: WARNING/MainProcess] Traceback (most recent call last):
celery_service        | [2024-04-01 10:09:41,091: WARNING/MainProcess]   File "/usr/local/lib/python3.9/threading.py", line 980, in _bootstrap_inner
celery_service        | [2024-04-01 10:09:41,092: WARNING/MainProcess]     
celery_service        | [2024-04-01 10:09:41,093: WARNING/MainProcess] self.run()
celery_service        | [2024-04-01 10:09:41,093: WARNING/MainProcess]   File "/usr/local/lib/python3.9/threading.py", line 917, in run
celery_service        | [2024-04-01 10:09:41,094: WARNING/MainProcess]     
celery_service        | [2024-04-01 10:09:41,094: WARNING/MainProcess] self._target(*self._args, **self._kwargs)
celery_service        | [2024-04-01 10:09:41,094: WARNING/MainProcess]   File "/usr/local/lib/python3.9/site-packages/ray/_private/worker.py", line 901, in print_logs
celery_service        | [2024-04-01 10:09:41,095: WARNING/MainProcess]     
celery_service        | [2024-04-01 10:09:41,095: WARNING/MainProcess] data = subscriber.poll()
celery_service        | [2024-04-01 10:09:41,096: WARNING/MainProcess]   File "python/ray/_raylet.pyx", line 3178, in ray._raylet.GcsLogSubscriber.poll
celery_service        | [2024-04-01 10:09:41,096: WARNING/MainProcess]   File "python/ray/_raylet.pyx", line 589, in ray._raylet.check_status
celery_service        | [2024-04-01 10:09:41,097: WARNING/MainProcess] ray.exceptions
celery_service        | [2024-04-01 10:09:41,097: WARNING/MainProcess] .
celery_service        | [2024-04-01 10:09:41,097: WARNING/MainProcess] RaySystemError
celery_service        | [2024-04-01 10:09:41,097: WARNING/MainProcess] : 
celery_service        | [2024-04-01 10:09:41,097: WARNING/MainProcess] System error: Missing :authority header
celery_service        | [2024-04-01 10:13:08,822: WARNING/MainProcess] Exception in thread 
celery_service        | [2024-04-01 10:13:08,822: WARNING/MainProcess] ray_listen_error_messages
celery_service        | [2024-04-01 10:13:08,822: WARNING/MainProcess] :
celery_service        | [2024-04-01 10:13:08,822: WARNING/MainProcess] Traceback (most recent call last):
celery_service        | [2024-04-01 10:13:08,822: WARNING/MainProcess]   File "/usr/local/lib/python3.9/threading.py", line 980, in _bootstrap_inner
celery_service        | [2024-04-01 10:13:08,823: WARNING/MainProcess]     
celery_service        | [2024-04-01 10:13:08,823: WARNING/MainProcess] self.run()
celery_service        | [2024-04-01 10:13:08,823: WARNING/MainProcess]   File "/usr/local/lib/python3.9/threading.py", line 917, in run
celery_service        | [2024-04-01 10:13:08,823: WARNING/MainProcess]     
celery_service        | [2024-04-01 10:13:08,823: WARNING/MainProcess] self._target(*self._args, **self._kwargs)
celery_service        | [2024-04-01 10:13:08,823: WARNING/MainProcess]   File "/usr/local/lib/python3.9/site-packages/ray/_private/worker.py", line 2139, in listen_error_messages
celery_service        | [2024-04-01 10:13:08,823: WARNING/MainProcess]     
celery_service        | [2024-04-01 10:13:08,823: WARNING/MainProcess] _, error_data = worker.gcs_error_subscriber.poll()
celery_service        | [2024-04-01 10:13:08,823: WARNING/MainProcess]   File "python/ray/_raylet.pyx", line 3133, in ray._raylet.GcsErrorSubscriber.poll
celery_service        | [2024-04-01 10:13:08,823: WARNING/MainProcess]   File "python/ray/_raylet.pyx", line 589, in ray._raylet.check_status
celery_service        | [2024-04-01 10:13:08,823: WARNING/MainProcess] ray.exceptions
celery_service        | [2024-04-01 10:13:08,823: WARNING/MainProcess] .
celery_service        | [2024-04-01 10:13:08,823: WARNING/MainProcess] RaySystemError
celery_service        | [2024-04-01 10:13:08,823: WARNING/MainProcess] : 
celery_service        | [2024-04-01 10:13:08,823: WARNING/MainProcess] System error: Missing :authority header got this error here
1 Like