Download an opensource LLM model in Raycluster yaml file?

ranimg · December 13, 2023, 10:04am

Dear Team,

How can i download an opensource LLM model from hugging transformer in Raycluster yaml configuration? image: Package text-generation-inference · GitHub and where should i configure the MODEL name.

Can i configure this while deploying the ray-cluster? Your support truly helps
Not using Kubernetes

Regards
Rani

Jules_Damji · December 13, 2023, 7:01pm

@ranimg You might be able to download as part of setup commands for each node in the cluster

github.com

ray-project/ray/blob/master/python/ray/autoscaler/aws/example-full.yaml

# An unique identifier for the head node and workers of this cluster.
cluster_name: default

# The maximum number of workers nodes to launch in addition to the head
# node.
max_workers: 2

# The autoscaler will scale up the cluster faster with higher upscaling speed.
# E.g., if the task requires adding more nodes then autoscaler will gradually
# scale up the cluster in chunks of upscaling_speed*currently_running_nodes.
# This number should be > 0.
upscaling_speed: 1.0

# This executes all commands on all nodes in the docker container,
# and opens all the necessary ports to support the Ray cluster.
# Empty string means disabled.
docker:
    image: "rayproject/ray-ml:latest-gpu" # You can change this to latest-cpu if you don't need GPU support and want a faster startup
    # image: rayproject/ray:latest-cpu   # use this one if you don't need ML dependencies, it's faster to pull
    container_name: "ray_container"

This file has been truncated. show original

ranimg · December 14, 2023, 4:08am

Hi @Jules_Damji
I tried this way initially and i face 2 problems here

During docker execution for this container, i always started getting time out exception. I couldn’t find solution to it. Exactly this issue (facing in Azure)
Where exactly i would say the particular LLM model to be downloaded? Is this in the cluster.yaml or in the python wrapper?

Please let me know.

-Rani

Topic		Replies	Views
Kubernetes YAML files for manual deployment Ray Clusters	0	351	August 10, 2021
Overriding resources per worker in ray-llm LLMs/Generative AI/Aviary	7	412	January 30, 2024
Deploying ML Model using Ray serve on K8s Ray Serve	1	1130	August 8, 2022
Create a cluster on azure the workers node no available when resource request Ray Clusters	0	48	April 4, 2024
vLLM example not working in Docker on VM	1	84	September 4, 2024

Download an opensource LLM model in Raycluster yaml file?

Related topics