Download an opensource LLM model in Raycluster yaml file?

ranimg · December 13, 2023, 10:04am

Dear Team,

How can i download an opensource LLM model from hugging transformer in Raycluster yaml configuration? image: Package text-generation-inference · GitHub and where should i configure the MODEL name.

Can i configure this while deploying the ray-cluster? Your support truly helps
Not using Kubernetes

Regards
Rani

Jules_Damji · December 13, 2023, 7:01pm

@ranimg You might be able to download as part of setup commands for each node in the cluster

github.com

ray-project/ray/blob/master/python/ray/autoscaler/aws/example-full.yaml

# An unique identifier for the head node and workers of this cluster.
cluster_name: default

# The maximum number of workers nodes to launch in addition to the head
# node.
max_workers: 2

# The autoscaler will scale up the cluster faster with higher upscaling speed.
# E.g., if the task requires adding more nodes then autoscaler will gradually
# scale up the cluster in chunks of upscaling_speed*currently_running_nodes.
# This number should be > 0.
upscaling_speed: 1.0

# This executes all commands on all nodes in the docker container,
# and opens all the necessary ports to support the Ray cluster.
# Empty string means disabled.
docker:
    image: "rayproject/ray-ml:latest-gpu" # You can change this to latest-cpu if you don't need GPU support and want a faster startup
    # image: rayproject/ray:latest-cpu   # use this one if you don't need ML dependencies, it's faster to pull
    container_name: "ray_container"

This file has been truncated. show original

ranimg · December 14, 2023, 4:08am

Hi @Jules_Damji
I tried this way initially and i face 2 problems here

During docker execution for this container, i always started getting time out exception. I couldn’t find solution to it. Exactly this issue (facing in Azure)
Where exactly i would say the particular LLM model to be downloaded? Is this in the cluster.yaml or in the python wrapper?

Please let me know.

-Rani

Topic		Replies	Views
Ray starts head node succesfully but no workers (Azure) Ray Clusters	2	501	June 29, 2022
Ray workers can't ssh to head node Ray Core	5	600	June 14, 2022
Submitting job to remote AWS cluster Ray Core	3	96	April 5, 2024
The example-full.yaml file questions Ray Clusters	1	399	January 11, 2023
Ray cluster's worker node is pending Ray Clusters	2	1002	February 8, 2022

Download an opensource LLM model in Raycluster yaml file?

Related Topics