# serveConfig with import path

**URL:** <https://discuss.ray.io/t/serveconfig-with-import-path/22080>\
**Category:** Ray Clusters\
**Created:** [March 18, 2025, 4:18pm UTC](https://discuss.ray.io/t/serveconfig-with-import-path/22080 "2025-03-18T16:18:40Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![puppadas](https://avatars.discourse-cdn.com/v4/letter/p/a4c791/32.png) [@puppadas](https://discuss.ray.io/u/puppadas)\
**Post date:** [March 18, 2025, 4:18pm UTC](https://discuss.ray.io/t/serveconfig-with-import-path/22080/1 "2025-03-18T16:18:40Z")

</div>

**1. Severity of the issue: (select one)**  
 Medium: Significantly affects my productivity but can find a workaround.

**2. Environment:**

- Ray version: latest 2.43
- Python version: 3.11
- OS: Ubuntu
- Cloud/Infrastructure: On prem with AMD GPU’s
- Other libs/tools (if relevant):

**3. What happened vs. what you expected:**

- Expected: Copied vllm-serve.py script to /root folder in the image (also tried with mounting voulme where script included as configmap) and when I start rayservice, it is saying vllm serve module not found and failing.

Below is my rayservice manifest file:

apiVersion: [ray.io/v1](http://ray.io/v1)  
kind: RayService  
metadata:  
name: llama-3-2-3b  
spec:  
serveConfigV2: |  
applications:  
- name: fm-llama-3-2-3b  
route\_prefix: /  
import\_path: vllm-serve:model  
deployments:  
- name: VLLMDeployment  
#num\_replicas: 16  
ray\_actor\_options:  
num\_cpus: 2  
#num\_gpus: 1  
# NOTE: num\_gpus is set automatically based on TENSOR\_PARALLELISM  
runtime\_env:  
env\_vars:  
BACKEND: “ray”  
MODEL\_ID: “meta-llama/Llama-3.1-70B-Instruct”  
TENSOR\_PARALLELISM: “2”  
PIPELINE\_PARALLELISM: “2”  
rayClusterConfig:  
headGroupSpec:  
rayStartParams:  
dashboard-host: ‘0.0.0.0’  
num-gpus: “0”  
template:  
spec:  
containers:  
- name: ray-head  
image: cloud/ce:raycluster-rocm  
resources:  
limits:  
cpu: “8”  
memory: “20Gi”  
requests:  
cpu: “8”  
memory: “20Gi”  
ports:  
- containerPort: 6379  
name: gcs-server  
- containerPort: 8265  
name: dashboard  
- containerPort: 10001  
name: client  
- containerPort: 8000  
name: serve  
- containerPort: 8888  
name: grpc  
env:  
- name: HUGGING\_FACE\_HUB\_TOKEN  
valueFrom:  
secretKeyRef:  
name: hf-secret  
key: hf\_api\_token  
envFrom:  
- configMapRef:  
name: server-config

```
workerGroupSpecs:
- replicas: 1
  minReplicas: 1
  maxReplicas: 2
  numOfHosts: 2
  groupName: gpu-group
  rayStartParams:
    num-gpus: "2"
  template:
    spec:
      containers:
      - name: fm-llama-3-2-3b
        image: cloud/ce:raycluster-rocm
        volumeMounts:
        - mountPath: /dev/shm
          name: dshm
        - mountPath: /root/.cache/huggingface/hub
          name: ray-pvc-source
          subPath: .cache/huggingface/hub
        env:
        - name: HUGGING_FACE_HUB_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-secret
              key: hf_api_token
        - name: NODE_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        resources:
          limits:
            cpu: "50"
            memory: "200Gi"
            amd.com/gpu: "2"
          requests:
            cpu: "50"
            memory: "200Gi"
            amd.com/gpu: "2"
      volumes:
      - name: dshm
        emptyDir:
          medium: Memory
          sizeLimit: 100Gi
      - name: ray-pvc-source
        persistentVolumeClaim:
          claimName: ray-pv-claim
      # Please add the following taints to the GPU node.
      tolerations:
        - key: "amd.com/gpu"
          operator: "Exists"
          effect: "NoSchedule"

```

~

- Actual: