# Ray Train **URL:** https://discuss.ray.io/c/ray-train/17.md?page=2 [Latest](https://discuss.ray.io/latest.md) · [Categories](https://discuss.ray.io/categories.md) · [Tags](https://discuss.ray.io/tags.md) **Page:** 3 --- ## [\[ray dataset\] Ray\_import\_thread blocked causing ray data hanging?](https://discuss.ray.io/t/ray-dataset-ray-import-thread-blocked-causing-ray-data-hanging/13953)
**Author:** [@raytune\_kuberay\_user](https://discuss.ray.io/u/raytune_kuberay_user)\ **Replies:** 0\ **Last updated:** [March 8, 2024, 10:07pm UTC](https://discuss.ray.io/t/ray-dataset-ray-import-thread-blocked-causing-ray-data-hanging/13953 "2024-03-08T22:07:30Z")
we are doing some ray data repartition, batch etc. And then the program seems to hangs at repartition. ray 2.4.0 did a py-spy for ray::main on head node and found that $ py-spy dump -p 737 Process 737: ray::main Pytho… --- ## [Access ray train checkpoint after training](https://discuss.ray.io/t/access-ray-train-checkpoint-after-training/13890)
**Author:** [@arunppsg](https://discuss.ray.io/u/arunppsg)\ **Replies:** 2\ **Last updated:** [March 8, 2024, 7:36pm UTC](https://discuss.ray.io/t/access-ray-train-checkpoint-after-training/13890 "2024-03-08T19:36:57Z")
How severe does this issue affect your experience of using Ray? Medium: It contributes to significant difficulty to complete my task, but I can work around it. I am working on using TorchTrainer to train models from t… --- ## [Installing TensorRT LLM on Ray Docker Image as Custom Docker](https://discuss.ray.io/t/installing-tensorrt-llm-on-ray-docker-image-as-custom-docker/13306)
**Author:** [@rifkybujana](https://discuss.ray.io/u/rifkybujana)\ **Replies:** 2\ **Last updated:** [March 7, 2024, 12:20am UTC](https://discuss.ray.io/t/installing-tensorrt-llm-on-ray-docker-image-as-custom-docker/13306 "2024-03-07T00:20:38Z")
Hi, I’m trying to use TensorRT LLM (GitHub - NVIDIA/TensorRT-LLM: TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art… --- ## [How to stream data directly from s3](https://discuss.ray.io/t/how-to-stream-data-directly-from-s3/13838)
**Author:** [@Aleksei](https://discuss.ray.io/u/Aleksei)\ **Replies:** 2\ **Last updated:** [March 4, 2024, 10:59am UTC](https://discuss.ray.io/t/how-to-stream-data-directly-from-s3/13838 "2024-03-04T10:59:14Z")
How severe does this issue affect your experience of using Ray? Medium: It contributes to significant difficulty to complete my task, but I can work around it. We need to train pytorch models on large datasets 500GB-2… --- ## [How to set TORCH\_DISTRIBUTED\_DEBUG evn var](https://discuss.ray.io/t/how-to-set-torch-distributed-debug-evn-var/13692)
**Author:** [@nimaous](https://discuss.ray.io/u/nimaous)\ **Replies:** 0\ **Last updated:** [February 11, 2024, 11:04am UTC](https://discuss.ray.io/t/how-to-set-torch-distributed-debug-evn-var/13692 "2024-02-11T11:04:11Z")
Hello, I’m using Ray TochTrainer() to train my model on a multi-cluster setup. I’m getting some errors and I want to set TORCH\_DISTRIBUTED\_DEBUG=Info and also need to somehow pass find\_unused\_parameters to Pytorch DDP… --- ## [Best practices to run multiple models in multiple GPUs in RayLLM](https://discuss.ray.io/t/best-practices-to-run-multiple-models-in-multiple-gpus-in-rayllm/13663)
**Author:** [@lizzzcai](https://discuss.ray.io/u/lizzzcai)\ **Replies:** 0\ **Last updated:** [February 8, 2024, 3:57am UTC](https://discuss.ray.io/t/best-practices-to-run-multiple-models-in-multiple-gpus-in-rayllm/13663 "2024-02-08T03:57:11Z")
Hi team, I am trying out RayLLM with KubeRay recently following example here, I am trying to deploy a Mistral-7b model on 1xA10 GPU and 4x A10 GPU. I am expecting 4x throughput but only get 1.x improvement. Would like t… --- ## [Training time not change linearly when changing sample/batch size](https://discuss.ray.io/t/training-time-not-change-linearly-when-changing-sample-batch-size/13653)
**Author:** [@Ziqi\_Jiang](https://discuss.ray.io/u/Ziqi_Jiang)\ **Replies:** 0\ **Last updated:** [February 6, 2024, 6:14pm UTC](https://discuss.ray.io/t/training-time-not-change-linearly-when-changing-sample-batch-size/13653 "2024-02-06T18:14:20Z")
How severe does this issue affect your experience of using Ray? None: Just asking a question out of curiosity Low: It annoys or frustrates me for a moment. Medium: It contributes to significant difficulty to complete m… --- ## [ScalingConfig() num\_workers not corresponding to training runs?](https://discuss.ray.io/t/scalingconfig-num-workers-not-corresponding-to-training-runs/12744)
**Author:** [@ray\_user\_11](https://discuss.ray.io/u/ray_user_11)\ **Replies:** 8\ **Last updated:** [February 5, 2024, 9:25pm UTC](https://discuss.ray.io/t/scalingconfig-num-workers-not-corresponding-to-training-runs/12744 "2024-02-05T21:25:05Z")
I am running a ray.tune.Tuner() that depends on a ScalingConfig(). My understanding from the documentation is that, when using only CPUs, the number of simultaneous training runs should be equal to num\_workers. However,… --- ## [Error in databricks](https://discuss.ray.io/t/error-in-databricks/10833)
**Author:** [@Akshat\_Suwalka](https://discuss.ray.io/u/Akshat_Suwalka)\ **Replies:** 1\ **Last updated:** [February 1, 2024, 2:54am UTC](https://discuss.ray.io/t/error-in-databricks/10833 "2024-02-01T02:54:39Z")
Libraries HOROVOD\_WITH\_GLOO=1 … pip install ‘horovod\[ray\]’ pip install horovod\[ray\] pip install ray\[default\]\>=2.3.0 pip install raydp Code is :- from ray.train.horovod import HorovodTrainer Error is :- TypeErr… --- ## [Get Trial Directory](https://discuss.ray.io/t/get-trial-directory/13554)
**Author:** [@sgerber-hf](https://discuss.ray.io/u/sgerber-hf)\ **Replies:** 0\ **Last updated:** [January 26, 2024, 3:39pm UTC](https://discuss.ray.io/t/get-trial-directory/13554 "2024-01-26T15:39:09Z")
How severe does this issue affect your experience of using Ray? Low I am trying to figure out how to get the trial directory being used by ray train (TorchTrainer in my case), such that I can use ray.train.Result.fro… --- ## [XGBoostTrainer Warning: Saving into deprecated binary model format](https://discuss.ray.io/t/xgboosttrainer-warning-saving-into-deprecated-binary-model-format/12916)
**Author:** [@daviddwlee84](https://discuss.ray.io/u/daviddwlee84)\ **Replies:** 4\ **Last updated:** [December 19, 2023, 2:48am UTC](https://discuss.ray.io/t/xgboosttrainer-warning-saving-into-deprecated-binary-model-format/12916 "2023-12-19T02:48:42Z")
There are warning messages that keep spamming during the training progress when using XGBoostTrainer + RunConfig & CheckpointConfig. WARNING: /workspace/src/c\_api/c\_api.cc:1240: Saving into deprecated binary model forma… --- ## [Checking if TorchTrainer is using the available GPUs](https://discuss.ray.io/t/checking-if-torchtrainer-is-using-the-available-gpus/13066)
**Author:** [@psr.ai](https://discuss.ray.io/u/psr.ai)\ **Replies:** 2\ **Last updated:** [December 6, 2023, 6:29pm UTC](https://discuss.ray.io/t/checking-if-torchtrainer-is-using-the-available-gpus/13066 "2023-12-06T18:29:10Z")
How severe does this issue affect your experience of using Ray? High: It blocks me to complete my task. I am trying to train on a given dataset using distributed training in ray. I ran the sample code given in the ex… --- ## [DEADLINE\_EXCEEDED when training using xgboost\_ray on Sagemaker](https://discuss.ray.io/t/deadline-exceeded-when-training-using-xgboost-ray-on-sagemaker/12974)
**Author:** [@Bhargav\_Srinivasan](https://discuss.ray.io/u/Bhargav_Srinivasan)\ **Replies:** 2\ **Last updated:** [November 30, 2023, 11:26pm UTC](https://discuss.ray.io/t/deadline-exceeded-when-training-using-xgboost-ray-on-sagemaker/12974 "2023-11-30T23:26:20Z")
How severe does this issue affect your experience of using Ray? High: It blocks me to complete my task. I’m trying to train a xgboost model on xgboost\_ray in Sagemaker. The train works for a smaller dataset (11.5 GB t… --- ## [Can I catch the original error in code outside train\_func?](https://discuss.ray.io/t/can-i-catch-the-original-error-in-code-outside-train-func/12976)
**Author:** [@KepingYan](https://discuss.ray.io/u/KepingYan)\ **Replies:** 5\ **Last updated:** [November 30, 2023, 11:38am UTC](https://discuss.ray.io/t/can-i-catch-the-original-error-in-code-outside-train-func/12976 "2023-11-30T11:38:58Z")
Hi, I want to catch the original error when trainer.fit() reports an error. Is it possible? from typing import Any, Dict from ray.train.torch import TorchTrainer from ray.air.config import ScalingConfig from ray.air imp… --- ## [OOM when I decoupled ray from GPTj finetune script](https://discuss.ray.io/t/oom-when-i-decoupled-ray-from-gptj-finetune-script/12879)
**Author:** [@Purvang](https://discuss.ray.io/u/Purvang)\ **Replies:** 0\ **Last updated:** [November 17, 2023, 11:39pm UTC](https://discuss.ray.io/t/oom-when-i-decoupled-ray-from-gptj-finetune-script/12879 "2023-11-17T23:39:54Z")
I replicated gptj\_ray with batch size = 48 on 8xH100 and able to run without any error. Later, I decoupled all ray related functionality and tried to run without ray with Batch size=1 on same server, but I am keep getti… --- ## [Pytorch+ray train example not working](https://discuss.ray.io/t/pytorch-ray-train-example-not-working/12740)
**Author:** [@MakGulati](https://discuss.ray.io/u/MakGulati)\ **Replies:** 4\ **Last updated:** [November 9, 2023, 9:02pm UTC](https://discuss.ray.io/t/pytorch-ray-train-example-not-working/12740 "2023-11-09T21:02:07Z")
How severe does this issue affect your experience of using Ray? High: It blocks me to complete my task. I was trying pytorch+ray train example \[Get Started with PyTorch — Ray 2.8.0\](ray torch train example) impor… --- ## [Horovod Trainer hangs](https://discuss.ray.io/t/horovod-trainer-hangs/9942)
**Author:** [@mizhazha](https://discuss.ray.io/u/mizhazha)\ **Replies:** 5\ **Last updated:** [November 3, 2023, 5:14am UTC](https://discuss.ray.io/t/horovod-trainer-hangs/9942 "2023-11-03T05:14:07Z")
High: It blocks me to complete my task. I use Horovod trainer. The train job hangs after many iterations without failing. The horovod timeout should be whatever ray default. I expect the job to fail with exception thr… --- ## [RayTrainReportCallback error using in Pytorch Lightning](https://discuss.ray.io/t/raytrainreportcallback-error-using-in-pytorch-lightning/12243)
**Author:** [@serena.hyesun](https://discuss.ray.io/u/serena.hyesun)\ **Replies:** 8\ **Last updated:** [October 26, 2023, 7:47pm UTC](https://discuss.ray.io/t/raytrainreportcallback-error-using-in-pytorch-lightning/12243 "2023-10-26T19:47:31Z")
How severe does this issue affect your experience of using Ray? High: It blocks me to complete my task. Hi all, I’m trying ray tune for hyperparameter searching with Pytorch Lightning. I have followed the tutorial … --- ## [Distributed training with uneven inputs](https://discuss.ray.io/t/distributed-training-with-uneven-inputs/12427)
**Author:** [@ben1](https://discuss.ray.io/u/ben1)\ **Replies:** 3\ **Last updated:** [October 26, 2023, 5:17pm UTC](https://discuss.ray.io/t/distributed-training-with-uneven-inputs/12427 "2023-10-26T17:17:19Z")
I’m trying to replace torch.DDP with ray.train, and hit the problem of different DDP instance having different number of input batches. DDP works with the ‘Join’ context manager can fix this problem, as shown below. H… --- ## [Is it correct for this sample code?](https://discuss.ray.io/t/is-it-correct-for-this-sample-code/12156)
**Author:** [@Li\_Bin](https://discuss.ray.io/u/Li_Bin)\ **Replies:** 1\ **Last updated:** [September 25, 2023, 9:09pm UTC](https://discuss.ray.io/t/is-it-correct-for-this-sample-code/12156 "2023-09-25T21:09:08Z")
In the doc https://docs.ray.io/en/latest/ray-air/computer-vision.html where a CV sample is given to show how the AIR works for CV . Here is a code snippet, dataset = per\_epoch\_preprocessor.transform(dataset) train… --- ## [Ray data read hdfs slowly and process slowly](https://discuss.ray.io/t/ray-data-read-hdfs-slowly-and-process-slowly/11958)
**Author:** [@yiwei00000](https://discuss.ray.io/u/yiwei00000)\ **Replies:** 3\ **Last updated:** [August 31, 2023, 3:54pm UTC](https://discuss.ray.io/t/ray-data-read-hdfs-slowly-and-process-slowly/11958 "2023-08-31T15:54:16Z")
Our data volume is 2 million pieces, each with 4000 columns, totaling about 80G. The data is stored on hdfs. Ray cluster resources 6 cores and 24GB of memory per node. There are a total of three nodes. We use ray. data… --- ## [Running torch profiler](https://discuss.ray.io/t/running-torch-profiler/11289)
**Author:** [@asaiacai](https://discuss.ray.io/u/asaiacai)\ **Replies:** 5\ **Last updated:** [August 29, 2023, 6:16pm UTC](https://discuss.ray.io/t/running-torch-profiler/11289 "2023-08-29T18:16:31Z")
--- ## [How to use fraction GPU in \`ray.tune.Tuner\`?](https://discuss.ray.io/t/how-to-use-fraction-gpu-in-ray-tune-tuner/11863)
**Author:** [@Xinchengzelin](https://discuss.ray.io/u/Xinchengzelin)\ **Replies:** 6\ **Last updated:** [August 24, 2023, 1:03am UTC](https://discuss.ray.io/t/how-to-use-fraction-gpu-in-ray-tune-tuner/11863 "2023-08-24T01:03:09Z")
I use ray.tune.Tuner where the trainable is a TorchTrainer(the basic code structure as follows). I have 1 GPU, I want to run 2 tune trials at one GPU at the same time, so only 0.5GPU for one trial. I only find tune.wi… --- ## [Ray on spark support for windows?](https://discuss.ray.io/t/ray-on-spark-support-for-windows/11896)
**Author:** [@vj1210](https://discuss.ray.io/u/vj1210)\ **Replies:** 0\ **Last updated:** [August 22, 2023, 6:21pm UTC](https://discuss.ray.io/t/ray-on-spark-support-for-windows/11896 "2023-08-22T18:21:46Z")
How severe does this issue affect your experience of using Ray? High: It blocks me to complete my task. Hi, I’m trying to use the example provided here – Deploying on Spark Standalone cluster — Ray 2.6.1 It seems l… --- ## [Enable retries when training xgboot on ray](https://discuss.ray.io/t/enable-retries-when-training-xgboot-on-ray/11689)
**Author:** [@mlts](https://discuss.ray.io/u/mlts)\ **Replies:** 1\ **Last updated:** [August 9, 2023, 6:38pm UTC](https://discuss.ray.io/t/enable-retries-when-training-xgboot-on-ray/11689 "2023-08-09T18:38:30Z")
Hello, is there a way to enable retries when training xgboost on ray? I saw a lot of msg as below and I am wondering if enabling retries might fix some of them. A Ray actor died during training and the maximum number of… --- ## [🚀 Unleash the Power of Ray: Bring Your Own Model for Training and Fine-Tuning!](https://discuss.ray.io/t/unleash-the-power-of-ray-bring-your-own-model-for-training-and-fine-tuning/11626)
**Author:** [@Wendi](https://discuss.ray.io/u/Wendi)\ **Replies:** 0\ **Last updated:** [July 31, 2023, 10:49pm UTC](https://discuss.ray.io/t/unleash-the-power-of-ray-bring-your-own-model-for-training-and-fine-tuning/11626 "2023-07-31T22:49:07Z")
Hey everyone! :smile: Are you eager to unlock the potential of Ray and fine-tune your models with ease? We’re excited to announce a 1-hour session dedicated to exploring Ray’s newest feature for model training/fine-tun… --- ## [Incorrect steps calculation in GPT-J fine-tuning example](https://discuss.ray.io/t/incorrect-steps-calculation-in-gpt-j-fine-tuning-example/11415)
**Author:** [@amogkam](https://discuss.ray.io/u/amogkam)\ **Replies:** 3\ **Last updated:** [July 17, 2023, 5:43pm UTC](https://discuss.ray.io/t/incorrect-steps-calculation-in-gpt-j-fine-tuning-example/11415 "2023-07-17T17:43:50Z")
How severe does this issue affect your experience of using Ray? Low: It annoys or frustrates me for a moment. Hi, I’m running the GPT-J fine-tuning example: GPT-J-6B Fine-Tuning with Ray AIR and DeepSpeed — Ray 2.5.1.… --- ## [OOM when Passing Large Object to Ray Trainer Config](https://discuss.ray.io/t/oom-when-passing-large-object-to-ray-trainer-config/11358)
**Author:** [@pratkpranav](https://discuss.ray.io/u/pratkpranav)\ **Replies:** 2\ **Last updated:** [July 16, 2023, 2:38pm UTC](https://discuss.ray.io/t/oom-when-passing-large-object-to-ray-trainer-config/11358 "2023-07-16T14:38:01Z")
How severe does this issue affect your experience of using Ray? High: It blocks me to complete my task. We are working on building a distributed training framework using Ray Trainer. We are using the DataParallelTrain… --- ## [XGBoost on Ray can not find GPUs](https://discuss.ray.io/t/xgboost-on-ray-can-not-find-gpus/11218)
**Author:** [@mlts](https://discuss.ray.io/u/mlts)\ **Replies:** 3\ **Last updated:** [June 30, 2023, 6:37pm UTC](https://discuss.ray.io/t/xgboost-on-ray-can-not-find-gpus/11218 "2023-06-30T18:37:45Z")
Hello, we are having an issue running XGBoost on Ray on GCE GPU machines. What appears to us is that XGBoost can not find any GPU although we can see GPUs on nvidia-smi. We tried different GPU types and got same result. … --- ## [Failed to initialize Rabit when running XGBoost on Ray](https://discuss.ray.io/t/failed-to-initialize-rabit-when-running-xgboost-on-ray/10811)
**Author:** [@mlts](https://discuss.ray.io/u/mlts)\ **Replies:** 4\ **Last updated:** [June 8, 2023, 6:21am UTC](https://discuss.ray.io/t/failed-to-initialize-rabit-when-running-xgboost-on-ray/10811 "2023-06-08T06:21:38Z")
I kept getting the following err when running XGBoost on Ray. There is no additional err msg. What might be the cause? /build/build/ext/public/python/xgboost/1/7/3/build/python3.9/src/collective/rabit\_communicator.h:47:… [Previous page](https://discuss.ray.io/c/ray-train/17.md?page=1) [Next page](https://discuss.ray.io/c/ray-train/17.md?page=3)