# Ray Train **URL:** https://discuss.ray.io/c/ray-train/17.md [Latest](https://discuss.ray.io/latest.md) · [Categories](https://discuss.ray.io/categories.md) · [Tags](https://discuss.ray.io/tags.md) --- ## [About the Ray Train category](https://discuss.ray.io/t/about-the-ray-train-category/3383)
**Author:** [@rliaw](https://discuss.ray.io/u/rliaw)\ **Replies:** 0\ **Last updated:** [August 29, 2021, 4:56am UTC](https://discuss.ray.io/t/about-the-ray-train-category/3383 "2021-08-29T04:56:19Z")
For questions on distributed model training. Use Train to scale a model’s training process across CPUs, GPUs, or multiple nodes. Ray Train is a scalable machine learning library for distributed training and fine-tuning. … --- ## [Dynamic Resource Allocation](https://discuss.ray.io/t/dynamic-resource-allocation/23571)
**Author:** [@Saaketh](https://discuss.ray.io/u/Saaketh)\ **Replies:** 1\ **Last updated:** [June 25, 2026, 5:40pm UTC](https://discuss.ray.io/t/dynamic-resource-allocation/23571 "2026-06-25T17:40:05Z")
Hey there, I just had a quick question, does Ray have anything for automatically/dynamically distributing compute and resources during training? Such as detecting a device/clusters total resources and scaling up the para… --- ## [Only 10 ray data actor is working with ray train and the rest is just idle](https://discuss.ray.io/t/only-10-ray-data-actor-is-working-with-ray-train-and-the-rest-is-just-idle/23568)
**Author:** [@wymli](https://discuss.ray.io/u/wymli)\ **Replies:** 18\ **Last updated:** [June 12, 2026, 5:07pm UTC](https://discuss.ray.io/t/only-10-ray-data-actor-is-working-with-ray-train-and-the-rest-is-just-idle/23568 "2026-06-12T17:07:36Z")
1. Severity of the issue: (select one) High: Completely blocks me. 2. Environment: Ray version: 2.51.2 Python version: py3.9 OS: Cloud/Infrastructure: Other libs/tools (if relevant): 3. What happened vs. what you e… --- ## [Error with "column\_names" when using Ray with TRL's sft\_trainer](https://discuss.ray.io/t/error-with-column-names-when-using-ray-with-trls-sft-trainer/23404)
**Author:** [@markoov](https://discuss.ray.io/u/markoov)\ **Replies:** 6\ **Last updated:** [June 2, 2026, 5:59pm UTC](https://discuss.ray.io/t/error-with-column-names-when-using-ray-with-trls-sft-trainer/23404 "2026-06-02T17:59:39Z")
1. Severity of the issue: (select one) Medium: Significantly affects my productivity but can find a workaround. 2. Environment: Ray version: 2.50.0 Python version: 3.11.10 Trl: 0.25.1 transformers: 4.57.1 3. What h… --- ## [Are there any hacks to use nsys in Ray?](https://discuss.ray.io/t/are-there-any-hacks-to-use-nsys-in-ray/8097)
**Author:** [@yzs](https://discuss.ray.io/u/yzs)\ **Replies:** 12\ **Last updated:** [April 13, 2026, 4:04pm UTC](https://discuss.ray.io/t/are-there-any-hacks-to-use-nsys-in-ray/8097 "2026-04-13T16:04:12Z")
How severe does this issue affect your experience of using Ray? High: It blocks me to complete my task. It is very common to use NVIDIA nsight system/compute for GPU related workloads profiling. However, AFAIK, nsys … --- ## [VScode breakpoint will be bypassed even with local\_mode=True](https://discuss.ray.io/t/vscode-breakpoint-will-be-bypassed-even-with-local-mode-true/8493)
**Author:** [@heng2j](https://discuss.ray.io/u/heng2j)\ **Replies:** 7\ **Last updated:** [February 13, 2026, 5:47pm UTC](https://discuss.ray.io/t/vscode-breakpoint-will-be-bypassed-even-with-local-mode-true/8493 "2026-02-13T17:47:02Z")
How severe does this issue affect your experience of using Ray? Medium: It contributes to significant difficulty to complete my task, but I can work around it. I am writing a DL training framework that leverages Ray … --- ## [How to enable debug logs from Ray train's internal checkpoint manager?](https://discuss.ray.io/t/how-to-enable-debug-logs-from-ray-trains-internal-checkpoint-manager/23467)
**Author:** [@psnilesh](https://discuss.ray.io/u/psnilesh)\ **Replies:** 3\ **Last updated:** [February 5, 2026, 6:16am UTC](https://discuss.ray.io/t/how-to-enable-debug-logs-from-ray-trains-internal-checkpoint-manager/23467 "2026-02-05T06:16:56Z")
Ray is deleting checkpoints from s3 for some reason and I need to know why --- ## [Model Parallelism in Ray](https://discuss.ray.io/t/model-parallelism-in-ray/4617)
**Author:** [@Vipul\_Gupta](https://discuss.ray.io/u/Vipul_Gupta)\ **Replies:** 10\ **Last updated:** [January 9, 2026, 4:07am UTC](https://discuss.ray.io/t/model-parallelism-in-ray/4617 "2026-01-09T04:07:10Z")
Hi folks, it seems like Ray Train focuses on distributed training with data parallelism. I am wondering if there is a use case with model parallelism. In our specific use-case, we are training large-scale embeddings, and… --- ## [How to integrate Megatron-Core with Ray Train v2 for large language model training?](https://discuss.ray.io/t/how-to-integrate-megatron-core-with-ray-train-v2-for-large-language-model-training/23443)
**Author:** [@psnilesh](https://discuss.ray.io/u/psnilesh)\ **Replies:** 3\ **Last updated:** [January 9, 2026, 3:56am UTC](https://discuss.ray.io/t/how-to-integrate-megatron-core-with-ray-train-v2-for-large-language-model-training/23443 "2026-01-09T03:56:57Z")
I need to train 70B+ parameter models using Megatron-Core’s tensor/pipeline parallelism, but want Ray Train v2’s orchestration. My guess is this isn’t supported because: Ray Train’s TorchConfig assumes data parallelis… --- ## [What are the advantages of calling \`ray.train.report\` with upload\_mode as \`NO\_UPLOAD\`?](https://discuss.ray.io/t/what-are-the-advantages-of-calling-ray-train-report-with-upload-mode-as-no-upload/23429)
**Author:** [@psnilesh](https://discuss.ray.io/u/psnilesh)\ **Replies:** 5\ **Last updated:** [January 1, 2026, 7:46pm UTC](https://discuss.ray.io/t/what-are-the-advantages-of-calling-ray-train-report-with-upload-mode-as-no-upload/23429 "2026-01-01T19:46:08Z")
I’ve been reading checkpointing capabilities of Ray Saving and Loading Checkpoints — Ray 2.53.0 , especially custom checkpointing. I understand how ray.train.report with ASYNC upload mode can improve training goodput. H… --- ## [How to quickly check how many times a TorchTrainer job has restarted?](https://discuss.ray.io/t/how-to-quickly-check-how-many-times-a-torchtrainer-job-has-restarted/23424)
**Author:** [@psnilesh](https://discuss.ray.io/u/psnilesh)\ **Replies:** 3\ **Last updated:** [December 29, 2025, 10:26pm UTC](https://discuss.ray.io/t/how-to-quickly-check-how-many-times-a-torchtrainer-job-has-restarted/23424 "2025-12-29T22:26:40Z")
Following the guide, Handling Failures and Node Preemption — Ray 2.53.0 I have setup worker level fault tolerance like so, trainer = ray.train.torch.TorchTrainer( train\_fn\_per\_worker, scaling\_config=ray.train.… --- ## [High GPU Memory (DeepSpeed+HuggingFace+Ray)](https://discuss.ray.io/t/high-gpu-memory-deepspeed-huggingface-ray/23408)
**Author:** [@markoov](https://discuss.ray.io/u/markoov)\ **Replies:** 2\ **Last updated:** [December 23, 2025, 10:19am UTC](https://discuss.ray.io/t/high-gpu-memory-deepspeed-huggingface-ray/23408 "2025-12-23T10:19:06Z")
1. Severity of the issue: (select one) None: I’m just curious or want clarification. Low: Annoying but doesn’t hinder my work. Medium: Significantly affects my productivity but can find a workaround. High: Comple… --- ## [Training issues with MultiworkerMirroredStrategy](https://discuss.ray.io/t/training-issues-with-multiworkermirroredstrategy/23335)
**Author:** [@0x6A4B](https://discuss.ray.io/u/0x6A4B)\ **Replies:** 5\ **Last updated:** [November 25, 2025, 6:00pm UTC](https://discuss.ray.io/t/training-issues-with-multiworkermirroredstrategy/23335 "2025-11-25T18:00:46Z")
1. Severity of the issue: (select one) None: I’m just curious or want clarification. Low: Annoying but doesn’t hinder my work. Medium: Significantly affects my productivity but can find a workaround. High: Comple… --- ## [Long initialization time to initialize\_session with large scale dataset](https://discuss.ray.io/t/long-initialization-time-to-initialize-session-with-large-scale-dataset/21545)
**Author:** [@Tairui\_Wang](https://discuss.ray.io/u/Tairui_Wang)\ **Replies:** 5\ **Last updated:** [October 7, 2025, 7:02pm UTC](https://discuss.ray.io/t/long-initialization-time-to-initialize-session-with-large-scale-dataset/21545 "2025-10-07T19:02:14Z")
How severe does this issue affect your experience of using Ray? Medium: It contributes to significant difficulty to complete my task, but I can work around it. Issue I am working on distributed training with Ray. The … --- ## [Help needed with a simple demo](https://discuss.ray.io/t/help-needed-with-a-simple-demo/23109)
**Author:** [@JinLiang](https://discuss.ray.io/u/JinLiang)\ **Replies:** 1\ **Last updated:** [September 22, 2025, 11:50pm UTC](https://discuss.ray.io/t/help-needed-with-a-simple-demo/23109 "2025-09-22T23:50:35Z")
1. Severity of the issue: (select one) High: Completely blocks me. 2. Environment: Ray version: ray, version 2.49.1 Python version: 3.9 OS: Linux iZuf69e0i0z4iawhy8o915Z 6.8.0-60-generic #63-Ubuntu SMP PREEMPT\_DYNAMI… --- ## [How to get dataset shard size in each train worker](https://discuss.ray.io/t/how-to-get-dataset-shard-size-in-each-train-worker/23094)
**Author:** [@ypp](https://discuss.ray.io/u/ypp)\ **Replies:** 0\ **Last updated:** [September 11, 2025, 8:16am UTC](https://discuss.ray.io/t/how-to-get-dataset-shard-size-in-each-train-worker/23094 "2025-09-11T08:16:41Z")
1. Severity of the issue: (select one) Low: Annoying but doesn’t hinder my work. 2. Environment: Ray version: 2.46.0 Python version: 3.12 OS: wsl2 ubuntu 3. What happened vs. what you expected: Expected: handy me… --- ## [What is the correct way of using get\_dataset\_shard?](https://discuss.ray.io/t/what-is-the-correct-way-of-using-get-dataset-shard/23093)
**Author:** [@mitul93](https://discuss.ray.io/u/mitul93)\ **Replies:** 0\ **Last updated:** [September 11, 2025, 7:59am UTC](https://discuss.ray.io/t/what-is-the-correct-way-of-using-get-dataset-shard/23093 "2025-09-11T07:59:42Z")
1. Severity of the issue: (select one) Low: Annoying but doesn’t hinder my work. 2. Environment: Ray version: 2.49.0 Python version: 3.10 OS: Ubuntu 24.04 Cloud/Infrastructure: AWS Other libs/tools (if relevant): W… --- ## [Training loop stuck at StreamSplitDataIterator](https://discuss.ray.io/t/training-loop-stuck-at-streamsplitdataiterator/23069)
**Author:** [@mitul93](https://discuss.ray.io/u/mitul93)\ **Replies:** 1\ **Last updated:** [September 5, 2025, 1:52pm UTC](https://discuss.ray.io/t/training-loop-stuck-at-streamsplitdataiterator/23069 "2025-09-05T13:52:53Z")
1. Severity of the issue: (select one) High: Completely blocks me. 2. Environment: Ray version: 2.49.0 Python version: 3.10.12 OS: Ubuntu 24.04 Cloud/Infrastructure: AWS Other libs/tools (if relevant): 3. What happ… --- ## [\_collate\_fn argument removed from ray.data.DataIterator.iter\_batches](https://discuss.ray.io/t/collate-fn-argument-removed-from-ray-data-dataiterator-iter-batches/23052)
**Author:** [@mitul93](https://discuss.ray.io/u/mitul93)\ **Replies:** 2\ **Last updated:** [August 28, 2025, 8:52am UTC](https://discuss.ray.io/t/collate-fn-argument-removed-from-ray-data-dataiterator-iter-batches/23052 "2025-08-28T08:52:28Z")
Hi I noticed that \_collate\_fn argument was removed from ray.data.DataIterator.iter\_batches in 2.47.0 release. My understanding was that ray.train.get\_dataset\_shard should be used to iterate through batches of data when… --- ## [Ray Tensorflow/Pytorch trainer metrics](https://discuss.ray.io/t/ray-tensorflow-pytorch-trainer-metrics/23009)
**Author:** [@gauravsuman](https://discuss.ray.io/u/gauravsuman)\ **Replies:** 0\ **Last updated:** [August 18, 2025, 8:58am UTC](https://discuss.ray.io/t/ray-tensorflow-pytorch-trainer-metrics/23009 "2025-08-18T08:58:49Z")
Environment: Ray version: 2.48.0 Python version: 3.12 OS: Linux Cloud/Infrastructure: GCP Other libs/tools (if relevant): I am working on distributed training on ray using TorchTrainer and TensorflowTrainer.I am tryin… --- ## [Save log for train v2 in specify dir](https://discuss.ray.io/t/save-log-for-train-v2-in-specify-dir/22858)
**Author:** [@richardrex.xu](https://discuss.ray.io/u/richardrex.xu)\ **Replies:** 1\ **Last updated:** [July 21, 2025, 5:16pm UTC](https://discuss.ray.io/t/save-log-for-train-v2-in-specify-dir/22858 "2025-07-21T17:16:06Z")
1. Severity of the issue: (select one) None: I’m just curious or want clarification. Low: Annoying but doesn’t hinder my work. Medium: Significantly affects my productivity but can find a workaround. High: Comple… --- ## [TorchDiagGaussian from logits](https://discuss.ray.io/t/torchdiaggaussian-from-logits/22568)
**Author:** [@H-Park](https://discuss.ray.io/u/H-Park)\ **Replies:** 4\ **Last updated:** [June 5, 2025, 6:41pm UTC](https://discuss.ray.io/t/torchdiaggaussian-from-logits/22568 "2025-06-05T18:41:50Z")
For box action spaces, are the logits in the form \[mean1, std1, mean2, std2, …\] or \[mean1, mean2, …, std1, std2, …\] ? --- ## [\[Ray Train\] XGBoostTrainer crashes with ActorDiedError when using num\_workers \> 1 and use\_gpu=False](https://discuss.ray.io/t/ray-train-xgboosttrainer-crashes-with-actordiederror-when-using-num-workers-1-and-use-gpu-false/22562)
**Author:** [@OmarAlmusa](https://discuss.ray.io/u/OmarAlmusa)\ **Replies:** 0\ **Last updated:** [May 26, 2025, 8:38pm UTC](https://discuss.ray.io/t/ray-train-xgboosttrainer-crashes-with-actordiederror-when-using-num-workers-1-and-use-gpu-false/22562 "2025-05-26T20:38:05Z")
Severity of the issue: High Environment: Ray version: 2.44.1 Python version: 3.10.16 OS: WSL Cloud/Infrastructure: none Other libs/tools (if relevant): xgboost 3.0.0 Ray resources: 0.0/28.0 CPU 0.0/1.0 GPU 0B/11.9… --- ## [How to report loss when using more than one worker?](https://discuss.ray.io/t/how-to-report-loss-when-using-more-than-one-worker/22513)
**Author:** [@mitul93](https://discuss.ray.io/u/mitul93)\ **Replies:** 2\ **Last updated:** [May 20, 2025, 6:17am UTC](https://discuss.ray.io/t/how-to-report-loss-when-using-more-than-one-worker/22513 "2025-05-20T06:17:09Z")
1. Severity of the issue: (select one) Low: Annoying but doesn’t hinder my work. 2. Environment: Ray version: 2.45 Python version: 3.10 OS: Ubuntu 24.04 Cloud/Infrastructure: AWS I’m exploring ray and I’m very new … --- ## [Ray train job gets killed with no errors!](https://discuss.ray.io/t/ray-train-job-gets-killed-with-no-errors/11831)
**Author:** [@milad\_heidari](https://discuss.ray.io/u/milad_heidari)\ **Replies:** 3\ **Last updated:** [May 19, 2025, 5:53am UTC](https://discuss.ray.io/t/ray-train-job-gets-killed-with-no-errors/11831 "2025-05-19T05:53:36Z")
I tried to run a single XGBoost training job but the job gets killed without any errors/exceptions. checked the logs but it seems nothing is wrong. How should I find the problem? Is there any way to increase the log leve… --- ## [XGBoostTrainer crashes with ActorDiedError when using num\_workers \> 1 and use\_gpu=False](https://discuss.ray.io/t/xgboosttrainer-crashes-with-actordiederror-when-using-num-workers-1-and-use-gpu-false/22507)
**Author:** [@OmarAlmusa](https://discuss.ray.io/u/OmarAlmusa)\ **Replies:** 0\ **Last updated:** [May 18, 2025, 7:10pm UTC](https://discuss.ray.io/t/xgboosttrainer-crashes-with-actordiederror-when-using-num-workers-1-and-use-gpu-false/22507 "2025-05-18T19:10:43Z")
Severity of the issue: High Environment: Ray version: 2.44.1 Python version: 3.10.16 OS: WSL Cloud/Infrastructure: none Other libs/tools (if relevant): xgboost 3.0.0 Ray resources: 0.0/28.0 CPU 0.0/1.0 GPU 0B/11.9… --- ## [WorkerCrashedError: The worker died unexpectedly while executing this task. Check python-core-worker-\*.log files for more information](https://discuss.ray.io/t/workercrashederror-the-worker-died-unexpectedly-while-executing-this-task-check-python-core-worker-log-files-for-more-information/22506)
**Author:** [@OmarAlmusa](https://discuss.ray.io/u/OmarAlmusa)\ **Replies:** 0\ **Last updated:** [May 18, 2025, 6:38pm UTC](https://discuss.ray.io/t/workercrashederror-the-worker-died-unexpectedly-while-executing-this-task-check-python-core-worker-log-files-for-more-information/22506 "2025-05-18T18:38:46Z")
Severity of the issue: High Environment: Ray version: 2.44.1 Python version: 3.10.16 OS: WSL Cloud/Infrastructure: - Other libs/tools (if relevant): Name: xgboost Version: 3.0.0 My Rig shown in Ray status: 0.0/28… --- ## [OSError when saving checkpoint with ray.train.lightning.RayTrainReportCallback](https://discuss.ray.io/t/oserror-when-saving-checkpoint-with-ray-train-lightning-raytrainreportcallback/21905)
**Author:** [@JmPearl](https://discuss.ray.io/u/JmPearl)\ **Replies:** 6\ **Last updated:** [May 7, 2025, 10:06pm UTC](https://discuss.ray.io/t/oserror-when-saving-checkpoint-with-ray-train-lightning-raytrainreportcallback/21905 "2025-05-07T22:06:49Z")
How severe does this issue affect your experience of using Ray? High: It blocks me to complete my task. The following error continues to prevent me from training a model with checkpointing. OSError: \[Errno 22\] Error… --- ## [Init device mesh in pytorch distributed](https://discuss.ray.io/t/init-device-mesh-in-pytorch-distributed/22371)
**Author:** [@navmarri1](https://discuss.ray.io/u/navmarri1)\ **Replies:** 2\ **Last updated:** [April 26, 2025, 6:08pm UTC](https://discuss.ray.io/t/init-device-mesh-in-pytorch-distributed/22371 "2025-04-26T18:08:48Z")
1. Severity of the issue: (select one) None: I’m just curious or want clarification. Low: Annoying but doesn’t hinder my work. Medium: Significantly affects my productivity but can find a workaround. High: Comple… --- ## [Ray Train V2 with Ray Tune does not start another trial after a training run is TERMINATED](https://discuss.ray.io/t/ray-train-v2-with-ray-tune-does-not-start-another-trial-after-a-training-run-is-terminated/22330)
**Author:** [@Logan\_Kinch](https://discuss.ray.io/u/Logan_Kinch)\ **Replies:** 3\ **Last updated:** [April 17, 2025, 7:51pm UTC](https://discuss.ray.io/t/ray-train-v2-with-ray-tune-does-not-start-another-trial-after-a-training-run-is-terminated/22330 "2025-04-17T19:51:41Z")
1. Severity of the issue: (select one) None: I’m just curious or want clarification. Low: Annoying but doesn’t hinder my work. Medium: Significantly affects my productivity but can find a workaround. High: Comple… [Next page](https://discuss.ray.io/c/ray-train/17.md?page=1)