# Ray train job gets killed with no errors!

**URL:** <https://discuss.ray.io/t/ray-train-job-gets-killed-with-no-errors/11831>\
**Category:** Ray Train\
**Created:** [August 17, 2023, 11:20am UTC](https://discuss.ray.io/t/ray-train-job-gets-killed-with-no-errors/11831 "2023-08-17T11:20:20Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![milad\_heidari](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/milad_heidari/32/4743_2.png) [@milad\_heidari](https://discuss.ray.io/u/milad_heidari)\
**Post date:** [August 17, 2023, 11:20am UTC](https://discuss.ray.io/t/ray-train-job-gets-killed-with-no-errors/11831/1 "2023-08-17T11:20:20Z")

</div>

I tried to run a single XGBoost training job but the job gets killed without any errors/exceptions. checked the logs but it seems nothing is wrong. How should I find the problem? Is there any way to increase the log level?  
My script and screenshots from the terminal and Ray’s dashboard are attached.

Code:

```auto
# Load data.
dataset = ray.data.read_csv("s3://anonymous@air-example-data/breast_cancer.csv")

# Split data into train and validation.
train_dataset, valid_dataset = dataset.train_test_split(test_size=0.3)

scaling_config = ScalingConfig(
    # Number of workers to use for data parallelism.
    num_workers=2,
    # Whether to use GPU acceleration.
    use_gpu=False,
)

sync_config = SyncConfig(
    syncer=CustomCommandSyncer(
        sync_up_template="aws s3 --endpoint-url=$AWS_ENDPOINT_URL sync {source} {target}",
        sync_down_template="aws s3 --endpoint-url=$AWS_ENDPOINT_URL sync {source} {target}",
        delete_template="aws s3 --endpoint-url=$AWS_ENDPOINT_URL rm {target} --recursive",
    ),
)

trainer = XGBoostTrainer(
    scaling_config=scaling_config,
    run_config=RunConfig(
        sync_config=sync_config,
        verbose=1,
        storage_path="s3://mlops/milad/"
    ),
    label_column="target",
    num_boost_round=10,
    params={
        # XGBoost specific params
        "objective": "binary:logistic",
        # "tree_method": "gpu_hist", # uncomment this to use GPU for training
        "eval_metric": ["logloss", "error"],
    },
    datasets={"train": train_dataset, "valid": valid_dataset},
)

result = trainer.fit()

print(result.metrics)

```

Terminal:  
[https://mega.nz/file/XJkjyTYb#8AFhRnFTxzojkUIHlkE-wW8q-qRSeFnE7CVJaDVOA\_A](https://mega.nz/file/XJkjyTYb#8AFhRnFTxzojkUIHlkE-wW8q-qRSeFnE7CVJaDVOA_A)

Dashboard:  
[https://mega.nz/file/mMNmTKDC#ePShAMUO-WCtcOkyx8zCsys9wQqT8N6DhjFP0J8ejTs](https://mega.nz/file/mMNmTKDC#ePShAMUO-WCtcOkyx8zCsys9wQqT8N6DhjFP0J8ejTs)

---

<div class="post-metadata">

**Author:** ![matthewdeng](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/matthewdeng/32/1446_2.png) [@matthewdeng](https://discuss.ray.io/u/matthewdeng)\
**Post date:** [August 19, 2023, 4:20pm UTC](https://discuss.ray.io/t/ray-train-job-gets-killed-with-no-errors/11831/2 "2023-08-19T16:20:16Z")

</div>

Hey it’s not clear to me from the logs that you’ve shown - can you copy/paste the full log from the Dashboard view?

---

<div class="post-metadata">

**Author:** ![milad\_heidari](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/milad_heidari/32/4743_2.png) [@milad\_heidari](https://discuss.ray.io/u/milad_heidari)\
**Post date:** [August 19, 2023, 4:42pm UTC](https://discuss.ray.io/t/ray-train-job-gets-killed-with-no-errors/11831/3 "2023-08-19T16:42:00Z")

</div>

Update

Thanks for the response.  
The problem was insufficient memory of the head node. After I increased the memory of the head node, the problem was solved. I wonder how can we detect such problems. The logs on the dashboard don’t say anything about the memory shortage or insufficient CPU cores.

---

<div class="post-metadata">

**Author:** ![OmarAlmusa](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/omaralmusa/32/7963_2.png) [@OmarAlmusa](https://discuss.ray.io/u/OmarAlmusa)\
**Post date:** [May 19, 2025, 5:53am UTC](https://discuss.ray.io/t/ray-train-job-gets-killed-with-no-errors/11831/4 "2025-05-19T05:53:36Z")

</div>

how did you increase the memory of the head node?