# Simple hello\_world example crashes badly

**URL:** <https://discuss.ray.io/t/simple-hello-world-example-crashes-badly/13272>\
**Category:** Ray Core\
**Created:** [December 27, 2023, 11:02pm UTC](https://discuss.ray.io/t/simple-hello-world-example-crashes-badly/13272 "2023-12-27T23:02:26Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Florin\_Andrei](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/florin_andrei/32/5545_2.png) [@Florin\_Andrei](https://discuss.ray.io/u/Florin_Andrei)\
**Post date:** [December 27, 2023, 11:02pm UTC](https://discuss.ray.io/t/simple-hello-world-example-crashes-badly/13272/1 "2023-12-27T23:02:26Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- High: It blocks me to complete my task.

I’m just getting started with Ray, and I’m comparing it with Optuna. I’m trying to run a trivial example - minimizing the Rastrigin function.

```python
import numpy as np
from ray import train, tune
from ray.tune.search.bayesopt import BayesOptSearch
import os

search_space = {
    "x": tune.uniform(-5.12, 5.12),
    "y": tune.uniform(-5.12, 5.12),
}

def rastrigin(config):
    # the Rastrigin function
    score = (
        config['x'] ** 2
        - 10 * np.cos(2 * np.pi * config['x'])
        + config['y'] ** 2
        - 10 * np.cos(2 * np.pi * config['y'])
        + 20
    )
    return {"score": score}

bayes_search = BayesOptSearch(metric='score', mode='min')
# tune_config = tune.TuneConfig(search_alg=bayes_search, num_samples=-1, max_concurrent_trials=os.cpu_count(), time_budget_s=10)
tune_config = tune.TuneConfig(mode='min', metric='score', num_samples=-1, max_concurrent_trials=os.cpu_count(), time_budget_s=10)

run_config = train.RunConfig(verbose=0)
tuner = tune.Tuner(rastrigin, param_space=search_space, tune_config=tune_config, run_config=run_config)

results = tuner.fit()

```

Running this simple code generates a huge error:

> <https://gist.github.com/FlorinAndrei/c4b9fed2e2bf63998e521526bb8b49e2>

Enabling Bayesian search fixes it temporarily, but if I increase the time budget to 60, it crashes the same way.

Ubuntu 22.04, Python 3.11.7, the latest versions of all relevant Python modules including Ray. I’ve tried to install Ray a couple different ways, but it makes no difference:

```auto
pip install --user ray

pip install --user "ray[all]"

```

I’m not trying to do any rocket science here, just following the simplest examples from the website, but it just doesn’t run properly. ☹

Would be nice if this project had a solid hello-world example, in a prominent place on the website, that actually works. This would help people like me who are completely new to the project.

---

<div class="post-metadata">

**Author:** ![fordaz](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/fordaz/32/5549_2.png) [@fordaz](https://discuss.ray.io/u/fordaz)\
**Post date:** [December 28, 2023, 9:22pm UTC](https://discuss.ray.io/t/simple-hello-world-example-crashes-badly/13272/2 "2023-12-28T21:22:04Z")

</div>

I played a little bit with this code, and was able to reproduce the error. After just changing the following parameter `num_samples=1` instead of -1, the code seems to be running fine.

I don’t know the root cause of this error, probably a bug, but I wanted to share this in case this helps and gets you unblocked.

---

<div class="post-metadata">

**Author:** ![Florin\_Andrei](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/florin_andrei/32/5545_2.png) [@Florin\_Andrei](https://discuss.ray.io/u/Florin_Andrei)\
**Post date:** [December 28, 2023, 11:52pm UTC](https://discuss.ray.io/t/simple-hello-world-example-crashes-badly/13272/3 "2023-12-28T23:52:56Z")

</div>

I’ve tried that, but then only 1 trial is performed. What I want is to run as many trials as possible within a given time (10 sec).

---

<div class="post-metadata">

**Author:** ![fordaz](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/fordaz/32/5549_2.png) [@fordaz](https://discuss.ray.io/u/fordaz)\
**Post date:** [December 29, 2023, 12:16am UTC](https://discuss.ray.io/t/simple-hello-world-example-crashes-badly/13272/4 "2023-12-29T00:16:47Z")

</div>

yeah It makes sense. Just in case, using 10K or 100K seem to work as well, but I see your point. I’m also getting started Ray, let’s see what other folks say.

---

<div class="post-metadata">

**Author:** ![Florin\_Andrei](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/florin_andrei/32/5545_2.png) [@Florin\_Andrei](https://discuss.ray.io/u/Florin_Andrei)\
**Post date:** [December 29, 2023, 12:59am UTC](https://discuss.ray.io/t/simple-hello-world-example-crashes-badly/13272/5 "2023-12-29T00:59:29Z")

</div>

I’m also getting the error if I try to do a more substantial search using OptunaSearch.

```python
import numpy as np
from ray import train, tune
from ray.tune.search.optuna import OptunaSearch

def rastrigin(config):
    score = (
        config['x'] ** 2
        - 10 * np.cos(2 * np.pi * config['x'])
        + config['y'] ** 2
        - 10 * np.cos(2 * np.pi * config['y'])
        + 20
    )
    return {"score": score}

search_space = {
    "x": tune.uniform(-5.12, 5.12),
    "y": tune.uniform(-5.12, 5.12),
}

optuna_search = OptunaSearch(metric='score', mode='min')
tune_config = tune.TuneConfig(search_alg=optuna_search, num_samples=-1, time_budget_s=20)
run_config = train.RunConfig(name='rastrigin', verbose=0)

tuner = tune.Tuner(rastrigin, param_space=search_space, tune_config=tune_config, run_config=run_config)
results = tuner.fit()

```

Using a large integer value for num\_samples does not fix it.

The error:

```auto
2023-12-28 16:54:42,007	ERROR worker.py:405 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): The worker died unexpectedly while executing this task. Check python-core-worker-*.log files for more information.
(bundle_reservation_check_func pid=664883) Traceback (most recent call last):
(bundle_reservation_check_func pid=664883) File "python/ray/_raylet.pyx", line 1788, in ray._raylet.execute_task
(bundle_reservation_check_func pid=664883) File "python/ray/_raylet.pyx", line 1790, in ray._raylet.execute_task
(bundle_reservation_check_func pid=664883) File "/home/florin/.local/lib/python3.11/site-packages/ray/_private/worker.py", line 790, in deserialize_objects
(bundle_reservation_check_func pid=664883) context = self.get_serialization_context()
(bundle_reservation_check_func pid=664883) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=664883) File "/home/florin/.local/lib/python3.11/site-packages/ray/_private/worker.py", line 678, in get_serialization_context
(bundle_reservation_check_func pid=664883) context_map[job_id] = serialization.SerializationContext(self)
(bundle_reservation_check_func pid=664883) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=664883) File "/home/florin/.local/lib/python3.11/site-packages/ray/_private/serialization.py", line 153, in __init__
(bundle_reservation_check_func pid=664883) serialization_addons.apply(self)
(bundle_reservation_check_func pid=664883) File "/home/florin/.local/lib/python3.11/site-packages/ray/util/serialization_addons.py", line 29, in apply
(bundle_reservation_check_func pid=664883) from ray._private.pydantic_compat import register_pydantic_serializers
(bundle_reservation_check_func pid=664883) File "/home/florin/.local/lib/python3.11/site-packages/ray/_private/pydantic_compat.py", line 2, in <module>
(bundle_reservation_check_func pid=664883) from pkg_resources import packaging
(bundle_reservation_check_func pid=664883) File "/home/florin/.local/lib/python3.11/site-packages/pkg_resources/ __init__.py", line 31, in <module>
(bundle_reservation_check_func pid=664883) import pkgutil
(bundle_reservation_check_func pid=664883) File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
(bundle_reservation_check_func pid=664883) File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
(bundle_reservation_check_func pid=664883) File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
(bundle_reservation_check_func pid=664883) File "<frozen importlib._bootstrap_external>", line 936, in exec_module
(bundle_reservation_check_func pid=664883) File "<frozen importlib._bootstrap_external>", line 1069, in get_code
(bundle_reservation_check_func pid=664883) File "<frozen importlib._bootstrap_external>", line 729, in _compile_bytecode
(bundle_reservation_check_func pid=664883) File "/home/florin/.local/lib/python3.11/site-packages/ray/_private/worker.py", line 841, in sigterm_handler
(bundle_reservation_check_func pid=664883) raise_sys_exit_with_custom_error_message(
(bundle_reservation_check_func pid=664883) File "python/ray/_raylet.pyx", line 846, in ray._raylet.raise_sys_exit_with_custom_error_message
(bundle_reservation_check_func pid=664883) SystemExit: 1

```

---

<div class="post-metadata">

**Author:** ![fordaz](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/fordaz/32/5549_2.png) [@fordaz](https://discuss.ray.io/u/fordaz)\
**Post date:** [December 29, 2023, 3:58am UTC](https://discuss.ray.io/t/simple-hello-world-example-crashes-badly/13272/6 "2023-12-29T03:58:04Z")

</div>

Nothing concrete yet, just a few observations/hints:

- Looking at this log `/tmp/ray/session_latest/logs/raylet.out`  
I noticed “main.cc:372: Raylet received SIGTERM”, so I get the impression the OS is killing the ray workers, probably due to memory/resource exhaustion.

- There are other logs in the same folder, but I could not find any specific errors, which makes sense if the processes are being killed.

- I’m planning to try the suggestions in this [page](https://docs.ray.io/en/latest/ray-observability/user-guides/debug-apps/debug-memory.html#troubleshooting-out-of-memory-how-to-detect) in more detail.

- It seems that [high parallelism](https://docs.ray.io/en/latest/ray-observability/user-guides/debug-apps/debug-memory.html#reduce-parallelism) can trigger OOM errors, maybe a large `num_samples` for HPO is exceeding the capacity of our local machines, or maybe it needs proper configs.

- I’m using macos, so the memory monitor is not running, as it is only supported in Linux, I wonder if you can use it in Linux and get additional information.

- Running the code with a low number of `num_samples` seems to work, only after increasing it the error starts to show, which makes sense if the issue is about resource exhaustion.

---

<div class="post-metadata">

**Author:** ![Florin\_Andrei](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/florin_andrei/32/5545_2.png) [@Florin\_Andrei](https://discuss.ray.io/u/Florin_Andrei)\
**Post date:** [December 29, 2023, 4:16am UTC](https://discuss.ray.io/t/simple-hello-world-example-crashes-badly/13272/7 "2023-12-29T04:16:39Z")

</div>

Good to know!

At least in my case, it’s not exhausting RAM. I have 64 GB of RAM on this machine, and 75% of it is unused while the code is running. I run `htop` all the time. I’m pretty sure `htop` is available on MacOS via Brew.

I need to be able to generate 10k trials total, using all CPUs, as fast as possible - lowering that limit is not an option for me.

Using multiple instances of Optuna (one for each CPU core) in joblib is actually faster for me. For a different problem (not the Rastrigin function, but an actual model optimization), I get 10k trials in 5 minutes with Optuna, with the MySQL storage. With Ray, I only get about 8k. I was hoping Ray would be faster.

I’ll also try to investigate some more tomorrow.