# Error occurs under high memory use **URL:** \ **Category:** Ray Core\ **Created:** [July 23, 2022, 9:35am UTC](https://discuss.ray.io/t/error-occurs-under-high-memory-use/6934 "2022-07-23T09:35:12Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![mathew](https://avatars.discourse-cdn.com/v4/letter/m/5daacb/32.png) [@mathew](https://discuss.ray.io/u/mathew)\ **Post date:** [July 23, 2022, 9:35am UTC](https://discuss.ray.io/t/error-occurs-under-high-memory-use/6934/1 "2022-07-23T09:35:12Z")
Hi, - None: Just asking a question out of curiosity I am new to Ray. I was testing it out and got the following error. The error only occurs under high memory use. System I am using Microsoft Windows 10 Pro Version 10.0.19044 Build 19044 Python 3.7.9; ray==1.13.0 Intel Xeon E5-2630 v4 @2.2GHz \* 2 Physical memory 64 GB, Virtual memory = 110GB (SSD) The code is given below. import numpy as np import os import time import ray def poly2d(x,y,z,order): A = for i in range(order+1): for j in range(i+1): A.append(x\*\*(i-j) \* y\*\*(j)) A = np.array(A).T coeff, r, rank, s = np.linalg.lstsq(A, z, rcond=None) return coeff def polyval2d(x,y,coeff,order): z = 0 count = 0 for i in range(order+1): for j in range(i+1): z = z + x\*\*(i-j) \* y\*\*(j) \* coeff[count] count += 1 return z @ray.remote def calc(): ni = 500 [x,y] = np.meshgrid(np.linspace(-1,1,ni),np.linspace(-1,1,ni)) z = np.sqrt(x **2 + y** 2) + np.random.normal(0,0.1,(ni,ni)) coeff = poly2d(x.flatten(),y.flatten(),z.flatten(),18) z1 = polyval2d(x.flatten(),y.flatten(),coeff,18) z1 = z1.reshape(ni,ni) return [np.sqrt(np.mean((z-z1)\*\*2))] def test(): start\_time = time.time() results = ray.get([calc.remote() for \_ in range(os.cpu\_count())]) duration = time.time() - start\_time print(‘Remote execution time: {}’.format(duration)) ``` print(results) ``` try: ray.init() test() ray.shutdown() except Exception: ray.shutdown() The error I am getting. (pid=) [2022-07-23 14:47:29,885 E 11508 13272] (raylet.exe) agent\_manager.cc:107: The raylet exited immediately because the Ray agent failed. The raylet fate shares with the agent. This can happen because the Ray agent was unexpectedly killed or failed. See `dashboard_agent.log` for the root cause. (pid=) [2022-07-23 14:47:30,700 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnknown: RPC Error message: Stream removed; RPC Error details: (pid=) [2022-07-23 14:47:33,745 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:47:33,768 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:47:34,326 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:47:34,789 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:47:36,612 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:47:37,336 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:47:38,973 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:47:39,881 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:47:40,666 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:47:41,826 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:47:42,870 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:47:44,266 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:47:45,235 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:47:46,488 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:47:47,996 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:47:48,968 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:47:50,241 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:47:51,757 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:47:52,860 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:47:54,089 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:47:55,863 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:47:56,975 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:47:57,850 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:47:58,944 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:48:00,099 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:48:01,075 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:48:02,229 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:48:05,557 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:48:05,558 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:48:05,559 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:48:06,323 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:48:07,383 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:48:08,432 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: 2022-07-23 14:48:11,909 WARNING worker.py:1404 – The node with node id: d429ece4708fb61efa00bbba0e320ea111bb89e4d8e49b59a920e142 and address: 127.0.0.1 and node name: 127.0.0.1 has been marked dead because the detector has missed too many heartbeats from it. This can happen when a raylet crashes unexpectedly or has lagging heartbeats. (pid=) [2022-07-23 14:48:09,428 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:48:10,441 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: (pid=) [2022-07-23 14:48:11,461 E 37748 31676] (gcs\_server.exe) gcs\_server.cc:283: Failed to get the resource load: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: ---
**Author:** ![rickyyx](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rickyyx/32/2671_2.png) [@rickyyx](https://discuss.ray.io/u/rickyyx)\ **Post date:** [July 23, 2022, 5:09pm UTC](https://discuss.ray.io/t/error-occurs-under-high-memory-use/6934/2 "2022-07-23T17:09:57Z")
Hey @mathew , thanks for sharing the context. Would you also share the `dashboard_agent.log`? Looks like the dashboard agent dies first. (Default should be in `/tmp/ray/session_latest/logs/dashboard_agent.log`) Or with `ray logs dashboard_agent.log` if you are on nightly. ---
**Author:** ![mathew](https://avatars.discourse-cdn.com/v4/letter/m/5daacb/32.png) [@mathew](https://discuss.ray.io/u/mathew)\ **Post date:** [July 23, 2022, 5:53pm UTC](https://discuss.ray.io/t/error-occurs-under-high-memory-use/6934/3 "2022-07-23T17:53:30Z")
| 2022-07-23 23:15:17,178 | INFO agent.py:109 – Dashboard agent grpc address: 127.0.0.1:56805 | | --- | --- | | 2022-07-23 23:15:17,180 | INFO utils.py:99 – Get all modules by type: DashboardAgentModule | | 2022-07-23 23:15:17,193 | INFO utils.py:111 – Module ray.dashboard.modules.actor.actor\_head cannot be loaded because we cannot import all dependencies. Download `pip install ray[default]` for the full dashboard functionality. Error: No module named ‘aiohttp’ | | 2022-07-23 23:15:17,583 | INFO utils.py:111 – Module ray.dashboard.modules.event.event\_head cannot be loaded because we cannot import all dependencies. Download `pip install ray[default]` for the full dashboard functionality. Error: No module named ‘aiohttp’ | | 2022-07-23 23:15:17,600 | INFO utils.py:111 – Module ray.dashboard.modules.job.job\_head cannot be loaded because we cannot import all dependencies. Download `pip install ray[default]` for the full dashboard functionality. Error: No module named ‘aiohttp’ | | 2022-07-23 23:15:17,612 | INFO utils.py:111 – Module ray.dashboard.modules.log.log\_agent cannot be loaded because we cannot import all dependencies. Download `pip install ray[default]` for the full dashboard functionality. Error: No module named ‘aiohttp’ | | 2022-07-23 23:15:17,616 | INFO utils.py:111 – Module ray.dashboard.modules.log.log\_head cannot be loaded because we cannot import all dependencies. Download `pip install ray[default]` for the full dashboard functionality. Error: No module named ‘aiohttp’ | | 2022-07-23 23:15:17,623 | INFO utils.py:111 – Module ray.dashboard.modules.node.node\_head cannot be loaded because we cannot import all dependencies. Download `pip install ray[default]` for the full dashboard functionality. Error: No module named ‘aiohttp’ | | 2022-07-23 23:15:17,635 | INFO utils.py:111 – Module ray.dashboard.modules.reporter.reporter\_agent cannot be loaded because we cannot import all dependencies. Download `pip install ray[default]` for the full dashboard functionality. Error: No module named ‘opencensus’ | | 2022-07-23 23:15:17,639 | INFO utils.py:111 – Module ray.dashboard.modules.reporter.reporter\_head cannot be loaded because we cannot import all dependencies. Download `pip install ray[default]` for the full dashboard functionality. Error: No module named ‘aiohttp’ | | 2022-07-23 23:15:17,660 | INFO utils.py:111 – Module ray.dashboard.modules.serve.serve\_head cannot be loaded because we cannot import all dependencies. Download `pip install ray[default]` for the full dashboard functionality. Error: No module named ‘aiohttp’ | | 2022-07-23 23:15:17,668 | INFO utils.py:111 – Module ray.dashboard.modules.snapshot.snapshot\_head cannot be loaded because we cannot import all dependencies. Download `pip install ray[default]` for the full dashboard functionality. Error: No module named ‘aiohttp’ | | 2022-07-23 23:15:17,675 | INFO utils.py:111 – Module ray.dashboard.modules.state.state\_head cannot be loaded because we cannot import all dependencies. Download `pip install ray[default]` for the full dashboard functionality. Error: No module named ‘aiohttp’ | | 2022-07-23 23:15:17,682 | INFO utils.py:111 – Module ray.dashboard.modules.test.test\_agent cannot be loaded because we cannot import all dependencies. Download `pip install ray[default]` for the full dashboard functionality. Error: No module named ‘aiohttp’ | | 2022-07-23 23:15:17,688 | INFO utils.py:111 – Module ray.dashboard.modules.test.test\_head cannot be loaded because we cannot import all dependencies. Download `pip install ray[default]` for the full dashboard functionality. Error: No module named ‘aiohttp’ | | 2022-07-23 23:15:17,693 | INFO utils.py:111 – Module ray.dashboard.modules.test.test\_utils cannot be loaded because we cannot import all dependencies. Download `pip install ray[default]` for the full dashboard functionality. Error: No module named ‘async\_timeout’ | | 2022-07-23 23:15:17,700 | INFO utils.py:111 – Module ray.dashboard.modules.tune.tune\_head cannot be loaded because we cannot import all dependencies. Download `pip install ray[default]` for the full dashboard functionality. Error: No module named ‘aiohttp’ | | 2022-07-23 23:15:17,708 | INFO utils.py:132 – Available modules: [\] | | 2022-07-23 23:15:17,708 | INFO agent.py:130 – Loading DashboardAgentModule: \ | | 2022-07-23 23:15:17,714 | INFO agent.py:134 – Loaded 1 modules. | ---
**Author:** ![rickyyx](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/rickyyx/32/2671_2.png) [@rickyyx](https://discuss.ray.io/u/rickyyx)\ **Post date:** [July 25, 2022, 4:36pm UTC](https://discuss.ray.io/t/error-occurs-under-high-memory-use/6934/4 "2022-07-25T16:36:23Z")
@mathew Thanks for the logs - a couple of follow-up questions from this: 1. Was this log above produced when the system crashed with high memory usage as well? 2. If you start your workload script w/o dashboard, i.e. `ray.init(include_dashboard=False)`, does the error still occur?