ValueError: Attempting to cap object store memory usage at 67108864 bytes, but the minimum allowed is 78643200 bytes

Ray Core
1

I’m getting the following error when running in docker container (OpenShift is the orchestrator).
What is the best way to resolve this error?

2

This literally means the plasma store memory limitation of your docker container is too low (60MB). Ray requires at least 75~isMB of memory in your machine available.

Note that Ray automatically set 30% of your machine (container) memory for plasma store. That says there are two solutions;

  1. Increase the memory capacity of your container. I think your container probably has 200~isMB of memory limitation.
  2. Specify the plasma store memory larger explicitly when you start a Ray instance. e.g., ray start --object-store-memory=$(10010241024). This is not recommended.
3

@sangcho Can you please elaborate re 2nd option? I’m not running a real cluster so how to set it for ray.init(local_mode=True)?

P.S I’ve re-checked, and request & limit for containers are set to 8GB! Pods are created in OpenShift v4.5. Plasma dir is /tmp

4

That’s pretty weird. Usually when ray is started (by ray start --head for example), it uses the 30% of the machine memory for object store memory. And you can overwrite this whenever you want to in this way;

ray start --head --object-store-memory=[memory_in_bytes]

You can also do this when you start ray from a python script.

```python3`
ray.init(object_store_memory=[memory_in_bytes])

1 Like
5

@sangcho Thanks, using the ‘object_store_memory’ param has resolved the issue.