ValueError: Attempting to cap object store memory usage at 67108864 bytes, but the minimum allowed is 78643200 bytes

I’m getting the following error when running in docker container (OpenShift is the orchestrator).
What is the best way to resolve this error?

This literally means the plasma store memory limitation of your docker container is too low (60MB). Ray requires at least 75~isMB of memory in your machine available.

Note that Ray automatically set 30% of your machine (container) memory for plasma store. That says there are two solutions;

  1. Increase the memory capacity of your container. I think your container probably has 200~isMB of memory limitation.
  2. Specify the plasma store memory larger explicitly when you start a Ray instance. e.g., ray start --object-store-memory=$(10010241024). This is not recommended.

@sangcho Can you please elaborate re 2nd option? I’m not running a real cluster so how to set it for ray.init(local_mode=True)?

P.S I’ve re-checked, and request & limit for containers are set to 8GB! Pods are created in OpenShift v4.5. Plasma dir is /tmp

That’s pretty weird. Usually when ray is started (by ray start --head for example), it uses the 30% of the machine memory for object store memory. And you can overwrite this whenever you want to in this way;

ray start --head --object-store-memory=[memory_in_bytes]

You can also do this when you start ray from a python script.


1 Like

@sangcho Thanks, using the ‘object_store_memory’ param has resolved the issue.