You cannot directly enforce a hard memory cap on RLlib policies during setup; RLlib’s memory resource specification is only for bookkeeping and does not limit actual memory usage (memory management discussion). To avoid OOM, you should reduce the number of policies in memory (lower policy_map_capacity), shrink the observation space, or run fewer workers. RLlib will otherwise allocate as much memory as needed for all policies and their dummy batches during initialization.
If you need to prevent your system from being killed, limit the number of concurrent policies or processes, and monitor memory usage externally. RLlib does not provide a built-in mechanism to enforce a strict memory ceiling during algorithm setup (memory management discussion). Would you like more detail or practical workarounds?
Sources:
Hint: Mention @RunLLM in the post for followups.