Ray actor CPU affinity

Ray Core
1

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.
  1. I wonder if ray will set cpu affinity to actor ?
  2. Is ray core can ensure different actors which own cpu resources are not scheduled on same physical cpu core at same time ?

Here is my experiment code. The result of cpu count is not meet my expectations.

image
image1920×826 97.8 KB

demo.py

import ray
import subprocess
from ray.util.placement_group import placement_group


@ray.remote
class Actor:

    def exec(self):
        p = subprocess.Popen("python /gpfs/public/infra/zhangrb/AlignOrch/env.py", shell=True)
        
        return True

ray.init()

bundles = [{"GPU": 4, "CPU": 4*8}]

pg = placement_group(bundles)

actor = Actor.options(num_cpus=4*8,num_gpus=4,placement_group=pg).remote()

ref = actor.exec.remote()

ray.get(ref)

env.py

import os
import psutil

process = psutil.Process()
cpu_affinity_count = len(process.cpu_affinity())

print(f"CUDA_VISIBLE_DEVICES: {os.environ.get('CUDA_VISIBLE_DEVICES')}")
print(f"CPU Count: {cpu_affinity_count}")
2

Ray’s resources are logical resources; it doesn’t handle physical resources, as I understand it. cc @Ruiyang_Wang to confirm.

3

Assume I have an actor which num_cpus=4 but I launch 100 threads (more than 4) on this actor. If these threads can be scheduled on all physical cpu cores it may hurts other actors which are running at the same time.