Hard limit on CPUs

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

Is it possible to enforce the Trainable does not use more than the configured CPUs for that trial?

With the below spec I see the CPU usage > 400% for each of the trials running in parallel on my 16 core machine,

analysis = tune.run(
    training_function,
    resources_per_trial={"cpu": 4, "gpu": 0},
    config={
        ...
    },
    num_samples=2)
2

Hi @Nitin_Pasumarthy, just to be clear, does this means your 2 parallel trials are using more than 8 CPUs in total? How many? All 16?

3

Hi Kai,
Yes I see the activity on all 16. The objective function launches a subprocess which tries to use all available multiple cores. So, I was wondering why Ray is not restricting the number of visible CPUs for each of these multi core trials, something like what linux’s cgroup does

4

I’m running into the same issue with a hypertuning process. Even when I specify only one cpu is allowed to be used in tune.run’s resources_per_trial, it occupies most of my 12 cores. It’s completely unexpected behaviour, so I’d like to know what is going on as well.

–edit

Apparently this was implemented on purpose: “If specifying CPUs, Ray does not enforce isolation (i.e., your task is expected to honor its request” (Ray Core Walkthrough — Ray v2.0.0.dev0).