When I run locally on a single machine I can specify num_cpus on ray.init(num_cpus = 2). But i cant specify this on cluster
I want to limit the number of cpus being used per worker like this:
worker_node:
max_cpus: 3
Head_node:
max_cpus: 4
Do you know how I would edit my yaml to do so, as what I have tried hasn’t worked?
The reason why is I run out of RAM on my ec2 when all cores are running on it.
cluster_name: autoscale
initial_workers: 5
min_workers: 5
max_workers: 5
initialization_commands:
- aws configure set aws_access_key_id --------------------
- aws configure set aws_secret_access_key -------------------
- aws configure set default.region eu-west-2
#access to docker login
- eval $(aws ecr get-login --no-include-email --region eu-west-2)
- sudo aws s3 cp s3:/ pipeline/ --recursive
docker:
image: "/pipeline:ray"
container_name: "hello_ray_container"
pull_before_run: True
run_options:
- "-v /home/ubuntu/pipeline/data:/opt/pipeline/data"
provider:
type: aws
region: eu-west-2
auth:
ssh_user: ubuntu
head_node:
InstanceType: c5.12xlarge
ImageId: latest_dlami # Default Ubuntu 16.04 AMI
BlockDeviceMappings:
- DeviceName: /dev/sda1
Ebs:
VolumeSize: 200
worker_nodes:
InstanceType: c5.12xlarge
ImageId: latest_dlami # Default Ubuntu 16.04 AMI.
BlockDeviceMappings:
- DeviceName: /dev/sda1
Ebs:
VolumeSize: 200