How to enable huge pages in ray

Ray Core
1

I want plasma store to work with huge pages

I tried creating a a file system and activate huge pages with. Like the example in plasmastore
sudo mkdir -p /mnt/hugepages
gid=id -g
uid=id -u
sudo mount -t hugetlbfs -o size=170G -o uid=$uid -o gid=$gid none /mnt/hugepages
sudo bash -c “echo $gid > /proc/sys/vm/hugetlb_shm_group”
sudo bash -c “echo 20000 > /proc/sys/vm/nr_hugepages”

and I set in ray.init _plasma_directory="/mnt/hugepages"

This is the output of raylet.out

2021-12-16 10:50:07,727 I 83753 83753] io_service_pool.cc:35: IOServicePool is running with 1 io_service.
[2021-12-16 10:50:07,728 I 83753 83753] store_runner.cc:30: Allowing the Plasma store to use up to 161.631GB of memory.
[2021-12-16 10:50:07,728 I 83753 83753] store_runner.cc:46: Starting object store with directory /mnt/hugepages, fallback /tmp/ray, and huge page support disabled
[2021-12-16 10:50:07,728 I 83753 83782] dlmalloc.cc:146: create_and_mmap_buffer(161630912520, /mnt/hugepages/plasmaXXXXXX)
[2021-12-16 10:50:07,751 C 83753 83782] dlmalloc.cc:164: failed to ftruncate file /mnt/hugepages/plasmaUeo901, errorInvalid argument
*** StackTrace Information ***
_ZN3ray12GetCallTraceB5cxx11Ev
ray::SpdLogMessage::Flush()
ray::RayLog::~RayLog()
plasma::create_and_mmap_buffer()
plasma::fake_mmap()
dlmalloc
plasma::internal_memalign()
plasma::PlasmaAllocator::Allocate()
plasma::PlasmaAllocator::PlasmaAllocator()
plasma::PlasmaStoreRunner::Start()
std::thread::_State_impl<>::_M_run()
start_thread

As a secondery issue I saw that it prints in the raylet.out
“and huge page support disabled”

I see in raylet main.cc the following argument
DEFINE_bool(huge_pages, false, “Whether enable huge pages”);
and it is used in the code regarding the plasma store

However I don’t see anywhere were it is set to true

So how can I enable it?

2

I found a bit more

  1. ftruncate must receive size which is a multiplication of the page size. But since some alignment is done somewhere inside the code, no matter what value I put in ray.init(object_store_memory=) I can’t get it to align to 2MB, therefore it won’t run.
  2. in ray.init
    Worker.py code
    the huge_pages is set to false without ability to pass it as an argument to ray.init. The rest of the code is implmented, but this part is missing
3

Hi @shiranbi, support for huge pages in Ray was removed in Ray 1.0 since it added complexity to our APIs and the Ray system without much benefit to users. Do you see huge page support as very important for your use case?