1. Severity of the issue: (select one)
Medium: Significantly affects my productivity but can find a workaround.
2. Environment:
- Ray version: 2.37.0
- Python version: 3.9.15
- OS: centos8
- Cloud/Infrastructure: k8s
- Other libs/tools (if relevant): I am using data-juicer
3. What happened vs. what you expected:
- Expected: read and process very bit dataset with a raycluster with 30+ pods and 3000+ cpus
- Actual: Too many tasks and actors created and queened, and also see the log in console: WARNING util.py:260 –
The requested parallelism of 39587 is more than 4x the number of available CPU slots in the cluster of 5700.0. This can lead to slowdowns during the data reading phase due to excessive task creation. Reduce the parallelism to match with the available CPU slots in the cluster, or set parallelism to -1 for Ray Data to automatically determine the parallelism. You can ignore this message if the cluster is expected to autoscale. - my question: how is the parallelism(39587) to be calculated? How to set the parallelism? what api use this parallelism?