I am using ray air TorchTrainer within my ray cluster. The dataset is created from a csv file and then multiple workers work with it, the issue is that when calling TorchTrainer it appears that the file is read all into memory by just one node. Is there anywhere to avoid loading the complete file into memory?
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
|
Accessing Large Static Datasets with Ray Clusters
|
3 | 544 | May 27, 2023 | |
| Tuning Settings for Big Data | 0 | 286 | November 3, 2023 | |
|
Debugging Ray Data out of memory errors
|
0 | 227 | February 8, 2024 | |
|
Shared dataset on a local desktop
|
1 | 287 | March 7, 2023 | |
|
Using ray datasets with pytorch lightning
|
0 | 307 | November 22, 2023 |