Have parallel iterators been removed?

ceteri · September 4, 2021, 11:26pm

The current (1.6) docs return a 404 for the page about parallel iterators: https://docs.ray.io/en/latest/iter.html

Has this capability been moved to another part of the API? Using a different name?

Or is it an error in the documentation?

mannyv · September 5, 2021, 11:01am

It seems to be an error in the documentation. The current version of the documentation points to the link below but that update is in latest not 1.6.

github.com

ray-project/ray/blob/master/python/ray/util/iter.py

from contextlib import contextmanager
import collections
import random
import threading
import time
from typing import TypeVar, Generic, Iterable, List, Callable, Any

import ray
from ray.util.iter_metrics import MetricsContext, SharedMetrics

# The type of an iterator element.
T = TypeVar("T")
U = TypeVar("U")


def from_items(items: List[T], num_shards: int = 2,
               repeat: bool = False) -> "ParallelIterator[T]":
    """Create a parallel iterator from an existing set of objects.

    The objects will be divided round-robin among the number of shards.

This file has been truncated. show original

The iter documentation file was deleted in this commit. I am not sure why but presumably @ericl does.

ericl · September 6, 2021, 5:16am

Yes, parallel iterators have been soft deprecated (we want to encourage users to use Datasets: Datasets: Flexible Distributed Data Loading — Ray v2.0.0.dev0).

They’re still supported since they’re part of the public API of RLlib. But for most use cases we’ve seen the Dataset-like approach is much easier to use and has less ways to “shoot yourself in the foot”.

Topic		Replies	Views
Converting IterableDataset to Ray iterator Ray Core	2	305	April 12, 2022
Improve and verify the performance of code on Ray Ray Core	0	272	March 3, 2021
[Core] Issue with running Ray code from Jupyter notebook Ray Core	1	317	April 21, 2021
Converting torch.utils.data.IterableDataset to Ray's Dataset Ray Data	4	595	April 13, 2022
Check Logging for parallel Iterator Ray Core	2	270	March 27, 2021

Have parallel iterators been removed?

Related Topics