# Is there any method for returning a false data in ray.data

**URL:** <https://discuss.ray.io/t/is-there-any-method-for-returning-a-false-data-in-ray-data/11650>\
**Category:** Uncategorized\
**Created:** [August 2, 2023, 2:00pm UTC](https://discuss.ray.io/t/is-there-any-method-for-returning-a-false-data-in-ray-data/11650 "2023-08-02T14:00:26Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![kyoka\_gong](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kyoka_gong/32/927_2.png) [@kyoka\_gong](https://discuss.ray.io/u/kyoka_gong)\
**Post date:** [August 2, 2023, 2:00pm UTC](https://discuss.ray.io/t/is-there-any-method-for-returning-a-false-data-in-ray-data/11650/1 "2023-08-02T14:00:26Z")

</div>

Hello.

I was using ray.data to batch inference my image dataset.  
In my case, some of my image dataset could not be read by PIL.  
Then I created a class which read image and preprocessed it.

```auto
class ImageReader:
    def __init__ (self) -> None:
        self.logger = Logger()
        self.bli2_processor = Blip2ProcessorWrapper() 

    def __call__ (self, datum):
        image_path = datum['image_path']
        try:
            img = Image.open(image_path)
            if not img.mode == "RGB":
                img = img.convert("RGB")
            prepro = self.bli2_processor.processor(img, return_tensors="pt")['pixel_values']
        except Exception as e:
            self.logger.exception(f"read error: {str(e)}, origin_url: {image_path}")
            return {"batch_images": None, **datum}
        return {"batch_images": prepro, **datum}

```

In my class, when one image could be read, then I returned the result after blip2\_preprocessor. When image could not be read, i returned None. And then, I used a filter to filter that `batch_images` is None.

Howerver, if I used this, I would found `pyarrow exception` in data serialization.

SO, is there any method for returning a false data in ray.data, so that i can filter in the follow up filter operator

---

<div class="post-metadata">

**Author:** ![Jules\_Damji](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jules_damji/32/4058_2.png) [@Jules\_Damji](https://discuss.ray.io/u/Jules_Damji)\
**Post date:** [August 2, 2023, 8:15pm UTC](https://discuss.ray.io/t/is-there-any-method-for-returning-a-false-data-in-ray-data/11650/2 "2023-08-02T20:15:35Z")

</div>

@kyoka_gong Instead of return a None, can you create a fake image, with tensors, which you can filter out, that way serailzation of None won’t fail.

cc: @ericl @chengsu Any ideas here?

---

<div class="post-metadata">

**Author:** ![ericl](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/ericl/32/133_2.png) [@ericl](https://discuss.ray.io/u/ericl)\
**Post date:** [August 2, 2023, 9:20pm UTC](https://discuss.ray.io/t/is-there-any-method-for-returning-a-false-data-in-ray-data/11650/3 "2023-08-02T21:20:31Z")

</div>

Yup, I think that would be the way to go. You can also mark a record as errored with a separate “valid” or “error” column.

---

<div class="post-metadata">

**Author:** ![kyoka\_gong](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kyoka_gong/32/927_2.png) [@kyoka\_gong](https://discuss.ray.io/u/kyoka_gong)\
**Post date:** [August 2, 2023, 11:48pm UTC](https://discuss.ray.io/t/is-there-any-method-for-returning-a-false-data-in-ray-data/11650/4 "2023-08-02T23:48:56Z")

</div>

thx ericl’s & jules’s suggestion. 🤝