Ray snowflake connector

Ray Libraries (Data, Train, Tune, Serve) Ray Data
1

How severe does this issue affect your experience of using Ray?

  • None: Just asking a question out of curiosity

I have been trying to motivate my team towards using Ray with our snowflake setup. I recently came across this anyscale blog – Introducing the Anyscale Snowflake Connector | Anyscale which excited my further. After some searching on github, I could find the code for this connector as well.

I am a complete complete newbie to ray and hence couldn’t get this connector to work consistently. However, I am thoroughly familiar with snowflake and its python connector and I have a complete understanding how a snowflake connector in ray should work, so I went ahead and wrote my connector while seeking motivation from the above codebase as well as from the ray custom datasource documentation.

:drum: GitHub - prabodh1194/ray_snowflake :drum:

Please go through my repo and fire away with questions and PRs and ideas on how can we go about using these.

Thanks a ton.

P.S.: I’ll be spending sometime this week to create an optimal write step which can directly write an arrow table to snowflake instead of taking the Pandas route.

2

Hey @pbd,

I’m excited that your team is exploring Ray!

The fork you linked (datasets-database) is actually unmaintained and unrelated to the implementation described in the blog post. (The official implementation is only available on Anyscale). For more up-to-date documentation, see Read and Write from Snowflake.

3

Thanks in that case :smiley:. I thought so too. I’m feeling even better now that I have written this now.

1 Like
4

Hey @pbd, nice you recreated the snowflake reader which is close-sourced by anyscale. Did you try replicating the snowflake datasource writer?

Thanks!