Ray Expert help?

1

Hi,

I’m interested in hiring a Ray expert to help me add parallel processing into a Python forecasting tool that I’ve built.

Note – I did read the TOS / Acceptable Use first thing, and didn’t see anything prohibiting or discouraging this kind of post. I’ll promptly remove it if there’s any concerns from the admins or community.

Ray looks like a superior choice for this kind of work, and, I haven’t been able to find specific experts on general sites, so I’m hoping this group may be the right place to ask.
Down the road, could this be added as a category of its own?

Here’s the gist:

I have a couple hundred optimization rules that run in sequence right now as part of a forecast tool.

Each rule starts with the “core data”, optimizes a piece, and then pushes its findings back into the “core data”.

I want to shift this to a pattern where rules can run in parallel on different cores at the same time.

The tricky bit:
Whenever a rule is completed, I want it to push the findings back to the “core dataset” just like it does today. Any NEW rules that are later started should use the new data state, not the original.

Rule processing order is not important, so whatever rules finish first can influence the next ones. This is sort of a multi-dimensional grid search where I iteratively wind my way to the global solution.

Is this something that an expert on this site could help me with?

Thanks much!

Keith

2

Hey Keith, this question definitely doesn’t violate any TOS or acceptable use. Happy to have people seeking to solve problems. :slight_smile:

From a quick read / at a high level, what you want to do is likely store the data “in” (or wrap with) an actor that is the one that kicks off the “rule process” so that that actor can ensure that any new “rule process” that is kicked off is using the most up to date version. Basically this actor is the “orchestrator” to kick off jobs with the most up to date “core data”.

On another note, one thing that we do at Anyscale (company created by creators of Ray) is work with users on these sorts of architectures to set up best practices. This is something that’s part of our managed product. If you’re not looking for a managed product, I can refer you to some consulting companies that might be interested in working with you. You can ping me on the Ray slack or here if you’d like to chat in more detail.

3

Hi Bill,

Thanks for the reply, this is great information.

I don’t think the managed product makes sense at this point, but I’ll keep it in mind for future efforts we undertake.

Separately, for the shorter-term need, I’d love to get connected to the consultants you had in mind.

Keith