Request for comments: Adding LangChain support to Aviary

Waleed_Kadous · June 2, 2023, 3:33am

Hey folks,

We just put out an RFC for our planned changes to support LangChain in Aviary. Feel free to leave comments on it here:

REQUEST FOR COMMENTS: Aviary <-> LangChain Integration

opened 02:40AM - 02 Jun 23 UTC

# Purpose Aviary is an open source LLM management toolkit built on top of Ray… Serve and Ray. LangChain is an incredibly popular open source toolkit for building LLM applications. The question is how do these things fit together? # Possible integrations This first integration is focused on LLM (non chat) to begin with. When we add streaming to Aviary, we will also integrate that for the chat application too. There are 3 possible integration points with LangChain. ## 1. Aviary as an LLM provider for LangChain Make Aviary a model backend for LangChain (the same way that OpenAI is done currently). This would enable you to do things like: ```python import os from langchain.llms import Aviary aviary_url = os.environ["AVIARY_URL"] # Token is optional aviary_token = os.environ["AVIARY_TOKEN"] or None from langchain.llms import Aviary llm = Aviary(model_name = 'amazon/LightGPT') #single query llm.predict('How do you make fried rice?') #uses Aviary's batch interface for greater efficiency llm.generate(['How do you make fried rice?', 'What are the most influential punk bands?']) ``` The only real decision here is do we use our SDK or do we allow direct connection to our endpoints. Since our Web API right now is so simple, it might be easier to code against it in the short term, and use the SDK when it is justified. ## 2. Aviary “wraps” LLMs provided by LangChain Allow any [model supported by LangChain](https://python.langchain.com/en/latest/modules/models/llms/integrations.html) to be wired up through Aviary (the same way that Aviary currently “wires up” Hugging Face). This would give a way for centrally managed Aviaries to control access to models from OpenAI and to impose additional limits on length. For every model you want to wrap, you would have to set up a models/ file in the [https://github.com/ray-project/aviary/tree/master/models](https://github.com/ray-project/aviary/tree/master/models) directory. We would expand that file format to also support LangChain LLMs as well. ## 3. Integrate LangChain LLM support directly into Aviary Explorer and Aviary CLI Allow users to query any model supported by LangChain directly. This would be useful for example to do cross OSS <-> commercial comparisons e.g. with GPT-3.5-turbo. What we would do there is allow Aviary CLI to do something like this: ``` aviary query -–model amazon/LightGPT -–model model-configs/langchain-openai-gpt-35,yaml examples/qa-prompts.txt ``` In the aviary command, we would read openai://gpt-3.5-turbo and use the LangChain OpenAI LLM tool allowing for cross evaluation. We would have add new functionality to Aviary Explorer to support adding arbitrarily configured LangChain LLMs. In essence the difference between proposals 2 and 3 is: where do the config files for specifying LLM properties live? # Decision We are not limited to doing one of these. The most immediate need and highest impact is perhaps #1. #2 and #3 are similar in many ways. Perhaps #3 is more impactful. The Aviary Explorer changes, however, are more complicated. It’s slightly ugly in the sense that we now have yaml files both on the Aviary backend and in the Aviary CLI and Explorer.

Topic		Replies	Views
Announcing Ray Aviary LLMs/Generative AI/Aviary	1	770	May 31, 2023
About the LLMs/Generative AI/Aviary category LLMs/Generative AI/Aviary	0	601	April 20, 2023
Does ray-llm support only CPU? LLMs/Generative AI/Aviary	0	384	October 25, 2023
How about a Discord Server?	1	915	July 16, 2022
Serving LLM with multiple gpus LLMs/Generative AI/Aviary	0	104	July 3, 2024

Request for comments: Adding LangChain support to Aviary

Related topics