Relay: Self-Hosted LLM Gateway with Model Routing
A solution supporting OpenAI and Anthropic APIs with customizable routing policies.
1What is Relay?
Relay is a self-hosted LLM gateway and model router that supports OpenAI and Anthropic APIs. It operates with a single Go binary and offers flexible routing policies. This allows developers to easily switch between different providers using their own API keys.
2Flexible Routing Policies
Relay provides a range of routing policies, from static aliases to learned smart layers. These policies can be customized according to user needs, balancing performance and cost.
3Evaluation Mechanism
Relay ensures routing decisions with an evaluation mechanism. This mechanism operates on models trained on user traffic, enhancing the reliability of smart routing.
4Privacy and Security
Relay collects no telemetry data and is entirely privacy-focused. This prevents user data from being shared with third parties, offering a secure solution.
5Easy Setup and Usage
Relay is a solution that can be set up in 60 seconds and requires zero configuration. Developers can quickly start using their own API keys and switch between various providers.
6Performance and Scalability
Relay stands out with low latency and high performance. While provider latency is the dominant factor, Relay's job is to minimize this latency and provide a seamless experience for users.
7Future and Development
Relay continues to develop its smart routing layers. These layers, trained on user traffic, aim to provide a more powerful and reliable solution in the future.