Respan: An Open-Source Rust AI Gateway Built on Plugins
A decoupled, modular AI gateway designed for extensibility through a lightweight core architecture.
1The Shift to Modularity
Respan introduces an open-source AI gateway built in Rust, emphasizing a decoupled architecture where every component functions as a plugin. Unlike monolithic gateways that bundle numerous providers and features, Respan provides a minimal core. This design allows development teams to assemble specific AI infrastructure requirements without modifying the underlying gateway source code.
2Overcoming Monolithic Limitations
The project originated from challenges faced while wrapping LiteLLM, where the codebase ballooned to over 100,000 lines. A critical failure during a January upgrade forced the team to manually backport fixes, highlighting the risks of monolithic dependencies. This experience drove the decision to build a custom, plugin-first architecture inspired by Nginx.
3Core Event Dispatching Mechanism
The gateway operates by handing HTTP or WebSocket connections to client formats that recognize the traffic. The core then executes registered plugins in order, enabling them to filter streams, wrap requests, or make secondary calls. This event-driven model supports both simple HTTP requests and complex, long-lived bidirectional sessions for real-time AI interactions.
4Extensibility Through Plugin Ecosystem
Everything within Respan is a plugin, including provider integrations, protocols like MCP, and security features like PII redaction. Developers can implement custom logic without touching the core, making it simple to add unique providers. For instance, a Mistral provider implementation requires only 22 lines of code, demonstrating the system's lightweight nature.
5Handling Diverse AI Modalities
Respan goes beyond simple model routing by automatically handling tools, images, and reasoning protocols. It allows applications built for the OpenAI Realtime API to interact with Gemini Live through the gateway. By parsing requests into a shared structure while preserving native fields, it ensures compatibility without dropping provider-specific data during transmission.
6Deployment and Open Source
The gateway is available under MIT or Apache-2.0 licenses on GitHub, allowing users to build from source or run it locally. Developers can swap base URLs in existing clients to leverage the gateway immediately. This flexibility ensures that teams maintain control over their infrastructure while benefiting from the modular plugin architecture.
7Business Model and Observability
While the core gateway remains open source, the team monetizes through a hosted platform. This service provides native observability, evaluation tools, and unified billing for organizations. By offloading the operational burden of running the gateway, the platform offers a managed alternative for teams that prefer not to self-host the infrastructure.
8Future Roadmap and Contribution
The primary roadmap focuses on expanding the plugin ecosystem and encouraging community contributions. Documentation in the repository provides guidance for developers to write and share their own plugins. By fostering this collaborative environment, Respan aims to become a standard, extensible foundation for routing, security, and observability in AI agent applications.
Original article: Hacker News: Launches — Respan: An Open-Source Rust AI Gateway Built on Plugins