Cloudflare has announced a new feature for its AI Gateway: the Auto Router. This model router evaluates the complexity of each incoming request using an edge-deployed classifier and automatically selects the optimal AI model. By balancing expected output quality against token costs, organizations can significantly reduce AI spending while maintaining performance.
The Auto Router analyzes request complexity in real-time at the edge, choosing from a set of available models. This eliminates the need for manual model selection or complex logic in applications. Key concrete changes include:
- Edge-deployed classifier: The router uses a lightweight classifier running on Cloudflare's edge network to assess request complexity without adding latency.
- Model selection: Based on complexity, it routes requests to the most appropriate model, which can vary in cost and capability.
- Cost optimization: By matching requests to models, the system minimizes token costs while aiming to preserve output quality.
- Integration with AI Gateway: The feature is built into the existing AI Gateway, so developers can enable it through the dashboard or API.
Developers can take advantage of this by simply configuring their AI Gateway to use the Auto Router. It requires no changes to application code, as the routing happens transparently. The classifier is designed to handle a variety of request types, from simple queries to complex tasks, ensuring that each request uses the most efficient model.
This matters to developers because it reduces the operational burden of managing multiple AI models and their costs. Instead of manually deciding which model to use, developers can rely on the Auto Router to make intelligent decisions. This can lead to substantial savings, especially for applications with high volumes of AI requests. Additionally, the edge deployment ensures low-latency routing, maintaining user experience. As AI costs continue to be a concern, tools like the Auto Router help make AI more accessible and sustainable for businesses of all sizes.