Skip to main content
FastAPI AI Kit
All templatesTemplate

A unified LLM gateway that routes, caches, meters, and bills.

The LLM Gateway template configures FastAPI AI Kit as a centralized LLM access point for organizations. Route requests to different LLM providers, cache common responses, meter token usage per team or project, and enforce access policies — all through a single, unified API.

FastAPIRedisPostgreSQLOpenAIAnthropicGoogle GeminiStripe

What's included

  • Multi-provider LLM routing (OpenAI, Anthropic, Gemini — switchable per request)
  • Response caching with configurable TTL for repeated queries
  • Per-key token metering and cost attribution
  • Provider fallback: auto-switch to backup provider on failures
  • Request/response logging for audit and debugging
  • Per-team API keys with independent quotas
  • Model access policies (restrict which keys can use which models)
  • Cost dashboard data endpoints for finance visibility

Code preview

main.py
# Unified LLM endpoint with provider routing
@router.post("/v1/completions")
async def completions(
    body: CompletionRequest,
    key: APIKey = Depends(verify_api_key),
):
    # Check model access policy
    if body.model not in key.allowed_models:
        raise HTTPException(403, f"Key not authorized for model: {body.model}")

    # Try cache first
    cached = await cache.get(body.cache_key)
    if cached:
        return CompletionResponse(content=cached, cached=True)

    # Route to correct provider with fallback
    response = await llm.chat(
        messages=body.messages,
        model=body.model,
        provider=resolve_provider(body.model),
        fallback_provider=body.fallback_provider,
    )

    await cache.set(body.cache_key, response.content, ttl=body.cache_ttl)
    await meter.record(key.id, response.tokens, model=body.model)
    return CompletionResponse(content=response.content, tokens=response.tokens)

Other templates

FastAPI AI Chatbot API TemplateFastAPI RAG Search API TemplateFastAPI SaaS API Template

Start with this template

Get FastAPI AI Kit, select this template configuration, and deploy your fastapi llm gateway template in minutes.

Ready to ship your AI backend this weekend?

Join developers who skipped weeks of boilerplate and went straight to building.

Read the docs
No subscriptions · One-time payment · Lifetime updates