Groq Integration

Sub-second inference on Llama and Mixtral via Groq's LPU cloud.

Groq exposes an OpenAI-compatible API for open-weight models running on their LPU hardware. Point the kit's LLM base URL at Groq's endpoint for dramatically faster token generation on supported models.

Setup in 4 steps

1Add GROQ_API_KEY to your .env file
2Set LLM_PROVIDER=openai and LLM_BASE_URL=https://api.groq.com/openai/v1
3Set LLM_MODEL to a supported model (e.g. llama-3.3-70b-versatile)
4Streaming and token tracking work unchanged via the existing OpenAI provider

What's included

OpenAI-compatible endpoint — no custom adapter required
Significantly lower time-to-first-token for supported open-weight models
Good fit for latency-sensitive endpoints like chat or code review
Same token tracking flows into Stripe metering as any other provider
Easy to A/B test latency against OpenAI/Anthropic by env var only

Code example

example.py

# .env — Groq via OpenAI-compatible endpoint
GROQ_API_KEY=gsk_...
LLM_PROVIDER=openai
LLM_BASE_URL=https://api.groq.com/openai/v1
LLM_MODEL=llama-3.3-70b-versatile

response = await llm.chat(
    messages=[{"role": "user", "content": prompt}],
    stream=True,
    track_tokens=True,
)

Full documentation: /docs/llm-providers

Groq integration pre-wired and ready.

FastAPI AI Kit ships with Groq configured out of the box. No manual setup required.

Ready to ship your AI backend this weekend?

Join developers who skipped weeks of boilerplate and went straight to building.

Read the docs

No subscriptions · One-time payment · Lifetime updates