RunPod Integration

Self-hosted GPU inference for models you don't want to send to a third-party API.

RunPod rents GPU pods for self-hosted model inference — useful when you're running a fine-tuned or open-weight model yourself for cost, latency, or data residency reasons rather than calling a hosted API.

Setup in 4 steps

1Deploy your model as a RunPod serverless endpoint or persistent pod
2Add RUNPOD_ENDPOINT_URL and RUNPOD_API_KEY to your .env
3Point LLM_PROVIDER at a custom adapter that calls your RunPod endpoint
4Handle cold-start latency by keeping a minimum number of workers warm if the endpoint is latency-sensitive

What's included

Full control over the model and hardware, useful for data residency requirements
Can be cheaper than hosted APIs at sustained high volume, depending on utilization
Fits behind the same llm.chat() interface with a custom provider adapter
Good option for fine-tuned open-weight models not available via hosted LLM APIs

Code example

example.py

# app/llm/providers/runpod_provider.py
import httpx

class RunPodProvider:
    def __init__(self, endpoint_url: str, api_key: str):
        self.endpoint_url = endpoint_url
        self.headers = {"Authorization": f"Bearer {api_key}"}

    async def chat(self, messages: list[dict]) -> str:
        async with httpx.AsyncClient() as client:
            resp = await client.post(
                f"{self.endpoint_url}/runsync",
                json={"input": {"messages": messages}},
                headers=self.headers,
            )
            return resp.json()["output"]["text"]

Full documentation: /docs/llm-providers

RunPod integration pre-wired and ready.

FastAPI AI Kit ships with RunPod configured out of the box. No manual setup required.

Ready to ship your AI backend this weekend?

Join developers who skipped weeks of boilerplate and went straight to building.

Read the docs

No subscriptions · One-time payment · Lifetime updates