A reverse proxy sits in front of an API you call, not in front of a site you serve. Same word, opposite direction — and it is worth being clear about that first, because most proxy writing means the other thing.
The reason teams put one in front of OpenAI comes down to a single uncomfortable fact: your API key is a bearer token with a company credit card behind it. Anywhere that key exists is a place it can leak from.
What it actually gets you
The key stops travelling. Without a proxy, every service that calls OpenAI holds the key. Mobile apps are the worst case — a key shipped in a client is a key published, and no amount of obfuscation changes that. With a proxy the key lives in one place and your clients authenticate to you.
You get a spend cap that actually exists. OpenAI gives you an org-level limit. It does not tell you that one runaway loop in staging is eating the budget until the bill arrives. A proxy can hold per-key, per-user, per-day ceilings and refuse the request that crosses one.
Caching becomes possible. Deterministic calls — classification, extraction, embeddings — return the same answer for the same input. Caching those is free money. Without a proxy there is no shared place to put that cache.
You can swap models without redeploying. The proxy decides which model a given route uses, so a downgrade from a large model to a cheap one is a config change rather than a release.
The minimum viable version
Twelve lines of nginx will do it:
location /ai/ {
proxy_pass https://api.openai.com/;
proxy_set_header Authorization "Bearer $OPENAI_KEY";
proxy_set_header Host api.openai.com;
proxy_read_timeout 300s; # streaming responses run long
proxy_buffering off; # required, or tokens arrive in one lump
limit_req zone=ai burst=20 nodelay;
}Two of those lines are the ones people miss. proxy_buffering off is not optional if you stream — with buffering on, nginx holds the whole response and your users watch a spinner instead of tokens appearing. And the default read timeout will cut long generations off mid-sentence.
Doing it at the edge instead
If you have no server to put nginx on, a Cloudflare Worker covers the same ground and adds per-user attribution:
export default {
async fetch(req, env) {
const user = await authenticate(req); // your auth, not OpenAI's
if (!user) return new Response("Unauthorized", { status: 401 });
if (await overBudget(env, user.id))
return new Response("Budget exceeded", { status: 429 });
const body = await req.text();
const res = await fetch("https://api.openai.com/v1/chat/completions", {
method: "POST",
headers: {
"Authorization": `Bearer ${env.OPENAI_KEY}`,
"Content-Type": "application/json",
},
body,
});
await recordUsage(env, user.id, res.headers);
return res; // stream passes straight through
},
};Returning the Response object directly preserves streaming. Read the body to inspect it and you have quietly turned a streaming endpoint into a blocking one.
The mistakes that cost money
Caching things that should not be cached. Cache on a hash of the entire request — model, temperature, every message. Two prompts differing by one word are different prompts. And if the prompt contains user data, a shared cache is a data leak wearing a performance costume.
Logging prompts in full, forever. Useful for debugging, and a liability the moment a user pastes something personal. Log metadata by default and content only behind a flag with a retention limit.
Forgetting the proxy is now a single point of failure. Everything that used to fail independently now fails together. Health checks and a bypass path are not optional.
Where it does not belong
If you have one backend service calling OpenAI and one team, a proxy is architecture for its own sake. The key already lives in exactly one place. Add the proxy when you have a second consumer, an untrusted client, or a bill you cannot attribute.
And to be clear about the naming: this has nothing to do with the forward proxies used for scraping. If you are collecting the training or evaluation data rather than calling the model, that is the other side of the pipeline.
Frequently asked questions
Why put a reverse proxy in front of the OpenAI API?
So the API key never leaves your infrastructure. A reverse proxy holds the key server-side and your apps authenticate to you instead. That also gives you one place to enforce spend caps, rate limits, caching and audit logging, none of which OpenAI gives you per-client.
Is it against OpenAI's terms to proxy their API?
Routing your own traffic through your own infrastructure is ordinary architecture and is not the issue. What does breach terms is reselling access, sharing one key across unrelated customers, or using a proxy to evade region or rate restrictions. The line is whether you are managing your own usage or redistributing someone else's.
Does a reverse proxy slow down responses?
By a few milliseconds if it sits near your application. That is invisible next to a model response measured in seconds. Put the proxy in the same region as your app, not the same region as OpenAI.
Can I cache OpenAI responses?
Only where the same prompt should give the same answer - classification, extraction, embeddings. Cache on a hash of the full request including model and temperature. Never cache open-ended generation, and never cache across users unless the prompt has no user-specific content in it.