Running Our Own AI Gateway Instead of Handing Everyone a Provider Key

The problem with individual provider keys

Giving every developer, service, and internal tool a direct provider key is convenient at first. It is also difficult to unwind later.

Each key becomes a credential we must issue, store, rotate, revoke, and investigate. Usage is split across accounts or projects, so answering a basic question such as which team generated this cost? may require exporting data from several provider dashboards. We also lose a practical control point when an experiment starts sending unexpected traffic.

We prefer to give applications access to an internal AI gateway rather than to a model provider directly. The gateway owns the provider credentials. Applications receive scoped credentials that we can disable or change without modifying our upstream provider accounts.

What we put behind the gateway

Our gateway sits between clients and model providers. A client sends a request to one internal endpoint, and the gateway decides how to authenticate, authorize, route, log, and account for it.

We can run this design with LiteLLM because it provides an OpenAI-compatible interface while supporting multiple model providers. That compatibility matters in practice: many existing SDKs and tools can point to our gateway by changing a base URL and API key, rather than requiring application-specific provider integrations.

A self-hosted AI gateway does not remove our relationship with providers. We still need provider accounts, upstream keys, contracts where appropriate, and an understanding of each provider's retention and data policies. What changes is where those keys are held and where usage controls are applied.

Virtual keys are not provider keys

The main implementation decision is to issue virtual keys to users and workloads.

A virtual key is an internal credential created by the gateway. It can be tied to a person, service, environment, project, or automation. The gateway maps that key to permitted models and one or more upstream provider credentials.

This gives us controls that are hard to enforce when everyone has a provider key:

• We can revoke one application's access without affecting other applications.
• We can limit a key to selected models or providers.
• We can set rate limits appropriate to an interactive user, background job, or shared service.
• We can attach metadata such as team, project, environment, and cost center.
• We can rotate virtual keys independently from upstream provider keys.

We do not treat virtual keys as harmless simply because they are internal. They are still secrets, and we keep them out of source control, browser code, and unprotected configuration files. A gateway is most useful when clients call it from controlled server-side workloads. For browser-based features, we usually keep the model call behind our own application backend rather than exposing any long-lived AI credential to the browser.

Budget controls need a clear policy

A budget is only useful if we decide what should happen when it is approached or exceeded. The gateway can track spend by virtual key, team, project, or model, but it cannot make the policy decision for us.

We normally define several levels of budget control:

• A per-key limit protects against a single leaked credential or faulty deployment.
• A project budget makes experimental work visible before it becomes shared infrastructure spend.
• A team or cost-center budget supports internal planning and review.
• A global provider limit provides a final guardrail, although it should not be the only one.

There is a trade-off between hard blocking and alerts. Hard limits contain cost immediately, but they can interrupt a production workflow. Alerts preserve availability, but they require someone to respond. For critical workloads, we prefer a documented fallback model or a degraded feature path over assuming that budget enforcement will never trigger.

We also distinguish between input and output behavior when reviewing costs. A system with large prompts, long conversation histories, tool results, or verbose outputs can consume budget rapidly even when request counts look ordinary. Request count alone is not a reliable measure of AI spend.

Cost attribution should start with request metadata

Cost attribution is much easier when we require metadata at the gateway boundary. We want each request to carry enough context to answer who initiated it, which product or service used it, and why it was allowed to use a particular model.

Useful fields often include:

• team
• project
• environment
• service
• feature
• cost_center
• request_id

The exact naming is less important than consistency. We define a small required set and reject or flag requests that omit it. If every client invents its own labels, reporting becomes a cleanup project.

We also keep attribution separate from personal data where possible. A project identifier is usually more useful for routine cost analysis than a user's email address. If we need to investigate an incident, we can correlate a request identifier with application logs under the appropriate access controls.

Routing is where flexibility becomes complexity

One benefit of LiteLLM and similar gateway patterns is model routing. We can expose a logical model name internally and map it to a provider model. This lets us change an upstream model, move a workload to another provider, or establish a fallback without updating every client.

That flexibility introduces risk. Different providers can differ in tool calling, structured output behavior, context limits, safety settings, latency, and pricing. We should not assume that two models with similar names are interchangeable.

We make routing decisions explicit:

1. We define which workloads may use which models.
2. We test important prompt and tool flows against proposed fallback models.
3. We document expected behavior when a primary provider is unavailable.
4. We keep model changes reviewable rather than silently changing routes for every workload.

For some use cases, routing to a lower-cost model is reasonable. For others, quality or compatibility matters more than unit cost. The gateway gives us a place to apply that decision, but it does not eliminate the need for evaluation.

Operating a self-hosted AI gateway

Self-hosted AI means we accept operational responsibilities. We need to patch the gateway, protect its administrative interface, manage secrets, back up configuration where needed, monitor availability, and decide how long logs are retained.

We also decide what request data should be logged. Full prompts and responses can be valuable for debugging, but they can contain sensitive information. We prefer to log operational metadata by default and enable detailed content logging only when there is a justified need, suitable access control, and a retention policy.

At minimum, we monitor:

• Gateway errors and upstream provider errors.
• Request volume, latency, and timeout patterns.
• Spend and token usage by project, model, and provider.
• Rate-limit events and budget enforcement events.
• Authentication failures and unusual key activity.

The gateway itself can become a dependency. We plan for that by running it with the same discipline as other internal production services: health checks, controlled configuration changes, secret rotation procedures, and a documented incident path.

A practical migration path

We do not need to move every application at once. A staged migration reduces risk and gives us time to validate reporting and policy.

1. We deploy the gateway with a limited set of provider credentials and administrative access.
2. We define a virtual key format and required metadata fields.
3. We migrate one non-critical internal workload first.
4. We verify that its cost attribution, logs, budgets, and model behavior are understandable.
5. We migrate services incrementally and revoke direct provider access when each migration is complete.
6. We review unused virtual keys and direct provider keys on a regular schedule.

During migration, we avoid keeping permanent bypasses. A temporary direct provider key may be necessary for diagnosis, but it should have an owner, an expiry, and a documented reason. Otherwise the old access pattern quietly returns.

The decision we are making

Running our own AI gateway is not about hiding providers behind another layer for its own sake. We are choosing a control plane for AI access.

With LiteLLM, virtual keys, budget rules, consistent cost attribution, and careful self-hosted operations, we can give teams useful model access without making every person responsible for a provider account and credential. The trade-off is that we must operate and govern the gateway well. For us, that is preferable to distributing powerful upstream keys and reconstructing control after usage has already spread.