Private AI for Organizations That Cannot Send Content to a Public Model

Some organizations cannot send their documents, tickets, or customer records to a model endpoint that sits outside their network boundary. Healthcare providers, government agencies, and companies with contractual data-residency requirements fall into this group. The constraint is not a preference for smaller models. It is a requirement that the content used for retrieval or for generation stays inside a network the organization controls.

Azure provides several ways to meet that requirement. The practical choices are smaller models hosted inside a virtual network, Azure OpenAI with private endpoints and no public network access, and, for the narrowest cases, models that run on infrastructure the organization already operates. Each option has different limits on capability, cost, and operational effort.

Private endpoints and network isolation come first

The first decision is network placement, not model size. An Azure OpenAI resource can be configured with a private endpoint so that traffic from the application stays on the Microsoft backbone and does not traverse the public internet. The same pattern applies to Azure AI Search or to a vector store used for retrieval. The application, the model endpoint, and the retrieval store all sit in virtual networks that are peered or connected through private link.

This configuration does not by itself prevent the model provider from seeing the prompts. It does prevent the prompts from being exposed on a public endpoint, and it lets the organization apply its existing network controls and logging. For many regulated workloads, that distinction is the one that matters. We document the private endpoint configuration, the DNS setup, and the identity that the application uses, so that a later security review can verify the path without reconstructing it from the Azure portal.

Smaller models when the task fits inside their limits

When the organization also wants to avoid sending content to a large proprietary model, even over a private endpoint, the next option is a smaller model that can be hosted inside the same virtual network. These models handle classification, extraction, summarization of short passages, and structured generation more reliably than open-ended question answering over long documents. They are a reasonable choice for routing tickets, extracting fields from forms, or producing a first-pass summary that a person will edit.

They are a poor choice for assistants that need to synthesize answers from many pages of policy or technical documentation. In those cases the retrieval quality and the context window matter more than the model's general knowledge. A smaller model with a well-built retrieval pipeline can still be useful, but the pipeline has to be tighter, and the refusal behavior has to be more conservative, because the model has less capacity to notice when the retrieved passages do not actually answer the question.

Data handling and logging stay inside the boundary

Whatever model is chosen, the logs and the traces have to follow the same network rules as the prompts. A trace that includes the retrieved document chunks cannot be shipped to a monitoring service that sits outside the virtual network. We keep the detailed traces in a store that is reachable only from the application's network, and we send only aggregated counts, such as request volume and error rates, to any external dashboard.

Retention follows the organization's existing rules for application logs. If those rules require that prompts containing personal information be deleted after a set number of days, the trace store implements the same deletion. The model endpoint itself is configured so that it does not retain prompts for provider-side training or improvement, using the options Azure exposes for that purpose.

Operational cost is part of the choice

Private networking and smaller hosted models both add cost and operational work. Private endpoints require DNS and identity configuration that must be maintained when the application is updated. Smaller models may need more careful prompt design and more frequent evaluation because they fail in different ways than larger ones. The organization should expect to spend time on evaluation sets that reflect its own documents, because public benchmarks do not measure performance on its policies or its ticket language.

We help clients compare the options against the actual constraint. If the constraint is only that traffic must not cross the public internet, private endpoints with a larger model are usually sufficient and require less ongoing evaluation effort. If the constraint is that content must not leave infrastructure the organization controls, then a smaller model or a model running on their own compute is the path, and the project should include the evaluation and monitoring work that path requires.

What we recommend documenting

The decision record should state which content is in scope, which network boundary applies, which model and endpoint configuration was chosen, and how traces are stored and deleted. It should also name the person who will review evaluation results when the model or the retrieval corpus changes. Without that owner, the private deployment becomes a system that no one is measuring against the organization's own documents.

Private AI is not a different category of project from the other AI work we do. It is the same retrieval, skill, and evaluation practices, placed inside a network boundary the organization already understands how to operate. The boundary changes the infrastructure choices. It does not remove the need for a reviewed corpus, a versioned procedure, and a way to see when the answers stop matching the sources.