Microsoft Foundry and Foundry Local: Where should your AI run?

Microsoft Foundry and Foundry Local address these questions from two different directions. Microsoft Foundry provides a managed cloud platform for building and governing AI applications. Foundry Local brings AI inference directly to user devices. Used together, they enable cloud, local, and hybrid application architectures.

Why local AI is becoming relevant

Cloud-hosted models remain the natural choice for many applications. They offer access to powerful models without requiring customers to manage the underlying infrastructure.

However, not every workload needs—or should use—a cloud model. Local inference can be useful when teams want to:

• Test and compare models quickly
• Prototype applications without per-call cloud inference costs
• Continue development when cloud credits or service access are limited
• Keep confidential information on the device
• Support offline or limited-connectivity scenarios
• Reduce latency by avoiding network round trips

The important question is therefore not whether cloud or local AI is universally better. It is which option is more suitable for each workload.

What is Microsoft Foundry?

Microsoft Foundry is a unified platform for building, grounding, operating, and governing AI applications and agents.

Its model catalog includes more than 1,900 models from Microsoft, OpenAI, Anthropic, Mistral, xAI, Meta, DeepSeek, Hugging Face, and other providers. This gives development teams a broad choice without requiring every model experiment to use a completely different toolchain.

Foundry is not limited to model access. Teams can build agents that combine models with tools, knowledge sources, memory, routines, and guardrails. SDKs for languages including C#, JavaScript, Python, and Rust provide a consistent developer experience.

Foundry also supports evaluation, tracing, monitoring, identity management, role-based access control, policies, networking, security, and compliance controls.

Understanding the cost model

The cost of an AI application is not determined only by the selected model. It depends on how the model is hosted and on the supporting architecture.

With Azure-hosted models, billing is normally handled through an Azure subscription. This includes Azure OpenAI models and selected partner models, commonly charged according to usage - pay-as-you-go.

Foundry can also act as the development and orchestration layer while the application calls an external provider. In this bring-your-own-key model, authentication, quotas, pricing, and contractual terms are managed by the external provider.

Additional Azure services can create separate costs. An AI application may also need storage, search, compute resources, or monitoring. Product teams should therefore estimate the cost of the complete application architecture rather than looking only at the price per model token.

What is Foundry Local?

Foundry Local is an on-device runtime for AI inference. Instead of sending each request to a remote service, an application can execute supported models directly on local hardware.

This approach offers several practical benefits:

• Data privacy: Text, audio, images, prompts, and generated outputs can remain on the device.

• Offline operation: After the runtime and models have been downloaded, applications can work with limited or no connectivity.

• Lower latency: Local processing removes the network round trip, which can improve interactive and real-time scenarios.

• Lower cloud inference costs: Suitable workloads can use existing device hardware instead of generating a charge for every cloud request.

Local inference still has limits. Performance depends on the available CPU, GPU, or NPU, as well as memory and storage. Smaller optimized models are usually better suited to this environment than the largest cloud models.

How the local runtime works

Applications communicate with Foundry Local through an SDK. The SDK calls the Foundry Local Core API, which manages model lifecycles, inference sessions, and hardware abstraction.

The actual model execution is handled by ONNX Runtime. This open-source, cross-platform inference engine runs models in the ONNX format independently of the framework originally used to train them.

ONNX Runtime applies graph optimizations and optimized execution kernels. Through its Execution Providers, it can use the best supported hardware accelerator available on the device. Inference may run on a GPU or NPU, with CPU available as a fallback.

The Foundry catalog provides model discovery, versioning, and hardware-optimized model variants. Models are downloaded into a local cache and can then run directly from disk.

This architecture allows application developers to work with a consistent API while the runtime handles many hardware-specific details.

A developer workflow

The Foundry CLI provides a fast way to explore local AI before adding it to an application.

On Windows, Foundry Local can be installed with:

winget install Microsoft.FoundryLocal

On macOS:

brew install foundrylocal

Developers can verify the installation and inspect available commands:

foundry --version
foundry --help

They can then discover and test models:

foundry model list
foundry model info phi-4-mini
foundry model run phi-4-mini

The CLI also provides commands for inspecting the model cache and checking the local service:

foundry cache list
foundry cache location
foundry service status

This creates a short development loop: discover a model, inspect its requirements, run a prompt, evaluate its behavior, and only then integrate it into the application.

Cloud, local, or hybrid?

Each deployment model has different strengths.

A hybrid architecture is often the most practical option. The application can use a local model as its fast and private default path, then escalate more complex requests to a cloud model.

For example, a document application might summarize confidential content locally while using a cloud model for a complex reasoning task—but only after removing sensitive information or receiving user approval.

Routing can be based on:

• Data sensitivity
• Task complexity
• Device capabilities
• Network availability
• Response-time requirements
• Cost limits
• Organizational policies

Showcase: A hybrid AI in action

How could an application that uses cloud, local, and hybrid AI look? Imagine an application for processing meeting notes. The notes may contain sensitive information and can vary in length.

As the demo shows, a hybrid approach is chosen, where the local model processes notes that contain sensitive information. The cloud model is used for summarizing notes that do not contain sensitive information. Routing is determined based on whether the data contains sensitive information or based on the length of the notes.