If you are serious about using AI on sensitive data – customer records, health information, financial transactions, source code – there is a brutally simple problem you run into fast:

At some point, the data has to be decrypted so your model can use it.

You can encrypt data “at rest” in storage, and “in transit” over the network, but during training or inference, it traditionally sits in memory as plain text. Anyone who can compromise the host, hypervisor, or cloud admin layer can potentially see everything. For many teams, that single fact is the difference between “cool demo” and “absolutely not going to production.”

That is the gap confidential computing is trying to close – and it is especially important now that you are feeding tools like ChatGPT, Claude, Gemini, or custom LLMs with highly sensitive inputs and proprietary models. You want the benefits of modern AI, without trusting every layer of the stack or every person who might have admin access.

What is confidential computing, really?

Confidential computing is about protecting data in use – while it is being actively processed – using hardware-based isolation.

Under the hood, this is usually done with a Trusted Execution Environment (TEE) on the CPU or GPU. A TEE is an isolated, hardware-enforced “bubble” where code runs and data stays encrypted in memory, invisible even to the operating system, hypervisor, and cloud provider admins.

Industry definitions and standards bodies describe confidential computing as extending cryptographic protections to data while in use, typically using TEEs and memory encryption so that workloads are shielded from the underlying infrastructure and other tenants in the cloudsource.

For AI, that means:

  • Your training data is decrypted only inside the enclave/TD (trust domain), not visible to the host.
  • Your model weights live in encrypted memory, protecting valuable IP.
  • Your prompts and outputs stay shielded from cloud operators and many common attack paths.

Why AI workloads are a perfect (and scary) target

AI workloads are unusually attractive for attackers and regulators alike:

  • They concentrate highly sensitive data (think entire medical histories, legal case archives, or private chats).
  • They run on shared cloud infrastructure with complex software stacks – drivers, containers, orchestration, monitoring.
  • They may embed extremely valuable IP in model weights and fine-tuning data.

Research and industry guidance explicitly call out that TEEs and confidential computing can protect the integrity and confidentiality of data during AI training and inference, as well as safeguard non-public models from theft or adversarial attackssource.

So if you are building anything like:

  • A retrieval-augmented chatbot over private documents using something like ChatGPT, Gemini, or Claude as a backend.
  • A fine-tuned LLM on customer support transcripts or financial records.
  • A healthcare or government model where regulations require strict control over where and how data is processed.

…then you care not only about secure APIs and VPCs, but about what happens on the actual silicon.

How the big clouds are doing confidential AI today

The good news is that you no longer have to assemble this from scratch – all major clouds now offer confidential computing options, and they are starting to be wired directly into AI services.

Azure Confidential Computing and Confidential AI

Microsoft Azure provides confidential VMs and services that use hardware-backed TEEs, and explicitly positions this as the foundation for “Confidential AI.” With Azure Confidential Computing, workloads run inside hardware-based protected environments so that even Microsoft cannot access unencrypted customer data when the feature is correctly configuredsource. Azure documents how customers are already using these capabilities to build confidential AI solutions, including AI workloads that rely on GPUs in collaboration with NVIDIAsource.

In practical terms for you, this looks like:

  • Spinning up confidential VMs for model serving.
  • Running your own inference server (for example with an open-source LLM) inside a TEE.
  • Using Azure services that add confidential computing under the hood for AI workloads.

Intel SGX and TDX: protecting AI in CPU-based environments

On the CPU side, Intel has been driving much of the confidential computing evolution.

  • Intel SGX (Software Guard Extensions) introduced enclave-style protection for applications, and helped define the broader “confidential computing” paradigmsource.
  • Intel Trust Domain Extensions (Intel TDX) extend this idea to full confidential VMs, called Trust Domains, where the memory of the VM is encrypted and isolated from the hypervisor and other tenantssource.

Intel explicitly calls out confidential AI as a key use case: its confidential AI capabilities are built on technologies like Intel SGX and Intel TDX, intended to help secure and isolate sensitive data, AI models, and workloads in multi-tenant environmentssource.

This is already live in real AI deployments. For example, Baidu AI Cloud has built confidential computing IaaS products for large language model workloads using 5th Gen Intel Xeon processors with Intel TDX, to protect data confidentiality and code integrity while maintaining performancesource.

Confidential AI on GPUs: NVIDIA Hopper and beyond

GPUs are where most serious AI training and high-throughput inference runs today, so protecting data and models on GPUs is crucial.

NVIDIA’s Hopper architecture (H100, H200) introduced the world’s first accelerated computing platform that integrates confidential computing features directly on the GPU. According to NVIDIA, Hopper-based GPUs provide confidential computing capabilities that protect the confidentiality and integrity of data and applications while still delivering high performance for AI training and inferencesource. Technical documentation on secure AI with Hopper and the successor Blackwell architecture further details how the hardware enforces confidentiality, integrity, and availability for sensitive AI workloadssource.

In practice, this enables:

  • Confidential inference on GPUs, where prompts, activations, and model weights stay encrypted and isolated from the host.
  • Confidential containers that run entire AI workloads inside protected environments using supported GPUs like the H100source.

This matters if you are offloading heavy inference from something like ChatGPT or Gemini to your own infrastructure and need cloud-level acceleration without fully trusting the infrastructure provider.

Key building blocks: how confidential AI actually works

While vendors name things differently, confidential AI typically relies on a few common building blocks.

1. Hardware TEEs and memory encryption

On CPUs (Intel TDX, AMD SEV, etc.) and GPUs (NVIDIA Hopper/Blackwell), the hardware:

  • Encrypts memory regions for a given VM or enclave.
  • Isolates that memory so even the hypervisor or host OS cannot read it directly.
  • Protects registers and other state when switching between secure and normal worlds.

This is what lets you say “even the cloud provider should not be able to see my data while my model is running.”

2. Remote attestation

A TEE by itself is not enough – you need a way to know that:

  • Your code is running on real, genuine hardware.
  • The firmware and TEE configuration have not been tampered with.
  • The exact measurement (hash) of the code you expect is what is actually running.

This is what remote attestation provides. For example, Intel’s documentation on Intel Trust Authority and TDX describes how confidential VMs can produce attestation reports that external verifiers can check before releasing secrets or sensitive data to the workloadsource.

In an AI context, you might:

  • Verify an attestation report.
  • Only then send encryption keys for your training data or model weights.
  • Keep your data and model unusable unless the right environment proves itself.

3. Secure I/O and key management

AI stacks are complex – you still need to move data in and out and manage encryption keys:

  • Secure I/O paths ensure that data is encrypted as it enters and exits the enclave/TD so it is not exposed in the clear on the PCIe bus or other components.
  • Key management solutions (such as hardware-backed key vaults integrated with confidential computing) can generate and store keys in a way that even infrastructure operators cannot access. Industry solution briefs show how secret management systems like HashiCorp Vault can leverage Intel TDX-based confidential computing to keep secrets protected from hosts and administratorssource.

Where confidential computing helps AI – and where it does not

It is important not to treat confidential computing as a magic shield. It is powerful, but it has a specific job.

Helpful for:

  • Protecting against infrastructure insiders – rogue admins or compromised hypervisors seeing your training data or model.
  • Multi-tenant isolation – ensuring one tenant’s AI workload cannot spy on another’s in shared cloud environments.
  • Regulatory arguments – showing that even your cloud provider cannot access plain-text data, which can matter under regimes like GDPR and sector-specific rules.
  • IP protection – safeguarding proprietary model weights and training pipelines when running in third-party clouds.

Not a silver bullet for:

  • Bad prompts or access control – if you give too many people access to internal chatbots or model APIs, confidential computing will not fix over-sharing.
  • Model misuse – it does not stop someone with legitimate access from abusing the system.
  • Application-layer bugs – data can still leak if your code logs the wrong things or exposes unexpected APIs.

Think of confidential computing as moving your AI workload into a locked, shielded room. It still matters who you let in and what they run inside that room.

How to start using confidential computing for your AI projects

You do not have to redesign your whole stack to get practical benefits. You can start incrementally:

  1. Map your highest-risk AI workloads.
    Identify where you are training or serving models on the most sensitive data – healthcare, finance, internal source code, proprietary datasets, or customer-specific deployments (for example, a fine-tuned version of a model like Claude or Gemini for a single client).

  2. Check what your current cloud already offers.

    • On Azure, look at Azure Confidential Computing and confidential VM SKUs, and how they tie into confidential AI options on CPUs and GPUssource.
    • On other clouds, look for confidential VMs or enclave-based services, plus GPU support if you need it.

  3. Pilot a single confidential AI workload.
    Take one high-value microservice or model (for example, a retrieval-augmented generation API over sensitive documents) and:

    • Run its inference server inside a confidential VM (e.g., using Intel TDX-based instances or equivalent).
    • Integrate attestation: only send the encryption key for the vector store or model when the environment proves it is genuine.
    • Measure performance and operational overhead so you know what to expect as you scale.

As AI becomes the default interface for your most sensitive data, the cost of ignoring “data in use” security will only go up. Confidential computing is not hype at this point – it is real hardware, in real clouds, already underpinning real-world AI deployments. The sooner you get hands-on with it, the more confidently you can say “yes” to ambitious AI projects without quietly worrying about who can see what under the hood.