Lesson 7 of 13 · 13 min read

Multi-Tenant Inference Isolation: Caches, Batches, GPUs and Side Channels

Serving models cheaply means sharing: prompt caches, response caches, batches, adapters, GPUs and network paths. Each shared resource leaks wherever it is keyed by something coarser than the trust boundary. Researchers found providers sharing prompt caches across customers, recovered prompts from scheduling order, read GPU memory left by other processes, and inferred chat topics from encrypted packet sizes. Scope every cache by a server-derived tenant key, pick GPU sharing modes by isolation guarantee, pad streams, and remember confidential computing removes the operator, not co-tenants.