Lesson 6 of 13 · 5 min read
Self-Hosted Inference Servers: A Secure Reference Deployment
Teams increasingly run their own model servers, such as Ollama, vLLM, Triton, Ray Serve and llama.cpp. Most were designed for trusted networks: some have no authentication by default, expose model management next to inference, and rely on internal channels that deserialise data. Vulnerabilities and exposed servers followed. A secure deployment binds servers to loopback or a private segment behind an authenticating proxy, separates admin endpoints from inference, loads models only from an internal registry by digest, gives nodes no unnecessary cloud identity or egress and patches on a schedule.
free sign-in, no card required
Practise it as an interviewer asks it
Containers, gVisor, microVMs and VMs: where is each isolation boundary?Why are cloud metadata endpoints an SSRF target, and how do modern defences work?What is membership inference, and why does overfitting raise the risk?What does the shared responsibility model mean for a managed AI service?Threat-model an LLM inference API that receives untrusted input at scale.A Linux service running as a non-root user is compromised. What decides whether the attacker can become root, and what do you check?