Insecure Output Handling

TL;DR: Treat model output exactly as you would treat input from an anonymous user, because an attacker who can put text into the model's context can shape it. Rendered as HTML, it can carry script (XSS). Concatenated into SQL, it is SQL injection. Passed to a shell, eval or a template engine, it is code execution. Used as a URL your server fetches, it is SSRF. The controls are the classic ones, applied at each sink: context-aware encoding, parameterised queries, allowlisted commands with structured arguments, schema-validated tool calls, a content security policy and egress limits.

Why output is untrusted

The model writes what its context leads it to write. That context contains the user's message and possibly retrieved documents, web pages and tool results written by third parties. Any of them can influence output through prompt injection, and some do not even need injection: a user can simply ask for the output they want. So model output is not "our" data in the security sense; it is attacker-influenced data that happens to pass through our model.

The bugs that follow are not new. They are the classic injection vulnerabilities, reached through a new path.

Sinks and what goes wrong

When model output reached exec: three CVEs

This is not a theoretical class. Early LLM frameworks shipped features whose design was "the model writes code and we run it", and each became a remote code execution vulnerability once someone asked what an attacker could put in the model's context.

The pattern is the same in all three: the model call is a source of untrusted data, exec is a code-execution sink, and nothing in between constrained what could flow from one to the other. The fix in each case is the same too: stop executing generated code in the application process.

Worked example: reviewing a text-to-chart feature

A product lets users ask questions about their sales data and get a chart, in the style of the Vanna case. The code generates SQL and Python plotting code with a model and runs both. A source-to-sink review goes like this:

1. Mark the source. Every model call is a taint source. Its input includes the user's question and any text the model reads from the database, such as product names that customers can edit.
2. Find the sinks. Generated Python goes to exec: code injection. Generated SQL goes to the database driver: SQL injection, with whatever privileges the connection has.
3. Fix the chart sink by removing it. Have the model return a chart specification instead of code: a chart type from an enum, column names checked against the result schema, an aggregation from an enum. Fixed plotting code renders the specification.
4. If generated code really is the product, run it in a sandbox with no network, no credentials and strict limits on time and memory, never in the web process. See sandboxing untrusted code.
5. Constrain the SQL leg as described below: ideally the model produces parameters for fixed queries; if it must write SQL, run it with a read-only role over views limited to the user's own rows.

Structured output is a format guarantee

Structured outputs and constrained decoding make the model's response match a JSON schema. That removes a whole class of parsing bugs, and it is worth using. It does not make the values safe. A schema-valid {"image_url": "..."} can hold an attacker's URL; a valid order_id can name someone else's order; a valid sql string is still SQL. After the shape is guaranteed, validate the values: URLs against an allowlist, identifiers against what this user may access, free text encoded for wherever it is going.

Text-to-SQL when it is the product

Some products exist to answer questions with generated SQL. Then the goal is to make the worst possible query harmless.

The tenant or user filter must be set by the server from the session, for example as a session variable that a row-level security policy reads, never as a value the model writes into the query.

Rendering: sanitise the HTML you produce

Markdown is not one syntax. Links and images can be written inline, in reference style with the URL defined elsewhere, as autolinks, or as raw HTML if the renderer allows it. The EchoLeak vulnerability in Microsoft 365 Copilot used reference-style markdown to get past redaction that handled only the inline form. The robust approach is to render markdown to HTML first and then sanitise the resulting HTML with an allowlist of elements and attributes, so every syntax is checked in one form. Do not auto-load images or link previews from model output; the channels and the vendor fixes are covered in data exfiltration channels.

Patterns that make output safe by construction

• Ask for data, not code. Have the model produce structured values (an order ID, a date range, a filter enum) validated against a schema, and let your code build the query or command. A model that never writes SQL cannot write malicious SQL.
• Least privilege at the sink. If generated queries are unavoidable, run them on a read-only role against views limited to what the user may see, with timeouts and row limits.
• No general interpreters. Replace "run this shell command" with a small set of named operations. Where code execution is the product, isolate it in a sandbox (see sandboxing untrusted code).
• Render safely by default. Encode text for its context, sanitise markdown with an allowlist, set a strict content security policy, and do not load images or previews from arbitrary origins (see data exfiltration channels).
• Validate tool calls against schemas and authorize them in the backend.

Misconceptions interviewers listen for

• "Structured output makes it safe." It guarantees format, not that a URL, ID or string is acceptable.
• "We sanitise the markdown." Which syntaxes? Sanitise the rendered HTML instead.
• "A content security policy solves image exfiltration." Allowed first-party proxies and redirectors can still carry data.
• "Our model is aligned, so it will not emit script." A user can simply ask, and injected text can arrange it.

How this shows up in interviews

Expect both conceptual and practical versions: "why is LLM output untrusted?" and a code review where model output flows into subprocess, a SQL string or innerHTML. In a code review, trace from source (the model call) to sink, name the vulnerability class, and fix it at the sink with the classic control. Saying "we'll tell the model not to output dangerous content" is the answer that fails.

Say it in 30 seconds: "Model output is shaped by everything in its context, including attacker-controlled text, so I treat it like anonymous user input. If it reaches a browser it can be XSS, SQL it can be injection, a shell or eval it's code execution, a URL fetcher it's SSRF. The fixes are the classic ones at each sink: encoding, parameterised queries, structured operations instead of interpreters, schema validation for tool calls, CSP and egress allowlists. Best of all, have the model produce validated data and let code build the query."