Secure Model Output Handling for Web and API Applications

Every application security engineer knows that user input is untrusted. SQL queries are parameterized. HTML output is escaped. File paths from user requests are validated. The same discipline must extend to model output, which is also untrusted data from the perspective of the application consuming it.

Model output can contain attacker influenced content for two reasons. The model may have been prompted with attacker controlled input through prompt injection. The model may have memorized or reproduced malicious content from its training data. Either pathway can result in model output that, when rendered or processed without validation, introduces injection vulnerabilities.

Web rendering and XSS through model output

Any application that renders model output in a browser without HTML escaping is vulnerable to XSS if the model can be prompted with attacker controlled content. This includes chatbots, document summarizers, email assistants, and any other UI that displays model generated text.

The control is the same as for any other potentially unsafe HTML. Use a rendering framework that escapes by default. Never call innerHTML or equivalent APIs with model output. If Markdown rendering is required, use a sanitizing Markdown renderer that strips script tags, event handlers, and other active content.

Structured output and schema validation

The most effective control for reducing the output attack surface is requiring structured output from the model. When a model is constrained to produce valid JSON matching a defined schema, the possible output values are bounded by the schema. An attacker who wants to inject a script tag cannot do so if the output schema only permits alphanumeric identifiers.

Validate structured output against the schema before using it in any security sensitive context. Reject outputs that do not match the expected structure entirely rather than attempting to extract values from malformed output. A fail closed validation pattern prevents partial validation bypasses where an attacker crafts output that is almost correct.

1. 01Define a strict schema for the expected model output using a validation library. The schema in this example declares four fields. An action field constrained to the values create, read, or update. A resource type field constrained to ticket, comment, or attachment. A resource identifier field that must match a pattern of alphanumeric characters up to 64 characters long. A summary field limited to 256 characters.
2. 02Wrap the parsing and validation in a function that accepts the raw model output string and returns either the validated object or null. The return type is explicit so callers must handle the null case.
3. 03Inside the function, first parse the raw string as JSON. If parsing fails, the function returns null immediately. Do not attempt to extract values from malformed output.
4. 04After successful JSON parsing, run the parsed object through the schema validator. If any field is missing, has the wrong type, or violates a constraint, the validator throws and the function returns null.
5. 05Never attempt to sanitize or partially recover from a validation failure. Return null and log the failure with the raw output for later investigation. Callers that receive null must not proceed with the action.
6. 06This fail closed pattern means a model that produces unexpected output, whether due to a prompt injection or a reliability issue, results in no action being taken rather than an unpredictable one.

Downstream agent systems and output as instructions

When model output is passed to another system as instructions or input, the downstream system inherits the prompt injection risk of the upstream model. An attacker who can influence the upstream model output can potentially inject instructions into the downstream system.

Any architectural pattern where model output is used as input to another agent, tool, or language model should be treated as a high risk data flow. Apply schema validation at the boundary between systems. The schema should be as narrow as possible to support the legitimate use case. Free text passthrough from one model to another without a schema boundary is an uncontrolled injection pathway.

Testing model output handling

Include model output handling in standard AppSec testing. Provide test inputs to the model that attempt to produce XSS payloads, SQL injection strings, command injection patterns, and instruction injection for downstream agents. Verify that the application correctly handles each case through escaping, validation, or rejection.

Track output handling security coverage as a metric. The percentage of model output rendering paths that have been validated through security testing and confirmed to apply correct escaping or validation should be 100 percent for any application exposed to external users.