Structured Output Generation

Source Excerpt

Techniques for constraining Large Language Model output to specific formats — JSON, XML, CSV, or custom schemas — enabling reliable parsing and downstream processing. Critical for production applications where output must be machine-readable.

Approaches

JSON Mode (Provider-Native)

Most providers offer a "JSON mode" that constrains output to valid JSON.

OpenAI: response_format: { "type": "json_object" }

Anthropic: System prompt: "Always respond in valid JSON"

Gemini: response_mime_type: "application/json"

Limitations: Ensures valid JSON but not valid schema. Model may produce JSON with wrong structure.

JSON Schema (Provider-Native)

Providers increasingly support JSON schema validation at the model level.

OpenAI strict mode: strict: true with JSON schema. Model is trained to produce schema-valid output.

Anthropic: Tool-use format inherently produces structured output.

Advantage: Model-level enforcement, highest reliability.

Regex Constraints

Use regex patterns to constrain output format.

Libraries: guidance, outlines, lm-format-enforcer

from outlines import generate, models

model = models.transformers("mistralai/Mistral-7B-Instruct-v0.2")
generator = generate.regex(model, r"\d{4}-\d{2}-\d{2}")  # Date format
output = generator("What is today's date?")

Advantage: Precise control over output format.

Limitation: Complex schemas become unwieldy regex.

Grammar-Based Constraints

Use formal grammars (EBNF, CFG) to define output structure.

Libraries: guidance, xgrammar

from guidance import models, gen

grammar = """
json_object ::= '{' key_value_pairs '}'
key_value_pairs ::= key_value (',' key_value)*
key_value ::= '"' string '"' ':' value
value ::= '"' string '"' | number | 'true' | 'false' | 'null'
"""

Advantage: Expressive, handles complex nested structures.

Limitation: Requires grammar expertise.

Pydantic Validation (Post-Generation)

Generate free-text, then validate with Pydantic.

from pydantic import BaseModel, Field

class WeatherResponse(BaseModel):
    location: str = Field(description="City and state")
    temperature: float = Field(description="Temperature in Fahrenheit")
    conditions: str = Field(description="Weather conditions")

# Parse LLM output
import json
response = json.loads(llm_output)
validated = WeatherResponse(**response)

Advantage: Leverages existing validation infrastructure.

Limitation: May require retry loops for invalid output.

Library Comparison

Best Practices

Prompt Design

• Explicit format instructions: "Respond in JSON format with the following structure:"
• Provide examples: Few-shot examples of valid output
• Describe each field: Include descriptions in schema
• Specify types: "temperature should be a number, not a string"

Schema Design

• Keep it simple: Fewer fields = higher compliance
• Use enums: Constrain string fields to known values
• Required vs optional: Mark only truly required fields
• Nested objects: Limit nesting depth (2-3 levels max)

Error Handling

• Retry with feedback: On parse failure, send error to model for retry
• Partial parsing: Extract what you can, request missing fields
• Fallback: If structured output fails, fall back to free-text parsing

Reliability Techniques

Temperature and Top-P

• Lower temperature (0.1-0.3): More deterministic, higher structure compliance
• Lower top-p (0.9-0.95): Reduces creative variation
• Temperature 0: Most deterministic, but may reduce quality

Max Tokens

• Set reasonable max_tokens to prevent truncation
• Estimate output size based on schema complexity

Validation Pipeline

Large Language Model output → JSON parse → Schema validate → Pydantic model → Success/Fail
                                    ↓ (fail)
                              Retry with error feedback (max 3 retries)
                                    ↓ (fail)
                              Fallback to free-text parsing

Common Use Cases

Data Extraction

{
  "name": "John Smith",
  "email": "john@example.com",
  "phone": "555-123-4567",
  "company": "Acme Corp"
}

Classification

{
  "sentiment": "positive",
  "confidence": 0.92,
  "topics": ["product", "shipping"],
  "urgency": "low"
}

Code Generation

{
  "language": "python",
  "code": "def hello():\n    print('Hello, world!')",
  "explanation": "Simple greeting function"
}

Search Queries

{
  "query": "industrial automation PLC programming",
  "filters": {
    "date_range": "2024-2026",
    "type": "technical_documentation"
  }
}

Related

• ai-ml/function-calling
• ai-ml/tool-calling
• ai-ml/model-context-protocol