Reliable JSON From LLMs: Structured Outputs in Practice
Free-form text is fine for chat, but application code needs predictable structure. Here is how to get it consistently.
Most LLM integrations eventually need structured data: a classification, a list of extracted fields, a decision with a reason. Asking the model to "respond in JSON" gets you there most of the time, and "most of the time" is exactly the problem. A 2% parsing failure rate becomes hundreds of broken requests a day at scale.
Use the provider's structured output or tool-calling features when they are available. Passing a JSON schema lets the model generate output constrained to that shape, which removes trailing commentary, mismatched quotes and invented keys. It also documents the contract in one place that both the prompt and the parser share.
Validate the result anyway. A schema-constrained response can still contain semantically wrong values: an impossible date, an enum spelled with a different case, a confidence score of 120. Runtime validation with clear error messages catches those before they reach the database. When validation fails, retry once with the error message appended; models are good at correcting a specific, named mistake.
Design the schema for the model, not only for your database. Fewer, well-named fields with short descriptions work better than deeply nested objects. Put a "reasoning" field before the decision field if you want the model to think before it commits; order matters because generation is sequential.
Keep a corpus of real inputs and expected outputs, and run it whenever you change the schema, the prompt or the model version. Structure is only reliable if you measure it.
