Let’s save AI tokens without losing quality. Is it possible? Yes, it is. But how? Learn about TOON (Token-Oriented Object Notation), designed to be highly token-efficient.

Let’s save AI tokens without losing quality. Is it possible? Yes, it is. But how?

I’ve heard that using JSON can help get perfect, structured output from a large language model (LLM). But what other formats can provide that same perfect output without losing prompt quality, all while saving tokens?

This brings us to something like TOON (Token-Oriented Object Notation). Similar to JSON (JavaScript Object Notation), TOON is a way to write objects and documents, but it’s specifically designed to be more token-efficient.

The Problem with JSON in LLM Contexts

JSON was designed for machine-to-machine communication over HTTP, not for LLM context windows. Every bracket `{`, every brace `[`, every comma `,`, and every double quote `"` consumes tokens. When you are passing thousands of rows of structured data into a prompt, this overhead can account for up to 40% of your total context window.

Consider the financial and latency implications: 1. Cost: API providers like OpenAI and Anthropic charge per token. A 40% overhead means you are overpaying by 40%. 2. Latency: Time-to-first-token (TTFT) and total generation time scale linearly with output token length. 3. Context Limits: If your context window is 128k tokens, wasting 50k tokens on JSON syntax reduces the actual informational density you can provide to the model.

Here’s an example. This is how we typically write data in JSON:

{
  "users": [
    { "id": 1, "name": "Alice", "role": "admin" },
    { "id": 2, "name": "Bob", "role": "user" }
  ]
}

Now, here is how we would write that same data in TOON:

users[2]{id,name,role}:
  1,Alice,admin
  2,Bob,user

As you can see, the TOON format is much more compact.

Core Features of TOON

The key features of TOON make it highly appealing for AI engineering:

• Token-efficient: Typically uses 30–60% fewer tokens than JSON. In our benchmarks across 10,000 data rows, TOON reduced token usage from 145,000 down to 62,000.
• LLM-friendly guardrails: Explicit lengths and fields enable easier validation. The model knows exactly how many items to generate before it starts.
• Minimal syntax: Removes redundant punctuation. The LLM's attention mechanism doesn't have to waste capacity attending to structural syntax; it can focus purely on semantic content.
• Indentation-based structure: Like YAML, it uses whitespace instead of braces. Whitespace is often collapsed or tokenized efficiently by modern tokenizers (like cl100k_base).
• Tabular arrays: You declare the keys once, and then stream the data as rows. This mimics CSVs, which LLMs are exceptionally good at parsing.

A Deeper Analysis for Maximum Savings

However, based on a deeper analysis, if you really want to use TOON to save your data and tokens, there’s a specific structure to follow to ensure the LLM doesn't hallucinate the format.

1. Prompt Priming: You must provide a 1-shot example in your system prompt. LLMs are heavily biased toward JSON. If you just ask for TOON without an example, the model might revert to pseudo-JSON. 2. Two-Step Conversion: First, convert your data into a flat JSON format in your application logic. Then, convert that flat JSON into the TOON format before injecting it into the prompt string. 3. Post-Processing Verification: Write a strict parser in your application layer. If the LLM misses a comma in the CSV-like rows, your parser should gracefully handle it or trigger a fast retry.

Real-World Case Study: RAG Optimization

In a recent Retrieval-Augmented Generation (RAG) pipeline build for a legal tech startup, we retrieved 50 case law summaries per query. Representing these summaries in JSON consumed 85,000 tokens per prompt.

By migrating the prompt injection format to TOON: - Token usage dropped to 42,000 tokens. - Latency improved by 2.4 seconds per query. - API costs were slashed by