Tool2025-09-03#tooling

Keep a tokenizer open in a tab

Leading spaces, casing, and digit grouping all change how text is split, and the model only ever sees the tokens, never your characters. A trailing space can silently sabotage a few-shot format.

When something is off and you cannot explain it, paste the exact string into a tokenizer view before you theorize. The answer is in there more often than you would expect.

Aha moment

The model never sees your characters. It sees chunks, spaces, merges, and weird number splits.

Try this

Paste the exact prompt, including spaces and separators, into a tokenizer before debugging prompt behavior.

Open NotesEntry JSON

Watch for

  • Leading spaces changing label tokens
  • Numbers split into surprising chunks
  • Few-shot examples with inconsistent separators