neural / tokens
Token Explorer
Real tokenization via js-tiktoken and llama-tokenizer-js. Every token ID, boundary, and byte count is computed client-side from production tokenizer vocabularies.
Input
10
Tokens
44
Characters
4.40
Chars/Token
44
Bytes
200,019
Vocab Size
$<0.001
Cost (input)
Context window (claude-3.5-sonnet)10 / 200,000 (0.005%)
Token Boundaries
The·quick·brown·fox·jumps·over·the·lazy·dog.
Top Tokens
the2
quick1
brown1
fox1
jumps1
over1
lazy1
dog1
Compression
Chars/Token4.40x
Tokens/Word1.11x
Bytes/Token4.40x
English prose ~4 chars/token with BPE. Code and non-Latin scripts compress less.
How these tokenizers work
o200k_base (GPT-4o) — BPE with 200k vocabulary optimized for multilingual text and code.
cl100k_base (GPT-4) — BPE with 100k tokens. Fewer merges = more splits on non-English text.
Llama 2 — SentencePiece (BPE-based) with 32k tokens. Smaller vocab = more tokens per text.