neural / tokens

Token Explorer

Real tokenization via js-tiktoken and llama-tokenizer-js. Every token ID, boundary, and byte count is computed client-side from production tokenizer vocabularies.

Input
10
Tokens
44
Characters
4.40
Chars/Token
44
Bytes
200,019
Vocab Size
$<0.001
Cost (input)
Context window (claude-3.5-sonnet)10 / 200,000 (0.005%)
Token Boundaries
The·quick·brown·fox·jumps·over·the·lazy·dog.
Top Tokens
the
2
quick
1
brown
1
fox
1
jumps
1
over
1
lazy
1
dog
1
Compression
Chars/Token4.40x
Tokens/Word1.11x
Bytes/Token4.40x
English prose ~4 chars/token with BPE. Code and non-Latin scripts compress less.
How these tokenizers work

o200k_base (GPT-4o) — BPE with 200k vocabulary optimized for multilingual text and code.

cl100k_base (GPT-4) — BPE with 100k tokens. Fewer merges = more splits on non-English text.

Llama 2 — SentencePiece (BPE-based) with 32k tokens. Smaller vocab = more tokens per text.