Inside Byte-Pair Encoding (BPE): How LLMs Segment Characters into Subword Tokens
Modern transformer architectures—including OpenAI's GPT-5 and GPT-4.1 series, Anthropic's Claude 4.6, Google's Gemini 3.1 and 2.5, xAI's Grok 4, Meta's Llama 4, and DeepSeek V3/R1—process text through deterministic subword tokenization algorithms such as Byte-Pair Encoding (BPE) and SentencePiece. Rather than maintaining an infinite dictionary of every word in every language, BPE starts with individual UTF-8 bytes and iteratively merges the most frequently adjacent character pairs into subword tokens.
As a result, a common English word with a leading space like ` intelligence` is encoded as a single integer token ID, whereas an unfamiliar compound term, a hexadecimal hash, or a heavily indented Python block is fractured into dozens of individual tokens. Multinook's visual tokenizer reconstructs these subword boundaries in real time inside your browser RAM.

