How LLMs convert text to tokens: a primer

Tokens are the smallest units of text that language models process, not necessarily whole words. Tokenization converts raw text into token IDs, which are then embedded into vectors for transformer computation. The autoregressive loop predicts the next token repeatedly until a stopping condition is met.