What are tokens?
Tokens are the basic units of text that Large Language Models read and write. Instead of processing whole words or individual letters, models break text into these manageable chunks to predict what comes next.
When you use an LLM, you might think of it as reading a sentence word by word. In reality, the model sees a sequence of tokens. A token is the fundamental unit of text a model processes.
- Tokens are the units of text a model reads and writes.
- They are often a short word or a part of a word.
- Models do not read whole words or individual letters; they use a tokenizer to split text into these pieces.
For example, a simple word like "apple" might be a single token. However, a complex or rare word might be split into several smaller pieces to help the model understand it.
How Tokenization Works
To turn human language into something a machine can process, we use a process called tokenization. A tokenizer is the tool that performs this split.
Most modern tokenizers use a method like byte-pair encoding. This method learns frequent chunks of text from the massive datasets used during training. This allows the model to handle a vast vocabulary efficiently without needing a unique entry for every single possible word in existence.
- A tokenizer splits text into tokens using methods like byte-pair encoding.
- Common words often become a single token.
- Rare words are split into several smaller pieces.
Practical Impact: Cost, Speed, and Language
For business owners and developers, tokens are not just a technical detail—they are a direct driver of your bottom line. If you are building an application, you must account for how tokens affect your operational metrics.
| Factor | Impact of Token Usage |
|---|---|
| Cost | Most LLM providers charge you based on the number of tokens processed and generated. |
| Speed | The total number of tokens in a request directly affects inference speed (latency). |
| Language Support | The efficiency of a language depends on how well the tokenizer was trained on it. |
This leads to a significant disparity in costs across different languages. Languages that were under-represented in a tokenizer's training data—such as many Indian languages written in their own scripts—often require more tokens to represent the same sentence. This makes them slower and more expensive to process on many models.
If you are building a tool for a local market, always test the token efficiency of your target language. A sentence in English might be 10 tokens, but the same sentence in a different script might be 30 tokens, tripling your cost for that specific task.
From Tokens to Embeddings
Once a tokenizer has split your text into tokens, the model needs to understand what those tokens actually mean. This is where embeddings come in. Each token is mapped to an embedding, which is a list of numbers known as a vector.
These vectors are not random. The mapping is designed so that tokens with related meanings end up with similar vectors in a high-dimensional space. This mathematical relationship is what enables advanced AI features like semantic search and Retrieval-Augmented Generation (RAG).
For example, in a vector space, the token for "doctor" will be mathematically closer to "hospital" than it is to "bicycle." This allows the model to understand context and meaning rather than just matching exact words.
Tokens and the Context Window
Every model has a limit on how much information it can "keep in mind" at any one time. This limit is known as the context window. It is the maximum number of tokens a model can consider at once.
It is important to remember that the context window is not just for the user's current question. It includes:
- Your specific instructions (system prompts).
- Any documents or data you have provided.
- The entire conversation history.
- The model's own previous replies.
While larger context windows allow you to feed in massive amounts of data, such as long legal contracts or entire books, more data isn't always better. Key information is most effective when placed clearly near the question within that window. For more on how these limits work, you can read our guide on the context window.
Predicting the Next Token
At its core, an LLM is a prediction engine. When you give a model a prompt, it doesn't "know" the answer in the human sense. Instead, it calculates a probability for every possible next token in its vocabulary. It then picks one of those tokens to output.
To control how "creative" or "strict" the model is, developers use sampling settings. These settings decide which token is actually picked from the list of probabilities:
- Temperature: This setting sharpens or flattens the probability distribution. A low temperature leads to predictable, consistent output, which is great for tasks like extracting data or writing code. A higher value allows for more variety and "creativity," which is useful for brainstorming.
- Top-p (Nucleus Sampling): This method keeps only the smallest set of tokens whose probabilities add up to a specific value (p), then samples from that set. This helps cut off unlikely, nonsensical tokens.
Understanding these settings is vital for business applications. If you are building a bot to fill out a booking form from a WhatsApp message, you want low temperature to ensure the output is structured and accurate.
Tokens in Model Training
The intelligence of a model is built through a massive process called pretraining. During this phase, the model is shown trillions of tokens of text and code.
The model learns by playing a game of "what comes next?" It looks at a sequence of tokens and tries to predict the next one. Every time it makes a wrong guess, the underlying mathematical parameters are adjusted slightly to reduce the error. This process is repeated billions of times until the model develops an understanding of grammar, facts, and even reasoning patterns.
The goal of training is to reduce the error in these next-token predictions. The more high-quality tokens a model sees during this stage, the more capable it becomes at understanding the nuances of human language.
Frequently Asked Questions
- Do LLMs read whole words?
- No, models use a tokenizer to split text into tokens, which can be whole words, parts of words, or even single characters.
- Why are some languages more expensive to process?
- If a language was under-represented in the tokenizer's training data, it may require more tokens to represent the same sentence, increasing cost and latency.
- What is a context window in terms of tokens?
- The context window is the limit on how many tokens (including instructions and history) the model can process at one time.
- How do tokens relate to embeddings?
- Each token is converted into an embedding, which is a vector of numbers that represents the token's meaning in a high-dimensional space.
- What is the difference between a word and a token?
- A word might be a single token, but complex or rare words are often broken down into multiple tokens by the tokenizer.

Questions people ask
Do LLMs read whole words?
No, models use a tokenizer to split text into tokens, which can be whole words, parts of words, or even single characters.
Why are some languages more expensive to process?
If a language was under-represented in the tokenizer's training data, it may require more tokens to represent the same sentence, increasing cost and latency.
What is a context window in terms of tokens?
The context window is the limit on how many tokens (including instructions and history) the model can process at one time.
How do tokens relate to embeddings?
Each token is converted into an embedding, which is a vector of numbers that represents the token's meaning in a high-dimensional space.
What is the difference between a word and a token?
A word might be a single token, but complex or rare words are often broken down into multiple tokens by the tokenizer.
