Posts tagged with “large language models”

Memory Bandwidth, Not Just Compute, Speeds Up AI Chatbots

When a chatbot types out its answer word by word, the speed you see is often limited by how fast a chip can move data, not by how fast it can calculate. That is why memory bandwidth has become one of the most important specifications in AI hardware.

Every token needs the whole model

Language models generate text one token at a time. For each new token, the chip has to read the model's weights, often billions of numbers, from memory into its compute units. For a single user, the math performed on each weight is small. As a result, the processor spends much of its time waiting for data to arrive.

Sponsored Recommended for you – discover more →

What a Context Window Is, and Why It Matters

Every large language model works with a fixed amount of text at a time. This limit is called the context window. It includes everything the model can see in a single exchange: your instructions, any documents you share, earlier messages in the conversation, and the reply it is writing.

Measured in tokens

Context windows are measured in tokens, not words. A token is a small piece of text, often a short word or part of a longer one. When a conversation grows past the limit, the oldest material drops out of view or has to be shortened. The model does not keep it anywhere else. For that exchange, it is simply gone.