There are nine things you should know about how AI models actually run.
1. Model weights
The billions of numbers a model learned during training, stored in a file. Loading a model means loading this file into memory.
Example: a 7B model has about 7 billion weights, stored in a file several gigabytes in size.
2. Quantization
Storing each weight with fewer bits, trading a small amount of precision for a much smaller file and less memory use.
Example: a weight stored at 16-bit precision compressed to 4-bit can shrink a 14 GB model to roughly 4 GB, with a small quality cost.
3. Tokenizer
The step that breaks text into smaller pieces called tokens, and turns those pieces into numbers the model can read.
Example: "running" might become two tokens, "run" and "ning", encoded as two numeric IDs.
4. Context window
The maximum number of tokens a model can hold in view at once, covering both what you send in and what it generates back.
Example: an 8K context window means the prompt and the response together cannot exceed about 8,000 tokens.
5. KV cache
Stored results from tokens the model already processed, kept in memory so each new token does not require recomputing everything before it.
Example: on token 500, the model reuses the cached math from tokens 1 through 499 instead of redoing it.
6. Prefill
The first pass, where the model reads your entire prompt at once and builds the initial cache before it writes anything back.
Example: sending a 2,000-word prompt means prefill processes those tokens in parallel before the first reply token appears.
7. Decoding
The step after prefill, where the model generates one token at a time, each one built from everything that came before it.
Example: a 100-token reply means 100 separate decoding steps, one per word or word-fragment.
8. Batching
Running several requests through the model together instead of one at a time, so the GPU's capacity is not left idle between them.
Example: a server handling three chat requests at once can batch them into a single pass instead of queuing them.
9. Device offloading
Splitting a model across GPU and CPU memory when it does not fully fit on the GPU, keeping some layers fast and some slower.
Example: a 20 GB model on a 12 GB GPU might keep 15 GB of layers on the GPU and offload the rest to system RAM.