Take a name like Qwen3.8-27B-Instruct-Q4_K_M.gguf. The 27B is the parameter count: 27 billion. The Q4 says each parameter is stored in roughly four bits. That’s enough to estimate how much memory the weights need.
WHAT THE NAME SAYSILLUSTRATIVE
Qwen3.8-27B-Instruct-Q4_K_M.gguf
↳27B27 billion parameters.
↳Q4Roughly four bits per weight.
03 / Parameters1 of 3
What is a parameter?
A parameter, often called a weight, is just a number. Each one nudges the model a little toward some patterns and away from others, and billions of them together produce its answers. Training is what sets them: the model makes a prediction, the training software compares it with the known answer, and the numbers that contributed to the mistake get adjusted slightly.
Learned valuesConceptual illustration
0.37 → 0.4321One example weight, nudged during training
↓One of billions of learned values
Training adjusts the values
03 / Parameters2 of 3
A model file is a set of trained parameters.
A model file is those finished numbers, saved to disk. They sit on your drive until a runner, the program that runs the model, loads them into memory.
Learned valuesConceptual illustration
0.4321One example weight, fixed once training is over
↓One of billions of learned values
Training is over; the values are saved
03 / Parameters3 of 3
What do extra parameters buy you?
More parameters give a model more room to hold the patterns and distinctions it picked up in training. That isn’t the same as knowing more: what a model knows depends on the training data and on how it was trained and designed. A bigger model can do better on broader or harder work.
At the same precision, a bigger model needs more memory, and size alone doesn’t promise better answers.
Learned valuesConceptual illustration
0.4321One example weight, fixed once training is over
↓One of billions of learned values
Prompt→Fixed weights→Response
04 / Quantization1 of 3
One model can be stored at several precisions.
Quantization stores each weight in fewer bits than a full 16-bit weight, giving up some precision to save space. At 16 bits per weight, our 27B example takes about 54 GB for the weights alone. Compare the four precisions to see what changes.
Same 27B modelWeights only
16-bit · example weight
━ Illustrated value 0.4321
54 GB
054 GB
The number of weights stays the same; the bits stored for each one can vary.
Each stop on the dial is a value a weight can be rounded to. The stops and numbers are simplified, not values read from a real file.
Show the numbers
These values show one weight with less detail at each setting. A weight is one number in a tensor, not a whole vector. The model still has the same number of weights.
One example weight across four precisions. Simplified illustration.
Precision
Illustrated value
16-bit (FP16)
0.4321
Q8
0.432
Q6
0.431
Q4
0.43
The shorter decimals make the idea easy to see, but real quantization doesn’t just trim decimal places. Q4_K_M uses a different scheme. One weight alone can’t tell you how good the model’s answers will be.
Same 27B modelWeights only
16-bit · example weight
━ Illustrated value 0.4321
54 GB
054 GB
The number of weights stays the same; the bits stored for each one can vary.
Each stop on the dial is a value a weight can be rounded to. The stops and numbers are simplified, not values read from a real file.
04 / Quantization2 of 3
Less precision can cost some nuance.
Each weight is rounded to one of a smaller set of possible values than at 16 bits, and that’s where the saving comes from. The model usually keeps its general behaviour, but the rounding can make answers a little less accurate or less nuanced, or tip it toward a different next word. How much depends on the model, the quantization method and the task. At Q6 the weights come to a theoretical 20.25 GB.
Same 27B modelWeights only
Q6 · example weight
━ Illustrated value 0.431
20.25 GB
054 GB
The number of weights stays the same; the bits stored for each one can vary.
Each stop on the dial is a value a weight can be rounded to. The stops and numbers are simplified, not values read from a real file.
04 / Quantization3 of 3
At Q4 the weights need a quarter of the memory they need at 16 bits.
At four bits per weight the same 27B model needs about 13.5 GB for its weights, against 54 GB at 16 bits. Context, the runner and your other apps still need memory on top of that.
Pick a precision that leaves room to run the model, then try it on your own work to see whether the answers still hold up.
Same 27B modelWeights only
Q4 · example weight
━ Illustrated value 0.43
13.5 GB
054 GB
The number of weights stays the same; the bits stored for each one can vary.
Each stop on the dial is a value a weight can be rounded to. The stops and numbers are simplified, not values read from a real file.
The four precisions side by side
Same 27B modelWeights only
16-bit · example weight
━ Illustrated value 0.4321
54 GB
054 GB
The number of weights stays the same; the bits stored for each one can vary.
Each stop on the dial is a value a weight can be rounded to. The stops and numbers are simplified, not values read from a real file.
Same 27B modelWeights only
Q8 · example weight
━ Illustrated value 0.432
27 GB
054 GB
The number of weights stays the same; the bits stored for each one can vary.
Each stop on the dial is a value a weight can be rounded to. The stops and numbers are simplified, not values read from a real file.
Same 27B modelWeights only
Q6 · example weight
━ Illustrated value 0.431
20.25 GB
054 GB
The number of weights stays the same; the bits stored for each one can vary.
Each stop on the dial is a value a weight can be rounded to. The stops and numbers are simplified, not values read from a real file.
Same 27B modelWeights only
Q4 · example weight
━ Illustrated value 0.43
13.5 GB
054 GB
The number of weights stays the same; the bits stored for each one can vary.
Each stop on the dial is a value a weight can be rounded to. The stops and numbers are simplified, not values read from a real file.
The numbers for one weight
These values show one weight with less detail at each setting. A weight is one number in a tensor, not a whole vector. The model still has the same number of weights.
One example weight across four precisions. Simplified illustration.
Precision
Illustrated value
16-bit (FP16)
0.4321
Q8
0.432
Q6
0.431
Q4
0.43
The shorter decimals make the idea easy to see, but real quantization doesn’t just trim decimal places. Q4_K_M uses a different scheme. One weight alone can’t tell you how good the model’s answers will be.
05 / Estimating weight memory
Calculate the memory required.
Billions of parameters×Bits per weight÷ 8≈ GB
Multiply the parameters (in billions) by the bits per weight, then divide by eight, because a byte is eight bits. The answer is in gigabytes. For 27B at 4 bits that’s 27 × 4 ÷ 8 = 13.5 GB. That figure covers the weights only; the runtime needs more on top.
12B × 4 ÷ 8
6 GBWeights only
27B × 4 ÷ 8
13.5 GBWeights only
70B × 4 ÷ 8
35 GBWeights only
These assume exactly four bits for every weight and use decimal GB (one billion bytes). Real files often mix precisions and add metadata and extra model components, so expect their size to differ.
06 / Your memory layout
Unified VS Pool memory
The CPU uses your computer’s RAM. A dedicated graphics card has its own memory, called VRAM. With a compatible runner, using the GPU often gives you quicker replies. If the model won’t fit entirely in VRAM, you may still be able to run it from RAM on the CPU, but you may wait longer. On a unified-memory computer, the CPU and GPU share memory with the system and your apps; there’s no separate VRAM figure to add.
If quicker replies matter most, check whether your GPU can run the model. If you want to try a larger model, check your RAM too, then see whether the wait is acceptable.
Three illustrative memory layouts
Unified-memory laptop
CPUGPU
↘ ↙
16 GBShared with system and apps
One shared pool
Desktop PC
CPU↓
64 GBRAM
GPU↓
16 GBVRAM
Two separate pools
Workstation
CPU↓
128 GBRAM
GPU↓
24 GBVRAM
Two separate pools
What if there’s no dedicated graphics memory?
A compatible runner can still use the CPU. Integrated graphics generally borrow system RAM, so the “shared memory” figure your system reports isn’t extra capacity on top of the RAM you installed. Your runner’s documentation will say whether it supports your exact hardware.
07 / Total runtime memory1 of 2
The weights aren’t the only thing in memory.
Put 13.5 GB of weights in a 16 GB pool and 2.5 GB is left. That 2.5 GB isn’t free: before you’ve typed a word, the operating system, your open apps and the runner itself are already using some of it, and what they use changes as you work. Whatever remains has to hold the conversation, with a little to spare.
A 16 GB memory poolWeights only
13.5 GB of weights
Weights
016 GB
2.5 GB left ↓ for everything else
Already using memory
Operating system & open appsModel runner
The system, apps and runner use different amounts on different machines, and context needs room too. No figure for any of them is measured here.
What if the weights are much larger?
On the example PC with 64 GB of RAM and 16 GB of VRAM, 128 GB of weights can’t fit. Even with all 16 GB of VRAM holding weights, the other 112 GB would still have to sit in RAM, and the PC has 64. Free disk space doesn’t make it a practical fit.
07 / Total runtime memory2 of 2
Long sessions need more RAM.
Context is everything the model can see while it writes an answer: your instructions, the conversation so far, any documents you’ve added, and the text it has already generated. It’s measured in tokens, which are small chunks of text. As the conversation grows or you add documents, the runner may need more memory to hold all of it, and that comes out of what’s left after the weights, the system and your apps. So a model that handles a short question comfortably can struggle with a long one.
Let your runner estimate the memory for your model and context setting; there’s no universal conversion from tokens to GB. A model’s advertised maximum context may be more than your computer can handle.
Same model and computerContext varies
Short questionInstructionsQuestion
→
Long conversation with documentsInstructionsConversation historyDocument excerptsGenerated text
08 / When VRAM isn’t enough
Can the GPU still help?
On our example PC, the model has 32 GB of weights, but the graphics card has only 16 GB of VRAM. The PC also has 64 GB of RAM. If there’s enough free RAM for the model and the rest of the system, a compatible runner can run it on the CPU. With partial GPU offload, the runner puts some of the model in VRAM for the GPU and leaves the rest in RAM for the CPU.
RAM and VRAM remain separate pools. If your runner supports offload, try it and compare the wait with CPU-only. Whether it helps depends on your hardware and the work you’re doing.
Example PCOne 0–64 GB scale
32 GB weights64 GB RAM16 GB VRAM
016324864 GB
System RAM32 / 64 GB · fills half of it
Graphics VRAM32 / 16 GB · twice its capacity
Weights Capacity boundary Beyond capacity
09 / Base or Instruct
For an assistant, start with Instruct.
Instruct models are trained to respond to requests, which is what you want for questions, summaries and other assistant-style work. Give a Base model the same request and it may simply carry on your text instead of doing what you asked. Base models can suit text completion or further training, so check the publisher’s description before you choose one.
Match the variant to the job, then check that your runner can open the file format.
Illustrative behaviour, not measured outputs
Base · continues the text
Text completion
Explain why leaves change colour.
Include a diagram and a short explanation of the pigments involved…
Instruct · answers the instruction
Assistant response
Explain why leaves change colour.
As chlorophyll breaks down, other pigments become visible.
Back to the filename: what do Instruct, Q4_K_M and .gguf mean?
Instruct
A variant trained to follow instructions. Publishers sometimes use other names, such as IT.
Q4_K_M
One specific quantization recipe. It mixes precisions, so not every stored value is exactly four bits.
.gguf
A file format that some runners can open. Being able to read the format doesn’t mean a runner supports this particular model’s architecture.
That filename is only an example. For a real download, check the actual file and whether your runner supports it.
Local means the model runs on your computer. Hosted means it runs on remote infrastructure you don’t control, and you reach it over the internet.
Sometimes the choice is made for you: if the material can’t leave your device, hosted is off the table. Otherwise, pick whichever setup does the job well enough at a price you can live with.
10 / Confidential documents
A confidential document can stay on your device.
Prefer local
When the model runs on your own computer, the document doesn’t have to go to a model provider at all.
For example, you can have a local model summarise an internal report and save the summary in the same folder.
If that folder syncs to the cloud, the report leaves the device anyway, and connected tools or the app’s logging can carry it off too. The local model also has to be good enough for the job.
If a smaller model does it well, every run uses hardware you already have, and no model provider charges you for it.
For example, a batch of short notes arrives every morning, and each one needs filing as an idea, an action or a reference.
Local isn’t free, though: setup time, electricity, upkeep and the hardware itself all cost something. Whether it saves money depends on how often you run the job and what a hosted service would charge for it.
The same small task, every day
New noteNew noteNew note
↓
One local modelUsed for each batch
↓
IdeasActionsReference
Notes sorted by category · the category names are illustrative
12 / Demanding reasoning
Some problems are worth sending to a stronger model.
Prefer hosted
The model you’d want may not fit in your computer’s memory, or may run too slowly there. Through a hosted service you can use it anyway.
For example, it could compare a system design’s conflicting requirements, such as reliability against budget, and explain the trade-offs behind what it proposes.
A capable model can help whether it runs on your computer or a hosted service. Check its suggestions as you would a colleague’s, and only send the service material you’re allowed to share.
Requirements that pull against each other
ReliabilityBudgetMaintenance
↓
Provider infrastructure
Capable hosted model
↓
Proposal with reasonsTrade-offs to review
13 / Many agents at once
Swarms of subagents each need their own RAM
Prefer hosted
An agent is an AI helper that works through a task in steps and can use tools. Agents on one computer can share a loaded model, but each keeps its own conversation, and every conversation takes processing and memory. When the jobs don’t depend on each other, a hosted service can do that processing on remote infrastructure instead.
For example, three agents could each take one part of a research question, with a final step combining what they find.
Splitting the job up doesn’t automatically make it faster: service limits, queues and the work of coordinating the agents still apply.
Try the whole task with a local model first. If it misreads a tool’s result or keeps choosing the wrong next step, a hosted model may do better. The number of calls alone isn’t a reason to switch.
For example, it might look up a field in the documentation, then use it to query a database.
If you use a hosted model, any tool results you send back to it go to the provider, even if the tool ran on your computer.
Here is how the memory checks apply to two machines you might develop with.
Linux PC with an RTX 3090
Memory
24 GB VRAM · 64 GB system RAM (owner’s example)
Model
Qwen3.8 27B, Q4_K_XL file
Fit
projectedThe weights file is 16.35 GiB, not the 13.5 GB that exactly four bits per weight gives, because this quant keeps some tensors at higher precision. Add about 2 GiB of context at 32,000 tokens and roughly 0.5 GiB of runtime and it projects to about 19 GiB of the GPU’s 24 GiB VRAM. The 64 GB of system RAM is separate and can help with partial CPU offload.
Tasks to try
Explain a function and draft tests for a small change.