01 / A field guide to running AI yourself
Understand which AI models your hardware can run, choose one worth trying, and decide where your work belongs.
Start with the modelUnderstand
02 / Read the model filename
27B tells you the parameter count. Q4 describes how precisely the weights are stored. Together, they help you estimate weight memory.
We’ll unpack these terms next. The rest of the filename can wait until you choose a variant.
Example filename. Check the name of the actual file before downloading.
03 / Parameters1 of 3
A parameter is an adjustable number in the model, usually called a weight. During training, these numbers change as the model learns patterns from examples.
A few values represent the idea. 27B means 27 billion parameters, not 27 billion separate facts.
03 / Parameters2 of 3
The model file contains trained weights. They stay on your storage drive until the runner—the software that runs the model—loads them into memory.
A few values represent the idea. 27B means 27 billion parameters, not 27 billion separate facts.
03 / Parameters3 of 3
When you ask a question, the model uses its learned values to generate a response. This is inference; it normally leaves the weights unchanged.
More parameters usually need more memory. They do not guarantee better answers.
A few values represent the idea. 27B means 27 billion parameters, not 27 billion separate facts.
04 / Quantization1 of 3
Quantization reduces the precision used to store weights. Our 27B example at 16 bits per weight needs about 54 GB for weights alone. Select a precision to see what changes.
┄ Original weight━ Stored value
54 GB
Fewer bits per weight.
The same number of weights.
Fewer stops mean coarser rounding. These stops illustrate the effect, not the actual Q4/Q6/Q8 encodings. Real files include additional information.
┄ Original weight━ Stored value
54 GB
Fewer bits per weight.
The same number of weights.
Fewer stops mean coarser rounding. These stops illustrate the effect, not the actual Q4/Q6/Q8 encodings. Real files include additional information.
04 / Quantization2 of 3
At Q6, the theoretical weight estimate is 20.25 GB. Rounding can change the answers; the effect depends on the model, quantization method, and your task. The dial exaggerates the rounding so you can see it.
┄ Original weight━ Stored value
20.25 GB
Fewer bits per weight.
The same number of weights.
Fewer stops mean coarser rounding. These stops illustrate the effect, not the actual Q4/Q6/Q8 encodings. Real files include additional information.
04 / Quantization3 of 3
At four bits per weight, the same 27 billion values need about 13.5 GB. The parameter count stays fixed as storage precision changes.
Choose a precision that leaves room to run the model, then check its answers on your work.
┄ Original weight━ Stored value
13.5 GB
Fewer bits per weight.
The same number of weights.
Fewer stops mean coarser rounding. These stops illustrate the effect, not the actual Q4/Q6/Q8 encodings. Real files include additional information.
┄ Original weight━ Stored value
54 GB
Fewer bits per weight.
The same number of weights.
Fewer stops mean coarser rounding. These stops illustrate the effect, not the actual Q4/Q6/Q8 encodings. Real files include additional information.
┄ Original weight━ Stored value
27 GB
Fewer bits per weight.
The same number of weights.
Fewer stops mean coarser rounding. These stops illustrate the effect, not the actual Q4/Q6/Q8 encodings. Real files include additional information.
┄ Original weight━ Stored value
20.25 GB
Fewer bits per weight.
The same number of weights.
Fewer stops mean coarser rounding. These stops illustrate the effect, not the actual Q4/Q6/Q8 encodings. Real files include additional information.
┄ Original weight━ Stored value
13.5 GB
Fewer bits per weight.
The same number of weights.
Fewer stops mean coarser rounding. These stops illustrate the effect, not the actual Q4/Q6/Q8 encodings. Real files include additional information.
05 / Estimate weight memory
There are eight bits in a byte. Multiply the parameter count by the bits stored for each weight, then divide by eight.
12B × 4 ÷ 8
6 GBWeights only27B × 4 ÷ 8
13.5 GBWeights only70B × 4 ÷ 8
35 GBWeights onlyUniform four-bit estimates in decimal GB (one billion bytes). Actual files can contain mixed precisions, metadata, and extra model components. Runtime memory is more than weights.
06 / Your memory layout
RAM is your computer’s main memory. A dedicated graphics card has its own memory, called VRAM. In a unified-memory computer, CPU, GPU, operating system, and apps share one capacity.
Identify your memory layout before comparing model sizes.
Three illustrative memory layouts
One shared pool
Two separate pools
Two separate pools
You can use a compatible runner on the CPU. Integrated graphics generally share system RAM; their reported shared memory is not extra installed capacity. Check support for your exact hardware. These illustrations describe memory layouts, not compatibility guarantees.
07 / The whole runtime1 of 2
13.5 GB of weights leaves only 2.5 GB in a 16 GB pool. The conversation, running software, operating system, and other apps need memory too. Leave spare room for their needs to change.
13.5 GB of weights
2.5 GB left must cover other needs
These needs are not measured here. Spare room is capacity deliberately left unused.
07 / The whole runtime2 of 2
Context includes instructions, conversation, documents, and generated text. It is counted in tokens—small chunks of text. Longer context can require more runtime memory; there is no fixed tokens-to-GB formula.
A weights-only calculation is a starting point, not the full requirement. An advertised context limit is not a guarantee your setup can run at that limit; use the relevant material and check the original passages.
13.5 GB of weights
2.5 GB left must cover other needs
These needs are not measured here. Spare room is capacity deliberately left unused.
Instructions + conversation + documents + generated text
On the 64 GB RAM / 16 GB VRAM example PC, 128 GB of weights is beyond the combined physical capacity. Even if all 16 GB VRAM could hold weights, 112 GB would still need RAM. Storage space does not make that a practical in-memory fit.
08 / One worked PC
Now take 32 GB of weights: large enough to exceed a 16 GB graphics card, but half of a PC’s 64 GB RAM. Compatible software can use the CPU if sufficient memory remains for the rest of the runtime.
Not fitting fully in VRAM does not rule out CPU execution. Useful speed still needs a real trial.
Weights Capacity boundary Beyond capacity
CPU execution may work. The weights leave 32 GB of installed RAM before other needs. The full model cannot reside in this 16 GB VRAM.
09 / Let the GPU help1 of 2
In CPU-only execution, the model uses weights in system RAM. The graphics card does not generate this answer. Whether that is fast enough for your work depends on the actual setup.
Not used for this answer
Conceptual allocation; the runner determines the split. Equal blocks show order, not GB, actual layers, or processing time. RAM and VRAM remain separate.
09 / Let the GPU help2 of 2
A GPU can perform many calculations at once. With a compatible runner, some model work can move to the GPU while the CPU handles the rest. This is partial GPU offload.
RAM and VRAM stay separate. Try the supported offload settings to find out whether they help.
Conceptual allocation; the runner determines the split. Equal blocks show order, not GB, actual layers, or processing time. RAM and VRAM remain separate.
Choose
10 / Base or Instruct
Instruct models are trained to follow instructions. Base models continue text and can be useful for further training or completion tasks. Names vary, so check the publisher’s description.
Choose the variant for the task, then check that the file format works with your runner.
Illustrative behaviour, not measured outputs
Base · continues the text
Explain why leaves change colour.
Include a diagram and a short explanation of the pigments involved…
Instruct · answers the instruction
Explain why leaves change colour.
As chlorophyll breaks down, other pigments become visible.
The opening filename is illustrative. Verify the actual artifact and your runner’s support.
11 / Model scale
Theoretical four-bit weights range from 6 GB for a 12B model to 1,400 GB for a 2.8T model. The second scale enlarges the small end so you can compare it without hiding the full difference.
An available model is not necessarily a practical model for a personal computer.
Enlarged view — different scale
2.25× Qwen’s estimated weights compared with Gemma’s
≈ 104× Kimi’s estimated weights compared with Qwen’s
Idealized weight arithmetic, not file sizes or runtime measurements. Sources: Google · Qwen · Moonshot AI. Checked 16 September 2026.
12 / Narrow the candidates
Choose an example machine and compare the models on it. These bars account for weights only. The capacity left still needs to cover the rest of the runtime.
Shortlist a plausible candidate; verify the actual file and runtime before treating it as a fit.
The model, runner, operating system, and apps share these 16 GB. Each bar represents the capacity labelled above it; compare the numbers across different pools.
12B parameters
≈ 6 GB weights
10 GB left before other needs
A plausible candidate. The remaining memory must also cover the system, apps, runner, and conversation.
27B parameters
≈ 13.5 GB weights
2.5 GB left before other needs
Only 2.5 GB for the system, apps, and everything beyond the weights is too constrained for a normal setup. Try a smaller model.
2.8T parameters
≈ 1,400 GB weights
1,384 GB of weights beyond this capacity
About 1,400 GB of theoretical weights is far beyond this 16 GB shared pool.
The CPU uses system RAM. The graphics card has its own VRAM. Each bar represents the capacity labelled above it; compare the numbers across different pools.
12B parameters
≈ 6 GB weights
10 GB left before other needs
58 GB left before other needs
The weights fit on the graphics card with room for other GPU memory needs. The CPU and apps use separate RAM. Check the full estimate.
27B parameters
≈ 13.5 GB weights
2.5 GB left before other needs
50.5 GB left before other needs
Little VRAM remains, but the weights have plenty of room in RAM. A compatible runner can use the CPU or partial offload; running entirely on the GPU needs a careful memory check.
2.8T parameters
≈ 1,400 GB weights
1,384 GB of weights beyond this capacity
1,336 GB of weights beyond this capacity
About 1,400 GB of theoretical weights is far beyond both the 64 GB RAM and the separate 16 GB VRAM.
More memory in each place, still separate capacities. Each bar represents the capacity labelled above it; compare the numbers across different pools.
12B parameters
≈ 6 GB weights
18 GB left before other needs
122 GB left before other needs
The graphics card has more room beyond the weights. Check how much the runner needs for your conversation length.
27B parameters
≈ 13.5 GB weights
10.5 GB left before other needs
114.5 GB left before other needs
More VRAM remains for the conversation and runner. Check their memory needs before loading the model.
2.8T parameters
≈ 1,400 GB weights
1,376 GB of weights beyond this capacity
1,272 GB of weights beyond this capacity
About 1,400 GB of theoretical weights is far beyond both the 128 GB RAM and the separate 24 GB VRAM.
16 GB shared
The model, runner, operating system, and apps share these 16 GB.
| Model | Weights | What to check |
|---|---|---|
| Gemma 3 12B | 6 GB | A plausible candidate. The remaining memory must also cover the system, apps, runner, and conversation. |
| Qwen3.8 27B | 13.5 GB | Only 2.5 GB for the system, apps, and everything beyond the weights is too constrained for a normal setup. Try a smaller model. |
| Kimi K3 | 1,400 GB | About 1,400 GB of theoretical weights is far beyond this 16 GB shared pool. |
64 GB RAM · 16 GB VRAM
The CPU uses system RAM. The graphics card has its own VRAM.
| Model | Weights | What to check |
|---|---|---|
| Gemma 3 12B | 6 GB | The weights fit on the graphics card with room for other GPU memory needs. The CPU and apps use separate RAM. Check the full estimate. |
| Qwen3.8 27B | 13.5 GB | Little VRAM remains, but the weights have plenty of room in RAM. A compatible runner can use the CPU or partial offload; running entirely on the GPU needs a careful memory check. |
| Kimi K3 | 1,400 GB | About 1,400 GB of theoretical weights is far beyond both the 64 GB RAM and the separate 16 GB VRAM. |
128 GB RAM · 24 GB VRAM
More memory in each place, still separate capacities.
| Model | Weights | What to check |
|---|---|---|
| Gemma 3 12B | 6 GB | The graphics card has more room beyond the weights. Check how much the runner needs for your conversation length. |
| Qwen3.8 27B | 13.5 GB | More VRAM remains for the conversation and runner. Check their memory needs before loading the model. |
| Kimi K3 | 1,400 GB | About 1,400 GB of theoretical weights is far beyond both the 128 GB RAM and the separate 24 GB VRAM. |
These are examples, not a list of supported computers. Record your own RAM and dedicated VRAM separately, or your single unified-memory capacity. Compare weights with the pool the runner will use, then account for context and other software.
An integrated GPU usually uses system memory. A “shared GPU memory” allowance is not additional installed RAM. Check your runner’s support for the exact processor and graphics hardware.
CPU and GPU rows are alternative placements, not two model copies or a measured offload split. These examples do not establish full compatibility or useful speed. Neither model size nor execution location establishes answer quality.
Find your memory & estimate a modelDecide
Local means the model runs on your computer. Hosted means it runs on a provider’s infrastructure. The better choice depends on the work.
These are starting points. If the information cannot leave your device, that restriction comes first. Otherwise, choose a model and workflow capable of doing the job within your budget.
13 / Sensitive document analysis
Prefer local
A fully local workflow keeps document contents on your device instead of sending them to a model provider. That makes local processing a useful starting point when the material cannot be shared.
For example Summarise a confidential internal report, with the document and model both on your computer.
The whole application matters: connected tools, syncing, and logging must also keep the contents local. The model still needs to be capable of the task.
A fully local workflow
On your device
No document content sent out
14 / Small, frequent jobs
Prefer local
When a smaller model handles the task well, running it on hardware you already own can reduce recurring usage charges. Repetition makes this worth considering: the same model can process each new batch.
For example Categorise short notes or extract the same fields from incoming text every day.
Local is not free: setup time, electricity, maintenance, and hardware have costs. Compare those with hosted subscription or usage charges; savings depend on how much you use it.
Repeat a useful small task
Categorised notes · illustrative output labels
15 / Demanding reasoning
Prefer hosted
A provider may offer a capable model that exceeds your computer’s practical memory or compute limits. Hosted access can make that model available without running its weights on your own hardware.
For example Compare conflicting system-design requirements and explain the trade-offs behind a proposed approach.
Choose the model for the job. Hosting alone does not make a model reason better, and a proposal still needs review. Share only information the provider is permitted to receive.
Work through competing requirements
Provider infrastructure
Capable hosted model16 / Many agents working together
Prefer hosted
Agents are AI helpers that can take several steps and use tools. Running many at once can increase context-memory and processing demands. Hosted capacity can help with work that has genuinely independent parts.
For example Separate agents investigate different parts of a research question, then combine their findings.
More agents consume more usage. Service limits, queues, and coordination still apply; parallel work is not a guaranteed speedup. Agents can share a model rather than each loading their own copy.
Independent parts of one question
Provider capacity · service limits apply
Research questionRequests may queue; more agents use more tokens.
17 / Complex tool-driven workflows
Usually prefer hosted for demanding coordination
A capable hosted model can help choose tools and interpret substantial results across many steps. Tool-call count alone does not require hosted compute, and hosted inference does not supply integrations or move tool computation automatically.
For example Search documentation, query a database, analyse results, then choose the next action.
A tool running locally does not keep its results local if they are returned to a hosted model.
Reasoning and tools run separately
Sent to the model provider
Hosted modelCalls and results
Tools run locally or remotely, as configured. Results sent upward leave for the provider.