Local inference

01 /

A field guide to running AI yourself

What can your computer run?

Learn to understand which AI models your hardware can run, how to choose ones worth trying, and learn what kind of work would be better for local AI inferences compared to hosted.

Learn to read a model name
ONE PROMPT. TWO PLACES TO RUN.
Help me rewrite this email.
01A remote serverRuns on provider hardware
02Your computerRuns on your hardware

Understand

Learn what your computer can run locally

02 / Read a model name

Read the size
and precision.

27B tells you the parameter count. Q4 describes how precisely the weights are stored. Together, they help you estimate weight memory.

READ THE MODEL NAMEILLUSTRATIVE
Qwen3.8-27B-Instruct-Q4_K_M.gguf
27B27 billion parameters.
Q4Roughly four bits per weight.

03 / Parameters1 of 3

What is a parameter?

A parameter, often called a weight, is a number that gives the model a tiny bias toward some patterns and away from others. Together, billions of these numbers shape its responses. During training, the model makes a prediction; training software compares it with the known answer and slightly adjusts the numbers that contributed to the mistake.

Learned valuesConceptual illustration
0.37 → 0.42One illustrative weight adjusts during training
One of many learned values
Training adjusts the values

A few values represent the idea. 27B means 27 billion parameters, not 27 billion separate facts.

03 / Parameters2 of 3

You download the learned values.

The model file contains trained weights. They stay on your storage drive until the runner—the software that runs the model—loads them into memory.

Learned valuesConceptual illustration
0.42One illustrative weight stays fixed during inference
One of many learned values
The learned values are ready to run

A few values represent the idea. 27B means 27 billion parameters, not 27 billion separate facts.

03 / Parameters3 of 3

Why have more parameters?

More parameters give a model more capacity to represent the patterns and distinctions it learned during training. They do not measure how much the model knows: its training data, training process, and design shape that. More capacity can help with broader or harder work, but only when the model is trained and designed well. When you ask a question, it uses those learned values without normally changing them.

Larger models can be more capable, but they need more memory and do not guarantee better answers.

Learned valuesConceptual illustration
0.42One illustrative weight stays fixed during inference
One of many learned values
PromptFixed weightsResponse

A few values represent the idea. 27B means 27 billion parameters, not 27 billion separate facts.

04 / Quantization1 of 3

Store the same weights with fewer bits.

Quantization reduces the precision used to store weights. Our 27B example at 16 bits per weight needs about 54 GB for weights alone. Select a precision to see what changes.

Same 27B modelWeights only
16-bit · one weight

┄ Original weight━ Stored value

54 GB

054 GB

Fewer bits per weight.
The same number of weights.

Fewer stops mean coarser rounding. These stops illustrate the effect, not the actual Q4/Q6/Q8 encodings. Real files include additional information.

04 / Quantization2 of 3

Coarser rounding can change a close call.

Quantization rounds learned numbers to fewer possible values. Think of redrawing a smooth line on a square grid: the overall route remains, but fine turns snap to nearby positions. Each small change shifts the model’s preferences slightly. When two next words are close, that can select a different one; later choices may then diverge. At Q6, the theoretical weight estimate is 20.25 GB. The dial exaggerates the rounding so you can see it.

Same 27B modelWeights only
Q6 · one weight

┄ Original weight━ Stored value

20.25 GB

054 GB

Fewer bits per weight.
The same number of weights.

Fewer stops mean coarser rounding. These stops illustrate the effect, not the actual Q4/Q6/Q8 encodings. Real files include additional information.

04 / Quantization3 of 3

Q4 brings the estimate to 13.5 GB.

At four bits per weight, the same 27 billion values need about 13.5 GB. The parameter count stays fixed as storage precision changes.

Choose a precision that leaves room to run the model, then check its answers on your work.

Same 27B modelWeights only
Q4 · one weight

┄ Original weight━ Stored value

13.5 GB

054 GB

Fewer bits per weight.
The same number of weights.

Fewer stops mean coarser rounding. These stops illustrate the effect, not the actual Q4/Q6/Q8 encodings. Real files include additional information.

All four precisions, at a glance

Same 27B modelWeights only
16-bit · one weight

┄ Original weight━ Stored value

54 GB

054 GB

Fewer bits per weight.
The same number of weights.

Fewer stops mean coarser rounding. These stops illustrate the effect, not the actual Q4/Q6/Q8 encodings. Real files include additional information.

Same 27B modelWeights only
Q8 · one weight

┄ Original weight━ Stored value

27 GB

054 GB

Fewer bits per weight.
The same number of weights.

Fewer stops mean coarser rounding. These stops illustrate the effect, not the actual Q4/Q6/Q8 encodings. Real files include additional information.

Same 27B modelWeights only
Q6 · one weight

┄ Original weight━ Stored value

20.25 GB

054 GB

Fewer bits per weight.
The same number of weights.

Fewer stops mean coarser rounding. These stops illustrate the effect, not the actual Q4/Q6/Q8 encodings. Real files include additional information.

Same 27B modelWeights only
Q4 · one weight

┄ Original weight━ Stored value

13.5 GB

054 GB

Fewer bits per weight.
The same number of weights.

Fewer stops mean coarser rounding. These stops illustrate the effect, not the actual Q4/Q6/Q8 encodings. Real files include additional information.

05 / Estimate weight memory

Calculate weight memory.
Weights only.

Billions of parameters×Bits per weight÷ 8≈ GB

There are eight bits in a byte. Multiply the parameter count by the bits stored for each weight, then divide by eight.

12B × 4 ÷ 8

6 GBWeights only

27B × 4 ÷ 8

13.5 GBWeights only

70B × 4 ÷ 8

35 GBWeights only

Uniform four-bit estimates in decimal GB (one billion bytes). Actual files can contain mixed precisions, metadata, and extra model components. Runtime memory is more than weights.

06 / Your memory layout

One shared pool, or two separate pools?

RAM is your computer’s main memory. A dedicated graphics card has its own memory, called VRAM. In a unified-memory computer, CPU, GPU, operating system, and apps share one capacity.

Identify your memory layout before comparing model sizes.

Three illustrative memory layouts

Unified-memory laptop

CPUGPU
16 GBShared with system and apps

One shared pool

Desktop PC

CPU
64 GBRAM
GPU
16 GBVRAM

Two separate pools

Workstation

CPU
128 GBRAM
GPU
24 GBVRAM

Two separate pools

No dedicated graphics memory?

You can use a compatible runner on the CPU. Integrated graphics generally share system RAM; their reported shared memory is not extra installed capacity. Check support for your exact hardware. These illustrations describe memory layouts, not compatibility guarantees.

07 / Total runtime memory1 of 2

The weights fit. What about everything else?

13.5 GB of weights leaves only 2.5 GB in a 16 GB pool. The conversation, running software, operating system, and other apps need memory too. Leave spare room for their needs to change.

A 16 GB memory poolWeights only

13.5 GB of weights

016 GB

2.5 GB left must cover other needs

ContextRunning softwareSystem & appsSpare room

These needs are not measured here. Spare room is capacity deliberately left unused.

07 / Total runtime memory2 of 2

Context is the material available for this answer.

Context includes instructions, conversation, documents, and generated text. It is counted in tokens—small chunks of text. Longer context can require more runtime memory; there is no fixed tokens-to-GB formula.

A weights-only calculation is a starting point, not the full requirement. An advertised context limit is not a guarantee your setup can run at that limit; use the relevant material and check the original passages.

A 16 GB memory poolWeights only

13.5 GB of weights

016 GB

2.5 GB left must cover other needs

ContextRunning softwareSystem & appsSpare room

These needs are not measured here. Spare room is capacity deliberately left unused.

Context for this answer

Instructions + conversation + documents + generated text

What if the weights are much larger?

On the 64 GB RAM / 16 GB VRAM example PC, 128 GB of weights is beyond the combined physical capacity. Even if all 16 GB VRAM could hold weights, 112 GB would still need RAM. Storage space does not make that a practical in-memory fit.

08 / Compare RAM and VRAM

Fits in RAM. Too large for this graphics card.

32 GB of weights exceeds a 16 GB graphics card, but is half of a PC’s 64 GB RAM. Compatible software can use the CPU if sufficient memory remains for the rest of the runtime.

Not fitting fully in VRAM does not rule out CPU execution. Useful speed still needs a real trial.

Example PCOne 0–64 GB scale
32 GB weights64 GB RAM16 GB VRAM
016324864 GB
System RAM32 / 64 GB · half the pool
Graphics VRAM32 / 16 GB · twice the pool

Weights Capacity boundary Beyond capacity

CPU execution may work. The weights leave 32 GB of installed RAM before other needs. The full model cannot reside in this 16 GB VRAM.

09 / Let the GPU help1 of 2

The CPU can do the work.

In CPU-only execution, the model uses weights in system RAM. The graphics card does not generate this answer. Whether that is fast enough for your work depends on the actual setup.

CPU-only executionConceptual allocation
One model, in order A → B → C → D → E → F

CPU / RAM

ABCDEF

GPU / VRAM

Not used for this answer

Conceptual allocation; the runner determines the split. Equal blocks show order, not GB, actual layers, or processing time. RAM and VRAM remain separate.

09 / Let the GPU help2 of 2

The GPU can handle part of the model.

A GPU can perform many calculations at once. With a compatible runner, some model work can move to the GPU while the CPU handles the rest. This is partial GPU offload.

RAM and VRAM stay separate. Try the supported offload settings to find out whether they help.

Partial GPU offloadConceptual allocation
One model, in order A → B → C → D → E → F

CPU / RAM

ABC

GPU / VRAM

DEF

Conceptual allocation; the runner determines the split. Equal blocks show order, not GB, actual layers, or processing time. RAM and VRAM remain separate.

Choose

Choose a model worth trying

10 / Base or Instruct

For an assistant, start with Instruct.

Instruct models are trained to follow instructions. Base models continue text and can be useful for further training or completion tasks. Names vary, so check the publisher’s description.

Choose the variant for the task, then check that the file format works with your runner.

Illustrative behaviour, not measured outputs

Base · continues the text

Text completion

Explain why leaves change colour.

Include a diagram and a short explanation of the pigments involved…

Instruct · answers the instruction

Assistant response

Explain why leaves change colour.

As chlorophyll breaks down, other pigments become visible.

Return to the filename: what do the other parts mean?
Instruct
A variant intended to follow instructions. Publishers may use other names, such as IT.
Q4_K_M
A particular quantization recipe, using mixed precisions. It is not exactly four bits for every stored value.
.gguf
A file format supported by some runners. The format alone does not establish support for the model architecture.

The opening filename is illustrative. Verify the actual artifact and your runner’s support.

11 / Model scale

Larger models need far more memory.

Theoretical four-bit weights range from 6 GB for a 12B model to 1,400 GB for a 2.8T model. The second scale enlarges the small end so you can compare it without hiding the full difference.

An available model is not necessarily a practical model for a personal computer.

Four-bit weight estimatesLinear scales
01,400 GB
Gemma 3 12B6 GB
Qwen3.8 27B13.5 GB
Kimi K31,400 GB
The first 16 GB, enlarged below

Enlarged view — different scale

016 GB
Gemma 3 12B6 GB
Qwen3.8 27B13.5 GB

2.25× Qwen’s estimated weights compared with Gemma’s

≈ 104× Kimi’s estimated weights compared with Qwen’s

Idealized weight arithmetic, not file sizes or runtime measurements. Sources: Google · Qwen · Moonshot AI. Checked 16 September 2026.

12 / Compare candidates on your hardware

Compare candidates with your hardware first.

Choose an example machine and compare the models on it. These bars account for weights only. The capacity left still needs to cover the rest of the runtime.

Shortlist a plausible candidate; verify the actual file and runtime before treating it as a fit.

Unified-memory laptop

16 GB shared

The model, runner, operating system, and apps share these 16 GB.

Weights only; other runtime needs remain
ModelWeightsWhat to check
Gemma 3 12B6 GBA plausible candidate. The remaining memory must also cover the system, apps, runner, and conversation.
Qwen3.8 27B13.5 GBOnly 2.5 GB for the system, apps, and everything beyond the weights is too constrained for a normal setup. Try a smaller model.
Kimi K31,400 GBAbout 1,400 GB of theoretical weights is far beyond this 16 GB shared pool.

Desktop PC

64 GB RAM · 16 GB VRAM

The CPU uses system RAM. The graphics card has its own VRAM.

Weights only; other runtime needs remain
ModelWeightsWhat to check
Gemma 3 12B6 GBThe weights fit on the graphics card with room for other GPU memory needs. The CPU and apps use separate RAM. Check the full estimate.
Qwen3.8 27B13.5 GBLittle VRAM remains, but the weights have plenty of room in RAM. A compatible runner can use the CPU or partial offload; running entirely on the GPU needs a careful memory check.
Kimi K31,400 GBAbout 1,400 GB of theoretical weights is far beyond both the 64 GB RAM and the separate 16 GB VRAM.

Workstation

128 GB RAM · 24 GB VRAM

More memory in each place, still separate capacities.

Weights only; other runtime needs remain
ModelWeightsWhat to check
Gemma 3 12B6 GBThe graphics card has more room beyond the weights. Check how much the runner needs for your conversation length.
Qwen3.8 27B13.5 GBMore VRAM remains for the conversation and runner. Check their memory needs before loading the model.
Kimi K31,400 GBAbout 1,400 GB of theoretical weights is far beyond both the 128 GB RAM and the separate 24 GB VRAM.
Apply this to your computer

These are examples, not a list of supported computers. Record your own RAM and dedicated VRAM separately, or your single unified-memory capacity. Compare weights with the pool the runner will use, then account for context and other software.

An integrated GPU usually uses system memory. A “shared GPU memory” allowance is not additional installed RAM. Check your runner’s support for the exact processor and graphics hardware.

CPU and GPU rows are alternative placements, not two model copies or a measured offload split. These examples do not establish full compatibility or useful speed. Neither model size nor execution location establishes answer quality.

Find your memory & estimate a model

Decide

Choose local or hosted AI for the work

Local means the model runs on your computer. Hosted means it runs on a provider’s infrastructure. The better choice depends on the work.

These are starting points. If the information cannot leave your device, that restriction comes first. Otherwise, choose a model and workflow capable of doing the job within your budget.

13 / Sensitive document analysis

Sensitive document analysis

Prefer local

A fully local workflow keeps document contents on your device instead of sending them to a model provider. That makes local processing a useful starting point when the material cannot be shared.

For example Summarise a confidential internal report, with the document and model both on your computer.

The whole application matters: connected tools, syncing, and logging must also keep the contents local. The model still needs to be capable of the task.

Local and cloud behaviour ↗

A fully local workflow

On your device

  1. Confidential report
  2. Local model
  3. SummarySaved locally

No document content sent out

14 / Small, frequent jobs

Small, frequent jobs

Prefer local

When a smaller model handles the task well, running it on hardware you already own can reduce recurring usage charges. Repetition makes this worth considering: the same model can process each new batch.

For example Categorise short notes or extract the same fields from incoming text every day.

Local is not free: setup time, electricity, maintenance, and hardware have costs. Compare those with hosted subscription or usage charges; savings depend on how much you use it.

Repeat a useful small task

New noteNew noteNew note
One local modelUsed for each batch
IdeasActionsReference

Categorised notes · illustrative output labels

15 / Demanding reasoning

Demanding reasoning

Prefer hosted

A provider may offer a capable model that exceeds your computer’s practical memory or compute limits. Hosted access can make that model available without running its weights on your own hardware.

For example Compare conflicting system-design requirements and explain the trade-offs behind a proposed approach.

Choose the model for the job. Hosting alone does not make a model reason better, and a proposal still needs review. Share only information the provider is permitted to receive.

Work through competing requirements

ReliabilityBudgetMaintenance

Provider infrastructure

Capable hosted model
Proposal with reasonsTrade-offs to review

16 / Many agents working together

Many agents working together

Prefer hosted

Agents are AI helpers that can take several steps and use tools. Running many at once can increase context-memory and processing demands. Hosted capacity can help with work that has genuinely independent parts.

For example Separate agents investigate different parts of a research question, then combine their findings.

More agents consume more usage. Service limits, queues, and coordination still apply; parallel work is not a guaranteed speedup. Agents can share a model rather than each loading their own copy.

Multi-agent resource trade-offs ↗

Independent parts of one question

Provider capacity · service limits apply

Research question
Part AAgentOwn context
Part BAgentOwn context
Part CAgentOwn context
Combine findingsOne synthesis

Requests may queue; more agents use more tokens.

17 / Complex tool-driven workflows

Complex tool-driven workflows

Usually prefer hosted for demanding coordination

A capable hosted model can help choose tools and interpret substantial results across many steps. Tool-call count alone does not require hosted compute, and hosted inference does not supply integrations or move tool computation automatically.

For example Search documentation, query a database, analyse results, then choose the next action.

A tool running locally does not keep its results local if they are returned to a hosted model.

How models and tools interact ↗

Reasoning and tools run separately

Sent to the model provider

Hosted model
Tool requests Returned results
Application / tool runnerConnects the model to configured tools

Calls and results

Documentation searchDatabase queryAnalysis tool

Tools run locally or remotely, as configured. Results sent upward leave for the provider.