Local inference

01 /

A field guide to running AI yourself

What can your computer run?

Learn to understand which AI models your hardware can run, how to choose ones worth trying, and learn what kind of work would be better for local AI inferences compared to hosted.

Learn to read a model name
ONE PROMPT. TWO PLACES TO RUN.
Help me rewrite this email.
01A remote serverRuns on provider hardware
02Your computerRuns on your hardware

Understand

Learn what your computer can run locally

02 / Read a model name

Read the size
and precision.

27B tells you the parameter count. Q4 describes how precisely the weights are stored. Together, they help you estimate weight memory.

READ THE MODEL NAMEILLUSTRATIVE
Qwen3.8-27B-Instruct-Q4_K_M.gguf
27B27 billion parameters.
Q4Roughly four bits per weight.

03 / Parameters1 of 3

What is a parameter?

A parameter, often called a weight, is a number that gives the model a tiny bias toward some patterns and away from others. Together, billions of these numbers shape its responses. During training, the model makes a prediction; training software compares it with the known answer and slightly adjusts the numbers that contributed to the mistake.

Learned valuesConceptual illustration
0.37 → 0.42One illustrative weight adjusts during training
One of many learned values
Training adjusts the values

A few values represent the idea. 27B means 27 billion parameters, not 27 billion separate facts.

03 / Parameters2 of 3

You download the learned values.

The model file contains trained weights. They stay on your storage drive until the runner—the software that runs the model—loads them into memory.

Learned valuesConceptual illustration
0.42One illustrative weight stays fixed during inference
One of many learned values
The learned values are ready to run

A few values represent the idea. 27B means 27 billion parameters, not 27 billion separate facts.

03 / Parameters3 of 3

Why have more parameters?

More parameters give a model more capacity to represent the patterns and distinctions it learned during training. They do not measure how much the model knows: its training data, training process, and design shape that. More capacity can help with broader or harder work, but only when the model is trained and designed well. When you ask a question, it uses those learned values without normally changing them.

Larger models can be more capable, but they need more memory and do not guarantee better answers.

Learned valuesConceptual illustration
0.42One illustrative weight stays fixed during inference
One of many learned values
PromptFixed weightsResponse

A few values represent the idea. 27B means 27 billion parameters, not 27 billion separate facts.

04 / Quantization1 of 3

Store the same weights with fewer bits.

Quantization reduces the precision used to store weights. Our 27B example at 16 bits per weight needs about 54 GB for weights alone. Select a precision to see what changes.

Same 27B modelWeights only
16-bit · one weight

┄ Original weight━ Stored value

54 GB

054 GB

Fewer bits per weight.
The same number of weights.

Fewer stops mean coarser rounding. These stops illustrate the effect, not the actual Q4/Q6/Q8 encodings. Real files include additional information.

04 / Quantization2 of 3

Fewer bits can reduce the accuracy and nuance of the next word.

Quantization saves memory by storing each learned number with fewer possible values. The model usually keeps its broad behaviour, but lost precision can reduce the accuracy and nuance of its answers or change which word it chooses next. The effect depends on the model, quantization method, and task. At Q6, the theoretical weight estimate is 20.25 GB. The dial exaggerates the rounding so you can see it.

Same 27B modelWeights only
Q6 · one weight

┄ Original weight━ Stored value

20.25 GB

054 GB

Fewer bits per weight.
The same number of weights.

Fewer stops mean coarser rounding. These stops illustrate the effect, not the actual Q4/Q6/Q8 encodings. Real files include additional information.

04 / Quantization3 of 3

Q4 reduces the memory needed for weights.

At four bits per weight, the same 27B model needs about 13.5 GB for its weights, compared with 54 GB at 16 bits. The saving comes from storing each weight less precisely. Context, running software, and other apps still need memory.

Choose a precision that leaves room to run the model, then check its answers on your work.

Same 27B modelWeights only
Q4 · one weight

┄ Original weight━ Stored value

13.5 GB

054 GB

Fewer bits per weight.
The same number of weights.

Fewer stops mean coarser rounding. These stops illustrate the effect, not the actual Q4/Q6/Q8 encodings. Real files include additional information.

All four precisions, at a glance

Same 27B modelWeights only
16-bit · one weight

┄ Original weight━ Stored value

54 GB

054 GB

Fewer bits per weight.
The same number of weights.

Fewer stops mean coarser rounding. These stops illustrate the effect, not the actual Q4/Q6/Q8 encodings. Real files include additional information.

Same 27B modelWeights only
Q8 · one weight

┄ Original weight━ Stored value

27 GB

054 GB

Fewer bits per weight.
The same number of weights.

Fewer stops mean coarser rounding. These stops illustrate the effect, not the actual Q4/Q6/Q8 encodings. Real files include additional information.

Same 27B modelWeights only
Q6 · one weight

┄ Original weight━ Stored value

20.25 GB

054 GB

Fewer bits per weight.
The same number of weights.

Fewer stops mean coarser rounding. These stops illustrate the effect, not the actual Q4/Q6/Q8 encodings. Real files include additional information.

Same 27B modelWeights only
Q4 · one weight

┄ Original weight━ Stored value

13.5 GB

054 GB

Fewer bits per weight.
The same number of weights.

Fewer stops mean coarser rounding. These stops illustrate the effect, not the actual Q4/Q6/Q8 encodings. Real files include additional information.

05 / Estimate weight memory

Estimate weight memory
before choosing a model.

Billions of parameters×Bits per weight÷ 8≈ GB

Multiply the parameter count in billions by the bits per weight, then divide by eight to estimate weight memory in GB. For example, 27 × 4 ÷ 8 gives 13.5 GB. This estimates the weights; the complete runtime needs additional memory.

12B × 4 ÷ 8

6 GBWeights only

27B × 4 ÷ 8

13.5 GBWeights only

70B × 4 ÷ 8

35 GBWeights only

Uniform four-bit estimates in decimal GB (one billion bytes). Actual files can contain mixed precisions, metadata, and extra model components. Runtime memory is more than weights.

06 / Your memory layout

Your memory layout affects which models fit.

A dedicated graphics card has its own working memory, called VRAM, separate from the computer’s main memory, RAM. A model that fits in RAM may be too large for VRAM. On a unified-memory computer, the CPU and GPU share one pool with the operating system and apps. Count that shared capacity once.

Compare the model with the memory its runner will use. Separate RAM and VRAM are not one seamless pool.

Three illustrative memory layouts

Unified-memory laptop

CPUGPU
16 GBShared with system and apps

One shared pool

Desktop PC

CPU
64 GBRAM
GPU
16 GBVRAM

Two separate pools

Workstation

CPU
128 GBRAM
GPU
24 GBVRAM

Two separate pools

No dedicated graphics memory?

You can use a compatible runner on the CPU. Integrated graphics generally share system RAM; their reported shared memory is not extra installed capacity. Check support for your exact hardware. These illustrations describe memory layouts, not compatibility guarantees.

07 / Total runtime memory1 of 2

Your system and apps already use some memory.

13.5 GB of weights leaves 2.5 GB in a 16 GB pool. Before you ask the model anything, the operating system, open apps, and model runner already use memory. That leaves less room for the conversation and spare capacity. Their usage can change, so fitting the weights alone is not enough.

A 16 GB memory poolWeights only

13.5 GB of weights

016 GB

2.5 GB left must cover more than weights

Already using memory

Operating system & open appsModel runner

Their usage varies. Context and spare room also need capacity; none of these amounts is measured here.

What if the weights are much larger?

On the 64 GB RAM / 16 GB VRAM example PC, 128 GB of weights is beyond the combined physical capacity. Even if all 16 GB VRAM could hold weights, 112 GB would still need RAM. Storage space does not make that a practical in-memory fit.

07 / Total runtime memory2 of 2

Longer context can use more of the remaining memory.

Context is the material available for the current answer: instructions, conversation, documents, and generated text. It is counted in tokens—small chunks of text. As the conversation grows or you add documents, the runner may need more memory to keep that material available. The weights, system, and apps still occupy memory, so a model that answers a short question may struggle with a longer one.

Check the runner’s estimate for your model and context setting. There is no universal tokens-to-GB conversion, and the advertised context limit may exceed what your computer can run.

Same model and computerContext varies
Short questionInstructionsQuestion
Longer conversation + documentsInstructionsConversation historyDocument excerptsGenerated text

More active text can need more memory.

Conceptual comparison; the cards are not a memory scale. Weights, system, and apps still use the same memory pool.

08 / Compare RAM and VRAM

A model too large for GPU memory may still run on the CPU.

Here, 32 GB of weights exceeds the graphics card’s 16 GB VRAM but occupies half of the computer’s 64 GB RAM. A compatible runner may run the model on the CPU if enough RAM remains for the rest of the runtime. You still need to check whether responses arrive quickly enough for your work.

Not fitting fully in VRAM does not rule out CPU execution. Useful speed still needs a real trial.

Example PCOne 0–64 GB scale
32 GB weights64 GB RAM16 GB VRAM
016324864 GB
System RAM32 / 64 GB · half the pool
Graphics VRAM32 / 16 GB · twice the pool

Weights Capacity boundary Beyond capacity

CPU execution may work. The weights leave 32 GB of installed RAM before other needs. The full model cannot reside in this 16 GB VRAM.

09 / Let the GPU help1 of 2

You can run a model without a dedicated graphics card.

A compatible runner can use the CPU and system RAM to generate answers. This gives you a way to try local AI without dedicated graphics memory. Whether it is fast enough depends on the model, your hardware, and the task.

CPU-only executionConceptual allocation
One model, in order A → B → C → D → E → F

CPU / RAM

ABCDEF

GPU / VRAM

Not used for this answer

Conceptual allocation; the runner determines the split. Equal blocks show order, not GB, actual layers, or processing time. RAM and VRAM remain separate.

09 / Let the GPU help2 of 2

The GPU can help even when the whole model will not fit.

With a compatible runner, part of the model can use the GPU and its VRAM while the CPU handles the rest using RAM. This is partial GPU offload. It may shorten the wait for answers, but the benefit depends on the hardware, the split, and the work involved.

RAM and VRAM stay separate. Try the supported offload settings to find out whether they help.

Partial GPU offloadConceptual allocation
One model, in order A → B → C → D → E → F

CPU / RAM

ABC

GPU / VRAM

DEF

Conceptual allocation; the runner determines the split. Equal blocks show order, not GB, actual layers, or processing time. RAM and VRAM remain separate.

Choose

Choose a model worth trying

10 / Base or Instruct

For an assistant, start with Instruct.

Instruct models are trained to respond to requests, making them a useful starting point for questions, summaries, and other assistant tasks. A Base model may continue your text instead of carrying out the instruction. Base models can suit text completion or further training; check the publisher’s description before choosing.

Choose the variant for the task, then check that the file format works with your runner.

Illustrative behaviour, not measured outputs

Base · continues the text

Text completion

Explain why leaves change colour.

Include a diagram and a short explanation of the pigments involved…

Instruct · answers the instruction

Assistant response

Explain why leaves change colour.

As chlorophyll breaks down, other pigments become visible.

Return to the filename: what do the other parts mean?
Instruct
A variant intended to follow instructions. Publishers may use other names, such as IT.
Q4_K_M
A particular quantization recipe, using mixed precisions. It is not exactly four bits for every stored value.
.gguf
A file format supported by some runners. The format alone does not establish support for the model architecture.

The opening filename is illustrative. Verify the actual artifact and your runner’s support.

11 / Model scale

Larger models need far more memory.

At the same four-bit precision, a 12B model has about 6 GB of theoretical weights, while a 2.8T model has about 1,400 GB. The first may be practical on personal hardware; the second exceeds every computer shown here before context and software are included.

An available model is not necessarily a practical model for a personal computer.

Four-bit weight estimatesLinear scales
01,400 GB
Gemma 3 12B6 GB
Qwen3.8 27B13.5 GB
Kimi K31,400 GB
The first 16 GB, enlarged below

Enlarged view — different scale

016 GB
Gemma 3 12B6 GB
Qwen3.8 27B13.5 GB

2.25× Qwen’s estimated weights compared with Gemma’s

≈ 104× Kimi’s estimated weights compared with Qwen’s

Idealized weight arithmetic, not file sizes or runtime measurements. Sources: Google · Qwen · Moonshot AI. Checked 16 September 2026.

12 / Compare candidates on your hardware

Shortlist models that leave room for the whole task.

Select an example machine and compare each model’s weights with its RAM, VRAM, or unified memory. Leave room for context and running software in the relevant pool; system RAM or unified memory must also accommodate the operating system and apps. Then check runner support and try the task to judge quality and responsiveness.

Shortlist a plausible candidate; verify the actual file and runtime before treating it as a fit.

Unified-memory laptop

16 GB shared

The model, runner, operating system, and apps share these 16 GB.

Weights only; other runtime needs remain
ModelWeightsWhat to check
Gemma 3 12B6 GBA plausible candidate. The remaining memory must also cover the system, apps, runner, and conversation.
Qwen3.8 27B13.5 GBOnly 2.5 GB for the system, apps, and everything beyond the weights is too constrained for a normal setup. Try a smaller model.
Kimi K31,400 GBAbout 1,400 GB of theoretical weights is far beyond this 16 GB shared pool.

Desktop PC

64 GB RAM · 16 GB VRAM

The CPU uses system RAM. The graphics card has its own VRAM.

Weights only; other runtime needs remain
ModelWeightsWhat to check
Gemma 3 12B6 GBThe weights fit on the graphics card with room for other GPU memory needs. The CPU and apps use separate RAM. Check the full estimate.
Qwen3.8 27B13.5 GBLittle VRAM remains, but the weights have plenty of room in RAM. A compatible runner can use the CPU or partial offload; running entirely on the GPU needs a careful memory check.
Kimi K31,400 GBAbout 1,400 GB of theoretical weights is far beyond both the 64 GB RAM and the separate 16 GB VRAM.

Workstation

128 GB RAM · 24 GB VRAM

More memory in each place, still separate capacities.

Weights only; other runtime needs remain
ModelWeightsWhat to check
Gemma 3 12B6 GBThe graphics card has more room beyond the weights. Check how much the runner needs for your conversation length.
Qwen3.8 27B13.5 GBMore VRAM remains for the conversation and runner. Check their memory needs before loading the model.
Kimi K31,400 GBAbout 1,400 GB of theoretical weights is far beyond both the 128 GB RAM and the separate 24 GB VRAM.
Apply this to your computer

These are examples, not a list of supported computers. Record your own RAM and dedicated VRAM separately, or your single unified-memory capacity. Compare weights with the pool the runner will use, then account for context and other software.

An integrated GPU usually uses system memory. A “shared GPU memory” allowance is not additional installed RAM. Check your runner’s support for the exact processor and graphics hardware.

CPU and GPU rows are alternative placements, not two model copies or a measured offload split. These examples do not establish full compatibility or useful speed. Neither model size nor execution location establishes answer quality.

Check your memory and estimate runtime needs.

Decide

Choose local or hosted AI for the work

Local means the model runs on your computer. Hosted means it runs on a provider’s infrastructure. The better choice depends on the work.

These are starting points. If the information cannot leave your device, that restriction comes first. Otherwise, choose a model and workflow capable of doing the job within your budget.

13 / Sensitive document analysis

Keep confidential document contents on your device.

Prefer local

A fully local workflow lets you analyse documents without sending their contents to a model provider. This is useful when the material cannot be shared. The document, model processing, and saved result must all remain local.

For example Summarise a confidential internal report, with the document and model both on your computer.

The whole application matters: connected tools, syncing, and logging must also keep the contents local. The model still needs to be capable of the task.

Local and cloud behaviour ↗

A fully local workflow

On your device

  1. Confidential report
  2. Local model
  3. SummarySaved locally

No document content sent out

14 / Small, frequent jobs

Repeated local jobs can reduce recurring usage charges.

Prefer local

If a smaller model handles the task well, you can use hardware you already own for each new batch without paying a model provider for those runs. This can be useful for routine work you repeat often; the overall saving depends on your usage and local costs.

For example Categorise short notes or extract the same fields from incoming text every day.

Local is not free: setup time, electricity, maintenance, and hardware have costs. Compare those with hosted subscription or usage charges; savings depend on how much you use it.

Repeat a useful small task

New noteNew noteNew note
One local modelUsed for each batch
IdeasActionsReference

Categorised notes · illustrative output labels

15 / Demanding reasoning

Hosted access can put a more capable model within reach.

Prefer hosted

A difficult task may benefit from a model that exceeds your computer’s memory or processing limits. Hosted access lets you try that model without running it yourself. The benefit comes from the model’s capability; hosting the same model does not make it reason better.

For example Compare conflicting system-design requirements and explain the trade-offs behind a proposed approach.

Choose the model for the job. Hosting alone does not make a model reason better, and a proposal still needs review. Share only information the provider is permitted to receive.

Work through competing requirements

ReliabilityBudgetMaintenance

Provider infrastructure

Capable hosted model
Proposal with reasonsTrade-offs to review

16 / Many agents working together

Hosted capacity can help when many agents strain your computer.

Prefer hosted

Agents are AI helpers that can take several steps and use tools. Running several at once increases demand for processing and memory for their separate conversations. A hosted service can handle that model processing on provider hardware, which may help when the jobs can proceed independently.

For example Separate agents investigate different parts of a research question, then combine their findings.

More agents consume more usage. Service limits, queues, and coordination still apply; parallel work is not a guaranteed speedup. Agents can share a model rather than each loading their own copy.

Multi-agent resource trade-offs ↗

Independent parts of one question

Provider capacity · service limits apply

Research question
Part AAgentOwn context
Part BAgentOwn context
Part CAgentOwn context
Combine findingsOne synthesis

Requests may queue; more agents use more tokens.

17 / Complex tool-driven workflows

A capable hosted model can help choose and interpret tool actions.

Usually prefer hosted for demanding coordination

A demanding workflow may need a model to choose a tool, interpret its results, and decide what to do next. Hosted access can provide a suitable model when local options struggle. Your application still connects and runs the tools; more tool calls alone do not require hosted processing.

For example Search documentation, query a database, analyse results, then choose the next action.

A tool running locally does not keep its results local if they are returned to a hosted model.

How models and tools interact ↗

Reasoning and tools run separately

Sent to the model provider

Hosted model
Tool requests Returned results
Application / tool runnerConnects the model to configured tools

Calls and results

Documentation searchDatabase queryAnalysis tool

Tools run locally or remotely, as configured. Results sent upward leave for the provider.