01 /

A field guide to running AI yourself

What can your computer run?

On this page, you’ll learn:

  • How to read a model’s name and make a rough estimate of the memory its weights need.
  • Why running the model needs more memory than that estimate.
  • When to use your own computer and when a hosted service makes more sense.
How to read a model name
WHERE THE MODEL RUNS
Help me rewrite this email.
01A remote serverRuns on hardware you don’t control
02Your computerRuns on your own hardware

Understand

How local models work

02 / Reading a model name

The name tells you
the size and the precision.

Take a name like Qwen3.8-27B-Instruct-Q4_K_M.gguf. The 27B is the parameter count: 27 billion. The Q4 says each parameter is stored in roughly four bits. That’s enough to estimate how much memory the weights need.

WHAT THE NAME SAYSILLUSTRATIVE
Qwen3.8-27B-Instruct-Q4_K_M.gguf
27B27 billion parameters.
Q4Roughly four bits per weight.

03 / Parameters1 of 3

What is a parameter?

A parameter, often called a weight, is just a number. Each one nudges the model a little toward some patterns and away from others, and billions of them together produce its answers. Training is what sets them: the model makes a prediction, the training software compares it with the known answer, and the numbers that contributed to the mistake get adjusted slightly.

Learned valuesConceptual illustration
0.37 → 0.4321One example weight, nudged during training
One of billions of learned values
Training adjusts the values

03 / Parameters2 of 3

A model file is a set of trained parameters.

A model file is those finished numbers, saved to disk. They sit on your drive until a runner, the program that runs the model, loads them into memory.

Learned valuesConceptual illustration
0.4321One example weight, fixed once training is over
One of billions of learned values
Training is over; the values are saved

03 / Parameters3 of 3

What do extra parameters buy you?

More parameters give a model more room to hold the patterns and distinctions it picked up in training. That isn’t the same as knowing more: what a model knows depends on the training data and on how it was trained and designed. A bigger model can do better on broader or harder work.

At the same precision, a bigger model needs more memory, and size alone doesn’t promise better answers.

Learned valuesConceptual illustration
0.4321One example weight, fixed once training is over
One of billions of learned values
PromptFixed weightsResponse

04 / Quantization1 of 3

One model can be stored at several precisions.

Quantization stores each weight in fewer bits than a full 16-bit weight, giving up some precision to save space. At 16 bits per weight, our 27B example takes about 54 GB for the weights alone. Compare the four precisions to see what changes.

Same 27B modelWeights only
16-bit · example weight

━ Illustrated value 0.4321

54 GB

054 GB

The number of weights stays the same;
the bits stored for each one can vary.

Each stop on the dial is a value a weight can be rounded to. The stops and numbers are simplified, not values read from a real file.

04 / Quantization2 of 3

Less precision can cost some nuance.

Each weight is rounded to one of a smaller set of possible values than at 16 bits, and that’s where the saving comes from. The model usually keeps its general behaviour, but the rounding can make answers a little less accurate or less nuanced, or tip it toward a different next word. How much depends on the model, the quantization method and the task. At Q6 the weights come to a theoretical 20.25 GB.

Same 27B modelWeights only
Q6 · example weight

━ Illustrated value 0.431

20.25 GB

054 GB

The number of weights stays the same;
the bits stored for each one can vary.

Each stop on the dial is a value a weight can be rounded to. The stops and numbers are simplified, not values read from a real file.

04 / Quantization3 of 3

At Q4 the weights need a quarter of the memory they need at 16 bits.

At four bits per weight the same 27B model needs about 13.5 GB for its weights, against 54 GB at 16 bits. Context, the runner and your other apps still need memory on top of that.

Pick a precision that leaves room to run the model, then try it on your own work to see whether the answers still hold up.

Same 27B modelWeights only
Q4 · example weight

━ Illustrated value 0.43

13.5 GB

054 GB

The number of weights stays the same;
the bits stored for each one can vary.

Each stop on the dial is a value a weight can be rounded to. The stops and numbers are simplified, not values read from a real file.

The four precisions side by side

Same 27B modelWeights only
16-bit · example weight

━ Illustrated value 0.4321

54 GB

054 GB

The number of weights stays the same;
the bits stored for each one can vary.

Each stop on the dial is a value a weight can be rounded to. The stops and numbers are simplified, not values read from a real file.

Same 27B modelWeights only
Q8 · example weight

━ Illustrated value 0.432

27 GB

054 GB

The number of weights stays the same;
the bits stored for each one can vary.

Each stop on the dial is a value a weight can be rounded to. The stops and numbers are simplified, not values read from a real file.

Same 27B modelWeights only
Q6 · example weight

━ Illustrated value 0.431

20.25 GB

054 GB

The number of weights stays the same;
the bits stored for each one can vary.

Each stop on the dial is a value a weight can be rounded to. The stops and numbers are simplified, not values read from a real file.

Same 27B modelWeights only
Q4 · example weight

━ Illustrated value 0.43

13.5 GB

054 GB

The number of weights stays the same;
the bits stored for each one can vary.

Each stop on the dial is a value a weight can be rounded to. The stops and numbers are simplified, not values read from a real file.

The numbers for one weight

These values show one weight with less detail at each setting. A weight is one number in a tensor, not a whole vector. The model still has the same number of weights.

One example weight across four precisions. Simplified illustration.
PrecisionIllustrated value
16-bit (FP16)0.4321
Q80.432
Q60.431
Q40.43

The shorter decimals make the idea easy to see, but real quantization doesn’t just trim decimal places. Q4_K_M uses a different scheme. One weight alone can’t tell you how good the model’s answers will be.

05 / Estimating weight memory

Calculate
the memory required.

Billions of parameters×Bits per weight÷ 8≈ GB

Multiply the parameters (in billions) by the bits per weight, then divide by eight, because a byte is eight bits. The answer is in gigabytes. For 27B at 4 bits that’s 27 × 4 ÷ 8 = 13.5 GB. That figure covers the weights only; the runtime needs more on top.

12B × 4 ÷ 8

6 GBWeights only

70B × 4 ÷ 8

35 GBWeights only

These assume exactly four bits for every weight and use decimal GB (one billion bytes). Real files often mix precisions and add metadata and extra model components, so expect their size to differ.

06 / Your memory layout

Unified VS Pool memory

The CPU uses your computer’s RAM. A dedicated graphics card has its own memory, called VRAM. With a compatible runner, using the GPU often gives you quicker replies. If the model won’t fit entirely in VRAM, you may still be able to run it from RAM on the CPU, but you may wait longer. On a unified-memory computer, the CPU and GPU share memory with the system and your apps; there’s no separate VRAM figure to add.

If quicker replies matter most, check whether your GPU can run the model. If you want to try a larger model, check your RAM too, then see whether the wait is acceptable.

Three illustrative memory layouts

Unified-memory laptop

CPUGPU
16 GBShared with system and apps

One shared pool

Desktop PC

CPU
64 GBRAM
GPU
16 GBVRAM

Two separate pools

Workstation

CPU
128 GBRAM
GPU
24 GBVRAM

Two separate pools

What if there’s no dedicated graphics memory?

A compatible runner can still use the CPU. Integrated graphics generally borrow system RAM, so the “shared memory” figure your system reports isn’t extra capacity on top of the RAM you installed. Your runner’s documentation will say whether it supports your exact hardware.

07 / Total runtime memory1 of 2

The weights aren’t the only thing in memory.

Put 13.5 GB of weights in a 16 GB pool and 2.5 GB is left. That 2.5 GB isn’t free: before you’ve typed a word, the operating system, your open apps and the runner itself are already using some of it, and what they use changes as you work. Whatever remains has to hold the conversation, with a little to spare.

A 16 GB memory poolWeights only

13.5 GB of weights

016 GB

2.5 GB left for everything else

Already using memory

Operating system & open appsModel runner

The system, apps and runner use different amounts on different machines, and context needs room too. No figure for any of them is measured here.

What if the weights are much larger?

On the example PC with 64 GB of RAM and 16 GB of VRAM, 128 GB of weights can’t fit. Even with all 16 GB of VRAM holding weights, the other 112 GB would still have to sit in RAM, and the PC has 64. Free disk space doesn’t make it a practical fit.

07 / Total runtime memory2 of 2

Long sessions need more RAM.

Context is everything the model can see while it writes an answer: your instructions, the conversation so far, any documents you’ve added, and the text it has already generated. It’s measured in tokens, which are small chunks of text. As the conversation grows or you add documents, the runner may need more memory to hold all of it, and that comes out of what’s left after the weights, the system and your apps. So a model that handles a short question comfortably can struggle with a long one.

Let your runner estimate the memory for your model and context setting; there’s no universal conversion from tokens to GB. A model’s advertised maximum context may be more than your computer can handle.

Same model and computerContext varies
Short questionInstructionsQuestion
Long conversation with documentsInstructionsConversation historyDocument excerptsGenerated text

08 / When VRAM isn’t enough

Can the GPU still help?

On our example PC, the model has 32 GB of weights, but the graphics card has only 16 GB of VRAM. The PC also has 64 GB of RAM. If there’s enough free RAM for the model and the rest of the system, a compatible runner can run it on the CPU. With partial GPU offload, the runner puts some of the model in VRAM for the GPU and leaves the rest in RAM for the CPU.

RAM and VRAM remain separate pools. If your runner supports offload, try it and compare the wait with CPU-only. Whether it helps depends on your hardware and the work you’re doing.

Example PCOne 0–64 GB scale
32 GB weights64 GB RAM16 GB VRAM
016324864 GB
System RAM32 / 64 GB · fills half of it
Graphics VRAM32 / 16 GB · twice its capacity

Weights Capacity boundary Beyond capacity

09 / Base or Instruct

For an assistant, start with Instruct.

Instruct models are trained to respond to requests, which is what you want for questions, summaries and other assistant-style work. Give a Base model the same request and it may simply carry on your text instead of doing what you asked. Base models can suit text completion or further training, so check the publisher’s description before you choose one.

Match the variant to the job, then check that your runner can open the file format.

Illustrative behaviour, not measured outputs

Base · continues the text

Text completion

Explain why leaves change colour.

Include a diagram and a short explanation of the pigments involved…

Instruct · answers the instruction

Assistant response

Explain why leaves change colour.

As chlorophyll breaks down, other pigments become visible.

Back to the filename: what do Instruct, Q4_K_M and .gguf mean?
Instruct
A variant trained to follow instructions. Publishers sometimes use other names, such as IT.
Q4_K_M
One specific quantization recipe. It mixes precisions, so not every stored value is exactly four bits.
.gguf
A file format that some runners can open. Being able to read the format doesn’t mean a runner supports this particular model’s architecture.

That filename is only an example. For a real download, check the actual file and whether your runner supports it.

Decide

Hosted VS Local inference

Local means the model runs on your computer. Hosted means it runs on remote infrastructure you don’t control, and you reach it over the internet.

Sometimes the choice is made for you: if the material can’t leave your device, hosted is off the table. Otherwise, pick whichever setup does the job well enough at a price you can live with.

10 / Confidential documents

A confidential document can stay on your device.

Prefer local

When the model runs on your own computer, the document doesn’t have to go to a model provider at all.

For example, you can have a local model summarise an internal report and save the summary in the same folder.

If that folder syncs to the cloud, the report leaves the device anyway, and connected tools or the app’s logging can carry it off too. The local model also has to be good enough for the job.

Ollama FAQ: local processing versus cloud features ↗

A fully local workflow

On your device

  1. Confidential report
  2. Local model
  3. SummarySaved locally

No document content sent out

11 / Small, frequent jobs

Running the same small job day after day?

Prefer local

If a smaller model does it well, every run uses hardware you already have, and no model provider charges you for it.

For example, a batch of short notes arrives every morning, and each one needs filing as an idea, an action or a reference.

Local isn’t free, though: setup time, electricity, upkeep and the hardware itself all cost something. Whether it saves money depends on how often you run the job and what a hosted service would charge for it.

The same small task, every day

New noteNew noteNew note
One local modelUsed for each batch
IdeasActionsReference

Notes sorted by category · the category names are illustrative

12 / Demanding reasoning

Some problems are worth sending to a stronger model.

Prefer hosted

The model you’d want may not fit in your computer’s memory, or may run too slowly there. Through a hosted service you can use it anyway.

For example, it could compare a system design’s conflicting requirements, such as reliability against budget, and explain the trade-offs behind what it proposes.

A capable model can help whether it runs on your computer or a hosted service. Check its suggestions as you would a colleague’s, and only send the service material you’re allowed to share.

Requirements that pull against each other

ReliabilityBudgetMaintenance

Provider infrastructure

Capable hosted model
Proposal with reasonsTrade-offs to review

13 / Many agents at once

Swarms of subagents each need their own RAM

Prefer hosted

An agent is an AI helper that works through a task in steps and can use tools. Agents on one computer can share a loaded model, but each keeps its own conversation, and every conversation takes processing and memory. When the jobs don’t depend on each other, a hosted service can do that processing on remote infrastructure instead.

For example, three agents could each take one part of a research question, with a final step combining what they find.

Splitting the job up doesn’t automatically make it faster: service limits, queues and the work of coordinating the agents still apply.

Anthropic on subagent contexts, coordination and token use ↗

One question, split into independent parts

Provider capacity · service limits apply

Research question
Part AAgentOwn context
Part BAgentOwn context
Part CAgentOwn context
Combine findingsOne combined answer

Each extra agent uses more tokens.

14 / Using tools

Chained requests

Consider hosted for harder multi-step tasks

Try the whole task with a local model first. If it misreads a tool’s result or keeps choosing the wrong next step, a hosted model may do better. The number of calls alone isn’t a reason to switch.

For example, it might look up a field in the documentation, then use it to query a database.

If you use a hosted model, any tool results you send back to it go to the provider, even if the tool ran on your computer.

Hugging Face docs: the model requests a tool call, your code runs it ↗

How a tool call works

Sent to the model provider

Hosted model
Tool requests Returned results
Application / tool runnerConnects the model to configured tools

Calls and results

Documentation searchDatabase queryAnalysis tool

Each tool runs wherever it’s configured to, on your machine or elsewhere.

Develop

Hardware and model examples

Model scores on Artificial Analysis

Intelligence Index v5.0.1 · checked 2026-11-05 · higher is better

  1. Claude Opus 5.5Max, Default Fallback59
  2. Claude Sonnet 5.5Max, Default Fallback55
  3. GPT-6.1 SolMax55
  4. GPT-6.1 TerraMax51
  5. Claude Opus 5Max44
  6. Claude Haiku 5.5Max36
  7. Grok 4.7Xhigh35
  8. DeepSeek V4.1 Flash 1017Max · open weights33
  9. GLM-5.3Max · open weights32
  10. GPT-6 LunaMax28
  11. Qwen3.8 27BXhigh · open weights20

Source: Artificial Analysis model comparison. API scores, not tests of the local files below.

Two local machine examples

Here is how the memory checks apply to two machines you might develop with.

Linux PC with an RTX 3090

Memory
24 GB VRAM · 64 GB system RAM (owner’s example)
Model
Qwen3.8 27B, Q4_K_XL file
Fit
projectedThe weights file is 16.35 GiB, not the 13.5 GB that exactly four bits per weight gives, because this quant keeps some tensors at higher precision. Add about 2 GiB of context at 32,000 tokens and roughly 0.5 GiB of runtime and it projects to about 19 GiB of the GPU’s 24 GiB VRAM. The 64 GB of system RAM is separate and can help with partial CPU offload.
Tasks to try
Explain a function and draft tests for a small change.
Est. US price (Nov 2026)
$2,830

Sources: GPU spec · Model files · PC price · 64 GB RAM price

Mac Studio with M5 Ultra

Memory
256 GB unified memory
Model
DeepSeek V4.1 Flash 1017, 2-bit file
Fit
vendor-reportedUnsloth estimates 176–205 GB of memory for this file. Metal support and GPU access to the Mac’s 256 GB still need testing.
Tasks to try
Trace a bug across files or propose a multi-file change.
Est. US price (Nov 2026)
$9,500

Sources: Mac spec · Model files · Memory estimate · Mac price

Which work suits local or hosted AI

If most of these jobs would use hosted models, you may not need either machine for AI work.