Small language models explained: what runs on a laptop and when to pick one

A small language model is one compact enough to run on hardware you already own, such as a laptop or a phone. It does not need a rack of data center GPUs. In practice that means a few billion parameters (the learned numbers inside the model) rather than hundreds of billions. For a team, the useful question is narrower than the hype: which jobs survive the drop in size, and which fall apart.

The short answer from the vendors’ own model cards is that small models hold up on scoped, repeatable work and lose badly on long, multi-step reasoning and agent tasks. Every benchmark figure below comes from the company that built the model, measured on its own test setup, unless we say otherwise.

What counts as small

There is no official cutoff. NVIDIA researchers, in a position paper posted on arXiv, define a small model by what it does. It fits on a common consumer device and answers one user fast enough to be practical. They add that, as of 2025, they would count most models below 10 billion parameters as small.

That definition is honest about something parameter counts hide. Google’s Gemma 4 model card lists an E2B model with 2.3 billion “effective” parameters, but 5.1 billion once its embedding tables are counted. The E4B model is 4.5 billion effective and 8 billion in total. The smaller number is what does the per-token work, the larger one is what you have to store.

Mixture of experts models blur the line further. A mixture of experts model splits its weights into many sub-networks and uses only a few for each token. Google’s 26B A4B uses about 3.8 billion active parameters per token out of 25.2 billion in total. It computes like a small model but still needs the memory of a large one.

What actually fits on a laptop

Memory, not processing power, is usually the first wall. Google’s Gemma 4 overview publishes approximate memory needed just to load each model, including 20% overhead:

• E2B: 11.4 GB at 16-bit precision, 5.7 GB at 8-bit, 2.9 GB at 4-bit
• E4B: 17.9 GB, 8.9 GB and 4.5 GB
• 12B: 26.7 GB, 13.4 GB and 6.7 GB
• 31B: 69.9 GB, 34.9 GB and 17.5 GB

The 4-bit figures come from quantization, which stores each weight with fewer bits so the model takes less memory at some cost in accuracy. Google’s mobile builds of E2B go down to 1.1 GB.

Two caveats sit in Google’s own notes. The table covers static weights only, and memory use grows with the length of the prompt and answer. Google also says the numbers may change with your inference tool.

In practice, a 4-bit E4B at 4.5 GB leaves room to spare on a laptop with 16 GB of memory. The 31B at 17.5 GB does not fit there at all.

Once a model fits, speed depends mostly on how fast the chip can read those weights. We went through that in our explainer on why memory bandwidth matters more than TOPS for local chatbots.

Where small models hold up

The NVIDIA authors argue that most calls inside AI agents are “repetitive, scoped, and non-conversational”: parse a command, extract fields, fill a template, route a request. Those are the jobs where a small model, ideally fine-tuned on examples of the task, is good enough. They suggest 10,000 to 100,000 examples as a rule of thumb for that fine-tuning.

They also estimate how much of three open-source agent projects could switch: about 60% of language model calls in MetaGPT, 40% in Open Operator and 70% in Cradle. Treat those as the authors’ judgment from reading the code. They are estimates, not measured results, and the paper is a position piece meant to start a debate.

There is outside evidence on efficiency. For a UNESCO report with University College London, UCL researchers ran experiments on open-source models. UNESCO says small models tailored to tasks such as translation or summarization cut energy use by up to 90% without losing performance. Compression such as quantization saved up to 44%, and shorter prompts and answers cut energy by more than 50%.

Microsoft makes a similar case for its 3.8 billion parameter Phi-4-mini-instruct, released in February 2025. On Microsoft’s internal benchmark platform it scores 88.6 on GSM8K, a grade-school math test, against 82.4 for Llama 3.1 8B, a model about twice its size. Microsoft notes it changed system prompts for some models that broke the output format, so the comparison is not perfectly like for like.

Where they fall apart

The same Gemma 4 table that sells the small models also shows the gap, and one comparison in it deserves a second look. On Tau2, an agent benchmark, Google reports 76.9% for the 31B model, 42.2% for E4B and 24.5% for E2B.

Long documents hurt too. On MRCR v2, a long-context retrieval test that Google ran at 128,000 tokens, 31B scores 66.4% and E4B 25.4%. On AIME 2026 competition math without tools, the gap is 89.2% against 42.5%.

Knowledge is the other limit. Microsoft says Phi-4-mini “does not have the capacity to store too much factual knowledge” and suggests pairing it with search, so the model looks facts up instead of recalling them. Its red team also found the model could sometimes hallucinate function names or URLs when calling tools, which matters if you plan to hand it an agent job.

Google lists E4B at 69.4% on MMLU Pro, a broad knowledge test, above the previous generation Gemma 3 27B at 67.6%. But the Gemma 3 column is marked “no think”, meaning it ran without step-by-step reasoning, and Google does not spell out the settings behind the Gemma 4 columns. A small model beating an old large one is real progress, but this table is not a clean test of size alone. If you want to know how much weight to put on a single score, our guide to what the big AI leaderboards measure covers the usual traps.

What cost looks like

Citing earlier work, the NVIDIA paper says a 7 billion parameter model is 10 to 30 times cheaper to serve than a 70 to 175 billion one. The comparison covers latency, energy and compute, and it is the authors’ summary of other studies, not a measurement of their own.

People cost money too. The same paper concedes that hosting your own model brings setup and staffing costs that are easy to leave out, and that the economics are “highly case-specific”. Training costs belong to the vendor, but they show the scale: Microsoft trained Phi-4-mini on 5 trillion tokens using 512 A100 GPUs for 21 days.

Fine-tuning is where small models change the math for a team. The NVIDIA authors say tuning a small model takes only a few GPU-hours, so a narrow open model can be retrained on your own examples. Nous Research did this at a larger 27B size when it fine-tuned an open Qwen model to cut propaganda-style answers.

Who should pick one

A small model makes sense when the task is narrow and checkable, the data must stay on the device, or the call runs all day. Classification, field extraction from forms, short summaries and routing a request to the right tool all fit that description.

It is the wrong choice for long multi-step agent work, questions over very long documents, or anything that relies on the model remembering facts. NVIDIA’s authors recommend mixing both: a small model handles the routine calls and passes the hard ones to a larger model.

What to do this week

Pick one repetitive task your team already sends to a large model, such as tagging support tickets or pulling fields from invoices. Collect 50 real examples with known correct answers, run them through a 4-bit small open model and your current model, and count the errors. If the small model’s error rate is acceptable, you have a candidate to move on device. If not, you have a test set ready for the next release.

Sources

• Google DeepMind: Gemma 4 model card
• Google: Gemma 4 model overview and inference memory requirements
• Microsoft: Phi-4-mini-instruct model card on Hugging Face
• Belcak et al., NVIDIA Research: Small language models are the future of agentic AI (arXiv)
• UNESCO: small changes can reduce LLM energy use by 90%