Working AI domestically or on CPUs often means accepting slower responses or utilizing smaller, much less succesful fashions than GPU-hosted massive language fashions (LLMs).
On the similar time, small language fashions (SLMs) can run totally on CPUs and are more and more able to dealing with specialised duties akin to buyer help, translation, doc processing, and code era.
Whereas SLMs aren’t designed to exchange LLMs, they provide a sensible, lower-cost method to deploy AI for a lot of real-world functions with out devoted GPU infrastructure.
With so many SLMs and deployment choices obtainable, the larger problem is figuring out which of them are literally value deploying.
To search out out, we benchmarked 34 CPU-only SLM configurations throughout 4 open-weight mannequin households to determine which of them delivered the largest efficiency beneficial properties.
The experiment we ran
We examined 34 CPU-only SLM configurations constructed from 4 open-weight mannequin households. Every household included a number of parameter sizes (the place M = million parameters and B = billion parameters):
- Qwen3.5 (0.8B and 2B)
- Gemma-4 E2B
- Gemma-3n E2B
- LFM2.5 (350M and 1.2B)
Every mannequin household was examined at a number of compression ranges, from full precision all the way down to aggressively compressed. Each configuration ran underneath the identical mounted settings every time, so the numbers are instantly comparable and reproducible.
Then, we measured the SLM efficiency throughout 5 axes:
- Reasoning accuracy, examined with MMLU-Professional
- Cross-language comprehension, examined with Belebele
- Single-turn latency, that means time-to-first-token and decode pace on standalone requests
- Multi-turn latency, measuring the identical two metrics underneath a back-and-forth chat workload
- Lengthy-context dealing with, examined with a needle-in-a-haystack setup that measures the utmost context size a mannequin can deal with earlier than timing out
Testing ran on two intentionally completely different CPU platforms – a Google Cloud Intel Xeon (8 cores/16 threads, no AVX-512) and a Hostinger VPS on AMD EPYC (8 cores/8 threads, with AVX-512).
The total 34-config sweep ran on the Intel Xeon machine, and an 18-config shared subset ran on each Intel Xeon and AMD EPYC CPUs. Cross-machine comparisons are drawn from the shared subset.
Comply with our ongoing SLM on CPU benchmark
This benchmark is an ongoing venture. We’ll proceed including extra fashions, configurations, and outcomes over time. To remain updated with the most recent benchmarks and methodology, learn our full report on Hugging Face.
Is compressing an SLM impactful?
One comparability sample stored showing all through our benchmark. Fewer-parameter fashions matched compressed higher-parameter fashions in inference pace whereas delivering considerably weaker reasoning efficiency.
These two Qwen mannequin configurations confirmed a transparent instance:
| Configuration | Reasoning accuracy | Decode pace |
| Qwen3.5-2B, compressed (Q8) | 36.6% | 18.8 tokens/second |
| Qwen3.5-0.8B, uncompressed (BF16) | 20.5% | 18.7 tokens/second |
And it wasn’t an remoted consequence. Comparable trade-offs appeared all through the benchmark, with compressed fashions repeatedly outperforming their full-precision counterparts.
It is because downsizing the AI mannequin configuration removes parameters, decreasing each the mannequin’s reminiscence utilization and its reasoning capability. In the meantime, compression retains the identical SLM structure intact and shops its weights extra effectively, preserving rather more of the mannequin’s functionality at an analogous latency.
Our experiment additionally revealed that not all compression ranges are equally helpful. Evaluating the identical Qwen3.5-2B mannequin throughout completely different compression ranges makes that clear:
| Configuration | Reasoning accuracy | Decode pace |
| Qwen3.5-2B (Q8) | 36.6% | 18.8 tokens/second |
| Qwen3.5-2B (This autumn) | 31.3% | 20.2 tokens/second |
| Qwen3.5-2B (BF16) | 34.8% | 10.5 tokens/second |
On this case, the Q8 variant (the “Q” denotes a compressed mannequin and “8” refers to 8-bit precision) was nearly a free optimization, delivering almost double the throughput with out sacrificing reasoning accuracy.
However the beneficial properties didn’t proceed indefinitely. Compressing additional to 4-bit precision (This autumn variant) traded a number of factors of reasoning accuracy from 36.6% to 31.3% for added pace.
SLM compression proved to be a more practical optimization technique than downsizing, however solely up to some extent. Our testing outcomes counsel that average compression hits the candy spot between pace and reasoning efficiency.
Which SLM carried out the perfect?
As soon as we plotted all 34 mannequin configurations by reasoning accuracy and decoding pace, solely 5 have been by no means crushed on each fronts directly.
And amongst these 5, solely three SLM configurations confirmed outstanding efficiency insights:
- Smartest – Qwen3.5-2B (Q8) delivers the best reasoning accuracy whereas sustaining interactive response speeds. It’s your best option for assistants and workflows the place reply high quality issues most.
- Most balanced – Qwen3.5-0.8B (This autumn) gives among the best speed-to-quality trade-offs within the benchmark, exhibiting a big soar in throughput for a comparatively small drop in reasoning efficiency. For a lot of manufacturing workloads, that is doubtless the candy spot.
- Quickest – LFM2.5-350M (This autumn) prioritizes latency above every thing else, making it well-suited for routing, classification, extraction, and different light-weight duties.
Apparently, not one of the surviving tiers use a full-precision mannequin. Every is a compressed variant.
The place these finest SLMs fell brief
Pace and compression apart, these fashions are nonetheless small language fashions, and that reveals up most clearly within the matter and the language it’s requested in.
On reasoning, the hole is subject-dependent.
These SLMs deal with knowledge-and-memory duties, like biology, historical past, and common well being questions, moderately properly. They wrestle extra with issues that require step-by-step considering, like math or enterprise logic.
Language utilization reveals an analogous sample.
All fashions carried out finest in English, with a modest drop in accuracy throughout German, Spanish, French, and Portuguese. Gemma held up essentially the most evenly throughout all 5, making it the higher decide if multilingual output issues greater than squeezing out the previous few factors of reasoning accuracy.
Does newer {hardware} at all times imply quicker SLM outcomes?
Working the identical 18 SLM configurations on Intel Xeon and AMD EPYC servers, we anticipated the newer AVX-512 EPYC processor to constantly outperform the older AVX2 Xeon.
On common, it did, delivering 1.12x larger decode throughput throughout the shared configurations. However that common hid two completely different tales.
Uncompressed SLMs almost doubled throughput on newer EPYC processors, due to AVX-512 and better clock speeds that accelerated massive matrix multiplications.
Alternatively, compressed fashions ran quicker on the older Xeon server as a result of it uncovered 16 processing threads, in comparison with solely 8 on the EPYC digital machine. For these workloads, thread depend mattered greater than newer vector directions.
Reasoning accuracy remained successfully unchanged throughout each CPU varieties. This means that your alternative of {hardware} shall be primarily pushed by price and latency, not by output high quality.
Why do some fashions nonetheless really feel gradual to reply, even once they generate textual content shortly?
In our experiment, we measured time-to-first-token for each SLM configuration. Whereas tokens per second measures how shortly a mannequin generates textual content, time-to-first-token captures how lengthy customers watch for the primary phrase.
A roughly 1,000-token immediate, akin to a retrieval-augmented era (RAG), lengthy system instruction, or doc to summarize, was added anyplace from 2 to fifteen seconds earlier than the primary token appeared.
The penalty grew with mannequin measurement. Qwen3.5-2B Q8 waited round 10.5 seconds, whereas Gemma 3n Q8 took almost 15 seconds.
For AI functions constructed round RAG, lengthy system prompts, or massive context home windows, bettering throughput alone gained’t get rid of the wait earlier than the mannequin begins responding. Lowering delay additionally means preserving prompts concise, caching repeated system prompts, and selecting smaller mannequin tiers for retrieval-heavy workloads.
Selecting the perfect SLM on your venture
Our benchmark confirmed that optimizing SLM deployments is much less about selecting the “finest” mannequin and extra about choosing the proper optimization technique.
Compression constantly delivered extra worth than merely selecting a smaller mannequin, and each configuration that survived our mixed reasoning-and-speed lower was a compressed variant.
The identical sample appeared elsewhere within the benchmark. Newer {hardware} solely improved throughput underneath particular workloads, whereas immediate size usually had an even bigger affect on perceived responsiveness than decoding pace.
Collectively, these findings counsel that getting essentially the most out of an SLM isn’t about making use of as many optimizations as doable. It’s about figuring out which one can have the largest affect.
Larassatti Dharma is a content material author with 4+ years of expertise within the internet hosting trade. She has populated the web with over 100 YouTube scripts and articles round internet hosting, digital advertising, and e-mail advertising. When she’s not writing, Laras enjoys solo touring across the globe or attempting new recipes in her kitchen. Comply with her on LinkedIn
Tomas Rasymas, AI Analysis Lead at Hostinger, drives the combination of AI to reinforce our merchandise and buyer expertise. With over 15 years of tech expertise, he leads AI analysis, mentors a proficient workforce, and advocates for transformative AI energy. Throughout leisure, Tomas enjoys path working and studying books.
