The Literature Already Knew: Why 4-Bit Quants Tie and Thinking Budgets Bite

The research on LLM quantization and test-time compute makes two clear predictions. A 67-hour, 4,800-task community benchmark of Qwen3.8-27B just confirmed both.

Short Summary

If you run language models on your own hardware, you have probably worried about the wrong thing. Two separate bodies of research point to the same conclusion. The first, on quantization, says that compressing a model’s weights to 4 bits costs almost no accuracy when done well. Real damage only appears at 3 bits and below, or in rare fragile configurations. The second, on test-time compute, says the dial people ignore is the dangerous one: letting a model “think” longer helps up to a point, then flattens out, and sometimes makes results worse.

This post explains both research threads in plain terms. Then it examines a recent independent benchmark that tested exactly what the papers predict, in the wild. One person ran Qwen3.8-27B through five different inference setups, four reasoning settings, and 4,800 tasks, with no limits on how long the model could think. The result: all five setups tied statistically at high reasoning effort. Changing the reasoning setting moved accuracy by 8 to 12 points. The highest setting burned 7 to 11 times more tokens than the low setting, and bought almost nothing in return.

The theory and the field data agree. You are probably tuning the wrong dial.

Introduction

Science works best when two kinds of evidence line up. Labs produce controlled experiments. The real world produces observations. When an observation lands exactly where the experiments pointed, you can trust both a little more.

Local LLM inference rarely gets this luxury. Researchers test models on standardized benchmarks with lab tooling. Practitioners run Ollama on a MacBook and argue in comment sections. The two worlds barely talk to each other.

This post connects them, on one specific question. When you run a model at home, what actually hurts quality: the 4-bit compression you feel guilty about, or the reasoning settings you never look at?

The research already has an answer. And in August 2026, an independent tester named Alexey Fateev produced the cleanest real-world confirmation of that answer I have seen. He did not frame it that way himself. The data does.

What the Quantization Research Says

First, some vocabulary. Quantization means storing a model’s weights with fewer bits to save memory. A “4-bit quant” stores each weight in 4 bits instead of 16. The shorthand W4A16 means 4-bit weights with 16-bit activations (the intermediate values the model computes while running). The question is always the same: how much intelligence do you lose in the compression?

The answer, at scale: almost none. Red Hat and Neural Magic ran over half a million evaluations on quantized models, formalized in the paper “Give Me BF16 or Give Me Death?”. Their finding: well-made 8-bit and 4-bit quantization recovers essentially all of the original model’s accuracy, and W4A16 is the sweet spot for running models locally. An independent study of instruction-tuned models up to 405B parameters found the same thing across model families. So did the AWQ paper, which won the MLSys 2024 best paper award for showing that smart 4-bit compression preserves capability.

Then reasoning models arrived, models that think step by step before answering, and researchers checked whether the story still held. “Quantization Hurts Reasoning?” (COLM 2025) tested quantized reasoning models from 1.5B to 70B parameters and drew a precise boundary. At W4A16 or 8-bit, quantization stays lossless, even for long chains of reasoning. Below that, real risk appears, and it depends on model size, model family, and task difficulty.

Below the boundary, things get genuinely strange. A 2026 study titled “Quantized Reasoning Models Think They Need to Think Longer, but They Do Not” found that 3-bit compression made a model’s reasoning 4.5 times longer while its math accuracy collapsed from 85.6% to 47.0%. In over half the failures, the model actually found the right answer mid-reasoning, then talked itself out of it.

Put together, the research makes a precise prediction. At 4-bit weights with sensible settings, expect a statistical tie with the uncompressed model. A cliff exists, but it sits at 3 bits and in unusual corner cases, not where most people stand.

What the Test-Time Compute Research Says

The other dial is the reasoning setting, and its research is less comforting.

The good news came first. A well-known paper showed that giving a model more thinking time can beat making the model bigger. This is why every modern model now ships with a thinking mode and effort settings. More thinking buys more accuracy.

Until it does not. A paper bluntly titled “Do NOT Think That Much for 2+3=?” documented reasoning models spending enormous token budgets on problems a calculator solves instantly, gaining nothing. A survey on efficient reasoning catalogs this “overthinking” pattern across many models. A 2025 paper on inverse scaling in test-time compute goes further: on some tasks, longer reasoning makes accuracy actively worse, because the model amplifies its own distractions and errors. There is now even a benchmark, OptimalThinkingBench, that measures both failure modes at once, because thinking too little on hard problems and too much on easy ones are the same disease. The model cannot price its own thought.

So the second prediction: the reasoning setting should move quality far more than the choice of quant. Returns should diminish hard at the top. Cost should grow in multiples. And occasionally, more thinking should just dig a deeper wrong hole.

The Field Test: 4,800 Tasks, Five Setups, No Limits

Now the real-world data, read against those two predictions.

Fateev spent 67 hours of machine time testing Qwen3.8-27B, a 27-billion-parameter open model with a 262K-token context window and an adjustable reasoning-effort setting, on his home rig of four RTX 3090 GPUs. The full thread is public.

Three design choices make this test unusually trustworthy.

First, he tested complete setups, not just weight files. Each of the five configurations below is something you could actually deploy, with its own engine and cache format:

Setup	Engine	GPUs	KV cache	Context
FP8 (baseline, 8-bit)	vLLM	2x 3090	FP8	262K
NVFP4 (4-bit)	vLLM	2x 3090	FP8	262K
AWQ INT4 (4-bit)	vLLM	2x 3090	FP8	262K
GGUF Q4_K_M (4-bit)	llama.cpp	2x 3090	Q8_0	262K
NInfer (mixed 4-6 bit)	ninfer-3090	1x 3090	INT8	187K

(The KV cache is the model’s working memory of the conversation. It gets compressed too, which matters later.)

Second, he crossed every setup with all four reasoning settings the model offers: off, low, medium, and xhigh. Each combination ran 150 real-world coding and agent tasks plus 90 reasoning problems drawn from three published benchmarks: HumanEval+, LiveCodeBench, and GSM-Symbolic. In total: 4,800 tasks, 10,120 requests, 14.5 million reasoning tokens.

Third, and rarest of all: no token limits anywhere. The model thought as long as it wanted, and every token was logged. He even audited himself afterward and found that three of his baseline cells (FP8 at off, low, and medium) had accidentally kept an old token cap. He flagged them publicly and promised a clean rerun. Those three cells are marked below.

Here are the scores. Each cell is the percentage of the 150 tasks passed on the first attempt:

Reasoning	FP8	NVFP4	GGUF Q4_K_M	AWQ INT4	NInfer
off	80.7*	81.3	77.3	78.0	79.3
low	86.7*	87.3	89.3	86.7	83.3
medium	87.3*	86.7	88.0	86.0	84.0
xhigh	88.7	89.3	89.3	90.0	88.0

*Affected by the leftover token cap. Clean rerun pending.

Prediction one, confirmed. Look at the xhigh row. The best setup (AWQ INT4 at 90.0%) and the worst (NInfer at 88.0%) differ by three tasks out of 150. Fateev ran the proper statistical test (McNemar’s, which compares paired results task by task) and got p-values between 0.69 and 1.00 for every quant against the baseline. In plain terms: no detectable difference. A tie, exactly where the W4A16 research said the tie would be. He started the project expecting to document quantization damage. He found nothing to document.

Prediction two, confirmed. Turning reasoning from off to xhigh added 8 to 12 points on every setup. That effect is roughly ten times larger than any difference between quants. And here is what the top setting costs:

Setup	Tokens at low	Tokens at xhigh	Ratio	Accuracy gain
AWQ INT4	83,440	578,123	6.9x	+3.3 pts
NInfer	98,093	699,018	7.1x	+4.7 pts
GGUF Q4_K_M	86,367	651,281	7.5x	+0.0 pts
FP8	88,513	724,404	8.2x	+2.0 pts
NVFP4	73,797	819,808	11.1x	+2.0 pts

Read the GGUF row twice. Same score at low and at xhigh. Seven and a half times the thinking.

The stranger corners of the research showed up too. The medium setting scored below low on the reasoning problems for all five setups. That is the overthinking signature from the papers: extra deliberation hurting short coding problems. And one agent task defeated every single setup at xhigh. The five setups poured a combined 700,000 reasoning tokens into it. Individual attempts burned 93,000 to 174,000 tokens each. The longest ran 25 minutes. All five failed. That is inverse scaling, live: more thinking funding a longer excavation of the wrong hole.

One more result completes the picture. The only statistically significant quantization failure in the entire run was a corner case, exactly the kind the reasoning papers warn about. NVFP4 with reasoning turned off collapsed on HumanEval+ (13 out of 30, against the baseline’s perfect 30), with p = 0.0041. Turn reasoning up to just low, and it scored a perfect 30. Quantization fragility is real. But it lives in specific configurations, not in the format itself.

How Much to Trust Each Number

Not every number above deserves the same confidence. Three tiers.

Solid, tested within the run. The tie between quants at xhigh (p = 0.69 to 1.00). The 8-to-12-point effect of the reasoning setting. The NVFP4 collapse with reasoning off (p = 0.0041). These have proper statistical tests behind them, run by the author.

Suggestive, but untested. The medium-below-low dip appeared on all five setups, but no significance test was reported for it. The token-cost ratios come from single runs with no repeats. And any gap of 1 to 3 points between setups is invisible at this sample size. With 150 tasks per cell, differences that small are noise.

Not claimable at all. Any ranking of the five setups at xhigh. “AWQ beats FP8” would need evidence this run cannot provide, and to his credit, the author never claims it. This is one model, one run, no repeats, a custom task suite that is not public, self-reported numbers, and three baseline cells awaiting a rerun.

The right way to hold this is the scientific way. One field observation proves little on its own. Its value is agreement: every major pattern in this run landed where independent lab research, published over three years by unrelated groups, said it would land.

What This Does and Does Not Prove

Together, the research plus the field test support three working conclusions. One: modern 4-bit quantization of a well-supported model costs you approximately nothing, on real deployed workloads and not just lab benchmarks. Two: the reasoning setting is the dominant quality lever, with such steep diminishing returns that maximum effort should be a specialist tool, never a default. Three: where quantization failures still exist, they are configuration-shaped. A specific format, times a specific setting, times a specific task family. That is why aggregate scores keep missing them, and why you should test your own workload.

What none of this proves is that quantization is free everywhere. Push to 3 bits and the 2026 study shows accuracy halving while thinking quadruples. Small distilled reasoning models are more fragile than large ones, per the COLM study.

The field test also has a design limitation worth stating plainly: it cannot separate weights from everything else. The GGUF setup compresses its working memory (KV cache) differently than the vLLM setups do, so any difference between them mixes two causes. A sharp reply in the thread pointed this out, and the author agreed, with a fair defense: he tested configurations he would actually deploy, so the comparison is complete setups against complete setups, not a pure weights experiment. NInfer is similarly entangled. It mixes several bit-widths and runs on one GPU at reduced context, and an independent NYU Shanghai audit of its parent engine found that its impressive speed can coexist with broken output on structured tasks. Real deployments are tangled objects. That is exactly why testing them whole is useful, and why the results describe setups, not formats in isolation.

The lab and the field now agree. The community argues about quants and shrugs at reasoning settings. It should be doing the opposite.

What to Actually Do on Your Own Machine

The numbers above came from a four-GPU rig with a 262K context window. Your machine is smaller. Here is what transfers and what does not.

The quant verdict transfers directly. Pick your quant by what fits your memory, not by fear. A Q4_K_M or MLX build of a 27B model is around 18 GB and runs comfortably on a Mac with 24 GB or more of unified memory, or on a single 24 GB GPU. The research says you are not paying a meaningful quality tax. Just do not chase 3-bit compression on reasoning workloads. That is where the published cliff is.

The reasoning economics transfer with interest, because your limits are tighter. Ollama gives models only a few thousand tokens of context by default unless you raise the num_ctx setting (see the FAQ), and realistic local usage sits around 16K to 64K. Now recall what xhigh spends: single failed attempts in the field test used 93,000 to 174,000 reasoning tokens. On an 8K context window, maximum reasoning effort is not merely expensive. It often cannot physically fit. The window cannot contain the digging.

So the practical playbook is the low row, not the xhigh row. Run reasoning at low by default. Never turn it fully off: that cost 8 to 12 points on every setup, and the one genuine quantization failure in the whole run lived there. Save high effort for individual hard problems, launched deliberately, with a large context window behind them. And spend one evening running your own five or ten representative tasks through your own setup, because the failures that remain are configuration-shaped, and nobody has benchmarked your configuration.

Weight precision is the number on the box. Thinking budget is the bill.

Limitations

Every piece of evidence here has edges. The lab studies test standardized benchmarks under uniform tooling and can miss effects that only appear in messy deployments. The field test is one person, one model, one run: self-reported, never repeated, with a private task suite, three baseline cells awaiting a clean rerun, and cache formats tangled with weight formats. Agreement between weak evidence and strong evidence strengthens the conclusion only modestly. And reasoning-effort economics are a property of how each model was trained. Other models price their thinking differently, so none of these specific numbers should be quoted for a model nobody tested. Treat everything here as a well-supported starting assumption, not a law.

Conclusion

For three years the research has been converging on two findings the local LLM community keeps not hearing. Compressing weights to 4 bits, done well, costs approximately nothing. And the thinking budget is the expensive, dangerous, badly-defaulted dial that actually decides your quality and your cost.

Now a 67-hour, uncapped, honestly-audited field test has observed both findings at once, on hardware people actually own.

When the lab and the field agree this cleanly, update.

Audit the dial, not the weights.

Sources and References

Academic literature

The field test and its stack

Benchmark thread: Alexey Fateev (@superalesha) on X, August 20, 2026
Model: Qwen/Qwen3.8-27B and Qwen/Qwen3.8-27B-FP8
Don-Chad/ninfer-3090 and the NYU Shanghai RITS audit
NVFP4 background: Red Hat Developer and vLLM llm-compressor docs
Local deployment: Ollama FAQ and Qwen3.8-27B on Ollama