---
title: "Tech | A Popular Trick for Making AI More Efficient Has Limits"
author: "Unique Research"
sourcePublication: "Unique Research (FFCAP English library)"
originalPublishedAt: "2025-01-02T12:32:54+08:00"
canonical: "https://ffcap.cn/en/research/20250102a04mwy00"
source: "https://ffcap.cn/en/research/20250102a04mwy00"
language: "en"
cnSourceUrl: "https://view.inews.qq.com/a/20250102A04MWY00"
---

# Tech | A Popular Trick for Making AI More Efficient Has Limits

_Unique Research · Tech · January 2, 2025_

Quantization is one of the most widely used ways to make AI models more efficient — and it has limits the industry may soon hit.

In AI, quantization means using fewer bits to represent information. Analogy: asked the time, you might say “noon” instead of “12:00:01.004.” Both are right; one is less precise. How much precision you need depends on context.

Models have many components that can be quantized, especially parameters — internal variables used for predictions. With millions of computations at runtime, lower-bit parameters are mathematically (and computationally) cheaper. (This is not distillation, which is a more selective pruning process.)

But trade-offs may be steeper than assumed.

## Shrinking models

Research from Harvard, Stanford, MIT, Databricks, and Carnegie Mellon finds quantized models perform worse when the original full-precision model was trained long and hard on lots of data. In some regimes, training a smaller model directly may beat shrinking a large one.

That is awkward for labs that train huge models (for answer quality) then quantize to cut serving cost. Effects are already visible: developers and academics reported quantized Meta Llama 3 often looking “more harmful” than peers — possibly due to how it was trained.

“In my view the biggest cost in AI now and ahead is inference, and our work shows one important way to cut inference cost will not work forever,” first author Tanishq Kumar, a Harvard math student, told TechCrunch.

Contrary to popular belief, inference (ChatGPT answering a question) is often more expensive in aggregate than training. Google reportedly spent ~$191M training one flagship Gemini — large, but answering half of Google Search queries with a 50-word model reply could cost ~$6B a year.

Big labs have bet that “scale” — more data and compute — yields ever-stronger AI. Meta trained Llama 3 on 15 trillion tokens (~750,000 words per million tokens); Llama 2 used ~2 trillion. In early December Meta released Llama 3.3 70B, claiming better core performance at lower cost.

There are signs of diminishing returns: Anthropic and Google reportedly trained huge models that missed internal benchmarks. Few signs the industry is ready to leave entrenched scaling recipes.

## How precise is precise enough?

If labs won’t train on smaller datasets, can models be made less fragile? Possibly. Kumar and coauthors found training at “low precision” can make models more robust.

Precision here is how many bits a numeric type accurately represents — e.g., FP8 uses 8 bits for floats. Most models train at 16-bit (“half”) precision then post-training quantize to 8-bit: some components convert to lower precision at a small accuracy cost — like computing to many decimals then rounding to tenths.

Hardware vendors such as NVIDIA push even lower precision for inference. Blackwell chips support 4-bit FP4, pitched as a win for memory- and power-constrained data centers.

But very low precision may hurt. Kumar says below 7–8 bits, quality can drop sharply unless the original model is extremely large in parameter count.

The takeaway is simple: AI models are not fully understood, and shortcuts that work elsewhere don’t always apply here. You wouldn’t start a 100-meter race at “noon.”

“You can’t naively work around these limits,” Kumar said. “We hope our work adds nuance to defaults that keep chasing lower training and inference precision.”

The study is relatively small — more models to come — but one insight should hold: no free lunch on inference cost. “Bit precision matters, and it isn’t free. You can’t lower it forever without hurting performance. Model capacity is finite — so rather than stuffing trillions of tokens into a tiny model, people will put more care into curating and filtering data for smaller models. I’m optimistic about new architectures that stabilize low-precision training — those will matter.”

— END —

---

Editorial English edition published on UniqueCapital: https://ffcap.cn/en/research/20250102a04mwy00
On-site reading page: https://ffcap.cn/en/research/20250102a04mwy00