PeakWeights: Data-Free Discovery of Critical LLM Parameters

Thiyagarajan M (Kalmantic Labs) & Vamshi Ambati (CMU, IIIT Hyderabad)

December 2025

Abstract

Post-training quantization of large language models introduces errors that degrade quality, but not all weights contribute equally to this degradation. PeakWeights identifies critical weights in a single forward pass without calibration data. Weight magnitude multiplied by peak activation closely tracks worst-case quantization error. Across five architectures (SmolLM2-1.7B, Qwen2.5-7B, DeepSeek-R1-7B, Mistral-7B, Phi-3-mini), protecting 50 weights during 4-bit quantization recovers 61-99% of lost perplexity. The optimal number of protected weights is architecture-dependent: four models achieve 90%+ recovery at K=50, while Mistral requires K=100. Weight importance follows a power law (alpha = 0.39-0.49), but critical weight locations vary: MLP layers for Qwen/DeepSeek, output projection for Mistral.

Key Results

Model	FP16	4-bit	+PeakWeights	Recovery
Qwen2.5-7B	10.62	11.51	10.63	99%
SmolLM2-1.7B	17.92	24.56	17.99	99%
Phi-3-mini	11.65	12.67	11.68	97%
DeepSeek-R1-7B	70.73	73.52	70.91	93%
Mistral-7B	13.44	13.80	13.58	61%

Perplexity on WikiText-103. Lower is better.

Method

Scoring Function

Each weight is scored by worst-case output perturbation:

score(w_ij) = |w_ij| × max_t |x_t,j|

where x_t,j is the activation at position j for token t. The maximum captures worst-case behavior.

Key Findings

Power Law Distribution

Weight importance follows score ∝ rank^-α with α = 0.39-0.49.

Architecture-Dependent K

Optimal K varies by 5x across models. Four achieve 90%+ at K=50.

Critical Weight Locations

MLP for SwiGLU models, lm_head for tied-embedding models.

Data-Free Works

Synthetic input (16 tokens) suffices to identify critical channels.

Citation

@article{peakweights2025,
  title={PeakWeights: Data-Free Discovery of
         Critical LLM Parameters},
  author={Maruthavanan, Thiyagarajan and
          Ambati, Vamshi},
  year={2025},
  url={https://github.com/Kalmantic/peakweights}
}

Authors

Thiyagarajan M

Kalmantic Labs

Founder/Researcher. Built PeakInfer.

thiyagarajan@kalmantic.com

Vamshi Ambati

CMU, IIIT Hyderabad

PhD from CMU. 1000+ citations. Founded PredEra.

vamshi.ambati@gmail.com