SVDQuant supports NVFP4 on NVIDIA Blackwell GPUs with 3× speedup over BF16 and better image quality than INT4. Try our interactive demo below or at https://svdquant.mit.edu/! Our code is all available at https://github.com/mit-han-lab/nunchaku.
With Moore's law slowing down, hardware vendors are shifting toward low-precision inference. NVIDIA's latest Blackwell architecture introduces a new 4-bit floating point format (NVFP4), improving upon the previous MXFP4 format. NVFP4 features more precise scaling factors and a smaller microscaling group size (16 v.s. 32), enabling it to maintain 16-bit model accuracy even at 4-bit precision while delivering 4× higher peak performance.
In our previous blog, we shared a tutorial on setting up a 5090 workspace with the Blackwell architecture. In this blog, we’re excited to announce that SVDQuant now supports NVFP4 on the 5090 GPU, delivering better image quality and performance! Our code and demo are all publicly available!
SVDQuant is a new 4-bit quantization paradigm. Compared to other quantization methods which redistribute the outliers between weights and activations, Unlike traditional methods that redistribute outliers between weights and activations, it employs a lightweight high-precision low-rank branch to absorb them.
The table above compares image quality across various datatypes on four popular text-to-image diffusion models using the MJHQ prompt set. Image Reward assesses overall image quality, while LPIPS and PSNR measure perceptual and numerical similarity between images generated by quantized and original models.
Across all models, NVFP4 outperforms INT4, particularly in similarity metrics, thanks to the native hardware support of smaller microscaling group size on Blackwell. Additionally, SVDQuant consistently improves upon naive quantization, as its low-rank branch effectively absorbs outliers. Notably, combining SVDQuant with NVFP4 delivers the best results, achieving a PSNR of 21.5 on FLUX.1-dev, closely matching the image quality of the original 16-bit model.
Below, we present qualitative comparisons between different precision settings:
Both INT4 and NVFP4 SVDQuant compress the 12B FLUX.1 model by 3.6×, reducing memory usage by 3.5×. On both RTX 4090 and 5090, SVDQuant achieves 3× speedups over the original BF16 model, even while retaining 16-bit attention.
Notably, the NVFP4 model on the 5090 runs 1.3× faster than the INT4 model on the 4090, aligning with the theoretical 1.3× performance boost from the RTX 4090 to 5090, as documented in NVIDIA’s specifications. These results are just initial benchmarks, and we are actively working on further kernel optimizations.
Furthermore, both our INT4 and NVFP4 kernels are fully open-source. We welcome you to explore and contribute!
In this blog, we introduced SVDQuant’s support for NVFP4 on NVIDIA 50-series GPUs. In the future, we will continue optimizing our kernels and extending support to more models (e.g., video models) to further benefit the community.
