HERA: High-efficiency Matrix Compression via Element Replacement

Matrix quantization involves encoding matrix elements in a more space-efficient manner to minimize storage requirements, with dequantization used to reconstruct the original matrix for practical use. We define the Quantization Error Minimization (QEM) problem as minimizing the difference between a matrix before and after quantization while ensuring that the quantized matrix occupies the same amount of memory. Matrix quantization is essential in various fields, including weight quantization in Large Language Models (LLMs), vector databases, KV cache quantization, graph compression, and image compression. The growing scale of LLMs, such as GPT-4 and BERT, underscores the need for matrix compression due to the large size of parameters and KV caches, which are stored as matrices. To address the QEM problem, we introduce HETA, an algorithm that leverages the local orderliness of matrix elements by iteratively swapping elements to create a locally ordered matrix. This matrix is then grouped and quantized by columns. To further improve HETA, we present two optimizations: additional quantization of residuals to reduce mean squared error (MSE) and the application of masking and batch processing to accelerate the algorithm. Our experiments show that HETA effectively reduces MSE to 12.3% of its original value at the same compression ratio, outperforming leading baseline algorithms. Our contributions include formalizing the QEM problem, developing the HETA algorithm, and proposing two optimizations to enhance both accuracy and processing speed.