Store only the precision a matrix actually needs.
Conventional quantization usually starts by choosing a target width — 8-bit, 4-bit, 3-bit — and then forcing weights into that budget. ElasticBit reverses the decision. It measures the output error of a matrix on representative inputs, finds thesmallest storage widththat satisfies the requested error threshold, packs the weights at that width, and uses the next supported compute precision when the matrix is executed.
[ THE STORAGE PROBLEM ]
Large models are expensive not only because they perform many operations, but because their weights must be downloaded, stored, moved into memory and repeatedly read. Quantization helps — but a single fixed bit-width gives every matrix the same precision budget even when their sensitivity is different.
A predetermined 4-bit or 8-bit target is applied broadly. Modern methods can preserve quality remarkably well, but lower precision is still a lossy transformation and sensitivity is not uniform across a model.
ONE BUDGET → MANY MATRICESThe real-model exporter analyzes eachnn.Linearmatrix independently. A matrix gets the smallest supported storage width that stays inside the calibration-error threshold you set.
It changes how weights are represented. The current implementation chooses precision per matrix, not a different width for every individual scalar parameter. Quality preservation is threshold-controlled and must still be validated on the target model and dataset.
[ HOW IT WORKS ]
The core idea is to separate the precision needed forstoragefrom the precision that the runtime uses forarithmetic.
Run representative input vectors through the original matrix so the analyzer has a functional reference.
Measure candidate widths against the requested error threshold instead of assuming 4, 8 or 16 bits in advance.
Keep the smallest width whose measured calibration error satisfies the target.
Intermediate widths use exact bit-packed integer payloads with row-wise symmetric scales.
At execution, storage is widened into the next supported compute bucket rather than forcing hardware to calculate at an awkward width.
[ STORAGE ≠ COMPUTE ]
ElasticBit can store a matrix at an intermediate width while executing it through a precision class the hardware already understands. That decoupling is the practical bridge between compact representation and available compute.
Use the native 4-bit compute bucket.
Keep the packed width, widen for INT8 execution.
For example, 9-bit storage can execute through FP16.
Higher-precision fallback when the error target requires it.
[ VALIDATED RUNTIME SNAPSHOT / TESLA T4 ]
This is a runtime validation snapshot, not a universal model-compression claim. The test used a 1024×512 matrix, a 0.01 calibration threshold and 100 benchmark iterations on swapped Tesla T4 GPUs.
smallest width selected bybitsAnaliser
GPU/file weight payload
vs 1,048,576 B native FP16less than the FP16 weight baseline
for this matrix and selected widthruntime speed in this validation
fast layout widened to the FP16 compute bucketRuntimeMatrix(..., "compact")retains the exact packed representation on GPU, maximizing memory reduction while the runtime handles conversion for execution.
RuntimeMatrix(..., "fast")widens to the selected compute bucket ahead of execution. In the T4 snapshot it was roughly 25–27% faster than compact and essentially matched the FP16 baseline speed.
Evidence scope: synthetic runtime matrix validation from the final MLBricks dual-T4 merge gate. Storage savings, selected width and latency vary by matrix distribution, calibration inputs, threshold, GPU and model. Full-model quality must be measured after export.
[ QUANTIZATION RESEARCH CONTEXT ]
ElasticBit sits in a broader research direction where quantization decisions are informed by sensitivity or hardware constraints rather than blindly applying one precision everywhere. Its implementation makes a different engineering choice: search every integer storage width from 4 through 32, then decouple packed storage from the runtime compute bucket.
GPTQ showed that large language models can be compressed to low fixed widths such as 3–4 bits with very small accuracy loss using approximate second-order information.
Read GPTQ ↗AWQ observes that weights are not equally important and uses activation statistics to protect salient channels while keeping a hardware-friendly low-bit weight format.
Read AWQ ↗HAWQ-V3 treats bit precision as an optimization variable, balancing model perturbation with constraints such as memory footprint and latency.
Read HAWQ-V3 ↗ElasticBit tests 4–32-bit storage widths on representative inputs, selects the smallest width meeting the error target, stores that matrix compactly, then executes using the next available INT4 / INT8 / FP16 / FP32 bucket.
Explore ElasticBit ↗These projects are related by the general goal of preserving useful model behavior while reducing numerical cost. ElasticBit should not be described as the first adaptive or mixed-precision quantizer; its differentiator is the specific 4–32-bit storage-selection and storage/compute-decoupling runtime implemented by MLBricks.
[ CAPABILITIES ]
bitsAnaliserevaluates candidate widths and returns the selected width, selected error, compute type and a report for every tested precision.
The real-model exporter captures representative inputs for everynn.Linearmatrix and analyzes them independently instead of requiring one model-wide bit-width.
Keep packed weights for memory efficiency or widen them to the selected compute bucket when latency is the stronger requirement.
MEMORY ↔ SPEEDRuntime matrices can be saved to the ElasticBit matrix format, loaded in compact or fast mode, dequantized for inspection, and benchmarked against native FP16.
DEPLOYMENT WORKFLOW[ WHY IT MATTERS ]
The objective is not “make everything 4-bit.” It is to stop paying 16 or 32 bits everywhere when a specific matrix can meet the desired error envelope with less.
Smaller weight files can reduce artifact size, download cost and local disk pressure for models that ship to many devices.
Compact mode can keep packed representations resident, leaving more device memory for larger models, context, batches or other runtime state.
A matrix can retain more than 4 or 8 bits when calibration shows that aggressive quantization would exceed the allowed error target.
Intermediate storage widths do not require matching arithmetic units: values are promoted into the next supported compute precision at runtime.
[ OPTIONAL NATIVE CUDA API ]
The optional CUDA runtime exposes 4–32-bit analysis plus compact and fast runtime matrices. Use the portable API above when the native extension is not installed.
from mlbricks import ElasticBit
analysis = ElasticBit.bitsAnaliser(
weights,
calibration,
threshold=0.01,
min_bits=4,
max_bits=32,
)
bits = analysis["selected_bits"]
matrix = ElasticBit.RuntimeMatrix(
weights,
bits,
"compact",
)
y = matrix.forward(x)[ RESEARCH STATUS ]
ElasticBit is experimental because bit selection is only one part of the deployment problem. Calibration coverage, end-to-end quality, kernel cost, hardware support and model-specific sensitivity all matter.
The analyzer sweeps candidate widths and reports measured calibration error, payload size and compute type.
Compact and fast runtime modes, save/load, dequantization, reference execution and native FP16 comparison are available.
Each PyTorch linear matrix can be calibrated independently and reconstructed for loss/perplexity evaluation.
An error threshold is not an unconditional quality guarantee. The exported model should be evaluated on the actual validation tasks that matter.
The native 4–32-bit runtime currently targets CUDA builds withnvcc. MLBricks retains PyTorch-compatible ElasticLinear fallbacks separately.
Future work can explore broader model classes, calibration strategies, device architectures and end-to-end bandwidth/energy effects.
[ INSTALLATION · MLBRICKS KIT 1.0.0B1 ]
mlbricks.This component ships inside the unifiedmlbricks-kitdistribution. Python imports continue to use themlbricksnamespace.
pip install mlbricks-kit==1.0.0b1[ API · MLBRICKS KIT 1.0.0B1 ]
The public package surface supports PyTorch-compatible tensor and module quantization in every install. The 4–32-bit CUDA runtime remains an optional native capability.
from mlbricks import ElasticBit
elastic = ElasticBit(
bits=4,
group_size=128,
backend="auto",
)
packed = elastic.quantize(weights)
restored = elastic.dequantize(packed)
qlinear = elastic.linear(torch_linear)quantize(tensor) / dequantize(...)Pack a tensor and reconstruct it for inspection or fallback execution.
linear(layer) / embedding(layer)Wrap PyTorch Linear or Embedding modules with packed ElasticBit storage.
quantize_module(...)Convert supported modules across a larger PyTorch model.
set_backend(...) / resolved_backend()Use the standard MLBricks backend surface.
ElasticBit.bitsAnaliser(...)Optional native CUDA 4–32-bit analysis API when the native runtime is packaged.
ElasticBit.RuntimeMatrix(...)Optional native compact/fast runtime matrix.
bitsAnaliserandRuntimeMatrixdo.[ ELASTICBIT ]
ElasticBit turns precision into a storage decision informed by observed model behavior — compact when it can be, higher precision when it needs to be, and mapped onto real compute formats when execution begins.