Every token can still look back at individual tokens.
Bolt retains causal softmax over a token-indexed history rather than collapsing the entire past into one recurrent state.
TOKEN × TOKENTOKEN-TO-TOKEN ATTENTION · COMPACT CACHE
Bolt
Attention
ResearchKeep token × token attention.
Carry 73% less context memory.
Bolt is a compact variant of causal attention for models that still want every previous token to remain individually addressable. Instead of carrying a conventional full-width K/V pair for every cached token, Bolt stores a narrower learned latent record and a scale value.
The result is a different trade-off:attention-style access to history with a much smaller persistent cache footprint.
Persistent cache73.44% smaller
Context accessToken × token
Full-sequence pathSDPA compatible
StageResearch
[ THE IDEA ]
The expensive part of attention is not only compute.
It is the history you keep carrying.
During autoregressive generation, conventional attention stores keys and values for previous tokens. That cache grows with context. Bolt keeps the same idea of querying individual historical tokens, but makes each stored record substantially smaller.
34 bytes instead of 128.
At FP16 with latent width 16 and head width 32, Bolt's compact C + ρ cache uses 73.44% less persistent storage per head/token.
73.44% LESS CACHEContext is not the only thing competing for RAM.
Saved cache capacity can become room for model weights, a longer prompt, larger batch, more concurrent sessions, or simply a smaller device.
MORE HEADROOMMove less context data through the system.
Decode is often memory-bound. A smaller persistent representation can reduce capacity and bandwidth pressure, especially as context grows.
EDGE-ORIENTED[ MEMORY / ATTENTION ]
Same history length.
A very different memory bill.
This comparison is raw persistent-cache payload, not total GPU allocation. Temporary buffers, weights, activations and framework workspaces are separate.
Store K + V for every token.
With head dimension 32, two FP16 vectors require 128 bytes per head/token.
128 BK + V · per head / token
Store C + ρ for every token.
A 16-wide FP16 latent vector plus one FP16 scale requires 34 bytes per head/token.
34 BC + ρ · 73.44% less persistent payload
THE DIFFERENCE128 B → 34 B−73.44%
Not by deleting history. By representing each historical token more compactly.
Bolt cache / layer
vs 2.00 MiB K/V1.47 MiB savedB4 · H4 example
Bolt cache / layer
vs 32.00 MiB K/V23.50 MiB savedB4 · H4 example
Bolt cache / layer
vs 128.00 MiB K/V94.00 MiB savedper layer · B4 · H4
Bolt is not constant-memory attention.Its cache still grows linearly with context because it preserves a token-indexed history. The advantage is a much smaller slope.
[ QUALITY / BEHAVIOR ]
Compress the representation.
Not the token-to-token interaction.
Bolt is aimed at the space between standard attention and recurrent-state models: keep explicit causal attention over individual history positions, but learn a narrower representation for what each position stores.
Attention behavior
- Causal token-to-token scoring
- Softmax over historical positions
- Individual cached tokens remain addressable
- Sequence history is not collapsed into one global state
The memory representation
- Learned gated latent content C
- One normalization scalar ρ per cached token/head
- Configurable latent width below head width
- Far fewer persistent bytes per token
FULL-SEQUENCE IDENTITYKBOLT= ρC · VBOLT= C
For training and full-sequence processing, the Bolt equation can be expressed directly through scaled dot-product attention. This preserves Bolt's causal softmax exactly; it is not an approximation of Bolt's own attention equation.
QUALITY OBJECTIVEAttention-class quality at lower cache cost.
Smaller cache does not automatically guarantee identical language-model quality. Latent width is a learned bottleneck and must be validated on the target model and dataset. Bolt remains aResearchcomponent while larger-scale quality parity is being established.
[ WHY EDGE ]
On constrained hardware,
memory becomes a product feature.
Phones, laptops, robots and embedded systems have tighter RAM, bandwidth, thermal and power budgets than datacenter accelerators. Bolt's value grows when context memory competes directly with the model itself.
Longer useful context
A smaller per-token cache can let the same memory budget hold more history before cache capacity becomes the limiting factor.
MORE CONTEXT / SAME RAMSmaller-device fit
Reducing context storage leaves more memory available for weights and runtime buffers, making attention-based SLMs easier to explore on consumer hardware.
MORE ROOM FOR THE MODELLower bandwidth pressure
Autoregressive decode is frequently memory-bound. Smaller cached records mean less persistent context data to read and maintain.
LESS DATA MOVEMENTPotentially lower energy pressure
Less memory traffic can improve efficiency, but actual energy savings depend on hardware and implementation and should be measured end to end.
MEASURED, NOT ASSUMEDLESS CACHE→LESS MEMORY PRESSURE→MORE DEVICE HEADROOM→MORE PRACTICAL ON-DEVICE AI
[ WHERE BOLT FITS ]
When you want attention's explicit history
without paying the full K/V bill.
Bolt is most interesting when token-level access matters, but memory is becoming the constraint that decides context length, concurrency or whether the model fits on the device at all.
Keep attention-like access to conversation history while making cache growth easier to accommodate on laptops and mobile-class hardware.
Preserve explicit sequence history for commands, observations or action tokens where edge memory and power budgets are limited.
Use more of the device's memory budget for the model itself instead of spending it on a conventional full-width K/V history.
Explore longer context or higher concurrency with a lower per-token persistent memory cost.
[ RESEARCH STATUS ]
Promising enough to build on.
Early enough to keep measuring.
Research status is intentional. Bolt already has a defined compact-attention equation and cache format; the remaining work is to prove the quality/efficiency trade-off across larger models and finalize the best decode implementation.
Compact cache representation
C + ρ storage gives the 73.44% raw persistent-cache reduction in the validated latent-16/head-32 FP16 configuration.
Exact SDPA formulation
Full-sequence Bolt can map to SDPA using K=ρC and V=C without changing Bolt's mathematical output definition.
Large-scale quality parity
Quality must be tested model by model. The goal is attention-class quality with a smaller context-memory footprint.
Final decode path
Decoder optimization is still being refined, so the public page avoids presenting experimental decode speed as a settled product claim.
Edge deployment
Mobile and embedded evaluation should measure actual latency, RAM, bandwidth, thermals and joules per token—not infer them from cache size alone.
Latent width
Width can trade memory for representational capacity. The right operating point depends on model scale, task and quality target.
[ QUICK START ]
Build with compact attention.
Choose the model width, number of heads and latent width. A smaller latent reduces the persistent cache payload, while the model still attends over individual cached positions.
PYTHON
from mlbricks import Bolt
bolt = Bolt(
d_model=384,
num_heads=6,
latent_dim=32,
position="rope",
)
y = bolt(x)[ BOLT ATTENTION ]
Keep the history.
Reduce what it costs to carry.
Bolt is for models that still want explicit causal attention but need a more memory-conscious path toward long-context and on-device intelligence.