Long-context throughput
Associative state composition creates a parallel path for full-sequence processing without materializing a dense token-to-token attention score matrix inside the ESA mixer.
SEQUENCE SCALEMEMORY-EFFICIENT SEQUENCE MIXER
Entangled State
Attention
ActiveMore context. Less memory pressure. Compact generation.
ESA is a state-based sequence architecture built for efficient long-context processing and recurrent decode.
ESA changes the part of sequence models that becomes expensive as history grows. It replaces explicit token-to-token score mixing with an associative state recurrence, then generates from compact recurrent state instead of a key/value cache that grows with every cached token.
64K training29.57M tok/s
64K reported peak1,325.3 MB
Decode average6,580.16 tok/s
Decode state0.000488 MB
[ WHY ESA ]
Efficiency that becomes more important as context grows.
ESA changes both the full-sequence computation pattern and the persistent memory pattern used during autoregressive generation.
Lower GPU RAM pressure
In the selected H100 long-context benchmark, the measured peak-memory gap widened strongly as context increased, reaching a 97.2% reduction at 64K versus the displayed FA4 baseline.
MEASURED / H100Constant-state decode
The persistent recurrent state has a fixed shape for a given model and batch. It does not acquire a new K/V entry for every prompt or generated token.
GENERATIONEdge-oriented design
Compact persistent state reduces cache-growth pressure during generation, making ESA a practical research direction for offline assistants, mobile inference, robotics, and continuous streams.
MOBILE / EDGE[ ARCHITECTURE ]
Parallel over the sequence.
Recurrent at generation time.
The same state recurrence can be composed across a sequence for training and prefill, then carried forward one token at a time for generation.
01Input representation[B,T,C]
→02State coefficientslearned update + write terms
→03Associative compositionparallel sequence processing
→04Compact final stateper-layer recurrent summary
→05One-token updatereuse state for next output
CORE RECURRENCEEt= At⊙ Et−1+ Bt
The state summarizes prior sequence information. Associative composition makes the recurrence parallelizable for full-sequence work while retaining a compact recurrent form for decoding.
History remains explicit.
Keys and values are retained for cached tokens. A conventional dynamic KV cache therefore grows with sequence length and is read again as new tokens are produced.
Persistent cache grows with cached context.
History moves through state.
Each layer carries a compact state shaped by batch, heads and head dimension. Prompt length is not a dimension of that persistent state.
Persistent state shape stays fixed as prompt length grows.
Inference/decode distinction: fixed persistent state doesnotmean every training allocation is constant. Training still uses activations, parameters, optimizer state and implementation-specific temporary memory.
[ H100 / LONG CONTEXT ]
The measured advantage widened with context.
Selected internal research measurements. Training uses batch 64, FP16 and three-seed averages. Values below compare the optimized ESA profile used in the supplied benchmark with the displayed FA4 baseline.
tokens/s
vs 2.67M tok/s · 11.06×
ESA · C16 Compile Reduce OH
vs 48,166.6 MB · 97.2% lower
ESA · C16 Compile Reduce OH
vs 11.95 PPL · FA4 Native
tokens/s across 512, 1K, 4K and 8K prompts
vs 1,098.38 tok/s · 5.99× average ratio
training throughput
16.22M vs 4.38M tok/s182.4 MBvs 978.1 MB peak · 81.4% lower
training throughput
28.62M vs 7.06M tok/s454.5 MBvs 12,213.4 MB peak · 96.3% lower
training throughput
29.57M vs 2.67M tok/s1,325.3 MBvs 48,166.6 MB peak · 97.2% lower
These are configuration-specific benchmark results, not universal guarantees. Hardware, model size, precision, batch, compiler and implementation materially affect both throughput and memory.Full methodology ↗
[ GENERATION / MEMORY ]
Prompt length grows.
Persistent ESA state does not.
In the displayed H100 decoder benchmark, ESA recurrent state remained 0.000488 MB at 512, 1K, 4K and 8K prompt lengths while the native Attention KV memory increased with prompt length.
6,658.09ESA tok/s
Attention: 1,103.19 tok/s · 6.04×
0.000488 MBESA state0.750000 MB attention KV
6,574.91ESA tok/s
Attention: 1,118.56 tok/s · 5.88×
0.000488 MBESA state1.250000 MB attention KV
6,518.19ESA tok/s
Attention: 1,109.38 tok/s · 5.88×
0.000488 MBESA state4.250000 MB attention KV
6,569.44ESA tok/s
Attention: 1,062.37 tok/s · 6.18×
0.000488 MBESA state8.250000 MB attention KV
AVERAGE DECODE5.99×
6,580.16 ESA tok/s vs 1,098.38 Attention KV tok/s across the four displayed prompt lengths.
STATE SHAPE[B,H,D]
Prompt length is not a dimension of the per-layer recurrent state.
PRACTICAL EFFECTPredictable
Persistent generation storage is easier to budget as conversations get longer.
The recurrent state still scales with model width, number of layers, batch size and precision. ESA trades explicit token-by-token K/V history for a learned compressed state summary.
[ WHERE ESA FITS ]
From H100s to constrained devices.
The same compact-state idea is useful anywhere memory growth, long streams or limited device RAM make a full token history expensive.
MOBILE VULKAN SNAPSHOT29.94
tokens/s · ESA 512×16 research test
A working mobile/Vulkan prototype shows that ESA's recurrent decode can be mapped beyond datacenter GPUs. The result is a feasibility signal—not a claim that the mobile path is production complete.
Sequence workloads where attention memory and token-pair computation become increasingly expensive as context expands.
Fixed-shape persistent state can reduce generation-cache pressure when memory is limited and conversations continue for many turns.
Continuous language, sensor or action streams can be summarized into recurrent state instead of retaining every prior token representation.
Reducing per-request context storage can leave more device memory available for model weights, larger batches or additional concurrent work.
[ PROS / TRADE-OFFS ]
What ESA changes—and what it does not.
A serious architecture page should make both the advantages and the constraints visible.
No growing persistent KV history
Generation carries learned recurrent state rather than appending key/value tensors for every cached token.
Strong long-context benchmark scaling
The supplied H100 measurements show larger throughput and peak-memory advantages at 16K and 64K than at 1K.
One architecture, two execution modes
Associative composition serves full-sequence work; the same recurrence naturally becomes a one-token update for generation.
History is compressed
Unlike attention with explicit cached K/V, ESA does not preserve direct token-by-token representations. Quality depends on what the learned state retains.
Training memory is implementation dependent
Fixed decode state does not make all training memory constant. Activations, optimizer state, temporary buffers and compilation strategy still matter.
Performance is workload dependent
Hardware, context, batch, precision and model scale can change the winner. Benchmark the real workload rather than assuming a universal speedup.
[ QUICK START ]
Use ESA through the MLBricks API.
The public surface stays simple: create an ESA layer, run full sequences, then use prefill and one-token recurrent updates when generating.
PYTHON
from mlbricks import ESA
esa = ESA(embd=512, head=8)
# Full sequence
output = esa(x)
# Recurrent generation
prefill_out, state = esa.prefill(prompt)
next_out, state = esa.decode_step(next_token, state)[ ENTANGLED STATE ATTENTION ]
Long context without carrying the whole past as a growing cache.
Study the recurrence, reproduce the benchmarks, test the fixed-state decode behavior, and evaluate ESA on the hardware and workloads that matter to you.