CONVENTIONAL DEPTHLayer depends on layer
L1→L2→L3→L4→L5→L6
Capability is repeatedly refined through a long sequence of transformations. More depth can mean more serial work before the output is ready.
SOUP
Architecture
ExperimentalWhat if a model did not need more serial layers every time it needed to think deeper?
SOUP explores a different place to put intelligence: into an evolvingstate, anobserver memory, and a learnedfusionpath. Instead of asking depth alone to carry every stage of understanding, SOUP lets each processing block work with the present representation plus information that has already been carried, observed and remembered.
StatusExperimental
Core ideaState + Memory + Fusion
CompositionMixer + State + Memory
Research directionEdge → Large-scale AI
[ RETHINK THE STACK ]
Transformers build understanding
through a deep dependency chain.
Each conventional block receives the output of the block before it. That structure works extremely well, but increasing depth also lengthens the serial path through the network. SOUP explores whether part of that burden can move into persistent state, memory and fusion.
SOUP DIRECTIONGive each block more than the previous output
S1→S2+ STATE+ MEMORY+ FUSION
SOUP does not remove layer dependencies. It tries to reduce how much capability must comeonlyfrom stacking more dependent layers.
ARCHITECTURE THESISDepth becomes one source of intelligence — not the only source.
State carries what is changing. Memory preserves what was observed. Fusion decides what should influence the present computation.
[ CAPABILITIES ]
Carry evolving state
A SOUP block can condition computation on an internal state rather than treating every transformation as an isolated pass. The model can reason about how its representation is changing, not only what the representation is now.
CONTINUITYObserve into memory
Observer State Memory turns internal state into a learned historical signal and retrieves relevant memory for the current representation. Information does not have to be rediscovered only by going deeper.
OBSERVATIONFuse three views
SOUP Fusion learns how much to use from the current hidden representation, projected state and observed memory. The final representation can be informed by the present, the evolving internal condition and remembered context.
HIDDEN + STATE + MEMORYCompose the processing path
Mixers and FFNs can be selected layer-by-layer. ESA, Bolt and compatible custom components can coexist, turning SOUP into an architecture for exploring how different forms of sequence processing cooperate.
COMPOSITION[ WHY THIS COULD MATTER ]
A shorter dependency chain
changes what becomes possible.
If quality can be preserved with fewer serial transformations, the benefit is larger than a prettier architecture diagram. It creates a path toward lower latency, more device-friendly models and less repeated processing — provided the state, memory and fusion paths are implemented efficiently.
Phones, laptops, robots and embedded systems have tight latency, memory, power and thermal limits. SOUP is a candidate for testing whether richer stateful blocks can reduce the physical depth required for useful models.
At datacenter scale, every sequential block sits on the token-generation critical path. A future SOUP-style model that reaches target quality with fewer physical stages could reduce dependency depth and improve serving efficiency.
Persistent state and observer memory naturally fit workloads where the system must carry context across a continuous stream rather than treat every input as an isolated computation.
SOUP gives architecture search another axis beyond “more layers” or “more parameters”: how much intelligence should live in state, memory, fusion, routing and specialised processing blocks?
01Fewer required serial stages
→02Shorter dependency path
→03Potentially lower latency
→04More practical local AI
[ RESEARCH STATUS ]
Promising enough to pursue.
Early enough to keep testing.
SOUP remains an experimental architecture. Current work focuses on validating where state, Observer memory and Fusion provide useful trade-offs across model sizes, workloads and deployment targets.
The integrated implementation supports end-to-end training, configurable mixers and FFNs, built-in Observer State Memory, SOUP Fusion and recurrent generation.
Quality, latency and efficiency advantages must be evaluated under controlled settings for each target model and task.
[ HOW SOUP THINKS ]
Process the present.
Carry state. Observe memory. Fuse.
The architecture gives every stage multiple information channels rather than relying only on a single hidden stream passed through deeper and deeper blocks.
01Input representationwhat is present now
→02Sequence mixerESA / Bolt / custom
→03State-aware transformhow the representation is changing
→04Observer memorywhat should be remembered
→05SOUP Fusionwhat matters now
FUSION VIEWY = Fusion(H, S, M)
The hidden representationH, evolving stateSand observed memoryMbecome learnable contributors to the next representation.
[ COMPOSITION ]
Use recurrent state-based sequence processing while SOUP adds state-aware transformation, observation and fusion around it.
Use compact causal attention in selected layers while other layers use state-based mixers. SOUP is designed to let processing strategies coexist.
Experiment with conventional or state-aware feed-forward paths instead of forcing the same computational brick into every layer.
Change the number of physical layers, state width, memory capacity and fusion design independently to study where model capability actually comes from.
[ QUICK START ]
Build the stack around
the problem, not convention.
SOUP can use one mixer/FFN choice throughout the model or different choices layer-by-layer.
PYTHON
from mlbricks import soup
model = soup(
dim=512,
width=1116,
depth=2,
mixer=["esa", "bolt"],
ffn="saffn",
precision="fp16",
)
y = model(x)[ STATE ORIENTED UNIFIED PROCESSING ]
Think beyond
“just add more layers.”
SOUP is an experiment in where intelligence can live inside a model: not only in depth, but in state, memory, observation and learned fusion.