Dispatches from the edge of chaos — on nonlinear dynamics, AI, emergence, and the mathematics of complex systems.
Hyena is a sub-quadratic attention replacement that uses long convolutions and element-wise gating to achieve Transformer-quality performance with significantly reduced computational cost, particularly for long sequences.
xLSTM (Extended Long Short-Term Memory) modernizes the classic LSTM architecture with exponential gating and novel memory structures, challenging Transformers and SSMs on language modeling while retaining
RWKV (Receptance Weighted Key Value) is a novel architecture that combines the efficient parallelizable training of Transformers with the efficient O(1) inference of RNNs, achieving
State Space Models (SSMs) including S4 and Mamba offer an alternative to Transformers for sequence modeling, achieving linear-time complexity during training and constant-time per-step inference, while