Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention

Claims

• Gemma 4's cross-layer KV sharing reuses key-value projections from earlier layers across later layers, reducing KV cache memory by approximately half — saving 2.7 GB for the E2B model at 128K context.
• Gemma 4's per-layer embeddings allow small models to increase representational capacity through cheap lookup-style embedding parameters without scaling the expensive transformer blocks themselves.
• Laguna XS.2 uses per-layer query-head budgeting, giving sliding-window layers 8 query heads per KV head and full-attention layers only 6, spending attention capacity where it is most useful.
• ZAYA1-8B's Compressed Convolutional Attention performs attention directly in a compressed latent space and adds convolutional mixing on compressed Q and K tensors, reducing both KV cache size and attention FLOPs during prefill and training.
• DeepSeek V4's CSA/HCA compressed sequence attention achieves 27% of single-token inference FLOPs and 10% of KV cache size compared to DeepSeek V3.2 at 1M-token context length.
• DeepSeek V4's manifold-constrained hyper-connections widen the transformer residual stream across four parallel streams with doubly-stochastic constraints for stability, adding only 6.7% training time overhead.