Figure 1 · Kimi K3.1 Pro sparse attention (DSA)
Every query head reads only the tokens the indexer picks
Toy example: 3 index heads, 4 query heads, 24 past tokens, top-8. K3.1 Pro: 32 index heads, 96 query heads, top-2,048. Not to scale.
Sizes: Kimi K3.1 Pro config.json (dsa_config: index_num_heads = 32, index_head_dim = 128, index_topk = 2048; kv_lora_rank = 512, qk_rope_head_dim = 64).
indexer (FP8 keys)
selected tokens (top-k)
MLA latent cache
not read
query heads, outputs