Release 5.17.0
# Release v5.17.0 ## New Model additions ### HYV4 Hy4-Preview is a 780B-parameter mixture-of-experts language model that activates 49B parameters per token. Each MoE layer holds 256 routed experts plus one always-active shared expert and routes every token to 8 of them. The context window is 1M tokens. The architecture combines four features: - **Multi-head Latent Attention (MLA)** compresses keys and values into a low-rank latent (`kv_lora_rank`) that `kv_b_proj` expands back to one key/value per query head. - **DeepSeek Sparse Attention (DSA)** selects `index_topk` keys per query with a lightweight indexer. Following [IndexShare](https://hu
来自 Transformers 发布记录 的官方公开更新；本站仅提供短摘要与原始出处链接。
Release 5.17.0
# Release v5.17.0 ## New Model additions ### HYV4 Hy4-Preview is a 780B-parameter mixture-of-experts language model that activates 49B parameters per token. Each MoE layer holds 256 routed experts plus one always-active shared expert and routes every token to 8 of them. The context window is 1M tokens. The architecture combines four features: - **Multi-head Latent Attention (MLA)** compresses keys and values into a low-rank latent (`kv_lora_rank`) that `kv_b_proj` expands back to one key/value per query head. - **DeepSeek Sparse Attention (DSA)** selects `index_topk` keys per query with a lightweight indexer. Following [IndexShare](https://hu
来自 Transformers 发布记录 的官方公开更新；本站仅提供短摘要与原始出处链接。
自动收录自 Transformers 发布记录 的公开订阅信息；请以原始出处为准。
前往原始出处阅读