docs: rewrite all DeepEP documentation with style improvements by kaniel-outis · Pull Request #567 · sgl-project/sgl-kernel-npu

kaniel-outis · 2026-06-24T02:36:35Z

No description provided.

- Move A2_DEEPEP_CN.md from python/deep_ep/ into python/deep_ep/doc/ (only README files remain at top level) - Merge FUSED_DEEP_MOE_EN.md + FUSED_DEEP_MOE_CN.md into single FUSED_DEEP_MOE.md with English | 中文 toggle - Add bilingual toggle header to NORMAL_API.md (English | 中文) - Add bilingual toggle header to LOW_LATENCY_API.md (English | 中文) - Update README.md and README_CN.md links to point to reorganized files - Delete FUSED_DEEP_MOE_EN.md and FUSED_DEEP_MOE_CN.md (merged into FUSED_DEEP_MOE.md) - All content meaning preserved unchanged, only structure reorganized

- NORMAL_API.md: A2 | A3 | A5 - LOW_LATENCY_API.md: A2 | A3 | A5 - FUSED_DEEP_MOE.md: A3 only - A2_DEEPEP_CN.md: A2 only

…ions - NORMAL_API: Mode-Normal / Platform-A2|A3|A5 / Quantization badges - LOW_LATENCY_API: Mode-Low-Latency / Platform / Latency badges - FUSED_DEEP_MOE: Mode-Fused / Platform-A3 only / Quantization badges + [!IMPORTANT] admonition - A2_DEEPEP_CN: Platform-A2 only badge + [!IMPORTANT] admonition

gemini-code-assist · 2026-06-24T02:36:39Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

kaniel-outis · 2026-06-24T02:44:52Z

 | `DEEP_USE_MODE=default` | default | `DefaultNormalCommStrategy` (deep_ep_cpp custom ops) | `DefaultLowLatencyCommStrategy` (deep_ep_cpp custom ops) |
 | `DEEP_USE_MODE=alltoall` | alltoall | `AlltoAllNormalCommStrategy` (torch.distributed alltoallv) | `AllToAllLowLatencyCommStrategy` (torch.distributed alltoall) |
-| `DEEP_USE_MODE=default` | ops | `DefaultNormalCommStrategy` (deep_ep_cpp custom ops) | `OpsLowLatencyCommStrategy` (torch_npu ops) |
+| `DEEP_USE_MODE=ops` | ops | `DefaultNormalCommStrategy` (deep_ep_cpp custom ops) | `OpsLowLatencyCommStrategy` (torch_npu ops) |


This needs to be confirmed again.

1329009851 · 2026-06-25T06:21:49Z


 **依赖**：`torch`, `deep_ep_cpp`

 **目的**：在 **多 NPU（Intranode）** 与 **跨节点（Internode）** 环境下，高效完成 低时延**Token Dispatch** 与 **Token Combine**（即分发?归约）操作。


"分发?归约" should be changed to "分发/归约"

1329009851 · 2026-06-25T07:18:18Z

        - A2系列双机取值范围：(0, 4096]；单机取值范围：(0, 8192]；
        - A3系列取值范围，不开蚂蚁搬家：(0, 8192]，开蚂蚁搬家：(0, 32k]；
    - hidden: 表示hidden size隐藏层大小。
        - A2系列取值范围：(0, 7168]，且保证是32的整数倍；


csrc/deepep/ops2/op_host/dispatch_normal_a2_tiling.cpp:581-583
(xDim1!= SUPPORT_HIDDEN_SIZE) requires that the hidden size must be exactly 7168, not in the range of (0, 7168] as described in the document.

1329009851 · 2026-06-25T07:38:19Z

+
+| Parameter | Type | Shape | Description |
+|-----------|------|-------|-------------|
+| **x** | `torch.Tensor` | `[bs, hidden]` | Input token representations, where each row is the hidden vector of a token (commonly `bfloat16`). **bs** range **[1, 256]**. **hidden** range **[512, 7168]**, must be divisible by **32**. |


csrc/deepep/ops/op_host/fused_deep_moe_tiling.cpp:85-93
constexpr uint32_t MIN_BATCH_SIZE = 0;
The document specifies that the value range is [1, 256]. However, the code MIN_BATCH_SIZE = 0 allows the value range to be [0, 256]. Therefore, the value range in the document should be changed to [0, 256].

1329009851 · 2026-06-25T07:57:50Z

+
+| 参数 | 类型 | 形状 | 说明 |
+|------|------|------|------|
+| **x** | `torch.Tensor` | `[bs, hidden]` | 输入 token 表示，每行一个 token 的隐藏向量（常用 `bfloat16`）。**bs** 取值范围 **[1, 256]**。**hidden** 取值范围 **[512, 7168]**，且必须能被 **32** 整除。 |


csrc/deepep/ops/op_host/fused_deep_moe_tiling.cpp:85-93
constexpr uint32_t MIN_BATCH_SIZE = 0;
The document specifies that the value range is [1, 256]. However, the code MIN_BATCH_SIZE = 0 allows the value range to be [0, 256]. Therefore, the value range in the document should be changed to [0, 256].

1329009851 · 2026-06-25T08:07:12Z

        - A3系列取值范围：[1024, 7168]；
    - num_experts：表示专家数量，取值范围：(0, 512]。
    - num_topk：表示选取topk个专家。
        - A2系列双机取值范围：[2, 16]；单机取值范围：(0, 16]；


csrc/deepep/ops2/op_host/dispatch_normal_a2_tiling.cpp:109-110
K_MAX is set to 10 (line 128), and MIN_K_VALUE_A2 is set to 2. The actual verification range in the code is [2, 10], but the value range in the document is [2, 16]. The value range in the document should be changed to [2, 10].

1329009851 · 2026-06-25T08:40:10Z


 | **参数类别** | **参数名**                           | **类型**                 | **默认值** | **详细描述**                                                 | **是否必要** | **注意事项**                                                 |
 | ------------ | ------------------------------------ | ------------------------ | ---------- | ------------------------------------------------------------ | ---------- | ------------------------------------------------------------ |
 | **核心输入** | `x`                                  | `torch.Tensor`           | -          | 输入token数据，形状`[num_tokens, hidden]`，类型`torch.bfloat16`。`num_tokens <= 512`| 是        | token数必须小于`num_max_dispatch_tokens_per_rank`, A2 算子实现要求 0 < hidden <= 7168 and hidden % 32 = 0           |


csrc/deepep/ops2/op_host/moe_distribute_dispatch_v2_tiling.cc:100-101
constexpr int64_t H_MIN = 1024;
constexpr int64_t H_MAX = 8192;
The document writes (0, 7168] and %32=0, but the code actually verifies [1024, 8192]. It should be changed to [1024, 8192].

1329009851 · 2026-06-25T09:06:57Z

        - A3系列取值范围，不开蚂蚁搬家：(0, 8192]，开蚂蚁搬家：(0, 32k]；
    - hidden: 表示hidden size隐藏层大小。
        - A2系列取值范围：(0, 7168]，且保证是32的整数倍；
        - A3系列取值范围：[1024, 7168]；


csrc/deepep/ops/op_host/cam_moe_dispatch_normal_tiling.cc:74-86
constexpr int64_t H_MIN = 1024;
constexpr int64_t H_MAX = 8192;
The document value is [1024, 7168], and the code verification value is [1024, 8192]. The correct value should be [1024, 8192].

1329009851 · 2026-06-25T09:13:53Z

+| Parameter | Type | Shape | Description |
+|-----------|------|-------|-------------|
+| **x** | `torch.Tensor` | `[bs, hidden]` | Input token representations, where each row is the hidden vector of a token (commonly `bfloat16`). **bs** range **[1, 256]**. **hidden** range **[512, 7168]**, must be divisible by **32**. |
+| **topk_idx** | `torch.Tensor` | `[bs, num_topk]` | Expert indices for each token, `int64` type. A value of `-1` indicates the token is not dispatched. |


csrc/deepep/ops/op_host/fused_deep_moe_tiling.cpp:92
SUPPORT_TOP_K = 12
Topk upper limit ≤ 12

1329009851 · 2026-06-25T09:14:41Z

+| **gmm2_weight** | `torch.Tensor` | e.g., `[G, 7168, 2048]` | Second-stage (down-projection) expert weights. |
+| **gmm2_weight_scale** | `torch.Tensor` | e.g., `[G, 7168]` | Quantization scale for second-stage weights. Same dtype rules as `gmm1_permuted_weight_scale`. |
+| **num_max_dispatch_tokens_per_rank** | `int` | Scalar | For `FUSED_DEEP_MOE`: maximum number of tokens to dispatch per rank, used for buffer/memory allocation. For `DISPATCH_FFN_COMBINE`: maximum number of tokens received in dispatch (typically `max_bs × num_ranks × topk`). All ranks must hold the same value. |
+| **num_experts** | `int` | Scalar | Total number of global experts. |


csrc/deepep/ops/op_host/fused_deep_moe_tiling.cpp:86
MAX_MOE_EXERT_NUM = 512
num_experts upper limit 512
num_experts > 0
The range of num_experts needs to be specified in detail as (0, 512]，moeExpertNum must be divisible by (epRankSize - sharedExpertRankNum).

1329009851 · 2026-06-25T12:12:55Z

+
+针对小 batch 推理优化（128 tokens/batch）：
+- **A3**：支持 `default`、`ops`、`alltoall` 策略。`ops` 策略支持 `comm_alg` 选项：`hierarchy`、`fullmesh_v1`、`fullmesh_v2`、`ccu`。
+- **A5**：支持 `default` 和 `ops` 策略，支持 MXFP8 per-block 量化（`use_ue8m0=True`，quant_mode=3）。数据格式：`float8_e4m3fn` + `float8_e8m0fnu` 缩放因子。


The conflict between MXFP8 quant_mode=3 and the ops policy verification [0,2] — An error is reported for the MXFP8 of A5.
csrc/deepep/ops2/op_host/moe_distribute_dispatch_v2_tiling.cc:91-93
MXFP8_SCALES=3 is not defined.
moe_distribute_dispatch_v2_tiling.cc:257-261:
The verification range is [0, 2]. If quant_mode is set to 3, the error message "quantMode is invalid, only support [0, 2]" is displayed.

MXFP8 quant_mode=3 与 ops 策略校验[0,2]冲突 — A5 的 MXFP8 会报错
csrc/deepep/ops2/op_host/moe_distribute_dispatch_v2_tiling.cc:91-93
没有定义 MXFP8_SCALES=3。
moe_distribute_dispatch_v2_tiling.cc:257-261：
校验范围 [0, 2]，quant_mode=3 会触发报错 "quantMode is invalid, only support [0, 2]"。

1329009851 · 2026-06-25T12:28:10Z

+高吞吐的 dispatch 和 combine，适用于训练和 Prefill 阶段：
+- **A3**：纯 HCCS 节点内通信，全互联 HCCS 节点间通信。无需分层实现。
+- **A2 单机**：纯 HCCS 通信，normal dispatch/combine 最大支持 `bs=8000`。
+- **A2 双机**：分层（节点内 HCCS + 节点间 RDMA）或不分层（纯 RDMA）实现。最大支持 `bs=4096`。


The normal dispatch fp8_e5m2 / fp4_e2m1 is not mentioned in the document. Check whether the normal dispatch quantization is supported, add the response description here, and update the related content in NORMAL_API.md.
Normal dispatch fp8_e5m2 / fp4_e2m1 支持未在文档提及，请确认 Normal dispatch 的量化支持，补充响应的说明至此，并更新 NORMAL_API.md 的相关内容

1329009851 · 2026-06-25T12:41:12Z

+| **gmm2_weight** | `torch.Tensor` | 例如 `[G, 7168, 2048]` | 第二阶段（下投）专家权重。 |
+| **gmm2_weight_scale** | `torch.Tensor` | 例如 `[G, 7168]` | 第二阶段权重量化 scale。数据类型规则同 `gmm1_permuted_weight_scale`。 |
+| **num_max_dispatch_tokens_per_rank** | `int` | 标量 | `FUSED_DEEP_MOE` 模式：每个 rank 最多分发的 token 数，用于 buffer/内存分配。`DISPATCH_FFN_COMBINE` 模式：dispatch 中最多接收的 token 数（通常为 `max_bs × num_ranks × topk`）。所有 rank 必须持有相同值。 |
+| **num_experts** | `int` | 标量 | 全局专家总数。 |


Same as above, it is necessary to specify in detail that the total number of experts ranges from (0, 512]. moeExpertNum must be divisible by (epRankSize - sharedExpertRankNum).
同上，需要详细说明专家总数范围是(0,512]，添加moeExpertNum must be divisible by (epRankSize - sharedExpertRankNum)的整除说明

Kaniel_Zhou added 5 commits June 23, 2026 19:46

docs: add platform scope header to each doc file

fb8e314

- NORMAL_API.md: A2 | A3 | A5 - LOW_LATENCY_API.md: A2 | A3 | A5 - FUSED_DEEP_MOE.md: A3 only - A2_DEEPEP_CN.md: A2 only

update README.md for origin

72f3366

update README.md

1fc0608

kaniel-outis commented Jun 24, 2026

View reviewed changes

1329009851 reviewed Jun 25, 2026

View reviewed changes

1329009851 reviewed Jun 26, 2026

View reviewed changes


		依赖：`torch`, `deep_ep_cpp`

		目的：在多 NPU（Intranode）与跨节点（Internode）环境下，高效完成低时延Token Dispatch 与 Token Combine（即分发?归约）操作。

Uh oh!

Conversation

kaniel-outis commented Jun 24, 2026

Uh oh!

gemini-code-assist Bot commented Jun 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

1329009851 Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

1329009851 Jun 25, 2026 •

edited

Loading