whereismyai
DeepEP Ascend
DeepEP Ascend is a high-performance communication library providing expert-parallel operations for machine learning on Huawei Ascend NPUs.
DeepEP-Ascend is a high-performance communication library for machine learning training and inference on Huawei Ascend NPUs. It provides expert-parallel all-to-all operations for mixture-of-experts dispatch and combine, including FP8 dispatch and deferred epilogues. The library also offers experimental communication primitives for pipeline parallelism, context and data parallelism through bucket collectives, and remote memory access. Its public buffer APIs align with the NVIDIA version of DeepEP to ease adoption. Ascend C kernels use HCCL, HCOMM, UBMEM, and URMA, and are compiled at runtime through DeepJIT for execution on supported Linux systems.
DeepEP Ascend at a glance
Key features
- Expert-parallel all-to-all operations for MoE dispatch and combine
- FP8 dispatch with row-major scales and BF16 combine
- Pipeline parallelism communication between adjacent ranks
- Bucket collectives for batched all-gather operations
- Remote memory access primitives through Engram
- Runtime kernel compilation through DeepJIT
- EPBuffer interface with cached handles and expert padding
Use it for
- Training large mixture-of-experts models on Ascend clusters
- Running inference prefill and decoding workloads
- Executing expert-parallel dispatch and combine operations
Alternatives to DeepEP Ascend
- vLLM Ascend — It enables vLLM functionality on Huawei Ascend NPU hardware.
- llm-d — Deploys production-grade LLM serving on existing hardware clusters.
- Strata — Runs large AI models locally on consumer gaming PCs without server reliance.
- AirLLM — Runs massive 70B+ parameter models on small 4GB consumer GPUs.
- MNN — It delivers high-performance inference and training with a lightweight footprint.
- Edge0 — Stream expert weights from storage on demand to bound peak memory usage.
- fal — Access 1,000+ generative models and high-performance serverless GPUs.
- LMDeploy — Accelerates inference with high-performance engines and quantization tools.
- ncnn — No runtime dependencies and supports CPU and Vulkan GPU backends.
- Strata Mlx — Runs 125B MoE models on Apple Silicon by streaming experts directly from disk.
- Kyojin — Runs 100 GB-class models in 128 GB with speculative decoding.
- Resin — Builds high-performance inference engines for complex models like DeepSeek and GLM.
All DeepEP Ascend alternatives