SLAI T-Rex: Post-Training DeepSeek-V4 on Ascend NPUs

1. The Big Picture

DeepSeek-V4 is a Mixture-of-Experts model with 1.6 trillion total parameters and 371 billion activated parameters per token. It was originally trained on NVIDIA H800 GPUs — the gold standard for large-scale AI training. But what happens when you try to post-train a model of this scale entirely on a different hardware ecosystem, one built around Huawei’s Ascend NPUs?

The answer, it turns out, is SLAI T-Rex: the first public recipe demonstrating viable full-parameter post-training of a trillion-parameter MoE model on Ascend 910C NPUs, running on the CloudMatrix384 SuperPOD architecture. The results speak for themselves: a 34.22% Model FLOPS Utilization (MFU) — a 2.93x improvement over the baseline Ascend configuration — and a 71.81% zero-shot Pass@1 average across mathematical optimization benchmarks.

The paper, authored by 64 researchers from the SLAI (Shanghai AI Laboratory) and partner institutions, earned #1 Paper of the Day on Hugging Face on July 23, 2026. This article breaks down what they did, how they did it, and why it matters.

2. What Is Post-Training (And Why Is It Hard on NPUs)?

Pre-training vs. Post-training

Pre-training is the initial, compute-intensive phase where a model learns from internet-scale data. DeepSeek-V3, the predecessor, consumed roughly 2.788 million H800 GPU-hours for pre-training alone. Post-training, by contrast, covers everything after pre-training: continued pre-training (domain adaptation), supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), and alignment. It is typically smaller in scale but far more sensitive to precision, stability, and memory footprint.

The NPU Challenge

Post-training a 1.6T MoE model on NPUs presents three core difficulties:

• Memory pressure: The 1.6T model requires aggressive sharding across devices, and Ascend NPUs have different memory hierarchies than GPUs.
• Non-overlapped communication: The Ascend Collective Communication Library (HCCL) does not naturally overlap with computation, creating idle GPU cycles during all-reduce and all-to-all operations.
• Inefficient kernel execution: Many operators critical to DeepSeek’s architecture — SparseAttn, SwiGLU, RoPE — lack optimized AscendC kernels out of the box.

The Ascend Ecosystem

The CloudMatrix384 SuperPOD is the hardware foundation. Each unit packs 384 Ascend 910C NPUs, 192 Kunpeng CPUs, and a Unified Bus interconnect that ties them together in a high-bandwidth mesh. The 910C NPU itself is a dual-die processor with 64 GB HBM and 800 TFLOPS FP16 throughput — competitive on paper, but the software ecosystem (CANN, MindSpeed) has historically lagged behind CUDA in maturity.

3. Three Layers of Optimization

T-Rex’s optimization strategy operates on three concentric layers: parallelism strategy, compute-communication orchestration, and kernel-level tuning. Together they drive MFU from a baseline of 11.67% to 34.22% — a 2.93x improvement.

Parallelism (TP + PP + EP + CP)

Handling a 1.6T MoE model requires combining four parallelism dimensions:

• Tensor Parallelism (TP): Splits individual operations across devices.
• Pipeline Parallelism (PP): Distributes layers across stages.
• Expert Parallelism (EP): Distributes MoE experts across devices so each device handles a subset.
• Context Parallelism (CP): Splits the sequence dimension — critical for long-context post-training.

The combination allows the 1.6T parameter footprint to fit within aggregate NPU memory while keeping per-device compute balanced.

Compute-Communication Orchestration

On GPUs, NCCL overlaps communication with computation almost transparently. On Ascend, HCCL requires explicit orchestration. T-Rex implements a coordinated scheduling layer that carefully overlaps all-reduce, all-to-all (for MoE routing), and reduce-scatter operations with forward and backward compute. CPU cores handle gradient aggregation and schedule communication windows so NPU cycles are never wasted waiting for data.

AuraKernel: Profiling-Driven Kernel Optimization

AuraKernel is T-Rex’s custom AscendC optimization workflow. The team profiled the full training graph and identified the most expensive operators, then hand-tuned AscendC kernels for each bottleneck.

The top compute-time consumer was SparseAttnSharedkvGrad at 16.86% of total compute time. Other critical targets included RMS norm, lightning indexer gradients, RoPE embeddings, SwiGLU activations, and multi-head concatenation (mHC) chains. Each received custom kernel optimizations that reduced wall-clock time significantly.

Metric	Baseline	T-Rex Optimized	Improvement
MFU	11.67%	34.22%	2.93x

4. Making DeepSeek-V4-Flash an OR Expert

T-Rex targets the DeepSeek-V4-Flash variant: 284 billion activated parameters with 16 MoE experts. The goal is to transform this general-purpose model into an Operations Research (OR) specialist — capable of solving mathematical optimization problems zero-shot.

Two-Stage Pipeline

The post-training pipeline has two phases:

• Continued Pre-Training (CPT): Domain-adaptive pre-training at 4K context length for 280 iterations. This re-exposes the model to optimization-domain text — problem statements, mathematical formulations, solver outputs — without losing its general capabilities.
• Supervised Fine-Tuning (SFT): Instruction tuning at 8K context for 250 iterations. This teaches the model to produce correct, well-structured optimization solutions in response to natural-language queries.

Key training configuration: learning rate 3e-6 (CPT) / 5e-6 (SFT), global batch size (GBS) of 128, micro batch size (MBS) of 1.

The Data Flywheel

High-quality OR training data is scarce. T-Rex builds a data flywheel: problem-answer seeds are expanded by a solver-verified synthetic document generator, then self-distilled into SFT pairs. A Clean-CoT (Chain-of-Thought) quality gate filters out hallucinated solutions, and contract-aware filtering removes malformed or ambiguous entries. The final dataset consists of 10,000 high-quality SFT samples across 4 OR task categories and 3 problem representations (DP, DT, DPS).

CPT-to-SFT Transfer Gain

The value of the two-stage approach is clear from the B4O-Feasible benchmark: the base model scores 60.47%. SFT alone lifts it to 65.93%. But CPT followed by SFT reaches 71.22% — a gain of +5.29 points over SFT-only. The domain-adaptive pre-training step is clearly not optional.

5. Benchmark Results

The paper evaluates zero-shot Pass@1 performance on mathematical optimization benchmarks. Here is how the models compare:

Model	NL4OPT	OptiBench	B4O-Feasible	B4O-ORGEval	Average
DeepSeek-V4-Flash (base)	—	—	60.47%	34.26%	60.54%
DeepSeek-V4-Flash (SFT-only)	—	—	65.93%	48.73%	—
DeepSeek-V4-Flash (CPT+SFT)	—	—	71.22%	59.39%	—
GPT-5.4-Mini	—	—	—	—	67.83%
SLAI T-Rex (DeepSeek-V4-Flash-OR)	—	—	—	—	71.81%

SLAI T-Rex achieves a 71.81% average across all OR benchmarks, outperforming GPT-5.4-Mini by +3.98 points and the base DeepSeek-V4-Flash by +11.27 points.

A critical caveat: these are Operations Research benchmarks — mathematical optimization problems (linear programming, constraint satisfaction, scheduling). They do not measure general reasoning, commonsense knowledge, or language understanding. A high score here does not imply superiority on MMLU, MATH, or GPQA.

6. What’s in the Open Source Repo?

The project is available on GitHub at SLAI-AITP/SLAI-T-Rex under the MIT license. The repository is organized into several modules:

• cpt_data_construction — Design notes and methodology for domain-adaptive pre-training data.
• cpt_training — Training scripts for the CPT phase.
• sft_data_construction — A runnable toolkit for generating and filtering OR instruction data.
• sft_training — SFT training scripts and configurations.
• model_download_deployment — Scripts for model export and inference deployment.
• docs and examples — Usage documentation and example workflows.

Quick Start

Getting started with the data construction toolkit is straightforward:

git clone https://github.com/SLAI-AITP/SLAI-T-Rex.git
cd SLAI-T-Rex/sft_data_construction
pip install -r requirements.txt
python run_pipeline.py --mode dry_run    # preview without generation
python run_pipeline.py --mode full       # full data generation

What is not included: the MindSpeed-LLM and CANN runtime environment (proprietary Huawei SDKs), private model checkpoints, and specific cluster configuration files. You will need access to an Ascend NPU cluster to run the training scripts.

7. Critical Analysis

No paper is beyond scrutiny. Here are the areas where T-Rex’s claims warrant careful interpretation:

• Self-reported MFU: The 34.22% MFU is reported by the authors without third-party verification. While the methodology appears sound, independent reproduction would strengthen confidence.
• Context from prior work: Pangu Ultra (718B params) achieved 30.0% MFU on 6,000 Ascend NPUs. TorchTitan with DeepSeek-V4 reached 28.78% on 64 NPUs. T-Rex’s 34.22% is the highest publicly reported MFU for any Ascend-based training recipe — but direct comparisons are difficult due to different model sizes and cluster scales.
• Narrow benchmark scope: The OR benchmarks are domain-specific. The paper does not evaluate on standard general-purpose benchmarks (MMLU, MATH, GPQA, HumanEval), so we cannot assess whether the model retained its broad capabilities after domain-adaptive training.
• GPT-5.4-Mini comparison: GPT-5.4-Mini is not a widely documented model, making the comparison difficult to contextualize. The paper provides limited details on GPT-5.4-Mini’s architecture or training setup.
• Missing operational details: The paper does not disclose total NPU-hours consumed, wall-clock training time, or the exact number of NPUs used for the post-training run. These details are essential for practitioners evaluating cost and feasibility.

8. Why This Matters

SLAI T-Rex is significant for three reasons:

1. First viable trillion-parameter post-training recipe on Ascend. While there have been Ascend inference demos and smaller-scale training experiments, T-Rex is the first public demonstration of full-parameter post-training at the 1.6T scale — and it works well enough to beat GPT-5.4-Mini on the chosen benchmarks.
2. Ascend’s transition from inference to training. The Ascend ecosystem has long been viewed as an inference-only alternative to NVIDIA. T-Rex shows that with sufficient engineering effort — custom kernels, parallelism tuning, orchestration — the 910C NPU can be a credible training platform. This has geopolitical implications: Ascend hardware is not subject to the same export controls as NVIDIA’s H100/B200 series.
3. The software stack is the moat. NVIDIA’s advantage is CUDA — a mature, battle-tested software ecosystem. Ascend’s stack (CANN, MindSpeed, HCCL, and now AuraKernel) is less mature but harder for export restrictions to control. As the ecosystem matures, the gap narrows.

What to Watch

The next milestone is the Ascend 950DT, a training-focused NPU expected in Q4 2026. If the 910C can reach 34% MFU with custom engineering, a purpose-built training chip with a more mature software stack could close the gap with NVIDIA significantly.

T-Rex is not the end of the story. It is the first chapter of a new one.