OSDI '26 · Training Reliability & Silent Errors

Determinism Makes
Silent Errors Observable

AEGIS · SDCHunter · OpGuard

从真实事故、技术机制到分布式系统方法论

What is a pre-training task?

一个长期运行、同步推进、每一步都依赖上一步状态的
distributed optimization process

global batch→ distributed forward→ loss→ distributed backward→ gradient collectives→ optimizer state

Long-lived

持续数周；checkpoint 只离散保存部分状态。

Large-scale

成百上千设备共同决定一个 logical step 的结果。

Path-dependent

一个错误 update 会污染后续 parameter、optimizer 和 checkpoint。

The setup behind one training step

Application statesamples · model / optimizer · RNG · loss scaling · checkpoint / dataloader cursor

Parallel semanticsDP / TP / PP / CP / EP · microbatch order · recomputation · sharding

Runtimemultiple streams · overlap · allocator · graph capture · async collectives

Compiler & kernelsfusion · autotuning · atomics · reduction trees · precision policy

Fleetaccelerators · HBM · links · topology · host processes · aging · intermittent failure

训练轨迹是一个跨模型、软件、通信与硬件的分布式状态机。

Many ways for a training run to be wrong

Model / data / config

bad sample · wrong mask · optimizer / routing policy

OpGuard Bug 14: MoE routing drift ← histogram-bin mismatch

Numerics

overflow · precision change · reduction associativity

OpGuard Bug 10: different NCCL topology ← different FP reduction tree

Software correctness

race · stale buffer · missing barrier · bad fusion

OpGuard Bug 1 / 18: embedding race / Triton missing barrier

Distributed runtime

collective order · rank mapping · async dependency

OpGuard Bug 4 / 17: stream race / activation-offload stream misuse

Hardware

DUE · HBM / compute corruption · defective GPU

AEGIS: matrix-unit fault + HBM bit flip without ECC signal

Orchestration / restart

mixed version · data cursor · missing durable state

OpGuard Bug 20: resume divergence ← optimizer step variable overwrite

同一个 loss spike 可以来自其中任何一层。

What exactly is SDC?

Silent Data Corruption
数据被非预期改变，但改变发生时没有 error signal。

“silent”描述的是 corruption point，不是说运行永远没有后续症状。

Term	At fault time	Possible outcome
DUE	hardware / software detects unrecoverable error	abort, reset, explicit failure
SDC	bad value escapes without signal	wrong update, later NaN, shape error, crash
Software bug	may be detected or silent	can look identical to SDC
Numerical variation	often expected	may be benign or amplify a real fault

Three incidents · no clean “hardware error” signature

Software race

Seen: grad-norm alert after >3,000 steps, at different steps across reruns.

Origin: a race in fused embedding backward changed only a few rows.

OpGuard Bug 1 · manual 5 days → reported 15 min

Compute-unit SDC

Seen: periodic wrong positions in a GEMM output.

Origin: matrix-core MMA on one execution unit; vector-core execution stayed correct.

AEGIS Case 1 · workload-specific hardware fault

HBM SDC

Seen: forward activation and backward read no longer matched.

Origin: one bit in a saved fp32 intermediate changed; ECC did not fire.

AEGIS Case 2 · caught by deterministic recomputation

可见症状、corruption point、root-cause class 是三件不同的事。

Paper-reported cases: OpGuard §6.2 / Table 2; AEGIS §7.6.

What the algorithm user sees

loss spike / drift grad-norm anomaly NaN / Inf shape / CUDA error quality regression resume divergence

↕ many-to-many ↕

data / algorithm numerics software race compiler / kernel distributed runtime hardware SDC

Paper cases: grad-norm alert after 3,000+ steps ← embedding-backward race (OpGuard Bug 1)；shape mismatch about 30 ops later ← one-bit scatter SDC (OpGuard Bug 12)；resume divergence ← optimizer-step state overwrite (OpGuard Bug 20)。

loss 和 gradient norm 适合报警，不适合归因。

Why pre-training correctness bugs are hard to debug

Delayed

坏值数十个 op 或数千 step 后才可见。

Bug 1: backward race，后续 forward 才出现症状

Diluted

少数 float 被 aggregation 与低精度 metric 掩盖。

Bug 1 / 16: only a few gradient entries changed

Nondeterministic

健康重跑也可能因 reduction / schedule 不同而变化。

Bug 10: NCCL topology changed accumulation order

Expensive

一次 rerun 占用大规模集群，可尝试实验很少。

OpGuard cases report 3–14 day manual investigations

Intermittent

fault 依赖 input、unit、placement 和 pressure。

SDCHunter: observed replay rate 100% → 10⁻⁶%

Observer-sensitive

同步、dump、缩模型、换卡可能让 bug 消失。

OpGuard: heavy tensor dump made stream races harder to reproduce

The operational contract with algorithm teams

“机器看起来没问题”
不是可消费的结论。

系统团队不能证明一个无限的 negative；能做的是缩小 claim，并交付可审计证据。

ReproduceSDCHunter: 保存 checkpoint、input window、placement、communication order

CompareAEGIS / OpGuard: 对 intended-equivalent execution 做 exact comparison

LocalizeOpGuard certified prefix；SDCHunter group → tensor → physical GPU

ExcludeSDCHunter: workload replay on healthy reference + isolated suspect hardware

ReportAEGIS: warning 与 confirmed incident 分开；明确 coverage boundary

A concrete trap · the exception is not the origin

报错行可能完全正确。
坏状态早已在上游产生。

token count 被静默破坏，最后表现为 shape mismatch；
取模结果被破坏，最后表现为 embedding 越界；
如果只沿 exception stack 排查，会系统性地追错方向。

SDCHunter paper figures showing first observed errors and delayed shape or bounds failures — SDCs in the Wild, Figures 1–3 · official paper

Core thesis

Determinism is an observability substrate.

模糊症状→ 定义“同一计算”→ 精确比较→ 第一个不一致

它把“感觉训练不对”变成 oracle violation；
但还没有自动回答 hardware or software?

Determinism is not one contract

local

Operator determinism

Fix: input + implementation + launch
Expect: bitwise-identical output
Use: AEGIS local replay

trajectory

Run-to-run determinism

Fix: checkpoint + data + RNG + layout + communication
Expect: same training trajectory
Use: SDCHunter exact replay

recovery

Restart determinism

Fix: durable state + post-restart execution plan
Expect: resumed step = uninterrupted step

token semantics

Batch invariance

同一 token 的结果不依赖 batch position / composition。
Example: DeepSeek-V4 kernel design.

implementation independent

Cross-stack comparability

不同 kernel、fusion、framework 在共享 semantic boundary 产生相同 tensor。
Example: OpGuard.

MAI-Thinking-1 展示的是更完整的 data、checkpoint、kernel、collective 和 restart contract。先说清 scope，才有资格讨论 exactness。

The reusable primitive

Reference

algebraic identity
self-equivalence
peer majority
known-good run

Fingerprint

把 tensor dump 压成可在线记录的 compact evidence。

Boundary

从 operator、recompute chain、PP tensor 到 model-level op。

Budget

sample、coarse-to-fine、pipeline bubble、offline replay。

observable correctness = reference × fingerprint × boundary × cost budget

One problem, different operational questions

during the live job

AEGIS

Trigger: continuous, sampled sensing
Question: 受保护计算刚才是否被 corruption？
Output: confirmed corruption report + device context

after a symptom or alert

SDCHunter

Trigger: loss / NaN / shape error / detector alert
Question: 是不是 defective hardware？是哪张 GPU？
Output: suspicious group → physical device

when two runs can be compared

OpGuard

Trigger: correctness escalation; separate online mode
Question: 两次执行第一次在哪里语义分叉？
Output: first divergent boundary + trace context

Our synthesis. These are overlapping operating modes, not three mutually exclusive pipeline stages.

Paper 1 · Safeguarding LLM Training at Scale

AEGIS

当前正在运行的 job，某个受保护计算是否发生了 silent corruption？

cheap sensing on-path · definitive verification off-path

AEGIS · architecture

cSensor→vTask→cVerifier

sense: inline、便宜、允许可疑；
vTask: checksum / fingerprint + 最小 replay context；
verify: replay 或 self-equivalence 后才上报 confirmed SDC；
多种 sensor 共用 queue、scheduler 和 report path。

AEGIS · sensor family 1

AEGIS higher precision algorithmic checksum figures — Safeguarding LLM Training at Scale, Figures 5–6 · official paper

Mixed-precision-aware algebraic checks

Δmatmul = ‖(AB)1 − A(B1)‖

直接从 bf16 output 求 checksum，round-off noise 会淹没小 corruption；
从 MMA 的 fp32 accumulator 累积 checksum，再与代数等价路径比较；
FlashAttention 使用结构化 invariant：1ᵀdV = 1ᵀdO；
只保留可疑 row / column 和 verification 所需上下文。

方法论：checker 必须理解数值路径；“加一个 checksum”并不自动可靠。

AEGIS · sensor family 2 + verifier

AEGIS recomputation fingerprint and vTask scheduling figures — Safeguarding LLM Training at Scale, Figures 9–11 · official paper

Natural redundancy

forward activation 与 backward recomputation 本就执行同一逻辑；比较 XOR fingerprint。

Selective boundary

deterministic chain 只在末端观察，仍覆盖链内 corruption，减少 vTask。

Lazy verification

优先使用 PP bubble；step 尾部只留固定小 time slice，并动态调整 sampling。

AEGIS · what 35M GPU-hours teach us

35M

monitored GPU-hours

18

confirmed incidents / 13 GPUs

3

surfaced as visible training failures

0.86%

reported production overhead

Bug story

AEGIS 也抓到过 framework 越界写造成的 SDC-like mismatch：exact mismatch 不是硬件分类器。

Coverage story

一个已知 faulty GPU 的 corruption 落在 Transformer backbone 外；轻量 elementwise / 未保护区域仍可能漏。

Paper-reported deployment results; not independently reproduced.

Paper 2 · SDCs in the Wild

SDCHunter

loss spike、NaN、shape error 或 fail-stop，究竟是不是坏卡？是哪张物理 GPU？

exact-workload replay · group first · device later

SDCHunter · field characterization

SDCHunter loss spikes vary across replays from the same checkpoint — SDCs in the Wild, Figure 4 · official paper

23 张 confirmed defective GPUs 带来的三个结论

生命周期：不只 burn-in；aging 后也会出现。
条件性：hardware unit、kernel sequence 和 input data 共同决定是否触发。
保护缺口：ECC、thermal sensor、generic stress test 会漏 logic-level fault。

observed replay rate: 100% → 10⁻⁶%

同一 checkpoint 的 spike 可以出现在完全不同 step；“这次没复现”几乎不构成反证。

SDCHunter · methodology

不要先做 minimal reproducer。
先保存 the workload that made the fault observable.

Preserve semantics

model · checkpoint · data window · RNG · operator sequence

Preserve placement

parallel layout · rank mapping · physical device · topology

Preserve pressure

memory footprint · overlap · communication order · timing-sensitive behavior

缩模型、换卡、加同步、dump 全 tensor，都可能“修好”这个 bug。

SDCHunter · system-wide replay contract

Inputssame checkpoint, shards, microbatch order, RNG

Computedisable autotuning, pin kernels, deterministic reductions / MoE dispatch

Communicationpin collective algorithm, channel count, topology-sensitive order

Executionsame parallel config and recorded message ordering

健康系统应产生
bitwise-aligned trace

这些约束在正常训练时也开启，不只在 replay 临时开启；
否则 diagnosis 运行的已不是触发异常的 execution regime；
论文报告 fixed layout 下 step time 差异 < 0.01%。

SDCHunter · phase 1