rillr
About Guidelines Privacy Terms Log in Join Rillr
← AI
aws.amazon.com · Nov 11

Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod

Silent data corruption is a GPU fault that returns wrong but plausible numbers with no error, quietly poisoning distributed training. This post adds three checks to a PyTorch DDP loop to detect it, localize the faulty node on Amazon SageMaker HyperPod, and hand the node to automatic recovery for replacement.

Visit article ↗ ▲ 0 💬 0
Discussion

0 comments

No comments yet. Start the discussion.