[W1105 04:19:13.033074459 socket.cpp:209] [c10d] The hostname of the client socket cannot be retrieved. err=-3 █████ █████ ░░███ ░░███ ████████ ██████ ████████ ██████ ██████ ░███████ ██████ ███████ ░░███░░███ ░░░░░███ ░░███░░███ ███░░███ ███░░███ ░███░░███ ░░░░░███░░░███░ ░███ ░███ ███████ ░███ ░███ ░███ ░███░███ ░░░ ░███ ░███ ███████ ░███ ░███ ░███ ███░░███ ░███ ░███ ░███ ░███░███ ███ ░███ ░███ ███░░███ ░███ ███ ████ █████░░████████ ████ █████░░██████ ░░██████ ████ █████░░███████ ░░█████ ░░░░ ░░░░░ ░░░░░░░░ ░░░░ ░░░░░ ░░░░░░ ░░░░░░ ░░░░ ░░░░░ ░░░░░░░░ ░░░░░ Overriding: depth = 26 Overriding: device_batch_size = 16 Autodetected device type: cuda /home/seonglae/nanochat/.venv/lib/python3.10/site-packages/torch/__init__.py:1617: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.) _C._set_float32_matmul_precision(precision) /home/seonglae/nanochat/.venv/lib/python3.10/site-packages/torch/cuda/__init__.py:283: UserWarning: Found GPU0 NVIDIA GB10 which is of cuda capability 12.1. Minimum and Maximum cuda capability supported by this version of PyTorch is (8.0) - (12.0) warnings.warn( 2025-11-05 04:19:16,035 - nanochat.common - [32m[1mINFO[0m - Distributed world size: 2 Vocab size: 65,536 num_layers: 26 model_dim: 1664 num_heads: 13 num_kv_heads: 13 Tokens / micro-batch / rank: 16 x 2048 = 32,768 Tokens / micro-batch: 65,536 Total batch size 524,288 => gradient accumulation steps: 8 Number of parameters: 1,081,999,360 Estimated FLOPs per token: 6.900941e+09 Calculated number of iterations from target data:param ratio: 41,275 Total number of training tokens: 21,639,987,200 Tokens : Params ratio: 20.00 Total training FLOPs estimate: 1.493363e+20 Scaling the LR for the AdamW parameters ∝1/√(1664/768) = 0.679366 Muon: Grouping 104 params of shape torch.Size([1664, 1664]), device cuda:0, dtype torch.float32 Muon: Grouping 26 params of shape torch.Size([1664, 6656]), device cuda:0, dtype torch.float32 Muon: Grouping 26 params of shape torch.Size([6656, 1664]), device cuda:0, dtype torch.float32 Step 00000 | Validation bpb: 3.3071 [rank0]:W1105 04:43:56.152000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/utils.py:1558] [2/0] Not enough SMs to use max_autotune_gemm mode ================================================================ Internal Triton PTX codegen error `ptxas` stderr: ptxas fatal : Value 'sm_121a' is not defined for option 'gpu-name' Repro command: /home/seonglae/nanochat/.venv/lib/python3.10/site-packages/triton/backends/nvidia/bin/ptxas -lineinfo -v --gpu-name=sm_121a /tmp/tmpx9u3mic8.ptx -o /tmp/tmpx9u3mic8.ptx.o // // Generated by LLVM NVPTX Back-End // .version 8.7 .target sm_121a .address_size 64 // .globl triton_red_fused__to_copy_linalg_vector_norm_0 // -- Begin function triton_red_fused__to_copy_linalg_vector_norm_0 .extern .shared .align 16 .b8 global_smem[]; // @triton_red_fused__to_copy_linalg_vector_norm_0 .visible .entry triton_red_fused__to_copy_linalg_vector_norm_0( .param .u64 .ptr .global .align 1 triton_red_fused__to_copy_linalg_vector_norm_0_param_0, .param .u64 .ptr .global .align 1 triton_red_fused__to_copy_linalg_vector_norm_0_param_1, .param .u32 triton_red_fused__to_copy_linalg_vector_norm_0_param_2, .param .u32 triton_red_fused__to_copy_linalg_vector_norm_0_param_3, .param .u64 .ptr .global .align 1 triton_red_fused__to_copy_linalg_vector_norm_0_param_4, .param .u64 .ptr .global .align 1 triton_red_fused__to_copy_linalg_vector_norm_0_param_5 ) .reqntid 512 { .reg .pred %p<16>; .reg .b32 %r<108>; .reg .b64 %rd<73>; .loc 1 18 0 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:18:0 $L__func_begin0: .loc 1 18 0 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:18:0 // %bb.0: ld.param.b64 %rd3, [triton_red_fused__to_copy_linalg_vector_norm_0_param_1]; ld.param.b64 %rd2, [triton_red_fused__to_copy_linalg_vector_norm_0_param_0]; $L__tmp0: .loc 1 23 28 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:23:28 mov.u32 %r1, %ctaid.x; .loc 1 25 21 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:25:21 setp.lt.u32 %p1, %r1, 338; .loc 1 26 37 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:26:37 mov.u32 %r2, %tid.x; shl.b32 %r6, %r2, 2; and.b32 %r7, %r6, 2044; .loc 1 36 46 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:36:46 shl.b32 %r8, %r1, 13; or.b32 %r3, %r7, %r8; .loc 1 36 51 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:36:51 // begin inline asm mov.u64 %rd4, 0x0; createpolicy.fractional.L2::evict_first.b64 %rd4, 1.0; // end inline asm @%p1 bra $L__BB0_2; bra.uni $L__BB0_1; $L__BB0_2: // %.split.us.preheader .loc 1 36 34 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:36:34 mad.wide.u32 %rd16, %r3, 4, %rd2; mov.b32 %r46, 0; mov.pred %p6, -1; .loc 1 36 51 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:36:51 // begin inline asm mov.u32 %r42, %r46; mov.u32 %r43, %r46; mov.u32 %r44, %r46; mov.u32 %r45, %r46; @%p6 ld.global.L1::evict_first.L2::cache_hint.v4.b32 { %r42, %r43, %r44, %r45 }, [ %rd16 + 0 ], %rd4; // end inline asm .loc 1 36 34 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:36:34 add.s64 %rd19, %rd16, 8192; .loc 1 36 51 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:36:51 // begin inline asm mov.u64 %rd20, 0x0; createpolicy.fractional.L2::evict_first.b64 %rd20, 1.0; // end inline asm // begin inline asm mov.u32 %r50, %r46; mov.u32 %r51, %r46; mov.u32 %r52, %r46; mov.u32 %r53, %r46; @%p6 ld.global.L1::evict_first.L2::cache_hint.v4.b32 { %r50, %r51, %r52, %r53 }, [ %rd19 + 0 ], %rd20; // end inline asm .loc 1 36 34 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:36:34 add.s64 %rd22, %rd16, 16384; .loc 1 36 51 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:36:51 // begin inline asm mov.u64 %rd23, 0x0; createpolicy.fractional.L2::evict_first.b64 %rd23, 1.0; // end inline asm // begin inline asm mov.u32 %r58, %r46; mov.u32 %r59, %r46; mov.u32 %r60, %r46; mov.u32 %r61, %r46; @%p6 ld.global.L1::evict_first.L2::cache_hint.v4.b32 { %r58, %r59, %r60, %r61 }, [ %rd22 + 0 ], %rd23; // end inline asm .loc 1 36 34 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:36:34 add.s64 %rd25, %rd16, 24576; .loc 1 36 51 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:36:51 // begin inline asm mov.u64 %rd26, 0x0; createpolicy.fractional.L2::evict_first.b64 %rd26, 1.0; // end inline asm // begin inline asm mov.u32 %r66, %r46; mov.u32 %r67, %r46; mov.u32 %r68, %r46; mov.u32 %r69, %r46; @%p6 ld.global.L1::evict_first.L2::cache_hint.v4.b32 { %r66, %r67, %r68, %r69 }, [ %rd25 + 0 ], %rd26; // end inline asm cvt.u64.u32 %rd27, %r42; cvt.u64.u32 %rd28, %r43; shl.b64 %rd29, %rd28, 32; or.b64 %rd30, %rd27, %rd29; cvt.u64.u32 %rd31, %r50; cvt.u64.u32 %rd32, %r51; shl.b64 %rd33, %rd32, 32; or.b64 %rd34, %rd31, %rd33; .loc 1 39 22 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:39:22 mul.f32x2 %rd35, %rd34, %rd34; .loc 1 41 23 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:41:23 fma.rn.f32x2 %rd36, %rd30, %rd30, %rd35; .loc 1 36 51 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:36:51 cvt.u64.u32 %rd37, %r58; cvt.u64.u32 %rd38, %r59; shl.b64 %rd39, %rd38, 32; or.b64 %rd40, %rd37, %rd39; .loc 1 41 23 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:41:23 fma.rn.f32x2 %rd41, %rd40, %rd40, %rd36; .loc 1 36 51 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:36:51 cvt.u64.u32 %rd42, %r66; cvt.u64.u32 %rd43, %r67; shl.b64 %rd44, %rd43, 32; or.b64 %rd45, %rd42, %rd44; .loc 1 41 23 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:41:23 fma.rn.f32x2 %rd46, %rd45, %rd45, %rd41; .loc 1 36 51 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:36:51 cvt.u64.u32 %rd47, %r45; cvt.u64.u32 %rd48, %r44; shl.b64 %rd49, %rd48, 32; or.b64 %rd50, %rd47, %rd49; cvt.u64.u32 %rd51, %r53; cvt.u64.u32 %rd52, %r52; shl.b64 %rd53, %rd52, 32; or.b64 %rd54, %rd51, %rd53; .loc 1 39 22 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:39:22 mul.f32x2 %rd55, %rd54, %rd54; .loc 1 41 23 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:41:23 fma.rn.f32x2 %rd56, %rd50, %rd50, %rd55; .loc 1 36 51 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:36:51 cvt.u64.u32 %rd57, %r61; cvt.u64.u32 %rd58, %r60; shl.b64 %rd59, %rd58, 32; or.b64 %rd60, %rd57, %rd59; .loc 1 41 23 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:41:23 fma.rn.f32x2 %rd61, %rd60, %rd60, %rd56; .loc 1 36 51 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:36:51 cvt.u64.u32 %rd62, %r69; cvt.u64.u32 %rd63, %r68; shl.b64 %rd64, %rd63, 32; or.b64 %rd65, %rd62, %rd64; .loc 1 41 23 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:41:23 fma.rn.f32x2 %rd66, %rd65, %rd65, %rd61; .loc 1 26 37 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:26:37 mov.b64 {_, %r74}, %rd46; mov.b64 %rd67, {%r74, %r75}; add.f32x2 %rd68, %rd46, %rd67; mov.b64 {_, %r76}, %rd66; mov.b64 %rd69, {%r76, %r77}; add.f32x2 %rd70, %rd69, %rd68; add.f32x2 %rd71, %rd66, %rd70; mov.b64 {%r107, _}, %rd71; bra.uni $L__BB0_3; $L__BB0_1: // %.split.preheader .loc 1 36 34 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:36:34 mad.wide.s32 %rd5, %r3, 4, %rd2; mov.b32 %r13, 0; mov.pred %p2, 0; .loc 1 36 51 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:36:51 // begin inline asm mov.u32 %r9, %r13; mov.u32 %r10, %r13; mov.u32 %r11, %r13; mov.u32 %r12, %r13; @%p2 ld.global.L1::evict_first.L2::cache_hint.v4.b32 { %r9, %r10, %r11, %r12 }, [ %rd5 + 0 ], %rd4; // end inline asm .loc 1 36 34 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:36:34 add.s64 %rd8, %rd5, 8192; .loc 1 36 51 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:36:51 // begin inline asm mov.u64 %rd9, 0x0; createpolicy.fractional.L2::evict_first.b64 %rd9, 1.0; // end inline asm // begin inline asm mov.u32 %r17, %r13; mov.u32 %r18, %r13; mov.u32 %r19, %r13; mov.u32 %r20, %r13; @%p2 ld.global.L1::evict_first.L2::cache_hint.v4.b32 { %r17, %r18, %r19, %r20 }, [ %rd8 + 0 ], %rd9; // end inline asm .loc 1 36 34 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:36:34 add.s64 %rd11, %rd5, 16384; .loc 1 36 51 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:36:51 // begin inline asm mov.u64 %rd12, 0x0; createpolicy.fractional.L2::evict_first.b64 %rd12, 1.0; // end inline asm // begin inline asm mov.u32 %r25, %r13; mov.u32 %r26, %r13; mov.u32 %r27, %r13; mov.u32 %r28, %r13; @%p2 ld.global.L1::evict_first.L2::cache_hint.v4.b32 { %r25, %r26, %r27, %r28 }, [ %rd11 + 0 ], %rd12; // end inline asm .loc 1 36 34 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:36:34 add.s64 %rd14, %rd5, 24576; .loc 1 36 51 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:36:51 // begin inline asm mov.u64 %rd15, 0x0; createpolicy.fractional.L2::evict_first.b64 %rd15, 1.0; // end inline asm // begin inline asm mov.u32 %r33, %r13; mov.u32 %r34, %r13; mov.u32 %r35, %r13; mov.u32 %r36, %r13; @%p2 ld.global.L1::evict_first.L2::cache_hint.v4.b32 { %r33, %r34, %r35, %r36 }, [ %rd14 + 0 ], %rd15; // end inline asm mov.b32 %r107, 0f00000000; $L__BB0_3: // %.split2.us .loc 1 26 37 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:26:37 and.b32 %r85, %r2, 31; $L__tmp1: .loc 2 291 36 // standard.py:291:36 @[ cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:43:25 ] shfl.sync.bfly.b32 %r86, %r107, 16, 31, -1; .loc 2 261 15 // standard.py:261:15 @[ cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:43:25 ] add.f32 %r87, %r107, %r86; .loc 2 291 36 // standard.py:291:36 @[ cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:43:25 ] shfl.sync.bfly.b32 %r88, %r87, 8, 31, -1; .loc 2 261 15 // standard.py:261:15 @[ cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:43:25 ] add.f32 %r89, %r87, %r88; .loc 2 291 36 // standard.py:291:36 @[ cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:43:25 ] shfl.sync.bfly.b32 %r90, %r89, 4, 31, -1; .loc 2 261 15 // standard.py:261:15 @[ cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:43:25 ] add.f32 %r91, %r89, %r90; .loc 2 291 36 // standard.py:291:36 @[ cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:43:25 ] shfl.sync.bfly.b32 %r92, %r91, 2, 31, -1; .loc 2 261 15 // standard.py:261:15 @[ cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:43:25 ] add.f32 %r93, %r91, %r92; .loc 2 291 36 // standard.py:291:36 @[ cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:43:25 ] shfl.sync.bfly.b32 %r94, %r93, 1, 31, -1; .loc 2 261 15 // standard.py:261:15 @[ cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:43:25 ] add.f32 %r79, %r93, %r94; .loc 2 291 36 // standard.py:291:36 @[ cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:43:25 ] setp.eq.b32 %p10, %r85, 0; shr.u32 %r95, %r2, 3; and.b32 %r96, %r95, 60; mov.b32 %r97, global_smem; add.s32 %r78, %r97, %r96; // begin inline asm @%p10 st.shared.b32 [ %r78 + 0 ], %r79; // end inline asm bar.sync 0; setp.lt.u32 %p11, %r2, 16; add.s32 %r81, %r97, %r6; // begin inline asm @%p11 ld.shared.b32 %r80, [ %r81 + 0 ]; // end inline asm shfl.sync.bfly.b32 %r99, %r80, 8, 31, -1; .loc 2 261 15 // standard.py:261:15 @[ cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:43:25 ] add.f32 %r100, %r80, %r99; .loc 2 291 36 // standard.py:291:36 @[ cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:43:25 ] shfl.sync.bfly.b32 %r101, %r100, 4, 31, -1; .loc 2 261 15 // standard.py:261:15 @[ cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:43:25 ] add.f32 %r102, %r100, %r101; .loc 2 291 36 // standard.py:291:36 @[ cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:43:25 ] shfl.sync.bfly.b32 %r103, %r102, 2, 31, -1; .loc 2 261 15 // standard.py:261:15 @[ cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:43:25 ] add.f32 %r104, %r102, %r103; .loc 2 291 36 // standard.py:291:36 @[ cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:43:25 ] shfl.sync.bfly.b32 %r105, %r104, 1, 31, -1; .loc 2 261 15 // standard.py:261:15 @[ cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:43:25 ] add.f32 %r83, %r104, %r105; .loc 2 291 36 // standard.py:291:36 @[ cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:43:25 ] setp.eq.b32 %p12, %r2, 0; // begin inline asm @%p12 st.shared.b32 [ %r81 + 0 ], %r83; // end inline asm bar.sync 0; ld.shared.b32 %r84, [global_smem]; $L__tmp2: .loc 1 44 25 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:44:25 mad.wide.u32 %rd72, %r1, 4, %rd3; .loc 1 44 36 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:44:36 and.b32 %r106, %r2, 511; setp.eq.b32 %p15, %r106, 0; and.pred %p13, %p1, %p15; // begin inline asm @%p13 st.global.b32 [ %rd72 + 0 ], { %r84 }; // end inline asm .loc 1 44 4 // cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py:44:4 ret; $L__tmp3: $L__func_end0: // -- End function } .file 1 "/tmp/torchinductor_seonglae/jd/cjdzrpiowtq6cqdctsghweanujgqn5x2bapyl7sbliz2sy6qvhe2.py" .file 2 "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/triton/language/standard.py" .section .debug_abbrev { .b8 1 // Abbreviation Code .b8 17 // DW_TAG_compile_unit .b8 1 // DW_CHILDREN_yes .b8 37 // DW_AT_producer .b8 8 // DW_FORM_string .b8 19 // DW_AT_language .b8 5 // DW_FORM_data2 .b8 3 // DW_AT_name .b8 8 // DW_FORM_string .b8 16 // DW_AT_stmt_list .b8 6 // DW_FORM_data4 .b8 27 // DW_AT_comp_dir .b8 8 // DW_FORM_string .b8 0 // EOM(1) .b8 0 // EOM(2) .b8 2 // Abbreviation Code .b8 46 // DW_TAG_subprogram .b8 0 // DW_CHILDREN_no .b8 3 // DW_AT_name .b8 8 // DW_FORM_string .b8 32 // DW_AT_inline .b8 11 // DW_FORM_data1 .b8 0 // EOM(1) .b8 0 // EOM(2) .b8 3 // Abbreviation Code .b8 46 // DW_TAG_subprogram .b8 1 // DW_CHILDREN_yes .b8 17 // DW_AT_low_pc .b8 1 // DW_FORM_addr .b8 18 // DW_AT_high_pc .b8 1 // DW_FORM_addr .b8 49 // DW_AT_abstract_origin .b8 19 // DW_FORM_ref4 .b8 0 // EOM(1) .b8 0 // EOM(2) .b8 4 // Abbreviation Code .b8 29 // DW_TAG_inlined_subroutine .b8 0 // DW_CHILDREN_no .b8 49 // DW_AT_abstract_origin .b8 19 // DW_FORM_ref4 .b8 17 // DW_AT_low_pc .b8 1 // DW_FORM_addr .b8 18 // DW_AT_high_pc .b8 1 // DW_FORM_addr .b8 88 // DW_AT_call_file .b8 11 // DW_FORM_data1 .b8 89 // DW_AT_call_line .b8 11 // DW_FORM_data1 .b8 87 // DW_AT_call_column .b8 11 // DW_FORM_data1 .b8 0 // EOM(1) .b8 0 // EOM(2) .b8 0 // EOM(3) } .section .debug_info { .b32 204 // Length of Unit .b8 2 // DWARF version number .b8 0 .b32 .debug_abbrev // Offset Into Abbrev. Section .b8 8 // Address Size (in bytes) .b8 1 // Abbrev [1] 0xb:0xc5 DW_TAG_compile_unit .b8 116 // DW_AT_producer .b8 114 .b8 105 .b8 116 .b8 111 .b8 110 .b8 0 .b8 2 // DW_AT_language .b8 0 .b8 99 // DW_AT_name .b8 106 .b8 100 .b8 122 .b8 114 .b8 112 .b8 105 .b8 111 .b8 119 .b8 116 .b8 113 .b8 54 .b8 99 .b8 113 .b8 100 .b8 99 .b8 116 .b8 115 .b8 103 .b8 104 .b8 119 .b8 101 .b8 97 .b8 110 .b8 117 .b8 106 .b8 103 .b8 113 .b8 110 .b8 53 .b8 120 .b8 50 .b8 98 .b8 97 .b8 112 .b8 121 .b8 108 .b8 55 .b8 115 .b8 98 .b8 108 .b8 105 .b8 122 .b8 50 .b8 115 .b8 121 .b8 54 .b8 113 .b8 118 .b8 104 .b8 101 .b8 50 .b8 46 .b8 112 .b8 121 .b8 0 .b32 .debug_line // DW_AT_stmt_list .b8 47 // DW_AT_comp_dir .b8 116 .b8 109 .b8 112 .b8 47 .b8 116 .b8 111 .b8 114 .b8 99 .b8 104 .b8 105 .b8 110 .b8 100 .b8 117 .b8 99 .b8 116 .b8 111 .b8 114 .b8 95 .b8 115 .b8 101 .b8 111 .b8 110 .b8 103 .b8 108 .b8 97 .b8 101 .b8 47 .b8 106 .b8 100 .b8 0 .b8 2 // Abbrev [2] 0x70:0x31 DW_TAG_subprogram .b8 116 // DW_AT_name .b8 114 .b8 105 .b8 116 .b8 111 .b8 110 .b8 95 .b8 114 .b8 101 .b8 100 .b8 95 .b8 102 .b8 117 .b8 115 .b8 101 .b8 100 .b8 95 .b8 95 .b8 116 .b8 111 .b8 95 .b8 99 .b8 111 .b8 112 .b8 121 .b8 95 .b8 108 .b8 105 .b8 110 .b8 97 .b8 108 .b8 103 .b8 95 .b8 118 .b8 101 .b8 99 .b8 116 .b8 111 .b8 114 .b8 95 .b8 110 .b8 111 .b8 114 .b8 109 .b8 95 .b8 48 .b8 0 .b8 1 // DW_AT_inline .b8 3 // Abbrev [3] 0xa1:0x2e DW_TAG_subprogram .b64 $L__func_begin0 // DW_AT_low_pc .b64 $L__func_end0 // DW_AT_high_pc .b32 112 // DW_AT_abstract_origin .b8 4 // Abbrev [4] 0xb6:0x18 DW_TAG_inlined_subroutine .b32 112 // DW_AT_abstract_origin .b64 $L__tmp1 // DW_AT_low_pc .b64 $L__tmp2 // DW_AT_high_pc .b8 1 // DW_AT_call_file .b8 43 // DW_AT_call_line .b8 25 // DW_AT_call_column .b8 0 // End Of Children Mark .b8 0 // End Of Children Mark } .section .debug_macinfo { } ================================================================ please share the reproducer above with Triton project. [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] Triton compilation failed: triton_red_fused__to_copy_linalg_vector_norm_0 [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] def triton_red_fused__to_copy_linalg_vector_norm_0(in_ptr0, out_ptr0, xnumel, r0_numel, XBLOCK : tl.constexpr, R0_BLOCK : tl.constexpr): [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] xnumel = 338 [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] r0_numel = 8192 [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] rnumel = r0_numel [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] RBLOCK: tl.constexpr = R0_BLOCK [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] xoffset = tl.program_id(0) * XBLOCK [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] xindex = xoffset + tl.arange(0, XBLOCK)[:, None] [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] xmask = xindex < xnumel [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] r0_base = tl.arange(0, R0_BLOCK)[None, :] [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] rbase = r0_base [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] x0 = xindex [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] _tmp5 = tl.full([XBLOCK, R0_BLOCK], 0, tl.float32) [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] for r0_offset in range(0, r0_numel, R0_BLOCK): [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] r0_index = r0_offset + r0_base [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] r0_mask = r0_index < r0_numel [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] roffset = r0_offset [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] rindex = r0_index [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] r0_1 = r0_index [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] tmp0 = tl.load(in_ptr0 + (r0_1 + 8192*x0), r0_mask & xmask, eviction_policy='evict_first', other=0.0) [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] tmp1 = tmp0.to(tl.float32) [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] tmp2 = tmp1.to(tl.float32) [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] tmp3 = tmp2 * tmp2 [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] tmp4 = tl.broadcast_to(tmp3, [XBLOCK, R0_BLOCK]) [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] tmp6 = _tmp5 + tmp4 [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] _tmp5 = tl.where(r0_mask & xmask, tmp6, _tmp5) [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] tmp5 = tl.sum(_tmp5, 1)[:, None] [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] tl.store(out_ptr0 + (x0), tmp5, xmask) [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] metadata: {'signature': {'in_ptr0': '*fp32', 'out_ptr0': '*fp32', 'xnumel': 'i32', 'r0_numel': 'i32', 'XBLOCK': 'constexpr', 'R0_BLOCK': 'constexpr'}, 'device': 0, 'constants': {'XBLOCK': 1, 'R0_BLOCK': 2048}, 'configs': [{(0,): [['tt.divisibility', 16]], (1,): [['tt.divisibility', 16]], (3,): [['tt.divisibility', 16]]}], 'device_type': 'cuda', 'num_warps': 16, 'num_stages': 1, 'debug': True, 'cc': 121} [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] Traceback (most recent call last): [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] File "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/triton/backends/nvidia/compiler.py", line 468, in make_cubin [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] subprocess.run(ptxas_cmd, check=True, close_fds=False, stderr=flog) [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] File "/home/seonglae/.local/share/uv/python/cpython-3.10.19-linux-aarch64-gnu/lib/python3.10/subprocess.py", line 526, in run [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] raise CalledProcessError(retcode, process.args, [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] subprocess.CalledProcessError: Command '['/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/triton/backends/nvidia/bin/ptxas', '-lineinfo', '-v', '--gpu-name=sm_121a', '/tmp/tmpx9u3mic8.ptx', '-o', '/tmp/tmpx9u3mic8.ptx.o']' returned non-zero exit status 255. [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] During handling of the above exception, another exception occurred: [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] Traceback (most recent call last): [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] File "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 778, in _precompile_config [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] binary = triton.compile(*compile_args, **compile_kwargs) [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] File "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/triton/compiler/compiler.py", line 320, in compile [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] next_module = compile_ir(module, metadata) [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] File "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/triton/backends/nvidia/compiler.py", line 520, in [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] stages["cubin"] = lambda src, metadata: self.make_cubin(src, metadata, options, self.target.arch) [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] File "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/triton/backends/nvidia/compiler.py", line 503, in make_cubin [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] raise PTXASError(error) [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] triton.runtime.errors.PTXASError: PTXAS error: Internal Triton PTX codegen error [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] `ptxas` stderr: [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] ptxas fatal : Value 'sm_121a' is not defined for option 'gpu-name' [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] Repro command: /home/seonglae/nanochat/.venv/lib/python3.10/site-packages/triton/backends/nvidia/bin/ptxas -lineinfo -v --gpu-name=sm_121a /tmp/tmpx9u3mic8.ptx -o /tmp/tmpx9u3mic8.ptx.o [rank0]:E1105 04:43:56.518000 300217 .venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py:780] [2/0] [rank0]: Traceback (most recent call last): [rank0]: File "/home/seonglae/.local/share/uv/python/cpython-3.10.19-linux-aarch64-gnu/lib/python3.10/runpy.py", line 196, in _run_module_as_main [rank0]: return _run_code(code, main_globals, None, [rank0]: File "/home/seonglae/.local/share/uv/python/cpython-3.10.19-linux-aarch64-gnu/lib/python3.10/runpy.py", line 86, in _run_code [rank0]: exec(code, run_globals) [rank0]: File "/home/seonglae/nanochat/scripts/base_train.py", line 286, in [rank0]: opt.step() [rank0]: File "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/torch/optim/optimizer.py", line 517, in wrapper [rank0]: out = func(*args, **kwargs) [rank0]: File "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context [rank0]: return func(*args, **kwargs) [rank0]: File "/home/seonglae/nanochat/nanochat/muon.py", line 176, in step [rank0]: g = zeropower_via_newtonschulz5(g, steps=group["ns_steps"]) [rank0]: File "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 845, in compile_wrapper [rank0]: raise e.remove_dynamo_frames() from None # see TORCHDYNAMO_VERBOSE=1 [rank0]: File "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 990, in _compile_fx_inner [rank0]: raise InductorError(e, currentframe()).with_traceback( [rank0]: File "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 974, in _compile_fx_inner [rank0]: mb_compiled_graph = fx_codegen_and_compile( [rank0]: File "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1695, in fx_codegen_and_compile [rank0]: return scheme.codegen_and_compile(gm, example_inputs, inputs_to_check, graph_kwargs) [rank0]: File "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1505, in codegen_and_compile [rank0]: compiled_module = graph.compile_to_module() [rank0]: File "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/torch/_inductor/graph.py", line 2319, in compile_to_module [rank0]: return self._compile_to_module() [rank0]: File "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/torch/_inductor/graph.py", line 2329, in _compile_to_module [rank0]: mod = self._compile_to_module_lines(wrapper_code) [rank0]: File "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/torch/_inductor/graph.py", line 2397, in _compile_to_module_lines [rank0]: mod = PyCodeCache.load_by_key_path( [rank0]: File "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 3548, in load_by_key_path [rank0]: mod = _reload_python_module(key, path, set_sys_modules=in_toplevel) [rank0]: File "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/torch/_inductor/runtime/compile_tasks.py", line 33, in _reload_python_module [rank0]: exec(code, mod.__dict__, mod.__dict__) [rank0]: File "/tmp/torchinductor_seonglae/ru/cru4n75qryyfhybd6gs3feyrxnkst4woonaugrrh6j6thesvegdk.py", line 49, in [rank0]: triton_red_fused__to_copy_linalg_vector_norm_0 = async_compile.triton('triton_red_fused__to_copy_linalg_vector_norm_0', ''' [rank0]: File "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/torch/_inductor/async_compile.py", line 500, in triton [rank0]: kernel.precompile( [rank0]: File "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 448, in precompile [rank0]: self._precompile_worker() [rank0]: File "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 474, in _precompile_worker [rank0]: raise NoTritonConfigsError( [rank0]: torch._inductor.exc.InductorError: NoTritonConfigsError: No valid triton configs. PTXASError: PTXAS error: Internal Triton PTX codegen error [rank0]: `ptxas` stderr: [rank0]: ptxas fatal : Value 'sm_121a' is not defined for option 'gpu-name' [rank0]: Repro command: /home/seonglae/nanochat/.venv/lib/python3.10/site-packages/triton/backends/nvidia/bin/ptxas -lineinfo -v --gpu-name=sm_121a /tmp/tmpx9u3mic8.ptx -o /tmp/tmpx9u3mic8.ptx.o [rank0]: Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo" [rank0]:[W1105 04:43:59.801267274 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) [rank0]:[W1105 04:44:03.362613554 AllocatorConfig.cpp:28] Warning: PYTORCH_CUDA_ALLOC_CONF is deprecated, use PYTORCH_ALLOC_CONF instead (function operator()) E1105 04:44:06.417000 300096 torch/distributed/elastic/multiprocessing/api.py:882] failed (exitcode: 1) local_rank: 0 (pid: 300217) of binary: /home/seonglae/nanochat/.venv/bin/python3 Traceback (most recent call last): File "/home/seonglae/nanochat/.venv/bin/torchrun", line 10, in sys.exit(main()) File "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 357, in wrapper return f(*args, **kwargs) File "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/torch/distributed/run.py", line 936, in main run(args) File "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/torch/distributed/run.py", line 927, in run elastic_launch( File "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 156, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/home/seonglae/nanochat/.venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 293, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ scripts.base_train FAILED ------------------------------------------------------------ Failures: ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2025-11-05_04:44:06 host : localhost rank : 0 (local_rank: 0) exitcode : 1 (pid: 300217) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================

DGX Spark 놀이

nanochat

성공

config 변경

Recommendations