nanoGPT train.py

Training speed technique

• torch.distributed
  torch.nn.parallel

• torch.amp

• torch.compile()

model_args

• n_layer

• n_head

• n_embd

• block_size

• bias

• dropout

functions

• estimate_loss() - get loss

• get_batch() - get batch

• get_lr() - learning_rate

Training Loop