Learning rate Warmup

Creator

Creator

Seonglae Cho

Created

Created

2024 Mar 1 14:13

Editor

Editor

Seonglae Cho

Edited

Edited

2024 Mar 3 4:19

Refs

Refs

initialization

초기에는 작은 learning rate 사용하다 안정되면 높이는 방식

warmup_steps Number of steps used for a linear warmup from 0 to learning_rate

warmup_ratio Ratio of total training steps used for a linear warmup from 0 to learning_rate.

Recommendations

///////