Seonglae ChoAbstractive Non-Deterministic Turing Machine with probability
Autoregression is a statistical technique that uses past values to predict future values in a time series. It's a regression of a variable against itself.
only a predetermined number of best partial solutions are kept as candidates
explores a graph by expanding the most promising node in a limited set
use BFS to build Search Tree
A beam search is most often used to maintain tractability in large systems with insufficient amount of memory to store the entire search tree
sampling strategy
greedy
top_k- 상위 k개의 확률이 높은 토큰 중에서 무작위로 선택
nucleus- 확률이 높은 토큰을 일정한 확률 분포 내에서 선택하는 방식으로 가장 다양한 결과
It's similar to how technology evolves, with a Paradigm that changes through divergence and convergence
top-
arxiv.org
https://arxiv.org/pdf/2411.07641
Maxime Labonne - Decoding Strategies in Large Language Models
A Guide to Text Generation From Beam Search to Nucleus Sampling
https://mlabonne.github.io/blog/posts/2022-06-07-Decoding_strategies.html#beam-search
Beam search
In computer science, beam search is a heuristic search algorithm that explores a graph by expanding the most promising node in a limited set. Beam search is an optimization of best-first search that reduces its memory requirements. Best-first search is a graph search which orders all partial solutions (states) according to some heuristic. But in beam search, only a predetermined number of best partial solutions are kept as candidates.[1] It is thus a greedy algorithm.
https://en.wikipedia.org/wiki/Beam_search