Speculative Decoding

Medusa

https://arxiv.org/pdf/2401.10774.pdf

Fast Inference from Transformers via Speculative Decoding

[arXiv](2023/05/18 version v2) Abstract 여러 개의 토큰을 병렬로 계산하여 더 빠르게 샘플링하는 Speculative Decoding 제안 Speculative Decoding 효율적인 모델 Mq가 토큰 시퀀스를 생성하고 목표 모델 Mp가 해당 시퀀스를 평가하여 토큰을 수용하거나 거부하고, 대안을 생성한다. 각 라인은 한 번의 decoding step이다. Standardized Sampling Argmax, top-k, nucleus, temperature 등 다양한 샘플링 설정이 있지만 본문에서는 생략하고 일반적인 경우만 가정. Speculative Sampling 준비물: 각 모델, 토큰 시퀀스 γ개의 예측 생성 Mp를 병렬로 실행하여 γ개의 예측을 각각 생성 q(x)..

https://ostin.tistory.com/402

Fast Inference from Transformers via Speculative Decoding

Speculative Decoding for 2x Faster Whisper Inference

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

https://huggingface.co/blog/whisper-speculative-decoding

Speculative Decoding for 2x Faster Whisper Inference

arxiv.org

https://arxiv.org/pdf/2211.17192.pdf

arxiv.org

https://arxiv.org/pdf/2402.01528.pdf

Accelerating Generative AI with PyTorch II: GPT, Fast

This post is the second part of a multi-series blog focused on how to accelerate generative AI models with pure, native PyTorch. We are excited to share a breadth of newly released PyTorch performance features alongside practical examples to see how far we can push PyTorch native performance. In part one, we showed how to accelerate Segment Anything over 8x using only pure, native PyTorch. In this blog we’ll focus on LLM optimization.

https://pytorch.org/blog/accelerating-generative-ai-2/

Accelerating Generative AI with PyTorch II: GPT, Fast

How to Make LLMs 3X Faster

In this article, we will look at how speculative decoding works.

https://blog.bytebytego.com/p/how-to-make-llms-3x-faster

RL-based speculative decoding system

Aurora

1.25x over a well-trained static speculator. Aurora is an open-source RL framework that turns speculative decoding from a one-time offline setup into a self-improving system that learns from every request it serves.

https://www.together.ai/blog/aurora

FlashMap: Flow Map Speculative Decoding

FlashMap conditions a continuous flow drafter on the target model's hidden states to propose an entire token block in one forward pass while capturing dependencies between tokens. On Qwen3-8B, the authors report up to 6.1× lossless acceleration and a 7% higher average speedup than a matched DFlash baseline. These are results for the evaluated configurations.

flashmap

al-jshen • Updated 2026 Oct 30 12:58

FlashMap: Flow Map Speculative Decoding

Speculative decoding accelerates large language model (LLM) inference by using a lightweight drafter to propose a block of tokens that the target model verifies in one parallel pass....

https://arxiv.org/abs/2610.24365

Speculative Decoding

Medusa

FlashMap: Flow Map Speculative Decoding

Backlinks

Recommendations