Seonglae ChoSSM (State Space Model)
Selective scan which allows the model to filter (ir)relevant information
특정 조건을 만족하는 데이터 또는 요소들만을 선택해 공간을 형성하는 개념 Selectively Retain Information
Selectively Copy with hardware-aware algorithm between DRAM and SRAM
The matrices A, B, and C are the same for every token the SSM generates. LTI System
- Update the state based on input and time (like RNN) (state equation)
- (output equation)
이 block 은 attention mechanism보다는 mlp, residual connection이 포함된 transformer block 전체 의 대체 표현이고 핵심은 rnn과 달리 hidden state를 좀더 활용한다는 정도
Selective State Space Model
Selective State Space Notion
