Long-form Context Injection with Recurrent Compression
- Recurrent Context Compression
- Compressed Context Injection
Perceiver module
The compressed features are obtained through the Perceiver module as follows
where represents the Perceiver module with query and input features
Recurrent compression process
Segments are sequentially fed into the Perceiver module, with the compressed features from the previous segment serving as the query features for the next segment:
Compressed Context Injection
Injected this compressed information into the pretrained transformer using gated cross-attention layers with residual connections. The embeddings are then contextualized through Gated Cross Attention Block.
Query Dependent Context Modeling
