Gated Retention

Gated retention (gRet, aka gRetNet or RetNet-3) augments retention with a data-dependent gating mechanism, which achieves training parallelism, good performance, and low inference cost simultaneously for sequence modeling.