EleutherAI/gpt-neo
An implementation of model & data parallel GPT2 & GPT3-like models, with the ability to scale up to full GPT3 sizes (and possibly more!), using the mesh-tensorflow library. Training and inference supported on both TPUs and GPUs.
https://github.com/EleutherAI/gpt-neo