Seonglae Cho- quant_linear
- Quantizer
text-generation-server quantize ORIGINAL_MODEL_ID NEW_MODEL_ID
simple
--dtype float16 --dtype bfloat16
Main document and quantization list
github.com
https://github.com/huggingface/text-generation-inference/blob/main/docs/source/conceptual/quantization.md
AWQ
GPTQ
github.com
https://github.com/huggingface/text-generation-inference/tree/main/server/text_generation_server/utils/gptq