Triton Inference

Creator

Creator

Seonglae Cho

Created

Created

2023 Oct 5 5:9

Editor

Editor

Seonglae Cho

Edited

Edited

2024 Mar 8 15:54

Refs

Refs

Transformer Engine

Faster Transformer

open-source inference serving software

Triton Inference Usages

Triton Inference Server

Triton Huggingface Inference

Triton Model Analyzer

Document

Triton Inference Server — NVIDIA Triton Inference Server

Getting Started

Triton Inference Server — NVIDIA Triton Inference Server

https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html

Accelerated Inference for Large Transformer Models Using NVIDIA Triton Inference Server | NVIDIA Technical Blog

Learn about FasterTransformer, one of the fastest libraries for distributed inference of transformers of any size, including benefits of using the library.

Accelerated Inference for Large Transformer Models Using NVIDIA Triton Inference Server | NVIDIA Technical Blog

https://developer.nvidia.com/blog/accelerated-inference-for-large-transformer-models-using-nvidia-fastertransformer-and-nvidia-triton-inference-server/

Accelerated Inference for Large Transformer Models Using NVIDIA Triton Inference Server | NVIDIA Technical Blog

Recommendations

//////