AI Inference

AI Inference

Creator

Creator

Seonglae Cho

Created

Created

2023 Oct 26 17:15

Editor

Editor

Seonglae Cho

Edited

Edited

2026 Jul 21 9:5

Refs

Refs

Model Training Tool

pytorch-labs • Updated 2024 Nov 25 20:41

Prompt Engineering Tool

LLM Development

AI Optimization,
Inference Optimization, Inference Engine

Fast inference but usually no API access to hidden states

Model Inference Tools

Triton Inference

Transformer Engine

FasterTransformer

AI Server

Model Inference Servers

AI Inference Notion

In-Flight Batching

AI Performance Libraries

LLM inference cost is going down fast

Welcome to LLMflation - LLM inference cost is going down fast ⬇️ | Andreessen Horowitz

For LLM of equivalent performance, the inference cost is decreasing by 10x every year. What cost $60/million tokens in 2021 costs $.06/million tokens today.

https://a16z.com/llmflation-llm-inference-cost

Welcome to LLMflation - LLM inference cost is going down fast ⬇️ | Andreessen Horowitz

Inference cost with model size

Observations About LLM Inference Pricing — LessWrong

This work was done as part of the MIRI Technical Governance Team. It reflects my views and may not reflect those of the organization. …

Observations About LLM Inference Pricing — LessWrong

https://www.lesswrong.com/posts/mRKd4ArA5fYhd2BPb/observations-about-llm-inference-pricing

Observations About LLM Inference Pricing — LessWrong

What is moat

NeoSigma on Twitter / X

https://t.co/PDwGDe6d0y— NeoSigma (@neosigma_ai) July 17, 2026

NeoSigma on Twitter / X

https://x.com/neosigma_ai/status/2078159114254057612?s=20

Backlinks

AWS Inferentia Local LLM LLM AI Development

Recommendations

/////