Research · Inference
Inference
LLM inference benchmarks, model by model and GPU by GPU — throughput, latency, and memory, with the limits of each result stated.
AUG 2026 · BENCHMARKS · LLM
Outperforming vLLM and Llama.cpp on Gemma4-12B
Emmy, an AI-driven compiler, generates CUDA kernels automatically instead of by hand. On Gemma 4 12B in FP16 its kernels reach up to 1.6× over cuBLAS on the RTX 5090, achieved via TMA transport, hybrid FP16/FP32 accumulation, and FlashAttention-2 parity, shipped as a drop-in vLLM plugin.
JUL 2026 · LLM · OPEN SOURCE
The State of Open Coding AI Models in July 2026
A data report on open-weight coding models in July 2026: what agentic developers actually run, how far the open-to-closed gap has closed, and why the best open coders have outgrown the hardware most people own.
JUL 2026 · TUTORIALS · GPU
Learning FlashAttention the Hard Way — Part 2
Part 2 takes Part 1's twisted monoid down to the metal: a generic compiler moveset walks one attention kernel from 25959µs to 206µs on an RTX 5090 — on par with FlashAttention-2 — with a measured number for every rung, the optimizations measurement refused, and the reasons FA-3/4 cannot land on consumer silicon.
JUL 2026 · TUTORIALS · ALGEBRA
Learning FlashAttention the Hard Way — Part 1
FlashAttention looks like a bespoke GPU kernel, but it is not much different from a regular parallel reduction. Safe softmax becomes an associative reduction via denominator trick, Welford's variance, and flash attention is the same concept. This part is the complete theoretical foundation, with the numerical-stability analysis and a test for when any loop is secretly associative.
MAR 2026 · BENCHMARKS · LLM
Optimizing Qwen3 Coder for RTX 5090 and PRO 6000
I got Qwen3 Coder from 277 tok/s to 1,207 tok/s on a PRO 6000, and from 556 to 1,157 tok/s on an RTX 5090. Here's exactly how, with reproducible recipes.
JAN 2026 · BENCHMARKS · LLM
Blackwell Dominates. Benchmarking LLM Inference on NVIDIA B200, H200, H100, and RTX PRO 6000
We benchmarked NVIDIA B200, H200, H100, and RTX PRO 6000 for long-context LLM inference using 8K input + 8K output (16K total). B200 delivers up to 4.9× the throughput of RTX PRO 6000 and is now the cost efficiency leader across all models.
NOV 2025 · BENCHMARKS · LLM
RTX PRO 6000 vs H100, H200, and L40S: LLM Inference
RTX PRO 6000 beats H100 SXM on single-GPU LLM inference at 28% lower cost per token. H100 and H200 NVLink pull 3-4x ahead on 8-way tensor parallel.
OCT 2025 · BENCHMARKS · LLM
RTX 4090 vs 5090 vs PRO 6000: LLM Inference Benchmark
RTX 4090, 5090, and PRO 6000 benchmarked on vLLM with three quantized models (24GB, 48GB, 96GB). PRO 6000 hits 3.7x the 4090 throughput on single GPU.
SEP 2025 · BENCHMARKS · LLM
Benchmarking LLM Inference on RTX 4090, RTX 5090, and RTX PRO 6000
We ran a series of benchmarks across multiple GPU cloud servers to evaluate their performance for LLM workloads, specifically serving LLaMA and Qwen models and on RTX 4090, RTX 5090, and RTX PRO 6000 GPUs.
AUG 2025 · AI TOOLS & WORKFLOWS · LLM
How to Give Your RTX GPU Nearly Infinite Memory for LLM Inference
Network-Attached KV Cache for Long-Context, Multi-Turn Workloads. Let's be honest — we can't afford an H100. Learn how to extend your RTX GPU's effective memory using innovative KV cache offloading techniques.