LLM inference engine from scratch — paged KV cache, continuous batching, chunked prefill, prefix caching, speculative decoding, CUDA graph, tensor parallelism, OpenAI-compatible serving

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling

🚀 Accelerate LLM inference with Mini-Infer, a high-performance engine designed for efficiency and power in AI model deployment.