Instead of optimizing one part of an inference system at a time, SEIS applies agentic self-evolution to optimize an entire LLM inference engine end-to-end, through iterative sessions that inherit code and reports from earlier ones. Serving Qwen3-0.6B on H100, the evolved engine reaches 3.27× the throughput of the original mini-sglang and beats vLLM, TensorRT-LLM, and SGLang in single-request serving. We validate the numerical fidelity and downstream task accuracy of the evolved engine, and identify reward-hacking failures in automated inference-system optimization.

SEIS: Self-Evolving Inference Systems
Zhen Xu, Jingyu Liu, Zongze Li, Tahseen Rabbani, Ce Zhang
arXiv preprint, 2026
[Paper] [Code]