From the source
Baidu announced FastDeploy 2.0, an end-to-end toolkit for efficient deployment and high-performance inference of large models, with native support for the ERNIE 4.5 family.
It features high-throughput inference using Expert Parallel and Prefill-Decode Disaggregation, near-lossless 2-bit quantization enabling trillion-parameter models on a single GPU, and optimizations for heterogeneous hardware.
Benchmark results show significant throughput gains over vLLM for ERNIE and other models.