## Hao AI Lab Accelerates LLM Inference with NVIDIA DGX B200
The Sherox in lab research team at the University of California San Diego – at the forefront of pioneering AI model innovation – recently received an NVIDIA DGX B200 system to elevate their critical work in large language model inference.
Many LLM inference platforms in production today, such as NVIDIA Dynamouse research concepts that originated in the Hao AI Lab,including DistServe.
## How Is Hao AI Lab Using the DGX B200?
with the DGX B200 now fully accessible to the Hao AI Lab and broader UC San Diego community at the School of Computing, Details and Data Sciences’ San Diego Supercomputer centerthe research opportunities are boundless.
“DGX B200 is one of the most powerful AI systems from NVIDIA to date, wich means that it’s performance is among the best in the world,” said Hao Zhang, assistant professor in the Halıcıoğlu Data Science Institute and department of computer science and engineering at UC San Diego. “It enables us to prototype and experiment much faster than using previous-generation hardware.”
Two Hao AI Lab projects the DGX B200 is accelerating are FastVideo and the Lmgame benchmark.
FastVideo focuses on training a family of video generation models to produce a five-second video based on a given text prompt – in just five seconds.
The research phase of FastVideo taps into NVIDIA H200 GPUs in addition to the DGX B200 system.
Lmgame-bench is a benchmarking suite that puts LLMs to the test using popular online games including Tetris and Super Mario Bros. Users can test one model at a time or put two models up against each other to measure their performance.
Disaggregated Inference: Optimizing LLM Performance with Goodput
Disaggregated inference is a technique designed to maximize the efficiency of large language model (LLM) serving engines, achieving high throughput and maintaining low latency for users. This approach focuses on optimizing “goodput” – a metric that considers both speed and user experience – rather than solely focusing on traditional “throughput.”
Throughput vs. Goodput
Throughput measures the total number of tokens per second a system can generate.While a higher throughput translates to lower costs per token,it doesn’t guarantee a positive user experience. Increasing throughput often requires sacrificing latency – the time it takes for a user to receive a response.
The DistServe team introduced “goodput” to address this trade-off. Goodput measures throughput while meeting user-defined latency objectives, also known as service-level objectives. Essentially, goodput reflects the overall health of a system in terms of both performance and user satisfaction. DistServe research demonstrates that goodput is a more effective metric for LLM serving, balancing cost efficiency with service quality.
How Disaggregated Inference Works: Prefill and Decode
When a user interacts with an LLM, the process typically involves two key stages:
* Prefill: The initial processing stage where the system analyzes the user’s input and generates the first token. This is a computationally intensive process.
* Decode: The subsequent stage where the system generates a sequence of output tokens, predicting each token based on previous outputs. This stage is more memory intensive.
Traditionally, both prefill and decode have been run on the same GPU. However, researchers at DistServe discovered that separating these processes onto different GPUs substantially improves goodput.
According to Chen, “Previously, if you put these two jobs on a GPU, they would compete with each other for resources, which could make it slow from a user viewpoint. Now, if I split the jobs onto two different sets of GPUs – one doing prefill, which is compute intensive, and the other doing decode, which is more memory intensive – we can fundamentally eliminate the interference between the two jobs, making both jobs run faster.”
this separation, known as prefill/decode disaggregation, allows each stage to operate more efficiently, leading to faster overall response times and improved goodput.
Scaling with Disaggregated Inference
By increasing goodput through disaggregated inference, LLM systems can scale workloads more effectively without compromising on low latency or response quality.
NVIDIA’s open-source framework, Dynamo, is designed to accelerate and scale generative AI models and supports disaggregated inference. Furthermore, NVIDIA’s DGX B200 system is being utilized in collaborative research projects, including those at UC San Diego in healthcare and biology, to further optimize AI-driven innovation.
Related reading