LLM Coaching and Inference with Intel Gaudi 2 AI Accelerators

At Databricks, we need to assist our prospects construct and deploy generative AI purposes on their very own information with out sacrificing information privateness or management. For patrons who need to prepare a customized AI mannequin, we assist them accomplish that simply, effectively, and at a low price. One lever now we have to handle this problem is ML {hardware} optimization; to that finish, now we have been working tirelessly to make sure our LLM stack can seamlessly assist a wide range of ML {hardware} platforms (e.g., NVIDIA [1][2], AMD [3][4]).

At this time, we’re excited to debate one other main participant within the AI coaching and inference market: the Intel® Gaudi® household of AI Accelerators! These accelerators can be found by way of AWS (first-generation Gaudi), the Intel Developer Cloud (Gaudi 2), and for on-premise implementations, Supermicro and WiWynn (Gaudi and Gaudi 2 accelerators).

On this weblog, we profile the Intel^R Gaudi^R 2 for LLM coaching utilizing our open-source LLM Foundry and for inference utilizing the open-source Optimum Habana library. We discover that general, the Intel^R Gaudi^R 2 accelerator has the 2nd greatest coaching performance-per-chip we have examined (solely bested by the NVIDIA H100), with over 260 TFLOP/s/machine achieved when coaching MPT-7B on 8 x Gaudi 2. For multi-node coaching, we had entry to 160x Intel^R Gaudi^R 2 accelerators and located near-linear scaling throughout the cluster. For LLM inference, we profiled LLaMa2-70B and located that the 8 x Gaudi 2 system matches the 8 x H100 system in decoding latency (the most costly part of LLM inference).

As a result of the Gaudi 2 accelerator is publicly accessible by way of the Intel Developer Cloud (IDC), we may additionally estimate performance-per-dollar. Based mostly on public, on-demand pricing on AWS and IDC, we discover that Intel^R Gaudi^R 2 has the very best coaching and inference performance-per-dollar, beating out the NVIDIA A100-40GB, A100-80GB, and even H100!

All outcomes on this weblog had been measured utilizing SynapseAI 1.12 and BF16 combined precision coaching. Sooner or later, we sit up for exploring SynapseAI 1.13, which unlocks Gaudi 2’s assist for FP8 coaching, efficiently demonstrated of their MLPerf Coaching 3.1 GPT3 submission as reaching 379 TFLOP/s/machine on a cluster of 256xGaudi 2 and 368 TFLOP/s/machine on a cluster of 384xGaudi 2. These FP8 coaching outcomes are practically 1.5x quicker than our outcomes with BF16. SynapseAI 1.13 can be anticipated to deliver a carry in LLM inference efficiency.

Learn on for particulars on the Intel Gaudi platform, LLM coaching and inference outcomes, LLM convergence outcomes, and extra. And, in fact, ensure to ask your cloud service supplier when they are going to be providing Intel^R Gaudi^R 2 (accessible now) and Gaudi 3, coming in 2024.

Intel^R Gaudi^R 2 {Hardware}

The Intel^R Gaudi^R 2 accelerator helps each deep studying coaching and inference for AI fashions like LLMs. The Intel^R Gaudi^R 2 accelerator is constructed on a 7nm course of expertise. It has a heterogeneous compute structure that features twin matrix multiplication engines (MME) and 24 programmable tensor processor cores (TPC). When in comparison with common cloud accelerators in the identical era, such because the NVIDIA A100-40GB and A100-80GB, the Gaudi 2 has extra reminiscence (96GB of HBM2E), greater reminiscence bandwidth, and better peak FLOP/s. Be aware that the AMD MI250 has greater specs per chip however is available in smaller system configurations of solely 4xMI250. In Desk 1a) we examine the Intel^R Gaudi^R 2 in opposition to the NVIDIA A100 and AMD MI250, and in Desk 1b) we examine the upcoming Intel Gaudi 3 in opposition to the NVIDIA H100/H200 and AMD MI300X.

The Intel^R Gaudi^R 2 accelerator ships in programs of 8x Gaudi 2 accelerators and has a singular scale-out networking design constructed on RDMA over Converged Ethernet (RoCEv2). Every Gaudi 2 accelerator integrates 24 x 100 Gbps Ethernet ports, with 21 ports devoted to all-to-all connectivity inside a system (3 hyperlinks to every of the seven different gadgets) and the remaining three hyperlinks devoted to scaling out throughout programs. Total, this implies every 8x Gaudi 2 system has 3 [scale-out links] * 8 [accelerators] * 100 [Gbps] = 2400 Gbps of exterior bandwidth. This design permits quick, environment friendly scale-out utilizing commonplace Ethernet {hardware}.

At Databricks, we obtained entry to a big multi-node Intel^R Gaudi^R 2 cluster by way of the Intel Developer Cloud (IDC), which was used for all profiling executed on this weblog.

Desk 1a (prime) and Desk 1b (backside): {Hardware} specs for NVIDIA, AMD, and Intel accelerators. Please observe that solely a subset of specs have been launched publicly as of Dec 2023. The TFLOP/s numbers for Gaudi 2 are estimated utilizing microbenchmarks run by Databricks, and the TFLOP/s and Reminiscence Bandwidth for Gaudi 3 are projected utilizing public data from Intel.

Intel SynapseAI Software program

Intel® Gaudi SynapseAI software program suite crucially permits PyTorch packages to run seamlessly on Gaudi gadgets with minimal modifications. SynapseAI gives a graph compiler, a runtime, optimized kernels for tensor operations, and a collective communication library. Determine 1 exhibits Intel Gaudi’s software program stack with all its key parts. Extra particulars about SynapseAI can be found within the official documentation.

One of the common libraries for ML programmers is PyTorch, and builders ought to be excited to know that PyTorch runs in keen mode or lazy mode on Intel accelerators like Gaudi 2. All you’ll want to do is set up the Gaudi-provided construct of PyTorch (set up matrix right here) or begin from one in all their Docker pictures. Typically, working HuggingFace/PyTorch code on Gaudi {hardware} requires minimal code modifications. These modifications usually embody substituting `tensor.cuda()` instructions with `tensor.to('hpu')` instructions or utilizing a PyTorch coach like Composer or HF Coach, which already has Gaudi assist inbuilt!

LLM Coaching Efficiency

For single-node LLM coaching efficiency, we discovered that the Intel^R Gaudi^R 2 is the 2nd-fastest AI chip available on the market right now, reaching over 260 TFLOP/s/machine with SynapseAI 1.12 and BF16 combined precision coaching. See Determine 2 for a comparability of the Intel^R Gaudi^R 2 vs. NVIDIA and AMD GPUs.

On every platform, we ran the identical coaching scripts from LLM Foundry utilizing MPT fashions with a sequence size of 2048, BF16 combined precision, and the ZeRO Stage-3 distributed coaching algorithm. On NVIDIA or AMD programs, this algorithm is carried out by way of PyTorch FSDP with sharding_strategy: FULL_SHARD. On Intel programs, that is at present executed by way of DeepSpeed ZeRO with `stage: 3`, however FSDP assist is predicted to be added within the close to future.

On every system, we additionally used probably the most optimized implementation of scalar-dot-product-attention (SDPA) accessible:

• NVIDIA: Triton FlashAttention-2
• AMD: ROCm ComposableKernel FlashAttention-2
• Intel: Gaudi TPC FusedSDPA

Lastly, we tuned the microbatch measurement for every mannequin on every system to attain most efficiency. All coaching efficiency measurements are reporting mannequin FLOPs somewhat than {hardware} FLOPs. See our benchmarking README for extra particulars.

When evaluating the Intel^R Gaudi^R 2 in opposition to the same-generation NVIDIA A100 and AMD MI250 GPUs, we discovered that Gaudi 2 is a transparent winner, with a mean speedup of 1.22x vs. the A100-80GB, 1.34x vs. the A100-40GB, and 1.59x vs. the MI250.

Compared in opposition to the NVIDIA H100, there may be nonetheless a major hole to shut: on common, the Intel^R Gaudi^R 2 performs at 0.55x the H100. Wanting ahead, the next-gen Intel Gaudi 3 has been publicly introduced to have 4x the BF16 efficiency of the Gaudi 2 and 1.5x the HBM bandwidth, making it a powerful competitor to the H100.

Though this weblog focuses on BF16 coaching, we anticipate that FP8 coaching will quickly change into a typical amongst main accelerators (NVIDIA H100/H200, AMD MI300X, Intel^R Gaudi^R 2/Gaudi 3), and we sit up for reporting on these numbers in future blogs. For the Intel^R Gaudi^R 2, FP8 coaching is supported, starting with SynapseAI model 1.13. For preliminary FP8 coaching outcomes for the NVIDIA H100 and Intel^R Gaudi^R 2, take a look at the MLPerf Coaching 3.1 GPT3 outcomes.

Now, efficiency is nice, however what about performance-per-dollar? Fortunately, pricing for each NVIDIA and Intel accelerators is publicly accessible and can be utilized to compute a mean coaching performance-per-dollar. Be aware that the costs listed under are on-demand hourly charges and will not replicate costs after discounting, which might be cloud service supplier (CSP) and buyer particular:

Desk 2: Coaching Efficiency-per-Greenback for numerous AI accelerators accessible in Amazon Internet Companies (AWS) and Intel Developer Cloud (IDC).

Based mostly on these public on-demand quoted costs from AWS and IDC, we discovered that the Intel^R Gaudi^R 2 has the greatest coaching performance-per-dollar, with a mean benefit of 4.8x vs the NVIDIA A100-80GB, 4.2x vs. the NVIDIA A100-40GB, and 5.19x vs. the NVIDIA H100. As soon as once more, these comparisons might range based mostly on customer-specific reductions on completely different cloud suppliers.

Transferring on to multi-node efficiency, the Intel^R Gaudi^R 2 cluster has nice scaling efficiency for LLM coaching. In Determine 3, we educated an MPT-7B mannequin with fastened world prepare batch measurement samples on [1, 2, 4, 10, 20] nodes and noticed a efficiency of 262 TFLOP/s/machine at one node (8x Gaudi 2) to 244 TFLOP/s/machine at 20 nodes (160x Gaudi 2). Be aware that for these outcomes, we used DeepSpeed with ZeRO Stage-2 somewhat than Stage-3 for max multi-node efficiency.

For extra particulars about our coaching configs and the way we measured efficiency, see our public LLM Foundry coaching benchmarking web page.

Given these outcomes, along with Intel’s MLPerf Coaching 3.1 outcomes coaching GPT3-175B on as much as 384x Gaudi 2, we’re optimistic about Intel^R Gaudi^R 2 efficiency at greater machine counts, and we sit up for sharing outcomes on bigger Intel^R Gaudi^R  2/Gaudi 3 clusters sooner or later.

LLM Convergence

To check coaching stability on Intel^R Gaudi^R 2, we educated MPT-1B, MPT-3B, and MPT-7B fashions from scratch on the C4 dataset utilizing Chinchilla-optimal token budgets to substantiate that we may efficiently prepare high-quality fashions on the Intel Gaudi platform.

We educated every mannequin on the multi-node Gaudi 2 cluster with BF16 combined precision and DeepSpeed ZeRO Stage-2 for max efficiency. The 1B, 3B, and 7B fashions had been educated on 64, 128, and 128 Gaudi 2 accelerators, respectively. Total, we discovered that coaching was secure, and we noticed no points attributable to floating level numerics.

Once we evaluated the ultimate fashions on commonplace in-context-learning (ICL) benchmarks, we discovered that our MPT fashions educated on Intel^R Gaudi^R 2 achieved related outcomes to the open-source Cerebras-GPT fashions. These are a household of open-source fashions educated with the identical parameter counts and token budgets, permitting us to check mannequin high quality instantly. See Desk 2 for outcomes. All fashions had been evaluated utilizing the LLM Foundry eval harness and utilizing the identical set of prompts.

These convergence outcomes give us confidence that prospects can prepare high-quality LLMs on Intel AI Accelerators.

Desk 2: Coaching particulars and analysis outcomes for MPT-[1B, 3B, 7B] vs. Cerebras-GPT-[1.3B, 2.7B, 6.7B]. Every pair of fashions is educated with related configurations and reaches related zero-shot accuracies (greater is healthier) on commonplace in-context-learning (ICL) duties.

LLM Inference

Transferring on to inference, we leveraged the Optimum Habana package deal to run inference benchmarks with LLMs from the HuggingFace Transformer library on Gaudi 2 {hardware}. Optimum Habana is an environment friendly middleman, facilitating seamless integration between the HuggingFace Transformer library and Gaudi 2 structure. This compatibility permits for direct, out-of-the-box deployment of supported HuggingFace fashions on Gaudi 2 platforms. Notably, after the required packages had been put in, we noticed that the LLaMa2 mannequin household was appropriate with the Gaudi 2 atmosphere, enabling inference benchmarking to proceed easily with out requiring any modifications.

For our inference benchmarking on Nvidia’s A100 and H100 gadgets, we utilized TRT-LLM (most important department as of 16/11/2023 when the experiments had been executed), a complicated inference answer just lately launched for LLMs. TRT-LLM is specifically designed to optimize the inference efficiency of LLMs on Nvidia {hardware}, leveraging the very best accessible software program from numerous NVIDIA libraries. We use the commonplace benchmark config offered by NVidia’s TRT-LLM library to acquire these numbers. The usual benchmark config runs Llama-70B with GPT consideration and GEMM plugins utilizing the FP16 datatype. Please observe that every one comparative numbers are executed with BF16 datatype on each the {hardware}. FP8 is offered on each the {hardware} however not used for benchmarking right here. We hope to make the outcomes of FP8 experiments accessible in a future weblog.

Determine 5 exhibits the latency-throughput curves for the Llama2-70B mannequin for all three profiled {hardware} configurations. We used the identical mannequin and the identical enter/output lengths on all three {hardware} configurations. We used BF16 mannequin weights and math for all programs. This plot permits customers to estimate the achievable throughput on specified {hardware} configurations based mostly on a set latency goal. 8x Gaudi 2 sits between 8x A100-80GB and 8x H100 based mostly on these outcomes. Nonetheless, for smaller batch sizes (i.e., low latency regime), 8x Gaudi 2 is way nearer to 8xH100. Additional evaluation exhibits that prefill time (i.e., immediate processing time) is way slower on Gaudi 2 than H100, whereas token era time on Gaudi 2 is just about an identical, as proven within the subsequent figures. For instance, for 64 concurrent customers, the prefill time on H100 is 1.62 secs, whereas on Gaudi 2 it’s 2.45 secs. Slower prefill time might be improved by incorporating higher software program optimizations, however that is principally anticipated because of the giant hole in peak FLOPs (989.5 for H100 and ~400 for Gaudi 2). The subsequent-gen Gaudi 3 is predicted to shut this hole.

Determine 6 illustrates the time required for every {hardware} configuration to generate a single token per person. Notably, it highlights a key development: Time Per Output Token (TPOT) per person is +/-8% for Intel^R Gaudi^R 2 vs. NVIDIA H100 underneath many of the person load eventualities. In some settings, the Gaudi 2 is quicker, and in different settings, the H100 is quicker. Basically, throughout all {hardware}, because the variety of concurrent customers will increase, the time taken to generate every token additionally will increase, resulting in lowered efficiency for each person. This correlation gives beneficial insights into the scalability and effectivity of various {hardware} configurations underneath various person masses.

In Determine 7, we present a metric known as Mannequin Bandwidth Utilization (MBU). The idea of MBU is elaborated in higher element in our earlier LLM Inference weblog submit. Briefly, it denotes the measure of effectivity in eventualities the place efficiency is primarily constrained by reminiscence bandwidth. The method of token era at smaller batch sizes predominantly hinges on achievable reminiscence bandwidth. Efficiency on this context is closely influenced by how successfully the software program and {hardware} can make the most of the height reminiscence bandwidth accessible. In our experiments, we noticed that the Intel^R Gaudi^R 2 platform demonstrates superior MBU in comparison with the H100 for token era duties. This means that Intel^R Gaudi^R 2 is at present extra environment friendly at leveraging accessible reminiscence bandwidth than the A100-80GB or H100.

Lastly, in Determine 8, we illustrate the price of producing 1 million tokens per person throughout completely different {hardware} platforms. For example, with 4 concurrent customers, the entire price depicted is for 4 million tokens (1 million per person). This calculation relies on the pricing particulars offered in Desk 2.

Key insights from this information embody:

1. H100 is costlier than A100, primarily as a result of its hourly price is greater than double, whereas its efficiency features don’t proportionately match this enhance (as detailed in Determine 8).
2. Gaudi 2 stands out attributable to its aggressive pricing, supplied by IDC, and efficiency ranges corresponding to H100, making it a compelling possibility.

It is very important observe that not one of the benchmarked {hardware} makes use of FP8 (accessible on H100 and Gaudi 2) or quantization.

What’s Subsequent?

On this weblog, we have demonstrated that the Intel^R Gaudi^R 2 is a compelling possibility for LLM coaching and inference. Single-node and multi-node coaching on Gaudi 2 works nice. Efficiency-per-chip is the 2nd highest out of any accelerator now we have examined, and the performance-per-dollar is the very best when evaluating on-demand pricing on AWS and IDC. For convergence testing, we educated Chinchilla-style 1B, 3B, and 7B parameter fashions from scratch and located that the ultimate mannequin qualities matched or exceeded reference open-source fashions.

On the inference facet, we discovered that the Gaudi 2 punches above its weight, outperforming the NVIDIA A100 in all settings and matching the NVIDIA H100 in decoding latency (the most costly part of LLM inference). That is due to the Intel^R Gaudi^R 2 software program and {hardware} stack, which achieves greater reminiscence bandwidth utilization than each NVIDIA chips. Placing it one other manner, with solely 2450 GB/s of HBM2E reminiscence bandwidth, the Gaudi 2 matches the H100 with 3350 GB/s of HBM3 bandwidth.

We consider these outcomes strengthen the AI story for Intel^R Gaudi^R 2 accelerators, and due to the interoperability of PyTorch and open-source libraries (e.g., DeepSpeed, Composer, StreamingDataset, LLM Foundry, Habana Optimum), customers can run the identical LLM workloads on NVIDIA, AMD, or Intel and even swap between the platforms.

Waiting for the Intel Gaudi 3, we count on the identical interoperability however with even greater efficiency. The projected public data specs (See Desk 1b) counsel that Intel Gaudi 3 ought to have extra FLOP/s and reminiscence bandwidth than all the main opponents (NVIDIA H100, AMD MI300X). Given the nice coaching and inference utilization numbers we already see right now on Gaudi 2, we’re very enthusiastic about Gaudi 3 and sit up for profiling it when it arrives. Keep tuned for future blogs on Intel Gaudi accelerators and coaching at a good bigger scale.

1. https://www.mosaicml.com/weblog/coreweave-nvidia-h100-part-1
2. https://www.databricks.com/weblog/llm-inference-performance-engineering-best-practices
3. https://www.databricks.com/weblog/training-llms-scale-amd-mi250-gpus
4. https://www.mosaicml.com/weblog/amd-mi250