{"id":94525,"date":"2023-06-26T08:00:00","date_gmt":"2023-06-26T15:00:00","guid":{"rendered":""},"modified":"2023-08-01T17:35:34","modified_gmt":"2023-08-02T00:35:34","slug":"automate-optimization-techniques-for-transformer-models","status":"publish","type":"post","link":"https:\/\/opensource.microsoft.com\/blog\/2023\/06\/26\/automate-optimization-techniques-for-transformer-models\/","title":{"rendered":"Automate optimization techniques for transformer models"},"content":{"rendered":"\n

Olive <\/a>is an easy-to-use hardware-aware model optimization tool by Microsoft which builds up a unified optimization framework to enable independent hardware vendors (IHVs) extend their capabilities to include their state-of-the-art and hardware-specific optimization toolchains. The Intel\u00ae Neural Compressor is an open-source library supporting popular advanced model compression technologies, from techniques used in the industry to the latest state-of-the-art from research. Intel has collaborated with Microsoft to integrate Intel\u00ae Neural Compressor into Olive, enabling developers to easily take advantage of model compression techniques in their deployment platform, including Intel processors and accelerators.<\/p>\n\n\n\n

The rest of this blog is organized as follows:<\/p>\n\n\n\n

1) We first provide an introduction to the toolchains, including Intel Neural Compressor<\/a>.<\/p>\n\n\n\n

2) We then walk through a step-by-step example, on how to optimize popular workloads like transformer-based models from Hugging Face.<\/p>\n\n\n\n

3) Finally, we conclude by summarizing performance gains and accuracy results obtained by using Olive and Intel Neural Compressor to optimize GPT-J, Bert-Base, and RoBERTa.<\/p>\n\n\n\n

Olive with Intel\u00ae Neural Compressor<\/h2>\n\n\n\n

Olive is a user-friendly tool for optimizing models with hardware awareness. It combines top-notch techniques across model compression, optimization, and compilation. By considering a given model, target hardware, as well as deployment constraints such as accuracy and latency, Olive tunes the most suitable optimization techniques to generate highly efficient models for inferencing across operating environments, platforms, and devices.<\/p>\n\n\n\n

Intel\u00ae Neural Compressor provides popular model compression techniques such as quantization, pruning (sparsity), distillation, and neural architecture search for mainstream frameworks<\/a> such as PyTorch*, TensorFlow*, and ONNX Runtime.<\/p>\n\n\n\n

By integrating the Intel\u00ae Neural Compressor quantizer into Olive\u2019s optimizations, as shown in figure 1, <\/em>developers can benefit from automatic selection of optimizations based on their target deployment platform. For example, developers can easily optimize their models with Olive to take advantage of Intel hardware acceleration technologies such as Intel\u00ae Advanced Matrix Extensions (Intel\u00ae AMX) and Intel\u00ae DL Boost. With Intel\u00ae DL Boost, developers can get up to 4 times theoretical performance speedups compared to the FP32 baseline models, and even higher speedups when using Intel\u00ae AMX while meeting your model accuracy requirements.<\/p>\n\n\n

\"here<\/figure>\n\n\n\n

Figure 1: Olive Architecture<\/p>\n\n\n\n

Over two times speedup with Hugging Face transformer models<\/h2>\n\n\n\n

Model Enabling<\/h3>\n\n\n\n

In this section, we\u2019ll explain how to optimize Hugging Face Bert-Base<\/a> model. We\u2019ll leverage Intel\u00ae Neural Compressor\u2019s quantization capabilities, which supports static and dynamic, and have been integrated into Olive. Olive model optimization workflows are defined using JSON files and each optimization is named as a pass. Intel\u00ae Neural Compressor\u2019s quantization techniques have been incorporated into Olive as a single pass named, \u2018IncQuantization\u2019, providing developers with the ability to tune quantization methods and hyperparameters at the same time.  <\/p>\n\n\n\n

The following steps illustrate how developers can use Olive with \u2018IncQuantization\u2019 to accelerate a Hugging Face Bert-Base model:<\/p>\n\n\n\n

1. Add \u2018IncQuantization\u2019 to \u2018passes\u2019 in a config.json file<\/p>\n\n\n

\"code<\/figure>\n\n\n\n

Figure 2: IncQuantization pass<\/p>\n\n\n\n

2. Define the dataset and dataloader needed by \u2018IncQuantizaton\u2019 for model calibration and validation. To achieve this, Olive provides the flexibility for developers to feed their data via a separate Python file: \u2018user_script.py\u2019. The following shows how IncQuantizaton users feed calibration data to tune and validate quantization accuracy.<\/p>\n\n\n

\"code<\/figure>\n\n\n\n

Figure 3: Dataloader<\/p>\n\n\n\n

3. Run the Olive optimization workflow using the config file.<\/p>\n\n\n\n