{"id":94513,"date":"2023-06-26T09:00:00","date_gmt":"2023-06-26T16:00:00","guid":{"rendered":""},"modified":"2023-09-01T11:37:51","modified_gmt":"2023-09-01T18:37:51","slug":"olive-a-user-friendly-toolchain-for-hardware-aware-model-optimization","status":"publish","type":"post","link":"https:\/\/opensource.microsoft.com\/blog\/2023\/06\/26\/olive-a-user-friendly-toolchain-for-hardware-aware-model-optimization\/","title":{"rendered":"Olive: A user-friendly toolchain for hardware-aware model optimization"},"content":{"rendered":"\n

Hardware-aware model optimization is the process of optimizing machine learning models to make the most efficient use of specific hardware architectures\u2014like CPUs, GPUs, and neural processing units (NPUs)\u2014to meet production requirements such as accuracy, latency, and throughput. However, it can be challenging. Firstly, it requires expertise in various independent hardware vendor (IHV) toolkits to handle the unique characteristics and optimizations needed for each hardware architecture. Secondly, aggressive optimizations can have an impact on model quality, balancing accuracy and efficiency within hardware constraints needs to be carefully managed. Additionally, the rapidly evolving hardware landscape necessitates constant updates and adaptations.  <\/p>\n\n\n\n

To alleviate this burden, we introduce Olive<\/a>, an easy-to-use toolchain for optimizing models with hardware awareness. With Olive, you don’t need to be an expert to explore diverse hardware optimization toolchains. It handles the complex optimization process for you, ensuring you achieve the best possible performance without the hassle. <\/p>\n\n\n\n

Easing model optimization across hardware with Olive<\/h2>\n\n\n\n

As a hardware-aware model optimization solution, Olive composes effective techniques in model compression, optimization, and compilation. As shown in Figure 1, for a given model and target hardware, Olive intelligently tunes the most appropriate optimization techniques to generate highly efficient models for inference. Currently, a range of optimization techniques is supported in Olive, including model quantization tuning, transformer optimization, ONNX Runtime performance tuning, and more. Moreover, Olive considers various constraints such as accuracy and latency to ensure the optimized models meet your specific requirements. Olive streamlines the process of optimizing machine learning models to make the most efficient use of specific hardware architectures. Whether you’re working on cloud-based applications or edge devices, Olive enables you to optimize your models effortlessly and effectively. It works with ONNX Runtime<\/a>, a high-performance inference engine, as an end-to-end inference optimization solution.  <\/p>\n\n\n

\"here<\/figure>\n\n\n\n

Figure 1: Olive architecture<\/em> <\/p>\n\n\n\n

By providing a configuration file specifying your model and scenario-specific information, Olive tunes optimization techniques to generate the optimal model(s) on the Pareto frontier based on the metrics goal you set. When working with the configuration file, you typically need to provide information about the input model\u2014including input names, shapes, and the location where the model is stored. Moreover, you specify your performance preferences, such as desired latency, accuracy, or other relevant factors. In addition to this information, you can choose from a range of optimizations<\/a> provided by Olive that you wish to apply to your specific hardware target. You also have the option to define the target hardware and utilize any additional features offered by Olive. By utilizing the configuration file, all you need to do is execute a simple command, eliminating the need for any Python code. \u00a0<\/p>\n\n\n\n

python -m olive.workflows.run --config my_model_acceleration_description.json<\/strong> <\/code><\/p>\n\n\n\n

Here are comprehensive examples<\/a> that demonstrate the process of optimizing models with Olive for various hardware targets. During the Microsoft Build 2023<\/a> conference, we showcase how Olive and the ONNX Runtime (ORT) optimize a whisper model<\/a>, demonstrating a remarkable reduction in end-to-end latency by over two times on Intel Xeon device and a decrease in model size by 2.25 times, as shown in Figure 2. <\/p>\n\n\n

\"The<\/figure>\n\n\n\n

Figure 2: Whisper model optimization with Olive and ORT<\/em> <\/p>\n\n\n\n

Intel and AMD optimization innovations in Olive <\/h2>\n\n\n\n

In addition to simplifying the model optimization experience for model developers, Olive also provides a unified framework that allows industry experts to plug in their own optimization innovations as optimization passes into Olive, resulting in a comprehensive and ready-to-use solution. Intel and AMD have integrated their optimization innovations in Olive. Learn more about contributing your optimization techniques<\/a>.\u00a0<\/p>\n\n\n\n