{"id":85277,"date":"2021-03-01T15:45:35","date_gmt":"2021-03-01T23:45:35","guid":{"rendered":"https:\/\/cloudblogs.microsoft.com\/opensource\/?p=85277"},"modified":"2025-06-23T08:42:19","modified_gmt":"2025-06-23T15:42:19","slug":"optimizing-bert-model-for-intel-cpu-cores-using-onnx-runtime-default-execution-provider","status":"publish","type":"post","link":"https:\/\/opensource.microsoft.com\/blog\/2021\/03\/01\/optimizing-bert-model-for-intel-cpu-cores-using-onnx-runtime-default-execution-provider\/","title":{"rendered":"Optimizing BERT model for Intel CPU Cores using ONNX runtime default execution provider"},"content":{"rendered":"\n

This blog was co-authored with Manash Goswami, Principal Program Manager, Machine Learning Platform.<\/em><\/p>\n\n\n\n

\"A<\/figure>\n\n\n\n

The performance improvements provided by ONNX Runtime powered by Intel\u00ae Deep Learning Boost: Vector Neural Network Instructions (Intel\u00ae DL Boost: VNNI) greatly improves performance of machine learning model execution for developers. In the past, machine learning models mostly relied on 32-bit floating point instructions using AVX512. Now, machine learning models can use 8-bit integer instructions (Intel\u00ae DL Boost: VNNI) to achieve substantial speed increases without significant loss of accuracy. To fully understand these performance improvements, you must first understand ONNX Runtime, Bi-Directional Encoder Representations from Transformers (BERT), Intel DL Boost: VNNI, and steps to achieve the best performance with ONNX Runtime on Intel platforms. Keep reading to learn more about accelerating BERT model inference with ONNX Runtime and Intel\u00ae DL Boost: VNNI.<\/p>\n\n\n\n

What is ONNX Runtime?<\/h2>\n\n\n\n

ONNX Runtime<\/a> is an open-source project that is designed to accelerate machine learning across a wide range of frameworks, operating systems, and hardware platforms. It enables acceleration of machine learning inferencing across all of your deployment targets using a single set of APIs.1<\/sup>Intel has partnered with the Microsoft ONNX Runtime team to add support for Intel\u00ae DL Boost and take advantage of microarchitectural improvements, such as non-exclusive caches on the new 11th Gen Intel\u00ae Core\u2122 processors to significantly improve performance. Read more to learn how to achieve the best performance using Intel\u00ae Deep Learning Boost: VNNI on ONNX Runtime\u2019s default CPU backend (Microsoft Linear Algebra Subroutine (MLAS)).<\/p>\n\n\n

\"ONNX<\/figure>\n\n\n\n

Figure 1: ONNX Runtime Architecture<\/em><\/p>\n\n\n\n

What is BERT?<\/h2>\n\n\n\n

BERT was originally created and published in 2018 by Jacob Devlin and his colleagues at Google. It\u2019s a machine learning technique that greatly improves machine natural language processing (NLP) capabilities. This technique does not process individual words (as previously done), but instead, it processes complete sentences. Machine learning models can now understand the relationship between words within a sentence and understand the context of a sentence. This approach to neuro-linguistic programming (NLP) has revolutionized language processing tasks such as search, document classification, question answering, sentence similarity, text prediction, and more. BERT class models are widely applied in the industry. Recently techniques such as knowledge distillation and quantization have been successfully applied to BERT, making this model deployable on Windows PCs.<\/p>\n\n\n\n

What is Deep Learning Boost: VNNI?<\/h2>\n\n\n\n

Intel Deep Learning Boost: VNNI is designed to deliver significant deep learning acceleration, as well as power-saving optimizations. A single vector instruction (such as VPDPBUSD) can be used to multiply two 8-bit integers and combining the result into a 32-bit output.<\/p>\n\n\n\n

\"A<\/figure>\n\n\n\n

Steps to build and execute ONNX Runtime for Windows 10 on 11th<\/sup> Gen Intel\u00ae Core\u2122 Processors<\/h2>\n\n\n\n

Pre-requisites:<\/p>\n\n\n\n