{"id":94179,"date":"2023-01-25T09:00:00","date_gmt":"2023-01-25T17:00:00","guid":{"rendered":"https:\/\/cloudblogs.microsoft.com\/opensource\/?p=94179"},"modified":"2024-06-19T10:50:42","modified_gmt":"2024-06-19T17:50:42","slug":"improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure","status":"publish","type":"post","link":"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/","title":{"rendered":"Improve BERT inference speed by combining the power of Optimum, OpenVINO\u2122, ONNX Runtime, and Azure"},"content":{"rendered":"\n

In this blog, we will discuss one of the ways to make huge models like BERT smaller and faster with OpenVINO\u2122 Neural Networks Compression Framework (NNCF) and ONNX Runtime with OpenVINO\u2122 Execution Provider through Azure Machine Learning<\/a>.<\/p>\n\n\n

\n\t
\n\t\t
\n\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\"a\t\t\t\t<\/div>\n\t\t\t\n\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\n\t\t\t\t\t

Azure Machine Learning<\/h2>\n\n\t\t\t\t\t
\n\t\t\t\t\t\t

Business-critical machine learning models at scale.<\/p>\n\t\t\t\t\t<\/div>\n\n\t\t\t\t\t\t\t\t\t\t\t

\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\tLearn more<\/span>\n\t\t\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t<\/div>\n\n\t\t\t\t\t<\/div>\n\t<\/div>\n<\/div>\n\n\n\n

Big models are slow, we need to make them faster<\/h2>\n\n\n\n

Today’s best-performing language processing models use huge neural architectures with hundreds of millions of parameters. State-of-the-art transformer-based architectures like BERT are available as pretrained models for anyone to use for any language task.<\/p>\n\n\n\n

The big models have outstanding accuracy, but they are difficult to use in practice. These models are resource hungry due to a large number of parameters. These issues become worse when serving the fine-tuned model and it requires a lot of memory and time to process a single message. A state-of-the-art model is not good if it can handle only one message per second. To improve the throughput, we need to accelerate the well-performing BERT model, by reducing the computation or the number of operations with the help of quantization.<\/p>\n\n\n\n

Overview of Optimum Intel and quantization aware training<\/h2>\n\n\n\n

Optimum Intel is an extension for the Hugging Face Optimum library with OpenVINO\u2122 runtime as a backend for the Transformers architectures. It also provides an interface to Intel\u00ae NNCF (Neural Network Compression Framework) package. It helps implement Intel\u2019s optimizations through NNCF with changes to just a few lines of code in the training pipeline.<\/p>\n\n\n\n

Quantization aware training (QAT) is a widely used technique for optimizing models during training. It inserts nodes into the neural network during training that simulates the effect of lower precision. This allows the training algorithm to consider quantization errors as part of the overall training loss that gets minimized during training. QAT has better accuracy and reliability than carrying out quantization after the model has been trained. The output after training with our tool is a quantized PyTorch model, ONNX model, and IR.xml.<\/p>\n\n\n\n

Overview of ONNXRuntime, and OpenVINO\u2122 Execution Provider<\/h2>\n\n\n\n

ONNX Runtime is an open source project that is designed to accelerate machine learning across a wide range of frameworks, operating systems, languages, and hardware platforms. It enables the acceleration of machine learning inferencing across all of your deployment targets using a single set of APIs.<\/p>\n\n\n\n

Intel and Microsoft joined hands to create the OpenVINO\u2122 Execution Provider<\/a> (OVEP) for ONNX Runtime, which enables ONNX models for running inference using ONNX Runtime APIs while using the OpenVINO\u2122 Runtime as a backend. With the OpenVINO\u2122 Execution Provider, ONNX Runtime delivers better inferencing performance on the same hardware compared to generic acceleration on Intel\u00ae CPU, GPU, and VPU. Now you\u2019ve got a basic understanding of quantization, ONNX Runtime, and OVEP, let’s take the best of both worlds and stitch the story together.<\/p>\n\n\n\n

Putting the tools together to achieve better performance<\/h2>\n\n\n\n

In our next steps, we will be doing quantization aware training using Optimum-Intel and Inference using Optimum-ORT with OpenVINO\u2122 Execution Provider through Azure Machine Learning. Optimum can be used to load optimized models from the Hugging Face Hub<\/a> and create pipelines to run accelerated inferences.<\/p>\n\n\n\n

Converting PyTorch FP32 model to INT8 ONNX model with QAT<\/h2>\n\n\n\n

When utilizing the Hugging Face training pipelines all you need is to update a few lines of code and you can invoke the NNCF optimizations for quantizing the model. The output of this would be an optimized INT8 PyTorch model, ONNX model, and OpenVINO\u2122 IR. See the flow diagram below:<\/p>\n\n\n

\"An<\/figure>\n\n\n\n

For this case study, we have chosen the bert-squad pretrained model<\/a> from Hugging Face. This has been pretrained on the SQuAD dataset for the question-answering use case. QAT can be applied by replacing the Transformers Trainer<\/a> with the Optimum (OVTrainer). See below:<\/p>\n\n\n

\nfrom trainer_qa import QuestionAnsweringOVTrainer\n\nRun the training pipeline\n1. Import OVConfig:\n\nfrom optimum.intel.openvino import OVConfig\nfrom trainer_qa import QuestionAnsweringOVTrainer\n\n2. Initialize a config from the \n    ov_config = OVConfig()\n    \n3. Initialize our Trainer\n    trainer = QuestionAnsweringOVTrainer()\n<\/pre><\/div>\n\n\n
Comparison of FP32 model and INT8 ONNX model with Netron model visualization tool<\/h2>\n\n\n\n
When compared with FP32, the INT8 model has QuantizeLinear<\/strong> and DequantizeLinear<\/strong> operations added to mimic the lower precision after the QAT.<\/p>\n\n\n\n
\n
\"A
Fig1: FP32 model<\/strong><\/em><\/figcaption><\/figure>\n<\/div>\n\n\n
\"An
Fig2: INT8 model<\/strong><\/em><\/figcaption><\/figure>\n\n\n\n
<\/p>\n<\/div>\n\n\n\n
To replicate this example check out the reference code with detailed instructions on QAT and Inference using OpenVINO<\/a> and Azure Machine Learning Jupyter Notebooks<\/a> on GitHub.<\/p>\n\n\n\n
Performance improvement results<\/h2>\n\n\n\n
Accuracy<\/th>Original FP32<\/th>QAT INT8<\/th>Explanation<\/th><\/tr><\/thead>
F1<\/td>93.1<\/td>92.83<\/td>In this case, it’s computed over the individual words in the prediction against those in the True Answer. The number of shared words between the prediction and the truth is the basis of the F1 score: precision is the ratio of the number of shared words to the total number of words in the prediction, and recall is the ratio of the number of shared words to the total number of words in the ground truth.<\/td><\/tr>
Eval_exact<\/td>86.91<\/td>86.94<\/td>This metric is as simple as it sounds. For each question + answer pair, if the characters<\/em> of the model’s prediction exactly match the characters of (one of) the True Answer(s), EM = 1, otherwise EM = 0. This is a strict all-or-nothing metric; being off by a single character results in a score of 0. When assessing against a negative example, if the model predicts any text at all, it automatically receives a 0 for that example.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Comparison of ONNXRUNTIME_PERF_TEST application for ONNX-FP32 and ONNX-INT8 models<\/h2>\n\n\n\n
We\u2019ve used ONNXRuntime APIs for running inference for the BERT model. As you can see the performance for the INT8 optimized model improved almost to 2.95x when compared to FP32 without much compromise in the accuracy.<\/p>\n\n\n
\"A<\/figure>\n\n\n\n
Quantized PyTorch, ONNX, and INT8 models can also be served using OpenVINO\u2122 Model Server for high-scalability and optimization for Intel\u00ae solutions so that you can take advantage of all the power of the Intel\u00ae Xeon\u00ae processor or Intel\u2019s AI accelerators and expose it over a network interface.<\/p>\n\n\n\n
Optimize speed and performance <\/h2>\n\n\n\n
As neural networks move from servers to the edge, optimizing speed and size becomes even more important. In this blog, we gave an overview of how to use open source tooling to make it easy to improve performance.<\/p>\n\n\n\n
References:<\/h3>\n\n\n\n
    \n
  1. Enhanced Low-Precision Pipeline to Accelerate Inference with OpenVINO\u2122 toolkit<\/a>.<\/li>\n\n\n\n
  2. Developer Guide: Model Optimization with the OpenVINO\u2122 Toolkit<\/a>. <\/li>\n\n\n\n
  3. Evaluating QA: Metrics, Predictions, and the Null Response<\/a>.<\/li>\n<\/ol>\n\n\n\n
    SW\/HW configuration<\/h4>\n\n\n\n
    \n
    Framework configuration: <\/strong>ONNXRuntime, Optimum-Intel [NNCF]
    Application configuration: <\/strong>ONNXRuntime, EP: OpenVINO\u2122 .\/onnx_perf_test<\/strong> OPENVINO 2022.2: .\/benchmark_app
    Input:<\/strong> Question and context
    Application Metric:<\/strong> Normalized throughput
    Platform<\/strong>: Intel Icelake-8380
    Number of Nodes<\/strong>: 2
    Number of Sockets<\/strong>: 2
    CPU or Accelerator<\/strong><\/strong>: Intel(R) Xeon(R) Platinum 8380 CPU @ 2.30GHz
    Cores\/socket, Threads\/socket or EU\/socket: <\/strong>40,2
    ucode<\/strong>: 0xd000375
    HT:<\/strong> Enabled
    Turbo: <\/strong>Enabled
    BIOS Version<\/strong>: American Megatrends International, LLC. V1.4
    System DDR Mem Config: slots \/ cap \/ run-speed<\/strong>: 32\/32 GB\/3200 MT\/s
    Total Memory\/Node (DDR+DCPMM)<\/strong>: 1024GB
    Storage \u2013 boot<\/strong>: INTEL_SSDSC2KB019T8 1.8T
    NIC<\/strong>: 2 x Ethernet Controller X710 for 10GBASE-T
    OS<\/strong>: Ubuntu 20.04.4 LTS
    Kernel<\/strong>: 5.15.0-46-generic<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
    Make large models smaller and faster with OpenVino Execution Provider, NNCF and ONNX Runtime leveraging Azure Machine Learning.<\/p>\n","protected":false},"author":5562,"featured_media":95489,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msxcm_post_with_no_image":false,"ep_exclude_from_search":false,"_classifai_error":"","_classifai_text_to_speech_error":"","footnotes":""},"post_tag":[1824,1827],"content-type":[340],"topic":[2238,2252],"programming-languages":[],"coauthors":[1972,1973,1969,1974],"class_list":["post-94179","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","tag-onnx-runtime","tag-transformer","content-type-tutorials-and-demos","topic-ai-machine-learning","topic-tools","review-flag-1593580428-734","review-flag-1593580419-521","review-flag-1-1593580432-963","review-flag-2-1593580437-411","review-flag-3-1593580442-169","review-flag-4-1593580448-609","review-flag-5-1593580453-725"],"yoast_head":"\nOpenVINO\u2122, ONNX Runtime, and Azure improve BERT inference speed | Microsoft Open Source Blog<\/title>\n<meta name=\"description\" content=\"Make large models smaller and faster with OpenVino Execution Provider, NNCF and ONNX Runtime leveraging Azure Machine Learning.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"OpenVINO\u2122, ONNX Runtime, and Azure improve BERT inference speed | Microsoft Open Source Blog\" \/>\n<meta property=\"og:description\" content=\"Make large models smaller and faster with OpenVino Execution Provider, NNCF and ONNX Runtime leveraging Azure Machine Learning.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/\" \/>\n<meta property=\"og:site_name\" content=\"Microsoft Open Source Blog\" \/>\n<meta property=\"article:published_time\" content=\"2023-01-25T17:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-06-19T17:50:42+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/06\/STB13_Michelle_03.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1170\" \/>\n\t<meta property=\"og:image:height\" content=\"640\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Cassie Breviu, Akhila Vidiyala, Devang Aggarwal, Sachin Rastogi\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@OpenAtMicrosoft\" \/>\n<meta name=\"twitter:site\" content=\"@OpenAtMicrosoft\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Cassie Breviu, Akhila Vidiyala, Devang Aggarwal, Sachin Rastogi\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 min read\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/\"},\"author\":[{\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/author\/cassie-breviu\/\",\"@type\":\"Person\",\"@name\":\"Cassie Breviu\"},{\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/author\/akhila-vidiyala\/\",\"@type\":\"Person\",\"@name\":\"Akhila Vidiyala\"},{\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/author\/devang-aggarwal\/\",\"@type\":\"Person\",\"@name\":\"Devang Aggarwal\"},{\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/author\/sachin-rastogi\/\",\"@type\":\"Person\",\"@name\":\"Sachin Rastogi\"}],\"headline\":\"Improve BERT inference speed by combining the power of Optimum, OpenVINO\u2122, ONNX Runtime, and Azure\",\"datePublished\":\"2023-01-25T17:00:00+00:00\",\"dateModified\":\"2024-06-19T17:50:42+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/\"},\"wordCount\":1129,\"publisher\":{\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/06\/STB13_Michelle_03.webp\",\"keywords\":[\"ONNX Runtime\",\"Transformer\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/\",\"url\":\"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/\",\"name\":\"OpenVINO\u2122, ONNX Runtime, and Azure improve BERT inference speed | Microsoft Open Source Blog\",\"isPartOf\":{\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/06\/STB13_Michelle_03.webp\",\"datePublished\":\"2023-01-25T17:00:00+00:00\",\"dateModified\":\"2024-06-19T17:50:42+00:00\",\"description\":\"Make large models smaller and faster with OpenVino Execution Provider, NNCF and ONNX Runtime leveraging Azure Machine Learning.\",\"breadcrumb\":{\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/#primaryimage\",\"url\":\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/06\/STB13_Michelle_03.webp\",\"contentUrl\":\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/06\/STB13_Michelle_03.webp\",\"width\":1170,\"height\":640},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/opensource.microsoft.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Improve BERT inference speed by combining the power of Optimum, OpenVINO\u2122, ONNX Runtime, and Azure\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/#website\",\"url\":\"https:\/\/opensource.microsoft.com\/blog\/\",\"name\":\"Microsoft Open Source Blog\",\"description\":\"Open dialogue about openness at Microsoft \u2013 open source, standards, interoperability\",\"publisher\":{\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/opensource.microsoft.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/#organization\",\"name\":\"Microsoft Open Source Blog\",\"url\":\"https:\/\/opensource.microsoft.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2019\/08\/Microsoft-Logo.png\",\"contentUrl\":\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2019\/08\/Microsoft-Logo.png\",\"width\":259,\"height\":194,\"caption\":\"Microsoft Open Source Blog\"},\"image\":{\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/x.com\/OpenAtMicrosoft\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"OpenVINO\u2122, ONNX Runtime, and Azure improve BERT inference speed | Microsoft Open Source Blog","description":"Make large models smaller and faster with OpenVino Execution Provider, NNCF and ONNX Runtime leveraging Azure Machine Learning.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/","og_locale":"en_US","og_type":"article","og_title":"OpenVINO\u2122, ONNX Runtime, and Azure improve BERT inference speed | Microsoft Open Source Blog","og_description":"Make large models smaller and faster with OpenVino Execution Provider, NNCF and ONNX Runtime leveraging Azure Machine Learning.","og_url":"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/","og_site_name":"Microsoft Open Source Blog","article_published_time":"2023-01-25T17:00:00+00:00","article_modified_time":"2024-06-19T17:50:42+00:00","og_image":[{"width":1170,"height":640,"url":"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/06\/STB13_Michelle_03.png","type":"image\/png"}],"author":"Cassie Breviu, Akhila Vidiyala, Devang Aggarwal, Sachin Rastogi","twitter_card":"summary_large_image","twitter_creator":"@OpenAtMicrosoft","twitter_site":"@OpenAtMicrosoft","twitter_misc":{"Written by":"Cassie Breviu, Akhila Vidiyala, Devang Aggarwal, Sachin Rastogi","Est. reading time":"5 min read"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/#article","isPartOf":{"@id":"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/"},"author":[{"@id":"https:\/\/opensource.microsoft.com\/blog\/author\/cassie-breviu\/","@type":"Person","@name":"Cassie Breviu"},{"@id":"https:\/\/opensource.microsoft.com\/blog\/author\/akhila-vidiyala\/","@type":"Person","@name":"Akhila Vidiyala"},{"@id":"https:\/\/opensource.microsoft.com\/blog\/author\/devang-aggarwal\/","@type":"Person","@name":"Devang Aggarwal"},{"@id":"https:\/\/opensource.microsoft.com\/blog\/author\/sachin-rastogi\/","@type":"Person","@name":"Sachin Rastogi"}],"headline":"Improve BERT inference speed by combining the power of Optimum, OpenVINO\u2122, ONNX Runtime, and Azure","datePublished":"2023-01-25T17:00:00+00:00","dateModified":"2024-06-19T17:50:42+00:00","mainEntityOfPage":{"@id":"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/"},"wordCount":1129,"publisher":{"@id":"https:\/\/opensource.microsoft.com\/blog\/#organization"},"image":{"@id":"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/#primaryimage"},"thumbnailUrl":"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/06\/STB13_Michelle_03.webp","keywords":["ONNX Runtime","Transformer"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/","url":"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/","name":"OpenVINO\u2122, ONNX Runtime, and Azure improve BERT inference speed | Microsoft Open Source Blog","isPartOf":{"@id":"https:\/\/opensource.microsoft.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/#primaryimage"},"image":{"@id":"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/#primaryimage"},"thumbnailUrl":"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/06\/STB13_Michelle_03.webp","datePublished":"2023-01-25T17:00:00+00:00","dateModified":"2024-06-19T17:50:42+00:00","description":"Make large models smaller and faster with OpenVino Execution Provider, NNCF and ONNX Runtime leveraging Azure Machine Learning.","breadcrumb":{"@id":"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/#primaryimage","url":"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/06\/STB13_Michelle_03.webp","contentUrl":"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/06\/STB13_Michelle_03.webp","width":1170,"height":640},{"@type":"BreadcrumbList","@id":"https:\/\/opensource.microsoft.com\/blog\/2023\/01\/25\/improve-bert-inference-speed-by-combining-the-power-of-optimum-openvino-onnx-runtime-and-azure\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/opensource.microsoft.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Improve BERT inference speed by combining the power of Optimum, OpenVINO\u2122, ONNX Runtime, and Azure"}]},{"@type":"WebSite","@id":"https:\/\/opensource.microsoft.com\/blog\/#website","url":"https:\/\/opensource.microsoft.com\/blog\/","name":"Microsoft Open Source Blog","description":"Open dialogue about openness at Microsoft \u2013 open source, standards, interoperability","publisher":{"@id":"https:\/\/opensource.microsoft.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/opensource.microsoft.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/opensource.microsoft.com\/blog\/#organization","name":"Microsoft Open Source Blog","url":"https:\/\/opensource.microsoft.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/opensource.microsoft.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2019\/08\/Microsoft-Logo.png","contentUrl":"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2019\/08\/Microsoft-Logo.png","width":259,"height":194,"caption":"Microsoft Open Source Blog"},"image":{"@id":"https:\/\/opensource.microsoft.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/OpenAtMicrosoft"]}]}},"msxcm_display_generated_audio":false,"msxcm_animated_featured_image":null,"distributor_meta":false,"distributor_terms":false,"distributor_media":false,"distributor_original_site_name":"Microsoft Open Source Blog","distributor_original_site_url":"https:\/\/opensource.microsoft.com\/blog","push-errors":false,"_links":{"self":[{"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/posts\/94179","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/users\/5562"}],"replies":[{"embeddable":true,"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/comments?post=94179"}],"version-history":[{"count":1,"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/posts\/94179\/revisions"}],"predecessor-version":[{"id":95746,"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/posts\/94179\/revisions\/95746"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/media\/95489"}],"wp:attachment":[{"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/media?parent=94179"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/post_tag?post=94179"},{"taxonomy":"content-type","embeddable":true,"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/content-type?post=94179"},{"taxonomy":"topic","embeddable":true,"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/topic?post=94179"},{"taxonomy":"programming-languages","embeddable":true,"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/programming-languages?post=94179"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/coauthors?post=94179"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}