![\"Open](\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2022\/11\/PreviewImage-2-1024x577.png\")
\t\t\t\t<\/div>\n\t\t\t\n\t\t\tPyTorch on Azure<\/h2>\n\n\t\t\t\t\t\n\t\t\t\t\t\tGet an enterprise-ready PyTorch experience in the cloud.<\/p>\n\t\t\t\t\t<\/div>\n\n\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\tLearn more<\/span>\n\t\t\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t<\/div>\n\n\t\t\t\t\t<\/div>\n\t<\/div>\n<\/div>\n\n\n\nPyTorch <\/a>is a machine learning framework used for applications such as computer vision and natural language processing, originally developed by Meta AI and now a part of the Linux Foundation umbrella, under the name of PyTorch Foundation. PyTorch has a powerful, TorchScript-based<\/a> implementation that transforms the model from eager to graph mode for deployment scenarios.<\/p>\n\n\n\nOne of the biggest challenges PyTorch developers face in their deep learning projects is model optimization and performance. Oftentimes, the question arises: How can I improve the performance of my PyTorch models? As you might have read in our previous blog, Intel\u00ae and Microsoft have joined hands<\/a> to tackle this problem with OpenVINO\u2122 Integration with Torch-ORT<\/a>. Initially, Microsoft had released Torch-ORT<\/a>, which focused on accelerating PyTorch model training using ONNX Runtime. Recently, this capability was extended to accelerate PyTorch model inferencing by using the OpenVINO\u2122 toolkit<\/a> on Intel\u00ae central processing unit (CPU), graphical processing unit (GPU), and video processing unit (VPU) with just two lines of code.<\/p>\n\n\n\n![\"diagram\"](\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2022\/11\/Picture1.jpg\")
Figure 1: OpenVINO\u2122 Integration with Torch-ORT Application Flow. This figure shows how OpenVINO\u2122 Integration with Torch-ORT can be used in a Computer Vision Application.<\/em><\/figcaption><\/figure>\n\n\n\nBy adding just two lines of code, we achieved 2.15 times faster inference for PyTorch Inception V3 model on an 11th<\/sup> Gen Intel\u00ae Core\u2122 i7 processor1<\/sup>. In addition to Inception V3, we also see performance gains for many popular PyTorch models such as ResNet50, Roberta-Base, and more. Currently, OpenVINO\u2122 Integration with Torch-ORT supports over 120 PyTorch models<\/a> from popular model zoo\u2019s, like Torchvision and Hugging Face.<\/p>\n\n\n\n![\"Performance](\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2022\/11\/Picture2.jpg\")
Figure 2: FP32 Model Performance of OpenVINO\u2122 Integration with Torch-ORT as compared to PyTorch. This chart shows average inference latency (in milliseconds) for 100 runs after 15 warm-up iterations on an 11th Gen Intel(R) Core (TM) i7-1185G7 @ 3.00GHz.<\/em><\/figcaption><\/figure>\n\n\n\nFeatures<\/h2>\n\n\n\n
OpenVINO\u2122 Integration with Torch-ORT introduces the following features:<\/p>\n\n\n\n
\n- Inline conversion of static\/dynamic input shape models<\/li>\n\n\n\n
- Graph partitioning<\/li>\n\n\n\n
- Support for INT8 models<\/li>\n\n\n\n
- Dockerfiles\/Docker Containers<\/li>\n<\/ul>\n\n\n\n
Inline conversion of static\/dynamic input shape models<\/h3>\n\n\n\n
OpenVINO\u2122 Integration with Torch-ORT performs inferencing of PyTorch models by converting these models to ONNX inline and subsequently performing inference with OpenVINO\u2122 Execution Provider<\/a>. Currently, both static and dynamic input shape models are supported with OpenVINO\u2122 Integration with Torch-ORT. You also have the ability to save the inline exported ONNX model using the DebugOptions<\/a> API.<\/p>\n\n\n\nGraph partitioning<\/h3>\n\n\n\n
OpenVINO\u2122 Integration with Torch-ORT supports many PyTorch models by leveraging the existing graph partitioning feature from ONNX Runtime. With this feature, the input model graph is divided into subgraphs depending on the operators supported by OpenVINO and the OpenVINO-compatible subgraphs run using OpenVINO\u2122 Execution Provider and unsupported operators fall back to MLAS CPU Execution Provider.<\/p>\n\n\n\n
Support for INT8 models<\/h3>\n\n\n\n
OpenVINO\u2122 Integration with Torch-ORT extends the support for lower precision inference through post-training quantization (PTQ) technique. Using PTQ<\/a>, developers can quantize their PyTorch models with Neural Network Compression Framework<\/a> (NNCF) and then run inferencing with OpenVINO\u2122 Integration with Torch-ORT. Note: <\/strong>Currently, our INT8 model support is in the early stages, only including ResNet50<\/a> and MobileNetv2. We are continuously expanding our INT8 model coverage.<\/p>\n\n\n\nDocker Containers<\/h3>\n\n\n\n
You can now use OpenVINO\u2122 Integration with Torch-ORT on Mac OS and Windows OS through Docker. Pre-built Docker images<\/a> are readily available on Docker Hub for your convenience. With a simple docker pull, you will now be able to unleash the key to accelerating performance of PyTorch models. To build the docker image yourself, you can also find dockerfiles<\/a> readily available on Github.<\/p>\n\n\n\nCustomer story<\/h2>\n\n\n\n
Roboflow<\/strong>\u2014Roboflow<\/a> empowers ISVs to build their own computer vision applications and enables hundreds of thousands of developers with a rich catalog of services, models, and frameworks to further optimize their AI workloads on a variety of different Intel\u00ae hardware. An easy-to-use developer toolkit to accelerate models, properly integrated with AI frameworks, such as OpenVINO\u2122 integration with Torch-ORT provides the best of both worlds\u2014an increase in inference speed as well as the ability to reuse already created AI application code with minimal changes. The Roboflow team has showcased a case study<\/a> that demonstrates performance gains with OpenVINO\u2122 Integration with Torch-ORT as compared to Native PyTorch for YOLOv7 model on Intel\u00ae CPU. The Roboflow team is continuing to actively test OpenVINO\u2122 integration with Torch-ORT with the goal of enabling PyTorch developers in the Roboflow Community.<\/p>\n\n\n\nTry it out<\/h2>\n\n\n\n
Try out OpenVINO\u2122 Integration with Torch-ORT through a collection of Jupyter Notebooks<\/a>. Through these sample tutorials, you will see how to install OpenVINO\u2122 Integration with Torch-ORT and accelerate performance for PyTorch models with just two additional lines of code. Stay in the PyTorch framework and leverage OpenVINO\u2122 optimizations\u2014it doesn\u2019t get much easier than this.<\/p>\n\n\n\nLearn more<\/h2>\n\n\n\n
Here is a list of resources to help you learn more:<\/p>\n\n\n\n
\n- Github Repository<\/a><\/li>\n\n\n\n
- Sample Notebooks<\/a><\/li>\n\n\n\n
- Supported Models<\/a><\/li>\n\n\n\n
- Usage Guide<\/a><\/li>\n\n\n\n
- PyTorch on Azure<\/a><\/li>\n<\/ul>\n\n\n\n
Notes:<\/em><\/strong><\/p>\n\n\n\n1<\/sup>Framework configuration: ONNXRuntime 1.13.1
Application configuration: torch_ort_infer 1.13.1, python timeit module for timing inference of models<\/em>
Input: Classification models: torch.Tensor; NLP models: Masked sentence; OD model: .jpg image
Application Metric: Average Inference latency for 100 iterations calculated after 15 warmup iterations
Platform: Tiger Lake
Number of Nodes: 1 Numa Node
Number of Sockets: 1
CPU or Accelerator: 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
Cores\/socket, Threads\/socket or EU\/socket: 4, 2 Threads\/Core
ucode: 0xa4
HT: Enabled
Turbo: Enabled
BIOS Version: TNTGLV57.9026.2020.0916.1340
System DDR Mem Config: slots \/ cap \/ run-speed: 2\/32 GB\/2667 MT\/s
Total Memory\/Node (DDR+DCPMM): 64GB
Storage \u2013 boot: Sabrent Rocket 4.0 500GB – size: 465.8G
OS: Ubuntu 20.04.4 LTS
Kernel: 5.15.0-1010-intel-iotg<\/em><\/p>\n\n\n\nNotices and disclaimers:<\/em><\/strong><\/p>\n\n\n\n
Get an enterprise-ready PyTorch experience in the cloud.<\/p>\n\t\t\t\t\t<\/div>\n\n\t\t\t\t\t\t\t\t\t\t\t
PyTorch <\/a>is a machine learning framework used for applications such as computer vision and natural language processing, originally developed by Meta AI and now a part of the Linux Foundation umbrella, under the name of PyTorch Foundation. PyTorch has a powerful, TorchScript-based<\/a> implementation that transforms the model from eager to graph mode for deployment scenarios.<\/p>\n\n\n\n One of the biggest challenges PyTorch developers face in their deep learning projects is model optimization and performance. Oftentimes, the question arises: How can I improve the performance of my PyTorch models? As you might have read in our previous blog, Intel\u00ae and Microsoft have joined hands<\/a> to tackle this problem with OpenVINO\u2122 Integration with Torch-ORT<\/a>. Initially, Microsoft had released Torch-ORT<\/a>, which focused on accelerating PyTorch model training using ONNX Runtime. Recently, this capability was extended to accelerate PyTorch model inferencing by using the OpenVINO\u2122 toolkit<\/a> on Intel\u00ae central processing unit (CPU), graphical processing unit (GPU), and video processing unit (VPU) with just two lines of code.<\/p>\n\n\n\n By adding just two lines of code, we achieved 2.15 times faster inference for PyTorch Inception V3 model on an 11th<\/sup> Gen Intel\u00ae Core\u2122 i7 processor1<\/sup>. In addition to Inception V3, we also see performance gains for many popular PyTorch models such as ResNet50, Roberta-Base, and more. Currently, OpenVINO\u2122 Integration with Torch-ORT supports over 120 PyTorch models<\/a> from popular model zoo\u2019s, like Torchvision and Hugging Face.<\/p>\n\n\n\n OpenVINO\u2122 Integration with Torch-ORT introduces the following features:<\/p>\n\n\n\n OpenVINO\u2122 Integration with Torch-ORT performs inferencing of PyTorch models by converting these models to ONNX inline and subsequently performing inference with OpenVINO\u2122 Execution Provider<\/a>. Currently, both static and dynamic input shape models are supported with OpenVINO\u2122 Integration with Torch-ORT. You also have the ability to save the inline exported ONNX model using the DebugOptions<\/a> API.<\/p>\n\n\n\n OpenVINO\u2122 Integration with Torch-ORT supports many PyTorch models by leveraging the existing graph partitioning feature from ONNX Runtime. With this feature, the input model graph is divided into subgraphs depending on the operators supported by OpenVINO and the OpenVINO-compatible subgraphs run using OpenVINO\u2122 Execution Provider and unsupported operators fall back to MLAS CPU Execution Provider.<\/p>\n\n\n\n OpenVINO\u2122 Integration with Torch-ORT extends the support for lower precision inference through post-training quantization (PTQ) technique. Using PTQ<\/a>, developers can quantize their PyTorch models with Neural Network Compression Framework<\/a> (NNCF) and then run inferencing with OpenVINO\u2122 Integration with Torch-ORT. Note: <\/strong>Currently, our INT8 model support is in the early stages, only including ResNet50<\/a> and MobileNetv2. We are continuously expanding our INT8 model coverage.<\/p>\n\n\n\n You can now use OpenVINO\u2122 Integration with Torch-ORT on Mac OS and Windows OS through Docker. Pre-built Docker images<\/a> are readily available on Docker Hub for your convenience. With a simple docker pull, you will now be able to unleash the key to accelerating performance of PyTorch models. To build the docker image yourself, you can also find dockerfiles<\/a> readily available on Github.<\/p>\n\n\n\n Roboflow<\/strong>\u2014Roboflow<\/a> empowers ISVs to build their own computer vision applications and enables hundreds of thousands of developers with a rich catalog of services, models, and frameworks to further optimize their AI workloads on a variety of different Intel\u00ae hardware. An easy-to-use developer toolkit to accelerate models, properly integrated with AI frameworks, such as OpenVINO\u2122 integration with Torch-ORT provides the best of both worlds\u2014an increase in inference speed as well as the ability to reuse already created AI application code with minimal changes. The Roboflow team has showcased a case study<\/a> that demonstrates performance gains with OpenVINO\u2122 Integration with Torch-ORT as compared to Native PyTorch for YOLOv7 model on Intel\u00ae CPU. The Roboflow team is continuing to actively test OpenVINO\u2122 integration with Torch-ORT with the goal of enabling PyTorch developers in the Roboflow Community.<\/p>\n\n\n\n Try out OpenVINO\u2122 Integration with Torch-ORT through a collection of Jupyter Notebooks<\/a>. Through these sample tutorials, you will see how to install OpenVINO\u2122 Integration with Torch-ORT and accelerate performance for PyTorch models with just two additional lines of code. Stay in the PyTorch framework and leverage OpenVINO\u2122 optimizations\u2014it doesn\u2019t get much easier than this.<\/p>\n\n\n\n Here is a list of resources to help you learn more:<\/p>\n\n\n\n Notes:<\/em><\/strong><\/p>\n\n\n\n 1<\/sup>Framework configuration: ONNXRuntime 1.13.1 Notices and disclaimers:<\/em><\/strong><\/p>\n\n\n\nFeatures<\/h2>\n\n\n\n
\n
Inline conversion of static\/dynamic input shape models<\/h3>\n\n\n\n
Graph partitioning<\/h3>\n\n\n\n
Support for INT8 models<\/h3>\n\n\n\n
Docker Containers<\/h3>\n\n\n\n
Customer story<\/h2>\n\n\n\n
Try it out<\/h2>\n\n\n\n
Learn more<\/h2>\n\n\n\n
\n
Application configuration: torch_ort_infer 1.13.1, python timeit module for timing inference of models<\/em>
Input: Classification models: torch.Tensor; NLP models: Masked sentence; OD model: .jpg image
Application Metric: Average Inference latency for 100 iterations calculated after 15 warmup iterations
Platform: Tiger Lake
Number of Nodes: 1 Numa Node
Number of Sockets: 1
CPU or Accelerator: 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
Cores\/socket, Threads\/socket or EU\/socket: 4, 2 Threads\/Core
ucode: 0xa4
HT: Enabled
Turbo: Enabled
BIOS Version: TNTGLV57.9026.2020.0916.1340
System DDR Mem Config: slots \/ cap \/ run-speed: 2\/32 GB\/2667 MT\/s
Total Memory\/Node (DDR+DCPMM): 64GB
Storage \u2013 boot: Sabrent Rocket 4.0 500GB – size: 465.8G
OS: Ubuntu 20.04.4 LTS
Kernel: 5.15.0-1010-intel-iotg<\/em><\/p>\n\n\n\n