{"id":87048,"date":"2021-07-09T09:00:18","date_gmt":"2021-07-09T16:00:18","guid":{"rendered":""},"modified":"2025-05-30T15:36:04","modified_gmt":"2025-05-30T22:36:04","slug":"simple-steps-to-create-scalable-processes-to-deploy-ml-models-as-microservices","status":"publish","type":"post","link":"https:\/\/opensource.microsoft.com\/blog\/2021\/07\/09\/simple-steps-to-create-scalable-processes-to-deploy-ml-models-as-microservices\/","title":{"rendered":"Simple steps to create scalable processes to deploy ML models as microservices"},"content":{"rendered":"\n

This post was co-authored by Alejandro Saucedo, Director of Machine Learning Engineering at Seldon Technologies.<\/em><\/strong><\/p>\n\n\n\n

About the co-author: Alejandro leads teams of machine learning engineers focused on the scalability and extensibility of machine learning deployment and monitoring products with over five million installations. Alejandro is also the Chief Scientist at the Institute for Ethical AI and Machine Learning, where he leads the development of industry standards on machine learning explainability, adversarial robustness, and differential privacy. With over 10 years of software development experience, Alejandro has held technical leadership positions across hyper-growth scale-ups and has a strong track record building cross-functional teams of software engineers.<\/em><\/p>\n\n\n\n

As organizations adopt machine learning in production, they face growing challenges that arise when the number of production machine learning models starts to increase. In this article, we provide a practical tutorial that will enable AI practitioners to leverage production-ready workflows to deploy their machine learning models at scale. More specifically, we will demonstrate the benefits of open source tools and frameworks like ONNX Runtime, Seldon Core, and HuggingFace, as well as how these can be integrated with Azure Kubernetes Services to achieve robust and scalable machine learning operations capabilities.<\/p>\n\n\n\n

By the end of this blog post, you will have a simple, repeatable, and scalable process to deploy complex machine learning models. You will learn by example, deploying the OpenAI GPT-2 natural language processing (NLP) model as a fully-fledged microservice with real-time metrics, and robust monitoring capabilities. Try out our  GPT-2 Azure AKS Deployment Notebook<\/a> that demonstrates the full process.<\/p>\n\n\n\n

The steps that will be carried out in this blog are outlined in the image below, and include the following:<\/p>\n\n\n\n

    \n
  1. Fetch the pre-trained GPT2 Model using HuggingFace and export to ONNX.<\/li>\n\n\n\n
  2. Setup Kubernetes Environment and upload model artifact.<\/li>\n\n\n\n
  3. Deploy ONNX Model with Seldon Core to Azure Kubernetes Service.<\/li>\n\n\n\n
  4. Send inference requests to Kubernetes deployed GPT2 Model.<\/li>\n\n\n\n
  5. Visualize real-time monitoring metrics with Azure dashboards.<\/li>\n<\/ol>\n\n\n\n
    \"Placeholder\"\/<\/figure>\n\n\n\n

    Furthermore, the tools that we\u2019ll be using in this framework will be the following:<\/p>\n\n\n\n