{"id":73619,"date":"2018-06-18T13:21:43","date_gmt":"2018-06-18T20:21:43","guid":{"rendered":""},"modified":"2025-01-29T11:49:48","modified_gmt":"2025-01-29T19:49:48","slug":"tutorial-canary-deployment-for-azure-virtual-machine-scale-sets","status":"publish","type":"post","link":"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/","title":{"rendered":"Tutorial: Canary Deployment for Azure Virtual Machine Scale Sets"},"content":{"rendered":"\n

Canary deployment is a pattern that rolls out releases to a subset of users or servers. It deploys the changes to a small set of servers, which allows you to test and monitor how the new release works before rolling the changes to the rest of the servers.<\/p>\n\n\n\n

Virtual machine scale sets (VMSS) are an Azure compute resource that you can use to deploy and manage a set of identical VMs. With all VMs configured the same, scale sets are designed to support true autoscale, and no pre-provisioning of VMs is required. So it’s easier to build large-scale services that target big compute, large data, and containerized workloads.<\/p>\n\n\n\n

VMSS allow you to manage large number of identical VMs with simple instructions, yet allow you to update specific VMs. You can build your VMSS with a customized image or publicly available OS images along with VM extension scripts to setup all required environments. When it comes to update existing VMSS, you need to update its configuration with a new image or extension scripts, and then manually trigger the update of the VMSS instances, either all in one instruction, or selectively pick some VMs to be updated.<\/p>\n\n\n\n

The ability to update individual VMs in VMSS allows us to control the number of VMs that will be updated to the new releases, i.e., allows us to do canary deployment:<\/p>\n\n\n\n

    \n
  1. (Existing) Create the initial VMSS which host your services.<\/li>\n\n\n\n
  2. Update the VMSS configuration, either point to new customized image, or update the extension scripts, which contains the new release of your services.<\/li>\n\n\n\n
  3. Selectively update individual instances to the new release according to the configuration changes.<\/li>\n\n\n\n
  4. Verify the new release works.<\/li>\n\n\n\n
  5. Update the rest of instances to the new release.<\/li>\n<\/ol>\n\n\n\n

    Nginx Canary Deployment Example<\/h2>\n\n\n\n

    Here we demonstrate the canary deployment for VMSS using the Nginx binary release.<\/p>\n\n\n\n

    Prepare the VMSS<\/h3>\n\n\n\n

    We use the public Ubuntu Server 16.04 LTS together with an extension script which installs the Nginx service to setup the VMSS. In your project, you can customize the extension script to install the service on demand, or create a customized image (reference: Packer \/ Azure Resource Manager Builder<\/a>).<\/p>\n\n\n\n

    <\/a>Prepare variable configurations<\/h3>\n\n\n\n

    First we setup some variables that will be used in the following preparation steps. You can update the variables based on your needs.<\/p>\n\n\n

    \n# resource group and VMSS name\nresource_group=the-resource-group-name\nlocation=the-resource-location\nvmss_name=the-vmss-name\n# admin user and SSH login credentials setup\nadmin_user=azureuser\nssh_pubkey=\"$(readlink -f ~\/.ssh\/id_rsa.pub)\"\n# the storage account that will be created to store the init scripts. Note that '-' is not allowed in a storage account name.\nexport AZURE_STORAGE_ACCOUNT=the-storage-account-name\n<\/pre><\/div>\n\n\n
    Create VMSS from public Ubuntu image<\/h3>\n\n\n\n
    We create a VMSS with 3 instances, using the public image “UbuntuLTS”.<\/p>\n\n\n
    \naz group create --name \"$resource_group\" --location \"$location\"\n# create the VMSS with 3 instances using the public Ubuntu LTS image\naz vmss create --resource-group \"$resource_group\" --name \"$vmss_name\" \\\n    --image UbuntuLTS \\\n    --admin-username \"$admin_user\" \\\n    --ssh-key-value \"$ssh_pubkey\" \\\n    --vm-sku Standard_D2_v3 \\\n    --instance-count 3 \\\n    --lb \"${vmss_name}LB\"\n<\/pre><\/div>\n\n\n
    Prepare the init scripts<\/h3>\n\n\n\n
    In order to use the custom script extension to configure the VMSS, we need to store the script at some location that’s accessible via HTTP(s). Here we create a storage account for the script storage, and expose the scripts publicly to allow the script extension to pick it up.<\/p>\n\n\n\n
    The custom script is fairly simple in this case. It installs the Nginx package from the Ubuntu Apt source. In your project, you may update the script to fetch dependencies, install, configure and start services, etc.<\/p>\n\n\n
    \n# create the storage account and container that store the extension scripts\naz storage account create --name \"$AZURE_STORAGE_ACCOUNT\" --location \"$location\" --resource-group \"$resource_group\" --sku Standard_LRS\nexport AZURE_STORAGE_KEY=\"$(az storage account keys list --resource-group \"$resource_group\" --account-name \"$AZURE_STORAGE_ACCOUNT\" --query '[0].value' --output tsv)\"\naz storage container create --name init --public-access container\n# upload the init script to the blob container\ncat <install_nginx.sh\n#!\/bin\/bash\nsudo apt-get update\nsudo apt-get install -y nginx\nEOF\naz storage blob upload --container-name init --file install_nginx.sh --name install_nginx.sh\ninit_script_url=\"$(az storage blob url --container-name init --name install_nginx.sh --output tsv)\"\n<\/pre><\/div>\n\n\n
    <\/a>Install the custom script extension<\/h3>\n\n\n
    \n# prepare the script config\ncat <script-config.json\n{\n  \"fileUris\": [\"$init_script_url\"],\n  \"commandToExecute\": \".\/install_nginx.sh\"\n}\nEOF\n# install the CustomScript extension to the VMSS\naz vmss extension set \\\n    --publisher Microsoft.Azure.Extensions \\\n    --version 2.0 \\\n    --name CustomScript \\\n    --resource-group \"$resource_group\" \\\n    --vmss-name \"$vmss_name\" \\\n    --settings @script-config.json\n# update all the instances so that they will have nginx installed\naz vmss update-instances --resource-group \"$resource_group\" --name \"$vmss_name\" --instance-ids \\*\n<\/pre><\/div>\n\n\n
    <\/a>Update load balancer endpoint<\/h3>\n\n\n\n
    We need to create a load balancer rule to route the public traffic to the Nginx services running in the VMSS backend.<\/p>\n\n\n
    \n# create load balancer rule to allow public access to the backend Nginx service\naz network lb probe create \\\n    --resource-group \"$resource_group\" \\\n    --lb-name \"${vmss_name}LB\" \\\n    --name nginx \\\n    --port 80 \\\n    --protocol Http \\\n    --path \/\naz network lb rule create \\\n    --resource-group \"$resource_group\" \\\n    --lb-name \"${vmss_name}LB\" \\\n    --name nginx \\\n    --frontend-port 80 \\\n    --backend-port 80 \\\n    --protocol Tcp \\\n    --backend-pool-name \"${vmss_name}LBBEPool\" \\\n    --probe nginx\n<\/pre><\/div>\n\n\n
    <\/a>Verify everything works<\/h3>\n\n\n\n
    Check that we can access the Nginx service from the public endpoint of the load balancer.<\/p>\n\n\n
    \n# check that the Nginx service is working properly\nlb_ip=$(az network lb show --resource-group \"$resource_group\" --name \"${vmss_name}LB\" --query \"frontendIpConfigurations[].publicIpAddress.id\" --output tsv | head -n1 | xargs az network public-ip show --query ipAddress --output tsv --ids)\ncurl -s \"$lb_ip\" | grep title\n#>> Welcome to nginx!<\/title>\n<\/pre><\/div>\n\n\n<h2 class=\"wp-block-heading\" id=\"deploy-new-release-in-canary-deployment-pattern\"><a id=\"user-content-deploy-new-release-in-canary-deployment-pattern\" class=\"anchor\" href=\"https:\/\/github.com\/ArieShout\/blogs\/blob\/master\/canary-deployment\/vmss-canary-deployment.md#deploy-new-release-in-canary-deployment-pattern\" aria-hidden=\"true\"><\/a>Deploy New Release in Canary Deployment Pattern<\/h2>\n\n\n\n<p>In the new release, we make a simple update in the Nginx landing page, and deploy it to 1 instance in the early stage. So after the deployment, we should have 1 instance serving the updated landing page, and 2 instances serving the original page.<\/p>\n\n\n\n<p>First, we need to update and upload the new custom script. Some points to call out here:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The custom script will be executed on a fresh VM after it is created from the given OS image. It is not an incremental update process based on the existing VM. So we need to install all the dependencies and services again, with the changes in the new release included.<\/li>\n\n\n\n<li>Any updates you make to your application are not exposed to the Custom Script Extension unless that install script changes. To force VMSS to pick up the custom script changes, you need to change the script name so that it results in a different file URI.<\/li>\n\n\n\n<li>This will not affect the existing instances until we manually update those instances.<\/li>\n<\/ul>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; auto-links: false; gutter: false; title: ; quick-code: false; notranslate\" title=\"\">\n# prepare the updated nginx service\ncat <<EOF >install_nginx.v1.sh\n#!\/bin\/bash\nsudo apt-get update\nsudo apt-get install -y nginx\nsudo sed -i -e 's\/Welcome to nginx\/Welcome to nginx on Azure VMSS\/' \/var\/www\/html\/index*.html\nEOF\naz storage blob upload --container-name init --file install_nginx.v1.sh --name install_nginx.v1.sh\ninit_script_url=\"$(az storage blob url --container-name init --name install_nginx.v1.sh --output tsv)\"\n# prepare the script config\ncat <<EOF >script-config.v1.json\n{\n  \"fileUris\": [\"$init_script_url\"],\n  \"commandToExecute\": \".\/install_nginx.v1.sh\"\n}\nEOF\n# install the CustomScript extension to the VMSS\naz vmss extension set \\\n    --publisher Microsoft.Azure.Extensions \\\n    --version 2.0 \\\n    --name CustomScript \\\n    --resource-group \"$resource_group\" \\\n    --vmss-name \"$vmss_name\" \\\n    --settings @script-config.v1.json\n<\/pre><\/div>\n\n\n<p>Now that the custom script configuration is update for the VMSS, we can update 1 instance to pick up the new custom script.<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; auto-links: false; gutter: false; title: ; quick-code: false; notranslate\" title=\"\">\n# pick up the first instance ID\ninstance_id=\"$(az vmss list-instances --resource-group \"$resource_group\" --name \"$vmss_name\" --query '[].instanceId' --output tsv | head -n1)\"\n# update the instance VM\naz vmss update-instances --resource-group \"$resource_group\" --name \"$vmss_name\" --instance-ids \"$instance_id\"\n# (optional) check the latest model applied status\naz vmss list-instances --resource-group \"$resource_group\" --name \"$vmss_name\" | grep latest\n#>>    \"latestModelApplied\": true,\n#>>    \"latestModelApplied\": false,\n#>>    \"latestModelApplied\": false,\n<\/pre><\/div>\n\n\n<p>Check the load balancer public endpoint and we should see the old version and new version interleaved.<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; auto-links: false; gutter: false; title: ; quick-code: false; notranslate\" title=\"\">\ncurl -s \"$lb_ip\" | grep title\n#>> <title>Welcome to nginx!<\/title>\ncurl -s \"$lb_ip\" | grep title\n#>> <title>Welcome to nginx on Azure VMSS!<\/title>\ncurl -s \"$lb_ip\" | grep title\n#>> <title>Welcome to nginx!<\/title>\ncurl -s \"$lb_ip\" | grep title\n#>> <title>Welcome to nginx!<\/title>\n<\/pre><\/div>\n\n\n<p>Now you can do more checks to verify if the new version works as expected. Note that the VMSS sits behind the frontend load balancer, <strong>you do not know the service status for a individual node through the public access point<\/strong>. If you need to check a specific instance, you can SSH login to the that instance through the NAT mapping defined in the load balancer and check the service in the SSH session; or you can open an tunnel to the remote service through the SSH channel, and check the service in detail through the local port.<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; auto-links: false; gutter: false; title: ; quick-code: false; notranslate\" title=\"\">\n# obtain the NAT SSH port for the updated instance\nssh_port=\"$(az network lb inbound-nat-rule show --resource-group \"$resource_group\" --lb-name \"${vmss_name}LB\" --name \"${vmss_name}LBNatPool.${instance_id}\" --query frontendPort --output tsv)\"\n# map the localhost:8080 endpoint to the remote 80 port through the SSH channel\nssh -L localhost:8080:localhost:80 -p \"$ssh_port\" azureuser@\"$lb_ip\"\n<\/pre><\/div>\n\n\n<p>After this, you can visit the web page through <code>http:\/\/localhost:8080<\/code> and it will show you the page served by the updated instance.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"complete-the-release-of-the-new-version\"><a id=\"user-content-complete-the-release-of-the-new-version\" class=\"anchor\" href=\"https:\/\/github.com\/ArieShout\/blogs\/blob\/master\/canary-deployment\/vmss-canary-deployment.md#complete-the-release-of-the-new-version\" aria-hidden=\"true\"><\/a>Complete the Release of the New Version<\/h2>\n\n\n\n<p>At this point you have 1 instance serving the updated web page and 2 instances serving the original page in the VMSS. When you have verified that the new version works, you can update the rest of the instances to the new version.<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; auto-links: false; gutter: false; title: ; quick-code: false; notranslate\" title=\"\">\naz vmss update-instances --resource-group \"$resource_group\" --name \"$vmss_name\" --instance-ids \\*\n<\/pre><\/div>\n\n\n<p>Note that this will update all the instances with models not aligned with the latest state in parallel. So all the outdated instances will be brought down, updated, and brought up again. It will not cause service downtime as long as the load balancer noticed some of the backends are down, as we have at least 1 instance updated in the previous steps. However, during the update window of the outdated instances, all the client traffic will be routed to up-to-date instances, which will increase the load and latency on those instances.<\/p>\n\n\n\n<p>A better approach may be querying the outdated instances list first, and then update them with smaller granularity:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; auto-links: false; gutter: false; title: ; quick-code: false; notranslate\" title=\"\">\naz vmss list-instances --resource-group \"$resource_group\" --name \"$vmss_name\" --query '[?latestModelApplied==`false`].instanceId' --output tsv\n#>> 2\n#>> 4\naz vmss update-instances --resource-group \"$resource_group\" --name \"$vmss_name\" --instance-ids 2\n# wait for instance 2 to be up and running\naz vmss update-instances --resource-group \"$resource_group\" --name \"$vmss_name\" --instance-ids 4\n<\/pre><\/div>\n\n\n<p>In this way, only a small number of instances are being updated at a given point of time. The rest of the instances are not touched and will serve the traffic as per normal.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"work-with-image-based-canary-deployment\"><a id=\"user-content-work-with-image-based-canary-deployment\" class=\"anchor\" href=\"https:\/\/github.com\/ArieShout\/blogs\/blob\/master\/canary-deployment\/vmss-canary-deployment.md#work-with-image-based-canary-deployment\" aria-hidden=\"true\"><\/a>Work with Image Based Canary Deployment<\/h2>\n\n\n\n<p>The above steps demonstrates how we can do canary deployments for VMSS using the custom script extension. VMSS also supports custom images. If specified, all the VMs will be created from the given image. Compared to the custom script extension based VMSS, the image based VMSS:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Provisions faster: the service creation and configuration is done at the image creation process, and when VMSS needs to provision new instance, it creates the VM from that image. It doesn’t need to execute extra scripts after the VM provision. (Although you can still add a custom script extension if needed.)<\/li>\n\n\n\n<li>Service and dependency versions are more stable. The service and dependencies are fetched when the image is created, and all the VMs created from the image get the same binaries. If working with custom script extension, you need to be careful if the service or dependencies is upgraded when the VMSS is scaling.<\/li>\n<\/ul>\n\n\n\n<p><a href=\"https:\/\/www.packer.io\/docs\/builders\/azure.html\" rel=\"nofollow\">Packer<\/a> is widely used to create OS images in different cloud platforms. Consider if we need to transform the above custom script extension based deployments to image based, we can create the base image using the following packer configuration (filename: <code>packer-nginx.json<\/code>):<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; auto-links: false; gutter: false; title: ; quick-code: false; notranslate\" title=\"\">\n{\n    \"variables\": {\n        \"client_id\": null,\n        \"client_secret\": null,\n        \"subscription_id\": null,\n        \"tenant_id\": null,\n        \"resource_group\": null,\n        \"location\": null,\n        \"vm_size\": \"Standard_DS2_v2\"\n    },\n    \"builders\": [\n        {\n            \"type\": \"azure-arm\",\n            \"client_id\": \"{{user `client_id`}}\",\n            \"client_secret\": \"{{user `client_secret`}}\",\n            \"subscription_id\": \"{{user `subscription_id`}}\",\n            \"tenant_id\": \"{{user `tenant_id`}}\",\n            \"managed_image_resource_group_name\": \"{{user `resource_group`}}\",\n            \"managed_image_name\": \"nginx-base-image\",\n            \"os_type\": \"Linux\",\n            \"image_publisher\": \"Canonical\",\n            \"image_offer\": \"UbuntuServer\",\n            \"image_sku\": \"16.04-LTS\",\n            \"location\": \"{{user `location`}}\",\n            \"vm_size\": \"{{user `vm_size`}}\"\n        }\n    ],\n    \"provisioners\": [\n        {\n            \"execute_command\": \"chmod +x {{ .Path }}; {{ .Vars }} sudo -E sh '{{ .Path }}'\",\n            \"inline\": [\n                \"apt-get update\",\n                \"apt-get dist-upgrade -y\",\n                \"apt-get install -y nginx\",\n                \"\/usr\/sbin\/waagent -force -deprovision+user && export HISTSIZE=0 && sync\"\n            ],\n            \"inline_shebang\": \"\/bin\/sh -x\",\n            \"type\": \"shell\"\n        }\n    ]\n}\n<\/pre><\/div>\n\n\n<p>and build it with:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; auto-links: false; gutter: false; title: ; quick-code: false; notranslate\" title=\"\">\npacker build \\\n  -var \"client_id=$your_service_principal_id\" \\\n  -var \"client_secret=$your_service_principal_key\" \\\n  -var \"subscription_id=$your_subscription_id\" \\\n  -var \"tenant_id=$your_tenant_id\" \\\n  -var \"resource_group=$resource_group\" \\\n  -var \"location=$location\" \\\n  packer-nginx.json\n<\/pre><\/div>\n\n\n<p>When this completes, we will get an image <code>nginx-base-image<\/code> in the resource group specified. Similarly, we can create the updated image (<code>nginx-updated-image<\/code>) by adding the following line to the provisioners script, updating the <code>managed_image_name<\/code> to <code>nginx-updated-image<\/code> and build the image.<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; auto-links: false; gutter: false; title: ; quick-code: false; notranslate\" title=\"\">\nsed -i -e 's\/Welcome to nginx\/Welcome to nginx on Azure VMSS\/' \/var\/www\/html\/index*.html\n<\/pre><\/div>\n\n\n<p>After that we can get the VMSS image ID for the base image and the updated one:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; auto-links: false; gutter: false; title: ; quick-code: false; notranslate\" title=\"\">\nbase_image_id=\"$(az image show --resource-group \"$resource_group\" --name nginx-base-image --query id --output tsv)\"\nexport updated_image_id=\"$(az image show --resource-group \"$resource_group\" --name nginx-updated-image --query id --output tsv)\"\n<\/pre><\/div>\n\n\n<p>Now that we have two images, we can do the canary deployment as follows:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><\/li>\n<\/ol>\n\n\n\n<p>1 – Initially, we need to specify the base image ID when we create the VMSS:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; auto-links: false; gutter: false; title: ; quick-code: false; notranslate\" title=\"\">\n# create the VMSS with 3 instances using the public Ubuntu LTS image\naz vmss create --resource-group \"$resource_group\" --name \"$vmss_name\" \\\n    --image \"$base_image_id\" \\\n    --admin-username \"$admin_user\" \\\n    --ssh-key-value \"$ssh_pubkey\" \\\n    --vm-sku Standard_D2_v3 \\\n    --instance-count 3 \\\n    --lb \"${vmss_name}LB\"\n<\/pre><\/div>\n\n\n<p>2 – When we deploy new release, we update the image in VMSS configuration:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; auto-links: false; gutter: false; title: ; quick-code: false; notranslate\" title=\"\">\naz vmss update --resource-group \"$resource_group\" --name \"$vmss_name\" --set \"virtualMachineProfile.storageProfile.imageReference.id=$updated_image_id\"\n<\/pre><\/div>\n\n\n<p>3 – Now we can selectively update certain instance to using the latest image with command <code>az vmss update-instances<\/code>, or upgrade all instances with <code>--instance-ids<\/code> setting to <code>*<\/code>.<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; auto-links: false; gutter: false; title: ; quick-code: false; notranslate\" title=\"\">\n# pick up the first instance ID\ninstance_id=\"$(az vmss list-instances --resource-group \"$resource_group\" --name \"$vmss_name\" --query '[].instanceId' --output tsv | head -n1)\"\n# update the instance VM\naz vmss update-instances --resource-group \"$resource_group\" --name \"$vmss_name\" --instance-ids \"$instance_id\"\n<\/pre><\/div>\n\n\n<h2 class=\"wp-block-heading\" id=\"canary-deployment-with-jenkins\">Canary Deployment with Jenkins<\/h2>\n\n\n\n<p>In canary deployment we may roll out new releases to the servers gradually, which may involve multiple deployments that updates the old releases \/ new releases server ratio. This may not be suitable to automate in limited number of Jenkins jobs.<\/p>\n\n\n\n<p>However, if we simplify the process, and we can model the process with parameterized Jenkins jobs. We have published <a href=\"https:\/\/plugins.jenkins.io\/azure-vmss\" rel=\"nofollow\">Azure Virtual Machine Scale Set<\/a> Jenkins plugin which helps to deploy new images to VMSS.<\/p>\n\n\n\n<p>The above image based canary deployment can be modeled as two Jenkins Pipeline jobs:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Deploy to a subset of instances<\/li>\n<\/ul>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; auto-links: false; gutter: false; title: ; quick-code: false; notranslate\" title=\"\">\nnode {\n    \/\/ ...\n    stage('Update Image Configuration') {\n       azureVMSSUpdate azureCredentialsId: '<azure-credential-id>', resourceGroup: env.resource_group, name: env.vmss_name,\n                       imageReference: [id: env.updated_image_id]\n    }\n    stage('Update A Subset of Instances') {\n       azureVMSSUpdateInstances azureCredentialsId: '<azure-credential-id>', resourceGroup: env.resource_group, name: env.vmss_name,\n                                instanceIds: '0,1'\n    }\n    \/\/ ...\n}\n<\/pre><\/div>\n\n\n<ul class=\"wp-block-list\">\n<li>Upgrade all the rest instances to the latest image<\/li>\n<\/ul>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; auto-links: false; gutter: false; title: ; quick-code: false; notranslate\" title=\"\">\nnode {\n   stage('Update All Instances') {\n       azureVMSSUpdateInstances azureCredentialsId: '<azure-credential-id>', resourceGroup: env.resource_group, name: env.vmss_name,\n                                instanceIds: '*'\n   }\n}\n<\/pre><\/div>\n\n\n<p>As mentioned in the previous example, you need to implement extra logic to test and validate if the new image is working properly.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"further-reading-blue-green-deployment\"><a id=\"user-content-further-reading-blue-green-deployment\" class=\"anchor\" href=\"https:\/\/github.com\/ArieShout\/blogs\/blob\/master\/canary-deployment\/vmss-canary-deployment.md#further-reading-blue-green-deployment\" aria-hidden=\"true\"><\/a>Further Reading: Blue-green Deployment<\/h2>\n\n\n\n<p>We can also do blue-green deployment on VMSS, in which you have two nearly identical backends, you can upgrade one of them and switch the routing to the upgraded backend without interrupting the user traffic. We have prepared a quick start template and you can find more details at <a href=\"https:\/\/aka.ms\/azjenkinsvmssqs\" rel=\"nofollow\">Jenkins Blue-green Deployment to VMSS<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Canary deployment is a pattern that rolls out releases to a subset of users or servers. It deploys the changes to a small set of servers, which allows you to test and monitor how the new release works before rolling the changes to the rest of the servers.<\/p>\n","protected":false},"author":5562,"featured_media":95489,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"msxcm_post_with_no_image":false,"ep_exclude_from_search":false,"_classifai_error":"","_classifai_text_to_speech_error":"","footnotes":""},"post_tag":[2272,166],"content-type":[340],"topic":[2240,2241],"programming-languages":[2273],"coauthors":[2344],"class_list":["post-73619","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","tag-microsoft","tag-azure","content-type-tutorials-and-demos","topic-application-development","topic-cloud","programming-languages-bash","review-flag-1593580428-734","review-flag-1593580415-931","review-flag-1593580419-521","review-flag-1593580771-946","review-flag-1-1593580432-963","review-flag-2-1593580437-411","review-flag-3-1593580442-169","review-flag-4-1593580448-609","review-flag-new-1593580248-669","review-flag-vm-1593580807-312"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.2 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Tutorial: Canary Deployment for Azure Virtual Machine Scale Sets | Microsoft Open Source Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Tutorial: Canary Deployment for Azure Virtual Machine Scale Sets | Microsoft Open Source Blog\" \/>\n<meta property=\"og:description\" content=\"Canary deployment is a pattern that rolls out releases to a subset of users or servers. It deploys the changes to a small set of servers, which allows you to test and monitor how the new release works before rolling the changes to the rest of the servers.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/\" \/>\n<meta property=\"og:site_name\" content=\"Microsoft Open Source Blog\" \/>\n<meta property=\"article:published_time\" content=\"2018-06-18T20:21:43+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-01-29T19:49:48+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/06\/STB13_Michelle_03.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1170\" \/>\n\t<meta property=\"og:image:height\" content=\"640\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Microsoft + Open Source\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@OpenAtMicrosoft\" \/>\n<meta name=\"twitter:site\" content=\"@OpenAtMicrosoft\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Microsoft + Open Source\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 min read\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/\"},\"author\":[{\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/author\/microsoft-open-source\/\",\"@type\":\"Person\",\"@name\":\"Microsoft + Open Source\"}],\"headline\":\"Tutorial: Canary Deployment for Azure Virtual Machine Scale Sets\",\"datePublished\":\"2018-06-18T20:21:43+00:00\",\"dateModified\":\"2025-01-29T19:49:48+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/\"},\"wordCount\":1577,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/06\/STB13_Michelle_03.webp\",\"keywords\":[\"Microsoft\",\"Microsoft Azure\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/\",\"url\":\"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/\",\"name\":\"Tutorial: Canary Deployment for Azure Virtual Machine Scale Sets | Microsoft Open Source Blog\",\"isPartOf\":{\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/06\/STB13_Michelle_03.webp\",\"datePublished\":\"2018-06-18T20:21:43+00:00\",\"dateModified\":\"2025-01-29T19:49:48+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/#primaryimage\",\"url\":\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/06\/STB13_Michelle_03.webp\",\"contentUrl\":\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/06\/STB13_Michelle_03.webp\",\"width\":1170,\"height\":640},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/opensource.microsoft.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Tutorial: Canary Deployment for Azure Virtual Machine Scale Sets\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/#website\",\"url\":\"https:\/\/opensource.microsoft.com\/blog\/\",\"name\":\"Microsoft Open Source Blog\",\"description\":\"Open dialogue about openness at Microsoft \u2013 open source, standards, interoperability\",\"publisher\":{\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/opensource.microsoft.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/#organization\",\"name\":\"Microsoft Open Source Blog\",\"url\":\"https:\/\/opensource.microsoft.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2019\/08\/Microsoft-Logo.png\",\"contentUrl\":\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2019\/08\/Microsoft-Logo.png\",\"width\":259,\"height\":194,\"caption\":\"Microsoft Open Source Blog\"},\"image\":{\"@id\":\"https:\/\/opensource.microsoft.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/x.com\/OpenAtMicrosoft\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Tutorial: Canary Deployment for Azure Virtual Machine Scale Sets | Microsoft Open Source Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/","og_locale":"en_US","og_type":"article","og_title":"Tutorial: Canary Deployment for Azure Virtual Machine Scale Sets | Microsoft Open Source Blog","og_description":"Canary deployment is a pattern that rolls out releases to a subset of users or servers. It deploys the changes to a small set of servers, which allows you to test and monitor how the new release works before rolling the changes to the rest of the servers.","og_url":"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/","og_site_name":"Microsoft Open Source Blog","article_published_time":"2018-06-18T20:21:43+00:00","article_modified_time":"2025-01-29T19:49:48+00:00","og_image":[{"width":1170,"height":640,"url":"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/06\/STB13_Michelle_03.png","type":"image\/png"}],"author":"Microsoft + Open Source","twitter_card":"summary_large_image","twitter_creator":"@OpenAtMicrosoft","twitter_site":"@OpenAtMicrosoft","twitter_misc":{"Written by":"Microsoft + Open Source","Est. reading time":"11 min read"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/#article","isPartOf":{"@id":"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/"},"author":[{"@id":"https:\/\/opensource.microsoft.com\/blog\/author\/microsoft-open-source\/","@type":"Person","@name":"Microsoft + Open Source"}],"headline":"Tutorial: Canary Deployment for Azure Virtual Machine Scale Sets","datePublished":"2018-06-18T20:21:43+00:00","dateModified":"2025-01-29T19:49:48+00:00","mainEntityOfPage":{"@id":"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/"},"wordCount":1577,"commentCount":0,"publisher":{"@id":"https:\/\/opensource.microsoft.com\/blog\/#organization"},"image":{"@id":"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/#primaryimage"},"thumbnailUrl":"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/06\/STB13_Michelle_03.webp","keywords":["Microsoft","Microsoft Azure"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/","url":"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/","name":"Tutorial: Canary Deployment for Azure Virtual Machine Scale Sets | Microsoft Open Source Blog","isPartOf":{"@id":"https:\/\/opensource.microsoft.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/#primaryimage"},"image":{"@id":"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/#primaryimage"},"thumbnailUrl":"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/06\/STB13_Michelle_03.webp","datePublished":"2018-06-18T20:21:43+00:00","dateModified":"2025-01-29T19:49:48+00:00","breadcrumb":{"@id":"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/#primaryimage","url":"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/06\/STB13_Michelle_03.webp","contentUrl":"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2024\/06\/STB13_Michelle_03.webp","width":1170,"height":640},{"@type":"BreadcrumbList","@id":"https:\/\/opensource.microsoft.com\/blog\/2018\/06\/18\/tutorial-canary-deployment-for-azure-virtual-machine-scale-sets\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/opensource.microsoft.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Tutorial: Canary Deployment for Azure Virtual Machine Scale Sets"}]},{"@type":"WebSite","@id":"https:\/\/opensource.microsoft.com\/blog\/#website","url":"https:\/\/opensource.microsoft.com\/blog\/","name":"Microsoft Open Source Blog","description":"Open dialogue about openness at Microsoft \u2013 open source, standards, interoperability","publisher":{"@id":"https:\/\/opensource.microsoft.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/opensource.microsoft.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/opensource.microsoft.com\/blog\/#organization","name":"Microsoft Open Source Blog","url":"https:\/\/opensource.microsoft.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/opensource.microsoft.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2019\/08\/Microsoft-Logo.png","contentUrl":"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2019\/08\/Microsoft-Logo.png","width":259,"height":194,"caption":"Microsoft Open Source Blog"},"image":{"@id":"https:\/\/opensource.microsoft.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/OpenAtMicrosoft"]}]}},"msxcm_display_generated_audio":false,"msxcm_animated_featured_image":null,"distributor_meta":false,"distributor_terms":false,"distributor_media":false,"distributor_original_site_name":"Microsoft Open Source Blog","distributor_original_site_url":"https:\/\/opensource.microsoft.com\/blog","push-errors":false,"_links":{"self":[{"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/posts\/73619","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/users\/5562"}],"replies":[{"embeddable":true,"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/comments?post=73619"}],"version-history":[{"count":3,"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/posts\/73619\/revisions"}],"predecessor-version":[{"id":97062,"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/posts\/73619\/revisions\/97062"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/media\/95489"}],"wp:attachment":[{"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/media?parent=73619"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/post_tag?post=73619"},{"taxonomy":"content-type","embeddable":true,"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/content-type?post=73619"},{"taxonomy":"topic","embeddable":true,"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/topic?post=73619"},{"taxonomy":"programming-languages","embeddable":true,"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/programming-languages?post=73619"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/opensource.microsoft.com\/blog\/wp-json\/wp\/v2\/coauthors?post=73619"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}