{"id":72802,"date":"2017-11-09T08:00:18","date_gmt":"2017-11-09T16:00:18","guid":{"rendered":""},"modified":"2025-01-22T15:46:16","modified_gmt":"2025-01-22T23:46:16","slug":"s3cmd-amazon-s3-compatible-apps-azure-storage","status":"publish","type":"post","link":"https:\/\/opensource.microsoft.com\/blog\/2017\/11\/09\/s3cmd-amazon-s3-compatible-apps-azure-storage\/","title":{"rendered":"Using s3cmd and other Amazon S3-compatible apps with Azure Blob Storage"},"content":{"rendered":"\n

Object storage is one of the core services offered by cloud platforms, as it allows developers to store any kind of unstructured data, conveniently and flexibly. Even if you’re not a developer, there are countless apps, libraries and tools that you can integrate in your daily workflow to leverage object storage on the cloud, for example for backups or just to store your data in a safe, reliable place.<\/p>\n\n\n\n

Microsoft Azure provides excellent object storage with\u202fAzure Blob Storage<\/a><\/strong>, which offers unmatched durability, virtually infinite capacity and multiple tiers of storage, all at very convenient rates. However, because Azure Blob Storage was developed before the world decided to \u201cstandardize\u201d on the S3 APIs, the two use different interfaces, and so most applications and libraries designed to work with Amazon S3 do not support Azure out-of-the-box. Many third-party and open source apps, libraries and tools are built to take advantage of S3, including very popular tools like s3cmd<\/a>; not all of them have built-in support for Azure Blob Storage, however.<\/p>\n\n\n\n

Enter Minio<\/h2>\n\n\n\n

The solution is incredibly simple, and it’s a free, lightweight, open source app called Minio<\/a><\/strong>: an object storage server that exposes S3-compatible APIs. Minio’s main goal was to simply expose local storage as object storage, but recently the developers have implemented a gateway feature that allows proxying requests to \u2014 you guessed it \u2014 Azure Blob Storage.<\/p>\n\n\n\n

In short, Minio allows us to convert Azure Blob Storage APIs to Amazon S3! Best of all: Minio itself is very lightweight (written in Golang), and it’s available as a Docker container.<\/p>\n\n\n\n

Running Minio as a Docker container is really simple:<\/p>\n\n\n

\n$ docker run -p 9000:9000 \\\n    -e \"MINIO_ACCESS_KEY=azureaccountname\" \\\n    -e \"MINIO_SECRET_KEY=azureaccountkey\" \\\n    minio\/minio gateway azure\n<\/pre><\/div>\n\n\n
Minio on Web Apps on Linux<\/h2>\n\n\n\n
Minio is a self-hosted solution, which means that you will need a server to run it. Thanks to Minio developers publishing Docker containers, however, a very simple and cost-effective solution is to use Azure Web Apps on Linux<\/a><\/strong> and custom container support<\/strong>.<\/p>\n\n\n\n
To deploy Minio on an Azure Web App, follow these simple steps. You will need the Azure CLI 2.0<\/a> (or “az cli”) installed on your machine, and, of course, an Azure subscription (or a free trial<\/a>).<\/p>\n\n\n\n
If you haven’t done it already, log in to Azure using the CLI with this command, then follow the instructions on the terminal to authenticate using the web browser:<\/p>\n\n\n
\n$ az login\n<\/pre><\/div>\n\n\n
The first thing we need to do is to create a Resource Group<\/strong>, which is nothing but a logical grouping unit where all of our Azure resources are deployed:<\/p>\n\n\n
\n$ az group create --name \"Minio\" --location \"WestUS\"\n<\/pre><\/div>\n\n\n
Next, we will create the actual Blob Storage Account<\/strong> in which our data will be stored; you can skip this if you already have a Storage Account that you want to use. There are multiple tiers of storage (hot vs cool), and multiple levels of redundancy (LRS, GRS, RA-GRS); for an explanation of the different options, I’m better off referring you to the official documentation<\/a>. In this example, we’re using a “Blob Storage Account” (a new kind of Storage Account that offers hot and cool storage, and lower rates for both), in LRS (Locally Redundant Storage: data is replicated 3 times within the same Azure datacenter) and “cool” tier (data is always online, but retrievals are charged a small fee per GB). Note that we’re adding two recurring parameters for the Resource Group and location as well. The full list of options for the command below is in the CLI reference<\/a>. Lastly, remember that the name of the storage account has to be globally<\/em> unique.<\/p>\n\n\n
\n$ az storage account create \\\n    --name \"aleminiostorage\" \\\n    --kind BlobStorage \\\n    --sku Standard_LRS \\\n    --access-tier Cool \\\n    --resource-group \"Minio\" \\\n    --location \"WestUS\"\n<\/pre><\/div>\n\n\n
After the account is created, we need to get the Account Key<\/strong>:<\/p>\n\n\n
\n$ az storage account show-connection-string \\\n    --name \"aleminiostorage\" \\\n    --resource-group \"Minio\"\n# Output\n{\n  \"connectionString\": \"DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net;AccountName=aleminiostorage;AccountKey=rOduFZr22jJ+...\"\n}\n<\/pre><\/div>\n\n\n
The Account Key is at the end of the connectionString<\/code> parameter (in bold above, truncated), and it’s a base64-encoded string. Take note of that, as we’ll need it in the next steps.<\/p>\n\n\n\n
It’s now time to deploy Minio to the Web App on Linux<\/strong>. First, we need to create an App Service Plan, which represents the managed VM(s) that will serve our app; after that, we’re creating a Web App inside it. We’re picking a “B1” (Basic 1) tier, which should be enough to run our lightweight Minio app; note also the --is-linux<\/code> flag, to create a Linux-based Web App. As in the case of the storage account, the name of the Web App (aleminio<\/code> in the example below) has to be globally unique too. We’re also configuring the Web App to run the minio\/minio<\/code> image from Docker Hub.<\/p>\n\n\n\n
Note: Traffic between your Azure Web App and the Azure Storage Account is free of charge, for as long as the two are in the same Azure Region.<\/p>\n\n\n
\n# Create the App Service Plan\n$ az appservice plan create \\\n    --name \"MinioAppPlan\" \\\n    --is-linux \\\n    --sku B1 \\\n    --resource-group \"Minio\" \\\n    --location \"WestUS\"\n# Create the Web App configured with the minio\/minio container\n$ az webapp create \\\n    --name \"aleminio\" \\\n    --deployment-container-image-name \"minio\/minio\" \\\n    --plan \"MinioAppPlan\" \\\n    --resource-group \"Minio\"\n<\/pre><\/div>\n\n\n
The Web App on Linux should now be up and running, at the URL webappname<\/em>.azurewebsites.net – in my example, https:\/\/aleminio.azurewebsites.net<\/code>.<\/p>\n\n\n\n
On the last step, let’s configure Minio on the Web App. First, we need to pass the configuration as environmental variables, similarly to what we did with the -e<\/code> flag in the Docker run command above. We then need to tell the Web App what command to execute on the Docker container to start Minio in gateway mode.<\/p>\n\n\n
\n# Environmental variables\n# The value for MINIO_ACCESS_KEY is the name of the Storage Account\n# Fill MINIO_SECRET_KEY with the Storage Account Key instead\n$ az webapp config appsettings set \\\n    --settings \"MINIO_ACCESS_KEY=aleminiostorage\" \"MINIO_SECRET_KEY=rOduFZr22jJ+...\" \"PORT=9000\" \\\n    --name \"aleminio\" \\\n    --resource-group \"Minio\"\n# Startup command\n$ az webapp config set \\\n    --startup-file \"gateway azure\" \\\n    --name \"aleminio\" \\\n    --resource-group \"Minio\"\n<\/pre><\/div>\n\n\n
We\u2019re done! Give the Web App a few seconds to start, then you\u2019ll have your Minio Amazon S3-compatible gateway working<\/strong>!<\/p>\n\n\n\n
Configure your client apps\/libraries<\/strong> with the following settings:<\/p>\n\n\n\n