What are use cases for Monitoring-as-code?<\/h3>\n\n\n\n\n- Continuous performance feedback<\/strong>: by collecting these metrics in an automated and consistent way, the PerfSpec result provides continuous performance feedback to engineers, allowing them to define what information they are interested in and control that information using source control repositories such as GIT.<\/li>\n\n\n\n
- Quality gates<\/strong>: by adding threshold criteria for each defined metric, these same metrics can be used to stop the code delivery pipeline by adding rules to determine \u201cpass\u201d or \u201cfail\u201d for the pipeline stage. We refer to these as quality gates and they provide the benefits of:<\/li>\n<\/ol>\n\n\n\n
\n- \n
\n- enforcing standards<\/li>\n\n\n\n
- eliminating manual review of logs and dashboards for every build<\/li>\n\n\n\n
- increasing confidence for build quality<\/li>\n\n\n\n
- allowing the team to receive targeted information for triaging issues<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
The diagram below expresses this concept:<\/p>\n\n\n![\"Quality](\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2019\/04\/Dynatrace-image-1-1024x491.webp\")
Quality gates in a software delivery pipeline contain rules to determine “pass or “fail” within the pipe<\/figcaption><\/figure>\n\n\n\n3. Automated deployment \/ Canary validation<\/strong>: the use of a PerfSpec is not limited to code delivery pipelines. Following a deployment to an environment, health check queries can be made using a PerfSpec to determine if all is well with the application. This is perfect for Canary deployment models where these metrics can inform the decision to enable the feature to all users or turn it off.<\/p>\n\n\n\nThese are just a few use cases for inspiration. See the 2019 Performance Advisory Council presentation slides for additional ideas \u201cPerformance as Code \u2013 Let\u2019s Make it a Standard<\/a>.\u201d<\/p>\n\n\n\nGetting started with Pitometer<\/h3>\n\n\n\n
Pitometer<\/a> is an open source project that is designed to qualify an application or service by using a well-defined performance specification. This performance specification file, referred to as PerfSpec, is a declarative way to define which metrics to monitor. Pitometer will parse the PerfSpec, pull the specified metrics, evaluate them, and give you a pass, fail, or warning result for each metric, as well as across all metrics where Pitometer calculates a total observation score.<\/p>\n\n\n\nPitometer is not an application, rather a set of several Node.js modules that one needs to include and incorporate within a custom Node.js client. Pitometer libraries are modular with extensible metric sources and graders. Right now, there are source modules for Dynatrace and Prometheus and a grader for thresholds, but it’s easy to write new sources and graders.<\/p>\n\n\n\n
We have developed two open source Pitometer client implementations that can be easily implemented within your CI tool, whether it be Jenkins, Azure DevOps, Bamboo, or something else.<\/p>\n\n\n\n
\n- The first implementation is a Dockerized webservice that can be deployed to your desktop, any VM running Docker, Azure container application, or Kubernetes<\/li>\n\n\n\n
- The second implementation is an Azure function application for a serverless option as Azure<\/li>\n<\/ol>\n\n\n\n
Both implementations accept an HTTP POST request to the service with these elements in a JSON structure.<\/p>\n\n\n\n
Pitometer client processing example<\/h3>\n\n\n\n
This example POST request defines the start\/stop time frame of the request along with the PerfSpec comprised of two indicators. Each indicator defines the sources as Dynatrace, the query of the metric sought, grading thresholds, and metric score. At the end of the request is an overall \u201cPerfSpec\u201d objective.<\/p>\n\n\n![\"The](\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2019\/04\/Dynatrace-image-2-1024x671.webp\")
The Pitometer request defines the start\/stop time frame of the request along with the PerfSpec<\/figcaption><\/figure>\n\n\n\nThe HTTP response is a JSON body with a result of \u201cpass,\u201d \u201cwarning,\u201d or \u201cfail.\u201d In our example response below, the sum of the indicator scores is between the pass and warning overall objectives, so the PerfSpec result is a \u201cwarning.\u201d If there were no violations, then the indicator would receive the total metricScore and the violations array would be empty.<\/p>\n\n\n![\"Pitometer](\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2019\/04\/Dynatrace-image-3-1024x649.webp\")
Pitometer response – this PerfSpec result is a “warning”<\/figcaption><\/figure>\n\n\n\nTry Pitometer locally<\/h3>\n\n\n\n
Follow these steps to call Pitometer using a PerfSpec from your laptop.<\/p>\n\n\n\n
\n- To provision the resources to host your application and configure these resources with the Dynatrace OneAgent for monitoring, click here for a Dynatrace Free Trial<\/a>. See these instructions<\/a> for how to generate the Dynatrace API token that Pitometer requires.<\/li>\n\n\n\n
- With Docker installed, just do a \u201cdocker pull\u201d of the pre-built Pitometer webservice image. See these instructions<\/a> for how to configure and run Docker locally.<\/li>\n\n\n\n
- Using a tool like Postman or VS Code with the \u201cREST client\u201d extension, open an example REST file.<\/li>\n<\/ol>\n\n\n\n
You will need to adjust the start\/stop times and then send the POST request to http:\/\/localhost:8080\/api\/pitometer<\/p>\n\n\n![\"VS](\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2019\/04\/Dynatrace-image-4-1024x473.webp\")
A VS Code REST Client extension<\/figcaption><\/figure>\n\n\n![\"Image](\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2019\/04\/Dynatrace-image-5-1024x725.webp\")
This shows a Pitometer request and response in VS Code<\/figcaption><\/figure>\n\n\n\nImplementing Pitometer quality gate in a pipeline<\/h3>\n\n\n\n
Below is a how to add a PerfSpec and quality gate using Pitometer into you code delivery pipelines.<\/p>\n\n\n\n![\"Pitometer](\"https:\/\/opensource.microsoft.com\/blog\/wp-content\/uploads\/2019\/04\/Dynatrace-image-6-1024x428.png\")
Pitometer quality gate in a pipeline<\/figcaption><\/figure>\n\n\n\nFollow these steps to add a quality gate using the \u201cPerfSpec\u201d file and Pitometer:<\/p>\n\n\n\n
\n- To provision the resources to host your application and configure these resources with the Dynatrace OneAgent for monitoring, click here for a Dynatrace Free Trial<\/a>. See these instructions<\/a> for how to generate a Dynatrace API token that Pitometer requires.<\/li>\n\n\n\n
- Provision the resources to host the Pitometer service and deploy Pitometer to it. There are multiple ways to deploy the Pitometer implementations such as:<\/li>\n<\/ol>\n\n\n\n
\n- \n
\n- Docker in a VM<\/li>\n\n\n\n
- Docker in a Azure container service<\/li>\n\n\n\n
- Docker in Kubernetes<\/li>\n\n\n\n
- Azure function application<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
See the README in the Pitometer web service and Pitometer Azure function GIT repo for more details.<\/p>\n\n\n\n
\n- Create a \u201cPerfSpec\u201d file and check it into your source code repo. Note: \u00a0there is an example PerfSpec file in the Pitometer reference implementation GIT samples folder<\/li>\n\n\n\n
- In the build pipeline, add a task that will add the \u201cPerfSpec\u201d as a built artifact<\/li>\n\n\n\n
- In the release pipeline import the build artifact. Then before the load test step, save the timestamp into a variable.\u00a0 After the load test, save timestamp into a variable.\u00a0 Then call the Pitometer service with the start and stop timestamp and the PerfSpec file contained in the artifact.<\/li>\n<\/ol>\n\n\n\n
Next steps<\/h3>\n\n\n\n
We envision the either the Pitometer web services or Pitometer Azure function application to be great for demos of monitoring-as-code and how-to building resiliency in your software delivery pipelines.<\/p>\n\n\n\n
Our example services are stateless and will scale to support multiple build pipelines for a given business critical application and shared service across applications and teams. The code is open source, so feel free to extend it to support other use cases and share back what you did.<\/p>\n\n\n\n
Please see the README files in these Pitometer example projects for more details to get started.<\/p>\n\n\n\n
- \n
- \n
- \n
- enforcing standards<\/li>\n\n\n\n
- eliminating manual review of logs and dashboards for every build<\/li>\n\n\n\n
- increasing confidence for build quality<\/li>\n\n\n\n
- allowing the team to receive targeted information for triaging issues<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
The diagram below expresses this concept:<\/p>\n\n\n
Quality gates in a software delivery pipeline contain rules to determine “pass or “fail” within the pipe<\/figcaption><\/figure>\n\n\n\n 3. Automated deployment \/ Canary validation<\/strong>: the use of a PerfSpec is not limited to code delivery pipelines. Following a deployment to an environment, health check queries can be made using a PerfSpec to determine if all is well with the application. This is perfect for Canary deployment models where these metrics can inform the decision to enable the feature to all users or turn it off.<\/p>\n\n\n\n
These are just a few use cases for inspiration. See the 2019 Performance Advisory Council presentation slides for additional ideas \u201cPerformance as Code \u2013 Let\u2019s Make it a Standard<\/a>.\u201d<\/p>\n\n\n\n
Getting started with Pitometer<\/h3>\n\n\n\n
Pitometer<\/a> is an open source project that is designed to qualify an application or service by using a well-defined performance specification. This performance specification file, referred to as PerfSpec, is a declarative way to define which metrics to monitor. Pitometer will parse the PerfSpec, pull the specified metrics, evaluate them, and give you a pass, fail, or warning result for each metric, as well as across all metrics where Pitometer calculates a total observation score.<\/p>\n\n\n\n
Pitometer is not an application, rather a set of several Node.js modules that one needs to include and incorporate within a custom Node.js client. Pitometer libraries are modular with extensible metric sources and graders. Right now, there are source modules for Dynatrace and Prometheus and a grader for thresholds, but it’s easy to write new sources and graders.<\/p>\n\n\n\n
We have developed two open source Pitometer client implementations that can be easily implemented within your CI tool, whether it be Jenkins, Azure DevOps, Bamboo, or something else.<\/p>\n\n\n\n
- \n
- The first implementation is a Dockerized webservice that can be deployed to your desktop, any VM running Docker, Azure container application, or Kubernetes<\/li>\n\n\n\n
- The second implementation is an Azure function application for a serverless option as Azure<\/li>\n<\/ol>\n\n\n\n
Both implementations accept an HTTP POST request to the service with these elements in a JSON structure.<\/p>\n\n\n\n
Pitometer client processing example<\/h3>\n\n\n\n
This example POST request defines the start\/stop time frame of the request along with the PerfSpec comprised of two indicators. Each indicator defines the sources as Dynatrace, the query of the metric sought, grading thresholds, and metric score. At the end of the request is an overall \u201cPerfSpec\u201d objective.<\/p>\n\n\n
The Pitometer request defines the start\/stop time frame of the request along with the PerfSpec<\/figcaption><\/figure>\n\n\n\n The HTTP response is a JSON body with a result of \u201cpass,\u201d \u201cwarning,\u201d or \u201cfail.\u201d In our example response below, the sum of the indicator scores is between the pass and warning overall objectives, so the PerfSpec result is a \u201cwarning.\u201d If there were no violations, then the indicator would receive the total metricScore and the violations array would be empty.<\/p>\n\n\n
Pitometer response – this PerfSpec result is a “warning”<\/figcaption><\/figure>\n\n\n\n Try Pitometer locally<\/h3>\n\n\n\n
Follow these steps to call Pitometer using a PerfSpec from your laptop.<\/p>\n\n\n\n
- \n
- To provision the resources to host your application and configure these resources with the Dynatrace OneAgent for monitoring, click here for a Dynatrace Free Trial<\/a>. See these instructions<\/a> for how to generate the Dynatrace API token that Pitometer requires.<\/li>\n\n\n\n
- With Docker installed, just do a \u201cdocker pull\u201d of the pre-built Pitometer webservice image. See these instructions<\/a> for how to configure and run Docker locally.<\/li>\n\n\n\n
- Using a tool like Postman or VS Code with the \u201cREST client\u201d extension, open an example REST file.<\/li>\n<\/ol>\n\n\n\n
You will need to adjust the start\/stop times and then send the POST request to http:\/\/localhost:8080\/api\/pitometer<\/p>\n\n\n
A VS Code REST Client extension<\/figcaption><\/figure>\n\n\n This shows a Pitometer request and response in VS Code<\/figcaption><\/figure>\n\n\n\n Implementing Pitometer quality gate in a pipeline<\/h3>\n\n\n\n
Below is a how to add a PerfSpec and quality gate using Pitometer into you code delivery pipelines.<\/p>\n\n\n\n
Pitometer quality gate in a pipeline<\/figcaption><\/figure>\n\n\n\n Follow these steps to add a quality gate using the \u201cPerfSpec\u201d file and Pitometer:<\/p>\n\n\n\n
- \n
- To provision the resources to host your application and configure these resources with the Dynatrace OneAgent for monitoring, click here for a Dynatrace Free Trial<\/a>. See these instructions<\/a> for how to generate a Dynatrace API token that Pitometer requires.<\/li>\n\n\n\n
- Provision the resources to host the Pitometer service and deploy Pitometer to it. There are multiple ways to deploy the Pitometer implementations such as:<\/li>\n<\/ol>\n\n\n\n
- \n
- \n
- \n
- Docker in a VM<\/li>\n\n\n\n
- Docker in a Azure container service<\/li>\n\n\n\n
- Docker in Kubernetes<\/li>\n\n\n\n
- Azure function application<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
See the README in the Pitometer web service and Pitometer Azure function GIT repo for more details.<\/p>\n\n\n\n
- \n
- Create a \u201cPerfSpec\u201d file and check it into your source code repo. Note: \u00a0there is an example PerfSpec file in the Pitometer reference implementation GIT samples folder<\/li>\n\n\n\n
- In the build pipeline, add a task that will add the \u201cPerfSpec\u201d as a built artifact<\/li>\n\n\n\n
- In the release pipeline import the build artifact. Then before the load test step, save the timestamp into a variable.\u00a0 After the load test, save timestamp into a variable.\u00a0 Then call the Pitometer service with the start and stop timestamp and the PerfSpec file contained in the artifact.<\/li>\n<\/ol>\n\n\n\n
Next steps<\/h3>\n\n\n\n
We envision the either the Pitometer web services or Pitometer Azure function application to be great for demos of monitoring-as-code and how-to building resiliency in your software delivery pipelines.<\/p>\n\n\n\n
Our example services are stateless and will scale to support multiple build pipelines for a given business critical application and shared service across applications and teams. The code is open source, so feel free to extend it to support other use cases and share back what you did.<\/p>\n\n\n\n
Please see the README files in these Pitometer example projects for more details to get started.<\/p>\n\n\n\n
- Provision the resources to host the Pitometer service and deploy Pitometer to it. There are multiple ways to deploy the Pitometer implementations such as:<\/li>\n<\/ol>\n\n\n\n
- With Docker installed, just do a \u201cdocker pull\u201d of the pre-built Pitometer webservice image. See these instructions<\/a> for how to configure and run Docker locally.<\/li>\n\n\n\n
- To provision the resources to host your application and configure these resources with the Dynatrace OneAgent for monitoring, click here for a Dynatrace Free Trial<\/a>. See these instructions<\/a> for how to generate the Dynatrace API token that Pitometer requires.<\/li>\n\n\n\n