DORA (DevOps Research and Assessment) offers a standardized set of metrics for assessing the performance and maturity of DevOps processes. These metrics offer insights into DevOps’ responsiveness to changes, the average code deployment time, iteration frequency, and understanding of failures. This guide outlines the four key DORA metrics, their significance, and how teams can utilize Open DevOps for performance measurement.

DORA metrics in DevOps

 

DORA began as a team within Google Cloud with a specific focus on evaluating DevOps performance. Their objective is to enhance performance and collaboration while increasing velocity. These metrics act as a tool for continuous improvement for DevOps teams worldwide. This can aid in setting goals based on current performance and then tracking progress towards those objectives.

DevOps plays a vital role in maintaining the smooth operation of business software and processes, allowing users to concentrate on their tasks. DORA metrics are essential for helping DevOps teams:

• Offer realistic response time estimates
• Enhance work planning
• pinpoint areas needing improvement
• Foster agreement on technical and resource investments

What do DevOps DORA metrics entail?

DORA metrics for DevOps teams concentrate on four key indicators:

• Deployment frequency
• Time elapsed between acceptance and deployment
• Deployment failure rate
• Time taken to restore service or recover from a failure

The subsequent section elaborates on why these metrics are considered best practices in DevOps, how they are measured, and strategies teams can employ to enhance their performance.

Deployment Frequency

DevOps teams typically release software in smaller, more frequent deployments. This allows to minimize the volume of changes and risks in each cycle. Increased deployment frequency enables teams to obtain feedback more quickly, leading to faster iterations.

Deployment frequency is the average number of completed code deployments per day to any specific environment. This serves as a gauge of DevOps’ overall efficiency, assessing the speed of the development team and their level of automation.

Diminishing the workload or the scope of each deployment can boost deployment frequency.

Lead Time for Changes

Lead time for changes evaluates the average speed at which the DevOps team delivers code, from commitment to deployment. It reflects the team’s efficiency, the complexity of the code, and DevOps’ overall responsiveness to environmental changes.

This metric allows businesses to quantify the speed of code delivery to the customer or business. For instance, some highly proficient teams might have an average lead time of 2-4 hours for changes, while others might take a week.

To reduce lead time for changes, it is beneficial to decrease the workload in the deployment, enhance code reviews, and augment automation.

Change Failure Rate

Change failure rate represents the percentage of deployments that result in a failure in production. While deployment frequency and lead time for changes are appropriate measures of DevOps automation and capabilities, they are only effective if the deployments are successful. The change failure rate serves as a counterbalance to frequency and speed.

This metric can be challenging to gauge because numerous deployments, particularly critical response ones, can introduce bugs in production. Understanding the seriousness and frequency of these issues helps DevOps teams balance stability with speed.

To decrease the change failure rate, reducing the amount of work in progress during deployment and increasing automation can be beneficial.

Calculation of Change Failure Rate

In GitLab, we determine the change failure rate as the percentage of deployments that cause an incident in production within a specified time frame.

GitLab computes the change failure rate by dividing the number of incidents by the number of deployments to a production environment. This calculation operates under the assumptions that:

• GitLab incidents are consistently tracked.
• All incidents, irrespective of the environment, are considered production incidents.

The change failure rate primarily serves as a high-level measure of stability, which is why all incidents and deployments within a day are aggregated into a combined daily rate. A proposal to establish specific relationships between deployments and incidents is outlined in issue 444295.

For instance, if there are 10 deployments (assuming one deployment per day) with two incidents on the first day and one incident on the last day, the change failure rate would be 0.3.

How to Enhance Change Failure Rate?

The initial step is to benchmark quality and stability across different groups and projects. Subsequently, you should focus on:

• Striking the appropriate balance between stability and throughput (Deployment frequency and Lead time for changes) without compromising quality for speed.
• Enhancing the effectiveness of code review procedures.
• Implementing automated testing.

Multi-Branch Rule for Lead Time for Changes

Unlike the standard calculation for lead time for changes, this calculation rule enables the measurement of multi-branch operations using a single deployment job for each operation.

For instance, transitioning from a development job on the development branch to a staging job on the staging branch, and finally to a production job on the production branch.

To implement this calculation rule, the dora_configurations table is updated with the target branches involved in the development flow. This allows GitLab to recognize these branches as a single unit and filter out other merge requests.

This configuration modifies how the daily DORA metrics are calculated for the selected project but does not impact other projects, groups, or users.

This feature is exclusively supported at the project level.

Measuring DORA Metrics

Measuring DORA metrics without relying on GitLab CI/CD pipelines: 

Deployment frequency is determined based on the deployment records generated for standard push-based deployments. These deployment records are not produced for pull-based deployments, such as when Container Images are linked to GitLab through an agent.

To monitor DORA metrics in these scenarios, you can establish a deployment record using the Deployments API. It is essential to specify the environment name where the deployment tier is configured since the tier variable is assigned to the particular environment, not the deployments.

Measuring DORA Metrics with Jira

Deployment frequency and lead time for changes are computed using GitLab CI/CD and Merge Requests (MRs) and do not depend on Jira data.

Time to restore service and change failure rate necessitate GitLab incidents for the calculation. For additional details, refer to the “Measure DORA Time to Restore Service and Change Failure Rate with External Incidents” and the Jira Incident Replicator Guide.

Measuring DORA Time to Restore Service and Change Failure Rate with External Incidents

For PagerDuty, you can configure a webhook to automatically generate a GitLab incident for each PagerDuty incident. This setup entails making adjustments in both PagerDuty and GitLab.
For other incident management tools, you can establish an HTTP integration to automatically:

• Generate an incident when an alert is activated.
• Close incidents through recovery alerts.

DORA Metrics and Value Stream Management

Value stream management focuses on delivering frequent, high-quality releases to customers, with a successful measure being the realization of value from the changes by the customer.

DORA metrics are integral to value stream management as they offer foundational measures for capturing:

• Deployment frequency
• Lead time for changes
• Failure rate
• Time to restore service

When paired with customer feedback, DORA metrics guide DevOps teams on where to concentrate their improvement efforts and how to position their services competitively.

After extensive research, Google’s DevOps Research and Assessment (DORA) team identified four key metrics to evaluate a team’s performance:

• Lead time for changes
• Deployment frequency
• Mean time to recovery
• Change failure rate

DORA metrics have become the industry standard for assessing the effectiveness of software development teams and can offer vital insights into areas for improvement. These metrics are crucial for organizations aiming to modernize and those striving for a competitive advantage. Below, we will delve into each metric to explore what they reveal about development teams.

Lead Time for Changes (LTC)

Lead time for changes (LTC) represents the duration between a commit and production. LTC serves as an indicator of a team’s agility, revealing not only the time taken to implement changes but also the team’s responsiveness to the continuously evolving demands and requirements of users.

In their “Accelerate State of DevOps 2021” report, the DORA team identified the following benchmarks for performance:

• Elite performers: Less than one hour
• High performers: One day to one week
• Medium performers: One month to six months
• Low performers: More than six months

LTC can expose signs of inadequate DevOps practices. If teams require weeks or even months to release code into production, it indicates inefficiencies in their process. To reduce LTC, it is essential to implement continuous integration and continuous delivery (CI/CD).

It is beneficial to foster close collaboration between testers and developers to ensure a comprehensive understanding of the software. Additionally, developing automated tests can further save time and enhance the CI/CD pipeline.

Given the multiple phases between the initiation and deployment of a change, it is prudent to define each step of the process and monitor the duration of each. Analyzing the cycle time provides a comprehensive view of the team’s functioning and offers further insights into potential time-saving opportunities.

However, it is crucial not to compromise the quality of software delivery in pursuit of faster changes. While a low LTC may suggest that a team is efficient, if they are unable to support the changes they implement or are operating at an unsustainable pace, they risk undermining the user experience.

Instead of comparing the team’s Lead Time for Changes with that of other teams or organizations, it is advisable to evaluate this metric over time and view it as an indicator of growth or stagnation.

Deployment Frequency (DF)

Deployment frequency (DF) refers to the frequency of  changes, reflecting the consistency of your software delivery. This metric is useful for assessing whether a team is achieving its goals for continuous delivery. According to the DORA team, the benchmarks for Deployment Frequency are:

• Elite performers: Multiple times a day
• High performers: Once a week to once a month
• Medium performers: Once a month to once every six months
• Low performers: Less than once every six months

The most effective way to improve DF is by shipping a series of small changes, which offers several advantages. A high deployment frequency may uncover bottlenecks in the development process or suggest that projects are overly complex.

Frequent shipping indicates that the team is continually refining their service, and if there is a code issue, it is easier to identify and address.

For larger teams, this approach may not be practical. Instead, one might consider establishing release trains and shipping at regular intervals. This strategy enables the team to deploy more frequently without overwhelming team members.

Mean Time to Recovery (MTTR)

Mean time to recovery (MTTR) is the average duration it takes for your team to restore service following a disruption, such as an outage. This metric provides insight into both the stability of your software and the agility of your team when faced with challenges. The benchmarks identified in the State of DevOps report are:

• Elite performers: Less than one hour
• High performers: Less than one day
• Medium performers: One day to one week
• Low performers: Over six months

To minimize the impact of service degradation on your value stream, downtime should be kept to a minimum. If your team takes more than a day to restore services, it may be beneficial to implement feature flags.

This allows you to quickly disable a change without causing significant disruption. Additionally, shipping in small batches can make it easier to identify and resolve issues.

Although mean time to detect (MTTD) differs from mean time to recovery, your team’s detection time influences your MTTR—the quicker your team identifies an issue, the faster service can be restored.

Similar to lead time for changes, it is important not to implement sudden changes at the expense of a quality solution. Instead of deploying a quick fix, ensure that the change you are implementing is durable and comprehensive. It is advisable to track MTTR over time to monitor your team’s improvement and strive for consistent, stable growth.

Change Failure Rate (CFR)

Change failure rate (CFR) represents the percentage of releases that lead to downtime, degraded service, or rollbacks, offering insights into a team’s effectiveness in implementing changes. The performance benchmarks for CFR are as follows:

• Elite performers: 0-15%
• High, medium, and low performers: 16-30%

Change Failure Rate is a particularly valuable metric as it prevents teams from being misled by the total number of failures they encounter. Teams that are not implementing many changes will experience fewer failures, but this does not necessarily mean they are more successful with the changes they do deploy.

Conversely, teams following CI/CD practices may experience a higher number of failures. If CFR is low, these teams will have an advantage due to the speed of their deployments and their overall success rate.

This rate can also have significant implications for the value stream. This indicates how much time is spent on solving issues rather than developing new projects.

As high, medium, and low performers all fall within the same range, it is best to set goals based on the team and the specific business rather than comparing with other organizations.

Putting it All Together with DORA Metrics

As with any data, DORA metrics require context. It is essential to consider the story that all four of these metrics collectively convey. Lead time for changes and deployment frequency offer insights into a team’s velocity and responsiveness to the evolving needs of users.

Conversely, mean time to recovery and change failure rate indicate the stability of a service and the team’s responsiveness to service outages or failures.

By comparing all four key metrics, one can assess how well their organization balances speed and stability. For instance, if the LTC is within a week with weekly deployments but the change failure rate is high. This suggests that teams may be releasing changes prematurely or may not be able to support the changes they are deploying.

On the other hand, if deployments occur once a month and both MTTR and CFR are high, the team may be spending more time correcting code than enhancing the product.

Because DORA metrics offer a high-level view of a team’s performance, they can be invaluable for organizations striving to modernize. DORA metrics can pinpoint areas that require improvement and provide guidance on how to enhance them. Over time, teams can gauge their growth and identify areas that have stagnated.

However, it is crucial to remember that data will only take you so far. To derive the most value from DORA metrics, engineering leads must understand their organization and teams and use this knowledge to guide their goals and determine how to effectively allocate their resources.

[Want to learn more about DevOps DORA Metrics? Click here to reach us.]

Conclusion

DORA metrics have become essential in DevOps, offering a comprehensive view of performance and enabling organizations to optimize their software delivery processes effectively. Bobcares’ advanced DevOps support provides an easy way for businesses to access and leverage the full range of DevOps functionalities.

By partnering with Bobcares, organizations can harness the power of DORA metrics to identify areas for improvement, streamline their practices, and achieve greater efficiency and reliability in software delivery.

As the number of elite performers continues to grow, implementing these best practices with Bobcares’ support positions organizations to drive innovation, enhance operational excellence, and gain a competitive edge in the market.