Getting Buy-In for DORA Metrics

EverQuote prides itself on having a data-driven culture. But even with this organization-wide commitment, its engineering team initially struggled to embrace metrics that would help them understand and improve performance. Many leaders would give up after a failed attempt, but Virginia Toombs, VP of Engineering Operations, took a step back to understand what went wrong so they could try again — and succeed.

Along the way, the EverQuote team learned what to avoid when implementing engineering metrics and how to successfully roll them out. For them, it was all about empowering team members, collecting actionable data in the right platform, and correlating metrics for a more holistic view of what was happening across the organization.

Lessons Learned About Measuring Engineering Productivity

When EverQuote decided to measure engineering productivity a few years ago, it started by purchasing a tool, like many organizations commonly do. But it encountered problems as it hadn’t considered what to measure or how to leverage those insights to improve performance. Because EverQuote didn’t know which engineering metrics best suited its unique team and processes, it ended up with a tool that didn’t have mature throughput or flow metrics — two things it would learn were core to its success. The result? Virginia and her team saw detailed engineering metrics but lacked a comprehensive view of the organization’s performance.

This issue caused a domino effect. Measuring only granular metrics made team members feel that individual performance was being judged, rather than the process itself, and enthusiasm for the program came to a halt. That’s when the engineering operations team decided to rethink the approach and start from scratch.

Using Metrics to Improve Developer Experience

EverQuote’s engineering operations team is a central function within engineering whose main goal is to create an environment where engineers can thrive. This team optimizes processes, encourages collaboration, and coaches on agile techniques. For them, it’s essential to understand how engineering processes are performing so they can make data-driven decisions to improve. This team made two important decisions when rolling out engineering metrics for the second time.

First, they took the time to understand which engineering metrics applied to their organization. Rather than starting with granular metrics, they decided to lead with the big picture, adopting the four original DORA metrics: Deployment Frequency (DF), Mean Lead Time for Changes (MLTC), Mean Time to Recover (MTTR), and Change Failure Rate (CFR). From these high-level metrics, they would still be able to identify bottlenecks or issues and drill down into more granular metrics as needed.

To support DORA, and to provide visibility into its corresponding metrics, EverQuote adopted Code Climate. With Code Climate's Software Engineering Intelligence platform, they could identify organizational trends, look at data by teams or applications, and dig into specific DORA metrics. For example, if they see that MLTC is high, they can click into it to see exactly where the holdup is — maybe a long Time to Open or Time to First Review is preventing the PRs from getting to production as expected. Starting at a high level helps them understand their systems holistically, and then they can drill down as needed, which is more efficient and saves team members from metric fatigue.

Second, they empowered teams to own their metrics by educating them in how to read and interpret the data, and creating processes to discuss performance at the end of a sprint. They held these conversations as a team, not during one-on-ones, and focused on how they could better collaborate to improve as a unit. This strategy exemplifies one of EverQuote’s core principles: If you work as a team, you succeed as a team.

Successfully Implementing DORA DevOps Metrics

The EverQuote journey to measurement has come full circle. Now, engineers embrace engineering metrics as a tool for continuous improvement. After two iterations of implementing metrics, the team has learned three major lessons for successful adoption:

• Collect data you plan to act on. Although measuring and tracking every possible engineering metric may be tempting, it can prevent you from seeing the forest for the trees. Instead, be intentional about the metrics that your organization can derive insights from to take action.
• Correlate metrics and drill down as needed. Measuring DORA metrics gives EverQuote a full view of how engineering systems work at any given time. Being able to double-click into them in Code Climate's platform lets them quickly identify and resolve the root cause of an issue when it arises.
• Use consistent data. EverQuote has 17 engineering teams spread across multiple functions and locations. To maintain consistency, they align on how metrics will be defined and calculated. This process is essential to ensure they speak the same language and can benchmark against other teams and the industry.

Combining DORA DevOps metrics with other engineering metrics in Code Climate's insights platform has helped EverQuote nurture its data-driven culture. To learn more about successfully rolling out engineering metrics within your organization, request a consultation.

Understand and Optimize Developer Productivity with the SPACE Framework

Engineering productivity is notoriously difficult to measure. Classic metrics tend to focus exclusively on outputs, but this approach is flawed — in software engineering, quantity of output isn’t necessarily the most productive goal. Quality is an important complement to quantity, and a team that’s measuring success in lines of code written may sacrifice conciseness for volume. This can lead to bloated, buggy code. A team focused on the number of features shipped may prioritize simple features that they can get out the door quickly, rather than spending time on more complex features that can move the needle for the business. The SPACE Framework aims to address this by replacing output-focused measurements with a more nuanced approach to understanding and optimizing developer productivity.

This research-based framework offers five key dimensions that, viewed in concert, offer a more comprehensive view of a team’s status. These dimensions guide leaders toward measuring and improving factors that impact outputs, rather than focusing solely on the outputs themselves.

Importantly, SPACE foregrounds the developer. It recognizes that leaders must empower their team members with the ability to take ownership of their work, grow their skills, and get things done while contributing to the success of the team as a whole.

What is the SPACE Framework?

The SPACE Framework is a systematic approach to measuring, understanding, and optimizing engineering productivity. Outlined by researchers from GitHub, Microsoft, and the University of Victoria, it encourages leaders to look at productivity holistically, placing metrics in conversation with each other and linking them to team goals. It breaks engineering productivity into five dimensions, Satisfaction and Well-being; Performance; Activity; Communication and Collaboration; and Efficiency and Flow.

Satisfaction and Well-being – Most often measured by employee surveys, this dimension asks whether team members are fulfilled, happy, and practicing healthy work habits. Satisfaction and well-being are strongly correlated with productivity, and they may even be important leading indicators; unhappy teams that are highly productive are likely headed toward burnout if nothing is done to improve their well-being.

Of all five SPACE dimensions, satisfaction and well-being is the hardest to quantify. Objective metrics can’t directly measure developer happiness, but they can point to circumstances that are likely to cause dissatisfaction or burnout. By viewing quantitative metrics in tandem with qualitative information and survey data, engineering leaders can better assess team satisfaction.

With Code Climate's Software Engineering Intelligence (SEI) solution, leaders can view engineering metrics like Coding Balance to get a sense of developer workload. Coding Balance displays the percentage of contributors responsible for 80% of your team’s most impactful work. If work is not evenly distributed, overloaded team members may be overwhelmed, while those doing less impactful work may not be challenged enough.

Another informative measure, Work in Progress PRs, offers a count of Pull Requests with activity over a 72-hour period. A high Work in Progress number could mean that developers are forced to context switch often, which can prevent developers from entering a state of flow. The ideal number of Work in Progress PRs is usually 1-2 PRs per developer.

To get a better understanding of overall satisfaction, engineering leaders should look at both short term changes and overall trends. For example, when an engineering manager at Code Climate noticed a spike in volume of PRs and reviews in a short amount of time, she took it as a sign to further investigate the workload of each developer. When she discovered that engineers were working overtime and on weekends in order to ship new features, she successfully advocated for extra time off to recharge and avoid potential burnout.

Performance –

The originators of the SPACE Framework recommend assessing performance based on the outcome of a developer’s work. This could be a measure of code quality, or of the impact their work has on the product’s success.

Metrics that help evaluate performance include Defect Rate and Change Failure Rate. Change Failure Rate is a key performance metric, as it measures the percentage of deployments causing a failure in production, helping engineering leaders understand the quality of the code developed and shipped to customers. Every failure in production takes away time from developing new features and ultimately has a negative impact on customers.

When assessing team performance, PR Throughput is an important counterpoint to quality metrics. This metric counts PRs merged (a bug fix, new feature, or improvement) over time, and is correlated to output and progress. To get a full picture of performance, PR Throughput should be viewed alongside quality metrics; you’ll want to ensure that your team is delivering at volume while maintaining quality.

Activity –

This dimension is most reminiscent of older measures of productivity as it refers to developer outputs like on-call participation, pull requests opened, volume of code reviewed, or documents written. Still, the framework reminds leaders that activity should not be viewed in isolation, and it should always be placed in context with qualitative information and other metrics.

Useful Activity metrics to look at include coding metrics, such as Commits Per Day, the average number of times Active Contributors commit code per coding day, and Pushes Per Day, the average number of times Active Contributors are pushing per day.

These metrics offer an overview of developer activity so you can gauge whether or not work is balanced, if workload matches expectations, and whether delivery capacity is in line with strategic goals.

It can also be helpful to look at Deployment Frequency, which measures how often teams deliver new features or bug fixes to production.

Communication and Collaboration –

The most effective teams have a high degree of transparency and communication. This helps ensure developers are aligned on priorities, understand how their work fits into broader initiatives, and can learn from each other. Proxies for measuring communication and collaboration might include review coverage or documentation quality.

Code Climate supports Pair Programming, which encourages collaboration. Credit can be attributed to two developers who write code in a pairing session.

Metrics like Review Speed, Time to first Review, and Review Coverage, can help you assess the health of your collaborative code review process. These metrics can help identify whether your team is keeping collaboration top of mind, has enough time to review PRs, and is giving thoughtful feedback in comments.

Efficiency and Flow –

In the SPACE framework, flow refers to a state of individual efficiency where work can be completed quickly, with limited interruption. Efficiency is similar, but it occurs at the team level. Both are important for minimizing developer frustration. (Though it’s worth noting that when an organization has too much efficiency and flow, it’s not always a good thing, as it may be at the expense of collaboration or review.) Perceived efficiency and flow can be important information gathered via survey, while speed metrics can capture a more objective measurement.

In Code Climate's SEI platform, Cycle Time is an important metric to view within the context of efficiency and flow, as well as Mean Lead Time for Changes. Cycle Time, aka your speedometer, is representative of your team’s time to market. A low Cycle Time often means a higher output of new features and bug fixes being delivered to your customers. Cycle Time is an objective way to evaluate how process changes are affecting your team’s output. Often, a high output with quality code can speak to a team’s stability.

Mean Lead Time for Changes, which refers to DevOps speed, measures how quickly a change in code makes it through to production successfully. This can be correlated to team health and ability to react quickly. Like Cycle Time, it can indicate whether your team’s culture and processes are effective in handling a large volume of requests.

How can you implement the SPACE Framework?

The framework’s authors caution that looking at all five dimensions at once is likely to be counterproductive. They recommend choosing three areas that align with team priorities and company goals. What a team chooses to measure will send a strong signal about what the team values, and it’s important to be deliberate about that decision. They also remind leaders to be mindful of invisible work and to check for biases in their evaluations.

At Code Climate, we’ve spent years helping software engineering leaders leverage engineering data to boost their teams’ productivity, efficiency, and alignment. With that experience in mind, we have a few additional recommendations:

• Be transparent. When introducing data in your organization or using data in a new way, it’s important to be open. Make sure your team members know what data you’re looking at, how you plan to use it, and what value you hope to gain from it.
• Place all metrics in context. Quantitative data can only tell you so much. It’s critical to supplement metrics with qualitative information, or you’ll miss key insights. For example, if you’re measuring activity metrics and see a decrease in pull requests opened, it’s important to talk to your team members about why that’s happening. You may find out that they’ve been spending more time in meetings, so they can’t find time to code. Or you may learn that they’re coding less because they’re working to make sense of the technical direction. Both these situations may impact activity, but will need to be addressed in very different ways.
• Work data into your existing processes. The most effective way to build a new habit is to layer it onto an existing one. If you’re already holding regular 1:1s, standups, and retros, you can use them as an opportunity to present data to your team, assess progress toward goals, and discuss opportunities for improvement. By bringing data to your meetings, you can help keep conversations on track and grounded in facts.

Why does the SPACE Framework matter?

The SPACE Framework helps engineering leaders think holistically about developer productivity. The five dimensions serve as critical reminders that developer productivity is about more than the work of one individual, but about the way a team comes together to achieve a goal. And perhaps somewhat counterintuitively, understanding productivity is about more than simply measuring outputs. In looking at multiple dimensions, teams can gain a fuller understanding of the factors influencing their productivity, and set themselves up for sustainable, long-term success.

Learn how you can use data to enhance engineering performance and software delivery by requesting a consultation.

How Upskilling Helps Retain Top Engineering Talent and Build High-Performance Teams

Recent headlines might lead one to conclude that it’s more difficult than ever to build a high-performing team. Hiring is increasingly competitive, salaries are on the rise, and a growing number of people are choosing to switch jobs or exit the workforce entirely. But building a stellar team is about more than just recruiting great talent — it’s about investing in the talent you have. And that’s not investment in the financial sense (though salaries and benefits are important!), it’s a commitment to coaching and upskilling your existing team.

Focusing on professional development is a win-win. Helping developers excel will boost team performance. Ensuring that developers feel both challenged and supported will increase their job satisfaction and make them more likely to stick around.

How Can You Help Developers Upskill?

Of course, helping engineers level up their skills is a multi-layered process. Time and money set aside for learning are important, but it’s not enough. As a leader, there are things you can do to create a culture where positive feedback is welcomed, missteps are seen as learning opportunities, and developers feel comfortable openly discussing their professional goals. Once the cultural foundation is set, you can make adjustments to incorporate coaching into your team’s processes and help ensure that it remains a priority.

Culture Is Key to Leveling Up

Psychological safety is a prerequisite to the success of any coaching or professional development initiatives. In order for developers to have honest conversations about their career goals, or to be comfortable receiving feedback, they must trust that they will not be penalized for aspirations that are out of alignment with current responsibilities.

Though psychological safety is essential, it is just a baseline. An organization looking to prioritize professional development may also benefit from adopting elements of Continuous Improvement. In Continuous Improvement, every member of a team is on the lookout for opportunities to make incremental improvements. The underlying belief is that even small changes to processes, products, and more can have a big impact.

At the individual level, it would be detrimental to engage every team member in a conversation about one engineer’s professional development. The critical takeaway from Continuous Improvement is that improving should not be a top-down process. When it comes to coaching, it’s important to empower individuals with an active role in their professional development. They can actively contribute by identifying areas of incremental improvement, making plans for their own development, and setting and tracking progress toward goals. When they are involved in making plans, they’ll be more likely to see them through. As they realize the value of making small, positive changes, they’ll be motivated to keep learning.

Create Regular Touchpoints

At the process level, effective upskilling requires consistent check-ins and conversations. Regular 1:1s are a great opportunity to surface opportunities for upskilling and to evaluate progress toward goals. Come prepared with observations and discussion points, and encourage your team members to do the same. Give them the chance to raise their questions and concerns first, so you can get a more complete understanding of what blockers are impacting them the most, and what skills they’d most like to improve. Make their goals a priority whenever possible, and seek out opportunities to challenge team members to envision how their goals align with business priorities.

These touchpoints will be most effective when a baseline of safety has already been established, though it’s still important to be proactive about reinforcing trust during 1:1s. Practicing vulnerability can help establish the right tone. You may also want to remind team members that 1:1s are not meant for work-related status updates but for bigger picture conversations about their role, skills, and aspirations.

Leverage Data To Coach More Effectively

Leaders can supplement qualitative conversations with Engineering Intelligence data from a platform like Code Climate. With the help of objective data, it’s possible to cut through biases, check assumptions, and more accurately assess how a particular developer is working.

For example, you may observe that a particular team member rarely contributes in meetings and only speaks when spoken to. You may conclude that this team member is not engaged or invested in their work, or that they don’t value collaboration. Engineering data can help you test that hypothesis. You might find that this same team member is an active participant in Code Reviews, frequently leaving thorough, impactful feedback for their peers. Where you once might have encouraged this team member to be more collaborative, you can now offer more specific feedback around participating in meetings. Alternatively, you may decide to accept their participation in reviews as evidence of their commitment to teamwork and instead work with them on another area of growth.

You can also use engineering data to identify specific units of work that may present learning opportunities. For example, if you notice that a developer has an abnormally large or long-running PR, you can have a conversation about the circumstances that are drawing things out. This allows you to surface potential anti-patterns or areas of weakness that may benefit from coaching. You may learn that the developer is having an issue with that particular area of the codebase, or you may find that they would benefit from coaching around coding hygiene.

It’s important to remember that metrics are not diagnostic, and quantitative data must always be placed in context. Different projects will naturally progress at different speeds, and non-code-related factors can impact the data. One engineer may appear to be struggling when in reality, they’re simply working through a tricky problem. Another engineer may be adding value through glue work that isn’t as recognizable as shipped code. If you’re gathering relevant context and having open, honest conversations with your team, you’ll be able to determine whether a concerning data point has a reasonable explanation, is an anomaly, or indicates something that needs to be addressed.

Data can do more than help you surface potential areas for improvement. It can help you make those improvements a reality. Goals are more effective when paired with objective data. Metrics make it possible to set and track progress towards specific, actionable targets, which will set your team members up for success. You and your team members will be able to align on exactly what they’re working toward and see how effectively they’re getting there. If progress seems to stall, you can check-in and re-evaluate your tactics — or the goal itself.

Upskilling Is Key to Building a High-Performance Team

Coaching and professional development take time, but they’re critical to driving success and retaining your top performers. It’s not enough to simply hire talented people, as even the most skilled developers will be looking for opportunities to keep growing. With a mixture of cultural and process-level adjustments, you can help create an environment that encourages development while still advancing business priorities.

To find out how to leverage data from a Software Engineering Intelligence (SEI) platform to upskill team members and boost retention, request a consultation.

Knowledge Silos are Holding Back Your Engineering Team — Here’s How to Break Them Down

Knowledge silos are more than just a frustration or an inconvenience — they’re a serious threat to productivity. Yet with a few tweaks to team processes, and a commitment to intentional communication, engineering leaders can break down existing knowledge silos, and make it less likely that new ones will form.

What are Knowledge Silos?

Knowledge silos are individuals or groups in possession of information that is not readily accessible to the rest of an organization. Sometimes knowledge silos are formed by the intentional withholding of information — an individual or team wants to be the go-to source for answers on a certain topic — but often they’re the result of workflows that fail to encourage collaboration and knowledge sharing. Generally, “knowledge silo” is not used to describe situations where the information in question is privileged or confidential, but rather, situations where sharing information would enable team members to work more efficiently or effectively.

What Do you Stand to Gain from Knowledge Sharing?

When engineers lack access to key information, they might end up spending time devising a new solution to a problem that has already been solved elsewhere in the codebase. Or, they may be forced to make assumptions, some of which will be incorrect, about how their work fits into the broader context of a feature or roadmap. When your team is solving the same problems over and over, or acting on incorrect assumptions, they’ll be less efficient, more frustrated, and more likely to perform unnecessary rework. In more extreme cases, when knowledge only resides with one person on a team, that person’s absence can bring work to a standstill.

At Code Climate, an initiative to promote intentional knowledge sharing, including logging key decisions and documenting new tracks of work, helped boost throughput almost 70%. A broader study, GitHub’s 2021 State of the Octoverse, found that specific knowledge-sharing practices like creating and maintaining up-to-date, quality documentation can increase developer productivity by 50%.

How Can you Break Down Knowledge Silos on Your Engineering Team?

Knowledge silos are a natural and expected part of any evolving organization, particularly as it scales, but there are process-level measures leaders can take to actively encourage knowledge sharing.

• Documentation – Though writing code may get all the glory, writing documentation is critical to the success of engineering projects. Documentation allows teams to share context and to align on everything from best practices to project specifications. When documentation isn’t written or maintained, information is harder to access. Engineers not only need to know who to ask about a particular feature or project, they also have to feel comfortable reaching out with their questions.
• Pair Programming – Knowledge exchange is one of the most well-known benefits of pair programming. When two developers are working on the same piece of code, they’ll exchange context and perspectives to arrive at a shared solution. With pair programming, more junior team members will have the opportunity to ask questions; more senior team members can share wisdom culled from their own experiences; and team members with different specialties and skill sets can learn from each other.
• Code Review – While different organizations employ different best practices for Code Review, it is a fundamental opportunity for engineers to share knowledge and offer feedback to help improve each other’s work. If you’re using a Software Engineering Intelligence (SEI) solution like Code Climate, metrics like Review Speed and Review Influence can help provide a sense of whether engineers are prioritizing Code Reviews, whether reviewers are leaving impactful feedback or rubber-stamping code, and if PRs are evenly distributed among reviewers. If one reviewer is disproportionately commenting on certain kinds of PRs, or PRs associated with a particular area of the codebase, that could indicate a potential knowledge silo.
• Draft PRs – Designed to encourage collaboration, Draft PRs enable developers to signal to potential reviewers that code is a work in progress while still opening the door for feedback. This allows developers to gain perspective from teammates at the earliest stages while ensuring that work in progress code does not get merged or prematurely factor into any review metrics being tracked in an SEI platform.

By implementing processes that require communication and collaboration, teams can actively encourage information sharing, rather than leaving it up to chance. With more information exchanged, existing knowledge silos will break down, fewer new ones will form, and overall team productivity will get a boost.

To learn more about using an SEI platform to spot blockers like knowledge silos and improve engineering processes, request a consultation .

Combat Performance Review Biases with Objective Engineering Data

It’s no secret that performance reviews are flawed. Not only is occasional feedback unlikely to affect meaningful growth, but the feedback itself can also be suspect — studies indicate that numerical ratings reveal more about the rater than the person being reviewed, while open-ended evaluations are subject to a host of performance review biases.

Despite this research, most companies still rely on some form of performance review. They’re not ready to pivot from the idea of using reviews to promote professional development, and many employees aren’t either. As an engineering leader, you may not be able to overhaul your company’s review process, but you can still take steps to minimize some of its flaws. Engineering data found in your VCS and project management tools can help by acting as a check against anecdotal evidence and gut feel, helping you combat some common performance review biases.

Common Performance Review Biases

Biases may be evident in nearly every aspect of your day-to-day, but the open-ended format of most performance review frameworks is particularly vulnerable to some common biases. If you’re aware of their existence, you can take steps to counteract them.

Recency Bias When reviews happen infrequently, the period of time right around review season is freshest in the reviewer’s mind and tends to be given the most weight.

What you can do: A skim of the Issues a developer worked on over the past year can be an important reminder of all they contributed, and a great way to refresh your memory. In addition, a deep dive into specific engineering metrics can help you distinguish longstanding patterns from recent anomalies. For example, you may have noticed that a developer is prioritizing their own work and putting off reviewing teammates’ Pull Requests. By looking at trends in a metric like Review Speed, you can determine whether or not that’s a new development, so you can calibrate your conversation accordingly.

Halo/Horns Effect The Halo/Horns Effect occurs when a manager lets one trait — good or bad — skew their entire impression of an individual.

What you can do: There may be an engineer on your team who rarely speaks during meetings, and only participates when directly spoken to. You could take that as evidence that they’re generally disengaged at work, but data might reveal otherwise. If that same engineer has a consistently high PR Throughput and frequently participates in Code Review, it’s more likely that they just don’t like speaking up in group settings. With this information, you can offer specific feedback about their participation in meetings, rather than general (and inaccurate) feedback about their overall level of engagement, or you can adapt your expectations to match their work style.

Gender Bias Studies show that reviewers tend to focus more on the personality traits of women and female-presenting individuals than on their skills or accomplishments.

What you can do: Make a conscious effort to focus on an individual’s work, and be sure to check any of your assumptions against objective data. For example, you can look at a developer’s history of commits, pushes, and review activity to confirm whether their work is in line with your expectations for a developer in their role. You might expect a more senior developer to spend less time committing new code and more time giving feedback to their teammates, but your expectations may be the opposite for a more junior team member.

Similarity Bias This is the tendency of a manager to look more favorably on team members who remind them of themselves, perhaps due to a particular personality trait, a shared work style, or an aspect of their training or background.

What you can do: You may feel a particular kinship with a developer and assume that they’re achieving at the level you would in their role, but a look at the data — whether it’s their PR Throughput or a review of their contributions to Code Reviews —  can help you regain perspective and ground your assessment in the reality of their work.

Using Data to Check Your Performance Review Biases

Data is not only a valuable tool for dismantling performance review biases, it can help you deliver specific, actionable feedback and collaborate with developers to set clear goals for the future. As with any tool, however, it must be used carefully.

Quantitative data should always be contextualized with qualitative data, and it’s important to resist the urge to compare or rank the members of your team. Developers coding new features will naturally work at a different pace than those working through technical debt, and team leads will likely be focused more on coaching or project management than committing code. You’ll need to pair data with an understanding of the circumstances surrounding each team member’s work in order to have a complete picture of their overall performance.

If you’re interested in learning more about incorporating data into your performance reviews, request a consultation.