How to Measure Engineering Performance Uniformly Across Teams

This post is part of our historical archive. It represents the beliefs, actions, products, and services of Code Climate as of its publication date. Today, Code Climate focuses on providing enterprise leaders the software development data, context layer, and playbooks needed to build the AI-native software organization their enterprise needs. Head to codeclimate.com to learn more.

Feb 29, 2024

4 min read

Engineering teams are more distributed than ever. Nearly 70% of active engineering positions are open to remote applicants, and many companies have a mix of remote, hybrid, in-person, and contracted employees. So, how do engineering leaders measure performance uniformly across all of these teams? By creating a framework for understanding performance, asking the right questions, and using data to answer them.

Creating a Performance Framework

Engineering leaders want to mitigate surprises before they impact software delivery or the business. It’s not enough to make decisions based on gut feel or anecdotal performance reviews — especially when an engineering organization is made up of multiple teams with unique working styles and deliverables. To truly understand performance across teams, leaders must establish which metrics are important to their company and create a framework to measure them. Establishing a performance framework ensures that leaders are measuring engineering teams in a consistent and equitable way so they can identify and resolve bottlenecks faster to optimize the flow of work.

Tailoring a Framework for Your Team

Using a common framework like DORA is a great starting point, but leaders must tailor measurement to the needs of their unique team. Traditional engineering metrics, and even frameworks like DORA, can overrotate on the quantity of code that’s produced and underrotate on the quality of that code, how efficiently it was written, or how effectively it solves a specific problem. Solely measuring quantity can result in bloated, buggy code because engineers may prioritize simple features they can get out the door quickly rather than spending time on more complex features that can move the needle for the business.

Adding metrics and context that apply to your specific team can provide a more accurate look at engineering performance. For example, to understand team productivity, leaders may look at engineering metrics like Mean Lead Time for Change (MLTC) alongside Cycle Time. If MLTC is high, it could indicate that Cycle Time is also high. These metrics can be viewed in tandem with other metrics like Time to Open, Time to Merge, and Time to First Review to understand where changes need to be made. These metrics can then be compared across teams to understand which teams are performing well and establish best practices across the organization.

Monthly Engineering Metrics to Understand Team Performance

Data-driven insights can provide engineering leaders with objective ways to evaluate developer competency, assess individual progress, and spot opportunities for improvement. While quarterly KPIs and annual performance reviews are great goalposts, managers are constantly thinking about how their teams are progressing toward those targets. Reviewing engineering metrics on a monthly basis is a good way to assess month-over-month progress and performance fluctuations on an individual level and a team level. Which metrics a team considers depends on its defined framework and overall company goals. Here are a few to consider:

PRs Merged vs. PRs Reviewed

Looking at these metrics together can show how the two key responsibilities of writing and reviewing code are spread across a team.

Review Coverage vs. Review Influence

This helps leaders understand what amount of thoroughness of Code Reviews results in a desired action.

Review Cycles vs. Cycle Time

To understand the effect that back-and-forth cycles in Code Review have on shipping speed, leaders can look at Review Cycles vs. Cycle Time.

Impact vs. Rework

Comparing Impact and Rework will show which teams are making the most significant changes to the codebase and how efficiently they are doing so.

Communicating Engineering Team Performance

Understanding and communicating engineering team performance is an effective way to ensure teams are aligned and that all requirements are understood and met. Making this a standard across the engineering organization — especially in a distributed or hybrid environment — is essential to its success. How leaders communicate their findings is equally important as gathering the information. When feedback is a fundamental part of a blameless team culture, team members understand that feedback is critical to growing as a team and achieving key goals, and will likely feel more secure in sharing ideas, acknowledging weaknesses, and asking for help. Leaders can tailor the questions listed above to meet the unique needs of their organizations and use engineering metrics as a way to understand, communicate, and improve team performance.

Why a Strategy Matters

At the heart of every successful software engineering team is a drive for three things:

1. A culture of continuous improvement
2. The ability to move from idea to impact quickly, frequently, and with confidence
3. A software organization delivering meaningful value

These goals sound simple enough, but in reality, achieving them requires more than just wishing for better performance. It takes data, action, and, most importantly, a cultural shift.
And here's the catch: those three things don't come together by accident.

In my experience, whenever a large-scale change fails, there's one common denominator: a lack of a cohesive strategy. Every time I’ve witnessed a failed attempt at implementing new technology or making a big shift, the missing piece was always that strategic foundation. Without a clear, aligned strategy, you're not just wasting resources—you’re creating frustration across the entire organization.

Step 1: Define Your Purpose

The first step in any successful engineering insights strategy is defining why you're doing this in the first place. If you're rolling out developer productivity metrics or an insights platform, you need to make sure there’s alignment on the purpose across the board.

Too often, organizations dive into this journey without answering the crucial question: Why do we need this data? If you ask five different leaders in your organization, are you going to get five answers, or will they all point to the same objective? If you can’t answer this clearly, you risk chasing a vague, unhelpful path.

One way I recommend approaching this is through the "Five Whys" technique. Ask why you're doing this, and then keep asking "why" until you get to the core of the problem. For example, if your initial answer is, “We need engineering metrics,” ask why. The next answer might be, “Because we're missing deliverables.” Keep going until you identify the true purpose behind the initiative. Understanding that purpose helps avoid unnecessary distractions and lets you focus on solving the real issue.

Step 2: Understand Your People

Once the purpose is clear, the next step is to think about who will be involved in this journey. You have to consider the following:

1. Who will be using the developer productivity tool/insights platform?
2. Are these hands-on developers or executives looking for high-level insights?
3. Who else in the organization might need access to the data, like finance or operations teams?

It’s also crucial to account for organizational changes. Reorgs are common in the enterprise world, and as your organization evolves, so too must your insights platform. If the people responsible for the platform’s maintenance change, who will ensure the data remains relevant to the new structure? Too often, teams stop using insights platforms because the data no longer reflects the current state of the organization. You need to have the right people in place to ensure continuous alignment and relevance.

Step 3: Define Your Process

The next key component is process—a step that many organizations overlook. It's easy to say, "We have the data now," but then what happens? What do you expect people to do with the data once it’s available? And how do you track if those actions are leading to improvement?

A common mistake I see is organizations focusing on metrics without a clear action plan. Instead of just looking at a metric like PR cycle times, the goal should be to first identify the problem you're trying to solve. If the problem is poor code quality, then improving the review cycle times might help, but only because it’s part of a larger process of improving quality, not just for the sake of improving the metric.

It’s also essential to approach this with an experimentation mindset. For example, start by identifying an area for improvement, make a hypothesis about how to improve it, then test it and use engineering insights data to see if your hypothesis is correct. Starting with a metric and trying to manipulate it is a quick way to lose sight of your larger purpose.

Step 4: Program and Rollout Strategy

The next piece of the puzzle is your program and rollout strategy. It’s easy to roll out an engineering insights platform and expect people to just log in and start using it, but that’s not enough. You need to think about how you'll introduce this new tool to the various stakeholders across different teams and business units.

The key here is to design a value loop within a smaller team or department first. Get a team to go through the full cycle of seeing the insights, taking action, and then quantifying the impact of that action. Once you've done this on a smaller scale, you can share success stories and roll it out more broadly across the organization. It’s not about whether people are logging into the platform—it’s about whether they’re driving meaningful change based on the insights.

Step 5: Choose Your Platform Wisely

And finally, we come to the platform itself. It’s the shiny object that many organizations focus on first, but as I’ve said before, it’s the last piece of the puzzle, not the first. Engineering insights platforms like Code Climate are powerful tools, but they can’t solve the problem of a poorly defined strategy.

I’ve seen organizations spend months evaluating these platforms, only to realize they didn't even know what they needed. One company in the telecom industry realized that no available platform suited their needs, so they chose to build their own. The key takeaway here is that your platform should align with your strategy—not the other way around. You should understand your purpose, people, and process before you even begin evaluating platforms.

Looking Ahead

To build a successful engineering insights strategy, you need to go beyond just installing a tool. An insights platform can only work if it’s supported by a clear purpose, the right people, a well-defined process, and a program that rolls it out effectively. The combination of these elements will ensure that your insights platform isn’t just a dashboard—it becomes a powerful driver of change and improvement in your organization.

Remember, a successful software engineering insights strategy isn’t just about the tool. It’s about building a culture of data-driven decision-making, fostering continuous improvement, and aligning all your teams toward achieving business outcomes. When you get that right, the value of engineering insights becomes clear.