Blog Author: Bryan Helmkamp | Code Climate

Company Updates

Code Climate Quality is Now Qlty Software

Nov 11, 2024

2 min read

Today, we’re excited to share that Code Climate Quality has been spun out into a new company: Qlty Software. Code Climate is now focused entirely on its next phase of Velocity, our Software Engineering Intelligence (SEI) solution for enterprise organizations

How It Started

I founded Code Climate in 2011 to help engineering teams level up with data. Our initial Quality product was a pioneer for automated code review, helping developers merge with confidence by bringing maintainability and code coverage metrics into the developer workflow.

Our second product, Velocity, was launched in 2018 as the first Software Engineering Intelligence (SEI) platform to deliver insights about the people and processes in the end-to-end software development lifecycle.

All the while, we’ve been changing the way modern software gets built. Quality is reviewing code written by tens of thousands of engineers, and Velocity is helping Fortune 500 companies drive engineering transformation as they adopt AI-enabled workflows.

Today, Quality and Velocity serve different types of software engineering organizations, and we are investing heavily in each product for their respective customers.

Where We're Going

To serve both groups better, we’re branching out into two companies. We’re thrilled to introduce Qlty Software, and to focus Code Climate on software engineering intelligence.

Over the past year, we’ve made more significant upgrades to Quality and our SEI platform, Velocity, than ever before. Much of that is limited early access, and we’ll have a lot to share publicly soon. As separate companies, each can double down on their products.

Qlty Software is dedicated to taking the toil out of code maintenance. The new company name represents our commitment to code quality. We’ve launched a new domain, with a brand new, enhanced edition of the Quality product.

I’m excited to be personally moving into the CEO role of Qlty Software to lead this effort. Josh Knowles, Code Climate’s General Manager, will take on the role of CEO of Code Climate, guiding the next chapter as an SEI solutions partner for technology leaders at large, complex organizations.

We believe the future of developer tools to review and improve code automatically is brighter than ever – from command line tools accelerating feedback loops to new, AI-powered workflows – and we’re excited to be on that journey with you.

-Bryan

CEO, Qlty Software

Leadership

The Secret to CTO and CEO Alignment

Jul 20, 2022

4 min read

For many CTOs, communicating with the CEO (or any member of the executive team) can be an unending source of frustration. Though I’m a CEO today, I’ve also been a CTO, and I’ve seen firsthand the challenges that brings. It’s not easy to convey a complete picture of the state of an engineering organization to a technical leader who isn’t involved in the team’s day-to-day, and it’s even harder when you’re speaking to someone without a technical background. You may be working towards the same goal, but if you’re not aligned on how to get there or how things are going, you’ll face unnecessary challenges at every turn.

Unless you know the secret — the key to enhancing alignment with executive stakeholders, including the CEO — clear, objective reporting.

Reporting isn’t just for your boss

CTOs often face difficulties securing budget for critical initiatives, facilitating agreement on the state of a situation, explaining that engineering isn’t always the bottleneck, and more. These may seem like distinct challenges, but in reality they share a common foundation — they’re all difficulties rooted in issues of communication and alignment.

A key responsibility of executive leadership is improving communication and facilitating alignment. No matter how well your team performs, no matter how stellar your software, your department’s success will likely be limited if you can’t get stakeholders on the same page. In order to promote alignment, you’ll need to leverage one of the most underappreciated, oft-maligned tools at your disposal: reporting.

Though it has a bad reputation — Office Space’s TPS reports always come to mind — reporting has a lot to offer. Not timecards, not compulsory bureaucratic tracking, but great reporting (more on what that means in a moment), can offer enormous benefit to you and your team. Done well, reporting allows you to frame the conversations you need to have, and inform the decisions that need to be made.

Every other department has already learned this lesson. Sales, Marketing, HR, and Finance are all reporting on objective data, using it to advocate for their departments and drive critical discussions with the rest of the executive team. It’s time for engineering leaders to do the same.

What is great reporting?

In this context, reporting is the process of gathering and sharing quantitative and qualitative information in order to create the opportunity for shared, fact-based understanding. It ensures that everyone comes to the table with the same data, and that they’re operating on the basis of facts, not feelings. Understanding occurs when that data is contextualized and internalized, and can be used to drive conversations and decisions.

Great reporting goes above and beyond the requirements of that definition. It involves:

Reporting establishes a shared foundation for having critical conversations and making key decisions, but it’s just a starting point. Your report might show your CEO that things are going well, or that a major initiative is off-track, but it can’t explain why, nor can it solve problems. Still, when done well, reporting can be the basis for productive collaboration, and can help you drive success in your organization.

To find out how to leverage clear, objective reporting to meet your organizational goals, request a consultation.

Company Updates

Build Your Own Code Climate Analysis Plugin

Jul 7, 2015

8 min read

Recently, we announced the release of the Code Climate Quality platform, which lets anyone create and deploy static analysis tools to an audience of over 50,000 developers. These Open Source static analysis tools are called “engines" or "plugins," and in this post I’ll show you how to create one from scratch.

We’ll create a plugin that greps through your source code and looks for instances of problematic words like FIXME, TODO, and BUG. This plugin conforms to the Code Climate plugin specification, and all of the code for this plugin is available on GitHub. When we get up and running, you’ll see results like this on your command line:

Plugins that you create can be tested with the Code Climate command line tool. Once you’ve got an Open Source plugins that works, we’d love to chat with you about making it available on our cloud platform, so that your whole community can have access to it!

For more information join our Developer Program.

What’s a plugin made of?

Instead of asking you to dive into the Code Climate plugin specification, to learn what a plugin is, I’ll give you a brief overview here.

A Code Climate plugin is a containerized program which analyzes source code and prints issues in JSON to STDOUT.

Sound simple? We really think it is! Hopefully this blog post will illustrate what we mean.

More concretely, the FIXME plugin we’re going to create contains three important files:

There are other requirements in the specification regarding resource allocation, timing, and the shape of the output data (which we’ll see more of below), but that’s really all there is to it.

A little bit of setup

Before we write any code, you need a few things running locally to test your plugin, so you might as well get that out of the way now. You’ll need the following things running locally:

Run codeclimate -v when you’re done. If it prints a version number, you should be ready to go!

FIXME

The idea for FIXME came to us when we were brainstorming new plugin ideas which were both high value and easy to implement. We wanted to release a sort of Hello, world plugin, but didn’t want it to be one that did something totally pointless. Thus, FIXME was born.

The FIXME plugin looks for (case-insensitive, whole word) instances of the following strings in your project’s files:

This is not a novel idea. It’s well known that instances of these phrases in your code are lurking problems, waiting to manifest themselves when you least expect it. We also felt it worth implementing because running a FIXME plugin in your workflow has the following benefits:

Pretty nifty for around 75 lines of code.

To achieve this, the plugin performs a case insensitive grep command on all of the files you specify, and emits Code Climate issues wherever it finds one.

Implementing an plugin in JavaScript

The meat of the actual plugin is in the index.js file, which contains around 50 lines of JavaScript. The entirety of the file can be found here. I’ll highlight a few important sections of the code for the plugin below, but if you have any questions, please open an issue on the GitHub repo and I’ll try my best to answer promptly!

On to the code. After requiring our dependencies and typing out the module boilerplate, we put the phrases we want to find in grep pattern format:

var fixmeStrings = "'(FIXME|TODO|HACK|XXX|BUG)'";

This will be used in a case insensitive search against all of the files the plugin we’ll analyze.

Next, we create a function that we will use to print issues to STDOUT according to the issue data type specification in the plugin spec. The printIssue function accepts a file name, a line number, and the issue string,

`var printIssue = function(fileName, lineNum, matchedString){ var issue = {    "type": "issue",    "check_name": "FIXME found",    "description": matchedString + " found",    "categories": ["Bug Risk"],    "location":{      "path": fileName,      "lines": {        "begin": lineNum,        "end": lineNum      }    } };

// Issues must be followed by a null byte var issueString = JSON.stringify(matchedString)+"\0"; console.log(issueString); } `

This data format contains information about the location, category, and description of each issue your plugin emits. It’s at the heart of our plugin specification and massaging data from an existing tool to conform to this format is typically straightforward.

The data in the JSON your plugin prints will be consumed by the CLI and if you join our Developer Program and work with us, it can also be made available to all users of Quality. We’ll work with you to ensure your plugin is spec compliant and meets our security and performance standards, and get your work in front of a lot of people!

The actual code that greps each file isn’t super interesting, but you should check it out on GitHub and open an issue on the repo if you have a question.

Because it’s a requirement of plugins to respect the file exclusion rules passed to it by the CLI or our cloud services, though, I’ll show a bit of how that works:

`// Uses glob to traverse code directory and find files to analyze, // excluding files passed in with by CLI config var fileWalk = function(excludePaths){ var analysisFiles = []; var allFiles = glob.sync("/code//", {});

allFiles.forEach(function(file, i, a){    if(excludePaths.indexOf(file.split("/code/")[1]) < 0) {      if(!fs.lstatSync(file).isDirectory()){        analysisFiles.push(file);      }    } });

return analysisFiles; } `

Here I am using the NPM glob module to iterate over all of the files starting at /code recursively. This location also comes from the plugin specification. The fileWalk function takes an array of excludePaths, which it extracts from /config.json (this will be made available to your plugin after the CLI parses a project’s .codeclimate.yml file). This all happens in the main function of the plugin, runEngine:

`FixMe.prototype.runEngine = function(){ // Check for existence of config.json, parse exclude paths if it exists if (fs.existsSync("/config.json")) {    var engineConfig = JSON.parse(fs.readFileSync("/config.json"));    var excludePaths = engineConfig.exclude_paths; } else {    var excludePaths = []; }

// Walk /code/ path and find files to analyze var analysisFiles = fileWalk(excludePaths);

// Execute main loop and find fixmes in valid files analysisFiles.forEach(function(f, i, a){    findFixmes(f); }); } `

This main function gives hopefully gives you a clear picture of what this plugin does:

Packaging it up

How plugins are packaged as Docker containers is important: it has its own section of the plugin specification. The Dockerfile for FIXME is pretty typical:

`FROM node

MAINTAINER Michael R. Bernstein

RUN useradd -u 9000 -r -s /bin/false app

RUN npm install glob

WORKDIR /code COPY . /usr/src/app

USER app VOLUME /code

CMD ["/usr/src/app/bin/fixme"] `

Here’s a breakdown of each line (for more information about each directive, see the official Docker documentation):

  1. The official node Docker container is the basis for this plugin container. It has nodeand npm installed, and generally makes our lives easier.
  2. Declare a maintainer for the container.
  3. Create the app user to run the command as specified.
  4. Install packages with npm install glob so that the external dependency is available when the plugin runes.
  5. Set the WORKDIR to /code, where the source to be analyzed will be mounted.
  6. Copy the plugin code to /usr/src/app.
  7. Use the app user that we created earlier.
  8. Mount /code as a VOLUME per the spec
  9. Our plugin specification says that the plugin should launch and run immediately, so we use CMD to achieve this. In the case of FIXME, the executable wrapper script instantiates the plugin we wrote in JavaScript above, and runs it. Check it out:

`#!/usr/bin/env node

var FixMe = require('../index'); var fixMe = new FixMe();

fixMe.runEngine(); `

We now have all of the pieces in places. Let’s test it out.

Testing your plugin locally

If you want to test the code for this plugin locally, you can clone the codeclimate-fixme repository locally, and follow these steps:

 engines:    fixme:      enabled: true

And you should see some results from test/test.js! Pretty cool, right?

Note that if you want to test modifications you are making to this plugin, you should build the image with a different image name, e.g. codeclimate/codeclimate-fixme-YOURNAME. You would then add fixme-YOURNAME to your .codeclimate.yml file as well.

If you get stuck during development, invoke codeclimate console and run:

Analyze.new(['-e', 'my-engine', '--dev']).run

And you should be able to see what’s going on under the hood.

What will you build?

Hopefully seeing how straightforward an plugin can be will give you lots of great ideas for plugins you can implement on your own. If tools for your language don’t exist, contact us, and maybe we can help you out!

Simple ideas like FIXME have a lot of power when your entire team has access to them. Wire up the codeclimate CLI tool in your build process, push your repositories to Code Climate, and keep pursuing healthy code. We can’t wait to see what you’ll build.

Productivity

Known Good Testing

Feb 20, 2014

5 min read

Automatically Validating Millions of Data Points

At Code Climate, we feel it’s critical to deliver dependable and accurate static analysis results. To do so, we employ a variety of quality assurance techniques, including unit tests, acceptance tests, manual testing and incremental rollouts. They all are valuable, but we still had too much risk of introducing hard-to-detect bugs. To fill the gap, we’ve added a new tactic to our arsenal: known good testing.

"Known good" testing refers to capturing the result of a process, and then comparing future runs against the saved or known good version to discover unexpected changes. For us, that means running full Code Climate analyses of a number of open source repos, and validating every aspect of the result. We only started doing this last week, but it’s already caught some hard-to-detect bugs that we otherwise may not have discovered until code hit production.

Why known good testing?

Known good testing is common when working with legacy code. Rather than trying to specify all of the logical paths through an untested module, you can feed it a varied set of inputs and turn the outputs into automatically verifying tests. There’s no guarantee the outputs are correct in this case, but at least you can be sure they don’t change (which, in some systems is even more important).

For us, given that we have a relatively reliable and comprehensive set of unit tests for our analysis code, the situation is a bit different. In short, we find known good testing valuable because of three key factors:

These add up to mean that traditional unit and acceptance testing is necessary but not sufficient. We use unit and acceptance tests to provide faster results and more localized detection of regressions, but we use our known good suite (nicknamed Krylon) to sanity check our results against a dozen or so repositories before deploying changes.

How to implement known good testing

The high level plan is pretty straightforward:

  1. Choose (or randomly generate, using a known seed) a set of inputs for your module or program.
  2. Run the inputs through a known-good version of the system, persisting the output.
  3. When testing a change, run the same inputs through the new version of the system and flag any output variation.
  4. For each variation, have a human determine whether or not the change is expected and desirable. If it is, update the persisted known good records.

The devil is in the details, of course. In particular, if the outputs of your system are non-trivial (in our case a set of MongoDB documents spanning multiple tables), persisting them was a little tricky. We could keep them in MongoDB, of course, but that would not make them as accessible to humans (and tools like diff and GitHub) as a plain-test format like JSON would. So I wrote a little bit of code to dump records out as JSON:

dir = "krylon/\#{slug}" repo_id = Repo.create!(url: "git://github.com/\#{slug}") run_analysis(repo_id) FileUtils.mkdir_p(dir) %w[smells constants etc.].each do |coll|  File.open("\#{dir}/\#{coll}.json", "w") do |f|    docs = db[coll].find(repo_id: repo_id).map do |doc|      round_floats(doc.except(*ignored_fields))    end    sorted_docs = JSON.parse(docs.sort_by(&:to_json).to_json)    f.puts JSON.pretty_generate(sorted_docs)  end end

Then there is the matter of comparing the results of a test run against the known good version. Ruby has a lot of built-in functionality that makes this relatively easy, but it took a few tries to get a harness set up properly. We ended up with something like this:

dir = "krylon/\#{slug}" repo_id = Repo.create!(url: "git://github.com/\#{slug}") run_analysis(repo_id) %w[smells constants etc.].each do |coll|  actual_docs = db[coll].find(repo_id: repo_id).to_a  expected_docs = JSON.parse(File.read("\#{dir}/\#{coll}.json"))  actual_docs.each do |actual|    actual = JSON.parse(actual.to_json).except(*ignored_fields)    if (index = expected_docs.index(actual))      \# Delete the match so it can only match one time      expected_docs.delete_at(index)    else      puts "Unable to find match:"      puts JSON.pretty_generate(JSON.parse(actual.to_json))      puts      puts "Expected:"      puts JSON.pretty_generate(JSON.parse(expected_docs.to_json))      raise    end  end  if expected_docs.empty?    puts "    PASS \#{coll} (\#{actual_docs.count} docs)"  else    puts "Expected not empty after search. Remaining:"    puts JSON.pretty_generate(JSON.parse(expected_docs.to_json))    raise  end end

All of this is invoked by a couple Rake tasks:

Our CI system runs the rake krylon:validate task. If it fails, someone on the Code Climate team reviews the results, and either fixes an issue or uses rake krylon:save to update the known good version.

Gotchas

In building Krylon, we ran into a few issues. They were all pretty simple to fix, but I’ll list them here to hopefully save someone some time:

Wrapping up

Known good testing is not a substitute for unit tests and acceptance tests. However, it can be a valuable tool in your toolbox for dealing with legacy systems, as well as certain specialized cases. It’s a fancy name, but implementing a basic system took less than a day and began yielding benefits right away. Like us, you can start with something simple and rough, and iterate it down the road.

Productivity

High-Speed Rails Deploys with Git

Oct 2, 2013

5 min read

Deploying in 5 seconds with simpler, faster Capistrano tasks

TL;DR — We reduced deploy times from ten minutes to less than five seconds by replacing the standard Capistrano deploy tasks with a simpler, Git-based workflow and avoiding slow, unnecessary work.

At Code Climate, we try to minimize the time between when code is written and when it is live in production. When deploys slowed until they left enough time to make a pot of coffee, we invested in speeding them up.

What’s in a deploy?

At its core, deploying a modern Rails application consists of a few simple steps:

  1. Update the application code
  2. Run bundle install (if the Gemfile was updated)
  3. Precompile assets (if assets were updated)
  4. Restart the application processes (e.g. Unicorn)

If the deploy fails, the developer needs to be alerted immediately. If application processes fail to rollover to the latest code, we need to detect that.

For kicks, I wrote a Bash script to perform those steps, to determine our theoretical lowest deploy time (just the time for SSH and running the minimum, required commands). It took about three seconds when there were no Gemfile or asset changes. So I set out to reduce our ten minute deploys to as close to that number as possible.

Enter Capistrano

If you take anything away from this article, make it this: Capistrano is really two tools in one. It provides both:

  1. A runtime allowing you to run arbitrary commands against sets of remote servers via SSH
  2. A set of default tasks for deploying Rails applications

The runtime is incredibly useful. The default tasks, which originated back in 2005, come from a pre-Git era and are unnecessarily slow and complex for most Rails applications today.

By default, Capistrano creates a releases directory to store each deployed version of the code, and implicitly serve as a deployment history for rollback. The current symlink points to the active version of the code. For files that need to be shared across deployments (e.g. logs and PID files), Capistrano creates symlinks into the shared directory.

Git for faster, simpler deploys

We avoid the complexity of the releases, current and shared directories, and the slowness of copying our application code on every deploy by using Git. To begin, we clone our Git repo into what will become our deploy_to directory (in Capistrano speak):

git clone ssh://github.com/codeclimate/codeclimate.git /data/codeclimate/app

To update the code, a simple git fetch followed by git reset —hard will suffice. Local Git tags (on the app servers) work beautifully for tracking the deployment history that the releases directory did. Because the same checkout is used across deployments, there’s no need for shared symlinks. As a bonus, we use Git history to detect whether post-update work like bundling Gems needs to be done (more on that later).

The Results

Our new deploy process is heavily inspired by (read: stolen from) Recap, a fantastic set of modern Capistrano tasks intended to replace the defaults. We would have used Recap directly, but it only works on Ubuntu right now.

In the end we extracted a small set of Capistrano tasks that work together to give us the simple, extremely fast deploys:

echo 'curl -sS http://127.0.0.1:3000/system/revision | grep "c7fe01a813" > /dev/null || echo "Expected SHA: c7fe01a813" | mail -s "Unicorn restart failed" ops@example.com' | at now + 2 minutes

Here’s where we ended up: ( Note: I edited the output a bit for clarity.)

If your deploys are not as zippy as you’d like, consider if a similar approach would work for you. The entire project took me about a day of upfront work, but it pays dividends each and every time we deploy.