Improve your Ruby code reviews with actionable code coverage and Undercover

This is part 2 of Stop Shipping Untested Ruby Code with Undercover. If you’re wondering What The Heck™️ all of this is about, you may want to read the first post in a new tab.

Ruby’s SimpleCov lets us track test coverage and is common when developing Ruby on Rails apps. Many code review services that rely on SimpleCov allow us to see the coverage delta or diff coverage as a percentage. I’ve been recently talking to Ruby developer friends and gathering feedback about an alternative approach, which is to turn untested code into actionable pull request warnings, because coverage as percentage is insufficient.

This post will teach you how to get started with Undercover. We’ll set it up locally, so that commits are blocked until we have tests in place. Then, we’ll learn how to add a code build status check in CI services and finally, how to post automated review comments to GitHub.

It’s almost Christmas, so let’s get started and give our Ruby projects some love!

Getting started with local coverage checks

First fetch the gem from RubyGems with:

gem install undercover

Once installed, the undercover command should be available to your system. However, to make it fully functional we need to set up test coverage tracking.

Update your Gemfile with SimpleCov and an LCOV reporter, which is Undercover’s coverage input file format:

gem 'simplecov'
gem 'simplecov-lcov'
gem 'simplecov-html' # if you'd like classic html coverage reports

After installing them with bundle install, we enable coverage tracking at the very top of the test_helper.rb or spec_helper.rb, before any of the application code is required.

require 'simplecov'
require 'simplecov-lcov'
SimpleCov::Formatter::LcovFormatter.config.report_with_single_file = true
SimpleCov.formatter = SimpleCov::Formatter::MultiFormatter.new([
SimpleCov::Formatter::LcovFormatter,
SimpleCov::Formatter::HTMLFormatter,
])
SimpleCov.start do
add_filter(/^\/spec\//) # For RSpec
add_filter(/^\/test\//) # For Minitest
end

To verify the above set up, we can now run specs and see that a nonempty coverage/ directory has appeared in the project root.

When undercover is invoked with no arguments, the command-line interface inspects the most recent coverage report and looks for untested methods that appear in the current changeset. What are these methods exactly? These are all code blocks that have any source lines appearing in the current git index (staged) and work tree (unstaged). Let’s illustrate that with an example.

Assuming no changes have been made since last commit, undercover prints out a message and exits cleanly.

$ undercover
✅ No reportable changes
Undercover finished in 0.0742s

The fun begins when we start making changes to our code and test suite. Let’s assume we change a method, run our test suite and invoke undercover again. Now, we get a different output:

Whoops! We need to add tests to the validate instance method to keep Undercover happy. If we add those and run undercover again, we’ll be back to green.

We have analysed test coverage for the current diff, but what about bigger chunks of work? Undercover can perform the same analysis with respect to any branch or commit. This is where the -c or --compare option comes in handy.

Try this set of command arguments:

undercover --compare master

It will look at the most recent coverage report and find untested methods in the entire diff between HEAD and what we’ve set in --compare. Comes in handy for various branching workflows!

We now have what we need to create the beginning of an automated feedback loop. Let’s get started with git hooks.

What are they? Git hooks are customisable scripts that live in .git/hooks of every repository and are executed before or after git events. They can be edited manually, but I suggest to start with a helper gem called overcommit, which does an excellent job managing community standard git hooks and custom ones.

Follow Overcommit’s readme to install it and place the contents of this suggested config in .overcommit.yml. An starter config should have been created during installation.

PrePush:
Undercover:
enabled: true
required: true
command: ['undercover', '--compare', 'master']

This is my config of choice that ensures all tests are in place before code is pushed (PrePush). You might want to change this to suite your preferences and perhaps use a PreCommit variant instead. It’s also good to know that hooks can be disabled in an emergency by setting OVERCOMMIT_DISABLE=1 or using the --no-verify option in commands like commit or push.

Set up Continuous Integration

The next steps will cover adding Undercover to analyse commits project-wide on the CI server, so that we close the feedback loop and make it part of the code review process.

Step 1: Build status check

We’ll start with a status check for test coverage that runs tests and then Undercover. I have prepared a few starter configs for a few popular CI services: Travis CI, CircleCI, Semaphore and Codeship. They are also stored in the repo in case you need them later.

The sample .travis.yml config was taken straight from the undercover repository. It fails the build based on coverage results and failures can be inspected within the output log. This is not ideal, but works just fine.

# .travis.yml
language: ruby
rvm:
- 2.5.1
- 2.4.4
- 2.3.4
before_install:
- gem install bundler undercover
- gem update --system
script:
- bundle exec rake
- git pull origin master:master # required for --compare below
- undercover --compare master

It should be also possible to run a coverage check a separate TravisCI job, but I haven’t looked into it yet. It appears that passing a coverage report (or any data, really) between Travis test executors requires an external storage layer like S3.

Sample single-job config for CircleCI that runs specs and validates coverage with Undercover.

# .circleci/config.yml
version: 2
jobs:
build:
docker:
- image: circleci/ruby:2.5-browsers
steps:
- checkout
- run:
name: Install dependencies
command: |
sudo apt-get install cmake
bundle install
- run:
name: Run RSpec
command: bundle exec rspec
- run:
name: Check coverage
command: |
gem install undercover
undercover --compare origin/master

We can go even further and create a separate job for analysing coverage, leveraging features of CircleCI 2.0. That is useful, because we can benefit from workflows and build conditions, but requires a bit more work. Having that in mind, there is another more advanced starter config available on GitHub, where the LCOV report file is shared between multiple CircleCI jobs to create individual build stages for test and coverage.

Semaphore pipeline example with a single job running rspec and undercover:

version: v1.0
name: RSpec + Undercover
agent:
machine:
type: e1-standard-2
os_image: ubuntu1804
blocks:
- name: "RSpec"
task:
jobs:
- name: Run Specs and Check Coverage
commands:
- checkout
- bundle install
- bundle exec rspec
- gem install undercover
- undercover --compare origin/master

To run specs and analyse coverage in Codeship, go to Project Settings and place the following instructions in the Setup Commands section:

rvm use 2.5.3 --install
bundle install
gem install undercover

Then, run specs and undercover as part of the Test Pipeline:

bundle exec rspec --format documentation --color# pull origin/master to have a ref to compare against
git remote set-branches --add origin master
git fetch
undercover --compare origin/master

One caveat is that Codeship uses a shallow git clone, so we need to specifically add the master branch (or any other branch we want to compare with) to the cloned repository. This way we can use undercover --compare origin/master.

Another common caveat that applies to most CI services supporting parallel execution is about partial test results. Undercover does not currently support partial coverage reports, so we need to merge them into one before running the undercover command. If this applies to you, this gist might be a good starting point.

Step 2: Automated GitHub comments

We have already closed the feedback loop with the build status, but that’s not all we can do. How about not having to inspect the build output every time untested code is detected? We can create automated pull request comments thanks to Pronto and its integration with Undercover!

Pronto is an open-source automated code review framework that integrates with multiple tools (Rubocop,Reek, Brakeman included) and offers integrations with SCMs like GitHub, GitLab and Bitbucket.

That’s how automated code review looks like at Rainforest

To start setting up review comments, we need a GitHub API OAuth token. It’s up to you to create a new bot user account or use a personal one. Either way, please sign in to GitHub and go to Settings → Developer Settings → Personal Access Token. Then click on “Generate New Token” illustrated below.

Let’s create a token with repo or public_repo privilege scope and make it available to the CI environment. From now on we’ll use Codeship as an example, but these instructions should require minimal changes no matter which CI you are using.

To set build environment variables in Codeship, visit Project Settings → Environment and specify each new variable as a key value pair. Please note that Pronto expects the variable holding our access token to be named PRONTO_GITHUB_ACCESS_TOKEN.

Next, we need update the setup instructions to install pronto and pronto-undercover gems instead of just undercover. This is how the Setup Commands section should look like:

# Setup Commands
rvm use 2.5.3 --install
bundle install
gem install pronto
gem install pronto-undercover

Then, modify the Test Pipeline to run tests and then analyse coverage with pronto. We will use the github_pr formatter option to enable the GitHub integration.

# Test Pipeline
bundle exec rspec --format documentation --color
# pull origin/master to have a ref to compare against
git remote set-branches --add origin master
git fetch
pronto run -f github_pr -c origin/master

Now, when we save our changes, re-run the build and Pronto triggers any coverage warning, it will create comments in GitHub instead of failing the CI build. Yas!

Summary

After reading this post you should know how to:

  1. Set up the undercover gem to run locally, so that untested code changes do not escape the development environment
  2. Configure your CI to do the same thing, but across all commits and remote branches
  3. Create an automated code coverage feedback loop with Pronto and code review comments that appear directly in a GitHub pull request diff.

I hope Undercover will help you deliver better and well tested code with more confidence! If you have any feedback or run into issues with any of the presented setups, please leave comments below, create a GitHub issue or get in touch with me.

Happy Holidays!

And don’t forget to leave a ⭐️ at https://github.com/grodowski/undercover.

software engineer @rainforestqa, hardware hacker and fixed-gear cycling enthusiast from Warsaw, PL.