Improve your Ruby code reviews with actionable code coverage and Undercover

8 min readDec 19, 2018

This is part 2 of Stop Shipping Untested Ruby Code with Undercover. If you’re wondering What The Heck™️ all of this is about, you may want to read the first post in a new tab.

Ruby’s SimpleCov lets us track test coverage and is common when developing Ruby on Rails apps. Many code review services that rely on SimpleCov allow us to see the coverage delta or diff coverage as a percentage. I’ve been recently talking to Ruby developer friends and gathering feedback about an alternative approach, which is to turn untested code into actionable pull request warnings, because coverage as percentage is insufficient.

This post will teach you how to get started with Undercover. We’ll set it up locally, so that commits are blocked until we have tests in place. Then, we’ll learn how to add a code build status check in CI services and finally, how to post automated review comments to GitHub using Pronto.

It’s almost Christmas, so let’s get started and give our Ruby projects some love!

Getting started with local coverage checks

Setup

First fetch the gem from RubyGems with:

gem install undercover

Once installed, the undercover command should be available to your system. However, to make it fully functional we need to set up test coverage tracking.

Update your Gemfile with SimpleCov and an LCOV reporter, which is Undercover’s coverage input file format:

gem 'simplecov'
gem 'simplecov-lcov'
gem 'simplecov-html' # if you'd like classic html coverage reports

After installing them with bundle install, we enable coverage tracking at the very top of the test_helper.rb or spec_helper.rb, before any of the application code is required.

require 'simplecov'
require 'simplecov-lcov'
SimpleCov::Formatter::LcovFormatter.config.report_with_single_file = true
SimpleCov.formatter = SimpleCov::Formatter::MultiFormatter.new([
  SimpleCov::Formatter::LcovFormatter,
  SimpleCov::Formatter::HTMLFormatter,
])
SimpleCov.start do
  add_filter(/^\/spec\//) # For RSpec  
  add_filter(/^\/test\//) # For Minitest
  enable_coverage(:branch) # Enables branch coverage
end

To verify the above set up, we can now run specs and see that a nonempty coverage/ directory has appeared in the project root.

Using the CLI

When undercover is invoked with no arguments, the command-line interface inspects the most recent coverage report and looks for untested methods that appear in the current changeset. What are these methods exactly? These are all code blocks that have any source lines appearing in the current git index (staged) and work tree (unstaged). Let’s illustrate that with an example.

Assuming no changes have been made since last commit, undercover prints out a message and exits cleanly.

$ undercover
✅ No reportable changes
Undercover finished in 0.0742s

The fun begins when we start making changes to our code and test suite. Let’s assume we change a method, run our test suite and invoke undercover again. Now, we get a different output:

Whoops! We need to add tests to the validate instance method to keep Undercover happy. If we add those and run undercover again, we’ll be back to green.

Compare to branch or commit

We have analysed test coverage for the current diff, but what about bigger chunks of work? Undercover can perform the same analysis with respect to any branch or commit. This is where the -c or --compare option comes in handy.

Try this set of command arguments:

undercover --compare master

It will look at the most recent coverage report and find untested methods in the entire diff between HEAD and what we’ve set in --compare. Comes in handy for various branching workflows!

Automate

We now have what we need to create the beginning of an automated feedback loop. Let’s get started with git hooks.

What are they? Git hooks are customisable scripts that live in .git/hooks of every repository and are executed before or after git events. They can be edited manually, but I suggest to start with a helper gem called overcommit, which does an excellent job managing community standard git hooks and custom ones.

Follow Overcommit’s readme to install it and place the contents of this suggested config in .overcommit.yml. An starter config should have been created during installation.

PrePush:
  Undercover:
    enabled: true
    required: true
    command: ['undercover', '--compare', 'master']

This is my config of choice that ensures all tests are in place before code is pushed (PrePush). You might want to change this to suite your preferences and perhaps use a PreCommit variant instead. It’s also good to know that hooks can be disabled in an emergency by setting OVERCOMMIT_DISABLE=1 or using the --no-verify option in commands like commit or push.

Set up Continuous Integration

The next steps will cover adding Undercover to analyse commits project-wide on the CI server, so that we close the feedback loop and make it part of the code review process.

Step 1: Build status check

We’ll start with a status check for test coverage that runs tests and then Undercover. I have prepared a few starter configs for a few popular CI services: Travis CI, CircleCI, Semaphore and Codeship. They are also stored in the repo in case you need them later.

GitHub Actions

The sample workflow config was taken straight from the undercover repository. It fails the build based on coverage results and failures can be inspected within the output log. This is not ideal, but works just fine.

# .github/workflows/ruby.yml
name: Rails Unit Tests
on: [push, pull_request]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - name: Set up Ruby 3.2
      uses: ruby/setup-ruby@v1
      with:
        ruby-version: 3.2
    - name: Build and test with Rake
      env:
        RAILS_ENV: test
      run: |
        # other pre-requisite setup steps...
        gem install bundler undercover
        bundle install --jobs 4 --retry 3
        bundle exec rake test
        undercover --compare origin/master

CircleCI

Sample single-job config for CircleCI that runs specs and validates coverage with Undercover.

# .circleci/config.yml
version: 2
jobs:
  build:
    docker:
      - image: circleci/ruby:2.5-browsers
    steps:
      - checkout
      - run:
          name: Install dependencies
          command: |
            sudo apt-get install cmake
            bundle install
      - run:
          name: Run RSpec
          command: bundle exec rspec
      - run:
          name: Check coverage
          command: |
            gem install undercover
            undercover --compare origin/master

We can go even further and create a separate job for analysing coverage, leveraging features of CircleCI 2.0. That is useful, because we can benefit from workflows and build conditions, but requires a bit more work. Having that in mind, there is another more advanced starter config available on GitHub, where the LCOV report file is shared between multiple CircleCI jobs to create individual build stages for test and coverage.

Semaphore

Semaphore pipeline example with a single job running rspec and undercover:

version: v1.0
name: RSpec + Undercover
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804
blocks:
  - name: "RSpec"
    task:
      jobs:
      - name: Run Specs and Check Coverage
        commands:
          - checkout
          - bundle install
          - bundle exec rspec
          - gem install undercover
          - undercover --compare origin/master

Codeship

To run specs and analyse coverage in Codeship, go to Project Settings and place the following instructions in the Setup Commands section:

rvm use 2.5.3 --install
bundle install
gem install undercover

Then, run specs and undercover as part of the Test Pipeline:

bundle exec rspec --format documentation --color
# pull origin/master to have a ref to compare against
git remote set-branches --add origin master
git fetch
undercover --compare origin/master

One caveat is that Codeship uses a shallow git clone, so we need to specifically add the master branch (or any other branch we want to compare with) to the cloned repository. This way we can use undercover --compare origin/master.

Merging test results from multiple test runners

Another common caveat that applies to most CI services supporting parallel execution is about partial test results. Undercover does not currently support partial coverage reports, so we need to merge them into one before running the undercover command. If this applies to you, this gist might be a good starting point.

Step 2: Automated GitHub comments

We have already closed the feedback loop with the build status, but that’s not all we can do. How about not having to inspect the build output every time untested code is detected? We can create automated pull request comments thanks to Pronto and its integration with Undercover!

Pronto is an open-source automated code review framework that integrates with multiple tools (Rubocop, Reek, Brakeman included) and offers integrations with SCMs like GitHub, GitLab and Bitbucket.

That’s how automated code review looks like at Rainforest

To start setting up review comments, we need a GitHub API OAuth token. It’s up to you to create a new bot user account or use a personal one. Either way, please sign in to GitHub and go to Settings → Developer Settings → Personal Access Token. Then click on “Generate New Token” illustrated below.

Let’s create a token with repo or public_repo privilege scope and make it available to the CI environment. From now on we’ll use Codeship as an example, but these instructions should require minimal changes no matter which CI you are using.

To set build environment variables in Codeship, visit Project Settings → Environment and specify each new variable as a key value pair. Please note that Pronto expects the variable holding our access token to be named PRONTO_GITHUB_ACCESS_TOKEN.

Next, we need update the setup instructions to install pronto and pronto-undercover gems instead of just undercover. This is how the Setup Commands section should look like:

# Setup Commands
rvm use 3.2.0 --install
bundle install
gem install pronto
gem install pronto-undercover

Then, modify the Test Pipeline to run tests and then analyse coverage with pronto. We will use the github_pr formatter option to enable the GitHub integration.

# Test Pipeline
bundle exec rspec --format documentation --color
# pull origin/master to have a ref to compare against
git remote set-branches --add origin master
git fetch
pronto run -f github_pr -c origin/master

Now, when we save our changes, re-run the build and Pronto triggers any coverage warning, it will create comments in GitHub instead of failing the CI build. Yas!

Summary

After reading this post you should know how to:

Set up the undercover gem to run locally, so that untested code changes do not escape the development environment
Configure your CI to do the same thing, but across all commits and remote branches
Create an automated code coverage feedback loop with Pronto and code review comments that appear directly in a GitHub pull request diff.

I hope Undercover will help you deliver better and well tested code with more confidence! If you have any feedback or run into issues with any of the presented setups, please leave comments below, create a GitHub issue or get in touch with me.

Happy Holidays!

And don’t forget to leave a ⭐️ at https://github.com/grodowski/undercover. Also make sure to check out the Undercover GitHub App, which is free to use for public repos: https://undercover-ci.com.