The Deception of Code Coverage

I'm writing this post to express an opinion I’ve had for a long time, since for years I’ve been hearing the same old tune: “What’s the code coverage percentage?” and I’m tired of explaining the same thing over and over. Code coverage percentage doesn’t matter one bit.

I’m writing this post, providing arguments and examples so you can send it to your boss instead of trying to discuss topics he possibly won’t understand. 

1 - High Percentage Is a Deception

Let’s start with the most obvious, and personally, I think developers don’t do enough to avoid this situation. 

A high percentage of code coverage is useless, it measures absolutely nothing. The only reason that number exists is to report it to people who have no idea what it is, and basically just need to see a number as close to 100 as possible to feel happy. 

In part, I understand that we just give them the number, because in the long run, it’s harder to explain why it’s a useless metric than to just give the number. I say this from experience, no matter how many times you explain it, or even prove you can reach 100% coverage without actually testing anything. 

public void Test(){
    FooService service = new FooService();
    int result = service.Execute(1);

    Assert.Equal(5, 3+2 )
}

This test is clearly an extreme example taken to the absurd, and you might say that test wouldn’t pass code review, right? Really? Want to bet? If I ask a colleague, “hey, review my PR” and 10 seconds later it’s approved, do you think they’ll notice that test? 

But anyway, you get the idea with this example: you execute something, the code hits all the lines, and then you just check random things. The code percentage calculator will show 100% covered, but not the logic, which is what it should really measure. 

But hey, that’s the way it is. I also have to say, just as having 100% coverage proves nothing, having 0% proves you don’t have a single test.

2 - False Sense of Security

Reporting a high percentage is deceptive, because we can say we have 95% tested, and the “machine” says so. So if there’s a bug? Is it going to be in that 5%? Probably that’s what the chain of command thinks, but in reality, the bug that crashed production and kept us up at 2am probably happened in that 95% that’s supposedly tested. 

This is just more proof that the percentage is useless. A real-life example: when we have a dependency, and for some reason the dependency fails, or returns an unexpected result due to a bug in the dependency. What do we do then? Are we covering that case? Most likely, NO.

But, wait, if we’re covering 100% of the code, how can there be an error? 

3 - The Impact on Productivity

Having 100% code coverage is just stupid. I’m not saying there isn't some piece of software we must cover completely; every application has its “core” or extremely important parts that absolutely must be covered, and that’s a good thing. But in programming, everything is a tradeoff, everything has pros and cons. 

Trying to cover 100% is usually a waste of time, not because it’s useless, but because it takes so long to actually test everything well that you’d spend more time writing tests than on the rest of the project. But hey, everyone decides how to spend their time and money. To have a complete test suite, we have different types of tests (unit, component, integration, end to end, UI, etc.) and… surprise! End-to-end, UI, and even integration tests are not included in code coverage, but are just as important as unit tests… So if we have these kinds of tests, we know the percentage is not only a waste of time, but also not even the real number.

4 - Writing Relevant and Quality Tests

This is where the most important part comes in: the team in charge, whether they’re responsible for the functionality or the code change. Why do I say this? Because the code is written by one, but reviewed by many. Having low-quality tests isn’t the fault of the person writing them, but of the reviewer. 

There are lots of people, and I’ve done this in the past, who see a PR with 20 files where 5 are tests, and because it’s “large” they just skim it and don’t even look at the tests. I’m not going to blame anyone, because we’re all busy, but I’ll let you in on a secret: ignoring the tests in a PR leads to one thing, bugs in production. My recommendation is to block out 2 slots each day of 30 minutes each for code review. I do mine at 9am and after lunch.

As software engineers, we must make sure everything we touch ends up in better shape than when we got it, and that also includes reviewing code. We have to make sure the tests cover EVERYTHING the person has created, especially the business logic. 

Here’s a practice I like: cover all exits. Let’s imagine we’re testing a method that calls a dependency.

public Article InsertArticle(string content, string title, int authorId)
{
    if (!_authorRepository.IsValid(authorId))
    {
        throw new Exception("Author not valid");
    }
        
        
    int aritcleId = _articleRepository.Insert(content, title, authorId);

    return GetArticle(aritcleId);
}

In this particular case, we have two exits: the exception and the regular return value. 

The reviewer should make sure both exits are tested and work as expected. That’ll mean this bit of code is 100% tested, and maybe your boss will even give you a pat on the back.

But really, ideally, and for me, it’s mandatory, you should have at least one extra test. That’s why our favorite testing framework (whichever you prefer) lets us reuse tests with different parameters to cover many scenarios. For example, what if “content” is null? Now in C# we have nullable, but a few years ago we didn’t, so we must have a test (in this case, an integration test) to check what happens if the value is null, because spoiler: in the database that value is required, but in the code, for whatever reason, no one checks and it’s just assumed.

These extra tests DO NOT increase code coverage percentage at all, so why write them if your boss won’t see any benefit? The answer is simple: because you care about the software you write and don’t want bugs in production. These extra tests might not catch every edge case, but at least the vast majority. 

So, to decide when a test suite is complete, the little number isn't enough, you need to understand the tests, understand the code, and of course, run the suite on your machine and make sure everything works as it should. 

And as you’ve seen throughout this post, the percentage doesn’t matter at all, it’s not a metric that measures anything, and in most cases, it’s a wrong number.