Complementing Unit Testing with Static Analysis, with NUnit as an Example

When discussing static analysis tools for C# projects, programmers will often deny the necessity of static analysis arguing that most errors can be caught through unit testing. So, I decided to find out how well one of the most popular unit-testing frameworks, NUnit, was tested and see if our analyzer could find anything of interest there.

Introduction

NUnit is a popular unit-testing library for .NET projects ported from Java to C#. Its code is open and can be downloaded from the project website http://www.nunit.org/.

It should be noted that JUnit — the project that NUnit was ported from — was created by such renowned programmers as Erich Gamma, a co-author of the textbook on object-oriented design patterns, and Kent Beck, the creator of the test-driven development and extreme programming methodologies. I recall reading his book Test Driven Development By Example once, where he explains test-driven development by the example of creating a unit-testing framework, like JUnit, following all of his methodologies. What I mean to say is that there is no doubt that JUnit and NUnit were developed in keeping with the best traditions of unit testing, which is also confirmed by Kent Beck's comment at the NUnit site: "… an excellent example of idiomatic design. Most folks who port xUnit just transliterate the Smalltalk or Java version. That's what we did with NUnit at first, too. This new version is NUnit as it would have been done had it been done in C# to begin with."

I looked through NUnit's source files: there are piles of tests; it looks like they have tested everything that could be tested. Taking into account the project's great design and the fact that NUnit has been used by thousands of developers over a number of years, I didn't expect PVS-Studio to find a single bug there. Well, I was mistaken: it did find one bug.

About the bug found

It triggered the V3093 diagnostic, which deals with an issue when programmers use operators & and | instead of && and ||. This issue may cause troubles when it is critical that the right part of an expression should not execute under certain conditions. Let's see what this error looks like in NUnit.

public class SubPathConstraint : PathConstraint
{
    protected override bool Matches(string actual)
    {
        return actual != null &
            IsSubPath(Canonicalize(expected), Canonicalize(actual));
    }
}
public abstract class PathConstraint : StringConstraint
{
    protected string Canonicalize(string path)
    {
        if (Path.DirectorySeparatorChar !=
            Path.AltDirectorySeparatorChar)
            path = path.Replace(Path.AltDirectorySeparatorChar,
                                Path.DirectorySeparatorChar);
        ....
    }
}

Even if the Matches method receives the value null as the actual parameter, the right operand of the & operator will be evaluated anyway, which means that the Canonicalize method will be called, too. If you look at this method's definition, you'll see that the value of its path parameter is not tested for null and method Replace is called on it right away — this is where NullReferenceException might be raised. Let's try to reproduce this issue. For that purpose, I wrote a simple unit-test:

[Test]
public void Test1()
{
    Assert.That(@"C:\Folder1\Folder2", Is.SubPathOf(null));
}

Now let's run it and here's what we get:

Here it is: NUnit crashed with NullReferenceException. PVS-Studio managed to find a real bug even in such a well-tested product as NUnit is. Note that it was no harder than writing a unit-test: you just run project analysis from the menu and check the grid with the results.

Conclusion

Unit testing and static analysis are not alternative, but complementary software-development strategies [1]. Download PVS-Studio analyzer and run it on your projects to see if it can find errors that tests didn't.

References

Andrey Karpov. How to complement TDD with static analysis.

Ilya Ivanov. An unusual bug in Lucene.Net.