Platinum Solutions Corporate Website


Exception Handling in JUnit tests

The answer you entered to the math problem is incorrect.

While preparing a training class this past month on Effective Exception Handling for a client, I came up with a list of "Ten Commandments of Exception Handling."  Some of them were the basic, standard items ("Thou shalt not catch or throw java.lang.Throwable", "Thou shalt not throw raw java.lang.RuntimeExceptions--thou shalt throw a subclass instead", "Thou shalt not catch an Exception unless thou art going to truly handle it, or are going to wrap it in another Exception").  Other items on the list were specifically tailored for the client.  I liked one of the "commandments" better than the others, however:

"XIII: Thou shalt not catch any Exceptions in JUnit tests, unless testing that your code under test throws the Exceptions the API says it does."

If you read this and say, "well, duh, Matt!" then you and I are thinking alike and I am preaching to the choir.  If this statement has even a whiff of controversy to it, however, read on.

Think about JUnit tests for a second.  If an Exception is thrown by the test, or the code under test and it perks up out of the test and gets caught by the JUnit framework, it is reported as a "test error" (well, if you are still using good ol' JUnit 3.8, anyway - JUnit 4, I understand, is dropping the distinction between test failures and test errors -- BOO!!!!!).  A test error means that the test did not run to successful completion; that is, the set of assertions that the test tried to make did not get made.  This is quite a bit different from stating that the assertions were made and passed, or that the assertions were made and failed.

If your test case catches and handles the exceptions thrown by code under test that blows up, you interrupt this process.  If you catch an exception and use a fail message to fail the test because you caught an exception, you have skewed reality - rather than showing that the code failed to run correctly, you have represented that the code ran to successful completion, but did not give the expected outputs.  Even worse, if you catch the exception and just print the stack trace, the fact that the code under test failed to execute is reported by JUnit as a success!

The best thing to do when code under test throws an Exception is to just let it perk up.  My preference is to explicitly list the checked exceptions on the throws line as needed for each test method, and this is the standard we have put in place for the client for which I developed the training I spoke of.  It is sufficient to simply put throws Exception on each test method, but I don't like this as much, as it is less rigorous and has the potential for encouraging similar sloppiness in the rest of the code.  I like encouraging developers to consider test cases as first-order clients of the code they write, and not second-class citizens.  (All coding standards for the main code applies to the test code at my client's site.)

In my mind there is one, and only one, good reason to catch Exceptions thrown by code under test in a JUnit test:  when you are actually testing that the code under tests throws Exceptions according to its contract.

If you have written a method that throws an exception under a certain condition, that's part of the API of your class.  It's part of your contract.  That's something you should test, and you should be writing specific tests whose job is to make sure that when the right conditions exist, the Exceptions actually occur as promised.

Here's an example.  Suppose I've written a class UserFetcher, which has a method getUser(String id), which is supposed to return a User object that matches the supplied ID, or throw a NoSuchUserException if no user can be found for the supplied ID.  Since I've defined this contract, I need to test that the Exception is actually thrown as advertised :

    public void testGetUserException() {
        User u = null;
        try {
            u = UserFetcher.getUser("ThisUserIdDoesNotExist");
            fail("Should have gotten NoSuchUserException - used a bad user ID");
        } catch (NoSuchUserException expected) {
            ; // Expected - intentional
        }
    } 

 

The call to the code under test is surrounded in a try/catch block; the call is immediately followed by an explanatory fail message -- if the code throws its exception, this fail message will never execute, but if the exception does not occur, the test will fail and terminate.  The catch block catches only the specific exception expected.  Note that the caught exception is named something self-documenting. The catch block is empty and contains a comment indicating that the exception was intentional and expected.  All other exceptions are allowed to perk up as JUnit test errors.

That much is not very controversial.  But here's the more difficult habit for developers to discipline themselves into:  At no other time should I catch this NoSuchUserException in any JUnit test!!! They should simply not occur any other time, and if they do, I have an error either in my code or in my test and I want to know about it.

Exceptions by definition only occur under exceptional circumstances.  While it's appropriate for an application to gracefully handle and often hide code failures, we write tests to expose them. We want to see failures when they occur in a test case, not to hide or handle them.  The only exceptions you should handle in your JUnit tests are the ones you create on purpose by deliberately manipulating the circumstances to make them happen, like we did in the code example above.

Earlier I said there was one and only one good reason to catch exceptions in JUnit tests.  There may be other necessary reasons that are not "good."  Now that I am near the end of my "sermon," I need to make a few
clarifying comments about the -- no pun intended -- "exceptions" to my
commandment about catching exceptions in JUnit tests. 

  1. Although the commandment says "any Exceptions" I mean Exceptions thrown from the code under test, although other code should fall under that umbrella as well.  Sometimes the test itself has to do some things for itself with test fixtures like parse a string as an integer, and you may want to catch NumberFormatException for that -- but I still say to avoid it. If you do catch exceptions, do it only for housekeeping code that is not related to the class or method under test.  I still think it's probably better to let these exceptions just perk up to let you know your test had a problem unless you can write code to fully and completely recover from the situation.
  2. Sometimes in order to get to certain execution points to test them, you need to suppress (i.e., ignore) certain errors.  This is one of those rare instances where reality exceeds dogma and you do what you have to do to get the job done.  But you need to do this deliberately, with full knowledge that you are doing something risky and assuming the consequences; and you need to document this thoroughly in your code so the person who comes up behind you knows you did it on purpose.

In closing, I can't help but wonder if I've been simply stating the obvious, or if I've added any new insights.  My gut tells me that will depend largely on the reader, of course.  My hope, however, is that someone out there will read this and say "wow, he's got a point."  They will look at their test code and remove a catch block from a test case that's been eating an exception for weeks or months and causing JUnit to erroneously report that a buggy piece of code is just fine.  The exception will start perking up and showing up as an error like it should have all along.  This person will then say "I never knew my code had all these bugs.  I thought everything was fine because all my tests were passing."

Comments

Anonymous (not verified) Wed, 1969-12-31 20:00

Why do you need to catch the exception? Why don't you have a throws clause in the signature of the test method?

public void testSomething throws Exception {
//Do stuff
}

Anonymous (not verified) Wed, 1969-12-31 20:00

Thanks! I had forgotten why the assert and fail methods didn't take an exception as an optional argument, and know I remember it is because junit enforces this kind of crap to try to remind testers that they don't have to use try catch in the test since the framework does it for you. Anyone looking to print stacktrace for an exception caught in a junit test needs to remember this.

Anonymous (not verified) Wed, 1969-12-31 20:00

In my opinion you should not be throwing exception in your code especially when it might be inherited by a subclass. Things may go funky and the object may partially be constructed. Instead you could add a new "setter" method to initialize status of the instance member and have that method to throw an exception.

Anonymous (not verified) Wed, 1969-12-31 20:00

Thanks Matt.

It helped me figure out how til test for exceptions

Anonymous (not verified) Wed, 1969-12-31 20:00

I don't like try catch in my unit tests, but how do you test a class that has a constructor that can throw exceptions?  I am currently doing something like...

try{

    My Class myObject = new MyClass(args);

    //rest of test case

} catch (MyException e) {

     e.printStackTrace();

    fail("Should not have been thrown"); 

Post new comment

Please solve the math problem above and type in the result. e.g. for 1+1, type 2.
The content of this field is kept private and will not be shown publicly.
  • Lines and paragraphs break automatically.

More information about formatting options