I think, therefore I blog

Maven and TestNG

By , 8 November 2006

Maven and TestNG

I'm fairly new to maven, but so far it has really impressed me with its simplicity, productivity gains and IDE independence. So I was really looking forward to porting some unit tests to TestNG, running mvn test and watching in awe as maven found my tests, ran them and generated a pretty html report of the test results. Unfortunately, it didn't turn out quite like that.

Maven and TestNG

It would seem there are some unfortunate bugs in surefire (maven's test runner) and these extended my five minute job into six hours of bewilderment - with occasional profanities. Anyway, to cut a long story short, here is a list of problems I encountered using TestNG with maven, and the ultimate solution I settled for.

The Problems

  • Each of the Test* classes were found by surefire, but none of the tests in them were run. It turns out that this is a known bug [1], and the workaround is use an xml test suite instead of relying on class name matching.
  • A TestNG exception caused by a missing parameter for an @BeforeSuite method was being hidden by surefire, and the build failing mysteriously. I only discovered the problem when I changed @BeforeSuite to @Test.
  • The surefire-report reruns the test cases. Again, this is a known bug [2] and isn't critical, but it is quite annoying. To only execute the tests once, you can run mvn surefire-report:report instead of mvn test.
  • The test report repeats the results from all the previous tests in each section, so you end up with longer and longer lists of repeated data (another known bug [3]).
  • The straw which broke the camels back: Nested exceptions are not shown in the test output. Many of my test failures were caused by "An expection has occurred, please check the nested exception for details".

A Solution

Fortunately, maven can run ant scripts and the testng jars include some ant tasks for executing testng 'natively'. We can configure surefire not to run the test cases and use the <testng> ant task instead. Also, the <junitreport> task can be used to generate a similar html report to that generated by the surefire-report plugin.


        <!-- this ant script runs testng natively -->
              <taskdef resource="testngtasks" classpath="testng.jar"
              <testng classpathref="maven.test.classpath"
                <xmlfileset dir="src/test/suites" includes="*.xml"/>
              <junitreport todir="target/test-reports">
                <fileset dir="target/test-reports">
                  <include name="**/*.xml" />
                <report format="noframes" todir="target/test-reports" />

    <!-- disable surefire plugin (too many problems!) -->

Of course you'll need to make your project dependent on TestNG also:



[1] http://jira.codehaus.org/browse/MSUREFIRE-134
[2] http://jira.codehaus.org/browse/MSUREFIREREP-6
[3] http://jira.codehaus.org/browse/MSUREFIRE-172


About Roger Keays

Maven and TestNG

Roger Keays is an artist, an engineer, and a student of life. Since he left Australia in 2009, he has been living as a digital nomad in over 40 different countries around the world. Roger is addicted to surfing. His other interests are music, psychology, languages, and finding good food. Click here to subscribe to his weekly blog, or stalk him on Facebook and Twitter.

Leave a Comment

Please visit https://RogerKeays.com/blog/maven_and_testng to add your comments.

Comment posted by: Daren, 8 years ago

Thanks for the great post!  I can't believe we're halfway through 2008 and the surefire/TestNG combination is still so unstable!  I've tried every version combination of surefire and TestNG and have lost all hope of getting it to work.  This Ant workaround is just what the Doctor ordered.

Thanks! - Daren

Comment posted by: Tomek, 9 years ago

maven-surefire-plugin ver. 2.3 fixes bug nr 1
the other two should be fixed in 2.4

thx for this post

Comment posted by: , 9 years ago

Hey Binil,

I don't know if that can be done with Surefire, but it's another one that can be done with the ant task by using the listeners attribute.

Comment posted by: Binil Thomas, 9 years ago

Hi Roger,

Thanks for the informative post.
Do you know a way to configure a TestNG listener when running the tests from Surefire?


Join 3,559 People Who Think Outside The Box

I write every Sunday about travel, psychology, technology, and all sorts of interesting stuff. It's completely free, and you can leave whenever you like. Subscribe now, and you'll be the first to get new articles each week.

Read a Good Book

“A spellbinding true story of love, passion and adventure. One can’t help but be swept away by 100% Love Guaranteed.” —Dr Tammie Matson, author of Elephant Dance.

Chat For A While

Your Vote Matters

Which animal will take over when humans go extinct?