Improving TestNG Reporting
December 09, 2012Recently, the test suite I maintain kept crashing in the reporting phase. After some digging, and a number of report-free runs, I found the problem.
There's a bug in TestNG's reporter that is a known issue with using a StringBuffer
to generate large amounts of html. This is simple to fix. We just have to disable the html reporter.
The following goes into our pom.xml under failsafe:
<configuration>
<properties>
<property>
<name>usedefaultlisteners</name>
<value>false</value>
</property>
<property>
<name>reporter</name>
<value>org.testng.reporters.XMLReporter</value>
</property>
</properties>
</configuration>
If you use Jenkins, the basic Maven build plugin should be able to 'just' pick the TestNG xml.
The only problem with this is that you get slightly less reporting from your build when an issue occurs. The HTML reports are really good at telling you in a page-by-page fashion exactly where an error is occuring. To combat that issue, I inline all TestNG failures via this class:
public class ImmediateTestListener extends TestListenerAdapter {
@Override
public void onTestFailure(ITestResult tr) {
Throwable t = tr.getThrowable();
if (t != null) {
LoggerFactory.getLogger(tr.getTestClass().getRealClass()).error(t.toString(), t);
}
}
}
TestListenerAdapter is the TestNG interface for test listeners and reporters. The failure method tells our logging framework to use our logger (slf4j) to report the error via whatever logger is configured for the build.
<property>
<name>listener</name>
<value>your_package_name_here.ImmediateTestListener</value>
</property>
When a test fails, you get something like this:
03:29:22.638 ERROR [stalledLocalContainer] - tld.package.CrazyException: Couldn't generate all the things. Sending 500.
03:29:22.638 ERROR [c.l.e.i.t.shelves.TestingStuffTest ] - org.openqa.selenium.NoSuchElementException: Unable to locate element: {"method":"name","selector":"pageHeading"}
Command duration or timeout: 1.28 seconds
And if you use something like cargo, it's inline with your server logs for the application under test.