|
|
Overview
Code coverage describes the degree to which the source code of a program has been tested.
The basic information about code coverage can be found at http://www.bullseye.com/coverage.html .
Code coverage runs in most frameworks in these phases :
- instrumentation changes the bytecode of a classes and inject custom calls to the program
- run of tests and collection of metrics reported by the injected custom calls
- processing of the metrics
- reporting of results
The main java projects which are working or worked on code coverage are :
- JVMDI, JBLANKET, InsectJ, noUnit, Hansel, gretel, Groboutils, Quilt, Codecover, Coverclipse, Jester, Cobertura, Emma, Clover
Selection
The criteria for selection of right tool for our projects are:
- Maven integration
- Hudson integration
- Reporting in HTML
- Fast processing
- Good documentation
- Progress on project support and development
The mentioned tools will be divided into there groups :
a) Tools that don't fullfill criteria
- dead tools
-JVMDI, JBLANKET, InsectJ, noUnit, Hansel, gretel, Groboutils, quilt
- different testing method
-jester
- other
-codecover, coverclipse
- licensed
Jcoverage
Clover
b) Tools that fullfils criteria
a) Tools that don't fullfill criteria
QUILT
| Source |
http://quilt.sourceforge.net/ |
| Description |
Quilt is a Java software development tool that measures coverage , the extent to which unit testing exercises the software under test.
Optimized for use with the JUnit unit test package |
| Start |
Ant, Maven |
| Technology |
statement coverage |
| Frameworks |
BCEL |
| Reports |
|
| Report example |
|
| Last release |
20 October 2003 |
| License |
Apache License, Artistic license |
| IDE support |
No |
GROBOUTILS
| Source |
http://groboutils.sourceforge.net/ |
| Description |
The popular tools included with GroboUtils include multi-threaded tests, hierarchial unit tests, and a code coverage tool. |
| Start |
Ant |
| Technology |
logging statements for tracking coverage |
| Frameworks |
BCEL |
| Reports |
XML, xsl-t transformations, HTML |
| Reports example |
http://groboutils.sourceforge.net/codecoverage/CoverageReport-LineCount.html |
| Last release |
2004-11-05 |
| Licence |
MIT licence |
| Additional information |
- well documented
- instruments ear files, war files, zip files within zip file
- slower performance
|
GRETEL
| Source |
http://www.cs.uoregon.edu/research/perpetual/dasada/Software/Gretel/ |
| Description |
Gretel is a test coverage monitoring tool for Java programs. The current version provides statement
coverage monitoring (identifying which lines of Java have been executed, and which have not been touched by testing). |
| Start |
GUI, command line |
| Technology |
|
| Frameworks |
BCEL |
| Reports |
GUI |
| Last realease |
June 2002 |
| License |
BSD |
HANSEL
| Source |
http://hansel.sourceforge.net/ |
| Description |
Hansel is an extension to JUnit that adds code coverage testing to the testing framework.
Much of the source code is taken from Gretel a "Residual Test Coverage Monitoring"-Tool developed at the University of Oregon. |
| Start |
ANT |
| Technology |
|
| Frameworks |
ASM |
| Reports |
|
| Last release |
2006-10-04 |
| License |
BSD |
| Aditional Information |
|
NOUNIT
| Source |
http://nounit.sourceforge.net/ |
| Description |
NoUnit allows you to see how good your JUnit tests are. It generates a report from your code to graphically show you how many of your project's methods are being tested , and how well |
| Start |
command line |
| Reports |
XML, xsl, HTML |
| Report example |
http://nounit.sourceforge.net/sample.html |
| Last release |
2003-12-15 |
| License |
GNU GPL |
| IDE support |
|
| Addition information |
dont need source codes |
JESTER
COVERLIPSE
CODECOVER
| Source |
http://codecover.org/ |
| Description |
|
| Start |
Eclipse, Command line, ant |
| Technology |
CodeCover measures statement, branch, loop, term coverage (subsumes MC/DC), question mark operator coverage, and synchronized coverage |
| Frameworks |
|
| Reports |
template engine Velocity, html, csv |
| Report example |
http://codecover.org/documentation/tutorials/SimpleJavaAppReport.html |
| Last release |
19.04.2010 |
| Licence |
Eclipse Public Licence (EPL) |
| Comments |
- there is no maven support
- nice reports
- good documentation |
- Licensed
JCoverage
Clover
b) Tools that fullfills criteria
CLOVER
| Source |
http://www.atlassian.com/software/clover/ |
| Description |
Clover delivers actionable Java code coverage metrics to assess the impact of your tests. It goes beyond raw test coverage to help you focus your limited testing resources where they will be most effective. |
| Start |
Eclipse, Netbeans, IntelliJ Command line, ant, maven |
| Technology |
Instrumentation of source codes |
| Frameworks |
|
| Reports |
html, xml, pdf, json |
| Report example |
http://www.atlassian.com/software/clover/sample-reports.jsp |
| Last release |
9-September-2009 |
| Licence |
Commercial, sale for acamedics, free for non-profit and open source projects |
| Comments |
- Most complete reports and statistics
- integration with other atlassian products
- good documentation
- Distributed Test Coverage Aggregation
- Extensibility via java api
- Hudson plugin
|
COBERTURA
| Source |
http://cobertura.sourceforge.net/ |
| Description |
Cobertura is a free Java tool that calculates the percentage of code accessed by tests.
It can be used to identify which parts of your Java program are lacking test coverage.
It is based on jcoverage. |
| Start |
Ant,Command line, Maven |
| Technology |
McCabe cyclomatic code complexity of each class (http://en.wikipedia.org/wiki/Cyclomatic_complexity) |
| Frameworks |
ASM |
| Reports |
Can generate reports in HTML or XML. |
| Report example |
http://cobertura.sourceforge.net/sample/ |
| Last release |
last version 1.9.4.1 released 2010-03-03 |
| License |
GNU GPL 2 |
| IDE support |
http://ecobertura.johoop.de/
written in scala,
supported by one man
http://www.eclipse-plugins.info/eclipse/plugin_details.jsp?id=1151
added 3,2005 modified sep 13 2006 |
| Additional Information |
- cobertura plugin for hudson http://wiki.hudson-ci.org/display/HUDSON/Cobertura+Plugin
- is a fork of jcoverage started at 2005
- When instrumenting, you can now specify a zip, jar, war,ear or sar file and Cobertura will instrument any classes inside of the archive and archives within archives
- Ability to instrument classes on a given classpath instead of specifying filesets
- Can now use multiple filesets in the cobertura-instrument task
- support for classes compiled with AspectJ
- Support for instrumenting classes written in Groovy
- thread safe
- data obtained in different instrumentation or test runs can be merged together
|
| Comment |
|
EMMA
Summary
From mentioned frameworks Clover provides the best services. Clover has the best reports, good documentation and easy to use configuration.
As a open source project we can use the product for free. To get the free license the project must be registred and approved by atlassian.
There is also term in license that code coverage reports must be publicly accessible.
Emma provides good enough functionality but the last release was at 2005-07-27 and there is no progress on the project anymore.
Cobertura provides all functions which are needed for code coverage. The main difference between Clover and Cobertura is in reports.
Cobertura report task provides the same basic data but clover offers much more statistics about projects and coverage.
Cobertura maven plugin has also problems with coverage of integration tests, but there is a workaround for this problem.
I vote for Cobertura because it is open source project and provides sufficient data about code coverage of our project.
(Interesting tool is also Jester which uses mutation testing, but it is not what we are looking for right now.)
|
