EMMA: Frequently Asked Questions

Code coverage refers to a software engineering technique whereby you track quality and comprehensiveness of your test suite by determining simple metrics like the percentage of {classes, methods, lines, etc} that got executed when the test suite ran.

Practice shows that coverage percentages below, say, 60-70% correspond to poorly tested software. You can expect undiscovered bugs in such software (to be discovered by your users, of course). Because of this, "good" software companies instill internal processes whereby a team cannot release a piece of software unless it passes release gates like "line coverage must be 80% or higher", etc.

Incidentally, reaching for 100.0% coverage isn't profitable either. You just get a lot less quality improvement for considerably more effort to reach such perfection. Coverage close to 85-90% is "good enough" for all practical purposes.

The topic of which coverage metric is "better" could be somewhat religious. There have been academic studies showing that, for example, path coverage at a certain level detects somewhat more bugs than, say, line coverage at the same level. I personally think the actual metric definition is not that important. I'd rather empower all developers on my team with a free and fast tool so that they can track their own coverage (of some kind) early and frequently. An experienced developer will look at the coverage report that links to the source code, drill down a bit, look at the "red" areas, and figure out which, if any, areas of the product he left somewhat under-tested. This is why EMMA opts for a set of simple metrics that are easy to obtain without a lot of runtime overhead.

Yes. You need to be able to run your application to get coverage. (Your application does run, doesn't it?) Ideally, the process is automated with a functional test suite, but in a pinch a human "test driver" and a test script on a piece of paper will do.

Getting code coverage for a completely untested piece of software is a lower cost starting point for improving your test situation than writing a comprehensive test suite. With a code coverage profile in hand, you can analyze which areas in code need attention first.

Instrumentation approach. EMMA belongs to a class of pure Java (i.e. not JVMPI- or JVMDI-based) coverage tools based on Java bytecode instrumentation. This means you don't need special JVM switches for enabling coverage: rather, you run EMMA-processed .class files more or less as you normally would (after adding EMMA as a runtime dependency). It also means EMMA does not instrument your .java sources (as is done by Clover, for example).

As a somewhat unique feature, EMMA offers two different options for when to instrument: either as an offline step ("second" compiler) or on-the-fly, when the classes are loaded by EMMA's custom classloader. The former is a generic approach that works for pretty much all contexts (J2SE, J2EE, distributed client/server applications, etc) while the latter is a handy and lightweight shortcut for simple standalone programs.

To reiterate, bytecode instrumentation means EMMA takes in .class content and outputs an enhanced version of it, with added coverage tracking hooks. In the offline mode this happens explicitly (you start with a .class file on disk and get an instrumented version of it also on disk) and in the on-the-fly mode it happens inside the JVM (the instrumented classes are never written out to disk).

Supported coverage metrics. EMMA supports class, method, line, and basic block coverage, aggregated at "all", package, source file, class, and method levels. Line and block metrics are offered in "normal" and "weighted" varieties. Other entries in this FAQ provide details on these.

EMMA does not do branch or path coverage. However, EMMA reports fractional line coverage to help you visualize untested branches in code.

Performance and scale. EMMA trades off the advantages of using full syntactical data in .java sources for the speed with which you can process .class content. It was created as a fast tool for both personal development as well as for massive enterprise software projects. During performance testing, EMMA's instrumentor had been run on more than 20000 .class files in a single run. By now, EMMA has been used on a real-world app with at least 16000 classes (NetBeans).

Java platform support. EMMA pays attention to the needs of enterprise software developers who can't always run in the latest Java version or track down obscure 3rd-party library incompatibilities: EMMA runs in any Java 1.2+ JVM and has no library dependencies.

Licensing. EMMA is free and uses a very liberal open source license.

Yes, it does, in the sense of keeping your fast and incremental development work fast and incremental.

The old approach to collecting code coverage stats is to do it for the entire application, some time during the final testing stages. EMMA was created by a practicing Java developer and is very much a "developer's tool": it encourages personal use, e.g., with the coverage filters and reports scoped to just "your" part of the codebase. In EMMA's vision, all developers on a team check in code that is already well covered and the painful step of bringing the entire codebase to the release levels of coverage becomes unnecessary (or at least much less painful).

Additionally, EMMA tools were purposefully designed for performance, so that coverage could be obtained as a quick side-effect of testing itself. You do test as you write code, right? Why not glance at the coverage report at the same time and get an idea of what to test next?

There is no way to predict how your application will execute and not everybody uses JUnit as the test driver. Because of that, EMMA does not assume any particular testing methodology. Instead, it offers generic tools to coverage-enable your application classes and assumes you will then deploy them as you have been doing it so far. EMMA does not know whether you use Websphere or Swing or whatever. EMMA assumes just a standard J2SE JVM and will collect coverage statistics unintrusively, with a small runtime performance overhead and as a quiet side effect of your testing.

This explains why EMMA does not tout special "JUnit integration". Assuming this project gets some volunteer help, it may happen in the future, but for now JUnit is supported just like any other test framework (you either (a) use emmarun or (b) you instrument your classes after you compile them and before you run your tests).

There is no dire need for special application server/J2EE support in EMMA, because its offline mode works for a J2EE application server as for any standalone JVM. However, due to the complex nature of J2EE application packaging it helps to follow the steps outlined in How do I use EMMA in {WebLogic, Websphere, Tomcat, JBoss, ...}?.

EMMA's fundamental units of coverage are basic blocks (all other types of coverage are derived from basic block coverage in one way or another). In case you are not familiar with the term, a basic block is a sequence of bytecode instructions without any jumps or jump targets. In other words, a basic block always (in the absense of exceptions) executes as one atomic unit (if it is entered at all). Because several Java source lines can be in the same basic block, for efficiency reasons at execution time it makes more sense to keep track of basic blocks rather than individual lines. Furthermore, a given Java source file contains a large number of lines that are not executable (they do not map to any "real" bytecode), such as comments, import statements, field declarations, method signatures, etc.

Now, in Java just about any bytecode instruction can potentially throw an exception. For example, aaload, an instruction to read an object reference from an array, can throw an ArrayIndexOutOfBoundsException if the array index is incorrect. This means that a basic block is not quite as atomic as it was made out above: block execution can be aborted half-way and the control can be transferred to some exception handler per Java language rules. How does EMMA deal with this?

The key observation here is that such failures are usually genuine dynamic program failures and not something the programmer anticipated (by contrast, explicit error handling via Java's try/catch creates its own additional basic blocks). Thus, EMMA does not attempt to track the success or failure of every single bytecode instruction: instead, it marks a basic block covered when the control reaches its last instruction. A covered basic block is thus guaranteed to have executed without failures at least once in a given coverage run session.

The number of basic blocks in a given method is a better measure of that method's complexity than, say, sheer line count. As you create branching logic in your Javacode, you create basic blocks. Thus, basic block coverage is a very desirable type of coverage metric and at the same time it can be obtained with less overhead than, say, path coverage. And as another advantage over line coverage, block coverage is available for classes compiled with no debug information.

Note that 100% basic block coverage always implies 100% executable line coverage. However, a given basic block always belongs to a single method but a given source line may be split between multiple methods or classes (see What's the deal with (fractional) line coverage?), which further lowers the utility of traditional line coverage.

Once again, the motivation behind this has been that most legitimate code is written so that methods return normally (without throwing an exception). If this were to change, every trivial method call like String.length() would need to be considered a block boundary and the coverage instrumentation overhead would be much higher.

Line coverage is a classical but somewhat poorly defined concept. Except for the convenience of linking line coverage to the original source code, I don't think it should be used too much. EMMA's approach to analyzing line coverage is to project basic block coverage onto the source code line structure.

Specifically, EMMA figures out how the basic block structure in a method and the source line information (in LineNumberTable class attribute) map to each other and for each line it determines whether all of the basic blocks that this line is responsible for have been covered. If so, the line is 100% covered. Otherwise, the coverage for the line is a fraction less than 100%.

Although conceptually straightforward, this analysis is complicated by a few factors:

• a given Java source file can contain several classes;
• a given source line may be spread over several methods or even classes;
• code behind a given source line may actually be cloned into several independent methods (e.g., instance initialization code) by the compiler.

EMMA tries to keep line analysis consistent in all such cases, but it might explain why you can see line coverage stats not roll up in an intuitive way from methods to classes or from classes to source files, etc.

See also I am seeing a yellow line in {... code ...}? Why is this line partially covered?.

First of all, a class needs to be considered executable to even be considered for coverage. An executable class is considered to have been covered if it has been loaded and initialized by the JVM. Class initialization implies that the class static constructor (if any) is executed. Note that a class can be covered even though none of its other methods has been executed. It is common to see a small number of loaded but uninitialized classes when you use emmarun without the -f option.

EMMA reports class coverage so that you could spot classes that do not seem to be "touched" by your test suite: they could be either dead code or in need of more test attention.

EMMA considers a method covered when it has been entered (its very first basic block has been covered). Tracking whether the method has completed execution instead would be problematic: a given method can have any number of normal or abnormal exit points and it is not clear which of the many exit paths should be considered the "official" one.

Looking out for uncovered methods is a good technique for detecting either dead code or code than needs more test attention.

Independently of any user-provided instrumentation filters, EMMA always excludes the following from coverage:

• methods that have no executable Java bytecode: abstract methods, native methods;
• Java interface classes (even though they may have executable static initializers).

Additionally, the following types of methods are excluded by default, but can be included by setting the appropriate instrumentation property:

• methods marked as Synthetic by the compiler: these are usually "helper" methods needed by the compiler to implement certain Java language constructs like inner class access, etc. These methods have no user-provided code.
• bridge methods created by Java 1.5 compilers to graft new language features on the existing class format.

A feature to allow EMMA users to mark arbitrary methods as excluded from coverage is being considered for future versions.

Since EMMA bases some coverage types on basic blocks, it gives you a choice to treat all blocks as "equal" (every block contributes the same weight) or not (every block contributes a weight proportional to its size, i.e. the number of bytecode instructions it contains). This decision is made entirely at report generation time, via the report units setting. The default is weighted mode.

In practice, weighted block coverage is also a good approximation of (or alternative to) line coverage if the latter is not available (e.g., the application classes were not compiled with debug info). It is recommended that you choose either weighted block coverage or line coverage for a project, but not both.

See What's the deal with (fractional) line coverage?

They are Java source lines that EMMA determined to have been only partially covered. They usually mean that there is a branch in your code that was not exercised, although EMMA presents no direct way of seeing in the report which branch it is.

In some rare cases, a line could be split across several methods or classes and could be partially covered because one of those methods or classes has not been covered.

It has to be that your code has branches not exercised by your tests, although it is not always obvious what those branches are. Here are a few common examples:

Implicit branches on the same line. Consider both partially covered lines in following example:

Line 6 is really an if-then statement and only one branch of it is taken. Because vj ends up being negative, in the following for-loop only the initialization and loop termination expressions are evaluated (the loop body is never executed). Because vk is never incremented, line 8 has not executed 100%.

Shared constructor code. Can you see why line 17 is not 100% covered in this case?:

Field m_i is initialized with an expression that requires code execution at class construction time. What happens behind Java syntax is that that code is emitted by the compiler effectively twice, once per each of MyClass constructors. Only the copy inside the non-default constructor is executed by main() and hence line 17 is not 100% covered. The execution path going via the default constructor is never tested and EMMA detects that.

I can make this more explicit if I re-write the original code to use an instance initializer:

By the way, re-writing the original code this way highlights an inefficiency: field m_i is assigned twice by the non-default constructor. It is best to declare Java fields final whenever possible to avoid such mistakes.

Implicit branches due to finally blocks. Why line 16 is partially covered in the next example is a little harder to grasp at first:

The reason lies in how finally blocks are implemented by the compiler. The compiler must ensure that the cleanup code in lines 16-25 is always executed, whether or not there is an exception thrown in the try block. There are thus two possible paths leading into the finally block: with and without an exception pending. Execute this example with an invalid filename input to simulate the failure situation and you will see a difference. However, it has to be said that it is not clear why javac assigns that implicit branch code to line 16 and not, say, 14 (the exact behavior here is compiler-specific as well).

Implicit branches due to a hidden Class.forName(). This case is rather unfortunate because it is pretty common and yet the programmer has almost no control over it. Line 10 uses a <ClassName>.class construct, which is currently implemented via a hidden call to Class.forName():

Because Class.forName() can throw checked exceptions, the compiler emits a catch block that rethrows them as unchecked. This catch block hardly ever executes in practice, but it succeeds in marking the line as partially covered.

Future EMMA versions may be able to detect and exclude this code pattern. However, it appears already that <ClassName>.class construct will be implemented differently in Java 1.5.

To calculate coverage, EMMA combines data of two types: metadata (static info about your Java classes, methods, lines, and basic blocks) and runtime coverage data (which basic blocks have been executed). These are stored in files with *.em and *.ec extensions, respectively. This is pure convention: EMMA does not care about file extensions (so you should probably choose whatever works best for your ANT filesets, etc). In fact, each EMMA file acts as a mini-database capable of storing data of both types, even if generated and added at different times.

Some EMMA tools (<emmajava>/emmarun and <merge>/merge) assume that a file dump will contain mixed metadata and coverage data and use *.es extension (session data). Again, this is just convention.

Note that you can mix and match as many files of both types to generate a coverage report as you want. For example, during some client-server testing you can get one *.ec file on a client machine and another *.ec file on an entirely different server machine, and later combine both.

To decide whether a given class is included in a report, EMMA examines all of the metadata you gave it: if a class X is present, it will be reported even if it lacks a runtime coverage data entry. Once you understand this behavior, you can exploit it and partition data by files to achieve various desirable combinations. See I would like to generate one coverage report for my entire project as well as individual reports for sub-projects ... for one such example.

Yes. You can use the offline class instrumentor (<instr>/instr) to instrument your classes, which will produce metadata files (*.em). Then, when you execute your tests, runtime coverage data (*.ec) files are created. You can then use the offline coverage report generator (<report>/report) to generate reports for the entire codebase or a selected subset of code.

As should be clear from Can you provide details on various intermediate data files used by EMMA ..., the key to making this possible is to keep EMMA metadata and coverage data files partitioned by sub-project:

1. instrument all sub-projects and obtain separate metadata files prj1.em, prj2.em, etc. That is, do not merge all of your metadata into a single global *.em file;

2. similarly, run tests and obtain coverage data files prj1.ec, prj2.ec, etc;

3. now, to report on a selected project only, generate a report based on, say, {prj1.em, prj1.ec} files. And to report on the entire codebase, report based on all available files {prj1.em, prj2.em, ..., prj1.ec, prj2.ec, ...}.

Most of EMMA documentation uses internal names for properties. For example, these are the names used in the EMMA property summary in the reference manual. However, there are many (quite frankly, too many) ways to set properties depending on which EMMA tool is being used and whether it is used from command line or from ANT. For example, because the set of JVM system properties is shared by all applications, EMMA carves out its own namespace by expecting a property name prefix emma. when its properties are set through the JVM system (-D) properties. How do you know when to use emma. prefix? The rule of thumb here is that if the properties come from a resource already specific to EMMA (e.g., emma.properties), then the internal names should be used. This is explained in excruciating detail in the EMMA property summary, but here is an example as well (it assumes you want to override the default for session.out.file):

(a) >java emmarun -Dsession.out.file=mydir/myfile ...
(b) >java -Demma.session.out.file=mydir/myfile emmarun ... (compare with (a))
(c) >java emmarun -properties my.properties ...
(d) >java -Demma.properties=my.properties emmarun ... (compare with (c))
(e) >java -cp {directory containing emma.properties} emmarun ... (compare with (c))

where file my.properties sets session.out.file to some mydir/myfile value. Note the difference between -D options for command line tools and -D JVM system properties.

You have both emma.jar and emma-ant.jar copied into the <jre dir>/lib/ext directory. Check the instructions again: you will note that only emma.jar should be installed as a standard JRE extension. Remove emma-ant.jar from <jre dir>/lib/ext and all should be well.

This question has been asked several times and in all cases the users forgot to package or deploy the instrumented versions of their classes. Please check that your build places EMMA-instrumented classes ahead of the normal application classpath at runtime.

Remember that in order to compute coverage <report>/report needs both metadata and runtime coverage data. Chance are, you are missing one of the data types (it's common for ANT users to misplace their *.ec file and not notice it because of relying on ANT's wildcard matching).

In order to see which file you are not picking up, turn on verbose logging and check your input file list.

Because you have not compiled all your classes with full debug data, EMMA is "gracefully" degrading to not doing any source linking and not generating any line coverage data.

Depending on your original intentions, you have several options:

• If you want line coverage and source code linking, recompile and reinstrument with full debug data. If you are compiling in several steps, you might be forgetting the -g (debug='true' in ANT) flag in just one of the steps -- in this case, EMMA's trace ("showing up to 3 classes without full debug info per package...") should help identify which packages were compiled improperly. (Also, see this if the problem persists.)

• You do want line coverage and source code linking, but a 3rd party lib (not debug-compiled) is messing things up. The right thing to do is to exclude that lib from EMMA's instrumentation in the first place.

• You are using EMMA on someone else's compiled classes (no sources) or you only want basic block coverage. The right thing to do is to remove the line coverage column from the report and ignore all remaining warnings.

In order to properly map bytecode to your Java sources, the report generator needs some of the information that is added by javac when its -g option is used (in ANT, it would be the debug='true' attribute of <javac> task).

Even if the generator were allowed to skip linking for the offending classes, it also needs to compute aggregate line coverage percentages for the rest of the report. Without knowing line coverage for all classes in the instrumentation set there is no way to roll up such statistics.

EMMA should log some of the offending classes either when instrumenting or reporting, which should help track the problem down. Some common reasons for appearance of classes with incomplete debug info are:

• Your application may be using RMI. Sun's rmic tool will incorrectly fail to generate complete debug info even when it is given the -g option. The correct solution here is to exclude the offending classes using a coverage filter. The classes you would be excluding will be rmic-generated RMI proxy/stub classes anyway, not anything you coded yourself.

• You may be using some other bytecode enhancer (e.g., a JDO implementation). Unlike EMMA, most enhancers do not preserve the debug info in the classes they touch. The correct solution here is to make sure that EMMA is the first tool in the build chain: do EMMA instrumentation first and then apply JDO, etc enhancing.

• You may have chosen to let EMMA process an EJB .jar file that is ready to be deployed into a J2EE container. Some J2EE vendors' tools will inject proprietary classes into J2EE archives behind the scenes (e.g., Websphere adds loadbalancing-aware EJB proxies). To work around this, either let EMMA do its instrumentation before the J2EE tool gets to your classes or use a coverage filter to exclude the injected stuff.

• You may be using debuglevel attribute of ANT's <javac> task and you have spaces between lines, source, vars keywords. ANT passes this option unchanged to the actual compiler executable and the spaces cause some of the keywords to be silently ignored. Remove the spaces.

First of all, there is little chance that you will be able to use the on-the-fly mode (emmarun) with a full-blown J2EE container. The reason lies in the fact that many J2EE features require specialized classloading that will happen outside of EMMA instrumenting classloader. The server might run fine, but you will likely get no coverage data.

Thus, the correct procedure is to instrument your classes prior to deployment (offline mode). Offline instrumentation always follows the same compile/instrument/package/deploy/get coverage/generate reports sequence. Follow these steps:

1. use EMMA's instr tool to instrument the desired classes. This can be done as a post-compilation step, before packaging. However, many users also find it convenient to let EMMA process their jars directly (either in-place, using overwrite mode, or by creating separate instrumented copies of everything, in fullcopy mode);

2. do your J2EE packaging as normal, but do not include emma.jar as a lib at this level, that is, within your .war, .ear, etc;

3. locate whichever JRE is used by the container and copy emma.jar into its <jre dir>/lib/ext directory. If that is impossible, add emma.jar to the server classpath (in a server-specific way);

4. deploy your instrumented classes, .jars, .wars, .ears, etc and exercise/test your J2EE application via your client-side testcases or interactively or whichever way you do it;

5. to get a coverage dump file, you have three options described in What options exist to control when EMMA dumps runtime coverage data?. It is highly recommended that you use coverage.get control command with the ctl tool available in v2.1.

Starting with build 2.1.5320, you can use the new ctl tool to either get coverage data remotely from a running JVM or ask it to dump the data in a file. This can be done as many times as needed without terminating the JVM. See 2.1.5320 release notes for usage instructions.

In v2.0, it is possible as well, but with a little bit of extra work: see What options exist to control when EMMA dumps runtime coverage data?.

The options are:

1. The default behavior is to dump coverage data in a JVM shutdown hook. This happens when the VM exits or is killed via Ctrl-C (in Java 1.2.x, only the latter works).

2. NEW in v2.1

   ctl tool

   Whenever your instrumented code is running you can request a code coverage dump using EMMA's ctl tool. The tool can even run on a different machine (and different OS) from your instrumented code. You get to choose where to save coverage data: on the machine that is running ctl or on the machine that is running your instrumented code.

3. Finally, your application can request a coverage dump programmatically, by invoking the following method:

   com.vladium.emma.rt.RT.dumpCoverageData(File outFile, boolean merge, boolean stopDataCollection)

   Use false for the second argument to ignore any data in outFile if it exists and use false for the last argument if you would like to invoke this method repeatedly.

   Of course, this option only works from within the JVM that is running your instrumented code, so you would need to invoke this from a JSP, a servlet, a servlet filter, an application listener, etc. Prior to build 2.1.5320, this was the only way to trigger a coverage dump on demand. This option is now deprecated in favor of the new ctl tool.

Not in v2.0, but starting with early access build 2.1.5320 you can try coverage.reset control command with the ctl tool. You can even reset coverage several times and run a different test suite each time while observing which parts of the application get covered.

This question usually applies when running EMMA-instrumented classes in offline mode. By default, the coverage dump file is created in what the JVM considers to be its user.dir. When the JVM is started from command line, this is normally the user's current working directory and things are reasonably intuitive. However, for JVMs running as OS services or launched via convoluted shell scripts you might want to set the coverage.out.file property to a known absolute pathname.

If you are still stumped, perhaps it is time to try the ctl tool available in v2.1.5320.

You are probably using EMMA's default output mode for offline instrumentation: copy. In this mode, <instr>/instr only outputs classes that got instrumented. So, a class that was not instrumented (for example, because it was not executable or because it was filtered out) will never show up in the output directory. When the instrumented classes are added to the start of the execution classpath, the uninstrumented classes "show through" in the original classpath. (This is by design)

You are probably using EMMA's default output mode for offline instrumentation: copy. In this mode, <instr>/instr only outputs classes that got instrumented. So, a class that was not instrumented (for example, because it was not executable or because it was filtered out) will never show up in the output directory. When the instrumented classes are added to the start of the execution classpath, the uninstrumented classes "show through" in the original classpath. (This is by design)

You might be getting tripped up by the incremental nature of EMMA instrumentation. When debugging an EMMA-enabled build, try either a clean recompile and/or delete all instrumentation output directories and all *.em/*.ec files by hand to reset EMMA to a clean state.

When used repeatedly (in copy and overwrite modes), <instr>/instr tool will re-instrument only those classes that appear to have been changed since the last instrumentation run (much like javac itself). In copy mode this is detected via the output .class file being older than the input file. In overwrite mode this is detected via the output .class file not having any EMMA instrumentation.

See the reference manual for further details.

This appears to be a problem for users of Apache web servers. The default Apache charset mapping (UTF-8) incorrectly takes precedence over the META tag in EMMA HTML files (ISO-8859-1).

See this user forum thread for more details.

Either switch to using the ANT task or use -Dreport.out.file=<pathname> command line tool option. If you are generating more than one report type in one go, use -Dreport.<report type>.out.file=<pathname>.

(Note: EMMA properties can also be set as JVM system properties. In this case, prefix the property names with emma., see How do I change an EMMA property default setting?.)

This means the application is being loaded by the JVM system classloader, not EMMA itself. Assuming the emmarun command is

>java -cp A emmarun -cp B ...

move your application classes from A to B.

Yes. Starting with build 2.0.4217, you can set the verbosity.level property to silent. This will suppress all but severe error messages.

Start by adding -verbose option to command line tools and/or by setting verbosity ANT task attribute to verbose. EMMA logging can be made even more verbose by increasing the verbosity level to trace1, trace2, and trace3. (These attributes and options were added in build 2.0.4217).

Please use the bug tracker at EMMA's SourceForge project home page.

Please use the RFE tracker at EMMA's SourceForge project home page.