Comparing Code Coverage of Randoop and EvoSuite using EclEmma and JD-Eclipse

The first thing you need to do is make sure you have EclEmma and JD-Eclipse installed.  I ran into a little trouble decompiling .class files with JD-Eclipse at first because there isn't good documentation for how to actually use it once it is installed.  The first thing you need to do is to make sure that the class files are on your build path.  To do this I just did new->source folder. Then I made sure that the within that folder I had the proper folder structure for my class.  In this instance the class file was part of the "example.tritype2" package so I needed to make sure that the directory structure was example/tritype2/.  Then in that folder I put my class file.

From there you should be able to double click on it using the Navigation view and see a decompiled Java program.  If not try right clicking on the file and selecting Open With->Class File Editor.  If that doesn't work for you, then I am afraid I can't be of much help because I am new to using the tool myself.

Now that you have JD-Eclipse working you can right click on your project and go to New->Package. Create the package(s) that correspond to each class file in the JAR that you need to run coverage analysis on.  Then for each class file create new empty Java class in the appropriate package.  Finally just open the class file and copy and paste the decompiled Java into the java file.

Next you need to add two new source folders.  one for EvoSuite tests and the other for Randoop tests.  Then copy your EvoSuite test folder into the corresponding Eclipse source folder.  From there you need to do something similar with Randoop tests, but since they aren't in a nice directory structure like EvoSuite you can just copy all the tests directly into the randoop tests source folder.

Fix EvoSuite Tests

You will need to do is to edit the EvoSuite test becuase it will have an error.  You need to remove 3 lines.  First remove from the imports section

import org.junit.runner.RunWith;
import org.evosuite.junit.EvoSuiteRunner;

Then from right above the class header remove the following annotation.

@RunWith(EvoSuiteRunner.class)

I originally put EvoSuite on my classpath to get the errors to go away but that caused problems when I tried to get coverage.  I found that by just removing them it fixed everything.

The next thing I did was create coverage runs to make it easier to quickly run each and not get confused.  For the first one right click on the EvoSuite test folder and go to Coverage As -> Coverage Configuration.  Then in the JUnit section edit the Test area so that it runs all of the tests in that folder.  In the Coverage tab make sure only the src folder is selected, since it is the only folder we are trying to get coverage on.

EvoSuite test coverage configuration.

EvoSuite coverage selection.

Randoop will be  a bit more tricky because we have to create a coverage run for each class (EclEmma will allow us to combine these together later on to get a single coverage profile).  It is probably best to create a sub folder in the Randoop folder for each class.  Then right-click on the first sub-folder and go to Coverage As -> Coverage Configuration again.  This time you want to click Run a single test because if you try to run all tests in the folder it will run out of memory.  However if you select just the test class without a number, then that one will call all of the other ones.  If you try to run the entire folder than the tests get run twice and it overloads the JVM.

Randoop test coverage configuration

Now after you run each of the Randoop coverage tests it is quite easy to merge them together.  Just select the Merge Sessions option in the Coverage window that opens at the bottom and select which ones you want merged.  Then you will have a fair comparison between the two.  The sad news is that you will have to use a calculator and add up all the JUnit times yourself so that you can figure out how long it took each to run.


Hope this helps. Peace!