Android studio code coverage

Android+Kotlin Code coverage with Jacoco, Sonar and Gradle plugin 6.x

Sep 21, 2020 · 7 min read

At Wandera, we have been measuring our code coverage for quite a while, but we relied on a 3rd party plugin that solved the Jacoco plugin configuration for us because we didn’t want to deal with Groovy language for speed and simplicity.

We were dealing with two issues — we were using an outdated Gradle plugin and we were missing the instrumentation test coverage in our reports.

When we upgraded the Gradle plugi n to the newest version ( 6.6.1 as of writing this blogpost), it wasn’t all sunshine and rainbows with the then-used 3rd party plugin as it had not been maintained for a good year.

I learned the plugins are quite simple to implement, so to encourage you not to use 3rd party plugins when it is not necessary, I’d like to present you with a thorough guideline.

We are going to go through how to setup Jacoco plugin, how to setup the client part of Sonar, how to be able to choose one or both the jvm and instrumentation tests code coverage and what to watch out for when using version 6.x of the Gradle plugin.

Jacoco plugin

Firstly, you will want to add Jacoco plugin to your project.
Add classpath “ org.jacoco:org.jacoco.core:0.8.5 ” to your project.

—
Version 0.8.5 contains a small bug so if you want to avoid any issues, use a different version instead
https://github.com/jacoco/jacoco/issues/968
—

Coverage configuration

Now we will want to compute coverage of JVM unit tests and Android instrumentation tests.

You can configure Jacoco in your module’s build.gradle file, but it is much cleaner to have larger configurations separated in new gradle files for easier maintenance. A separate file may also be reused across modules or projects.

Contents of the jacoco.gradle file (located where the module’s build.gradle file is) explained below:

Gradle tasks

We created a task for each variant of our app. So if you have a debug, dev and release variant, you should see testDebugUnitTestCoverage, testDevUnitTestCoverage and testReleaseUnitTestCoverage Gradle tasks in your right-side Gradle tab in Android Studio. You may name the jobs whatever you like.

Then we need to declare that our testXxxUnitTestCoverage task depends on running the tests so that we have some data to compute the coverage from. We cannot start with computation until the necessary output files are generated.

To enable Android Instrumentation tests coverage, your module’s build.gradle file should look like this:

Important: the Android Instrumentation test coverage works only if you set your application variant’s property testCoverageEnabled true

When the test coverage is not enabled, the testXxxUnitTestCoverage tasks for such variants won’t work unless you exclude androidTestCoverageData from dependencies (for instance by not depending on the task and by not expecting the report to be available for execution data).

On the other hand, you only need to keep track of one of your variants which usually is the debug one.

Report files

Moving on, we specify the types of reports we want to have generated.

HTML format is the most human-friendly. XML and CSV reports are used for other reporting tools, for instance Sonar or a CI plugin.

Sources

We need to tell Jacoco where to find our Java and Kotlin classes

You can see the build directory can be referenced variably.

If you don’t include the java classes, any tests written in java won’t be taken into account. Same for Kotlin.

Then we want to exclude generated classes that we cannot cover with tests and would make the coverage unnecessarily low. These are usually classes generated by 3rd parties during compilation, such as Dagger/Hilt, Butterknife, Kotlin etc. You may exclude anything you consider unnecessary to appear in your reports.

I enriched my list by looking in the following sources.

Gradle plugin 6.x update

In one of the newest Gradle plugin versions, some properties were made read-only. That’s one of the reasons most of the abandoned 3rd party plugins have stopped working. Setter methods must be used instead.

So instead of a direct assignment, setters must be used.

You may want to add certain directories for a specific variant type.

You can do so in

Tests data

At last we need to specify where to find the execution data.

An .exec file is created for the JVM unit tests and a similar file of .ec type is created for Android instrumentation files.

We define both of these paths using the Jacoco’s executionData property.

If you only want to track coverage of your unit tests, don’t add the androidTestsData path. The same goes for the instrumentation tests.

For future reference, some Android plugin versions change the build dir hierarchy, so in case it is not working in the future, make sure the files are generated at the path you provided. If not, change the path to the directory the files now appear in.

You can find the description of each Jacoco property by following the below link.

Apply the plugin

Now that we have the plugin configuration done, we need to make sure it is applied.

In your app’s build.gradle file, add the following line apply from: “ jacoco.gradle” so it looks like the following:

Now when you run

you should see your reports appear in the Jacoco directory.

Watch out for the Gradle plugin version!

This whole story started with updating the Android Gradle plugin to version 4 as the Android Studio likes to warn us about new updates.

When you update the Android Gradle plugin version, it usually comes with updating the Gradle wrapper version to be compatible as per the documentation on the official site https://developer.android.com/studio/releases/gradle-plugin

And when you are updating, you usually select the latest version possible. So we did update the gradle-wrapper.properties file:

After that, our automated builds started failing on not having the Jacoco reports available!

After the whole day and night of searching for all possible issues with Jacoco I finally bumped into the Gradle issues Github site saying the Gradle plugin version 6.4 makes the Android connected tests run separately and that Jacoco does not wait for them to finish to pick up the report.

All you need to do though is to downgrade to 6.3 and wait for a fix

Configure Sonar

You want to continue reading if you have your Jacoco reports ready and you want to be able to automate the visualisation and you would like to use the static analysis that Sonar offers.

This section assumes you know how to setup the server side and you know a little about Sonar.

You may again configure sonar in a separate gradle file, such as sonar.gradle and put it next to jacoco.gradle and not to forget, add “ apply from” to the main build.gradle file.

Add a new dependency in your module’s gradle file

We still use the 2.7.1 version, but I think it is safe to update to 2.8.

Then add this at the bottom (or in your newly created sonar.gradle file)

The pasted code snippet should work for all Sonar plugin versions, although some of the parameters may be deprecated.

Источник

Измерение покрытия кода тестами в Android с помощью JaCoCo

С тех пор как эта возможность появилась в Android Gradle плагине версии 0.10.0 было написано много статей об измерении покрытия кода тестами (test coverage) — и я не испытываю никаких иллюзий по этому поводу. Однако, что меня раздражает, так это необходимость заглядывать в несколько таких статей и даже в документацию Gradle прежде чем вы получите полностью работающее решение. Так что вот, еще одна статья которая попытается это исправить и сберечь ваше время.

Постановка задачи

Имеется Android проект с модульными тестами (unit tests), и мы хотим создать отчет о покрытии кода для выполненных тестов. Решение должно поддерживать различные режимы сборки (build types) и вариации продукта (product flavours).

Решение

Решение состоит из несколько частей, поэтому давайте рассмотрим его по шагам.

Включите сбор данных о покрытии кода

Вам нужно включить поддержку сбора данных о покрытии кода тестами для режима сборки (build type) в котором вы будете выполнять тесты. Ваш build.gradle должен содержать следующее:

Настройте JaCoCo

Хотя всё из этого раздела можно было бы поместить в build.gradle , такой «навесной монтаж» сделает ваш сценарий сборки не читаемым, поэтому я рекомендую поместить всё это в отдельный сценарий сборки, а затем импортировать.

Мы начнем настройку JaCoCo с создания файла jacoco.gradle в корневом каталоге проекта. Можете создать его в любом месте где пожелаете, но держать его в корневом каталоге проекта позволит легко на него ссылаться из всех подпроектов.

Самая простая часть — импорт JaCoCo:

Обратите внимание, что вам не нужно объявлять какие-либо зависимости чтобы использовать плагин «jacoco» — всё что нужно подключит плагин Android.

Чтобы проверить какая версия последняя, поищите org.jacoco:org.jacoco.core в jCenter, но обновляйте осторожно — самая последняя версия может оказаться пока еще несовместимой, что может привести к каким-нибудь странностям, например к пустому отчету.

Следующий шаг это создание задач Gradle (Gradle tasks) для всех вариаций продукта и режимов сборки (на самом деле вы будете тестировать только отладочную сборку (debug), однако очень удобно иметь такую возможность для любой специальной конфигурации отладочной сборки).

Обратите внимание что collect в Groovy получает на вход список, вызывает функцию с каждым элементом списка, а результаты возвращает в новом списке. В данном случае на вход поступают списки объектов «режим сборки» и «вариация продукта» которые преобразуются в списки их названий.

В угоду проектам в которых не заданы вариации продукта мы добавим пустое имя:

Теперь мы можем вот так пролистать их, что по существу является вложенным циклом в Groovy:

Самая важная часть — то, что мы поместим внутрь цикла, поэтому давайте рассмотрим это более подробно.

Сначала мы подготовим имена задач с правильной расстановкой заглавных букв:

• sourceName – название источника сборки (build source name), н-р: blueDebug
• sourcePath – путь к исходным кодам сборки (build source path), н-р: blue/debug
• testTaskName – задача для выполнения тестов от которой будет зависеть задача измерения покрытия кода, н-р: testBlueDebug

Вот как мы их определяем:

Теперь задача, как она выглядит на самом деле:

Вы могли встречать похожий код в других статьях про JaCoCo, поэтому надеюсь, что большая его часть понятна без объяснений.

Части заслуживающие дополнительного внимания:

• classDirectories – в «excludes» вы можете перечислить шаблоны для исключения из отчета; это может быть сгенерированный код (класс R , код внедряющий зависимости и т.д.) или что-угодно, что вы захотите игнорировать
• reports – разрешает HTML и/или XML отчеты, в зависимости от того, нужны ли они для публикации или для анализа, соответственно.

Вот и всё о jacoco.gradle , поэтому вот полное содержимое файла:

В заключение вам нужно импортировать этот сценарий сборки в ваш сценарий в app как-то так:

(Обратите внимание: Здесь подразумевается, что jacoco.gradle расположен в корневой директории вашего проекта, как было описано выше)

Вот и всё! Вы можете убедиться, что задачи создаются, выполнив gradle tasks и поискав в секции «Reporting» что-то похожее на следующее:

Чтобы создать отчет, выполните gradle testBlueDebugUnitTestCoverage и вы найдете его в «build/reports/jacoco/testBlueDebugUnitTestCoverage/» .

Источник

Android studio code coverage

Locate and schedule your COVID-19 vaccination at a pharmacy near you.

• Rally
• Code Coverage for Android Testing

Code Coverage for Android Testing

By Santosh Astagi | June 4, 2018

At Rally, we focus a lot on automated testing. We spend a significant amount of time writing tests and thus need to make important decisions around testing strategies. As a good practice, we try to measure and maintain a high code coverage for our projects. With a variety of testing tools available, this post is about how we measure code coverage and the common gotchas encountered in Android projects. Most of this knowledge is already out there in some form of Stack Overflow question or forum question, but we have made an attempt here to consolidate all the information a developer needs to set up unified code coverage for an Android project.

For Android, any test that requires a hardware or an emulator is an instrumented test. Such tests are placed in the androidTest folder. Tests that do not require an emulator/hardware are placed in the tests folder. This blog post covers extracting code coverage for unit tests, instrumentation tests, Robolectric tests and tests written in Kotlin.

If you are writing tests in Kotlin you may encounter an issue mocking final classes. Since Kotlin classes are final by default, one may have to open the classes or functions, or create interfaces. Both these options are not appealing. Mockito 2 allows mocking final classes. In order to do that, one needs to create a file named org.mockito.plugins.MockMaker and place this under the resources/mockito-extensions directory in the test folder. This file contains a single line:

That’s it! Now you can mock final classes in Kotlin using Mockito 2.

Code coverage setup

Android Studio generates HTML coverage reports only for instrumentation test. To view HTML coverage reports for unit testing, one will need Jacoco (Java Code Coverage). With Android plugin 3.0.0 onwards, one cannot configure Jacoco using the android DSL block. Thus, in the project level gradle file, one needs to add the Jacoco dependency with the gradle dependency, forcing the instrumented tests to use Jacoco.

Now, for the module containing the tests, update the respective gradle file to apply the Jacoco plugin and specify the same toolVersion as used in the project gradle file.

You may encounter an issue while getting test coverage for Robolectric tests. To include Robolectric tests in the Jacoco report, one will need to set the includeNolocationClasses flag to true. This can no longer be configured using the android DSL block, thus we search all tasks of Test type and enable it.

To get coverage reports for instrumentation tests, in the android DSL block, set the testCoverageEnabled flag to true.

AndroidJUnitRunner runs on the same instrumentation process so basically your tests are running stateful, this might wreak havoc if your tests have some sort of dependency on the process state. To avoid this, Android recommends using ORCHESTRATOR . In Android test orchestrator each test is run in its own process so dependencies aren’t an issue.

You may also disable the animations during the test execution, and enable the Android resources for the unit tests so that Robolectric tests won’t fail.

For the purpose of this post, we will be running the tests for the debug build variant.

The following is what your gradle file might look like after incorporating the above changes.

Instrumentation tests

To get coverage reports for the intrumentation tests, one can run the gradle task createDebugCoverageReport . This will generate the coverage report for the instrumentation tests only. This report is generated under build/reports/coverage/debug . If you don’t see createDebugCoverageReport , check the testCoverageEnabled flag, that should be true for the build variant, you are interested in.

Local unit tests

To get a quick peek at the code coverage for unit tests, one can click the “Run test with coverage”, to the next of the debug button(highlighted with a yellow circle). This will show the code coverage within Android Studio:

However, if you are interested in getting the raw coverage files, you can run the gradle task testDebugUnitTest . This generates the raw coverage file in the jacoco folder within build/outputs . The raw coverage file will be called testDebugUnitTest.exec .

As mentioned before, Android Studio does not generate Jacoco reports for the unit tests. However, you do have the raw coverage file ( testDebugUnitTest.exec ). This can be used to generate the Jacoco reports by writing a gradle task:

This task first runs its dependency, to make sure the raw unit test coverage files are available. The next set of commands specify the source code to evaluate coverage against. We then filter to remove some of the unwanted files (e.g. R files, manifest, configs, etc.) from the code coverage calculations. javaClasses gives the location of the files in the build directory. All this data is then fed into the relevant parameters for the Jacoco report (e.g. classDirectories , sourceDirectories ). The executionData is then made to point to the raw coverage data file. Finally, one can specify the types of reports to be generated (HTML, XML). After running this task, the coverage report will be available under build/jacoco/jacocoUnitTestReport . You should be able to see something along the following lines:

Kotlin

To include Kotlin files, add «src/main/kotlin» to the coverageSourceDirs first. For Kotlin, the location of the debugTree files are different than that for Java. The Kotlin files are located at tmp/kotlin-classes/debug instead of /intermediates/classes/debug . Thus, you will need to add the following after the javaClasses variable:

And then include that in the classDirectories : classDirectories = files([ javaClasses ], [ kotlinClasses ]) . This should be sufficient to generate the coverage reports for Kotlin tests.

Aggregate coverage report (Instrumentation + Unit)

Unit test raw files are of the .exec format, while the instrumentation tests are of the .ec format. That is not an issue while creating the aggregate coverage report. We can use both these raw files while setting up the execution data for the Jacoco report tasks. Thus to get an aggregate coverage report, update the executionData to include the instrumentation test coverage reports:

Multi module app

If you have multiple modules in your project and tests are spread across these modules, you could create a gradle task in every module to get the unit test coverage reports, and then create a root level gradle task to generate an aggregate coverage report by specifying the classDirectories , sourceDirectories and executionData from the different modules.

Since, we were going to use a similar task in every module to generate the coverage reports for unit tests, we decided to abstract this out into a gradle method, which accepted the module and build variant as the parameters and generated the coverage reports for that module:

We then call this from the gradle file of all the submodules and pass the project and the build variant as follows:

And finally, we apply the plugin and this function in the root gradle file:

Now you have the unit test coverage reports for all modules with the instrumentation test reports using createDebugCoverageReport . To generate the aggregate coverage report across all these modules and test types (instrumentation, unit etc.), there is a gradle plugin created by the folks at Palantir — com.palantir.jacoco-full-report . This needs to be applied in the root level gradle file:

Now, to create an aggregate coverage report across all modules (basically for the Android project), run the following gradle command: ./gradlew jacocoFullReport . The aggregate report will be available in the root level build directory under reports/jacoco/jacocoFullReport .

Firebase TestLab

You probably want to generate the coverage reports in your continuous integration (CI) environments. In order for instrumentation tests, the CI will need to run an emulator or hardware device. Jenkins has an Android emulator plugin, which can be used for this purpose. However, setting up an emulator in an emulated environment comes with its own set of challenges. But wouldn’t you love to run your tests on an actual device? Or a range of devices and OS versions? That is possible using Firebase TestLab.

Firebase Test Lab is a service provided by Firebase to run instrumentation tests on actual devices or emulators with specific configurations. This obviates the need to run tests on an emulator on Jenkins, when you can run your tests on a variety of devices in the cloud. The test lab reports include many things such as videos of the test runs, error reports and coverage reports too. These coverage reports are stored on the gcloud bucket associated with the Firebase account of the Android project. In order to get a simple run of firebase test lab, you can execute the following command in your continuous integration environment:

A few points to consider here:

1. This command requires that the apks and test apk be available. That is possible using the gradle assemble command.
2. The Android manifest for the test apk needs to enable external storage, else the coverage reports will not be generated as the coverage files are stored on the device and downloaded from there.

You can then download the coverage reports from the firebase gcloud bucket for the project by setting up gcloud and using the gsutil cp command to specify the coverage source location and the destination to copy the coverage files to.

Flank

A common problem with mobile ui tests it that it could take a significant amount of time to run all the ui tests for a mature product (nearly an hour or more). Android actually added annotations to signify small, medium, and large tests, so that you could exclude certain tests when you run so they don’t take forever. Enter Flank — an open source project built by Walmart labs with support from the Google Firebase team. Flank effectively helps scale the number of tests through test sharding to keep the total test execution time low so that all the tests can be run as part of the CI deployment process. This could also help one reduce their costs on Firebase. In a nutshell, while Firebase will run every test on a separate device and charge rounding up to the minute, Flank, stores the test run times and then aggregates to optimize the number of devices used. E.g. If you have around 300 tests and each test takes 10 seconds, the first execution will run on 300 devices and will charge for one minute for each device. The test execution times are saved by flank. Consecutive runs will take into account the test times and thus will use 50 devices (6 tests on each device) for the test runs. This helps save some setup/tear down time. Flank is well explained on this blog post and the project resides here. After the tests are run, all the details regarding the tests such as videos, coverage reports are downloaded in the root folder of the Android project in a folder called results .

The entities and logos identified on this page may have trademarks and those trademarks are owned by the respective entity.

Источник