Checking dependencies excluding mac html5 android linux

Compiling on Linux: Missing zlib 1.2.8, libPNG-1.5.27 etc. #64

Comments

Cygon commented Sep 9, 2020

I’m trying to build this fork under Linux and I’m running into missing / wrongly set up third-party libraries.

I cloned the git repo:

(Just to be safe, I then ran git submodule update —init —recursive , but there were no submodules)

I pulled in UE4 dependencies:

(The excludes are perhaps a bit excessive, but my Unreal Engine 4.24 install, when built for Linux from Git, is already a honking 59 GiB — I’m currently doing a full checkout to see if this will affect the error, but I suspect not).

I attempted to run HTML5Setup.sh

But this one encountered issues. First, zlib-1.2.8.tar.gz was missing. I manually downloaded it and put it in the right folder and re-ran HTML5Setup.sh , which brought me a bit further.

Next, libPNG was missing options.awk — the sources were there, but it kept looking for files that weren’t there. options.awk was the first problem, so I assume it is the original issue and the rest may be follow-up problems.

What might be going wrong?

• The 4.24-html5 branch is the default, so I think I’m supposed to use that one
• The —single-branch and —depth 1 shouldn’t affect the checkout (I don’t need the full repo history on my hard drive)
• The —exclude set worked fine with a normal 4.24 build. I’m currently checking if things change when I download all dependencies (in case the HTML5 build script is ninja-ing something out of another Platform’s dependency folder)

The text was updated successfully, but these errors were encountered:

Источник

Использование сканера уязвимостей в используемых библиотеках Dependency-Check в GitlabCI

Важная часть управления уязвимостями состоит в том, чтобы хорошо понимать и обеспечить безопасность цепочки поставок тех компонентов ПО, из которых строятся современные системы. Команды, практикующие гибкие методики и DevOps, широко применяют библиотеки и каркасы с открытым исходным кодом, чтобы сократить время и стоимость разработки. Но эта медаль имеет и обратную сторону: возможность получить в наследство чужие ошибки и уязвимости.

Очевидно, команда должна обязательно знать, какие компоненты с открытым исходным кодом включены в ее приложения, следить за тем, чтобы заведомо надежные версии скачивались из заведомо надежных источников, и загружать обновленные версии компонентов после исправления вновь обнаруженных уязвимостей.

В этом посте рассмотрим использование OWASP Dependency Check для прерывания сборки в случае обнаружения серьезных проблем с вашим кодом.

В книге «Безопасность разработки в Agile-проектах» описан так. OWASP Dependency Check – это бесплатный сканер, который каталогизирует все используемые в приложении компоненты с открытым исходным кодом и показывает имеющиеся в них уязвимости. Имеются версии для Java, .NET, Ruby (gemspec), PHP (composer), Node.js и Python, а также для некоторых проектов на C/C++. Dependency Check интегрируется с распространенными средствами сборки, в т. ч. Ant, Maven и Gradle и серверами непрерывной интеграции типа Jenkins.

Dependency Check сообщает обо всех компонентах с известными уязвимостями из Национальной базы данных уязвимос­тей (NVD) NIST и обновляется на основании данных из новостных каналов NVD.

По счастью, все это можно делать автоматически с помощью таких инструментов, как проект OWASP Dependency Check, или коммерческих программ типа Black Duck, JFrog Xray, Snyk, Nexus Lifecycle компании Sonatype или SourceClear.

Эти инструменты можно включить в сборочные конвейеры, чтобы автоматически составлять опись зависимостей с открытым исходным кодом, выявлять устаревшие версии библиотек и библиотеки, содержащие известные уязвимости, и прерывать сборку в случае обнаружения серьезных проблем.

OWASP Dependency Check

Для тестирования и демонстрации работы Dependency Check используем этот репозиторий dependency-check-example.

Для просмотра HTML отчета нужно настроить web-сервер nginx на вашем gitlab-runner.

Пример минимального конфига nginx:

В конце сборки вы можете видеть вот такую картину:

Переходим по ссылке и видим отчет Dependency Check.

Первый скриншот — верхняя часть отчета с кратким содержанием.

Второй скриншот подробности CVE-2017-5638. Здесь мы видим уровень CVE и ссылки на эксплоиты.

Третий скриншот — подробности log4j-api-2.7.jar. Видим что уровни CVE 7.5 и 9.8.

Четвертый скриншот — подробности commons-fileupload-1.3.2.jar. Видим что уровни CVE 7.5 и 9.8.

Если вы хотите использовать gitlab pages, то не получится — упавшая таска не создаст артефакт.

Вывод сборки: артефактов нет, html отчета не вижу. Надо попробовать Artifact: always

Регулирование уровня CVE уязвимостей

Самая главная строка в файле gitlab-ci.yaml:

Параметром failBuildOnCVSS вы можете регулировать уровень CVE уязвимостей, на которые нужно реагировать.

Скачивание с интернета базы данных уязвимос­тей (NVD) NIST

Вы заметили что постоянно скачивает базы данных уязвимос­тей (NVD) NIST с интернета:

Для скачивания можно использовать утилиту nist_data_mirror_golang

Установим и запустим ее.

Nist-data-mirror закачивает CVE JSON NIST в /var/www/repos/nist-data-mirror/ при запуске и обновляет данные каждые 24 часа.

Для скачивания CVE JSON NIST нужно настроить web-сервер nginx (например на вашем gitlab-runner).

Пример минимального конфига nginx:

Чтобы не делать длинную строку где запускается mvn вынесем параметры в отдельную переменную DEPENDENCY_OPTS.

Источник

How to exclude Gradle dependencies

Last Updated on November 29, 2021

Some combinations of transitive dependencies in a project can cause issues, but fortunately Gradle has several ways to exclude those unwanted dependencies. In this article you’ll learn why you’d want to exclude dependencies in the first place, as well as how to use each of Gradle’s exclude options.

Reasons to exclude dependencies

When you declare a dependency in your build script, Gradle automatically pulls any transitive dependencies (dependencies of that dependency) into your project. In Java projects these dependencies make their way onto the compile or runtime classpaths.

Let’s explore some scenarios where certain combinations of dependencies can cause an issue in your project.

1. Multiple SLF4J bindings

The SLF4J logging library requires that only one binding appears on the classpath, otherwise it doesn’t know which implementation to use for logging. For example, if you included both the Logback and Log4J bindings you’d get this error.

SLF4J: Class path contains multiple SLF4J bindings.

This means one of those bindings needs to be removed from the classpath, which you’ll see shortly how to do using exclude rules.

2. Unused transitive dependency

Sometimes we only need to use a very small part of a dependency artifact. One or more of its transitive dependencies may not be needed at compile or runtime.

For example, Google’s popular Guava utility library pulls in several transitive dependencies, such as com.google.code.findbugs:jsr305. This artifact, which we’ll refer to as findbugs, contains annotations that might not be needed at runtime. If we’re only using one method from Guava that we’re confident doesn’t use findbugs, then findbugs is a potential candidate for exclusion.

Of course if we’re taking such an approach we need to be confident that the excluded library won’t be required now or at any time in the future. This can be validated with automated tests to exercise that area of code.

The benefits of excluding unused transitive dependencies include:

• cleaner compile classpath improves performance
• smaller application deployable due to less artifacts on the runtime classpath

Now you know what problems you might see with transitive dependencies, let’s explore how Gradle solves them. All the following code samples are in Groovy, although each approach is also given in Kotlin where stated.

Option 1) per-dependency exclude rules

When you specify a dependency in your build script, you can provide an exclude rule at the same time telling Gradle not to pull in the specified transitive dependency.

For example, say we have a Gradle project that depends on Google’s Guava library, or more specifically com.google.guava:guava:30.1.1-jre .

Here’s how the dependencies look in the build script.

Inspecting the compile and runtime classpaths shows us they’re identical with the following transitive dependencies.

Guava’s pulling in a lot of extra stuff! You’ll see later how to generate such a dependency graph yourself.

Say we just wanted to use a tiny subset of Guava, like the endlessly helpful ImmutableMap.of(K k1, V v1) method. If we wanted to we could exclude, for example, the findbugs dependency. Nothing against findbugs, but it seems particularly unnecessary!

Here’s how the exclude syntax looks in the Groovy build.gradle.

And in the Kotlin build.gradle.kts.

Within the closure we call exclude , passing:

• group the group of the artifact we want to exclude
• module the name of the artifact we want to exclude. This is equivalent to the name used to declare a dependency in Gradle.

It’s also entirely valid to pass only group or only module to match more generically. In the above example though, all combinations would result in the following updated transitive dependencies appearing on the classpaths.

This shows that we have one less dependency now. Findbugs is no more! 🔫

A caveat

The problem with adding an exclude per-dependency is that if another dependency also pulls in the excluded dependency, the exclude is ignored.

Using the above example, imagine that your well-meaning colleague decided to add another dependency jackson-datatype-guava.

Our build script dependencies now look like this.

On the surface this seems like an innocent change, but how do the classpaths look now?

Findbugs has sneaked its way back in again! 😩 That’s because jackson-datatype-guava depends on guava, meaning all of guava‘s transitive dependencies get pulled in again by Gradle.

This functionality can be helpful, since it means we have to think carefully how an exclusion applies to each dependency. If we’re confident that jackson-datatype-guava also doesn’t need findbugs, we can add another exclude rule to its dependency definition.

What if we have a dependency that we’re absolutely sure should never be included though? Can we end this game of whack-a-mole permanently?

Fortunately Gradle has another trick up it’s sleeve…

Option 2) per-configuration exclude rules

For the scenario where we’re confident that a transitive dependency should be excluded across all dependencies, Gradle offers exclusion rules against dependency configurations.

Let’s use example from earlier, where we declared an implementation dependency on guava, which transitively depended on findbugs. The per-configuration exclude rule for this in the Groovy build.gradle looks like this.

And in the Kotlin build.gradle.kts.

This time we pass a closure to the dependency configuration. Once again, within the closure we call the exclude method with a group and/or module .

Dependency configurations primer

A dependency configuration is just a grouping of dependencies. One such dependency configuration is implementation , which is a group of dependencies that appear on the runtime and compile classpaths. Other dependency configurations in Java projects include runtimeOnly and compileOnly . You can probably guess which classpaths they end up on. 😉

With only the per-configuration exclude rule applied, the compile and runtime classpath dependencies look like this.

Findbugs successfully squished again! Even if we add more dependencies which transitively depend on findbugs, it won’t appear on our classpaths.

An SLF4J + Spring Boot example

Consider a Spring Boot web application in which we want to use the SLF4J logging framework with a Log4j2 implementation. The project’s build script has the following dependencies.

The application contains the following Java code to get and use a logger, using only the SLF4J APIs.

When we run the application there are 2 problems:

1. at startup we get the SLF4J: Class path contains multiple SLF4J bindings warning mentioned at the start of this article
2. when we try to get an instance of org.slf4j.Logger we get the following exception.

Let’s deal with the exception first.

Here are the dependencies on the runtime classpath ( . indicates entries left out for clarity).

Highlighted above are the dependencies listed in the exception. log4j-to-slf4j according to its docs allows applications coded to the Log4j 2 API to be routed to SLF4J. We can safely exclude it as we won’t be calling the Log4j 2 APIs within our application, only the SLF4J APIs.

Here’s how the exclude rule looks.

This fixes the exception and now the logging framework can actually be used. But we still get this unfriendly warning on startup.

So it’s findings bindings for both log4j and logback. Let’s review the classpath dependencies again.

When using the SLF4J logging framework, we should only have one binding to a logging implementation on the runtime classpath. The highlighted line above shows that spring-boot-starter-logging is bringing in logback-classic, which we don’t need.

At this point rather than excluding logback-classic, we can actually exclude the whole spring-boot-starter-logging dependency in favour of spring-boot-starter-log4j2, so our exclude rule looks like this.

No warnings on startup now. 👍 This is also the same approach suggested by the Spring Boot Logging docs.

Exclude dependency from all configurations

Excluding the dependency from the implementation dependency configuration is enough to fix the Spring Boot + SLF4J error. In fact, this excludes the dependency from the compile, runtime, testCompile, and testRuntime classpaths!

But, Gradle offers a way to exclude dependencies from all dependency configurations. This could be helpful, for example, if you had the same dependency on the annotationProcessor path and wanted to exclude it.

Option 3) module replacement exclude alternative

One alternative suggested in the Spring Boot documentation is to replace a dependency rather than exclude it.

We can achieve this with the following entry in our Groovy build.gradle.

And here’s how that looks in the Kotlin build.gradle.kts.

This has the advantage of being more explicit than an exclude rule, as it ties the dependency to be excluded and the replacement together in one place. We can also pass a reason for the replacement to help with issue diagnosis.

Reviewing your project’s dependencies

You can check the dependencies in your own project using Gradle’s built-in dependencies task. Here’s how to review all dependencies across all dependency configurations:

Or to review the dependencies for a specific dependency configuration.

The most interesting dependency configurations to plug into the above command are probably compileClasspath and runtimeClasspath as they’re used by Gradle directly when compiling and running your application.

All the dependency graphs in this article were generated using this mechanism. For a full understanding of using the dependencies task, as well as other troubleshooting options available in Gradle, I highly recommend signing up to the Gradle Hero course.

Common pitfalls

If you’re having problems check out these common errors from my own experience and those of others online.

Trying to exclude with name instead of module

For reasons still unknown, when specifying an exclude in Gradle you pass a module key as opposed to a name key used when specifying a dependency. They represent the same value, the name/id of the artifact.

Here’s the INVALID syntax.

Which produces this error in the build.

Easily fixed by replacing name with module .

Trying to exclude using string instead of map notation

Another difference between using the exclude method and declaring dependencies is that you can’t pass a string instead of a map.

Here’s an example of an INVALID build script exclusion rule.

Which results in this error.

This is fixed by replacing the string with a map.

Comparison of Gradle’s dependency exclude approaches

Approach name	Description	When to use	How to specify
Per-dependency exclude	Add an exclude rule to a specific dependency	You want to exclude a transitive dependency from one specific dependency, but not necessarily if it gets pulled in by another	Map of group and/or module
Per-configuration exclude	Add an exclude rule to an entire dependency configuration or all dependency configurations	You’re sure you want to exclude the transitive dependency across all dependencies	Map of group and/or module
Module replacement	Replace one dependency with another	You know that whenever there’s one dependency, it should always be replaced with another	String of ‘ : ’

Resources

Try out for yourself the SLF4J + Spring Boot example from this article in this GitHub repository.

If you’ve come across a relevant scenario which hasn’t been mentioned then please leave a comment below to start a discussion.

Want to learn more about Gradle?
Check out the full selection of Gradle tutorials.

Related Posts

• 
• 
• 
• 

One thought on “ How to exclude Gradle dependencies ”

Please note that changes done with module replacement rules will not be reflected in a potentially published pom while excludes in dependencies and configurations are.
This has to be considered with downstream projects that consume those poms.
I think it depends on the use case which way you want or need to go.

Leave a Reply Cancel reply

Is your dev team productive?

Popular Gradle articles

Categories

• Agile (1)
• AWS (22)
• Build Automation (31)
• Business (2)
• Continuous integration (16)
• Docker (6)
• Documentation (1)
• General coding (11)
• Monitoring (8)

Talk to Tom!

If you want to improve your dev & devOps skills then I sincerely hope there’s something for you here.

If there’s nothing that tickles your tech-tastebuds, let me know what subjects you’d like to read about. I’d love to hear from you at tom@tomgregory.com.

If you want to keep in touch, feel free to connect on LinkedIn.

Источник