You certainly may have heard about the recent Log4Shell Remote Code Execution (RCE) vulnerability targeting Log4J, one of the most used logging libraries in the Java ecosystem.

When analyzing the impact of this vulnerability in several of our systems at work, I came across one particular application: self-hosted, legacy, closed-source and using one of the affected Log4J 2.x version; this kind of applications that have been running for months, and which no one wanted to touch, because it works. Even when it had clear evidence of application issues (like memory leaks), a simple Process reboot was the quick and dirty way to relieve it again whenever people ever had to touch it. It is a public-facing Spring Boot-based application, configured with Log4J2 and running in production, so highly at risk with regard to this attack.

Unfortunately, as it can be the case sometimes with accumulated technical debt within organizations, it had never been a priority to update this application. And in the context of this vulnerability, upgrading this service to see if a most recent version of Log4J was used could not be an option, due to a way too high risk of functional breaking changes.

Identifying the version of Log4J

As depicted above, this application is closed-source. The only resource at hand was the Spring Boot self-contained (or executable) JAR. Without access to the source code, it is therefore not quite possible to display a tree of all its dependencies (including direct and transitive ones).

Under the cover however, JAR files are simply ZIP-compressed files that contain a collection of JVM Bytecode class files along with other resources. Spring Boot goes beyond by also nesting the application dependencies (also JAR files) into the application JAR file. Find out more in the official and detailed documentation.

As a consequence, this makes it easy to inspect a self-contained JAR in a simple yet effective manner: using either the zip or jar commands, like so:

1

jar tf /path/to/jar | grep -i log4j

Or:

1

zip -sf /path/to/jar | grep -i log4j

This allows to quickly list all the files in the archive, without actually decompressing it.

For example:

1
2
3
4
5
6

❯ jar tf demo-0.0.1-SNAPSHOT.jar | grep -i log4j
BOOT-INF/lib/spring-boot-starter-log4j2-2.0.7.RELEASE.jar
BOOT-INF/lib/log4j-slf4j-impl-2.10.0.jar
BOOT-INF/lib/log4j-api-2.10.0.jar
BOOT-INF/lib/log4j-core-2.10.0.jar
BOOT-INF/lib/log4j-jul-2.10.0.jar

The jar command is part of any JDK installation. If you do not have the jar command installed, maybe you are running a JRE instead (like in pre Java 11 environnments). In this case, rather than installing a complete JDK, you can use tools like fastjar, which is a standalone program that provides the same features as jar.

Note that since Java 11, only JDKs are distributed by default, but users may create their custom runtime distributions using tools like jlink.

Mitigating the vulnerability

The simplest remediation recommended at the time was to set the “log4j2.formatMsgNoLookups” JVM Property or the “LOG4J_FORMAT_MSG_NO_LOOKUPS” environment variable to “true”.

This was quick to set, with no modification to the application JAR, apart from rebooting the running JVM Process. However, this proved to be insufficient, because it reportedly only limited exposure while leaving some attack vectors open. Fortunately, the Apache Foundation was therefore very prompt to propose other mitigation measures, and we had to explicitly remove the org.apache.logging.log4j.core.lookup.JndiLookup class from the vulnerable Log4J 2 JAR.

In a Spring Boot self-contained JAR however, the log4j-core-*.jar files certainly are nested in the application JAR. Therefore, we need to extract the main application JAR first, patch the log4j-core JAR and repackage the former.

Extracting the main application JAR

This can be done either with the jar command, like below, or even using the zip command. We will use jar, which should be available as part of your JDK installation. More on why jar is required in a moment.

1

jar xf /path/to/springboot_main_app.jar

Patching the vulnerable log4j-core JAR

Once extracted, the log4j-core will generally live under the BOOT-INF/lib folder, or WEB-INF/lib or WEB-INF/lib-provided, depending on the project packaging format (JAR or WAR). Anyway, we can patch the Log4J JAR right away with the mitigation measure recommended:

1
2

zip -q -d BOOT-INF/lib/log4j-core-*.jar \
  org/apache/logging/log4j/core/lookup/JndiLookup.class

Repackaging the main application JAR

You might be tempted to repackage the main application JAR using the standard zip command, like so:

1

zip -r /path/to/springboot_main_app.jar ./*

However, doing so will likely prevent your Spring Boot application from starting again, with error lines similar to the following ones if you attempt to restart the process:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

Exception in thread "main" java.lang.IllegalStateException: Failed to get nested archive for entry BOOT-INF/lib/spring-expression-5.0.11.RELEASE.jar
        at org.springframework.boot.loader.archive.JarFileArchive.getNestedArchive(JarFileArchive.java:108)
        at org.springframework.boot.loader.archive.JarFileArchive.getNestedArchives(JarFileArchive.java:86)
        at org.springframework.boot.loader.ExecutableArchiveLauncher.getClassPathArchives(ExecutableArchiveLauncher.java:70)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:49)
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)
Caused by: java.io.IOException: Unable to open nested jar file 'BOOT-INF/lib/spring-expression-5.0.11.RELEASE.jar'
        at org.springframework.boot.loader.jar.JarFile.getNestedJarFile(JarFile.java:254)
        at org.springframework.boot.loader.jar.JarFile.getNestedJarFile(JarFile.java:239)
        at org.springframework.boot.loader.archive.JarFileArchive.getNestedArchive(JarFileArchive.java:103)
        ... 4 more
Caused by: java.lang.IllegalStateException: Unable to open nested entry 'BOOT-INF/lib/spring-expression-5.0.11.RELEASE.jar'. It has been compressed and nested jar files must be stored without compression. Please check the mechanism used to create your executable jar file
        at org.springframework.boot.loader.jar.JarFile.createJarFileFromFileEntry(JarFile.java:282)
        at org.springframework.boot.loader.jar.JarFile.createJarFileFromEntry(JarFile.java:262)
        at org.springframework.boot.loader.jar.JarFile.getNestedJarFile(JarFile.java:250)
        ... 6 more

As you might guess from the root cause message, the application can no longer start because the nested dependencies have been compressed and the Spring Boot loader expects them to be nested without compression at all:

Unable to open nested entry ‘BOOT-INF/lib/…’. It has been compressed and nested jar files must be stored without compression. Please check the mechanism used to create your executable jar file

This is where the jar command can come to the rescue to fix this startup issue:

1

jar u0f /path/to/springboot_main_app.jar BOOT-INF/lib/log4j-core-*.jar

This updates only the log4j-core-* files inside the main application JAR, but without compressing them again. This is the purpose of the “0” option, which means “no compression”. The “u” option is for “update”, and the “f” for the JAR file being updated.

Now the main application should be up and running happily again.

Wrapping Up

This is how we patched this legacy application to protect it against the critical Log4Shell vulnerability.

At first sight, this seemed a pretty straightforward mitigation task, but ended up causing some application startup trouble, which fortunately got sorted out using the jar JDK tool.

For now, this vulnerability appears to be remediated, at least until we hear more recent news about it again or until the next vulnerabilities with a similar severity are disclosed.

Jokes aside, this is typically why I love and contribute to open-source, just for having access to the source code in case such kind of issues need to be fixed, or for maintenance purposes.

I hope this article will be helpful to people struggling with such kind of applications, so they can quickly mitigate this vulnerability in their legacy Spring Boot applications.

Again, thanks to all the people working on such a sensitive library like Log4J, and for being prompt and responsive in dealing with this case.

As usual, feel free to share your thoughts in the comments below.

Merry Christmas and Happy New Year 2022 !