Remediating the Log4Shell CVE in a legacy closed-source Spring Boot application
You certainly may have heard about the recent Log4Shell Remote Code Execution (RCE) vulnerability targeting Log4J, one of the most used logging libraries in the Java ecosystem.
When analyzing the impact of this vulnerability in several of our systems at work, I came across one particular application: self-hosted, legacy, closed-source and using one of the affected Log4J 2.x version; this kind of applications that have been running for months, and which no one wanted to touch, because it works. Even when it had clear evidence of application issues (like memory leaks), a simple Process reboot was the quick and dirty way to relieve it again whenever people ever had to touch it. It is a public-facing Spring Boot-based application, configured with Log4J2 and running in production, so highly at risk with regard to this attack.
Unfortunately, as it can be the case sometimes with accumulated technical debt within organizations, it had never been a priority to update this application. And in the context of this vulnerability, upgrading this service to see if a most recent version of Log4J was used could not be an option, due to a way too high risk of functional breaking changes.
Identifying the version of Log4J
As depicted above, this application is closed-source. The only resource at hand was the Spring Boot self-contained (or executable) JAR. Without access to the source code, it is therefore not quite possible to display a tree of all its dependencies (including direct and transitive ones).
Under the cover however, JAR files are simply ZIP-compressed files that contain a collection of JVM Bytecode class files along with other resources. Spring Boot goes beyond by also nesting the application dependencies (also JAR files) into the application JAR file. Find out more in the official and detailed documentation.
As a consequence, this makes it easy to inspect a self-contained JAR in a simple yet effective manner: using either the zip or jar commands, like so:
jar tf /path/to/jar | grep -i log4j
zip -sf /path/to/jar | grep -i log4j
This allows to quickly list all the files in the archive, without actually decompressing it.
❯ jar tf demo-0.0.1-SNAPSHOT.jar | grep -i log4j BOOT-INF/lib/spring-boot-starter-log4j2-2.0.7.RELEASE.jar BOOT-INF/lib/log4j-slf4j-impl-2.10.0.jar BOOT-INF/lib/log4j-api-2.10.0.jar BOOT-INF/lib/log4j-core-2.10.0.jar BOOT-INF/lib/log4j-jul-2.10.0.jar
The jar command is part of any JDK installation. If you do not have the jar command installed, maybe you are running a JRE instead (like in pre Java 11 environnments). In this case, rather than installing a complete JDK, you can use tools like fastjar, which is a standalone program that provides the same features as jar.
Note that since Java 11, only JDKs are distributed by default, but users may create their custom runtime distributions using tools like jlink.
Mitigating the vulnerability
The simplest remediation recommended at the time was to set the "log4j2.formatMsgNoLookups" JVM Property or the "LOG4J_FORMAT_MSG_NO_LOOKUPS" environment variable to "true".
This was quick to set, with no modification to the application JAR, apart from rebooting the running JVM Process. However, this proved to be insufficient, because it reportedly only limited exposure while leaving some attack vectors open. Fortunately, the Apache Foundation was therefore very prompt to propose other mitigation measures, and we had to explicitly remove the org.apache.logging.log4j.core.lookup.JndiLookup class from the vulnerable Log4J 2 JAR.
In a Spring Boot self-contained JAR however, the log4j-core-*.jar files certainly are nested in the application JAR. Therefore, we need to extract the main application JAR first, patch the log4j-core JAR and repackage the former.
Extracting the main application JAR
This can be done either with the jar command, like below, or even using the zip command. We will use jar, which should be available as part of your JDK installation. More on why jar is required in a moment.
jar xf /path/to/springboot_main_app.jar
Patching the vulnerable log4j-core JAR
Once extracted, the log4j-core will generally live under the BOOT-INF/lib folder, or WEB-INF/lib or WEB-INF/lib-provided, depending on the project packaging format (JAR or WAR). Anyway, we can patch the Log4J JAR right away with the mitigation measure recommended:
zip -q -d BOOT-INF/lib/log4j-core-*.jar \ org/apache/logging/log4j/core/lookup/JndiLookup.class
Repackaging the main application JAR
You might be tempted to repackage the main application JAR using the standard zip command, like so:
zip -r /path/to/springboot_main_app.jar ./*
However, doing so will likely prevent your Spring Boot application from starting again, with error lines similar to the following ones if you attempt to restart the process:
Exception in thread "main" java.lang.IllegalStateException: Failed to get nested archive for entry BOOT-INF/lib/spring-expression-5.0.11.RELEASE.jar at org.springframework.boot.loader.archive.JarFileArchive.getNestedArchive(JarFileArchive.java:108) at org.springframework.boot.loader.archive.JarFileArchive.getNestedArchives(JarFileArchive.java:86) at org.springframework.boot.loader.ExecutableArchiveLauncher.getClassPathArchives(ExecutableArchiveLauncher.java:70) at org.springframework.boot.loader.Launcher.launch(Launcher.java:49) at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) Caused by: java.io.IOException: Unable to open nested jar file 'BOOT-INF/lib/spring-expression-5.0.11.RELEASE.jar' at org.springframework.boot.loader.jar.JarFile.getNestedJarFile(JarFile.java:254) at org.springframework.boot.loader.jar.JarFile.getNestedJarFile(JarFile.java:239) at org.springframework.boot.loader.archive.JarFileArchive.getNestedArchive(JarFileArchive.java:103) ... 4 more Caused by: java.lang.IllegalStateException: Unable to open nested entry 'BOOT-INF/lib/spring-expression-5.0.11.RELEASE.jar'. It has been compressed and nested jar files must be stored without compression. Please check the mechanism used to create your executable jar file at org.springframework.boot.loader.jar.JarFile.createJarFileFromFileEntry(JarFile.java:282) at org.springframework.boot.loader.jar.JarFile.createJarFileFromEntry(JarFile.java:262) at org.springframework.boot.loader.jar.JarFile.getNestedJarFile(JarFile.java:250) ... 6 more
As you might guess from the root cause message, the application can no longer start because the nested dependencies have been compressed and the Spring Boot loader expects them to be nested without compression at all:
Unable to open nested entry 'BOOT-INF/lib/...'. It has been compressed and nested jar files must be stored without compression. Please check the mechanism used to create your executable jar file
This is where the jar command can come to the rescue to fix this startup issue:
jar u0f /path/to/springboot_main_app.jar BOOT-INF/lib/log4j-core-*.jar
This updates only the log4j-core-* files inside the main application JAR, but without compressing them again. This is the purpose of the "0" option, which means "no compression". The "u" option is for "update", and the "f" for the JAR file being updated.
Now the main application should be up and running happily again.
This is how we patched this legacy application to protect it against the critical Log4Shell vulnerability.
At first sight, this seemed a pretty straightforward mitigation task, but ended up causing some application startup trouble, which fortunately got sorted out using the jar JDK tool.
For now, this vulnerability appears to be remediated, at least until we hear more recent news about it again or until the next vulnerabilities with a similar severity are disclosed.
Jokes aside, this is typically why I love and contribute to open-source, just for having access to the source code in case such kind of issues need to be fixed, or for maintenance purposes.
I hope this article will be helpful to people struggling with such kind of applications, so they can quickly mitigate this vulnerability in their legacy Spring Boot applications.
Again, thanks to all the people working on such a sensitive library like Log4J, and for being prompt and responsive in dealing with this case.
As usual, feel free to share your thoughts in the comments below.
Merry Christmas and Happy New Year 2022 !