On November 6, security researchers of FoxGlove Security released five zero day exploits for WebSphere, WebLogic, JBoss, Jenkins and OpenNMS. These software products are used everywhere in enterprise organizations and with the published exploits remote malicious code can be unauthenticated executed.
The underlying vulnerability was known for years but it was vague and believed to be hard to exploit. That situation changed when Chris Frohoff (@frohoff) and Gabriel Lawrence (@gebl) published in January of an exploit generator for it in their talk on AppSecCali . Their generator didn’t get much attention on the internet. FoxGlove didn’t find the original (underlying) vulnerability, but they did find vulnerable products and wrote exploits for it. They call the vulnerability “the most underrated, underhyped vulnerability of “. When I read about FoxGlobe’s research on Twitter on November 8, , my guts told me immediately that this was something big.
Validating the security research
I needed to validate this research and so I tried out the exploits against a fully patched and hardened WebSphere server on November 9, . It worked! Woah! I could easily run arbitrary code on WebSphere servers. This is something really big! For organizations that have a large Java stack, it’ll be certainly a lot bigger than Heartbleed.
After validating the research I informed my hacker friends John de Kroon and Ger Schinkel to start investigating the vulnerability further. We exploited a WebLogic and Jenkins server and improved the published WebLogic exploit code. We also directly started developing vulnerability scan software to scan large computer networks for vulnerable servers and put our scan software on Github.
Escalating the vulnerability to enterprise organizations
We saw that a lot of enterprise organisations and critical infrastructure were vulnerable. These large organizations didn’t know about it yet and at the time that we found out on November 10, , not much attention was given to the zero day exploits on the internet. We informed the Dutch National Cyber Security Center and also shared our intelligence with security departments of quite of lot of enterprise organizations in The Netherlands and financial institutions world wide. That had effect and security departments were starting their investigations after our notification.
As the underlying vulnerability is a complex one, some security departments didn’t give the vulnerability the attention and priority that it deserved. Hopefully that will change with this post.
Via this article I want to share our knowledge about scanning large enterprise networks for vulnerable software components. I’ll update the article continuously with new information about the situation (last update was November 21, ). Security departments can use the information in this article to get up to speed and in control of the vulnerability. This article functions as a security test plan and we’re also releasing a vulnerability scanner so you can check your own networks for vulnerable WebSphere, WebLogic, JBoss and Jenkins servers.
The Java deserialization vulnerability
The zero day exploits that were released for five different products on November 6, , all exploit the same underlying vulnerability in the Apache Commons Collection library (VU#576313) that occurs when deserializing Java serialized objects. Custom Java applications using vulnerable versions of third-party libraries, such as Apache Commons Collection, Groovy, or Spring, may also be vulnerable.
This means that a lot more products are also vulnerable for the Java deserialization vulnerability. The vulnerability has the potential to affect almost all applications written in Java (!). When you search on Github for applications that use Apache Commons Collection then you’ll find more than 1,300 applications.
The security problems with Java deserialization was already known for quite some time:
- Pierre Ernst wrote in 2013: “When Java serialization is used to exchange information between a client and a server, attackers can try to replace the legitimate serialized stream with malicious data. [..] Java serialization enables developers to save a Java object to a binary format so that it can be persisted to a file or transmitted over a network. Remote Method Invocation (RMI) uses serialization as a communication medium between a client and a server. Several security problems can arise when a service accepts binary data from a client and deserializes the input to construct a Java instance. [..] [an] attacker [can] cause a denial-of-service condition or €” in extreme cases €” to inject arbitrary code. You might believe that this kind of attack is impossible [..]”. The extreme case Pierre Ernst referred to was CVE-2012-4858.
- Those “extreme impossible cases” are not that extreme and impossible anymore because in January of a generator called Ysoserial was released to easily exploit applications that made use of Java deserialization functionality that was available in the Apache Commons Collection, Groovy and the Spring Java library. This exploit generator didn’t get any attention in the security community.
- A Java deserialization vulnerability was found by Matthias Kaiser in Atlassian Bamboo (CVE–6576) and an advisory was published on October 21, .
- The Ysoserial exploit generator was used by FoxGlove to create the five zero day exploits that were released on November 6, . This certainly got the attention of the security community now.
- Jonas released an exploit for Atlassian Bamboo on November 20, .
Vulnerable internet-facing servers
When you want to find vulnerable servers on the internet, then the Shodan search engine is a great place to start. When you:
- Search for WebLogic servers, you’ll find 931 servers:
- Searching for WebSphere will result in 6,079 servers:
- Searching for JBoss will result in 52,575 servers:
- Searching for Jenkins will result in 18,939 servers:
- Searching for OpenNMS will result in 177 servers:
Via a quick Shodan search, we already found 78,701 potentially vulnerable servers.
Security research from ZoomEye
According to ZoomEye, the number of vulnerable internet-facing:
- JBoss servers is 8,398 out of 45,198 detected JBoss servers.
- Jenkins servers is 1,811 out of 15,000 detected Jenkins servers:
- WebLogic servers is 1,966 out of 2 million detected IP addresses that have port 7001 open:
This means that currently it’s possible to easily hack 12,175 servers. This number is probably is just the tip of the iceberg of vulnerable internet-facing servers.
Impact and security risk rating
According to our research it seems that the vulnerability mostly occurs on internal networks due to (management) interfaces that are most of the time not internet-facing. If a vulnerable interface can be reached, then it’s game over: an attacker can execute arbitrary malicious code on the vulnerable server.
We observed that the published WebSphere, WebLogic and Jenkins exploits are very reliable and haven’t resulted in denial of service of servers.
The injected malicious code will be executed under the privileges of the operating system user that the vulnerable service runs under. In hardened services this will not be a root or administrator account. Within WebSphere an attacker will be ‘root’ under all applications that WebSphere runs: all applications can get compromised. An attacker will also have access to database connections that WebSphere Java applications use. An attacker can thus also compromise linked databases. This all makes it a critical security vulnerability.
Via a privilege escalation vulnerability that the server might have, an attacker can gain root or administrator rights on the server, which results in full compromise of the server.
Attacks possible on internal network launched via internet
Even through most vulnerable servers can only be reached via an internal network, we know ways to exploit these servers via the internet. For now, we won’t disclose that attack scenario, as most companies haven’t patched their systems yet.
Metasploit exploit for JBoss servers
On November 18, , security researcher Nick Fox released an exploit for the Metasploit framework that targets vulnerable JBoss servers:
This Metasploit exploit makes it much easier to exploit JBoss servers in comparison with the already released JBoss exploit. Via this new polished exploit JBoss servers now face also the threat of script kiddies.
Security mitigation phases
The security mitigation plan should be based around the following phases:
- Inventory of vulnerable assets and set-up security monitoring alerts.
- Damage control (short term).
- Prevention of Java deserialization vulnerabilities (long term).
Phase 1: Inventory of vulnerable assets and set-up security monitoring alerts
First things first. A threat analysis should be made for all known vulnerable products, after that your custom written software should be reviewed and finish by finding yet unknown deserialization vulnerabilities in other products.
Phase 1.1: Vulnerability scan plan for known vulnerable products
- An organization should first find out if they have IBM WebSphere, Oracle WebLogic, JBoss, Jenkins, OpenNMS and Atlassian Bamboo servers running in their network.
- If this is the case, then it should be validated if the released exploits work against these servers:
- It’s important to test if existing security controls are blocking or minimizing the attack surface, such as firewalls and intrusion prevention systems.
- It could be the case that the version of the software that is used is not vulnerable for the exploit. In other words: find out if your organization is running the latest version of the software (I hope so, otherwise you’ll probably have a lot of security updates that are missing).
- But perhaps the most important one: have hard proof in hand that your organization is vulnerable for the exploit(s). This is very important information to convince higher management to form a security incident response team to start controlling the vulnerability and have the mandate to use human resources with high priority of your organization.
- Now that you found out which of the six products are running in your network and which of those are exploitable (probably all of them), this issue should be escalated to management so you’ll have the time and team to work on it.
- The next action is to make an inventory of all vulnerable servers located in your network:
- Hopefully your organization has Configuration Management Databases (CMDB) were this information is stored in. I specifically name the existence of multiple CMDBs, because most IT departments have their owns lists of servers that are not (yet) consolidated into the main CMDB of the company. If you have found a CMDB, hopefully it’s up-to-date. It probably won’t be the case, because IT assets will get attached, detached and relocated on a network all the time. Administration is always outdated.
- Contact should be made with the IT departments that deliver and maintain the vulnerable software packages and servers. The IT department that initially delivers a software package/server can be different from the department that maintains it. Ask these departments to log in on all their servers to get a list of vulnerable components from a white box perspective on their landscape. It’s very important to get to know on which TCP and/or UDP ports the vulnerable services are listening, so let them run Netstat on the server. We need this information to validate the vulnerability from a black box scan perspective and to find other vulnerable servers on the network that the IT department is not aware of.
- Apart from the administration retrieved, you should fire up port scanner Nmap and start scanning the internal network to find servers that are not on the official lists (you’ll be surprised!). It’s important to limit the amount of ports that you’ll scan with Nmap in order to minimize the risk of denial of servicing production machines and network components. You’ll not be the first security analyst who brings down a network firewall and cause management to panic because production servers are not reachable anymore. Limit the port list to all default ports (see further in this article for a list) that the vulnerable software components use and complete it with the custom ports that your organization uses for these components.
- Combine above gathered lists and start removing duplicates. Convert fully qualified domain names (FQDNs) to IP addresses (or reverse).
- Now that you have a list of potential vulnerable servers, you need to confirm the vulnerability on these server from a black box scan perspective. First scan the internet-facing side, and then the internal network.
Indicators of the vulnerability
Indicators of the Java deserializing vulnerability:
- When the hexadecimal string AC ED 00 05 and in base64 encoded format rO0 is visible in HTTP requests or responses (parameters, ViewStates or cookies).
- The vulnerable Java library is commons-collections*.jar. This is a file that exists on the file system in most cases (fat .jar files are an exception to this rule). The star functions as a wildcard, as it can contain various version numbers. This library is responsible for the code execution possibility when Java objects are deserialized. Existence of this library means that the software that uses it has the possibility of the deserialization vulnerability. The following steps should be followed to analyse and determine the security risk further:
- Application Java code should be analyzed if untrusted Java objects are deserialized. If this is the case, then the application is vulnerable.
- To determine the risk value associated with the vulnerability, the attack surface should be determined: how easy is it for an attacker to exploit the vulnerability?
- Everywhere where Java objects are deserialized. Some examples:
- Java Remote Method Invocation (Java RMI). Java RMI is a Java API that performs the object-oriented equivalent of remote procedure calls (RPC), with support for direct transfer of serialized Java classes and distributed garbage collection.
- Java Management Extensions (JMX). This is a RMI management protocol.
- Remote EJB.
- A Java method call to (source):
- The readObject() method, typically used by developers when standard serialization cannot be used, such as when a transient member needs to be set.
- The readResolve() method, typically used to serialize singleton instances.
- The readExternal() method, used for externalizable objects.
- The value Content-type: application/x-java-serialized-object in a HTTP response.
Naming the vulnerability
Because the hexadecimal string starts with AC ED, we started calling internally the deserialization vulnerability ACED. Easier to pronounce than “the Java deserialization vulnerability”.
On Twitter someone also called the vulnerability DeJa Vu. This abbreviation stands for Deserialization of Java objects Vulnerability.
Default ports of vulnerable services
The table below lists the default ports on which the vulnerability can be exploited:
|IBM WebSphere||5005: SOAP over HTTPS
8880: SOAP over HTTPS
|Oracle WebLogic||7001: binary T3 protocol
7002: binary T3S secure protocol
16200: binary T3 protocol
After connecting to port 8080 or 9080, Jenkins will give the client a ‘random’ port back to connect to: binary communication.
|Atlassian Bamboo||Not researched so far.|
Java serialized objects could be used anywhere on any port. TCP would be used in 99% of the cases, but also UDP could be used in rare cases. Java objects are frequently communicated via HTTP, HTTPS and SOAP. Most web servers use TCP port 80 and 443, but also 81, 444, 8080, 8443, 8888, 9000 and 9443 are frequently used as (alternative) web server ports.
Common used ports according to Shodan
It’s wise to also scan for vulnerable servers on common used ports. According to Shodan the most used ports are:
|IBM WebSphere||80, 443, 8880, 9080, 9443|
|Oracle WebLogic||80, 443, 8080, 8086|
|Jenkins||80, 443, 8080, 8443|
|JBoss||80, 443, 8080, 8443|
|OpenNMS||80, 443, 8080,|
|Atlassian Bamboo||80, 443, 8080, 8443|
Developing a WebSphere, WebLogic, JBoss and Jenkins vulnerability scanner
We had a huge list of servers that we wanted to check and unfortunately there was no vulnerability scan software available that had signatures to test for the Java deserialization vulnerability. So we created our own scanner, dubbed it SerializeKiller and put it on Github on November 11, . The scanner is written in Python can check for vulnerable WebSphere, WebLogic, JBoss and Jenkins servers.
Our scanner can scan 1,000 servers in less than two minutes (via multi threading) and is non intrusive. No exploits will be fired and only the default installation ports will be scanned. You can easily add extra ports by modifying the code.
To run our scanner, Python, CURL and Nmap should be installed. After that the scanner can be called via command line:
The contents of targetfile.txt:
The contents of serializekiller.py can be found on Github.
Other vulnerability scanners
Burp Suite extension SuperSerial
On November 9, , DirectDefense released a Burp Suite extension called SuperSerial that can detect via black box passive content sniffing Java deserialization vulnerabilities:
I haven’t tested this extension yet, but it looks like it can only detect vulnerable JBoss servers. This free unofficial extension can only be loaded in the commercial edition of vulnerability scanner and HTTP proxy Burp Suite.
Nessus plugin #86898 detects vulnerable Jenkins servers
On November 17, , Tenable Nessus (commercial vulnerability scanner) released plug-in #86898 that can detect vulnerable Jenkins servers.
Phase 1.2: Vulnerability scan plan for your own custom written Java code
If custom Java code is developed for your organisation, then all Java code should be reviewed if object serialization is used. As this can take up quite some time, you should prioritize on your internet-facing applications, and then first review your most important applications based on their Confidentiality Integrity Availability (CIA) rating.
Apart from white box code analysis, you should also search for Java serialized objects from a black box perspective. Crawl your web sites via a HTTP web proxy such as Burp Suite (commercial) or OWASP ZAP (open source) and search for the hexadecimal string ac ed 00 05 and in base64 encoded format rO0.
Fortify Static Source Code Analyzer
The (commercial) static source code analyzer (SCA) of HP Fortify can detect the usage of Java serialization. This white box detection is however very simple (compare it to a simple grep search) and doesn’t take into account possible attack surface. All Fortify SCA findings should be manually verified, because the rudimentary signature is prone to quite a lot of false positives.
Detailed information about relevant rules:
- Missing SecurityManager Check: Serializable
- J2EE Bad Practices: Non-Serializable Object Stored in Session
On November 13, , Sonatype CLM released detection signatures for vulnerable Apache Commons Collection libraries:
Sonatype CLM (commercial) scans source code repositories for vulnerable libraries that are included in custom written software.
Phase 1.3: Vulnerability scan plan for possible third party vulnerable products
System administrators should log into the servers they maintain and scan file systems for existence of the Apache Commons Collection library. A list of all the hits should be collected:
root@server:/opt/IBM# find . -iname "*commons*collection*"
Next thing for them to do is to check which network ports are opened by Java:
root@server:/opt/IBM# lsof -i -P | grep LISTEN | grep java
Apart from servers, work stations can also be vulnerable. Analyzing servers should have a higher priority than clients.
A security analyst should then process the results of the collected data to research new Java based products that are vulnerable for the deserialization vulnerability. Key is to check if serialized Java objects are visible in attack surface in the products that are to be researched.
Network based intrusion detection systems can help with the inventory
Signatures for Java deserialization can be created by yourself via the vulnerability indicators above. The most simple indicator is to search network traffic for the hexadecimal string AC ED 00 05 and in base64 encoded format rO0. We found out that searching for JRMI also gives interesting results (possible new zero days).
Detection of vulnerable services via an IDS is particularly helpful in a complex network to get the inventory of known and unknown vulnerable services as complete as technically and organizationally possible.
Emerging Threats has rules
On November 9, , Emerging Threats released in their daily ruleset update the following (commercial) rules for Snort (open source IDS) and Suricata intrusion detection systems:
2814811 - ETPRO EXPLOIT Serialized Java Object Calling Common Collection Function (exploit.rules) 2814812 - ETPRO EXPLOIT Serialized Java Object Calling Common Collection Function (exploit.rules) 2814813 - ETPRO EXPLOIT Serialized Java Object Generated by ysoserial (exploit.rules) 2814814 - ETPRO EXPLOIT Serialized Groovy Java Object Generated by ysoserial (exploit.rules) 2814815 - ETPRO EXPLOIT Serialized Spring Java Object Generated by ysoserial (exploit.rules)
I’ve seen the rules in action and they are of great support in the detection process of vulnerable applications.
On November 18, , Emerging Threats decided to freely release the Snort rules (great decision!), so here they are:
alert tcp any any -> $HOME_NET any (msg:" ETPRO EXPLOIT Serialized Java Object Calling Common Collection Function"; flow:to_server,established; content:"rO0ABXNyA"; content:"jb21tb25zLmNvbGxlY3Rpb25z"; fast_pattern; distance:0; reference:url,github.com/foxglovesec/JavaUnserializeExploits; classtype:misc-activity; sid:2814811; rev:1;) alert tcp any any -> $HOME_NET any (msg:" ETPRO EXPLOIT Serialized Java Object Calling Common Collection Function"; flow:to_server,established; content:"|ac ed 00 05 73 72 00|"; fast_pattern; content:"commons.collections"; nocase; distance:0; reference:url,github.com/foxglovesec/JavaUnserializeExploits; classtype:misc-activity; sid:2814812; rev:1;) alert tcp any any -> $HOME_NET any (msg:" ETPRO EXPLOIT Serialized Java Object Generated by ysoserial"; flow:to_server,established; content:"|ac ed 00 05 73 72 00|"; fast_pattern; content:"java/io/Serializable"; nocase; distance:0; content:"ysoserial/payloads/util/Gadgets"; reference:url,github.com/foxglovesec/JavaUnserializeExploits; classtype:misc-activity; sid:2814813; rev:1;) alert tcp any any -> $HOME_NET any (msg:" ETPRO EXPLOIT Serialized Groovy Java Object Generated by ysoserial"; flow:to_server,established; content:"|ac ed 00 05 73 72 00|"; fast_pattern; content:"org.codehaus.groovy.runtime.ConversionHandler"; nocase; distance:0; content:"ysoserial/payloads/util/Gadgets"; reference:url,github.com/foxglovesec/JavaUnserializeExploits; classtype:misc-activity; sid:2814814; rev:1;) alert tcp any any -> $HOME_NET any (msg:" ETPRO EXPLOIT Serialized Spring Java Object Generated by ysoserial"; flow:to_server,established; content:"|ac ed 00 05 73 72 00|"; fast_pattern; content:"org.springframework.core.SerializableTypeWrapper"; nocase; distance:0; content:"ysoserial/payloads/util/Gadgets"; reference:url,github.com/foxglovesec/JavaUnserializeExploits; classtype:misc-activity; sid:2814815; rev:1;)
HP TippingPoint has also rules
TippingPoint (commercial IDS) has released detection rules on November 18, . I’ve seen those in action and first impression is they work great!
Phase 2: Damage control (short term)
Java deserialization is broken in the design and in its core it’s not an implementation vulnerability. The vulnerability has not been patched in the Apache Commons Collection library and that probably won’t be the case any time soon.
Just for the record: Apache released a patch (in their nightly build) which added a kill switch for the deserialization functionality, but I don’t call that a real patch. If your application is using Java serialized objects, then you won’t get safe if you just install Apache’s patch.
Solution for custom written Java software
Using Java object deserialization functionality is deprecated and should not be used. If it’s nonetheless used, then the exchanged serialized objects should be protected via strong encryption so a malicious person won’t be able to view and modify the Java object and inject malicious exploit code. To further protect the exchange of Java objects, Java object white listing should be implemented.
Will Sargent has wrote an extensive web log post detailing all the ins and outs regarding deserialization mitigation advice. I strongly recommend that you read that article for patching custom build Java software.
Solution for IBM WebSphere
WebSphere has released a patch. We’ve confirmed that the exploit won’t work anymore after applying this patch. WebSphere should be restarted after applying the patch.
Alternative for the official patch is to close the management interface of WebSphere on port 8880 and/or 5005 via iptables on the WebSphere server itself. This should have minimal impact. When iptables is used, the server doesn’t have to be rebooted in order to close the port; this means no down time for the server and thus this mitigation shouldn’t have customer impact.
Solution for Oracle WebLogic
WebLogic is working on a fix, but hasn’t released a patch yet. The patch is planned to be released in week 48. Please note that it’s expected that the patch will only be available for the latest version and will not be backported.
The Oracle WebLogic vulnerability is tracked under CVE–4852. Oracle has released mitigation advice under MOS Note 2076338.1. The advice was put however behind a paywalled login screen. The Oracle mitigation advice is just to block the WebLogic port (that uses the T3/T3S protocol) on the firewall.
Solution for Jenkins
Jenkins has released this week a patch for the vulnerability via version 1.638 and 1.625.2. We’ve confirmed that this patch results in the fact that the Java deserialization exploit won’t work anymore.
Solution for JBoss
We’ve not researched the solution for this product so far. You can always use iptables to block remote access to vulnerable TCP port 8080.
Solution for OpenNMS
We’ve not researched the solution for this product so far. You can always use iptables to block remote access to vulnerable TCP port 1099.
Solution for Atlassian Bamboo
Atlassian has released a patch and you should upgrade to the latest version.
Solution via WAF or IPS?
Currently no web application firewalls (WAF) and intrusion prevention systems (IPS) are known that can mitigate exploits against the Java deserialization vulnerability.
Update November 21, : I’ve been told HP TippingPoint has IPS signatures now. I’ll validate and test that.
Phase 3: Prevention of Java deserialization vulnerabilities (long term)
The long term solution would be to add an entry in the Java development security policy to deprecate the usage of serialized Java objects. If serialized Java objects are used, these should be protected via strong encryption and object white listing should be implemented.
Check developed Java software via security testing and code reviewing before deploying it to production.
A chronological overview on all the dates and events named in this article:
|January 15, 2013||Warning||Pierre Ernst warns that Java deserialization is vulnerable|
|January 28,||Exploit generator||Chris Frohoff and Gabriel Lawrence released their Java deserialization exploit generator Ysoserial|
|November 6,||Exploits||FoxGlobe released 5 zero day exploits for WebSphere, WebLogic, JBoss, Jenkins and OpenNMS based on Ysoserial|
|November 9,||Scanner||DirectDefense released a Burp Suite extension called SuperSerial that detects vulnerable JBoss servers|
|November 9,||IDS signature||Emerging Threats released commercial IDS signatures for Snort and Suricata|
|November 11,||Scanner||We’ve published our SerializeKiller scanner on Github|
|November 13,||Scanner||Sonatype CLM released detection signatures for vulnerable Apache Commons Collection libraries|
|November 14,||Research||This web log article was published|
|November 17,||Scanner||Tenable Nessus released plug-in #86898 that can detect vulnerable Jenkins servers|
|November 18,||Exploit||Security researcher Nick Fox released an exploit for the Metasploit framework that targets vulnerable JBoss servers|
|November 18,||IDS signature||Emerging Threats decided to freely release the IDS rules|
|November 18,||IDS signature||TippingPoint has released IDS signatures|
|November 20,||Exploit||Jonas released an exploit for Atlassian Bamboo|
Do you have suggestions or improvements?
Please leave a comment below or e-mail me at sijmen[at]ruwhof.net if you have improvements for this article. This security research is work in progress. Let’s work together and get secure again.
Last update of article is November 21, .
Sites that link to this research: