It was relatively widely reported in the popular press as well as many technical sites that a Heartbleed exploitation "leaves no trace".
That of course is not true.
Packets almost always tell a detailed story of what has happened, including in the case of Heartbleed.
In this post, we will describe a technique for on-going monitoring of Heartbleed exploitations, but even more importantly, if you have a sufficiently large rolling buffer of packet data, many people will be able to use this technique to reach back in time to prior to the public disclosure of Heartbleed to check if an actual exploit seems to have occurred prior to patching vulnerable servers.
This this technique uses a BPF packet filter to automatically flag larger-than-typical TLS heartbeat responses from the server, and can be used with Wireshark and tcpdump as well as with the Riverbed AppResponse and Shark products. (AppResponse and Shark support many terabytes of stored packets, coupled with the ability to quickly analyze those packets; more Riverbed product specific hints to follow).
This is still an emerging threat, and this is a heuristic based approach that in theory can have both false positives and false negatives. However, given the urgency and severity of the threat, Riverbed is sharing this publicly in parallel to internally continuing to test and validate this.
Suggested improvements to this BPF filter are welcome.
This BPF filter currently only looks at traffic from port 443 (default HTTPS port). That section can be adjusted if HTTPS is running on other ports, as well as for other protocols (e.g., an email server running IMAP on port 143). The size threshold is 30 bytes. This can be adjusted upwards to reduce false positives, if needed.
-----------
Heartbleed BPF expression
-----------
tcp src port 443 and (tcp[((tcp[12] & 0xF0) >> 4 ) * 4] = 0x18) and (tcp[((tcp[12] & 0xF0) >> 4 ) * 4 + 1] = 0x03) and (tcp[((tcp[12] & 0xF0) >> 4 ) * 4 + 2] < 0x04) and ((ip[2:2] - 4 * (ip[0] & 0x0F) - 4 * ((tcp[12] & 0xF0) >> 4) > 30))
NOTE: this ignores VLANs for now. (VLAN-specific update to follow).
For clarity purposes, as well as to increase the ability of people to comment and suggest improvements, I will now try to break down the various sub-expressions of this specific BPF expression, and then finally at the bottom, for similar reasons, I have included some related "Simpler" BPF expressions that parse related information.
----------------------------
Breaking Down the Heartbleed BPF expression
----------------------------
# A response from server on port 443.
# This can be modified if server not using 443.
tcp src port 443
# This calculates the start of payload data beyond TCP.
tcp[((tcp[12] & 0xF0) >> 4 ) * 4
# Use that start-of-payload calculation to see if first payload byte is 0x18 (SSL Heartbeat message)
(tcp[((tcp[12] & 0xF0) >> 4 ) * 4] = 0x18)
# Use that start-of-payload calculation to see if second payload byte is 0x03 (SSL/TLS version major version 3)
(tcp[((tcp[12] & 0xF0) >> 4 ) * 4 + 1] = 0x03)
# Use that start-of-payload calculation to see if third payload byte is less than 0x04 (SSL/TLS minor version 1-3)
(tcp[((tcp[12] & 0xF0) >> 4 ) * 4 + 2] < 0x04)
# Determine if the TCP payload length is greater than 16.
# NOTE: if this generates too many false positives, this number can be increased
((ip[2:2] - 4 * (ip[0] & 0x0F) - 4 * ((tcp[12] & 0xF0) >> 4) > 30))
-----------------------------
Related "warm ups" - simpler BPF examples that demonstrate parsing the needed fields
-----------------------------
# ip datagram len is 40 bytes
ip[2:2] = 40
# ip header len is 5 32-bit words, or 20 bytes (5 32-bit words, length of 5×32 = 160 bits = 20 bytes, or 5x(4 bytes/word) = 20 bytes)
ip[0] & 0x0F = 5
# tcp header len is 5 32-bit words, or 20 bytes (5 32-bit words, length of 5×32 = 160 bits = 20 bytes, 5x(4 bytes/word) = 20 bytes)
((tcp[12] & 0xF0) >> 4 ) = 5
# now find packets with zero tcp payload len (tcp LEN pseudo field is 0)
# datagram len - 4 * ip header len - 4 * tcp header len
tcp and ( ip[2:2] - 4 * (ip[0] & 0x0F) - 4 * ((tcp[12] & 0xF0) >> 4 ) = 0)
Finally, if interested, here’s a quick pointer to best code-level description I’ve seen so far of the actual bug:
http://blog.existentialize.com/diagnosis-of-the-openssl-heartbleed-bug.html
And here’s an overview of heartbleed in general:
http://arstechnica.com/security/2014/04/critical-crypto-bug-in-openssl-opens-two-thirds-of-the-web-to-eavesdropping/
It is being widely reported in the popular press as well as many technical sites that a Heartbleed exploit "leaves behind no trace".
That of course is not true.
Packets almost always tell a detailed story of what has happened, including in the case of Heartbleed. The reason it's being repeated so frequently that a Heartbleed exploit "leaves behind no trace" is because on the server, a Heartbleed-based exploit almost never leaves behind any evidence. Stored packets, on the other hand, do tell the story of a Heartbleed exploit even after the hacker has stopped an active attack.
In this post, I will describe a technique for on-going monitoring of Heartbleed exploits, but even more importantly, if you have a sufficiently large rolling buffer of packet data, many people will be able to use this technique to reach back in time to prior to the public disclosure of Heartbleed to check if an actual exploit seems to have occurred prior to patching vulnerable servers.
This technique uses a BPF packet filter to automatically flag larger-than-typical TLS heartbeat responses from the server, and can be used with Wireshark and tcpdump as well as with the Riverbed AppResponse, Shark, and Pilot products. (AppResponse and Shark support many terabytes of stored packets, coupled with the ability to quickly analyze those packets; more Riverbed product specific hints will follow in a separate blog post).
In addition, for the majority of published Heartbleed exploits so far (which are moving the compromised data in the clear on the wire), this technique also identifies what exact data was compromised (e.g., the set of user passwords exposed, vs. the "crown jewels" of a server's private keys, vs. ____).
[ADDED 04-15-2014] One question currently being asked is: have Heartbleed exploits been going on for months or even years? To help answer this question, I just added a Heartbleed detection script that searches a hard drive (on a desktop, laptop, or file server, or ____) for packet capture files to see if it can find an old Heartbleed exploit sitting in an old packet capture. This new script is also useful for examining recent packet captures. It builds on the BPF filter described below, and improves upon it by looking more statefully at heartbeat requests and responses (with the result being fewer false positives).
[ADDED 04-11-2014] Interestingly, standard Intrusion Detection Systems have been unknowingly struggling to properly identify Heartbleed exploits since the public disclosure. For example, the standard Snort rules for detecting Heartbleed were updated yesterday (Thursday 2014-04-10), after having been broken and missing real exploits for ~2 days, and the widely circulated EmergingThreat Heartbleed rules did not flag many real exploits. (Part of the issue with Snort and EmergingThreat and other IDS rules, and why the incorrect logic was not noticed for so long, was that rules were successfully identifying many "test" exploits that many non-malicious people around the Internet were doing via "am I vulnerable?" on-line checkers or via the many "toy" test exploit scripts that had quickly thrown together after public disclosure of Heartbleed, but those IDS rules were simultaneously missing "real" exploits due to logic flaws. For example, some IDS packages were missing Heartbleed exploits done after the start of encryption, which is more likely what a real attacker would be doing). The live & retrospective packet-based BPF approach described below is not affected by either of the issues that "broke" the Snort & EmergingThreat rules. More than anything, these examples emphasize the importance of using multiple techniques in parallel to take advantage of the different strength and weakness of different approaches.
Speed round on OpenSSL-- This is the software package that secures the majority of the world wide web, in addition to its other uses. SSL (and its successor TLS) encrypt network packets and securely prevent eavesdropping and tampering.
Speed round on Heartbleed itself-- In part because of OpenSSL’s memory allocation optimizations, there ends up being more caching of sensitive data and less crashing when exploited by Heartbleed, and a wide variety of data from a "secure" server can be remotely downloaded by an attacker without any trace left on the server. If not for these memory allocation optimizations, a Heartbleed exploit would have triggered a crash more frequently, and hence the vulnerability very likely would have been noticed and fixed a long time ago by the security community. Up to approximately 64KB of sensitive data can be downloaded per attack by sending a malformed TLS "heartbeat" message (and hence the name "Heartbleed"), but much more sensitive data than 64KB can be downloaded by repeated attacks, especially if the attacks are not noticed. Arguably, there has been no larger known Internet security vulnerability affecting more people than Heartbleed.
Why BPF? BPF engines are fast, but more importantly, BPF is basically a "lingua franca" / common tongue for packet processing, understood by the large majority of operating systems and packet-analyzing software. BPF’s broad applicability is the main reason we put energy into detecting Heartbleed via BPF. BPF is available on Linux, Mac, in the Cloud, etc. Thanks to "The Italian Guys", you can run BPF on Windows too. BPF / tcpdump is also present on most network appliances (firewalls / load balancers / ADCs / etc.), and often accessible by an admin for troubleshooting purposes. And of course, packet analysis & storage engines in the Network Performance Management space almost all support BPF.
This is still an emerging threat, and this BPF-based technique below is a heuristic based approach that in theory can have both false positives and false negatives. However, given the urgency and severity of the threat, Riverbed is sharing this publicly in parallel to Riverbed continuing to internally test and validate this.
Suggested improvements to this BPF filter are welcome.
This BPF filter currently looks at traffic from port 443 (default HTTPS port). That section can be adjusted if HTTPS is running on other ports, as well as for other protocols (e.g., an email server running IMAP on port 143). The size threshold is 69 bytes. This can be adjusted upwards or downwards to reduce false positives or false negatives, if needed. This BPF filter still functions to identify a Heartbleed exploit regardless of whether or not the actual Heartbleed exploit occurs before or after the start of encryption within a given TCP connection (and this BPF filter does not rely on attempting to peer into the portion of the payload that might be encrypted).
--------- Heartbleed BPF expression (IPv4, non-VLAN version) ---------
tcp src port 443 and (tcp[((tcp[12] & 0xF0) >> 4 ) * 4] = 0x18) and (tcp[((tcp[12] & 0xF0) >> 4 ) * 4 + 1] = 0x03) and (tcp[((tcp[12] & 0xF0) >> 4 ) * 4 + 2] < 0x04) and ((ip[2:2] - 4 * (ip[0] & 0x0F) - 4 * ((tcp[12] & 0xF0) >> 4) > 69))
--------- Heartbleed BPF expression (IPv4, VLAN-friendly version) ---------
[ADDED 04-09-2014] In general, handling VLANs in BPF is more complex than it should be. Here is the VLAN-friendly version of the exact same BPF expression as above (where the BPF expression above ends up being replicated twice here-- once for the VLAN case, and once for the non-VLAN case, both of which are then OR'ed together).
((not ether proto 0x8100) and (tcp src port 443 and (tcp[((tcp[12] & 0xF0) >> 4 ) * 4] = 0x18) and (tcp[((tcp[12] & 0xF0) >> 4 ) * 4 + 1] = 0x03) and (tcp[((tcp[12] & 0xF0) >> 4 ) * 4 + 2] < 0x04) and ((ip[2:2] - 4 * (ip[0] & 0x0F) - 4 * ((tcp[12] & 0xF0) >> 4) > 69)))) or (vlan and (tcp src port 443 and (tcp[((tcp[12] & 0xF0) >> 4 ) * 4] = 0x18) and (tcp[((tcp[12] & 0xF0) >> 4 ) * 4 + 1] = 0x03) and (tcp[((tcp[12] & 0xF0) >> 4 ) * 4 + 2] < 0x04) and ((ip[2:2] - 4 * (ip[0] & 0x0F) - 4 * ((tcp[12] & 0xF0) >> 4) > 69))))
--------- Heartbleed BPF expression (IPv6-only, non-VLAN version) ---------
[ADDED 04-11-2014] Added IPv6-specific version. BPF support for IPv6 is unfortunately relatively weak (e.g., doing something like like "tcp[20]" does not work if above IPv6), so this expression enforces that there are no IPv6 optional headers.
tcp src port 443 and ip6 and ip6[6]=6 and (ip6[40 + ((ip6[40+12] & 0xF0) >> 4) * 4 + 0] = 0x18) and (ip6[40 + ((ip6[40+12] & 0xF0) >> 4) * 4 + 1] = 0x03) and (ip6[40 + ((ip6[40+12] & 0xF0) >> 4) * 4 + 2] < 0x04) and ((ip6[4:2] - 4*( (ip6[40+12] & 0xF0) >> 4) ) > 69)
--------- Heartbleed BPF expression (Combined IPv4 and IPv6, non-VLAN version) ---------
[ADDED 04-11-2014] Added version that works for mixed IPv4/IPv6 traffic. This expression also enforces that there are no IPv6 optional headers.
tcp src port 443 and (((ip and tcp[((tcp[12] & 0xF0) >> 4 ) * 4] = 0x18) and (tcp[((tcp[12] & 0xF0) >> 4 ) * 4 + 1] = 0x03) and (tcp[((tcp[12] & 0xF0) >> 4 ) * 4 + 2] < 0x04) and ((ip[2:2] - 4 * (ip[0] & 0x0F) - 4 * ((tcp[12] & 0xF0) >> 4) > 69)) ) or ( (ip6 and ip6[6]=6 and (ip6[40 + ((ip6[40+12] & 0xF0) >> 4) * 4 + 0] = 0x18) and (ip6[40 + ((ip6[40+12] & 0xF0) >> 4) * 4 + 1] = 0x03) and (ip6[40 + ((ip6[40+12] & 0xF0) >> 4) * 4 + 2] < 0x04) and ((ip6[4:2] - 4*( (ip6[40+12] & 0xF0) >> 4) ) > 69))))
For clarity purposes, as well as to increase the ability of people to comment and suggest improvements, I will now breakdown the various sub-expressions of this specific BPF expression. Also, for purposes of clarity and in part because this BPF filter is slightly trickier than a typical BPF filter (including using BPF to offset past variable length headers such as TCP options), I also included further down several smaller BPF filters that were effectively used as "unit tests" in building up the full expression.
-------- Breaking Down the Heartbleed BPF expression (IPv4, non-VLAN version) ----------
# A response from server on port 443. This can be modified if server not using 443.
tcp src port 443
# This calculates the start of payload data beyond TCP.
tcp[((tcp[12] & 0xF0) >> 4 ) * 4]
# Use that start-of-payload calculation to see if first payload byte is 0x18 (TLS Heartbeat message)
(tcp[((tcp[12] & 0xF0) >> 4 ) * 4] = 0x18)
# Use that start-of-payload calculation to see if second payload byte is 0x03 (TLS major version 3)
(tcp[((tcp[12] & 0xF0) >> 4 ) * 4 + 1] = 0x03)
# Use that start-of-payload calculation to see if third payload byte is less than 0x04 (TLS minor version 0-3)
(tcp[((tcp[12] & 0xF0) >> 4 ) * 4 + 2] < 0x04)
# Determine if the TCP payload length is greater than 69. NOTE: if this generates too many false positives, this number can be increased
((ip[2:2] - 4 * (ip[0] & 0x0F) - 4 * ((tcp[12] & 0xF0) >> 4) > 69))
----------Related BPF "warm ups" for IPv4 ----------
These are simpler test BPF examples that demonstrate parsing the related fields. Variations of these expressions are used in the Heartbleed BPF expression.
# test BPF expression: checks if ip datagram len is equal 40 bytes
ip[2:2] = 40
# test BPF expression: checks if ip header len is 5 32-bit words, or 20 bytes
ip[0] & 0x0F = 5
# test BPF expression: checks if tcp header len is 5 32-bit words, or 20 bytes
((tcp[12] & 0xF0) >> 4 ) = 5
# test BPF expression: checks if tcp payload len is zero (tcp LEN pseudo field is 0)
# datagram len - 4 * ip header len - 4 * tcp header len
tcp and ( ip[2:2] - 4 * (ip[0] & 0x0F) - 4 * ((tcp[12] & 0xF0) >> 4 ) = 0)
----------Related BPF "warm ups" for IPv6 ----------
# test BPF expression: checks if source port is 443
tcp src port 443
# test BPF expression: checks if IPv6, and next header is TCP (6), meaning no optional IPv6 extension headers
ip6 and ip6[6]=6
# test BPF expression: checks if IPv6, and TCP header len (which is selected from start of IPv6) is
# 32 bytes (and that there are no IPv6 optional extension headers)
ip6 and ip6[6]=6 and (4*((ip6[40+12] & 0xF0) >> 4) = 32)
# test BPF expression: checks if IPv6, and no optional IPv6 optional headers, and
# IPv6 payload size - TCP header size = TCP LEN pseudo field = 165
ip6 and ip6[6]=6 and ((ip6[4:2] - 4*((ip6[40+12] & 0xF0) >> 4))=165)
---------- End of BPF Expressions ----------
Some related links:
Example use on Wednesday (04-09-2014) at wireshark.org to process 100s of GB of traffic (from time before public exposure to after patching vulnerable servers on wireshark.org):
https://blog.wireshark.org/2014/04/heartbleed-traffic/
My colleague Chris White wrote a nice script that automates using this BPF expression to detect Heartbleed using Wireshark, tcpdump, or a Shark appliance:
https://splash.riverbed.com/docs/DOC-4083
Here’s an overview of Heartbleed in general:
http://arstechnica.com/security/2014/04/critical-crypto-bug-in-openssl-opens-two-thirds-of-the-web-to-eavesdropping/
Here’s a pointer to best code-level description I’ve seen so far of the actual Heartbleed bug:
http://blog.existentialize.com/diagnosis-of-the-openssl-heartbleed-bug.html
If you are looking for how Heartbleed (CVE-2014-0160) does or does not affect Riverbed products, that has already been separately covered by Riverbed Support here:
https://supportkb.riverbed.com/support/index?page=content&id=S23635
Finally, Riverbed Pilot is also a very good way to look at this data, including using the BPF expression described above. You can freely download Pilot and be up and running very easily.
[UPDATED 04-09-2014] increased default size threshold above to greater than 69 bytes based on additional experimentation with default client OpenSSL heartbeats.
2 Responses
P. J. Malloy
Apr 11, 2014
See update above that adds support for IPv6
Christopher Maynard
Apr 10, 2014
Forgive me if this is a dumb question, but would an equivalent ip6 filter also be needed/useful?