Heartbleed, a long-undiscovered bug in cryptographic software called OpenSSL that secures Web communications, may have left roughly two-thirds of the Web vulnerable to eavesdropping for the past two years. Heartbleed isn’t your garden-variety vulnerability, so here’s a quick guide to what it is, why it’s so serious, and what you can do to keep your data safe.
Serious.
This bug is patched in Debian, RHEL, and CentOS.
Also on Ubuntu. There were patches for this already yesterday, so I’m glad it’s fixed fast.
Add Fedora to the list.
(You may need to download the RPMs directly from dl.fedoraproject.org or wait for your local mirror to sync).
– Gilboa
It’s available here: http://arstechnica.com/security/2014/04/critical-crypto-bug-exposes…
At first I thought Heartbleed or CVE-2014-0160 was just about being able to decrypt SSL messages without needing a man in the middle on the session but that’s not the case.
It allows an attacker to return data from the servers memory that could be ssl keys or data like user/passwords of users communicating with the server.
Most importantly: It does not require being able to intercept a users traffic to obtain their login credentials.
I could exploit an unpatched server and it will freely offer up user/pass combos of people communicating with it. It is a much much more serious vulnerability than simply allowing me to decrypt traffic I had to capture on the wire.
Euh… well, it’s a bit more complicated than that.
You can’t tell the server what data you want.
The library on the server copies a part of memory of the same process and sends it to the client.
So you can’t predict what you get and you might not even know what you got. It’s just some a random block of data.
Now let’s take some examples, a PHP-website. On the server we have 3 types of installations if they use nginx or Apache.
nginx or Apache with FastCGI like PHP-FPM would mean that all the data of the application isn’t in the server process. So it should be pretty safe.
The webserver process might contain small chunks that are being send to users (which could contain a session-IDs) if you are using nginx or an Apache threaded/worker model (no prefork).
In the case of Apache with prefork and maybe mod_php, probably the only data the process has is for your own connection (read: boring information).
So if you look at these examples username/password isn’t very likely. Session-ID maybe, SSL-private key maybe. Maybe even just a part of it and would you be able to recognize it ? As a key is just a blob of random data. Most likely just some random boring memory.
Should server operators patch: Yes, should they replace SSL certificate keys ? Yes.
Did bad things happen ? We don’t know.
A couple of people published that they were able to successfully fetch usernames and passwords from yahoo mail. There is a screenshot of one of the examples in the link I posted.
Because login/password are known fields you would be able to query the server in bulk and grep through the data for those strings knowing the data after it would be the actual credentials.
Here is an example of what I mean: http://i.imgur.com/GL2J8O8.png
You could build a database of user/pass combos with some fairly simple shell scripts.
Edited 2014-04-09 20:38 UTC
I got aware of that a little later after I commented, I even mentioned it in a later comment how it is done:
http://www.osnews.com/thread?586753
Purely by coincidence I even know the researcher personally that found out about it, we worked at the same company many years ago before he was founded the security company.
Edited 2014-04-09 21:00 UTC
I’m guessing this is going to be a pain as patching will not be enough on it’s own. If the ssl keys have been compromised they will need to be regenerated and you don’t know if your system has been compromised as it is not in the logs – you will need to regenerate the ssl keys.
Generating new key is not the problem. blacklisting the existing key is not the problem. Distributing the blacklists and making sure that EVERY application/appliance consults the blacklist before trusting the other party is the problem. There can be millions of keys that must be blacklisted. Perhaps it would be easier to blacklist the intermediate CA keys? But that would require all the previously generated keys to be regenerated. This is massive issue and many appliances use offline blacklists and might not have (easy) means to update the blacklist or might not have enough storage available to hold a list of two thirds of internet keys that have been blacklisted.
Also I’ve seen the programs where the CRL is not consulted due to performance requirements.
Therefore the man-in-the-middle attack has become every day reality where the stolen keys are used to create fake sites.
Edited 2014-04-09 10:54 UTC
wigry,
Unfortunately it is a big problem. Public key encryption can scale extremely well due to the fact that no communications are necessary to validate a certificate. This increases not only certificate validation speed but reliability. However the revocation protocol (OCSP) lacks this scalability, and due to the slow/unreliable nature of it many browsers intentionally ignore non-responses.
http://news.netcraft.com/archives/2013/04/16/certificate-revocation…
Something I learned is that many browsers change their revocation checking behavior depending on whether certs are EV certs or not. Some browsers don’t even bother checking revocation on non-EV certs.
http://blog.spiderlabs.com/2011/04/certificate-revocation-behavior-…
Closing this hole completely means each and every SSL certificate needs to be untrusted until completing validation against the CA’s revocation database. It’s doable but mostly negates the benefits of using PKI in the first place. Assuming every certificate has to be checked anyways, then logically it makes more sense for the client to ask the CA if the certificate is in the “good list” rather than whether it’s in the “bad list”.
There’s a poorly supported hack in place today called “OCSP stapling” that allows the negative revokation status information to be cached by the web server to reduce the need for clients to individually lookup certificate revocation, which should dramatically lesson the bottleneck on a CA’s servers. However at this point I’d argue we have a system of hacks built on top of hacks, and we’d be better off ditching the revocation infrastructure all together and just using short lived certificates. Not only is this approach simpler, but it would be more secure too.
Ultimately I will be glad if/when DNSSEC reaches a point where we will no longer have to deal with CAs at all.
Edited 2014-04-09 14:51 UTC
My own web servers were affected because they used OpenSSL. Just goes to show why monoculture is bad.
If something powers 66% of the web, that’s too much risk. Independent implementations are important part of diversity, however in the security community that tends to be discouraged.
A small part of that 66% of all webservers. Is actually Apache servers with GNU TLS instead of OpenSSL, if that makes you feel better. 😉
An even bigger portion of Apache servers don’t use SSL at all…
Plenty of non-apache webservers also use openssl…
Lots of people are running old versions which date from before this bug was introduced, and thus were never vulnerable.
This is an issue with openssl rather than apache, and apache itself is quite diverse – many different versions running on many different platforms with many different configurations. It’s not ideal but it could be a lot worse.
Let’s say that more than 50% of all HTTPS-websites run on nginx or Apache and probably 99% of those use OpenSSL.
I picked lower than 66%, because many have loadbalancers in front and the share for Apache/nginx is lower for HTTPS than in general.
One article pointed to SSL Pulse which says deployment of TLS 1.2 currently stands at about 30%. Older versions of the library don’t support TLS 1.2
So it’s 30% of 50% is: very, very roughly more than 15% of all HTTPS sites in total were vulnerable.
Obviously it isn’t all that simple:
For example in the case of login.yahoo.com using a loadbalancer actually makes it worse.
They had a loadbalancer which uses OpenSSL which was single process and gave out username/password in HTTP POST-data of people logging in.
Doing everything centralized in a single process in this case turns out to be really bad.
Edited 2014-04-09 10:28 UTC
PL/I and Mesa had bounds checking, but the world decided to go C instead, so keep having fun.
http://blog.existentialize.com/diagnosis-of-the-openssl-heartbleed-…
Borland Pascal did also… a lot of non-C compilers do.
Sure you are right.
I just wanted to mention a programming language older than C and another one of the same age to state that the C language designers ought to know better.
Perhaps, they went with “fast” and “code correctness”? The problem is, on a very complex scenario, not having bounds checking intrinsic to the language proved to be a bad choice for general development.
It’s more that the OpenSSL team made some terminally stupid decisions:
http://thread.gmane.org/gmane.os.openbsd.misc/211952/focus=211963
Indeed, I was about to post the same link. Some systems have these checks in place but when people actively work around them…
If someone really wants to shoot himself in the foot, there’s only so much you can do to stop him. Unfortunately they shot in pretty much everyone’s foot with this one. Hopefully this will cause them to reconsider some practices.
cfgr,
Conceptually sure, but in this particular case it wouldn’t have helped. The vulnerability didn’t corrupt memory such that free could have detected it, it simply wrote memory out to a socket.
clint might have detected it though.
If your goal is to discourage coders from shooting their own feet, then C is probably not the best choice as a language. Not many languages are as notorious as C for things like buffer/stack overruns.
Why not? For example, the OpenBSD malloc (with the G option enabled) may place guards around each allocated block which would prevent this exploit from reading outside the allocated memory within the same application as it would trigger one of the guards and cause a segfault.
The problem here is that rather than using the system malloc/free, it keeps its memory cached to reuse it. Hence why the guards are ineffective.
Edited 2014-04-10 14:07 UTC
cfgr,
Yes, you could use the mmu for mallocs instead of using a local/caching allocator, but MMUs only work on page sizes (ie 4k and up). That results in lots of wasted memory and probably poor cache utilization too. This is fine for debugging, but on a production system…the negatives are a show stopper, IMHO. Can you name any performance critical production systems that enable this by default?
In any case, if it’s important to you, then it already looks like you can generate a version of openssl with the malloc cache disabled.
The reason I don’t think it would have helped is because it would not have come up on any testing/debugging build. The normal test cases wouldn’t have triggered the conditions. The best shot they’d have to catch bugs like this might be with an input fuzzer (in conjunction with multiple safety checks like malloc page guards). I suspect moving forward this is exactly what they’ll do, which they arguably should have been doing all along.
The impact of malloc guarding on the performance is minimal (according to the OpenBSD developers*), however, there’s a lot of software that misbehaves and would crash, which is why it’s disabled by default.
The post on the mailing list mentions that it doesn’t seem to be that easy. Also, here’s a blog post of his, it seems to be worse than previously thought:
http://www.tedunangst.com/flak/post/analysis-of-openssl-freelist-re…
Mayhaps, but an attack would have crashed the service at once, rather than quietly leaking keys for two years.
[*] http://download.cdn.yandex.net.cache-default05d.cdn.yandex.net/comp…
Edited 2014-04-10 18:50 UTC
cfgr,
Ah, yes that would be another bug then.
I don’t know much about OpenBSD, so I will try reading more on that (your link is interesting!). On linux, freelists do improve on the performance of the standard GCC malloc even without guard pages. It’s particularly slow in multithreaded processes, this was maybe in 2012-2013, I should probably test it again.
However it’s not just about performance, it’s also granularity. Most allocated objects might range from a few dozen bytes to a few hundred, adding a guard page per each object would require one page for data, plus one page to trigger the fault. So a 100byte structure would still require 8K of ram. This is way too inefficient for a production system, IMHO.
https://developer.apple.com/library/mac/documentation/Darwin/Referen…
Lets say openssl did ship with guard pages by default. The person developing an exploit would have discovered that requests>4K trigger the guard page fault and tuned the exploit accordingly so as to not trigger the fault. I’ll concede that <4K reads are less useful than 64K ones, but it might still be done without raising a red flag.
Edited 2014-04-10 21:35 UTC
Soulbender,
Having profiled the standard malloc myself on linux+gcc, I actually believe both points are correct. There are very real performance gains to be had by abstracting the allocator behind a caching mechanism, especially in thread local storage.
For a project like OpenSSL, which is both performance sensitive and security sensitive, there can be divergent goals. Optimizing malloc wouldn’t ordinarily cause bugs on it’s own, however if it eliminated some of the sanity checks, it might allow some alloc/free/memory leak bugs elsewhere in code to go unnoticed and keep running instead of crashing.
In this case the ideal solution seems to be to compile these optimizations conditionally, and run both versions through a comprehensive barrage of unit test cases. I don’t know much about OpenSSL’s testing procedures, but the source code does reveal that the caching can be enabled/disabled conditionally by defining “OPENSSL_NO_BUF_FREELISTS”, so it’s possible OpenSSL is already doing the right thing.
This reminds me of the range checking directive in pascal programs {$R+} {$R-}. The idea was a good one, if you develop your program successfully with range checking turned on and it’s working, then that increases the confidence the code will be correct even with range checking turned off. The main caveat is that you don’t get any assurances for code paths that were not adequately tested when range checking was on.
Edited 2014-04-09 18:22 UTC
If your project is OpenSSL your first and most important priority is security. If some obscure platform have a shitty malloc you selectively fix it only there, you don’t make a nasty hack solution and apply it to all platforms.
No, you can’t disable it because openssl won’t work then.
It doesn’t exactly flatter the OpenSSL team that apparently no-one gave a shit about the bug report filed against this.
Edited 2014-04-11 04:33 UTC
Soulbender,
I don’t think it’s such a hack in the first place, but regardless of that the point is kind of mute in terms of security because a) malloc guard pages are not used on production servers, and b) a variant of the exploit might still be feasible with the malloc guard enabled anyways.
Quoting the openbsd researchers from cfgr’s link earlier: “malloc guarding (world is not ready for this)”. The reasoning can be extrapolated from the rest of the paper: “Don’t want to break normal/expected behaviors. But maybe change anything else which makes exploits hard/impossible. As long as the performance cost is insignificant/very low…”.
Well, it didn’t work because there was another bug (which cfgr also pointed out), however in principal there’s no reason it won’t work. I think it’s a good idea to test with guard pages, and clearly OpenSSL was not tested this way. I will learn from this and make changes to my own testing procedures. However I still think it is a stretch to suggest this particular vulnerability would have been caught if not for freelists, but of course that’s just my opinion.
Edited 2014-04-11 06:05 UTC
Closed and Open Source SSL libraries have all had bugs of different levels of disaster.
We have acid tests for html. There is no vendor neutral tests for SSL.
Remember with the GNUTLS issue it was like go use openssl. Reality we do need more than 1 SSL library/solution. Bugs will come.
http://arstechnica.com/information-technology/2010/02/microsoft-war… Yes those anti-open source will forget this.
At some point we have to get serous about secuirty.
1) If there is not a validation suite it cannot be secure.
Maybe because it was a fairly forgettable issue that didn’t have anywhere near the same impact as the OpenSSL flaw?
Not to mention being a flaw in the TLS/SSL protocol itself, rather than a flaw specific to Microsoft’s implementation:
http://lwn.net/Articles/362234/
Edited 2014-04-10 02:46 UTC
http://technet.microsoft.com/en-us/security/bulletin/MS10-049
BallmerKnowsBest read your own link. It details a flaw that does not come from protocol alone. No matter how badly you implement SChannel you should not end up with means to do remote code execution. If you can execute code you can request particular pages of memory. Microsoft had buffer overflow into executable space if you left protocol back in 2010. Should not have been possible if NX extensions and memory allocation in cpu had been used correctly.
MS10-049 is in fact for means to collect data worse than the current OpenSSL flaw. Current openssl flaw you get random blocks of data not exact data requests. The SChannel flaw in windows back in 2010 also worked in reverse from client to server. Luckily there are not many MS Windows servers on the internet.
Yes on scale of SSL screwups in implementations the current OpenSSL is not the worst that has happened. Heck even the Microsoft SChannel flaw is not the worst.
Yes you are right its was a flaw in the TLS/SSL protocol but Microsoft implementation managed to handle it worse than everyone else. Insanely worse.
The problem here is every SSL implementation closed or open has been having major issues. Still there is no unified test suite to confirm that a implementation is to standard. You will have some idiots come out and say its because of lack of professional management… Reality its more lack of conformance testing. Like most web browsers should not be SSL conforming because they don’t check if SSL certificates are revoked or not.
Reason why SSL works is more good luck than management.
Are we going to wake up now and demard good management. I don’t think would be a good idea to attempt to hold by breath waiting.
Try following some of the links from that page, which would quickly lead you to this:
http://support.microsoft.com/kb/977377
(emphasis: mine)
Or you could read the 2nd link I posted, which detailed how vulnerability to the TLS renegotiation issue was by no means limited to Microsoft’s software.
So yes, the UNDERLYING flaw came from flaws in the protocol itself – the problems with Microsoft’s implementation made the problem worse, but they obviously weren’t the sole/primary cause of the problem.
Yeahhhhhhh… [citation needed]. I’m guessing you missed this minor detail in the MS10-049 summary:
As opposed to the current OpenSSL flaw, where your login credentials can be exposed simply by signing in to a vulnerable server.
And was there any evidence that the SChannel flaw was ever actually exploited “in the wild”? Because I’ve already seen some fairly dramatic demonstrations of Heartbleed – E.g. comments in an ArsTechnica thread from someone who was able to retrieve another user’s credentials and then login & comment from their account:
http://arstechnica.com/security/2014/04/critical-crypto-bug-in-open…
(the linked comment and the one directly after it are the relevant ones)
Oh, well I’m sure that be an immense comfort to any people whose online accounts are compromised because of this flaw…
Hahahaha, yeah, only about… say, a third of all public-facing servers on the internet. Oh, and remind me: what was the estimate as to number of the servers on the internet that were vulnerable to Heartbleed? Something like two-thirds, wasn’t it…?
To the extent that I can glean a point from your post, it seems like an attempt to deflect criticisms that no one has actually made. Before it, I haven’t seen anyone try to ascribe the problem to the open source model as a whole (or anyone other than the OpenSSL developers directly responsible for it). Nor did I realize that the number & severity of SSL implementation flaws was the subject of a contest between open source and Microsoft/closed source.
If the shoe were on the other foot, I have no doubt that the usual FLOSS ideologues would immediately pounce on it as “proof” of the inferiority of Microsoft’s software – if not the closed-source development model as a whole. Yet you usually don’t see people making the same lazy conclusions in reverse, no matter how many times a lax approach to security in open source software development enables things like Darkleech or Heartbleed… not to mention the constant stream of recent problems caused by beginner-level security failures in open source web applications like WordPress, Joomla, Drupal, or Mailman (AKA “the backscatter spammer’s best friend”).
BUT, since you brought it up: if you really want to keep score, then it’s pretty clear that Heartbleed has vaulted open source into the “lead” as far as SSL-related security failures go. As you alluded to earlier, Linux is more widely-used as a server OS – and now Linux & the open source community at large are starting to experience the inevitable downsides of population density/network effects (WRT security & malware) that Windows has been subjected to on the desktop for years.
BallmerKnowsBest if you download metasploit you will find a demo of reverse back to webserver using MS10-049.
With MS10-049 we were lucky. Heartbleed not lucky. This is the thing with these flaws some we are lucky some we are not.
Interesting enough lots of Windows servers are behind Linux load balancers and filters. Some sites using Windows were got by Heartbleed because their Linux Balancer got hit yet those same Linux load balancers blocked other attacks. Why the load balancer is doing the ssl decode. Its way less than 1/3 of servers on the Internet that have Windows servers with internet facing ssl. You are looking at basically 90 percent Linux when you look at what is decoding ssl.
http://support.microsoft.com/kb/977377 and MS10-049 are two exploits. The fix for MS10-049 also contains the fix for 977377.
Over all SSL falures have been broad spread.
There have been security failures in closed source frameworks to use instead of open source solutions like WordPress, Joomla, Drupal, or Mailman.
http://www.cvedetails.com/vulnerability-list/vendor_id-26/product_i…
https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=wordpress
Over all its simpler to get executable code into a Windows server to alter its function.
There are constant stream of bugs in closed and open source. Yes beginner level bugs are turning up in Microsoft products as well. I can pull in other.
Even this recent openssl bug on Linux still was not remote run what ever you like.
population density/network effects makes a weakness worse. But if Linux was having the same flaws as windows a lot we would be looking at disasters that make the recent openssl flaw look minor.