Due to the difficulty of finding high-quality Heartbleed instructions, I have discovered that perfectly good, intelligent rationalists either didn't do all that was needed and ended up with a false sense of security or did things that increased their risk without realizing it and needed to take some additional steps.  Part of the problem is that organizations who write for end users do not specialize in computer security and vice versa, so many of the Heartbleed instructions for end users had issues.  The issues range from conflicting and confusing information to outright ridiculous hype.  As an IT person and a rationalist, I knew better than to jump to the proposing solutions phase before researching [1].  Recognizing the need for well thought out Heartbleed instructions, I spent 10-15 hours sorting through the chaos to create more comprehensive Heartbleed instructions.  I'm not a security expert, but as an IT person who has read about computer security out of a desire for professional improvement and also due to curiosity and is familiar with various research issues, cognitive biases, logical fallacies, etc, I am not clueless either.  In light of this being a major event that some sources are calling one of the worst security problems ever to happen on the Internet [2], that has been proven to be more than a theoretical risk (Four people hacked the keys to the castle out of Cloudflare's challenge in just one day.) [3], that has been badly exploited (900 Canadian social insurance numbers were leaked today. [4]), and some evidence exists that it may have been used for spying for a long time (EFF found evidence of someone spying on IRC conversations. [5]), I think it's important to share my compilation of Heartbleed instructions just so that a better list of instructions is out there.  More importantly, this disaster is a very rare rationality learning opportunity: reflecting on our behavior and comparing it with what we realize we should have done after becoming more informed may help us see patches of irrationality that could harm us during future disasters.  For that reason, I did some rationality checks on my own behavior by asking myself a set of questions.  I have of course included the questions.

 

Heartbleed Research Challenges this Post Addresses:

  - There are apparent contradictions between sources about which sites were affected by Heartbleed, which sites have updated for Heartbleed, which sites need a password reset, and whether to change your passwords now or wait until the company has updated for Heartbleed.  For instance, Yahoo said Facebook was not vulnerable. [6] LastPass said Facebook was confirmed vulnerable and recommended a password update. [7]

  - Companies are putting out a lot of "fluffspeek"*, which makes it difficult to figure out which of your accounts have been affected, and which companies have updated their software.

  - Most sources *either* specialize in writing for end-users *or* are credible sources on computer security, not both.

  - Different articles have different sets of Heartbleed instructions.  None of the articles I saw contained every instruction.

  - A lot of what's out there is just ridiculous hype. [8]

 

Disclaimer

I am not a security specialist, nor am I certified in any security-related area.  I am an IT person who has randomly read a bunch of security literature over the last 15 years, but there *is* a definite quality difference between an IT person who has read security literature and a professional who is dedicated to security.  I can't give you any guarantees (though I'm not sure it's wise to accept that from the specialists either).  Another problem here is time.  I wanted to act ASAP.  With hackers on the loose, I do not think it wise to invest the time it would take me to create a Gwern style masterpiece.  This isn't exactly slapped together, but I am working within time constraints, so it's not perfect.  If you have something important to protect, or have the money to spend, consult a security specialist.

 

Compilation of Heartbleed Instructions


  Beware fraudulent password reset emails and shiny Heartbleed fixes.

  With all the real password reset emails going around, there are a lot of scam artists out there hoping to sneak in some dupes.  A lot of people get confused.  It doesn't mean you're stupid.  If you clicked a nasty link, or even if you're not sure, call the company's fraud department immediately.  That's why they're there. [9]  Always be careful about anything that seems too good to be true, as the scam artists have also begun to advertise Heartbleed "fixes" as bait.


  If the site hasn't done an update, it's risky to change your password.

  Why: This may increase your risk.  If Heartbleed isn't fixed, any new password you type in could be stolen, and a lot of criminals are probably doing whatever they can to exploit Heartbleed right now since they just found out about it.  "Changing your password before receiving notice about a fixed service may only reveal your new password to an attacker." [10]


  If you use digital password storing, consider whether it is secure.

  Some digital password storing software is way better than others.  I can't recommend one, but be careful which one you choose.  Also, check them for Heartbleed.


  If you already changed your password, and then a site updates or says "change your password" do it again.

  Why change it twice?: If you changed it before the update, you were sending that new password over a connection with a nasty security flaw.  Consider that password "potentially stolen" and make a new one.  "Changing your password before receiving notice about a fixed service may only reveal your new password to an attacker." [10]


  If a company says "no need to change your password" do you really want to believe them?

  There's a perverse incentive for companies to tell you "everything is fine" when in fact it is not fine, because nobody wants to be seen as having bad security on their website.  Also, if someone did steal your password through this bug, it's not traceable to the bug.  Companies could conceivably claim "things are fine" without much accountability.  "Exploitation of this bug leaves no traces of anything abnormal happening to the logs." [11] I do not know whether, in practice, companies respond to similar perverse incentives, or if some unknown thing keeps them in check, but I have observed plenty of companies taking advantage of other perverse incentives.  Health care rescission for instance.  That affected much more important things than data.


  When a site has done a Heartbleed update, *then* change your password.

  That's the time to do it. "Changing your password before receiving notice about a fixed service may only reveal your new password to an attacker." [10]


  Security Questions

  Nothing protected your mother's maiden name or the street you grew up on from Heartbleed any more than your passwords or other data.  A stolen security question can be a much bigger risk than a stolen password, especially if you used the same one on multiple different accounts.  When you change your password, also consider whether you should change your security questions.  Think about changing them to something hard to guess, unique to that account, and remember that you don't have to fill out your security questions with accurate information.  If you filled the questions out in the last two years, there's a risk that they were stolen, too.


  How do I know if a site updated?

 

  Method One:

    Qualys SSL Labs, an Information Security Provider created a free SSL Server Test.  Just plug in the domain name and Qualys will generate a report.  Yes, it checks the certificate, too.  (Very important.)

    Qualys Server Test

 

  Method Two:

    CERT, a major security flaw advisory publisher, listed some (not all!) of the sites that have updated.  If you want a list, you should use CERT's list, not other lists. 

    CERT's List

    Why CERT's list?  Hearing "not vulnerable" on some news website's list does not mean that any independent organization verified that the site was fine, nor that an independent organization even has the ability to verify that the site has been safe for the entire last two years.  If anyone can do that job, it would be CERT, but I am not unaware of tests of their abilities in that regard.  Also, there is no fluffspeek*.


  Method Three:

    Search the site itself for the word "Heartbleed" and read the articles that come up.  If the site had to do a Heartbleed update, change your password.  Here's the quick way to search a whole site in Google (do not add "www"):

    site:websitename.com Heartbleed


  If an important site hasn't updated yet:

  If you have sensitive data stored there, don't log into that site until it's fixed.  If you want to protect it, call them up and try to change your password by phone or lock the account down.  "Stick to reputable websites and services, as those sites are most likely to have addressed the vulnerability right away." [10]


  Check your routers, mobile phones, and other devices.

  Yes, really. [13] [14]


  If you have even the tiniest website:

  Don't think "There's nothing to steal on my website".  Spammers always want to get into your website.  Hackers make software that exploits bugs and can share or sell that software.  If a hacker shares a tool that exploits Heartbleed and your site is vulnerable, spammers will get the tool and could make a huge mess out of everything.  That can get you blacklisted and disrupt email, it can get you removed from Google search engine results, it can disrupt your online advertising ... it can be a mess.

  Get a security expert involved to look for all the places where Heartbleed may have caused a security risk on your site, preferably one who knows about all the different services that your website might be using.  "Services" meaning things like a vendor that you pay so your website can send bulk text messages for two-factor authentication, or a free service that lets users do "social sign on" to log into your site with an external service like Yahoo.  The possibilities for Heartbleed to cause problems on your website, through these kinds of services, is really pretty enormous.  Both paid services and free services could be affected.

  A sysadmin needs to check the server your site is on to figure out if it's got the Heartbleed bug and update it.

  Remember to check your various web providers like domain name registration services, web hosting company, etc.


Rationality Learning Opportunity (The Questions)

We won't get many opportunities to think about how we react in a disaster.  For obvious ethical reasons, we can't exactly create disasters in order to test ourselves.  I am taking the opportunity to reflect on my reactions and am sharing my method for doing this.  Here are some questions I asked myself which are designed to encourage reflection.  I admit to having made two mistakes at first: I did not apply rigorous skepticism to each news source right from the very first article I read, and the mistake of underestimating the full extent of what it would take to address the issue.  What saved me was noticing my confusion.

  When you first heard about Heartbleed, did you fail to react?  (Normalcy bias)

  When you first learned about the risk, what probability did you assign to being affected by it?  What probability do you assign now?  (Optimism bias)

  Were you surprised to find out that someone in your life did not know about Heartbleed, and regret not telling them when it had occurred to you to tell them?  (Bystander effect)

  What did you think it was going to take to address Heartbleed?  Did you underestimate what it would take to address it competently?  (Dunning-Kruger effect)

  After reading news sources on Heartbleed instructions, were you surprised later that some of them were wrong?

  How much time did you think it would take to address the issue?  Did it take longer?  (Planning fallacy)

  Did you ignore Heartbleed?  (Ostrich effect)


*Fluffspeek:

Companies, of course, want to present a respectable face to customers, so most of them are not just coming out and saying "We were affected by Heartbleed.  We have updated.  It's time to change your password now."  Instead, some have been writing fluff like:

  "We see no evidence that data was stolen."

  According to the company that found this bug, Heartbleed doesn't leave a trail in the logs. [15] If someone did steal your password, would there be evidence anyway?  Maybe some really were able to rule that out somehow.  Positivity bias, a type of confirmation bias, is an important possibility here.  Maybe, like many humans, these companies simply failed to "Look into the dark" [16] and think of alternate explanations for the evidence they're seeing (or not seeing, which can sometimes be evidence [17], but not useful evidence in this case).

  "We didn't bother to tell you whether we updated for Heartbleed, but it's always a good idea to change your password however often."

  Unless you know each website has updated for Heartbleed, there's a chance that you're going to go out and send your new passwords right through a bunch of website's Heartbleed security holes as you're changing them.  Now that Heartbleed is big news, every hacker and script kiddie on planet earth probably knows about it, which means there are probably way more people trying to steal passwords through Heartbleed than before.  Which is the greater risk?  Entering in a new password while the site is leaking passwords in a potentially hacker-infested environment, or leaving your potentially stolen password there until the site has updated?  Worse, if people *did not* change their password after the update because they already changed it *before* the update, they've got a false sense of security about the probability that their password was stolen.  Maybe some these companies updated for Heartbleed before saying that.  Maybe the bug was completely non-applicable for them.  Regardless, I think end users deserve to know that updating their password before the Heartbleed update carries a risk.  Users need to be told whether an update has been applied.  As James Lynn wrote for Forbes, "Forcing customers to guess or test themselves is just negligent." [8]

"Fluffspeek" is a play on "leetspeek", a term used to describe bits of text full of numbers and symbols that is attributed to silly "hackers".  Some PR fluff may be a deliberate attempt to exploit others, similar in some ways to the manipulation techniques popular among black hat hackers, called social engineering.  Even when it's not deliberate, this kind of garbage is probably about as ugly to most people with half a brain as "I AM AN 31337 HACKER!!!1", so is still fitting.

 

References:

 1. http://lesswrong.com/lw/ka/hold_off_on_proposing_solutions/

 2. http://money.cnn.com/2014/04/09/technology/security/Heartbleed-bug/

 3. http://blog.cloudflare.com/the-results-of-the-cloudflare-challenge

 4. http://www.cra-arc.gc.ca/gncy/sttmnt2-eng.html

 5. https://www.eff.org/deeplinks/2014/04/wild-heart-were-intelligence-agencies-using-Heartbleed-november-2013

 6. http://finance.yahoo.com/blogs/breakout/Heartbleed-security-flaw--how-to-protect-yourself-172552932.html

 7. https://lastpass.com/Heartbleed/?h=facebook.com

 8. Forbes.com "Avoiding Heartbleed Hype, What To Do To Stay Safe" (I can't link to this for some reason but you can do a search.)

 9. http://www.net-security.org/secworld.php?id=16671

 10. http://www.cnbc.com/id/101569136

 11. http://Heartbleed.com/

 12. https://community.norton.com/t5/Norton-Protection-Blog/Heartbleed-Bug-What-You-Need-to-Know-and-Security-Tips/ba-p/1120128

 13. http://online.wsj.com/news/articles/SB10001424052702303873604579493963847851346

 14. Forbes.com "A Billion Smartphone Users May Be Affected by the Heartbleed Security Flaw" (I can't link to this for some reason but you can do a search.)

 15. http://Heartbleed.com/

 16. http://lesswrong.com/lw/iw/positive_bias_look_into_the_dark/

 17. http://lesswrong.com/lw/ih/absence_of_evidence_is_evidence_of_absence/

New to LessWrong?

New Comment
32 comments, sorted by Click to highlight new comments since: Today at 8:55 AM
[-][anonymous]10y140

Security questions can be answered with lies rather than facts. If might be a fact that I grew up on Elm Street but because that fact might be compromising I can instead lie and say I grew up on p5y77fhssr*:552sfvhdde. More difficult to remember, more difficult to compromise.

Security questions can be answered with lies rather than facts.

Security questions should be answered with lies rather than facts. In most situations "security" questions are a large vulnerability that is often exploited.

[-][anonymous]10y60

I did something like this over the past few years with some Gmail accounts. I've permanently lost access to those accounts because I didn't keep the answers. Just remember to protect yourself from yourself.

How do people remember all their passwords? I write all of mine down. Er, that is, store them in an encrypted file with a long password that isn't written down anywhere. At last count there were about 200 different sets of credentials. All of the passwords are meaningless, vaguely pronounceable strings, so just remembering them all isn't an option.

I have an internal hash algorithm that I use for all of my passwords, and have a set number of base words that I rotate. So an example of what I do would be if one had a list of 10 words that you use for all of your passwords and then use the rot13 hash on them as the actual password input. I basically have endless variations of the same 10 words as long as I change the hash algorithm.

How do people remember all their passwords?

My passwords are divided into two categories -- important (e.g. for a bank account or the main email address) and not important (mostly for a variety of online stores and other places which think that forcing me to register there is a good idea). The important passwords live in a password manager file which itself lives in Dropbox. The unimportant passwords are composed algorithmically, so that when I am looking at a site's login page I can reconstruct what the password for it should be.

I added a note about this to the post.

I have accounts on Google, Amazon, Apple, various financial institutions, work, and a lot of other lower-sensitivity sites, and I have had not a single email from any of them about Heartbleed, including those listed in various places as vulnerable. In fact, I would have written it off as a hoax but for seeing people like Bruce Schneier and Randall Munroe taking it seriously. Has this been other people's experience also?

I have had not a single email from any of them about Heartbleed

This delayed my finding out about it. I discovered later that most of them had posted their notice someplace a little out of the way - on some blog that's not even under their main domain name (Google did this and I think Microsoft may have done it), or in a little "news" box that you only see after signing in to the website. I don't know what immunized me against the instinct to write it off as a hoax. Perhaps it was all of the security literature I've read and my daily experiences that have informed me that security is difficult and that flawed code can easily be written by accident.

This delayed my finding out about it. I discovered later that most of them had posted their notice someplace a little out of the way

This seems like a good strategy (for them). Answer the questions of those who have security concerns without drawing negative attention to yourself among naive customers.

Think about their incentives while remembering that Heartbleed attacks are, generally speaking, invisible and leave no traces.

If it weren't for Munroe, I might not even know about it. I haven't seen any non-internet news stories on it, and the only online news articles I've seen are ones from me specifically looking for information on it.

I did get a mail from my hoster one day after but at that time I already had updated ubuntu (after determining the openssl version and checking for rootkits). I didn't hear from Google or other big sites but I reason that they rely on the news.

This is pretty basic, but if you're compiling a list of instructions, it include that one should not reset passwords by following a link provided in an email, but rather one should type in the URL of the website manually and navigate to the password reset page.

And companies should stop sending password reset emails with links in them. That's just priming people for phishing.

that has been badly exploited (200,000 Canadian social insurance numbers were leaked today [4])

but your link has the number 900, not 200,000. Moreover, my understanding is that numbers like this are a commodity: leaking the number is the least significant part of this process and criminals already have more than they can use, if not all 30 million.

That's been fixed. I was pretty surprised about this mistake. I remember finding that Canadian website link via a news article discussing the impact of Heartbleed. Maybe I copied the number from the news article, but somehow got mixed up while switching between the two pages and ended up grabbing a number that was relevant to some other topic on the news page. Thanks for catching that.

Is Heartbleed really untraceable?

I was sent a PM about this and a really good point was raised there. If EFF claims that networking logs showed enough detail to confirm Heartbleed exploit attempts going on, then why did Codenomicon's heartbleed.com website claim that the bug doesn't show up in logs?

Recognizing that this is outside my specialty, I did not venture into this topic but I have been wondering about this myself. To say that Heartbleed leaves no traces in the logs is a pretty big commitment. This is because there are a lot of different software options that you could potentially use. Within those, there may be settings you can use to turn logs on or off, limit the size of the log files, etc. Also, people can always create their own software which logs whatever they'd like logged. So, to say that it doesn't show up on most logs could be correct, but to say it doesn't show up in any logs would be a hasty generalization.

I think it's possible and maybe even likely that most companies would not have sufficient logs. There are multiple reasons for this:

  • Despite turning logs on, if you have a server and it loses a hard drive every 2-5 years (because they all die eventually), you might leave your logs behind. They're very large and can be undesirable when considered in the context of your back up (depending on the resource limitations that apply to your backup options... and backing up requires processing resources for compression, bandwidth for sending the compressed file as well as the storage itself).

  • Some websites don't have access to their logs since they're on a shared hosting environment. Attempting to get logs out of a shared hosting company is likely to be difficult to impossible.

  • Logs often have so many entries that trying to find a hacking attempt among the legitimate attempts could be extremely difficult. I do realize that in order to make use of the Heartbleed bug, one would probably generate an awful lot of entries on the log. However, on a big server, that could be a drop in an ocean. A hacker is probably going to be thinking about hiding the hacking attempt. In addition to obscuring their IP address, they probably also know better than to be "noisy", in other words, to create a huge number of weird requests of a type that might get noticed. If possible, they might do things with the specific intention of making it hard for you to tell the difference between their illegitimate request and real requests. Finding evidence in the logs could be extremely difficult or even impossible, depending on how well the attempts are camoflaged.

  • I've noticed that in IT, one tiny difference between the data format people are actually using and the data format you're imagining might mean that the data they actually have can't be used for the purpose you're thinking of. For instance, if the logs don't record every single packet or don't retain the input, then the logs may not be useful in this case.

I do not know exactly what is in network log files, or what data format they use, or how many different systems and settings there are, but I would not be surprised if many companies don't have logs that record enough detail for them to be useful for detecting Heartbleed exploit attempts. I am curious about this. Does anyone know about this? Would you please add a screen shot of some relevant logs, especially if they have the right format to detect a Heartbleed attempt, and let me know what type of device and software generated it?

If EFF claims that networking logs showed enough detail to confirm Heartbleed exploit attempts going on, then why did Codenomicon's heartbleed.com website claim that the bug doesn't show up in logs?

Because "logs" is a very generic term. You can set up your logging to record varying amount of information -- you can fully log every packet received, or you can log only errors, or you can do something in between.

If you record every packet received, you will be able to see Heartbleed attacks in your logs. However, for obvious reasons, few people do that and very few people do that on a permanent basis.

[-][anonymous]10y00

If EFF claims that networking logs showed enough detail to confirm Heartbleed exploit attempts going on, then why did Codenomicon's heartbleed.com website claim that the bug doesn't show up in logs?

Most loggers don't record the full contents of every packet that comes through. What Codenomicon means is the heartbleed doesn't show up in logs under the settings typically used. Apparently EFF found some sites that record enough in their logs to notice Heartbleed.

[This comment is no longer endorsed by its author]Reply

Should this learning experience / Heartbleed instructions post be in main or discussions?

[pollid:677]

I wasn't sure that this was on topic enough for main, but Gunnar_Zarncke argued that this post should be in main "because it is totally applicable to practical rationality and real risks".

If a significant number of people vote and the majority wants this to become an article, I'll put it there after fixing whatever errors I'm notified about in the comments, and after requesting or making a polished version of the poll for it that Gunnar_Zarncke started (the one that asks those questions to encourage you to review your reactions to Heartbleed). I really wanted to include such a poll. I just didn't have the time to construct a quality one.

The questions raised cry for a poll:

When you first heard about Heartbleed, did you fail to react? (Normalcy bias)

[pollid:668]

When you first learned about the risk, what probability did you assign to being affected by it? (enter 0.0 if not affected)

[pollid:669]

What probability do you assign now? (Optimism bias) (enter 0.0 if not affected)

[pollid:670]

Were you surprised to find out that someone in your life did not know about Heartbleed, and regret not telling them when it had occurred to you to tell them? (Bystander effect)

[pollid:671]

EDIT: The second option should read "I did tell other people delayed" (poll options cannot be altered later)

What did you think it was going to take to address Heartbleed? Did you underestimate what it would take to address it competently? (Dunning-Kruger effect)

[pollid:672]

After reading news sources on Heartbleed instructions, were you surprised later that some of them were wrong?

[pollid:673]

How much time did you think it would take to address the issue? Enter the time in days, enter 0.0 to see results.

[pollid:674]

Did it take longer? (Planning fallacy) Enter the ''factor'' it took longer than expected (e.g. if it took double as long enter 2.0; to see results enter 1.0).

[pollid:675]

Did you ignore Heartbleed? (Ostrich effect)

[pollid:676]


I would have posted this in Main because it is totally applicable to practical rationality and real risks and it is well written and sourced. That it is somewhat specialized and contains (very good!) technical advice shouldn't matter (this is still much better and more applicable that relaying purely personal experiences which were also OK earlier). But my judgement on this issue cannot be trusted.

For #1, "I reacted immediately" and "I reacted when the urgency became evident" are probably the same thing for most people. I heard about the bug 20 minutes after it was announced, from the Cloudflare blog of all places. Not even USN had posted about it. I patched my servers within an hour, and spent the next 5 hours waiting for my CA to respond to my revocation and re-key requests. Apparently they were inundated.

On the bright side, I prepared for security issues like this. I used multi-factor auth for our admin tools and perfect forward secrecy cipher suites for our TLS. Even with our private key, previously recorded traffic cannot be decrypted. And if an attacker got ahold of our passwords, they would still need to steal our YubiKeys to get access to our admin tools.

Hooray for being paranoid about security.

Note that a sysadmin might e.g. react immediately to patch their company's servers, revoke keys, etc., but be far more lax about changing their own passwords.

Meanwhile, and possibly distorting the poll, I still have not reacted despite wanting to do so because my internet connection is currently a wet piece of string (slow GPRS). Perhaps there should be an option for "I couldn't react"?

Not offered answer 1: I acted immediately by not logging in to anything, but not really doing anything else Not offered answer 2: I thought I told someone, but she didn't remember later so maybe I didn't?

How long questions: The quickest folks would have it taken care of in hours, the slowest in years. When is it really taken care of?

Well done, but your link to the list of updated websites is broken.

Thanks. That's fixed. I also checked all the other links in the post for 404 errors and fixed two others.

My bank's web server got an "F" on the Qualys SSL Labs test because it supports SSL v2. Should I be worried?

Probably, but not about Heartbleed.

Apparently SSL v2 is vulnerable to man-in-the-middle attacks, but it's only used by obsolete browsers (IE v6) and it can't leak your data if you don't use it. So this particular vulnerability is not one I personally need to worry about, but secure servers are supposed to have SSL v2 disabled - and if it isn't, it generally means that the people who set up the server either don't know what they're doing or don't care.

I think Eugine_Nier means to imply (and I would agree) that anybody using SSL2 is incompetent when it comes to security.

If you have a significant amount of money in your account, I recommend asking your bank about multi-factor authentication. I had to pay a small fee for it, but Wells Fargo gave me an RSA token for my accounts. Its use is required when transferring funds to other banks. So even if my password is stolen, my money is safe. Silicon Valley Bank has a similar scheme using SMS authentication.

Update: The bank uses a separate domain name for "online banking" services. That one got an A-.