Like many online services, we at Telerivet spent much of last week responding to the OpenSSL Heartbleed vulnerability (see our post-mortem here).
While the lessons of Heartbleed are still fresh in our minds, now is the time for the software community to consider how we can improve our collective security and prepare for the next zero-day vulnerability. Whether the next vulnerability is in an open-source library, operating system, firmware, one’s own software, or a third-party service, many of the same lessons apply.
Our top 4 recommendations for online services to limit the fallout from the next vulnerability like Heartbleed:
1. Improve your SSL configuration
Heartbleed is the perfect advertisement for Perfect Forward Secrecy. Thousands of web services all realized this week that their SSL private keys may have been compromised. Those without PFS enabled also realized that someone who had been recording encrypted sessions could have used their private keys to decrypt up to two years worth of previous traffic to their servers.
SSL configuration is hard to get right. Qualys SSL Labs makes it easy to test your server’s SSL setup and make sure it’s both secure and compatible with all the browsers out there. Try running their SSL server test on your own server, and follow their suggestions to improve your SSL config until you get an A or A+.
2. Subscribe to security announcement lists
Many of us were lucky to learn about Heartbleed very quickly on Monday when it quickly reached #1 on Hacker News. But system administrators shouldn’t rely on visiting the right website at the right time in order to react quickly to major vulnerabilities.
Fortunately, most major projects have an email list to alert users of security vulnerabilities.
Since it can be impractical to subscribe to announcement lists for every library and software package you depend on (like openssl-announce, where Heartbleed was first publicly announced), joining the security announcement list for your server operating system would be good enough for almost everyone. Here are a few:
When the next big zero-day vulnerability happens, don’t leave it up to chance whether you find out about it immediately or after it's already too late.
3. Limit your exposure when using third-party services
Vulnerabilities in any of these third-party services could compromise customer data just as easily as vulnerabilities in the main service itself. Yet most services’ Heartbleed post-mortems haven't discussed the risk of vulnerabilities in the third party services they use.
In the case of Heartbleed, all of the services above were affected for at least some time before being patched. It’s hard to say with certainty that nobody used Heartbleed to extract user data from any of these services. And now that we know for sure that attackers can use Heartbleed to steal SSL private keys, it’s hard to say that nobody MITMed them either, though that would be more difficult to do on a large scale.
4. Design your systems with defense in depth
The Heartbleed vulnerability was surprising in some ways — how was such a critical part of the world’s security infrastructure broken for so long with so little review? But it isn’t as surprising as one might think. There is an unmeasurably massive amount of code in the wild that almost everyone online relies on every day. And in any of those obscure lines of code there might be one simple programming mistake that was overlooked in code review, missed by a static analyzer, and deployed into production. It has happened before and it will surely happen again.
As a developer, this means that it’s necessary to design your systems with multiple levels of security to mitigate the damage if one part of your system’s security fails, whether because of a vulnerability in your own code or someone else’s code.
In the case of Heartbleed, helpful mitigation methods would have included:
For session ID exposure: requiring users to re-authenticate when changing IP addresses.
For password exposure: requiring two-factor authentication, or requiring users to approve logins from new IP addresses.
For secret key exposure: enabling Perfect Forward Secrecy so that a MITM attacker couldn't decrypt previous encrypted sessions.
In general, imagine that an attacker was able to get through one level of your security, and think about how you can still prevent them from doing something even worse.
For example, imagine that an attacker somehow gained read-access to your database through a SQL injection vulnerability, or somehow gained access to your database backups. Then it would help if sensitive fields in your database were encrypted with a key stored outside the database.
Or imagine one of your employees’ laptops was stolen. It’d help if any customer data or secret keys on it were encrypted in a TrueCrypt volume.
Or imagine that an attacker already had root access to one of your database servers. Then you're in big trouble, but it would still help if you used Tarsnap with a write-only key so that they couldn’t just delete all your data along with all your backups.
And although “security by obscurity” is a bad foundation for a cryptosystem, obscurity is a perfectly valid additional layer of “swiss cheese” in a security system. Simply making things harder to find — like choosing a non-default port for your SSH servers — can at least buy you valuable time when the next zero-day is being exploited.
Since it is inevitable that another “Heartbleed” will happen again, defense in depth is the best way to make sure your users stay as protected as possible.
The two most secure types of software are the software that never ships and the software that does nothing at all. Unfortunately, neither type is very useful to anyone. In practice, we all have to make tradeoffs between competing priorities like shipping, adding features, and improving security. While the lessons of Heartbleed are still fresh, let’s all at least take a moment to make sure we’re well prepared for the next big zero-day vulnerability.
What do you think the software community should do to prepare for the next Heartbleed?